View on GitHub

FOnline Engine

Flexible cross-platform isometric game engine

Testing

Engine-owned documentation. This page maps the current engine test executable, generated test targets, coverage targets, and every Source/Tests/Test_*.cpp suite currently present in the checkout.

Purpose

Use this page when choosing validation for an engine change or when adding/removing tests. The source-tree README at ../Source/Tests/README.md is a short entry point; this page is the maintained full test map.

Source paths inspected

Test runner model

Source/Applications/TestingApp.cpp is the test application entry point. It requires FO_TESTING_APP, initializes the application layer with InitApp(-1, nullptr), marks IsTestingInProgress, and delegates execution to Catch::Session().run(argc, argv).

BuildTools/cmake/stages/EngineSources.cmake owns FO_TESTS_SOURCE, the explicit list of test source files compiled into test builds. BuildTools/cmake/stages/Applications.cmake builds test executables through SetupTestBuild(name):

BuildTools/check_windows7_imports.py <binary> [...] is a standalone PE-level regression check for Windows 7 artifacts. It rejects the reported CreateFile2 import; embedding-project CI should run it after linking and before packaging.

For an embedding project with dev name LF, the standard generated names are LF_UnitTests, RunUnitTests, LF_CodeCoverage, RunCodeCoverage, GenerateCodeCoverageReport, and AnalyzeCodeCoverage. Treat the prefix as project-generated, not universal.

Running tests

Client script probes can deliver lifecycle notifications through Game.SimulateDisconnect(), Game.SimulateConnectingFailed() and Game.SimulateInfoMessage(infoMessage, extraText). These APIs invoke the native subscriber chains without changing the transport, so a probe can observe notification handling and still report over its existing connection. Use an actual connection to validate transport behavior.

Preferred local baseline from a configured build:

cmake --build . --config RelWithDebInfo --target RunUnitTests

With FO_EFFEKSEER_PARTICLES enabled, the focused [particle] Catch2 cases invoke the published helper through the production ParticleBaker path. They cover text compilation, dependency invalidation, malformed XML, and rejection of cooked files presented as authored inputs.

The executable target can also be invoked directly when you need Catch2 arguments. In Last Frontier-style layouts, test binaries are emitted under Binaries/Tests-*, for example Binaries/Tests-Windows-win64/LF_UnitTests.exe or Binaries/Tests-Linux-x64/LF_UnitTests.

With Visual Studio/MSBuild generators, RunUnitTests writes the test process output to <build-dir>/<ProjectDevName>_UnitTests.log and uses the test process exit code as the pass/fail signal. This keeps expected negative-case diagnostics such as compiler error lines from being reclassified as MSBuild errors. When the run fails, the helper also echoes the captured output before failing, so a failure is diagnosable from the build output alone — on CI the log file never leaves the runner, and the exit code by itself does not say which test or assertion broke. The generated RunUnitTests target captures the complete test process output under the configured build tree’s Testing/ directory and prints the Catch2 success summary. On a real non-zero process exit it replays the captured output before failing. This keeps expected diagnostics from negative compiler/parser tests from being reclassified as build errors by native build frontends such as MSBuild.

The validate workflow also runs a standalone windows-file-io job on a hosted Windows runner. CMake discovers Visual Studio and builds the diagnostic with both static and dynamic CRTs; this job has no engine or game build dependency. Its windows-file-io artifact retains the factual JSON, compiler logs, executables and available embedded manifests even when a probe fails. See the filesystem diagnostic contract.

For broad validation scenarios, the BuildTools validators can run selected scenarios:

Engine/BuildTools/validate.sh unit-tests
Engine/BuildTools/validate.sh android-arm64-client linux-client linux-server

Use the smallest focused tests first, then the broader run target when the change crosses subsystem boundaries.

The validation project (Engine/BuildTools/validation-project) defaults to FO_ANGELSCRIPT_SCRIPTING with FO_MANAGED_SCRIPTING off. Ordinary validators retain these defaults and avoid the heavy Mono source build. The explicit managed-mac-client, managed-ios-simulator-client and managed-ios-device-client scenarios instead build the same engine-owned scaffold with managed scripting enabled and AngelScript disabled. They run normal native client compilation and linking, including SetupManagedRuntime and the generated runtime identity. They require an Apple host, Xcode, a .NET 10 SDK and network access to the pinned dotnet/runtime source.

The manual validate workflow accepts job=managed-apple to run only four managed Apple builds: native macOS x64 and arm64, iOS x64 simulator and unsigned iOS arm64 device. job=all also runs the ordinary matrix. Automatic push/PR validation keeps the existing ordinary matrix; managed Apple builds are explicit because of their additional runtime build cost. The device build disables code signing and proves compilation/linking, not installation, signing or on-device execution. These engine-only builds need no embedding-project code, resources or credentials. Embedding projects must still validate their own managed assemblies, packages and live runtime behavior.

The unit-test executable follows the configured scripting backends. AngelScript-only test translation units are compiled only with FO_ANGELSCRIPT_SCRIPTING; Test_ManagedScriptBaker is compiled only with FO_MANAGED_SCRIPTING. A managed-only embedding project can therefore build and run its local RunUnitTests target without re-enabling the retired runtime backend. Ordinary unit validators retain the full AngelScript backend boundary.

BakerTests::TestRig keeps sources and outputs in memory and leaves BakeOutput empty, so map/proto bakers cannot load unrelated managed assemblies or particle caches from the process working directory. Tests that exercise disk output, assembly packaging or dependency caches must explicitly set a private bake directory; this includes dry-run managed project generation. The MapBaker regression plants a foreign assembly under the working directory and verifies isolation plus explicit disk opt-in.

Test_ServerEntityLifetime runs for every scripting-backend configuration. Its two [lifetime] cases start a real server using in-memory metadata/prototypes and retain native owners of Critter, Item, Map, Location and Player. One releases those owners on another joined thread after shutdown and destruction of the server; the other releases them before shutdown to exercise normal destructor invariants. ASan runs detect stale engine access during deferred release. When AngelScript is enabled, the fixture compiles its own minimal server bytecode against the same in-memory metadata before startup. The fixture uses no embedding-project assemblies, resource packs or database files.

Managed core-script regression tests

With a .NET 10 SDK, run the offline console harness:

dotnet run --project Source/Scripting/Managed/Tests/FOnline.CoreScripts.Tests.csproj

It compiles the real managed invocation, registration and value-type helpers against a minimal generated-API fixture. Cases cover ref-result conversion and failure accounting, qualified modules/enums, overload selection, cached dispatch allocation, native fallback, isolation from foreign enum assemblies, dictionary signatures, async completion, signed duration boundaries, direction normalization for both map geometries and narrow/full-width signed inputs, and isolated bootstrap runs with and without neighboring source files. The native baker suite verifies that generated direction structs cannot bypass CoreScript normalization, and geometry tests pin the matching native constructor boundaries. A failing static constructor must stop startup before module initialization. Native calls are fixture boundaries; embedding projects must also bake and run their managed gameplay tests against the actual Mono backend.

The native callback GC probe uses an existing Linux Mono embedding runtime (its include/mono-2.0 and lib directories), Clang, and the .NET 10 SDK on PATH:

FO_MANAGED_CALLBACK_RUNTIME=/path/to/mono/linux.x64.Release \
  python3 -m pytest BuildTools/tests/test_managed_callback_gc_roots.py

It compiles the canonical DispatchManagedCallbackInContext body and managed callback helpers against small argument-conversion fixtures. Real Mono collections cover eleven mixed scalar arguments, a mutable string with a return value, and cleanup after a boxing exception. The Mono profiler checks strong-handle lifetime at the boxing and copy-back boundaries: native conservative stack scanning can otherwise keep an unrooted object alive. The same probe runs 10,000 frame-pump scopes on one external worker under hybrid suspension, verifies one attachment for the worker lifetime, and forces a collection while that worker is parked GC-safe before checking its one final detach. This proves the native ownership and worker-lifetime contracts; WebAssembly collection and browser behavior still require a Web runtime check.

An existing Linux Makefiles unit-test build also supplies the actual SyncContext and EntityLock implementations for the callback scope probe:

FO_MANAGED_CALLBACK_BUILD=/path/to/native/build \
  python3 -m pytest BuildTools/tests/test_managed_callback_context.py

This probe compiles the canonical callback wrapper and ServerEngine::RunScriptContext method on a small fixture host. Releasing or replacing the callback’s cover, including an exceptional return, must preserve the caller’s context and physical lock while releasing the callback’s own lock. It records the native link inputs and verifies that they remain unchanged during linking. A running server with real managed remote calls remains the end-to-end acceptance check.

Unit tests under sanitizers

The unit tests also run under Clang sanitizers via dedicated validators, which select the matching San_* build type and run RunUnitTests instrumented:

Engine/BuildTools/validate.sh unit-tests-san-address    # AddressSanitizer (+LeakSanitizer)
Engine/BuildTools/validate.sh unit-tests-san-memory     # MemorySanitizer (requires Workspace/msan-libcxx)
Engine/BuildTools/validate.sh unit-tests-san-undefined  # UndefinedBehaviorSanitizer
Engine/BuildTools/validate.sh unit-tests-san-thread     # ThreadSanitizer

The validate.yml workflow runs these as a unit-tests-sanitizers matrix job. ASan/MSan/UBSan/TSan are blocking legs. The unit-tests-san-memory validator prepares Workspace/msan-libcxx by building LLVM’s libc++, libc++abi, and libunwind with MSan instrumentation, then configures San_Memory with FO_MSAN_LIBCXX_ROOT. The runtime build applies a narrow libunwind ignorelist so C++ exception and sanitizer-report unwinding do not self-report on ABI register snapshots. Engine native stack capture and the backward-cpp signal handler are disabled under MSan and TSan so the sanitizer runtimes own their reports; backward-cpp/libbfd symbolization under TSan also produces prohibitive shadow-memory growth. The embedded Mono archive and its generated JIT code are not instrumented by the host sanitizer toolchain. Managed-script builds therefore reject San_Memory*: valid runtime writes otherwise retain poisoned shadow bytes and report as soon as Mono loads CoreLib. They also reject San_Thread: Mono suspends mutators with signals for stop-the-world collection, which does not publish a happens-before edge to the host TSan runtime; valid nursery allocation and collection then report as races. Changing the SGen clear or collector mode only moves those reports between Mono’s intercepted memcpy/memset calls. The Linux source patch initializes and publishes POSIX signal-action bytes for bounded MSan diagnostics, but does not qualify the whole runtime for either sanitizer. Use the managed-disabled engine unit validators for native MSan/TSan coverage and ASan/UBSan for managed runtime execution. unit-tests-san-memory-with-origins is available locally as the slower diagnostic variant when a future MSan finding needs origin tracking. San_DataFlow remains intentionally unwired: DataFlowSanitizer is a taint-tracking framework, not a defect detector.

Applications that load BakerLib while running under a sanitizer must use a baker built with the same San_* configuration. Hiding the plugin’s ELF exports prevents direct symbol interposition, but calls implemented inside the shared C++ runtime may still allocate through the host and return to an inline deallocator in the plugin. Matching configurations keep the sanitizer runtime and allocator contract identical on both sides of that module boundary.

On MSVC, the San_Address/Debug_San_Address configs additionally link executables with /STACK:8388608 (AddExecutableApplication in BuildTools/cmake/helpers/Build.cmake): ASan’s stack-frame inflation overflows the 1 MiB Windows executable default on recursion depths that fit every production configuration, so sanitizer runs get the same 8 MiB reserve that Linux runs already have from the default rlimit. Production configs keep the 1 MiB default.

Vendored third-party libraries are excluded from UBSan’s -fsanitize=function and -fsanitize=alignment checks (the rest of -fsanitize=undefined still applies to them). DisableLibWarnings adds -fno-sanitize=function,alignment on the San_Undefined/San_Address_Undefined configs because several vendored libraries trip those two checks by design:

Both are third-party idioms, not undefined behaviour in engine code, so they must not fail the UBSan leg (which CI runs with halt_on_error=1). First-party engine code keeps both checks fully active.

LeakSanitizer runs as part of the address-sanitizer leg (CI sets ASAN_OPTIONS=detect_leaks=1). It runs with no suppression list — every leak it can report is fixed at the source rather than masked. Notable cases:

Code coverage

When FO_CODE_COVERAGE is enabled, BuildTools/cmake/stages/Init.cmake selects the backend from the compiler:

Coverage builds use AngelScript’s portable generic calling convention. The native x64 GCC trampoline adjusts the stack inside inline assembly and cannot reliably unwind an application C++ exception once coverage instrumentation changes the surrounding frame; the portable path keeps the same registered-function behavior in ordinary C++ so expected exception tests remain catchable.

BuildTools/cmake/stages/Applications.cmake wires coverage command targets through BuildTools/codecoverage.py:

Coverage output is rooted under CodeCoverage/<Toolchain>/<Platform-Config>/. Coverage-only configurations also provide the ordinary <DevName>_ServerHeadless and <DevName>_Baker executable targets. They link the same instrumented core libraries as <DevName>_CodeCoverage; no second configuration or production runtime rebuild is required. They do not enable the windowed applications or the baker plugin. Clang/GCC companion applications, including the managed script baker, register a quick_exit coverage flush on platforms where ExitApp uses it (Linux/Windows; Apple, Android, and Web retain exit), because the engine’s ordinary shutdown bypasses the compiler runtime’s atexit writer.

For native LLVM coverage of script-driven integration tests, first run RunCodeCoverage, then run the embedding project’s real integration tests with an absolute LLVM_PROFILE_FILE=<coverage-output>/raw/integration-%m-%p.profraw. Keep bake/setup profiles in a separate directory so setup execution cannot replace gameplay acceptance. Verify every integration process succeeds and produces its own nonempty profile; a unit-test profile alone does not prove that an integration process contributed. Use the original instrumented executables as coverage objects, even if the tests run byte-identical staged copies.

Finally invoke BuildTools/codecoverage.py report directly with the existing --workspace-root, --build-dir, --binary, --backend llvm, and --output-dir arguments, adding --object <instrumented-server> for the integration executable. --object is repeatable for additional executables/shared libraries and supported by LLVM report/full only. The collector disables debuginfod lookup and rejects binary IDs missing from the supplied objects. LLVM merges profiles before exporting all supplied objects together; shared source lines remain a union, while uncovered lines in integration-only source files stay in the denominator. Do not invoke GenerateCodeCoverageReport or AnalyzeCodeCoverage after integration tests: the former depends on RunCodeCoverage, and both start a fresh unit collection that removes previous profiles. full likewise starts a fresh run; use report to preserve integration data. BuildTools/tests/test_codecoverage_llvm_objects.py exercises the collector with actual instrumented processes, including quick exit, shared source mapping, and failing inputs.

The engine validation workflow uploads coverage through the pinned Codecov action release 7.0.0. Its composite action uses a Node 24 helper and preserves CLI signature verification, token authentication and failure propagation for upload errors. BuildTools/codecoverage.py reports first-party production engine sources under Engine/Source/; it excludes Source/Tests/, ThirdParty/, GeneratedSource/, and Applications/ from the denominator. See ../Source/Tests/README.md for current local task notes.

Coverage is a per-platform, per-environment measurement, and the denominator reflects that in two different ways:

The summary prints the scoped headline, a combined all-sources figure, and the excluded bucket file-by-file with reasons, so the split stays auditable. Adding an entry there is a routing decision, not a write-off: it must be covered by the layer that can run it — a windowed/rendering run on the owning platform, or an integration suite with real endpoints.

Covering ImGui diagnostic panels

DrawGui() implementations normally only run inside the windowed application, but they are reachable from unit tests through a backend-less ImGui context: no renderer is attached and the draw data is discarded, while every panel builder runs for real. Test_ServerEngine.cpp shows the pattern. These details matter:

Pressing a widget so the branch behind it runs

Drawing a panel covers its layout, not its behaviour: the body of every button, checkbox, selectable and tree node stays unreachable because nothing is ever clicked. Test_ImGuiHarness.h closes that gap, and Test_ImGui.cpp pins the harness itself against a window the test owns. The rules that matter:

What an inbound remote call can reach

The handler is entered with the calling player covered, plus - transitively - the critter it controls. Everything else needs an explicit Game.Sync(...), and some native paths reach further than any cover a script can prepare. Reachable on a second critter after Game.Sync(npc): Map.AddCritter, Critter.SetDir, Critter.Action and synchronized property writes. Not reachable on a critter the caller does not control: TransferToHex, SetCondition, DestroyItem, AttachToCritter and Game.DestroyCritter, plus moving a map item into an inventory and reusing one location across two logins.

A script-level catch around a failing call does not contain the damage: the sync violation still tears the session down, so the next remote call never arrives. A test that probes these operations behind try/catch therefore reports “the last step was never sent” rather than the operation that actually failed - drive only what is reachable.

Covering the crash reporter

ExceptionHandling.cpp publishes SetCrashStackTrace, SetCrashSignalInfo, SetCrashSehInfo, SetCrashTerminationInfo and GetCrashStream to backward.hpp only — they carry no engine namespace and appear in no engine header, so a test declares them exactly as that header does. The report is emitted through the base log on the first write to the crash stream, so point LogToFile at a private file, write one line into GetCrashStream() and read the report back instead of letting “FATAL ERROR!” leak into the test console. Restore the log with LogToFile("/dev/null") ("NUL" on Windows); there is no “stop logging to a file” call. Terminating reporters are covered out of process through DiagnosticSelfTest: main_strong_assert covers ReportExceptionAndExit, main_basic_strong_assert and main_fatal_exit cover the early FatalError layer, and main_failure_exit pins the raw status-only ExitApp(false) contract. The embedding project’s Tools/PipelineTests/test_crash_diagnostics_linux.py asserts their log and exit contracts without killing the unit-test process.

Covering the text formatter without a real font asset

FontManager refuses to answer any metric for an unbound slot, so text measurement, wrapping and drawing are unreachable until a font exists. Both loader formats can be synthesized in-memory, which is cheaper and more stable than shipping a binary asset:

Bind with a scale in (0..1] — larger scales are rejected on purpose, because the intended fix for bigger text is a bigger font asset.

SplitLines paginates into rect-sized pages rather than into individual lines: it emits an entry only once the text overflows the rect height, so a test that wants several entries needs a short rect, not merely embedded newlines.

Driving a logged-in client↔server session

A connected client is not a logged-in one: the pre-login session accepts only a remote call, so login is script-driven from the client. The pieces that have to line up:

Reaching the world-reload path

Server tests default to the in-memory database, which means the branch a real server takes on every restart — “Restore world” and EntityManager::LoadEntities — never runs. Point the settings at the file-backed JSON storage instead (DbStorage = "JSON <dir>"), let one server write the world and shut down, then start a second server on the same directory. Two constraints:

Instantiating a 3D model headlessly

The Null renderer serves the whole model path, so a ModelInstance can be created, posed and drawn without a GPU. The fixture chain is what makes it work:

Get the manager from the live client with client->SprMngr.GetSpriteFactory(typeid(ModelSpriteFactory)).dyn_cast<ModelSpriteFactory>()->GetModelMngr().

Authoring static map content for a server fixture

A .fomap-bin-server blob is the format header, then the hash table, then the critter records, then the item records. Each record is ident (int64), the prototype hash (uint64) and a properties blob preceded by its uint32 size. Writing a zero size fails with “Unexpected end of buffer” — a default-constructed Properties still serializes to a non-empty payload, so produce it with props.StoreAllData(...) rather than assuming empty means zero bytes. With content present, map creation runs the content generator instead of skipping it.

The client-side .fomap-bin-client blob is a different, shorter layout (header, hash table and static items only).

A per-map static item removal is only observable end to end when the same static item id appears in both blobs: the server needs it in StaticItemsById to remove it, and the client needs a view built from it to drop. Test_ClientServerIntegration carries one such item (props with Static, Ownership = MapHex and a Hex) in both map blobs, so a server-side Map.RemoveStaticItem is checked against the live client’s MapView::GetItem.

Writing into a real Maps root from the mapper

SaveMap / SaveMapToDir resolve the on-disk Maps root from an existing map container, so a memory-only fixture cannot reach them. Point a resource pack at a temp directory with InputDirs = <dir> (the plural key — the singular one is silently ignored), drop a reference .fomap there, and set ProtoFileExtensions to include fomap so the container is recognised. The same fixture gives DrawMapListWindowImGui real entries to enumerate.

Prefer SaveMapToDir in tests: plain SaveMap falls back to the first source file’s directory when the map has no container of its own, which in a test process is the working directory — it will write into the repository.

Current test inventory

Current count: 108 Test_*.cpp suites.

Essentials and low-level utilities

Configuration, data sources, files, and caches

Common runtime model

Networking and server/client integration

Scripting and script-visible APIs

Bakers and tools

The model-animation tests divide the production contract explicitly. Test_ModelMeshData.cpp exercises the mandatory LFMODMSH schema-1 mesh-only header and complete recursive payload codec. It covers geometry, skin palettes, children, structural validation, trailing data, every truncated header size, rejection of old headerless data, and exact byte compatibility with the original schema-1 writer layout. Test_ClientEngine.cpp also bakes a position-only OBJ through ModelMeshBaker and preloads the resulting bytes through the real ModelManager parser. This crosses the BakerLib/ClientLib boundary and catches payload-layout drift that a second test-only parser could reproduce instead of detecting. Test_ModelSourceLoader.cpp covers complete source validation, real minimal OBJ/ASCII-FBX extraction, per-call cache single-flight behavior, shared results, exception fan-out, and missing inputs. Test_ModelAnimationData.cpp exercises the little-endian archive, joint-remap, and rig-manifest contracts, including truncation, count/length bombs, ordering, metadata mismatches, and bindings. Test_ModelAnimationConverter.cpp covers canonical conversion and the per-instance runtime pose: unaligned/owned loading, body blending, movement replacement, reverse and nearest sampling, stable storage, canonical resolution, and numeric limits. Test_ModelAnimationPoseProcedural.cpp covers bounded procedural pre-rotations and exact world-matrix overrides; Test_ModelAnimationRuntime.cpp covers the validated direct-model rest path, canonical contributed-joint lookup, and cross-model joint-link resolution without physical bones. Test_ModelBaker.cpp covers source-backed model-info generation, dependency-mtime invalidation, exact animation-geometry exceptions, Base, reverse, case-insensitive lookup, and clip deduplication. Test_ModelAnimation.cpp is the timeline/binding behavior gate: controller copies own mutable event state while sharing only immutable Ozz clip metadata.

After source-loader, mesh-wire, or converter changes, ForceBakeResources is the positive real-content gate: it must parse the project’s actual selected FBX sources and extract their animations successfully. Run ordinary BakeResources afterward to check that the dependency-mtime contract leaves an unchanged tree incremental-clean.

Rendering/frontend smoke tests

Validation routing by change type

Adding or removing tests

  1. Add the new Source/Tests/Test_*.cpp file with deterministic Catch2 tests.
  2. Add it to FO_TESTS_SOURCE in BuildTools/cmake/stages/EngineSources.cmake.
  3. Update this page and ../Source/Tests/README.md so the inventory stays complete.
  4. Run the focused test binary and, when practical, RunUnitTests.
  5. If coverage behavior changed, verify the relevant coverage target.

Validation checklist

  1. Every current Source/Tests/Test_*.cpp file should appear in this page.
  2. No deleted/nonexistent test file should be listed.
  3. Target names should be described as generated from FO_DEV_NAME, not hard-coded as universal engine names.
  4. If TestingApp.cpp, FO_TESTS_SOURCE, or coverage target wiring changes, update this page in the same change.