Essentials

Engine-owned documentation. This page maps the low-level Source/Essentials/ layer: platform/compiler prerequisites, process-wide lifecycle helpers, logging, memory, strings, serialization, filesystem, sockets, and utility types used by every higher engine layer.

Purpose

Use this page when changing code that sits below Source/Common/ or when you need to know whether a utility belongs in the reusable engine foundation instead of client, server, tools, or game-specific code.

For the memory model’s exception contract — SafeAlloc / SafeAllocator terminate on OOM (so std::bad_alloc is not a recoverable error) and the throw / FO_VERIFY_* / FO_STRONG_ASSERT error tiers built on ExceptionHandling.h — see ExceptionSafety.md.

The essentials layer should stay dependency-light. It is included by most of the engine through Source/Essentials/Essentials.h, so changes here can affect every application target.

Cross-layer decision

Essentials follows a strict dependency DAG: a dependency must appear earlier in the umbrella block, and a reverse dependency must be moved upward through parameters or a higher owning layer. Register new implementation files in FO_ESSENTIALS_SOURCE; EssentialsLib is the owning target, and consumers must link at the correct dependency point rather than bypassing the layer.

The exact umbrella order is BasicCore, GlobalData, StackTrace, BaseLogging, FatalError, FunctionObjects, SmartPointers, MemorySystem, StringObject, DequeObject, Containers, StringUtils, WinApi, Posix, Platform, ExceptionHandling, RandomGenerator, Threading, SafeArithmetics, DataSerialization, HashedString, StrongType, TimeRelated, ExtendedTypes, Compressor, WorkThread, Logging, DiskFileSystem, CommonHelpers, and NetSockets. Do not reorder that list to repair a cycle. Move reverse dependency pressure upward through a parameter or split the responsibility into the higher owning layer. Every new Essentials .h / .cpp must enter the checked FO_ESSENTIALS_SOURCE list, from which CoreLibs.cmake builds EssentialsLib; a source-path inventory alone is not build wiring. Only headers participate in the Essentials.h umbrella order; never add a .cpp file to that umbrella. Register both headers and implementation files in FO_ESSENTIALS_SOURCE, then verify that EssentialsLib owns them and that its consumers link at the correct dependency point.

When the same change affects script-visible metadata, keep ownership separate. The Engine owns the reusable metadata/codegen machinery; the embedding project supplies project configuration, extra metadata sources, common headers, and script/content inputs; generated files are build artifacts. Diff all eighteen canonical generated models and require the reviewed exact domain-bound disposition for every gated compatibility break; an Essentials unit test does not bypass the contract-change gate.

Source paths inspected

• Source/Essentials/Essentials.h
• Source/Essentials/Essentials.cpp
• Source/Essentials/BasicCore.h
• Source/Essentials/BasicCore.cpp
• Source/Essentials/GlobalData.h
• Source/Essentials/GlobalData.cpp
• Source/Essentials/StackTrace.h
• Source/Essentials/StackTrace.cpp
• Source/Essentials/BaseLogging.h
• Source/Essentials/BaseLogging.cpp
• Source/Essentials/FatalError.h
• Source/Essentials/FatalError.cpp
• Source/Essentials/FunctionObjects.h
• Source/Essentials/FunctionObjects.cpp
• Source/Essentials/SmartPointers.h
• Source/Essentials/SmartPointers.cpp
• Source/Essentials/MemorySystem.h
• Source/Essentials/MemorySystem.cpp
• Source/Essentials/StringObject.h
• Source/Essentials/StringObject.cpp
• Source/Essentials/DequeObject.h
• Source/Essentials/DequeObject.cpp
• Source/Essentials/Containers.h
• Source/Essentials/Containers.cpp
• ThirdParty/small_vector/README.md
• ThirdParty/small_vector/source/include/gch/small_vector.hpp
• Source/Essentials/StringUtils.h
• Source/Essentials/StringUtils.cpp
• Source/Essentials/WinApi.h
• Source/Essentials/WinApi.cpp
• Source/Essentials/Posix.h
• Source/Essentials/Posix.cpp
• Source/Essentials/Platform.h
• Source/Essentials/Platform.cpp
• Source/Essentials/ExceptionHandling.h
• Source/Essentials/ExceptionHandling.cpp
• Source/Essentials/RandomGenerator.h
• Source/Essentials/RandomGenerator.cpp
• Source/Essentials/Cryptography.h
• Source/Essentials/Cryptography.cpp
• ThirdParty/Monocypher/src/monocypher.h
• ThirdParty/Monocypher/src/monocypher.c
• Source/Essentials/Threading.h
• Source/Essentials/Threading.cpp
• Source/Essentials/SafeArithmetics.h
• Source/Essentials/SafeArithmetics.cpp
• Source/Essentials/DataSerialization.h
• Source/Essentials/DataSerialization.cpp
• Source/Essentials/HashedString.h
• Source/Essentials/HashedString.cpp
• Source/Essentials/StrongType.h
• Source/Essentials/StrongType.cpp
• Source/Essentials/TimeRelated.h
• Source/Essentials/TimeRelated.cpp
• Source/Essentials/ExtendedTypes.h
• Source/Essentials/ExtendedTypes.cpp
• Source/Essentials/Compressor.h
• Source/Essentials/Compressor.cpp
• Source/Essentials/WorkThread.h
• Source/Essentials/WorkThread.cpp
• Source/Essentials/Logging.h
• Source/Essentials/Logging.cpp
• Source/Essentials/DiskFileSystem.h
• Source/Essentials/DiskFileSystem.cpp
• Source/Essentials/CommonHelpers.h
• Source/Essentials/CommonHelpers.cpp
• Source/Essentials/NetSockets.h
• Source/Essentials/NetSockets.cpp
• Source/Essentials/UcsTables.inc
• Source/Essentials/WinApiUndef.inc
• BuildTools/natvis/essentials.natvis
• BuildTools/cmake/stages/EngineSources.cmake
• BuildTools/tests/test_essentials_layering.py
• related tests under Source/Tests/

Include and dependency model

Source/Essentials/Essentials.h is the umbrella include. Its exact include order is the dependency order for the foundation layer:

BasicCore → GlobalData → StackTrace → BaseLogging → FatalError → FunctionObjects → SmartPointers → MemorySystem → StringObject → DequeObject → Containers → StringUtils → WinApi → Posix → Platform → ExceptionHandling → RandomGenerator → Threading → SafeArithmetics → DataSerialization → HashedString → StrongType → TimeRelated → ExtendedTypes → Compressor → Cryptography → WorkThread → Logging → DiskFileSystem → CommonHelpers → NetSockets.

This list is intentionally exact rather than thematic. Essentials.h defines a strict dependency DAG: every Essentials header and its .cpp may include and call only modules that appear earlier in the umbrella block. Declaring an API early but defining it in a later .cpp is still a reverse link dependency. BuildTools/tests/test_essentials_layering.py checks direct includes and namespace-level external-definition ownership. Do not reorder the list to hide a cycle; move data through parameters or split the responsibility at the correct layer.

Keep new essentials APIs free of dependencies on Source/Common/, Source/Client/, Source/Server/, Source/Tools/, or embedding-project headers.

global_data::destroy(observer, context) calls the observer with each registered set name immediately before its delete callback. This makes a stuck teardown attributable to a set without letting the observer depend on global data that may already be gone. A host/runtime module must destroy only a set it created, and it must join its own workers before returning to a continuing host. Process-lifetime crash-record state stays outside this sweep.

Subsystem map

Platform and compiler gate

BasicCore.h also declares FO_TRACE_COLOR_<Category> names and the FO_TRACE_ZONE(Category) / FO_TRACE_ZONE_NAMED(Category, name) macros. Profiling builds use generated TraceCategories.gen.h to compile in selected zones; other builds validate category names but emit no zones. See the profiling guide.

BasicCore.h enforces the selected OS macro (FO_WINDOWS, FO_LINUX, FO_MAC, FO_ANDROID, FO_IOS, or FO_WEB) and requires C++20. It also binds frequently used standard types into the engine namespace and declares core macros such as FO_EXPORT_FUNC, FO_KEEP_DATA_SYMBOL, and namespace helpers. Warning-suppression helpers also live here: FO_DISABLE_WARNINGS_PUSH/POP silence all warnings (for wrapping third-party header includes), while the per-compiler FO_GCC_IGNORE_WARNINGS_PUSH/POP, FO_CLANG_IGNORE_WARNINGS_PUSH/POP, and FO_MSVC_IGNORE_WARNINGS_PUSH/POP silence one named diagnostic and are active only on their matching compiler (so a single-toolchain false positive can be suppressed at one site without other toolchains rejecting an unknown -W name or warning number). Prefer fixing warnings at their root; reach for the per-compiler helpers only for documented compiler false positives.

Platform.h / .cpp owns host-specific helpers that are deliberately small: informational logging, thread names, executable path lookup, per-user data directory lookup, process id formatting, fork support where available, process memory usage, CPU usage snapshots, and dynamic module loading. Platform::GetUserDataBase() prefers environment values and is shell/SDL-free: Windows uses %LOCALAPPDATA% (else %APPDATA%), macOS/iOS use $HOME/Library/Application Support, and Linux/Android/other use $XDG_DATA_HOME (else $HOME/.local/share). When no environment path exists, Windows falls back to SHGetKnownFolderPath and supported POSIX hosts to getpwuid_r; a host without either source returns no path. Higher layers append the application name and decide whether absence is fatal. Platform::GetCpuUsageSnapshot() returns cumulative per-core system counters plus the current process CPU time; callers compare two snapshots to compute percentages and keep any sampling/cache state outside the Platform layer. Platform stays above ExceptionHandling and uses the earlier FO_BASIC_STRONG_ASSERT for terminating host-API invariants rather than importing later exception macros. Platform-specific application/window/rendering behavior lives under Source/Frontend/, not here.

platform::get_last_module_error() captures the Windows loader error text or POSIX dlerror() immediately after a failed load or symbol lookup. platform::get_os_version() uses RtlGetVersion on Windows and uname elsewhere for client diagnostics.

WinApi.* and Posix.* own the operating-system calls behind the winapi:: and posix:: namespaces. Their public boundaries use engine strings, optionals, fixed-width integers, and nptr<void> rather than leaking HANDLE, pid_t, or other OS types. Platform dispatches to those modules; ordinary consumers add or call a wrapper instead of including <Windows.h> or POSIX headers directly. The structural exceptions are lower Essentials implementations that the module order cannot depend on (BasicCore.cpp, BaseLogging.cpp, and StringUtils.cpp) plus NetSockets.* and ServerServiceApp.cpp, which are OS wrappers themselves rather than ordinary consumers.

Windows builds retain the _WIN32_WINNT=0x0601 compile baseline. One Windows build-platform registry owns the CMake architecture, toolset, and canonical packaging architecture for the regular, -clang, and -win7 variants. The Win7 pair pins MSVC 14.44, while FO_BINARY_OUTPUT_POSTFIX remains independent of the platform. In the package DSL the corresponding BINARY entry can select its own postfix, for example BINARY Client Windows win32-win7 Raw+Zip+Wix POSTFIX Win7, without affecting sibling binaries in the package. Compatibility checks are kept outside application targets.

platform::process_identity pairs PID with process start time. An ID alone can be reused, so client-session diagnostics match both values. On Windows the liveness check polls the process handle with zero timeout rather than reading exit code 259 (STILL_ACTIVE), which a terminated process can retain while another process holds its handle. BuildTools/tests/test_process_identity.py covers live and terminated retained-handle cases when clang++ is available.

Temporary compatibility

FO_TEMPORARY_COMPAT(Id, "YYYY-MM-DD"); in BasicCore.h marks necessary code that recognizes older builds or data, including a refusal path. Its static_assert checks the date’s shape and can stand at namespace, class, or block scope; it does not make the build expire. Managed code uses [TemporaryCompat("Id", "YYYY-MM-DD")] from CoreScripts/Attributes.cs. Repeat the same identifier and last-valid date on every related implementation, member, and test.

BuildTools/temporary_compat.py scans Engine Source/ by default, rejects malformed or inconsistent markers and dates more than 366 days ahead, and fails from the day after expiration with all marked locations. The Engine validation workflow runs its tests and scan. An embedding project can pass its own source directories alongside Engine/Source; the marker never substitutes for a reviewed reason to retain old-format handling. At expiry, remove the code or move the date in a reviewed change if the reason remains. Keeping expiry in CI rather than the macro leaves old release builds reproducible.

Diagnostics and failure handling

BaseLogging.* and Logging.* provide the logging foundation. WriteLogMessage() collapses immediate duplicates by LogType and message text: repeated copies are skipped, then the next different log line first emits a summary such as ...and 25 more same messages. LogToFile() opens the file without an exclusive lock, and every WriteSync seeks to the end, so separately linked Engine modules in one process can append safely. WriteLog/WriteBaseLog degrade to the base log and then std::cout before full global data exists.

FatalError.* is the early native-only fatal layer. It follows StackTrace and BaseLogging, suspends asynchronous writes, emits one synchronous message plus native trace, and delegates only mechanical termination to BasicCore::ExitApp(false). It owns ReportFatalAndExit, ReportStrongAssertAndExit, and FO_BASIC_STRONG_ASSERT without constructing exception objects or depending on the later ExceptionHandling module. ExitApp(false) itself remains status-only because controlled command failures and fatal invariant failures both use it.

StackTrace.* captures and formats native/script stacks, while ExceptionHandling.* owns the later exception-object reporting helpers. For debugger-facing workflows, use Native, AngelScript, and Managed Debugging.

Memory, pointers, and lifetime utilities

MemorySystem.* owns backup-memory chunks, bad-allocation reporting, and SafeAllocator. SmartPointers.* contains pointer wrappers used to make ownership, nullability, and raw-reference intent explicit; see SmartPointers.md for the native ptr / nptr vocabulary and migration rules. Use this layer for generic ownership utilities only; entity lifetime and holder semantics belong in Entity Model.

Callable vocabulary

FunctionObjects.* replaces std::function with two engine-owned wrappers. function<Signature> is an alias of move-only move_only_function<Signature> and is the default. Use copyable_function<Signature> only when copying the stored target is the real ownership contract, such as a callback snapshot distributed to more than one owner. Both keep a small nothrow-movable target inline and allocate larger targets through a fail-fast path, so construction does not introduce a recoverable std::bad_alloc. If migration exposes a copy, first decide whether the owner should move instead. The StackTrace.h script-provider hook is the sole retained std::function, because it precedes FunctionObjects in the dependency order.

Allocation vocabulary

Engine code allocates through one of two surfaces, and nothing else:

• The fo container aliases from Containers.h — string, wstring, vector, map, unordered_map, set, list, deque, stringstream, small_vector and friends. string and wstring use the engine basic_string from StringObject.h; deque uses basic_deque from DequeObject.h; the remaining allocator-aware aliases use SafeAllocator. Use these, never the std:: originals.
• SafeAlloc — MakeUnique / MakeShared / MakeRefCounted / MakeRawArr / MakeUniqueArr for typed objects, and the raw tier MallocRaw / CallocRaw / ReallocRaw / FreeRaw plus MallocAlignedRaw / FreeAlignedRaw for C-ABI boundaries.

The raw tier exists because third-party allocator hooks are C-shaped: they demand realloc, or an untyped byte block, or both, which a C++ allocator cannot express. It carries the same out-of-memory policy as SafeAllocator — report, drain the backup pool, retry, then exit deterministically — so wiring a library through it does not silently opt that library out of the contract. A zero-size request is passed through rather than treated as failure.

The underlying rpmalloc primitives are deliberately not exported from MemorySystem.h. They return null on failure and would be a second, equally reachable entry point that skips the contract; they live as file-local statics in MemorySystem.cpp. The vendored allocator propagates both an on-demand commit failure in a reserved span and a failed recommit of a decommitted free page as null. It recommits that page before publishing it as available; on failure it returns the page to the free list, so the backup-pool retry cannot hand out reserve-only memory. Test_MemorySystem.cpp injects both failures. Bad-allocation reports are serialized across threads; if stack-trace generation itself cannot allocate, the nested report writes its header without recursing. Backup chunks released by another thread return to their original thread heap and do not guarantee retry headroom for every thread. Mono’s lock-free allocator is outside this contract and can abort on OOM. The MemCopy / MemMove / MemFill / MemCompare / MemReadUnaligned / MemWriteUnaligned block operations are unrelated to allocation and remain public.

The vendored rpmalloc keeps its upstream 256 MiB spans on 64-bit targets. On 32-bit targets one span is reduced to LARGE_PAGE_SIZE (16 MiB). Older Windows allocation APIs reserve size + alignment; an aligned 256 MiB span can therefore require a contiguous 512 MiB hole inside a 2 GiB x86 address space and make the first small allocation fail. The 16 MiB x86 span preserves all built-in page classes while removing that startup dependency on one enormous contiguous reservation.

Three distinct things are at stake when code bypasses this vocabulary, and they are not equally severe:

Known and accepted limits: std::future/std::promise/std::packaged_task, std::thread, std::filesystem::path and the file streams have no allocator parameter at all, so they reach the engine heap through global new but throw on exhaustion. The sole std::function in StackTrace.h is also above the engine callable module. Separately, BasicCore, StackTrace and BaseLogging sit above MemorySystem in the Essentials.h include order and therefore use std:: containers by design — MemorySystem.cpp calls GetStackTrace() from ReportBadAlloc, so the reporting path must not depend on the allocator that just failed.

Allocator occupancy diagnostics

memory::get_allocator_statistics() returns an allocation-free snapshot. With rpmalloc it is enabled in Debug and Tracy builds, or by FO_MEMORY_DIAGNOSTICS=ON in regular builds. Otherwise available is false. The shared AngelScript/Managed C# export Game.GetAllocatorStatistics() returns an empty dictionary when unavailable; the script dictionary itself allocates, unlike the native snapshot.

• mappedBytes, committedBytes, hugeAllocatedBytes and heapCount describe the rpmalloc instance in the calling native module. Commitment is global.active, not cumulative commit traffic.
• threadSizeClassAllocatedBytes is occupied block capacity in the calling thread, including size rounding and deferred cross-thread releases. It excludes huge allocations and is not requested payload size or a process-wide live-byte count.
• threadReusableBlockBytes counts immediately reusable slots, including unused page tails. threadFreeCommittedPageBytes separately counts cached committed pages.
• class<block-size>AllocatedBlocks and class<block-size>ReusableBlocks retain the size-class distribution; a slot cannot generally satisfy a larger allocation.

The snapshot never walks another live thread’s heap, drains deferred frees, forces GC, or trims caches. Global atomic counters are not one atomic multi-field snapshot. CRT allocations, GC-object storage, drivers, padding and GPU fragmentation are outside this measurement. Do not subtract thread occupancy from global commitment to infer fragmentation; reusable capacity alone is not evidence of harmful fragmentation. Test_MemorySystem.cpp checks holes and reuse on the owning thread without cache flushing.

Third-party allocators

Vendored vkd3d-shader exposes no allocator hook. Its bake-scoped calls use plain malloc and release returned objects through vkd3d’s own free functions; errors fail the effect bake. Do not count these temporary allocations as Engine-heap or Tracy allocation events.

The bson vtable is worth reading before copying its shape elsewhere: it supplies aligned_alloc but releases those blocks through the plain free member, never recording the alignment. That is sound only while both paths end in the same release function. Under rpmalloc they do (rpaligned_alloc and rpmalloc both end in rpfree), and so do they on POSIX without it (posix_memalign blocks are free()-able by definition). The one combination that breaks is Windows without rpmalloc — the sanitizer configs, where expr_RpmallocEnabled turns the allocator off so the sanitizer can interpose — because there the aligned path is _aligned_malloc/_aligned_free. BsonAlignedAlloc therefore falls back to plain safe_alloc::malloc_raw in exactly that case, which is what bson’s own default vtable does on MSVC and for the same stated reason (_aligned_alloc_impl in libbson memory.c deliberately does not use _aligned_malloc); every aligned request in mongoc is a BSON_ALIGNOF of an ordinary C struct, so malloc’s fundamental alignment covers them. The vtable is process-global, so every BSON-backed factory installs the same callbacks before its backend can allocate; changing it later could pair an old allocation with a new free callback. Dropping aligned_alloc from the vtable is not an alternative — bson then substitutes an internal fallback that discards the requested alignment on every platform, not just the one that needs it.

SQLite’s hook needs an xSize callback and hands the free/realloc/size functions only a pointer, so each block carries an 8-byte size header. Its configuration must also be installed before sqlite3_initialize, which is why the library is built with SQLITE_OMIT_AUTOINIT and every caller goes through one exported initializer.

Not hooked, with reasons: Monocypher allocates nothing; its callers own every buffer. LibreSSL exports CRYPTO_set_mem_functions but its body is an inert return 0; — custom allocators were removed upstream, so calling it would be dead code that reads like coverage. ogg / vorbis / theora expose no allocator hook.

When vendoring or updating a library, check whether it has an allocator hook and either wire it or record why not — and read the hook’s implementation, not just its declaration. Two of the entries above were initially misjudged from the call site or the symbol name alone.

Vector containers and inline storage

Containers.h exposes two sequence aliases backed by SafeAllocator<T>:

• vector<T> is the normal dynamically allocated sequence and remains the default for unbounded data, persistent collections that amortize their allocation, move-heavy pipelines, and exact engine interfaces.
• small_vector<T, InlineCapacity> stores up to InlineCapacity elements inside the object and spills to SafeAllocator storage above that limit. The engine alias requires an explicit capacity; GCH_SMALL_VECTOR_DEFAULT_SIZE configures the vendored implementation and is not a capacity-selection policy.

Use small_vector only when measurements or a hard protocol limit show that a frequently constructed collection is usually small. Choose the capacity from observed typical cardinality, keep rare large cases correct through heap spill, and account for the inline bytes in every instance. A per-call scratch list can be a strong candidate; adding several inline buffers to every cell in a dense map can cost more memory than the avoided first allocation. inlined() reports the current storage mode and is useful in focused tests and profiling instrumentation.

The representation has several correctness consequences:

1. Moving an inline small_vector relocates its elements into the destination object’s inline buffer. Pointers, references, and iterators into the source do not follow the move as they commonly do when a heap-backed vector transfers its allocation. Audit every address that survives a container move.
2. Inline moves and swaps perform element work and are only conditionally noexcept; re-derive the touched function’s exception-safety guarantee instead of inheriting assumptions from vector.
3. A small_vector<T, N> data member instantiates inline element destruction at the containing class boundary. T must therefore be complete there; it is not a drop-in replacement for a vector member whose element type is only forward-declared.
4. With the vendored implementation, a member whose element is a nested type with default member initializers can make default-inserting operations ill-formed while the enclosing class is incomplete, notably under Clang. For a shrink, prefer erase(begin() + new_size, end()); otherwise move the element type out of the enclosing class or make construction requirements explicit.
5. Heap spill still follows the engine’s terminate-on-OOM policy because the alias uses SafeAllocator. Element construction, conversion, and move operations may still throw; see ExceptionSafety.md.

Do not substitute small_vector across an exact-type boundary merely because the operations look vector-like:

• script export/codegen signatures and ScriptSystem registration use exact vector<T> / readonly_vector<T> spellings and type identities;
• property writes, serialized backing stores, DataReader / DataWriter, NetBuffer, and CScriptArray have exact vector contracts at selected boundaries;
• a span-taking internal helper may accept either representation without exposing the concrete container type, which is the preferred seam when both are valid;
• FO_ENTITY_PROPERTY cannot take small_vector<T, N> directly because the comma also separates macro arguments.

vector_collection admits both engine aliases for generic readers. The producing helpers vec_filter, vec_transform, and vec_sorted preserve vector versus small_vector and retain the inline capacity through rebind_vector_t; non-vector ranges materialize as vector. to_vector intentionally always materializes a vector, while copy_hold_ref exposes an opaque ref-held snapshot rather than a concrete sequence contract. The generic formatter accepts both aliases for ordinary numeric elements, but its string and boolean special cases currently match exact vector<string> and vector<bool> types; do not assume equivalent formatting for small_vector<string, N> or small_vector<bool, N> without extending and testing that formatter.

For an adoption, record the measured distribution and object count, audit address lifetime plus move/swap sites, verify complete-type and exact-interface constraints, and add focused coverage for inline operation, spill, and any generic-helper result type. Then run the complete native unit suite, the embedding project’s exception-safety and smart-pointer audits when present, and representative bake/gameplay/profiling paths for the changed subsystem.

Deque containers

DequeObject.* owns basic_deque<T, BlockBytes>, exposed by Containers.h as deque<T, BlockBytes = DEQUE_BLOCK_BYTES>. The default block holds 512 bytes of elements and never fewer than four elements, while a call site may select a different block size when measurements justify it. Growth at either end does not move existing elements, preserving the reference-stability contract used by engine message, packet, task, input, and database queues. Do not replace it with std::deque: the standard implementation’s fixed block policy is outside the Engine allocator and performance contract.

Random generation

RandomGenerator.* owns random_generator, the Engine’s xoshiro256++ source. Default construction seeds it from the OS; explicit seeding expands the seed through SplitMix64. Use next() for raw bits, next_below(bound) for [0, bound), next_between(min, max) for an inclusive signed range, and next_normalized() for [0, 1). These bounded mappings are Engine-owned and therefore deterministic across supported standard libraries. Do not use std::mt19937 or std::uniform_int_distribution for engine behavior.

Cryptography

Cryptography.* owns the crypto:: primitives used by the secure network channel: X25519, BLAKE2b-512 and HMAC-BLAKE2b, and RFC 8439 ChaCha20-Poly1305. The implementation uses the same vendored Monocypher on native, Web and Android targets. crypto::fill_random obtains key material from the operating system through platform::fill_system_random (BCryptGenRandom on Windows, getentropy on Linux/Web, arc4random_buf on Apple and Android); it throws if the platform cannot provide it. Never use random_generator to create a channel key. AEAD sealing/opening creates a fresh Monocypher context per message; the library’s streaming context rekeys after its first message. crypto::is_equal compares in constant time and crypto::wipe clears secret buffers; owners of plain key arrays wipe them during teardown.

Serialization, values, strings, and hashes

StringObject.* owns the engine basic_string implementation. Its API follows std::basic_string, while FO_STRING_INLINE_CAPACITY selects the compiled small-string buffer for both string and wstring; this is a build-wide ABI choice, not a per-call optimization. Three standard-library boundaries require explicit adapters: copy text into a standard stream with make_stream_string, construct std::filesystem::path through fs_make_path, and call getline unqualified so argument-dependent lookup selects the engine overload. String growth uses the same deterministic terminate-on-OOM policy as other engine storage.

DataSerialization.* contains binary read/write helpers used by network, persistence, resources, and tests. DataReader::Read<T>() and DataWriter::Write<T>() copy standard-layout values through byte copies so serialized streams do not depend on buffer alignment. The zero-copy ReadPtr<T>(size) overload is only for raw byte/string views (uint8_t, char, or void); typed values that need alignment must use Read<T>() or ReadPtr(destination, size). StringUtils.*, HashedString.*, StrongType.*, ExtendedTypes.*, SafeArithmetics.*, and TimeRelated.* provide the small reusable values that higher layers treat as primitives. iround rejects non-finite and out-of-int64-range floating-point input before rounding so no value undefined for std::llround can reach it. HashStorage::SetResolveHashFailureHandler lets higher layers observe failed hash resolution in both throwing and flagged no-throw lookup paths without teaching essentials about a specific recovery policy.

Filesystem, compression, sockets, and work threads

DiskFileSystem.* is the low-level disk abstraction. fs::disk_read_file captures a descriptor and length, and read_at performs bounded positional reads; a region constructor opens an uncompressed .fores inside an APK without copying it. fs::disk_write_file excludes competing writers, appends or truncates an uncommitted suffix, and flushes via fsync/_commit. fs::disk_directory_lock serializes resource mutations without a lock file (flock on POSIX, named mutex on Windows); unlike a named mutex, a second flock on the same directory is not reentrant even in one process. fs::rename_durable persists a rename, and fs::sync_parent persists the directory entry; fs::available_space permits admission without preallocating a resumable download. fs::is_contained_relative_path rejects rooted, traversal, colon, NUL, and malformed UTF-8 paths before joining a writable root. fs::make_writable_path layers a relative path under the user root. Use fs::make_path for UTF-8 to std::filesystem::path and fs::path_to_string on the way back: path.string() on Windows converts through the ANSI code page and can fail for a Cyrillic profile. The higher mounted view is Source/Common/FileSystem.*, described in Configuration and Data Sources. Compressor.* owns generic compression, NetSockets.* raw sockets, and WorkThread.* background workers.

On Windows, fs::make_io_path presents a literal extended-length path to standard-library filesystem/file operations without changing the logical resource path. The disk tests exercise Unicode names past 320 native characters and Windows trailing-name behavior. Use this conversion at the native I/O boundary rather than truncating or shortening project paths.

Threading.h exposes coarse_sleep and precise_sleep; Engine code does not use std::this_thread::sleep_for. coarse_sleep parks without consuming CPU and is intended for polling or other approximate waits. precise_sleep uses a high-resolution timer and spins the final short interval, so it is reserved for deliberate sub-millisecond deadlines such as synchronization back-off and frame pacing. Both functions are noexcept; choose by latency intent rather than interchanging them mechanically.

When a WorkThread job throws, a registered exception handler owns the reporting: it knows what the failure means to its owner and when to say so, and it can also update worker-owned policy such as clearing queued jobs. The thread reports through the global non-fatal exception reporter only when no handler is registered — reporting in both places would repeat the exception behind everything the handler already set in motion.

Build integration

BuildTools/cmake/stages/EngineSources.cmake lists every authored Essentials .h / .cpp pair plus the two .inc files and debugger visualization in FO_ESSENTIALS_SOURCE. BuildTools/cmake/stages/CoreLibs.cmake then creates EssentialsLib from that list. The library is part of the core dependency chain used by applications, tools, tests, and generated-code consumers. If a new Essentials file is added, place it at the correct dependency point in Essentials.h, wire it through FO_ESSENTIALS_SOURCE, and add focused coverage where possible.

Tests to inspect

The essentials layer has direct test coverage in:

• Source/Tests/Test_BaseLogging.cpp
• Source/Tests/Test_BasicCore.cpp
• Source/Tests/Test_CommonHelpers.cpp
• Source/Tests/Test_Compressor.cpp
• Source/Tests/Test_Containers.cpp
• Source/Tests/Test_Cryptography.cpp
• Source/Tests/Test_DequeObject.cpp
• Source/Tests/Test_DataSerialization.cpp
• Source/Tests/Test_DiskFileSystem.cpp
• Source/Tests/Test_ExceptionHandling.cpp
• Source/Tests/Test_ExtendedTypes.cpp
• Source/Tests/Test_FunctionObjects.cpp
• Source/Tests/Test_GenericUtils.cpp
• Source/Tests/Test_GlobalData.cpp
• Source/Tests/Test_HashedString.cpp
• Source/Tests/Test_Logging.cpp
• Source/Tests/Test_MemorySystem.cpp
• Source/Tests/Test_NetSockets.cpp
• Source/Tests/Test_Platform.cpp
• Source/Tests/Test_RandomGenerator.cpp
• Source/Tests/Test_SafeArithmetics.cpp
• Source/Tests/Test_SmartPointers.cpp
• Source/Tests/Test_StackTrace.cpp
• Source/Tests/Test_StringObject.cpp
• Source/Tests/Test_StringUtils.cpp
• Source/Tests/Test_StrongType.cpp
• Source/Tests/Test_TimeRelated.cpp
• Source/Tests/Test_Threading.cpp
• Source/Tests/Test_WorkThread.cpp

Test_Containers.cpp pins the engine alias, allocator, inline-to-heap transition, move, swap, and formatting behavior. Test_DequeObject.cpp covers block growth, both-ended mutation, iterators, reference stability, copy/move, and destruction. Test_RandomGenerator.cpp pins the seeded cross-platform sequence and bounded ranges. Test_Threading.cpp checks the sleep primitives and their sub-millisecond behavior. Test_CommonHelpers.cpp pins container-kind preservation through rebind_vector_t and the producing vec_* helpers.

See Testing for the complete test-suite map and target wiring.

Change routing

• Compiler/OS gates, namespace, base aliases, and low-level macros: Source/Essentials/BasicCore.*.
• Global create/delete callback registration: Source/Essentials/GlobalData.*.
• Stack traces, logging, and exception reporting: Source/Essentials/StackTrace.*, BaseLogging.*, Logging.*, ExceptionHandling.*, and Native, AngelScript, and Managed Debugging.
• Generic memory/pointer utilities: Source/Essentials/MemorySystem.*, SmartPointers.*, and SmartPointers.md.
• Callable ownership and inline targets: Source/Essentials/FunctionObjects.*.
• Engine strings and the build-wide inline-capacity contract: Source/Essentials/StringObject.*; deque storage: DequeObject.*; aliases and stream interop: Containers.h.
• OS-call confinement and dispatch: Source/Essentials/WinApi.*, Posix.*, and Platform.*.
• Cross-platform random sequences: Source/Essentials/RandomGenerator.*; coarse and precise waits: Threading.*.
• File bytes and low-level writable-path composition on disk: Source/Essentials/DiskFileSystem.*; mounted engine resources and installed-client overlays: Configuration and Data Sources.
• Socket primitives: Source/Essentials/NetSockets.*; protocol/command/network runtime: Networking.

Validation checklist

1. Confirm the change does not introduce a dependency from essentials back into higher engine layers.
2. Update BuildTools/cmake/stages/EngineSources.cmake when adding/removing essentials files.
3. Run the smallest matching essentials test and then the broader RunUnitTests target when behavior crosses utility boundaries.
4. For diagnostics changes, also verify Native, AngelScript, and Managed Debugging stays accurate.
5. For filesystem/socket/threading changes, validate at least one higher-level consumer if the low-level contract changed.
6. For a small_vector adoption, prove capacity and object-count assumptions with data, audit move/address lifetime and exact-type boundaries, and re-run exception-safety plus pointer-ownership gates.