Persistence

This document explains the server-side database abstraction, collection/key model, commit queue, backend-consistent snapshots, recovery logs, and backend implementations.

Use it when changing Source/Server/DataBase.*, database settings, entity save/load code, or persistence tests.

DataBase-Json.cpp publishes complete inserted and updated JSON records through fs::write_file; unique sibling staging replaces the former fixed <record>.tmp sequence. Record keys, JSON content and commit-queue semantics remain unchanged. SQLite snapshot restore uses the same atomic helper for its unpublished scratch database before opening it for restore.

Ownership model

The engine owns the reusable database abstraction, backend implementations, and source-backed recovery boundaries. Backup and Recovery defines the provider-neutral backup/restore procedure; an embedding project owns deployment choices, connection strings, provider tooling, schedules, retention, RPO/RTO, data migration policy, and production authority.

Do not put live credentials, production connection strings, or host-specific recovery steps in this engine document.

Graceful shutdown and entity ownership

Persistent entity-holder properties describe durable ownership. ServerEngine::Shutdown must leave those IDs and custom-entity documents intact. EntityManager::DestroyAllEntities clears runtime inner references while the registry still retains every entity, then drops parent links and registry ownership. This is distinct from ordinary DestroyCustomEntity, which updates holder IDs and deletes documents. The distinction applies to Game-owned trees and custom children of standard holders alike. ServerShutdownPreservesPersistentCustomEntityTreesAcrossRestarts exercises the boundary through two real file-backed restarts; a live-world read before shutdown is not durability evidence.

Public database facade

Source/Server/DataBase.h defines the public facade DataBase. It wraps a DataBaseImpl backend and exposes collection/document operations:

• state/metrics: InValidState(), GetDbRequestsPerMinute();
• enumeration: GetAllIds(), GetAllIntIds(), GetAllStringIds();
• reads: Get(), GetMany(), Valid();
• writes: Insert(), TryInsert(), Update(), CompareExchange(), Delete();
• commit control: StartCommitChanges(), WaitCommitChanges(), ClearChanges();
• backend snapshot: CreateSnapshot() and RestoreSnapshot(bytes);
• debug UI: DrawGui().

ConnectToDataBase() constructs the facade from settings, connection info, collection schemas, and a panic callback. JSON and DbSQLite use the trimmed entire remainder after the provider name as their storage directory, so JSON C:/Game Data/world and DbSQLite C:/Game Data/world preserve internal spaces. Do not put literal quote characters around that remainder. Empty directories fail; Memory and Mongo retain their token grammar.

GetMany(collection, ids) reads several records of one collection through one backend GetRecords() request and returns documents aligned with the requested ids: a missing record produces an empty document and a repeated id repeats the same result. Each document follows the Get() contract, with pending commit operations overlaid; only a record committed during the batch read is fetched again. A commit stops being overlaid when its backend write completes and it leaves the pending queue. If that occurs after a read fetched the old record but before it checked the queue, the affected record is fetched again; unaffected records are not. Get() itself uses the same path for one id.

Active readers share a record generation, with each backend request capturing its own expected generation. Commit advances it under the same lock that removes the pending operation. Another reader cannot replace an invalidated read’s evidence of that commit. The tracking entry is removed only when its last reader leaves, so completed historical records do not accumulate. DataBaseConditionalReadCannotBorrowAnotherReadersCommitMarker pins the two-reader/commit ordering as well as CAS refusal of the old value.

GetAllIds(collection) and its typed GetAllIntIds() / GetAllStringIds() variants enumerate backend keys with pending inserts and deletes overlaid in queue order, matching what Get() can read. A pending delete removes a listed key; a pending insert adds it. If an insert or delete for that collection commits while backend keys are being listed, the listing is retried. Updates do not change the key set. DataBaseGetAllDocumentIdsAppliesPendingInsertsAndDeletes and DataBaseGetAllDocumentIdsRelistsKeysCommittedDuringListing cover the overlay and concurrent commit.

Conditional mutations

TryInsert(collection, id, document) returns false when the logical record already exists. CompareExchange(collection, id, key, expected, desired) changes an existing field only when its typed value equals expected; a missing record, missing field or mismatch returns false without queuing a write. Both include pending inserts, updates and deletes in the condition. Invalid collection/key types, empty insert documents and non-finite desired values still throw.

One producer mutex per DataBaseImpl holds the condition check and enqueue together and serializes them with ordinary inserts/updates/deletes, RestoreSnapshot() and ClearChanges(). The commit thread takes only its existing queue/backend locks. A true result means an operation entered the logical queue, not that it was durably committed: StartCommitChanges() enables processing and WaitCommitChanges() retains its existing failure/recovery limits. This is neither a cross-instance/cross-process transaction nor a backend-native Mongo/SQLite compare-and-swap. Consumers must keep a single owning instance for records needing this guarantee.

Server scripts expose Game.DbTryInsertRecord(collection, id, keyValues) and Game.DbCompareExchangeRecordString(collection, id, key, expected, desired) for both integer and string ids. These string-field methods validate collection/id, nonempty insert values/key and UTF-8 before delegating to the shared database path. They require no live entity cover. Existing DbInsertRecord duplicate-error behavior is unchanged.

Collections and keys

The database layer stores AnyData::Document values in named collections.

DataBaseImpl validates documents recursively before queuing inserts or updates and rejects non-finite Float64 values in nested documents and arrays. JSON and BSON conversion applies the same rule in both directions, so invalid floating-point data fails at the persistence boundary instead of entering storage or runtime state.

Core types:

• DataBaseKeyType — IntId or String.
• DataBaseStringKeyEscaping — Raw, File, or Hex.
• DataBaseKey — variant<ident_t, string>.
• DataBaseCollection — map from DataBaseKey to AnyData::Document.
• DataBaseCollectionSchema — pair of collection name and key type.
• DataBaseCollectionSchemas — list of collection schemas used at initialization.

DataBaseImpl::ValidateCollectionKey() enforces that collection schemas and record IDs agree. When adding a new persistent collection, add the schema at the server/entity-manager layer and validate all backend implementations.

Backend interface

DataBaseImpl is the backend base class. Backends must implement:

• GetStringKeyEscaping();
• GetAllRecordIds();
• EnsureCollection();
• GetRecord();
• InsertRecord();
• UpdateRecord();
• DeleteRecord().

Backends can override:

• GetRecords() for a batched read. Mongo uses _id: {$in: [...]} chunks of at most 1000 ids with matching batch size/limit, SQLite uses one key IN (...) statement per 1000 ids, and Memory/JSON read the batch under one storage lock. The base implementation falls back to one GetRecord() per id;
• CreateSnapshotData() and RestoreSnapshotData() when the backend can represent its whole content as bytes;
• TryReconnect();
• DrawGui();
• test hooks such as OnCommitOperationWrittenToOpLog() and OnPendingChangesRestored().

Provider factories live in adjacent guarded headers and source pairs:

• DataBaseMemory::Create() in DataBase-Memory.h, always available;
• DataBaseJson::Create() in DataBase-Json.h, with FO_DATABASE_JSON;
• DataBaseSQLite::Create() in DataBase-SQLite.h, with FO_DATABASE_SQLITE;
• DataBaseMongo::Create() in DataBase-Mongo.h, with FO_DATABASE_MONGO.

DataBase.cpp owns dispatch, commit/recovery, keys and metrics. It includes every provider header unconditionally; a disabled provider has no declarations or implementation. DataBaseSerialization.h/.cpp owns the neutral JSON/CBOR document conversion. See component selection.

Commit queue

Writes are represented as commit operations:

• Insert
• Update
• Delete

DataBaseImpl queues pending commit operations and processes them through commit-thread machinery:

• StartCommitChanges() schedules/starts commit processing;
• WaitCommitChanges() waits for the commit thread to drain;
• ClearChanges() clears pending state;
• CommitNextChange() applies one operation;
• CommitThreadEntry() runs the background loop.

The public DataBase facade forwards write calls into this machinery. Backend write implementations should remain focused on durable record operations, while shared logic handles scheduling, operation logs, panic/retry policy, and metrics.

Backend-consistent snapshots

DataBase::CreateSnapshot() returns the whole database content as bytes and RestoreSnapshot(bytes) puts such a content back. The database never names a file, creates a directory, or decides where a snapshot lives: storing the bytes is the caller’s business. Shared DataBaseImpl logic requires an active commit thread for a capture, drains the pending commit queue, rejects a failed backend, then blocks new Insert() / Update() / Delete() producers until the backend operation returns or throws. A restore requires an already drained queue and blocks producers the same way. Concurrent snapshot operations serialize. Reads may continue, subject to the backend’s own storage lock.

A backend without a byte representation throws DataBaseException in both directions, so an unsupported backend fails loudly instead of returning an empty snapshot. These operations do not select slot names, publish packages, write manifests, capture Engine/runtime state, or authorize a save; those policies belong to the embedding session controller. The caller must first stop gameplay/runtime producers and materialize any exact state that needs to enter the commit queue.

DbSQLite::CreateSnapshotData() serializes the database with sqlite3_serialize() while holding the source storage lock, so the bytes are the exact page image the database would have on disk with WAL content folded in. RestoreSnapshotData() loads those bytes into a private in-memory database with sqlite3_deserialize() and copies it into the live storage with the online-backup API: deserializing straight into the live handle would detach it from its file. The buffer handed to sqlite3_deserialize() is allocated with sqlite3_malloc64() and freed on close.

ServerEngine::CreateSnapshot() is the higher Engine-state composition for a stable authoritative world. It first reaches quiescence and returns typed blockers for runtime-only script contexts, delayed closures, time events, or movement; only a ready world flushes exact time/id, creates the backend payload, and writes the paired versioned RNG/compatibility manifest. This does not change the narrower DataBase contract and is not atomic slot publication. Restore callers strictly read the manifest, copy the immutable database payload to an isolated live directory, and pass its state into a fresh ServerEngine; opening a selected snapshot directly as writable live storage is outside the supported contract.

DataBaseSnapshotDrainsAndBlocksNewProducers pins the shared commit/producer barrier. SQLiteDataBaseCreatesReopenableSnapshotWithoutOverwriting pins pending-write inclusion, later-source-mutation exclusion, live-source independent reopen, no-overwrite behavior, and a failed destination that leaves no database file.

Recovery logs and panic policy

DataBaseImpl::RecoveryLogHandle owns an operation-log file:

• GetPath();
• GetLinesCount();
• GetTextSize();
• GetContent();
• Append();
• Truncate().

DataBaseImpl can keep pending and committed change logs:

• _pendingChangesLog
• _committedChangesLog

When operation logging is enabled, DataBase.OpLogPath must be non-empty. The configured path owns the pending log; the committed-progress path is derived by replacing .oplog with -committed.oplog. InitializeOpLogs() does not enforce that suffix, so keep the conventional .oplog name and verify that the derived path is distinct rather than relying on an arbitrary filename.

Recovery/panic settings include:

• _pendingChangesPanicThreshold
• _panicShutdownTimeout
• _reconnectRetryPeriod
• _panicCallback

Relevant methods:

• InitializeOpLogs();
• RestorePendingChanges();
• StartPanic();
• TryReconnect().

When changing commit durability, validate failed-write recovery and pending-log restoration, not only successful writes.

Backend-specific notes

• JSON backend: file/directory-oriented storage and string-key escaping suitable for filesystem paths.
• SQLite backend: enabled only when the build has FO_DATABASE_SQLITE, which is server-only — clients link no embedded database. Every collection is a table inside one Storage.sqlite file, journalled in WAL mode, SQLite allocates through the engine memory system via SQLITE_CONFIG_MALLOC, and CreateSnapshot() returns the serialized page image rather than copying the live files.
• Mongo backend: enabled only when the build has FO_DATABASE_MONGO; only this provider owns the BSON conversion and allocator setup, in DataBase-Mongo.cpp.
• Memory backend: useful for tests and non-durable runtime paths.

DocumentToBson() and BsonToDocument() are private Mongo helpers. JSON uses ordinary JSON text and SQLite uses CBOR BLOBs through DataBaseSerialization. GetDbKeyType() reports whether a runtime key is integer- or string-backed.

Storage formats and upgrade boundary

JSON files contain ordinary UTF-8 objects; SQLite record BLOBs contain CBOR objects. Both preserve signed Int64 (including limits and values above 2^53), finite Float64, Bool, String, Array and Dict, including embedded NUL in strings/keys. Null/binary, unsigned overflow, non-object roots, malformed/truncated input and trailing CBOR bytes fail. Limits: 64 MiB per record and 64 nesting levels. CBOR SAX validation checks types/depth/declared lengths before DOM allocation; validation precedes writes.

Use empty JSON/SQLite storage: old EJSON/BSON has no reader/converter. Preserve old binaries/storage for rollback. Mongo BSON/Memory semantics and SQLite tables/keys/WAL/snapshots remain unchanged; snapshots require matching current metadata/binaries. Components defines provider selection; backup/recovery defines consistent sets. Guarded Test_DataBase.cpp checks codecs, reopen/update/delete and recovery.

Relationship to entity state

Persistence stores documents; entity state reaches those documents through property serialization and server entity-management code.

Relevant entity/property concepts from Entity Model:

• persistent property flags;
• temporary property exclusion;
• base/overlay property data;
• ExplicitlyPersistent;
• custom holder IDs/entries;
• prototype-derived runtime state.

Do not add database-specific assumptions to Entity or Properties unless all backends and tests can support the behavior.

Metrics and diagnostics

GetDbRequestsPerMinute() reports recent database request volume using per-second buckets; a batch counts once per backend GetRecords() call, regardless of its record count. Backend failures and reconnect attempts are tracked in DataBaseImpl state.

DrawGui() is available at both facade and backend levels for debug/inspection UI.

Tests to inspect

Relevant tests include:

• Source/Tests/Test_DataBase.cpp
• entity-management tests such as Test_LocationAndEntityMgmt.cpp when persistence affects saved entities;
• backend-specific tests when enabled by build options.

The hidden MongoDataBaseRecoversAfterIsolatedServerRestart test requires FO_TEST_MONGO_URI (loopback) and FO_TEST_MONGO_CONTROL (an isolated writable directory). A coordinator waits for ready, stops its owned mongod, writes stopped, waits for spilled, restarts that instance and writes restarted; complete confirms replay and reopen.

Change routing

• Public facade and shared commit/recovery logic: Source/Server/DataBase.h and Source/Server/DataBase.cpp.
• JSON backend: Source/Server/DataBase-Json.cpp.
• SQLite backend: Source/Server/DataBase-SQLite.cpp.
• Mongo-only BSON allocator/conversion: Source/Server/DataBase-Mongo.cpp, InitializeBsonMemory(), DocumentToBson(), and BsonToDocument(); Mongo-specific operations stay in Source/Server/DataBase-Mongo.cpp.
• Memory backend: Source/Server/DataBase-Memory.cpp.
• Entity/property serialization: Entity Model and PropertiesSerializer.*.
• Build feature toggles: Build Workflow and BuildTools Pipeline.
• Backup sets, oplog limits, isolated restore, and disaster-recovery drills: Backup and Recovery. Deployment stop/rollback sequencing: Release Operations. Provider choice, schedules, retention, RPO/RTO, concrete migrations, and production authority remain project-owned.

Validation checklist

1. Run Source/Tests/Test_DataBase.cpp or the embedding project’s equivalent database test target.
2. Validate insert, update, delete, get, valid, and ID enumeration for every affected backend.
3. Validate integer-key and string-key collections when changing key handling.
4. Validate commit queue drain with StartCommitChanges() / WaitCommitChanges().
5. Validate operation-log restore after a simulated failed commit when durability/recovery behavior changes.
6. Validate that an enabled DataBase.OpLogPath is non-empty and that replacing .oplog with -committed.oplog produces a distinct committed-log path.
7. For snapshot changes, prove pending changes are included, later source mutations are excluded, the destination reopens while the source remains live, completed destinations are not overwritten, and failed destinations leave no database artifact.
8. Validate entity save/load paths when persistent property semantics change.
9. Never put production credentials or live connection strings into repository docs or tests.