Server Runtime

Engine-owned documentation. This page describes reusable server runtime behavior in Source/Server/; game rules, world content, concrete balance, quests, and project-specific deployment policy remain in the embedding project.

Purpose

The server runtime owns authoritative game state. It loads resources, initializes scripts and metadata, accepts network connections, creates and persists entities, validates client input, processes player/critter/map/item state, broadcasts visible changes, and runs the game loop jobs that make the world advance.

For how this state stays internally consistent when an exception is thrown mid-operation — the WorkerPool catch-and-continue model, the entity-lifecycle throw-as-signal contract, and the throw / FO_VERIFY_* / FO_STRONG_ASSERT error tiers — see ExceptionSafety.md.

Read this page together with:

• Entity Model for entity/property/prototype ownership.
• Maps, Movement, and Geometry for map positions, blockers, path finding, and movement contexts.
• Networking for command buffers, transports, and client/server message boundaries.
• Persistence for database backends and entity serialization.
• Backup and Recovery for durable backup sets, oplog limits, isolated restore, and disaster-recovery evidence.
• Client Runtime for the client-side view of server messages.
• Client Updater for the updater backend that is hosted by ServerEngine.

Source paths inspected

• Source/Server/Server.h
• Source/Server/Server.cpp
• Source/Server/EntityManager.h
• Source/Server/EntityManager.cpp
• Source/Server/MapManager.h
• Source/Server/MapManager.cpp
• Source/Server/CritterManager.h
• Source/Server/CritterManager.cpp
• Source/Server/ItemManager.h
• Source/Server/ItemManager.cpp
• Source/Server/Player.h
• Source/Server/Player.cpp
• Source/Server/Critter.h
• Source/Server/Critter.cpp
• Source/Server/Map.h
• Source/Server/Map.cpp
• Source/Server/StaticMap.h
• Source/Server/StaticMap.cpp
• Source/Server/Location.h
• Source/Server/Location.cpp
• Source/Server/Item.h
• Source/Server/Item.cpp
• Source/Server/ClientDataValidation.h
• Source/Server/ClientDataValidation.cpp
• Source/Server/UpdaterBackend.h
• Source/Server/UpdaterBackend.cpp
• Source/Server/WorkerPool.h
• Source/Server/WorkerPool.cpp
• Source/Essentials/WorkThread.h
• Source/Essentials/WorkThread.cpp
• Source/Scripting/ServerCritterScriptMethods.cpp
• Source/Scripting/ServerMapScriptMethods.cpp
• Source/Scripting/ServerPlayerScriptMethods.cpp
• Source/Tests/Test_ServerEngine.cpp
• Source/Tests/Test_Timer.cpp
• Source/Tests/Test_WorkerPool.cpp
• Source/Tests/Test_EntityLifecycle.cpp
• Source/Tests/Test_ServerItems.cpp
• Source/Tests/Test_ServerMapOperations.cpp
• Source/Tests/Test_ServerAdvancedOps.cpp
• Source/Tests/Test_ServerScriptMethods.cpp
• Source/Tests/Test_ClientServerIntegration.cpp
• Source/Tests/Test_DataBase.cpp

Runtime owner: ServerEngine

ServerEngine in Source/Server/Server.h is the server-side composition root. It derives from BaseEngine and implements EntityManagerApi, so scripts and runtime systems can create, load, destroy, and query entities through one authoritative owner.

Major responsibilities:

• load server resources through GetServerResources(GlobalSettings&);
• initialize storage, metadata, language packs, maps, client packs, scripts, networking, and game logic;
• run the server job loop and frame-time synchronization;
• accept network connections and create unlogged players;
• process handshake, ping, command, movement, direction, property, and remote-call messages;
• create, load, unload, destroy, and switch critters;
• move critters by paths and movement contexts;
• dispatch entity lifecycle and gameplay events to scripts;
• persist entity/property changes through DataBase and PropertiesSerializer;
• host UpdaterBackend for client resource/runtime updates;
• publish health information and optional health-file output.

ServerEngine is intentionally authoritative: client views can request movement, commands, property changes, and remote calls, but the server validates and applies the state that matters.

Initialization and server jobs

ServerEngine startup is organized as scheduled jobs rather than one monolithic constructor. The private job list in Source/Server/Server.h shows the runtime phases:

• InitHealthFileJob()
• InitScriptSystemJob()
• InitNetworkingJob()
• InitStorageJob()
• InitMetadataJob()
• InitLanguageJob()
• InitMapsJob()
• InitClientPacksJob()
• InitGameLogicJob()
• InitDoneJob()
• SyncPointJob()
• FrameTimeJob()
• TimeEventJob()
• NotLoggedInPlayerJob()
• PlayerJob()
• CritterMovingJob()

InitNetworkingJob() validates ServerNetwork.ChannelSecretKey and constructs the server’s static secure-channel identity before accepting any connection. This is required even when external listeners are disabled: interthread and test connections use the same channel. It logs only the public key. The receive path latches an invalid Noise frame under its buffer lock; the owning worker later closes that connection with ProtocolError. See Networking.

After startup, only SyncPointJob() and FrameTimeJob() recur on the main worker. Time events, connection processing, and critter movement use keyed, self-rescheduling WorkerPool jobs; data-arrival callbacks wake the associated connection key without restoring aggregate per-frame polling.

Startup runs on the _starter worker thread, so a failure surfaces asynchronously. If any mandatory init job throws — for example InitStorageJob() when the database is unreachable — the starter’s exception handler reports the exception, sets IsStartingError(), and clears the remaining jobs: IsStarted() never becomes true, and the worker pool, database connection, and time synchronization are never established. The handler owns that reporting because WorkThread reports a job exception itself only when no handler is registered. Host apps must observe this rather than block forever: ServerHeadlessApp, ServerDaemonApp, and ServerServiceApp wait on IsQuitRequested() || IsStartingError() and turn a start error into a non-success quit, instead of leaving the process listening but non-functional (the failure mode behind a Staging incident where a down MongoDB left the headless server half-initialized for hours). Shutdown() is correspondingly safe to call on a partially-initialized engine: the worker-pool drain and the database / sync-time flushes are gated on reached_running_state (the presence of _workerPool, which is created last in InitMetadataJob after the DB connect and time-sync), so an aborted startup tears down cleanly instead of dereferencing the null pool (WorkerPool::Clear locking a null pool’s mutex) or tripping the connected/synchronized invariants. Source/Tests/Test_ServerEngine.cpp (ServerEngineShutdownIsSafeAfterStartupFailure) pins this by forcing an unrecognized DbStorage, asserting the start error, and requiring Shutdown() to complete without crashing.

InitGameLogicJob() calls EntityManager::InitEntityIdBoundary() as soon as the globals document is loaded, before OnInit and before either world generation or persisted-world restoration. A generated world draws the next id above max(stored LastEntityId, Server.EntityStartId); an explicit snapshot restore retains its exact stored boundary. Keep Server.EntityStartId above every map-authored id, because static and runtime items share the client item index and an id collision would replace authored scenery.

Release Operations turns these startup/error/shutdown states, log markers, and optional health-file output into a deployment readiness and rollback procedure. It does not make process existence or an open listener equivalent to readiness.

The public Lock() / Unlock() pair is used by tests, tooling, and controlled operations that need a consistent view of server state. Source/Tests/Test_ServerEngine.cpp repeatedly waits for server startup, locks the server, performs entity/script checks, and unlocks on scope exit.

The frame path stays lock-free: GameTimer publishes its pause flag and accumulated offset as atomics, and the mutex only keeps Pause() and Resume() exclusive with each other. WorkerPool counts anonymous scheduled jobs incrementally rather than scanning its queue, because diagnostics run periodically while the health file is enabled.

RunInQuiescence() is the stronger reusable boundary for an authoritative capture operation. It must be called from outside a server execution/synchronization context. The Engine serializes the operation, closes new connection admission, reaches the existing SyncPoint() and drained WorkerPool pause, freezes GameTimer frame/synchronized time and the worker scheduling clock, covers every registered server entity plus current not-logged-in players, and invokes the caller callback with the captured synchronized time and the four-word random_generator state. Existing connection buffers remain transport state, but their player jobs cannot apply gameplay input while the pool is paused. New accepted transport connections are disconnected until admission reopens.

The callback boundary is exception-safe: scheduling time, game time, the whole-world synchronization context, the worker/main sync point, and admission are unwound in order when the callback returns or throws. Server shutdown is serialized with quiescence and waits for an active callback; shutdown from inside the callback is rejected before lock acquisition to avoid self-deadlock. Delayed jobs retain their remaining scheduling delay across the pause, including jobs submitted while scheduling time is frozen; an explicit Wake() during the pause makes its keyed job due when execution resumes. GameTimer::FrameAdvance() is inert while paused and removes the paused wall-clock interval from later frame/synchronized-time projection.

ServerEngine::CreateSnapshot() is the first storage composition built on that boundary. Inside quiescence it inspects AngelScript context diagnostics, anonymous delayed worker callbacks, runtime time events, and active critter movements. Any retained state produces counted ServerSnapshotBlockerKind entries and no payload. A ready world flushes the exact generated-entity id and synchronized time, invokes DataBase::CreateSnapshot(), and returns the payload bytes together with a ServerSnapshotState carrying compatibility and metadata versions, exact synchronized time, id boundary, and the four generator words. The Engine writes no file and defines no container format: naming, versioning, packaging and atomic publication belong to the embedder. Restoring is the mirror image — ServerEngine is constructed with a ServerSnapshotRestore pair, loads the payload into storage before anything reads the world, and rejects a state that disagrees with its payload before gameplay hooks run.

ReadServerSnapshotState() parses that manifest strictly and requires the SQLite payload. The embedding controller must copy the immutable payload into a separate writable live-session directory before construction. Passing the parsed state to ServerEngine validates format/compatibility/metadata synchronously and restores RNG before startup jobs; after the copied database globals load, exact time and id are cross-checked before script module/init hooks run. A corrupt or mismatched pair fails startup rather than combining manifest state with a different database.

This is still a staging primitive rather than a player-facing slot/checkpoint system. Version 1 blocks every runtime time event and movement instead of serializing them, cannot classify project-owned state, and does not choose slot names, calculate package integrity, atomically publish/rotate saves, own pause/UI authority, or coordinate network reload and identity. Lock() / Unlock() retain their existing diagnostic semantics and do not freeze clocks; callers needing only a custom frozen operation use RunInQuiescence(), while stable Engine/database capture uses CreateSnapshot(). ServerEngineQuiescenceFreezesAndCleansUp, ServerEngineSnapshotEligibilityRejectsRuntimeOnlyState, ServerEngineSnapshotRoundTripsThroughFreshSQLiteSession, PauseFreezesFrameAndSynchronizedTime, and WorkerPoolSchedulingTimeFreezePreservesDelays pin the source contract.

Script-exported map critter queries (Map.GetCritters(...), “who sees” variants, and property-filtered lookups) rely on the map access validation performed by the script dispatch layer: callers must already hold map coverage, and concurrent map membership mutation under that cover is a bug to surface rather than mask by taking extra critter refs.

Enumerating independent roots so the caller can cover them. Three things a native call graph touches are not reachable through the map/location ancestry the caller already holds, so the script cannot derive them from the entities it covers — the engine has to let it read them first:

• Map spectators. MapManager::DestroyMapInternal() ejects each spectator with ValidateEntityAccess(player) + Player::ResetViewMap(), and a spectator Player is a separate root, not a map descendant. Map.GetSpectatorPlayers() returns the map’s current spectators (the _spectatorLock-guarded owning snapshot also used by the broadcast fan-out) so a caller preparing a map/location destroy can cover them and re-read the list to prove the membership did not change while it was acquiring that cover.
• Global-map group members. A travelling critter’s group members are independent Critter roots that its own cover does not include, and each Send_AddCritter(member) validates the member. ServerEngine::SendCritterInitialInfo() therefore does not fan the group out: it sends only the critter itself, and the group fan-out is the separate script-driven Critter.SendGlobalMapGroupInfo() export, which validates every member of Critter::GetGlobalMapGroup() before its first send (so an incomplete cover throws instead of half-delivering the group) and throws for a mapped critter. This split exists because the critter to attach is commonly chosen inside the login/attach callback, after the caller’s cover was prepared — the caller could not have covered a group it did not yet know about. Critter.GetGlobalMapCritterIds(uint64& revision) returns the member ids plus the group’s membership revision (empty with revision 0 for a mapped critter). Membership lives in a shared GlobalMapGroup object — one instance per group, held by every member — whose shared_mutex makes a read taken under one member’s cover safe against a join/leave performed under another member’s cover, and whose revision advances on every membership change. A caller resolves the reported ids, covers them, then re-reads ids and revision; an unchanged pair proves the cover it acquired is the current membership. Critter::GetGlobalMapGroup() correspondingly returns an owning copy taken under that lock rather than a live span.
• Spectated map. A spectating Player holds no parent link to the map it views, so Player.GetControlledCritter() does not reach it and the player’s own cover does not include it. Player.GetViewMapTarget() returns the map the player currently spectates (null when it spectates none), so a caller rebuilding a player’s dependency graph — reconnect above all — covers that map and re-reads the handle to prove the view did not change while it was acquiring that cover.

None of these exports changes cover on the script’s behalf: all three are ordinary reads under the receiver’s cover, which the script dispatch layer validates before the export body runs.

Property serialization through Properties::StoreData() returns pointer/size lists backed by the entity’s live property storage plus the per-call send cache. Callers copy those chunks immediately into network or persistence buffers and must already hold the entity cover that protects property reads. ThreadSanitizer builds annotate the engine’s custom EntityLock so this external cover is visible to the sanitizer without adding per-property snapshot copies.

Sync-free server→client broadcasts (the “rassylka”). A broadcast to observers/spectators is the one place where a sender legitimately cannot hold the recipient’s cover — the fan-out runs under the subject critter’s (or map’s) cover only, then must reach every recipient’s player. The whole broadcast surface is sync-free: property, movement (Send_Moving/Send_MovingSpeed), facing (Send_Dir), action (Send_Action), inventory move (Send_MoveItem), teleport (Send_Teleport), and attachments (Send_Attachments), plus their serialization helpers SendItem/SendInnerEntities/SendCritterMoving. The pattern:

• Recipient sends validate the SUBJECT, not the recipient — and this always carries its own explicit marker. Two independent decisions, both stated at the top of every send: (1) the this-marker declares how the method treats its own entity — a recipient send writes only the recipient’s connection and never reads recipient state, so its this-marker is FO_NO_VALIDATE_ENTITY_ACCESS(). This marker is mandatory and is not implied by the value check — FO_VALIDATE_ENTITY_ACCESS_VALUE(x) validates x, it is not a this-decision, so every send pairs the two: FO_NO_VALIDATE_ENTITY_ACCESS(); then FO_VALIDATE_ENTITY_ACCESS_VALUE(subject);. (2) The subject validation: every send is handed an entity it serializes or reads — even just its id — and validates it via FO_VALIDATE_ENTITY_ACCESS_VALUE(subject) (= the null-tolerant throwing ValidateEntityAccess(subject)). The subject must be in sync, and the broadcaster holds its cover for the whole fan-out so the check passes — an uncovered subject is caught (a ScriptException reported at the job/script frontier, which continues; escaping a noexcept send still terminates the process). This is intentionally aggressive diagnostic validation: validate every sent entity to surface every desync immediately (this validation layer is temporary and will be removed after the multithreaded logic system stabilizes; see the TODO below). The recipient connection is guarded by a per-player _connectionLock so a concurrent reconnect SwapConnection cannot swap _connection mid-write. It is a plain mutex, not a shared_mutex: same-player sends already serialize on the connection’s own single output-buffer lock (ServerConnection::_outBufLocker, held by the OutBufAccessor for the whole WriteMsg), so a shared “many concurrent send readers” lock would buy nothing — and mutex::lock() is cheaper on the hot path than shared_mutex::lock_shared(). Cross-player concurrency (the actual win) comes from each player owning its own lock; sends and SwapConnection both take it exclusively, and no send re-enters it (the SendItem/SendInnerEntities/SendCritterMoving helpers take the already-opened buffer as a parameter, so a non-recursive mutex cannot self-deadlock). is_chosen is a lock-free atomic identity compare against Player::_controlledCr (no deref). Only sends that are handed no entity at all — Send_TimeSync/Send_InfoMessage/Send_PlaceToGameComplete (no entity), Send_HashList (bare strings; used by the reported-hash broadcast fan-out and the handshake-time full-list push), Send_RemoteCall (a name + opaque payload; the outbound remote-call channel — the recipient may be uncovered and mid-reconnect, so the send pins the live connection under _connectionLock), Send_Ping/Send_HandshakeAnswer/Send_InitData/Send_UpdateFileData (connection-stage protocol replies, isolated in Player so no code outside Player writes a player-directed NetMessage), Send_RemoveCustomEntity (a bare ident_t), and Send_SomeItems (a span — each item is validated downstream in SendItem) — are pure FO_NO_VALIDATE with no value check. (The validation gap the subject check closes: StoreData/GetRawData do not validate entity access, so a send serializing a subject through them must validate the subject explicitly.) The Critter::Send_* forwarders (per-critter “send to my own player”) follow the same two-marker shape: FO_NO_VALIDATE_ENTITY_ACCESS() for the recipient critter (this) plus FO_VALIDATE_ENTITY_ACCESS_VALUE(subject) for the forwarded subject. They must not validate the recipient critter (the old this-check fired spuriously on an uncovered NPC group member with no player during DestroyCritter cleanup — the subject being removed is covered, only the recipient was not), and they read _player atomically before forwarding.
• Fan-outs resolve a refcount-pinned recipient set. Critter::Broadcast_* / SendAndBroadcast_* and the generic SendAndBroadcast(ignore_player, player_callback) build Critter::GetBroadcastRecipients(ignore_player) under the subject cover: each observer’s player via Critter::GetPlayerForSend() (the no-validate, link-lock-protected TryAddRef accessor mirroring ServerEntity::GetParentRaw) plus map spectators via Map::GetSpectatorPlayersForSend() (a FO_NO_VALIDATE snapshot guarded by the map’s _spectatorLock shared_mutex, so the broadcaster needs neither the observer’s nor the map’s cover). The pinned vector<refcount_ptr<Player>> keeps every recipient alive through the lock-free dispatch (GetBroadcastRecipients/GetMapSpectators/GetSpectatorPlayersForSend return an owning refcount_ptr vector; ref_hold_vector is reserved for the transient copy_hold_ref(...) loop-helper use).
• Property broadcasts read the subject live. Every property broadcast trigger fans out the ordinary Player::Send_Property(type, prop, subject) to each pinned recipient: it validates the subject (FO_VALIDATE_ENTITY_ACCESS_VALUE), reads the subject’s serialized bytes live via Properties::GetRawData (safe because the broadcaster holds the subject’s cover for the whole fan-out), and writes only the recipient’s connection under _connectionLock. The triggers: critter/critter-item (Critter::Broadcast_Property), global (OnSendGlobalValue → all players), map (Map::SendProperty Map case → map critters + spectators), location (OnSendLocationValue → Map::SendProperty Location case for each map in the location → map critters + spectators), and custom entity (OnSendCustomEntityValue → viewers resolved by the covered ForEachCustomEntityView, then dispatched lock-free). (A byte-snapshot optimization — capturing the payload once under the subject’s cover and blitting it lock-free — was prototyped and reverted; the live-read fan-out is the current shape.)
• TSA-guarded. Both fine-grained locks are Clang Thread Safety Analysis capabilities (fo::mutex / fo::shared_mutex), and the state they guard carries FO_TSA_GUARDED_BY: Player::_connection FO_TSA_GUARDED_BY(_connectionLock) and Map::_spectatorPlayers FO_TSA_GUARDED_BY(_spectatorLock) (each lock declared before the field it guards). Every lock-free path holds the lock via scoped_lock/shared_lock, so TSA statically enforces the guard on exactly the threads that lack the entity cover. The accessors that legitimately reach the guarded state under the entity cover instead — the cooperative scheme TSA cannot model, and which also excludes the swap/mutation — are FO_TSA_NO_ANALYSIS with a comment: Player::GetConnection (hands the pointer to entity-cover callers), Player::SwapConnection (cross-object other->_connection swap), Map::HasSpectatorPlayers / Map::GetSpectatorPlayers (leak a span), and the single-threaded ~Map teardown invariant. The Map::AddItem / RemoveItem / SendProperty (MapItem) spectator legs route through GetSpectatorPlayersForSend() (which takes the shared lock) rather than touching _spectatorPlayers directly, so they stay TSA-clean without an escape hatch.

This is why Critter::_player, Player::_controlledCr, and Player::_sendIgnoreEntity/_sendIgnoreProperty are atomics published under their owner’s cover — the broadcaster reads them without the recipient’s cover. Message ordering survives because recipient resolution stays under the subject cover (the visibility grant MapManager::ProcessVisibleCritters inserts an observer into the reverse-visible set and the broadcast reads that set both under the subject cover, so a delta can never be enqueued ahead of its AddCritter; AddCritter ships a full snapshot, so even a reordered client message self-heals — the client decoder drops an unknown-entity message rather than faulting).

Covered-by-design exceptions (intentionally NOT sync-free). Sends that read the recipient’s own state stay validated, because that state needs the recipient’s cover by construction — they are single-recipient sends issued under that cover, not broadcast distribution: Send_LoginSuccess (serializes the recipient itself), Send_ViewMap (reads the recipient’s own _viewMap), Send_AddCritter (reads the recipient’s controlled-critter visibility mode; sent from the visibility grant / map-load / transfer, which hold the recipient’s cover — convertible only by threading the grant-computed vis-mode through its call sites). Every other Player::Send_* is recipient-lock-free (this-marker FO_NO_VALIDATE_ENTITY_ACCESS() — the recipient is never validated): every one that is handed an entity validates that subject via FO_VALIDATE_ENTITY_ACCESS_VALUE, including the ones that read only the subject’s id (Send_RemoveCritter/Send_CritterVisibilityMode/Send_RemoveItemFromMap/Send_ChosenRemoveItem/Send_Teleport) alongside the ones that serialize it (Send_Property/Send_Moving/Send_MovingSpeed/Send_Dir/Send_Action/Send_MoveItem/Send_Attachments/Send_LoadMap/Send_AddItemOnMap/Send_ChosenAddItem/Send_AddCustomEntity and the SendItem/SendInnerEntities/SendCritterMoving helpers). Only the sends handed no entity are pure FO_NO_VALIDATE with no value check: Send_RemoveCustomEntity (a bare ident_t), Send_InfoMessage, Send_PlaceToGameComplete, Send_TimeSync, and Send_SomeItems (a span — each item is validated in SendItem). Map::SendProperty’s MapItem case and the Map::AddItem/RemoveItem item-appearance loops are also covered-by-design: not pure fan-outs but per-critter notify-and-react loops (AddVisibleItem/RemoveVisibleItem + a re-entrant OnItemOnMap* event with an early-return on item-context change), which require each critter covered; their spectator legs are lock-free. The critter destructor’s teardown-invariant diagnostics likewise read only raw members (no validating accessor) so a refcount-driven ~Critter on a worker thread outside the critter’s cover cannot fault.

FO_VALIDATE_ENTITY(...) declares a method’s preconditions:

Manual server-side entity methods that read or mutate their own entity state declare the LOCKED precondition at method entry. Generated C++ property accessors validate the owning entity through FO_VALIDATE_ENTITY_ACCESS_VALUE(entity) before touching storage. Low-level raw Properties access remains reserved for serialization, loading, tooling, and paths with their own storage-access contract. Validator, persistence, and lock-mechanism internals use explicit FO_NO_VALIDATE_ENTITY_ACCESS() escape hatches so validation cannot recurse while proving cover or reporting a fault.

Two ordering contracts follow from this model. (1) Script-export functions validate an entity argument before its first property read. Property accessors are noexcept, so the throwing validator inside one cannot be recovered — an uncovered read there terminates the process. Every FO_SCRIPT_API function that reads a property of an entity argument therefore calls ValidateEntityAccess(arg) first, converting a caller’s sync-scope violation into the recoverable frontier ScriptException; the Server_Game_DestroyCritter(s) family is the reference pattern. (2) Ordinary manager entry points consume caller-owned cover; they do not acquire a missing container/holder. The script caller covers every existing destination, source holder, map/location, or destroy dependency before crossing the native boundary. Manager code may use EnsureEntitySynced() only to retain the own lock of an entity already in that package across a detach or reparent. A genuinely fresh unpublished entity is different: EntityManager::CaptureFreshEntity() uses the private EnsureFreshEntitySynced() boundary before publication, without granting access to any pre-existing dependency. An omitted existing container or holder is therefore a caller error that fails validation; native create/destroy code must not repair it by replacing or widening the cover.

Script-facing typed destruction is handle-only. Game.DestroyEntity, DestroyItem, DestroyCritter, DestroyLocation, DestroyMap, and the bulk DestroyEntities / DestroyItems / DestroyCritters variants accept a live handle or handle array; the former ident and ident[] overloads were removed at Engine revision 845bdcce4a5e3707bb7bb9a1b7d39bd313a16760. A project that stores only an id must resolve it with the matching Game.Get* lookup, narrow the nullable result, acquire the entity and required parent cover, and pass that handle. Missed id call sites fail AngelScript compilation instead of performing a second registry lookup inside destruction. Game.DestroyUnloadedCritter(ident) remains intentionally id-based because an unloaded persistent critter has no live handle; it returns successfully when the Critters collection no longer contains that id, making cleanup retries idempotent, and deletes the stored row otherwise. Critter.DestroyItem(hstring|ProtoItem[, count]) is also unchanged: it selects inventory content by prototype rather than destroying an arbitrary entity by id.

TODO: remove the entire entity-access validation system after multithreaded logic stabilization; it is a high-overhead diagnostic layer, not a permanent runtime safety mechanism. If any FO_VALIDATE_ENTITY_ACCESS* check fires, fix the owning top-level path (job dispatch, script entry, sync widening, entity creation/registration, or holder transfer) so the entity cover is acquired before the code can reach the checked access at all. Do not treat the checked method or property accessor as the synchronization boundary unless that method is itself the top-level entry point.

Connected players are processed by keyed WorkerPool jobs. OnPlayerConnected() submits a NotLoggedInPlayerJob() for the temporary player object, and OnPlayerLoggedIn() cancels the not-logged-in job key and submits the logged-in PlayerJob(). Player.HardDisconnect() and other hard-disconnect paths only mark the underlying connection as disconnected; logged-in player teardown (OnPlayerLogout, critter detach, view reset, destroyed mark, and unregister) belongs to the next PlayerJob() pass through ProcessPlayer(). Code that continues after a script-visible player event should validate possible connection/control changes, but it should not treat hard disconnect as an inline player-destruction path.

SwitchPlayerCritter() sends the new critter’s initial info before OnPlayerCritterSwitched. OnCritterSendInitialInfo can re-enter scripts and detach or switch the player again, so the switch notification is emitted only if the player still controls the same critter after initial-info scripts return. Switching to no critter sends RemoveCritter, detaches the previous chosen server-side, and sends AddCritter for the same entity as an ordinary non-chosen view. An active client therefore clears HasChosen immediately without making the still-loaded critter disappear. Initial info for a critter on the global map covers only that critter — the script attaching it delivers the rest of its travelling group with Critter.SendGlobalMapGroupInfo() (see the independent-roots section above).

Typed entity destruction has a single active owner once the target is marked Destroying. OnItemFinish, OnCritterFinish, and OnLocationFinish handlers may observe the entity and may issue redundant destroy calls, but they must not complete the same teardown inline; the native owner asserts that the entity still exists after the finish event. Map and location destruction apply the same rule across the owning pair. Once DestroyMap() marks a map as destroying, scripted events in that flow may not destroy the owning location to take over the same map; DestroyLocation() asserts that none of its maps is already in another destroy-flow before it marks them. OnMapFinish and OnMapRemoved handlers therefore run while the map still exists, but native continuation asserts that the same map and location were not destroyed behind the current owner. Map content destruction may still detach an already-Destroying non-player critter from the map without issuing another finish event; this only completes the map containment edge when the critter’s own destroy owner is still active. For the same reason, removing an item from a critter that is already Destroying (inventory teardown inside DestroyCritter) does not fire OnCritterItemMoved: the item is being destroyed with its owner rather than relocated, and re-entering scripts there would let an item-movement handler attach a new inner entity (for example a modifier StartEvent) to the already-destroying critter, which the entity layer rejects. Normal item moves on a live critter still fire the event.

WorkThread and WorkerPool each expose a raw completed-job counter through a GetDiagnostics() snapshot. ServerEngine keeps a separate throughput counter for jobs that should be visible in server stats: the _starter initialization sequence is excluded, and recurring service jobs that mostly reflect scheduler cadence (SyncPointJob, TimeEventJob, FrameTimeJob, HealthFileJob, and HealthFileWriteJob) are excluded too. The always-open Info summary reports jobs per second, jobs per minute, total completed visible jobs, and CPU load for the machine and current process. The separate Performance details panel is closed by default and expands raw per-executor job counts, worker-pool internals, and per-core system CPU load. Job throughput is the live server cadence metric. The former loop-based metrics — per-loop time statistics (average/min/max/last loop time), the loops-per-second counter (and its Tracy plot), and the Server.LoopAverageTimeInterval setting — were all removed as the server moves from loop-based to event-based execution; only the Tracy “Server jobs per second” plot remains.

FrameTimeJob updates the engine FrameTime cache on a dedicated high-frequency Server.FrameTimePeriodNs cadence. Server movement uses this cached frame time for MovingContext start times, speed changes, step advancement, and outgoing movement snapshots instead of calling nanotime::now() in those hot paths.

Every frame writes frame-time properties while holding the Game entity lock exclusively, then pumps script continuations. A microsecond-scale period therefore adds contention for every reader of Game properties without improving movement; tune this cadence as a lock and worker-load setting, not only a clock precision setting.

Managed Sync acquisition/restoration helpers publish an externally returned false to Sync.OnFailure (Action<Sync.FailureInfo>). No subscriber means no snapshot, entity-context read, or stack capture. With subscribers, each receives the same immutable operation, caller file/member/line, terminal reason/helper location, entity IDs and lifecycle snapshots, proto IDs, and managed stack. Subscriptions are synchronous and unsampled; an observer must not acquire cover, mutate gameplay state, or use async void. Its exception is reported and counted without suppressing other observers or changing the helper’s false. Successful calls, recovered internal retries, probes, and best-effort cleanup are silent; native acquisition exceptions still propagate. This event proves a particular call refused cover, not that a gameplay transaction failed or rolled back. Validate with dotnet run --project Source/Scripting/Managed/SyncTests/FOnline.Sync.Tests.csproj and the embedding project’s subscriber route.

Sync.OnRetry records reason and caller/retry sites, including nested helpers. External loops use Sync.ReportRetry(reason). No subscriber means no allocation; frequency distinguishes handoffs from later-frame waits.

CPU percentages come from Platform::GetCpuUsageSnapshot(): ServerEngine samples it about once per second and diffs consecutive snapshots. System load is the busy fraction of the whole machine (and per core); process load is this process’s share normalized to one machine’s worth of capacity, while Performance details also shows the un-normalized “process core load” (which can exceed 100% on multiple cores, like top).

The stat fields are updated on _mainWorker (inside SyncPointJob) and read only on _mainWorker itself (GetHealthInfo()) and by the visible server app’s DrawGui, which reads them behind Lock() (serialized against the main worker by the sync point). Because nothing reads them from another thread, the fields are plain (no atomics needed).

Server events

ServerEngine declares script-facing events for lifecycle, players, critters, maps, locations, items, movement, and static-item triggers. Important event groups include:

• lifecycle: OnInit, OnGenerateWorld, OnStart, OnFinish;
• player flow: OnPlayerLogin, OnPlayerLogout, OnPlayerCritterSwitched;
• player-controlled motion: OnPlayerMoveCritter, OnPlayerDirCritter;
• critter motion/lifecycle: OnCritterMoved, OnCritterStartMoving, OnCritterStopMoving, OnCritterTransfer, OnCritterPreLoad, OnCritterInit, OnCritterFinish, OnCritterLoad, OnCritterUnload;
• map/location lifecycle: OnLocationInit, OnLocationFinish, OnMapInit, OnMapFinish;
• map presence: OnMapCritterIn, OnMapCritterOut, OnGlobalMapCritterIn, OnGlobalMapCritterOut;
• item lifecycle: OnItemInit, OnItemFinish, OnCritterItemMoved;
• static item trigger: OnStaticItemWalk.

These are engine extension points. The scripts that implement actual game rules belong to the embedding project.

All three login entrypoints (LoginPlayerToNewRecord, LoginPlayerToExistentRecord, and LoginPlayerToTempSession) preserve a client-visible failure boundary specifically for OnPlayerLogin: if the event chain stops (including because a subscriber throws), the server queues EngineInfoMessage::NetLoginScriptFail on the connection that owns the active attempt before requesting its graceful disconnect. Other exceptions still unwind through the entrypoint’s rollback guard and retain the existing hard DisconnectReason::LoginFailed path. The embedding client is responsible for turning an unexpected connection failure during an unfinished login attempt into a general localized error; exception text and stack details are never sent to the client.

OnCritterPreLoad is the persistence-migration boundary. EntityManager::LoadCritter() fires it once after the critter properties, inventory, and inner entities have been restored, while the critter is registered but still detached from any map. The event precedes map/global-map entry, OnCritterInit(cr, false), visibility processing, and OnCritterLoad. New critters do not receive it. Player-bound loads mark the critter ControlledByPlayer before the event, so handlers observe the real controllable state. Map transfers are locked for the callback, so handlers can normalize persisted state and inventory without relocating the critter. The critter has no global-map group yet, and during world startup the rest of the world may be only partially restored, so handlers must confine themselves to the critter’s own persisted state and inventory — resolving, loading, or relocating other persisted entities is not supported at this boundary. A handler may explicitly destroy the restored critter as a clean migration drop: successful destruction returns null without setting the load-error flag, allowing an owning map to remove the stale id and continue startup; dropping a player-bound critter makes the direct-load wrapper throw, and pruning outside references to the dropped id (rosters, follower links) stays with the embedding project. An exception thrown by a handler stops the event chain, and EntityManager::LoadCritter() converts that stopped chain into a load error, so the database load fails instead of exposing partly migrated state; the persisted record itself is kept.

MapManager::Transfer() emits OnCritterTransfer only after the critter transfer, attached-critter transfers, and final visibility refresh finish. Nested event paths may destroy the transferred critter or previous-map argument before that final notification attempt; ValidateEntityAccess() accepts that state and event dispatch suppresses script callbacks whose entity arguments are already destroyed. While the transfer lock is held, the critter’s target map/global ownership remains an asserted invariant rather than a recoverable branch.

Script event handlers may re-enter item movement while an item is already in its committed add state. Native helpers that report a completed move therefore validate the final ownership after firing the event: AddItemToCritter() throws if the committed item no longer belongs to the target critter, CreateItemOnHex() / script Map.AddItem() throw if the created item no longer belongs to the target map hex, and MoveItem(..., Map*) returns it only if it still belongs to the target map. A partial-stack MoveItem() splits the source before delivery, and the split’s init event can re-enter scripts and destroy the destination; if it does, the helper folds the split count back into the surviving source stack and destroys the undeliverable split item, so a failed split move is lossless rather than leaving an orphaned Nowhere item. ChangeItemSlot() swap notification still attempts the second OnCritterItemMoved after the displaced-item event, even if that handler moves or destroys the original moving item; redundant or stale notifications are handled by the event path and final item ownership. Map-item add, visibility, and property broadcasts snapshot the item’s map/hex context; if OnItemOnMapAppeared, OnItemOnMapDisappeared, or OnItemOnMapChanged moves, destroys, or otherwise detaches the item from that context, the outer broadcast stops before notifying more observers or spectators. Removing an item from a holder fires events after the item has already been detached, so handlers can destroy that detached item, but ordinary script movement APIs require a current holder and do not move Nowhere items.

Walk trigger processing is scoped to the critter’s current trigger context. If OnStaticItemWalk or an item’s OnCritterWalk moves, transfers, destroys, or otherwise detaches the critter from that context, VerifyTrigger() stops processing the remaining triggers from the old map/hex. A static item the map instance has removed contributes no trigger at all, because VerifyTrigger() reads the same per-instance static overlay as the rest of the static queries.

Entity synchronization and locking

Server.SingleThreadedLogic is a fixed opt-out from the concurrent entity-cover contract. When enabled, ServerEngine pins its worker pool to one thread and finishes startup work before resuming that pool, so keyed player, connection, movement, and time-event jobs run serially. IsEntityAccessValid() and SyncContext::ValidateAccess() then accept every entity, while SyncEntities(), EnsureEntitySynced(), and EnsureFreshEntitySynced() acquire nothing. Game.Sync and Game.SyncRelease are inert; Game.Lock and Game.Unlock still take the engine singleton bucket, which is uncontended.

The setting removes cover acquisition, not entity lifetime checks. A handle captured by one job can still refer to an entity destroyed before a later job runs, so project wrappers that combine synchronization with a destroyed-entity guard must keep their liveness half. Engine instances read the mode through the entity they operate on, allowing differently configured servers to coexist in one process. Disabling the setting restores the complete multithreaded cover contract without changing scripts that retained their ordinary synchronization calls. ServerEngineSingleThreadedLogicRunsWithoutEntityCover pins the opt-out path in Source/Tests/Test_ServerEngine.cpp.

Source/Server/EntitySync.{h,cpp} implements the cover model. Every ServerEntity owns an EntityLock; a thread proves read access by holding that lock or an appropriate ancestor/widen-chain cover. Parentage mutations require the entity’s own lock directly.

Raw atomic links are used where covered identity checks must stay lock-free: ServerEntity::_parent, Critter::_player, and Player::_controlledCr. An uncovered accessor cannot safely perform a bare pointer load followed by TryAddRef(), because replacement could release the link’s last reference in between. Each link therefore has an atomic_mutex shared by load-plus-pin and replacement. The old owner is released only after the new pointer is published. A reader either pins the old target while the link still owns it or observes the new target; it never tries to resurrect a freed object. atomic_mutex is used because these accessors are noexcept and an OS mutex acquisition failure would have no reporting path.

The controlled-critter link is non-owning. Critter::DetachPlayer() clears it under the same lock before the critter can be destroyed, and ~Critter verifies the attachment is already gone. SyncContext::SyncEntities() keeps owning handles for requested and widened entities, acquires the candidate cover, then re-reads each parent/widen link under that cover. A concurrent reparent or relink invalidates the candidate and triggers a bounded recomputation rather than letting the target disappear between discovery and acquisition. Focused coverage is in ServerEngineSyncContextWidenAndAncestorCover, ServerEngineSyncContextReparentStress, and ServerEngineEntityLinkPinSurvivesConcurrentDetach.

DescendantHold is bookkeeping, not access. It prevents an ancestor writer from cutting underneath active descendant work and prevents taking a descendant below an ancestor owned exclusively by another thread. A pending exclusive writer has priority over new descendant registrations, so sibling traffic cannot starve it.

Waiters are FIFO. A per-waiter atomic state distinguishes waiting, granted, and aborted during shutdown; shared waiters batch until the first exclusive waiter. Multi-lock Ensure is all-or-nothing and escalates in global order, restoring any released ancestor recursion and descendant-hold counts exactly. Shutdown aborts parked waiters and rejects new acquisitions.

Each active SyncContext accumulates only the time its thread is parked in the atomic wait inside EntityLock::Acquire, AcquireShared, or RegisterDescendantHold. The duration is also added to every outer context in the synchronous call chain, because an outer script’s wall time includes a nested script callback’s wait. Queue insertion, uncontended acquisition, lock bookkeeping, and ordinary native/script execution are not counted as lock wait. ServerEngine::RunScriptContext() returns that accumulated duration to the scripting backend so diagnostics can separate contention from execution cost.

Holder counts use an inline linear vector because millions of entity locks usually have only a few concurrent holders and must allocate nothing while idle. Per-sync cover and held-lock lists use small_vector sized to measured common paths; owner collections remain vector where incomplete ServerEntity types prevent inline storage.

SyncContext::YieldLocks() hands all thread locks, including outer cover, to waiters and reacquires them in address order with recursion restored. Re-prove cover and re-read relations. A held Game singleton forbids it. Scripts use Game.SyncYield() via Sync.Yield(); unavailable entities defer via ScriptTask.Delay(0) so teardown can finish.

An entity being destroyed

An entity marked Destroying remains in the cover held by its destroyer’s thread until it is actually Destroyed. SyncContext::WidenEntities() retains that owner while adding other entities, so a finish handler can still operate on its subject. A foreign thread must not acquire it: waiting on the destroyer could close a lock cycle. Managed Sync helpers accept a destroying handle only when Game.IsEntityLocked confirms current-thread cover; a destroyed handle is always unavailable. Lifecycle races that end in an unavailable handle return false without publishing a failure diagnostic; unrelated synchronization failures still report. ServerSyncWidenKeepsHeldEntityBeingDestroyed and the managed Sync harness pin both sides of this boundary.

Entity ownership and persistence

EntityManager (Source/Server/EntityManager.h) is the central registry and persistence boundary for server entities.

It owns:

• loading persisted locations, maps, critters, items, custom entities, and inner entities;
• registering/unregistering players, locations, maps, critters, items, and custom entities;
• persistent/non-persistent state through MakePersistent() and recursive persistence helpers;
• entity destruction and inner-entity destruction;
• custom entity creation/loading/view enumeration;
• entity document storage through StoreEntityDoc() and LoadEntityDoc() / LoadEntityDocs().

Item trees are restored level by level. LoadItems() uses one DataBase::GetMany() for every container nesting level, registers that batch, and then loads its children as the next level. A critter inventory or map item tree therefore costs one database request per depth (or one query per 1000 ids of that depth on Mongo), not one sequential request per item. A missing record is logged, marks the load failed, and is pruned from its holder while sibling records from the same batch still restore.

Custom inner entities are batched per holder entry in the same way: LoadInnerEntitiesEntry() passes the id list to LoadCustomEntities(), and a missing record is pruned without discarding siblings.

Custom entities held directly by the global game object share its singleton EntityLock. Managed scripts take that lock through using GameLock scope = GameLock.Acquire();; raw Game.Lock() / Game.Unlock() are reserved for the wrapper. When an engine operation synchronizes one of those entities inside the scope, the current context reuses the singleton acquisition instead of tracking the same physical lock twice, and scope disposal releases it completely.

Entity changes are persisted when relevant properties are saved by ServerEngine::OnSaveEntityValue() through PropertiesSerializer. The database facade and backends are documented in Persistence.

A persisted entity whose proto resolves through a Proto <Type> <Name> __remove__ migration rule is dropped on load: CheckMigrationRule() represents the __remove__ deletion token as an engaged optional containing an empty hash, while nullopt continues to mean that no rule exists. LoadEntityDoc() detects that empty replacement and returns an empty document without flagging a load error, so each loader (LoadCritter / LoadItem / LoadLocation / LoadMap) returns null and its owner removes the id from its child list while the rest of the load continues. OnCritterPreLoad destruction is the script-controlled equivalent for a fully restored critter and also returns null without a load error after DestroyCritter() removes its persistent graph. A proto that is simply absent (covered by no migration rule) still surfaces as a fatal proto not found load error, so deliberate deletion and accidental content gaps stay distinct. A dropped critter requested directly through ServerEngine::LoadCritter() is not silently returned — the wrapper raises so a player login cannot continue without its character.

Persisted properties whose base type is a proto reference use the same distinction while their owning entity is restored. A rename stores the replacement proto id in memory. An empty replacement produced by the __remove__ deletion token clears the value only when the property is Nullable; a non-nullable property still rejects it because the embedding project must provide a valid replacement. An unknown proto with no migration rule remains an error. This conversion happens before script load events, so a script migration can repair related nullable fields after the entity is structurally loadable.

Player and connection flow

A newly accepted NetworkServerConnection enters the runtime through ServerEngine::OnNewConnection() and becomes a not-logged-in Player through CreateNotLoggedInPlayer().

The server then processes the player in two broad states:

1. Not-logged-in player — ProcessNotLoggedInPlayer() reads initial protocol messages and runs handshake/login logic.
2. Logged player — ProcessPlayer() handles normal game messages for an attached player/critter session.

Player in Source/Server/Player.h owns the server-side per-client send surface:

• login success;
• movement/direction/speed;
• map load and view-map messages;
• property updates;
• add/remove critters and items;
• chosen inventory updates;
• teleport, time sync, info messages;
• critter actions and item moves;
• place-to-game-complete;
• custom entity add/remove;
• selected item batches through Send_SomeItems().

Player also tracks its controlled critter, connection, ignored property-send pair, and optional view-map context.

Network validation and inbound messages

The server receives client messages through ServerConnection and dispatches them from ServerEngine methods such as:

• Process_Handshake()
• Process_Ping()
• Process_Move()
• Process_StopMove()
• Process_Dir()
• Process_Property()
• Process_RemoteCall()

Inbound remote-call and property payload validation is centralized in Source/Server/ClientDataValidation.h:

• ValidateInboundRemoteCallData()
• ValidateInboundPropertyData()

Source/Tests/Test_ClientDataValidation.cpp exercises invalid UTF-8, invalid enum values, non-finite floats, unknown hashed strings, invalid bools, truncated payloads, and ref-type payload validation.

ProcessPlayer() drains up to MaxMessagesPerProcessPass buffered messages per job pass in a loop, with the player synced once by PlayerJob() before the loop. Script reached from a message handler never runs in the job’s primary SyncContext: every AngelScript execution goes through the engine’s virtual RunScriptContext() hook, and ServerEngine::RunScriptContext() creates the nested ScopedSyncContext used by that script context. Event, setter, and remote-call dispatchers do not create additional synchronization scopes. When SwitchPlayerCritter() creates a new Player/Critter sync-widen pair, it recursively retains both locks in every active context that already owns either half before publishing the link. An ancestor that entered with only the player therefore keeps continuous physical cover of the newly attached critter and its subtree after the temporary script context releases. A script Sync::Lock(...) replaces the held lock set of the current context with exactly the requested entities, so under the universal script layer it cannot drop the primary’s player cover — the player stays locked across every buffered message. Non-server engine implementations run the callback without synchronization. The engine-side handlers that re-sync the primary themselves (Process_Move() / Process_StopMove() / Process_Dir()) always include the player in the requested set, and Process_Property() does not touch the held set at all. The drain loop still re-checks player->IsDestroyed() each iteration because handler script may legitimately destroy the player. The invariant matters because a later message’s first player access goes through a noexcept accessor (e.g. Player::GetConnection() in Process_RemoteCall()): an uncovered access there would trip the access-without-sync invariant and terminate the process, since a throw cannot escape the noexcept frontier to be recovered at the job boundary.

Immediately before invoking an inbound server remote call, Process_RemoteCall() syncs its Player argument in the job’s primary SyncContext: the remote-call dispatcher creates no scope of its own, and RunScriptContext()’s nested context only opens once the script itself executes. Because SyncContext::SyncEntities() replaces the current context’s held lock set, this re-establishes the primary’s cover as exactly {player + widened critter} for the remainder of the drain loop. That sync follows Player::GetSyncWidenEntity(), so the cover includes the player’s current controlled Critter; the reverse Critter → Player link is symmetric. A server RPC may therefore read player.GetControlledCritter() and use that critter without another script-side lock or widen operation. Only independently resolved entities outside the linked pair need an explicit wider set. Repeating Sync::Lock(cr) or Sync::Widen(cr) for the same controlled critter is redundant and obscures the boundary contract.

For the wire-level model, see Networking. For client behavior, see Client Runtime.

Managers

MapManager

MapManager owns map/location creation, destruction, transfer, visibility, and map content generation:

• load map data from resources;
• create/destroy locations and maps;
• regenerate maps;
• add/remove critters to/from maps;
• transfer critters between maps or to global state;
• process visible critters and items;
• create temporary map views for players;
• calculate critter visibility modes;
• generate and destroy map content.

The reusable geometry, path finding, blockers, line tracing, and map-loading concepts are documented in Maps, Movement, and Geometry. MapManager applies those concepts to authoritative server state.

CritterManager

CritterManager owns critter creation/destruction and inventory-holder operations:

• create a critter on a map;
• destroy a critter;
• destroy a critter inventory;
• add and remove items from a critter.

Critter itself owns visibility sets, attached player/critter relationships, moving state, map-transfer locking, visible item checks, broadcasting helpers, and critter-specific script events.

ItemManager

ItemManager owns item creation, splitting, destruction, and movement between holders:

• create loose items and map items;
• add items to containers and critters;
• subtract/set critter item counts;
• split stacks;
• move items between critters, maps, and containers;
• remove item-holder relationships.

Item owns container membership and multihex entries. StaticItem is the static-map specialization used by map content. Static items are built once per ProtoMap into the shared StaticMap (Source/Server/StaticMap.h), carry the ident_t their map file authored, and are never registered, persisted, or destroyed as runtime entities. A map instance drops individual static items through its own RemovedStaticItemIds list rather than by mutating that shared data; the model, the accessors it filters, and the client half are described in Maps and Movement.

Map, location, item, and critter entities

Server entity classes combine Common-layer property/prototype behavior with server-only ownership rules:

• Location groups maps and raises OnMapAdded / OnMapRemoved.
• Map owns map fields, critter/item presence, spectators, item visibility, manual blocks, trigger verification, and OnCheckLook / OnCheckTrapLook.
• Critter owns visibility, current map/location/global state, inventory, moving state, player attachment, and broadcast helpers.
• Item owns holder/container relationships, stack/multihex behavior, and OnCritterWalk.
• Player owns connection/session state and the send surface to one client.

Do not duplicate the Common entity taxonomy here; Entity Model owns the base entity/property/prototype explanation.

Movement and authoritative state

Client movement requests enter through Process_Move(), Process_StopMove(), and Process_Dir(). The server validates the request, applies script events such as OnPlayerMoveCritter and OnPlayerDirCritter, then updates the authoritative Critter and broadcasts the resulting state. Stop-move packets include the client’s current hex and hex offset; the server normalizes that pair to a canonical in-bounds hex/offset, reconciles positions that lie on the critter’s current authoritative MovingContext path, and allows a small pathfinding-validated correction for rapid start/stop input that stopped between path centers. Normalization uses the same passability predicate as client movement: if rounding a sub-hex offset would cross into a blocked neighbor, it retains the reported logical hex and clamps the offset instead of snapping toward the blocker. This lets client and server converge without accepting arbitrary stop teleports. If the reported stop position cannot be reconciled, the server stops at its authoritative position and sends that final position back to the controlling player; only a successfully reconciled stop may omit the redundant self-update.

Process_StopMove() also fires OnPlayerDirCritter during stop reconciliation, before it can stop the active MovingContext. Scripts may hard-disconnect the connection, detach or switch the player’s controlled critter, or move the critter to another map; the native continuation revalidates those possible outcomes before applying the final stop to avoid completing a stale client command.

An arrival the client predicted is reconciled before the request behind it

Process_MoveFinished() handles SendCritterMoveFinished after a predicted plan ends. The server begins the plan one uplink transit after the client, so an action sent at arrival could otherwise be checked while the server critter is still moving. The ordered player-message loop reconciles the arrival along the authoritative path before reading the action behind it. The report identifies the plan by its end hex, and can advance only min(round trip / 2, Server.MoveFinishCatchUpMaxMs) + Server.CritterMovingPeriodMs; an excessive or stale report is refused. This is completion of an existing move, so it does not fire OnPlayerMoveCritter. Server.MoveBridgeReportHexes sets the threshold for logging the async-fix bridge on new movement requests.

A teleport ends the plan it interrupts

MapManager::Transfer stops active movement and lands the critter with zero hex offset. On receiving CritterTeleport, the client also stops its old plan and clears the offset before placing the critter; otherwise interpolation could walk it back onto the interrupted route.

A refused move request ends the route it interrupts

When Critter.MoveToHex replaces an active route, a successful new path produces one CritterMove without an intermediate stop. If the new request is refused (including zero speed, an already reached goal, or a pathfinding failure), the old route still ends through StopCritterMoving: observers receive CritterPos and OnCritterStopMoving fires. Merely stopping the server-side context left clients animating the previous route. ServerCritterMovePositionReconciliation checks both outcomes.

Server scripts can call Player.RefreshCritterMoving(cr) to resend the authoritative movement snapshot for a critter on the player’s current map. Moving critters are sent as CritterMove; stationary critters are sent as CritterPos, which lets the client stop prediction and apply the server hex, hex offset, and direction without inventing a project-specific correction packet.

Runtime movement is independent of CritterCondition: alive, knockout, dead, and any future condition use the same MovingContext processing. Game scripts own gameplay-level movement permissions and must stop or reject movement when a creature state should forbid it. Attached critters still stop their active MovingContext because attachment is a transport/ownership relationship rather than a condition.

Server-side movement helpers include:

• StartCritterMoving() overloads for an existing MovingContext or raw path data;
• StopCritterMoving();
• ChangeCritterMovingSpeed();
• CritterMovingJob() as the self-rescheduling worker-pool body for active movement;
• ProcessCritterMovingBySteps().

Source/Tests/Test_ServerEngine.cpp includes overdue movement tests that verify route completion, condition-independent movement processing, and blocked-hex stopping behavior. Coordinate/pathfinding mechanics remain documented in Maps, Movement, and Geometry.

Client update backend

During server construction, ServerEngine can create and load UpdaterBackend from client resources (Source/Server/Server.cpp, Source/Server/UpdaterBackend.*). This backend is responsible for describing and serving client resource/runtime update files to connecting clients.

UpdaterBackend responsibilities:

• scan client resources and binaries through LoadFromClientResources(const GlobalSettings&);
• build an update descriptor grouped by update targets;
• serve requested file portions through ProcessUpdateFile(ServerConnection*, int32_t);
• respond with NetMessage::UpdateFileData chunks;
• expose target-specific descriptors selected by binary target name.

The client host/runtime updater flow is documented in Client Updater. Keep protocol-level details there and runtime hosting/ownership details here.

Tests and validation map

Use the smallest relevant test scope when changing server behavior:

• Source/Tests/Test_ServerEngine.cpp — server startup, critter creation, player-controlled critter unload, script module init/events, admin remote-call allowlist, script marshalling, overdue movement.
• Source/Tests/Test_EntityLifecycle.cpp — entity init events, C++ entity/manager APIs, player registration and reconnect cover, and (IndependentRootCoverEnumeration) global-map group / map spectator enumeration plus the cover it lets a caller acquire, including that initial info leaves the group fan-out to Critter.SendGlobalMapGroupInfo() and that the export refuses to send without the members covered.
• Source/Tests/Test_ServerItems.cpp — item creation/destruction, critter inventory, critter lifecycle, entity-manager queries.
• Source/Tests/Test_ServerMapOperations.cpp — map item/critter/hex/path/static-item/location/proto/property-filter operations.
• Source/Tests/Test_ServerAdvancedOps.cpp — location creation, entity-manager bulk operations, advanced critter/item operations, utility/database/string/array/dict/math/time/proto script operations.
• Source/Tests/Test_ServerScriptMethods.cpp — server script method surface for critter inventory/state, game queries, item operations, entity lifecycle, database/text/player ids.
• Source/Tests/Test_ClientServerIntegration.cpp — client/server handshake and connection event behavior.
• Source/Tests/Test_DataBase.cpp — persistence backend behavior used by server entity storage.

Exact test target names are generated by the embedding project’s CMake/BuildTools configuration; do not hard-code one project’s target names in engine docs.

Change checklist

When changing server runtime behavior, verify:

• The changed entity type has a clear owner: EntityManager, MapManager, CritterManager, ItemManager, Player, or ServerEngine.
• Persistent state changes go through documented property/entity serialization boundaries from Persistence.
• Client-originated data is validated before mutating authoritative state.
• New or changed network messages are cross-linked in Networking and Client Runtime.
• Movement changes preserve Maps, Movement, and Geometry invariants and server blocked-hex behavior.
• Script-facing events and methods are covered by server tests and the owning scripting documentation.
• Updater changes preserve the boundary between UpdaterBackend hosting here and client host/runtime behavior in Client Updater.
• Process-host changes preserve the operational lifecycle and evidence boundary in Release Operations.