Measuring runs

To tune a room game, you play it with bots over many runs and read what happened in each: the wave the run reached, how long each wave lasted, what the players earned and lost, and what each wave cost the room and the network. spawnite play room does this with one command and no page. It runs the game’s own room once for each bot count and seed, and prints a table for each run and a table over the runs of each bot count.

This page says how a scene declares what a run records, how to run the runs, how to read the tables, and where each number comes from. Play the room with bots says how the bots play.

Declare what a run records

The scene’s file exports a timeline beside its component, its plugins and its bot. The room reads each measure before its first step and after every step, and logs each change with the step it landed on. Holdfast’s timeline, shortened:

import { createQuery } from "koota";
import {
    MeasureKind,
    WalletTrait,
    type SceneTimeline,
} from "@spawnite/engine/core";
import { LifeMachine } from "./life";
import { PhaseTrait, readPhase } from "./phase";
import { MonsterTrait, SiegeTrait } from "./traits";
import { wardens } from "./wardens";

const monsters = createQuery(MonsterTrait);

export const timeline: SceneTimeline = {
    measures: {
        wave: {
            kind: MeasureKind.Gauge,
            read: (world) =>
                world.queryFirst(SiegeTrait)?.get(SiegeTrait)?.wave ?? 0,
        },
        phase: {
            kind: MeasureKind.Gauge,
            read: (world) =>
                readPhase(world.queryFirst(PhaseTrait)) ?? "waiting",
        },
        //  The wardens down now: a counter sums each rise, so each fall.
        downs: {
            kind: MeasureKind.Counter,
            read: (world) =>
                world
                    .query(wardens)
                    .filter((warden) => warden.has(LifeMachine.is.down)).length,
        },
        monsters: {
            kind: MeasureKind.Gauge,
            read: (world) => world.query(monsters).length,
        },
    },
    parts: ["wave", "phase"],
    over: (world) => {
        const phase = readPhase(world.queryFirst(PhaseTrait));
        return phase === "over" || phase === "dawn";
    },
};

A timeline holds the following:

measures: each measure by the name that heads its column. A measure reads the room’s world and returns a value.
- A counter returns a number, such as kills or coins. The table shows how far it rose over each part, summing each rise, so a count that starts again at a new run still counts, and a level such as the wardens down counts each time it rose.
- A gauge returns a number, a word or a boolean, such as the wave, the phase or the monsters standing. The table shows its highest number, or its last word.
parts: the measures whose change starts a new part of the run. The table has a row for each part. Holdfast’s parts are the wave and the phase, so each wave’s fight and each breather are rows of their own. Left out, the run is one part.
over: whether the run is over, such as every player’s character down or the last wave held. A run ends when it holds. Left out, a run lasts its --seconds.

The room refuses a timeline of another shape as it loads the scene, naming what is wrong. A read runs after each of the room’s steps, so keep it to a query and a sum. A read that throws closes the room, naming the measure, as a system that throws does. The reads change nothing, so a run the room recorded still continues exactly.

A measure the game does not keep, such as how long a monster lives, needs state of the game’s own that a measure can read: for example a counter of damage dealt to each kind of monster beside a counter of kills of each kind, from which the mean damage to kill each kind follows.

Run the runs

From the game’s folder:

pnpm spawnite play room --bots 1-4 --seed 1-3 --speed 8

The command takes the following flags:

--bots takes a count, a list such as 1,2,4, or a range such as 1-4, from 1 to 12, and no more than the room holds, as the game’s maxPlayers or --room-env MAX_PLAYERS sets it. Each count plays its own runs.
--seed takes a seed, a list or a range. Each seed is one run for each bot count, 12 runs above. The room’s world draws its random numbers from the seed, so a run on one seed draws the same night each time. The bots’ timing still varies from run to run, so two runs on one seed differ in their details. Without --seed, each command draws one seed and prints it.
--speed runs the room’s clock that many times faster than the wall’s, up to 16. The bots keep the room’s time, so the game plays as it would at real time. Each run prints the speed the room kept up: a busy room runs slower than asked, and the numbers stay right, since every time and rate counts the game’s own seconds.
--seconds caps a run’s game time: 60 seconds by default, or 30 minutes when the scene’s timeline says when a run is over.
--invulnerable spares every player’s character, so a run plays to its last wave.
--room-env NAME=VALUE sets the room’s environment over its spawnite.room’s, such as SEND_RATE=30.
--room-heap <megabytes> sets the room’s V8 old space, 256 by default, the hosted room’s 259 MB heap, so a game that keeps more memory every wave fails here with JavaScript heap out of memory as it would in production. A larger number raises it, and none lifts it. The run’s output names the limit in force.
--out <file> keeps the runs in that file. Without it, they go to spawnite/room-runs in the system’s temp folder, which keeps the newest 20. The file is written as each run ends, so a run that fails loses none before it.
--heap-snapshot <when> has each run’s room write a heap snapshot, which Chrome’s DevTools Memory panel loads: exit as the run ends and the room stops, or a share of the heap limit such as 0.6, the first time a full collection leaves the heap past it. The snapshots go to spawnite/heap-snapshots in the system’s temp folder, which keeps the newest 6, and the command prints each one’s path. Writing one holds the room for as long as the write takes, 25 s for a 616 MB heap on a desktop, so use it to measure memory, never time.

Each run starts a fresh room on its seed. The room holds its clock until every bot has joined, so the run starts with every player in it, and then runs at --speed. The room runs unwatched, so an edit to the game’s files during a long pass does not restart it. spawnite play room report <file> prints a file of runs again, and --json carries every table as numbers.

A room that exits during a run, such as one out of memory, stops the command with its exit code and the last 20 lines it printed, where V8’s out-of-memory error sits.

Read the tables

Each run prints what each bot did, then a row for each part:

1 bot, seed 1: stopped after its 0:30 of play, at wave 2, phase fight (4.0× real time).
  Bot 1, character 15: 603 moves, 38 shots, 34 hits for 304 damage, walked 153 m; sent siege.pick ×2, siege.ready ×4.
  wave  phase     starts  lasts  kills  downs  coins  monsters peak  step ms  longest  B/s a page  peak B/s
     0  waiting     0:00   0:05      0      0      0              0     0.48     9.63       1,634     3,124
     0  breather    0:05   0:03      0      0      0              0     0.40     2.40       1,370     1,639
     1  fight       0:08   0:11      5      0      2              1     0.36     9.99       1,812     2,885
     1  breather    0:19   0:03      0      0      4              0     0.32     7.17       1,812     2,885
     2  fight       0:22   0:08      4      0      0              1     0.29     7.73       1,942     2,175
        run         0:00   0:30     10      0      6              1     0.36     9.99       1,770     3,124

A room whose heartbeat logs its heap and counts adds four columns, and the run’s head line names its heap limit. Holdfast’s room with two bots, its measure columns cut here:

2 bots, seed 4: stopped after its 0:30 of play, at wave 1, phase fight (1.0× real time), on a heap limit of 4,288 MB.
  wave  phase     starts  lasts  step ms  longest  B/s a page  peak B/s  heap MB  kept MB  entities  objects
     0  waiting     0:00   0:05     0.78     9.96       2,916     9,164      178      111        20       15
     0  breather    0:05   0:08     0.56     1.99       2,103     2,154      167      115        20       15
     1  fight       0:13   0:17     0.73     2.89       3,560     4,568      170      115        27       15
        run         0:00   0:30     0.69     9.96       3,063     9,164      178      115        27       15

Each row has the following columns:

The part measures’ values, then starts and lasts: when the part started, from the run’s start, and how long it lasted, in minutes and seconds of play.
Each other measure: a counter’s rise over the part, or a gauge’s highest, headed peak.
step ms and longest: the room’s step, on average over the part and at its longest. A step runs 60 times a second, so a room holds real time while its mean stays under 16.7 ms, and a player feels a step past that as a stall. Both leave out the replay checkpoint a recording room takes every 600 steps, which a production room never takes, so they read as the game’s own step: the room times each step without its checkpoint, so the longest is the longest step whichever step took the checkpoint.
B/s a page and peak B/s: the bytes each player’s page received a second of play, on average and in the second that sent one page the most.
heap MB and kept MB: the most heap the room held in the part, and the most a full collection left it holding, which is what the room keeps. A kept heap that climbs from wave to wave is a leak; one that nears the heap limit on the run’s head line is a room about to run out of memory. --heap-snapshot takes a snapshot to find what holds it.
entities and objects: the most entities the room’s world held in the part, and the most objects in its headless scene, which no frame draws. A count that climbs from wave to wave, such as the objects of a view that adds one on each shot and removes it only on a frame, shows the leak before the heap does.

The last row sums the whole run. To look inside a part, such as a wave that lasted far longer than the rest, read the run the room recorded at a moment of it, with no page: spawnite replay state --run <run> --at 14:25 --trait monster prints the room’s world at 14:25 of play, and spawnite replay log --run <run> what its players did, as the replay page says. Where a bot count ran more than once, a table over its runs follows: how far its runs reached and how long they lasted, then each part’s means over the runs that reached it, with a runs column that counts them.

Where the numbers come from

The room logs one JSON line on each change of the timeline, a heartbeat for each second of play, and a heartbeat as each part ends, so no heartbeat spans two parts. The heartbeat names the room’s step, the steps since the last heartbeat and their milliseconds, the bytes sent to its players since the last heartbeat, over all of them and the most to one, its heap now, after the last full collection and at its limit, and its entities and scene objects. A recording room also names its longest checkpoint since the last heartbeat, and the steps’ milliseconds again with each step’s checkpoint taken out, which the command reads in place of the logged ones. A log from a room built before it named them reads as logged, with the checkpoints in. The command prices each part from the heartbeats inside it.

The bytes are the payloads of the frames the room sent: the snapshot on join, each delta, each acknowledgement and each word to one player. WebSocket’s own framing, two to fourteen bytes a frame, is left out, as is voice, which does not travel over the socket. A page’s rate is the heartbeat’s total shared among its players, so the join’s snapshot lifts the first part’s rate.

A step’s time is the room’s own, measured in the room’s process. The bots run in a process of their own on the same machine, so a pass run beside other work reads slower steps than a room alone would.

Keep to a range of ports

Every server a command starts, such as the dev server, the build’s preview and the room, listens on a free port the system picks. To keep to a range, give --ports 8960-8979, and --room-ports 4360-4379 for the rooms, to any command, or set GAME_PORTS and GAME_ROOM_PORTS. The command gives each server the first free port of the range that it has not already given another, and says so when every port is taken. A room the cli starts runs no voice, as every development room does, so it takes no UDP port and no media worker’s CPU, and a profile measures the game without voice’s cost.

Every room the cli starts listens on this machine alone, 127.0.0.1, the room’s own default, so Windows’ firewall never asks about it.

What the other engines do

Unreal’s CSV profiler records a game’s own stats and events beside the engine’s for each frame of a headless or dedicated server, and its report tool splits a run’s summary at the events. Unreal’s Gauntlet runs automated sessions with bots and collects their metrics. Godot’s custom monitors are named functions the engine calls each frame. Unity’s profiler counters and Roblox’s server stats show the same live. The timeline takes Godot’s shape, a named read of the game, and Unreal’s split into parts. None of them prints a table of a game’s outcomes over many seeded runs from one command, which is what an agent tuning a game reads.