Playing a held session

A session that spawnite play start holds, a game’s page, a room’s page or a replay played back, takes a player’s input from a command: any key, each mouse button, the mouse’s motion and its wheel, a pad, and a finger on a phone-sized page. The pad is the tool’s: it stands one up in the page for a game that reads a pad itself, since the engine reads none. Three reads answer what a player would check by eye: what the camera sees of an entity, where a shot lands against the crosshair, and how a value moves frame by frame. When no command answers a question, spawnite play eval runs a few lines of your own against the page with the engine’s world, scene, input and time in reach.

spawnite play list, or the MCP’s list_playtests, names each playtest that runs for the games under the folder, with its address and what started it: spawnite play start, which spawnite play stop ends, or an MCP server’s start_playtest, which its stop_playtest ends. Read it before a start, to reuse a session that runs or stop one left behind. spawnite play start refuses while a session of the game that it began runs, and names that session; spawnite play start --replace stops it and starts a new one.

This page covers the input, the reads and the scripts. Directing a moment covers opening, holding, stepping and shooting a session. The MCP’s director tool runs see, aim and eval on the same session, as action, with the same reply; the input commands are the command line’s alone.

Name a control

Every input command names the control a player touches. The names are the following:

Control	Names
A key	Its `KeyboardEvent.code`, such as `KeyW`, `Digit1` or `ShiftLeft`, or Playwright’s name for it, such as `Space`, `Shift` or `r`
A mouse button	`MouseLeft`, `MouseRight`, `MouseMiddle`
The mouse’s motion and its wheel	`Mouse`, `Wheel`
A standard pad’s buttons, in its own order	`PadA`, `PadB`, `PadX`, `PadY`, `PadLB`, `PadRB`, `PadLT`, `PadRT`, `PadBack`, `PadStart`, `PadLS`, `PadRS`, `PadUp`, `PadDown`, `PadLeft`, `PadRight`, `PadHome`
A pad’s sticks	`PadLeftStick`, `PadRightStick`
A finger	`Touch1` to `Touch10`

Controls joined by + are a chord, such as Shift+KeyW: pressed in order and let go in reverse.

Press, hold, release and move

A control stays held between commands until you let it go, and the session’s time moves only when it steps or runs, as Unity’s InputTestFixture and Godot’s GUT hold input until the next update. So a key held across a step walks the player’s character for that step:

spawnite play hold KeyW
spawnite play step --seconds 1
spawnite play release KeyW

The four commands are the following:

spawnite play press <controls...> presses each control and lets it go, in order. spawnite play press Digit1 takes a card where a game binds 1 to one. Without --hold, the control goes down and up at once, which a game that listens for the press takes and one that reads the control held each frame, such as a trigger, misses: --hold 0.3 keeps it down 0.3 s of the game’s time, which a held session steps and a running one runs.
spawnite play hold <controls...> holds each control down until release lets it go.
spawnite play release [controls...] lets each control go, or with none everything held, the last held first.
spawnite play move <control> moves what moves. Mouse --by 400,0 moves the mouse 400 pixels to the right, which turns the camera while the cursor is captured, and --to x,y takes a free cursor to a point on the page. Wheel --by 100 turns the wheel a notch down. PadLeftStick --to 0,-1 stands a stick, each axis from -1 to 1, where it stays until it moves again. PadRT --to 0.5 pulls a trigger halfway. A finger held with hold slides with --to or --by.

Each command prints what is held after it, and spawnite play state prints the same under the engine’s state. --at x,y on press and hold presses a mouse button at a point of the page, in its pixels from the top-left corner, and says where a finger lands. --on <subject> on press presses on an entity instead: where it shows nearest the middle of what shows, the point spawnite play see reads, as Playwright’s locator.click() clicks an element’s middle. The subject picks one entity, and one that picks several fails naming them, as Playwright’s strict locator does; so does one that shows nothing, naming why. --page 2 sends the input to the second player’s page.

The MCP’s send_input also clicks a control by its name, such as a menu’s Play button, and a name that matches nothing fails naming the visible controls. A name reaches the game’s own controls only: the devtools’ controls, such as the Spawnite toggle at the top left, are neither clicked by name nor named in that list, so a test of the game never opens the editor by accident and what it reads is about the game. A point reaches them, --at x,y here or a point for send_input, and so does their shortcut, such as ⌘E for edit mode. Playwright’s and Testing Library’s role queries see the whole page and leave the app to scope them to a region; Unity’s and Unreal’s UI automation address the game’s own interface apart from the editor’s, and this click does the same.

On the MMORPG, a click on Mira walks the player’s character to her and opens the talk:

$ spawnite play press MouseLeft --on '[npc.name=Mira]'
Pressed MouseLeft.
Nothing is held. The cursor is free at (412, 301).

On Holdfast, a burst from her gun and a turn of the camera:

$ spawnite play press MouseLeft --hold 0.3
Pressed MouseLeft, each held 0.3 s of the game's time.
Nothing is held. The cursor is captured: the mouse turns the camera, and a button presses at the crosshair.
$ spawnite play move Mouse --by 400,0
Nothing is held. The cursor is captured: the mouse turns the camera, and a button presses at the crosshair.
$ spawnite play state
Camera: position 4.68, 2.17, 2.32; yaw 160.0°, pitch -10.0°, fov 70°; turned 160° since the loop mounted; lock on, third person.

Press what a game declares

A key reaches only what the game binds to it. A command with no key, such as an ability’s cast, or one that carries fields, has no key to press. Name it instead: press, hold and release take a name the game declares, plugin.key, in place of a control, and the page presses it as its key would. The key alone, such as strike for boss.strike, does too where one plugin declares that key; where two do, the page names both and asks for the plugin. A command goes out through sendCommand, through the entity a key bound to it would act through, so a page in a room sends it to the room, which runs a predicted one on the tick her page runs it, and the corrections play state counts mean something. Each kind of declaration takes the following:

A command: play press sends it once, with --payload as its fields, a JSON object, and {} where --payload is left out. A payload of another shape fails with the page’s refusal and exit code 1.
A held button: play press taps it for the next step, --hold 0.5 holds it half a second of the game’s time, and play hold holds it until play release.
An axis: play hold <name> --value -0.5 holds it at a value inside its range, and play release lets it back to its rest.
A page action: play hold runs its listeners’ press and play release their release, as holding its key does.

One command takes controls or declared names, never both. Where a command and a page action share a name, as Holdfast’s siege.ready does, press sends the command and hold holds the action. play release with no argument also lets go of every declared input a hold holds. A name the world does not declare fails, naming every input, command and page action it does declare. Unreal’s Enhanced Input injects an input for an action by its name the same way, rather than by its key.

On the MMORPG, played alone, the mage casts Frost Nova, which costs 25 mana and cools down for 5 s:

$ spawnite play press abilities.cast --payload '{"ability":"frost-nova","target":null}'
Sent the command abilities.cast: on a page in a room, to the room, a predicted one run on her page on the tick she sent it, and otherwise to the world's next step; its result shows in the page's command results.
The tools hold no declared input.
$ spawnite play step --seconds 2
Stepped 120 steps.
$ spawnite play eval "JSON.stringify(player().mana)"
{"current":78,"maximum":100,"reserved":0}

How the input reaches the page

Each control reaches the page as the browser hands a player’s input over:

Keys, and the mouse over a free cursor, go through Chromium’s own input, as Playwright’s keyboard and mouse send it. The events are trusted, so a focused button presses on Enter, a text box types, and a click lands on whatever stands under the cursor.
The mouse while the cursor is captured reaches the canvas as a pointer lock delivers it: every event to the canvas, at its middle where the crosshair is, with the mouse’s motion in movementX and movementY. An automated browser gets no pointer lock, since one would hold the cursor of whoever runs the session, so the engine grants the capture itself, as the engine’s state reads it in cursor.captured, and the input layer delivers the mouse as the lock would. A game reads the capture with isCursorCaptured(canvas), not document.pointerLockElement, which stays empty. Replay playback delivers a recorded move the same way.
A finger goes through Chromium’s touch input, every finger on the screen in each event, as a touch screen reports them. A finger stays on the screen: one moved past the page’s edge stands at the edge.
A pad is a standard pad the page’s navigator.getGamepads() returns, connected on its first use with a gamepadconnected event. No browser takes a pad’s input from a tool yet, so the pad lives in the page, as Unity’s InputSystem.AddDevice<Gamepad>() adds a virtual one. The engine itself reads no pad; a game that reads one does.

Inputs at moments of a step

--press on spawnite play step lands inputs on moments of the step, in seconds of its game time: control@seconds presses a control there, and control@from-to holds it between the two. A press on the step’s last moment or past its end fails the step, naming it, since no step is left to take it; a hold may end with the step. The session holds between the moments, so each lands on its step exactly. --keys holds controls down for the whole step. On Holdfast, walk forward for a second, jump at 0.3 s and fire from 0.5 s to 0.9 s:

spawnite play step --seconds 1 --press "KeyW@0-1,Space@0.3,MouseLeft@0.5-0.9"

A replay played on takes the input too, on top of the input it recorded, which it plays as it goes. What the page would send a room goes nowhere, so the input changes only what the page computes: its camera, its HUD and its own prediction.

A phone-sized page

spawnite play start --device <name> opens the session’s pages as a device Playwright names, such as "Pixel 7" or "iPhone 15": its screen’s size and pixel ratio, a touch screen, and a phone’s coarse pointer, so the game lays itself out and picks its controls as it does on a phone. On Holdfast as a Pixel 7, the page is 412 by 839 at a pixel ratio of 2.625, (pointer: coarse) matches, and the camera starts unlocked. A finger dragged across the sky turns the camera:

spawnite play start --device "Pixel 7"
spawnite play hold Touch1 --at 150,250
spawnite play move Touch1 --by 120,0
spawnite play release Touch1

Read what the camera sees

spawnite play see [subject], or the MCP’s director with action see and subject, prints what the player’s camera sees of each entity a subject picks, or of every entity it draws, most showing first: the share of it that shows with the rest of the scene in front, its pixels, the box they fill on the page, and its share of the screen. --part <name> narrows each entity to the objects inside its drawn object whose name holds the text, such as the gun in a character’s hand. A part that matches nothing fails with the names the entity’s meshes carry.

On Holdfast, her gun from the player’s camera over her shoulder:

$ spawnite play see 18 --part blaster
What the player's camera sees of 18:
  18 (blaster-d): 97% of it shows, 3,849 of the 3,963 pixels it covers in view, in a 45 by 171 box at (313, 308); 0.7% of the screen.

The camera draws each entity once in a flat colour of its own, with the scene in front, and once alone, at most 960 pixels wide, as three.js’s GPU picking draws. What a player sees through, glass, a flame or a glow, hides nothing behind it. A vertex shader’s own motion, such as grass that sways, is left out, since the flat colour draws the mesh as it stands. Roblox’s WorldToViewportPoint says whether a point is on screen and GetPartsObscuringTarget what stands in front of one.

Check a shot against the crosshair

spawnite play aim, or the MCP’s director with action aim, prints where the crosshair points and each shot the page sent its room, the newest twenty, beside where the crosshair pointed as the shot left. The engine’s Weapon records each shot with recordSentShot and each refusal with recordRefusedShot; a game’s own weapon component calls the same two to appear here. The crosshair’s point is the first thing the camera’s ray through the canvas’s middle meets by the engine’s own reads: the targets as the page draws them and the cover a shot meets, so grass and a model’s render mesh, which no shot meets, are not it. For each shot, the command prints how far its line passed that point, in metres and degrees. A shot that lands on what the crosshair shows passes a few centimetres from it at most. Where the room refused a shot, the line says under which rule and why, since a shot on the crosshair’s point that the room refused dealt nothing.

On Holdfast, after a burst at the ground:

$ spawnite play aim
The cursor is captured, so the crosshair is the canvas's middle: from (8.15, 2.14, 11.71) it meets cover at (8.15, 0.35, 1.52), 10.35 m from the eye.
The shots the page sent its room, oldest first, beside where the crosshair pointed as each left:
  Shot 0 of blaster, at the room's step 423, from (7.57, 1.08, 10.19): it passed 0.000 m from the crosshair's point, 0.00° off; the crosshair met cover at (8.15, 0.35, 1.52), 10.35 m from the eye.

The page keeps each shot as it sends it, in a development build alone.

Sample a value over time

--sample <expression> on spawnite play step reads the expression on each frame the page draws while the step runs, as eval reads one, and prints each frame’s value beside the world’s step and the canvas’s clock. A room session runs with its page at real time for it, so each frame is one a player would see; a frame that repeats the one before it, its step, its clock and its value, is left out. --json carries every frame as samples. A recoil’s curve is --sample camera.pitch over the burst, and a hit’s knockback is the monster’s transform.position over the half second after it.

On Holdfast, her character walking and jumping:

$ spawnite play step --seconds 0.5 --sample "player()?.transform.position"
Stepped 30 steps.
Sampled player()?.transform.position on 143 frames, the world's step and the canvas's clock beside each:
  frame  step    clock  value
   3889    69  1.987 s  [7.3708,1.2729,9.3992]
   3913    74  2.083 s  [7.4541,1.0237,9.5435]
   3929    82  2.218 s  [7.4541,0.4835,9.5435]
   3945    90  2.346 s  [7.4541,0.385,9.5435]

A timeline after a press

spawnite play timeline <control> presses the control, then shoots the page every --every seconds of the game’s time, 0.1 by default, for --seconds, 2 by default, and tiles the frames into one PNG, each under its time. It answers the question a clip’s numbers do not: which frame an effect should land on. A person reads the sheet, picks the frame where the hands are highest or the wand points, and the label under it is the second to write as the clip’s release mark in registerClip, which is the ability’s cast time, as the abilities page says. Each frame is cropped round the player’s character, where the page shows her on the first frame, so the sheet stays small; --crop '[monster.kind=wolf]' crops round another entity, and --crop none keeps the whole page, shrunk to the cell. The session is held for the sheet, so each frame is exact to its step, and a session that ran runs on after. The gap snaps to whole steps, 60 a second, and the labels count the gap stepped.

--clip <name> plays a clip the game registered on the player’s character in place of a press, so a clip with no ability yet shows its frames too, and the output ends with the registerClip line to write the second into, as its release mark. spawnite model check on the clip’s .vrma prints its motion in numbers first, where the arms peak, which a mark starts from; the sheet settles it. The MCP’s timeline tool is the same for an agent, with keys or clip. For a person to pick the frame by eye, the model viewer scrubs a bundled clip with its marks drawn on the track: tools.spawnite.com/model?avatar=chifa&clip=charged-spell-cast&marks=release:1.4, and the time under the track is the second to write.

On the mmorpg, her mage’s third spell from beside a wolf:

$ spawnite play timeline Digit3 --every 0.1 --seconds 2 --out timeline.png
Saved to /…/timeline.png: 21 frames, one every 6 steps, 0.1 s of the game's time, from the press of Digit3, cropped round [controlled].

Record a video clip

spawnite play record start starts a video clip of the session’s page, spawnite play record stop saves it, and any play command between them moves the world. The clip is the one the dev tools’ Record button writes, as the devtools page describes it: an H.264 MP4 in the game’s .spawnite/clips, with its stills, contact sheet and clip.json. It runs on the world’s steps, one frame a step at 1/60 s, so the video shows the game’s time, with no pause while a command waits and no frame dropped: a held world waits for the encoder. Its time is not the replay’s, so it keeps no replay window. Holding the world pauses the clip, so there is no pause command: a held session records nothing between commands, and spawnite play pause stops a running one’s clip until the next step or resume. A look, a set or a from-here while held shows on the next step’s frame, as a cut. A step that fails, such as a press past its end, drops the clip, and a pausing modal that ends the steps early ends the clip there, which the output says. A step that holds a press records the press. --shape 9:16 letterboxes the page to a tall clip. spawnite play record --seconds <s> records that much of the game’s time in one go, with --press as play step takes it. The stop prints the clip as spawnite clip show does. The MCP’s record_clip does the same for an agent: action start and stop around other playtest calls, or seconds with keys held, and the reply carries the contact sheet.

$ spawnite play record start --shape 9:16
Recording 9:16 on the world's steps: each step is one frame at 1/60 s. spawnite play record stop saves it.
$ spawnite play step --seconds 3 --press Space@0.2
Stepped 180 steps.
$ spawnite play record stop
Clip 20261003-041307: 3.0 s, 1080x1920 at 60 fps, 3.7 MB.
Sheet: …/games/sled/.spawnite/clips/20261003-041307/sheet.webp

Godot’s Movie Maker and Unity’s Recorder in its Constant mode record the same way, on the game’s own fixed step rather than the wall’s clock.

Run a check of your own

When no command answers a question, spawnite play eval, or the MCP’s director with action eval and expression or file, runs your own lines on the page. An expression reads the engine’s scope, the names below, before the page’s own globals, as a browser’s console does. A function is called with the scope and a promise is awaited, for 30 s at most unless --timeout says otherwise.

Name	What it is
`entities`, `count`, `has`, `player`	The world as `spawnite play dump` files it, as a step’s `--until` reads it.
`camera`, `loop`, `cursor`	The engine’s state, as `spawnite play state` prints it.
`views`	What each view exposes through `useInspect`.
`steps`, `at`	The steps a step’s condition has taken, and the moment a replay stands on in seconds.
`world`, `find(subject)`	The Koota world the page runs, and the entities a subject picks, as `dump --where` reads one.
`three`	React Three Fiber’s state: the scene, the camera, the renderer and the canvas’s size.
`engine()`, `load(path)`	The engine’s module, and a promise of one of the game’s modules by its path from the game’s folder, as the game runs them: `find('[npc]')[0].get(engine().Transform)`.
`see(subject, { part })`, `aim()`	What `spawnite play see` and `aim` read.
`input`	`press`, `hold`, `release` and `move`, as the commands above take them.
`time`	`step({ steps, seconds, until })`, `frames(count)` to wait for drawn frames, and `sample(read, { steps, seconds })`.

input and time reach the director through the page, so they work on a session spawnite play start opened. A step from a script runs from the session’s own page, not from --page 2.

To try a camera setting with no edit and no restart, write it on the camera’s own entity:

spawnite play eval "() => { const { CameraTrait } = engine(); const rig = world.queryFirst(CameraTrait); rig.set(CameraTrait, { fov: 55, shoulderOffset: 0.7, pivotRise: 0.15 }); rig.get(CameraTrait).orbit.dollyTo(2.2, false); }"

--file <path> runs a script’s default export with the scope in place of an expression. The game’s dev server compiles the script, TypeScript or JavaScript, and resolves its imports to the modules the game runs, so a script in the game’s folder imports the game’s own traits as its code does. PlayScope from @spawnite/engine/devtools types the scope. The following script turns to the nearest entity with a monster trait, as Holdfast’s monsters hold, fires a half-second burst and reads where the monster stands on each frame. The dump files a transform as { position, rotation, scale }, so the script reads transform.position, and engine<typeof import("@spawnite/engine")>() types the engine’s module for CameraTrait:

import type { PlayScope } from "@spawnite/engine/devtools";

/** A transform as the dump files it. */
interface DumpedTransform {
    position: number[];
}

/** Turns to the nearest monster, fires a half-second burst, and reads
 *  where the monster stands on each frame. */
export default async function shootNearestMonster({
    entities,
    camera,
    input,
    time,
    world,
    engine,
}: PlayScope) {
    const [x, , z] = camera?.position ?? [0, 0, 0];
    const [key, monster] = Object.entries(entities)
        .filter(([, traits]) => traits.monster !== undefined)
        .map(([key, traits]) => {
            const [mx, , mz] = (traits.transform as DumpedTransform).position;
            return [key, { mx, mz, far: Math.hypot(mx - x, mz - z) }] as const;
        })
        .sort(([, a], [, b]) => a.far - b.far)[0];
    //  Yaw 0 looks down -Z and 90 down -X, as the engine's state reads it.
    const yaw =
        (Math.atan2(-(monster.mx - x), -(monster.mz - z)) * 180) / Math.PI;
    const turn = ((yaw - (camera?.yaw ?? 0) + 540) % 360) - 180;
    //  The lock turns the camera lookSpeed radians a pixel, the other way.
    const { CameraTrait } = engine<typeof import("@spawnite/engine")>();
    const rig = world.queryFirst(CameraTrait)?.get(CameraTrait);
    const pixels = -((turn * Math.PI) / 180) / (rig?.lookSpeed ?? 1);
    await input.move("Mouse", { by: [pixels, 0] });
    await input.hold("MouseLeft");
    try {
        const frames = await time.sample(
            ({ entities }) =>
                (entities[key]?.transform as DumpedTransform | undefined)
                    ?.position,
            { seconds: 0.5 },
        );
        return { key, frames };
    } finally {
        //  Let go even where the step failed, so nothing stays held.
        await input.release("MouseLeft");
    }
}

Saved in the game’s folder as checks/shootNearestMonster.ts, it runs with the following command:

spawnite play eval --file checks/shootNearestMonster.ts

What the other engines do

Playwright gives a page a keyboard and a mouse with down, up, press, move and wheel, device descriptors such as devices["Pixel 7"], and a tap on a touch screen. It has no relative mouse motion under a lock, no pad, and no multi-finger touch; Chromium’s DevTools protocol has the touch.
Unity’s InputTestFixture presses, releases, moves and sets any control by its path, such as <Gamepad>/leftStick, begins, moves and ends a touch by its number, and adds virtual devices. Input waits for the next InputSystem.Update().
Unreal drives its UI with the Automation Driver’s sequences of presses, clicks and waits, injects gameplay input into Enhanced Input for one frame or every frame, and runs whole sessions with bots under Gauntlet. Its CSV profiler records a stat each frame.
Roblox’s VirtualInput sends keys, the mouse, a pointer and text in a test, and Studio simulates a phone’s profile, orientation and resolution. RunService.Heartbeat reads a value each frame.

The director takes Unity’s model: named controls held between commands, virtual devices where the browser has none, and time that moves only when the session steps. It takes Playwright’s names for keys and devices, which an agent already knows, and Unreal’s per-frame sampling. It differs from all four in two ways. The input reaches the page as the browser would hand a player’s input over, with the engine’s own capture standing in for a lock. And a check that needs more than a command is a few lines against the engine’s own scope, run from the command line, with no test project to build.