Skip to content

Python API

This page covers the Python modules that run a task with a simulated participant: nkl.sim, which steps a run by hand, and nkl.agents, which lets a model act as the participant, one reply a turn.

Simulator API

nkl.sim runs a task in a headless browser through nkl-server, with a participant that sees only the screen and acts only with the pointer and the keys. Times are in µs.

python
run = nkl.sim.start(main(question="What should we call you?"), seed=0)
run, images = nkl.sim.run(run, [(0, nkl.sim.Keyboard.Press("KeyA")),
                                (80_000, nkl.sim.Keyboard.Release("KeyA")),
                                (0, nkl.sim.Look())])
nkl.sim.result(run)

Settings

typeDisplay(screen=(1920, 1080), ratio=1.0)

The simulated screen: its size in CSS pixels, and its device pixel ratio.

typeLook(size=None, channels=None)

What a look reads back: a size in pixels, or the canvas's buffer when None, and "rgb", "rgba" or "gray". In a schedule, it takes a look at that time, and its unset fields are the run's.

Acts

typePointer.Move(x, y)`, `Pointer.Down()`, `Pointer.Up()`, `Pointer.Connect(x, y, down)`, `Pointer.Disconnect()

The pointer's acts. x and y are in scene units, with the origin at the canvas's centre and y up.

typeKeyboard.Press(key)`, `Keyboard.Release(key)`, `Keyboard.Connect(held)`, `Keyboard.Disconnect()

The keyboard's acts. key is a Keyboard.Key, a W3C KeyboardEvent.code such as "KeyA" or "ShiftLeft". A press is translated to a character as a US Mac keyboard does, with shift, the option layers and dead keys.

typePointer.State(x, y, down, connected)`, `Keyboard.State(down, connected)

The input states one instant before a run begins.

typeAct

Any of the pointer's or keyboard's acts.

valueSTOP

A schedule item: the participant leaves, and the run ends "cancelled".

Runs

typeRun

A run, as a frozen value: key, the time t in µs, and its outcome (None while it runs). Each function returns a new Run, so an older one is a branch. A run the server no longer holds is started again and replayed from its history.

typeImage

One look: width, height, channels, the run time at, the request line of the screen shown, and the pixel data, top row first. .array() is a NumPy uint8 array of shape (height, width, channels), or (height, width) for gray. .png() is the bytes of a PNG file, 8 bits a channel.

functionstart(app, *, seed=0, display=Display(), look=Look(), pointer=Pointer.State(), keyboard=Keyboard.State(), server=None) -> Run

A run of a root @nkl.function applied to its arguments.

functionrun(run, schedule, until=None) -> (Run, list[Image])

Applies schedule, a list of (dt, item), each dt µs after the item before (the first after run.t). An item is an Act, a Look or STOP. Then, if until is given, the clock runs on to that absolute time, firing the task's timers. Acts at the same µs are separate ticks, and items after the run ends are not applied. The looks come back in order.

functionsubmit(run, *acts) -> Run`, `wait(run, duration) -> Run`, `look(run, size=None, channels=None) -> (Run, Image)`, `stop(run) -> Run

One act each at the run's time; the clock duration µs on; one look now; STOP now.

functionoutcome(run)`, `result(run)`, `trace(run)

How the run ended; its result as Python values, which raises RunEnded for a run that ended otherwise; and its trace lines so far.

Agents

nkl.agents lets a model act as the participant. Each turn the model sees the screen and its own context, and replies with lines of a small grammar. The reply's actions are applied with one nkl.sim.run call, and the next screen is looked at. Calls to the model keep no state: everything carried from turn to turn is in the context.

python
from pathlib import Path
import nkl
from nkl import agents

bedrock = agents.AwsBedrock.connect(
    "global.anthropic.claude-sonnet-5-5",
    region="us-east-1",
    credential=Path("~/.config/nkl/bedrock.env").expanduser(),
)
run = nkl.sim.start(main(question="What should we call you?"), seed=0)
run, turns = agents.session(
    run, agents.AwsBedrock.respond, bedrock,
    instructions="Type your name and press Enter.",
    pacing=agents.Frozen(),
    into=Path("runs/text-box-sonnet"),
)

The reply grammar

A reply's actions and memories are its lines that each hold one JSON object, after an optional leading - or * , done in order. A line that starts with a note and goes on with {"do" holds the command from there. Every other line is the model's own note and is ignored. Positions are in pixels of the current screen image: x from 0 at the left, y from 0 at the top.

  • {"do": "pointer_move", "x": X, "y": Y, "force": F}: moves the pointer to (X, Y). force is from above 0 to 1, the default; less force is slower. Consecutive pointer moves of the same force are one stroke, which passes exactly through each position on a smooth curve and is at rest only at its ends.
  • {"do": "pointer_down"}, {"do": "pointer_up"}: press or release the pointer's button.
  • {"do": "keys_down", "keys": [KEY, …]}, {"do": "keys_up", "keys": [KEY, …]}: press or release keys, named by Keyboard.Key. "keys": "all" on keys_up releases every key held.
  • {"do": "type", "text": TEXT}: TEXT is printable ASCII, newline and tab. Each character is typed with the presses of the US Mac keyboard's plain or shifted layer; newline is Enter, and tab is Tab.
  • {"do": "wait", "ms": MS}: MS milliseconds pass.
  • {"do": "look", "id": ID}: a picture of the screen at that point, shown to the model as ID from its next turn.
  • {"do": "memory_set", "id": ID, "text": TEXT}: keeps TEXT under ID.
  • {"do": "memory_delete", "id": ID}: forgets one memory, or all of them with "id": "all".

A reply holds between 1 and limits.actions actions; memories don't count. A reply with any wrong command does nothing. Its problems are shown to the model in its next turn. These are wrong:

  • a line that starts with { and is not one JSON object
  • an unknown "do", or a field the command doesn't take
  • a position off the screen
  • an unknown key
  • a character type can't type, or a modifier key held while typing
  • a memory longer than the limit
  • no action, or more actions than the limit allows

Context

typeLimits(memories=16, memory_chars=400, looks=4, actions=20, turns=100)

How many memories and looks are kept, with the oldest dropped first; the longest memory; the most actions a reply may hold, where a line is one action however many presses it makes and memory lines don't count; and the turns a session asks for.

typeBody(pointer: nkl.sim.Pointer.State, keyboard: nkl.sim.Keyboard.State)

What the participant knows of its own hands: where the pointer is, whether its button is down, and which keys are held. Positions are in scene units.

typeAgentContext

All the model knows at a turn:

  • screen: the current screen, an nkl.sim.Image
  • extent: the screen image's size in scene units
  • memories: (id, text) pairs, least recently set first
  • looks: (id, image) pairs, oldest first
  • body: a Body
  • problems: what was wrong with the previous reply
  • limits: a Limits
  • turns_left: the turns left, this one included

Response

typeWait(us)`, `TakeLook(id)

The actions that are not acts: time passing, and a look kept under id.

typeStroke(points, force=1.0)

A pointer move through points, in scene units, from rest to rest. The path is a centripetal Catmull-Rom spline through the points, which passes through each one with a continuous direction; between two points it is a straight line.

typeAction

An nkl.sim.Act, a Wait, a TakeLook or a Stroke.

typeMemorySet(id, text)`, `MemoryDelete(id)`, `MemoryChange

A change to the memories. MemoryDelete("all") forgets them all.

typeUsage(input_tokens, output_tokens, cache_read_tokens, cache_write_tokens)

The tokens of one model call.

typeAgentResponse

One reply:

  • actions: in order and in scene units, with type expanded into key presses, and consecutive pointer moves of one force joined into a Stroke
  • memory_changes: in order
  • problems
  • text: the reply as written
  • reasoning: the model's reasoning text, where the backend returns it
  • usage
  • seconds: the wall time of the attempt that answered
  • attempts
  • model, stop_reason and request_id

When problems is not empty, actions and memory_changes are empty.

Backends

A backend is a session record and a function that answers one turn:

python
respond(session, system: str, context: AgentContext) -> tuple[AgentResponse, AgentContext]

The returned context is context with the reply's memory changes applied, its problems set, and one turn fewer left. Its screen, looks and body are updated by observe once the actions have run.

typeAwsBedrock.Session(model, region, credential, max_tokens=16000, additional_fields=None, cache=True, attempts=6)

A model on Amazon Bedrock:

  • model: an inference profile ID, such as global.anthropic.claude-sonnet-5-5
  • credential: the path of an env file that sets AWS_BEARER_TOKEN_BEDROCK, a Bedrock API key. The session holds only the path, and the key is read from the file for each request.
  • additional_fields: sent as the Converse API's additionalModelRequestFields, for example to turn on reasoning
  • cache: puts a cache point after the system prompt
  • attempts: how many times a throttled or failed request is tried
functionAwsBedrock.connect(model, *, region, credential, max_tokens=16000, additional_fields=None, cache=True) -> AwsBedrock.Session

A session, once the key has listed the region's foundation models and model is among them.

functionAwsBedrock.respond(session, system, context)

One Converse API request over HTTPS. The system prompt is followed by a cache point when cache is set. The one user message holds, in order:

  • the problems
  • the memories
  • the looks, each with its id and time
  • the pointer and the keys held
  • the screen

Images are sent as PNG. Throttling and service errors are retried with jittered backoff.

typeScripted.Session(reply: Callable[[AgentContext], str], model="scripted")`, function `Scripted.respond

A backend that answers with reply(context), with no model and no cost.

Shared functions

functionsystem_prompt(instructions, *, screen, extent, limits, pacing, motor, seconds) -> str

The participant's briefing:

  • its goal, to complete the task
  • the screen's size and coordinates
  • the reply grammar
  • how many actions a reply may hold, and why a shorter one shows the screen sooner
  • how long acts take, with the times of three strokes at force 1 and at force 0.5 worked out by stroke_ms, and how the clock runs under pacing
  • the turns and the seconds of task time, and the limits
  • the key names

The experimenter's instructions come last.

functionparse(text, context) -> (actions, memory_changes, problems)

Reads a reply against the context. It converts pixels to scene units using extent, expands keys_up of "all" from the keys held, and expands type.

functionremember(memories, changes, limits)

The memories after changes, in order. A memory that is set becomes the newest, and the oldest beyond limits.memories are dropped.

functionkeys() -> tuple[str, ...]

The Keyboard.Key names.

Running a session

typeRealtime()`, `Frozen()`, `Dense(step=16_667)`, `Pacing

How task time relates to the model's answering:

  • Realtime: the response's seconds pass in the task before its first act.
  • Frozen: no task time passes while the model answers.
  • Dense: each turn moves the task on step µs, and a reply holds one action.
typeMotor(press=1_000, top_speed=2.5, top_acceleration=0.02, rate=120.0)

How the participant's hand moves:

  • press: the µs a key or button act takes
  • top_speed: a stroke's top speed at force 1, in scene units a ms
  • top_acceleration: its top acceleration, in scene units a ms a ms
  • rate: the pointer positions a second along a stroke
functionstroke_ms(length, force, motor) -> float

The duration of a stroke length scene units long. The stroke follows a minimum-jerk profile from rest to rest, and the duration is the shortest that keeps its peak speed (1.875 L/T) and peak acceleration (5.77 L/T²) within the motor's, scaled by force.

functionstroke_samples(start, stroke, motor) -> (list[(µs, point)], µs)

A stroke's pointer positions from start, timed from the stroke's start, and its duration. Each of the stroke's points is given exactly, at the moment the minimum-jerk profile reaches it along the path. Between them, positions on the path are given at motor.rate.

functionbegin(run, *, limits=Limits()) -> (Run, AgentContext)

The first context of a fresh run: a look at the screen, no memories or looks, and the input states the run started with.

functionact(run, context, response, pacing, motor=Motor()) -> (Run, list[Image])

The response's actions as one schedule:

  • A key or button act takes motor.press.
  • A Stroke is its stroke_samples, from where the pointer is.
  • A Wait adds its time.
  • A TakeLook is a look at that point.
  • A final look gives the next screen. Under Dense, it is at the latest of three times: step after the turn began, the reply's last act, and the end of its last wait or stroke.
functionobserve(context, response, images) -> AgentContext

The next context:

  • the last image as the screen
  • the response's looks added under their ids, within limits.looks
  • the body after the response's acts
typeTurn(at, context, response)

One turn: the run's time when the model was asked, what it saw, and what it replied.

functionsession(run, respond, backend, *, instructions="", pacing=Frozen(), limits=Limits(), motor=Motor(), seconds=60.0, into=None) -> (Run, tuple[Turn, ...])

Turns of respond, act and observe from a fresh run, until one of these:

  • the run ends
  • seconds of task time have passed
  • limits.turns turns have been taken

A run still going is then stopped. Under Dense, limits.actions is 1. With into, a run folder is written there:

  • system.txt: the system prompt
  • turns.jsonl: each turn's context, with images as frame paths, and its response
  • frames/: each screen and look, as PNG
  • trace.jsonl: the run's trace
  • summary.json: the outcome, the number of turns and of invalid replies, the total tokens, and the settings