tldraw-offline – Agents & Automation skill for AI agents

tldraw offline Skill

Work with the tldraw offline desktop app (offline.tldraw.com): read the open

canvas, make edits, and write document scripts — JavaScript embedded in a

.tldraw file that runs on load and gives the file durable behavior. The app

runs a local HTTP API (default localhost:7236) that a coding agent drives

with plain curl from its terminal — this is exactly how the app's own homepage

demo (Codex editing a canvas live) works. The agent does NOT use computer-use /

GUI clicking, and does NOT hand-edit the .tldraw file directly. Keep tldraw

offline open while you work.

When to Use

• The user has tldraw offline open and asks you to build or modify a canvas

(diagrams, wireframes, layouts).

• You want to add durable behavior to a drawing (reactive shapes, interactive

buttons, animation, connection logic) via an embedded document script.

Do NOT hand-place shapes to imitate a drawing — write the code that generates

them. Agents are far better at scripting the canvas than at drawing on it.

Prerequisites

• tldraw offline installed and running, with a document open. Releases:

https://github.com/tldraw/tldraw-offline/releases/latest (macOS DMG, Windows

x64/Arm64, Linux x86_64/arm64 AppImage or amd64/arm64 .deb).

• Agent skills installed in the app: Develop → Install Agent Skills. The

app writes its own tldraw skill into ~/.codex/skills/, ~/.claude/skills/,

~/.cursor/skills/, and ~/.gemini/skills/ — teaching that agent the curl

recipes below. (This Hermes skill mirrors that guidance for Hermes.)

• The local control API. On launch the app writes server.json to its config

dir (Linux ~/.config/tldraw/, macOS ~/Library/Application Support/tldraw/,

Windows %APPDATA%\tldraw\) with port (default 7236), a bearer token,

pid, and startedAt. Every request except GET / needs

Authorization: Bearer <token>. A clean quit removes server.json; if it's

present but the port doesn't answer, the app quit uncleanly — treat as not

running.

• Re-read port + token on EVERY shell call. Each terminal call is a fresh

shell, so an exported token does not persist — "export once and reuse" sends

an empty token and 401s. Read both inline at the top of each call:

PORT=$(jq -r .port <server.json>); TOKEN=$(jq -r .token <server.json>).

• No account or network needed for local editing.

How to Run

Two distinct workflows. Pick by whether the change must survive a reload.

A. One-off canvas edits (/exec) — layout, generating shapes, cleanup. This

is a live edit, not saved script:

BASE=http://localhost:7236
TOKEN=$(python -c "import json;print(json.load(open('$HOME/.config/tldraw/server.json'))['token'])")
# find the focused document id
DOC=$(curl -s "$BASE/api/search" -X POST -H 'content-type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"code":"return (await api.getFocusedDoc()).id"}' | python -c "import sys,json;print(json.load(sys.stdin)['result'])")
# run code with the live `editor` + `helpers` in scope
curl -s "$BASE/api/doc/$DOC/exec" -X POST -H 'content-type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"code":"const {createShapeId,toRichText}=await import(\"tldraw\"); editor.createShape({id:createShapeId(),type:\"geo\",x:0,y:0,props:{geo:\"rectangle\",w:200,h:100,color:\"blue\",fill:\"solid\",richText:toRichText(\"hello\")}}); return editor.getCurrentPageShapes().length"}'

B. Durable behavior (script/main.js) — reactive/interactive logic that must

survive reload. Edit the file on disk; the app's watcher applies it:

# get the live script file path for the doc
curl -s "$BASE/api/doc/$DOC/script-workspace" -X POST \
  -H "Authorization: Bearer $TOKEN"          # -> result.mainJsPath, result.isDefaultScript
# edit result.mainJsPath with read_file / patch / write_file (see scripts/main.js)
# then confirm the watcher applied it:
curl -s "$BASE/api/doc/$DOC/script-status" -H "Authorization: Bearer $TOKEN"

The ready-to-adapt document script is scripts/main.js.

Quick Reference

The document-script contract (verified against the app's bundled

script-context.d.ts):

import { createShapeId, toRichText } from 'tldraw'   // primitives: import, not globals

export default function ({ editor, helpers, signal }) {
  editor.run(() => {                                 // batch = one undo step
    helpers.createShapeIfMissing({                   // idempotent furniture
      id: createShapeId('node-1'), type: 'geo', x: 0, y: 0,
      props: { geo: 'rectangle', w: 200, h: 100, richText: toRichText('hi') },
    })
  })

  const stop = editor.store.listen(() => { /* react */ })  // fires the tick AFTER a commit
  signal.addEventListener('abort', () => stop())           // REQUIRED cleanup on rerun/close
}

• ctx.editor — the live Editor (createShape, updateShape, deleteShapes,

getCurrentPageShapes, getShape, getBindingsFromShape, zoomToFit,

on('tick'|'event', fn), run(fn, { history: 'ignore' })).

• ctx.helpers — createShapeIfMissing, createShapesIfMissing,

createArrowBetweenShapes(from, to, { arrowheadEnd }), translateShapes,

onShapeTranslate(id, fn, { signal }), richTextToPlainText, boxShapes,

getLints.

• ctx.signal — AbortSignal; attach every listener/interval teardown to it.
• config.js (separate file) registers custom shape/tool/component utils and

runs before mount; main.js runs against the mounted editor and reruns on save.

Interactive UI (clickable buttons that drive state)

Drawn shapes can behave like a real app — the thing a static whiteboard can't do.

Full example: scripts/counter.js (a number display + MINUS/RESET/PLUS buttons).

Verification boundary — read this before claiming interaction works or doesn't.

The app's OWN agent playbook says to verify a clickable-UI script with "one

simulated click and one state read" via /exec (editor.dispatch a pointer

event, await a tick, read the shape's state) — NOT by driving a real mouse. By

that standard the counter is verified: dispatched clicks stepped it

0 → 1 → 2 → 1 → 0. Two caveats worth writing down:

• The script only runs once the app's file-watcher applies it. On Linux that

watcher uses inotify; a host with an exhausted fs.inotify.max_user_instances

logs inotify_add_watch ... No space left on device, script-status shows

state: "not-watching" / hasEntry: false, and the script never executes.

This is a host limit, not a script bug — a normal desktop is unaffected.

• **Driving the canvas with computer-use needs FOREGROUND delivery, not

background.** cua-driver's default background delivery returns

background_unavailable against Electron's occluded renderer — but that is the

first rung, not a wall. cua-driver returns an escalation: "foreground" hint;

set delivery_mode: "foreground" (pair with bring_to_front) and its X11 XTest

path (x11_xtest_fg) clicks land — this dismisses the "Run Script" consent

dialog and clicks the canvas. Verified by the Cua team on this exact v1.11.0

AppImage (Linux/X11). Do NOT conclude "Electron rejects synthetic clicks" and

give up in background mode — climb to foreground. (The real product path is

still /exec, not clicks; this note is for computer-use-driven testing.)

The pattern:

export default function ({ editor, helpers, signal }) {
  // 1. Build buttons idempotently; tag each with meta so the handler finds them.
  //    Give buttons a visible label AND a meta.action.
  // 2. Hit-test pointer_down in PAGE coordinates against the button bounds:
  const inside = (b, p) => p.x >= b.x && p.x <= b.x + b.w && p.y >= b.y && p.y <= b.y + b.h
  function onEvent(info) {
    if (!info || info.name !== 'pointer_down') return
    let p = null
    try { if (info.point && editor.screenToPage) p = editor.screenToPage(info.point) } catch {}
    p = p ?? editor.inputs?.currentPagePoint
    if (!p) return
    const hit = editor.getCurrentPageShapes().find(
      (s) => s.meta?.ui === 'button' &&
        inside({ x: s.x, y: s.y, w: s.props.w, h: s.props.h }, p)
    )
    if (hit) runAction(hit.meta.action)   // mutate state; store it in a shape's meta
  }
  editor.on('event', onEvent)
  signal.addEventListener('abort', () => editor.off('event', onEvent))  // REQUIRED
}

• Find buttons by meta (or visible label via helpers.richTextToPlainText),

not by hard-coded coordinates.

• One script owns both build and read. If the shapes are created by one code

path (with meta.action: 'inc') and the handler reads another convention

(meta.action === 'PLUS'), clicks silently do nothing. Ship the buttons built

by the same script that handles them, or ship an empty canvas so the script

builds them fresh — never pre-bake mismatched shapes into the file's db.

• Keep app state in a shape's meta (e.g. meta.count) and render it as that

shape's `ric

… (full content is in the download package)