--- title: "Run it yourself | Pelican benchmark" url: https://evaljetty.com/pelicans/run.html description: "The pelican-on-a-bicycle runbook, the Jetty task workflow JSON and curl commands to run it in your own Jetty collection with an OpenRouter key." updated: 2026-09-28 publisher: Jetty (https://jetty.io) --- Run it yourself · one runbook · one API key # Draw your own pelican on Jetty Everything on this site came from one runbook and a small orchestrator. Here's all of it. You need a Jetty collection and an OpenRouter key; a single round costs roughly $0.50 to $2.50 depending on the model. [Download the runbook (.md)](https://evaljetty.com/pelicans/run/pelican-bicycle-svg.runbook.md) [Task workflow JSON](https://evaljetty.com/pelicans/run/task-workflow.json) [Get a Jetty account](https://jetty.io/?utm_source=evaljetty&utm_medium=referral&utm_campaign=pelicans&utm_content=run_yourself) 1. ### The runbook Plain markdown with YAML frontmatter. It tells the agent to draw, render with rsvg-convert, look at the PNG, score it on four axes and redraw, three times, then write final.svg, final.png and report.md. ``` --- name: pelican-bicycle-svg version: 2 description: Generate the best possible hand-written SVG depicting a pelican riding a bicycle, with iterative self-refinement. agent: claude-code model: anthropic/claude-sonnet-5 model_provider: openrouter parameters: max_rounds: type: integer default: 3 description: Number of refinement rounds (each round = critique previous + write improved) results_dir: type: string default: /app/results secrets: openrouter: env: OPENROUTER_API_KEY description: OpenRouter key (collection environment variable) used by the agent's model calls evaluation: pattern: self-judge --- # Pelican Riding a Bicycle — SVG Runbook ## Mission Produce the highest-quality hand-written SVG depicting **a pelican riding a bicycle**. Both subjects must be unmistakably recognizable, the composition coherent (pelican IS interacting with the bicycle, not floating next to it), and the file should be pure XML SVG (no `` tags, no base64 data, no external rasters). ## Hard constraints - Pure SVG XML — use ``, ``, ``, ``, ``, ``, ``, ``, ``, etc. - **No** `` elements with external or base64 data - Must validate as well-formed XML and render in any modern browser - viewBox approximately `0 0 800 600` (landscape) — adjust if you have a specific reason - Total file size under 50KB ## INPUTS - `max_rounds` = **3** (override only if explicitly told otherwise) - `results_dir` = `/app/results` ## Steps ### 1. Setup ```bash mkdir -p {{results_dir}}/rounds cd {{results_dir}} ``` Check that `rsvg-convert` or an SVG rasterizer is available: ```bash which rsvg-convert || which inkscape || which convert ``` If none are available, install one: ```bash apt-get update && apt-get install -y librsvg2-bin || true ``` ### 2. Round 1 — first draft Write your best first attempt to `{{results_dir}}/rounds/v1.svg`. Aim to depict: - **Pelican**: long beak with throat pouch (the iconic pelican silhouette), body, eye, wings, legs/feet - **Bicycle**: two wheels with spokes, frame (top tube + down tube + seat tube), handlebars, seat, pedals - **Riding**: pelican's body is on the seat, feet on or near pedals, "hands" (wing tips) on handlebars Render it: ```bash cd {{results_dir}} rsvg-convert -w 800 rounds/v1.svg -o rounds/v1.png ``` ### 3. Self-critique + refinement loop For each round R from 1 to `max_rounds - 1`: 1. **Inspect** `rounds/v${R}.png` visually (read it as an image). 2. **Score** on four axes, 0–10 each: - **Pelican recognizability** — would a stranger immediately say "that's a pelican"? - **Bicycle recognizability** — would they say "that's a bicycle"? - **Composition / riding** — is the pelican clearly riding the bike? - **Aesthetic polish** — line quality, color, balance 3. **Identify the lowest-scoring axis** and write down 2-3 concrete fixes. 4. **Write** `rounds/v$((R+1)).svg` applying those fixes. Keep what worked; rewrite what didn't. 5. **Render** `rounds/v$((R+1)).png`. 6. If the total (sum of 4 axes) is ≥ 36/40 — early-exit the loop. ### 4. Finalize After the loop, pick the round with the highest total score (break ties by preferring later rounds since they had more iteration). ```bash cp {{results_dir}}/rounds/vBEST.svg {{results_dir}}/final.svg cp {{results_dir}}/rounds/vBEST.png {{results_dir}}/final.png ``` Write `{{results_dir}}/report.md` containing: ```markdown # Pelican-Bicycle SVG Report ## Per-round scores | Round | Pelican | Bicycle | Composition | Polish | Total | |-------|---------|---------|-------------|--------|-------| | 1 | ? | ? | ? | ? | ? | | 2 | ? | ? | ? | ? | ? | | 3 | ? | ? | ? | ? | ? | ## Best round: vN (Total: X/40) ## What worked - ... ## What didn't work - ... ## SVG technique notes - viewBox used: - Total path / shape count: - File size: ``` ## Final checklist - [ ] `{{results_dir}}/final.svg` exists, is valid XML, renders in a browser, has no `` tags - [ ] `{{results_dir}}/final.png` exists (rasterized version) - [ ] `{{results_dir}}/rounds/v*.svg` and `v*.png` exist for every round attempted - [ ] `{{results_dir}}/report.md` exists with per-round score table and a notes section - [ ] File size of `final.svg` is under 50KB ``` 2. ### The Jetty task A single runbook step. Agent, model, provider, snapshot, instruction and template variables are all read from init\_params, so one task can run any agent/model pair. The runbook itself is inlined as instruction (elided here). ``` { "init_params": { "agent": "claude-code", "model": "anthropic/claude-sonnet-5", "model_provider": "openrouter", "snapshot": "python312-uv", "instruction": "", "vars": { "prompt": "Execute the runbook end-to-end.", "results_dir": "/app/results", "max_rounds": 3 }, "file_paths": [] }, "steps": [ "run" ], "step_configs": { "run": { "activity": "runbook", "agent_path": "init_params.agent", "model_path": "init_params.model", "model_provider_path": "init_params.model_provider", "snapshot_path": "init_params.snapshot", "instruction_path": "init_params.instruction", "template_variables_path": "init_params.vars", "files_path": "init_params.file_paths", "cpus": 4, "memory": "8G", "timeout_sec": 1800, "network_enabled": true } } } ``` 3. ### Set the key and create the task The collection needs one secret, OPENROUTER\_API\_KEY, stored as a collection environment variable. Secrets never go in init\_params. ``` export JETTY_TOKEN=mlc_... # your Jetty API key export COLLECTION=your-collection # a collection you own # 1. Store your OpenRouter key as a collection environment variable (sent via stdin, not argv) printf '{"environment_variables": {"OPENROUTER_API_KEY": "%s"}}' "$OPENROUTER_API_KEY" | \ curl -s -X PATCH "https://flows-api.jetty.io/api/v1/collections/$COLLECTION/environment" \ -H "Authorization: Bearer $JETTY_TOKEN" -H "Content-Type: application/json" --data-binary @- # 2. Create the task from the workflow JSON (the runbook is embedded as init_params.instruction) curl -sO https://evaljetty.com/pelicans/run/task-workflow.json jq '{name: "pelican-bicycle-svg", description: "Pelican riding a bicycle, SVG", workflow: .}' task-workflow.json | \ curl -s -X POST "https://flows-api.jetty.io/api/v1/tasks/$COLLECTION" \ -H "Authorization: Bearer $JETTY_TOKEN" -H "Content-Type: application/json" --data-binary @- ``` 4. ### Run it Launch with a multipart init\_params field. Top-level keys you pass replace the task defaults; keys you omit (snapshot, file\_paths) keep the task's values. ``` # 3. Run it. Anything in init_params overrides the task defaults (top-level keys are merged) curl -s -X POST "https://flows-api.jetty.io/api/v1/run/$COLLECTION/pelican-bicycle-svg" \ -H "Authorization: Bearer $JETTY_TOKEN" \ -F 'init_params={"agent": "claude-code", "model": "anthropic/claude-sonnet-5", "model_provider": "openrouter", "vars": {"prompt": "Execute the runbook end-to-end. You are headless: never stop to ask questions.", "results_dir": "/app/results", "max_rounds": 3}}' # => {"trajectory_id": "…", "workflow_id": "…"} # Other agent/model pairs we ran (all model_provider=openrouter): # claude-code anthropic/claude-opus-5.5 # opencode openrouter/google/gemini-3.8-flash # opencode openrouter/openai/gpt-5.6-terra # hermes openrouter/typesafe/jev-router ``` 5. ### Collect the drawing ``` # 4. Poll until status is completed, then download everything (final.svg, final.png, report.md, rounds/) curl -s "https://flows-api.jetty.io/api/v1/db/trajectory/$COLLECTION/pelican-bicycle-svg/$TRAJ" \ -H "Authorization: Bearer $JETTY_TOKEN" | jq '.status, .steps.run.outputs.usage' curl -s -o pelican.zip "https://flows-api.jetty.io/api/v1/trajectory/$COLLECTION/pelican-bicycle-svg/$TRAJ/download" \ -H "Authorization: Bearer $JETTY_TOKEN" # or open https://jetty.io/$COLLECTION/pelican-bicycle-svg/$TRAJ ``` 6. ### Hill-climb it The orchestrator we used for v2: each round embeds the previous best SVG into the runbook and targets the weakest self-scored axis. It stops launching runs past a spend budget. ``` # 5. Optional: hill-climb (5 rounds per agent, agents in parallel, $60 spend guard) curl -sO https://evaljetty.com/pelicans/run/hill_climb.py curl -so runbook.md https://evaljetty.com/pelicans/run/pelican-bicycle-svg.runbook.md # edit COLLECTION at the top of hill_climb.py, then: JETTY_TOKEN=$JETTY_TOKEN python3 hill_climb.py --agent all --rounds 5 --budget 60 ``` --- Source: https://evaljetty.com/pelicans/run.html · Evals by Jetty · Run your own evals: https://jetty.io/?utm_source=evaljetty&utm_medium=llms