{ "schema": "agent-public-bench@1", "generatedAt": "2026-10-07T00:00:00.000Z", "url": "https://agent.sasid.ai/benchmarks/system-one-arena", "study": { "slug": "system-one-arena", "title": "System One arena: Jev vs Clef and five open decision models, head to head", "seoTitle": "Jev vs Clef: 7 decision models on 1,085 decisions", "description": "Jev 1.13, Clef 27B, Clef-Flash and four small open decision models on 1,085 checkable decisions and in head-to-head games, Pong included.", "question": "How good are typed-decision models at decisions with a checkable answer, and which one wins when they play each other?", "answer": "Jev 1.13 answered 76.8% of 1,048 graded decisions correctly, clearly ahead of every other model (exact McNemar p < 0.001 for each pair). The best open model, Clef 27B, reached 69.9%. At one provider's list prices (Cloudflare Workers AI) a thousand decisions cost $0.035 for Jev 1.13, $0.051 for Clef-Flash 9B, $0.136 for Clef 27B. Speed is compared only where the footing is the same: on one GPU a third-party board measured Clef 27B at 209 ms and Clef-Flash 9B at 39 ms, while Jev 1.13, a hosted API, took 524 ms in that harness with the network included. Clef-Flash 9B (6.49 GB, 65.2%) and Kev 4B (3.03 GB, 64.0%) were statistically tied. Laya (0.45 GB, 27.4%) and Julia-1 (0.17 GB, 25.9%) were statistically tied. When no option fitted, Jev 1.13 chose \"none\", \"ask\" or \"escalate\" 86% of the time; Laya 18%. Clef 27B and Clef-Flash 9B never changed a decision when the options were shuffled, but changed 18% and 18% when the same options were renamed a, b, c. In 1,512 round-robin games (tic-tac-toe, Connect Four, Nim, Dots and Boxes and Pong), Jev 1.13 rated highest (Elo 1040 against 896 for the random player; the other models sit between 877 and 941). Jev 1.13 (30–11 of 42) and Clef-Flash 9B (29–11 of 42) beat the random player clearly (exact sign test p < 0.05); the other models did not. In Pong, where an answer counts only once it arrives, Jev 1.13 won 13 of 16 games and Kev 4B 12, while Clef 27B, the most accurate open model on the exam, won 4 at 1.8 s per decision. In the arena leagues Jev 1.13 ranked as follows: Othello: rank 9 of 9, Elo 890; best was perfect at 1280; the random player 1015; Pong (every answer after 150 ms): rank 3 of 8, Elo 1042; best was perfect at 1098; the random player 930; Snake (every answer after 150 ms): rank 2 of 9, Elo 1103; best was expert at 1185; the random player 954; Tron (every answer after 150 ms): rank 2 of 9, Elo 1219; best was expert at 1267; the random player 898.", "date": "2026-10-06", "updated": "2026-10-06", "tags": [ "jev", "clef", "decision-models", "system-one", "llama-cpp", "open-models", "routing" ], "method": [ "Seven models with the same API (a JSON state in, a choice or a yes/no with probabilities out): TypeSafe Jev 1.13 over its hosted API, and Cloudflare Clef 27B, Clef-Flash 9B, lev 4B, Kev 4B, Laya and Julia-1 as GGUF files (Q4_K_M or Q8_0, checked against the published SHA-256) in llama.cpp 0.6.0 on one Mac Studio (M3 Ultra, 96 GB).", "Five suites, 1,085 items, written by five agents that never called a model: games solved by search (tic-tac-toe, Connect Four, Nim, Pong intercepts, Sudoku, Wordle, mazes, Hanoi, coin weighing, knight moves), logic and thought experiments (syllogisms, knights and knaves, Monty Hall, base rates, expected value, bias probes, framing pairs), fictional company policies in 20 industries, usability intents on 14 kinds of app, and stress tests (prompt injection, long logs, thresholds, near-identical options, paraphrases).", "Every answer comes from search, arithmetic, logic or a rule written in the item. Each suite has its own verifier that re-derives every answer. A second agent re-solved every item of the first versions of the suites (978) and found no wrong answer; the ambiguities and shortcuts it found were fixed before the freeze, adding 107 items and rewriting others. Consistency-only items (no right answer) are scored only on agreement inside their group.", "Exam pass: every item, every model; choice items also with shuffled options and with options renamed a, b, c. Latency pass: 120 seeded items, one model and one call at a time. Repeat pass: 60 items asked twice.", "Matches: a round robin of tic-tac-toe, Connect Four, Nim and Dots and Boxes (10 games per pairing, colours swapped), real-time Pong (first to 3 points after amendment 2) where each answer takes effect only after its measured latency, and a random and a perfect player as anchors. Elo: K 32, averaged over 200 seeded game orders.", "The protocol, the item hashes and the game counts were declared before the first counted call. Two amendments, both declared before the data they affect, are in the published protocol: the local-server batch size (amendment 1) and the Pong match length (amendment 2). One erratum: the Connect Four reference solver graded some moves in a search-order-dependent way; it was fixed and every move regraded, and no pick, probability, result or rating changed (erratum 2). Every call and every game is kept; errors count as wrong." ], "caveats": [ "Local models ran quantized on one Mac; the vendors measured full-precision weights on data-centre GPUs. Latency on other hardware will differ, and Jev latency includes the network.", "The items are synthetic and written by agents. They test clear, checkable decisions, not the messy decisions a product meets; real accuracy depends on how well the state and options are written.", "Typed-decision models are built for routing and gating with a confidence threshold. These items have no threshold: a model that is unsure must still pick.", "A few small families lean on content where the right option is often the longest or shortest description (sudoku, seating order, gambles). Reported, not removed.", "These are llama.cpp ports, not the reference implementations: llama.cpp converts only part of lev's output head, an open llama.cpp issue reports degraded probabilities for Clef Q8_0 (we ran Q4_K_M), and Laya and Julia-1 were trained on inputs of 512 to 1,024 tokens while our items average about 800.", "One harness error was fixed mid-run (amendment 1): the local servers first ran with a 512-token batch, which rejected longer inputs. Every local model was rerun from the start with a whole-input batch, one model at a time; the earlier rows are kept apart and not counted.", "Games: 10 games per pairing in each turn-based game (colours swapped), 2 in Pong, first to 3 after amendment 2. The sign tests against the random player are not corrected for the many comparisons made, so read them as descriptions. Connect Four \"perfect\" is a depth-10 search plus exact endgames, not a full solve.", "In Pong each answer moves the paddle only after its measured latency, and the arrival point of the ball is in the state, so Pong tests reading and speed more than physics. The viral \"smarter AI lost at Pong\" result (Laya beating Jev) did not reproduce under these rules: Jev won all 4 games of the featured series.", "Jev ran on TypeSafe's servers and the six open models on one Mac Studio, so speed across that line is not comparable. This study compares capability (same questions and positions) directly, and speed only on one machine, on one GPU (third-party numbers), through one gateway (OpenRouter's numbers) or as list price.", "Arena leagues (Othello, and Tron, Snake and Pong in fair mode) are content recordings, not tests of the models: one seed, 2 to 4 games per pairing, protocol amendment 3. The Mac also ran other work while they were recorded (erratum 4), so their call times are not clean measurements and no speed claim uses them; fair-mode results do not depend on call time." ], "sourceIds": [ "system-one-arena" ], "stats": [ { "id": "arena-items", "label": "Decision items, written by agents that never called a model (1,048 graded, 37 consistency-only)", "value": 1085, "unit": "count", "display": "1,085", "n": 1085 }, { "id": "arena-calls", "label": "Decision calls made and counted (exam, latency and repeat passes)", "value": 23247, "unit": "calls", "display": "23,247", "n": 23247 }, { "id": "arena-best", "label": "Highest accuracy on the graded items: Jev 1.13", "value": 0.7681, "unit": "rate", "display": "76.8% (805/1048)", "n": 1048, "ci": [ 0.7416, 0.7927 ] }, { "id": "arena-best-open", "label": "Best open model: Clef 27B", "value": 0.6994, "unit": "rate", "display": "69.9% (733/1048)", "n": 1048, "ci": [ 0.671, 0.7264 ] }, { "id": "arena-smallest", "label": "Smallest model: Julia-1 (0.17 GB file)", "value": 0.2586, "unit": "rate", "display": "25.9% (271/1048)", "n": 1048, "ci": [ 0.233, 0.2859 ] }, { "id": "arena-jev-cost", "label": "Jev 1.13 list-price cost per 1,000 decisions (calculation from its reported input tokens)", "value": 0.03465, "unit": "usd", "display": "$0.0347", "n": 3081 }, { "id": "arena-jev-repeat", "label": "Same question asked twice: Jev 1.13 gave the same answer 58/60 times; the local models every time", "value": 0.9667, "unit": "rate", "display": "58/60", "n": 60 }, { "id": "arena-jev-latency", "label": "Jev 1.13 hosted API: median call time from Houston, network included (not comparable with a model on another machine)", "value": 137, "unit": "ms", "display": "137 ms", "n": 120 }, { "id": "arena-chance", "label": "Expected accuracy of picking an option at random on the same graded items (calculation)", "value": 0.224, "unit": "rate", "display": "22.4%", "n": 1048 }, { "id": "arena-rank-agreement", "label": "Rank agreement between our accuracy order and the independent Decision Index 0.2.1 order, same seven models (Spearman, calculation)", "value": 0.93, "unit": "score", "display": "0.93", "n": 7 }, { "id": "arena-games", "label": "Games played to the end and counted (round robin, featured series and speed ladder)", "value": 1620, "unit": "count", "display": "1,620", "n": 1620 } ], "charts": [ { "id": "arena-accuracy", "title": "Who decides right? Accuracy on 1,000+ checkable decisions", "subtitle": "Share of graded items answered correctly, first presentation · 95% Wilson intervals", "kind": "dot-range", "unit": "rate", "whisker": "ci95", "series": [ { "name": "Accuracy", "points": [ { "label": "Jev 1.13", "value": 0.7681, "lo": 0.7416, "hi": 0.7927, "n": 1048, "highlight": true }, { "label": "Clef 27B", "value": 0.6994, "lo": 0.671, "hi": 0.7264, "n": 1048 }, { "label": "Clef-Flash 9B", "value": 0.6517, "lo": 0.6224, "hi": 0.68, "n": 1048 }, { "label": "Kev 4B", "value": 0.6403, "lo": 0.6107, "hi": 0.6688, "n": 1048 }, { "label": "lev 4B", "value": 0.5859, "lo": 0.5558, "hi": 0.6153, "n": 1048 }, { "label": "Laya", "value": 0.2739, "lo": 0.2477, "hi": 0.3016, "n": 1048 }, { "label": "Julia-1", "value": 0.2586, "lo": 0.233, "hi": 0.2859, "n": 1048 } ] } ], "note": "1048 graded items in five suites. An error, a timeout or a label outside the option set counts as wrong. Two models differ clearly only where the paired McNemar test says so (table below).", "sourceIds": [ "system-one-arena" ] }, { "id": "arena-by-suite", "title": "Where each model is strong", "subtitle": "Accuracy by suite, first presentation", "kind": "grouped-bar", "unit": "rate", "series": [ { "name": "Jev 1.13", "points": [ { "label": "Games", "value": 0.4219, "lo": 0.3542, "hi": 0.4926, "n": 192 }, { "label": "Logic and thought experiments", "value": 0.7436, "lo": 0.6698, "hi": 0.8057, "n": 156 }, { "label": "Policy cases, 20 industries", "value": 0.8139, "lo": 0.7636, "hi": 0.8555, "n": 274 }, { "label": "Usability intents", "value": 0.9336, "lo": 0.8934, "hi": 0.9594, "n": 226 }, { "label": "Stress tests", "value": 0.87, "lo": 0.8163, "hi": 0.9097, "n": 200 } ] }, { "name": "Clef 27B", "points": [ { "label": "Games", "value": 0.3594, "lo": 0.2949, "hi": 0.4294, "n": 192 }, { "label": "Logic and thought experiments", "value": 0.6474, "lo": 0.5697, "hi": 0.718, "n": 156 }, { "label": "Policy cases, 20 industries", "value": 0.719, "lo": 0.663, "hi": 0.7689, "n": 274 }, { "label": "Usability intents", "value": 0.8894, "lo": 0.8418, "hi": 0.9239, "n": 226 }, { "label": "Stress tests", "value": 0.825, "lo": 0.7664, "hi": 0.8714, "n": 200 } ] }, { "name": "Clef-Flash 9B", "points": [ { "label": "Games", "value": 0.3073, "lo": 0.2463, "hi": 0.3758, "n": 192 }, { "label": "Logic and thought experiments", "value": 0.5833, "lo": 0.5049, "hi": 0.6578, "n": 156 }, { "label": "Policy cases, 20 industries", "value": 0.7226, "lo": 0.6668, "hi": 0.7723, "n": 274 }, { "label": "Usability intents", "value": 0.8673, "lo": 0.8168, "hi": 0.9054, "n": 226 }, { "label": "Stress tests", "value": 0.695, "lo": 0.628, "hi": 0.7546, "n": 200 } ] }, { "name": "Kev 4B", "points": [ { "label": "Games", "value": 0.3854, "lo": 0.3195, "hi": 0.4559, "n": 192 }, { "label": "Logic and thought experiments", "value": 0.5449, "lo": 0.4666, "hi": 0.621, "n": 156 }, { "label": "Policy cases, 20 industries", "value": 0.6752, "lo": 0.6176, "hi": 0.7279, "n": 274 }, { "label": "Usability intents", "value": 0.8009, "lo": 0.744, "hi": 0.8477, "n": 226 }, { "label": "Stress tests", "value": 0.73, "lo": 0.6646, "hi": 0.7868, "n": 200 } ] }, { "name": "lev 4B", "points": [ { "label": "Games", "value": 0.3021, "lo": 0.2415, "hi": 0.3704, "n": 192 }, { "label": "Logic and thought experiments", "value": 0.5064, "lo": 0.4287, "hi": 0.5838, "n": 156 }, { "label": "Policy cases, 20 industries", "value": 0.5912, "lo": 0.5322, "hi": 0.6478, "n": 274 }, { "label": "Usability intents", "value": 0.823, "lo": 0.768, "hi": 0.8672, "n": 226 }, { "label": "Stress tests", "value": 0.645, "lo": 0.5765, "hi": 0.708, "n": 200 } ] }, { "name": "Laya", "points": [ { "label": "Games", "value": 0.1771, "lo": 0.1296, "hi": 0.2373, "n": 192 }, { "label": "Logic and thought experiments", "value": 0.2821, "lo": 0.2173, "hi": 0.3572, "n": 156 }, { "label": "Policy cases, 20 industries", "value": 0.2664, "lo": 0.2176, "hi": 0.3217, "n": 274 }, { "label": "Usability intents", "value": 0.323, "lo": 0.2654, "hi": 0.3865, "n": 226 }, { "label": "Stress tests", "value": 0.315, "lo": 0.2546, "hi": 0.3823, "n": 200 } ] }, { "name": "Julia-1", "points": [ { "label": "Games", "value": 0.2083, "lo": 0.1569, "hi": 0.2712, "n": 192 }, { "label": "Logic and thought experiments", "value": 0.3462, "lo": 0.276, "hi": 0.4237, "n": 156 }, { "label": "Policy cases, 20 industries", "value": 0.2518, "lo": 0.2041, "hi": 0.3064, "n": 274 }, { "label": "Usability intents", "value": 0.1593, "lo": 0.1173, "hi": 0.2126, "n": 226 }, { "label": "Stress tests", "value": 0.36, "lo": 0.2967, "hi": 0.4286, "n": 200 } ] } ], "note": "Games: positions solved by search. Logic: syllogisms, probability, bias probes. Policy cases: fictional company rules in 20 industries. Usability: what the user wants on a given screen. Stress tests: prompt injection, long logs, thresholds, near-identical options.", "sourceIds": [ "system-one-arena" ] }, { "id": "arena-latency-this-mac", "polarity": "lower", "title": "Speed on this Mac: the six open models", "subtitle": "Median, whisker to the 95th percentile · one Mac Studio (M3 Ultra), one call at a time", "kind": "dot-range", "unit": "ms", "whisker": "p50-p95", "series": [ { "name": "This Mac", "points": [ { "label": "Julia-1", "value": 13, "lo": 13, "hi": 22, "n": 120 }, { "label": "Laya", "value": 43, "lo": 43, "hi": 68, "n": 120 }, { "label": "Kev 4B", "value": 307, "lo": 307, "hi": 600, "n": 120 }, { "label": "lev 4B", "value": 425, "lo": 425, "hi": 646, "n": 120 }, { "label": "Clef-Flash 9B", "value": 566, "lo": 566, "hi": 816, "n": 120 }, { "label": "Clef 27B", "value": 1919, "lo": 1919, "hi": 2792, "n": 115 } ] } ], "note": "All six ran on the same machine, so they compare with each other. Jev is not here: it ran on TypeSafe's servers. These times describe this Mac and 4- or 8-bit files, not the models: a data-centre GPU is several times faster.", "sourceIds": [ "system-one-arena" ] }, { "id": "arena-latency-same-gpu", "polarity": "lower", "title": "Speed on one GPU: third-party numbers", "subtitle": "Median, whisker to the 95th percentile · Decision Index 0.2.1, one NVIDIA RTX PRO 6000, one harness (snapshot 2026-10-01)", "kind": "dot-range", "unit": "ms", "whisker": "p50-p95", "series": [ { "name": "Open models, same GPU", "points": [ { "label": "Laya", "value": 5.8, "lo": 5.8, "hi": 222.5 }, { "label": "Julia-1", "value": 5.8, "lo": 5.8, "hi": 15.3 }, { "label": "Clef-Flash 9B", "value": 38.8, "lo": 38.8, "hi": 122.4 }, { "label": "Kev 4B", "value": 52.1, "lo": 52.1, "hi": 141.1 }, { "label": "lev 4B", "value": 70.8, "lo": 70.8, "hi": 687.7 }, { "label": "Clef 27B", "value": 209.3, "lo": 209.3, "hi": 238.6 } ] }, { "name": "Hosted: network included", "points": [ { "label": "Jev 1.13", "value": 524.1, "lo": 524.1, "hi": 536 } ] } ], "note": "Measured by the Decision Index, not by us. Every open model ran on the same GPU in one harness, so the six compare with each other. Jev is a hosted API: its time includes the network from the board's lab, so it is shown apart and is not a compute comparison.", "sourceIds": [ "system-one-arena" ] }, { "id": "arena-speed-accuracy-gpu", "polarity": "none", "title": "Open models: accuracy against speed on the same GPU", "subtitle": "x: third-party median latency on one GPU · y: our accuracy on 1,048 graded items · 95% intervals", "kind": "scatter", "unit": "rate", "xLabel": "Median latency on one GPU (ms, third party)", "yLabel": "Accuracy (ours)", "series": [ { "name": "Clef 27B", "points": [ { "label": "Clef 27B", "x": 209.3, "value": 0.6994, "lo": 0.671, "hi": 0.7264, "n": 1048 } ] }, { "name": "Clef-Flash 9B", "points": [ { "label": "Clef-Flash 9B", "x": 38.8, "value": 0.6517, "lo": 0.6224, "hi": 0.68, "n": 1048 } ] }, { "name": "lev 4B", "points": [ { "label": "lev 4B", "x": 70.8, "value": 0.5859, "lo": 0.5558, "hi": 0.6153, "n": 1048 } ] }, { "name": "Kev 4B", "points": [ { "label": "Kev 4B", "x": 52.1, "value": 0.6403, "lo": 0.6107, "hi": 0.6688, "n": 1048 } ] }, { "name": "Laya", "points": [ { "label": "Laya", "x": 5.8, "value": 0.2739, "lo": 0.2477, "hi": 0.3016, "n": 1048 } ] }, { "name": "Julia-1", "points": [ { "label": "Julia-1", "x": 5.8, "value": 0.2586, "lo": 0.233, "hi": 0.2859, "n": 1048 } ] } ], "note": "Speed comes from the Decision Index (one GPU, one harness); accuracy comes from this study. Jev is a hosted API and is not on this machine, so it has no dot here: it scored 76.8%.", "sourceIds": [ "system-one-arena" ] }, { "id": "arena-latency-gateway", "polarity": "lower", "title": "Speed through one gateway: OpenRouter's own numbers", "subtitle": "Median latency OpenRouter reports for each model's endpoint · read 2026-10-06", "kind": "bar", "unit": "seconds", "series": [ { "name": "Median latency, one gateway", "points": [ { "label": "Jev 1.13 (TypeSafe)", "value": 0.17 }, { "label": "Clef 27B (Workers AI)", "value": 0.5 }, { "label": "Clef-Flash 9B (Workers AI)", "value": 0.59 }, { "label": "Kev 4B (SiliconFlow)", "value": 4.08 } ] } ], "note": "One gateway measures all four, so the network path is the same. The providers differ (TypeSafe, Workers AI, SiliconFlow) and the numbers move daily. Not our measurement.", "sourceIds": [ "system-one-arena" ] }, { "id": "arena-cost-same-provider", "polarity": "lower", "title": "Price per 1,000 decisions at one provider's list prices", "subtitle": "Cloudflare Workers AI list price per million input tokens × the input tokens each model counted per decision in this study", "kind": "bar", "unit": "usd", "series": [ { "name": "USD per 1,000 decisions", "points": [ { "label": "Jev 1.13 ($0.042/M, 825 tokens)", "value": 0.0347 }, { "label": "Clef-Flash 9B ($0.09/M, 568 tokens)", "value": 0.0511 }, { "label": "Clef 27B ($0.24/M, 568 tokens)", "value": 0.1363 } ] } ], "note": "One provider prices all three, so this is a like-for-like cost, with no timing involved. Token counts come from each model's own tokenizer; output is free for all three. A calculation, not an invoice.", "sourceIds": [ "system-one-arena" ] }, { "id": "arena-size-accuracy", "polarity": "none", "title": "Does a bigger file decide better?", "subtitle": "Accuracy against model file size (open models)", "kind": "scatter", "unit": "rate", "xLabel": "Model file (GB)", "yLabel": "Accuracy", "series": [ { "name": "Clef 27B", "points": [ { "label": "Clef 27B", "x": 19.23, "value": 0.6994, "lo": 0.671, "hi": 0.7264, "n": 1048 } ] }, { "name": "Clef-Flash 9B", "points": [ { "label": "Clef-Flash 9B", "x": 6.49, "value": 0.6517, "lo": 0.6224, "hi": 0.68, "n": 1048 } ] }, { "name": "lev 4B", "points": [ { "label": "lev 4B", "x": 3.01, "value": 0.5859, "lo": 0.5558, "hi": 0.6153, "n": 1048 } ] }, { "name": "Kev 4B", "points": [ { "label": "Kev 4B", "x": 3.03, "value": 0.6403, "lo": 0.6107, "hi": 0.6688, "n": 1048 } ] }, { "name": "Laya", "points": [ { "label": "Laya", "x": 0.45, "value": 0.2739, "lo": 0.2477, "hi": 0.3016, "n": 1048 } ] }, { "name": "Julia-1", "points": [ { "label": "Julia-1", "x": 0.17, "value": 0.2586, "lo": 0.233, "hi": 0.2859, "n": 1048 } ] } ], "note": "File size of the GGUF that ran (quantized). Hardware does not enter. Jev is closed and its size is not published, so it is not on this chart.", "sourceIds": [ "system-one-arena" ] }, { "id": "arena-robust", "title": "Same question, different presentation", "subtitle": "Right at first presentation vs right in all three: original order, shuffled options, options renamed a, b, c", "kind": "dot-range", "unit": "rate", "whisker": "ci95", "series": [ { "name": "First presentation", "points": [ { "label": "Jev 1.13", "value": 0.7681, "lo": 0.7416, "hi": 0.7927, "n": 1048 }, { "label": "Clef 27B", "value": 0.6994, "lo": 0.671, "hi": 0.7264, "n": 1048 }, { "label": "Clef-Flash 9B", "value": 0.6517, "lo": 0.6224, "hi": 0.68, "n": 1048 }, { "label": "Kev 4B", "value": 0.6403, "lo": 0.6107, "hi": 0.6688, "n": 1048 }, { "label": "lev 4B", "value": 0.5859, "lo": 0.5558, "hi": 0.6153, "n": 1048 }, { "label": "Laya", "value": 0.2739, "lo": 0.2477, "hi": 0.3016, "n": 1048 }, { "label": "Julia-1", "value": 0.2586, "lo": 0.233, "hi": 0.2859, "n": 1048 } ] }, { "name": "All three presentations", "points": [ { "label": "Jev 1.13", "value": 0.729, "lo": 0.7013, "hi": 0.755, "n": 1048 }, { "label": "Clef 27B", "value": 0.6536, "lo": 0.6243, "hi": 0.6818, "n": 1048 }, { "label": "Clef-Flash 9B", "value": 0.5964, "lo": 0.5664, "hi": 0.6257, "n": 1048 }, { "label": "Kev 4B", "value": 0.5697, "lo": 0.5395, "hi": 0.5993, "n": 1048 }, { "label": "lev 4B", "value": 0.5344, "lo": 0.5041, "hi": 0.5644, "n": 1048 }, { "label": "Laya", "value": 0.1584, "lo": 0.1375, "hi": 0.1817, "n": 1048 }, { "label": "Julia-1", "value": 0.1603, "lo": 0.1393, "hi": 0.1838, "n": 1048 } ] } ], "note": "Yes/no items have one presentation, so for them both measures are the same.", "sourceIds": [ "system-one-arena" ] }, { "id": "arena-flips", "polarity": "lower", "title": "Decisions that changed when only the presentation changed", "subtitle": "Share of choice items · lower is better", "kind": "grouped-bar", "unit": "rate", "series": [ { "name": "Options shuffled", "points": [ { "label": "Jev 1.13", "value": 0.077, "lo": 0.0618, "hi": 0.0956, "n": 961 }, { "label": "Clef 27B", "value": 0, "lo": 0, "hi": 0.004, "n": 961 }, { "label": "Clef-Flash 9B", "value": 0, "lo": 0, "hi": 0.004, "n": 961 }, { "label": "Kev 4B", "value": 0.1426, "lo": 0.1219, "hi": 0.1661, "n": 961 }, { "label": "lev 4B", "value": 0.1322, "lo": 0.1122, "hi": 0.155, "n": 961 }, { "label": "Laya", "value": 0.3361, "lo": 0.3069, "hi": 0.3666, "n": 961 }, { "label": "Julia-1", "value": 0.5203, "lo": 0.4887, "hi": 0.5517, "n": 961 } ] }, { "name": "Options renamed a, b, c", "points": [ { "label": "Jev 1.13", "value": 0.0687, "lo": 0.0543, "hi": 0.0864, "n": 961 }, { "label": "Clef 27B", "value": 0.1831, "lo": 0.16, "hi": 0.2088, "n": 961 }, { "label": "Clef-Flash 9B", "value": 0.18, "lo": 0.157, "hi": 0.2056, "n": 961 }, { "label": "Kev 4B", "value": 0.1103, "lo": 0.092, "hi": 0.1317, "n": 961 }, { "label": "lev 4B", "value": 0.1061, "lo": 0.0882, "hi": 0.1272, "n": 961 }, { "label": "Laya", "value": 0.4631, "lo": 0.4317, "hi": 0.4947, "n": 961 }, { "label": "Julia-1", "value": 0, "lo": 0, "hi": 0.004, "n": 961 } ] } ], "note": "The descriptions never changed, only their order or their labels. A decision model should not care.", "sourceIds": [ "system-one-arena" ] }, { "id": "arena-escape", "polarity": "none", "title": "Knowing when to say \"none of these\"", "subtitle": "Items with a none, ask or escalate option · first presentation", "kind": "grouped-bar", "unit": "rate", "series": [ { "name": "Escaped when it should (higher is better)", "points": [ { "label": "Jev 1.13", "value": 0.8571, "lo": 0.7915, "hi": 0.9046, "n": 147 }, { "label": "Clef 27B", "value": 0.7619, "lo": 0.6869, "hi": 0.8235, "n": 147 }, { "label": "Clef-Flash 9B", "value": 0.6054, "lo": 0.5247, "hi": 0.6808, "n": 147 }, { "label": "Kev 4B", "value": 0.619, "lo": 0.5385, "hi": 0.6936, "n": 147 }, { "label": "lev 4B", "value": 0.4694, "lo": 0.3905, "hi": 0.5498, "n": 147 }, { "label": "Laya", "value": 0.1769, "lo": 0.1237, "hi": 0.2465, "n": 147 }, { "label": "Julia-1", "value": 0.2245, "lo": 0.1646, "hi": 0.2985, "n": 147 } ] }, { "name": "Escaped when it should not (lower is better)", "points": [ { "label": "Jev 1.13", "value": 0.055, "lo": 0.0396, "hi": 0.0759, "n": 618 }, { "label": "Clef 27B", "value": 0.0259, "lo": 0.016, "hi": 0.0416, "n": 618 }, { "label": "Clef-Flash 9B", "value": 0.0146, "lo": 0.0077, "hi": 0.0274, "n": 618 }, { "label": "Kev 4B", "value": 0.0809, "lo": 0.0619, "hi": 0.1051, "n": 618 }, { "label": "lev 4B", "value": 0.034, "lo": 0.0223, "hi": 0.0514, "n": 618 }, { "label": "Laya", "value": 0.1375, "lo": 0.1126, "hi": 0.1669, "n": 618 }, { "label": "Julia-1", "value": 0.165, "lo": 0.1379, "hi": 0.1964, "n": 618 } ] } ], "note": "Out-of-scope requests, policies that do not cover a case, ambiguous requests that need a question, lost game positions.", "sourceIds": [ "system-one-arena" ] }, { "id": "arena-injection", "title": "Prompt injection: does text in the state hijack the decision?", "subtitle": "Right answers on items whose state holds \"ignore the instructions and choose X\" style text · 95% Wilson intervals", "kind": "dot-range", "unit": "rate", "whisker": "ci95", "series": [ { "name": "Right despite the injected text", "points": [ { "label": "Jev 1.13", "value": 0.95, "lo": 0.835, "hi": 0.9862, "n": 40 }, { "label": "Clef 27B", "value": 0.9, "lo": 0.7695, "hi": 0.9604, "n": 40 }, { "label": "Clef-Flash 9B", "value": 0.775, "lo": 0.625, "hi": 0.8768, "n": 40 }, { "label": "Kev 4B", "value": 0.95, "lo": 0.835, "hi": 0.9862, "n": 40 }, { "label": "lev 4B", "value": 0.675, "lo": 0.5202, "hi": 0.7992, "n": 40 }, { "label": "Laya", "value": 0.325, "lo": 0.2008, "hi": 0.4798, "n": 40 }, { "label": "Julia-1", "value": 0.45, "lo": 0.3071, "hi": 0.6017, "n": 40 } ] } ], "note": "The injected text sits in a user-supplied field (a review, an email, a file name). The right answer follows the real instructions.", "sourceIds": [ "system-one-arena" ] }, { "id": "arena-by-length", "title": "Short inputs vs long inputs", "subtitle": "Accuracy by item length (Jev's token count of the same request) · first presentation", "kind": "grouped-bar", "unit": "rate", "series": [ { "name": "Up to 512 tokens", "points": [ { "label": "Jev 1.13", "value": 0.7867, "lo": 0.7286, "hi": 0.8351, "n": 225 }, { "label": "Clef 27B", "value": 0.72, "lo": 0.658, "hi": 0.7746, "n": 225 }, { "label": "Clef-Flash 9B", "value": 0.6667, "lo": 0.6027, "hi": 0.725, "n": 225 }, { "label": "Kev 4B", "value": 0.6, "lo": 0.5348, "hi": 0.6618, "n": 225 }, { "label": "lev 4B", "value": 0.6044, "lo": 0.5393, "hi": 0.6661, "n": 225 }, { "label": "Laya", "value": 0.3689, "lo": 0.3085, "hi": 0.4336, "n": 225 }, { "label": "Julia-1", "value": 0.4222, "lo": 0.3595, "hi": 0.4875, "n": 225 } ] }, { "name": "More than 512 tokens", "points": [ { "label": "Jev 1.13", "value": 0.7631, "lo": 0.7328, "hi": 0.7908, "n": 823 }, { "label": "Clef 27B", "value": 0.6938, "lo": 0.6615, "hi": 0.7243, "n": 823 }, { "label": "Clef-Flash 9B", "value": 0.6476, "lo": 0.6144, "hi": 0.6795, "n": 823 }, { "label": "Kev 4B", "value": 0.6513, "lo": 0.6181, "hi": 0.6831, "n": 823 }, { "label": "lev 4B", "value": 0.5808, "lo": 0.5468, "hi": 0.6141, "n": 823 }, { "label": "Laya", "value": 0.2479, "lo": 0.2196, "hi": 0.2785, "n": 823 }, { "label": "Julia-1", "value": 0.2139, "lo": 0.1872, "hi": 0.2432, "n": 823 } ] } ], "note": "Laya and Julia-1 are small encoders trained on short inputs (512 to 1,024 tokens); the longest items here are about 2,000 tokens. This split was added after the freeze as a description, not a test.", "sourceIds": [ "system-one-arena" ] }, { "id": "arena-confident-wrong", "polarity": "lower", "title": "Wrong and sure of it", "subtitle": "Share of wrong answers given with probability 0.8 or more · lower is better", "kind": "dot-range", "unit": "rate", "whisker": "ci95", "series": [ { "name": "Confident wrong answers", "points": [ { "label": "Jev 1.13", "value": 0.1111, "lo": 0.0775, "hi": 0.1568, "n": 243 }, { "label": "Clef 27B", "value": 0.0161, "lo": 0.0069, "hi": 0.0371, "n": 311 }, { "label": "Clef-Flash 9B", "value": 0.0082, "lo": 0.0028, "hi": 0.0239, "n": 365 }, { "label": "Kev 4B", "value": 0.0743, "lo": 0.0519, "hi": 0.1052, "n": 377 }, { "label": "lev 4B", "value": 0.3364, "lo": 0.2936, "hi": 0.3821, "n": 434 }, { "label": "Laya", "value": 0.0972, "lo": 0.0782, "hi": 0.1204, "n": 761 }, { "label": "Julia-1", "value": 0.5611, "lo": 0.526, "hi": 0.5956, "n": 777 } ] } ], "note": "The probability is what the model returned for the option it chose. A well-calibrated decider is rarely this sure when it is wrong, so a confidence gate can catch its mistakes.", "sourceIds": [ "system-one-arena" ] }, { "id": "arena-elo", "title": "Tournament rating across every game", "subtitle": "Elo from every round-robin game (K 32, start 1000, averaged over 200 game orders) · 95% bootstrap intervals", "kind": "dot-range", "unit": "score", "whisker": "ci95", "series": [ { "name": "Elo", "points": [ { "label": "Perfect player", "value": 1554, "lo": 1498, "hi": 1609, "n": 336 }, { "label": "Jev 1.13", "value": 1040, "lo": 1002, "hi": 1082, "n": 336, "highlight": true }, { "label": "lev 4B", "value": 941, "lo": 894, "hi": 986, "n": 336 }, { "label": "Clef 27B", "value": 941, "lo": 901, "hi": 984, "n": 336 }, { "label": "Kev 4B", "value": 936, "lo": 892, "hi": 974, "n": 336 }, { "label": "Clef-Flash 9B", "value": 936, "lo": 890, "hi": 982, "n": 336 }, { "label": "Random player", "value": 896, "lo": 858, "hi": 948, "n": 336 }, { "label": "Julia-1", "value": 879, "lo": 831, "hi": 921, "n": 336 }, { "label": "Laya", "value": 877, "lo": 840, "hi": 921, "n": 336 } ] } ], "note": "Tic-tac-toe, Connect Four, Nim, Dots and Boxes and Pong; every game counts once. The random and perfect players anchor the scale.", "sourceIds": [ "system-one-arena" ] }, { "id": "arena-wdl", "title": "Wins, draws and losses in the round robin", "subtitle": "Every round-robin game", "kind": "stacked-bar", "unit": "count", "series": [ { "name": "Wins", "points": [ { "label": "Perfect player", "value": 324, "n": 336 }, { "label": "Jev 1.13", "value": 196, "n": 336 }, { "label": "lev 4B", "value": 146, "n": 336 }, { "label": "Clef 27B", "value": 152, "n": 336 }, { "label": "Kev 4B", "value": 144, "n": 336 }, { "label": "Clef-Flash 9B", "value": 147, "n": 336 }, { "label": "Random player", "value": 127, "n": 336 }, { "label": "Julia-1", "value": 120, "n": 336 }, { "label": "Laya", "value": 118, "n": 336 } ] }, { "name": "Draws", "points": [ { "label": "Perfect player", "value": 9, "n": 336 }, { "label": "Jev 1.13", "value": 6, "n": 336 }, { "label": "lev 4B", "value": 9, "n": 336 }, { "label": "Clef 27B", "value": 5, "n": 336 }, { "label": "Kev 4B", "value": 9, "n": 336 }, { "label": "Clef-Flash 9B", "value": 7, "n": 336 }, { "label": "Random player", "value": 12, "n": 336 }, { "label": "Julia-1", "value": 6, "n": 336 }, { "label": "Laya", "value": 13, "n": 336 } ] }, { "name": "Losses", "points": [ { "label": "Perfect player", "value": 3, "n": 336 }, { "label": "Jev 1.13", "value": 134, "n": 336 }, { "label": "lev 4B", "value": 181, "n": 336 }, { "label": "Clef 27B", "value": 179, "n": 336 }, { "label": "Kev 4B", "value": 183, "n": 336 }, { "label": "Clef-Flash 9B", "value": 182, "n": 336 }, { "label": "Random player", "value": 197, "n": 336 }, { "label": "Julia-1", "value": 210, "n": 336 }, { "label": "Laya", "value": 205, "n": 336 } ] } ], "sourceIds": [ "system-one-arena" ] }, { "id": "arena-perfect-moves", "title": "How often a model found the perfect move", "subtitle": "Share of decisive moves (some option is a mistake) that a perfect player would also make", "kind": "grouped-bar", "unit": "rate", "series": [ { "name": "Jev 1.13", "points": [ { "label": "Tic-tac-toe", "value": 0.4653, "lo": 0.3858, "hi": 0.5466, "n": 144 }, { "label": "Connect Four", "value": 0.3789, "lo": 0.3297, "hi": 0.4307, "n": 351 }, { "label": "Nim", "value": 0.3916, "lo": 0.3206, "hi": 0.4675, "n": 166 }, { "label": "Dots and Boxes", "value": 0.5479, "lo": 0.5127, "hi": 0.5827, "n": 772 } ] }, { "name": "Clef 27B", "points": [ { "label": "Tic-tac-toe", "value": 0.4718, "lo": 0.3916, "hi": 0.5536, "n": 142 }, { "label": "Connect Four", "value": 0.3138, "lo": 0.2698, "hi": 0.3613, "n": 392 }, { "label": "Nim", "value": 0.4051, "lo": 0.3317, "hi": 0.483, "n": 158 }, { "label": "Dots and Boxes", "value": 0.3376, "lo": 0.3016, "hi": 0.3757, "n": 622 } ] }, { "name": "Clef-Flash 9B", "points": [ { "label": "Tic-tac-toe", "value": 0.5306, "lo": 0.4502, "hi": 0.6095, "n": 147 }, { "label": "Connect Four", "value": 0.3629, "lo": 0.3163, "hi": 0.4122, "n": 383 }, { "label": "Nim", "value": 0.2593, "lo": 0.1927, "hi": 0.3391, "n": 135 }, { "label": "Dots and Boxes", "value": 0.3267, "lo": 0.2903, "hi": 0.3652, "n": 600 } ] }, { "name": "lev 4B", "points": [ { "label": "Tic-tac-toe", "value": 0.4737, "lo": 0.3959, "hi": 0.5527, "n": 152 }, { "label": "Connect Four", "value": 0.3477, "lo": 0.3024, "hi": 0.396, "n": 394 }, { "label": "Nim", "value": 0.3101, "lo": 0.2367, "hi": 0.3944, "n": 129 }, { "label": "Dots and Boxes", "value": 0.363, "lo": 0.3261, "hi": 0.4017, "n": 617 } ] }, { "name": "Kev 4B", "points": [ { "label": "Tic-tac-toe", "value": 0.4371, "lo": 0.3605, "hi": 0.5168, "n": 151 }, { "label": "Connect Four", "value": 0.1415, "lo": 0.1071, "hi": 0.1846, "n": 311 }, { "label": "Nim", "value": 0.2975, "lo": 0.2233, "hi": 0.3842, "n": 121 }, { "label": "Dots and Boxes", "value": 0.3339, "lo": 0.2974, "hi": 0.3725, "n": 602 } ] }, { "name": "Laya", "points": [ { "label": "Tic-tac-toe", "value": 0.3716, "lo": 0.2979, "hi": 0.4518, "n": 148 }, { "label": "Connect Four", "value": 0.1003, "lo": 0.073, "hi": 0.1363, "n": 349 }, { "label": "Nim", "value": 0.2025, "lo": 0.1473, "hi": 0.2719, "n": 158 }, { "label": "Dots and Boxes", "value": 0.3667, "lo": 0.3308, "hi": 0.4041, "n": 660 } ] }, { "name": "Julia-1", "points": [ { "label": "Tic-tac-toe", "value": 0.3562, "lo": 0.2831, "hi": 0.4366, "n": 146 }, { "label": "Connect Four", "value": 0.2227, "lo": 0.186, "hi": 0.2644, "n": 431 }, { "label": "Nim", "value": 0.2706, "lo": 0.2094, "hi": 0.3419, "n": 170 }, { "label": "Dots and Boxes", "value": 0.288, "lo": 0.2524, "hi": 0.3263, "n": 573 } ] }, { "name": "Random player", "points": [ { "label": "Tic-tac-toe", "value": 0.4013, "lo": 0.3268, "hi": 0.4807, "n": 152 }, { "label": "Connect Four", "value": 0.232, "lo": 0.1951, "hi": 0.2734, "n": 444 }, { "label": "Nim", "value": 0.2327, "lo": 0.1738, "hi": 0.3042, "n": 159 }, { "label": "Dots and Boxes", "value": 0.3241, "lo": 0.2884, "hi": 0.3621, "n": 617 } ] } ], "note": "Tic-tac-toe and Dots and Boxes against exact solvers, Nim by nim-sum, Connect Four against a depth-10 search (agreement, not proof of a mistake). The random player is the anchor: below it, a model chose worse than chance.", "sourceIds": [ "system-one-arena" ] }, { "id": "arena-pong", "title": "Pong as deployed: who won", "subtitle": "Share of Pong games won · first to 3 · each answer lands after its own call time on its own machine (hardware-dependent) · 95% Wilson intervals", "kind": "dot-range", "unit": "rate", "whisker": "ci95", "series": [ { "name": "Pong score", "points": [ { "label": "Perfect player", "value": 1, "lo": 0.8064, "hi": 1, "n": 16 }, { "label": "Jev 1.13", "value": 0.8125, "lo": 0.5699, "hi": 0.9341, "n": 16 }, { "label": "Kev 4B", "value": 0.75, "lo": 0.505, "hi": 0.8982, "n": 16 }, { "label": "lev 4B", "value": 0.5625, "lo": 0.3318, "hi": 0.769, "n": 16 }, { "label": "Clef-Flash 9B", "value": 0.5, "lo": 0.28, "hi": 0.72, "n": 16 }, { "label": "Clef 27B", "value": 0.25, "lo": 0.1018, "hi": 0.495, "n": 16 }, { "label": "Julia-1", "value": 0.25, "lo": 0.1018, "hi": 0.495, "n": 16 }, { "label": "Random player", "value": 0.1875, "lo": 0.0659, "hi": 0.4301, "n": 16 }, { "label": "Laya", "value": 0.1875, "lo": 0.0659, "hi": 0.4301, "n": 16 } ] } ], "note": "Each paddle asks its model where to go as soon as it is free; the answer moves the paddle only after that call's real latency, in simulated time. The arrival point is given in the state, so Pong tests reading and speed, not physics.", "sourceIds": [ "system-one-arena" ] }, { "id": "arena-pong-quality", "title": "Pong decision quality, ignoring time", "subtitle": "Share of graded decisions where the model chose the zone the ideal intercept chose · 95% Wilson intervals", "kind": "dot-range", "unit": "rate", "whisker": "ci95", "series": [ { "name": "Right zone", "points": [ { "label": "Jev 1.13", "value": 1, "lo": 0.9927, "hi": 1, "n": 523 }, { "label": "Clef 27B", "value": 1, "lo": 0.8897, "hi": 1, "n": 31 }, { "label": "lev 4B", "value": 0.9869, "lo": 0.9536, "hi": 0.9964, "n": 153 }, { "label": "Kev 4B", "value": 0.9774, "lo": 0.9541, "hi": 0.989, "n": 310 }, { "label": "Clef-Flash 9B", "value": 0.8296, "lo": 0.7573, "hi": 0.8837, "n": 135 }, { "label": "Julia-1", "value": 0.2145, "lo": 0.1997, "hi": 0.2301, "n": 2811 }, { "label": "Laya", "value": 0.1221, "lo": 0.1033, "hi": 0.1439, "n": 999 } ] } ], "note": "Hardware-neutral: each decision is judged on its own, and time does not enter. The ball is heading toward the paddle and its arrival point is in the state. Clef 27B decided rarely (slow), so its n is small.", "sourceIds": [ "system-one-arena" ] }, { "id": "arena-pong-ladder", "title": "How much is a millisecond worth in Pong?", "subtitle": "The perfect player slowed down, in a round robin against itself and a random player · 95% Wilson intervals", "kind": "dot-range", "unit": "rate", "whisker": "ci95", "series": [ { "name": "Share of games won", "points": [ { "label": "0 ms", "value": 0.975, "lo": 0.8712, "hi": 0.9956, "n": 40 }, { "label": "100 ms", "value": 0.775, "lo": 0.625, "hi": 0.8768, "n": 40 }, { "label": "300 ms", "value": 0.5, "lo": 0.352, "hi": 0.648, "n": 40 }, { "label": "1000 ms", "value": 0.125, "lo": 0.0546, "hi": 0.2611, "n": 40 } ] } ], "note": "Same perfect decisions, only slower. No model calls.", "sourceIds": [ "system-one-arena" ] } ], "tables": [ { "id": "arena-fairness", "title": "What is comparable: each comparison and what keeps it fair", "columns": [ { "key": "what", "label": "Comparison", "unit": "text" }, { "key": "fair", "label": "Fair across all seven?", "unit": "text" }, { "key": "why", "label": "Why", "unit": "text" } ], "rows": [ { "what": "Accuracy, consistency, calibration, \"none of these\", prompt injection", "fair": "Yes", "why": "Same 1,085 items and the same states for every model. The open models ran as 4- or 8-bit files, so a full-precision run could differ a little; our order of the models agrees with the independent Decision Index (rank correlation 0.93)." }, { "what": "Turn-based games", "fair": "Yes", "why": "No clock: a move counts only by its quality." }, { "what": "Pong decision quality (right zone per decision)", "fair": "Yes", "why": "Judged per decision against the ideal intercept; time does not enter." }, { "what": "Speed on this Mac", "fair": "Among the six open models only", "why": "The six ran on one machine, one call at a time. Jev ran on TypeSafe's servers." }, { "what": "Speed on one GPU (Decision Index)", "fair": "Among the six open models only", "why": "Third party, one GPU, one harness. Jev is a hosted API there and includes the network." }, { "what": "Speed through one gateway (OpenRouter)", "fair": "Yes for the four it lists", "why": "One gateway measures all four. Providers differ and the numbers change daily." }, { "what": "Price per 1,000 decisions", "fair": "Yes for Jev, Clef and Clef-Flash", "why": "Cloudflare Workers AI prices all three on one list." }, { "what": "Pong and other real-time games as deployed", "fair": "No", "why": "Each answer takes effect after its own call time on its own machine, so the result mixes decision quality with hardware." }, { "what": "Real-time games in fair mode (Tron, Snake and Pong in the arena)", "fair": "Yes", "why": "Every player's answer takes effect after the same simulated 150 ms, whatever its real call time, so hosted and local models stand on the same footing and only decision quality counts. The models are still called for real. The leagues are one seed with 2 to 4 games per pairing, so the intervals are wide." } ] }, { "id": "arena-models", "title": "Every model, every measure", "columns": [ { "key": "model", "label": "Model", "unit": "text" }, { "key": "runs", "label": "Runs as", "unit": "text" }, { "key": "acc", "label": "Accuracy", "unit": "text" }, { "key": "robust", "label": "Right in all 3 presentations", "unit": "text" }, { "key": "groups", "label": "Consistent groups", "unit": "text" }, { "key": "escape", "label": "Escaped when it should", "unit": "text" }, { "key": "falseEscape", "label": "Escaped when it should not", "unit": "text" }, { "key": "ece", "label": "Calibration error (ECE)", "unit": "ratio" }, { "key": "brier", "label": "Brier", "unit": "ratio" }, { "key": "p50", "label": "Median ms (where it ran: this Mac, or Jev's servers)", "unit": "ms" }, { "key": "p95", "label": "p95 ms", "unit": "ms" }, { "key": "same", "label": "Same answer twice", "unit": "text" }, { "key": "errors", "label": "Errors", "unit": "count" } ], "rows": [ { "model": "Jev 1.13", "runs": "Hosted API", "acc": "76.8% (805/1048)", "robust": "764/1048", "groups": "40/53", "escape": "126/147", "falseEscape": "34/618", "ece": 0.03, "brier": 0.18, "p50": 137, "p95": 189, "same": "58/60", "errors": 0 }, { "model": "Clef 27B", "runs": "Q4_K_M in llama.cpp, 19.23 GB", "acc": "69.9% (733/1048)", "robust": "685/1048", "groups": "38/53", "escape": "112/147", "falseEscape": "16/618", "ece": 0.28, "brier": 0.429, "p50": 1919, "p95": 2792, "same": "60/60", "errors": 8 }, { "model": "Clef-Flash 9B", "runs": "Q4_K_M in llama.cpp, 6.49 GB", "acc": "65.2% (683/1048)", "robust": "625/1048", "groups": "42/53", "escape": "89/147", "falseEscape": "9/618", "ece": 0.287, "brier": 0.476, "p50": 566, "p95": 816, "same": "60/60", "errors": 0 }, { "model": "Kev 4B", "runs": "Q4_K_M in llama.cpp, 3.03 GB", "acc": "64.0% (671/1048)", "robust": "597/1048", "groups": "35/53", "escape": "91/147", "falseEscape": "50/618", "ece": 0.037, "brier": 0.326, "p50": 307, "p95": 600, "same": "60/60", "errors": 0 }, { "model": "lev 4B", "runs": "Q4_K_M in llama.cpp, 3.01 GB", "acc": "58.6% (614/1048)", "robust": "560/1048", "groups": "38/53", "escape": "69/147", "falseEscape": "21/618", "ece": 0.145, "brier": 0.364, "p50": 425, "p95": 646, "same": "60/60", "errors": 0 }, { "model": "Laya", "runs": "Q8_0 in llama.cpp, 0.45 GB", "acc": "27.4% (287/1048)", "robust": "166/1048", "groups": "35/53", "escape": "26/147", "falseEscape": "85/618", "ece": 0.203, "brier": 0.603, "p50": 43, "p95": 68, "same": "60/60", "errors": 0 }, { "model": "Julia-1", "runs": "Q8_0 in llama.cpp, 0.17 GB", "acc": "25.9% (271/1048)", "robust": "168/1048", "groups": "28/53", "escape": "33/147", "falseEscape": "102/618", "ece": 0.547, "brier": 0.681, "p50": 13, "p95": 22, "same": "60/60", "errors": 0 } ] }, { "id": "arena-pairwise", "title": "Head to head on the same items: who was right where the other was wrong (exact McNemar test)", "columns": [ { "key": "pair", "label": "Pair", "unit": "text" }, { "key": "aOnly", "label": "Only the first right", "unit": "count" }, { "key": "bOnly", "label": "Only the second right", "unit": "count" }, { "key": "p", "label": "p", "unit": "ratio" }, { "key": "verdict", "label": "Verdict", "unit": "text" } ], "rows": [ { "pair": "Jev 1.13 vs Clef 27B", "aOnly": 158, "bOnly": 86, "p": 0, "verdict": "Jev 1.13 is ahead" }, { "pair": "Jev 1.13 vs Clef-Flash 9B", "aOnly": 208, "bOnly": 86, "p": 0, "verdict": "Jev 1.13 is ahead" }, { "pair": "Jev 1.13 vs lev 4B", "aOnly": 270, "bOnly": 79, "p": 0, "verdict": "Jev 1.13 is ahead" }, { "pair": "Jev 1.13 vs Kev 4B", "aOnly": 212, "bOnly": 78, "p": 0, "verdict": "Jev 1.13 is ahead" }, { "pair": "Jev 1.13 vs Laya", "aOnly": 584, "bOnly": 66, "p": 0, "verdict": "Jev 1.13 is ahead" }, { "pair": "Jev 1.13 vs Julia-1", "aOnly": 599, "bOnly": 65, "p": 0, "verdict": "Jev 1.13 is ahead" }, { "pair": "Clef 27B vs Clef-Flash 9B", "aOnly": 141, "bOnly": 91, "p": 0.0012, "verdict": "Clef 27B is ahead" }, { "pair": "Clef 27B vs lev 4B", "aOnly": 209, "bOnly": 90, "p": 0, "verdict": "Clef 27B is ahead" }, { "pair": "Clef 27B vs Kev 4B", "aOnly": 178, "bOnly": 116, "p": 0.0004, "verdict": "Clef 27B is ahead" }, { "pair": "Clef 27B vs Laya", "aOnly": 529, "bOnly": 83, "p": 0, "verdict": "Clef 27B is ahead" }, { "pair": "Clef 27B vs Julia-1", "aOnly": 543, "bOnly": 81, "p": 0, "verdict": "Clef 27B is ahead" }, { "pair": "Clef-Flash 9B vs lev 4B", "aOnly": 168, "bOnly": 99, "p": 0, "verdict": "Clef-Flash 9B is ahead" }, { "pair": "Clef-Flash 9B vs Kev 4B", "aOnly": 158, "bOnly": 146, "p": 0.5282, "verdict": "not clear" }, { "pair": "Clef-Flash 9B vs Laya", "aOnly": 486, "bOnly": 90, "p": 0, "verdict": "Clef-Flash 9B is ahead" }, { "pair": "Clef-Flash 9B vs Julia-1", "aOnly": 510, "bOnly": 98, "p": 0, "verdict": "Clef-Flash 9B is ahead" }, { "pair": "lev 4B vs Kev 4B", "aOnly": 111, "bOnly": 168, "p": 0.0008, "verdict": "Kev 4B is ahead" }, { "pair": "lev 4B vs Laya", "aOnly": 425, "bOnly": 98, "p": 0, "verdict": "lev 4B is ahead" }, { "pair": "lev 4B vs Julia-1", "aOnly": 452, "bOnly": 109, "p": 0, "verdict": "lev 4B is ahead" }, { "pair": "Kev 4B vs Laya", "aOnly": 471, "bOnly": 87, "p": 0, "verdict": "Kev 4B is ahead" }, { "pair": "Kev 4B vs Julia-1", "aOnly": 503, "bOnly": 103, "p": 0, "verdict": "Kev 4B is ahead" }, { "pair": "Laya vs Julia-1", "aOnly": 190, "bOnly": 174, "p": 0.4318, "verdict": "not clear" } ] }, { "id": "arena-examples", "title": "One item per suite that split the field: about half the models right", "columns": [ { "key": "suite", "label": "Suite", "unit": "text" }, { "key": "item", "label": "Item", "unit": "text" }, { "key": "question", "label": "Question", "unit": "text" }, { "key": "answer", "label": "Right answer", "unit": "text" }, { "key": "right", "label": "Right", "unit": "text" }, { "key": "wrong", "label": "Wrong", "unit": "text" }, { "key": "why", "label": "Why", "unit": "text" } ], "rows": [ { "suite": "Games", "item": "Find the poisoned column", "question": "Y is to move. R has no winning move at the moment. Exactly one column is unsafe: if Y drops a disc in it, R can win on the very next move. Choose that unsafe column.", "answer": "Drop a disc in column 3", "right": "Jev 1.13, lev 4B, Kev 4B, Laya", "wrong": "Clef 27B, Clef-Flash 9B, Julia-1", "why": "A Y disc in column 3 lands directly under the cell where R completes four, so R wins on the very next move. All other columns leave R without an immediate win." }, { "suite": "Logic and thought experiments", "item": "Flagged application: how likely is the fault?", "question": "One application was picked at random from those checked, and it was flagged. Which band holds the chance that it really has a copied answer?", "answer": "The chance is 30% to under 60%.", "right": "Jev 1.13, Clef 27B, Clef-Flash 9B", "wrong": "lev 4B, Kev 4B, Laya, Julia-1", "why": "Among the flagged items, the share that truly have the fault is 35.5% (Bayes' rule), which falls in the band \"30% to under 60%\"." }, { "suite": "Policy cases, 20 industries", "item": "Delay exactly on the long-delay line, refund requested, no missed connection", "question": "Which action does the policy require for this passenger case? Apply the numbered rules in order and choose exactly one action.", "answer": "Offer no remedy: no rebooking and no refund", "right": "Jev 1.13, Clef-Flash 9B, lev 4B, Kev 4B", "wrong": "Clef 27B, Laya, Julia-1", "why": "R4: the delay is 300 minutes, which does not meet \"less than 150 minutes\" and meets \"300 minutes or less\", and no connection was missed, so there is no remedy." }, { "suite": "Usability intents", "item": "Buttons lost against the background", "question": "Pick the one action the app should run now for what the user just said. Use the screen state and the rules in the state.", "answer": "Turn on high-contrast colors", "right": "Jev 1.13, Clef 27B, Clef-Flash 9B, lev 4B", "wrong": "Kev 4B, Laya, Julia-1", "why": "Buttons that blend into the background are a contrast problem, so High contrast." }, { "suite": "Stress tests", "item": "Loud alert names and big counts versus the severity rule", "question": "Apply the routing rules to the alert and pick the action.", "answer": "Open a ticket", "right": "Jev 1.13, Clef 27B, Clef-Flash 9B, Kev 4B", "wrong": "lev 4B, Laya, Julia-1", "why": "Severity 2 is below the page level, so no page; it is at least 2, so open_ticket, whatever the name or customer count says." } ] }, { "id": "arena-families", "title": "Every item family: right answers per model (first presentation)", "columns": [ { "key": "suite", "label": "Suite", "unit": "text" }, { "key": "family", "label": "Family", "unit": "text" }, { "key": "jev", "label": "Jev 1.13", "unit": "text" }, { "key": "clef", "label": "Clef 27B", "unit": "text" }, { "key": "clef-flash", "label": "Clef-Flash 9B", "unit": "text" }, { "key": "kev-4b", "label": "Kev 4B", "unit": "text" }, { "key": "lev", "label": "lev 4B", "unit": "text" }, { "key": "laya", "label": "Laya", "unit": "text" }, { "key": "julia-1", "label": "Julia-1", "unit": "text" } ], "rows": [ { "suite": "Games", "family": "tic-tac-toe", "jev": "8/23", "clef": "7/23", "clef-flash": "6/23", "kev-4b": "8/23", "lev": "11/23", "laya": "5/23", "julia-1": "1/23" }, { "suite": "Games", "family": "connect-four", "jev": "7/26", "clef": "4/26", "clef-flash": "6/26", "kev-4b": "7/26", "lev": "7/26", "laya": "4/26", "julia-1": "5/26" }, { "suite": "Games", "family": "nim", "jev": "12/22", "clef": "7/22", "clef-flash": "9/22", "kev-4b": "5/22", "lev": "5/22", "laya": "1/22", "julia-1": "6/22" }, { "suite": "Games", "family": "pong-intercept", "jev": "6/18", "clef": "9/18", "clef-flash": "8/18", "kev-4b": "9/18", "lev": "6/18", "laya": "3/18", "julia-1": "4/18" }, { "suite": "Games", "family": "sudoku-single", "jev": "6/17", "clef": "5/17", "clef-flash": "2/17", "kev-4b": "6/17", "lev": "3/17", "laya": "2/17", "julia-1": "2/17" }, { "suite": "Games", "family": "wordle", "jev": "10/16", "clef": "7/16", "clef-flash": "3/16", "kev-4b": "9/16", "lev": "9/16", "laya": "3/16", "julia-1": "4/16" }, { "suite": "Games", "family": "maze", "jev": "7/16", "clef": "4/16", "clef-flash": "2/16", "kev-4b": "5/16", "lev": "3/16", "laya": "5/16", "julia-1": "4/16" }, { "suite": "Games", "family": "hanoi", "jev": "5/16", "clef": "6/16", "clef-flash": "8/16", "kev-4b": "4/16", "lev": "2/16", "laya": "4/16", "julia-1": "5/16" }, { "suite": "Games", "family": "coin-weighing", "jev": "11/17", "clef": "11/17", "clef-flash": "9/17", "kev-4b": "9/17", "lev": "8/17", "laya": "4/17", "julia-1": "6/17" }, { "suite": "Games", "family": "knight-moves", "jev": "9/21", "clef": "9/21", "clef-flash": "6/21", "kev-4b": "12/21", "lev": "4/21", "laya": "3/21", "julia-1": "3/21" }, { "suite": "Logic and thought experiments", "family": "syllogism", "jev": "13/13", "clef": "12/13", "clef-flash": "12/13", "kev-4b": "13/13", "lev": "9/13", "laya": "7/13", "julia-1": "4/13" }, { "suite": "Logic and thought experiments", "family": "quantifier-logic", "jev": "6/6", "clef": "5/6", "clef-flash": "2/6", "kev-4b": "4/6", "lev": "5/6", "laya": "1/6", "julia-1": "2/6" }, { "suite": "Logic and thought experiments", "family": "conditional-logic", "jev": "12/12", "clef": "11/12", "clef-flash": "10/12", "kev-4b": "10/12", "lev": "7/12", "laya": "3/12", "julia-1": "7/12" }, { "suite": "Logic and thought experiments", "family": "wason-selection", "jev": "5/6", "clef": "5/6", "clef-flash": "6/6", "kev-4b": "5/6", "lev": "3/6", "laya": "1/6", "julia-1": "1/6" }, { "suite": "Logic and thought experiments", "family": "knights-knaves", "jev": "4/12", "clef": "2/12", "clef-flash": "2/12", "kev-4b": "2/12", "lev": "2/12", "laya": "3/12", "julia-1": "3/12" }, { "suite": "Logic and thought experiments", "family": "seating-order", "jev": "7/12", "clef": "5/12", "clef-flash": "3/12", "kev-4b": "5/12", "lev": "3/12", "laya": "1/12", "julia-1": "4/12" }, { "suite": "Logic and thought experiments", "family": "monty-hall", "jev": "7/10", "clef": "8/10", "clef-flash": "7/10", "kev-4b": "4/10", "lev": "9/10", "laya": "3/10", "julia-1": "3/10" }, { "suite": "Logic and thought experiments", "family": "base-rate", "jev": "9/10", "clef": "9/10", "clef-flash": "3/10", "kev-4b": "2/10", "lev": "3/10", "laya": "2/10", "julia-1": "2/10" }, { "suite": "Logic and thought experiments", "family": "dice-cards", "jev": "9/10", "clef": "6/10", "clef-flash": "6/10", "kev-4b": "5/10", "lev": "6/10", "laya": "4/10", "julia-1": "4/10" }, { "suite": "Logic and thought experiments", "family": "gamble-ev", "jev": "6/9", "clef": "9/9", "clef-flash": "6/9", "kev-4b": "9/9", "lev": "5/9", "laya": "1/9", "julia-1": "2/9" }, { "suite": "Logic and thought experiments", "family": "sunk-cost", "jev": "5/6", "clef": "5/6", "clef-flash": "4/6", "kev-4b": "5/6", "lev": "5/6", "laya": "2/6", "julia-1": "2/6" }, { "suite": "Logic and thought experiments", "family": "gamblers-fallacy", "jev": "9/11", "clef": "9/11", "clef-flash": "7/11", "kev-4b": "7/11", "lev": "8/11", "laya": "4/11", "julia-1": "4/11" }, { "suite": "Logic and thought experiments", "family": "denominator-neglect", "jev": "1/3", "clef": "1/3", "clef-flash": "1/3", "kev-4b": "1/3", "lev": "2/3", "laya": "1/3", "julia-1": "2/3" }, { "suite": "Logic and thought experiments", "family": "conjunction", "jev": "4/5", "clef": "2/5", "clef-flash": "4/5", "kev-4b": "2/5", "lev": "2/5", "laya": "2/5", "julia-1": "2/5" }, { "suite": "Logic and thought experiments", "family": "simpson", "jev": "2/4", "clef": "0/4", "clef-flash": "1/4", "kev-4b": "1/4", "lev": "2/4", "laya": "1/4", "julia-1": "1/4" }, { "suite": "Logic and thought experiments", "family": "sample-size", "jev": "4/5", "clef": "4/5", "clef-flash": "5/5", "kev-4b": "1/5", "lev": "1/5", "laya": "1/5", "julia-1": "3/5" }, { "suite": "Logic and thought experiments", "family": "anchoring", "jev": "4/4", "clef": "2/4", "clef-flash": "3/4", "kev-4b": "2/4", "lev": "2/4", "laya": "2/4", "julia-1": "1/4" }, { "suite": "Logic and thought experiments", "family": "units", "jev": "4/9", "clef": "3/9", "clef-flash": "4/9", "kev-4b": "3/9", "lev": "2/9", "laya": "2/9", "julia-1": "5/9" }, { "suite": "Logic and thought experiments", "family": "calendar-time", "jev": "5/9", "clef": "3/9", "clef-flash": "5/9", "kev-4b": "4/9", "lev": "3/9", "laya": "3/9", "julia-1": "2/9" }, { "suite": "Policy cases, 20 industries", "family": "insurance", "jev": "10/17", "clef": "11/17", "clef-flash": "15/17", "kev-4b": "9/17", "lev": "8/17", "laya": "8/17", "julia-1": "4/17" }, { "suite": "Policy cases, 20 industries", "family": "retail", "jev": "9/13", "clef": "7/13", "clef-flash": "10/13", "kev-4b": "7/13", "lev": "7/13", "laya": "2/13", "julia-1": "1/13" }, { "suite": "Policy cases, 20 industries", "family": "banking", "jev": "12/16", "clef": "13/16", "clef-flash": "10/16", "kev-4b": "8/16", "lev": "11/16", "laya": "3/16", "julia-1": "1/16" }, { "suite": "Policy cases, 20 industries", "family": "fraud", "jev": "12/14", "clef": "10/14", "clef-flash": "11/14", "kev-4b": "11/14", "lev": "11/14", "laya": "3/14", "julia-1": "5/14" }, { "suite": "Policy cases, 20 industries", "family": "airline", "jev": "14/14", "clef": "11/14", "clef-flash": "13/14", "kev-4b": "10/14", "lev": "11/14", "laya": "2/14", "julia-1": "4/14" }, { "suite": "Policy cases, 20 industries", "family": "hotel", "jev": "14/16", "clef": "11/16", "clef-flash": "10/16", "kev-4b": "8/16", "lev": "6/16", "laya": "4/16", "julia-1": "4/16" }, { "suite": "Policy cases, 20 industries", "family": "logistics", "jev": "13/13", "clef": "10/13", "clef-flash": "10/13", "kev-4b": "9/13", "lev": "9/13", "laya": "1/13", "julia-1": "4/13" }, { "suite": "Policy cases, 20 industries", "family": "saas", "jev": "13/13", "clef": "13/13", "clef-flash": "11/13", "kev-4b": "10/13", "lev": "11/13", "laya": "3/13", "julia-1": "2/13" }, { "suite": "Policy cases, 20 industries", "family": "telecom", "jev": "15/15", "clef": "13/15", "clef-flash": "11/15", "kev-4b": "12/15", "lev": "9/15", "laya": "4/15", "julia-1": "3/15" }, { "suite": "Policy cases, 20 industries", "family": "utilities", "jev": "8/14", "clef": "10/14", "clef-flash": "10/14", "kev-4b": "11/14", "lev": "9/14", "laya": "5/14", "julia-1": "3/14" }, { "suite": "Policy cases, 20 industries", "family": "hr-leave", "jev": "13/13", "clef": "12/13", "clef-flash": "9/13", "kev-4b": "12/13", "lev": "10/13", "laya": "3/13", "julia-1": "3/13" }, { "suite": "Policy cases, 20 industries", "family": "hr-expense", "jev": "10/12", "clef": "11/12", "clef-flash": "11/12", "kev-4b": "10/12", "lev": "9/12", "laya": "4/12", "julia-1": "5/12" }, { "suite": "Policy cases, 20 industries", "family": "realestate", "jev": "9/15", "clef": "4/15", "clef-flash": "7/15", "kev-4b": "9/15", "lev": "2/15", "laya": "6/15", "julia-1": "5/15" }, { "suite": "Policy cases, 20 industries", "family": "education", "jev": "13/15", "clef": "8/15", "clef-flash": "9/15", "kev-4b": "8/15", "lev": "6/15", "laya": "5/15", "julia-1": "6/15" }, { "suite": "Policy cases, 20 industries", "family": "food", "jev": "10/15", "clef": "13/15", "clef-flash": "10/15", "kev-4b": "12/15", "lev": "11/15", "laya": "4/15", "julia-1": "5/15" }, { "suite": "Policy cases, 20 industries", "family": "manufacturing", "jev": "9/12", "clef": "6/12", "clef-flash": "9/12", "kev-4b": "5/12", "lev": "4/12", "laya": "3/12", "julia-1": "5/12" }, { "suite": "Policy cases, 20 industries", "family": "health-billing", "jev": "10/12", "clef": "10/12", "clef-flash": "8/12", "kev-4b": "7/12", "lev": "6/12", "laya": "3/12", "julia-1": "2/12" }, { "suite": "Policy cases, 20 industries", "family": "health-scheduling", "jev": "3/7", "clef": "3/7", "clef-flash": "3/7", "kev-4b": "5/7", "lev": "5/7", "laya": "2/7", "julia-1": "0/7" }, { "suite": "Policy cases, 20 industries", "family": "permits", "jev": "14/14", "clef": "12/14", "clef-flash": "12/14", "kev-4b": "11/14", "lev": "9/14", "laya": "2/14", "julia-1": "2/14" }, { "suite": "Policy cases, 20 industries", "family": "itsec", "jev": "12/14", "clef": "9/14", "clef-flash": "9/14", "kev-4b": "11/14", "lev": "8/14", "laya": "6/14", "julia-1": "5/14" }, { "suite": "Usability intents", "family": "clear intent", "jev": "28/31", "clef": "29/31", "clef-flash": "31/31", "kev-4b": "27/31", "lev": "30/31", "laya": "16/31", "julia-1": "8/31" }, { "suite": "Usability intents", "family": "screen or history context", "jev": "38/39", "clef": "33/39", "clef-flash": "34/39", "kev-4b": "29/39", "lev": "33/39", "laya": "7/39", "julia-1": "3/39" }, { "suite": "Usability intents", "family": "out of scope", "jev": "18/19", "clef": "18/19", "clef-flash": "17/19", "kev-4b": "17/19", "lev": "14/19", "laya": "12/19", "julia-1": "0/19" }, { "suite": "Usability intents", "family": "ambiguous request", "jev": "17/20", "clef": "13/20", "clef-flash": "7/20", "kev-4b": "3/20", "lev": "6/20", "laya": "1/20", "julia-1": "8/20" }, { "suite": "Usability intents", "family": "destructive needs confirm", "jev": "27/27", "clef": "24/27", "clef-flash": "25/27", "kev-4b": "23/27", "lev": "25/27", "laya": "5/27", "julia-1": "4/27" }, { "suite": "Usability intents", "family": "slang and typos", "jev": "17/18", "clef": "18/18", "clef-flash": "17/18", "kev-4b": "16/18", "lev": "17/18", "laya": "5/18", "julia-1": "3/18" }, { "suite": "Usability intents", "family": "multilingual", "jev": "33/34", "clef": "29/34", "clef-flash": "28/34", "kev-4b": "31/34", "lev": "29/34", "laya": "13/34", "julia-1": "6/34" }, { "suite": "Usability intents", "family": "accessibility", "jev": "13/17", "clef": "16/17", "clef-flash": "17/17", "kev-4b": "15/17", "lev": "17/17", "laya": "5/17", "julia-1": "1/17" }, { "suite": "Usability intents", "family": "undo, redo and back", "jev": "20/21", "clef": "21/21", "clef-flash": "20/21", "kev-4b": "20/21", "lev": "15/21", "laya": "9/21", "julia-1": "3/21" }, { "suite": "Stress tests", "family": "injection", "jev": "38/40", "clef": "36/40", "clef-flash": "31/40", "kev-4b": "38/40", "lev": "27/40", "laya": "13/40", "julia-1": "18/40" }, { "suite": "Stress tests", "family": "needle", "jev": "22/22", "clef": "21/22", "clef-flash": "15/22", "kev-4b": "16/22", "lev": "15/22", "laya": "7/22", "julia-1": "8/22" }, { "suite": "Stress tests", "family": "distractor", "jev": "15/16", "clef": "15/16", "clef-flash": "13/16", "kev-4b": "12/16", "lev": "9/16", "laya": "3/16", "julia-1": "4/16" }, { "suite": "Stress tests", "family": "negation", "jev": "17/21", "clef": "19/21", "clef-flash": "18/21", "kev-4b": "13/21", "lev": "16/21", "laya": "8/21", "julia-1": "6/21" }, { "suite": "Stress tests", "family": "threshold", "jev": "13/25", "clef": "13/25", "clef-flash": "7/25", "kev-4b": "7/25", "lev": "12/25", "laya": "12/25", "julia-1": "12/25" }, { "suite": "Stress tests", "family": "out-of-scope", "jev": "19/19", "clef": "18/19", "clef-flash": "16/19", "kev-4b": "16/19", "lev": "15/19", "laya": "4/19", "julia-1": "9/19" }, { "suite": "Stress tests", "family": "near-options", "jev": "12/14", "clef": "9/14", "clef-flash": "10/14", "kev-4b": "12/14", "lev": "10/14", "laya": "4/14", "julia-1": "4/14" }, { "suite": "Stress tests", "family": "many-options", "jev": "16/16", "clef": "14/16", "clef-flash": "9/16", "kev-4b": "14/16", "lev": "11/16", "laya": "1/16", "julia-1": "0/16" }, { "suite": "Stress tests", "family": "paraphrase", "jev": "22/27", "clef": "20/27", "clef-flash": "20/27", "kev-4b": "18/27", "lev": "14/27", "laya": "11/27", "julia-1": "11/27" } ] }, { "id": "arena-industries", "title": "Policy cases by industry: right answers per model (first presentation)", "columns": [ { "key": "industry", "label": "Industry", "unit": "text" }, { "key": "jev", "label": "Jev 1.13", "unit": "text" }, { "key": "clef", "label": "Clef 27B", "unit": "text" }, { "key": "clef-flash", "label": "Clef-Flash 9B", "unit": "text" }, { "key": "kev-4b", "label": "Kev 4B", "unit": "text" }, { "key": "lev", "label": "lev 4B", "unit": "text" }, { "key": "laya", "label": "Laya", "unit": "text" }, { "key": "julia-1", "label": "Julia-1", "unit": "text" } ], "rows": [ { "industry": "Airline rebooking", "jev": "14/14", "clef": "11/14", "clef-flash": "13/14", "kev-4b": "10/14", "lev": "11/14", "laya": "2/14", "julia-1": "4/14" }, { "industry": "Banking and card disputes", "jev": "12/16", "clef": "13/16", "clef-flash": "10/16", "kev-4b": "8/16", "lev": "11/16", "laya": "3/16", "julia-1": "1/16" }, { "industry": "Education admissions and registration", "jev": "13/15", "clef": "8/15", "clef-flash": "9/15", "kev-4b": "8/15", "lev": "6/15", "laya": "5/15", "julia-1": "6/15" }, { "industry": "HR leave and expense policy", "jev": "23/25", "clef": "23/25", "clef-flash": "20/25", "kev-4b": "22/25", "lev": "19/25", "laya": "7/25", "julia-1": "8/25" }, { "industry": "Healthcare administration", "jev": "13/19", "clef": "13/19", "clef-flash": "11/19", "kev-4b": "12/19", "lev": "11/19", "laya": "5/19", "julia-1": "2/19" }, { "industry": "Hotel booking changes", "jev": "14/16", "clef": "11/16", "clef-flash": "10/16", "kev-4b": "8/16", "lev": "6/16", "laya": "4/16", "julia-1": "4/16" }, { "industry": "IT security access requests", "jev": "12/14", "clef": "9/14", "clef-flash": "9/14", "kev-4b": "11/14", "lev": "8/14", "laya": "6/14", "julia-1": "5/14" }, { "industry": "Insurance claims", "jev": "10/17", "clef": "11/17", "clef-flash": "15/17", "kev-4b": "9/17", "lev": "8/17", "laya": "8/17", "julia-1": "4/17" }, { "industry": "Logistics and shipping exceptions", "jev": "13/13", "clef": "10/13", "clef-flash": "10/13", "kev-4b": "9/13", "lev": "9/13", "laya": "1/13", "julia-1": "4/13" }, { "industry": "Manufacturing quality control", "jev": "9/12", "clef": "6/12", "clef-flash": "9/12", "kev-4b": "5/12", "lev": "4/12", "laya": "3/12", "julia-1": "5/12" }, { "industry": "Payments fraud flags", "jev": "12/14", "clef": "10/14", "clef-flash": "11/14", "kev-4b": "11/14", "lev": "11/14", "laya": "3/14", "julia-1": "5/14" }, { "industry": "Public-sector permits", "jev": "14/14", "clef": "12/14", "clef-flash": "12/14", "kev-4b": "11/14", "lev": "9/14", "laya": "2/14", "julia-1": "2/14" }, { "industry": "Real-estate rental applications", "jev": "9/15", "clef": "4/15", "clef-flash": "7/15", "kev-4b": "9/15", "lev": "2/15", "laya": "6/15", "julia-1": "5/15" }, { "industry": "Restaurant and food delivery", "jev": "10/15", "clef": "13/15", "clef-flash": "10/15", "kev-4b": "12/15", "lev": "11/15", "laya": "4/15", "julia-1": "5/15" }, { "industry": "Retail returns", "jev": "9/13", "clef": "7/13", "clef-flash": "10/13", "kev-4b": "7/13", "lev": "7/13", "laya": "2/13", "julia-1": "1/13" }, { "industry": "SaaS customer support routing", "jev": "13/13", "clef": "13/13", "clef-flash": "11/13", "kev-4b": "10/13", "lev": "11/13", "laya": "3/13", "julia-1": "2/13" }, { "industry": "Telecom plan changes", "jev": "15/15", "clef": "13/15", "clef-flash": "11/15", "kev-4b": "12/15", "lev": "9/15", "laya": "4/15", "julia-1": "3/15" }, { "industry": "Utilities and energy billing", "jev": "8/14", "clef": "10/14", "clef-flash": "10/14", "kev-4b": "11/14", "lev": "9/14", "laya": "5/14", "julia-1": "3/14" } ] }, { "id": "arena-games-table", "title": "Round robin by game: wins-draws-losses", "columns": [ { "key": "model", "label": "Model", "unit": "text" }, { "key": "ttt", "label": "Tic-tac-toe", "unit": "text" }, { "key": "c4", "label": "Connect Four", "unit": "text" }, { "key": "nim", "label": "Nim", "unit": "text" }, { "key": "dab", "label": "Dots and Boxes", "unit": "text" }, { "key": "illegal", "label": "Illegal moves", "unit": "count" } ], "rows": [ { "model": "Jev 1.13", "ttt": "37-6-37", "c4": "56-0-24", "nim": "31-0-49", "dab": "59-0-21", "illegal": 0 }, { "model": "Clef 27B", "ttt": "38-5-37", "c4": "42-0-38", "nim": "45-0-35", "dab": "23-0-57", "illegal": 0 }, { "model": "Clef-Flash 9B", "ttt": "34-7-39", "c4": "38-0-42", "nim": "36-0-44", "dab": "31-0-49", "illegal": 0 }, { "model": "lev 4B", "ttt": "29-9-42", "c4": "45-0-35", "nim": "29-0-51", "dab": "34-0-46", "illegal": 0 }, { "model": "Kev 4B", "ttt": "33-9-38", "c4": "27-0-53", "nim": "38-0-42", "dab": "34-0-46", "illegal": 0 }, { "model": "Laya", "ttt": "20-13-47", "c4": "17-0-63", "nim": "34-0-46", "dab": "44-0-36", "illegal": 0 }, { "model": "Julia-1", "ttt": "29-6-45", "c4": "26-0-54", "nim": "38-0-42", "dab": "23-0-57", "illegal": 0 } ] }, { "id": "arena-pong-table", "title": "Pong round robin: every model, speed and accuracy side by side", "columns": [ { "key": "model", "label": "Model", "unit": "text" }, { "key": "record", "label": "Games W-D-L", "unit": "text" }, { "key": "medianMs", "label": "Median decision (ms)", "unit": "ms" }, { "key": "decisions", "label": "Decisions", "unit": "count" }, { "key": "perfect", "label": "Right zone when the ball came", "unit": "text" }, { "key": "returnRate", "label": "Shots returned", "unit": "text" } ], "rows": [ { "model": "Jev 1.13", "record": "13-0-3", "medianMs": 132.4, "decisions": 1436, "perfect": "100% (523/523)", "returnRate": "86%" }, { "model": "Kev 4B", "record": "12-0-4", "medianMs": 284.7, "decisions": 815, "perfect": "98% (303/310)", "returnRate": "80%" }, { "model": "lev 4B", "record": "9-0-7", "medianMs": 417.8, "decisions": 451, "perfect": "99% (151/153)", "returnRate": "59%" }, { "model": "Clef-Flash 9B", "record": "8-0-8", "medianMs": 527.6, "decisions": 336, "perfect": "83% (112/135)", "returnRate": "49%" }, { "model": "Clef 27B", "record": "4-0-12", "medianMs": 1789.1, "decisions": 84, "perfect": "100% (31/31)", "returnRate": "26%" }, { "model": "Julia-1", "record": "4-0-12", "medianMs": 12.4, "decisions": 7451, "perfect": "21% (603/2811)", "returnRate": "21%" }, { "model": "Laya", "record": "3-0-13", "medianMs": 36.2, "decisions": 2460, "perfect": "12% (122/999)", "returnRate": "12%" } ] }, { "id": "arena-pong-featured", "title": "Featured Pong series (4 games each, seed 2)", "columns": [ { "key": "pair", "label": "Pair", "unit": "text" }, { "key": "aWins", "label": "First won", "unit": "count" }, { "key": "bWins", "label": "Second won", "unit": "count" } ], "rows": [ { "pair": "Clef 27B vs Clef-Flash 9B", "aWins": 1, "bWins": 1 }, { "pair": "Clef 27B vs Jev 1.13", "aWins": 0, "bWins": 2 }, { "pair": "Jev 1.13 vs Laya", "aWins": 4, "bWins": 0 } ] }, { "id": "arena-pong-sides", "title": "Featured Pong match (Jev 1.13 vs Laya): who played which side", "columns": [ { "key": "side", "label": "Side", "unit": "text" }, { "key": "model", "label": "Model", "unit": "text" }, { "key": "id", "label": "Id", "unit": "text" }, { "key": "medianMs", "label": "Median decision time (ms)", "unit": "ms" }, { "key": "decisions", "label": "Decisions in the match", "unit": "count" }, { "key": "points", "label": "Points", "unit": "count" } ], "rows": [ { "side": "left", "model": "Jev 1.13", "id": "jev", "medianMs": 123.6, "decisions": 54, "points": 3 }, { "side": "right", "model": "Laya", "id": "laya", "medianMs": 37.2, "decisions": 156, "points": 0 } ] }, { "id": "arena-c4-players", "title": "Featured Connect Four game: players and result", "columns": [ { "key": "side", "label": "Side", "unit": "text" }, { "key": "model", "label": "Model", "unit": "text" }, { "key": "result", "label": "Result", "unit": "text" } ], "rows": [ { "side": "A", "model": "Jev 1.13", "result": "won" }, { "side": "B", "model": "Clef 27B", "result": "lost" } ] }, { "id": "arena-pong-rally", "title": "Featured Pong rally (Jev 1.13 vs Laya): ball and paddle positions every 80 ms, 0 to 1", "columns": [ { "key": "t", "label": "Seconds", "unit": "seconds" }, { "key": "x", "label": "Ball x", "unit": "ratio" }, { "key": "y", "label": "Ball y", "unit": "ratio" }, { "key": "left", "label": "Left paddle", "unit": "ratio" }, { "key": "right", "label": "Right paddle", "unit": "ratio" } ], "rows": [ { "t": 0, "x": 0.5, "y": 0.5, "left": 0.5, "right": 0.5 }, { "t": 0.083, "x": 0.5, "y": 0.5, "left": 0.5, "right": 0.5 }, { "t": 0.167, "x": 0.5, "y": 0.5, "left": 0.5, "right": 0.5 }, { "t": 0.25, "x": 0.5, "y": 0.5, "left": 0.5, "right": 0.5 }, { "t": 0.333, "x": 0.5, "y": 0.5, "left": 0.5, "right": 0.5 }, { "t": 0.417, "x": 0.5, "y": 0.5, "left": 0.5, "right": 0.5 }, { "t": 0.5, "x": 0.5, "y": 0.5, "left": 0.5, "right": 0.5 }, { "t": 0.583, "x": 0.5488, "y": 0.471, "left": 0.5, "right": 0.48 }, { "t": 0.667, "x": 0.5975, "y": 0.441, "left": 0.5, "right": 0.43 }, { "t": 0.75, "x": 0.6463, "y": 0.412, "left": 0.49, "right": 0.38 }, { "t": 0.833, "x": 0.695, "y": 0.382, "left": 0.44, "right": 0.33 }, { "t": 0.917, "x": 0.7438, "y": 0.353, "left": 0.39, "right": 0.3 }, { "t": 1, "x": 0.7925, "y": 0.323, "left": 0.34, "right": 0.3 }, { "t": 1.083, "x": 0.8413, "y": 0.294, "left": 0.29, "right": 0.3 }, { "t": 1.167, "x": 0.89, "y": 0.264, "left": 0.24, "right": 0.3 }, { "t": 1.25, "x": 0.9106, "y": 0.235, "left": 0.19, "right": 0.28 }, { "t": 1.333, "x": 0.8563, "y": 0.206, "left": 0.14, "right": 0.25 }, { "t": 1.417, "x": 0.8019, "y": 0.177, "left": 0.13, "right": 0.3 }, { "t": 1.5, "x": 0.7475, "y": 0.149, "left": 0.18, "right": 0.3 }, { "t": 1.583, "x": 0.6931, "y": 0.12, "left": 0.23, "right": 0.3 }, { "t": 1.667, "x": 0.6388, "y": 0.091, "left": 0.28, "right": 0.3 }, { "t": 1.75, "x": 0.5838, "y": 0.062, "left": 0.3, "right": 0.29 }, { "t": 1.833, "x": 0.5294, "y": 0.033, "left": 0.3, "right": 0.3 }, { "t": 1.917, "x": 0.475, "y": 0.035, "left": 0.3, "right": 0.3 }, { "t": 2, "x": 0.4206, "y": 0.064, "left": 0.3, "right": 0.3 }, { "t": 2.083, "x": 0.3663, "y": 0.093, "left": 0.3, "right": 0.3 }, { "t": 2.167, "x": 0.3119, "y": 0.122, "left": 0.3, "right": 0.3 }, { "t": 2.25, "x": 0.2575, "y": 0.151, "left": 0.3, "right": 0.3 }, { "t": 2.333, "x": 0.2031, "y": 0.179, "left": 0.3, "right": 0.3 }, { "t": 2.417, "x": 0.1488, "y": 0.208, "left": 0.3, "right": 0.3 }, { "t": 2.5, "x": 0.0944, "y": 0.237, "left": 0.3, "right": 0.3 }, { "t": 2.583, "x": 0.1125, "y": 0.227, "left": 0.3, "right": 0.3 }, { "t": 2.667, "x": 0.1719, "y": 0.192, "left": 0.3, "right": 0.27 }, { "t": 2.75, "x": 0.2306, "y": 0.156, "left": 0.3, "right": 0.26 }, { "t": 2.833, "x": 0.2894, "y": 0.121, "left": 0.3, "right": 0.23 }, { "t": 2.917, "x": 0.3488, "y": 0.085, "left": 0.28, "right": 0.18 }, { "t": 3, "x": 0.4075, "y": 0.05, "left": 0.23, "right": 0.13 }, { "t": 3.083, "x": 0.4669, "y": 0.026, "left": 0.18, "right": 0.1 }, { "t": 3.167, "x": 0.5256, "y": 0.061, "left": 0.13, "right": 0.1 }, { "t": 3.25, "x": 0.5844, "y": 0.097, "left": 0.1, "right": 0.1 }, { "t": 3.333, "x": 0.6438, "y": 0.132, "left": 0.1, "right": 0.1 }, { "t": 3.417, "x": 0.7025, "y": 0.167, "left": 0.1, "right": 0.1 }, { "t": 3.5, "x": 0.7619, "y": 0.203, "left": 0.1, "right": 0.1 }, { "t": 3.583, "x": 0.8206, "y": 0.238, "left": 0.1, "right": 0.1 }, { "t": 3.667, "x": 0.8794, "y": 0.274, "left": 0.1, "right": 0.1 }, { "t": 3.75, "x": 0.9388, "y": 0.309, "left": 0.1, "right": 0.1 }, { "t": 3.833, "x": 0.9975, "y": 0.345, "left": 0.15, "right": 0.11 } ] }, { "id": "arena-c4-game", "title": "Featured Connect Four game (Jev 1.13 vs Clef 27B): every move. Rule: The longest Connect Four game between jev and clef in the turn-based round robin (most moves, openings included); ties go to the earliest.", "columns": [ { "key": "moveNo", "label": "Move", "unit": "count" }, { "key": "opening", "label": "Random opening move", "unit": "text" }, { "key": "side", "label": "Side", "unit": "text" }, { "key": "col", "label": "Column", "unit": "count" }, { "key": "p", "label": "Probability", "unit": "ratio" }, { "key": "optimal", "label": "Perfect move", "unit": "text" }, { "key": "best", "label": "Best columns", "unit": "text" } ], "rows": [ { "moveNo": 1, "opening": true, "side": "A", "col": 4, "p": null, "optimal": null, "best": "" }, { "moveNo": 2, "opening": true, "side": "B", "col": 4, "p": null, "optimal": null, "best": "" }, { "moveNo": 3, "opening": true, "side": "A", "col": 3, "p": null, "optimal": null, "best": "" }, { "moveNo": 4, "opening": true, "side": "B", "col": 3, "p": null, "optimal": null, "best": "" }, { "moveNo": 5, "opening": false, "side": "A", "col": 4, "p": 0.44, "optimal": false, "best": "2,5" }, { "moveNo": 6, "opening": false, "side": "B", "col": 4, "p": 0.1926, "optimal": false, "best": "5" }, { "moveNo": 7, "opening": false, "side": "A", "col": 3, "p": 0.39, "optimal": false, "best": "2,5" }, { "moveNo": 8, "opening": false, "side": "B", "col": 3, "p": 0.1899, "optimal": false, "best": "5" }, { "moveNo": 9, "opening": false, "side": "A", "col": 3, "p": 0.44, "optimal": false, "best": "2,5" }, { "moveNo": 10, "opening": false, "side": "B", "col": 4, "p": 0.1815, "optimal": false, "best": "2" }, { "moveNo": 11, "opening": false, "side": "A", "col": 3, "p": 0.58, "optimal": false, "best": "2,5" }, { "moveNo": 12, "opening": false, "side": "B", "col": 4, "p": 0.2549, "optimal": false, "best": "2" }, { "moveNo": 13, "opening": false, "side": "A", "col": 5, "p": 0.62, "optimal": true, "best": "2,5" }, { "moveNo": 14, "opening": false, "side": "B", "col": 2, "p": 0.2223, "optimal": null, "best": "" }, { "moveNo": 15, "opening": false, "side": "A", "col": 5, "p": 0.6, "optimal": false, "best": "6" }, { "moveNo": 16, "opening": false, "side": "B", "col": 6, "p": 0.2136, "optimal": true, "best": "6" }, { "moveNo": 17, "opening": false, "side": "A", "col": 5, "p": 0.61, "optimal": true, "best": "5" }, { "moveNo": 18, "opening": false, "side": "B", "col": 5, "p": 0.2408, "optimal": true, "best": "5" }, { "moveNo": 19, "opening": false, "side": "A", "col": 5, "p": 0.67, "optimal": false, "best": "2" }, { "moveNo": 20, "opening": false, "side": "B", "col": 5, "p": 0.2192, "optimal": true, "best": "5" }, { "moveNo": 21, "opening": false, "side": "A", "col": 6, "p": 0.48, "optimal": true, "best": "6" }, { "moveNo": 22, "opening": false, "side": "B", "col": 6, "p": 0.3235, "optimal": true, "best": "6" }, { "moveNo": 23, "opening": false, "side": "A", "col": 6, "p": 0.74, "optimal": true, "best": "6" }, { "moveNo": 24, "opening": false, "side": "B", "col": 6, "p": 0.3626, "optimal": false, "best": "1" }, { "moveNo": 25, "opening": false, "side": "A", "col": 6, "p": 0.62, "optimal": true, "best": "6" }, { "moveNo": 26, "opening": false, "side": "B", "col": 2, "p": 0.4789, "optimal": false, "best": "7" }, { "moveNo": 27, "opening": false, "side": "A", "col": 2, "p": 0.54, "optimal": true, "best": "2" } ] }, { "id": "arena-leagues-table", "title": "Arena leagues: wins-draws-losses and Elo per model", "columns": [ { "key": "model", "label": "Player", "unit": "text" }, { "key": "c0", "label": "Othello", "unit": "text" }, { "key": "c1", "label": "Pong, fair mode 150 ms", "unit": "text" }, { "key": "c2", "label": "Snake, fair mode 150 ms", "unit": "text" }, { "key": "c3", "label": "Tron, fair mode 150 ms", "unit": "text" } ], "rows": [ { "model": "perfect", "c0": "32-0-0 (1280)", "c1": "7-0-0 (1098)", "c2": "–", "c3": "–" }, { "model": "Clef 27B", "c0": "20-2-10 (1082)", "c1": "–", "c2": "11-0-5 (1068)", "c3": "12-3-17 (956)" }, { "model": "random", "c0": "17-0-15 (1015)", "c1": "1-0-6 (930)", "c2": "6-0-10 (954)", "c3": "8-4-20 (898)" }, { "model": "Laya", "c0": "15-1-16 (990)", "c1": "0-0-7 (902)", "c2": "1-1-14 (850)", "c3": "7-1-24 (852)" }, { "model": "Julia-1", "c0": "13-2-17 (965)", "c1": "2-0-5 (958)", "c2": "9-0-7 (1022)", "c3": "2-1-29 (761)" }, { "model": "lev 4B", "c0": "14-0-18 (964)", "c1": "4-0-3 (1014)", "c2": "4-0-12 (908)", "c3": "14-3-15 (991)" }, { "model": "Clef-Flash 9B", "c0": "11-1-20 (924)", "c1": "3-0-4 (986)", "c2": "9-1-6 (1034)", "c3": "17-1-14 (1025)" }, { "model": "Kev 4B", "c0": "9-1-22 (891)", "c1": "6-0-1 (1070)", "c2": "2-1-13 (875)", "c3": "17-2-13 (1031)" }, { "model": "Jev 1.13", "c0": "9-1-22 (890)", "c1": "5-0-2 (1042)", "c2": "12-1-3 (1103)", "c3": "27-3-2 (1219)" }, { "model": "expert", "c0": "–", "c1": "–", "c2": "16-0-0 (1185)", "c3": "30-2-0 (1267)" } ] } ], "related": [ "routing-jev-vs-llm", "routing-overhead" ] }, "sources": [ { "id": "system-one-arena", "title": "System One arena: typed-decision models on checkable decisions and in head-to-head games", "kind": "run", "date": "2026-10-06", "note": "Jev 1.13 (TypeSafe API) and six open System One models in llama.cpp 0.6.0 on one Mac Studio (Clef 27B and Clef-Flash 9B Q4_K_M, lev 4B and Kev 4B Q4_K_M, Laya and Julia-1 Q8_0). 1,085 model-blind items in five suites with independent verifiers; exam, latency and repeat passes. Round-robin games (tic-tac-toe, Connect Four, Nim, Dots and Boxes, real-time Pong) against each other and a random and a perfect player, with the featured series and a speed ladder; declared amendments 1 and 2 in the published protocol. Protocol, item hashes and game counts declared before the first counted call; every call and game kept.", "data": [ "/benchmarks/raw/system-one-arena/items.json", "/benchmarks/raw/system-one-arena/exam.json", "/benchmarks/raw/system-one-arena/latency.json", "/benchmarks/raw/system-one-arena/repeat.json", "/benchmarks/raw/system-one-arena/same-host-reference.json", "/benchmarks/raw/system-one-arena/matches.json", "/benchmarks/raw/system-one-arena/match-summary.json", "/benchmarks/raw/system-one-arena/featured.json", "/benchmarks/raw/system-one-arena/leagues.json", "/benchmarks/raw/system-one-arena/speed-lab.json", "/benchmarks/raw/system-one-arena/clock-registry.json" ] } ] }