Real model decisions, matched tasks, and every recorded attempt. See where a skill kit changes the result — and where it does not.
Published experiment spend
0.019863036OpenRouter credits recorded
All published comparisons and infrastructure interruptions.
$5 total experiment allowance
One cumulative ceiling for the experiment program.
Published snapshots only — not a live balance or permission for further calls. The private ledger controls the remaining allowance.
SERIES SIDE BY SIDE
Task success and the skill difference.
First read each agent’s task success. Then compare the full kit with the baseline inside the same series. These are two different measurements.
2 of 4 retained series
Series 03paused
DeepSeek V4.1 Flash · 8 actions · paused
deepseek/deepseek-v4.1-flash
Output / request timeout
8,192 tokens / 120 s
Actions / context
8 / 32,768
Temperature / seed
0 / 0
Task repeats
3
16 / 18 attempts recorded · partial series
01 Tasks passed
Absolute results · passes / graded attempts
Baseline8 / 8
8 of 9 planned attempts graded · 0 failed · 1 not run
Full Genome kit8 / 8
8 of 9 planned attempts graded · 0 failed · 1 not run
02 Skill difference
Full kit against this series’ baseline
0net passes so far
8 of 9 pairs graded so far · same task and repetition
0improved0regressed8tied1unpaired
Improved: baseline failed, full kit passed. Regressed: the reverse. The net difference is improved minus regressed; passing with both is a tie.
Recorded failure reasons0 failures across all skill sets
Output limit
0
Action limit
0
Invalid format
0
Other
0
Settings: frozen campaign.
Series 04paused
DeepSeek V4.1 Flash · 16 actions · paused
deepseek/deepseek-v4.1-flash
Output / request timeout
8,192 tokens / 120 s
Actions / context
16 / 32,768
Temperature / seed
0 / 0
Task repeats
3
17 / 18 attempts recorded · partial series
01 Tasks passed
Absolute results · passes / graded attempts
Baseline9 / 9
9 of 9 planned attempts graded · 0 failed
Full Genome kit7 / 7
7 of 9 planned attempts graded · 0 failed · 1 indeterminate · 1 not run
02 Skill difference
Full kit against this series’ baseline
0net passes so far
7 of 9 pairs graded so far · same task and repetition
0improved0regressed7tied2unpaired
Improved: baseline failed, full kit passed. Regressed: the reverse. The net difference is improved minus regressed; passing with both is a tie.
Recorded failure reasons0 failures across all skill sets
Output limit
0
Action limit
0
Invalid format
0
Other
0
Settings: frozen campaign.
Recorded differences: Action limit.
A full kit can pass more tasks in a later series while its baseline also improves. That increase is different from the skill difference measured inside each series.
These are separate series, not a controlled test of one changed parameter. Different limits, timeouts, execution revisions or model variability can change outcomes. Cross-series differences do not identify their cause. Failure categories describe the recorded stopping reason across all skill sets; they do not prove a skill caused it.
Every configuration, with its own series baseline
Frozen skill configuration
Series 38,192 tokens · 120 s
Series 48,192 tokens · 120 s
baselineUnchanged baseline
8pass0fail0indeterminate1not run
Calls
35
Input tokens
17,819
Output tokens
11,709
Recorded charge
0.003692898OpenRouter credits · all dispatched calls
Reference in this series.
9pass0fail0indeterminate0not run
Calls
34
Input tokens
17,254
Output tokens
14,325
Recorded charge
0.004076064OpenRouter credits · all dispatched calls
Reference in this series.
fullFull selected kit
8pass0fail0indeterminate1not run
Calls
33
Input tokens
29,540
Output tokens
22,388
Recorded charge
0.006633144OpenRouter credits · all dispatched calls
0 improved0 regressed8 tied1 unpairedAgainst this series’ baseline · 8/9 graded pairs so far
7pass0fail1indeterminate1not run
Calls
34
Input tokens
Unknown29,255 recorded
Output tokens
Unknown15,897 recorded
Recorded charge
0.00540225OpenRouter credits · all dispatched calls
0 improved0 regressed7 tied2 unpairedAgainst this series’ baseline · 7/9 graded pairs so far
Rows align by skill kind and frozen lesson snapshots, including the omitted lesson in a removal test. A changed snapshot gets a separate row. There is no pooled score or automatic ranking across series. Usage includes dispatched calls from failed and unfinished attempts, attributed by the frozen attempt plan. Missing receipt accounting stays unknown; unspent reservations are not model calls.
What should we retest?Human review · no automatic skill changes
These flags identify follow-up questions, not good or bad skills. Keep each series separate, fix the next test conditions in advance, and check the same lesson on new tasks before changing it.
Review a kit regression
Verify the outcome, not the intention
Evidence is limited to these tasks
The kit result cannot identify an individual skill’s effect
The full kit lost at least one pair
The unrelated control matched or exceeded the kit in comparable wins
Removing a skill helped in at least one pair; interactions remain possible
Some pairs are missing or indeterminate
1,024 output tokens · 45 s
Full kit vs baseline: 1 improved · 2 regressed · 6 tied · 0 unpaired.
Without this skill vs full kit: 3 improved · 1 regressed · 5 tied · 0 unpaired.
8,192 output tokens · 120 s
Full kit vs baseline: 1 improved · 0 regressed · 8 tied · 0 unpaired.
Without this skill vs full kit: 0 improved · 2 regressed · 7 tied · 0 unpaired.
8,192 output tokens · 120 s
Full kit vs baseline: 0 improved · 0 regressed · 8 tied · 1 unpaired.
No removal test recorded for this skill.
8,192 output tokens · 120 s
Full kit vs baseline: 0 improved · 0 regressed · 7 tied · 2 unpaired.
No removal test recorded for this skill.
Review a kit regression
Recheck after the world changes
Evidence is limited to these tasks
The kit result cannot identify an individual skill’s effect
The full kit lost at least one pair
The unrelated control matched or exceeded the kit in comparable wins
Removing a skill helped in at least one pair; interactions remain possible
Some pairs are missing or indeterminate
1,024 output tokens · 45 s
Full kit vs baseline: 1 improved · 2 regressed · 6 tied · 0 unpaired.
Without this skill vs full kit: 4 improved · 1 regressed · 4 tied · 0 unpaired.
8,192 output tokens · 120 s
Full kit vs baseline: 1 improved · 0 regressed · 8 tied · 0 unpaired.
Without this skill vs full kit: 0 improved · 2 regressed · 7 tied · 0 unpaired.
8,192 output tokens · 120 s
Full kit vs baseline: 0 improved · 0 regressed · 8 tied · 1 unpaired.
No removal test recorded for this skill.
8,192 output tokens · 120 s
Full kit vs baseline: 0 improved · 0 regressed · 7 tied · 2 unpaired.
No removal test recorded for this skill.
SELECTED SNAPSHOT · 13 Oct 2026, 00:11 UTC
DeepSeek V4.1 Flash · 16 actions · paused
deepseek/deepseek-v4.1-flash · 8,192 output tokens · 120 s request timeout
paused
17 / 18 development attempts recorded · 18 confirmation attempts reserved
Provider requests680 unresolved
Recorded tokens7673146509 known input · 30222 known output1 call has unknown token usage
The same task and repeat, with a fresh world for each skill set. Outcome and constraints are checked together; compliance is reported separately.
Against Unchanged baseline
The complete selected kit compared with the unchanged baseline. An observed improvement belongs to this kit and task; it does not isolate a single skill.
0ImprovedFail → pass
0RegressedPass → fail
7Unchanged7 both pass · 0 both fail
2Unpaired1 missing attempts · 1 indeterminate
7 / 9 pairs have two supported grades0 percentage points in paired pass rate · among 7 graded pairs so far
Verify the changed artifact1 graded pairs · 2 unpaired+0−01 tied
Reference → compared skill set
Repeat 1pass→indeterminateUnpaired
At least one attempt has no supported outcome grade. Both receipts are retained.
Reference
Observed outcome and constraints pass. Compliance is scored separately.Receipt 355ab238619cf85fc0f1ebd14b0dc0cba003877c1c31d30cb4a019ad2a133616
Compared skill set
Provider charge reconciled; delivered action or checkpoint could not be independently recovered.Receipt d3e9b2a1b1d693362af4ca4758eb1b5661fe55b17d44225e8edf0e2c4bb9abdc
Repeat 2pass→passBoth passed
Both attempts passed.
Reference
Observed outcome and constraints pass. Compliance is scored separately.Receipt 9e79637b46916aa90c08d2908cfb140ed252cda3a6b8c35de33e8f0f48d5abea
Compared skill set
Observed outcome and constraints pass. Compliance is scored separately.Receipt 9a1b3630d92c6fc2b23e3a889039de724d946baf3883f473ab458c7b5a0e5c2b
Repeat 3pass→not runUnpaired
Compared attempt has not been recorded.
Reference
Observed outcome and constraints pass. Compliance is scored separately.Receipt b898464ef801bc09e30c1a4028bab881967220c058f2e0c03a701fa2dc269452
Compared skill set
Not run
Task ID: coding-artifact-dev
Duplicate order events3 graded pairs · 0 unpaired+0−03 tied
Reference → compared skill set
Repeat 1pass→passBoth passed
Both attempts passed.
Reference
Observed outcome and constraints pass. Compliance is scored separately.Receipt 3bcb8a9cea26ca32100ceb59d002d19189f8fce9bc59a6fdd9201b6d6374babc
Compared skill set
Observed outcome and constraints pass. Compliance is scored separately.Receipt 6f317a699c0f4e1a8ea9c9c098b2eed72ede294126ee6d486dfd6435e7599d0e
Repeat 2pass→passBoth passed
Both attempts passed.
Reference
Observed outcome and constraints pass. Compliance is scored separately.Receipt 96f6edc6130adf559b47ccd75e4aea44fc9b3e0fc04af5dedb23ef9f9adb9302
Compared skill set
Observed outcome and constraints pass. Compliance is scored separately.Receipt ef006aa2ceab85d41121fc7ed4fb452a4cb42c6c3c663e9e63f30581b17e8caf
Repeat 3pass→passBoth passed
Both attempts passed.
Reference
Observed outcome and constraints pass. Compliance is scored separately.Receipt 854663a18a870db73454c67d49b9f80fdebe39f597fdd1acdc144abf44eed88b
Compared skill set
Observed outcome and constraints pass. Compliance is scored separately.Receipt fef0e05f0bd67ef1688852910ee418a18ffb3397f2ffc3880e37af9ef67099b5
Observed outcome and constraints pass. Compliance is scored separately.Receipt 76d7f51f98271e065a6e37afe75aa8f8c3c42e234b0dd92582441f78af44b431
Compared skill set
Observed outcome and constraints pass. Compliance is scored separately.Receipt 49309afb418ec900eb5ea9798484dee76baff01e2218e84c1a95a52e2dcbf1f9
Repeat 2pass→passBoth passed
Both attempts passed.
Reference
Observed outcome and constraints pass. Compliance is scored separately.Receipt d8f802b8fe031a3ccc3ca86cf95eba05167b282d3688a729bf0d4589faf3d2a8
Compared skill set
Observed outcome and constraints pass. Compliance is scored separately.Receipt d07e2e8c7b8047bd5abd5cf89b997db8cfff48d70aaccd6459d25955a20a3545
Repeat 3pass→passBoth passed
Both attempts passed.
Reference
Observed outcome and constraints pass. Compliance is scored separately.Receipt fd0a2096619fce8cafa28c12e2632b90fef3da233e30b1ec04924d20fb916646
Compared skill set
Observed outcome and constraints pass. Compliance is scored separately.Receipt 5786d7dcbb529987e15ea2b23a6927dd135447246983299208f6ac0d21e724a4
Task ID: commerce-stock-dev
These are descriptive results from operator-authored development tasks. Repeats are not independent real-world tasks. Small samples and excluded pairs can change the picture; this is not statistical proof or a model ranking. Confirmation tasks remain reserved.
Every attempt stays visible.17 recorded · 1 not run · 1 indeterminate
Not executed in this series; retained as protocol metadata.
Archive retrieval does not change any tool-state transition in these worlds, and no library-search tool is available. It controls for additional skill text rather than a relevant recovery procedure.
Provider acknowledgements and retained accounting records are separate from task outcomes. Reconciled charges are operator-attested, as labeled. Unknown values are not counted as zero. No prompts or model outputs are published here.
This is a published snapshot, last checkpoint 13 Oct 2026, 00:11 UTC. Opening it never starts a model call or refreshes a private notebook. It includes failed and unfinished work; no replacement attempts are inserted to improve the score.
Infrastructure interruptions · 2 retained records
These interrupted preparations are retained separately from the comparison series. They contribute no paired skill-effect estimate and are not agent successes or failures. Their known charges still count toward the shared spending ceiling.
Budget study: deepseek8
deepseek/deepseek-v4.1-flash · paused · checkpoint 2026-10-12 22:56:35 UTC
The initial request stopped at an accounting check. The delivered action or checkpoint could not be independently recovered.
The charge and token counts were reconciled afterward. Billing recovery does not supply a task result, so the attempt remains indeterminate.
An earlier assumption about the upstream accounting field was removed from the adapter. This interruption does not establish a failure of the agent or provider.
Generation ID: gen-1791845562-9IgGe37ihlOv8N9M1MhbReceipt status: reconciled. No reservation remains held.
Retained record references
Campaign bf2e3080-d48d-45c1-a1f1-83135084c266
baseline · commerce-stock-dev · repetition 1: Provider charge reconciled; delivered action or checkpoint could not be independently recovered.
This download is a projection of the public campaign export. It does not include the private reconciliation record or independently recover the missing action.
Budget study: deepseek16
deepseek/deepseek-v4.1-flash · paused · checkpoint 2026-10-12 22:56:40 UTC
Paused before dispatch; no model outcome recorded
Attempt record
0 indeterminate · 18 not run
Provider requests
0
Known charge
0 OpenRouter credits
Recorded tokens
0 input · 0 output
Paused before any provider request was dispatched. No outcome or skill-effect estimate was recorded.
Kept as an undispatched preparation; all 18 planned development attempts remain not run.
Retained record references
Campaign 428dcea5-8afd-499c-b469-3f39c1d2aa47
This download is a projection of the public campaign export. It does not include the private reconciliation record or independently recover the missing action.