STAMPED principles and their role in the age of
AI-driven scientific software development

Agents often fail not for lack of "intelligence".
They fail for lack of structure to act on.

Yaroslav O. Halchenko @yarikoptic @yarikoptic@fosstodon.org
Center for Open Neuroscience Department of Psychological and Brain Sciences Center for Cognitive Neuroscience Dartmouth College New Hampshire, USA

LLM Workshop — MIT — October 16^th, 2026
Live slides/Sources: https://datasets.datalad.org/centerforopenneuroscience/talks/2026-reproAI-MIT-STAMPED.html

It’s 2026. Agents write the code.

So the bottleneck moved

HBR, Feb 2026: “AI doesn’t reduce work, it intensifies it”

Without structure, the agent has to guess

study_final2/
  scans/
    s01_ep2d_bold_moco_3.nii
    s01_t1_mprage_FINAL.nii
    s02_ep2d_bold_moco_3(1).nii
  analysis_new/
    run_it.sh
    run_it_OLD.sh
    results_use_this_one.mat
  notes.docx

ds000001/
  dataset_description.json
  participants.tsv
  sub-01/
    anat/sub-01_T1w.nii.gz
    anat/sub-01_T1w.json
    func/sub-01_task-rest_bold.nii.gz
    func/sub-01_task-rest_bold.json
  derivatives/
    mriqc/dataset_description.json

Not just rhetoric: a structured neuroimaging harness moved first-choice tool-selection accuracy from 23.3% to 93.6% across seven models — Chen, …, Poldrack, Bringing analytic rigor to agentic AI for science, arXiv:2608.19902, 2026 (brain-researcher.com).

This problem already has a name

YODA, 2018 — for humans composing studies

→

STAMPED, 2026 — generalized, gradeable, domain-agnostic

We did not invent these for AI. That is the point — they were already the right properties.

Seven properties a research object should have —
whoever, or whatever, picks it up next.

STAMPED in one table

A. Macdonald, C. Baker, I. To, Y. O. Halchenko. STAMPED: principles for reproducible research objects. Preprint, 2026. doi: 10.31222/osf.io/f3h82_v1  ·  sources: stamped-principles/stamped-paper  ·  paper.stamped-principles.org

Actionability is the one that agents live or die on

“All standards are ‘bad’, but some are used.” — D. Clunie, MICCAI 2017. A used standard beats a correct one — and now it has a second constituency.

… and AI changed what counts as actionable

So grade them too: a skill is a research object

A skill that only runs in your repo, on your paths, undocumented and unversioned, is a silo with a friendlier interface.

Walk it yourself: checklist.stamped-principles.org (MUST / SHOULD / MAY, item by item) · examples · principles · paper source (main.tex — small enough to paste into a context window, which is how we actually grade a study today).

What we built to make this real

Twenty years of it, in neuroimaging, before anyone said “agent”.

Standards are the grammar — BIDS as the worked example

K. J. Gorgolewski, …, Y. O. Halchenko, …, R. A. Poldrack. The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments. Scientific Data 3:160044, 2016.

Tracked: datalad run writes the record for you

$ datalad run -m "run brain extract workflow" \
    --input "inputs/ds*/sub-01/anat/sub-01_T1w.nii.gz" \
    --output "sub-*/anat" \
    bash -e code/brain_extraction.sh inputs/ds*/sub-01/anat/sub-01_T1w.nii.gz

$ git show --stat
commit 1223bc4bead3c7e6be15b78546c894c6465dd7a4

    [DATALAD RUNCMD] run brain extract workflow

    === Do not change lines below ===
    {
     "cmd": "bash -e code/brain_extraction.sh inputs/ds000001/sub-01/anat/sub-01_T1w.nii.gz ...",
     "dsid": "b656ac80-973c-45ef-887e-1b8d6f6a51dd",
     "exit": 0,
     "inputs": ["inputs/ds*/sub-01/anat/sub-01_T1w.nii.gz"],
     "outputs": ["sub-*/anat"],
     "pwd": "."
    }
    ^^^ Do not change lines above ^^^

 sub-001/anat.nii.gz      | 1 +
 sub-001/anat_mask.nii.gz | 1 +

• Machine-readable, in the commit message — no sidecar database to lose
• Clean git state in, clean git state out: no surprises
• datalad rerun reads it back and re-executes

Example from the DataLad Handbook. Found in the wild in ~15% of indexed DataLad repositories on GitHub.

Provenance does not care who wrote the code

Agent-written or hand-written, the [DATALAD RUNCMD] record is the same shape.

The tests do not know the code was AI-generated either.

The missing piece: con/duct

… and that log is the provenance you need later

“Why did this get OOM-killed on the cluster?” Peak RSS over time, per process. Not your memory of it. “How much do I request for 2,000 subjects?” Measured, from the run you already did. “What did it print before it died?” Captured, versioned, next to the outputs.

con-duct plot — MRIQC on one subject/session

Already wired in: set REPRONIM_USE_DUCT=1 with ///repronim/containers and every datalad containers-run gets a duct trace for free.

All three layers, one real artifact — built with AI

dandi/compression-comparisons-t261-study — a reproduction of Buccino et al. 2023 on ephys compression, plus a comparison against the new ITU-T T.261 codec.

dataset_description.json
code/
  compression-comparisons-tools/
  run_sweep.sh
sourcedata/
  aind-ephys-compression/      # 523 GB
derivatives/
  compbench-*/<cell>/
    metrics.json  manifest.json
    duct-info.json  duct-usage.jsonl
  reports-*/

$ git log --oneline
7733465 [DATALAD RUNCMD] cell aind-634568__zstd...

 "cmd": "duct --output-prefix .../duct- \
      --sample-interval 1 --report-interval 5 \
      compbench run --codec zstd ...",
 "inputs":  ["sourcedata/aind-ephys-compression/.."],
 "outputs": ["derivatives/../aind-634568__zstd.."],
 "exit": 0

So I tried to rerun it. Live, on this laptop.

$ datalad clone .../compression-comparisons-t261-study
$ datalad get code/compression-comparisons-tools
  ✗ fatal: repository not found

$ # .gitmodules says:  url = ../..
$ #  → resolves to https://github.com/ (not a repo)
$ # fine on the author's disk; breaks for a cloner

$ git clone .../dandi/compression-comparisons \
     code/compression-comparisons-tools # by hand
$ uv venv envs/compbench && uv pip install ...

$ datalad rerun 42dc5cfa
  run(ok): ... [duct --clobber --output-prefix ..]
  Wall Clock Time: 0.549 sec   RSS peak: 7.8 MB

So what do we build next?

Not more code. Harnesses and skills.

Agent = Model + Harness

formulation borrowed from Oscar Esteban, OHBM 2026

Where are you on the ladder?

Ladder figure & framing: Jo Van Eyck, “You are Not Ready: Agentic coding in 2026”

Level What it looks like The skill you have to acquire L1 Chat Browser tab; copy-paste Prompt & context engineering L2 Mid-loop In the IDE — “autocomplete on steroids” Critically evaluating alternatives, not accepting them L3 In-the-loop The babysitting phase — you watch it struggle Doom loops, guard rails, extracting reusable skills. 2–3 months, minimum. L4 On-the-loop Spec it, hand off, verify the artifact Everything above plus a real harness: skills, tests, sandboxes, scanners L5 Multi-agent Orchestration; the frontier Everyone is still figuring this out L4–L5 is exactly where a bad harness stops being survivable — which is why the harness, not the model, is the thing to invest in.

And the loop that gets you up it

That was the individual ladder. Here is the team one.

Figure reproduced from: E. C. Johnson, T. T. Nguyen, B. K. Dichter, …, M. E. Martone, F. Pestilli, D. Yatsenko. SciOps: Achieving Productivity and Reliability in Data-Intensive Research. arXiv:2401.00077, 2024.

Right now, everyone is rebuilding the same private harness

A hundred labs, each writing their own “how to work with BIDS” prompt.
None of them shared. None of them versioned. None of them tested.

We have seen this failure mode before. It was called
“every lab has its own analysis scripts”, and we fixed it with standards and packages.

We are already doing this — upstream, in public

K-Dense-AI/scientific-agent-skills — 177 scientific skills, Agent Skills spec, MIT-licensed, scanned and reviewed.

Note the shape: a community member wrote the DataLad skill, we reviewed it, and a third party then extended it with STAMPED. That is exactly how a standard spreads — and it is now happening to harnesses.

And domain-agnostic ones: con/skills

Deeper dive on the workflow around these: “A few words of intro into AI assisted coding” (CA Origami Retreat, 2026) — also in the appendix of this deck.

Two regimes — and be honest about which one you are in

Either the system checks itself against goals and spec — or the work items stay small enough that a human still can.

“Well, less is more” — Robert Browning, Andrea del Sarto, 1855; later the motto of Mies van der Rohe, who credited his mentor Peter Behrens rather than claiming it.

We are not the only ones saying this

HI ↔ AI — every project picks its own policy

Rows 1–2 live in regime B; rows 3–4 are only honest once regime A is actually built.

Common ground, all four: AI cannot be an author (ICMJE, Jan 2026); humans retain full responsibility; AI use is disclosed inside the artifact. STAMPED Tracking is what makes that mechanical rather than a promise.

Catalog of declared OSS stances: melissawm/open-source-ai-contribution-policies. — Note what the row-4 exemplars are: harnesses. Brain Researcher constrains the agent to acceptable analyses and required validations rather than trusting it — autonomy is bought with structure, not with a bigger model.

Take home — what to do Monday

1. Make one object actionable. Pick one study. Put it in a declared standard (BIDS, NWB) and run the validator. The agent gets a grammar; you get a type-checker.
2. Record one command. Wrap your next analysis step in datalad run — or at minimum in duct. A provenance record you did not have to remember to write.
3. Grade one thing against STAMPED. checklist.stamped-principles.org, fifteen minutes, and you will know which property is your weakest.
4. Write down your harness. An AGENTS.md in the repo beats a prompt in your notes app. Declare your AI policy there while you are at it.
5. Size the work to the loop you actually have. Until the project self-checks, keep AI-assisted items small, reviewable, and tried by your own hand — and say so out loud, so nobody mistakes regime B for regime A.
6. Push one skill upstream. The next person who needs your procedure should clone it, not re-derive it. scientific-agent-skills, con/skills, or your own repo — but somewhere public.

Appendix

Material cut for time — press ↓ to walk it, or follow the links.

STAMPED and FAIR do different jobs

Portable + Ephemeral: the environment travels with the record

$ datalad containers-run \ -n repronim-mriqc \ -m "run MRIQC on sub-01" \ --input sub-01 --output derivatives/mriqc \ '{inputs}' '{outputs}' participant Container image is a tracked subdataset, not a latest tag that drifts The commit records image and command together Bind-mount only the current module — look up you must not Throw the container away afterwards: that is Ephemerality, and it is already how agents run

ReproNim/containers — auto-rebuilt, versioned collection

“Depend on where you run, you should not.” An agent’s sandbox is not your laptop — which makes portability a correctness property, not a courtesy.