Project · Autonomous research · July 2026
ScholarLoop: an autonomous research loop that can't grade its own homework
Most "AI scientist" demos share a quiet flaw: the same model that runs the experiment also reports the result. Give an agent the ability to edit its own evaluation, and sooner or later it will — not out of malice, but because that's where the gradient points.
ScholarLoop is our answer. It's an autonomous multi-agent framework that reproduces a researcher's full experimental workflow — read the literature, form a grounded hypothesis, run real PyTorch experiments, reflect on failures, and draft a peer-reviewed write-up — with no human in the loop, on a single-GPU budget. The defining design decision: the LLM only does the open-ended thinking. Everything checkable is deterministic, tested code.
How it works
One governed research round
Every round is budgeted, gated, and scored against frozen ground truth
01 — Direct
A director reads the experiment ledger and literature hotspots, then commits to a research direction, a concrete topic, and a budget.
02 — Read
Real retrieval across arXiv and OpenAlex, ranked by citation impact and distilled into cited, testable techniques.
03 — Propose
Constraints mined from history — dead regions, duplicates, frozen surfaces — bound a structured experiment proposal.
04 — Debate
Persona critics vote on whether the idea is worth GPU time before a single seed is spent.
05 — Experiment
A smoke → verify → full funnel across seeds. The agent edits training code; a frozen, hash-verified scorer owns the metric.
06 — Reflect
Every agent's predictions are scored against measured truth; lessons decay into a relevance-ranked skill library.
07 — Write
A paper drafted only from verified numbers, then peer-reviewed by the system itself — which rejects marginal results.
scroll →
Honest by construction
The interesting engineering isn't the loop — it's the guardrails that make the loop trustworthy when nobody is watching:
- Two-phase frozen scoring. The agent may edit
train.py; it may never touch the scorer. Scoring runs from a pristine copy of the codebase, and the scorer's hash is verified at runtime. We ship an adversarial "cheater" engine in the test suite to prove the guards hold. - Number grounding. Every mean, std, and per-seed result is frozen into a verified registry. The paper writer can only cite numbers that exist in that registry — a deterministic audit rejects anything else.
- Predict-then-verify calibration. Each agent's checkable claims are scored against what actually happened, and the calibration record is fed back into future prompts. The loop learns which of its own agents to trust.
- A self-stopping governor. Unattended runs halt on dollar budget, round caps, or convergence — with cost tracking priced per model, per call.
Where it stands
The full loop runs end-to-end today — deterministically against a mock model with no API key or GPU, and live against frontier models on real experiment engines. In our captured live runs, the system's own reviewer rejected both generated papers as too marginal to publish. We consider that a feature: an autonomous researcher that knows when its results aren't good enough is worth more than one that always declares victory.
ScholarLoop is an active research preview. If you're working on autonomous research systems and want to compare notes — or break our guards — write to research@r2m.ai.