STaR-style canonical fallback
STaR-style canonical fallback is the project's historical RFT variant that supplies a known answer when every sampled response fails.
Project status: Implemented as historical --mode star; tested on Track B. This entry describes the source audit of 14 September 2026; historical measurements retain their original dates.
Mechanism
For mixed pass/fail groups, choose the shortest passing response as in RFT. Skip all-pass groups. For all-fail groups, synthesize Answer: <expected> from the task's known target and train on that answer instead of skipping.
Implementation and controls
experiment_pi_old_synth.py::filter_and_rank records index −1 for a canonical fallback. The target comes from task.expected; it is not a newly generated and verified explanatory rationale. The same SFT submission path trains selected responses.
Evidence and evaluation
The historical lattice gives Arm 2 −12.5 percentage points on Track B Qwen3-0.6B. It did not rescue the local RFT regression. The runner makes fallback frequency observable, which matters because this arm can train on more prompts than skip-all-fail RFT.
Limitations and interpretation
“STaR-style” is a local analogy, not a faithful implementation claim for every stage of the published STaR method. It assumes access to correct target answers. Canonical short answers can change response style and reasoning behavior; compare against vanilla SFT at matched update/token budget.
Sources
- agi: autoresearch/experiment_pi_old_synth.py — historical revision
3842fd8875ca. - agi: autoresearch/LATTICE.md — historical revision
3842fd8875ca.