Pre-sampled unlikelihood training
Pre-sampled unlikelihood training is the cont hybrid recipe that corrects cold-model mistakes before memorizing demonstrations.
Project status: Implemented as hybrid_presample_unlike in the autoresearch runner. This entry describes the source audit of 14 September 2026; historical measurements retain their original dates.
Mechanism
For each selected task, first sample K responses from the still-untrained distribution. Grade their extracted answers and submit correction samples for wrong responses. Then run greedy memorization on the canonical answer. The order addresses an earlier post-memorize variant whose sampling produced no wrong responses, leaving its correction stage inactive.
Implementation and controls
autoresearch/experiment.py has the branch; config.json contains K=4, temperature 0.8, top_p=0.95, max_new_tokens=384, five train samples, memorize cap=100, threshold=0.95, patience=10, and lr=5e-4. The branch records cold_wrong_count and unlike/memorize commit tokens. This branch emits prompt/response-only unlike samples, whereas the current primitive requires prefix/bad_token_id. Its correction stage is therefore incompatible with the audited daemon schema when wrong responses are submitted; the runner needs adaptation before that stage can run unchanged against current Trainfer.
Evidence and evaluation
The May 16 journal records heldout score 0.60, matching the memorize baseline; with CoT evaluation it tied at 0.70 at 3–5 times the wall time. Earlier post-sample correction had zero wrong rollouts on the five tasks. The data therefore did not show an added efficiency benefit from the correction stage.
Limitations and interpretation
A branch name and logged commit do not establish that current daemon schemas accept its correction payload or that the intended negative gradient was applied. Positive memorization can dominate or undo correction. Report effective correction count and output behavior, and do not call a tied result an improvement.
Sources
- cont: autoresearch/experiment.py — checkout audited
87946914c7b9. - cont: autoresearch/config.json — checkout audited
87946914c7b9. - cont: docs/research/JOURNAL.md — checkout audited
87946914c7b9. - trainfer: trainfer/objectives/unlike.py — checkout audited
1c6391f3773b.