Self-synthesized paraphrase training
Self-synthesized paraphrase training is cont's attempt to broaden training prompts using paraphrases generated by the same model.
Project status: Implemented as self_synth in the autoresearch runner. This entry describes the source audit of 14 September 2026; historical measurements retain their original dates.
Mechanism
Request paraphrases of each selected question. Ask the model to answer each paraphrase and retain those whose extracted answer still matches the original expected answer. Memorize the original and retained paraphrases toward the same target. This is data augmentation followed by SFT, not a new distillation loss.
Implementation and controls
Controls include number of source tasks, paraphrases per task, and the memorize cap/threshold/patience/rate. The checked-in block uses 20 tasks and m=3 with memorize cap=100, threshold=0.95, patience=10, lr=5e-4. The runner logs paraphrase and retention information. It depends on the task suite's answer extractor and expected answers.
Evidence and evaluation
The early journal's m=3 self-synthesis arm reached heldout 0.40 versus 0.60 for the memorize baseline, with probe 0.53. This small local result motivated describing the approach as a regression at that scale rather than a demonstrated distributional-generalization gain.
Limitations and interpretation
Answer agreement does not prove a paraphrase preserved meaning: the model can answer a different question with the same short label. Self-generation can reinforce shared errors and familiar wording. Sampling, filtering, and repeated targets also change the compute budget, so compare retained data and total tokens rather than only original K.
Sources
- cont: autoresearch/experiment.py — checkout audited
87946914c7b9. - cont: autoresearch/config.json — checkout audited
87946914c7b9. - cont: docs/research/JOURNAL.md — checkout audited
87946914c7b9.