Round 4 (s04) build: coding recovery on top of s03
Built 2026-10-04 (WAT) from eval/results/s03-vs-base.md (commit 1e49b79). Scripts: scripts/build_round4.py
(data), scripts/make_notebooks.py (s04_notebook), scripts/make_eval_notebook.py (s04_notebook),
scripts/dryrun_s04.py (CPU dry run). Reports: data/built/reports/r4_*.json.
Diagnosis (s03-vs-base samples, Oviezibe/ren-evals/s03-vs-base/samples_humaneval.jsonl)
- Score: HumanEval+ 0.755 vs base 0.804, 15 lost and 7 gained (McNemar p = 0.13, 95% CI [-0.104, 0.006]).
- Lost: HumanEval/20, 25, 39, 57, 64, 65, 86, 93, 97, 102, 113, 120, 122, 134, 155.
- Gained: 54, 59, 75, 95, 103, 106, 147.
- Error types of the 15 losses: 13 AssertionError and 2 sandbox timeouts. All are full functions with the right
signature in a closed code block, so none is a format problem. Code-block rate is 0.994 for s03 vs 0.963 for base.
Typical examples:
- /20: no sort, so the pair comes back in the wrong order.
- /97:
a % 10withoutabs(), wrong for negative inputs. - /64: only lower-case
ycounted. - /57: a broken monotonic check.
- /39: a Fibonacci walk that never ends (timeout).
- 14 of the 15 losses were already lost by s02. The drift sits in the inherited s02 LoRA. s03's base self-distillation (1,035 rows, 24% of tokens) won back 8 of s02's 23 losses (0.706 → 0.755) without losing new ones beyond 1.
- On the held-out OpenCodeInstruct problems (
code_check.json, 96 stubs) s03 already beats the base (85 vs 82). So the remaining gap is the HumanEval-style edge cases the s02 drift broke, not general coding ability. - Phase-0 yield in s03: 1,280 base answers in 68 min (60-min cap); 80.9% pass (stubs 88.9%, task statements 52%).
Levers: chosen and rejected
| lever | decision | evidence / reason |
|---|---|---|
| more base self-distilled code | yes: the main lever | It is the only lever with measured effect (s02 → s03 +4.9 pp on the same tasks). s04 trains 952 of the 1,009 reusable s03 answers plus ~700 fresh ones, about 0.75M tokens vs s03's 0.58M. |
| reuse s03's phase-0 answers | yes | Greedy base answers are deterministic, so regenerating them would waste about 70 GPU-minutes. 1,034/1,035 re-pass locally (bwrap), and their rendering is asserted identical to the s03 rows. They ship as the ordinary group open:general/selfdistill_s03. |
| fresh pool | yes | 1,098 new OpenCodeInstruct problems from shards 8-10, 85% stubs (stubs pass 89% vs 52% for tasks, so more rows per GPU-minute). Phase 0 is capped at 40 min, with 64 held-out problems for code_check.json. |
| best-of-n from the base | no | Within the same GPU minutes, greedy on fresh stubs yields about 4x more passing rows than resampling the ~11% of stubs the base fails greedily. Time is the binding constraint. |
| rejection-sampled self-distillation from s03 | no | s03's own answers carry the inherited s02 drift (14/15 losses persist) and need extra s03 generation time. On problems both pass they add little signal. |
| higher code share | yes: 55% (s03: 34%) | Coding is the only failing gate. |
| lower LR | yes: 2e-5 (s03: 3e-5), warmup 6 | Third continuation of the same LoRA, code-heavy. Less movement on the gates that pass. |
| short code-only phase | no | Terminal (0.881 vs 0.85) and interruption (0.810 vs 0.80) have the smallest margins. A code-only tail would put the recency effect on code and risk those. Code is interleaved instead. |
| small replay | yes: 45% | Every s03 group is kept at ≥ 0.5x its s03 weight, with agent and interruption slightly favoured. |
| weight interpolation | as a check, not a fix | The s04 eval re-runs HumanEval+ with the adapter scaled to 0.5 (LORA_SCALE_CHECK, about 18 min) to show whether a partial merge reaches parity if s04 alone does not. Only HumanEval is re-run at that scale. |
ren-data-v3 (private: Oviezibe/ren-data-v3 @ bc8703cb13856131b397b4206dcacb69dc84addf)
24,172 records = ren-data-v2 (22,065, every group unchanged) + 2 groups:
open:general/code_selfdistill_r4: 1,098 prompts (the phase-0 pool, weight 0 as ordinary rows). 1,150 written; the filter dropped 12 via 13-gram decontamination against HumanEval / MBPP / LiveCodeBench and some near-duplicates; 27 duplicate v2 prompts and 4 internal duplicates were also dropped.open:general/selfdistill_s03: 1,009 base answers. 1,034 re-verified; 16 dropped as dup-v2 prompts, the rest by the filter.
Overlap (r4_overlap_report.json, the eval's own check, heldout_v1 vs all v3 shards + templates): disjoint, 0 items
in every section, 0 HumanEval+ and 0 MMLU-Pro drops. No Llama-generated rows, no OpenAI / Anthropic outputs. Seed
20261301 is registered in heldout.TRAIN_SEEDS_R4.
Mix (ROUND4_MIX) and the CPU dry run on real v3
Mock tokenizer at 3.6 chars/token; phase 0 uses the reference answers on CPU, with the unit tests actually run.
| area | weights | dry-run share |
|---|---|---|
| coding | selfdistill_s03 0.27 (952 rows), selfdistill:code 0.20 (709 rows), code_only 0.08 | 54.8% |
| agent | first_action_r3 .035, r2 .005, terminal r3/r2/v0 .025/.02/.015, open terminal .025, tool_use .02, swe .015 | 16.2% |
| interruption | revise_r3 .055, interrupt_r2 .045, stop_r3 .02, dialogue .015, open_interruption .015 | 15.0% |
| format | template .045, web .01, safety .025 | 8.0% |
| decide | typed_decision .06 | 6.0% |
The dry run gave 3,824 rows, 1.61M tokens and 121 steps. Problems: none, including all the round-3 row contracts. If phase 0 underfills its 0.20, the remainder goes to the other groups (logged).
_mix_literal now writes weights finer than 2 decimals exactly. Before this fix 0.035 → 0.04; the s02 and s03 mixes
are unaffected.
Run plan, time and quota
- Training:
/workspace/kaggle_upload/ren_s04_smoke.ipynb(→Oviezibe/ren-s04-smoke, ~40 min), thenren_s04_full.ipynb(→Oviezibe/ren-s04).- Start point:
INIT_ADAPTER = Oviezibe/ren-s03 @ c142b458. - Settings: 1.6M tokens, lr 2e-5, cap 2.75 h.
- Session: load ~15 min, phase 0 ~45 min, training 2.1-2.55 h (214-175 tok/s; s03 measured 210), evals ~15 min. About 3.3-3.8 h in total.
- Start point:
- Eval:
/workspace/kaggle_upload/ren_eval_s04_smoke.ipynb(optional, ~25 min), thenren_eval_s04_full.ipynb.- Results go to
Oviezibe/ren-evals/s04-vs-base/. - Stop-at-
<|im_end|>is kept (STOP_AT_IM_END = True). - Full run about 1.7 h (s03's 83 min + ~18 min scale check).
- Results go to
- Quota: the weekly quota resets Sat Oct 10 01:00 WAT (00:00 UTC). About 8 h remain.
- Full training + full eval ≈ 5.0-5.5 h, within the 6 h target.
- Smoke training adds ~0.6 h, so ~5.6-6.1 h.
- Optional eval smoke adds ~0.4 h, giving ≤ ~6.5 h. Margin is about 1.5-2.4 h, which is not enough for a second full round. panel looks empty).
Gate (s04-vs-base)
- HumanEval+ ≥ base.
- No regression beyond noise on the gates s03 passes: format answer ≥ 0.95, agent format ≥ 0.95 / answer ≥ 0.90, Decide ≥ base, terminal ≥ 0.85, interruption all ≥ 0.80 / stop ≥ 0.80.
- With n = 163 a ±3-4 pp swing is within noise. Look at gained/lost and the scale check before concluding.
Risks
- Small residual signal: the lost tasks are specific edge cases. More base-style code may not flip all of them (s03 recovered 8 of 23). The scale check gives a fallback data point.
- LoRA scale check: it mutates PEFT
scalingdicts. It is tested only with mocks on CPU; on GPU a no-effect probe raises a warning note, and the scale is always restored. - Phase 0: the 40-min cap may yield fewer than ~700 rows. The remainder goes to replay, not code.
- Nemotron safety rows: Mixtral-generated, kept as before, flagged.