REN eval: round-4 adapter vs base (s04-vs-base)

• base_model: Qwen/Qwen3.5-9B
• adapter: Oviezibe/ren-s04 @ 2ca3a441 (continued from s03 @ c142b458)
• smoke_test: False (full heldout_v1)
• load_mode: lora16_2gpu
• finished: Mon Oct 5 00:08:35 2026 UTC (main eval, 76.9 min on 2×T4); LoRA scale check committed 01:24 WAT (15 min)
• Kaggle: nutinman/ren-s02-train Version 7 (scriptVersionId 355288092)
• HF results: Oviezibe/ren-evals s04-vs-base/ (results.json, results.md, lora_scale_check.json)
• Training: step 122/122, train_loss 0.636, skipped 0; data Oviezibe/ren-data-v3 @ bc8703cb13; 55% coding (base self-distill)
• Overlap: heldout_v1 vs ren-data-v3 + templates, 0 items dropped in every section

Same harness as s03 (stop at <|im_end|>), so s03 and s04 absolute numbers compare directly.

Headline: base vs s03 vs s04

Round-4 gate (round4-build.md): HumanEval+ >= base with no regressions

Diagnosis

• Coding moved the right way but did not fully close. s04 is 0.755 to 0.773 vs s03 (9 gained, 6 lost) and 0 timeouts (s03 had 2). The 11 problems still lost vs base are HumanEval/19, 20, 40, 57, 64, 67, 93, 97, 102, 113, 134; 8 of those (20, 57, 64, 93, 97, 102, 113, 134) were already s03 losses, i.e. the inherited s02 LoRA drift. All are AssertionError edge cases in well-formed full functions, not format problems (code-block rate 0.982 vs 0.963 base).
• Halving the LoRA barely helps (0.773 to 0.779, still loses 20, 57, 64, 93, 97, 102, 113, which shows the drift lives in the low-rank update itself, not in its magnitude). Weight interpolation is not a cheap fix, and halving would likely give up some of the interruption/agent gains it was not measured on.
• In-notebook code check (64 held-out OpenCodeInstruct problems): s04 61 vs base 60. General coding is at or above base; the HumanEval+ gap is a handful of benchmark-style edge cases.
• Everything else is REN's best yet: best Decide accuracy and calibration, best terminal score, interruption correction jumped to 0.929, stop still 14/14, agent answer 50/50, format answer 0.980.

Decision

1. Serve s04 at full strength (Oviezibe/ren-s04 @ 2ca3a441). It beats s03 on every area except one interruption add_info item and MMLU-Pro noise, and the 0.5 scale gains only one HumanEval problem while untested elsewhere. s03 stays the rollback.
2. Ship-ready: yes for a first product (beta), with an honest coding caveat. The strict gate missed by 5/163 problems and that gap is inside noise; REN's selling points (interrupts, agent tool calls, Decide, format) are all clearly above base. Do not market it as "better than Qwen at coding"; market interruption/agent work, with coding "on par".
3. Serving must keep the stop-at-<|im_end|> rule (part of the measured behaviour).
4. Next coding fix belongs to the bigger-model round, not another 9B continuation: the drift is baked into the s02-lineage LoRA and only ~4 h of Kaggle T4 remain until Sat Oct 10 01:00 WAT. Options for later: retrain a fresh LoRA from base on the v3 mix (no s02 inheritance), or apply the recipe to the 14-30B model.

Raw Hub table mirrored below for archival.

REN eval: pilot adapter vs base (s04-vs-base)

• base_model: Qwen/Qwen3.5-9B
• adapter: Oviezibe/ren-s04
• adapter_revision: 2ca3a4416cf846177ba7f7a0e056f3e2cde5d793
• smoke_test: False
• load_mode: lora16_2gpu
• sandbox: uid
• finished: Mon Oct 5 00:08:35 2026
• minutes: 76.9
• heldout_version: heldout_v1
• overlap_check: Oviezibe/ren-data-v3@bc8703cb13 (24172 records, sha256-checked) + templates

Paired over the items both runs completed (greedy decoding, same prompts). Δ = adapter − base; 95% CI from a paired bootstrap over items; p = exact McNemar on the items that changed (binary metrics).

LoRA scale check (HumanEval+ only)