REN eval: round-5 adapter vs base (s05-vs-base)
- base_model: Qwen/Qwen3.5-9B
- adapter: Oviezibe/ren-s05 @
3e097444 (fresh LoRA from base, lr 5e-5, warmup 15, 740 steps, 0 skipped)
- training data:
Oviezibe/ren-data-s05 @ 7f77c5fe, 20,441 rows / 9.51M tokens, 2 Kaggle sessions (time-cap restart at step 646)
- smoke_test: False (full heldout_v1)
- load_mode: lora16_2gpu
- finished: Sat Oct 17 2026 12:15:52 UTC (main eval, 73.2 min on 2×T4); LoRA scale check committed 12:32 UTC (16.8 min)
- Kaggle: nutinman/ren-s05-eval Version 1
- HF results:
Oviezibe/ren-evals s05-vs-base/ (results.json, results.md, lora_scale_check.json)
- Overlap: heldout_v1 vs ren-data-s05 (37,824 records, sha256-checked) + templates, 0 items dropped
Same harness as s03/s04 (stop at <|im_end|>), so absolute numbers compare directly.
Headline: base vs s04 vs s05
| Area |
base |
s04 |
s05 |
s05 Δ vs base |
s05 vs s04 |
| Format compliance (all, n=350) |
0.894 |
1.000 |
1.000 |
+0.106 |
= |
| Format answer (all) |
0.654 |
0.980 |
0.937 |
+0.283 |
-4.3 pp |
| Format answer, interruption_add_info |
0.260 |
0.920 |
0.660 |
+0.400 |
-26 pp |
| Format answer, interruption_correct |
0.000 |
1.000 |
0.960 |
+0.960 |
-4 pp |
| Decide accuracy (n=300) |
0.677 |
0.840 |
0.830 |
+0.153 |
-1.0 pp |
| Decide Brier (lower better) |
0.419 |
0.231 |
0.252 |
-0.167 |
worse |
| Decide ECE (lower better) |
0.075 |
- |
0.034 |
-0.040 |
- |
| HumanEval+ pass@1 (n=163) |
0.804 |
0.773 |
0.755 |
-0.049 (7 gained / 15 lost, p=0.13, CI [-0.104, 0.006]) |
-1.8 pp (5 gained / 8 lost) |
| HumanEval+ at LoRA scale 0.5 |
0.804 |
0.779 |
0.767 |
-0.037 (5 / 11) |
-1.2 pp |
| Terminal success (n=42) |
0.857 |
0.905 |
1.000 |
+0.143 (6 / 0, p=0.03) |
+9.5 pp |
| Terminal tool calls well-formed |
1.000 |
1.000 |
0.976 |
-0.024 (1 item) |
-1 item |
| Interrupt (all, n=42) |
0.786 |
0.833 |
0.833 |
+0.048 |
= |
| Interruption stop / correction / add_info |
0.857 / 0.857 / 0.643 |
1.000 / 0.929 / 0.571 |
1.000 / 0.929 / 0.571 |
+0.143 / +0.071 / -0.071 |
= |
| Interruption format (all) |
0.952 |
0.976 |
1.000 |
+0.048 |
+2.4 pp |
| Agent format / answer |
1.000 / 0.740 |
1.000 / 1.000 |
1.000 / 1.000 |
0 / +0.260 |
= |
| MMLU-Pro (n=200) |
0.615 |
0.635 |
0.630 |
+0.015 |
-0.5 pp (noise) |
s05 targets (s05_plan.md)
| Target |
s05 |
Result |
| Decide ≥ 97% |
0.830 |
miss (plan odds were ~5%, realistic 88–92%; came in below that range) |
| Terminal ≥ 97% |
1.000 (42/42) |
hit |
| HumanEval+ > 80.4 |
0.755 |
miss (still below base, not significant: p=0.13) |
| Above s04 on every benchmark |
- |
no: up on terminal and interruption format, tied on interrupt/agent/format compliance, down on HumanEval+, format answer (interruption add-info), Decide and MMLU-Pro |
Diagnosis
- Terminal is the clear win. 42/42 end-state checks, the first REN round to clear every terminal task, against 0.857 for base and 0.905 for s04. The new open + programmatic terminal data and strict first-action rows worked.
- Coding did not recover. Lost vs base: HumanEval/40, 49, 57, 64, 67, 86, 93, 97, 101, 102, 113, 115, 122, 134, 155; gained 75, 95, 103, 106, 109, 110, 147. 0 timeouts, code-block rate 0.994. 57, 64, 93, 97, 102, 113, 134 are the same edge cases s03 and s04 lost, so a fresh LoRA on open OpenCodeInstruct data reproduces the same drift; s05 had none of s04's base self-distilled code. Halving the LoRA only gets to 0.767.
- Interruption add-info answer regressed in the format section (0.92 → 0.66, 27 gained / 7 lost vs base) even though the dedicated add-info data was in the mix, while the exec-verified interrupt score is identical to s04. Worth reading
samples_format.jsonl before round 6.
- Decide is a hair below s04 and less well calibrated by Brier, though its ECE (0.034) is good. Doubling the weak families moved json_schema (+13 pp vs base) but not pep440_select or shell_destructive_random.
Decision
- Keep s04 as the served default. s05 is not better across the board; it trails s04 on coding, Decide and the interruption add-info answer.
- s05 is the best terminal/agent-task checkpoint (
Oviezibe/ren-s05 @ 3e097444) and is kept for terminal-heavy use and as data for the next round.
- Next round should combine s05's terminal data with s04's base self-distilled code and look into the add-info answer drop before training. Both s04 and s05 are Kaggle-trained and non-commercial.