REN eval: round-3 adapter vs base (s03-vs-base)
- base_model: Qwen/Qwen3.5-9B
- adapter: Oviezibe/ren-s03 @
c142b458 - smoke_test: False (full heldout_v1)
- load_mode: lora16_2gpu
- finished: Sun Oct 4 18:25:57 2026 (session), 83.1 min on 2×T4
- Kaggle: nutinman/ren-eval-pilot-vs-base Version 8 (scriptVersionId 355226099)
- HF results:
Oviezibe/ren-evalss03-vs-base/ - Training: step 187/187 from s02 adapter, train_loss ≈ 1.169, skipped=0; data
Oviezibe/ren-data-v2@558471996f;TRAINING_COMPLETE.jsonon Hub
Stop-at-turn-end caveat: every generate() now stops at <|im_end|>. That changes both base and adapter numbers vs s02-vs-base (notably base agent format 0.96 → 1.00). Cross-round comparison of absolute adapter scores is fairer than comparing Δ vs base across rounds.
Headline vs base (this eval harness)
| Area | base | s03 | Δ | vs s02 adapter |
|---|---|---|---|---|
| Format compliance (all, n=350) | 0.897 | 1.000 | +0.103 | 0.983 → 1.000 |
| Format answer (all) | 0.657 | 0.977 | +0.320 | 0.946 → 0.977 |
| Decide accuracy (n=300) | 0.680 | 0.827 | +0.147 | 0.787 → 0.827 |
| HumanEval+ pass@1 (n=163) | 0.804 | 0.755 | −0.049 | 0.706 → 0.755 (+4.9 pp) |
| Terminal success (n=42) | 0.857 | 0.881 | +0.024 | 0.905 → 0.881 (small dip) |
| Interrupt (n=42) | 0.786 | 0.810 | +0.024 | 0.762 → 0.810 |
| Interruption stop | 0.857 | 1.000 | +0.143 | 0.714 → 1.000 |
| Interruption correction | 0.857 | 0.786 | −0.071 | 0.857 → 0.786 |
| Interruption add_info | 0.643 | 0.643 | 0.000 | 0.714 → 0.643 |
| Agent format | 1.000 | 1.000 | 0.000 | 0.880 → 1.000 |
| Agent answer | 0.740 | 1.000 | +0.260 | 0.880 → 1.000 |
| MMLU-Pro (n=200) | 0.615 | 0.640 | +0.025 | 0.635 → 0.640 |
Round-3 gates (from round3-plan-from-s02.md)
| Gate | Target | s03 | Result |
|---|---|---|---|
| HumanEval+ ≥ base | ≥ 0.804 | 0.755 (−4.9 pp; 7 gained / 15 lost) | fail |
| Interrupt ≥ 0.80; stop ≥ 0.80 | 0.80 / 0.80 | 0.810 / 1.000 | pass |
| Agent format ≥ 0.95; answer ≥ 0.90 | 0.95 / 0.90 | 1.000 / 1.000 | pass |
| Format answer ≥ 0.95; terminal ≥ 0.85; Decide ≥ base | 0.95 / 0.85 / ≥base | 0.977 / 0.881 / 0.827 | pass |
Bottom line: Round 3 closed the stop-at-turn-end and agent-format holes and pushed format, Decide, and interruption stop to their best so far. HumanEval+ recovered another ~5 pp vs s02 but is still ~5 pp under base (mostly wrong implementations / 2 timeouts, not missing fences). Not deployment-ready yet — one more coding-focused round (round 4) before ship.
Decision: round 4 (coding recover), then ship check
- Keep s03 as the production candidate for interruption/agent/format/Decide behaviour.
- Round 4: coding-heavy continue or retrain from s03 — more on-policy self-distill + execution-verified code, lighter interruption/agent since those gates are green; gate HumanEval+ ≥ base (stretch: ≥ base + 1 pp without Decide/terminal regressions > 2 pp).
- After round-4 full eval clears coding, treat 9B as deployment-ready for a first product; AWS 27B stays reserved until then.
- Product note: the stop-at-
<|im_end|>rule must ship in the serving path — it is part of the measured behaviour, not only a training quirk.
Detail notes
- Stop fix: interruption stop 14/14 exec-verified; agent format 50/50. Base also benefits in this harness (agent format 1.0), so the paired Δ understates how much s02 was hurt by missing
<|im_end|>. - HumanEval+: code_block 0.994; 15 losses are AssertionError / sandbox timeout style failures, same pattern as s02. Phase-0 self-distill helped but did not fully close the gap.
- Decide: Brier 0.419 → 0.238; ECE 0.070 → 0.041 — best calibration yet.
- Terminal: 0.881 still above the 0.85 gate; slight absolute dip vs s02 0.905 is within noise (5 gained / 4 lost).
- Overlap: heldout_v1 vs ren-data-v2 + templates — Hub
overlap_report.json(disjoint claim from build).
Raw Hub table mirrored below for archival.
REN eval: pilot adapter vs base (s03-vs-base)
- base_model: Qwen/Qwen3.5-9B
- adapter: Oviezibe/ren-s03
- adapter_revision: c142b4580cb8a18ab1964b3338e99b13731ba072
- smoke_test: False
- load_mode: lora16_2gpu
- sandbox: uid
- finished: Sun Oct 4 18:25:57 2026
- minutes: 83.1
- heldout_version: heldout_v1
- overlap_check: Oviezibe/ren-data-v2@558471996f (22065 records, sha256-checked) + templates
Paired over the items both runs completed (greedy decoding, same prompts). Δ = adapter − base; 95% CI from a paired bootstrap over items; p = exact McNemar on the items that changed (binary metrics).
| section | group | metric | n | base | adapter | Δ | 95% CI | gained / lost | p |
|---|---|---|---|---|---|---|---|---|---|
| format | agent | format compliance | 50 | 1.000 | 1.000 | +0.000 | [0.000, 0.000] | 0 / 0 | 1 |
| format | decide | format compliance | 50 | 1.000 | 1.000 | +0.000 | [0.000, 0.000] | 0 / 0 | 1 |
| format | interruption_add_info | format compliance | 50 | 0.460 | 1.000 | +0.540 | [0.400, 0.680] | 27 / 0 | 1.49e-08 |
| format | interruption_correct | format compliance | 50 | 1.000 | 1.000 | +0.000 | [0.000, 0.000] | 0 / 0 | 1 |
| format | interruption_stop | format compliance | 50 | 1.000 | 1.000 | +0.000 | [0.000, 0.000] | 0 / 0 | 1 |
| format | fast | format compliance | 50 | 1.000 | 1.000 | +0.000 | [0.000, 0.000] | 0 / 0 | 1 |
| format | think | format compliance | 50 | 0.820 | 1.000 | +0.180 | [0.080, 0.280] | 9 / 0 | 0.00391 |
| format | all | format compliance | 350 | 0.897 | 1.000 | +0.103 | [0.071, 0.134] | 36 / 0 | 2.91e-11 |
| format | agent | answer correct | 50 | 0.740 | 1.000 | +0.260 | [0.160, 0.380] | 13 / 0 | 0.000244 |
| format | decide | answer correct | 50 | 0.880 | 1.000 | +0.120 | [0.040, 0.220] | 6 / 0 | 0.0312 |
| format | interruption_add_info | answer correct | 50 | 0.260 | 0.900 | +0.640 | [0.480, 0.780] | 33 / 1 | 4.07e-09 |
| format | interruption_correct | answer correct | 50 | 0.000 | 1.000 | +1.000 | [1.000, 1.000] | 50 / 0 | 1.78e-15 |
| format | interruption_stop | answer correct | 50 | 0.920 | 0.980 | +0.060 | [-0.020, 0.160] | 4 / 1 | 0.375 |
| format | fast | answer correct | 50 | 0.900 | 0.960 | +0.060 | [-0.020, 0.140] | 4 / 1 | 0.375 |
| format | think | answer correct | 50 | 0.900 | 1.000 | +0.100 | [0.020, 0.200] | 5 / 0 | 0.0625 |
| format | all | answer correct | 350 | 0.657 | 0.977 | +0.320 | [0.269, 0.369] | 115 / 3 | 1.65e-30 |
| decide | all | accuracy | 300 | 0.680 | 0.827 | +0.147 | [0.093, 0.200] | 58 / 14 | 1.65e-07 |
| decide | all | Brier (multi-class, lower=better) | 300 | 0.419 | 0.238 | -0.181 | [-0.233, -0.133] | - | - |
| decide | choice | accuracy | 66 | 0.667 | 0.788 | +0.121 | [0.000, 0.242] | 12 / 4 | 0.0768 |
| decide | score | accuracy | 28 | 0.357 | 0.714 | +0.357 | [0.143, 0.571] | 12 / 2 | 0.0129 |
| decide | yes_no | accuracy | 206 | 0.728 | 0.854 | +0.126 | [0.068, 0.184] | 34 / 8 | 6.88e-05 |
| decide | json_schema | accuracy | 23 | 0.739 | 0.739 | +0.000 | [-0.174, 0.174] | 2 / 2 | 1 |
| decide | pep440_contains | accuracy | 19 | 0.947 | 1.000 | +0.053 | [0.000, 0.158] | 1 / 0 | 1 |
| decide | pep440_count | accuracy | 13 | 0.231 | 0.692 | +0.462 | [0.231, 0.769] | 6 / 0 | 0.0312 |
| decide | pep440_resolve | accuracy | 16 | 0.500 | 0.750 | +0.250 | [-0.125, 0.625] | 7 / 3 | 0.344 |
| decide | pep440_select | accuracy | 13 | 0.538 | 0.385 | -0.154 | [-0.462, 0.154] | 1 / 3 | 0.625 |
| decide | py_exception | accuracy | 19 | 0.684 | 0.842 | +0.158 | [0.000, 0.316] | 3 / 0 | 0.25 |
| decide | py_output | accuracy | 34 | 0.706 | 0.912 | +0.206 | [0.059, 0.382] | 8 / 1 | 0.0391 |
| decide | py_raises | accuracy | 34 | 0.912 | 0.971 | +0.059 | [0.000, 0.147] | 2 / 0 | 0.5 |
| decide | py_raises_random | accuracy | 19 | 0.421 | 0.895 | +0.474 | [0.263, 0.684] | 9 / 0 | 0.00391 |
| decide | regex_count | accuracy | 15 | 0.467 | 0.733 | +0.267 | [-0.067, 0.600] | 6 / 2 | 0.289 |
| decide | regex_match | accuracy | 33 | 0.788 | 0.909 | +0.121 | [0.030, 0.242] | 4 / 0 | 0.125 |
| decide | regex_random | accuracy | 14 | 0.929 | 1.000 | +0.071 | [0.000, 0.214] | 1 / 0 | 1 |
| decide | shell_destructive | accuracy | 33 | 0.636 | 0.818 | +0.182 | [0.030, 0.333] | 7 / 1 | 0.0703 |
| decide | shell_destructive_random | accuracy | 15 | 0.533 | 0.467 | -0.067 | [-0.267, 0.133] | 1 / 2 | 1 |
| humaneval | all | HumanEval+ pass@1 (greedy) | 163 | 0.804 | 0.755 | -0.049 | [-0.104, 0.006] | 7 / 15 | 0.134 |
| humaneval | all | answer has a code block | 163 | 0.963 | 0.994 | +0.031 | [0.006, 0.061] | 5 / 0 | 0.0625 |
| terminal | all | task success (end-state checks) | 42 | 0.857 | 0.881 | +0.024 | [-0.119, 0.167] | 5 / 4 | 1 |
| terminal | all | every tool call well-formed | 42 | 1.000 | 1.000 | +0.000 | [0.000, 0.000] | 0 / 0 | 1 |
| terminal | all | used the tool at all | 42 | 1.000 | 1.000 | +0.000 | [0.000, 0.000] | 0 / 0 | 1 |
| interruption | add_info | interrupt handled (exec-verified) | 14 | 0.643 | 0.643 | +0.000 | [-0.357, 0.286] | 3 / 3 | 1 |
| interruption | correction | interrupt handled (exec-verified) | 14 | 0.857 | 0.786 | -0.071 | [-0.357, 0.214] | 2 / 3 | 1 |
| interruption | stop | interrupt handled (exec-verified) | 14 | 0.857 | 1.000 | +0.143 | [0.000, 0.357] | 2 / 0 | 0.5 |
| interruption | all | interrupt handled (exec-verified) | 42 | 0.786 | 0.810 | +0.024 | [-0.143, 0.190] | 7 / 6 | 1 |
| interruption | add_info | interruption format | 14 | 1.000 | 1.000 | +0.000 | [0.000, 0.000] | 0 / 0 | 1 |
| interruption | correction | interruption format | 14 | 1.000 | 1.000 | +0.000 | [0.000, 0.000] | 0 / 0 | 1 |
| interruption | stop | interruption format | 14 | 0.857 | 1.000 | +0.143 | [0.000, 0.357] | 2 / 0 | 0.5 |
| interruption | all | interruption format | 42 | 0.952 | 1.000 | +0.048 | [0.000, 0.119] | 2 / 0 | 0.5 |
| mmlu_pro | all | accuracy (letter logits) | 200 | 0.615 | 0.640 | +0.025 | [-0.010, 0.060] | 9 / 4 | 0.267 |
| mmlu_pro | all | top-1 token is a letter | 200 | 1.000 | 1.000 | +0.000 | [0.000, 0.000] | 0 / 0 | 1 |
| decide | all | ECE (10 bins, lower=better) | 300 | 0.070 | 0.041 | -0.029 | [-0.077, 0.009] | - | - |
| decide | all | ECE after cross-fitted temperature | 300 | 0.036 | 0.034 | -0.002 | - | - | - |
| decide | all | label-token probability mass | 300 | 0.995 | 1.000 | +0.005 | - | - | - |
| decide | all | top-1 token is a label | 300 | 1.000 | 1.000 | +0.000 | - | - | - |
| decide | all | MAE vs exact probability | 48 | 0.239 | 0.204 | -0.035 | - | - | - |