REN s06 final results vs base, s04 and s05 (heldout_v1)
Adapter Oviezibe/ren-s06 @6f42694a. Greedy decoding, same items for every model. Finished Oct 18 2026 (boost run done 11:00 WAT).
Single-pass (raw, no thinking)
| metric | n | base Qwen3.5-9B | s04 | s05 | s06 |
|---|---|---|---|---|---|
| HumanEval+ pass@1 | 163 | 0.804 | 0.773 | 0.755 | 0.822 |
| Interruption add-info answer (format) | 50 | 0.26 | 0.92 | 0.66 | 0.96 |
| Interruption correction answer (format) | 50 | 0.00 | 1.00 | 0.96 | 1.00 |
| All format answers correct | 350 | 0.654 | 0.980 | 0.937 | 0.989 |
| Decide accuracy | 300 | 0.677 | 0.840 | 0.830 | 0.853 |
| Decide Brier (lower is better) | 300 | 0.419 | 0.231 | 0.252 | 0.212 |
| Terminal task success | 42 | 0.857 | 0.905 | 1.000 | 0.976 |
| Interrupt handled (exec-verified) | 42 | 0.786 | 0.833 | 0.833 | 0.881 |
| Interruption add-info (exec-verified) | 14 | 0.643 | 0.571 | 0.571 | 0.714 |
| MMLU-Pro (letter logits) | 200 | 0.615 | 0.635 | 0.630 | 0.620 |
Thinking mode (NOT single-pass, s06 only)
s04 and s05 were never run in thinking mode, so there is no s04/s05 column here. Truncated thinking counts as wrong.
| boost | n | s06 single-pass on same items | s06 thinking |
|---|---|---|---|
| Decide, thinking only when confidence < 0.7 | 300 (55 routed) | 0.853 | 0.877 |
| MMLU-Pro, thinking (3,072 tokens) | 64 of 200 (time cap) | 0.656 | 0.812 (12 truncated) |
| HumanEval+, thinking (3,072 tokens) | 80 of 163 (time cap) | 0.912 | 0.800 (15 truncated) |
Honest reading
- s06 is the best REN round so far on most axes and the first above base on HumanEval+, but that gain (0.822 vs 0.804) is small and not statistically significant (95% CI -0.025 to +0.061).
- Add-info recovered from s05's 0.66 to 0.96. That test is an in-distribution template; exec-verified add-info (0.714, n=14) is the more honest signal.
- Terminal is 41 of 42, so it did not keep s05's 100%. Decide 0.853 single-pass is below the 0.88 goal; routing low-confidence items to thinking gets 0.877, still just short.
- MMLU-Pro thinking (0.812) and HumanEval+ thinking (0.800) cover only the subsets reached within the time cap, so they are not comparable to the full-set single-pass numbers. On HumanEval+, thinking did worse than single-pass on the same items, mostly from truncation.
- Stop gate (add-info >= 0.92, terminal 100, HE+ >= s04): add-info and HE+ pass, terminal misses by one task. The 90 MMLU-Pro and 95 HumanEval+ targets are not reachable for a 9B model.