REN s07b final results: heldout_v1 suite and Typed Decisions

Typed Decisions

Finished Oct 19 2026 at about 22:41 WAT on Kaggle 2xT4 (ren-s05-eval notebook, Version 10, about 25 min). Dataset LocalLLaMA/typed-decisions test split, 400 cases / 2,000 decisions. Adapter Oviezibe/ren-s07b @8b0cbf80 (final, step 93), base Qwen/Qwen3.5-9B, LoRA fp16 split over 2 GPUs, fast path (KV-prefix). s07 and s06 rows are from the earlier runs on the same items. Temperatures are fit only on the 180-case calib slice (a TRAIN-split slice never trained on). Full data: ren-evals/typed-decisions-s07b/summary.json.

Important: s07b (like s07) was fine-tuned on the typed-decisions TRAIN split (soft targets). It belongs in the "fitted or fine-tuned on train" group, not next to zero-shot models like Jev. The test split was never trained.

run	acc	Brier	ECE	KL	score-type acc	score MAE	agreed	disagreed	cov@0.7 (acc)	cov@0.9 (acc)	P50 ms	P95 ms
s06 (old path), raw	0.620	0.198	0.111	0.527	0.525	0.471	0.713	0.484	0.567 (0.775)	0.300 (0.853)	5911	7533
s06 + NLL TS (per type, T 2.3-3.8)	0.620	0.130	0.103	0.254	0.525	0.439	0.713	0.484	0.186 (0.895)	0.000 (n/a)	5911	7533
s06 + ECE TS (T 1.00, grid floor/ceiling 0.3-1.0)	0.620	0.198	0.111	0.527	0.525	0.471	0.713	0.484	0.567 (0.775)	0.300 (0.853)	5911	7533
s07 LoRA fast, raw	0.785	0.051	0.146	0.095	0.768	0.205	0.899	0.617	0.331 (0.958)	0.141 (1.000)	2093	2357
s07 + NLL TS (T 1.09-1.26)	0.785	0.049	0.175	0.091	0.768	0.204	0.899	0.617	0.278 (0.969)	0.095 (1.000)	2093	2357
s07 + ECE TS (T 0.42)	0.785	0.113	0.017	0.386	0.768	0.267	0.899	0.617	0.674 (0.879)	0.369 (0.957)	2093	2357
s07b LoRA fast, raw	0.789	0.049	0.155	0.092	0.763	0.203	0.903	0.622	0.314 (0.965)	0.132 (1.000)	2219	2467
s07b + NLL TS (T 1.15-1.20)	0.789	0.048	0.184	0.089	0.763	0.203	0.903	0.622	0.275 (0.969)	0.098 (1.000)	2219	2467
s07b + ECE TS (T 0.37)	0.789	0.119	0.026	0.462	0.763	0.275	0.903	0.622	0.703 (0.871)	0.391 (0.958)	2219	2467
Jev 1.13.0 (zero-shot, published)	0.740	0.148	0.045	n/a	n/a	0.389	0.841	0.591	0.576 (0.864)	0.245 (0.937)	687	777

No give-back on typed decisions. s07b scores 0.789 accuracy (95% CI 0.772-0.806) vs s07's 0.785, with Brier 0.049 vs 0.051, even though typed-decision rehearsal fell from ~60% to ~29% of training tokens. By type: choice 0.773, score 0.763, no/unclear 0.840.
Raw calibration is still the weak spot. s07b is underconfident (mean confidence 0.634 vs accuracy 0.789, ECE 0.155). The NLL-fitted temperature (T about 1.15-1.2) makes top-1 ECE worse (0.184), as it did for s07.
The new ECE-fitted temperature fixes ECE. One global T fit on the calib slice by minimising top-1 ECE (grid 0.30-1.00) picks T 0.37 for s07b (0.42 for s07). Test ECE drops 0.155 -> 0.026 (s07: 0.146 -> 0.017), below Jev's 0.045. Confidence gating gets much more useful: at >=0.7 it covers 70% of decisions at 87.1% accuracy (raw: 31% at 96.5%), and at >=0.9 it covers 39% at 95.8% (Jev: 24.5% at 93.7%).
The trade-off: sharpening with one global T also makes the wrong answers confident, so Brier rises 0.049 -> 0.119 and KL from the gold distribution 0.09 -> 0.46 (Brier is still better than Jev's 0.148). Use the ECE-fitted T for thresholds and routing on top-1 confidence, and the raw (or NLL-TS) probabilities when the full distribution matters.
For s06 the ECE grid stays at T 1.00 because s06 is overconfident and the grid stops at 1.0; NLL TS remains its better fix.
Latency (on-device, Kaggle 2xT4, LoRA fp16 fast path, one request at a time) is P50 2219 ms / P95 2467 ms, close to s07's 2093 / 2357. Jev's is client-side over the network, so they are not directly comparable. Fast path vs old path check: max |dp| 0.004 on 20-case checks.

REN is a solo proof of concept, not the base of any other product.

Next: s07b keeps every typed-decision gain but failed the suite gate on terminal (36/42) and HumanEval+ (0.791), so it is not promoted over s06 for code/terminal work. The terminal/code fix (s07c) waits for the Kaggle quota reset (~Oct 24).

heldout_v1 suite (single pass) + 50/50 merge

Finished Oct 19 2026 at about 21:11 WAT on Kaggle 2xT4 (ren-s05-eval notebook, Version 9). s07b part 43 min, merge part 33 min. Adapter Oviezibe/ren-s07b @8b0cbf80, merge m_s06w50_s07w50 from Oviezibe/ren-s07b-merges @aad5cd34, base Qwen/Qwen3.5-9B, greedy decoding, same heldout_v1 items as base/s04/s05/s06/s07. Overlap check: ren-data-s07 @6f33450c disjoint from heldout_v1. Full tables: ren-evals/s07b-vs-base/results.md and s07b_gate.md.

run	add-info (format)	Decide	MMLU-Pro	interruption all (exec)	interruption add-info (exec)	terminal	HumanEval+
base	0.260	0.677	0.615	0.786	0.643	0.857	0.804
s06	0.960	0.853	0.620	0.881	0.714	0.976	0.822
s07	1.000	0.863	0.625	0.905	0.786	0.857	0.779
s07b	0.960	0.870	0.625	0.905	0.714	0.857	0.791
merge s06/s07 50/50	1.000	0.857	0.625	0.881	0.714	0.905	0.791

No-regression gate (terminal >= 0.952, HumanEval+ >= 0.811, add-info >= 0.96, Decide >= 0.853, interruption >= 0.881, MMLU-Pro >= 0.615):

line	s07b	merge 50/50
terminal	0.857 (36/42) FAIL	0.905 (38/42) FAIL
HumanEval+	0.791 FAIL	0.791 FAIL
add-info	0.960 pass	1.000 pass
Decide	0.870 pass	0.857 pass
interruption	0.905 pass	0.881 pass on items (37/42, same as s06; flagged only by float rounding)
MMLU-Pro	0.625 pass	0.625 pass

Neither candidate passes. The repair pass did not recover terminal (still 36/42, same as s07 and base) and only partly recovered HumanEval+ (0.779 -> 0.791, s06 had 0.822).
s07b keeps s07's gains: Decide 0.870 (best of any round; Brier 0.200, ECE 0.043), interruption 0.905, format compliance 1.00 on all 7 kinds, answer accuracy 0.986.
The merge sits between s06 and s07, as expected: terminal 38/42, add-info 1.00, but HumanEval+ no better than s07b.
n is small (terminal and interruption 42 items each), so a few items are within noise; still, terminal is 4 items short of the line.

REN is a solo proof of concept, not the base of any other product. s07b trained on the typed-decisions TRAIN split; the typed-decisions test split was never trained.

Typed-decisions eval on s07b finished Oct 19 at about 22:41 WAT; see the section below.