REN s07 final results: heldout_v1 suite and Typed Decisions

Adapter Oviezibe/ren-s07 @6345e948. Both evals finished Oct 19 2026 (suite 03:55 WAT, typed decisions ~06:32 WAT).

Typed Decisions

Important: s07 was fine-tuned on the typed-decisions TRAIN split (soft targets). It belongs in the "fitted or fine-tuned on train" group, not next to zero-shot models like Jev. The test split was never trained.

run	acc	Brier	ECE	ECE +TS	score-type acc	score MAE	agreed	disagreed	cov@0.7 (acc)	cov@0.9 (acc)	P50 ms	P95 ms
s06 (old path)	0.620	0.198	0.111	0.103	0.525	0.471	0.713	0.484	0.567 (0.775)	0.300 (0.853)	5911	7533
s07 LoRA fp16 2-GPU, old path	0.786	0.051	0.147	0.177	0.770	0.205	0.900	0.618	0.332 (0.958)	0.141 (1.000)	5534	7095
s07 LoRA fp16 2-GPU, fast path	0.785	0.051	0.146	0.175	0.768	0.205	0.899	0.617	0.331 (0.958)	0.141 (1.000)	2093	2357
s07 merged fp16 2-GPU, fast path	0.787	0.051	0.148	0.177	0.770	0.205	0.900	0.621	0.331 (0.958)	0.141 (1.000)	1764	2037
s07 merged NF4 1-GPU, fast path	0.726	0.077	0.084	0.128	0.693	0.274	0.833	0.568	0.372 (0.930)	0.135 (0.996)	1970	2233
Jev 1.13.0 (zero-shot, published)	0.740	0.148	0.045	n/a	n/a	0.389	0.841	0.591	0.576 (0.864)	0.245 (0.937)	687	777

s07 beats s06 by +16.6 points of accuracy (0.620 to 0.786) and beats Jev's published 0.740 on the fp16 paths; Brier 0.051 vs 0.148 (Jev) vs 0.198 (s06).
Calibration is the weak spot: s07 is underconfident (mean confidence 0.64, overconfidence about -0.15), so ECE is 0.147 vs Jev's 0.045. Temperature scaling fit on calib (T about 1.1-1.3) did not help top-1 ECE (0.177), though it nudged Brier down slightly. For s06, TS cut Brier 0.198 to 0.130.
Because it is underconfident, s07 covers less at each threshold (33% at >=0.7, 14% at >=0.9) but is very accurate when it does (95.8%, 100%).
The fast path (prompt encoded once, answer prefixes scored on a copy of the cache) matches the old path (max |dp| 0.006-0.011 on 40-case checks) at about 2.6-3.1x lower latency. NF4 on one GPU costs about 6 points of accuracy.
REN latency is on-device on Kaggle 2xT4 (one request at a time); Jev's is client-side over the network, so they are not directly comparable.

REN is a solo proof of concept, not the base of any other product.

heldout_v1 suite (single pass)

Finished Oct 19 2026 at 03:55 WAT on Kaggle 2xT4 (ren-s05-eval notebook, Version 5, 43 min). Adapter Oviezibe/ren-s07 @6345e948, base Qwen/Qwen3.5-9B, greedy decoding, same heldout_v1 items as base/s04/s05/s06. Overlap check: ren-data-s07 @6f33450c disjoint from heldout_v1. Full table: ren-evals/s07-vs-base/results.md.

run	add-info (format)	Decide	Decide Brier	MMLU-Pro	interruption all (exec)	interruption add-info (exec)	terminal	HumanEval+
base	0.260	0.677	0.419	0.615	0.786	0.643	0.857	0.804
s04	0.920	0.840	0.231	0.635	0.833	0.571	0.905	0.773
s05	0.660	0.830	0.252	0.630	0.833	0.571	1.000	0.755
s06	0.960	0.853	0.212	0.620	0.881	0.714	0.976	0.822
s07	1.000	0.863	0.203	0.625	0.905	0.786	0.857	0.779

Gains vs s06: add-info 0.96 -> 1.00, Decide 0.853 -> 0.863 (Brier 0.212 -> 0.203; Decide score-type questions 0.821), interrupts 0.881 -> 0.905 (add-info 0.714 -> 0.786, stop 1.00), MMLU-Pro 0.620 -> 0.625. Format compliance 1.00 on all 7 kinds; answer accuracy 0.991.
Decide ECE 0.042 (base 0.075); after cross-fitted temperature 0.040.
Regressions vs s06: terminal 41/42 -> 36/42 (back to base level), HumanEval+ 0.822 -> 0.779 (vs base 0.804, -0.025, CI [-0.067, 0.012], not significant). The extra 2.2M typed-decision prompt tokens likely diluted the code/terminal replay.
n is small (terminal and interruption 42 items each), so differences of a few items are within noise.

REN is a solo proof of concept, not the base of any other product. s07 trained on the typed-decisions TRAIN split (soft targets); the typed-decisions test split was never trained.

Next: s07b repair pass to restore terminal and HumanEval+ (built, not launched yet).