Listen. Compare.
See what the model hears.

What is this model?

The encoder of OpenAI Whisper Base or Small is fine-tuned end to end. A learned mixture of pooled hidden states from every encoder layer helps separate task-specific information. Clip-level experts predict 192 scalar targets: 40 emotions, 57 VoiceNet dimensions, genuineness and burst blend, 19 Empathic Insight Plus extras, four AudioBox aesthetics, seven DNSMOS estimates, burst count and 61 VoiceCLAP attributes. Separate heads predict 128D Orange timbre, 250D Orange identity and literal characters per second. A 20 ms frame head finds burst regions; independent onset/duration proposals locate overlapping events and an event-local 53-class head assigns labels. This is not an ASR model; it has no Whisper decoder or transcript output.

A predicted z value is (raw prediction − training-set median) / training-set standard deviation for that target. It shows what is unusually high or low for the training distribution; it is not a probability. VoiceNet codes and all other output names are exposed below with raw values and z values.

Training in one pass through S1→S10

Each stage contributed one epoch of its Ladder mixture plus 10% P3 synthetic burst snippets. One cosine learning-rate schedule spans all ten stages, without restarts: 2% warmup, 10% minimum peak rate, 281,006 optimizer updates and global batch 256 on 16 GH200 GPUs. The encoder peak rate was 2×10⁻⁵ for Base and 1.5×10⁻⁵ for Small; task heads used 3×10⁻⁴. The gallery and independent test both use the final S10 stage-best checkpoint. Scalar heads used masked normalized Huber losses; speaker vectors used cosine distance plus Huber; burst timing and classes used frame/region and categorical objectives. Missing teacher targets were masked. Training code and stage checkpoints are in the model repository.

Independent S10 evaluation

Measured on 9,693 P3 test clips whose source identities were excluded from P3 train and validation. Real-speech characters/s comes from the separate Ladder validation split. Small leads on the summary metrics, while end-to-end burst classification remains notably harder than classifying a known span.

Score-family agreement with stored teacher annotations

Normalized MAE is the average absolute error per target divided by its training standard deviation. Pearson is agreement with the stored annotation model, not a human listening rating. Family values are unweighted means across valid dimensions.

How to read the listening examples

The purple curve is the model's 20 ms burst-frame probability. Purple blocks mark predicted onset/duration proposals; teal blocks on P3 examples mark inserted source events. A proposal lists its onset score and top three class softmax percentages. These percentages compare classes within this model and are not calibrated correctness probabilities. Both model columns predict the same clip independently. Each model's expandable tables contain every raw score and training-reference z value. Orange vector coordinates are in the accompanying JSON.

Split provenance: 10 examples come from the P3 source-disjoint test; 40 DE/EN real-speech examples come from the fixed Ladder validation split and were excluded by index from Ladder training; 10 Emolia originals show additional languages but exact-source exclusion from training has not been established. The gallery is qualitative; the metrics above use the full test sets.

Listening grid