From continuous emotion scores to benchmark classes

CREMA-D and RAVDESS: what the learned adapter actually does

A small classifier learns how the existing emotion predictions correspond to an acted-speech benchmark’s categories. Base S4 and Small S3 remain frozen. All reported adapted results below are predictions for actors excluded from that adapter’s training.

Accuracy is correct clips divided by all clips. UAR is mean recall across classes, also called balanced accuracy or UA in some papers. Macro F1 averages class-specific F1. Reported aggregate metrics pool each clip’s single out-of-fold prediction; they are not necessarily identical to an unweighted average of fold metrics. The intervals resample 2,000 actor clusters with their stored predictions; they do not include fresh model refits or variation across random split seeds.

Predictions are the adapter input

The selected inputs are the 40 emo_* scores distilled from Empathic Insight and 40 emotion entries from the voiceclap_attribute:* family. The latter are predictions made by the same released Whisper model; no external teacher is run at this calibration stage. These 80 values are concatenated. The 57 VoiceNet style scores, quality scores, timbre/identity embeddings, burst predictions and encoder hidden states are not adapter inputs. The encoder and all existing neural heads receive no gradient updates.

For each scalar, first apply the released training-reference transform z = (raw − training_mean) / training_std. Then fit an additional StandardScaler using only the current training fold: x = (z − fold_mean) / fold_std. A regularized multinomial logistic classifier learns logit[class] = Σ weight[class, j] × x[j] + bias[class]. The prediction is the class with the largest logit. Neutral gets its own learned weights and bias. Softmax can be displayed, but those probabilities were not independently calibrated.

This is a linear score-to-label readout with six outputs for CREMA-D (486 stored weight/bias values) or eight outputs for RAVDESS (648 stored weight/bias values). Fold means and scales add 160 statistics for an 80-feature adapter. Optimization uses an L2-regularized multiclass log-loss, lbfgs, up to 1,000 iterations, tolerance 10⁻⁴. It has no neural-training epoch schedule. The original multitask regression models were trained earlier with the objectives documented in their model card.

How the mapping and boundaries were selected

1. Outer test: split actors into five folds with StratifiedGroupKFold, seed 20261005. All clips from an actor stay together. Each clip is tested exactly once.
2. Inner selection: within the outer training actors, run three more actor folds. Compare five feature views: semantic pools from either emotion family, all 40 scores from either family, and their 80-score concatenation. For each view, try C ∈ {0.01, 0.1, 1, 10}; smaller C means stronger regularization. Choose by inner held-actor accuracy. Ties in C prefer stronger regularization.
3. Outer fit: refit the selected view and C on all outer training clips and score the excluded actors. Scaling is always fitted inside the corresponding training partition. Outer labels do not fit that fold’s weights or choose its C/view.
4. Deployment refit: after collecting test predictions, use a separate three-fold actor CV on the full benchmark to choose a final configuration, then fit it on all clips. That downloadable refit has no independent test score of its own.

Both models selected the 80-score concatenation in every outer fold on both datasets. The boundary improvement is therefore learned from the joint pattern across scores, rather than a manually adjusted threshold chosen against the outer test labels.

Exactly how many samples were used?

There is no single official train/test partition in this experiment. The tables show the actual five test partitions for each corpus; Base and Small share the same splits. “Outer training clips” is the number fitting the adapter that predicts that row’s test actors.

CREMA-D

RAVDESS

Is this a fair evaluation?

Yes, for supervised within-corpus adaptation to held-out actors. For the reported nested-CV figures, each test actor is absent from that adapter’s fitting and hyperparameter selection. We did not train the scored adapter on its own test clips. Training on other labeled actors from the same corpus is an explicit part of this protocol.

It is a post-hoc experiment. We inspected CREMA-D’s initial results and confusions before designing this adaptation. Nested folds protect the fitting and selection inside the stated candidate search, but cannot turn earlier benchmark inspection into a pristine blind final test. The results should be labeled exploratory supervised actor-CV adaptation. A stronger confirmatory result would freeze the entire procedure and use a new untouched corpus or a genuinely reserved actor set.

It is not the paper’s zero-shot protocol. The VoiceNet paper uses audio–text comparisons with no target-corpus label fitting. Our adapted rows use thousands of labeled CREMA-D or RAVDESS training clips. Their larger accuracies do not establish a zero-shot improvement over VoiceCLAP. The fixed-rule rows remain available as the benchmark-unfitted reference. These corpora and checkpoints are also different from the listening gallery’s final S10 models.

Scope of “unseen actor”. Actor separation is verified for this adapter. Overlap between upstream Whisper pretraining, annotation teachers, the original ladder training corpus and these public benchmark recordings has not been audited. The claim therefore does not guarantee that the complete model stack has never encountered an actor or recording. CREMA-D labels here are filename-based intended acted emotions, not crowd audio-majority perception labels. RAVDESS includes all eight emotions and only its 1,440 audio-only speech recordings from 24 actors; calm and neutral remain separate.

Published specialized classifiers: supervised comparison context

EmoBox (Interspeech 2024), Table 4 reports the following supervised readouts. CREMA-D and RAVDESS use four speaker folds. Its downstream network reads final-layer representations, with a hidden layer, ReLU and pooling; training lasts 100 epochs. These are useful reference values, with different folds, inputs and model sizes.

WA corresponds to clip accuracy; UA corresponds to class-average recall. Our CREMA-D accuracies are near the wav2vec 2.0 base reference, below the stronger specialized readouts. On RAVDESS, the Small score adapter lies between the HuBERT base and WavLM large reference accuracies. This numerical positioning is not a matched-fold ranking.

Luna-Jiménez et al. (Sensors 2021), Table 1 report 76.58% audio-only accuracy for fine-tuned PANNs CNN-14 on eight-class RAVDESS, using five subject folds. Their audio-and-video fusion reaches 80.08%; that multimodal result has additional visual information. Their subject folds differ from ours, and their audio model is fine-tuned on RAVDESS.

Paper zero-shot values: retain their separate interpretation

The following accuracies are from VoiceNet Table 2. Their target-corpus labels do not train a readout. They are shown for context; the supervised rows above answer a different evaluation question.

Downloads and reproduction

Both experiments reused cached scalar predictions. CREMA-D took 37.6 seconds; RAVDESS took 16.7 seconds, each with one CPU thread. No additional GPU hours were consumed by adapter training. Raw score arrays, row labels and actor IDs, source hashes, all inner trials, outer decisions, safe JSON adapters, training normalization and the exact fitting/inference code are published together.

Reproduction instructions and dependencies · Original CREMA-D fitting / JSON inference · RAVDESS fitting / JSON inference · Model card, weights, original training and inference code · CREMA-D source and label definitions

Applying the CREMA-D all-data adapter to RAVDESS was not evaluated: the class inventories differ. Each corpus has its own separately fitted adapter. No adaptation to EmoNet-Voice, VoiceNet-Emo, VoiceNet-Ext or vocal-burst targets was performed in this experiment.