SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data

What it claims

SemMSA is the first framework to use LLMs for semantic-level compensation in Multimodal Sentiment Analysis with incomplete data, rather than reconstructing missing features. It targets intra-modal missingness (corrupted tokens/frames/audio segments within each modality) and reports state-of-the-art results averaged over missing rates from 0.0 to 0.9, improving accuracy by 1.4% on average over prior methods on MOSI, MOSEI, and SIMS.

How it works

Two modules the summary names but does not detail:

• CSR (Cross-modal Semantic Refinement): Frozen modality-specific encoders produce visual/acoustic sequences; lightweight transformer adapters (B blocks) with M learnable prompt embeddings use cross-attention (prompts as queries against the features) plus self-attention to aggregate reliable evidence, projecting to M visual and M acoustic prefix tokens. These concatenate with LLM-embedded language into a fixed-order prefix U⁽⁰⁾. CSR then recurrently appends the LLM's last-position hidden state as a continuous latent token for O refinement steps (zₖ = f(U⁽ᵏ⁻¹⁾)), never decoding text, then pools + linearly projects to a semantic vector Hs.
• CSA (Cross-modal Spectral Alignment): Builds a 4×4 kernel Gram matrix K over normalized semantic/visual/acoustic/language vectors using an RBF kernel, eigendecomposes it, and uses a softmax-over-eigenvalues loss (temperature τ) to push the dominant eigenvalue λ₁ to dominate — driving all four toward a rank-one shared direction. A separation regularizer penalizes squared inner products between dominant eigenvectors (projected back to representation space) of different instances to prevent collapse. Final representation is element-wise sum Hs+Hv+Ha+Hl.

Evidence

On MOSI (averaged over missing rates 0.0–0.9), SemMSA reaches Acc-2 74.36/73.91, F1 74.18/73.82, Acc-5 40.93, Acc-7 36.49, versus the strongest baselines TF-Mamba (Acc-2 73.46/72.54, Acc-5 37.74, Acc-7 33.95) and P-RMF (Acc-2 71.53/72.81). SemMSA leads every listed MOSI metric. The MOSEI row and the full MOSI MAE/Corr values are cut off in the fetched text.

Limits

• Fetched page is truncated mid-Table 1: MOSEI numbers, SIMS results, ablations, and compute/training costs are not visible here — verify on the source.
• No install/run details, code link, LLM backbone identity or size, or licence appear in the fetched text.
• The method depends on a frozen LLM in the inference loop with O recurrent forward passes; the per-step overhead is claimed "token-efficient" but not quantified in the visible text.
• Preprint (arXiv v1, dated Sep 2026, NeurIPS 2026 listing); results not independently verified.