EmoGenie: Benchmarking and Evaluating Expressed and Evoked Emotion in T2I Generation with MLLMs

What it claims

EmoGenie is a benchmark of 6,400 AI-generated images that splits affect into expressed emotion (what a depicted person shows) and evoked emotion (what the viewer feels). The split is built into the prompts, the label taxonomy and the annotation dimensions. Prior generated-image emotion sets (MEMO-Bench, EmoNet-Face) cover only expressed emotion in portraits. Natural-image sets such as EEmo-Bench cover evoked emotion but not generated content. EmoProbe is an emotion-tuned MLLM built on InternVL3.5-8B-Instruct. It answers emotion questions and also outputs continuous valence, arousal, dominance (VAD) and aesthetic scores from the same model.

How it works

The two perspectives use separate prompt pools and get different annotation fields:

flowchart LR
  A[Expressed pool<br/>400 prompts] --> C[8 T2I models<br/>6,400 images]
  B[Evoked pool<br/>400 prompts] --> C
  C --> D{Perspective}
  D -->|expressed| E[Ekman 6 + neutral<br/>+ expression intensity]
  D -->|evoked| F[Mikels 8 categories<br/>+ VAD 1-9]
  E --> G[Aesthetic 1-5<br/>both]
  F --> G

• Taxonomies. Expressed images use Ekman's six basic emotions plus neutral. Evoked images use Mikels' eight: amusement, awe, contentment, excitement, anger, disgust, fear and sadness.
• Generators. Six open models: PixArt-Σ, SD3, SD3.5, FLUX.2, Z-Image and Qwen-Image. Two closed models: Seedream 5.0 and GPT-Image-2. Most run at 1024², FLUX.2 at 1360² and Qwen-Image at 1328².
• Annotation round 1. Each image gets 15 independent raters, for 384K dimension-level judgments in total. Categories are aggregated by majority vote and scores by the mean. If an image ties between categories, it stays in the dataset but is excluded from questions that need a unique answer. VAD uses the 9-point Self-Assessment Manikin scale.
• Annotation round 2. Evidence-grounded descriptions are collected for 300 single images and 100 pairs, covering 500 images.
• QA composition. 40K single-image and 50K image-pair perception questions. There are 1,057 description questions (957 single, 100 pair), and all of them are held out for evaluation only.
• EmoProbe Stage 1. Supervised fine-tuning on EmoGenie-QA, tuning the vision encoder, the projector and the language model.
• EmoProbe Stage 2. The projector is frozen, and the vision and language parts are trained with LoRA. Four regression heads, one per dimension, read response-aligned hidden states. The loss combines response cross-entropy with masked regression losses. Image pairs are encoded jointly in one context.

Evidence

The fetched text stops at the start of the method section, before any results. The authors claim state-of-the-art QA accuracy and better correlation with human ratings, but no metrics, baselines or numbers appear in what was retrieved. The authors also say T2I models and MLLMs vary markedly across perspectives and emotion categories, but this is likewise unquantified in the fetched text.

Limits

• Results not shown. The page is truncated before the experiments, so none of the claims above can be checked from this text.
• Release status unknown. Nothing in the fetched text mentions a code, weights or data release, or a licence.
• VAD only for evoked images. Expressed images get expression intensity instead, so continuous affect prediction cannot be evaluated on portraits.
• Aesthetic ratings are coarse. They use a 1–5 scale, averaged over 15 raters.
• Tied images drop out. Images with tied category votes are excluded from unique-answer questions, which may remove the most ambiguous generations.
• Annotation agreement not shown. The agreement analysis is deferred to Appendix B, which was not fetched.