FlowAct-R2: Beyond Talking Avatar via Streaming Multimodal References and Proactive Agent Planning

What it claims

FlowAct-R2 (ByteDance Intelligent Creation) is a live streaming avatar system. In a human preference study against Vidu-S1 it reports GSB (good/same/bad) margins of +54.76% for video quality and +40.48% for real-time interaction. Against Vidu-S2, it qualitatively shows cleaner product-reference switching. Vidu-S2 bleeds attributes from the previous product into the next one; FlowAct-R2 does not. Output is real-time 720p over hour-scale sessions. The paper gives no latency, frame-rate or hardware figures.

How it works

Generator. It adapts Seedance 2.0 Mini's reference-to-video backbone using Diffusion Forcing, which gives each token its own noise level. This allows autoregressive streaming inference while keeping the backbone's bidirectional spatiotemporal attention. Other components:

• Video-driven RoPE aligns reference chunks that arrive mid-stream with the generation timeline.
• (R + I)2V is explicitly trained reference-plus-image conditioning that anchors identity.
• Partially noised history frames are used as conditioning, so the model tolerates its own imperfect past output and accumulates less drift.
• Separate audio-driven training covers lip sync, expression and head pose.
• Coarse-to-fine inference progressively restores visual detail.

Agent. Expensive preparation happens offline, outside the latency loop. During the session, each audience event triggers two separate decisions: what to perform and when/how to perform it. A behavior can be initiated, overlaid, deferred, interrupted or resumed. The schedule is expressed as idle, listening, speaking and skill-execution periods. These periods are what produce the rolling prompts, audio and reference updates sent to the DiT.

flowchart LR
  A[Text/image/audio/video assets + manual draft] --> P[Offline: persona, agenda, skills]
  P --> D{Behavior + scheduling decision}
  E[Audience text/audio] --> D
  H[Summarized history] --> D
  D -->|initiate / overlay / defer / interrupt / resume| S[Idle · Listen · Speak · Skill]
  S --> G[Streaming MM-DiT]
  G --> H

Evidence

• vs Vidu-S1 (human eval, GSB): +54.76% video quality and +40.48% real-time interaction. In the listening phase, FlowAct-R2 varies gaze, head pose and expression. Vidu-S1 holds roughly the same frontal pose, smile and gesture.
• vs Vidu-S2 (qualitative only): Over three successive product references in one shopping stream, Vidu-S2 fused adjacent products. It produced white packaging, a branch graphic and a tall tin instead of the referenced items. In a separate example, it replaced a patterned cylinder wrap with stripes. FlowAct-R2 kept the layout, the pattern and the box shape.
• Demos: dance and garment try-on with a rear view, video chat turn-taking, and an outdoor vlog with scene changes.

How to use it

The page names a project page (https://bone-11.github.io/Flowact-R2/) and a Hugging Face Space (https://huggingface.co/spaces/ProAudience/FlowAct-R2). It says nothing about:

• released weights or code
• a licence
• model size
• hardware requirements
• an API

Treat this as a demo, not an adoptable artifact, until those are checked.

Limits

• Human evaluation is thin. The rater count, number of samples and prompts are not given, and there are no automatic metrics.
• The Vidu-S2 comparison is cherry-picked. It consists of selected frames only.
• No ablations. Nothing isolates the effect of video-driven RoPE, noised-history conditioning or the offline/online agent split.
• Hour-scale stability is asserted, not measured. No drift metric over time is reported.
• No limitations or failure modes are discussed.
• The scenario list is inconsistent. The introduction lists "interactive gaming" while the rest of the paper says "live vlogging".
• It depends on Seedance 2.0 Mini. The paper does not say whether that backbone is available to outside users.