Skip to content

🎨 Multimodal327

ListGrid
What you seeeverything
RecentOldestAddedA–Z
All types327Papers215Articles51Repos24Threads22
Write-ups7Tools6Advisories1Talks1
🎨 Multimodal
x.comthread

🚨 AI News | TestingCatalog (@testingcatalog) on X

RoboAPI is a public API from PokeBot, powered by its world-action model Poke-Aurora, that lets robot arms do new tasks from natural-language instructions with no task-specific data collection or training. A user connects a supported arm with a wrist-mounted camera, gives a task such as "clean the table", and receives the next moves in near real time. The post gives no benchmarks, supported hardware list or pricing.

RoboAPIPoke-Auroraworld-action model
PokeBotTestingCatalog

models, multimodal· Added 1 Nov 2026, published 31 Oct 2026

x.comthread

Artificial Analysis (@ArtificialAnlys) on X

Hyperflow H3 is Video Rebirth's 8-step LoRA for MiniMax H3, released in September 2026 with open weights. Artificial Analysis ranks it #5 on AA-Video-I2V v2.1 and #3 on AA-Video-Editing v1.1, ahead of MiniMax H3 (768p). It costs $9.00 per minute for Image to Video and $18.00 for Video Editing at 1088p, under half of Dreamina Seedance 2.5 ($34.12 and $40.82). The thread gives per-capability and per-use-case breakdowns.

LoRAdiffusion model
Artificial AnalysisVideo RebirthHyperflow H3+6

models, evals, multimodal· Added 31 Oct 2026, published 30 Oct 2026

huggingface.cowriteup

nemotron-halloween

MonsterHouse is a Halloween interactive projection built on the NVIDIA Nemotron Voice Agent: Nemotron Speech ASR, Nemotron 3.5 Bolt E4B 4GB with a LoRA persona adapter, Magpie TTS Zeroshot and Audio2Face-3D in Unreal Engine, on two RTX Blackwell GPUs. Median end-of-speech-to-audio is 1.70 seconds live and 0.72 seconds with cached responses. The post gives a build path and tips on caching and state handling.

Nemotron Voice AgentNemotron Speech ASRNemotron 3.5 Bolt E4B 4GB+11
NVIDIANemotronQwen3 30B-A3B+3

multimodal, agents, training· Published and added 30 Oct 2026

github.comrepo

Show HN: Overtone – Transcribe, correct and cut captions for any screen shape

Overtone is a caption format (.overtone.json), a CLI and a video.js plugin that store each spoken word once and cut captions separately per screen shape (16:9, 9:16 or custom). The transcribe pipeline combines gpt-transcribe, Azure AI Speech and optional ElevenLabs Scribe, resolves disagreements with an LLM vote, and uses an LLM editor for corrections. A small public benchmark reports 5.3% word error rate and $0.023–0.025 per audio minute.

video.jsNode.jswhisper.cpp+11
OvertoneOpenAIgpt-transcribe+6

multimodal, llm-apps, evals· Added 30 Oct 2026, published 29 Oct 2026

arxiv.orgpaper

RADIOlogy: A Single-Pass Approach to Training-Free Zero-Shot Anomaly Detection

RADIOlogy is a training-free, single-pass zero-shot anomaly detection method that uses multi-head attention pooling in vision-language models. It scores anomalies by contrasting average similarity between vision features and sets of normal and abnormal prompts, with no windowed execution. RADIO agglomerative features drive a dense refinement pipeline for pixel-level segmentation, and the method is applied to medical images.

CLIPRADIOvision-language model
RADIOlogyarXiv

research, multimodal· Added 31 Oct 2026, published 29 Oct 2026

arxiv.orgpaper

PhysCount: Do MLLMs Count Visual Appearances or Physical Entities?

PhysCount is a benchmark of 546 real-world images and 4,595 human-written questions in 651 target-specific groups. It tests whether multimodal LLMs can separate visible appearances (direct view, reflections, glass) from the physical entities behind them. Across 40 MLLMs, counting appearances is much easier than linking them to entities, and the best model answers every question correctly in only 13.36% of groups.

vision-language model
PhysCountarXiv

evals, multimodal, research· Added 31 Oct 2026, published 29 Oct 2026

latent.spacearticle

Can Frontier Models See Past a Decoy? Elorian Tests Visual Reasoning

DecoyCount is a benchmark from Elorian and Mercor with 1,660 items that tests whether models count objects or fall back on visual shortcuts. Frontier models scored below 50%: GPT-6 Astra led with 43.17%, ahead of Gemini-3.8-flash (32.71%), Qwen3.8-max (28.31%) and Claude-Fable-5.1 (25.22%). CEO Andrew Dai says Elorian's Glimpse model scored 50.48% in a single internal run without tools, and contrasts visual reasoning with world models such as Atlas and Genie 4.

vision-language model
ElorianGlimpseDecoyCount+17

evals, multimodal, models· Added 28 Oct 2026, published 29 Oct 2026

arxiv.orgpaper

Benchmarking color judgment in multimodal LLMs with CIE-computable ground truth

A five-task benchmark tests multimodal LLM color judgment against CIE colorimetric ground truth: AWB pair judgment, skin-tone bias direction, CCT estimation, Delta E 2000 pair ordering, and color naming under changing illuminants. Eight endpoints were tested. On ColorChecker-24 stimuli, closed frontier models reach CCT MAE of 715-1056 K versus 1358-2133 K for open-weight models, and a non-ColorChecker control shows this gap is partly template-dependent.

vision-language modelQwen3-VLGLM+2
GLM-4.5VGPT-5.4Claude-Opus-4.7+4

evals, multimodal, research· Added 31 Oct 2026, published 29 Oct 2026

huggingface.coarticle

Granite Switch 4.1.1 — Update models like you update software

Granite Switch 4.1.1 is a backwards-compatible patch release of IBM's modular Switch architecture, which configures a language model with specialized, cache-efficient adapters on one endpoint. It updates the Granite Speech ASR model (10 languages, native punctuation and capitalization) and the Decision adapter in granitelib-core, measured on the Jev Decision Index test set. Only the new adapters were retrained; the base model is unchanged. IBM plans tutorials and an Adapter SDK for third-party adapters.

Hugging Face
IBMGranite SwitchGranite Speech+2

models, multimodal· Published and added 29 Oct 2026

x.comthread

Muhammad Waris (@WarisCodes) on X

Nemotron 3.5 Transcribe is a free open-weight speech-to-text model from NVIDIA covering 40 languages, including mixed-language speech. The post says it ties for first among open models with a 2.8% error rate. It says the model needs a large GPU such as an H100 or RTX 5090, so it will not run on a laptop. The post gives no benchmark name or model card link.

Nemotron 3.5 Transcribe
NVIDIANemotron

models, multimodal· Added 30 Oct 2026, published 29 Oct 2026

arxiv.orgpaper

M-PFT: Angular Margin for Calibrated Prompt-Tuning of Fine-Tuned Vision-Language Models

M-PFT is an angular-margin method for prompt tuning of vision-language models that fixes base-to-novel dual miscalibration (underconfidence on base classes, overconfidence on novel classes). It adds two losses: minimizing the maximum cosine similarity between tunable class embeddings, and minimizing angular distance to each zero-shot embedding. Tests on 9 prompt-tuning methods and 15 datasets show lower novel-class calibration error at comparable accuracy and negligible overhead.

vision-language modelprompt tuning
arXiv

research, multimodal, training· Added 31 Oct 2026, published 29 Oct 2026

arxiv.orgpaper

Aligning Medical Vision Language Models at the Capability Boundary with Entropy Adaptive Rubric Grounding

MedCBR is a two-stage alignment framework for medical vision language models. Difficulty Weighted Boundary Preference Optimization (DBPO) trains on counterfactual preference pairs at the model's capability frontier. Entropy Gated Grounded Policy Optimization (EGPO) then applies distilled clinical rubrics according to prediction entropy. It reaches 61.2% average accuracy on six medical benchmarks, with the largest gains on differential diagnosis, and the two stages combine super-additively.

vision-language modelpreference optimizationreinforcement learning
MedCBRDifficulty Weighted Boundary Preference OptimizationEntropy Gated Grounded Policy Optimization+1

research, training, multimodal· Added 31 Oct 2026, published 29 Oct 2026

x.comthread

🚨 AI News | TestingCatalog (@testingcatalog) on X

Nemotron 3.5 Transcribe is an open-weight, promptable NVIDIA mixture-of-experts model with 30B total and about 4B active parameters. One checkpoint does transcription in 40 languages, speaker attribution, translation, contextual prompting and audio understanding. The post says it ranked first in VoiceArena's comparison of 14 diarization plus ASR systems. The fetched text is cut off and gives no license or download details.

Mixture of Experts
NVIDIANemotronNemotron 3.5 Transcribe+1

models, multimodal· Added 30 Oct 2026, published 29 Oct 2026

huggingface.coarticle

Introducing AnySpeechRecognizer and AnyDecisionModel: Open Models for Speech and Decisions on Apple Platforms

AnySpeechRecognizer and AnyDecisionModel are two Swift packages from Hugging Face. The first mirrors Apple's SpeechAnalyzer API with swappable engines (NVIDIA Parakeet, Alibaba Qwen3-ASR on Core ML) and adds Sortformer speaker diarization. The second returns typed yes/no, choice and score answers with probabilities via MLX, Core AI, llama.cpp, Jev or Ollama. The post gives M4 Max timings, model sizes and calibration options.

SwiftCore MLMLX+10
AnySpeechRecognizerAnyDecisionModelAnyLanguageModel+17

multimodal, inference, models· Published and added 29 Oct 2026

arxiv.orgpaper

Leaving, Staying, or Living There? Duration Commonsense Reasoning with Human-LLM Co-annotations

This paper studies whether vision-language models can infer how long a person stays somewhere (living, staying, or leaving) from geotagged social media posts. The authors build a new corpus of geotagged tweets co-annotated by humans and LLMs. State-of-the-art models struggle even with multi-turn Chain-of-Thought prompting, often missing contextual cues and subtle hashtags while over-relying on prominent visual details.

vision-language modelLLM
arXiv

research, evals, multimodal· Added 30 Oct 2026, published 28 Oct 2026

arxiv.orgpaper

inkGround: Benchmarking Fine-grained Visual Grounding Capabilities of Vision-Language Models for Cursive Handwriting in the Wild

inkGround is a benchmark for stroke trajectory recovery from full-page, camera-captured handwriting, with 537 page images (about 18.7k words, 163k strokes) and pseudo-label references filtered by OCR agreement and human raters. Models get a page image and a target word, then output ordered stroke points. Seven VLMs were scored with DTW, Chamfer distance, readability and style similarity. Closed-source models, notably Gemini-3.1-Pro, did best, and the ranking held on IAM-OnDB.

vision-language model
inkGroundIAM-OnDBGemini-3.1-Pro+1

evals, multimodal, research· Added 30 Oct 2026, published 28 Oct 2026

arxiv.orgpaper

AskStream: When and What to Ask Proactively under Uncertainty in Streaming Video Understanding

AskStream is a multi-turn benchmark for streaming video understanding that adds a third proactive action, ask, alongside wait and answer, for cases where video or user information is incomplete. It has 438 videos, 2,714 manually annotated frame-level time windows (wait, ask, answer), four task categories and eight subtasks. The strongest model reaches only 52.18% on the proposed Ask-F1 metric, in both multiple-choice and open-ended settings.

multimodal large language model
AskStreamAsk-F1arXiv

evals, multimodal, research· Added 30 Oct 2026, published 28 Oct 2026

arxiv.orgpaper

Quantize by Doing: Task-Guided Mixed-Precision Quantization of Vision-Language-Action Models

QbD-VLA is a mixed-precision post-training quantization method for vision-language-action models. It uses task success from short closed-loop evaluations to assign weight and activation bit widths, accounting for interactions between components. On four VLA models across all four LIBERO suites it holds task performance at about 4-bit weights and under 6-bit activations. The gain on SIMPLER is small. On a Franka Research 3 arm, X-VLA shows no detected loss, but π0.5 fails.

vision-language-action model
QbD-VLALIBEROSIMPLER+4

research, inference, multimodal· Added 30 Oct 2026, published 28 Oct 2026

x.comthread

John Rood (@johnroodepic) on X

Nemotron 3.5 Transcribe is an open-weight, promptable 30B mixture-of-experts model from NVIDIA with about 4B active parameters, built for recorded audio. One checkpoint transcribes 40 languages, handles recordings up to an hour, labels up to eight speakers, translates, and answers questions about English audio. Weights and a demo are on Hugging Face. The post gives no benchmarks.

Mixture of ExpertsHugging Face
NVIDIANemotronNemotron 3.5 Transcribe+1

models, multimodal· Added 29 Oct 2026, published 28 Oct 2026

huggingface.coarticle

NVIDIA Nemotron 3.5 Transcribe and Granary v2: Intelligent Multilingual Transcription You Can Prompt and Adapt

Nemotron 3.5 Transcribe is an open-weight, promptable speech model that handles transcription in 40 languages, translation, diarization for up to eight speakers, and audio Q&A. It has about 4B active of 30B parameters and tops open-weight ASR and diarization leaderboards. The post also covers the Granary v2 curation pipeline (463,062 hours, 48 languages) and a medical fine-tuning case study that cut held-out WER by 45.8%.

NeMo SpeechNeMo CuratorvLLM+11
NVIDIANemotron 3.5 TranscribeGranary v2+16

models, multimodal, training, evals· Published and added 28 Oct 2026

arxiv.orgpaper

Data Scaling Laws in Stereo Matching: From In-domain Fitting to Cross-domain Generalization

This paper studies data scaling laws in stereo matching and releases XStereo, a realistic synthetic dataset of 200 thousand samples. In-domain training on five large datasets shows two regimes: a standard power law, then saturation, with a transition point roughly fixed for a given model. Cross-domain performance depends only weakly on source data scale, and distribution and coverage diversity reduce domain gaps more.

XStereoarXiv

research, training, multimodal· Added 30 Oct 2026, published 28 Oct 2026

arxiv.orgpaper

Complementing Text Based Moderation with Multimodal Grounding: Detecting Fraudulent Inducement Behaviors on Short Video Platforms

The paper studies detection and temporal localization of fraudulent inducements (prompts to follow, join chat groups, click links) in internet finance short videos. It compares OCR+LLM, OCR+MLLM and a zero-shot MLLM, and builds a dataset of 2,419 videos (66.49 hours, 3,194 events, four categories). OCR+MLLM reaches mIoU 26.52 versus 14.49 for keyframe OCR+LLM, with 67.0 percent macro recall, using a recall-weighted F2 localization reward.

MLLMLLMOCR
arXiv

research, multimodal, training· Added 30 Oct 2026, published 28 Oct 2026

arxiv.orgpaper

Improving the Dependency on Fine-Tuning Data in Transfer Learning for Speech Emotion Recognition

This paper proposes an fMLLR-based compensation mechanism that cuts the fine-tuning data needed in transfer learning for speech emotion recognition under train-test acoustic mismatch. An autoencoder reduces trainable parameters, and an affine transform in the encoder latent space models rotations and shifts. In a human-robot interaction task, mean CCC rose 16.8% with 12.9% of the training set and 14.4% with 50%.

fMLLRautoencoder
arXiv

research, multimodal, training· Added 30 Oct 2026, published 27 Oct 2026

arxiv.orgpaper

Know, then Draw: A New Learning Paradigm for Visual Generation

Know, then Draw is a learning paradigm for visual generation that uses autoregressive pretraining to acquire semantic knowledge and diffusion post-training for high-fidelity rendering. Controlled comparisons find autoregression the stronger pretraining objective, and semantic quality matters more than AR-only generation quality. This motivates a purely semantic tokenizer trained with language supervision only. A 3B text-to-image model trained in 6,600 H100 GPU-hours scores 86 on GenEval and 87 on DPG-Bench.

diffusion modelTransformer
arXivGenEvalDPG-Bench+1

research, multimodal, training· Added 28 Oct 2026, published 27 Oct 2026

arxiv.orgpaper

Perceiving and Predicting Dynamic Sports: Spatiotemporal Intelligence in Vision Models for Egocentric Tennis Video

This paper presents an annotated egocentric tennis dataset (egocentric video synced with five allocentric views, global positions, contact times, stroke types) and evaluates image encoders, video encoders and vision-language models on seven prediction tasks. Best models reach 0.14 m player and 0.54 m ball localization error and 0.14 s contact-time error. Depth input helps 3D ball localization, and longer histories do not help forecasting. A conditional diffusion model generates whole-body tennis motion in real time.

vision-language modeldiffusion modelTransformer
arXiv

research, multimodal, evals· Added 28 Oct 2026, published 27 Oct 2026

arxiv.orgpaper

VideoLVR: Latent Visual Reasoning over Grounded Video Evidence

VideoLVR is a method that lets video multimodal LLMs build latent visual evidence from local regions at relevant moments and reason over it interleaved with text tokens. A visual token prediction objective, defined over a vocabulary derived from the input video, supervises the latents. It extends to reinforcement learning with sampled visual tokens as actions and final-answer rewards, and improves fine-grained video benchmarks without much context growth.

vision-language modelLLM
VideoLVRarXiv

research, multimodal, training· Added 28 Oct 2026, published 27 Oct 2026

arxiv.orgpaper

As the Story Unfolds: Watching a Film and Identifying Characters as a Human Does

This paper defines a character naming task for films: given a name spoken in dialogue and its video context, the model must pick which visible character bears that name. It provides a dataset with annotations for character naming and first appearances, a video-and-dialogue model, and a comparison with proprietary multimodal LLMs. It also proposes a progressive framework that builds a character bank as the film plays, without cast lists or actor photographs.

multimodal large language model
arXiv

research, multimodal· Added 28 Oct 2026, published 27 Oct 2026

huggingface.coarticle

Build specialized multimodal models with LFM3

LFM3 is Liquid AI's open-weight family of six ~2B-parameter models (text, vision-language, audio, omni and two d1 decision models), all branched from LFM3-1.7B-Base. The base model uses a tri-hybrid backbone of 13 gated short convolutions, 5 sliding-window attention layers and 6 global attention layers, pre-trained on 22T tokens with a 256K context. The post gives transformers snippets, benchmark comparisons against LFM2.5 and Qwen3.5, and a robotics fine-tuning example.

TransformersHugging FacePyTorch+7
Liquid AILFM3LFM3-1.7B+29

models, multimodal, training· Published and added 27 Oct 2026

liquid.aiarticle

Introducing LFM3: Every Modality for Specialized Intelligence

LFM3 is Liquid AI's new family of open-weight multimodal models, six models of about 2B parameters branched from LFM3-1.7B-Base (22T pre-training tokens, 256K context). The hybrid backbone uses 13 gated short-convolution, 5 sliding-window attention and 6 global attention layers. Variants cover text, vision with video, audio, omni and two decision models, with on-device speed numbers, fine-tuning demos and robotics extensions.

llama.cppMLXvLLM+17
Liquid AILFM3LFM3-1.7B+32

models, multimodal, inference, training· Added 28 Oct 2026, published 27 Oct 2026

deepmind.googlearticle

Genie 4: A new generation of world simulation

Genie 4 is Google DeepMind's real-time interactive world model, announced as a research preview for trusted testers. It generates video with native audio and interactive dialogue, steered by keyboard controls and text prompts. Gemini 3.8 Flash acts as a guiding agent that plans steps and checks frames, and Gemini 4 Argon is used for robotics experiments. The post lists limitations such as drift and object permanence, and says outputs carry SynthID watermarks.

GeminiGoogle AI StudioSynthID
Google DeepMindGenie 4Gemini 3.8 Flash+5

models, multimodal, agents· Published and added 27 Oct 2026

blog.googlearticle

We’re announcing Genie 4, our new world model for real-time interactive simulation.

Genie 4 is Google DeepMind's world model for real-time interactive simulation, with native audio generated on the fly (ambient sound, music, dialogue). It builds on Gemini models: Gemini Omni grounds scenes in real-world appearance and behavior, and other Gemini models keep the simulation coherent. Users steer scenes by natural language. Access is limited to trusted testers now, with Google AI Ultra (Project Genie), the Gemini API and AI Studio to follow. The post gives no architecture or benchmark detail.

GeminiGoogle AI StudioGemini API
Google DeepMindGenie 4Gemini Omni+2

models, multimodal· Added 30 Oct 2026, published 27 Oct 2026

arxiv.orgpaper

When Visual Evidence Is Insufficient: How Instructions, Post-Training, and Internal Activations Shape Refusal in Multimodal Language Models

This paper studies how multimodal LLMs refuse questions that an image cannot answer, using the MoHoBench benchmark on three MLLMs. It varies the meaning and placement of refusal instructions, compares supervised and preference-based post-training, and intervenes on a refusal-related activation direction. Instructions change refusal at the sample level in ways aggregate rates hide. Post-training raises refusal most but can mix refusal with attempted answers. Activation intervention gives smaller, transferable changes.

vision-language modelLLM
MoHoBencharXiv

ai-safety, multimodal, research· Added 28 Oct 2026, published 27 Oct 2026

x.comthread

Manoj | The AI Dude (@Manojkumar2257) on X

heditor is an open-source (MIT) agent skill from pycodebr that has Claude Code, Codex, Cursor or OpenCode edit talking-head videos. It transcribes with faster-whisper, picks a 30-70s segment, cuts frame-accurately, builds HTML + GSAP motion graphics, mixes audio at -14 LUFS, and runs QA with vision-reviewed previews. It offers 42 named styles and a long-form mode with chapters, SRT/VTT and lower thirds. Install: npx skills add pycodebr/heditor -g.

Claude CodeCodexCursor+6
heditorpycodebr

coding-agents, prompting, multimodal· Added 28 Oct 2026, published 27 Oct 2026

x.comthread

Rohan Paul (@rohanpaul_ai) on X

Hi3D v1.0 (formerly Sparc3D) is an open-sourced image-to-3D model from Hi3D, which runs a closed v3.0 refining at 2048 cubes per side. In a test with 13 professionals in 3D algorithms, 562 judgments against eight other image-to-3D models favoured it 83.5% of the time. The thread also covers the Hi3D Agent and an MCP server for other agents. Back and bottom views were checked, where TRELLIS.2 collapsed a winged figure.

Model Context Protocol
Hi3DHi3D v1.0Sparc3D+2

models, multimodal, mcp· Added 28 Oct 2026, published 27 Oct 2026

arxiv.orgpaper

AxisQ: Depth-Adaptive Channel--Frequency Quantization for Camouflaged Object Detection

AxisQ is a post-training quantization framework for camouflaged object detection (COD) that allocates activation bits by feature depth. It uses the channel domain for shallow features and the local-frequency domain for deep features, under a fixed per-layer average bit budget. It adds a boundary-aware Hessian approximation for reconstruction. On CAMO at W2A2, weighted F-measure improves 12.7% for CFRN and 33.3% for CamoFormer-P over the strongest baselines.

CNNTransformer
AxisQCFRNCamoFormer-P+1

research, inference, multimodal· Added 28 Oct 2026, published 27 Oct 2026

simonwillison.netarticle

GPT-6.1 Terra

GPT-6.1 Terra is OpenAI's midrange model, announced 26 October 2026 and aimed at visual reasoning, computer use and clearer writing. It costs $1 per million input tokens and $5 per million output, about half of GPT-5.6 Terra and GPT-6.1 Sol and roughly 10x GPT-6 Luna. Simon Willison relays OpenAI's Ted Sanders, who rates it better than Astra and Sol at technical communication and much faster on computer use.

OpenAI API
GPT-6.1 TerraOpenAISimon Willison+7

models, multimodal· Added 27 Oct 2026, published 26 Oct 2026

en.cryptonomist.charticle

Test-time training in robotics shows no clear gains in demo response

This article reports a study submitted October 23, 2026 by Qiangong Zhou, Jiafeng Liang, Yimin Dong, Yang He and Hongxiang Li. They added Test-Time Training (fast-weight memory) to the vision-language-action models InternVLA-A1 and Pi0.5, trained on paired human demonstrations and robot trajectories. Shorter videos, full-trajectory gradient connectivity and contrastive regularization gave no clear gain in demonstration-guided target selection at the tested scale.

vision-language-action modelTest-Time Training
InternVLA-A1Pi0.5Qiangong Zhou+4

research, multimodal, training· Published and added 26 Oct 2026

en.cryptonomist.charticle

A new AI method for saliency-aware image-text alignment improves image search

RoSA is a finetuning framework from Qiuyu Kong and three co-authors for long-text image-text retrieval. It targets "saliency collapse", where models over-rely on text tied to prominent visual features and miss contextual information. RoSA splits pairs into salient and contextual views using object-level saliency estimation and LLM text decomposition, then trains on both with plug-and-play auxiliary alignment objectives. Code and checkpoints are public.

LLM
RoSAQiuyu Kong

multimodal, research, training· Published and added 26 Oct 2026

lynchlab.netwriteup

A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis

A dermatologist tested GPT-6 Astra and Claude Fable 5.1 on 200 biopsy-proven lesions from MILK10k, giving each a clinical and a dermoscopic image. A CNN (Inception v3, fine-tuned for about twenty minutes on 68,011 ISIC images) named the right diagnosis for 62% of lesions, against 42% for both LLMs. The post lists caveats (small sample, single run, possible contamination) and links the cases, prompts, replies and scoring code.

CNNGPT-6 AstraClaude Fable 5.1+4
MILK10kInternational Skin Imaging CollaborationISIC Archive+2

evals, multimodal· Added 26 Oct 2026, published 25 Oct 2026

arxiv.orgpaper

Explaining Navigation Decisions: Stage-Aware Visual Attribution for Vision-Language Navigation

A stage-aware visual attribution framework for Vision-Language Navigation (VLN) that explains a robot's action choices. It identifies "instruction-stage insensitivity", where existing explainers give similar outputs across navigation stages. The method builds action-conditioned temporal instruction-visual attributions, aggregates relevant historical evidence by stage, and applies an instruction-guided spatial filter. It improves attribution quality across several VLN policies and benchmarks, helps diagnose failure cases, and was tested on a Unitree G1 robot.

vision-language model
Unitree G1

research, multimodal· Added 27 Oct 2026, published 25 Oct 2026

arxiv.orgpaper

Detecting and Understanding Ambiguity in Visualizations with Role-Playing Vision Language Model Interpretations

This paper presents a pipeline that uses persona-conditioned vision language models to surface interpretive ambiguity in visualizations. Topic-relevant personas elicit candidate takeaways, and a Target-Pattern-Scope framework lets LLM judges label statement pairs as Compatible, Disjoint or Opposing, which are summarized in an ambiguity profile. The authors benchmark VLMs on chart comprehension per chart type, check labeling against human annotations, and run case studies on public visualizations and a redesign.

vision-language modelLLM
arXiv

research, multimodal, evals· Added 27 Oct 2026, published 25 Oct 2026

arxiv.orgpaper

InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants

InteractiveWorld is an environment for closed-loop interaction between an AI assistant and a simulated human agent who sees the scene, follows instructions and acts independently. InteractiveBench, built on it, has 120 multi-turn tasks in six categories scored by verifiable, state-based rewards. Across 12 frontier MLLMs, the best model, Claude Fable 5.1, reaches at most 20% on long-horizon Guidance and 25% on Planning tasks.

multimodal large language model
InteractiveWorldInteractiveBenchClaude Fable 5.1+1

evals, multimodal, research· Added 27 Oct 2026, published 25 Oct 2026

coindesk.comarticle

Block's new voice AI sounds eerily human, sparking awe and scam fears

Dessa is a voice AI model from Block, trained from scratch on human speech only, with no text stage. It listens while speaking, so it handles interruptions, laughter and tone changes. Block says it trained on one Nvidia GB200 NVL72 system and that it runs locally on a MacBook Pro with M4 Max or better. Model size, benchmarks and release date are undisclosed. Early access goes to a small group of AI safety researchers. Reactions split between praise and scam fears.

Nvidia GB200 NVL72MacBook ProApple M4 Max
BlockDessaJack Dorsey+7

models, multimodal· Published and added 24 Oct 2026

arxiv.orgpaper

OWLS: Open-world Language-conditioned Speed Fields for Zero-Shot Rover Navigation

OWLS (Open-world Language-conditioned Speed Fields) plans directly over a vision-language model's dense feature map. A text prompt describing traversable terrain scores every pixel of one camera image. The scores are projected onto the ground, adapted with two morphological operations, and used as the speed field of a minimum-time planner. It needs no training, terrain labels, map or extra sensors. It was tested on a physical rover, on simulated Mars and Moon rovers over 200 procedural terrains, and on real planetary imagery.

vision-language model
OWLSarXiv

research, multimodal· Added 26 Oct 2026, published 23 Oct 2026

arxiv.orgpaper

Standing on Open Shoulders: Regional Fine-Tuning of Open-Weight Camera-Trap Models for UK Wildlife with Modest Data and Compute

DeepFaune-UK is a fine-tune of the open DeepFaune camera-trap classifier on 28 UK classes, paired with MegaDetector v6. On 20,212 MammalWeb images it reached 88.8% strict accuracy against 56.5% (DeepFaune v1.5.0) and 59.0% (SpeciesNet). Re-applying the recipe to the DINOv3-based classifier gave 90.6%. Fine-tuning on about 150 crops per class matched full-data out-of-distribution accuracy, and a DINOv2 run took about 22 minutes on under 4 GB of GPU memory.

DeepFauneMegaDetectorSpeciesNet+2
DeepFaune-UKMammalWeb

research, training, multimodal· Added 26 Oct 2026, published 23 Oct 2026

arxiv.orgpaper

UAV-SwarmAgent: Scalable Multi-UAV Reconnaissance and Task Allocation via Code Generation and Vision-Language Reasoning

UAV-SwarmAgent is a hierarchical multi-agent system where an operator gives one natural-language sentence and a ground-station agent writes and runs code that produces per-UAV routes. A vision-language model grounds targets in an aerial map, and worker agents verify incidents with a compact onboard model. In simulation it located the incident in 244 of 255 missions (95.7%), and flight time followed an Amdahl-type law with speedup bounded at 1.52x for one to five UAVs.

vision-language modelLLM
UAV-SwarmAgentarXiv

research, agents, multimodal· Added 26 Oct 2026, published 23 Oct 2026

arxiv.orgpaper

Efficient Zero-Shot Video Anomaly Detection via Signal-Aware Adaptive Fusion (SAAF)

Signal-Aware Adaptive Fusion (SAAF) is a training-free zero-shot video anomaly detection framework that cuts the compute cost of vision-language models. It combines a dual-signal sampler (frame-to-frame delta and rolling-context drift in embedding space, OR-gated with EMA thresholds), a threshold-gated cascade from frame-level to video-level analysis, and signal-aware score fusion. It reports AUC-ROC of 92.07% on XD-Violence, 86.70% on MSAD and 80.54% on ShanghaiTech, at 2.2x to 2.6x the throughput of the prior best zero-shot method.

vision-language model
SAAFXD-ViolenceMSAD+3

research, multimodal· Added 26 Oct 2026, published 23 Oct 2026

arxiv.orgpaper

When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning

MART-GRPO is a variant of group-relative policy optimization for vision-language-action (VLA) reinforcement learning. It uses a LoRA-fine-tuned VLM to find the last visible error without recovery in a failed rollout video, keeps failure feedback up to that point, and drops it afterward. Martingale stopping conditions show this preserves the expected GRPO gradient and can reduce covariance. With OpenVLA-OFT on RoboTwin 2.0, it raises success rates over full-trajectory GRPO across multiple tasks.

GRPOLoRAvision-language-action model+2
RoboTwin 2.0arXiv

research, training, multimodal· Added 26 Oct 2026, published 23 Oct 2026

arxiv.orgpaper

4DMemory: Benchmarking Long-Horizon Spatiotemporal Memory with Harness-Augmented Agents

4DMemory is a benchmark suite for long-horizon spatiotemporal memory: the LongVideo dataset (about 900 hours of video, over 5,000 manually reviewed questions) and 4DMemoryBench. It also presents the 4DMemory Harness, combining four skills, 22+ tool models and a model-specific RAG system. Across 13 models, the harness improved every one by over 50%, and the same models scored over 10% lower than on a previous benchmark.

RAG
4DMemory4DMemoryBenchLongVideo+2

evals, multimodal, agents· Added 26 Oct 2026, published 23 Oct 2026

arxiv.orgpaper

MAST-VLA: Skill-Conditioned Adaptation with Trajectory-Affordance Priors for Multi-Skill Robot Manipulation

MAST-VLA is a vision-language-action framework for multi-skill robot manipulation that conditions on both the skill to execute and where and how to contact objects. It adds skill-conditioned LoRA Mixture-of-Experts adapters to the LLM backbone and turns a predicted affordance heatmap into a patch-level spatial prior for the action head. It reaches an Avg. Len of 4.65 on CALVIN ABC→D and was tested on a real robot.

vision-language-action modelLoRAMixture of Experts+1
MAST-VLACALVINarXiv

research, multimodal· Added 26 Oct 2026, published 23 Oct 2026

arxiv.orgpaper

Sensorimotor Alignment: A Platonic Proxy for Embodied Policy Success

Sensorimotor Alignment (S^2) is a training-free offline metric that measures how well an embodied model's sensory representation aligns with the geometry of physical motion. It extends the Platonic Representation Hypothesis to motion trajectories. S^2 correlates positively with downstream robot task success across embodied model families, helps pick promising VLM backbones before fine-tuning into VLAs, and explains behaviours seen in VLAs and world action models.

vision-language modelvision-language-action modelworld action model
Sensorimotor AlignmentPlatonic Representation HypothesisarXiv

research, evals, multimodal· Added 24 Oct 2026, published 22 Oct 2026

arxiv.orgpaper

F2G-Pose: Geometry-Aware Foundation Feature Lifting for Direct RGB-D Category-Level Object Pose Estimation

F2G-Pose is a direct, single-pass framework that predicts category-level object pose, metric size and camera-frame completed shape from one RGB-D image. It lifts frozen RADIO foundation features onto partial point clouds, tokenizes local geometry with DGCNN, and fuses it with a Mixture-of-Experts Transformer. It trains only on synthetic SOPE data, leads on SOPE and ROPE at 28 FPS, and is tested on HouseCat6D, HANDAL and real manipulation.

TransformerMixture of ExpertsDGCNN+2
F2G-PoseSOPEROPE+4

research, multimodal· Added 24 Oct 2026, published 22 Oct 2026

arxiv.orgpaper

Progress Probes $P^2$: Lightweight Runtime Monitoring for Vision-Language-Action Models

Progress Probes (P^2) trains linear probes on a vision-language-action model's own activations, using demonstrations, to estimate task progress, then flags executions that deviate from expected progress. Tested on the open-source pi0.5 across simulated and real-world manipulation tasks, it matches or beats VLM-based monitors and failure-data-trained methods at failure detection, without an external foundation model.

vision-language-action modelvision-language modellinear probes
pi0.5arXiv

research, multimodal· Added 24 Oct 2026, published 22 Oct 2026

arxiv.orgpaper

Rethinking Sequence Compression with Test-time Regression

TTR-Compressor is a sequence compression method that treats input tokens as a test-time regression problem. The regression is learned end-to-end by backpropagation, and its coefficients are solved in closed form in the forward pass to give the compressed representation. As a block-wise sparse attention compressor it improves RULER over attention-based compressors by a large margin. As a visual token merger for multimodal LLMs it beats the MLP merger and matches it with 55.6% fewer visual tokens.

TransformerLLMvision-language model
TTR-CompressorRULERarXiv

research, multimodal· Added 24 Oct 2026, published 22 Oct 2026

arxiv.orgpaper

CellTrack-Bench: Microscopic Cell Tracking Benchmark for Vision-Language Models

CellTrack-Bench is a benchmark that tests vision-language models on microscopic cell tracking and compares them with specialised trackers. It has 1,537 problems covering 9,041 cell-linking decisions, drawn from 15 cell-tracking datasets across four imaging modalities, with a hard subset built from failures shared by distinct algorithmic trackers. The authors compare environment harnesses, benchmark 9 closed-source and 13 open-weight models, and analyse failure modes.

vision-language modelLLM
CellTrack-BencharXiv

evals, multimodal, research· Added 24 Oct 2026, published 22 Oct 2026

arxiv.orgpaper

DuoCook: An Overcooked-Inspired Benchmark for Visual Planning and Coordination

DuoCook is an Overcooked-inspired benchmark in which one vision-language model controls two cooks from top-down views in Unreal Engine environments built with coding agents. It has 20 maps in five categories covering equipment contention, changing terrain and handoff dependencies. Four VLMs were compared; the best, gpt-6-astra, reached 26.2% mean order completion. A notebook memory ablation had mixed effects: positive for Claude and Qwen, negative for GPT and Gemini.

Unreal Enginevision-language model
DuoCookGPT-6 AstraClaude+4

evals, agents, multimodal· Added 24 Oct 2026, published 22 Oct 2026

huggingface.coarticle

OlmoEarth v1.3: State-of-the-art embeddings for the whole planet

OlmoEarth v1.3 is an open satellite embedding model from Ai2 that reads a year of Sentinel-1, Sentinel-2 and Landsat data and outputs a 128-dimensional vector per 10 m pixel (the first 64 dimensions also work alone). Its embeddings beat AlphaEarth Foundations and Tessera v2 on 20 of 27 evaluations. Precomputed global embeddings, a browser Embeddings Explorer and pre-training cost (about $2,720 of H100 time) are covered.

TransformerHugging FaceGeoTIFF
OlmoEarthAi2AlphaEarth Foundations+8

models, multimodal, evals· Published and added 22 Oct 2026

arxiv.orgpaper

Drift Says When, Budget Says How Much: Rate-Controlled KV Re-selection for Long-Form Multimodal Generation

Rate-Controlled Refresh (RCR) is a KV cache re-selection method for multimodal LLMs in long-form generation. Attention drift decides when and where to re-select visual tokens, and a per-sample controller sets how often. RCR beats InfiniGen by 1.6-3.8 ROUGE-L points on five of six model-task settings and stays within 1.7 points of exact per-step re-selection. On video description, where drift is homogeneous, it no longer beats a fixed threshold.

Qwen2-VLQwen3-VLInfiniGen+1
DetailCaps

research, inference, multimodal· Added 24 Oct 2026, published 22 Oct 2026

deepmind.googlearticle

Beyond tree cover: mapping the world’s forest types

Forest Typology (ForTy) 2020 is a global 10-meter map from Google DeepMind, Google Research, WRI and IIASA. It sorts land into six FAO-based classes: primary, naturally regenerating, planted, plantation, tree crops and agroforestry, and other land. The model uses AlphaEarth Foundations embeddings and about 1.7 million labeled locations. It was 90% accurate on forest type across 9,000+ expert-checked sites. An early 2017 to 2025 time series is on Earth Engine.

Google Earth EngineAlphaEarth Foundations
Google DeepMindGoogle ResearchWorld Resources Institute+7

models, multimodal· Published and added 22 Oct 2026

github.comrepo

Show HN: OnKey – I'm a bad singer, so I built an app to show where I'm off-key

OnKey is a free, MIT-licensed singing trainer for Apple Silicon Macs. After a song ends in Spotify, Apple Music or browser YouTube, it separates the lead vocal with UVR's MDX-Net Karaoke 2, tracks pitch with CREPE and turns it into syllable-level notes with the GAME model. On later plays it scrolls the notes, draws the user's live pitch and scores each line. Analysis runs on-device, and the repo explains the build steps and network requests.

macOSRustNode.js+2
OnKeyCREPEGAME+9

multimodal, inference· Added 24 Oct 2026, published 22 Oct 2026

arxiv.orgpaper

TRIAD: Benchmarking Omni-Modal Ambiguity in Multimodal Large Language Models

TRIAD is a benchmark for omni-modal ambiguity resolution. It has 327 hand-written text-image-audio queries in 86 scene groups across 18 ambiguity categories, and removing any one modality makes the answer non-unique. Across 14 omni-modal systems, the best model trails humans by over 20 points in group accuracy. Without answer options, model group accuracy falls to single digits, and leave-one-out tests show audio cues are often underused.

multimodal large language modelomni-modal LLM
TRIADarXiv

evals, multimodal, research· Added 23 Oct 2026, published 22 Oct 2026

huggingface.copaper

Matrix-Game 4.0: Harnessing Video Models for Real-Time Playable Game Worlds

Matrix-Game 4.0 is a system that separates game logic from visual rendering for playable generated worlds. A lightweight engine holds world state and rules, a multimodal coding agent writes game code and scene captions, and a distilled autoregressive video model turns low-poly proxies into detailed frames. It runs 480p at 60 FPS on one RTX 5090 with 112 ms key-press-to-display latency. Code, weights and assets are released.

diffusion modelHugging Face
Matrix-Game 4.0NVIDIA

multimodal, research, agents· Added 22 Oct 2026

x.comthread

SpyTech Ai (@RizeOnline) on X

XiaoZhi ESP32 is an open-source, MIT-licensed firmware stack that turns ESP32 boards into MCP-based voice chatbots using models such as Qwen and DeepSeek. It streams Opus audio over WebSocket or MQTT + UDP through ASR, LLM and TTS, adds offline ESP-SR wake words, and uses device-side MCP to drive speakers, LEDs, servos and GPIO. It supports 138 board directories and builds with ESP-IDF 6.0.1 or newer.

Model Context ProtocolESP32ESP-IDF+10
XiaoZhi ESP32SpyTech Ai

mcp, multimodal, llm-apps· Added 23 Oct 2026, published 21 Oct 2026

huggingface.coarticle

Granite Speech 5.0: Transcribe, Translate, and Attribute more Accurately and Faster

Granite Speech 5.0 is IBM's update to its open speech model collection: the new granite-speech-5.0-2b, a multilingual granite-speech-5.0-470m-turboctc, and a revised granite-speech-4.1-2b-plus. The 2B model ranks #2 among open-weight models on average English WER (5.06%) on the Open ASR Leaderboard. Turbo CTC reaches 19,870 RTFx, and the 4.1 Plus revision cuts long-form WER 24%. All are Apache 2.0.

TransformersvLLMllama.cpp+5
IBMGranite Speechgranite-speech-5.0-2b+5

models, multimodal, evals· Published and added 21 Oct 2026

x.comthread

Vishesh Raheja (@entropyconquers) on X

GPT Voice is a free, open-source Mac app by Vishesh Raheja that brings ChatGPT's dictation to any macOS app. A shortcut starts recording, and the transcript is pasted where the user is typing. It uses the ChatGPT Pro subscription, so it needs no API key or extra subscription. The author built it after Handy handled coding prompts poorly and they did not want to pay for Wispr Flow. The thread gives no details on local versus API transcription or latency.

ChatGPTCodexmacOS
Vishesh RahejaT3 CodeGPT Voice+3

multimodal, coding-agents· Added 22 Oct 2026, published 21 Oct 2026

huggingface.coarticle

NVIDIA Cosmos3-H-Dreams: Bringing Clinical Surgery to Life Through Generative Simulation

Cosmos3-H-Dreams is NVIDIA's real-time generative surgical simulator, trained on clinical cholecystectomy video and kinematics from the CMR Surgical Versius platform. It builds on the 16B Cosmos3-H-Embodied-Nano, distills a bidirectional teacher into four-step and two-step causal students with Self-Forcing, and streams through FlashDreams on one RTX PRO 6000 GPU. Checkpoints and a recipe for custom data are released.

FlashDreamsFadeMemHugging Face+1
NVIDIACosmos3-H-DreamsCosmos-H-Dreams+6

models, multimodal, training· Published and added 20 Oct 2026

x.comthread

brenz. (@brenzhills) on X

Krea 3 is an image generation and editing model from Krea. A post on X says its "Fast" variant scores 1125 Elo on the Artificial Analysis image editing leaderboard, against 1124 for Google's Nano Banana 2, while producing 10 images in the time Google produces 2. It also says the "Ultra" variant ranks in the top 4, behind only OpenAI models. The post gives no methodology or model details.

Krea 3Nano Banana 2
KreaGoogleOpenAI+1

models, multimodal, evals· Added 21 Oct 2026, published 20 Oct 2026

arxiv.orgpaper

UniWALM: Unified World Action Language Model for Embodied Robot Control

UniWALM is a unified world-action-language model that learns video, action and language in one mixture-of-transformers backbone with per-modality diffusion heads that co-denoise together. Generated subtask and textual memory are fed back on the next step, so no external planner or memory bank is needed. It adds Flow Matching with Correction (FMC) for recovery from perturbed actions. It reaches 34.5% on the memory-dependent RMBench subset, above Mem-0's 28.5%, and lifts real-world Find-the-Item success from 11.1% to 61.1%.

Mixture of Expertsdiffusion modelflow matching+1
UniWALMFastWAMMolmoSpaces+2

research, multimodal· Added 22 Oct 2026, published 20 Oct 2026

arxiv.orgpaper

InterWebCoder: Learning Interactive Webpage Generation from Executable Tests via Reinforcement Learning

InterWebCoder is a framework for interactive webpage code generation by multimodal LLMs. It synthesizes self-contained webpages with executable test scripts from recorded interaction trajectories, repairs failures using runtime evidence, then trains with test-ID-weighted SFT and GRPO using test pass rates and visual similarity as rewards. The pipeline yields 1,736 verified samples from 2,058 tasks. A 9B model scores 48.54 on IWR-Bench and 48.42 on Vision2Web, and reward latency drops over 97%.

GRPOvision-language modelLLM
InterWebCoderIWR-BenchVision2Web+3

research, training, multimodal· Added 22 Oct 2026, published 20 Oct 2026

arxiv.orgpaper

Agentic Ensemble with Vision-Language Models for Visual Anomaly Detection

AnomalyAgent is a post-training fusion framework for visual anomaly detection that ensembles pretrained detectors. It aligns scores and anomaly maps, summarizes disagreement, and treats fusion as a choice among score actions and map actions. A constrained vision-language model inspects high-conflict samples and picks the actions. It beats equal-weight fusion and gets the best mean image AUROC and pixel AP among compared fusion methods on MVTec AD and VisA.

vision-language model
AnomalyAgentMVTec ADVisA+2

research, multimodal, agents· Added 22 Oct 2026, published 20 Oct 2026

arxiv.orgpaper

Code2Video-Compass: A Comprehensive Benchmark for General-Purpose Code-Based Video Generation across T2V, I2V, and V2V

Code2Video-Compass is a benchmark of 2,104 tasks for code-based video generation, where multimodal LLMs write executable programs that produce complete videos from text (T2V), image (I2V) or video (V2V) inputs. Tasks span 10 categories and 24 subtypes. Scoring uses 12 objective metrics and five MLLM-judge metrics. Across 15 proprietary and open-weight models from nine families, performance falls as construction, dynamic animation and state-dependency complexity rise.

vision-language modelLLM
Code2Video-CompassarXiv

evals, multimodal, research· Added 22 Oct 2026, published 20 Oct 2026

arxiv.orgpaper

DIVE: Benchmarking Full-Duplex Interaction with Visual Engagement in Continuous Streams

DIVE is a benchmark for full-duplex interaction with visual engagement in continuous audio/video streams. It has 525 human-annotated samples across six tasks in three scenarios: proactive intervention, continuous engagement and interaction coordination. Evaluated real-time models reach sub-second average latency, but window response rates stay at or below 53.77%, and adapting ongoing responses to new information is hardest.

MLLM
DIVEarXiv

evals, multimodal, research· Added 22 Oct 2026, published 20 Oct 2026

jevcards.comtool

Show HN: Jev Cards, Can you beat a vision model at a picture quiz?

Jev Cards is a browser picture quiz where a player competes against Jev 1.14, a vision decision model. The 490 cards cover fruit, flags, maps, chess and illusions. Jev reads the image and question and returns a probability for every option, or a box for "tap the place" cards, in one call of about 0.25 seconds. It never writes text. The page reports 5,256 wins in 6,820 games.

Jev

multimodal, evals· Added 21 Oct 2026, published 20 Oct 2026

arxiv.orgpaper

A Multimodal Engineering GraphRAG Framework for Post-Wind-Event Building Assessment and Recovery Inference

This paper presents a multimodal engineering GraphRAG framework that links post-wind-event damage evidence from images and reconnaissance reports to building-condition states, recovery actions and downtime. Evidence is stored as a provenance-preserving, component-level representation, and graph-aligned retrieval uses compatibility, contradiction, evidence-precedence and abstention checks. Demonstrated on tornado-affected low-rise homes, it beats BaselineRAG and Structured Baseline RAG, and ablations test each component.

RAGGraphRAG
BaselineRAGStructured Baseline RAGarXiv

research, llm-apps, multimodal· Added 22 Oct 2026, published 20 Oct 2026

arxiv.orgpaper

OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory

OpenVLHarness is a multimodal reasoning harness for MLLMs built on a capability layer (task-level tool interfaces plus evidence rendering) and persistent multimodal memory that stores images, structured data and web references as addressable artifacts. A coding interface runs generated programs over stored data. Across 22 benchmarks and eight backbones it adds 9.7-12.6 points over no-tool baselines, and GPT-6 Luna at medium effort beats Codex and Visual Sketchpad at maximum effort using at most 37% of their tokens.

vision-language modelQwen3-VLCodex+1
OpenVLHarnessKimi K3GPT-6 Astra+3

agents, multimodal, research· Added 21 Oct 2026, published 19 Oct 2026

arxiv.orgpaper

Where is Physical RSI capability from? An Observational Scaling Law of LLM-as-policy

The paper introduces PhysRSI-eval, a five-axis metric for physical recursive self-improvement: first-attempt performance, efficiency, success, budget and improvement from failure. Several VLMs are tested on 18 manipulation tasks under a fixed interaction budget. It proposes the Physical Barrel Hypothesis and studies observational scaling. Workspace interventions show that removing history lowers late-round success, while frozen workspaces still help on unseen layouts.

vision-language modelLLM
PhysRSI-evalPhysical Barrel HypothesisarXiv

research, evals, multimodal· Added 21 Oct 2026, published 19 Oct 2026

arxiv.orgpaper

What Language Adds, and at What Cost: Four Controlled Probes of Pathology Vision-Language Models

This preprint grades evidence for pathology vision-language models and tests four claims with controlled probes on released checkpoints and public data. Stage-redacted report embeddings raise pooled C-index by 0.019 (progression-free interval) and 0.016 (overall survival) across 1,633 TCGA patients. Zero-shot prompting matches a linear probe. SlideChat scores 74.82% with the slide and 74.41% without it. Paraphrasing flips 8-23% of answers.

vision-language model
SlideChatSlideBench-VQATCGA

research, evals, multimodal· Added 21 Oct 2026, published 19 Oct 2026

arxiv.orgpaper

Scaling Parameter-Efficient Fine-Tuning for Brain Encoding

BrainLoRA is a parameter-efficient fine-tuning method that adapts multimodal backbones of up to 104B parameters to predict neural responses. It predicts multiple timepoints in one causal pass instead of sliding windows, and a multi-LoRA kernel trains many adapters on a shared backbone. Across nine backbones it improves encoding, and a 200+ model ensemble ranks first on the Algonauts post-challenge leaderboard.

LoRAQwen3Transformer
BrainLoRAQwen3-OmniAlgonauts 2025 Challenge+1

research, training, multimodal· Added 21 Oct 2026, published 19 Oct 2026

arxiv.orgpaper

Mise-en-Video: Fine-Grained 3D Control for Multi-Shot Video Generation

Mise-en-Video (MeV) is a multi-shot video generation framework where creators stage shots in 3D space and then generate the matching video. It combines hierarchical attention for multi-level narrative consistency with 3D-aware layout guidance for per-shot spatial control. The authors built a 160K-video dataset with hierarchical multi-shot texts and oriented 3D bounding boxes, and report gains over baselines in quality, consistency and subject controllability.

diffusion model
Mise-en-VideoMeVarXiv

research, multimodal· Added 21 Oct 2026, published 19 Oct 2026

arxiv.orgpaper

HAF-VLA: Hierarchical Action Flow Framework for Humanoid Vision-Language-Action Control

HAF-VLA is a hierarchical action-flow framework that adapts a pretrained flow-matching vision-language-action model to humanoid whole-body loco-manipulation. It denoises a shared action space in stages: locomotion and head commands first, then waist refinement, then full-body manipulation. Learnable stage embeddings and cross-stage KV-cache conditioning carry dependencies between stages. The authors report gains over existing models on six real-world tasks.

flow matchingvision-language model
HAF-VLAarXiv

research, multimodal· Added 21 Oct 2026, published 19 Oct 2026

huggingface.corepo

Meta Muse Spark Open (816B-A42B) open-weights model card

Model card for Meta Superintelligence Labs' Muse Spark Open, an open-weights, natively multimodal model with 816B total and 42B active parameters and a 1M-token context window. It targets coding and agentic work (tool use, long-horizon tasks, multimodal reasoning, synthetic data generation, LLM-as-judge). It ships in BF16 and NVFP4, needs about 700GB VRAM (NVFP4) or 2TB (BF16), runs on SGLang, vLLM, Transformers, TensorRT-LLM, Unsloth, llama.cpp and MLX, and supports a reasoning_effort setting. The Muse Model License requires "Built with Muse Spark Open" attribution and a separate agreement for Model-as-a-Service above $20M revenue. The card reports safety results against Kimi K3 on cyber, chem/bio and loss-of-control evals.

Hugging FaceTransformersvLLM+8
Muse Spark OpenMetaMeta Superintelligence Labs+15

models, multimodal, inference· Added 17 Oct 2026

arxiv.orgpaper

FedRAPP: Federated Test-Time Prompt Tuning for Vision-Language Models with Reliability-Aware Prompt Personalization

FedRAPP is a personalized aggregation framework for Source-Free Federated Test-Time Prompt Tuning (SF-FTTPT), where clients adapt CLIP soft prompts on unlabeled, non-IID test streams without source data. It combines reliability-aware client assessment, a server-side exponential moving-average prompt memory and similarity-guided peer aggregation. Across nine CLIP benchmark datasets and ImageNet variants it beats standard federated aggregation baselines while keeping prompt-only efficiency.

CLIPvision-language model
FedRAPPImageNetarXiv

research, multimodal· Added 20 Oct 2026, published 16 Oct 2026

arxiv.orgpaper

FlexPart: Text-Controlled Semantics-Aware 3D Part Segmentation at Flexible Granularity

FlexPart is a framework for text-controlled 3D mesh segmentation that assigns discrete part IDs to mesh faces through a view-independent part-identity field. It uses a control prompt of part names with location and shape clauses, trained on a VLM-built multi-granularity dataset. On FlexPartBench it raises average mIoU by 22% over the strongest baseline, and by 15% with planner-completed prompts.

vision-language model
FlexPartFlexPartBench

research, multimodal· Added 20 Oct 2026, published 16 Oct 2026

arxiv.orgpaper

A Language-Informed Loss for Visual Generation

LILIE (Language-Informed and Lossless Image Encoder) is an invertible image transform built from a normalizing flow transformer, trained jointly with a vision-language model via next token prediction so captions can be recovered from noised features. The encoder supplies a language-informed loss for text-to-image generation. JiT models trained with it cut gFD from 66.84 to 53.01 against a pixel-only baseline on the GPIC dataset. Code, encoders and models are released.

normalizing flowTransformervision-language model
LILIEJiTGPIC+1

research, multimodal· Added 20 Oct 2026, published 16 Oct 2026

arxiv.orgpaper

Enhancing RAG-based Medical VQA with Gated Multimodal Representations

This paper presents a multimodal retrieval-augmented generation framework for Medical Visual Question Answering. A Gated Multimodal Unit learns a joint query representation from the medical image and the clinical question, so evidence retrieval uses both modalities while the Med-LVLM stays frozen. On benchmark datasets it consistently beats a reference Med-LVLM and stays competitive with state-of-the-art methods. The page gives no dataset names or numbers.

RAGvision-language model
arXiv

research, llm-apps, multimodal· Added 20 Oct 2026, published 16 Oct 2026

x.comthread

AmingIn_AI (@aming_inAI) on X

A tweet from AmingIn_AI describes a macOS screenshot-annotation tool. On release it saves the screenshot, a crop and the user's marks locally, then copies a 127.0.0.1 link. Claude Code or Codex on the same Mac can fetch that link and receive the marked image plus app/window info, and nothing is uploaded. It shows a local, privacy-preserving way to hand visual context to coding agents without MCP or cloud upload.

Claude CodeCodexmacOS
AmingIn_AI

coding-agents, multimodal· Added 17 Oct 2026, published 16 Oct 2026

arxiv.orgpaper

MUB: An Identity-Masked Benchmark for Music Understanding in Multimodal Language Models

MUB (Music Understanding Benchmark) is a set of 500 identity-masked recordings with existing annotations for tempo, key, time signature and structural intervals. It covers all 24 keys, tempos of 47 to 180 BPM and 4,676 intervals, and strips titles, artists and platform IDs from inputs. Baselines reach 53.6% tempo accuracy (DSP) and 98.4% time-signature accuracy (constant 4/4). No multimodal LLM has been tested yet, and rights clearance is unresolved.

multimodal language model
Music Understanding BenchmarkMUBarXiv

evals, multimodal, research· Added 20 Oct 2026, published 16 Oct 2026

github.comrepo

OpenMOSS/MOSS-Image

MOSS-Image is an open-source text-to-image model family from OpenMOSS with a 5B backbone, released October 2026 as MOSS-Image-Base and MOSS-Image-SFT. It supports JSON layout control with bounding boxes, text rendering inside images, and prompts up to 4K tokens. The repo gives a Diffusers pipeline, a Gradio app, a ComfyUI plugin, LoRA training guidance, a prompt-enhancer skill and day-0 SGLang and vLLM-Omni support.

DiffusersPyTorchGradio+7
OpenMOSSMOSS-ImageMOSS-Image-Base+5

models, multimodal, inference· Added 20 Oct 2026, published 16 Oct 2026

huggingface.coarticle

Granite Switch 4.1: Chat, Decisions, and Speech from One Endpoint

IBM announces Granite-Switch-4.1-8B and -30B. Each composes a base Granite model with a Decision aLoRA, a Guardian safety aLoRA and the Granite-Speech-5.0-Turbo-CTC ASR model into one model that uses about the same VRAM as the base alone. Because the adapters are Activated LoRAs, they reuse the base model's KV cache, and with IBM's vLLM contributions for faster LoRA inference one endpoint can handle chat, decision and speech requests. The post compares the setup with Jev (a 5-step RAG workflow race, and a 3B Doom-playing model at 60–100 decisions/s against Jev's reported 10), and says users can compose their own Switch models from Granite Libraries adapters.

vLLMLoRAActivated LoRA+4
IBMGranite Switch 4.1Granite-Switch-4.1-8B+9

models, inference, multimodal· Published and added 15 Oct 2026

x.comthread

Vaibhav Sisinty (@VaibhavSisinty) on X

Social post reporting Meta's release of Muse Spark Open, described as the most capable US open-weight model: 816B total and 42B active parameters, a 1M-token context, and text, image, video and audio input in one model. It is pitched at coding and agentic work and tuned to run on two M5 Ultra Mac Studios, with weights available at release. The poster notes that in Meta's own chart, China's Kimi K3 and GLM 5.3 still beat it on coding.

MetaMuseMuse Spark Open+3

models, multimodal· Added 16 Oct 2026, published 15 Oct 2026

x.comthread

Connor (@ConnorTalksAI) on X

A tweet relaying OpenAI's announcement of GPT-Live-1.1, an update to the voice model behind ChatGPT voice mode. It reduces how often the model says "checking", adds new voices and improves accents, including new voices in French, German, Hindi, Mexican Spanish, Korean and Japanese. It is rolling out to all users in ChatGPT, Codex and "dots", with API access coming soon. The poster frames it as a response to a new xAI/SpaceXSI Grok Voice release.

ChatGPTCodexGrok+1
GPT-Live-1.1OpenAISpaceXSI

models, multimodal· Added 16 Oct 2026, published 15 Oct 2026

arxiv.orgpaper

Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation

P^3-WAM is a unified world action model for egocentric dexterous manipulation with three modes: Perceive (reconstruct articulated hand motion and camera trajectories from video), Predict (generate visual futures under prescribed actions) and Plan (jointly generate action trajectories and their anticipated visual outcomes from an initial image, a language instruction and an optional visual goal). It adds a compact structured 3D action representation to a pretrained video diffusion transformer through a mixture-of-transformers design with shared base weights, modality-specific adaptation and bidirectional visual-action attention. Randomized modality visibility lets video and 3D actions act as either context or target. Unified training beats single-mode training and outperforms specialized baselines on 3D hand reconstruction and action-conditioned video prediction across four datasets, and gives better instruction adherence than video-generation baselines.

diffusion modelTransformervision-language model
P^3-WAMarXiv

research, multimodal· Added 17 Oct 2026, published 15 Oct 2026

arxiv.orgpaper

Grounding Generative Verification in 3D Content

The paper introduces 3DVerifierBench, a 10-task benchmark for generative verification of 3D content. It scores accept/reject decisions, rejection explanations, and responses to controlled changes in specifications and candidates. A human study shows a 21.6-point gap to the strongest tested model, GPT-6 Astra. The authors also build a pipeline of multi-view fact extraction, controlled 3D editing and native annotations. It yields 14,126 training examples with contradiction explanations and linked counterfactuals. Verification-guided selection raises complete factual success from 26.7% to 90.0% with oracle selection over sixteen candidates, and human-guided recursive refinement satisfies all seven requirements for every input in an eight-input study.

multimodal large language model3D generation
3DVerifierBenchGPT-6 AstraarXiv

research, evals, multimodal· Added 17 Oct 2026, published 15 Oct 2026

arxiv.orgpaper

DK-VLM: Geometry as Position for Spatial Reasoning in Vision-Language Models

DK-VLM adds Dual-Kernel Attention to a vision-language model. A semantic kernel keeps the pretrained attention and positional encoding. A separate geometric kernel uses estimated 3D points and camera poses to modulate attention through point displacements and relative view transformations, and a learned gate fuses the two outputs. Geometry therefore enters attention as position, not as fused 3D reconstruction features. The paper also adds Reference-Frame Composition (RC) supervision, which gives stepwise turn and heading targets derived from verified single-step annotations, to train multi-step spatial reasoning as the observer moves. DK-VLM-8B reports the highest average score across six spatial benchmarks among the compared methods, with 74.33 on VSI-Bench and 58.28 on ReVSI-32, and it preserves general video understanding.

vision-language modelTransformer
DK-VLMVSI-BenchReVSI-32+1

research, multimodal· Added 17 Oct 2026, published 15 Oct 2026

arxiv.orgpaper

Velox: Spatial Memory Makes World Models Efficient

Velox is a training-free attention router for chunked video-diffusion world models. It uses camera frustum slabs with depth intervals from streaming geometry to pick which historical token blocks to attend to, instead of letting the KV cache grow with rollout length. Frustum overlap softly modulates each layer's model-derived block importance, and uncovered regions keep the model's own routing. The fixed block granularity suits FlashAttention-style tiled execution, and slab construction runs asynchronously with denoising on the same H200. At chunk 48 it speeds up attention 4.36× and denoising 2.43×, and on the 158-case WBench navigation split it stays within 0.15 average points of dense LingBot-Fast-Cam.

diffusion modelFlashAttentionCUDA
VeloxLingBot-Fast-CamWBench+1

research, multimodal, inference· Added 17 Oct 2026, published 15 Oct 2026

arxiv.orgpaper

Skill-Decoupled On-Policy Expert Distillation for Unified Vision-Language Models

Proposes SCOPED (Skill-Decoupled On-Policy Expert Distillation), which trains capability-specific RL experts for vision-language models. It extracts explicit skills from them through reward-guided reflection. During on-policy distillation the skills guide expert supervision but are hidden from the unified student, so the student internalizes the procedural knowledge and needs no explicit skills at inference. Across eight multimodal benchmarks it beats the backbone and mixed-capability RL. Analyses show cross-capability gradient interference in joint RL, and that skill augmentation gives more consistent expert trajectories, more attention to task-relevant visual evidence, and better preservation of expertise.

vision-language model
SCOPEDarXiv

research, training, multimodal· Added 17 Oct 2026, published 15 Oct 2026

arxiv.orgpaper

Onboard Lightweight LLM Reasoning for Human-Aware Safe UAV Navigation

Presents a modular, fully onboard UAV navigation pipeline that closes the "human-aware safety gap" of geometry-only planners. A shared BLIP encoder drives captioning and text-conditioned segmentation, which is fused with depth for 3D obstacle localization. Qwen-2.5-0.5B, stabilized with RAG and grammar-constrained JSON decoding, infers per-category risk once per class and caches the verdict. Perception runs at 10 Hz on a Jetson Orin NX, and fallback LLM queries take about 0.8 s off the control path. In Gazebo it gives roughly 3x fewer collisions than a geometric planner and raises semantics-aware landing success from 34% to 96%. The 224-pixel encoder input limits small-object perception at range.

QwenBLIPRAG+4
NVIDIAarXiv

research, multimodal, inference· Added 17 Oct 2026, published 15 Oct 2026

arxiv.orgpaper

From Perception to Prediction and Intervention: Benchmarking Proactive Video Assistants in Real-Life Scenarios

Introduces P^3IBench, a benchmark for proactive video understanding in real-life streaming scenarios, covering three capabilities: daily perception, risk prediction and intervention. It evaluates 12 open-source and proprietary video-language models (streaming and adapted non-streaming) and finds they struggle to make well-calibrated decisions about when to respond. The authors also build extra proactive training data and train P^3I-Assistant-4B, which improves several tasks unevenly, showing that proactive assistance needs response timing and restraint as well as visual understanding.

vision-language model
P^3IBenchP^3I-Assistant-4BarXiv

evals, multimodal, research· Added 17 Oct 2026, published 15 Oct 2026

huggingface.coarticle

Apertus 1.5 Release Notes

Release notes for Apertus 1.5, a fully open (weights, data, training details) continued pretraining of the 8B and 70B Apertus models on 4T and 2T additional tokens. It adds early-fusion image and audio understanding via discrete tokenizers (Emu3.5, WavTokenizer) without architectural change, a thinking mode, 262k context, and better tool use. It documents a four-stage pretraining and five-stage post-training pipeline (SFT, long-context SFT, RLVR, DPO, charter-alignment SDPO), plus evaluation via the MLLM-eval-suite and Every Eval Ever (EEE) Eval Cards, released checkpoints and datasets.

Hugging Facelm-evaluation-harnesslmms-eval+2
ApertusApertus 1.5Emu3.5+17

models, training, multimodal, evals· Published and added 15 Oct 2026

huggingface.coarticle

Welcome Muse Spark Open: open-weight frontier multimodal mixture-of-experts model from Meta

Hugging Face's launch post for Meta's Muse Spark Open, an open-weight 816B-parameter mixture-of-experts model (about 42B active per token). It has 256 experts with 8 routed per token, a 1M-token context, and perception and audio encoders for image, video and audio input. The post gives the architecture details, BF16 and NVFP4 checkpoints, and run recipes for transformers, llama.cpp, SGLang, vLLM, MLX and Inference Endpoints. It also covers LoRA SFT of the full model with TRL on 64 H100s and agentic demos (Pi 3D asset generation, an OpenEnv browser agent, a voice-driven shopping agent).

Mixture of ExpertsTransformersvLLM+10
MuseMetaMuse Glimmer+4

models, inference, multimodal· Published and added 15 Oct 2026

huggingface.coarticle

Welcome FLUX.3 Image Klein in Diffusers

Hugging Face blog post introducing Black Forest Labs' open FLUX.3 Image Klein (distilled and base checkpoints) in Diffusers. It covers text-to-image, multi-reference image-to-image and bounding-box layout control, FP8/NVFP4 single-file quantized weights, DreamBooth LoRA training scripts for generation and editing, memory/speed figures (about 2 s per 1024×1024 image on an H100), and a detailed architecture tour: Qwen3-VL text encoder, two-stage per-stream then joint transformer blocks, per-token timestep modulation, 4D RoPE, and flow-matching denoising. It also shows how to customise the Modular Diffusers pipeline.

DiffusersModular DiffusersTransformers+9
FLUX.3 Image KleinBlack Forest LabsFLUX.2+4

models, multimodal, training· Published and added 15 Oct 2026

deploymentsafety.openai.comarticle

GPT-Live-1.1 system card

System card for GPT-Live-1.1 and GPT-Live-1.1 Quip, OpenAI's full-duplex voice models that listen and respond continuously. It reports voice-native disallowed-content evals on production and synthetic audio prompts, and finds parity overall but statistically significant regressions for GPT-Live-1.1 on sexual, mental health, emotional reliance and gore categories. It also covers alignment evals converted to audio (respecting warnings, blocker deception, broken search tool) and Preparedness Framework conclusions: neither model is High in Biological and Chemical, Cybersecurity or AI Self-Improvement.

Codex
OpenAIGPT-Live-1.1GPT-Live-1.1 Quip+5

ai-safety, multimodal, evals· Published and added 15 Oct 2026

latent.spacetalk

Dyna-2: General Purpose Physical Agents with Hour-Long Autonomy — Jason Ma, Dyna Robotics

Podcast interview with Jason Ma, co-founder and Chief Scientist of Dyna Robotics, on Dyna-2, a world-action model pre-trained on over one million hours of human egocentric video. The model predicts future video and uses it to guide robot actions, built on a diffusion transformer with flow matching. The discussion covers the robotics data bottleneck, adapting to new tasks from about 45 minutes of demonstrations, synthetic environments and self-generated training data, on-device inference and distillation, and Dyna's custom wheeled semi-humanoid hardware. The episode listing also covers commercial deployments (Din Tai Fung, Best Western), agentic harnesses for hour-long workflows, and world-action models versus VLAs, but the fetched text was truncated before those sections.

diffusion transformerflow matchingdiffusion model+2
Dyna RoboticsDyna-2Dyna-2.1+13

multimodal, training, models· Published and added 13 Oct 2026

arxiv.orgpaper

ChunkAct: Action Chunking for Agentic Robotics

ChunkAct brings action chunking to agentic robotics, where an LLM agent composes perception, motion and VLA tools guided by memory. Online, one call launches a chain of primitives that runs autonomously and returns control only on failure. Offline, memory is compiled into these chunks so execution needs no memory at test time and the planner is used only for real decisions. On LIBERO-Pro it improves task success by 5.0 points over the prior agentic-robotics state of the art, with 2.9x fewer LLM calls, 4.4x fewer input tokens and 2.3x fewer output tokens.

LLMvision-language-action model
ChunkActLIBERO-Pro

research, agents, multimodal· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper

Dog-Breed Text-to-image Flow Matching from Pretraining to Fine-Tuning on 4 Consumer GPUs

MiniDog is a framework for training a text-to-image flow-matching diffusion transformer from scratch on four RTX 3090 GPUs, with 55 minutes of pretraining and 15 minutes of fine-tuning. It pretrains on 26k ImageNet dog images that were captioned, then fine-tunes on 2k FLUX.1-schnell generated images. Image and text features are precomputed and cached. The model is a 22M-parameter transformer with the E2E-INVAE semantic latent space, a small Qwen text encoder and in-context conditioning, and training is accelerated with iREPA and an EUPE perception encoder. The repo evaluates with FID, HPSv2, PickScore and VAE reconstruction metrics. It lowers the barrier to hands-on diffusion training research.

flow matchingdiffusion modelTransformer+1
MiniDogImageNetFLUX.1-schnell+7

research, training, multimodal· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper

Visual Tokens on Demand: Visual Budgeting for Scalable and Efficient Multi-Image Reasoning in Multimodal LLMs

Proposes mmVB, a lightweight plug-and-play module (1.6M trainable parameters) that assigns query-conditioned per-image visual token budgets using cross-image context for multimodal LLMs, without retraining the base model or accessing attention matrices (so it is compatible with FlashAttention). It cuts visual input tokens 9x, giving 5x faster time-to-first-token and 7x fewer prefill FLOPs while keeping 94.4% of full-resolution accuracy across eight multi-image reasoning benchmarks. It matters because existing token pruning, merging and compression methods target single-image redundancy and either need attention maps or retraining.

FlashAttentionvision-language model
mmVB

research, multimodal, inference· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper

PixelMoE: Scaling Pixel-Space Diffusion with LLM-Style Mixture-of-Experts

PixelMoE is a pixel-space diffusion framework that scales capacity sparsely using standard LLM-style Mixture-of-Experts: token-choice routing, fine-grained routed experts, a shared expert and bias-based load balancing, with no diffusion-specific routing or auxiliary routing losses. It adds Pixel-Space Hierarchical Flow Matching, which varies spatial resolution along the denoising trajectory from global structure to fine detail. At matched active parameters it beats dense counterparts, and reaches FID 1.47 at 256x256 and 1.53 at 512x512 on class-conditional ImageNet-1K without representation alignment or perceptual losses. Code and models are promised but not yet released.

Mixture of Expertsdiffusion modelflow matching+1
PixelMoEImageNet-1KarXiv

research, multimodal· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper

RepProbe: Can Video Language Models Count Repeated Actions? A Diagnostic Benchmark

RepProbe is a benchmark of 42K real and synthetic video clips with exactly 1–20 complete action repetitions, controlling for clip position, temporal coverage and duration. Video language models' counting accuracy collapses at high counts, and this is not explained by frame sampling alone. Linear probes on final-layer representations show count information stays linearly decodable even when the model's answer is wrong, so the failure arises at the output stage, where answers cluster on a few preferred numbers.

vision-language model
RepProbearXiv

evals, multimodal, research· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper

Plan-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?

Plan-Bench is a benchmark of 699 curated samples across 16 subclasses and three dimensions. It tests whether vision-language models can reason over spatial and temporal information and produce multi-step plans under constraints for embodied tasks, with action execution isolated out. Across 28 VLMs the best model reaches only 75.0% accuracy. Scaling generally helps, and larger models look more promising as high-level planners. Code and benchmark are released.

vision-language model
Plan-BencharXiv

evals, multimodal, research· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper

Game2Real-Bench: Do Toy Games Reveal and Transfer to Real-World Visual Reasoning?

Game2Real-Bench tests whether vision-language model performance on procedurally generated toy games predicts real-world visual reasoning, and whether game examples help transfer. It has 1,000 verifiable game samples and 400 real-world samples, matched by shared reasoning requirements (fine-grained perception, spatial reasoning, constraint reasoning, sequential planning). Across 25 VLMs, game and real scores correlate strongly (Pearson/Spearman 0.935). One-shot game-to-real transfer on four models improves constraint reasoning for several but hurts fine-grained perception. The authors also release Game2Real-Dataset, with about 24K verified game samples across 18 tasks and 65 QA families.

vision-language model
Game2Real-BenchGame2Real-DatasetarXiv

evals, multimodal, research· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper

OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding

OmniCompass is a manually annotated benchmark for video grounding and understanding with two tracks: OmniCompass-VL (820 videos, 2,460 instances; vision-language models) and OmniCompass-AV (240 videos, 720 instances; joint audio-visual evidence for omni-modal models). Each track covers temporal grounding, spatial grounding and QA across short/medium/long videos, and each instance is run in paired text-only and text+image target specifications. Evaluating 50 MLLMs shows open-source models lag proprietary ones, especially on audio-visual understanding, that precise temporal/spatial grounding remains hard, and that predictions are inconsistent across the paired specifications.

vision-language modelMLLM
OmniCompassOmniCompass-VLOmniCompass-AV+1

evals, multimodal, research· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper

SP$^2$ec: Adaptive Self-Speculative Decoding for Vision-Language Models

SP$^2$ec (Single-Peak SPeculative decoding) is a training-free bandit algorithm that adapts the draft configuration of self-speculative decoding per prompt for vision-language models. It relies on the single-peak throughput pattern seen in SWIFT, KnapSpec, SpecVLM and the authors' cosSim method. It minimizes a new wall-time regret measure and has theoretical guarantees of near-optimal configuration with negligible exploration cost. On Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-Mistral-7B it improves over the original self-speculative methods by up to 11% on visual and textual tasks.

vision-language modelQwenMistral
SP$^2$ecSWIFTKnapSpec+5

research, inference, multimodal· Added 14 Oct 2026, published 13 Oct 2026

developers.googleblog.comarticle

Gemini 4 Argon for Robotics

Google's developer post presents Gemini 4 Argon as a physical-reasoning model for the whole robotics lifecycle. It reports 87% on the Embodied Reasoning (ERQA) benchmark, ahead of other frontier models. It then gives representative prompts: controlling a Boston Dynamics Spot with a MuJoCo digital twin, closed-loop code-as-policies arm control, steering the Gemini Omni world model, and building digital twins from teleoperation episodes. Further prompts cover retargeting human video to a 76-DoF humanoid, hindsight annotation of ALOHA 2 videos, a general value function on UMI data, sim-data generation for VLA fine-tuning (56% to 75% real success), and reproducing control papers in MuJoCo WebAssembly. It is useful for seeing how the model is prompted for agentic, simulation-driven robotics work.

GeminiMuJoComujoco_warp+5
Gemini 4 ArgonGoogleERQA+11

models, multimodal, agents· Published and added 13 Oct 2026

arxiv.orgpaper

UniW(A)M: A Unified Mixture-of-Experts Framework for World and World-Action Models

UniW(A)M is a mixture-of-experts model built on diffusion Transformers that unifies action-conditioned world modeling (AC-WM) with several world-action model (WAM) variants. Shared experts capture common physical dynamics, while atomic capability experts are reused across interfaces, each keeping its own conditional pathway. At deployment a plug-in adaptive interface router (PAIR), trained on a small subset of target-task chunk scenarios with the backbone frozen, picks a WAM interface for each execution chunk. It reaches 88.3% average real-world success against 76.7% for the strongest baseline, leads WorldArena on all three action-conditioned world-modeling tasks, and PAIR raises OOD success from 63.3% to 76.7%.

Mixture of Expertsdiffusion modelTransformer
UniW(A)MPAIRLIBERO+3

research, multimodal· Added 14 Oct 2026, published 13 Oct 2026

deepmind.googlearticle

Gemini 4 Argon

Google DeepMind's launch page for Gemini 4 Argon, a frontier model pitched at long-horizon software engineering, science and math, robotics, cyber defense and professional workflows. Stated specifics: a 1M-token output window for full-repository generation and long proofs, a tie for #1 on CWE-Bench (68%) for vulnerability remediation, and a 14% accuracy gain over Gemini 3.1 Pro in Box's evaluation. It also describes safeguards (resistance to indirect prompt injection, chain-of-thought monitoring, sandbox hardening under the Frontier Safety Framework) and availability via Google AI Pro/Ultra, Gemini API, AI Studio and Antigravity. Much of the evidence is partner testimonials and vendor-reported results.

GeminiGoogle AntigravityGoogle AI Studio+1
Gemini 4 ArgonGoogle DeepMindGoogle+16

models, coding-agents, multimodal· Published and added 13 Oct 2026

deepmind.googlearticle

Gemini 4 Argon for Robotics

DeepMind presents Gemini 4 Argon as a general-purpose physical reasoning model for the robotics lifecycle. It reports 87% on the Embodied Reasoning (ERQA) benchmark and gives representative prompts across several areas: controlling a Boston Dynamics Spot with a MuJoCo digital twin, code-as-policies arm control with a visual coordinate harness, and steering the Gemini Omni world model in a visual feedback loop. Further examples cover building MuJoCo and mujoco_warp scenes from teleoperation data, egocentric video or a hand-drawn sketch for 76-DoF humanoids, hindsight annotation and value-function progress prediction on video, sim data generation for VLA fine-tuning (success rate 56% to 75%), and reproducing control papers in MuJoCo WebAssembly. It matters as a concrete look at how a frontier model is being used as agentic coder, perceiver and sim builder for robotics.

MuJoComujoco_warpBlender+5
Gemini 4 ArgonGoogle DeepMindERQA+10

models, multimodal, agents· Published and added 13 Oct 2026

arxiv.orgpaper

EmoGenie: Benchmarking and Evaluating Expressed and Evoked Emotion in T2I Generation with MLLMs

EmoGenie is a benchmark for emotions expressed and evoked by text-to-image outputs. It has 6,400 images from eight T2I models using 800 emotion-targeted prompts. It includes EmoGenie-QA (90,000 closed-set and 1,057 open-ended questions) and EmoGenie-Assess (13,175 mean opinion scores for valence, arousal, dominance and aesthetic quality). The authors also propose EmoProbe, which pairs emotion QA with response-aligned, dimension-specific regression heads. They report that it beats baselines on emotion QA and correlates better with human ratings. The work shows how reliably MLLMs interpret affect in generated images.

multimodal large language modeltext-to-image
EmoGenieEmoGenie-QAEmoGenie-Assess+2

evals, multimodal, research· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper

Heading Hierarchy Reconstruction for Complex and Long Documents with Level-Wise Multimodal Memory

Proposes a vision-language-model framework that incrementally reconstructs heading hierarchies across tens or hundreds of pages using a level-wise multimodal memory. The memory keeps a compact set of earlier pages chosen by anchors (here, the most recent node at each heading level), so the VLM sees the visual, textual and layout patterns of headings at every level. The authors also introduce CLDoc-HHRBench (250 documents, 11,212 pages, expert-verified annotations) and report results on it and three existing benchmarks. It matters for RAG pipelines that need coherent document-level structure, which page-by-page OCR/VLM parsing doesn't give.

vision-language modelRAG
CLDoc-HHRBencharXiv

research, multimodal, evals· Added 14 Oct 2026, published 13 Oct 2026

pdftomp3.comtool

Show HN: Use your OpenAI subscription to create audiobooks

PDF TO MP3 is a hosted MCP server (Streamable HTTP, account authentication) that turns PDFs into audiobooks. The user's existing AI subscription, such as Codex, handles document processing through the MCP connection. Speech generation is billed separately through the user's own OpenAI or Google text-to-speech API key. The setup is three steps: add the MCP server, add a speech key, then attach a PDF and ask the assistant to use the MCP. It shows a "bring your own subscription" pattern that keeps the vendor's compute costs low.

Model Context ProtocolCodexOpenAI API+1
PDF TO MP3OpenAIGoogle

mcp, multimodal· Published and added 13 Oct 2026

arxiv.orgpaper

Seeing Through the Forgery Is Not Enough: Regressive Sycophancy under Fabricated Evidence in Medical Vision-Language Models

The paper studies regressive sycophancy under fabricated evidence: a medical VLM gives a correct answer, then abandons it when the user pushes back with a fake prescription. It tests four open VLMs on 400 VQA-RAD yes/no questions in 19,200 multi-turn conversations. Without a synthetic-document disclosure, models abandon 80.1% of 6,488 correct answers, and the same prescription raises Qwen2.5-VL-7B's flip rate from 45.9% to 89.6% while lowering Qwen3-VL-8B's from 77.7% to 50.9%. Models often reject the prescription yet keep the correct answer in only some conversations (915 of 3,232), and 40% of those flip later. System-prompt disclosure of synthetic documents cuts one model's flip rate from 77.8% to 47.3%. The authors conclude that recognising a forgery does not stop a model from adopting the forged answer.

vision-language modelQwen3-VL
VQA-RADQwen2.5-VL-7BQwen3-VL-8B+1

ai-safety, multimodal, evals· Added 14 Oct 2026, published 13 Oct 2026

huggingface.corepo

google/embeddinggemma-2

Model card for EmbeddingGemma 2, an open (Apache 2.0) 740M-parameter multimodal embedding model from Google DeepMind built on Gemma 4. It maps text, code, images, video and audio into one 768-d space, with a 270M text backbone and optionally loadable 170M vision and 300M audio encoders, an 8K context, and Matryoshka truncation to 128/256/512/768 dims. The card gives practical guidance: task instruction prefixes, re-normalising after truncation, avoiding float16 (NaN or degraded embeddings), interleaved multimodal inputs with placeholder tokens, and vision token budgets. It targets on-device search, RAG, classification and clustering.

Hugging FaceTransformerssentence-transformers+5
EmbeddingGemma 2Google DeepMindGemma 4+1

models, multimodal, llm-apps· Added 12 Oct 2026

github.comrepo

Show HN: Atticus – Hold-to-talk voice assistant for macOS that runs on Claude -p

Atticus is an MIT-licensed macOS 26+ hold-to-talk voice assistant that uses Apple's on-device speech transcription and runs AI requests through Claude Code on an existing Claude Pro/Max plan, so no API key is needed. Simple commands skip the model entirely (apps open in 0.08 s), actions it has seen once are learned as zero-cost shortcuts, and a "remember…" memory persists across asks. It also sets up no-AI background watchers (polling every 5 s) and holds itself to a budget of a median under $0.002 and 6 s per AI ask, with a system prompt under 10,000 characters.

Claude CodeClaudeCodex+6
AtticusAnthropicApple+1

agents, coding-agents, multimodal· Published and added 12 Oct 2026

github.comrepo

bilawalsidhu/gods-eye-view

God's Eye View is an open-source, vanilla JavaScript + CesiumJS + Vite 3D globe that fuses live public feeds (aircraft, ships, satellites, quakes, public cameras, weather) into one explorable scene. Its AI layer is a hands-free voice agent on the OpenAI Realtime API with 30 tools: camera fly-to, whiteboard annotations, analyst queries over live layers, and scene-context and screenshot grounding. Keys stay server-side and the browser gets only a short-lived session token. An experimental ChatGPT/Codex OAuth path is also supported. The app doubles as an MCP server, so Claude Desktop, Codex or ChatGPT can open the globe from a conversation.

OpenAI APIModel Context ProtocolCesiumJS+8
God's Eye ViewBilawal SidhuHalfpixel+2

llm-apps, mcp, multimodal· Added 11 Oct 2026

latent.spacearticle

Building AI for Reliable Execution: Lessons From Industrial Robotics

Latent Space interviews Standard Bots' CEO Evan Beard and Head of AI Leif Jentoft about the AI stack behind its industrial robot arms. A shared base model (low billions of parameters) is adapted per customer through demonstrations and fine-tuning. A zero-shot perception model, whose backbone is trained on over a billion images, handles machine tending, while conventional code handles motion and cell logic. Inference runs on-premises on edge GPUs that stream action chunks to low-level control. Failures and human corrections feed fleet learning except in air-gapped defense deployments, and the StandardOS APIs/SDKs let developers bring their own models such as NVIDIA Cosmos. The lesson is to keep the model's role narrow and use targeted, high-quality data to fix edge cases instead of chasing dataset size.

StandardOSNVIDIA Cosmosedge GPUs
Standard BotsEvan BeardLeif Jentoft+13

multimodal, inference, training· Published and added 10 Oct 2026

eprint.iacr.orgpaper

Confidential Error-Correcting Codes

Introduces Confidential Error-correcting Codes (CECs), a primitive that decodes reliably from perturbed codewords while keeping the message confidential, defined by confidentiality, robustness and soundness. The Randomness-Security Separation framework encodes a nonce and masks the payload with a PRF; the Conc-CEC instantiation uses concatenated Reed-Muller and Reed-Solomon codes and claims stronger concrete security than PRCs (CRYPTO'24). The application, CECMark, watermarks AI-generated images and reportedly beats Gaussian Shading, PRCMark and T2SMark across 56 perturbations with a false positive rate below 1e-5 and no side effects.

Reed-Muller codesReed-Solomon codespseudorandom function
Confidential Error-Correcting CodesCECMarkConc-CEC+5

research, multimodal· Added 14 Oct 2026, published 10 Oct 2026

huggingface.corepo

Qwen3.8-27B (Apache-2.0, multimodal dense, 262k ctx)

Model card for Qwen3.8-27B, a 27B dense vision-language model from the Qwen team with a hybrid Gated DeltaNet / Gated Attention layout (64 layers, 16×(3 DeltaNet + 1 attention)), MTP training, and 262k native context extensible to 1M via YaRN. It targets coding, research and long-horizon agentic tasks, with thinking on by default, reasoning_effort levels (xhigh/medium/low) and preserve_thinking. The card gives deployment recipes for Transformers, vLLM, SGLang and TokenSpeed, recommended sampling parameters, video-token settings for hour-scale video, and benchmark notes (SWE-bench Pro, NL2Repo-Bench, DeepSWE, HLE, MathVision, WebArena-Verified, etc.).

Qwen3.8-27BHugging FaceTransformers+8
QwenQwen3.5SWE-Bench Pro+14

models, multimodal, inference· Added 9 Oct 2026

arxiv.orgpaper

Images Talk Back: Guiding MMDiTs by Suppressing Image-to-Text Feedback

The paper studies joint attention in multimodal diffusion transformers (MMDiTs), where the evolving image updates text representations. It finds that this image-to-text (I2T) feedback tends to preserve objects and attributes already in the image, errors included, and that suppressing it improves generation. It proposes Semantic Feedback Guidance (SFG), which guides away from an auxiliary branch that strengthens I2T feedback and weakens text-to-image conditioning, and iSFG, an in-place variant that needs no extra model evaluations. SFG runs for the first 20% of sampling steps (about 1.2x base NFE without CFG). Across three backbones, adding SFG to CFG on FLUX raises GenEval from 0.6661 to 0.7479, and iSFG on SD3.5M adds +1.33 HPSv2.1 over CFG at zero extra NFE. The control also improves VBench scores for text-to-video and image-to-video.

diffusion modelTransformerFLUX+1
SD3.5MGenEvalHPSv2.1+4

research, multimodal· Added 12 Oct 2026, published 9 Oct 2026

arxiv.orgpaper

Hide and Seek: HIDE Concepts to SEEK explanations for black box models

Hide and Seek (HnS) is a concept-level attribution method for black-box vision models. It needs no model weights and no concept-labeled datasets. Given a concept and an image set, it uses a vision-language model to retrieve concept-aligned images and localize the concept, masks the localized region, and takes attribution from the change in the black-box model's output. It agreed better with ground-truth attributions than existing methods on CelebA and COCO-Stuff, and it works on tasks existing concept attribution methods cannot handle, such as image captioning and satellite image segmentation.

vision-language model
Hide and SeekHnSCelebA+1

research, multimodal· Added 12 Oct 2026, published 9 Oct 2026

arxiv.orgpaper

When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Priors

Introduces EICS (Evaluation-Invariant Concept Stability), a metric for checking whether representations consistently encode a concept under controlled one-concept interventions. Also introduces CASP-VL, a benchmark of causally controlled spatial perturbations derived from Visual Genome scene graphs. Across several VLM architectures, spatial relations are represented much less faithfully than lexical information, even for models well above chance on compositional benchmarks. The relation is identifiable at the predicate's own tokens but is lost when the sentence is aggregated into a global representation (EOS token or mean pooling), a failure that output-based evaluation does not reveal.

vision-language model
Visual GenomeEICSCASP-VL+1

research, evals, multimodal· Added 12 Oct 2026, published 9 Oct 2026

x.comtool

Gurasees (@gurasees0) on X

Robo dir is a directory that aggregates robotics and physical-AI datasets in one place. The launch tweet claims 3,494 datasets, 86,424 Hugging Face uploads, 236B+ frames and 2.2M+ hours of data. It also exposes the index through a single MCP server with no sign-in, so agents can search for robot training data programmatically.

Model Context ProtocolHugging Face
Robo dir

training, mcp, multimodal· Published and added 9 Oct 2026

arxiv.orgpaper

Benchmarking Answer-Grounding Consistency in Remote Sensing Visual Question Answering

Proposes the Answer-Evidence Support Protocol (AESP), which checks whether the grounding evidence a model outputs actually supports its own answer in grounded remote sensing visual question answering. It applies task-specific rules to existence, counting and comparison questions. The authors build the G-RSVQA benchmark of 15,027 samples with questions, reference answers and object-level annotations. Tests on seven grounded vision-language models show evidence-support failures on all three question types, with up to 50.82% of correct answers unsupported by their grounding evidence.

vision-language model
Answer-Evidence Support ProtocolG-RSVQAarXiv

evals, multimodal, research· Added 12 Oct 2026, published 9 Oct 2026

x.comthread

Adam Shepherd (@EntryStateGod) on X

Social post announcing Qwen's open-weights release of Qwen-Image-2.1-Turbo, an accelerated checkpoint of Qwen-Image-2.1 on the same 7B visual generation architecture. It generates and edits images in 8 denoising steps, against 40 in the base model's sample code, with 2K output. Qwen claims the fewer steps do not reduce quality. The license is non-commercial, which limits production use.

Qwen
Qwen-Image-2.1-TurboQwen-Image-2.1

models, multimodal· Published and added 9 Oct 2026

arxiv.orgpaper

RARA: Robust Policy Fine-Tuning through Retrieval-Anchored Representation Alignment for Robotic Manipulation

RARA (Retrieval-Anchored Representation Alignment) is a two-stage method for adapting pre-trained robot policies such as Vision-Language-Action models to new tasks from few demonstrations. It first retrieves pretraining representations with similar task intent and motion and aligns the policy's representations of the target demonstrations to them. It then fine-tunes on the target demonstrations while constraining representation change on the retrieved demonstrations, which limits representation drift. Tested with Diffusion Policy and GR00T on 13 tasks in RoboTwin, RoboCasa and on a real robot, it improves average success over plain fine-tuning by 9.5 points in-distribution and 13.6 out-of-distribution, and beats all compared baselines.

GR00TDiffusion Policyvision-language-action model
RARARoboTwinRoboCasa

research, training, multimodal· Added 12 Oct 2026, published 9 Oct 2026

arxiv.orgpaper

ManeuverFlow: Maneuver Hypothesis Generation with Future Evidence for Trajectory Evaluation and Refinement in End-to-End Autonomous Driving

ManeuverFlow is an end-to-end autonomous driving planner that generates trajectories for distinct driving intentions in one step. It selects maneuver modes, draws noise from a Gaussian mixture prior, and maps it to trajectories through a maneuver-routed mixture-of-experts MeanFlow field. Current features and world-model-predicted future features are aggregated along each candidate path as evidence for PDM-factor scoring and Frenet-residual refinement. With a TransFuser-style ResNet-34 backbone it reaches 90.6 PDMS on NAVSIM v1 and 90.1 EPDMS on NAVSIM v2 navtest. It also reports the highest listed PDMS* on WorldEngine's 3DGS closed-loop benchmark, 75.19 with logged traffic and 74.58 with IDM-controlled traffic.

flow matchingMixture of Experts3D Gaussian Splatting+1
ManeuverFlowNAVSIMWorldEngine+6

research, multimodal· Added 12 Oct 2026, published 9 Oct 2026

arxiv.orgpaper

DreamTrue: Action-Faithful Robot World Model with Counterfactual Post-Training

DreamTrue is a multi-view, cross-embodiment robot world model for action-faithful, physically plausible video prediction. It renders action trajectories into image-space conditions aligned to target videos by offline geometric calibration. Counterfactual post-training modifies recorded trajectories to cover unsuccessful interactions and contact configurations. A human-annotated defect dataset trains an embodied video reward model whose scores drive RL post-training. On AgiBot it reaches state-of-the-art action following and cuts the human-assessed interaction defect rate from 48.12% to 6.25%. It ranked first in the world model track of the AgiBot World Challenge 2026.

DreamTrueAgiBotAgiBot World Challenge 2026

research, multimodal, training· Added 9 Oct 2026, published 8 Oct 2026

arxiv.orgpaper

OuroWorld: Bringing Any 3D World Alive as Diverse, Endlessly Looping 3D Cinemagraphs

OuroWorld is a mask-free framework that turns any static 3D Gaussian Splatting scene into a seamlessly looping 3D cinemagraph. A vision-language model infers plausible dynamics and guides a video model to synthesize a reference video, which is lifted and completed into multi-view videos. Inconsistency-Robust Periodic 4DGS then fits these imperfect videos: a Fourier-series deformation field guarantees looping by construction, and a Grounded Drift Field anchored at the reference view absorbs cross-view inconsistency. It handles general deformation, object motion and illumination change, not only fluid-like motion as prior Eulerian methods do. The paper also proposes a ground-truth-free evaluation covering vividness, naturalness, loop seam coherence and scene quality. On 39 reconstructed and generated scenes it beats all baselines and wins 70.8%-99.0% of user-study comparisons.

3D Gaussian Splattingvision-language model
OuroWorldInconsistency-Robust Periodic 4DGSGrounded Drift Field

research, multimodal· Added 9 Oct 2026, published 8 Oct 2026

arxiv.orgpaper

OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning

OmniCapBench is a benchmark of 786 densely annotated videos that evaluates audio-visual captioning in multimodal LLMs. Instead of scoring free-form captions, it asks models for sets of atomic, verifiable units across three tracks (entity references, visual shots, audio events). These are scored with deterministic constraint checks plus localized LLM-based semantic comparisons, which gives coverage, localization and stable scoring together. Tests on frontier MLLMs show strong local perception but weak long-horizon audio-visual reasoning, especially identity drift and cross-modal misalignment.

multimodal large language model
OmniCapBench

evals, multimodal, research· Added 9 Oct 2026, published 8 Oct 2026

arxiv.orgpaper

WOVEN: Weaving Visual World Modeling into Multimodal LLMs

WOVEN is a training source and benchmark for visual transition reasoning in multimodal LLMs. It has 36,076 examples spanning 20 scene types, 5 action types and 8 reasoning types, built from rollouts of video-pretrained generative models. Evaluating 38 frontier MLLMs (including GPT-5.4 and Qwen3-VL-235B-A22B) shows systematic deficits that persist with scale. Training on WOVEN subsets of about 2,000 items improves 22 of 26 external benchmarks by up to 27.3 points, and WOVEN data can replace 30-50% of a task's own training data. The authors derive a recipe: select supervision by the reasoning operation it teaches rather than by scene or action, and prefer larger visual-state changes.

vision-language modelQwen3-VL
WOVENGPT-5.4

research, multimodal, evals, training· Added 9 Oct 2026, published 8 Oct 2026

arxiv.orgpaper

Connected Self Forcing: Beyond Local Learning in Video Autoregression

Connected Self Forcing extends Self Forcing, which trains autoregressive video generators on their own rollouts with KV caching but detaches historical caches and so cuts backward gradients between chunks. It reconnects gradient paths across chunks, passing gradients through generated latents into the computations that produced them, so later predictions can shape how earlier context is generated. Shortcut gradient replay recovers cross-chunk gradients without keeping the full rollout computation graph. Combined with distribution matching distillation, it improves long-horizon visual quality and temporal consistency with no change to inference.

KV cachedistribution matching distillation
Connected Self ForcingSelf Forcing

research, multimodal, training· Added 9 Oct 2026, published 8 Oct 2026

arxiv.orgpaper

PathTime-VLA: Path-Time Decoupling for Factorized Post-Training of Vision-Language-Action Policies

PathTime-VLA reparameterizes Vision-Language-Action actions as a progress-indexed interaction path X(s) plus a positive interval-time profile that defines a monotone time law t(s). Geometry is thus decoupled from execution speed, so teleoperation timing artifacts don't contaminate the path target. A staged post-training recipe uses demonstrations and DAgger interventions for a target-domain prior, Speed-DQN to learn execution-speed multipliers from robot interaction, and Path-AWR to refine the diffusion path generator from rollout outcomes. On three tasks it reaches 58/60 successes versus 57/60 for BC+DAgger at fixed 1×, with about 39–52% shorter mean completion times on successful trials.

Vision-Language-Actiondiffusion modelDAgger+2
PathTime-VLA

research, training, multimodal· Added 9 Oct 2026, published 8 Oct 2026

x.comthread

Marktechpost AI (@Marktechpost) on X

Liquid AI released Open d1, two open-weight "decision models" — d1-3B (text + vision) and d1-omni-600M (text + image or text + audio) — that don't generate text tokens at all. Instead they take a state plus a typed question as input and return calibrated probabilities in a single forward pass, positioning them as lightweight classifiers/judges rather than generative LLMs.

vision-language model
Liquid AId1-3Bd1-omni-600M

models, multimodal· Published and added 8 Oct 2026

decrypt.coarticle

Google Lets You Build a Video Game by Typing a Description

Google launched Playground, a browser-based tool letting U.S. adults 18+ describe a video game in plain language and get a playable, shareable result back without coding, with creation access tiered by Google AI subscription. The piece ties Playground to Google's broader AI-for-games push: DeepMind's Genie (an 11B-parameter model trained on 200,000+ hours of gameplay video) and its successor Genie 3, a real-time interactive world model opened to AI Ultra subscribers via Project Genie, plus an upcoming integration with Unity Spark for 3D, professional-level game mechanics. Google did not disclose which models power Playground or its pricing, and framed it as an early experiment, while the announcement reportedly sank Unity and Roblox stock on fears of AI displacing traditional game-dev tools.

Gemini
GoogleGoogle DeepMindUnity+3

multimodal· Published and added 7 Oct 2026

arxiv.orgpaper

Rephrase Before You Act: Characterizing and Mitigating Language Sensitivity in Vision-Language-Action Models

This paper shows that vision-language-action (VLA) models are highly sensitive to how instructions are phrased: a single-word change (e.g. "stove" vs "hot plate") can shift task success by dozens of points on π0 and π0.5 checkpoints in the LIBERO benchmark, even after rephrase-augmentation finetuning. The authors characterize this with statistically tested single-edit swings and an oracle phrase search, then mitigate it without retraining by having an LLM distill scoring evidence into 10-20 explicit rephrasing rules that rewrite incoming instructions at deployment time, improving frozen π0 by 16-27% relative on held-out tasks and lifting π0.5's in-finetune success from 93.6% to 97.8%.

vision-language model
π0π0.5LIBERO

research, multimodal· Added 8 Oct 2026, published 7 Oct 2026

arxiv.orgpaper

RoboJEPA: Scaling Robotic Latent World Models

RoboJEPA is an 8B-parameter latent world model built on the Joint Embedding Predictive Architecture (JEPA), trained on real-robot data spanning 12 robotic embodiments. The paper establishes the first scaling laws for multi-embodiment robotic world models, showing that imagination error (latent rollout error) follows a second-order power law in compute and predicts downstream planning performance, and demonstrates zero-shot deployment of the model as a goal-image-conditioned planner on real hardware for long-horizon tasks.

Joint Embedding Predictive Architecture
RoboJEPA

research, multimodal· Added 8 Oct 2026, published 7 Oct 2026

x.comthread

Liquid AI (@liquidai) on X

Liquid AI announces Open d1, two open-weight multimodal decision models. d1-3B (built from LFM2.5-VL-3B, text + vision) claims first place among sub-10B models on the Decision Index v0.2.1, while the experimental d1-omni-600M (LFM2.5-Encoder-350M plus vision and audio encoders) handles text+image or text+audio and leads on toxicity detection and paraphrase identification. Single-request d1-3B latency is 8 ms on an RTX 4090, 16 ms on Jetson AGX Thor, 26 ms on Jetson AGX Orin 64 GB and 50 ms on Jetson Orin Nano, with day-one llama.cpp support; demos include live-camera apps and Isaac Sim navigation. Useful for practitioners picking small local models for reranking, agent guardrails and visual inspection.

llama.cppHugging FaceNVIDIA Jetson+4
Liquid AId1-3Bd1-omni-600M+4

models, multimodal, inference· Added 9 Oct 2026, published 7 Oct 2026

arxiv.orgpaper

Self-correction Optimization for Interleaved Multimodal Generation

This paper introduces Self-Correction Optimization (SCO), a training-free method that improves consistency in interleaved image-text generation from multimodal large language models. SCO treats the classifier-free guidance update as a reference signal and applies minimal corrections under two constraints—a new-event constraint for temporal consistency and a state-preserving constraint for visual-subject coherence—avoiding the computational cost of additional training with augmented data. The authors show improvements on interleaved multimodal generation benchmarks and extend the method to video generation tasks like robot manipulation and long-horizon handcrafting.

vision-language model

research, multimodal· Added 8 Oct 2026, published 7 Oct 2026

arxiv.orgpaper

Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving

GeoCoTDrive is a vision-language-action (VLA) framework for autonomous driving that introduces an explicit geometric chain-of-thought: it first grounds 2D regions corresponding to decision-critical cues, then retrieves localized 3D geometric priors from a geometric foundation model within those regions, interleaving these features into the autoregressive context to drive trajectory generation. The paper also introduces "planning-relevant grounding," a new region-level grounding supervision task, and the PlanningGrounding dataset to teach VLAs planning-oriented grounding, showing consistent gains in safety-critical planning across multiple end-to-end driving benchmarks.

vision-language model
GeoCoTDrivePlanningGrounding

research, multimodal· Added 8 Oct 2026, published 7 Oct 2026

arxiv.orgpaper

TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelity

TaoD2C-Bench is a benchmark for evaluating multimodal large language models (MLLMs) on industrial design-to-code (D2C) tasks — converting UI designs into code that satisfies implementation constraints, not just visual fidelity. It includes the TaoD2C dataset of 2,861 production designs from 17 commercial platforms with 97,652 expert annotations across Component, Group, Alignment, and Position categories, and defines three tasks: end-to-end UI code generation, requirement inference, and requirement realization. Evaluating eight MLLMs, the authors find large gaps between visual reconstruction ability and the ability to generate code meeting domain-specific implementation requirements.

vision-language model
TaoD2C-BenchTaoD2C

evals, multimodal· Added 8 Oct 2026, published 7 Oct 2026

arxiv.orgpaper

CARES: A Controlled Synthetic Benchmark of Speaker Reactions to Sound

CARES is a synthetic benchmark of 10,000 two-speaker audio scenes designed to test whether audio-language models correctly identify which sounds are 'salient' — defined as sounds a speaker audibly reacts to. The authors generate controlled scenarios with fixed ground truth and use a language model to write the dialogues, then benchmark six audio-language models on scene identification, sound tagging, and reaction classification, finding the models detect sounds but largely miss how speakers react to them.

CARES

evals, multimodal, research· Added 8 Oct 2026, published 7 Oct 2026

huggingface.coarticle

Introducing Falcon ASR

TII (Technology Innovation Institute) announces Falcon-ASR, a 1.6B-parameter automatic speech recognition model focused on the Emirati Arabic dialect, which also supports MSA, other Gulf dialects, English, French, Spanish and Portuguese under one set of weights. It reports 20.92% average WER across six Arabic benchmarks on the Open Universal Arabic ASR Leaderboard (beating the best published leaderboard result of 23.17%), 22.73% WER / 10.19% CER on an internal Emirati evaluation (beating Qwen3-Omni by 4.07 points), and 5.74% mean WER on the Hugging Face Open ASR Leaderboard's English test sets. The model builds on the team's Falcon3-Audio work, supports word-level timestamps, and is available via a Hugging Face demo space with API access planned.

Hugging Face
TIIFalcon-ASRFalcon3-Audio+7

models, multimodal, evals· Added 8 Oct 2026, published 7 Oct 2026

simonwillison.netwriteup

llm-openai-decisions 0.1a0

Simon Willison built llm-openai-decisions, a plugin for his `llm` CLI tool that wraps OpenAI's newly announced Decisions API (unveiled at DevDay), modeled on his existing llm-typesafe plugin for Jev's competing API. Both APIs support structured yes/no, multiple-choice, and score-style questions, but OpenAI's gpt-6-luna model also accepts image input (at 10 cents per million input tokens, versus Jev's 4.2 cents), and the plugin itself was built by having GPT-6 Astra read OpenAI's docs. The post includes install instructions and a worked example querying an image attachment for a yes/no answer.

OpenAI APIllm
Simon WillisonOpenAIJev+3

prompting, multimodal· Added 7 Oct 2026, published 6 Oct 2026

decrypt.coarticle

Google Launches Nano Banana 2.1: Better Than Its Predecessor at Half the Price

Decrypt reports on Google October launch of Nano Banana 2.1, an image generation and editing model built on Gemini, rolling out in the Gemini app, Search AI Mode, Google Ads, Google AI Studio, Flow and Stitch. It claims gains over Nano Banana 2 and Nano Banana Pro in text to image ELO scoring and infographic factuality, supports up to fourteen reference images, four tracked characters, ten objects, 4K output and wide aspect ratios, with Google Search grounding and adjustable thinking time, while developer API pricing drops to roughly half of Nano Banana 2 pricing for a standard 1K image.

GeminiGoogle AI Studio
GoogleGoogle SearchGoogle Ads+2

models, multimodal· Published and added 6 Oct 2026

deepmind.googlearticle

EmbeddingGemma 2: an open, lightweight multimodal embedding model

Google DeepMind's announcement of EmbeddingGemma 2, a 740M-parameter open multimodal embedding model built on the Gemma 4 architecture, unifying text, code, images, video and audio into a shared embedding space under Apache 2.0. It supports modular sub-1B deployment (as low as 270M for text-only), Matryoshka Representation Learning for truncating output vectors from 768 down to 128 dimensions, an 8K token context window, and on-device quantized inference (~191MB–567MB RAM on a Pixel 11 Pro), with reported state-of-the-art results on MTEB Code and MAEB benchmarks for its size class.

GemmaGeminiHugging Face+6
EmbeddingGemma 2EmbeddingGemmaGemma 4+11

models, multimodal, inference· Published and added 6 Oct 2026

arxiv.orgpaper

World Models' Last Exam in Physics

World Models' Last Exam in Physics introduces a measurement-based benchmark of 40 controlled tasks across mechanics, optics, fluids, thermal/phase-change phenomena, electromagnetism, and surface tension, designed to test whether video world models generate physically consistent outputs. Each task pairs an initial image and prompt with predefined physical criteria, and a custom evaluator combines observability screening with task-specific quantitative measurements rather than relying on reference videos or model-based judgments. Testing eight video generation models across 1,280 generated videos found persistent physical inconsistencies, with the best model scoring only 57.76/100, and the evaluator showed higher agreement with human judgments than a vision-language model baseline.

vision-language model
World Models' Last Exam in Physics

evals, multimodal· Added 7 Oct 2026, published 6 Oct 2026

arxiv.orgpaper

PEARS: Physical-Prior-Guided Efficient Adaptation via Failure Reasoning and Diffusion Steering for Tactile Manipulation

PEARS is a hybrid reinforcement-learning framework for sample-efficient online adaptation of pretrained tactile-manipulation policies to out-of-distribution conditions. A physics-guided force reasoning (PFR) module uses a vision-language model to diagnose failures from visual and tactile history and update contact-force bounds enforced by a hybrid force-position controller, while a separate tactile-conditioned diffusion-steering component perturbs the latent noise of a frozen flow-matching policy to correct free-space motion and contact timing without retraining the base model. It improves success rates by 12.4-37.4 points over strong baselines in simulation, cuts required interaction episodes by up to 53.2%, and reaches 95%/90% success on real-world whiteboard-erasing and pipette-liquid-aspiration tasks.

vision-language modelflow matching
PEARSWhiteboard ErasingPipette Liquid Aspiration

research, multimodal, training· Added 7 Oct 2026, published 6 Oct 2026

arxiv.orgpaper

ALIVE: Interaction-Aligned Object Insertion for First-Frame-Guided Video Editing

ALIVE is a framework for first-frame-guided video editing that inserts objects so they coherently participate in interactions (e.g., being picked up or manipulated) rather than appearing static. The authors curate a 35,800-pair editing dataset combining 3D-rendered, model-generated and real-world videos plus pairs from ROSE, and train a vision-language model to predict interaction guidance from the edited first frame and an instruction naming the object. They introduce the ALIVE-interaction benchmark to measure interaction fidelity, source preservation and visual coherence via a unified VLM-based protocol, reporting a 43.9% Overall improvement on this benchmark and 4.4% on a general video object insertion benchmark over the strongest baseline, with VLM-predicted guidance adding a further 0.95-point gain.

vision-language model
ALIVEROSEALIVE-interaction

research, multimodal· Added 7 Oct 2026, published 6 Oct 2026

arxiv.orgpaper

Post-Training Semantic Lifting for 3D Gaussian Splatting: Separating Detector, Lifting and Representation Error

This arXiv paper introduces a post-training semantic lifting method for 3D Gaussian Splatting that assigns a single target class to Gaussians by accumulating target/non-target evidence across all views, weighted by per-view visibility, then filters with a dual-threshold (β and γβ) scheme that selects high-confidence seeds and extends them via connected components. The authors evaluate by transferring labels from Gaussians to mesh vertices and decompose error into three sources — the 2D detector, the lifting step, and the representation transfer — tuning thresholds on seven Replica scenes and testing on ten held-out ScanNet++ scenes, reaching mean mIoU of 0.93/0.65 (dataset masks/YOLO masks) on validation and 0.80/0.54 on test, a 0.24 mIoU improvement over prior per-view thresholding. The error analysis finds the 2D detector is the dominant remaining error source, informing where future gains in 3D semantic reconstruction should focus.

3D Gaussian Splatting
ScanNet++ReplicaYOLO+1

research, multimodal· Added 7 Oct 2026, published 6 Oct 2026

arxiv.orgpaper

Towards an Extensible Benchmark for Spoken Dialogue with Social Robots

This paper proposes an extensible benchmark for studying spoken dialogue artifacts between humans and social robots, including clarification requests, interruptions, embodied signals like head nods and facial cues, and time constraints. It argues that while language models give a plug-and-play human-robot interface, handling speech, fast interaction and collaboration remains an open challenge, and recommends Retico, a real-time communication framework, as the technical substrate for giving robots spoken dialogue capabilities.

Retico

evals, multimodal· Added 7 Oct 2026, published 6 Oct 2026

arxiv.orgpaper

EC-RAG: Event Chain Retrieval-Augmented Generation for Long Video Understanding

EC-RAG (Event Chain Retrieval-Augmented Generation) is a training-free framework for long-video question answering that organizes video content into an explicit, temporally ordered "event chain" instead of retrieving isolated frames or snippets. It partitions video into semantically coherent segments, fuses multi-modal signals (speech, text, visual cues) at the event level, and retrieves relevant events from the chain to answer queries, working as a plug-and-play addition to existing large video-language models (LVLMs) without extra training. Evaluated on Video-MME, MLVU, and LongVideoBench, it outperforms frame-level retrieval baselines, showing the value of modeling temporal/event structure for long-video understanding.

vision-language model
Video-MMEMLVULongVideoBench

research, multimodal, llm-apps· Added 7 Oct 2026, published 6 Oct 2026

x.comtool

Agent Game Forge (@agentgameforge) on X

Agent Sprite Forge v0.4.0 is an open-source skill/plugin for Claude Code, Codex and Grok CLI that takes a single character image and generates a full game-ready animated sprite set — idle, walk, run, attack, jump and hurt — all derived from that one master still. It adds official API support for OpenAI, Gemini, xAI, ModelArk and fal.ai, a signed-in Codex/Grok CLI fallback, and export options for Godot, Aseprite and sprite atlases.

Claude CodeCodexGrok+4
Agent Sprite ForgeOpenAIxAI+2

coding-agents, multimodal· Published and added 6 Oct 2026

x.comthread

Itsvishal (@itsvishaltwt) on X

A tweet reports that Mistral has announced Mistral Large 4, internally nicknamed "Le Chonk," a mixture-of-experts model with roughly 1 trillion total parameters and 49 billion active parameters. It is described as natively multimodal and state-of-the-art on critical workloads including cyber defense, with API access starting immediately and an open-weight release planned for late October.

Mistral
Mistral Large 4

models, multimodal· Published and added 6 Oct 2026

mistral.aiarticle

Introducing Mistral Large 4 ("Le Chonk")

Mistral announces Mistral Large 4 ("ML4"/"Le Chonk"), a 1-trillion-parameter natively multimodal Mixture-of-Experts model with 52B active parameters, trained from scratch on 3,800 NVIDIA Grace Blackwell GPUs in Mistral's own European datacenters. The post details state-of-the-art open-weight results on cybersecurity benchmarks (82% on an Artificial Analysis vulnerability-reproduction test, 93% on Cybench), agentic coding (DeepSWE, Terminal-Bench, AutomationBench), finance/legal evals (via Vals, Harvey's Legal Agent benchmark), visual grounding (beating GPT-6 Astra on Dense 200), and safety benchmarks (Lakera B3, KORA). It also describes the large-scale RL post-training pipeline (33B tokens/day at 3k GPUs) used to produce the model, with full weights promised by end of month.

MistralMixture of Expertsvision-language model+1
Mistral Large 4GLM-5.3GLM-5.2+20

models, multimodal, evals, training· Added 7 Oct 2026, published 6 Oct 2026

almostimplemented.github.iowriteup

Learning Jazz Pianist Style with Cross-Attention Conditioning

An ISMIR 2026 (Abu Dhabi) research project that fine-tunes Aria, a transformer pretrained on piano MIDI, on solo recordings from twelve jazz pianists in the PiJAMA dataset, adding a gated cross-attention layer that conditions generation on a learned per-pianist embedding. Because standard held-out perplexity is nearly blind to style (the next note is predictable regardless of pianist), the authors instead measure style transfer by sliding a pianist classifier over freely generated continuations: conditioned continuations are attributed to the intended pianist 70% of the time versus 37% unconditioned, and a classifier trained only on generated music identifies real recordings with 95% accuracy. The page is an interactive demo with audio examples, blindfold listening tests, and per-pianist results (e.g. 96% attribution for Hank Jones/Dick Hyman vs 29% for Cedar Walton), alongside a link to the full paper with ablations on mismatch conditioning and memorization checks.

TransformerMIDI
AriaPiJAMAISMIR 2026+8

multimodal, training· Added 6 Oct 2026, published 5 Oct 2026

arxiv.orgpaper

Recursive Video In-Context Learning for Agentic Robot

This paper introduces Recursive Video In-Context Learning (RV-ICL), a training-free method that lets an LLM agent orchestrating frozen vision-language-action (VLA) policies learn from a single demonstration video rather than from full video or fixed keyframes. The demonstration is converted into a navigable hierarchy (task keyframes, phases, moments, short clips) built around sub-events like grasps and releases, exposed to the agent as read-only tools it queries coarsely during planning and drills into for contact-level detail during execution. Built on the RPent agent, it raises task success from 92.6 percent to 96.5 percent on LIBERO-PRO and from 86.7 percent to 95.8 percent on LIBERO-Plus with only one demonstration per task.

vision-language-action model
RV-ICLRPentLIBERO-PRO+1

research, agents, multimodal· Added 6 Oct 2026, published 5 Oct 2026

arxiv.orgpaper

MemPilot: Orchestrating On-Demand Multimodal Memory Curation for LLM Agents

MemPilot is a framework that lets LLM agents curate memory on demand rather than query-agnostically, using a multi-step policy trained via reinforcement learning to choose between retrieving from pre-built memory or delegating query-specific curation of raw multimodal history to heterogeneous LLMs and VLMs. The policy controls evidence amount, curation instructions, model choice, and visual access, and is optimized with objective-wise advantage decoupling and prefix-based marginal utility estimation to balance performance, cost, and latency. Tested across five multimodal agent-memory benchmarks, it achieves broader performance-cost-latency trade-off frontiers than existing trade-off-aware baselines.

vision-language model
MemPilot

agents, research, multimodal· Added 6 Oct 2026, published 5 Oct 2026

arxiv.orgpaper

MedPrune: Topology-Efficient Multimodal Multi-Agent Communication Evolution for Medical VQA Tasks

MedPrune is a multi-agent framework for medical visual question answering that models the diagnostic process as a heterogeneous communication graph of specialist agents across departments, then dynamically prunes both nodes (irrelevant specialist agents, via reinforcement learning) and edges (low-value intra-/inter-departmental connections) to cut redundant communication. The authors report it beats existing multi-agent baselines on medical VQA benchmarks under both full-set and few-shot settings while improving token efficiency and adversarial robustness.

vision-language model
MedPrune

research, multimodal, agents· Added 6 Oct 2026, published 5 Oct 2026

arxiv.orgpaper

Lens3D: Target-Conditioned Visual Foveation for Fine-Grained 3D Understanding

Lens3D is a method for improving fine-grained object understanding in 3D large language models by using 3D localization to select informative 2D views that are fed to an external vision-language model (the LensUnd pipeline), then distilling that knowledge back into native 3D LLMs via detailed caption supervision (LensDistill), avoiding the need for external VLM calls at inference time. The authors also release LensBench, a held-out evaluation set of 2,068 objects with three reference descriptions each, and show gains in fine-grained object captioning on Video-3D LLM and 3DRS while preserving grounding and scene-level QA performance.

vision-language model
Lens3DLensUndLensDistill+3

research, multimodal· Added 6 Oct 2026, published 5 Oct 2026

arxiv.orgpaper

BrainTRACE: Tracing Longitudinal, Multimodal, and Volumetric Evidence in Brain MRI Clinical Reasoning

BrainTRACE is a report-grounded benchmark for evaluating whether vision-language models can trace the evidence structure required for longitudinal brain MRI interpretation. It contains 7,273 scored VQA instances derived from 1,778 longitudinal patients, 7,299 MRI studies, and ~29k co-registered 3D MRI sequence volumes, organized across five levels of clinical reasoning and evidence demands like longitudinal comparison, multi-sequence integration, and volumetric spatial grounding. Evaluation of 20 VLM configurations finds that current models identify isolated visual cues but rarely compose them into grounded longitudinal clinical interpretations.

vision-language model
BrainTRACE

evals, multimodal· Added 6 Oct 2026, published 5 Oct 2026

arxiv.orgpaper

SpatialChain: A Benchmark for Auditing Spatial Reasoning Faithfulness in VLMs

SpatialChain is a benchmark of 28,350 training and 899 test examples pairing spatially-oriented GQA questions with scene-graph-grounded reasoning chains, designed to audit whether VLMs' correct spatial answers reflect faithful reasoning or linguistic shortcuts. Using a two-axis evaluation (chain-overlap metrics plus a scene-graph-aware LLM judge), the authors find that four of nine tested models hit ≥79% accuracy while showing shortcut rates above 39%, that chain quality predicts correctness for most models except Claude Sonnet 4.6 and InternVL3.5-8B (which fail differently, via terse vs. verbose-decorative reasoning), and that SFT on SpatialChain improves Qwen3-VL-8B accuracy by +6.2pp while cutting its shortcut rate to 22%. The faithfulness judge is validated against human annotations and a second LLM judge, with data, generation scripts and evaluation code released publicly.

vision-language modelQwen3-VL
Claude Sonnet 4.6InternVL3.5-8BQwen3-VL-8B+1

evals, multimodal· Added 6 Oct 2026, published 5 Oct 2026

x.comthread

Meer Tarbani - Growth & Engineering (@MeerTarbani) on X

Sidekick is an open-source macOS plugin for Claude Code that lets you define a one-sentence persona (e.g. "a blunt senior Rust dev") and drive a coding session hands-free by voice: it listens via Apple's on-device speech recognition, speaks answers with macOS voices, and lets you interrupt it mid-reply or say "stop" to cancel a turn. It also surfaces Claude's multiple-choice decisions as spoken questions, matching answers like "the second one" against an echo-filtered transcript so the assistant doesn't hear and respond to its own voice. Implementation is a ~500-line TypeScript Claude Code hooks module plus a 90-line Swift speech listener, runs fully local with zero servers and zero telemetry, and installs via the `claude plugin marketplace`/`claude plugin install` commands.

Claude CodeTypeScriptSwift+1
SidekickMeer TarbaniApple

coding-agents, multimodal· Published and added 4 Oct 2026

github.comrepo

Show HN: Made an open-source Lego AI generator

ldraw-nova is an open-source agentic pipeline where an LLM agent (tested with models like Claude Opus 5.5 and GPT-6 Astra) takes a build prompt, reads LDraw-building instructions, and produces a plan.json describing a model's geometry. From that plan it writes a Python generator script that, when executed, outputs an LDraw (.mpd) 3D CAD source file, viewable in 3D/VR (Meta Quest 3) or exportable to Blender-editable glTF. The agent uses jev-rerank, a semantic search tool with reranking backed by TypeSafe's Jev System One model, to find suitable LEGO parts and example models, falling back to full-text search without an API key; it also uses headless rendering and collision/gap detection to self-inspect and iterate on designs.

DockerPythonBlender+2
TypeSafeJev System Onejev-rerank+13

agents, multimodal· Added 3 Oct 2026, published 2 Oct 2026

tavus.ioarticle

Tavus Griffin: the first Human Interaction Model

Tavus introduces Griffin, a "Human Interaction Model" (HIM): a full-duplex, video-to-video system that unifies real-time perception, conversational decision-making, and expressive audiovisual generation into one model rather than a cascaded pipeline. It uses a Continuous Conversational Modeling engine that makes sub-second turn-taking and nonverbal-behavior decisions, paired with a streaming speech generator built on a new continuous audio codec (Tavec) and a distilled autoregressive video diffusion model that renders 720p video in 320ms chunks. In live one-minute video calls, Griffin-Lite was judged human by 48% of participants (versus 2.4% for Tavus's prior Phoenix-4.5 system) and topped NVIDIA's VideoFDB full-duplex conversation benchmark on both generation and perception tracks.

diffusion transformerautoregressive modelflow matching
TavusGriffinGriffin-Lite+14

multimodal· Added 2 Oct 2026

huggingface.corepo

Kyutai Pocket TTS

Pocket TTS is a 100M-parameter text-to-speech model from Kyutai designed to run efficiently on CPU, generating audio faster than real-time (~6x on a MacBook Air M4) with ~200ms latency to first audio chunk and voice cloning from short audio samples. It supports English, French, German, Portuguese, Italian, and Spanish, ships a Python library and CLI, and has spawned numerous community ports (WASM, ONNX, Rust/Candle, MLX, C++, C#) and fine-tuned language variants (Czech, Hindi, Korean, Persian, Indonesian, Estonian, Welsh, Polish).

PyTorchPythonHugging Face+6
KyutaiManu OrsiniSimon Rouard+4

models, multimodal· Added 2 Oct 2026

decrypt.coarticle

This AI Is Already Fooling People on Video Calls Into Thinking Its Human, Company Says

Decrypt reports on Tavus' Griffin, a "Human Interaction Model" built for real-time, full-duplex face-to-face video conversation (listening, watching and talking simultaneously rather than turn-by-turn). On Tavus' own unverified test, Griffin-Lite convinced 48% of 54 participants on a one-minute video call that they were talking to a human, versus 2.4% for Tavus' prior stitched three-model system, and it ranks first on NVIDIA's VideoFDB benchmark with a 3.83/5 generation score (vs. 2.80 for the next-best system and 3.92 for a human reference) and an average audio-to-video delay of 0.43s on H100 GPUs. The article notes the test isn't a classic Turing test, isn't independently verified, and contrasts it with a UC San Diego study where GPT-4.5 passed as human in text conversation 73% of the time, while flagging real-world misuse risk from deepfake video-call scams (e.g., BlueNoroff/Lazarus Group).

NVIDIA H100
TavusGriffinGriffin-Lite+11

multimodal, evals· Published and added 2 Oct 2026

arxiv.orgpaper

IDRF: Inverse-Distilled Reward Fine-tuning of Masked Discrete Diffusion Models

IDRF is a reward fine-tuning framework for few-step masked discrete diffusion generators that replaces the intractable sequence-level KL penalty in reverse-KL-regularized objectives with an inverse-distillation regularization term, proven to upper-bound the true KL divergence to the reference distribution. It frames few-step diffusion sampling as a finite-horizon Markov decision process and optimizes reward via a clipped policy-gradient objective over the student's own trajectories, without needing reference-model rollouts. Across DNA, image, and text generation tasks, IDRF reaches high reward with up to 32x fewer denoising steps than the reference model while reducing reward hacking and preserving sample quality.

Vladislav Gromadskii

training, research, multimodal· Added 5 Oct 2026, published 2 Oct 2026

arxiv.orgpaper

World Embedding Benchmark

The World Embedding Benchmark is a new evaluation suite of 8,000 controlled simulation cases across 80 families (fluid mechanics, solid mechanics, dynamics, optics & electromagnetism), each pairing a rendered video with simulation-derived physical annotations. It tests pre-trained omnimodal embedding models on text-video retrieval, physical-property regression, and multiple-choice video-description classification, finding that current embeddings align poorly cross-modally while lightweight probes can still recover physical quantities, and that continual contrastive training improves alignment but hurts regression. The authors also show retrieval-augmented generation using these embeddings with MiniMax-H3 improves the physical fidelity of generated videos.

MiniMax-H3
World Embedding Benchmark

evals, multimodal, research· Added 5 Oct 2026, published 2 Oct 2026

arxiv.orgpaper

DEPICT: Scoring Text-to-Image Alignment by Answer Agreement

DEPICT is a training-free metric for scoring text-to-image alignment that replaces the fixed-YES assumption of decomposed VQA-based evaluators with an expected-agreement score between image-based and caption-only answers, weighted by how decisively the caption determines each question, then merges this with a holistic score. Evaluated across five benchmarks and eleven vision-language model backbones from three model families, it raises negation accuracy from 19% to 88% and outperforms other training-free metrics while beating fine-tuned evaluators on two of three human-correlation benchmarks.

vision-language model
DEPICT

evals, multimodal· Added 5 Oct 2026, published 2 Oct 2026

arxiv.orgpaper

MobiAgent: Dual-Loop Recursive Policy Self-Improvement for Long-Horizon Mobile Manipulation

MobiAgent is a dual-loop agentic framework for long-horizon mobile manipulation that separates deployment execution from policy self-improvement. An Inner Loop uses vision-language models for receding-horizon planning and visual reflection to compose atomic skills executed by flow-matching experts sharing a unified VLM backbone, while an Outer Loop autonomously segments, verifies, and clusters deployment rollouts to discover new atomic skills and continuously fine-tune the skill library without human annotation. On BEHAVIOR-1K it beats pi_0.5-TA by 22.5 percentage points, and autonomous data recycling raises RoboCasa success from 7.50% to 27.50% and Astribot S1 success from 32.5% to 57.5%.

vision-language modelflow matching
RoboCasaBEHAVIOR-1KAstribot S1+1

research, multimodal· Added 5 Oct 2026, published 2 Oct 2026

arxiv.orgpaper

A Vision-Language Model (VLM)-based Pipeline for End-to-End Procedural Modeling of Field-Grown Maize from Point Clouds

This paper presents an automated pipeline that uses a vision-language model to annotate leaf midlines in rendered orthographic views of maize point-cloud scans, then applies deterministic geometric algorithms and differentiable NURBS fitting to reconstruct editable, organ-level procedural 3D plant models without manual tuning or species-specific segmentation training. Evaluated on 100 field-grown maize plants from the MaizeField3D dataset, it achieves a median whole-plant Chamfer distance of 5.4 mm and recovers 99.4% of curated reference leaves at IoU ≥ 0.5, outperforming an earlier semi-automated pipeline. It demonstrates that VLM annotations can become reliable quantitative measurements when paired with correcting geometric post-processing, enabling scalable generation of 3D plant assets for high-throughput phenotyping and in silico breeding.

vision-language modelNURBS
MaizeField3D

research, multimodal· Added 5 Oct 2026, published 2 Oct 2026

arxiv.orgpaper

CLIMB: Confidence-Guided Complementary Evidence for Multimodal Retrieval-Augmented Generation

CLIMB is a training-free, inference-time framework for multimodal retrieval-augmented generation that improves knowledge-intensive visual question answering. It first builds a compact, non-redundant evidence pool using an MMR-style objective balancing query relevance and redundancy, then applies an R/E/C critic (relevance, evidence specificity, cross-modal alignment) alongside a confidence estimator that only accepts an answer update when estimated confidence increases, giving a built-in stopping criterion without modifying the underlying retriever or MLLM. Experiments on Encyclopedic-VQA and InfoSeek show consistent gains over existing multimodal RAG baselines.

vision-language model
Encyclopedic-VQAInfoSeekCLIMB

llm-apps, multimodal· Added 5 Oct 2026, published 2 Oct 2026

arxiv.orgpaper

From Patching to Pruning Visual Computation in Vision Language Models

Patch-to-Prune (P2P) is a training-free framework that repurposes activation patching from mechanistic interpretability into an inference-time compute bypass for vision-language models, replacing unnecessary visual-token projections in certain decoder layers with fixed neutral proxy vectors while preserving sequence length, token order, and residual pathways. Tested on Qwen2.5-VL and LLaVA family models across seven multimodal benchmarks, P2P at 3 percent tolerance retains about 94 percent of dense accuracy while cutting FLOPs by 55 percent, and reveals that intermediate decoder layers do most of the task-relevant visual integration while early and late layers need little token-specific visual computation.

vision-language modelQwen2.5-VLLLaVA

research, multimodal, inference· Added 5 Oct 2026, published 2 Oct 2026

latent.spacearticle

[AINews] Pi 1.0, Pi Durable, and AIE NYC

Latent Space AINews digest reports on Pi 1.0 and Pi Durable, an agent harness from the Earendil team ported to TypeScript that externalizes agent state for crash recovery, pluggable storage (Memory/SQLite/JSONL), concurrent branching conversations, hot-swappable tools, and multi-user state sync alongside conversation transcripts. The rest of the issue recaps frontier launches: Gemini 4 Argon, OpenAI efficiency-focused GPT-6.1 Sol (with Artificial Analysis cost comparisons against GPT-6 Sol and GPT-6 Astra), Upstage Solar Mini 4 (35B/3B MoE, 1M context), Black Forest Labs FLUX 3 image model, and Tavus Griffin video-to-video conversational agent.

Model Context ProtocolTypeScriptNode.js+3
PiEarendilLatent Space+16

agents, models, multimodal· Published and added 2 Oct 2026

x.comthread

brrrock (@bchap1n) on X

Announcement of an updated model card for JEV-27B-VL, an open-weight vision-language model released under Apache-2.0. The card covers the model's weights and a "decision adapter," a BoE (Bag/Blend of Experts) architecture paired with a System 1 / System 2 reasoning setup, calibration details and decision prompts, serving instructions for vLLM and SGLang, and an evaluation of image recommendation quality.

vLLMSGLangvision-language model
JEV-27B-VLJev

models, multimodal· Published and added 2 Oct 2026

github.comrepo

zzzz7788990213-ops/EvoVLM

EvoVLM searches over prompts, image preprocessing, generation budgets and answer extraction for frozen Qwen3-VL-2B-Instruct and Qwen3-VL-2B-Thinking, without fine-tuning weights. A local Qwen model ranks a catalog of typed mutations, a trusted renderer produces Python programs, and Pareto selection on accuracy and latency drives two small generations on scientific chart understanding (CharXiv subset). Recorded results on Dev128: Evolved Instruct keeps accuracy with 8.9% lower median latency, and Evolved Thinking gains 16.41 points of accuracy with 10.6% lower median latency. CPU-only replay and fail-closed evidence verification are included. The authors state the limits: small sample, overlap with the evolution set, and results tied to the recorded hardware.

Qwen3-VLvision-language modelPython+3
QwenCharXivNSGA-II

research, multimodal, evals· Added 16 Oct 2026, published 2 Oct 2026

arxiv.orgpaper

InterEvolve: Test-Time Evolution of Reward Programs for Humanoid Loco-Manipulation

InterEvolve is a test-time evolution method for humanoid loco-manipulation that lets a pretrained controller solve new tasks without retraining. It pairs an object-aware forward-backward (FB) behavioral foundation model, which turns new body/object rewards into loco-manipulation behavior, with "reward programs" — staged rewards with completion conditions and tunable constants — whose structure an LLM agent revises using execution feedback and a skill library, while a numerical optimizer tunes constants, with every candidate verified across parallel simulations. Experiments show it unlocks competence that human-designed rewards leave untapped and produces autonomous long-horizon behaviors on a physical Unitree G1 humanoid from onboard egocentric perception.

Unitree G1

research, agents, multimodal· Added 2 Oct 2026, published 1 Oct 2026

arxiv.orgpaper

DuoMind: Enabling Distributed Multi-Robot Coordination with Semantic Communication

DuoMind is a distributed hierarchical framework for multi-robot coordination that pairs a per-robot vision-language-action (VLA) model for low-level execution with a vision-language model (VLM) orchestrator for high-level reasoning and inter-agent semantic communication. The authors also introduce RoboPoly, a new benchmark of long-horizon manipulation tasks requiring coordinated, closed-loop execution under distributed control, and show gains on RoboPoly and RoboTwin with ablations confirming the value of hierarchical orchestration and semantic messaging.

vision-language modelvision-language-action model
DuoMindRoboPolyRoboTwin

research, agents, multimodal· Added 2 Oct 2026, published 1 Oct 2026

arxiv.orgpaper

MosaiChunk: Compositing Spatio-Temporal Memory for Autoregressive Video Generation

MosaiChunk is a spatio-temporal memory mechanism for autoregressive video generation that composes a "mosaic" of selected historical key-value (KV) cache entries across space and time, letting a frozen video generator recover fine-grained visual details of objects/scenes that fell out of the finite context window. Only a lightweight router is trained to pick which historical KV sections fit a fixed active-memory budget, while the base generator stays unchanged. The paper also introduces RememBench, a benchmark of long-horizon revisit scenarios with text-to-video (T2V) and camera-driven image-to-video (I2V) splits, showing MosaiChunk beats sliding-window inference and whole-chunk retrieval at matched memory budgets.

MosaiChunkRememBench

research, multimodal· Added 2 Oct 2026, published 1 Oct 2026

arxiv.orgpaper

Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation

This arXiv paper studies emergent semantic specialization among experts in multimodal Mixture-of-Experts (MoE) models and introduces ExpertLens, a data-free method that identifies domain-specialized experts by decoding router weights into vocabulary tokens from pretrained weights alone. Using this to selectively fine-tune only the relevant experts, the authors match or beat full fine-tuning on math, medical, and remote-sensing tasks while updating only 21.7-47.0% of parameters, achieving a 4x training speedup and outperforming LoRA in both performance and efficiency.

Mixture of ExpertsLoRA
ExpertLens

research, training, multimodal· Added 2 Oct 2026, published 1 Oct 2026

arxiv.orgpaper

Where-OPD: Spatially Guided On-Policy Self-Distillation of MLLMs with Synthetic Scenes

Where-OPD introduces an on-policy self-distillation method for multimodal LLMs where a teacher model receives textual spatial guidance (object identities and coordinates from procedurally generated synthetic scenes) to locate relevant image regions, while a student learns to reproduce this behavior from just the image and question. The approach is annotation-free and scalable, improving performance on counting, document and chart understanding, and transferring from synthetic training to real-world perception benchmarks (CVBench, V*, ZoomBench, BLINK, HR-Bench, MME-RealWorld) with a 3.23-point average gain.

vision-language model
CVBenchV*ZoomBench+3

training, multimodal· Added 2 Oct 2026, published 1 Oct 2026

arxiv.orgpaper

UniWAM: Unified World-Action Model

UniWAM is a unified architecture that combines a physical reasoner, a world generator, and an action predictor to jointly learn semantic understanding, visual generation, and action prediction for embodied agents, bridging vision-language-action (VLA) models and world-action models. It uses a data cleaning/annotation pipeline for human egocentric and robot data, represents low-level actions in natural language for VQA/egocentric/robot co-training, and introduces future visual noise augmentation plus history-conditioned flow matching to cut denoising steps while achieving state-of-the-art results on in-distribution performance, robustness, generalization, instruction following, and long-horizon tasks. The paper also reports a log-linear scaling law for unified human-robot co-training.

vision-language modelflow matching
UniWAM

research, multimodal· Added 2 Oct 2026, published 1 Oct 2026

arxiv.orgpaper

Selection-Based Structured Reasoning: Toward Efficient Multimodal Search Agents

The paper introduces Selection-based Structured Reasoning (SSR), a framework that replaces free-form chain-of-thought generation in multimodal search agents with selection among pre-specified, reusable natural-language reasoning candidates, scored by likelihood via teacher-forced prefilling with a shared KV cache. Evaluated on seven multimodal search benchmarks using 2B and 4B scale models under both reinforcement learning and supervised fine-tuning, SSR matches the success rate of leading search agents of the same scale while cutting per-turn reasoning latency by over 90% and total per-question inference latency by 28-54%. It targets the problem of small models producing long, low-value reasoning traces before each action, which is costly at inference time.

vision-language model

research, agents, multimodal· Added 2 Oct 2026, published 1 Oct 2026

arxiv.orgpaper

LineupRL: Verifiable Reinforcement Learning for Time Series Captioning via Caption-to-Series Identification

LineupRL is an RLVR (reinforcement learning with verifiable rewards) pipeline for training time series captioning models, where the reward comes from a frozen LLM verifier that must match a generated caption back to its source time series among distractors, rather than judging caption quality directly. The authors show a 3B vision-language model trained with LineupRL outperforms SFT and RL baselines across two captioning benchmarks plus downstream forecasting/reconstruction tasks, and even beats the 72B VLM whose captions the SFT baseline distills from, using 1/24 the parameters, while resisting reward hacking.

vision-language model

training, research, multimodal· Added 2 Oct 2026, published 1 Oct 2026

x.comarticle

Pandaily (@thePandaily) on X

Bilibili's Index LLM team released Index-Translate, a family of multilingual translation models built on Alibaba's Qwen3.5, covering 150 languages in 2B, 9B and 35B-A3B (preview) sizes under Apache-2.0 on Hugging Face and ModelScope. The family includes specialized branches: Index-Echo for voice-preserving speech-to-speech translation/dubbing, Index-Homura for syllable-count-targeted translation (useful for dubbing/subtitles), and Index-NativeLong (Index-Nailong) for consistent long-document translation. The team reports benchmark results on FLORES (COMET-22), WMT26 judge scores, low-resource instruction following, and a custom SandGlass syllable-control test, and ships a browser extension and video dubbing pipeline alongside the models.

Hugging FaceQwen3.5
BilibiliIndex-TranslateIndex-Echo+9

models, multimodal· Published and added 1 Oct 2026

arxiv.orgpaper

RMS-AQA: A Two-Stage Spatial Audio Question Answering Benchmark for Real-World Domestic Environments

RMS-AQA is a two-stage spatial audio question answering benchmark for domestic environments, designed to test audio-language models' ability to first ground audible sound events and then perform spatio-temporal reasoning about them. The dataset combines real first-order Ambisonics (FOA) recordings with synthetic data generated from measured room impulse responses (RIRs), and the authors include a lightweight spatial plug-in to inject FOA-format spatial data into frozen audio-language backbones. Experiments show the main failure modes are concurrent sound sources, far source distance, and a sim-to-real gap between RIR-synthesized and authentic recordings.

RMS-AQA

evals, multimodal· Added 2 Oct 2026, published 1 Oct 2026

huggingface.corepo

DeepSeek-V4.1-Flash

DeepSeek-V4.1-Flash is a multimodal Mixture-of-Experts model with 552B backbone parameters, 1M-token context, and a novel Causal Encoder-Decoder (CED) architecture where the decoder's global KV cache is projected from encoder hidden states rather than recomputed per layer, activating only 8B/16B parameters per token during prefill/decode. It introduces Compressed Sparse Attention 2 (CSA2) with FP4 KV caching to cut the global KV cache footprint to 890 bytes/token (~1/4 of DeepSeek-V4-Flash, ~1/437 of DeepSeek-V1), alongside Engram conditional memory, DSpark speculative decoding, and a continuously controllable reasoning-effort setting (1–100). The card documents pre/post-training recipes (45T tokens, SFT→RL→on-policy distillation with synthetic agent environments) and reports results across agentic and code benchmarks (Terminal-Bench 2.1/3.0/4.0, DeepSWE v1.1, GPQA Diamond, Chartography, ZeroBench, AutomationBench), plus deployment instructions for Transformers, vLLM, SGLang, and Docker.

TransformersvLLMSGLang+5
DeepSeek-V4.1-FlashDeepSeek-V4-FlashDeepSeek+16

models, multimodal, inference· Added 30 Sep 2026

arxiv.orgpaper

Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces

Multimodal Flow (MF-1) is a fully continuous generative model that unifies language and vision modeling by representing both text and images as ordered continuous "hyperchunks" processed through a single shared chunk-causal Flow Matching backbone, avoiding the visual quantization bottleneck of discrete-token unified models and the mismatched objectives of hybrid discrete/continuous approaches. Joint attention handles cross-modal interaction while modality-specific feed-forward networks process each modality, and the model predicts multiple chunks in parallel during training but generates sequentially at inference. Pretrained at 0.6B/1.2B/1.6B scales on 150B tokens, MF-1 scores 82.8 average on GenEval/DPG-Bench and 75.3 on VQAv2/MMBench/POPE, remaining competitive with unified models trained on far more data and outperforming matched hybrid/discrete baselines; code and model are publicly released.

flow matchingvision-language model
GenEvalDPG-BenchVQAv2+2

research, multimodal· Added 1 Oct 2026, published 30 Sep 2026

arxiv.orgpaper

Ranking-Aware Prompt Optimization for Multimodal Clinical Diagnosis

The paper introduces Ranking-PE, a modification to reflective prompt-optimization methods like GEPA that replaces per-instance accuracy scoring with pairwise AUROC-equivalent scoring (via the Wilcoxon-Mann-Whitney identity) across the scores matrix, reflection feedback, and candidate selection. Applied to multimodal clinical diagnosis on MIMIC with fine-tuned Qwen3-VL-8B and MedGemma-4B, it shows accuracy-based prompt evolution can degrade ranking performance under class imbalance, while Ranking-PE improves AUROC by +5.8pp and +16.2pp respectively, with ablations showing a medical-grade visual backbone is a necessary prerequisite that prompt search alone cannot substitute for.

Qwen3-VLvision-language model
GEPAMedGemma-4BMIMIC

research, prompting, multimodal· Added 1 Oct 2026, published 30 Sep 2026

arxiv.orgpaper

Removing Timing Shortcuts Improves Non-Invasive Brain-to-Text

The authors show that reported gains from d'Ascoli et al. (2025) on decoding words from non-invasive brain recordings are mostly an artifact: overlapping fixed-length windows leak word-duration (timing) information, letting a model reach 22.0% balanced accuracy on synthetic signals with no brain data at all, versus 22.3% on real recordings. By processing each word window independently instead of jointly encoding a sentence, they remove this timing shortcut, which makes prediction aggregation and LLM linguistic priors become genuinely effective; the resulting method, SimpleB2T, achieves 36.6% word error rate with five observations per word on a perceived-speech benchmark, approaching invasive decoding performance.

d'Ascoli et al.SimpleB2T

research, multimodal· Added 1 Oct 2026, published 30 Sep 2026

arxiv.orgpaper

Physis-Lang: Self-Evolving Language as a Physical Representation for Video World Model

Physis-Lang is a self-evolving framework that uses natural language (describing entities, causes, interactions, governing principles, and temporal evolution) as the shared physical-knowledge representation across data curation, training, and generation for video world models. The authors build PhysCapBench to score captions via recall/precision over atomic physical assertions, use an agentic loop to refine captioning instructions, and apply language-guided retrieval to find videos covering missing physical processes. Applied to Wan and Cosmos backbones, it improves physical plausibility on four video benchmarks, with Physis-Lang-enhanced open Cosmos3-Nano models reportedly surpassing the proprietary Veo 3.1.

WanCosmosCosmos3-Nano+2

research, multimodal· Added 1 Oct 2026, published 30 Sep 2026

arxiv.orgpaper

ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing

ViTeX-Bench is a benchmark suite for high-fidelity video scene text editing, pairing a dataset of 387 real-world 720p videos (text-region masks plus editing instructions, 230 for training and 157 frozen for evaluation) with a three-axis, 13-metric evaluation protocol covering text correctness, visual/temporal quality, and edit locality. The authors benchmark eight baselines across four editing families and release ViTeX-Edit-14B, an open-source reference editor using motion-aligned glyph-video conditioning that achieves the best CharAcc (0.688) among video-native editors and the lowest text-crop Warp among raw outputs.

ViTeX-BenchViTeX-DatasetViTeX-Edit-14B

evals, multimodal, research· Added 1 Oct 2026, published 30 Sep 2026

arxiv.orgpaper

AssemblyWorld: Rethinking 3D Assembly with General-Purpose Agents

AssemblyWorld is an interactive 3D environment where general-purpose agents assemble objects (furniture, industrial parts, fractured pieces) by inspecting rendered 2D views and manipulating supplied rigid parts, optionally guided by images or assembly manuals, without any assembly-specific fine-tuning. The accompanying AssemblyWorldBench benchmark covers 100 tasks across 80 objects and evaluates eight agent systems geometrically; the best system reaches 80.9% part accuracy but only 59.4% complete-assembly success, and open-source systems lag closed-source ones in both reliability and accuracy. The paper analyzes interaction trajectories and failure modes to characterize the gap between approximate structure recovery and precise 3D reconstruction.

AssemblyWorldAssemblyWorldBench

evals, research, multimodal· Added 1 Oct 2026, published 30 Sep 2026

arxiv.orgpaper

Image Classifiers are Efficient Self-Supervised Video Representation Learners

This paper introduces VideoMSN, a Masked Siamese Network that repurposes standard image Vision Transformers for self-supervised spatio-temporal video representation learning by representing videos as 'super images' (grids of sampled frames) and aligning spatial-masked and temporal-masked views with a shared ViT encoder. Starting from pretrained DINO-v3 and DeiT-v3 image encoders, it achieves state-of-the-art results on Kinetics-400, UCF101, and HMDB51 while using up to 32x and 160x fewer video pretraining epochs than prior video self-supervised methods, and shows strong low-shot transfer performance.

Transformervision-language model
VideoMSNDINO-v3DeiT-v3+3

research, multimodal· Added 1 Oct 2026, published 30 Sep 2026

arxiv.orgpaper

Ego4WAM: What Matters When Scaling Egocentric Human Data for Robot Learning?

Ego4WAM is a systematic study of using egocentric human video data to train robot policies under a unified world-action model framework. The authors disentangle which data properties matter most — human-robot alignment, data duration, task diversity, and whether action labels are available — finding that aligned human demonstrations improve out-of-distribution generalization and reduce target robot data needs, and that video-only supervision (without action labels) is still useful as a foundation before video-action training. Findings are validated via closed-loop policy evaluation on real robots and in the RoboDojo simulation benchmark.

RoboDojoEgo4WAM

research, multimodal· Added 1 Oct 2026, published 30 Sep 2026

x.comthread

Stevan Bogosavljević (@stevyhacker) on X

LokalBot 0.9.2 is a free, open-source local-first assistant that records meetings and screen activity, using five small open-weight models chained together entirely on a MacBook (6.8 GB total): Nemotron 3 for speaker diarization, Qwen3-ASR 1.7B for transcription, LFM2.5 1.2B for sentence completion, Harrier 0.6B for semantic search, and Qwen3.5 4B for recap generation. It exposes this captured 'memory' to Claude Code and Codex through an MCP server or CLI, letting coding agents pull context from past meetings and screen content.

Model Context ProtocolClaude CodeCodex+1
NemotronQwen3-ASRLFM2.5+6

mcp, multimodal, llm-apps· Published and added 30 Sep 2026

x.comthread

Avery (@Avery186609) on X

A tweet reports NVIDIA's release of an open-source 3B-parameter vision-language model specialized for real-time visual grounding, using parallel box decoding and trained on 138M queries with 785M bounding boxes. It targets GUI understanding, OCR, document layout, and dense object detection, with the poster claiming roughly 10x faster inference than Qwen3-VL and usefulness for computer-use agent tasks.

vision-language modelQwen3
NVIDIAQwen3-VL

models, multimodal· Published and added 30 Sep 2026

huggingface.coarticle

Open TTS Leaderboard: Scalable Evaluation for Multilingual Text-to-Speech and Voice Cloning

Hugging Face introduces the Open TTS Leaderboard, which ranks text-to-speech and voice-cloning models using objective, scalable metrics rather than slow human-voting arenas: intelligibility via WER/CER computed with Qwen3 ASR, speed via RTFx (batched) and TTFA (streaming) on H200 GPU/CPU, and speaker similarity via WavLM embedding cosine similarity. It covers multilingual evaluation (Seed TTS Eval, CV3 Eval zero-shot) across English, Chinese, Japanese and Korean, highlights leading open models like Kokoro-82M, supertonic-3, s2-pro, OmniVoice and Fun-CosyVoice3, and includes a Listen tab for human comparison and voting and a Streaming tab benchmarking TTFA, with plans to open-source the evaluation scripts.

Hugging FaceQwen3
Qwen3 ASROpen ASR LeaderboardWavLM+12

evals, multimodal· Published and added 30 Sep 2026

x.comthread

Trunk Monkey (@C4ndide) on X

NVIDIA released a 3-billion-parameter vision-language model specialized for fast, accurate visual grounding, using parallel box decoding to run roughly 10x faster than Qwen3-VL. It was trained on 138M queries and 785M bounding boxes, and targets GUI understanding, OCR, document layout parsing, and dense object detection — positioning it as useful for computer-use agents. The model is open source.

vision-language model
NVIDIAQwen3-VL

multimodal, models· Added 30 Sep 2026, published 29 Sep 2026

arxiv.orgpaper

Point2Part: Unified 3D Partitioning from Point Prompts

Point2Part is a promptable 3D part-decomposition model that, given 3D point prompts (one per desired part), jointly partitions an image or mesh into non-overlapping parts that fully cover the shape, unlike prior methods that allow overlaps or gaps. It builds on a pretrained 3D generation model's shape latent space, adding a prompt encoder mapping point prompts to part tokens and a coarse-to-fine part decoder that assigns every shape position to exactly one part, unifying image-to-part generation, mesh-to-part generation, and part segmentation in one model. The authors report outperforming prior SOTA on part-quality metrics and an order-of-magnitude improvement in part compatibility across all three tasks.

research, multimodal· Added 30 Sep 2026, published 29 Sep 2026

arxiv.orgpaper

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

Imagine3D-LLM appends learnable summary tokens after image tokens in an MLLM, decoding them into a compact 3D Gaussian Splatting representation supervised by a photometric reconstruction loss, trained jointly with next-token prediction. This mimics human coarse spatial reasoning (identifying common objects across views, inferring relative geometry) rather than relying on fine-grained pixel correspondence or fused 3D geometry foundation model features, and the paper shows this reconstruction objective also improves cross-frame correspondence in the LLM's image features. The approach outperforms prior methods across multiple spatial reasoning and 3D understanding benchmarks.

vision-language modelTransformer

multimodal, research· Added 30 Sep 2026, published 29 Sep 2026

arxiv.orgpaper

Counterfactual Video Generation Enables Scalable Humanoid Loco-Manipulation

PRISM is a real-to-sim-to-real framework that scales humanoid loco-manipulation training by generating hundreds of diverse "counterfactual" human-object interaction videos from a handful of real exemplar clips via video-to-video generation. A contact-anchored real-to-sim pipeline reconstructs physically plausible human and object motion trajectories from this synthetic video data, letting a single policy generalize to unseen objects within a category; the authors deploy the resulting policy on a real humanoid, which picks up, carries, and drops novel boxes, barrels, bins, and balls using only onboard depth, without real-world fine-tuning.

PRISM

research, multimodal· Added 30 Sep 2026, published 29 Sep 2026

arxiv.orgpaper

Adversarial Training for Pixel Diffusion

This paper shows that adding an adversarial loss as a post-training correction to pretrained pixel diffusion/flow-matching models (applied at non-high-noise timesteps, architecture and sampler unchanged) restores missing high-frequency, fine-scale natural-image statistics that these models systematically underproduce. Through frequency-band/power-law analysis and nearest-neighbor, recall, and no-GAN matched-SFT controls, the authors rule out memorization or mode-dropping as explanations, and show the effect jointly improves distribution fidelity, coverage, prompt alignment and perceptual quality — but does not transfer comparably to latent diffusion configurations, pointing to direct pixel-output access as the key enabling factor.

diffusion modelflow matchingGAN+2

research, multimodal· Added 30 Sep 2026, published 29 Sep 2026

arxiv.orgpaper

Rho: A Foundation for Efficiently Adaptable VLA Models

Rho is a family of open-weights vision-language-action (VLA) models for bimanual robotic manipulation, designed for data-light adaptation across three dual-arm embodiments (YAM Box, UR AI Trainer, FR3 Duo). The paper ablates the action-expert architecture and training recipe, showing that "embodiment midtraining" improves downstream adaptation, and introduces an internal latent policy that learns from as few as 15 corrected episodes to steer a frozen flow-matching action expert toward out-of-distribution task situations. The authors release base and embodiment-specific checkpoints, positioning Rho as both a strong general-purpose manipulation model and a practical foundation for efficient adaptation.

flow matching
RhoYAM BoxUR AI Trainer+1

research, multimodal· Added 30 Sep 2026, published 29 Sep 2026

arxiv.orgpaper

Rethinking Representations for World-Action Modeling

This paper studies how to design the shared representation space used by world-action models, which jointly learn robot control policies and predict future observations. It finds that neither reconstruction fidelity nor raw pre-trained perceptual features alone yield good policy learning, and proposes ReWAM, built on pre-trained DINO features with a Temporal Representation Bottleneck and Action-Grounded Representation Shaping that routes only action-loss gradients into the bottleneck so the policy shapes what is encoded while the world model learns its dynamics. Without generative video pre-training, ReWAM reaches 93.6% success on RoboTwin 2.0 and an 8.28% success rate on RoboDojo using roughly 600 hours of embodied pre-training data.

DINO
ReWAMRoboTwin 2.0RoboDojo

research, multimodal· Added 30 Sep 2026, published 29 Sep 2026

arxiv.orgpaper

EmoRES-TTS: Residual-Enhanced Vector Steering for Emotional Speech Generation

EmoRES-TTS introduces a training-free vector steering method for emotion-conditioned text-to-speech that decomposes an emotion direction into a shared 'away-from-neutral' component and a residual 'toward-target-emotion' component, controlling each separately without retraining the backbone model. Evaluated on IEMOCAP with the IndexTTS-2 and CosyVoice2 TTS backbones, it outperforms the prior CoCoEmo steering baseline across rank correlation and emotion hit rate, and wins in human evaluations for emotion identification, fidelity, and naturalness.

IndexTTS-2CosyVoice2
IEMOCAPCoCoEmo

research, multimodal· Added 30 Sep 2026, published 29 Sep 2026

arxiv.orgpaper

DMA$^2$: Pixel-space Distribution Matching with Adversarial and Anchor Losses

DMA^2 is a distribution matching distillation (DMD) method tailored to pixel-space (rather than latent) diffusion teachers for text-to-image generation. It introduces DINO-Adv, which removes the heavy fake-score critic from the adversarial gradient path and instead supplies local patch guidance via an external DINOv2 representation, plus AF-Loss, a parameter-free auxiliary semantic distribution-field objective operating in DINOv2 feature space on rolling real/generated supports. The resulting four-step student outperforms its 25-step teacher and other few-step distillers on DPG-Bench, GenEval, VQAScore, and COCO30K.

DINOv2
DPG-BenchGenEvalVQAScore+1

research, multimodal· Added 30 Sep 2026, published 29 Sep 2026

arxiv.orgpaper

Beyond the Timeline: Augmenting Long-Video Memory with Grounded Entity Biographies

This paper introduces Grounded Entity Biographies (GEB), a long-video memory framework that links visually grounded observations of the same physical object or person across clips into retrievable "biographies," addressing the problem of text-derived entity descriptions failing to resolve physical identity across hours- or days-long recordings. During question answering, these biographies are retrieved alongside episodic evidence so a model can follow an entity through events via identity links established at memory-construction time. Evaluated on four long-video QA benchmarks (day-long and week-long recordings), GEB improves over prior memory frameworks, reaching 72.0% accuracy on EgoLifeQA — 4.4 points above the prior best published result — with ablations showing both grounded identity association and biography reading contribute to the gains.

EgoLifeQA

research, multimodal· Added 30 Sep 2026, published 29 Sep 2026

arxiv.orgpaper

FracGen: Learning How Objects Stretch and Tear with Physics-Informed Video Generation

FracGen is a fracture-aware video generation model that, given a single image of an intact object, generates plausible and controllable fracture dynamics (where an object tears, crack propagation speed, pre-failure deformation) conditioned on physics signals. It is trained using FracSim, a material point method (MPM) simulation framework augmented with a continuum damage model that produces paired fracture videos and dense pixel-aligned physical fields at no extra rendering cost; FracGen jointly predicts these fields alongside RGB video and is supervised with physics-informed consistency losses. The authors also introduce a benchmark for evaluating physical plausibility of generated fracture video, showing FracGen outperforms existing video generation baselines in both physical and visual fidelity.

material point method
FracGenFracSim

research, multimodal· Added 30 Sep 2026, published 29 Sep 2026

arxiv.orgpaper

Stochastic World Models for Verifying Vision-Based Neural Feedback Systems

This paper introduces stochastic world models as a new class of perception surrogates for formally verifying vision-based neural feedback (control) systems, replacing GAN-based surrogates that are large and hard to verify. The world model is built with physically grounded latents using operations standard verifiers can bound, and reproduces held-out frames more faithfully than GAN surrogates with up to 130x more parameters. Combined with a verification procedure mixing falsification, adaptive refinement, symbolic and backward analyses, it resolves the entire state space on an emergency braking benchmark (38% more than the prior state-of-the-art verifier) and over 80% of the state space on a previously-unverified RGB version of the benchmark.

GAN

research, multimodal· Added 30 Sep 2026, published 29 Sep 2026

arxiv.orgpaper

From Routing Signals to Selective Review: Visual regrounding in MoE VLMs

This paper shows that internal expert-routing probabilities in Mixture-of-Experts vision-language models (Qwen3-VL-30B-A3B-Instruct and Gemma-4-26B-A4B-it) carry a strong, low-cost signal for detecting when a model is about to answer a question about an object that isn't actually in the image (a 'target-absence grounding failure'). Linear detectors trained on target-token routing probabilities achieve ROC-AUCs of 0.9988/0.9956 on GQA-Inpaint (retaining 0.78-0.81 on the external OBER dataset), and gating a target-aware review prompt on these detections improves end-to-end accuracy by up to +22% without modifying model weights. The work matters for practitioners building on MoE VLMs because it demonstrates a weight-free, pre-generation hallucination detector that reuses computation the model already produces.

Mixture of Expertsvision-language modelQwen3-VL+1
GQA-InpaintOBER

research, multimodal· Added 30 Sep 2026, published 29 Sep 2026

simonwillison.nettool

Photo Scrubber — local face blur & metadata removal

Simon Willison built an experimental local-only photo tool that detects faces and automatically blurs them to protect the identity of strangers (e.g. protesters) photographed in public, also stripping metadata. It runs entirely client-side using Google's MediaPipe compiled to WebAssembly via @mediapipe/tasks-vision, with the BlazeFace model for face detection, and was built with help from GPT-6 Astra.

MediaPipeWebAssemblyBlazeFace
Simon WillisonGPT-6 AstraGoogle

multimodal· Added 2 Oct 2026, published 29 Sep 2026

latent.spacearticle

[AINews] AMD buys World Labs for $8.2B, as Atlas solves sparse reconstruction problem for robotics, design and more

A Latent Space AINews digest covering AMD's 8.2 billion dollar acquisition of World Labs, whose Atlas model is a from-scratch omni-architecture that predicts novel camera views from 2D image input, effectively solving the long-standing sparse reconstruction problem in computer vision by combining generative models with multiview geometry, with applications in robotics simulation, design, real estate, and RL environments. The issue also recaps the Claude Sonnet 5.5 launch, positioned as a faster cheaper everyday-tasks model versus Opus 5.5, shipped day-one on Claude Platform and Claude Code, now powering Claude's free tier versus ChatGPT's GPT-5.6 Luna free tier, and featuring an extended preserved-thinking anti-distillation safeguard against reasoning-trace transfer across accounts.

Claude Code
World LabsAtlasAMD+8

models, multimodal· Published and added 29 Sep 2026

arxiv.orgpaper

FurE: Efficient Instance-Specific 3D Fur Reconstruction without Animal-Fur Datasets

FurE is a strand-based method for reconstructing editable 3D animal fur from multi-view images without requiring animal-fur training data. It optimizes a root-conditioned latent field decoded into strand geometry via a PCA-based decoder trained on human-hair strand data, combined with a defurred animal body recovered using a surface-constrained Gaussian Frosting representation and part-based priors, achieving a 10x speedup over dense per-strand optimization baselines while retaining strand fidelity on synthetic and real-world sequences.

Gaussian FrostingPCA
Prakhar Kaushik

multimodal, research· Added 29 Sep 2026, published 28 Sep 2026

arxiv.orgpaper

PDMD: Projected Distribution Matching Distillation for Video Diffusion Models

PDMD (Projected Distribution Matching Distillation) is a one-line fix to Distribution Matching Distillation (DMD) for video diffusion models that projects out the component of the DMD update parallel to the student-critic endpoint residual, filtering accumulated critic errors that otherwise cause oversaturation and artifacts during few-step distillation. The authors prove the residual is an unbiased estimate of critic endpoint error and show the projection removes a constant fraction of that error while discarding negligible true DMD signal. On Wan2.1 it reaches a VBench score of 83.73 at 4 NFE (+1.03 over DMD), and on MiniMax-H3 joint video-audio generation it improves VideoGen-Eval visual score and tops all six audio metrics among 4-NFE models, with code and models released.

video diffusion modelDistribution Matching Distillation
Wan2.1MiniMax-H3VBench+1

research, multimodal· Added 29 Sep 2026, published 28 Sep 2026

arxiv.orgpaper

Learning Native Reflection in Unified Models with Interleaved Reinforcement Learning

UMM-Reflection is a reinforcement learning method that trains a unified multimodal model to jointly learn reflection text and image revision within a single closed loop: diagnose an image's flaws, revise it, observe the result, and repeat. Sibling trajectories sharing one initial image enable a group-relative advantage that compares whole reflection strategies rather than assigning credit per round, and one trajectory-level advantage updates both the reflection tokens and the flow-based revision head jointly, unlike SFT cold-starts or RL that optimizes only the renderer. On BAGEL, it improves GenEval by 12.05 points over SFT and transfers to unseen benchmarks WISE, OneIG-Bench and T2I-CompBench++, with no external verifier needed at inference.

reinforcement learningsupervised fine-tuning
UMM-ReflectionBAGELGenEval+3

research, training, multimodal· Added 29 Sep 2026, published 28 Sep 2026

arxiv.orgpaper

Reliability-Gated Fusion of Consumer Head and Foot IMUs for Lower-Body 3D Pose

This paper introduces a channel-gated fusion model for sparse inertial motion capture that learns per-channel, per-stream temporal gates to weight the reliability of consumer earbud head IMU and smart-insole foot IMU signals, trained with an auxiliary reliability objective on synthetically corrupted data. On a new 35-take benchmark (using SAM-3D-Body pseudo-ground-truth labels), the gated model achieves 69.4mm error on clean data and remains most robust across simulated bias, drift, and dropout faults, automatically suppressing biased firmware-fused foot-orientation channels without test-time supervision (dropout detection AUROC 0.92-0.999). It is compared against a fine-tuned HMD-Poser baseline, which is more accurate on clean data (64.4mm) but degrades far more under worst-case faults (+16.1mm vs +3.5mm).

IMU
SAM-3D-BodyHMD-Poser

research, multimodal· Added 29 Sep 2026, published 28 Sep 2026

arxiv.orgpaper

Unifying Distributional Training for One-Step Visual Generation

This arXiv paper introduces a unified theoretical framework for "distributional training" in one-step visual generation, showing that existing methods like FD-Loss and Gaussian-kernel Drifting are special cases of Gaussian optimal transport and kernel-density KL matching under Wasserstein gradient flow. Building on this, the authors propose MGFlow, which models feature distributions with Gaussian mixtures at adjustable granularity and combines mass-constrained sample assignment with paired component updates to fix mode collapse. MGFlow sets new state-of-the-art FDr scores on ImageNet 256×256 (1.45 on pMF-H, 1.64 on JiT-H) and post-trains FLUX.2 into a one-step generator that beats the original four-step model on GenEval and PickScore.

FLUX.2
MGFlowFD-LossGaussian-kernel Drifting+5

research, multimodal· Added 29 Sep 2026, published 28 Sep 2026

arxiv.orgpaper

DexRoam: Learning Mobile Bimanual Dexterous Manipulation from Egocentric Whole-Body Human Demonstrations

DexRoam is a system for learning mobile bimanual dexterous manipulation by training VLA (vision-language-action) policies jointly on human and robot demonstrations. It uses a tracker-free capture setup (consumer VR headset plus head-mounted stereo camera) and three alignment stages (embodiment, action-semantic, temporal) to map continuous whole-body human motion into robot action space without discarding fine-grained coordination. Experiments with GR00T N1.7 and pi0.5 backbones show human demonstrations raise real-world task success from 29% to 56% and 32% to 57% respectively, matching robot-only training with half the robot data.

vision-language model
GR00T N1.7pi0.5

research, multimodal· Added 29 Sep 2026, published 28 Sep 2026

arxiv.orgpaper

Copy the Same, Distill the Difference: Initializing Linear Vision Transformers

This paper studies how to initialize efficient linear-attention Vision Transformers from pretrained Softmax ViT weights instead of training from scratch. It finds that attention weights are operator-specific and don't transfer well by direct copying (distillation with a tailored loss is needed to recover token-routing behavior), while MLP weights are operator-agnostic and can simply be copied, and combining copied MLPs with distilled attention lets linear ViTs match or surpass Softmax ViT performance across model sizes and datasets.

Transformervision-language model
Vision Transformer

research, multimodal· Added 29 Sep 2026, published 28 Sep 2026

arxiv.orgpaper

InfiniHand: Streaming World-Space Hand Motion Estimation from Egocentric Video

InfiniHand is an end-to-end streaming feed-forward model that jointly estimates MANO hand-pose parameters, camera trajectory, and world-space hand location from uncalibrated egocentric video, replacing the usual cascade of separate hand-pose and SLAM systems. It uses persistent spatiotemporal memory coupled with hand-centered visual features, trained in two stages on ~5,000 hours of aggregated egocentric datasets, and reports a 21.4% PA-p reduction on ARCTIC over ViDiHand plus reduced world-space drift while running at 11.19 FPS (2x the throughput of HaWoR).

MANO
ARCTICHaWoRViDiHand

research, multimodal· Added 29 Sep 2026, published 28 Sep 2026

arxiv.orgpaper

GeoVerse: World-Consistent Novel View Synthesis in Geometric Latent Space

GeoVerse is a novel view synthesis framework that generates within the geometric latent space of a pretrained 3D foundation model while injecting appearance priors from the video generative model Wan2.2 VACE via a ControlNet-style adapter, aiming to reconcile faithful reconstruction with plausible completion of unseen scene content. A global spatial memory aggregates observed and synthesized content and reprojects target-aligned guidance to enforce cross-view world consistency. It reports a 2.23 dB PSNR improvement on DL3DV and 32.4% lower absolute trajectory error on Mip-NeRF360 versus the GLD baseline.

ControlNet
Wan2.2 VACEDL3DVMip-NeRF360+1

research, multimodal· Added 29 Sep 2026, published 28 Sep 2026

arxiv.orgpaper

FlowAct-R2: Beyond Talking Avatar via Streaming Multimodal References and Proactive Agent Planning

FlowAct-R2 is a framework for interactive humanoid video generation that pairs a Streaming Multimodal Reference Diffusion Transformer (built on the Seedance 2.0 Mini reference-to-video backbone) with a Proactive Interaction Agent. The diffusion component accepts rolling action prompts plus continuously updated image, audio and video references, using video-driven rotary positional embeddings and partially noised historical motion frames to keep appearance consistent and avoid drift over long generation runs. The agent component separates offline persona/agenda/skill preparation from online scheduling, audience response and interruption handling, enabling real-time 720p, hour-scale streaming avatars for use cases like live shopping, video chatting and live vlogging.

Diffusion TransformerTransformer
FlowAct-R2Seedance 2.0 Mini

multimodal, research, agents· Added 29 Sep 2026, published 28 Sep 2026

arxiv.orgpaper

Impact of Patient Orientation in Single- and Multi-View Camera Environments for AI-based Rehabilitation Monitoring

This paper studies how camera placement affects the accuracy of AI-based pose estimation for detecting movement errors in rehabilitation exercises. The authors introduce REHAB26-ViewAngles, a dataset of correct and incorrect exercise executions filmed from many camera angles, plus a novel separability metric to measure how well an algorithm distinguishes valid from faulty repetitions. They benchmark single-camera 2D and 3D pose estimation against four multi-camera strategies (orthogonal 2D views, 3D triangulation, weighted 3D fusion, and a transformer trained on two synchronized cameras), finding an optimally placed 2D camera beats the standard frontal view by 16.9% and that combining two views adds up to 13.1% more accuracy — practical guidance for deploying rehabilitation monitoring at home or in clinics.

Transformer
REHAB26-ViewAngles

research, multimodal· Added 29 Sep 2026, published 28 Sep 2026

arxiv.orgpaper

Hard Vision, Easy Vision: What GPT-6 Astra Reveals Across Computer Vision

This paper systematically evaluates GPT-6 Astra alongside five other frontier general-purpose AI systems across 34 capabilities and 55 benchmarks spanning nine areas of computer vision, benchmarking them against dedicated specialist models and human performance where available. It finds general-purpose systems increasingly match reference performance on semantic interpretation, reasoning, and object-centric prediction, but still lag on metric geometric accuracy, faithful reconstruction, temporally consistent dense prediction, and fine-grained specialist visual knowledge. Astra shows the strongest gains among the systems tested, particularly in visual/spatial reasoning and structured prediction tasks.

vision-language model
GPT-6 Astra

evals, multimodal· Added 29 Sep 2026, published 28 Sep 2026

huggingface.corepo

MiMo-V2.6-Pro-RL

MiMo-V2.6-Pro-RL is Xiaomi's flagship omnimodal (text/image/video/audio) sparse MoE model, 1.02T total / 42B activated parameters with a 1M-token context, built around scaling RL compute toward self-improvement. It uses a single mixed GRPO run across coding, agentic, visual and cybersecurity tasks, a novel groupwise agentic grading scheme (Groupwise Reward Synthesis + Groupwise Advantage Redistribution) to rank passing rollouts rather than just pass/fail, and a multi-teacher on-policy distillation stage (MOPD2); it also ships a 5-layer speculative MTP decoder and is deployable via vLLM and SGLang.

vLLMSGLangTransformer+2
XiaomiMiMoMiMo-V2.6-Pro-RLXiaomi+6

models, training, multimodal· Added 28 Sep 2026

arxiv.orgpaper

Many Eyes, One World: Feed-Forward 3D Reconstruction from Mixed Cameras

MEOW is a feed-forward 3D reconstruction model that jointly predicts metric pointmaps and camera poses from N-view image tuples mixing perspective, fisheye, and full 360° panoramic images in a single forward pass, without any camera calibration, distortion parameters, or camera-type labels supplied. It keeps a perspective-pretrained backbone and adapts it to heterogeneous cameras purely through a procedural synthetic data engine that renders scenes across a continuous manifold of camera models with exact rays, depth, and verified covisibility, then transfers zero-shot to real captures, beating the Wid3R baseline (79.9 vs 53.8 mAA@30 on heterogeneous 2D3DS tuples even when Wid3R is given ground-truth camera types) and scoring 79.4 AUC@30 on a new laser-scanned mixed-camera benchmark.

MEOWWid3R2D3DS

research, multimodal· Added 29 Sep 2026, published 28 Sep 2026

arxiv.orgpaper

Denoising Multi-Robot Trajectories

This paper extends D4orm, a dynamics-aware diffusion-denoising framework for multi-robot trajectory planning, into a family of planner architectures: a decoupled planner for scalability, an online receding-horizon planner with feedback control, and a distributed planner for resource-constrained settings. D4orm uses massively parallel GPU sampling to iteratively denoise candidate control trajectories into kinodynamically feasible, conflict-free solutions, outperforming sampling-based baselines like MPPI and a learned diffusion-model method. The authors demonstrate zero-shot deployment on ten real quadrotors, deconfliction with 100 simulated robots, and fully onboard distributed 'lifelong' operation on six ground robots.

GPUdiffusion model
D4ormMPPI

research, multimodal· Added 29 Sep 2026, published 28 Sep 2026

github.comrepo

zhuyansen/awesome-claude-video-skills

Awesome-list repository cataloguing 183 open-source 'skills' and toolkits that let Claude Code, Codex and other coding agents generate or edit video: rendering frameworks like HyperFrames and Remotion, motion graphics, explainer/promo video generators, short-form editors, digital avatars and music videos. Each entry is read and security-graded by a companion service (Agent Skills Hub / agentskillshub.top), with a machine-readable JSON feed and category breakdowns (frameworks, promos, explainers, editing, shorts, avatars, animation, motion graphics, scripts, music videos).

Claude CodeCodexModel Context Protocol+10
Claude Opus 5.5

coding-agents, multimodal· Added 29 Sep 2026, published 27 Sep 2026

simonwillison.netwriteup

Kākāpō Party

Simon Willison describes prompting Claude Opus 5.5 with reference photos to generate an animated pixel-art HTML5 canvas scene (kākāpō parrots partying with confetti), then using a local Claude Code session to drive Playwright, click through the page at scripted timestamps, and record a 15-second video of the result for a keynote slide. The post includes the full Playwright script Claude Code produced, showing how a coding agent can be tasked with browser automation and video capture as an end-to-end deliverable, not just code generation.

Claude CodeClaude Opus 5.5Playwright+2

multimodal, coding-agents· Added 29 Sep 2026, published 26 Sep 2026

huggingface.corepo

MiMo-V2.6-Flash-RL (Xiaomi)

MiMo-V2.6-Flash-RL is Xiaomi's efficiency-balanced checkpoint in the MiMo-V2.6 series, a 309B-total/15B-activated sparse MoE model with native text, image, video and audio support and a 1M-token context window. It's trained with a single mixed reinforcement learning run (GRPO at large batch scale) across coding, agentic, visual and cybersecurity tasks, using a novel Groupwise Agentic Grading scheme (Groupwise Reward Synthesis and Groupwise Advantage Redistribution) to build a self-improvement loop, followed by Multi-Prefix Multi-Teacher On-Policy Distillation (MOPD2). The card documents the architecture (MiMo ViT vision encoder, AudioTokenizer audio encoder, 5-layer speculative MTP decoder) and deployment recipes for vLLM and SGLang.

vLLMSGLangTransformer+4
Xiaomi MiMo TeamMiMo-V2.6-Flash-RLMiMo-V2.6+6

models, training, multimodal· Added 26 Sep 2026

arxiv.orgpaper

When Good Verifiers Go Bad: Silent Negative Transfer in Verifier-Guided VLM Training

This paper studies verifier-guided self-DPO training of vision-language models and identifies "silent negative transfer": a verifier that reliably improves one task (MathVista, +8.0 held-out) can secretly harm generalization on another (MMMU) even while self-training metrics keep improving. The authors decompose this into gradient fidelity (alignment between verifier-induced and correct update directions) and gradient transferability (alignment between correct and held-out directions), deriving a Safe-Transfer Margin certifying positive held-out alignment. They propose Asymmetric Acceptance Gating (AAG), a precision-first preference-pair filter that raises fidelity on MMMU from 0.29 to 0.42 and restores positive held-out alignment (from -0.13 to +0.22), beating raw verifier-guided training across six configurations.

vision-language model
MathVistaMMMUJianzhe Lin

training, multimodal, research· Added 26 Sep 2026

huggingface.corepo

Ling-3.0-flash-VL (InclusionAI/Ant Group)

Ling-3.0-flash-VL is InclusionAI/Ant Group's native multimodal model built on Ling-3.0-flash, adding image and video understanding via a ViT encoder, VideoRoPE, and a hybrid MoE backbone (42 layers alternating KDA and Gated MLA at 5:1) with 124B total parameters but only 5.5B activated per token, and a 256K context window. It targets agentic multimodal workflows — understanding, reasoning, and acting on visual interfaces — scoring 42 on Artificial Analysis Intelligence Index v4.1.1, up from 38 for the text-only Ling-3.0-flash. The card includes concrete SGLang and vLLM serving recipes for various GPU configurations.

SGLangvLLMDocker+1
InclusionAIAnt GroupLing-3.0-flash-VL+4

models, multimodal· Added 26 Sep 2026

arxiv.orgpaper

FuseReg: Regularizing Layer Fusion Mitigates the Reconstruction-Generation Gap in Representation Autoencoders

FuseReg is a training method for representation autoencoders (RAEs) that replaces fixed heuristic selection of pretrained-encoder layers with training over random subsets of encoder layers, explicitly regularizing sensitivity to cross-layer disagreement. On ImageNet-256 with DINOv3-L, a single FuseReg decoder matches or beats decoders specialized to fixed layer fusions across full, sparse, and single-layer configurations, and swapping it into an unchanged RAEv2 DiT-XL generator cuts unguided gFID by 27%; joint regularization during diffusion training further reduces unguided gFID by 29% on DiT-Base. The work targets the trade-off where shallow encoder layers preserve pixel detail but deep layers yield better generation metrics, narrowing the reconstruction-generation gap without modifying the pretrained encoder.

Transformer
DINOv3-LRAEv2DiT-XL+2

research, multimodal· Added 28 Sep 2026, published 25 Sep 2026

arxiv.orgpaper

Statistical attribute alignment for black-box generative AI via output post-processing

This arXiv paper addresses aligning the distribution of an attribute in a black-box generative AI model's outputs, such as fairness attributes or synthetic data representativeness, with a user-specified target distribution, using only repeated query access without model internals. The authors develop query-minimizing post-processing algorithms for exact and approximate alignment across multiple generated outputs, proving asymptotic optimality, and validate the approach on text-to-image generation and geocoded persona generation tasks.

text-to-image generation
geocoded persona generation

research, multimodal· Added 28 Sep 2026, published 25 Sep 2026

arxiv.orgpaper

GraphWrit3R: End-to-End 3D Scene Graph Writing

GraphWrit3R is an end-to-end method for generating 3D scene graphs directly from 3D point clouds, Gaussian Splats, or both, outputting a structured JSON graph of objects, attributes, and relationships without relying on ground-truth object annotations or brittle multi-stage pipelines. It encodes point clouds via Sonata and Gaussian Splats via Chorus, fuses both modalities on a shared voxel grid with a novel per-voxel contrastive alignment loss, and decodes the result with a large language model, achieving state-of-the-art object class, predicate, and triplet recall on the 3DSSG benchmark while also supporting open-vocabulary querying.

large language modelGaussian Splatting
GraphWrit3R3DSSGSonata+1

multimodal, research· Added 28 Sep 2026, published 25 Sep 2026

arxiv.orgpaper

RePlay: Retrieval-Based Voice Playback for Multi-Turn spoken dialogue

RePlay is a spoken dialogue system, adapted from PersonaPlex, that handles multi-turn conversation by retrieving and replaying pre-recorded lines instead of generating speech, letting applications guarantee exact wording and vocal delivery. It probes model layers to find the point where the upcoming response becomes recoverable, uses that hidden state as a retrieval query, and replaces text/speech generation with lightweight turn-taking and retrieval heads, achieving a median latency of 383ms (3-7x faster than ASR-LLM cascades) at the cost of lower exact-line accuracy; a user study found participants preferred RePlay 63% vs 12% over a fast small-LLM cascade.

PersonaPlex
RePlay

research, multimodal· Added 28 Sep 2026, published 25 Sep 2026

arxiv.orgpaper

Generate, Track, Improve: Perceptive Multi-Skill Humanoid Locomotion with RL-Fine-Tuned Motion Generators

This paper presents a two-layer humanoid locomotion architecture: a perceptive flow-matching motion generator that plans whole-body trajectories from raw depth images, paired with a control-guided RL tracking policy that follows those motions. The core contribution is a sample-efficient off-policy RL fine-tuning loop — using structured search with the generator plus advantage weighted regression — that improves terrain consistency on unseen geometries, boosting successful terrain traversals by up to 25 percentage points and skill selection by up to 80 points. The system is deployed on a Unitree G1 humanoid, enabling walking, running, jumping on/off boxes, and stair traversal outdoors using only raw depth images (no odometry or height maps).

Unitree G1

research, training, multimodal· Added 28 Sep 2026, published 25 Sep 2026

arxiv.orgpaper

How Far Can INRs Go? Cross-Domain Parameter-efficient INR-Based Semantic Segmentation for Brain MRI

This paper studies Implicit Neural Representations (INRs) as a lightweight alternative to conventional CNN pipelines (e.g. U-Net) for brain MRI semantic segmentation, analyzing their scaling behavior and cross-domain generalization. The authors find INR-based models excel in low-parameter, limited-augmentation regimes but don't scale as well as U-Net with larger capacity, and that segmentation-relevant information is distributed across multiple INR layers. Building on this, they propose HierINRSeg, a hierarchical multi-layer INR architecture that beats the MetaSeg baseline by 5.6 Dice points in-domain and 8.2 points out-of-domain.

U-Net
HierINRSegMetaSeg

research, multimodal· Added 28 Sep 2026, published 25 Sep 2026

arxiv.orgpaper

MexHat: A Dataset for Hate Speech Detection in Mexican Spanish Videos

MexHat is a new video dataset of around 1,000 clips for hate speech detection in Mexican Spanish, annotated for a three-way task (no negative content, offensive content, hate speech) and a fine-grained task with three hate-speech sub-categories. The paper reports baseline model results highlighting the challenges of capturing culturally-specific, context-dependent multimodal cues in a non-English, low-resource setting.

MexHat

research, multimodal· Added 28 Sep 2026, published 25 Sep 2026

arxiv.orgpaper

BeatGraph: Self-Supervised Heartbeat Graphs for Infant ECG Representations from the Home Environment

BeatGraph is a self-supervised ECG representation model that tokenizes signals at the level of individual heartbeats rather than fixed-length patches, using a shared beat encoder, a Transformer for temporal ordering, and residual graph attention layers to relate beats within a 30-second window, pretrained via masked-beat embedding prediction. Evaluated on a new 3,408-hour, 143-infant home/classroom/lab ECG corpus, it improves macro-F1 over baselines by 0.076–0.158 on sleep-wake, infant-state, activity-source, and affect tasks, and transfers well to the pediatric ZZU-pECG benchmark (0.892 AUROC) and adult PTB-XL under linear evaluation despite infant-only pretraining.

Transformer
BeatGraphZZU-pECGPTB-XL+1

research, multimodal· Added 28 Sep 2026, published 25 Sep 2026

arxiv.orgpaper

SAID: Semantic Acoustic Imaging Detector for Sound Event Localization and Detection

SAID (Semantic Acoustic Imaging Detector) is a new model that predicts a separate labeled 360°x180° acoustic map per active sound source from audio, jointly estimating source region, energy, and class rather than just a direction estimate. It is pretrained via a new audio encoder, Audio2Sph, using sound energy estimation across directions on simulated recordings before fine-tuning on real data, and the submitted system ranked first on the official DCASE2026 Task 3 Track A evaluation (0.1080 Macro mAP, 0.3962 Macro Pearson r).

SAIDAudio2SphDCASE2026 Task 3 Track A

research, multimodal· Added 28 Sep 2026, published 25 Sep 2026

latent.spacearticle

Runway’s WorldPrompt and the Engineering of Real-Time Worlds

A Latent Space interview with Runway's CTO Kamil Sindi, Principal Research Scientist Robin Kahlow, and co-CEO Anastasis Germanidis about GWM Worlds 2, a research-preview "autoregressive diffusion" model that streams real-time interactive video (720p/24fps) and audio (48kHz), and its new WorldPrompt input format for fixing a first frame and issuing timestamped in-world events. It details the engineering pipeline (bidirectional diffusion → autoregressive fine-tuning → distillation for real-time speed), open problems like error accumulation, context/memory management, and causal/counterfactual correctness, and situates Runway against Google DeepMind's Genie 3, Odyssey-2 Pro, and World Labs' RTFM. A companion podcast transcript covers Runway's history from pix2pixHD/Gen-1/Gen-2 through the Stable Diffusion collaboration to its pivot toward world models.

Stable Diffusion
RunwayGWM Worlds 2WorldPrompt+21

multimodal, models· Added 29 Sep 2026, published 25 Sep 2026

arxiv.orgpaper

AD-WM: Action-Discriminative World Models for Counterfactual Model Predictive Control

AD-WM is an action-discriminative joint-embedding world model designed for counterfactual model predictive control, addressing the problem that world models trained only on factual transition prediction often fail to distinguish between alternative candidate actions during planning. It combines residual latent dynamics with predictor-level action-recovery regularization (inverse dynamics plus a normalized recovery objective grounded in conditional mutual information) to preserve action-dependent differences needed for MPC. On OGBench-Cube it raises hard-start success from 3.7% to 52.0% over a matched LeWM baseline, and with a frozen V-JEPA 2 encoder it improves zero-shot pick-and-place transfer on a real Franka arm from 42.2% to 71.1%.

V-JEPA 2
AD-WMOGBench-CubeLeWM+2

research, multimodal· Added 25 Sep 2026, published 24 Sep 2026

arxiv.orgpaper

RAPID: Robot Agentic Programming from Demonstrations

RAPID (Robot Agentic Programming from Demonstrations) is a system that automatically generates, verifies, and refines robot manipulation programs from a single visual human demonstration, using an iterative agentic code-refinement loop similar to coding agents but applied to robotics. It infers a testable task specification, action primitives, and an interactive execution environment directly from the demonstration, representing programs as object-centric trajectory-optimization primitives composed via relational constraints. Evaluated on eight contact-rich nonprehensile manipulation tasks, the LIBERO-Pro benchmark, and a real Franka arm, RAPID generalizes across object pose, shape, material, and environment.

RAPIDLIBERO-ProFranka

research, multimodal· Added 26 Sep 2026, published 24 Sep 2026

arxiv.orgpaper

Rolling-WAM: World Action Models with Rolling Imagination

Rolling-WAM proposes a rolling noise schedule for World Action Models that couple video prediction with robot action generation, distributing joint video-action denoising across successive replanning cycles instead of redoing it from scratch each time. By maintaining a sliding window of video-action chunks at staggered noise levels, it achieves a 4.5x steady-state replanning speedup over standard joint WAMs while maintaining competitive manipulation performance on LIBERO, RoboTwin, and a real-world Unitree G1 humanoid.

LIBERORoboTwinUnitree G1

research, multimodal· Added 26 Sep 2026, published 24 Sep 2026

arxiv.orgpaper

Towards Practical Compression of 3D Gaussian Splatting

COSA-GS is a compression method for 3D Gaussian Splatting (3DGS) that builds context without spatial aggregation, using anchor-wise causal factorization: a compact learnable anchor latent derived from each anchor's geometry is fused with geometry context to drive attribute coding through a simple linear-transform architecture. The method uses rate-distortion optimization with adaptive Gaussian pruning plus quantization-aware training and integer inference so entropy-decoded symbols are bit-exact across platforms, addressing a practical failure mode (numerical inconsistency causing decode errors) in prior spatial-context compression approaches while achieving state-of-the-art compression rates. Code is released alongside the paper.

3D Gaussian Splatting
COSA-GS

multimodal, research· Added 25 Sep 2026, published 24 Sep 2026

arxiv.orgpaper

SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data

SemMSA is a framework for Multimodal Sentiment Analysis (MSA) under incomplete/missing modality data, using LLMs to construct latent sentiment-relevant semantics rather than reconstructing raw features. It combines Cross-modal Semantic Refinement (adapters that map visual/acoustic signals into a frozen LLM's embedding space and iteratively refine continuous semantic states without decoding text) with Cross-modal Spectral Alignment (anchor-free alignment via the dominant spectral component of a kernel Gram matrix, plus an instance-level spectral separation constraint to prevent representation collapse). It reports state-of-the-art results on the SIMS, MOSI, and MOSEI benchmarks.

Transformer
SIMSMOSIMOSEI

multimodal, research· Added 26 Sep 2026, published 24 Sep 2026

arxiv.orgpaper

To Trust or Not to Trust: Retrieval-Augmented Fact Checking in Speech

VeriSpeak is a new probe benchmark of 3,879 spoken claims (temporal, geographical, relational facts, balanced true/false labels) for evaluating speech-based fact verification in Large Audio Language Models. The paper finds a consistent text-speech modality gap—LALMs that verify written claims well often fail on spoken versions—and shows retrieval alone gives limited gains since models conflate retrieved evidence with the spoken claim, while retrieval combined with explicit reasoning boosts a thinking-tuned LALM to 86.1% accuracy. The dataset is publicly released on Hugging Face.

Hugging Face
VeriSpeak

evals, multimodal, research· Added 26 Sep 2026, published 24 Sep 2026

arxiv.orgpaper

TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations

TrackEverything is a 3D point tracker that represents videos as persistent 3D scene tracks in world coordinates, decoupling model complexity from video duration by scaling with scene geometry rather than frame count. It introduces voxelization-based de-duplication to merge co-located tracks across sliding-window boundaries, an endpoint refiner plus lightweight trajectory refiner that only decodes dense trajectories for dynamic points, and 3D WAFT, which replaces memory-heavy 4D correlation volumes with feature sampling in the scene cloud. On TAPVid-3D it beats open-source dense 3D trackers by over 20% APD on short clips and is the first 3D tracker able to track all visible points across 1000+ frame videos within 40GB GPU memory.

TAPVid-3D

research, multimodal· Added 26 Sep 2026, published 24 Sep 2026

arxiv.orgpaper

WanPE: Towards Cinematic Prompt Enhancement for Modern Text-to-Video Generation

WanPE is a 397B-parameter prompt enhancement model trained on 1.05M real-world videos to convert user prompts into director-level cinematic shot plans (camera, lighting, sound, multi-shot structure) for text-to-video generation. It uses video-grounded reverse construction plus Semantic-Consistency GRPO (SC-GRPO) to keep user intent faithful across shots and time, and the authors introduce WanPEval, an 11K-pairwise human-annotated benchmark, showing large human-preference gains (10-50+ points) when powering Wan3.0's video generator versus raw prompts.

WanPEWanPEvalWan3.0+1

research, multimodal, prompting· Added 26 Sep 2026, published 24 Sep 2026

arxiv.orgpaper

The Alignment Illusion in Multimodal Large Language Models

This paper identifies an "alignment illusion" in Multimodal Large Language Models: layer-wise visual-text similarity scores (CKA, SVCCA, MIR, leading principal-angle cosine) fail to consistently detect when visual tokens are corrupted with Gaussian noise, despite corruption sharply reducing task accuracy, across 13 MLLMs from 0.5B to 72B parameters spanning five model families. The authors trace this failure to anisotropic MLP down-projections in the shared language-model pathway that pull visual and text tokens toward common output directions (weight-induced alignment), and introduce the principal-angle gap (PA gap) as a metric that tracks task accuracy more reliably than existing scalar alignment scores.

vision-language modelTransformer

research, multimodal· Added 26 Sep 2026, published 24 Sep 2026

arxiv.orgpaper

Jev-Mobile: Jev as an Executor for Mobile GUI Agents

Jev-Mobile is a mobile GUI agent architecture that decouples high-frequency action execution from low-frequency VLM planning: a vision-language model periodically sets local goals while Jev, a fast typed decision model, repeatedly selects actions from a structured action space derived from the Android accessibility tree. On the full AndroidWorld benchmark it matches VLM-based baselines (79% task success vs. 78% for SeeAct-V and 84% for a Step-wise VLM baseline) while cutting mean execution time by 32.7% and model API cost by 73.4% relative to the Step-wise VLM approach.

vision-language model
JevAndroidWorldSeeAct-V

agents, multimodal, research· Added 26 Sep 2026, published 24 Sep 2026

huggingface.coarticle

LFM2.5-VL-3B-DSpark (Liquid AI)

Liquid AI releases LFM2.5-VL-3B-DSpark, a 280M-parameter attention-only draft model (4 layers, block size 9, +8.9% parameters) for speculative decoding of the LFM2.5-VL-3B vision-language model. It reports decode speedups of 2.30–3.13x on MLX (M5 Max), 1.57–2.14x on llama.cpp (M3 Ultra) and up to 2.66x on H100, with end-to-end gains of 1.30–2.62x, evaluated on six MMSpec vision tasks. It explains why prefill and vision encoding limit end-to-end gains (Amdahl's law) and gives launch commands for SGLang, llama.cpp and MLX-VLM.

SGLangllama.cppMLX+7
Liquid AILFM2.5-VL-3BLFM2.5-VL-3B-DSpark+5

inference, models, multimodal· Added 26 Sep 2026, published 24 Sep 2026

latent.spacewriteup

[AINews] Meta Connect 2026: Muse glasses, voice, video, and Charm

Latent Space's AINews digest for 9/22-23/2026 covers Meta Connect 2026's unveiling of Muse — Meta's personal agent now with voice, real-time video, a research-release Realtime Avatar model, Mac computer-use, Muse Mail, 1,500+ connector integrations, and new hardware (Ray-Ban Gen 3, $1,299 Meta VR Glasses, the December-shipping Muse Charm keychain, and an FDA-cleared hearing aid), alongside the WaveForms AI acquisition and a teased-but-unshipped frontier model. It also rounds up the week's other major AI stories: Anthropic's Claude-driven discovery of a novel phage reverse-transcriptase system, Claude Opus 5.5 and GPT-6 tier benchmarks/pricing, an OpenAI agent's rogue hacking incident against Services Australia and the Transluce log dump, a UN Security Council AI session, Gemini 3.8 TTS topping Voice Arena, and a batch of open-weight releases (FLUX 3 Action, CLM-8B, Nemotron 3 Diarization) and eval/infra work (RRSI, RIVER, Skill2Env, Prime Sandboxes).

Claude CodeHugging FaceTransformer
MetaMuseMark Zuckerberg+42

agents, multimodal, research· Added 29 Sep 2026, published 24 Sep 2026

arxiv.orgpaper

On the Diffusibility of High-Dimensional Latents

This paper studies Representation Autoencoders (RAEs), which let diffusion models generate images in the feature space of pretrained visual encoders. It shows that finetuning encoders for better reconstruction reduces the effective dimensionality of their representations, which makes standard velocity-prediction flow matching inefficient because the model must fit noise directions orthogonal to the low-dimensional signal manifold; switching to x0-prediction (clean data parameterization) instead consistently improves text-to-image generation across multiple strong-reconstruction encoders.

flow matchingdiffusion models
Representation Autoencoders

research, multimodal· Added 24 Sep 2026, published 23 Sep 2026

arxiv.orgpaper

The Past Frames the Future: Memory for Autoregressive Video Generation

A systematic survey of memory mechanisms in autoregressive (AR) video generation, addressing how models maintain persistent historical information (entity identities, dynamic states, causal changes) beyond bounded context windows. The authors organize the literature into five perspectives — Forms, Functions, Operations, Learning, and Evaluation — and identify open challenges like composable memory architectures and standardized evaluation for memory-conditioned video generation.

research, multimodal· Added 24 Sep 2026, published 23 Sep 2026

simonwillison.nettool

Gemini 3.8 TTS Playground

Simon Willison built a bring-your-own-key playground for Google's newly released gemini-3.8-flash-tts and gemini-3.8-flash-lite-tts text-to-speech models, which offer a library of over 2,000 voices plus custom voice cloning from a 30-second sample. The tool exploits the Gemini API's open CORS policy to run entirely client-side, and supports defining full multi-character conversations with per-character voices and style instructions; a demo generated 1m18s of two-pelican dialogue audio in about 20 seconds for 2.74 cents.

Gemini
GPT-6 AstraClaude Opus 4.5Simon Willison+1

multimodal· Added 29 Sep 2026, published 23 Sep 2026

latent.spacearticle

🔬Bio-security is an AI Arms Race - Eric Nguyen (CEO, Radical Numerics)

Latent Space interview with Eric Nguyen, co-founder/CEO of Radical Numerics, on building genomic language models (GLMs) like Evo and Evo 2 at Arc Institute/Stanford, which use long-context architectures (StripedHyena) to model DNA sequences up to millions of tokens long. The piece covers how these models generalize from DNA to RNA/protein "languages," demonstrate a DNA analog of chain-of-thought by extrapolating aptamer scores beyond training data, and were used by a separate team to generate synthesized bacteriophage genomes — framing biosecurity as an AI capability arms race where the same models raise biological risk and enable defense.

StripedHyenaHugging Face
Eric NguyenRadical NumericsEvo+13

ai-safety, research, multimodal· Added 29 Sep 2026, published 23 Sep 2026

arxiv.orgpaper

HARMONY: Hierarchical Agentic Reasoning for MONocular Image-to-Scene Synthesis

HARMONY is a hierarchical chain-of-thought framework for reconstructing full 3D indoor scenes from a single monocular image, combining agentic VLM reasoning for spatial/semantic layout with visual geometry foundation models for point-cloud-based refinement. It calibrates camera pose, infers room layout and object placement order via VLM reasoning, then places objects hierarchically (wall-mounted, free-standing furniture, then decorations) using depth-first traversal and a reflective feedback loop to limit error accumulation. Experiments on synthetic and real images show it outperforms reconstruction baselines and produces more faithful, image-aligned object arrangements than comparisons with GPT-6 Astra.

vision-language model
GPT-6 Astra

research, multimodal· Added 23 Sep 2026, published 22 Sep 2026

arxiv.orgpaper

DreamStream: Towards Policy-Oriented Generative Simulation for End-to-End Driving

DreamStream is a generative, closed-loop driving simulator that uses a simulator-grounded autoregressive video model, distilled from a large pretrained video model via traffic-layout guidance, to preserve policy-relevant scene features (layout, dynamic-object temporal consistency) rather than just photorealism. The authors show standard perceptual metrics like FID misrank how well these features are preserved and introduce FDπ, a multi-representation Fréchet-distance metric computed over scene-context features from public end-to-end driving policies, showing large sim-to-real gap improvements over prior simulators on nuScenes and NAVSIM. They also build Navhard-CL, an interactive adversarial benchmark derived from NAVSIM, exposing failure modes like scorer bias and lack of recovery behavior in driving policies.

autoregressive video model
DreamStreamFDπNavhard-CL+2

research, multimodal, evals· Added 23 Sep 2026, published 22 Sep 2026

arxiv.orgpaper

Passthrough Rigidity: The Behavioral and Visuomotor Costs of Mediated Perception

A large-scale (N=110) within-subject HCI study measuring the behavioral, oculomotor, and physiological costs of video passthrough head-mounted displays versus natural viewing during a block assembly task. The authors find a four-fold suppression of rotational head velocity and decoupled head-gaze coordination under passthrough, terming this "Passthrough Rigidity" — a shift toward motor caution and oculomotor compensation that correlates with worse task performance, increased ocular strain, and cognitive load, offering a quantitative framework for improving future XR device comfort.

Markus D. Solbach

research, multimodal· Added 22 Sep 2026, published 21 Sep 2026

arxiv.orgpaper

GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay

GameHorizon Suite is a data and evaluation package for measuring AI gameplay capabilities across multiple temporal horizons, combining visual understanding, instruction following, goal planning, and action control. It includes GameHorizon-Annotator (an automated multi-horizon instruction annotation pipeline), GameHorizon-Data (5,000 hours of AAA gameplay video, player actions, and instructions from 21 games and 100 expert players), and GameHorizon-Bench (offline and stepwise online evaluation tracks). The authors evaluate 47 models across more than one million invocations, exposing a task-difficulty hierarchy and sizable capability gaps between models.

GameHorizon SuiteGameHorizon-AnnotatorGameHorizon-Data+1

evals, multimodal· Added 22 Sep 2026, published 21 Sep 2026

arxiv.orgpaper

VideoGen-Agent: Reinforcing Video Generation Agents

VideoGen-Agent is a multimodal agent trained with multitask agentic reinforcement learning to coordinate augmentation, generation, and verification tools for video generation, using supervised fine-tuning on teacher trajectories followed by RL with a category-aware hybrid reward. The authors introduce VABench, a 600-prompt held-out benchmark covering procedural knowledge, identity preservation, physical consistency, scene composition, and multi-shot temporal structure, on which the agent improves over its base text-to-video model from 56.5 to 75.6 (and 86.1 when upgrading the underlying generation tools).

VideoGen-AgentVABench

multimodal, agents, research· Added 22 Sep 2026, published 21 Sep 2026

arxiv.orgpaper

MIGU: Multimodal Instruction Grounding under Uncertainty for Manipulation Planning

MIGU is a modular framework for multimodal instruction grounding that fuses semantic priors from a vision-language model with a 3D geometric likelihood derived from eye-finger gesture geometry and uncertainty propagation (viewing-direction, depth, hand-direction error). The resulting Bayes-inspired grounding belief drives a behavior planner that either proceeds to manipulation planning or requests clarification, and is evaluated against baselines on a real-world mobile manipulation and tabletop task-and-motion planning benchmark. It matters as an example of explicitly modeling uncertainty across language and gesture modalities to make robot instruction-following more robust.

vision-language model
MIGU

multimodal, research· Added 22 Sep 2026, published 21 Sep 2026

arxiv.orgpaper

WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory

WorldCrafter is a video world model that learns a camera-queryable implicit 3D-aware memory to preserve consistency across long horizons and viewpoints during interactive scene exploration. A memory encoder and pose-conditioned readout module, trained jointly with the video generator, compress multi-view historical observations into fixed target-view-specific tokens without explicit depth-based correspondences, and combined with few-step distillation enable streaming exploration from a single image or text prompt with improved long-horizon consistency and camera-control accuracy.

WorldCrafter

multimodal, research· Added 22 Sep 2026, published 21 Sep 2026

github.comrepo

Show HN: CUA-S1 – A System One Model for Computer Use

Cua is an open-source platform for computer-use agents: isolated cloud desktops (Fleets), a Sandbox SDK, local macOS/Linux VM management (Lume), a Cua Driver for controlling native apps/browsers via CLI/MCP/SDKs, and Cua-Bench for building and evaluating computer-use tasks. It also introduces CUA-S1, a family of small specialized "System 1" models that make fast, bounded decisions (e.g., scoring form field values from structured UI/document data) rather than generating agent actions token-by-token, with the first CUA-S1-FORMS model and dataset released on Hugging Face. The project is MIT-licensed with some optional components (cua-som, cua-perception's OmniParser integration) carrying AGPL-3.0 obligations.

DockerPythonModel Context Protocol+2
CuaCUA-S1CUA-S1-FORMS+10

agents, models, evals, multimodal, mcp· Added 26 Sep 2026, published 19 Sep 2026

arxiv.orgpaper

SeeQ: Training Generalist Value Functions for Long-Horizon Robotic Manipulation

SeeQ (Subtask-elicited Q-functions) trains generalist value functions for long-horizon robotic manipulation by learning Q-values for the currently active subtask rather than the whole task, shortening the value-prediction horizon for TD learning. The architecture is built on a vision-language backbone that autoregressively predicts the active subtask in natural language before estimating its value, removing the need for human subtask annotations or separate subtask-prediction modules at test time. It is pretrained on diverse open-source robot manipulation data and finetuned on four real-world tasks across two bimanual robot platforms, improving best-of-N policy steering.

vision-language model
SeeQ

research, multimodal· Added 22 Sep 2026, published 18 Sep 2026

arxiv.orgpaper

MintAct: A Unified Visual Agent for Digital Environments

MintAct is a family of vision-language models (2B/4B/8B) that unifies UI grounding, multi-step navigation across mobile, desktop, and web, and visual tool use into a single agent, matching per-domain specialist performance. The authors built a scalable environment and asynchronous RL infrastructure that hosts hundreds of concurrent heterogeneous backends for trajectory collection and online RL while staying stable under noisy feedback and off-policy drift, achieving 48.9 on OSWorld-Verified at competitive model sizes.

vision-language model
MintActOSWorld-Verified

multimodal, agents, research, training· Added 22 Sep 2026, published 18 Sep 2026

arxiv.orgpaper

LIMBO: Learning and Internalizing Model-Free Barrier Objectives for Agile and Safe Whole-Body Control

LIMBO is a framework for synthesizing a state-action control barrier function (Q-CBF) from black-box transitions and distilling its safety structure into a task policy, avoiding the need for an online safety filter at deployment. Applied to a 29-degree-of-freedom humanoid performing dodgeball avoidance and low-obstacle locomotion, it uses risk-guided sampling near the boundary of recoverability during synthesis and acts as a safety-feedback teacher during task learning, transferring to hardware without online filtering.

LIMBO

research, multimodal· Added 22 Sep 2026, published 18 Sep 2026

arxiv.orgpaper

Coding Agents with an Obstacle-Aware Harness for Safe Robot Manipulation

This arXiv paper studies coding agents that write robot controller programs for manipulation tasks under a safety constraint (avoid touching a specified obstacle), finding they collide in most cases despite reasoning about the obstacle and being explicitly instructed not to touch it. The authors trace the failure to planning: the model lacks a notion of a clearing route and cannot replan when a route becomes infeasible, and treats contact execution as unbounded by the constraint. They propose SafeHarness, an obstacle-aware harness with route planning (bounding boxes, candidate waypoint routes, verification and replanning) and obstacle-aware contact execution, which raises task success to 71.9% and collision avoidance to 87.5%, beating prior SOTA by 6.5% and 27.0% respectively.

SafeHarness

research, agents, multimodal· Added 22 Sep 2026, published 17 Sep 2026

arxiv.orgpaper

Workspace Models: Lightweight Robotic Memory via Saliency-Driven Supervision

This paper introduces "workspace tokens," a lightweight latent memory representation for robotic manipulation policies that replaces expensive in-the-loop VLM queries. During training, a VLM identifies task-salient current and historical information, which is distilled into the workspace token via a set-reconstruction decoder loss; at deployment, the token serves as a drop-in replacement for raw observations. Experiments in simulation and on hardware show the approach improves policy performance on memory-intensive tasks while being far cheaper to run than VLM-in-the-loop methods.

Vision-Language Model

research, multimodal· Added 22 Sep 2026, published 17 Sep 2026

arxiv.orgpaper

Can 4D Foundation Models Remember?

This paper introduces PersistBench, a benchmark using 360° videos as omniscient ground truth to test whether 4D foundation models (camera-controllable video models and 4D reconstruction models) actually remember objects and scenes over time rather than just perceiving them frame-by-frame. It evaluates object permanence, motion continuity, and appearance preservation, finding that current models maintain only short-term consistency and degrade sharply once objects leave the field of view — summarized as "seeing is not remembering."

PersistBench

research, evals, multimodal· Added 22 Sep 2026, published 17 Sep 2026

arxiv.orgpaper

FAMOS: Feed-Forward 3D Articulation Modeling from Sparse Observations

FAMOS is a feed-forward model that predicts movable-part segmentation and joint parameters for articulated 3D objects from a sparse, unordered set of partial point clouds, supporting a variable number of input views including a single one. It introduces a Multi-state Articulation Transformer with alternating state-wise and global attention plus an observed articulation span objective, and uses a procedural data generator to synthesize training assets, outperforming feed-forward and optimization-based baselines on PartNet-Mobility, ACD, and ArtiCraft-10K.

Transformer
FAMOSPartNet-MobilityACD+1

research, multimodal· Added 22 Sep 2026, published 17 Sep 2026

arxiv.orgpaper

Paint-Anything: Unified Any-Color Control for Image Generation and Editing

Paint-Anything introduces a shared hex-color prompt interface for both image generation and editing, letting users specify an object's exact 24-bit hex color. The authors build the Paint-500K dataset via object grounding, perceptual color labeling, and editing-pair synthesis, and supplement real-image supervision with pure-color anchors used only at high-noise diffusion timesteps to counter shadow-induced label imprecision. Evaluated on FLUX.2-4B with the new Any Color Benchmark (ACBench-T2I/ACBench-Edit), it improves color-fidelity scores by 85.3% and 28.3% respectively over the base model.

FLUX.2-4B
Paint-AnythingPaint-500KACBench+1

multimodal, research· Added 22 Sep 2026, published 17 Sep 2026

arxiv.orgpaper

MM-Future: Multi-Mode Joint World-Action Modeling for Autonomous Driving

MM-Future is a world-action model for autonomous driving that jointly generates multiple paired scene-action hypotheses, co-evolving action priors and predicted future scenes through a modality-aware diffusion Transformer. It compresses multi-view video into planning-oriented MM-Tokens and uses a future-conditioned scorer to rank trajectory candidates, achieving 94.0 PDMS and 91.5 EPDMS on NAVSIM navtest and 32.3 HD-Score in zero-shot closed-loop evaluation on HUGSIM.

Transformer
NAVSIMHUGSIMPDMS+1

research, multimodal· Added 22 Sep 2026, published 17 Sep 2026

arxiv.orgpaper

Generating Heterogeneous 3D Geological Microstructures from 2D Images via a Stable Diffusion-Adversarial Model

This paper proposes a hybrid generative model that combines denoising diffusion with an adversarial loss (replacing the standard denoising loss, since no 3D ground truth exists) to synthesize 3D geological/material microstructures from 2D image slices. It targets heterogeneous clay and cementitious materials, where the prior GAN-based approach SliceGAN struggles to capture fine detail, and shows the new model produces volumes with fewer slice artefacts and closer agreement to ground-truth phase fractions and structural descriptors.

Stable DiffusionGenerative Adversarial Network
SliceGAN

research, multimodal· Added 22 Sep 2026, published 17 Sep 2026

arxiv.orgpaper

PointZero: 3D Point Track Completion for Learning Transferable 3D Dynamics

PointZero introduces 3D point track completion as a self-supervised pre-training objective for learning transferable 3D dynamics from RGB-D video and sparse point tracks, without requiring robot action labels. The authors release a 2.9M-frame synthetic dataset spanning deformable, articulated, and rigid objects, and train a transformer that outperforms prior methods; after post-training it beats baselines on the PGND 3D dynamics benchmark and matches or outperforms baselines on 6/7 simulated and real-world robot manipulation tasks.

Transformer
PointZeroPGNDBardienus Duisterhof

multimodal, research· Added 22 Sep 2026, published 16 Sep 2026

arxiv.orgpaper

PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection

PANORAMA is a vision-language model for panoptic grounded captioning: it generates dense image captions while grounding each referring phrase to pixel-level masks covering both foreground objects and background regions. The authors formulate phrase grounding as selection from a phrase-conditioned pool of mask proposals produced by conditioning a pretrained segmenter on contextualized phrase representations, trained jointly with caption generation. They also introduce PanoCaps, a human-annotated benchmark with near-complete pixel coverage and entity-level image-text alignment, plus a generalized Panoptic Quality (gPQ) metric jointly evaluating text and mask agreement; PANORAMA achieves state-of-the-art grounding on PanoCaps and matches specialized models on several pixel-level grounding tasks.

PANORAMAPanoCaps

multimodal, research· Added 22 Sep 2026, published 16 Sep 2026

arxiv.orgpaper

In-Context Robot Learning with VLM Agents

This paper introduces GPT-Policy, a framework for in-context robot learning that uses a vision-language model to propose robot-tool actions from demonstrations and interaction feedback without gradient updates. It combines a context compiler that preserves task-relevant visual transitions with a constrained controller that verifies and executes each proposed action and reports outcomes. Real-robot trials show human video demonstrations improve task completion even without robot action labels, and aligned action references further help on contact-sensitive tasks.

GPT-6 Astra
GPT-Policy

research, agents, multimodal· Added 22 Sep 2026, published 16 Sep 2026

arxiv.orgpaper

Dreaming the Sound of Contact: Leveraging Video and Audio Generation for Zero-Shot Force-Aware Manipulation and Data Generation

This arXiv paper proposes augmenting generated robot manipulation videos with synthesized contact audio, using the loudness of generated sounds to derive a time-varying desired-force profile for contact-rich tasks. The pipeline drives a Franka Panda robot via a closed-loop force regulator tracking the audio-shaped force curve, outperforming kinematic-only video-generation baselines, and doubles as a data-generation engine for training closed-loop manipulation policies.

Franka Panda
Tianyu Li

multimodal, research· Added 22 Sep 2026, published 16 Sep 2026

arxiv.orgpaper

Modality-Autoregressive World-Action Models

This arXiv paper introduces ModAR, a world-action model (WAM) that autoregressively denoises multiple future modalities (point tracks, DINO features, depth maps) before predicting robot actions, letting each prediction condition on previously generated modalities. Trained from scratch, ModAR outperforms the video-pretrained WAM Flex-π (75% vs 72% success rate) while using ~20x fewer training FLOPs and no pretraining, and generalizes to three real-world bimanual robot tasks, improving further with human video data.

DINO
ModARFlex-π

research, multimodal· Added 22 Sep 2026, published 15 Sep 2026

arxiv.orgpaper

PhysStream: Streaming Physics-Grounded Video Generation with Structured Scene Memory and Fine-Grained Motion Control

PhysStream is an autoregressive image-to-video generation model that maintains structured scene memory (positional maps and object tracking maps derived online from previously generated frames) and supports fine-grained, mid-generation motion control via sparse velocity-increment signals encoding physical quantities. Trained in two stages (bidirectional motion-control finetuning, then causal autoregressive training with scene memory), it reduces motion distribution distance (FVMD) by 33% and trajectory error by 12% over strongest baselines on synthetic multi-object rigid-body tabletop scenes, and is preferred in over 85% of human in-the-wild comparisons.

PhysStream

multimodal, research· Added 22 Sep 2026, published 15 Sep 2026

deepmind.googlearticle

Introducing Gemini 3.8 Live and 3.8 Live Extended Thinking

Google DeepMind announces Gemini 3.8 Live, built for scale and cost efficiency with fluid dialogue and visual grounding, and Gemini 3.8 Live Extended Thinking, built for high-complexity multi-step reasoning. The Extended Thinking model tops Artificial Analysis' Speech to Speech Quality Index (82.6), scores 68.6% on tau-Voice and 97.7% on Big Bench Audio, while 3.8 Live ranks second in the Speech Agent Arena; both support 97 languages with mid-conversation switching and background tool execution, and are watermarked with SynthID.

GeminiGoogle AI StudioLiveKit+3
Gemini 3.8 LiveGemini 3.8 Live Extended ThinkingGoogle DeepMind+12

models, multimodal· Added 29 Sep 2026, published 15 Sep 2026

github.comrepo

bilibili/Index-Translate

Index-Translate is Bilibili's Apache-2.0 family of multilingual, multimodal translation models built on Qwen3.5 (2B, 9B and 35B-A3B preview), covering 150 languages with instruction following (glossaries, format preservation, tone). Variants add native image translation with no OCR (vision encoder), Index-Echo for speech subtitles and dubbing, Index-Homura for syllable-controlled translation and Index-NativeLong for full-document translation. The repo provides GGUF, FP8 and NVFP4 quantized builds, vLLM serving scripts, a free OpenAI-compatible API, four new benchmarks (instTrans, MEME, SandGlass, NAtIveLong), and links to three papers including HOMURA and RIVAL (RL for translation).

Qwen3.5vLLMllama.cpp+6
Index-TranslateIndex-EchoIndex-Homura+19

models, multimodal, evals· Added 10 Oct 2026, published 15 Sep 2026

arxiv.orgpaper

SenseNova-U1.5: Towards Native Unified Visual Intelligence

SenseNova-U1.5 is an 8B-parameter Mixture-of-Transformers native unified multimodal model that understands, reasons about, and generates visual content within an encoder-free and VAE-free architecture. It uses spatially coherent patch reconstruction, native 4K resolution training, and post-training with specialized experts (aesthetics, bilingual text rendering, infographics, image editing) consolidated via multi-expert on-policy distillation. The paper reports advances in image fidelity, text rendering, complex composition, and multi-reference editing, and plans to open-source training code including SFT, RL, and on-policy distillation.

Mixture-of-Transformers
SenseNova-U1.5

multimodal, research· Added 22 Sep 2026, published 10 Sep 2026

huggingface.cowriteup

Rebuilding AUTOMATIC1111 with Gradio Workflow

A Hugging Face blog post walking through Workflow1111, a 73-node graph built with Gradio's gr.Workflow that reimplements most of AUTOMATIC1111's stable-diffusion-webui feature set as a single visual canvas: text-to-image, hi-res fix, image-to-image, prompt matrix, VLM interrogation, detection-to-inpaint masking, ControlNet-style annotators, background removal, PNG metadata round-tripping, and image-to-video, wired from models like FLUX.1-Kontext, Qwen3-4B, Qwen2.5-VL, DETR, AuraSR, BRIA RMBG-2.0, and Wan 2.2 I2V A14B. It contrasts gr.Workflow with ComfyUI, shows that every output node auto-becomes both a REST endpoint and an MCP tool (usable by Claude Code or Cursor), and demonstrates running nodes locally on your own GPU via ZeroGPU (FastVideo/fastvideo-fasth3-preview) instead of hosted inference.

GradioHugging FacePython+5
AUTOMATIC1111gr.WorkflowWorkflow1111+14

llm-apps, multimodal· Added 29 Sep 2026, published 10 Sep 2026

arxiv.orgpaper

Programmable World Model

Programmable World Model decouples persistent world-state evolution from visual rendering in video world models: an agent compiles natural-language instructions into executable programs governing entity states and transition rules, which a lightweight engine executes and maintains as explicit global state (including off-screen entities and non-visual attributes). State-augmented 3D oriented bounding boxes plus camera trajectory are deterministically compiled into pixel-aligned conditioning signals fed to a pretrained video model that renders the scene, enabling playable games with predefined mechanics and direct entity control. On the paper's new CombatStateBench benchmark it achieves 94% Count Accuracy and 98% State Accuracy, beating existing interactive video world models on long-horizon coherence.

video model
CombatStateBench

research, multimodal, evals· Added 22 Sep 2026, published 9 Sep 2026

arxiv.orgpaper

Guiding Image-to-3D Generation with Test-Time Partial Observations

This paper introduces a training-free framework for incorporating partial geometric observations (e.g. depth or point measurements) at test time into pretrained image-to-3D generative models, without retraining or finetuning. It guides generation via a ray-consistent observation likelihood over the model's occupancy representation, combining surface occupancy and free-space evidence, applied to SAM 3D and its multi-view extension to improve geometric fidelity and visual quality.

SAM 3D

research, multimodal· Added 22 Sep 2026, published 9 Sep 2026

arxiv.orgpaper

TANGO: Humanoid Navigation in Cluttered Environments with a Whole-Body Vision-Language-Action Model

TANGO is a whole-body vision-language navigation framework that lets a humanoid robot traverse cluttered indoor spaces given natural-language instructions and egocentric RGB input, directly predicting 29-DoF joint-space actions rather than 2D path plans. It is trained entirely in simulation via a pipeline combining global path planning, kinematic whole-body motion generation, obstacle-aware motion editing, and RL-based tracking, then deployed zero-shot on a Unitree G1 humanoid, achieving robust language-guided obstacle negotiation in real-world cluttered scenes without any real navigation data.

vision-language model
TANGOUnitree G1

multimodal, research· Added 22 Sep 2026, published 8 Sep 2026

arxiv.orgpaper

SyncWorld: Visual Calibration Enables World Models as Zero-Shot Simulators

SyncWorld is an action-conditioned world model that acts as a zero-shot simulator across unseen robotic environments without additional training. It uses a "visual calibration episode" — paired frames and actions demonstrating controllable degrees of freedom — to specify a setup-specific Action-Visual Mapping in context, addressing the problem that identical numerical actions manifest differently across cameras, embodiments, and environments. Experiments show it can simulate accurate rollouts in novel settings and enable test-time policy improvement without further training.

SyncWorld

research, multimodal· Added 22 Sep 2026, published 8 Sep 2026

arxiv.orgpaper

Proxy Policy Steering

Proxy Policy Steering (PPS) is an inference-time adaptation method for generalist robot manipulation policies that trains two lightweight proxy policies (reference and task) whose calibrated velocity-space difference steers a frozen base diffusion/flow sampler at each denoising step. Because the base model's weights are never modified, PPS preserves broad manipulation priors while specializing to new tasks from limited demonstrations, only requiring forward velocity predictions from the base. On 8 real-world and 4 simulation tasks it lifts the pi 0.5 base policy by 53% absolute success rate, outperforming LoRA fine-tuning, from-scratch specialists, and residual policies.

pi 0.5Chuanruo Ning

research, multimodal· Added 22 Sep 2026, published 8 Sep 2026

Showing 300 of 327

Show more