Skip to content

🎨 Multimodal327

ListGrid
What you seeeverything
RecentOldestAddedA–Z
All types327Papers215Articles51Repos24Threads22
Write-ups7Tools6Advisories1Talks1
🎨 Multimodal
x.comthread

🚨 AI News | TestingCatalog (@testingcatalog) on X

RoboAPI is a public API from PokeBot, powered by its world-action model Poke-Aurora, that lets robot arms do new tasks from natural-language instructions with no task-specific data collection or training. A user connects a supported arm with a wrist-mounted camera, gives a task such as "clean the table", and receives the next moves in near real time. The post gives no benchmarks, supported hardware list or pricing.

RoboAPIPoke-Auroraworld-action model
PokeBotTestingCatalog

models, multimodal· Added 1 Nov 2026, published 31 Oct 2026

x.comthread

Artificial Analysis (@ArtificialAnlys) on X

Hyperflow H3 is Video Rebirth's 8-step LoRA for MiniMax H3, released in September 2026 with open weights. Artificial Analysis ranks it #5 on AA-Video-I2V v2.1 and #3 on AA-Video-Editing v1.1, ahead of MiniMax H3 (768p). It costs $9.00 per minute for Image to Video and $18.00 for Video Editing at 1088p, under half of Dreamina Seedance 2.5 ($34.12 and $40.82). The thread gives per-capability and per-use-case breakdowns.

LoRAdiffusion model
Artificial AnalysisVideo RebirthHyperflow H3+6

models, evals, multimodal· Added 31 Oct 2026, published 30 Oct 2026

huggingface.cowriteup

nemotron-halloween

MonsterHouse is a Halloween interactive projection built on the NVIDIA Nemotron Voice Agent: Nemotron Speech ASR, Nemotron 3.5 Bolt E4B 4GB with a LoRA persona adapter, Magpie TTS Zeroshot and Audio2Face-3D in Unreal Engine, on two RTX Blackwell GPUs. Median end-of-speech-to-audio is 1.70 seconds live and 0.72 seconds with cached responses. The post gives a build path and tips on caching and state handling.

Nemotron Voice AgentNemotron Speech ASRNemotron 3.5 Bolt E4B 4GB+11
NVIDIANemotronQwen3 30B-A3B+3

multimodal, agents, training· Published and added 30 Oct 2026

github.comrepo

Show HN: Overtone – Transcribe, correct and cut captions for any screen shape

Overtone is a caption format (.overtone.json), a CLI and a video.js plugin that store each spoken word once and cut captions separately per screen shape (16:9, 9:16 or custom). The transcribe pipeline combines gpt-transcribe, Azure AI Speech and optional ElevenLabs Scribe, resolves disagreements with an LLM vote, and uses an LLM editor for corrections. A small public benchmark reports 5.3% word error rate and $0.023–0.025 per audio minute.

video.jsNode.jswhisper.cpp+11
OvertoneOpenAIgpt-transcribe+6

multimodal, llm-apps, evals· Added 30 Oct 2026, published 29 Oct 2026

arxiv.orgpaper

RADIOlogy: A Single-Pass Approach to Training-Free Zero-Shot Anomaly Detection

RADIOlogy is a training-free, single-pass zero-shot anomaly detection method that uses multi-head attention pooling in vision-language models. It scores anomalies by contrasting average similarity between vision features and sets of normal and abnormal prompts, with no windowed execution. RADIO agglomerative features drive a dense refinement pipeline for pixel-level segmentation, and the method is applied to medical images.

CLIPRADIOvision-language model
RADIOlogyarXiv

research, multimodal· Added 31 Oct 2026, published 29 Oct 2026

arxiv.orgpaper

PhysCount: Do MLLMs Count Visual Appearances or Physical Entities?

PhysCount is a benchmark of 546 real-world images and 4,595 human-written questions in 651 target-specific groups. It tests whether multimodal LLMs can separate visible appearances (direct view, reflections, glass) from the physical entities behind them. Across 40 MLLMs, counting appearances is much easier than linking them to entities, and the best model answers every question correctly in only 13.36% of groups.

vision-language model
PhysCountarXiv

evals, multimodal, research· Added 31 Oct 2026, published 29 Oct 2026

latent.spacearticle

Can Frontier Models See Past a Decoy? Elorian Tests Visual Reasoning

DecoyCount is a benchmark from Elorian and Mercor with 1,660 items that tests whether models count objects or fall back on visual shortcuts. Frontier models scored below 50%: GPT-6 Astra led with 43.17%, ahead of Gemini-3.8-flash (32.71%), Qwen3.8-max (28.31%) and Claude-Fable-5.1 (25.22%). CEO Andrew Dai says Elorian's Glimpse model scored 50.48% in a single internal run without tools, and contrasts visual reasoning with world models such as Atlas and Genie 4.

vision-language model
ElorianGlimpseDecoyCount+17

evals, multimodal, models· Added 28 Oct 2026, published 29 Oct 2026

arxiv.orgpaper

Benchmarking color judgment in multimodal LLMs with CIE-computable ground truth

A five-task benchmark tests multimodal LLM color judgment against CIE colorimetric ground truth: AWB pair judgment, skin-tone bias direction, CCT estimation, Delta E 2000 pair ordering, and color naming under changing illuminants. Eight endpoints were tested. On ColorChecker-24 stimuli, closed frontier models reach CCT MAE of 715-1056 K versus 1358-2133 K for open-weight models, and a non-ColorChecker control shows this gap is partly template-dependent.

vision-language modelQwen3-VLGLM+2
GLM-4.5VGPT-5.4Claude-Opus-4.7+4

evals, multimodal, research· Added 31 Oct 2026, published 29 Oct 2026

huggingface.coarticle

Granite Switch 4.1.1 — Update models like you update software

Granite Switch 4.1.1 is a backwards-compatible patch release of IBM's modular Switch architecture, which configures a language model with specialized, cache-efficient adapters on one endpoint. It updates the Granite Speech ASR model (10 languages, native punctuation and capitalization) and the Decision adapter in granitelib-core, measured on the Jev Decision Index test set. Only the new adapters were retrained; the base model is unchanged. IBM plans tutorials and an Adapter SDK for third-party adapters.

Hugging Face
IBMGranite SwitchGranite Speech+2

models, multimodal· Published and added 29 Oct 2026

x.comthread

Muhammad Waris (@WarisCodes) on X

Nemotron 3.5 Transcribe is a free open-weight speech-to-text model from NVIDIA covering 40 languages, including mixed-language speech. The post says it ties for first among open models with a 2.8% error rate. It says the model needs a large GPU such as an H100 or RTX 5090, so it will not run on a laptop. The post gives no benchmark name or model card link.

Nemotron 3.5 Transcribe
NVIDIANemotron

models, multimodal· Added 30 Oct 2026, published 29 Oct 2026

arxiv.orgpaper

M-PFT: Angular Margin for Calibrated Prompt-Tuning of Fine-Tuned Vision-Language Models

M-PFT is an angular-margin method for prompt tuning of vision-language models that fixes base-to-novel dual miscalibration (underconfidence on base classes, overconfidence on novel classes). It adds two losses: minimizing the maximum cosine similarity between tunable class embeddings, and minimizing angular distance to each zero-shot embedding. Tests on 9 prompt-tuning methods and 15 datasets show lower novel-class calibration error at comparable accuracy and negligible overhead.

vision-language modelprompt tuning
arXiv

research, multimodal, training· Added 31 Oct 2026, published 29 Oct 2026

arxiv.orgpaper

Aligning Medical Vision Language Models at the Capability Boundary with Entropy Adaptive Rubric Grounding

MedCBR is a two-stage alignment framework for medical vision language models. Difficulty Weighted Boundary Preference Optimization (DBPO) trains on counterfactual preference pairs at the model's capability frontier. Entropy Gated Grounded Policy Optimization (EGPO) then applies distilled clinical rubrics according to prediction entropy. It reaches 61.2% average accuracy on six medical benchmarks, with the largest gains on differential diagnosis, and the two stages combine super-additively.

vision-language modelpreference optimizationreinforcement learning
MedCBRDifficulty Weighted Boundary Preference OptimizationEntropy Gated Grounded Policy Optimization+1

research, training, multimodal· Added 31 Oct 2026, published 29 Oct 2026

x.comthread

🚨 AI News | TestingCatalog (@testingcatalog) on X

Nemotron 3.5 Transcribe is an open-weight, promptable NVIDIA mixture-of-experts model with 30B total and about 4B active parameters. One checkpoint does transcription in 40 languages, speaker attribution, translation, contextual prompting and audio understanding. The post says it ranked first in VoiceArena's comparison of 14 diarization plus ASR systems. The fetched text is cut off and gives no license or download details.

Mixture of Experts
NVIDIANemotronNemotron 3.5 Transcribe+1

models, multimodal· Added 30 Oct 2026, published 29 Oct 2026

huggingface.coarticle

Introducing AnySpeechRecognizer and AnyDecisionModel: Open Models for Speech and Decisions on Apple Platforms

AnySpeechRecognizer and AnyDecisionModel are two Swift packages from Hugging Face. The first mirrors Apple's SpeechAnalyzer API with swappable engines (NVIDIA Parakeet, Alibaba Qwen3-ASR on Core ML) and adds Sortformer speaker diarization. The second returns typed yes/no, choice and score answers with probabilities via MLX, Core AI, llama.cpp, Jev or Ollama. The post gives M4 Max timings, model sizes and calibration options.

SwiftCore MLMLX+10
AnySpeechRecognizerAnyDecisionModelAnyLanguageModel+17

multimodal, inference, models· Published and added 29 Oct 2026

arxiv.orgpaper

Leaving, Staying, or Living There? Duration Commonsense Reasoning with Human-LLM Co-annotations

This paper studies whether vision-language models can infer how long a person stays somewhere (living, staying, or leaving) from geotagged social media posts. The authors build a new corpus of geotagged tweets co-annotated by humans and LLMs. State-of-the-art models struggle even with multi-turn Chain-of-Thought prompting, often missing contextual cues and subtle hashtags while over-relying on prominent visual details.

vision-language modelLLM
arXiv

research, evals, multimodal· Added 30 Oct 2026, published 28 Oct 2026

arxiv.orgpaper

inkGround: Benchmarking Fine-grained Visual Grounding Capabilities of Vision-Language Models for Cursive Handwriting in the Wild

inkGround is a benchmark for stroke trajectory recovery from full-page, camera-captured handwriting, with 537 page images (about 18.7k words, 163k strokes) and pseudo-label references filtered by OCR agreement and human raters. Models get a page image and a target word, then output ordered stroke points. Seven VLMs were scored with DTW, Chamfer distance, readability and style similarity. Closed-source models, notably Gemini-3.1-Pro, did best, and the ranking held on IAM-OnDB.

vision-language model
inkGroundIAM-OnDBGemini-3.1-Pro+1

evals, multimodal, research· Added 30 Oct 2026, published 28 Oct 2026

arxiv.orgpaper

AskStream: When and What to Ask Proactively under Uncertainty in Streaming Video Understanding

AskStream is a multi-turn benchmark for streaming video understanding that adds a third proactive action, ask, alongside wait and answer, for cases where video or user information is incomplete. It has 438 videos, 2,714 manually annotated frame-level time windows (wait, ask, answer), four task categories and eight subtasks. The strongest model reaches only 52.18% on the proposed Ask-F1 metric, in both multiple-choice and open-ended settings.

multimodal large language model
AskStreamAsk-F1arXiv

evals, multimodal, research· Added 30 Oct 2026, published 28 Oct 2026

arxiv.orgpaper

Quantize by Doing: Task-Guided Mixed-Precision Quantization of Vision-Language-Action Models

QbD-VLA is a mixed-precision post-training quantization method for vision-language-action models. It uses task success from short closed-loop evaluations to assign weight and activation bit widths, accounting for interactions between components. On four VLA models across all four LIBERO suites it holds task performance at about 4-bit weights and under 6-bit activations. The gain on SIMPLER is small. On a Franka Research 3 arm, X-VLA shows no detected loss, but π0.5 fails.

vision-language-action model
QbD-VLALIBEROSIMPLER+4

research, inference, multimodal· Added 30 Oct 2026, published 28 Oct 2026

x.comthread

John Rood (@johnroodepic) on X

Nemotron 3.5 Transcribe is an open-weight, promptable 30B mixture-of-experts model from NVIDIA with about 4B active parameters, built for recorded audio. One checkpoint transcribes 40 languages, handles recordings up to an hour, labels up to eight speakers, translates, and answers questions about English audio. Weights and a demo are on Hugging Face. The post gives no benchmarks.

Mixture of ExpertsHugging Face
NVIDIANemotronNemotron 3.5 Transcribe+1

models, multimodal· Added 29 Oct 2026, published 28 Oct 2026

huggingface.coarticle

NVIDIA Nemotron 3.5 Transcribe and Granary v2: Intelligent Multilingual Transcription You Can Prompt and Adapt

Nemotron 3.5 Transcribe is an open-weight, promptable speech model that handles transcription in 40 languages, translation, diarization for up to eight speakers, and audio Q&A. It has about 4B active of 30B parameters and tops open-weight ASR and diarization leaderboards. The post also covers the Granary v2 curation pipeline (463,062 hours, 48 languages) and a medical fine-tuning case study that cut held-out WER by 45.8%.

NeMo SpeechNeMo CuratorvLLM+11
NVIDIANemotron 3.5 TranscribeGranary v2+16

models, multimodal, training, evals· Published and added 28 Oct 2026

arxiv.orgpaper

Data Scaling Laws in Stereo Matching: From In-domain Fitting to Cross-domain Generalization

This paper studies data scaling laws in stereo matching and releases XStereo, a realistic synthetic dataset of 200 thousand samples. In-domain training on five large datasets shows two regimes: a standard power law, then saturation, with a transition point roughly fixed for a given model. Cross-domain performance depends only weakly on source data scale, and distribution and coverage diversity reduce domain gaps more.

XStereoarXiv

research, training, multimodal· Added 30 Oct 2026, published 28 Oct 2026

arxiv.orgpaper

Complementing Text Based Moderation with Multimodal Grounding: Detecting Fraudulent Inducement Behaviors on Short Video Platforms

The paper studies detection and temporal localization of fraudulent inducements (prompts to follow, join chat groups, click links) in internet finance short videos. It compares OCR+LLM, OCR+MLLM and a zero-shot MLLM, and builds a dataset of 2,419 videos (66.49 hours, 3,194 events, four categories). OCR+MLLM reaches mIoU 26.52 versus 14.49 for keyframe OCR+LLM, with 67.0 percent macro recall, using a recall-weighted F2 localization reward.

MLLMLLMOCR
arXiv

research, multimodal, training· Added 30 Oct 2026, published 28 Oct 2026

arxiv.orgpaper

Improving the Dependency on Fine-Tuning Data in Transfer Learning for Speech Emotion Recognition

This paper proposes an fMLLR-based compensation mechanism that cuts the fine-tuning data needed in transfer learning for speech emotion recognition under train-test acoustic mismatch. An autoencoder reduces trainable parameters, and an affine transform in the encoder latent space models rotations and shifts. In a human-robot interaction task, mean CCC rose 16.8% with 12.9% of the training set and 14.4% with 50%.

fMLLRautoencoder
arXiv

research, multimodal, training· Added 30 Oct 2026, published 27 Oct 2026

arxiv.orgpaper

Know, then Draw: A New Learning Paradigm for Visual Generation

Know, then Draw is a learning paradigm for visual generation that uses autoregressive pretraining to acquire semantic knowledge and diffusion post-training for high-fidelity rendering. Controlled comparisons find autoregression the stronger pretraining objective, and semantic quality matters more than AR-only generation quality. This motivates a purely semantic tokenizer trained with language supervision only. A 3B text-to-image model trained in 6,600 H100 GPU-hours scores 86 on GenEval and 87 on DPG-Bench.

diffusion modelTransformer
arXivGenEvalDPG-Bench+1

research, multimodal, training· Added 28 Oct 2026, published 27 Oct 2026

arxiv.orgpaper

Perceiving and Predicting Dynamic Sports: Spatiotemporal Intelligence in Vision Models for Egocentric Tennis Video

This paper presents an annotated egocentric tennis dataset (egocentric video synced with five allocentric views, global positions, contact times, stroke types) and evaluates image encoders, video encoders and vision-language models on seven prediction tasks. Best models reach 0.14 m player and 0.54 m ball localization error and 0.14 s contact-time error. Depth input helps 3D ball localization, and longer histories do not help forecasting. A conditional diffusion model generates whole-body tennis motion in real time.

vision-language modeldiffusion modelTransformer
arXiv

research, multimodal, evals· Added 28 Oct 2026, published 27 Oct 2026

arxiv.orgpaper

VideoLVR: Latent Visual Reasoning over Grounded Video Evidence

VideoLVR is a method that lets video multimodal LLMs build latent visual evidence from local regions at relevant moments and reason over it interleaved with text tokens. A visual token prediction objective, defined over a vocabulary derived from the input video, supervises the latents. It extends to reinforcement learning with sampled visual tokens as actions and final-answer rewards, and improves fine-grained video benchmarks without much context growth.

vision-language modelLLM
VideoLVRarXiv

research, multimodal, training· Added 28 Oct 2026, published 27 Oct 2026

arxiv.orgpaper

As the Story Unfolds: Watching a Film and Identifying Characters as a Human Does

This paper defines a character naming task for films: given a name spoken in dialogue and its video context, the model must pick which visible character bears that name. It provides a dataset with annotations for character naming and first appearances, a video-and-dialogue model, and a comparison with proprietary multimodal LLMs. It also proposes a progressive framework that builds a character bank as the film plays, without cast lists or actor photographs.

multimodal large language model
arXiv

research, multimodal· Added 28 Oct 2026, published 27 Oct 2026

huggingface.coarticle

Build specialized multimodal models with LFM3

LFM3 is Liquid AI's open-weight family of six ~2B-parameter models (text, vision-language, audio, omni and two d1 decision models), all branched from LFM3-1.7B-Base. The base model uses a tri-hybrid backbone of 13 gated short convolutions, 5 sliding-window attention layers and 6 global attention layers, pre-trained on 22T tokens with a 256K context. The post gives transformers snippets, benchmark comparisons against LFM2.5 and Qwen3.5, and a robotics fine-tuning example.

TransformersHugging FacePyTorch+7
Liquid AILFM3LFM3-1.7B+29

models, multimodal, training· Published and added 27 Oct 2026

liquid.aiarticle

Introducing LFM3: Every Modality for Specialized Intelligence

LFM3 is Liquid AI's new family of open-weight multimodal models, six models of about 2B parameters branched from LFM3-1.7B-Base (22T pre-training tokens, 256K context). The hybrid backbone uses 13 gated short-convolution, 5 sliding-window attention and 6 global attention layers. Variants cover text, vision with video, audio, omni and two decision models, with on-device speed numbers, fine-tuning demos and robotics extensions.

llama.cppMLXvLLM+17
Liquid AILFM3LFM3-1.7B+32

models, multimodal, inference, training· Added 28 Oct 2026, published 27 Oct 2026

deepmind.googlearticle

Genie 4: A new generation of world simulation

Genie 4 is Google DeepMind's real-time interactive world model, announced as a research preview for trusted testers. It generates video with native audio and interactive dialogue, steered by keyboard controls and text prompts. Gemini 3.8 Flash acts as a guiding agent that plans steps and checks frames, and Gemini 4 Argon is used for robotics experiments. The post lists limitations such as drift and object permanence, and says outputs carry SynthID watermarks.

GeminiGoogle AI StudioSynthID
Google DeepMindGenie 4Gemini 3.8 Flash+5

models, multimodal, agents· Published and added 27 Oct 2026

blog.googlearticle

We’re announcing Genie 4, our new world model for real-time interactive simulation.

Genie 4 is Google DeepMind's world model for real-time interactive simulation, with native audio generated on the fly (ambient sound, music, dialogue). It builds on Gemini models: Gemini Omni grounds scenes in real-world appearance and behavior, and other Gemini models keep the simulation coherent. Users steer scenes by natural language. Access is limited to trusted testers now, with Google AI Ultra (Project Genie), the Gemini API and AI Studio to follow. The post gives no architecture or benchmark detail.

GeminiGoogle AI StudioGemini API
Google DeepMindGenie 4Gemini Omni+2

models, multimodal· Added 30 Oct 2026, published 27 Oct 2026

arxiv.orgpaper

When Visual Evidence Is Insufficient: How Instructions, Post-Training, and Internal Activations Shape Refusal in Multimodal Language Models

This paper studies how multimodal LLMs refuse questions that an image cannot answer, using the MoHoBench benchmark on three MLLMs. It varies the meaning and placement of refusal instructions, compares supervised and preference-based post-training, and intervenes on a refusal-related activation direction. Instructions change refusal at the sample level in ways aggregate rates hide. Post-training raises refusal most but can mix refusal with attempted answers. Activation intervention gives smaller, transferable changes.

vision-language modelLLM
MoHoBencharXiv

ai-safety, multimodal, research· Added 28 Oct 2026, published 27 Oct 2026

x.comthread

Manoj | The AI Dude (@Manojkumar2257) on X

heditor is an open-source (MIT) agent skill from pycodebr that has Claude Code, Codex, Cursor or OpenCode edit talking-head videos. It transcribes with faster-whisper, picks a 30-70s segment, cuts frame-accurately, builds HTML + GSAP motion graphics, mixes audio at -14 LUFS, and runs QA with vision-reviewed previews. It offers 42 named styles and a long-form mode with chapters, SRT/VTT and lower thirds. Install: npx skills add pycodebr/heditor -g.

Claude CodeCodexCursor+6
heditorpycodebr

coding-agents, prompting, multimodal· Added 28 Oct 2026, published 27 Oct 2026

x.comthread

Rohan Paul (@rohanpaul_ai) on X

Hi3D v1.0 (formerly Sparc3D) is an open-sourced image-to-3D model from Hi3D, which runs a closed v3.0 refining at 2048 cubes per side. In a test with 13 professionals in 3D algorithms, 562 judgments against eight other image-to-3D models favoured it 83.5% of the time. The thread also covers the Hi3D Agent and an MCP server for other agents. Back and bottom views were checked, where TRELLIS.2 collapsed a winged figure.

Model Context Protocol
Hi3DHi3D v1.0Sparc3D+2

models, multimodal, mcp· Added 28 Oct 2026, published 27 Oct 2026

arxiv.orgpaper

AxisQ: Depth-Adaptive Channel--Frequency Quantization for Camouflaged Object Detection

AxisQ is a post-training quantization framework for camouflaged object detection (COD) that allocates activation bits by feature depth. It uses the channel domain for shallow features and the local-frequency domain for deep features, under a fixed per-layer average bit budget. It adds a boundary-aware Hessian approximation for reconstruction. On CAMO at W2A2, weighted F-measure improves 12.7% for CFRN and 33.3% for CamoFormer-P over the strongest baselines.

CNNTransformer
AxisQCFRNCamoFormer-P+1

research, inference, multimodal· Added 28 Oct 2026, published 27 Oct 2026

simonwillison.netarticle

GPT-6.1 Terra

GPT-6.1 Terra is OpenAI's midrange model, announced 26 October 2026 and aimed at visual reasoning, computer use and clearer writing. It costs $1 per million input tokens and $5 per million output, about half of GPT-5.6 Terra and GPT-6.1 Sol and roughly 10x GPT-6 Luna. Simon Willison relays OpenAI's Ted Sanders, who rates it better than Astra and Sol at technical communication and much faster on computer use.

OpenAI API
GPT-6.1 TerraOpenAISimon Willison+7

models, multimodal· Added 27 Oct 2026, published 26 Oct 2026

en.cryptonomist.charticle

Test-time training in robotics shows no clear gains in demo response

This article reports a study submitted October 23, 2026 by Qiangong Zhou, Jiafeng Liang, Yimin Dong, Yang He and Hongxiang Li. They added Test-Time Training (fast-weight memory) to the vision-language-action models InternVLA-A1 and Pi0.5, trained on paired human demonstrations and robot trajectories. Shorter videos, full-trajectory gradient connectivity and contrastive regularization gave no clear gain in demonstration-guided target selection at the tested scale.

vision-language-action modelTest-Time Training
InternVLA-A1Pi0.5Qiangong Zhou+4

research, multimodal, training· Published and added 26 Oct 2026

en.cryptonomist.charticle

A new AI method for saliency-aware image-text alignment improves image search

RoSA is a finetuning framework from Qiuyu Kong and three co-authors for long-text image-text retrieval. It targets "saliency collapse", where models over-rely on text tied to prominent visual features and miss contextual information. RoSA splits pairs into salient and contextual views using object-level saliency estimation and LLM text decomposition, then trains on both with plug-and-play auxiliary alignment objectives. Code and checkpoints are public.

LLM
RoSAQiuyu Kong

multimodal, research, training· Published and added 26 Oct 2026

lynchlab.netwriteup

A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis

A dermatologist tested GPT-6 Astra and Claude Fable 5.1 on 200 biopsy-proven lesions from MILK10k, giving each a clinical and a dermoscopic image. A CNN (Inception v3, fine-tuned for about twenty minutes on 68,011 ISIC images) named the right diagnosis for 62% of lesions, against 42% for both LLMs. The post lists caveats (small sample, single run, possible contamination) and links the cases, prompts, replies and scoring code.

CNNGPT-6 AstraClaude Fable 5.1+4
MILK10kInternational Skin Imaging CollaborationISIC Archive+2

evals, multimodal· Added 26 Oct 2026, published 25 Oct 2026

arxiv.orgpaper

Explaining Navigation Decisions: Stage-Aware Visual Attribution for Vision-Language Navigation

A stage-aware visual attribution framework for Vision-Language Navigation (VLN) that explains a robot's action choices. It identifies "instruction-stage insensitivity", where existing explainers give similar outputs across navigation stages. The method builds action-conditioned temporal instruction-visual attributions, aggregates relevant historical evidence by stage, and applies an instruction-guided spatial filter. It improves attribution quality across several VLN policies and benchmarks, helps diagnose failure cases, and was tested on a Unitree G1 robot.

vision-language model
Unitree G1

research, multimodal· Added 27 Oct 2026, published 25 Oct 2026

arxiv.orgpaper

Detecting and Understanding Ambiguity in Visualizations with Role-Playing Vision Language Model Interpretations

This paper presents a pipeline that uses persona-conditioned vision language models to surface interpretive ambiguity in visualizations. Topic-relevant personas elicit candidate takeaways, and a Target-Pattern-Scope framework lets LLM judges label statement pairs as Compatible, Disjoint or Opposing, which are summarized in an ambiguity profile. The authors benchmark VLMs on chart comprehension per chart type, check labeling against human annotations, and run case studies on public visualizations and a redesign.

vision-language modelLLM
arXiv

research, multimodal, evals· Added 27 Oct 2026, published 25 Oct 2026

arxiv.orgpaper

InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants

InteractiveWorld is an environment for closed-loop interaction between an AI assistant and a simulated human agent who sees the scene, follows instructions and acts independently. InteractiveBench, built on it, has 120 multi-turn tasks in six categories scored by verifiable, state-based rewards. Across 12 frontier MLLMs, the best model, Claude Fable 5.1, reaches at most 20% on long-horizon Guidance and 25% on Planning tasks.

multimodal large language model
InteractiveWorldInteractiveBenchClaude Fable 5.1+1

evals, multimodal, research· Added 27 Oct 2026, published 25 Oct 2026

coindesk.comarticle

Block's new voice AI sounds eerily human, sparking awe and scam fears

Dessa is a voice AI model from Block, trained from scratch on human speech only, with no text stage. It listens while speaking, so it handles interruptions, laughter and tone changes. Block says it trained on one Nvidia GB200 NVL72 system and that it runs locally on a MacBook Pro with M4 Max or better. Model size, benchmarks and release date are undisclosed. Early access goes to a small group of AI safety researchers. Reactions split between praise and scam fears.

Nvidia GB200 NVL72MacBook ProApple M4 Max
BlockDessaJack Dorsey+7

models, multimodal· Published and added 24 Oct 2026

arxiv.orgpaper

OWLS: Open-world Language-conditioned Speed Fields for Zero-Shot Rover Navigation

OWLS (Open-world Language-conditioned Speed Fields) plans directly over a vision-language model's dense feature map. A text prompt describing traversable terrain scores every pixel of one camera image. The scores are projected onto the ground, adapted with two morphological operations, and used as the speed field of a minimum-time planner. It needs no training, terrain labels, map or extra sensors. It was tested on a physical rover, on simulated Mars and Moon rovers over 200 procedural terrains, and on real planetary imagery.

vision-language model
OWLSarXiv

research, multimodal· Added 26 Oct 2026, published 23 Oct 2026

arxiv.orgpaper

Standing on Open Shoulders: Regional Fine-Tuning of Open-Weight Camera-Trap Models for UK Wildlife with Modest Data and Compute

DeepFaune-UK is a fine-tune of the open DeepFaune camera-trap classifier on 28 UK classes, paired with MegaDetector v6. On 20,212 MammalWeb images it reached 88.8% strict accuracy against 56.5% (DeepFaune v1.5.0) and 59.0% (SpeciesNet). Re-applying the recipe to the DINOv3-based classifier gave 90.6%. Fine-tuning on about 150 crops per class matched full-data out-of-distribution accuracy, and a DINOv2 run took about 22 minutes on under 4 GB of GPU memory.

DeepFauneMegaDetectorSpeciesNet+2
DeepFaune-UKMammalWeb

research, training, multimodal· Added 26 Oct 2026, published 23 Oct 2026

arxiv.orgpaper

UAV-SwarmAgent: Scalable Multi-UAV Reconnaissance and Task Allocation via Code Generation and Vision-Language Reasoning

UAV-SwarmAgent is a hierarchical multi-agent system where an operator gives one natural-language sentence and a ground-station agent writes and runs code that produces per-UAV routes. A vision-language model grounds targets in an aerial map, and worker agents verify incidents with a compact onboard model. In simulation it located the incident in 244 of 255 missions (95.7%), and flight time followed an Amdahl-type law with speedup bounded at 1.52x for one to five UAVs.

vision-language modelLLM
UAV-SwarmAgentarXiv

research, agents, multimodal· Added 26 Oct 2026, published 23 Oct 2026

arxiv.orgpaper

Efficient Zero-Shot Video Anomaly Detection via Signal-Aware Adaptive Fusion (SAAF)

Signal-Aware Adaptive Fusion (SAAF) is a training-free zero-shot video anomaly detection framework that cuts the compute cost of vision-language models. It combines a dual-signal sampler (frame-to-frame delta and rolling-context drift in embedding space, OR-gated with EMA thresholds), a threshold-gated cascade from frame-level to video-level analysis, and signal-aware score fusion. It reports AUC-ROC of 92.07% on XD-Violence, 86.70% on MSAD and 80.54% on ShanghaiTech, at 2.2x to 2.6x the throughput of the prior best zero-shot method.

vision-language model
SAAFXD-ViolenceMSAD+3

research, multimodal· Added 26 Oct 2026, published 23 Oct 2026

arxiv.orgpaper

When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning

MART-GRPO is a variant of group-relative policy optimization for vision-language-action (VLA) reinforcement learning. It uses a LoRA-fine-tuned VLM to find the last visible error without recovery in a failed rollout video, keeps failure feedback up to that point, and drops it afterward. Martingale stopping conditions show this preserves the expected GRPO gradient and can reduce covariance. With OpenVLA-OFT on RoboTwin 2.0, it raises success rates over full-trajectory GRPO across multiple tasks.

GRPOLoRAvision-language-action model+2
RoboTwin 2.0arXiv

research, training, multimodal· Added 26 Oct 2026, published 23 Oct 2026

arxiv.orgpaper

4DMemory: Benchmarking Long-Horizon Spatiotemporal Memory with Harness-Augmented Agents

4DMemory is a benchmark suite for long-horizon spatiotemporal memory: the LongVideo dataset (about 900 hours of video, over 5,000 manually reviewed questions) and 4DMemoryBench. It also presents the 4DMemory Harness, combining four skills, 22+ tool models and a model-specific RAG system. Across 13 models, the harness improved every one by over 50%, and the same models scored over 10% lower than on a previous benchmark.

RAG
4DMemory4DMemoryBenchLongVideo+2

evals, multimodal, agents· Added 26 Oct 2026, published 23 Oct 2026

arxiv.orgpaper

MAST-VLA: Skill-Conditioned Adaptation with Trajectory-Affordance Priors for Multi-Skill Robot Manipulation

MAST-VLA is a vision-language-action framework for multi-skill robot manipulation that conditions on both the skill to execute and where and how to contact objects. It adds skill-conditioned LoRA Mixture-of-Experts adapters to the LLM backbone and turns a predicted affordance heatmap into a patch-level spatial prior for the action head. It reaches an Avg. Len of 4.65 on CALVIN ABC→D and was tested on a real robot.

vision-language-action modelLoRAMixture of Experts+1
MAST-VLACALVINarXiv

research, multimodal· Added 26 Oct 2026, published 23 Oct 2026

arxiv.orgpaper

Sensorimotor Alignment: A Platonic Proxy for Embodied Policy Success

Sensorimotor Alignment (S^2) is a training-free offline metric that measures how well an embodied model's sensory representation aligns with the geometry of physical motion. It extends the Platonic Representation Hypothesis to motion trajectories. S^2 correlates positively with downstream robot task success across embodied model families, helps pick promising VLM backbones before fine-tuning into VLAs, and explains behaviours seen in VLAs and world action models.

vision-language modelvision-language-action modelworld action model
Sensorimotor AlignmentPlatonic Representation HypothesisarXiv

research, evals, multimodal· Added 24 Oct 2026, published 22 Oct 2026

arxiv.orgpaper

F2G-Pose: Geometry-Aware Foundation Feature Lifting for Direct RGB-D Category-Level Object Pose Estimation

F2G-Pose is a direct, single-pass framework that predicts category-level object pose, metric size and camera-frame completed shape from one RGB-D image. It lifts frozen RADIO foundation features onto partial point clouds, tokenizes local geometry with DGCNN, and fuses it with a Mixture-of-Experts Transformer. It trains only on synthetic SOPE data, leads on SOPE and ROPE at 28 FPS, and is tested on HouseCat6D, HANDAL and real manipulation.

TransformerMixture of ExpertsDGCNN+2
F2G-PoseSOPEROPE+4

research, multimodal· Added 24 Oct 2026, published 22 Oct 2026

arxiv.orgpaper

Progress Probes $P^2$: Lightweight Runtime Monitoring for Vision-Language-Action Models

Progress Probes (P^2) trains linear probes on a vision-language-action model's own activations, using demonstrations, to estimate task progress, then flags executions that deviate from expected progress. Tested on the open-source pi0.5 across simulated and real-world manipulation tasks, it matches or beats VLM-based monitors and failure-data-trained methods at failure detection, without an external foundation model.

vision-language-action modelvision-language modellinear probes
pi0.5arXiv

research, multimodal· Added 24 Oct 2026, published 22 Oct 2026

arxiv.orgpaper

Rethinking Sequence Compression with Test-time Regression

TTR-Compressor is a sequence compression method that treats input tokens as a test-time regression problem. The regression is learned end-to-end by backpropagation, and its coefficients are solved in closed form in the forward pass to give the compressed representation. As a block-wise sparse attention compressor it improves RULER over attention-based compressors by a large margin. As a visual token merger for multimodal LLMs it beats the MLP merger and matches it with 55.6% fewer visual tokens.

TransformerLLMvision-language model
TTR-CompressorRULERarXiv

research, multimodal· Added 24 Oct 2026, published 22 Oct 2026

arxiv.orgpaper

CellTrack-Bench: Microscopic Cell Tracking Benchmark for Vision-Language Models

CellTrack-Bench is a benchmark that tests vision-language models on microscopic cell tracking and compares them with specialised trackers. It has 1,537 problems covering 9,041 cell-linking decisions, drawn from 15 cell-tracking datasets across four imaging modalities, with a hard subset built from failures shared by distinct algorithmic trackers. The authors compare environment harnesses, benchmark 9 closed-source and 13 open-weight models, and analyse failure modes.

vision-language modelLLM
CellTrack-BencharXiv

evals, multimodal, research· Added 24 Oct 2026, published 22 Oct 2026

arxiv.orgpaper

DuoCook: An Overcooked-Inspired Benchmark for Visual Planning and Coordination

DuoCook is an Overcooked-inspired benchmark in which one vision-language model controls two cooks from top-down views in Unreal Engine environments built with coding agents. It has 20 maps in five categories covering equipment contention, changing terrain and handoff dependencies. Four VLMs were compared; the best, gpt-6-astra, reached 26.2% mean order completion. A notebook memory ablation had mixed effects: positive for Claude and Qwen, negative for GPT and Gemini.

Unreal Enginevision-language model
DuoCookGPT-6 AstraClaude+4

evals, agents, multimodal· Added 24 Oct 2026, published 22 Oct 2026

huggingface.coarticle

OlmoEarth v1.3: State-of-the-art embeddings for the whole planet

OlmoEarth v1.3 is an open satellite embedding model from Ai2 that reads a year of Sentinel-1, Sentinel-2 and Landsat data and outputs a 128-dimensional vector per 10 m pixel (the first 64 dimensions also work alone). Its embeddings beat AlphaEarth Foundations and Tessera v2 on 20 of 27 evaluations. Precomputed global embeddings, a browser Embeddings Explorer and pre-training cost (about $2,720 of H100 time) are covered.

TransformerHugging FaceGeoTIFF
OlmoEarthAi2AlphaEarth Foundations+8

models, multimodal, evals· Published and added 22 Oct 2026

arxiv.orgpaper

Drift Says When, Budget Says How Much: Rate-Controlled KV Re-selection for Long-Form Multimodal Generation

Rate-Controlled Refresh (RCR) is a KV cache re-selection method for multimodal LLMs in long-form generation. Attention drift decides when and where to re-select visual tokens, and a per-sample controller sets how often. RCR beats InfiniGen by 1.6-3.8 ROUGE-L points on five of six model-task settings and stays within 1.7 points of exact per-step re-selection. On video description, where drift is homogeneous, it no longer beats a fixed threshold.

Qwen2-VLQwen3-VLInfiniGen+1
DetailCaps

research, inference, multimodal· Added 24 Oct 2026, published 22 Oct 2026

deepmind.googlearticle

Beyond tree cover: mapping the world’s forest types

Forest Typology (ForTy) 2020 is a global 10-meter map from Google DeepMind, Google Research, WRI and IIASA. It sorts land into six FAO-based classes: primary, naturally regenerating, planted, plantation, tree crops and agroforestry, and other land. The model uses AlphaEarth Foundations embeddings and about 1.7 million labeled locations. It was 90% accurate on forest type across 9,000+ expert-checked sites. An early 2017 to 2025 time series is on Earth Engine.

Google Earth EngineAlphaEarth Foundations
Google DeepMindGoogle ResearchWorld Resources Institute+7

models, multimodal· Published and added 22 Oct 2026

github.comrepo

Show HN: OnKey – I'm a bad singer, so I built an app to show where I'm off-key

OnKey is a free, MIT-licensed singing trainer for Apple Silicon Macs. After a song ends in Spotify, Apple Music or browser YouTube, it separates the lead vocal with UVR's MDX-Net Karaoke 2, tracks pitch with CREPE and turns it into syllable-level notes with the GAME model. On later plays it scrolls the notes, draws the user's live pitch and scores each line. Analysis runs on-device, and the repo explains the build steps and network requests.

macOSRustNode.js+2
OnKeyCREPEGAME+9

multimodal, inference· Added 24 Oct 2026, published 22 Oct 2026

arxiv.orgpaper

TRIAD: Benchmarking Omni-Modal Ambiguity in Multimodal Large Language Models

TRIAD is a benchmark for omni-modal ambiguity resolution. It has 327 hand-written text-image-audio queries in 86 scene groups across 18 ambiguity categories, and removing any one modality makes the answer non-unique. Across 14 omni-modal systems, the best model trails humans by over 20 points in group accuracy. Without answer options, model group accuracy falls to single digits, and leave-one-out tests show audio cues are often underused.

multimodal large language modelomni-modal LLM
TRIADarXiv

evals, multimodal, research· Added 23 Oct 2026, published 22 Oct 2026

huggingface.copaper

Matrix-Game 4.0: Harnessing Video Models for Real-Time Playable Game Worlds

Matrix-Game 4.0 is a system that separates game logic from visual rendering for playable generated worlds. A lightweight engine holds world state and rules, a multimodal coding agent writes game code and scene captions, and a distilled autoregressive video model turns low-poly proxies into detailed frames. It runs 480p at 60 FPS on one RTX 5090 with 112 ms key-press-to-display latency. Code, weights and assets are released.

diffusion modelHugging Face
Matrix-Game 4.0NVIDIA

multimodal, research, agents· Added 22 Oct 2026

x.comthread

SpyTech Ai (@RizeOnline) on X

XiaoZhi ESP32 is an open-source, MIT-licensed firmware stack that turns ESP32 boards into MCP-based voice chatbots using models such as Qwen and DeepSeek. It streams Opus audio over WebSocket or MQTT + UDP through ASR, LLM and TTS, adds offline ESP-SR wake words, and uses device-side MCP to drive speakers, LEDs, servos and GPIO. It supports 138 board directories and builds with ESP-IDF 6.0.1 or newer.

Model Context ProtocolESP32ESP-IDF+10
XiaoZhi ESP32SpyTech Ai

mcp, multimodal, llm-apps· Added 23 Oct 2026, published 21 Oct 2026

huggingface.coarticle

Granite Speech 5.0: Transcribe, Translate, and Attribute more Accurately and Faster

Granite Speech 5.0 is IBM's update to its open speech model collection: the new granite-speech-5.0-2b, a multilingual granite-speech-5.0-470m-turboctc, and a revised granite-speech-4.1-2b-plus. The 2B model ranks #2 among open-weight models on average English WER (5.06%) on the Open ASR Leaderboard. Turbo CTC reaches 19,870 RTFx, and the 4.1 Plus revision cuts long-form WER 24%. All are Apache 2.0.

TransformersvLLMllama.cpp+5
IBMGranite Speechgranite-speech-5.0-2b+5

models, multimodal, evals· Published and added 21 Oct 2026

x.comthread

Vishesh Raheja (@entropyconquers) on X

GPT Voice is a free, open-source Mac app by Vishesh Raheja that brings ChatGPT's dictation to any macOS app. A shortcut starts recording, and the transcript is pasted where the user is typing. It uses the ChatGPT Pro subscription, so it needs no API key or extra subscription. The author built it after Handy handled coding prompts poorly and they did not want to pay for Wispr Flow. The thread gives no details on local versus API transcription or latency.

ChatGPTCodexmacOS
Vishesh RahejaT3 CodeGPT Voice+3

multimodal, coding-agents· Added 22 Oct 2026, published 21 Oct 2026

huggingface.coarticle

NVIDIA Cosmos3-H-Dreams: Bringing Clinical Surgery to Life Through Generative Simulation

Cosmos3-H-Dreams is NVIDIA's real-time generative surgical simulator, trained on clinical cholecystectomy video and kinematics from the CMR Surgical Versius platform. It builds on the 16B Cosmos3-H-Embodied-Nano, distills a bidirectional teacher into four-step and two-step causal students with Self-Forcing, and streams through FlashDreams on one RTX PRO 6000 GPU. Checkpoints and a recipe for custom data are released.

FlashDreamsFadeMemHugging Face+1
NVIDIACosmos3-H-DreamsCosmos-H-Dreams+6

models, multimodal, training· Published and added 20 Oct 2026

x.comthread

brenz. (@brenzhills) on X

Krea 3 is an image generation and editing model from Krea. A post on X says its "Fast" variant scores 1125 Elo on the Artificial Analysis image editing leaderboard, against 1124 for Google's Nano Banana 2, while producing 10 images in the time Google produces 2. It also says the "Ultra" variant ranks in the top 4, behind only OpenAI models. The post gives no methodology or model details.

Krea 3Nano Banana 2
KreaGoogleOpenAI+1

models, multimodal, evals· Added 21 Oct 2026, published 20 Oct 2026

arxiv.orgpaper

UniWALM: Unified World Action Language Model for Embodied Robot Control

UniWALM is a unified world-action-language model that learns video, action and language in one mixture-of-transformers backbone with per-modality diffusion heads that co-denoise together. Generated subtask and textual memory are fed back on the next step, so no external planner or memory bank is needed. It adds Flow Matching with Correction (FMC) for recovery from perturbed actions. It reaches 34.5% on the memory-dependent RMBench subset, above Mem-0's 28.5%, and lifts real-world Find-the-Item success from 11.1% to 61.1%.

Mixture of Expertsdiffusion modelflow matching+1
UniWALMFastWAMMolmoSpaces+2

research, multimodal· Added 22 Oct 2026, published 20 Oct 2026

arxiv.orgpaper

InterWebCoder: Learning Interactive Webpage Generation from Executable Tests via Reinforcement Learning

InterWebCoder is a framework for interactive webpage code generation by multimodal LLMs. It synthesizes self-contained webpages with executable test scripts from recorded interaction trajectories, repairs failures using runtime evidence, then trains with test-ID-weighted SFT and GRPO using test pass rates and visual similarity as rewards. The pipeline yields 1,736 verified samples from 2,058 tasks. A 9B model scores 48.54 on IWR-Bench and 48.42 on Vision2Web, and reward latency drops over 97%.

GRPOvision-language modelLLM
InterWebCoderIWR-BenchVision2Web+3

research, training, multimodal· Added 22 Oct 2026, published 20 Oct 2026

arxiv.orgpaper

Agentic Ensemble with Vision-Language Models for Visual Anomaly Detection

AnomalyAgent is a post-training fusion framework for visual anomaly detection that ensembles pretrained detectors. It aligns scores and anomaly maps, summarizes disagreement, and treats fusion as a choice among score actions and map actions. A constrained vision-language model inspects high-conflict samples and picks the actions. It beats equal-weight fusion and gets the best mean image AUROC and pixel AP among compared fusion methods on MVTec AD and VisA.

vision-language model
AnomalyAgentMVTec ADVisA+2

research, multimodal, agents· Added 22 Oct 2026, published 20 Oct 2026

arxiv.orgpaper

Code2Video-Compass: A Comprehensive Benchmark for General-Purpose Code-Based Video Generation across T2V, I2V, and V2V

Code2Video-Compass is a benchmark of 2,104 tasks for code-based video generation, where multimodal LLMs write executable programs that produce complete videos from text (T2V), image (I2V) or video (V2V) inputs. Tasks span 10 categories and 24 subtypes. Scoring uses 12 objective metrics and five MLLM-judge metrics. Across 15 proprietary and open-weight models from nine families, performance falls as construction, dynamic animation and state-dependency complexity rise.

vision-language modelLLM
Code2Video-CompassarXiv

evals, multimodal, research· Added 22 Oct 2026, published 20 Oct 2026

arxiv.orgpaper

DIVE: Benchmarking Full-Duplex Interaction with Visual Engagement in Continuous Streams

DIVE is a benchmark for full-duplex interaction with visual engagement in continuous audio/video streams. It has 525 human-annotated samples across six tasks in three scenarios: proactive intervention, continuous engagement and interaction coordination. Evaluated real-time models reach sub-second average latency, but window response rates stay at or below 53.77%, and adapting ongoing responses to new information is hardest.

MLLM
DIVEarXiv

evals, multimodal, research· Added 22 Oct 2026, published 20 Oct 2026

jevcards.comtool

Show HN: Jev Cards, Can you beat a vision model at a picture quiz?

Jev Cards is a browser picture quiz where a player competes against Jev 1.14, a vision decision model. The 490 cards cover fruit, flags, maps, chess and illusions. Jev reads the image and question and returns a probability for every option, or a box for "tap the place" cards, in one call of about 0.25 seconds. It never writes text. The page reports 5,256 wins in 6,820 games.

Jev

multimodal, evals· Added 21 Oct 2026, published 20 Oct 2026

arxiv.orgpaper

A Multimodal Engineering GraphRAG Framework for Post-Wind-Event Building Assessment and Recovery Inference

This paper presents a multimodal engineering GraphRAG framework that links post-wind-event damage evidence from images and reconnaissance reports to building-condition states, recovery actions and downtime. Evidence is stored as a provenance-preserving, component-level representation, and graph-aligned retrieval uses compatibility, contradiction, evidence-precedence and abstention checks. Demonstrated on tornado-affected low-rise homes, it beats BaselineRAG and Structured Baseline RAG, and ablations test each component.

RAGGraphRAG
BaselineRAGStructured Baseline RAGarXiv

research, llm-apps, multimodal· Added 22 Oct 2026, published 20 Oct 2026

arxiv.orgpaper

OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory

OpenVLHarness is a multimodal reasoning harness for MLLMs built on a capability layer (task-level tool interfaces plus evidence rendering) and persistent multimodal memory that stores images, structured data and web references as addressable artifacts. A coding interface runs generated programs over stored data. Across 22 benchmarks and eight backbones it adds 9.7-12.6 points over no-tool baselines, and GPT-6 Luna at medium effort beats Codex and Visual Sketchpad at maximum effort using at most 37% of their tokens.

vision-language modelQwen3-VLCodex+1
OpenVLHarnessKimi K3GPT-6 Astra+3

agents, multimodal, research· Added 21 Oct 2026, published 19 Oct 2026

arxiv.orgpaper

Where is Physical RSI capability from? An Observational Scaling Law of LLM-as-policy

The paper introduces PhysRSI-eval, a five-axis metric for physical recursive self-improvement: first-attempt performance, efficiency, success, budget and improvement from failure. Several VLMs are tested on 18 manipulation tasks under a fixed interaction budget. It proposes the Physical Barrel Hypothesis and studies observational scaling. Workspace interventions show that removing history lowers late-round success, while frozen workspaces still help on unseen layouts.

vision-language modelLLM
PhysRSI-evalPhysical Barrel HypothesisarXiv

research, evals, multimodal· Added 21 Oct 2026, published 19 Oct 2026

arxiv.orgpaper

What Language Adds, and at What Cost: Four Controlled Probes of Pathology Vision-Language Models

This preprint grades evidence for pathology vision-language models and tests four claims with controlled probes on released checkpoints and public data. Stage-redacted report embeddings raise pooled C-index by 0.019 (progression-free interval) and 0.016 (overall survival) across 1,633 TCGA patients. Zero-shot prompting matches a linear probe. SlideChat scores 74.82% with the slide and 74.41% without it. Paraphrasing flips 8-23% of answers.

vision-language model
SlideChatSlideBench-VQATCGA

research, evals, multimodal· Added 21 Oct 2026, published 19 Oct 2026

arxiv.orgpaper

Scaling Parameter-Efficient Fine-Tuning for Brain Encoding

BrainLoRA is a parameter-efficient fine-tuning method that adapts multimodal backbones of up to 104B parameters to predict neural responses. It predicts multiple timepoints in one causal pass instead of sliding windows, and a multi-LoRA kernel trains many adapters on a shared backbone. Across nine backbones it improves encoding, and a 200+ model ensemble ranks first on the Algonauts post-challenge leaderboard.

LoRAQwen3Transformer
BrainLoRAQwen3-OmniAlgonauts 2025 Challenge+1

research, training, multimodal· Added 21 Oct 2026, published 19 Oct 2026

arxiv.orgpaper

Mise-en-Video: Fine-Grained 3D Control for Multi-Shot Video Generation

Mise-en-Video (MeV) is a multi-shot video generation framework where creators stage shots in 3D space and then generate the matching video. It combines hierarchical attention for multi-level narrative consistency with 3D-aware layout guidance for per-shot spatial control. The authors built a 160K-video dataset with hierarchical multi-shot texts and oriented 3D bounding boxes, and report gains over baselines in quality, consistency and subject controllability.

diffusion model
Mise-en-VideoMeVarXiv

research, multimodal· Added 21 Oct 2026, published 19 Oct 2026

arxiv.orgpaper

HAF-VLA: Hierarchical Action Flow Framework for Humanoid Vision-Language-Action Control

HAF-VLA is a hierarchical action-flow framework that adapts a pretrained flow-matching vision-language-action model to humanoid whole-body loco-manipulation. It denoises a shared action space in stages: locomotion and head commands first, then waist refinement, then full-body manipulation. Learnable stage embeddings and cross-stage KV-cache conditioning carry dependencies between stages. The authors report gains over existing models on six real-world tasks.

flow matchingvision-language model
HAF-VLAarXiv

research, multimodal· Added 21 Oct 2026, published 19 Oct 2026

huggingface.corepo

Meta Muse Spark Open (816B-A42B) open-weights model card

Model card for Meta Superintelligence Labs' Muse Spark Open, an open-weights, natively multimodal model with 816B total and 42B active parameters and a 1M-token context window. It targets coding and agentic work (tool use, long-horizon tasks, multimodal reasoning, synthetic data generation, LLM-as-judge). It ships in BF16 and NVFP4, needs about 700GB VRAM (NVFP4) or 2TB (BF16), runs on SGLang, vLLM, Transformers, TensorRT-LLM, Unsloth, llama.cpp and MLX, and supports a reasoning_effort setting. The Muse Model License requires "Built with Muse Spark Open" attribution and a separate agreement for Model-as-a-Service above $20M revenue. The card reports safety results against Kimi K3 on cyber, chem/bio and loss-of-control evals.

Hugging FaceTransformersvLLM+8
Muse Spark OpenMetaMeta Superintelligence Labs+15

models, multimodal, inference· Added 17 Oct 2026

arxiv.orgpaper

FedRAPP: Federated Test-Time Prompt Tuning for Vision-Language Models with Reliability-Aware Prompt Personalization

FedRAPP is a personalized aggregation framework for Source-Free Federated Test-Time Prompt Tuning (SF-FTTPT), where clients adapt CLIP soft prompts on unlabeled, non-IID test streams without source data. It combines reliability-aware client assessment, a server-side exponential moving-average prompt memory and similarity-guided peer aggregation. Across nine CLIP benchmark datasets and ImageNet variants it beats standard federated aggregation baselines while keeping prompt-only efficiency.

CLIPvision-language model
FedRAPPImageNetarXiv

research, multimodal· Added 20 Oct 2026, published 16 Oct 2026

arxiv.orgpaper

FlexPart: Text-Controlled Semantics-Aware 3D Part Segmentation at Flexible Granularity

FlexPart is a framework for text-controlled 3D mesh segmentation that assigns discrete part IDs to mesh faces through a view-independent part-identity field. It uses a control prompt of part names with location and shape clauses, trained on a VLM-built multi-granularity dataset. On FlexPartBench it raises average mIoU by 22% over the strongest baseline, and by 15% with planner-completed prompts.

vision-language model
FlexPartFlexPartBench

research, multimodal· Added 20 Oct 2026, published 16 Oct 2026

arxiv.orgpaper

A Language-Informed Loss for Visual Generation

LILIE (Language-Informed and Lossless Image Encoder) is an invertible image transform built from a normalizing flow transformer, trained jointly with a vision-language model via next token prediction so captions can be recovered from noised features. The encoder supplies a language-informed loss for text-to-image generation. JiT models trained with it cut gFD from 66.84 to 53.01 against a pixel-only baseline on the GPIC dataset. Code, encoders and models are released.

normalizing flowTransformervision-language model
LILIEJiTGPIC+1

research, multimodal· Added 20 Oct 2026, published 16 Oct 2026

arxiv.orgpaper

Enhancing RAG-based Medical VQA with Gated Multimodal Representations

This paper presents a multimodal retrieval-augmented generation framework for Medical Visual Question Answering. A Gated Multimodal Unit learns a joint query representation from the medical image and the clinical question, so evidence retrieval uses both modalities while the Med-LVLM stays frozen. On benchmark datasets it consistently beats a reference Med-LVLM and stays competitive with state-of-the-art methods. The page gives no dataset names or numbers.

RAGvision-language model
arXiv

research, llm-apps, multimodal· Added 20 Oct 2026, published 16 Oct 2026

x.comthread

AmingIn_AI (@aming_inAI) on X

A tweet from AmingIn_AI describes a macOS screenshot-annotation tool. On release it saves the screenshot, a crop and the user's marks locally, then copies a 127.0.0.1 link. Claude Code or Codex on the same Mac can fetch that link and receive the marked image plus app/window info, and nothing is uploaded. It shows a local, privacy-preserving way to hand visual context to coding agents without MCP or cloud upload.

Claude CodeCodexmacOS
AmingIn_AI

coding-agents, multimodal· Added 17 Oct 2026, published 16 Oct 2026

arxiv.orgpaper

MUB: An Identity-Masked Benchmark for Music Understanding in Multimodal Language Models

MUB (Music Understanding Benchmark) is a set of 500 identity-masked recordings with existing annotations for tempo, key, time signature and structural intervals. It covers all 24 keys, tempos of 47 to 180 BPM and 4,676 intervals, and strips titles, artists and platform IDs from inputs. Baselines reach 53.6% tempo accuracy (DSP) and 98.4% time-signature accuracy (constant 4/4). No multimodal LLM has been tested yet, and rights clearance is unresolved.

multimodal language model
Music Understanding BenchmarkMUBarXiv

evals, multimodal, research· Added 20 Oct 2026, published 16 Oct 2026

github.comrepo

OpenMOSS/MOSS-Image

MOSS-Image is an open-source text-to-image model family from OpenMOSS with a 5B backbone, released October 2026 as MOSS-Image-Base and MOSS-Image-SFT. It supports JSON layout control with bounding boxes, text rendering inside images, and prompts up to 4K tokens. The repo gives a Diffusers pipeline, a Gradio app, a ComfyUI plugin, LoRA training guidance, a prompt-enhancer skill and day-0 SGLang and vLLM-Omni support.

DiffusersPyTorchGradio+7
OpenMOSSMOSS-ImageMOSS-Image-Base+5

models, multimodal, inference· Added 20 Oct 2026, published 16 Oct 2026

huggingface.coarticle

Granite Switch 4.1: Chat, Decisions, and Speech from One Endpoint

IBM announces Granite-Switch-4.1-8B and -30B. Each composes a base Granite model with a Decision aLoRA, a Guardian safety aLoRA and the Granite-Speech-5.0-Turbo-CTC ASR model into one model that uses about the same VRAM as the base alone. Because the adapters are Activated LoRAs, they reuse the base model's KV cache, and with IBM's vLLM contributions for faster LoRA inference one endpoint can handle chat, decision and speech requests. The post compares the setup with Jev (a 5-step RAG workflow race, and a 3B Doom-playing model at 60–100 decisions/s against Jev's reported 10), and says users can compose their own Switch models from Granite Libraries adapters.

vLLMLoRAActivated LoRA+4
IBMGranite Switch 4.1Granite-Switch-4.1-8B+9

models, inference, multimodal· Published and added 15 Oct 2026

x.comthread

Vaibhav Sisinty (@VaibhavSisinty) on X

Social post reporting Meta's release of Muse Spark Open, described as the most capable US open-weight model: 816B total and 42B active parameters, a 1M-token context, and text, image, video and audio input in one model. It is pitched at coding and agentic work and tuned to run on two M5 Ultra Mac Studios, with weights available at release. The poster notes that in Meta's own chart, China's Kimi K3 and GLM 5.3 still beat it on coding.

MetaMuseMuse Spark Open+3

models, multimodal· Added 16 Oct 2026, published 15 Oct 2026

x.comthread

Connor (@ConnorTalksAI) on X

A tweet relaying OpenAI's announcement of GPT-Live-1.1, an update to the voice model behind ChatGPT voice mode. It reduces how often the model says "checking", adds new voices and improves accents, including new voices in French, German, Hindi, Mexican Spanish, Korean and Japanese. It is rolling out to all users in ChatGPT, Codex and "dots", with API access coming soon. The poster frames it as a response to a new xAI/SpaceXSI Grok Voice release.

ChatGPTCodexGrok+1
GPT-Live-1.1OpenAISpaceXSI

models, multimodal· Added 16 Oct 2026, published 15 Oct 2026

arxiv.orgpaper

Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation

P^3-WAM is a unified world action model for egocentric dexterous manipulation with three modes: Perceive (reconstruct articulated hand motion and camera trajectories from video), Predict (generate visual futures under prescribed actions) and Plan (jointly generate action trajectories and their anticipated visual outcomes from an initial image, a language instruction and an optional visual goal). It adds a compact structured 3D action representation to a pretrained video diffusion transformer through a mixture-of-transformers design with shared base weights, modality-specific adaptation and bidirectional visual-action attention. Randomized modality visibility lets video and 3D actions act as either context or target. Unified training beats single-mode training and outperforms specialized baselines on 3D hand reconstruction and action-conditioned video prediction across four datasets, and gives better instruction adherence than video-generation baselines.

diffusion modelTransformervision-language model
P^3-WAMarXiv

research, multimodal· Added 17 Oct 2026, published 15 Oct 2026

arxiv.orgpaper

Grounding Generative Verification in 3D Content

The paper introduces 3DVerifierBench, a 10-task benchmark for generative verification of 3D content. It scores accept/reject decisions, rejection explanations, and responses to controlled changes in specifications and candidates. A human study shows a 21.6-point gap to the strongest tested model, GPT-6 Astra. The authors also build a pipeline of multi-view fact extraction, controlled 3D editing and native annotations. It yields 14,126 training examples with contradiction explanations and linked counterfactuals. Verification-guided selection raises complete factual success from 26.7% to 90.0% with oracle selection over sixteen candidates, and human-guided recursive refinement satisfies all seven requirements for every input in an eight-input study.

multimodal large language model3D generation
3DVerifierBenchGPT-6 AstraarXiv

research, evals, multimodal· Added 17 Oct 2026, published 15 Oct 2026

arxiv.orgpaper

DK-VLM: Geometry as Position for Spatial Reasoning in Vision-Language Models

DK-VLM adds Dual-Kernel Attention to a vision-language model. A semantic kernel keeps the pretrained attention and positional encoding. A separate geometric kernel uses estimated 3D points and camera poses to modulate attention through point displacements and relative view transformations, and a learned gate fuses the two outputs. Geometry therefore enters attention as position, not as fused 3D reconstruction features. The paper also adds Reference-Frame Composition (RC) supervision, which gives stepwise turn and heading targets derived from verified single-step annotations, to train multi-step spatial reasoning as the observer moves. DK-VLM-8B reports the highest average score across six spatial benchmarks among the compared methods, with 74.33 on VSI-Bench and 58.28 on ReVSI-32, and it preserves general video understanding.

vision-language modelTransformer
DK-VLMVSI-BenchReVSI-32+1

research, multimodal· Added 17 Oct 2026, published 15 Oct 2026

arxiv.orgpaper

Velox: Spatial Memory Makes World Models Efficient

Velox is a training-free attention router for chunked video-diffusion world models. It uses camera frustum slabs with depth intervals from streaming geometry to pick which historical token blocks to attend to, instead of letting the KV cache grow with rollout length. Frustum overlap softly modulates each layer's model-derived block importance, and uncovered regions keep the model's own routing. The fixed block granularity suits FlashAttention-style tiled execution, and slab construction runs asynchronously with denoising on the same H200. At chunk 48 it speeds up attention 4.36× and denoising 2.43×, and on the 158-case WBench navigation split it stays within 0.15 average points of dense LingBot-Fast-Cam.

diffusion modelFlashAttentionCUDA
VeloxLingBot-Fast-CamWBench+1

research, multimodal, inference· Added 17 Oct 2026, published 15 Oct 2026

arxiv.orgpaper

Skill-Decoupled On-Policy Expert Distillation for Unified Vision-Language Models

Proposes SCOPED (Skill-Decoupled On-Policy Expert Distillation), which trains capability-specific RL experts for vision-language models. It extracts explicit skills from them through reward-guided reflection. During on-policy distillation the skills guide expert supervision but are hidden from the unified student, so the student internalizes the procedural knowledge and needs no explicit skills at inference. Across eight multimodal benchmarks it beats the backbone and mixed-capability RL. Analyses show cross-capability gradient interference in joint RL, and that skill augmentation gives more consistent expert trajectories, more attention to task-relevant visual evidence, and better preservation of expertise.

vision-language model
SCOPEDarXiv

research, training, multimodal· Added 17 Oct 2026, published 15 Oct 2026

arxiv.orgpaper

Onboard Lightweight LLM Reasoning for Human-Aware Safe UAV Navigation

Presents a modular, fully onboard UAV navigation pipeline that closes the "human-aware safety gap" of geometry-only planners. A shared BLIP encoder drives captioning and text-conditioned segmentation, which is fused with depth for 3D obstacle localization. Qwen-2.5-0.5B, stabilized with RAG and grammar-constrained JSON decoding, infers per-category risk once per class and caches the verdict. Perception runs at 10 Hz on a Jetson Orin NX, and fallback LLM queries take about 0.8 s off the control path. In Gazebo it gives roughly 3x fewer collisions than a geometric planner and raises semantics-aware landing success from 34% to 96%. The 224-pixel encoder input limits small-object perception at range.

QwenBLIPRAG+4
NVIDIAarXiv

research, multimodal, inference· Added 17 Oct 2026, published 15 Oct 2026

arxiv.orgpaper

From Perception to Prediction and Intervention: Benchmarking Proactive Video Assistants in Real-Life Scenarios

Introduces P^3IBench, a benchmark for proactive video understanding in real-life streaming scenarios, covering three capabilities: daily perception, risk prediction and intervention. It evaluates 12 open-source and proprietary video-language models (streaming and adapted non-streaming) and finds they struggle to make well-calibrated decisions about when to respond. The authors also build extra proactive training data and train P^3I-Assistant-4B, which improves several tasks unevenly, showing that proactive assistance needs response timing and restraint as well as visual understanding.

vision-language model
P^3IBenchP^3I-Assistant-4BarXiv

evals, multimodal, research· Added 17 Oct 2026, published 15 Oct 2026

huggingface.coarticle

Apertus 1.5 Release Notes

Release notes for Apertus 1.5, a fully open (weights, data, training details) continued pretraining of the 8B and 70B Apertus models on 4T and 2T additional tokens. It adds early-fusion image and audio understanding via discrete tokenizers (Emu3.5, WavTokenizer) without architectural change, a thinking mode, 262k context, and better tool use. It documents a four-stage pretraining and five-stage post-training pipeline (SFT, long-context SFT, RLVR, DPO, charter-alignment SDPO), plus evaluation via the MLLM-eval-suite and Every Eval Ever (EEE) Eval Cards, released checkpoints and datasets.

Hugging Facelm-evaluation-harnesslmms-eval+2
ApertusApertus 1.5Emu3.5+17

models, training, multimodal, evals· Published and added 15 Oct 2026

huggingface.coarticle

Welcome Muse Spark Open: open-weight frontier multimodal mixture-of-experts model from Meta

Hugging Face's launch post for Meta's Muse Spark Open, an open-weight 816B-parameter mixture-of-experts model (about 42B active per token). It has 256 experts with 8 routed per token, a 1M-token context, and perception and audio encoders for image, video and audio input. The post gives the architecture details, BF16 and NVFP4 checkpoints, and run recipes for transformers, llama.cpp, SGLang, vLLM, MLX and Inference Endpoints. It also covers LoRA SFT of the full model with TRL on 64 H100s and agentic demos (Pi 3D asset generation, an OpenEnv browser agent, a voice-driven shopping agent).

Mixture of ExpertsTransformersvLLM+10
MuseMetaMuse Glimmer+4

models, inference, multimodal· Published and added 15 Oct 2026

huggingface.coarticle

Welcome FLUX.3 Image Klein in Diffusers

Hugging Face blog post introducing Black Forest Labs' open FLUX.3 Image Klein (distilled and base checkpoints) in Diffusers. It covers text-to-image, multi-reference image-to-image and bounding-box layout control, FP8/NVFP4 single-file quantized weights, DreamBooth LoRA training scripts for generation and editing, memory/speed figures (about 2 s per 1024×1024 image on an H100), and a detailed architecture tour: Qwen3-VL text encoder, two-stage per-stream then joint transformer blocks, per-token timestep modulation, 4D RoPE, and flow-matching denoising. It also shows how to customise the Modular Diffusers pipeline.

DiffusersModular DiffusersTransformers+9
FLUX.3 Image KleinBlack Forest LabsFLUX.2+4

models, multimodal, training· Published and added 15 Oct 2026

deploymentsafety.openai.comarticle

GPT-Live-1.1 system card

System card for GPT-Live-1.1 and GPT-Live-1.1 Quip, OpenAI's full-duplex voice models that listen and respond continuously. It reports voice-native disallowed-content evals on production and synthetic audio prompts, and finds parity overall but statistically significant regressions for GPT-Live-1.1 on sexual, mental health, emotional reliance and gore categories. It also covers alignment evals converted to audio (respecting warnings, blocker deception, broken search tool) and Preparedness Framework conclusions: neither model is High in Biological and Chemical, Cybersecurity or AI Self-Improvement.

Codex
OpenAIGPT-Live-1.1GPT-Live-1.1 Quip+5

ai-safety, multimodal, evals· Published and added 15 Oct 2026

latent.spacetalk

Dyna-2: General Purpose Physical Agents with Hour-Long Autonomy — Jason Ma, Dyna Robotics

Podcast interview with Jason Ma, co-founder and Chief Scientist of Dyna Robotics, on Dyna-2, a world-action model pre-trained on over one million hours of human egocentric video. The model predicts future video and uses it to guide robot actions, built on a diffusion transformer with flow matching. The discussion covers the robotics data bottleneck, adapting to new tasks from about 45 minutes of demonstrations, synthetic environments and self-generated training data, on-device inference and distillation, and Dyna's custom wheeled semi-humanoid hardware. The episode listing also covers commercial deployments (Din Tai Fung, Best Western), agentic harnesses for hour-long workflows, and world-action models versus VLAs, but the fetched text was truncated before those sections.

diffusion transformerflow matchingdiffusion model+2
Dyna RoboticsDyna-2Dyna-2.1+13

multimodal, training, models· Published and added 13 Oct 2026

arxiv.orgpaper

ChunkAct: Action Chunking for Agentic Robotics

ChunkAct brings action chunking to agentic robotics, where an LLM agent composes perception, motion and VLA tools guided by memory. Online, one call launches a chain of primitives that runs autonomously and returns control only on failure. Offline, memory is compiled into these chunks so execution needs no memory at test time and the planner is used only for real decisions. On LIBERO-Pro it improves task success by 5.0 points over the prior agentic-robotics state of the art, with 2.9x fewer LLM calls, 4.4x fewer input tokens and 2.3x fewer output tokens.

LLMvision-language-action model
ChunkActLIBERO-Pro

research, agents, multimodal· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper

Dog-Breed Text-to-image Flow Matching from Pretraining to Fine-Tuning on 4 Consumer GPUs

MiniDog is a framework for training a text-to-image flow-matching diffusion transformer from scratch on four RTX 3090 GPUs, with 55 minutes of pretraining and 15 minutes of fine-tuning. It pretrains on 26k ImageNet dog images that were captioned, then fine-tunes on 2k FLUX.1-schnell generated images. Image and text features are precomputed and cached. The model is a 22M-parameter transformer with the E2E-INVAE semantic latent space, a small Qwen text encoder and in-context conditioning, and training is accelerated with iREPA and an EUPE perception encoder. The repo evaluates with FID, HPSv2, PickScore and VAE reconstruction metrics. It lowers the barrier to hands-on diffusion training research.

flow matchingdiffusion modelTransformer+1
MiniDogImageNetFLUX.1-schnell+7

research, training, multimodal· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper

Visual Tokens on Demand: Visual Budgeting for Scalable and Efficient Multi-Image Reasoning in Multimodal LLMs

Proposes mmVB, a lightweight plug-and-play module (1.6M trainable parameters) that assigns query-conditioned per-image visual token budgets using cross-image context for multimodal LLMs, without retraining the base model or accessing attention matrices (so it is compatible with FlashAttention). It cuts visual input tokens 9x, giving 5x faster time-to-first-token and 7x fewer prefill FLOPs while keeping 94.4% of full-resolution accuracy across eight multi-image reasoning benchmarks. It matters because existing token pruning, merging and compression methods target single-image redundancy and either need attention maps or retraining.

FlashAttentionvision-language model
mmVB

research, multimodal, inference· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper

PixelMoE: Scaling Pixel-Space Diffusion with LLM-Style Mixture-of-Experts

PixelMoE is a pixel-space diffusion framework that scales capacity sparsely using standard LLM-style Mixture-of-Experts: token-choice routing, fine-grained routed experts, a shared expert and bias-based load balancing, with no diffusion-specific routing or auxiliary routing losses. It adds Pixel-Space Hierarchical Flow Matching, which varies spatial resolution along the denoising trajectory from global structure to fine detail. At matched active parameters it beats dense counterparts, and reaches FID 1.47 at 256x256 and 1.53 at 512x512 on class-conditional ImageNet-1K without representation alignment or perceptual losses. Code and models are promised but not yet released.

Mixture of Expertsdiffusion modelflow matching+1
PixelMoEImageNet-1KarXiv

research, multimodal· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper

RepProbe: Can Video Language Models Count Repeated Actions? A Diagnostic Benchmark

RepProbe is a benchmark of 42K real and synthetic video clips with exactly 1–20 complete action repetitions, controlling for clip position, temporal coverage and duration. Video language models' counting accuracy collapses at high counts, and this is not explained by frame sampling alone. Linear probes on final-layer representations show count information stays linearly decodable even when the model's answer is wrong, so the failure arises at the output stage, where answers cluster on a few preferred numbers.

vision-language model
RepProbearXiv

evals, multimodal, research· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper

Plan-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?

Plan-Bench is a benchmark of 699 curated samples across 16 subclasses and three dimensions. It tests whether vision-language models can reason over spatial and temporal information and produce multi-step plans under constraints for embodied tasks, with action execution isolated out. Across 28 VLMs the best model reaches only 75.0% accuracy. Scaling generally helps, and larger models look more promising as high-level planners. Code and benchmark are released.

vision-language model
Plan-BencharXiv

evals, multimodal, research· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper

Game2Real-Bench: Do Toy Games Reveal and Transfer to Real-World Visual Reasoning?

Game2Real-Bench tests whether vision-language model performance on procedurally generated toy games predicts real-world visual reasoning, and whether game examples help transfer. It has 1,000 verifiable game samples and 400 real-world samples, matched by shared reasoning requirements (fine-grained perception, spatial reasoning, constraint reasoning, sequential planning). Across 25 VLMs, game and real scores correlate strongly (Pearson/Spearman 0.935). One-shot game-to-real transfer on four models improves constraint reasoning for several but hurts fine-grained perception. The authors also release Game2Real-Dataset, with about 24K verified game samples across 18 tasks and 65 QA families.

vision-language model
Game2Real-BenchGame2Real-DatasetarXiv

evals, multimodal, research· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper

OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding

OmniCompass is a manually annotated benchmark for video grounding and understanding with two tracks: OmniCompass-VL (820 videos, 2,460 instances; vision-language models) and OmniCompass-AV (240 videos, 720 instances; joint audio-visual evidence for omni-modal models). Each track covers temporal grounding, spatial grounding and QA across short/medium/long videos, and each instance is run in paired text-only and text+image target specifications. Evaluating 50 MLLMs shows open-source models lag proprietary ones, especially on audio-visual understanding, that precise temporal/spatial grounding remains hard, and that predictions are inconsistent across the paired specifications.

vision-language modelMLLM
OmniCompassOmniCompass-VLOmniCompass-AV+1

evals, multimodal, research· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper

SP$^2$ec: Adaptive Self-Speculative Decoding for Vision-Language Models

SP$^2$ec (Single-Peak SPeculative decoding) is a training-free bandit algorithm that adapts the draft configuration of self-speculative decoding per prompt for vision-language models. It relies on the single-peak throughput pattern seen in SWIFT, KnapSpec, SpecVLM and the authors' cosSim method. It minimizes a new wall-time regret measure and has theoretical guarantees of near-optimal configuration with negligible exploration cost. On Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-Mistral-7B it improves over the original self-speculative methods by up to 11% on visual and textual tasks.

vision-language modelQwenMistral
SP$^2$ecSWIFTKnapSpec+5

research, inference, multimodal· Added 14 Oct 2026, published 13 Oct 2026

developers.googleblog.comarticle

Gemini 4 Argon for Robotics

Google's developer post presents Gemini 4 Argon as a physical-reasoning model for the whole robotics lifecycle. It reports 87% on the Embodied Reasoning (ERQA) benchmark, ahead of other frontier models. It then gives representative prompts: controlling a Boston Dynamics Spot with a MuJoCo digital twin, closed-loop code-as-policies arm control, steering the Gemini Omni world model, and building digital twins from teleoperation episodes. Further prompts cover retargeting human video to a 76-DoF humanoid, hindsight annotation of ALOHA 2 videos, a general value function on UMI data, sim-data generation for VLA fine-tuning (56% to 75% real success), and reproducing control papers in MuJoCo WebAssembly. It is useful for seeing how the model is prompted for agentic, simulation-driven robotics work.

GeminiMuJoComujoco_warp+5
Gemini 4 ArgonGoogleERQA+11

models, multimodal, agents· Published and added 13 Oct 2026

arxiv.orgpaper

UniW(A)M: A Unified Mixture-of-Experts Framework for World and World-Action Models

UniW(A)M is a mixture-of-experts model built on diffusion Transformers that unifies action-conditioned world modeling (AC-WM) with several world-action model (WAM) variants. Shared experts capture common physical dynamics, while atomic capability experts are reused across interfaces, each keeping its own conditional pathway. At deployment a plug-in adaptive interface router (PAIR), trained on a small subset of target-task chunk scenarios with the backbone frozen, picks a WAM interface for each execution chunk. It reaches 88.3% average real-world success against 76.7% for the strongest baseline, leads WorldArena on all three action-conditioned world-modeling tasks, and PAIR raises OOD success from 63.3% to 76.7%.

Mixture of Expertsdiffusion modelTransformer
UniW(A)MPAIRLIBERO+3

research, multimodal· Added 14 Oct 2026, published 13 Oct 2026

deepmind.googlearticle

Gemini 4 Argon

Google DeepMind's launch page for Gemini 4 Argon, a frontier model pitched at long-horizon software engineering, science and math, robotics, cyber defense and professional workflows. Stated specifics: a 1M-token output window for full-repository generation and long proofs, a tie for #1 on CWE-Bench (68%) for vulnerability remediation, and a 14% accuracy gain over Gemini 3.1 Pro in Box's evaluation. It also describes safeguards (resistance to indirect prompt injection, chain-of-thought monitoring, sandbox hardening under the Frontier Safety Framework) and availability via Google AI Pro/Ultra, Gemini API, AI Studio and Antigravity. Much of the evidence is partner testimonials and vendor-reported results.

GeminiGoogle AntigravityGoogle AI Studio+1
Gemini 4 ArgonGoogle DeepMindGoogle+16

models, coding-agents, multimodal· Published and added 13 Oct 2026

deepmind.googlearticle

Gemini 4 Argon for Robotics

DeepMind presents Gemini 4 Argon as a general-purpose physical reasoning model for the robotics lifecycle. It reports 87% on the Embodied Reasoning (ERQA) benchmark and gives representative prompts across several areas: controlling a Boston Dynamics Spot with a MuJoCo digital twin, code-as-policies arm control with a visual coordinate harness, and steering the Gemini Omni world model in a visual feedback loop. Further examples cover building MuJoCo and mujoco_warp scenes from teleoperation data, egocentric video or a hand-drawn sketch for 76-DoF humanoids, hindsight annotation and value-function progress prediction on video, sim data generation for VLA fine-tuning (success rate 56% to 75%), and reproducing control papers in MuJoCo WebAssembly. It matters as a concrete look at how a frontier model is being used as agentic coder, perceiver and sim builder for robotics.

MuJoComujoco_warpBlender+5
Gemini 4 ArgonGoogle DeepMindERQA+10

models, multimodal, agents· Published and added 13 Oct 2026

arxiv.orgpaper

EmoGenie: Benchmarking and Evaluating Expressed and Evoked Emotion in T2I Generation with MLLMs

EmoGenie is a benchmark for emotions expressed and evoked by text-to-image outputs. It has 6,400 images from eight T2I models using 800 emotion-targeted prompts. It includes EmoGenie-QA (90,000 closed-set and 1,057 open-ended questions) and EmoGenie-Assess (13,175 mean opinion scores for valence, arousal, dominance and aesthetic quality). The authors also propose EmoProbe, which pairs emotion QA with response-aligned, dimension-specific regression heads. They report that it beats baselines on emotion QA and correlates better with human ratings. The work shows how reliably MLLMs interpret affect in generated images.

multimodal large language modeltext-to-image
EmoGenieEmoGenie-QAEmoGenie-Assess+2

evals, multimodal, research· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper

Heading Hierarchy Reconstruction for Complex and Long Documents with Level-Wise Multimodal Memory

Proposes a vision-language-model framework that incrementally reconstructs heading hierarchies across tens or hundreds of pages using a level-wise multimodal memory. The memory keeps a compact set of earlier pages chosen by anchors (here, the most recent node at each heading level), so the VLM sees the visual, textual and layout patterns of headings at every level. The authors also introduce CLDoc-HHRBench (250 documents, 11,212 pages, expert-verified annotations) and report results on it and three existing benchmarks. It matters for RAG pipelines that need coherent document-level structure, which page-by-page OCR/VLM parsing doesn't give.

vision-language modelRAG
CLDoc-HHRBencharXiv

research, multimodal, evals· Added 14 Oct 2026, published 13 Oct 2026

pdftomp3.comtool

Show HN: Use your OpenAI subscription to create audiobooks

PDF TO MP3 is a hosted MCP server (Streamable HTTP, account authentication) that turns PDFs into audiobooks. The user's existing AI subscription, such as Codex, handles document processing through the MCP connection. Speech generation is billed separately through the user's own OpenAI or Google text-to-speech API key. The setup is three steps: add the MCP server, add a speech key, then attach a PDF and ask the assistant to use the MCP. It shows a "bring your own subscription" pattern that keeps the vendor's compute costs low.

Model Context ProtocolCodexOpenAI API+1
PDF TO MP3OpenAIGoogle

mcp, multimodal· Published and added 13 Oct 2026

arxiv.orgpaper

Seeing Through the Forgery Is Not Enough: Regressive Sycophancy under Fabricated Evidence in Medical Vision-Language Models

The paper studies regressive sycophancy under fabricated evidence: a medical VLM gives a correct answer, then abandons it when the user pushes back with a fake prescription. It tests four open VLMs on 400 VQA-RAD yes/no questions in 19,200 multi-turn conversations. Without a synthetic-document disclosure, models abandon 80.1% of 6,488 correct answers, and the same prescription raises Qwen2.5-VL-7B's flip rate from 45.9% to 89.6% while lowering Qwen3-VL-8B's from 77.7% to 50.9%. Models often reject the prescription yet keep the correct answer in only some conversations (915 of 3,232), and 40% of those flip later. System-prompt disclosure of synthetic documents cuts one model's flip rate from 77.8% to 47.3%. The authors conclude that recognising a forgery does not stop a model from adopting the forged answer.

vision-language modelQwen3-VL
VQA-RADQwen2.5-VL-7BQwen3-VL-8B+1

ai-safety, multimodal, evals· Added 14 Oct 2026, published 13 Oct 2026

huggingface.corepo

google/embeddinggemma-2

Model card for EmbeddingGemma 2, an open (Apache 2.0) 740M-parameter multimodal embedding model from Google DeepMind built on Gemma 4. It maps text, code, images, video and audio into one 768-d space, with a 270M text backbone and optionally loadable 170M vision and 300M audio encoders, an 8K context, and Matryoshka truncation to 128/256/512/768 dims. The card gives practical guidance: task instruction prefixes, re-normalising after truncation, avoiding float16 (NaN or degraded embeddings), interleaved multimodal inputs with placeholder tokens, and vision token budgets. It targets on-device search, RAG, classification and clustering.

Hugging FaceTransformerssentence-transformers+5
EmbeddingGemma 2Google DeepMindGemma 4+1

models, multimodal, llm-apps· Added 12 Oct 2026

github.comrepo

Show HN: Atticus – Hold-to-talk voice assistant for macOS that runs on Claude -p

Atticus is an MIT-licensed macOS 26+ hold-to-talk voice assistant that uses Apple's on-device speech transcription and runs AI requests through Claude Code on an existing Claude Pro/Max plan, so no API key is needed. Simple commands skip the model entirely (apps open in 0.08 s), actions it has seen once are learned as zero-cost shortcuts, and a "remember…" memory persists across asks. It also sets up no-AI background watchers (polling every 5 s) and holds itself to a budget of a median under $0.002 and 6 s per AI ask, with a system prompt under 10,000 characters.

Claude CodeClaudeCodex+6
AtticusAnthropicApple+1

agents, coding-agents, multimodal· Published and added 12 Oct 2026

github.comrepo

bilawalsidhu/gods-eye-view

God's Eye View is an open-source, vanilla JavaScript + CesiumJS + Vite 3D globe that fuses live public feeds (aircraft, ships, satellites, quakes, public cameras, weather) into one explorable scene. Its AI layer is a hands-free voice agent on the OpenAI Realtime API with 30 tools: camera fly-to, whiteboard annotations, analyst queries over live layers, and scene-context and screenshot grounding. Keys stay server-side and the browser gets only a short-lived session token. An experimental ChatGPT/Codex OAuth path is also supported. The app doubles as an MCP server, so Claude Desktop, Codex or ChatGPT can open the globe from a conversation.

OpenAI APIModel Context ProtocolCesiumJS+8
God's Eye ViewBilawal SidhuHalfpixel+2

llm-apps, mcp, multimodal· Added 11 Oct 2026

latent.spacearticle

Building AI for Reliable Execution: Lessons From Industrial Robotics

Latent Space interviews Standard Bots' CEO Evan Beard and Head of AI Leif Jentoft about the AI stack behind its industrial robot arms. A shared base model (low billions of parameters) is adapted per customer through demonstrations and fine-tuning. A zero-shot perception model, whose backbone is trained on over a billion images, handles machine tending, while conventional code handles motion and cell logic. Inference runs on-premises on edge GPUs that stream action chunks to low-level control. Failures and human corrections feed fleet learning except in air-gapped defense deployments, and the StandardOS APIs/SDKs let developers bring their own models such as NVIDIA Cosmos. The lesson is to keep the model's role narrow and use targeted, high-quality data to fix edge cases instead of chasing dataset size.

StandardOSNVIDIA Cosmosedge GPUs
Standard BotsEvan BeardLeif Jentoft+13

multimodal, inference, training· Published and added 10 Oct 2026

eprint.iacr.orgpaper

Confidential Error-Correcting Codes

Introduces Confidential Error-correcting Codes (CECs), a primitive that decodes reliably from perturbed codewords while keeping the message confidential, defined by confidentiality, robustness and soundness. The Randomness-Security Separation framework encodes a nonce and masks the payload with a PRF; the Conc-CEC instantiation uses concatenated Reed-Muller and Reed-Solomon codes and claims stronger concrete security than PRCs (CRYPTO'24). The application, CECMark, watermarks AI-generated images and reportedly beats Gaussian Shading, PRCMark and T2SMark across 56 perturbations with a false positive rate below 1e-5 and no side effects.

Reed-Muller codesReed-Solomon codespseudorandom function
Confidential Error-Correcting CodesCECMarkConc-CEC+5

research, multimodal· Added 14 Oct 2026, published 10 Oct 2026

huggingface.corepo

Qwen3.8-27B (Apache-2.0, multimodal dense, 262k ctx)

Model card for Qwen3.8-27B, a 27B dense vision-language model from the Qwen team with a hybrid Gated DeltaNet / Gated Attention layout (64 layers, 16×(3 DeltaNet + 1 attention)), MTP training, and 262k native context extensible to 1M via YaRN. It targets coding, research and long-horizon agentic tasks, with thinking on by default, reasoning_effort levels (xhigh/medium/low) and preserve_thinking. The card gives deployment recipes for Transformers, vLLM, SGLang and TokenSpeed, recommended sampling parameters, video-token settings for hour-scale video, and benchmark notes (SWE-bench Pro, NL2Repo-Bench, DeepSWE, HLE, MathVision, WebArena-Verified, etc.).

Qwen3.8-27BHugging FaceTransformers+8
QwenQwen3.5SWE-Bench Pro+14

models, multimodal, inference· Added 9 Oct 2026

arxiv.orgpaper

Images Talk Back: Guiding MMDiTs by Suppressing Image-to-Text Feedback

The paper studies joint attention in multimodal diffusion transformers (MMDiTs), where the evolving image updates text representations. It finds that this image-to-text (I2T) feedback tends to preserve objects and attributes already in the image, errors included, and that suppressing it improves generation. It proposes Semantic Feedback Guidance (SFG), which guides away from an auxiliary branch that strengthens I2T feedback and weakens text-to-image conditioning, and iSFG, an in-place variant that needs no extra model evaluations. SFG runs for the first 20% of sampling steps (about 1.2x base NFE without CFG). Across three backbones, adding SFG to CFG on FLUX raises GenEval from 0.6661 to 0.7479, and iSFG on SD3.5M adds +1.33 HPSv2.1 over CFG at zero extra NFE. The control also improves VBench scores for text-to-video and image-to-video.

diffusion modelTransformerFLUX+1
SD3.5MGenEvalHPSv2.1+4

research, multimodal· Added 12 Oct 2026, published 9 Oct 2026

arxiv.orgpaper

Hide and Seek: HIDE Concepts to SEEK explanations for black box models

Hide and Seek (HnS) is a concept-level attribution method for black-box vision models. It needs no model weights and no concept-labeled datasets. Given a concept and an image set, it uses a vision-language model to retrieve concept-aligned images and localize the concept, masks the localized region, and takes attribution from the change in the black-box model's output. It agreed better with ground-truth attributions than existing methods on CelebA and COCO-Stuff, and it works on tasks existing concept attribution methods cannot handle, such as image captioning and satellite image segmentation.

vision-language model
Hide and SeekHnSCelebA+1

research, multimodal· Added 12 Oct 2026, published 9 Oct 2026

arxiv.orgpaper

When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Priors

Introduces EICS (Evaluation-Invariant Concept Stability), a metric for checking whether representations consistently encode a concept under controlled one-concept interventions. Also introduces CASP-VL, a benchmark of causally controlled spatial perturbations derived from Visual Genome scene graphs. Across several VLM architectures, spatial relations are represented much less faithfully than lexical information, even for models well above chance on compositional benchmarks. The relation is identifiable at the predicate's own tokens but is lost when the sentence is aggregated into a global representation (EOS token or mean pooling), a failure that output-based evaluation does not reveal.

vision-language model
Visual GenomeEICSCASP-VL+1

research, evals, multimodal· Added 12 Oct 2026, published 9 Oct 2026

x.comtool

Gurasees (@gurasees0) on X

Robo dir is a directory that aggregates robotics and physical-AI datasets in one place. The launch tweet claims 3,494 datasets, 86,424 Hugging Face uploads, 236B+ frames and 2.2M+ hours of data. It also exposes the index through a single MCP server with no sign-in, so agents can search for robot training data programmatically.

Model Context ProtocolHugging Face
Robo dir

training, mcp, multimodal· Published and added 9 Oct 2026

arxiv.orgpaper

Benchmarking Answer-Grounding Consistency in Remote Sensing Visual Question Answering

Proposes the Answer-Evidence Support Protocol (AESP), which checks whether the grounding evidence a model outputs actually supports its own answer in grounded remote sensing visual question answering. It applies task-specific rules to existence, counting and comparison questions. The authors build the G-RSVQA benchmark of 15,027 samples with questions, reference answers and object-level annotations. Tests on seven grounded vision-language models show evidence-support failures on all three question types, with up to 50.82% of correct answers unsupported by their grounding evidence.

vision-language model
Answer-Evidence Support ProtocolG-RSVQAarXiv

evals, multimodal, research· Added 12 Oct 2026, published 9 Oct 2026

x.comthread

Adam Shepherd (@EntryStateGod) on X

Social post announcing Qwen's open-weights release of Qwen-Image-2.1-Turbo, an accelerated checkpoint of Qwen-Image-2.1 on the same 7B visual generation architecture. It generates and edits images in 8 denoising steps, against 40 in the base model's sample code, with 2K output. Qwen claims the fewer steps do not reduce quality. The license is non-commercial, which limits production use.

Qwen
Qwen-Image-2.1-TurboQwen-Image-2.1

models, multimodal· Published and added 9 Oct 2026

arxiv.orgpaper

RARA: Robust Policy Fine-Tuning through Retrieval-Anchored Representation Alignment for Robotic Manipulation

RARA (Retrieval-Anchored Representation Alignment) is a two-stage method for adapting pre-trained robot policies such as Vision-Language-Action models to new tasks from few demonstrations. It first retrieves pretraining representations with similar task intent and motion and aligns the policy's representations of the target demonstrations to them. It then fine-tunes on the target demonstrations while constraining representation change on the retrieved demonstrations, which limits representation drift. Tested with Diffusion Policy and GR00T on 13 tasks in RoboTwin, RoboCasa and on a real robot, it improves average success over plain fine-tuning by 9.5 points in-distribution and 13.6 out-of-distribution, and beats all compared baselines.

GR00TDiffusion Policyvision-language-action model
RARARoboTwinRoboCasa

research, training, multimodal· Added 12 Oct 2026, published 9 Oct 2026

arxiv.orgpaper

ManeuverFlow: Maneuver Hypothesis Generation with Future Evidence for Trajectory Evaluation and Refinement in End-to-End Autonomous Driving

ManeuverFlow is an end-to-end autonomous driving planner that generates trajectories for distinct driving intentions in one step. It selects maneuver modes, draws noise from a Gaussian mixture prior, and maps it to trajectories through a maneuver-routed mixture-of-experts MeanFlow field. Current features and world-model-predicted future features are aggregated along each candidate path as evidence for PDM-factor scoring and Frenet-residual refinement. With a TransFuser-style ResNet-34 backbone it reaches 90.6 PDMS on NAVSIM v1 and 90.1 EPDMS on NAVSIM v2 navtest. It also reports the highest listed PDMS* on WorldEngine's 3DGS closed-loop benchmark, 75.19 with logged traffic and 74.58 with IDM-controlled traffic.

flow matchingMixture of Experts3D Gaussian Splatting+1
ManeuverFlowNAVSIMWorldEngine+6

research, multimodal· Added 12 Oct 2026, published 9 Oct 2026

arxiv.orgpaper

DreamTrue: Action-Faithful Robot World Model with Counterfactual Post-Training

DreamTrue is a multi-view, cross-embodiment robot world model for action-faithful, physically plausible video prediction. It renders action trajectories into image-space conditions aligned to target videos by offline geometric calibration. Counterfactual post-training modifies recorded trajectories to cover unsuccessful interactions and contact configurations. A human-annotated defect dataset trains an embodied video reward model whose scores drive RL post-training. On AgiBot it reaches state-of-the-art action following and cuts the human-assessed interaction defect rate from 48.12% to 6.25%. It ranked first in the world model track of the AgiBot World Challenge 2026.

DreamTrueAgiBotAgiBot World Challenge 2026

research, multimodal, training· Added 9 Oct 2026, published 8 Oct 2026

arxiv.orgpaper

OuroWorld: Bringing Any 3D World Alive as Diverse, Endlessly Looping 3D Cinemagraphs

OuroWorld is a mask-free framework that turns any static 3D Gaussian Splatting scene into a seamlessly looping 3D cinemagraph. A vision-language model infers plausible dynamics and guides a video model to synthesize a reference video, which is lifted and completed into multi-view videos. Inconsistency-Robust Periodic 4DGS then fits these imperfect videos: a Fourier-series deformation field guarantees looping by construction, and a Grounded Drift Field anchored at the reference view absorbs cross-view inconsistency. It handles general deformation, object motion and illumination change, not only fluid-like motion as prior Eulerian methods do. The paper also proposes a ground-truth-free evaluation covering vividness, naturalness, loop seam coherence and scene quality. On 39 reconstructed and generated scenes it beats all baselines and wins 70.8%-99.0% of user-study comparisons.

3D Gaussian Splattingvision-language model
OuroWorldInconsistency-Robust Periodic 4DGSGrounded Drift Field

research, multimodal· Added 9 Oct 2026, published 8 Oct 2026

arxiv.orgpaper

OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning

OmniCapBench is a benchmark of 786 densely annotated videos that evaluates audio-visual captioning in multimodal LLMs. Instead of scoring free-form captions, it asks models for sets of atomic, verifiable units across three tracks (entity references, visual shots, audio events). These are scored with deterministic constraint checks plus localized LLM-based semantic comparisons, which gives coverage, localization and stable scoring together. Tests on frontier MLLMs show strong local perception but weak long-horizon audio-visual reasoning, especially identity drift and cross-modal misalignment.

multimodal large language model
OmniCapBench

evals, multimodal, research· Added 9 Oct 2026, published 8 Oct 2026

arxiv.orgpaper

WOVEN: Weaving Visual World Modeling into Multimodal LLMs

WOVEN is a training source and benchmark for visual transition reasoning in multimodal LLMs. It has 36,076 examples spanning 20 scene types, 5 action types and 8 reasoning types, built from rollouts of video-pretrained generative models. Evaluating 38 frontier MLLMs (including GPT-5.4 and Qwen3-VL-235B-A22B) shows systematic deficits that persist with scale. Training on WOVEN subsets of about 2,000 items improves 22 of 26 external benchmarks by up to 27.3 points, and WOVEN data can replace 30-50% of a task's own training data. The authors derive a recipe: select supervision by the reasoning operation it teaches rather than by scene or action, and prefer larger visual-state changes.

vision-language modelQwen3-VL
WOVENGPT-5.4

research, multimodal, evals, training· Added 9 Oct 2026, published 8 Oct 2026

arxiv.orgpaper

Connected Self Forcing: Beyond Local Learning in Video Autoregression

Connected Self Forcing extends Self Forcing, which trains autoregressive video generators on their own rollouts with KV caching but detaches historical caches and so cuts backward gradients between chunks. It reconnects gradient paths across chunks, passing gradients through generated latents into the computations that produced them, so later predictions can shape how earlier context is generated. Shortcut gradient replay recovers cross-chunk gradients without keeping the full rollout computation graph. Combined with distribution matching distillation, it improves long-horizon visual quality and temporal consistency with no change to inference.

KV cachedistribution matching distillation
Connected Self ForcingSelf Forcing

research, multimodal, training· Added 9 Oct 2026, published 8 Oct 2026

arxiv.orgpaper

PathTime-VLA: Path-Time Decoupling for Factorized Post-Training of Vision-Language-Action Policies

PathTime-VLA reparameterizes Vision-Language-Action actions as a progress-indexed interaction path X(s) plus a positive interval-time profile that defines a monotone time law t(s). Geometry is thus decoupled from execution speed, so teleoperation timing artifacts don't contaminate the path target. A staged post-training recipe uses demonstrations and DAgger interventions for a target-domain prior, Speed-DQN to learn execution-speed multipliers from robot interaction, and Path-AWR to refine the diffusion path generator from rollout outcomes. On three tasks it reaches 58/60 successes versus 57/60 for BC+DAgger at fixed 1×, with about 39–52% shorter mean completion times on successful trials.

Vision-Language-Actiondiffusion modelDAgger+2
PathTime-VLA

research, training, multimodal· Added 9 Oct 2026, published 8 Oct 2026

x.comthread

Marktechpost AI (@Marktechpost) on X

Liquid AI released Open d1, two open-weight "decision models" — d1-3B (text + vision) and d1-omni-600M (text + image or text + audio) — that don't generate text tokens at all. Instead they take a state plus a typed question as input and return calibrated probabilities in a single forward pass, positioning them as lightweight classifiers/judges rather than generative LLMs.

vision-language model
Liquid AId1-3Bd1-omni-600M

models, multimodal· Published and added 8 Oct 2026

decrypt.coarticle

Google Lets You Build a Video Game by Typing a Description

Google launched Playground, a browser-based tool letting U.S. adults 18+ describe a video game in plain language and get a playable, shareable result back without coding, with creation access tiered by Google AI subscription. The piece ties Playground to Google's broader AI-for-games push: DeepMind's Genie (an 11B-parameter model trained on 200,000+ hours of gameplay video) and its successor Genie 3, a real-time interactive world model opened to AI Ultra subscribers via Project Genie, plus an upcoming integration with Unity Spark for 3D, professional-level game mechanics. Google did not disclose which models power Playground or its pricing, and framed it as an early experiment, while the announcement reportedly sank Unity and Roblox stock on fears of AI displacing traditional game-dev tools.

Gemini
GoogleGoogle DeepMindUnity+3

multimodal· Published and added 7 Oct 2026

arxiv.orgpaper

Rephrase Before You Act: Characterizing and Mitigating Language Sensitivity in Vision-Language-Action Models

This paper shows that vision-language-action (VLA) models are highly sensitive to how instructions are phrased: a single-word change (e.g. "stove" vs "hot plate") can shift task success by dozens of points on π0 and π0.5 checkpoints in the LIBERO benchmark, even after rephrase-augmentation finetuning. The authors characterize this with statistically tested single-edit swings and an oracle phrase search, then mitigate it without retraining by having an LLM distill scoring evidence into 10-20 explicit rephrasing rules that rewrite incoming instructions at deployment time, improving frozen π0 by 16-27% relative on held-out tasks and lifting π0.5's in-finetune success from 93.6% to 97.8%.

vision-language model
π0π0.5LIBERO

research, multimodal· Added 8 Oct 2026, published 7 Oct 2026

arxiv.orgpaper

RoboJEPA: Scaling Robotic Latent World Models

RoboJEPA is an 8B-parameter latent world model built on the Joint Embedding Predictive Architecture (JEPA), trained on real-robot data spanning 12 robotic embodiments. The paper establishes the first scaling laws for multi-embodiment robotic world models, showing that imagination error (latent rollout error) follows a second-order power law in compute and predicts downstream planning performance, and demonstrates zero-shot deployment of the model as a goal-image-conditioned planner on real hardware for long-horizon tasks.

Joint Embedding Predictive Architecture
RoboJEPA

research, multimodal· Added 8 Oct 2026, published 7 Oct 2026

x.comthread

Liquid AI (@liquidai) on X

Liquid AI announces Open d1, two open-weight multimodal decision models. d1-3B (built from LFM2.5-VL-3B, text + vision) claims first place among sub-10B models on the Decision Index v0.2.1, while the experimental d1-omni-600M (LFM2.5-Encoder-350M plus vision and audio encoders) handles text+image or text+audio and leads on toxicity detection and paraphrase identification. Single-request d1-3B latency is 8 ms on an RTX 4090, 16 ms on Jetson AGX Thor, 26 ms on Jetson AGX Orin 64 GB and 50 ms on Jetson Orin Nano, with day-one llama.cpp support; demos include live-camera apps and Isaac Sim navigation. Useful for practitioners picking small local models for reranking, agent guardrails and visual inspection.

llama.cppHugging FaceNVIDIA Jetson+4
Liquid AId1-3Bd1-omni-600M+4

models, multimodal, inference· Added 9 Oct 2026, published 7 Oct 2026

arxiv.orgpaper

Self-correction Optimization for Interleaved Multimodal Generation

This paper introduces Self-Correction Optimization (SCO), a training-free method that improves consistency in interleaved image-text generation from multimodal large language models. SCO treats the classifier-free guidance update as a reference signal and applies minimal corrections under two constraints—a new-event constraint for temporal consistency and a state-preserving constraint for visual-subject coherence—avoiding the computational cost of additional training with augmented data. The authors show improvements on interleaved multimodal generation benchmarks and extend the method to video generation tasks like robot manipulation and long-horizon handcrafting.

vision-language model

research, multimodal· Added 8 Oct 2026, published 7 Oct 2026

arxiv.orgpaper

Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving

GeoCoTDrive is a vision-language-action (VLA) framework for autonomous driving that introduces an explicit geometric chain-of-thought: it first grounds 2D regions corresponding to decision-critical cues, then retrieves localized 3D geometric priors from a geometric foundation model within those regions, interleaving these features into the autoregressive context to drive trajectory generation. The paper also introduces "planning-relevant grounding," a new region-level grounding supervision task, and the PlanningGrounding dataset to teach VLAs planning-oriented grounding, showing consistent gains in safety-critical planning across multiple end-to-end driving benchmarks.

vision-language model
GeoCoTDrivePlanningGrounding

research, multimodal· Added 8 Oct 2026, published 7 Oct 2026

arxiv.orgpaper

TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelity

TaoD2C-Bench is a benchmark for evaluating multimodal large language models (MLLMs) on industrial design-to-code (D2C) tasks — converting UI designs into code that satisfies implementation constraints, not just visual fidelity. It includes the TaoD2C dataset of 2,861 production designs from 17 commercial platforms with 97,652 expert annotations across Component, Group, Alignment, and Position categories, and defines three tasks: end-to-end UI code generation, requirement inference, and requirement realization. Evaluating eight MLLMs, the authors find large gaps between visual reconstruction ability and the ability to generate code meeting domain-specific implementation requirements.

vision-language model
TaoD2C-BenchTaoD2C

evals, multimodal· Added 8 Oct 2026, published 7 Oct 2026

arxiv.orgpaper

CARES: A Controlled Synthetic Benchmark of Speaker Reactions to Sound

CARES is a synthetic benchmark of 10,000 two-speaker audio scenes designed to test whether audio-language models correctly identify which sounds are 'salient' — defined as sounds a speaker audibly reacts to. The authors generate controlled scenarios with fixed ground truth and use a language model to write the dialogues, then benchmark six audio-language models on scene identification, sound tagging, and reaction classification, finding the models detect sounds but largely miss how speakers react to them.

CARES

evals, multimodal, research· Added 8 Oct 2026, published 7 Oct 2026

huggingface.coarticle

Introducing Falcon ASR

TII (Technology Innovation Institute) announces Falcon-ASR, a 1.6B-parameter automatic speech recognition model focused on the Emirati Arabic dialect, which also supports MSA, other Gulf dialects, English, French, Spanish and Portuguese under one set of weights. It reports 20.92% average WER across six Arabic benchmarks on the Open Universal Arabic ASR Leaderboard (beating the best published leaderboard result of 23.17%), 22.73% WER / 10.19% CER on an internal Emirati evaluation (beating Qwen3-Omni by 4.07 points), and 5.74% mean WER on the Hugging Face Open ASR Leaderboard's English test sets. The model builds on the team's Falcon3-Audio work, supports word-level timestamps, and is available via a Hugging Face demo space with API access planned.

Hugging Face
TIIFalcon-ASRFalcon3-Audio+7

models, multimodal, evals· Added 8 Oct 2026, published 7 Oct 2026

simonwillison.netwriteup

llm-openai-decisions 0.1a0

Simon Willison built llm-openai-decisions, a plugin for his `llm` CLI tool that wraps OpenAI's newly announced Decisions API (unveiled at DevDay), modeled on his existing llm-typesafe plugin for Jev's competing API. Both APIs support structured yes/no, multiple-choice, and score-style questions, but OpenAI's gpt-6-luna model also accepts image input (at 10 cents per million input tokens, versus Jev's 4.2 cents), and the plugin itself was built by having GPT-6 Astra read OpenAI's docs. The post includes install instructions and a worked example querying an image attachment for a yes/no answer.

OpenAI APIllm
Simon WillisonOpenAIJev+3

prompting, multimodal· Added 7 Oct 2026, published 6 Oct 2026

decrypt.coarticle

Google Launches Nano Banana 2.1: Better Than Its Predecessor at Half the Price

Decrypt reports on Google October launch of Nano Banana 2.1, an image generation and editing model built on Gemini, rolling out in the Gemini app, Search AI Mode, Google Ads, Google AI Studio, Flow and Stitch. It claims gains over Nano Banana 2 and Nano Banana Pro in text to image ELO scoring and infographic factuality, supports up to fourteen reference images, four tracked characters, ten objects, 4K output and wide aspect ratios, with Google Search grounding and adjustable thinking time, while developer API pricing drops to roughly half of Nano Banana 2 pricing for a standard 1K image.

GeminiGoogle AI Studio
GoogleGoogle SearchGoogle Ads+2

models, multimodal· Published and added 6 Oct 2026

deepmind.googlearticle

EmbeddingGemma 2: an open, lightweight multimodal embedding model

Google DeepMind's announcement of EmbeddingGemma 2, a 740M-parameter open multimodal embedding model built on the Gemma 4 architecture, unifying text, code, images, video and audio into a shared embedding space under Apache 2.0. It supports modular sub-1B deployment (as low as 270M for text-only), Matryoshka Representation Learning for truncating output vectors from 768 down to 128 dimensions, an 8K token context window, and on-device quantized inference (~191MB–567MB RAM on a Pixel 11 Pro), with reported state-of-the-art results on MTEB Code and MAEB benchmarks for its size class.

GemmaGeminiHugging Face+6
EmbeddingGemma 2EmbeddingGemmaGemma 4+11

models, multimodal, inference· Published and added 6 Oct 2026

arxiv.orgpaper

World Models' Last Exam in Physics

World Models' Last Exam in Physics introduces a measurement-based benchmark of 40 controlled tasks across mechanics, optics, fluids, thermal/phase-change phenomena, electromagnetism, and surface tension, designed to test whether video world models generate physically consistent outputs. Each task pairs an initial image and prompt with predefined physical criteria, and a custom evaluator combines observability screening with task-specific quantitative measurements rather than relying on reference videos or model-based judgments. Testing eight video generation models across 1,280 generated videos found persistent physical inconsistencies, with the best model scoring only 57.76/100, and the evaluator showed higher agreement with human judgments than a vision-language model baseline.

vision-language model
World Models' Last Exam in Physics

evals, multimodal· Added 7 Oct 2026, published 6 Oct 2026

arxiv.orgpaper

PEARS: Physical-Prior-Guided Efficient Adaptation via Failure Reasoning and Diffusion Steering for Tactile Manipulation

PEARS is a hybrid reinforcement-learning framework for sample-efficient online adaptation of pretrained tactile-manipulation policies to out-of-distribution conditions. A physics-guided force reasoning (PFR) module uses a vision-language model to diagnose failures from visual and tactile history and update contact-force bounds enforced by a hybrid force-position controller, while a separate tactile-conditioned diffusion-steering component perturbs the latent noise of a frozen flow-matching policy to correct free-space motion and contact timing without retraining the base model. It improves success rates by 12.4-37.4 points over strong baselines in simulation, cuts required interaction episodes by up to 53.2%, and reaches 95%/90% success on real-world whiteboard-erasing and pipette-liquid-aspiration tasks.

vision-language modelflow matching
PEARSWhiteboard ErasingPipette Liquid Aspiration

research, multimodal, training· Added 7 Oct 2026, published 6 Oct 2026

arxiv.orgpaper

ALIVE: Interaction-Aligned Object Insertion for First-Frame-Guided Video Editing

ALIVE is a framework for first-frame-guided video editing that inserts objects so they coherently participate in interactions (e.g., being picked up or manipulated) rather than appearing static. The authors curate a 35,800-pair editing dataset combining 3D-rendered, model-generated and real-world videos plus pairs from ROSE, and train a vision-language model to predict interaction guidance from the edited first frame and an instruction naming the object. They introduce the ALIVE-interaction benchmark to measure interaction fidelity, source preservation and visual coherence via a unified VLM-based protocol, reporting a 43.9% Overall improvement on this benchmark and 4.4% on a general video object insertion benchmark over the strongest baseline, with VLM-predicted guidance adding a further 0.95-point gain.

vision-language model
ALIVEROSEALIVE-interaction

research, multimodal· Added 7 Oct 2026, published 6 Oct 2026

arxiv.orgpaper

Post-Training Semantic Lifting for 3D Gaussian Splatting: Separating Detector, Lifting and Representation Error

This arXiv paper introduces a post-training semantic lifting method for 3D Gaussian Splatting that assigns a single target class to Gaussians by accumulating target/non-target evidence across all views, weighted by per-view visibility, then filters with a dual-threshold (β and γβ) scheme that selects high-confidence seeds and extends them via connected components. The authors evaluate by transferring labels from Gaussians to mesh vertices and decompose error into three sources — the 2D detector, the lifting step, and the representation transfer — tuning thresholds on seven Replica scenes and testing on ten held-out ScanNet++ scenes, reaching mean mIoU of 0.93/0.65 (dataset masks/YOLO masks) on validation and 0.80/0.54 on test, a 0.24 mIoU improvement over prior per-view thresholding. The error analysis finds the 2D detector is the dominant remaining error source, informing where future gains in 3D semantic reconstruction should focus.

3D Gaussian Splatting
ScanNet++ReplicaYOLO+1

research, multimodal· Added 7 Oct 2026, published 6 Oct 2026

arxiv.orgpaper

Towards an Extensible Benchmark for Spoken Dialogue with Social Robots

This paper proposes an extensible benchmark for studying spoken dialogue artifacts between humans and social robots, including clarification requests, interruptions, embodied signals like head nods and facial cues, and time constraints. It argues that while language models give a plug-and-play human-robot interface, handling speech, fast interaction and collaboration remains an open challenge, and recommends Retico, a real-time communication framework, as the technical substrate for giving robots spoken dialogue capabilities.

Retico

evals, multimodal· Added 7 Oct 2026, published 6 Oct 2026

arxiv.orgpaper

EC-RAG: Event Chain Retrieval-Augmented Generation for Long Video Understanding

EC-RAG (Event Chain Retrieval-Augmented Generation) is a training-free framework for long-video question answering that organizes video content into an explicit, temporally ordered "event chain" instead of retrieving isolated frames or snippets. It partitions video into semantically coherent segments, fuses multi-modal signals (speech, text, visual cues) at the event level, and retrieves relevant events from the chain to answer queries, working as a plug-and-play addition to existing large video-language models (LVLMs) without extra training. Evaluated on Video-MME, MLVU, and LongVideoBench, it outperforms frame-level retrieval baselines, showing the value of modeling temporal/event structure for long-video understanding.

vision-language model
Video-MMEMLVULongVideoBench

research, multimodal, llm-apps· Added 7 Oct 2026, published 6 Oct 2026

x.comtool

Agent Game Forge (@agentgameforge) on X

Agent Sprite Forge v0.4.0 is an open-source skill/plugin for Claude Code, Codex and Grok CLI that takes a single character image and generates a full game-ready animated sprite set — idle, walk, run, attack, jump and hurt — all derived from that one master still. It adds official API support for OpenAI, Gemini, xAI, ModelArk and fal.ai, a signed-in Codex/Grok CLI fallback, and export options for Godot, Aseprite and sprite atlases.

Claude CodeCodexGrok+4
Agent Sprite ForgeOpenAIxAI+2

coding-agents, multimodal· Published and added 6 Oct 2026

x.comthread

Itsvishal (@itsvishaltwt) on X

A tweet reports that Mistral has announced Mistral Large 4, internally nicknamed "Le Chonk," a mixture-of-experts model with roughly 1 trillion total parameters and 49 billion active parameters. It is described as natively multimodal and state-of-the-art on critical workloads including cyber defense, with API access starting immediately and an open-weight release planned for late October.

Mistral
Mistral Large 4

models, multimodal· Published and added 6 Oct 2026

mistral.aiarticle

Introducing Mistral Large 4 ("Le Chonk")

Mistral announces Mistral Large 4 ("ML4"/"Le Chonk"), a 1-trillion-parameter natively multimodal Mixture-of-Experts model with 52B active parameters, trained from scratch on 3,800 NVIDIA Grace Blackwell GPUs in Mistral's own European datacenters. The post details state-of-the-art open-weight results on cybersecurity benchmarks (82% on an Artificial Analysis vulnerability-reproduction test, 93% on Cybench), agentic coding (DeepSWE, Terminal-Bench, AutomationBench), finance/legal evals (via Vals, Harvey's Legal Agent benchmark), visual grounding (beating GPT-6 Astra on Dense 200), and safety benchmarks (Lakera B3, KORA). It also describes the large-scale RL post-training pipeline (33B tokens/day at 3k GPUs) used to produce the model, with full weights promised by end of month.

MistralMixture of Expertsvision-language model+1
Mistral Large 4GLM-5.3GLM-5.2+20

models, multimodal, evals, training· Added 7 Oct 2026, published 6 Oct 2026

almostimplemented.github.iowriteup

Learning Jazz Pianist Style with Cross-Attention Conditioning

An ISMIR 2026 (Abu Dhabi) research project that fine-tunes Aria, a transformer pretrained on piano MIDI, on solo recordings from twelve jazz pianists in the PiJAMA dataset, adding a gated cross-attention layer that conditions generation on a learned per-pianist embedding. Because standard held-out perplexity is nearly blind to style (the next note is predictable regardless of pianist), the authors instead measure style transfer by sliding a pianist classifier over freely generated continuations: conditioned continuations are attributed to the intended pianist 70% of the time versus 37% unconditioned, and a classifier trained only on generated music identifies real recordings with 95% accuracy. The page is an interactive demo with audio examples, blindfold listening tests, and per-pianist results (e.g. 96% attribution for Hank Jones/Dick Hyman vs 29% for Cedar Walton), alongside a link to the full paper with ablations on mismatch conditioning and memorization checks.

TransformerMIDI
AriaPiJAMAISMIR 2026+8

multimodal, training· Added 6 Oct 2026, published 5 Oct 2026

arxiv.orgpaper

Recursive Video In-Context Learning for Agentic Robot

This paper introduces Recursive Video In-Context Learning (RV-ICL), a training-free method that lets an LLM agent orchestrating frozen vision-language-action (VLA) policies learn from a single demonstration video rather than from full video or fixed keyframes. The demonstration is converted into a navigable hierarchy (task keyframes, phases, moments, short clips) built around sub-events like grasps and releases, exposed to the agent as read-only tools it queries coarsely during planning and drills into for contact-level detail during execution. Built on the RPent agent, it raises task success from 92.6 percent to 96.5 percent on LIBERO-PRO and from 86.7 percent to 95.8 percent on LIBERO-Plus with only one demonstration per task.

vision-language-action model
RV-ICLRPentLIBERO-PRO+1

research, agents, multimodal· Added 6 Oct 2026, published 5 Oct 2026

arxiv.orgpaper

MemPilot: Orchestrating On-Demand Multimodal Memory Curation for LLM Agents

MemPilot is a framework that lets LLM agents curate memory on demand rather than query-agnostically, using a multi-step policy trained via reinforcement learning to choose between retrieving from pre-built memory or delegating query-specific curation of raw multimodal history to heterogeneous LLMs and VLMs. The policy controls evidence amount, curation instructions, model choice, and visual access, and is optimized with objective-wise advantage decoupling and prefix-based marginal utility estimation to balance performance, cost, and latency. Tested across five multimodal agent-memory benchmarks, it achieves broader performance-cost-latency trade-off frontiers than existing trade-off-aware baselines.

vision-language model
MemPilot

agents, research, multimodal· Added 6 Oct 2026, published 5 Oct 2026

arxiv.orgpaper

MedPrune: Topology-Efficient Multimodal Multi-Agent Communication Evolution for Medical VQA Tasks

MedPrune is a multi-agent framework for medical visual question answering that models the diagnostic process as a heterogeneous communication graph of specialist agents across departments, then dynamically prunes both nodes (irrelevant specialist agents, via reinforcement learning) and edges (low-value intra-/inter-departmental connections) to cut redundant communication. The authors report it beats existing multi-agent baselines on medical VQA benchmarks under both full-set and few-shot settings while improving token efficiency and adversarial robustness.

vision-language model
MedPrune

research, multimodal, agents· Added 6 Oct 2026, published 5 Oct 2026

arxiv.orgpaper

Lens3D: Target-Conditioned Visual Foveation for Fine-Grained 3D Understanding

Lens3D is a method for improving fine-grained object understanding in 3D large language models by using 3D localization to select informative 2D views that are fed to an external vision-language model (the LensUnd pipeline), then distilling that knowledge back into native 3D LLMs via detailed caption supervision (LensDistill), avoiding the need for external VLM calls at inference time. The authors also release LensBench, a held-out evaluation set of 2,068 objects with three reference descriptions each, and show gains in fine-grained object captioning on Video-3D LLM and 3DRS while preserving grounding and scene-level QA performance.

vision-language model
Lens3DLensUndLensDistill+3

research, multimodal· Added 6 Oct 2026, published 5 Oct 2026

arxiv.orgpaper

BrainTRACE: Tracing Longitudinal, Multimodal, and Volumetric Evidence in Brain MRI Clinical Reasoning

BrainTRACE is a report-grounded benchmark for evaluating whether vision-language models can trace the evidence structure required for longitudinal brain MRI interpretation. It contains 7,273 scored VQA instances derived from 1,778 longitudinal patients, 7,299 MRI studies, and ~29k co-registered 3D MRI sequence volumes, organized across five levels of clinical reasoning and evidence demands like longitudinal comparison, multi-sequence integration, and volumetric spatial grounding. Evaluation of 20 VLM configurations finds that current models identify isolated visual cues but rarely compose them into grounded longitudinal clinical interpretations.

vision-language model
BrainTRACE

evals, multimodal· Added 6 Oct 2026, published 5 Oct 2026

arxiv.orgpaper

SpatialChain: A Benchmark for Auditing Spatial Reasoning Faithfulness in VLMs

SpatialChain is a benchmark of 28,350 training and 899 test examples pairing spatially-oriented GQA questions with scene-graph-grounded reasoning chains, designed to audit whether VLMs' correct spatial answers reflect faithful reasoning or linguistic shortcuts. Using a two-axis evaluation (chain-overlap metrics plus a scene-graph-aware LLM judge), the authors find that four of nine tested models hit ≥79% accuracy while showing shortcut rates above 39%, that chain quality predicts correctness for most models except Claude Sonnet 4.6 and InternVL3.5-8B (which fail differently, via terse vs. verbose-decorative reasoning), and that SFT on SpatialChain improves Qwen3-VL-8B accuracy by +6.2pp while cutting its shortcut rate to 22%. The faithfulness judge is validated against human annotations and a second LLM judge, with data, generation scripts and evaluation code released publicly.

vision-language modelQwen3-VL
Claude Sonnet 4.6InternVL3.5-8BQwen3-VL-8B+1

evals, multimodal· Added 6 Oct 2026, published 5 Oct 2026

x.comthread

Meer Tarbani - Growth & Engineering (@MeerTarbani) on X

Sidekick is an open-source macOS plugin for Claude Code that lets you define a one-sentence persona (e.g. "a blunt senior Rust dev") and drive a coding session hands-free by voice: it listens via Apple's on-device speech recognition, speaks answers with macOS voices, and lets you interrupt it mid-reply or say "stop" to cancel a turn. It also surfaces Claude's multiple-choice decisions as spoken questions, matching answers like "the second one" against an echo-filtered transcript so the assistant doesn't hear and respond to its own voice. Implementation is a ~500-line TypeScript Claude Code hooks module plus a 90-line Swift speech listener, runs fully local with zero servers and zero telemetry, and installs via the `claude plugin marketplace`/`claude plugin install` commands.

Claude CodeTypeScriptSwift+1
SidekickMeer TarbaniApple

coding-agents, multimodal· Published and added 4 Oct 2026

github.comrepo

Show HN: Made an open-source Lego AI generator

ldraw-nova is an open-source agentic pipeline where an LLM agent (tested with models like Claude Opus 5.5 and GPT-6 Astra) takes a build prompt, reads LDraw-building instructions, and produces a plan.json describing a model's geometry. From that plan it writes a Python generator script that, when executed, outputs an LDraw (.mpd) 3D CAD source file, viewable in 3D/VR (Meta Quest 3) or exportable to Blender-editable glTF. The agent uses jev-rerank, a semantic search tool with reranking backed by TypeSafe's Jev System One model, to find suitable LEGO parts and example models, falling back to full-text search without an API key; it also uses headless rendering and collision/gap detection to self-inspect and iterate on designs.

DockerPythonBlender+2
TypeSafeJev System Onejev-rerank+13

agents, multimodal· Added 3 Oct 2026, published 2 Oct 2026

tavus.ioarticle

Tavus Griffin: the first Human Interaction Model

Tavus introduces Griffin, a "Human Interaction Model" (HIM): a full-duplex, video-to-video system that unifies real-time perception, conversational decision-making, and expressive audiovisual generation into one model rather than a cascaded pipeline. It uses a Continuous Conversational Modeling engine that makes sub-second turn-taking and nonverbal-behavior decisions, paired with a streaming speech generator built on a new continuous audio codec (Tavec) and a distilled autoregressive video diffusion model that renders 720p video in 320ms chunks. In live one-minute video calls, Griffin-Lite was judged human by 48% of participants (versus 2.4% for Tavus's prior Phoenix-4.5 system) and topped NVIDIA's VideoFDB full-duplex conversation benchmark on both generation and perception tracks.

diffusion transformerautoregressive modelflow matching
TavusGriffinGriffin-Lite+14

multimodal· Added 2 Oct 2026

huggingface.corepo

Kyutai Pocket TTS

Pocket TTS is a 100M-parameter text-to-speech model from Kyutai designed to run efficiently on CPU, generating audio faster than real-time (~6x on a MacBook Air M4) with ~200ms latency to first audio chunk and voice cloning from short audio samples. It supports English, French, German, Portuguese, Italian, and Spanish, ships a Python library and CLI, and has spawned numerous community ports (WASM, ONNX, Rust/Candle, MLX, C++, C#) and fine-tuned language variants (Czech, Hindi, Korean, Persian, Indonesian, Estonian, Welsh, Polish).

PyTorchPythonHugging Face+6
KyutaiManu OrsiniSimon Rouard+4

models, multimodal· Added 2 Oct 2026

decrypt.coarticle

This AI Is Already Fooling People on Video Calls Into Thinking Its Human, Company Says

Decrypt reports on Tavus' Griffin, a "Human Interaction Model" built for real-time, full-duplex face-to-face video conversation (listening, watching and talking simultaneously rather than turn-by-turn). On Tavus' own unverified test, Griffin-Lite convinced 48% of 54 participants on a one-minute video call that they were talking to a human, versus 2.4% for Tavus' prior stitched three-model system, and it ranks first on NVIDIA's VideoFDB benchmark with a 3.83/5 generation score (vs. 2.80 for the next-best system and 3.92 for a human reference) and an average audio-to-video delay of 0.43s on H100 GPUs. The article notes the test isn't a classic Turing test, isn't independently verified, and contrasts it with a UC San Diego study where GPT-4.5 passed as human in text conversation 73% of the time, while flagging real-world misuse risk from deepfake video-call scams (e.g., BlueNoroff/Lazarus Group).

NVIDIA H100
TavusGriffinGriffin-Lite+11

multimodal, evals· Published and added 2 Oct 2026

arxiv.orgpaper

IDRF: Inverse-Distilled Reward Fine-tuning of Masked Discrete Diffusion Models

IDRF is a reward fine-tuning framework for few-step masked discrete diffusion generators that replaces the intractable sequence-level KL penalty in reverse-KL-regularized objectives with an inverse-distillation regularization term, proven to upper-bound the true KL divergence to the reference distribution. It frames few-step diffusion sampling as a finite-horizon Markov decision process and optimizes reward via a clipped policy-gradient objective over the student's own trajectories, without needing reference-model rollouts. Across DNA, image, and text generation tasks, IDRF reaches high reward with up to 32x fewer denoising steps than the reference model while reducing reward hacking and preserving sample quality.

Vladislav Gromadskii

training, research, multimodal· Added 5 Oct 2026, published 2 Oct 2026

arxiv.orgpaper

World Embedding Benchmark

The World Embedding Benchmark is a new evaluation suite of 8,000 controlled simulation cases across 80 families (fluid mechanics, solid mechanics, dynamics, optics & electromagnetism), each pairing a rendered video with simulation-derived physical annotations. It tests pre-trained omnimodal embedding models on text-video retrieval, physical-property regression, and multiple-choice video-description classification, finding that current embeddings align poorly cross-modally while lightweight probes can still recover physical quantities, and that continual contrastive training improves alignment but hurts regression. The authors also show retrieval-augmented generation using these embeddings with MiniMax-H3 improves the physical fidelity of generated videos.

MiniMax-H3
World Embedding Benchmark

evals, multimodal, research· Added 5 Oct 2026, published 2 Oct 2026

arxiv.orgpaper

DEPICT: Scoring Text-to-Image Alignment by Answer Agreement

DEPICT is a training-free metric for scoring text-to-image alignment that replaces the fixed-YES assumption of decomposed VQA-based evaluators with an expected-agreement score between image-based and caption-only answers, weighted by how decisively the caption determines each question, then merges this with a holistic score. Evaluated across five benchmarks and eleven vision-language model backbones from three model families, it raises negation accuracy from 19% to 88% and outperforms other training-free metrics while beating fine-tuned evaluators on two of three human-correlation benchmarks.

vision-language model
DEPICT

evals, multimodal· Added 5 Oct 2026, published 2 Oct 2026

arxiv.orgpaper

MobiAgent: Dual-Loop Recursive Policy Self-Improvement for Long-Horizon Mobile Manipulation

MobiAgent is a dual-loop agentic framework for long-horizon mobile manipulation that separates deployment execution from policy self-improvement. An Inner Loop uses vision-language models for receding-horizon planning and visual reflection to compose atomic skills executed by flow-matching experts sharing a unified VLM backbone, while an Outer Loop autonomously segments, verifies, and clusters deployment rollouts to discover new atomic skills and continuously fine-tune the skill library without human annotation. On BEHAVIOR-1K it beats pi_0.5-TA by 22.5 percentage points, and autonomous data recycling raises RoboCasa success from 7.50% to 27.50% and Astribot S1 success from 32.5% to 57.5%.

vision-language modelflow matching
RoboCasaBEHAVIOR-1KAstribot S1+1

research, multimodal· Added 5 Oct 2026, published 2 Oct 2026

arxiv.orgpaper

A Vision-Language Model (VLM)-based Pipeline for End-to-End Procedural Modeling of Field-Grown Maize from Point Clouds

This paper presents an automated pipeline that uses a vision-language model to annotate leaf midlines in rendered orthographic views of maize point-cloud scans, then applies deterministic geometric algorithms and differentiable NURBS fitting to reconstruct editable, organ-level procedural 3D plant models without manual tuning or species-specific segmentation training. Evaluated on 100 field-grown maize plants from the MaizeField3D dataset, it achieves a median whole-plant Chamfer distance of 5.4 mm and recovers 99.4% of curated reference leaves at IoU ≥ 0.5, outperforming an earlier semi-automated pipeline. It demonstrates that VLM annotations can become reliable quantitative measurements when paired with correcting geometric post-processing, enabling scalable generation of 3D plant assets for high-throughput phenotyping and in silico breeding.

vision-language modelNURBS
MaizeField3D

research, multimodal· Added 5 Oct 2026, published 2 Oct 2026

arxiv.orgpaper

CLIMB: Confidence-Guided Complementary Evidence for Multimodal Retrieval-Augmented Generation

CLIMB is a training-free, inference-time framework for multimodal retrieval-augmented generation that improves knowledge-intensive visual question answering. It first builds a compact, non-redundant evidence pool using an MMR-style objective balancing query relevance and redundancy, then applies an R/E/C critic (relevance, evidence specificity, cross-modal alignment) alongside a confidence estimator that only accepts an answer update when estimated confidence increases, giving a built-in stopping criterion without modifying the underlying retriever or MLLM. Experiments on Encyclopedic-VQA and InfoSeek show consistent gains over existing multimodal RAG baselines.

vision-language model
Encyclopedic-VQAInfoSeekCLIMB

llm-apps, multimodal· Added 5 Oct 2026, published 2 Oct 2026

Showing 180 of 327

Show more