🎨 Multimodal327

RoboAPI is a public API from PokeBot, powered by its world-action model Poke-Aurora, that lets robot arms do new tasks from natural-language instructions with no task-specific data collection or training. A user connects a supported arm with a wrist-mounted camera, gives a task such as "clean the table", and receives the next moves in near real time. The post gives no benchmarks, supported hardware list or pricing.

x.comthreadAdded 1 Nov 2026, published 31 Oct 2026

Hyperflow H3 is Video Rebirth's 8-step LoRA for MiniMax H3, released in September 2026 with open weights. Artificial Analysis ranks it #5 on AA-Video-I2V v2.1 and #3 on AA-Video-Editing v1.1, ahead of MiniMax H3 (768p). It costs $9.00 per minute for Image to Video and $18.00 for Video Editing at 1088p, under half of Dreamina Seedance 2.5 ($34.12 and $40.82). The thread gives per-capability and per-use-case breakdowns.

x.comthreadAdded 31 Oct 2026, published 30 Oct 2026

MonsterHouse is a Halloween interactive projection built on the NVIDIA Nemotron Voice Agent: Nemotron Speech ASR, Nemotron 3.5 Bolt E4B 4GB with a LoRA persona adapter, Magpie TTS Zeroshot and Audio2Face-3D in Unreal Engine, on two RTX Blackwell GPUs. Median end-of-speech-to-audio is 1.70 seconds live and 0.72 seconds with cached responses. The post gives a build path and tips on caching and state handling.

huggingface.cowriteupPublished and added 30 Oct 2026

Overtone is a caption format (.overtone.json), a CLI and a video.js plugin that store each spoken word once and cut captions separately per screen shape (16:9, 9:16 or custom). The transcribe pipeline combines gpt-transcribe, Azure AI Speech and optional ElevenLabs Scribe, resolves disagreements with an LLM vote, and uses an LLM editor for corrections. A small public benchmark reports 5.3% word error rate and $0.023–0.025 per audio minute.

github.comrepoAdded 30 Oct 2026, published 29 Oct 2026

RADIOlogy is a training-free, single-pass zero-shot anomaly detection method that uses multi-head attention pooling in vision-language models. It scores anomalies by contrasting average similarity between vision features and sets of normal and abnormal prompts, with no windowed execution. RADIO agglomerative features drive a dense refinement pipeline for pixel-level segmentation, and the method is applied to medical images.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

PhysCount is a benchmark of 546 real-world images and 4,595 human-written questions in 651 target-specific groups. It tests whether multimodal LLMs can separate visible appearances (direct view, reflections, glass) from the physical entities behind them. Across 40 MLLMs, counting appearances is much easier than linking them to entities, and the best model answers every question correctly in only 13.36% of groups.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

DecoyCount is a benchmark from Elorian and Mercor with 1,660 items that tests whether models count objects or fall back on visual shortcuts. Frontier models scored below 50%: GPT-6 Astra led with 43.17%, ahead of Gemini-3.8-flash (32.71%), Qwen3.8-max (28.31%) and Claude-Fable-5.1 (25.22%). CEO Andrew Dai says Elorian's Glimpse model scored 50.48% in a single internal run without tools, and contrasts visual reasoning with world models such as Atlas and Genie 4.

latent.spacearticleAdded 28 Oct 2026, published 29 Oct 2026

A five-task benchmark tests multimodal LLM color judgment against CIE colorimetric ground truth: AWB pair judgment, skin-tone bias direction, CCT estimation, Delta E 2000 pair ordering, and color naming under changing illuminants. Eight endpoints were tested. On ColorChecker-24 stimuli, closed frontier models reach CCT MAE of 715-1056 K versus 1358-2133 K for open-weight models, and a non-ColorChecker control shows this gap is partly template-dependent.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

Granite Switch 4.1.1 is a backwards-compatible patch release of IBM's modular Switch architecture, which configures a language model with specialized, cache-efficient adapters on one endpoint. It updates the Granite Speech ASR model (10 languages, native punctuation and capitalization) and the Decision adapter in granitelib-core, measured on the Jev Decision Index test set. Only the new adapters were retrained; the base model is unchanged. IBM plans tutorials and an Adapter SDK for third-party adapters.

huggingface.coarticlePublished and added 29 Oct 2026

Nemotron 3.5 Transcribe is a free open-weight speech-to-text model from NVIDIA covering 40 languages, including mixed-language speech. The post says it ties for first among open models with a 2.8% error rate. It says the model needs a large GPU such as an H100 or RTX 5090, so it will not run on a laptop. The post gives no benchmark name or model card link.

x.comthreadAdded 30 Oct 2026, published 29 Oct 2026

M-PFT is an angular-margin method for prompt tuning of vision-language models that fixes base-to-novel dual miscalibration (underconfidence on base classes, overconfidence on novel classes). It adds two losses: minimizing the maximum cosine similarity between tunable class embeddings, and minimizing angular distance to each zero-shot embedding. Tests on 9 prompt-tuning methods and 15 datasets show lower novel-class calibration error at comparable accuracy and negligible overhead.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

MedCBR is a two-stage alignment framework for medical vision language models. Difficulty Weighted Boundary Preference Optimization (DBPO) trains on counterfactual preference pairs at the model's capability frontier. Entropy Gated Grounded Policy Optimization (EGPO) then applies distilled clinical rubrics according to prediction entropy. It reaches 61.2% average accuracy on six medical benchmarks, with the largest gains on differential diagnosis, and the two stages combine super-additively.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

Nemotron 3.5 Transcribe is an open-weight, promptable NVIDIA mixture-of-experts model with 30B total and about 4B active parameters. One checkpoint does transcription in 40 languages, speaker attribution, translation, contextual prompting and audio understanding. The post says it ranked first in VoiceArena's comparison of 14 diarization plus ASR systems. The fetched text is cut off and gives no license or download details.

x.comthreadAdded 30 Oct 2026, published 29 Oct 2026

AnySpeechRecognizer and AnyDecisionModel are two Swift packages from Hugging Face. The first mirrors Apple's SpeechAnalyzer API with swappable engines (NVIDIA Parakeet, Alibaba Qwen3-ASR on Core ML) and adds Sortformer speaker diarization. The second returns typed yes/no, choice and score answers with probabilities via MLX, Core AI, llama.cpp, Jev or Ollama. The post gives M4 Max timings, model sizes and calibration options.

huggingface.coarticlePublished and added 29 Oct 2026

This paper studies whether vision-language models can infer how long a person stays somewhere (living, staying, or leaving) from geotagged social media posts. The authors build a new corpus of geotagged tweets co-annotated by humans and LLMs. State-of-the-art models struggle even with multi-turn Chain-of-Thought prompting, often missing contextual cues and subtle hashtags while over-relying on prominent visual details.

arxiv.orgpaperAdded 30 Oct 2026, published 28 Oct 2026

inkGround is a benchmark for stroke trajectory recovery from full-page, camera-captured handwriting, with 537 page images (about 18.7k words, 163k strokes) and pseudo-label references filtered by OCR agreement and human raters. Models get a page image and a target word, then output ordered stroke points. Seven VLMs were scored with DTW, Chamfer distance, readability and style similarity. Closed-source models, notably Gemini-3.1-Pro, did best, and the ranking held on IAM-OnDB.

arxiv.orgpaperAdded 30 Oct 2026, published 28 Oct 2026

AskStream is a multi-turn benchmark for streaming video understanding that adds a third proactive action, ask, alongside wait and answer, for cases where video or user information is incomplete. It has 438 videos, 2,714 manually annotated frame-level time windows (wait, ask, answer), four task categories and eight subtasks. The strongest model reaches only 52.18% on the proposed Ask-F1 metric, in both multiple-choice and open-ended settings.

arxiv.orgpaperAdded 30 Oct 2026, published 28 Oct 2026

QbD-VLA is a mixed-precision post-training quantization method for vision-language-action models. It uses task success from short closed-loop evaluations to assign weight and activation bit widths, accounting for interactions between components. On four VLA models across all four LIBERO suites it holds task performance at about 4-bit weights and under 6-bit activations. The gain on SIMPLER is small. On a Franka Research 3 arm, X-VLA shows no detected loss, but π0.5 fails.

arxiv.orgpaperAdded 30 Oct 2026, published 28 Oct 2026

Nemotron 3.5 Transcribe is an open-weight, promptable 30B mixture-of-experts model from NVIDIA with about 4B active parameters, built for recorded audio. One checkpoint transcribes 40 languages, handles recordings up to an hour, labels up to eight speakers, translates, and answers questions about English audio. Weights and a demo are on Hugging Face. The post gives no benchmarks.

x.comthreadAdded 29 Oct 2026, published 28 Oct 2026

Nemotron 3.5 Transcribe is an open-weight, promptable speech model that handles transcription in 40 languages, translation, diarization for up to eight speakers, and audio Q&A. It has about 4B active of 30B parameters and tops open-weight ASR and diarization leaderboards. The post also covers the Granary v2 curation pipeline (463,062 hours, 48 languages) and a medical fine-tuning case study that cut held-out WER by 45.8%.

huggingface.coarticlePublished and added 28 Oct 2026

This paper studies data scaling laws in stereo matching and releases XStereo, a realistic synthetic dataset of 200 thousand samples. In-domain training on five large datasets shows two regimes: a standard power law, then saturation, with a transition point roughly fixed for a given model. Cross-domain performance depends only weakly on source data scale, and distribution and coverage diversity reduce domain gaps more.

arxiv.orgpaperAdded 30 Oct 2026, published 28 Oct 2026

The paper studies detection and temporal localization of fraudulent inducements (prompts to follow, join chat groups, click links) in internet finance short videos. It compares OCR+LLM, OCR+MLLM and a zero-shot MLLM, and builds a dataset of 2,419 videos (66.49 hours, 3,194 events, four categories). OCR+MLLM reaches mIoU 26.52 versus 14.49 for keyframe OCR+LLM, with 67.0 percent macro recall, using a recall-weighted F2 localization reward.

arxiv.orgpaperAdded 30 Oct 2026, published 28 Oct 2026

This paper proposes an fMLLR-based compensation mechanism that cuts the fine-tuning data needed in transfer learning for speech emotion recognition under train-test acoustic mismatch. An autoencoder reduces trainable parameters, and an affine transform in the encoder latent space models rotations and shifts. In a human-robot interaction task, mean CCC rose 16.8% with 12.9% of the training set and 14.4% with 50%.

arxiv.orgpaperAdded 30 Oct 2026, published 27 Oct 2026

Know, then Draw is a learning paradigm for visual generation that uses autoregressive pretraining to acquire semantic knowledge and diffusion post-training for high-fidelity rendering. Controlled comparisons find autoregression the stronger pretraining objective, and semantic quality matters more than AR-only generation quality. This motivates a purely semantic tokenizer trained with language supervision only. A 3B text-to-image model trained in 6,600 H100 GPU-hours scores 86 on GenEval and 87 on DPG-Bench.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

This paper presents an annotated egocentric tennis dataset (egocentric video synced with five allocentric views, global positions, contact times, stroke types) and evaluates image encoders, video encoders and vision-language models on seven prediction tasks. Best models reach 0.14 m player and 0.54 m ball localization error and 0.14 s contact-time error. Depth input helps 3D ball localization, and longer histories do not help forecasting. A conditional diffusion model generates whole-body tennis motion in real time.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

VideoLVR is a method that lets video multimodal LLMs build latent visual evidence from local regions at relevant moments and reason over it interleaved with text tokens. A visual token prediction objective, defined over a vocabulary derived from the input video, supervises the latents. It extends to reinforcement learning with sampled visual tokens as actions and final-answer rewards, and improves fine-grained video benchmarks without much context growth.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

This paper defines a character naming task for films: given a name spoken in dialogue and its video context, the model must pick which visible character bears that name. It provides a dataset with annotations for character naming and first appearances, a video-and-dialogue model, and a comparison with proprietary multimodal LLMs. It also proposes a progressive framework that builds a character bank as the film plays, without cast lists or actor photographs.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

LFM3 is Liquid AI's open-weight family of six ~2B-parameter models (text, vision-language, audio, omni and two d1 decision models), all branched from LFM3-1.7B-Base. The base model uses a tri-hybrid backbone of 13 gated short convolutions, 5 sliding-window attention layers and 6 global attention layers, pre-trained on 22T tokens with a 256K context. The post gives transformers snippets, benchmark comparisons against LFM2.5 and Qwen3.5, and a robotics fine-tuning example.

huggingface.coarticlePublished and added 27 Oct 2026

LFM3 is Liquid AI's new family of open-weight multimodal models, six models of about 2B parameters branched from LFM3-1.7B-Base (22T pre-training tokens, 256K context). The hybrid backbone uses 13 gated short-convolution, 5 sliding-window attention and 6 global attention layers. Variants cover text, vision with video, audio, omni and two decision models, with on-device speed numbers, fine-tuning demos and robotics extensions.

liquid.aiarticleAdded 28 Oct 2026, published 27 Oct 2026

Genie 4 is Google DeepMind's real-time interactive world model, announced as a research preview for trusted testers. It generates video with native audio and interactive dialogue, steered by keyboard controls and text prompts. Gemini 3.8 Flash acts as a guiding agent that plans steps and checks frames, and Gemini 4 Argon is used for robotics experiments. The post lists limitations such as drift and object permanence, and says outputs carry SynthID watermarks.

deepmind.googlearticlePublished and added 27 Oct 2026

Genie 4 is Google DeepMind's world model for real-time interactive simulation, with native audio generated on the fly (ambient sound, music, dialogue). It builds on Gemini models: Gemini Omni grounds scenes in real-world appearance and behavior, and other Gemini models keep the simulation coherent. Users steer scenes by natural language. Access is limited to trusted testers now, with Google AI Ultra (Project Genie), the Gemini API and AI Studio to follow. The post gives no architecture or benchmark detail.

blog.googlearticleAdded 30 Oct 2026, published 27 Oct 2026

This paper studies how multimodal LLMs refuse questions that an image cannot answer, using the MoHoBench benchmark on three MLLMs. It varies the meaning and placement of refusal instructions, compares supervised and preference-based post-training, and intervenes on a refusal-related activation direction. Instructions change refusal at the sample level in ways aggregate rates hide. Post-training raises refusal most but can mix refusal with attempted answers. Activation intervention gives smaller, transferable changes.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

heditor is an open-source (MIT) agent skill from pycodebr that has Claude Code, Codex, Cursor or OpenCode edit talking-head videos. It transcribes with faster-whisper, picks a 30-70s segment, cuts frame-accurately, builds HTML + GSAP motion graphics, mixes audio at -14 LUFS, and runs QA with vision-reviewed previews. It offers 42 named styles and a long-form mode with chapters, SRT/VTT and lower thirds. Install: npx skills add pycodebr/heditor -g.

x.comthreadAdded 28 Oct 2026, published 27 Oct 2026

Hi3D v1.0 (formerly Sparc3D) is an open-sourced image-to-3D model from Hi3D, which runs a closed v3.0 refining at 2048 cubes per side. In a test with 13 professionals in 3D algorithms, 562 judgments against eight other image-to-3D models favoured it 83.5% of the time. The thread also covers the Hi3D Agent and an MCP server for other agents. Back and bottom views were checked, where TRELLIS.2 collapsed a winged figure.

x.comthreadAdded 28 Oct 2026, published 27 Oct 2026

AxisQ is a post-training quantization framework for camouflaged object detection (COD) that allocates activation bits by feature depth. It uses the channel domain for shallow features and the local-frequency domain for deep features, under a fixed per-layer average bit budget. It adds a boundary-aware Hessian approximation for reconstruction. On CAMO at W2A2, weighted F-measure improves 12.7% for CFRN and 33.3% for CamoFormer-P over the strongest baselines.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

GPT-6.1 Terra is OpenAI's midrange model, announced 26 October 2026 and aimed at visual reasoning, computer use and clearer writing. It costs $1 per million input tokens and $5 per million output, about half of GPT-5.6 Terra and GPT-6.1 Sol and roughly 10x GPT-6 Luna. Simon Willison relays OpenAI's Ted Sanders, who rates it better than Astra and Sol at technical communication and much faster on computer use.

simonwillison.netarticleAdded 27 Oct 2026, published 26 Oct 2026

This article reports a study submitted October 23, 2026 by Qiangong Zhou, Jiafeng Liang, Yimin Dong, Yang He and Hongxiang Li. They added Test-Time Training (fast-weight memory) to the vision-language-action models InternVLA-A1 and Pi0.5, trained on paired human demonstrations and robot trajectories. Shorter videos, full-trajectory gradient connectivity and contrastive regularization gave no clear gain in demonstration-guided target selection at the tested scale.

en.cryptonomist.charticlePublished and added 26 Oct 2026

RoSA is a finetuning framework from Qiuyu Kong and three co-authors for long-text image-text retrieval. It targets "saliency collapse", where models over-rely on text tied to prominent visual features and miss contextual information. RoSA splits pairs into salient and contextual views using object-level saliency estimation and LLM text decomposition, then trains on both with plug-and-play auxiliary alignment objectives. Code and checkpoints are public.

en.cryptonomist.charticlePublished and added 26 Oct 2026

A dermatologist tested GPT-6 Astra and Claude Fable 5.1 on 200 biopsy-proven lesions from MILK10k, giving each a clinical and a dermoscopic image. A CNN (Inception v3, fine-tuned for about twenty minutes on 68,011 ISIC images) named the right diagnosis for 62% of lesions, against 42% for both LLMs. The post lists caveats (small sample, single run, possible contamination) and links the cases, prompts, replies and scoring code.

lynchlab.netwriteupAdded 26 Oct 2026, published 25 Oct 2026

A stage-aware visual attribution framework for Vision-Language Navigation (VLN) that explains a robot's action choices. It identifies "instruction-stage insensitivity", where existing explainers give similar outputs across navigation stages. The method builds action-conditioned temporal instruction-visual attributions, aggregates relevant historical evidence by stage, and applies an instruction-guided spatial filter. It improves attribution quality across several VLN policies and benchmarks, helps diagnose failure cases, and was tested on a Unitree G1 robot.

arxiv.orgpaperAdded 27 Oct 2026, published 25 Oct 2026

This paper presents a pipeline that uses persona-conditioned vision language models to surface interpretive ambiguity in visualizations. Topic-relevant personas elicit candidate takeaways, and a Target-Pattern-Scope framework lets LLM judges label statement pairs as Compatible, Disjoint or Opposing, which are summarized in an ambiguity profile. The authors benchmark VLMs on chart comprehension per chart type, check labeling against human annotations, and run case studies on public visualizations and a redesign.

arxiv.orgpaperAdded 27 Oct 2026, published 25 Oct 2026

InteractiveWorld is an environment for closed-loop interaction between an AI assistant and a simulated human agent who sees the scene, follows instructions and acts independently. InteractiveBench, built on it, has 120 multi-turn tasks in six categories scored by verifiable, state-based rewards. Across 12 frontier MLLMs, the best model, Claude Fable 5.1, reaches at most 20% on long-horizon Guidance and 25% on Planning tasks.

arxiv.orgpaperAdded 27 Oct 2026, published 25 Oct 2026

Dessa is a voice AI model from Block, trained from scratch on human speech only, with no text stage. It listens while speaking, so it handles interruptions, laughter and tone changes. Block says it trained on one Nvidia GB200 NVL72 system and that it runs locally on a MacBook Pro with M4 Max or better. Model size, benchmarks and release date are undisclosed. Early access goes to a small group of AI safety researchers. Reactions split between praise and scam fears.

coindesk.comarticlePublished and added 24 Oct 2026

OWLS (Open-world Language-conditioned Speed Fields) plans directly over a vision-language model's dense feature map. A text prompt describing traversable terrain scores every pixel of one camera image. The scores are projected onto the ground, adapted with two morphological operations, and used as the speed field of a minimum-time planner. It needs no training, terrain labels, map or extra sensors. It was tested on a physical rover, on simulated Mars and Moon rovers over 200 procedural terrains, and on real planetary imagery.

arxiv.orgpaperAdded 26 Oct 2026, published 23 Oct 2026

DeepFaune-UK is a fine-tune of the open DeepFaune camera-trap classifier on 28 UK classes, paired with MegaDetector v6. On 20,212 MammalWeb images it reached 88.8% strict accuracy against 56.5% (DeepFaune v1.5.0) and 59.0% (SpeciesNet). Re-applying the recipe to the DINOv3-based classifier gave 90.6%. Fine-tuning on about 150 crops per class matched full-data out-of-distribution accuracy, and a DINOv2 run took about 22 minutes on under 4 GB of GPU memory.

arxiv.orgpaperAdded 26 Oct 2026, published 23 Oct 2026

UAV-SwarmAgent is a hierarchical multi-agent system where an operator gives one natural-language sentence and a ground-station agent writes and runs code that produces per-UAV routes. A vision-language model grounds targets in an aerial map, and worker agents verify incidents with a compact onboard model. In simulation it located the incident in 244 of 255 missions (95.7%), and flight time followed an Amdahl-type law with speedup bounded at 1.52x for one to five UAVs.

arxiv.orgpaperAdded 26 Oct 2026, published 23 Oct 2026

Signal-Aware Adaptive Fusion (SAAF) is a training-free zero-shot video anomaly detection framework that cuts the compute cost of vision-language models. It combines a dual-signal sampler (frame-to-frame delta and rolling-context drift in embedding space, OR-gated with EMA thresholds), a threshold-gated cascade from frame-level to video-level analysis, and signal-aware score fusion. It reports AUC-ROC of 92.07% on XD-Violence, 86.70% on MSAD and 80.54% on ShanghaiTech, at 2.2x to 2.6x the throughput of the prior best zero-shot method.

arxiv.orgpaperAdded 26 Oct 2026, published 23 Oct 2026

MART-GRPO is a variant of group-relative policy optimization for vision-language-action (VLA) reinforcement learning. It uses a LoRA-fine-tuned VLM to find the last visible error without recovery in a failed rollout video, keeps failure feedback up to that point, and drops it afterward. Martingale stopping conditions show this preserves the expected GRPO gradient and can reduce covariance. With OpenVLA-OFT on RoboTwin 2.0, it raises success rates over full-trajectory GRPO across multiple tasks.

arxiv.orgpaperAdded 26 Oct 2026, published 23 Oct 2026

4DMemory is a benchmark suite for long-horizon spatiotemporal memory: the LongVideo dataset (about 900 hours of video, over 5,000 manually reviewed questions) and 4DMemoryBench. It also presents the 4DMemory Harness, combining four skills, 22+ tool models and a model-specific RAG system. Across 13 models, the harness improved every one by over 50%, and the same models scored over 10% lower than on a previous benchmark.

arxiv.orgpaperAdded 26 Oct 2026, published 23 Oct 2026

MAST-VLA is a vision-language-action framework for multi-skill robot manipulation that conditions on both the skill to execute and where and how to contact objects. It adds skill-conditioned LoRA Mixture-of-Experts adapters to the LLM backbone and turns a predicted affordance heatmap into a patch-level spatial prior for the action head. It reaches an Avg. Len of 4.65 on CALVIN ABC→D and was tested on a real robot.

arxiv.orgpaperAdded 26 Oct 2026, published 23 Oct 2026

Sensorimotor Alignment (S^2) is a training-free offline metric that measures how well an embodied model's sensory representation aligns with the geometry of physical motion. It extends the Platonic Representation Hypothesis to motion trajectories. S^2 correlates positively with downstream robot task success across embodied model families, helps pick promising VLM backbones before fine-tuning into VLAs, and explains behaviours seen in VLAs and world action models.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026

F2G-Pose is a direct, single-pass framework that predicts category-level object pose, metric size and camera-frame completed shape from one RGB-D image. It lifts frozen RADIO foundation features onto partial point clouds, tokenizes local geometry with DGCNN, and fuses it with a Mixture-of-Experts Transformer. It trains only on synthetic SOPE data, leads on SOPE and ROPE at 28 FPS, and is tested on HouseCat6D, HANDAL and real manipulation.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026

Progress Probes (P^2) trains linear probes on a vision-language-action model's own activations, using demonstrations, to estimate task progress, then flags executions that deviate from expected progress. Tested on the open-source pi0.5 across simulated and real-world manipulation tasks, it matches or beats VLM-based monitors and failure-data-trained methods at failure detection, without an external foundation model.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026

TTR-Compressor is a sequence compression method that treats input tokens as a test-time regression problem. The regression is learned end-to-end by backpropagation, and its coefficients are solved in closed form in the forward pass to give the compressed representation. As a block-wise sparse attention compressor it improves RULER over attention-based compressors by a large margin. As a visual token merger for multimodal LLMs it beats the MLP merger and matches it with 55.6% fewer visual tokens.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026

CellTrack-Bench is a benchmark that tests vision-language models on microscopic cell tracking and compares them with specialised trackers. It has 1,537 problems covering 9,041 cell-linking decisions, drawn from 15 cell-tracking datasets across four imaging modalities, with a hard subset built from failures shared by distinct algorithmic trackers. The authors compare environment harnesses, benchmark 9 closed-source and 13 open-weight models, and analyse failure modes.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026

DuoCook is an Overcooked-inspired benchmark in which one vision-language model controls two cooks from top-down views in Unreal Engine environments built with coding agents. It has 20 maps in five categories covering equipment contention, changing terrain and handoff dependencies. Four VLMs were compared; the best, gpt-6-astra, reached 26.2% mean order completion. A notebook memory ablation had mixed effects: positive for Claude and Qwen, negative for GPT and Gemini.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026

OlmoEarth v1.3 is an open satellite embedding model from Ai2 that reads a year of Sentinel-1, Sentinel-2 and Landsat data and outputs a 128-dimensional vector per 10 m pixel (the first 64 dimensions also work alone). Its embeddings beat AlphaEarth Foundations and Tessera v2 on 20 of 27 evaluations. Precomputed global embeddings, a browser Embeddings Explorer and pre-training cost (about $2,720 of H100 time) are covered.

huggingface.coarticlePublished and added 22 Oct 2026

Rate-Controlled Refresh (RCR) is a KV cache re-selection method for multimodal LLMs in long-form generation. Attention drift decides when and where to re-select visual tokens, and a per-sample controller sets how often. RCR beats InfiniGen by 1.6-3.8 ROUGE-L points on five of six model-task settings and stays within 1.7 points of exact per-step re-selection. On video description, where drift is homogeneous, it no longer beats a fixed threshold.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026

Forest Typology (ForTy) 2020 is a global 10-meter map from Google DeepMind, Google Research, WRI and IIASA. It sorts land into six FAO-based classes: primary, naturally regenerating, planted, plantation, tree crops and agroforestry, and other land. The model uses AlphaEarth Foundations embeddings and about 1.7 million labeled locations. It was 90% accurate on forest type across 9,000+ expert-checked sites. An early 2017 to 2025 time series is on Earth Engine.

deepmind.googlearticlePublished and added 22 Oct 2026

OnKey is a free, MIT-licensed singing trainer for Apple Silicon Macs. After a song ends in Spotify, Apple Music or browser YouTube, it separates the lead vocal with UVR's MDX-Net Karaoke 2, tracks pitch with CREPE and turns it into syllable-level notes with the GAME model. On later plays it scrolls the notes, draws the user's live pitch and scores each line. Analysis runs on-device, and the repo explains the build steps and network requests.

github.comrepoAdded 24 Oct 2026, published 22 Oct 2026

TRIAD is a benchmark for omni-modal ambiguity resolution. It has 327 hand-written text-image-audio queries in 86 scene groups across 18 ambiguity categories, and removing any one modality makes the answer non-unique. Across 14 omni-modal systems, the best model trails humans by over 20 points in group accuracy. Without answer options, model group accuracy falls to single digits, and leave-one-out tests show audio cues are often underused.

arxiv.orgpaperAdded 23 Oct 2026, published 22 Oct 2026

Matrix-Game 4.0 is a system that separates game logic from visual rendering for playable generated worlds. A lightweight engine holds world state and rules, a multimodal coding agent writes game code and scene captions, and a distilled autoregressive video model turns low-poly proxies into detailed frames. It runs 480p at 60 FPS on one RTX 5090 with 112 ms key-press-to-display latency. Code, weights and assets are released.

huggingface.copaperAdded 22 Oct 2026

XiaoZhi ESP32 is an open-source, MIT-licensed firmware stack that turns ESP32 boards into MCP-based voice chatbots using models such as Qwen and DeepSeek. It streams Opus audio over WebSocket or MQTT + UDP through ASR, LLM and TTS, adds offline ESP-SR wake words, and uses device-side MCP to drive speakers, LEDs, servos and GPIO. It supports 138 board directories and builds with ESP-IDF 6.0.1 or newer.

x.comthreadAdded 23 Oct 2026, published 21 Oct 2026

Granite Speech 5.0 is IBM's update to its open speech model collection: the new granite-speech-5.0-2b, a multilingual granite-speech-5.0-470m-turboctc, and a revised granite-speech-4.1-2b-plus. The 2B model ranks #2 among open-weight models on average English WER (5.06%) on the Open ASR Leaderboard. Turbo CTC reaches 19,870 RTFx, and the 4.1 Plus revision cuts long-form WER 24%. All are Apache 2.0.

huggingface.coarticlePublished and added 21 Oct 2026

GPT Voice is a free, open-source Mac app by Vishesh Raheja that brings ChatGPT's dictation to any macOS app. A shortcut starts recording, and the transcript is pasted where the user is typing. It uses the ChatGPT Pro subscription, so it needs no API key or extra subscription. The author built it after Handy handled coding prompts poorly and they did not want to pay for Wispr Flow. The thread gives no details on local versus API transcription or latency.

x.comthreadAdded 22 Oct 2026, published 21 Oct 2026

Cosmos3-H-Dreams is NVIDIA's real-time generative surgical simulator, trained on clinical cholecystectomy video and kinematics from the CMR Surgical Versius platform. It builds on the 16B Cosmos3-H-Embodied-Nano, distills a bidirectional teacher into four-step and two-step causal students with Self-Forcing, and streams through FlashDreams on one RTX PRO 6000 GPU. Checkpoints and a recipe for custom data are released.

huggingface.coarticlePublished and added 20 Oct 2026

Krea 3 is an image generation and editing model from Krea. A post on X says its "Fast" variant scores 1125 Elo on the Artificial Analysis image editing leaderboard, against 1124 for Google's Nano Banana 2, while producing 10 images in the time Google produces 2. It also says the "Ultra" variant ranks in the top 4, behind only OpenAI models. The post gives no methodology or model details.

x.comthreadAdded 21 Oct 2026, published 20 Oct 2026

UniWALM is a unified world-action-language model that learns video, action and language in one mixture-of-transformers backbone with per-modality diffusion heads that co-denoise together. Generated subtask and textual memory are fed back on the next step, so no external planner or memory bank is needed. It adds Flow Matching with Correction (FMC) for recovery from perturbed actions. It reaches 34.5% on the memory-dependent RMBench subset, above Mem-0's 28.5%, and lifts real-world Find-the-Item success from 11.1% to 61.1%.

arxiv.orgpaperAdded 22 Oct 2026, published 20 Oct 2026

InterWebCoder is a framework for interactive webpage code generation by multimodal LLMs. It synthesizes self-contained webpages with executable test scripts from recorded interaction trajectories, repairs failures using runtime evidence, then trains with test-ID-weighted SFT and GRPO using test pass rates and visual similarity as rewards. The pipeline yields 1,736 verified samples from 2,058 tasks. A 9B model scores 48.54 on IWR-Bench and 48.42 on Vision2Web, and reward latency drops over 97%.

arxiv.orgpaperAdded 22 Oct 2026, published 20 Oct 2026

AnomalyAgent is a post-training fusion framework for visual anomaly detection that ensembles pretrained detectors. It aligns scores and anomaly maps, summarizes disagreement, and treats fusion as a choice among score actions and map actions. A constrained vision-language model inspects high-conflict samples and picks the actions. It beats equal-weight fusion and gets the best mean image AUROC and pixel AP among compared fusion methods on MVTec AD and VisA.

arxiv.orgpaperAdded 22 Oct 2026, published 20 Oct 2026

Code2Video-Compass is a benchmark of 2,104 tasks for code-based video generation, where multimodal LLMs write executable programs that produce complete videos from text (T2V), image (I2V) or video (V2V) inputs. Tasks span 10 categories and 24 subtypes. Scoring uses 12 objective metrics and five MLLM-judge metrics. Across 15 proprietary and open-weight models from nine families, performance falls as construction, dynamic animation and state-dependency complexity rise.

arxiv.orgpaperAdded 22 Oct 2026, published 20 Oct 2026

DIVE is a benchmark for full-duplex interaction with visual engagement in continuous audio/video streams. It has 525 human-annotated samples across six tasks in three scenarios: proactive intervention, continuous engagement and interaction coordination. Evaluated real-time models reach sub-second average latency, but window response rates stay at or below 53.77%, and adapting ongoing responses to new information is hardest.

arxiv.orgpaperAdded 22 Oct 2026, published 20 Oct 2026

Jev Cards is a browser picture quiz where a player competes against Jev 1.14, a vision decision model. The 490 cards cover fruit, flags, maps, chess and illusions. Jev reads the image and question and returns a probability for every option, or a box for "tap the place" cards, in one call of about 0.25 seconds. It never writes text. The page reports 5,256 wins in 6,820 games.

jevcards.comtoolAdded 21 Oct 2026, published 20 Oct 2026

This paper presents a multimodal engineering GraphRAG framework that links post-wind-event damage evidence from images and reconnaissance reports to building-condition states, recovery actions and downtime. Evidence is stored as a provenance-preserving, component-level representation, and graph-aligned retrieval uses compatibility, contradiction, evidence-precedence and abstention checks. Demonstrated on tornado-affected low-rise homes, it beats BaselineRAG and Structured Baseline RAG, and ablations test each component.

arxiv.orgpaperAdded 22 Oct 2026, published 20 Oct 2026

OpenVLHarness is a multimodal reasoning harness for MLLMs built on a capability layer (task-level tool interfaces plus evidence rendering) and persistent multimodal memory that stores images, structured data and web references as addressable artifacts. A coding interface runs generated programs over stored data. Across 22 benchmarks and eight backbones it adds 9.7-12.6 points over no-tool baselines, and GPT-6 Luna at medium effort beats Codex and Visual Sketchpad at maximum effort using at most 37% of their tokens.

arxiv.orgpaperAdded 21 Oct 2026, published 19 Oct 2026

The paper introduces PhysRSI-eval, a five-axis metric for physical recursive self-improvement: first-attempt performance, efficiency, success, budget and improvement from failure. Several VLMs are tested on 18 manipulation tasks under a fixed interaction budget. It proposes the Physical Barrel Hypothesis and studies observational scaling. Workspace interventions show that removing history lowers late-round success, while frozen workspaces still help on unseen layouts.

arxiv.orgpaperAdded 21 Oct 2026, published 19 Oct 2026

This preprint grades evidence for pathology vision-language models and tests four claims with controlled probes on released checkpoints and public data. Stage-redacted report embeddings raise pooled C-index by 0.019 (progression-free interval) and 0.016 (overall survival) across 1,633 TCGA patients. Zero-shot prompting matches a linear probe. SlideChat scores 74.82% with the slide and 74.41% without it. Paraphrasing flips 8-23% of answers.

arxiv.orgpaperAdded 21 Oct 2026, published 19 Oct 2026

BrainLoRA is a parameter-efficient fine-tuning method that adapts multimodal backbones of up to 104B parameters to predict neural responses. It predicts multiple timepoints in one causal pass instead of sliding windows, and a multi-LoRA kernel trains many adapters on a shared backbone. Across nine backbones it improves encoding, and a 200+ model ensemble ranks first on the Algonauts post-challenge leaderboard.

arxiv.orgpaperAdded 21 Oct 2026, published 19 Oct 2026

Mise-en-Video (MeV) is a multi-shot video generation framework where creators stage shots in 3D space and then generate the matching video. It combines hierarchical attention for multi-level narrative consistency with 3D-aware layout guidance for per-shot spatial control. The authors built a 160K-video dataset with hierarchical multi-shot texts and oriented 3D bounding boxes, and report gains over baselines in quality, consistency and subject controllability.

arxiv.orgpaperAdded 21 Oct 2026, published 19 Oct 2026

HAF-VLA is a hierarchical action-flow framework that adapts a pretrained flow-matching vision-language-action model to humanoid whole-body loco-manipulation. It denoises a shared action space in stages: locomotion and head commands first, then waist refinement, then full-body manipulation. Learnable stage embeddings and cross-stage KV-cache conditioning carry dependencies between stages. The authors report gains over existing models on six real-world tasks.

arxiv.orgpaperAdded 21 Oct 2026, published 19 Oct 2026

Model card for Meta Superintelligence Labs' Muse Spark Open, an open-weights, natively multimodal model with 816B total and 42B active parameters and a 1M-token context window. It targets coding and agentic work (tool use, long-horizon tasks, multimodal reasoning, synthetic data generation, LLM-as-judge). It ships in BF16 and NVFP4, needs about 700GB VRAM (NVFP4) or 2TB (BF16), runs on SGLang, vLLM, Transformers, TensorRT-LLM, Unsloth, llama.cpp and MLX, and supports a reasoning_effort setting. The Muse Model License requires "Built with Muse Spark Open" attribution and a separate agreement for Model-as-a-Service above $20M revenue. The card reports safety results against Kimi K3 on cyber, chem/bio and loss-of-control evals.

huggingface.corepoAdded 17 Oct 2026

FedRAPP is a personalized aggregation framework for Source-Free Federated Test-Time Prompt Tuning (SF-FTTPT), where clients adapt CLIP soft prompts on unlabeled, non-IID test streams without source data. It combines reliability-aware client assessment, a server-side exponential moving-average prompt memory and similarity-guided peer aggregation. Across nine CLIP benchmark datasets and ImageNet variants it beats standard federated aggregation baselines while keeping prompt-only efficiency.

arxiv.orgpaperAdded 20 Oct 2026, published 16 Oct 2026

FlexPart is a framework for text-controlled 3D mesh segmentation that assigns discrete part IDs to mesh faces through a view-independent part-identity field. It uses a control prompt of part names with location and shape clauses, trained on a VLM-built multi-granularity dataset. On FlexPartBench it raises average mIoU by 22% over the strongest baseline, and by 15% with planner-completed prompts.

arxiv.orgpaperAdded 20 Oct 2026, published 16 Oct 2026

LILIE (Language-Informed and Lossless Image Encoder) is an invertible image transform built from a normalizing flow transformer, trained jointly with a vision-language model via next token prediction so captions can be recovered from noised features. The encoder supplies a language-informed loss for text-to-image generation. JiT models trained with it cut gFD from 66.84 to 53.01 against a pixel-only baseline on the GPIC dataset. Code, encoders and models are released.

arxiv.orgpaperAdded 20 Oct 2026, published 16 Oct 2026

This paper presents a multimodal retrieval-augmented generation framework for Medical Visual Question Answering. A Gated Multimodal Unit learns a joint query representation from the medical image and the clinical question, so evidence retrieval uses both modalities while the Med-LVLM stays frozen. On benchmark datasets it consistently beats a reference Med-LVLM and stays competitive with state-of-the-art methods. The page gives no dataset names or numbers.

arxiv.orgpaperAdded 20 Oct 2026, published 16 Oct 2026

A tweet from AmingIn_AI describes a macOS screenshot-annotation tool. On release it saves the screenshot, a crop and the user's marks locally, then copies a 127.0.0.1 link. Claude Code or Codex on the same Mac can fetch that link and receive the marked image plus app/window info, and nothing is uploaded. It shows a local, privacy-preserving way to hand visual context to coding agents without MCP or cloud upload.

x.comthreadAdded 17 Oct 2026, published 16 Oct 2026

MUB (Music Understanding Benchmark) is a set of 500 identity-masked recordings with existing annotations for tempo, key, time signature and structural intervals. It covers all 24 keys, tempos of 47 to 180 BPM and 4,676 intervals, and strips titles, artists and platform IDs from inputs. Baselines reach 53.6% tempo accuracy (DSP) and 98.4% time-signature accuracy (constant 4/4). No multimodal LLM has been tested yet, and rights clearance is unresolved.

arxiv.orgpaperAdded 20 Oct 2026, published 16 Oct 2026

MOSS-Image is an open-source text-to-image model family from OpenMOSS with a 5B backbone, released October 2026 as MOSS-Image-Base and MOSS-Image-SFT. It supports JSON layout control with bounding boxes, text rendering inside images, and prompts up to 4K tokens. The repo gives a Diffusers pipeline, a Gradio app, a ComfyUI plugin, LoRA training guidance, a prompt-enhancer skill and day-0 SGLang and vLLM-Omni support.

github.comrepoAdded 20 Oct 2026, published 16 Oct 2026

IBM announces Granite-Switch-4.1-8B and -30B. Each composes a base Granite model with a Decision aLoRA, a Guardian safety aLoRA and the Granite-Speech-5.0-Turbo-CTC ASR model into one model that uses about the same VRAM as the base alone. Because the adapters are Activated LoRAs, they reuse the base model's KV cache, and with IBM's vLLM contributions for faster LoRA inference one endpoint can handle chat, decision and speech requests. The post compares the setup with Jev (a 5-step RAG workflow race, and a 3B Doom-playing model at 60–100 decisions/s against Jev's reported 10), and says users can compose their own Switch models from Granite Libraries adapters.

huggingface.coarticlePublished and added 15 Oct 2026

Social post reporting Meta's release of Muse Spark Open, described as the most capable US open-weight model: 816B total and 42B active parameters, a 1M-token context, and text, image, video and audio input in one model. It is pitched at coding and agentic work and tuned to run on two M5 Ultra Mac Studios, with weights available at release. The poster notes that in Meta's own chart, China's Kimi K3 and GLM 5.3 still beat it on coding.

x.comthreadAdded 16 Oct 2026, published 15 Oct 2026

A tweet relaying OpenAI's announcement of GPT-Live-1.1, an update to the voice model behind ChatGPT voice mode. It reduces how often the model says "checking", adds new voices and improves accents, including new voices in French, German, Hindi, Mexican Spanish, Korean and Japanese. It is rolling out to all users in ChatGPT, Codex and "dots", with API access coming soon. The poster frames it as a response to a new xAI/SpaceXSI Grok Voice release.

x.comthreadAdded 16 Oct 2026, published 15 Oct 2026

P^3-WAM is a unified world action model for egocentric dexterous manipulation with three modes: Perceive (reconstruct articulated hand motion and camera trajectories from video), Predict (generate visual futures under prescribed actions) and Plan (jointly generate action trajectories and their anticipated visual outcomes from an initial image, a language instruction and an optional visual goal). It adds a compact structured 3D action representation to a pretrained video diffusion transformer through a mixture-of-transformers design with shared base weights, modality-specific adaptation and bidirectional visual-action attention. Randomized modality visibility lets video and 3D actions act as either context or target. Unified training beats single-mode training and outperforms specialized baselines on 3D hand reconstruction and action-conditioned video prediction across four datasets, and gives better instruction adherence than video-generation baselines.

arxiv.orgpaperAdded 17 Oct 2026, published 15 Oct 2026

The paper introduces 3DVerifierBench, a 10-task benchmark for generative verification of 3D content. It scores accept/reject decisions, rejection explanations, and responses to controlled changes in specifications and candidates. A human study shows a 21.6-point gap to the strongest tested model, GPT-6 Astra. The authors also build a pipeline of multi-view fact extraction, controlled 3D editing and native annotations. It yields 14,126 training examples with contradiction explanations and linked counterfactuals. Verification-guided selection raises complete factual success from 26.7% to 90.0% with oracle selection over sixteen candidates, and human-guided recursive refinement satisfies all seven requirements for every input in an eight-input study.

arxiv.orgpaperAdded 17 Oct 2026, published 15 Oct 2026

DK-VLM adds Dual-Kernel Attention to a vision-language model. A semantic kernel keeps the pretrained attention and positional encoding. A separate geometric kernel uses estimated 3D points and camera poses to modulate attention through point displacements and relative view transformations, and a learned gate fuses the two outputs. Geometry therefore enters attention as position, not as fused 3D reconstruction features. The paper also adds Reference-Frame Composition (RC) supervision, which gives stepwise turn and heading targets derived from verified single-step annotations, to train multi-step spatial reasoning as the observer moves. DK-VLM-8B reports the highest average score across six spatial benchmarks among the compared methods, with 74.33 on VSI-Bench and 58.28 on ReVSI-32, and it preserves general video understanding.

arxiv.orgpaperAdded 17 Oct 2026, published 15 Oct 2026

Velox is a training-free attention router for chunked video-diffusion world models. It uses camera frustum slabs with depth intervals from streaming geometry to pick which historical token blocks to attend to, instead of letting the KV cache grow with rollout length. Frustum overlap softly modulates each layer's model-derived block importance, and uncovered regions keep the model's own routing. The fixed block granularity suits FlashAttention-style tiled execution, and slab construction runs asynchronously with denoising on the same H200. At chunk 48 it speeds up attention 4.36× and denoising 2.43×, and on the 158-case WBench navigation split it stays within 0.15 average points of dense LingBot-Fast-Cam.

arxiv.orgpaperAdded 17 Oct 2026, published 15 Oct 2026

Proposes SCOPED (Skill-Decoupled On-Policy Expert Distillation), which trains capability-specific RL experts for vision-language models. It extracts explicit skills from them through reward-guided reflection. During on-policy distillation the skills guide expert supervision but are hidden from the unified student, so the student internalizes the procedural knowledge and needs no explicit skills at inference. Across eight multimodal benchmarks it beats the backbone and mixed-capability RL. Analyses show cross-capability gradient interference in joint RL, and that skill augmentation gives more consistent expert trajectories, more attention to task-relevant visual evidence, and better preservation of expertise.

arxiv.orgpaperAdded 17 Oct 2026, published 15 Oct 2026

Presents a modular, fully onboard UAV navigation pipeline that closes the "human-aware safety gap" of geometry-only planners. A shared BLIP encoder drives captioning and text-conditioned segmentation, which is fused with depth for 3D obstacle localization. Qwen-2.5-0.5B, stabilized with RAG and grammar-constrained JSON decoding, infers per-category risk once per class and caches the verdict. Perception runs at 10 Hz on a Jetson Orin NX, and fallback LLM queries take about 0.8 s off the control path. In Gazebo it gives roughly 3x fewer collisions than a geometric planner and raises semantics-aware landing success from 34% to 96%. The 224-pixel encoder input limits small-object perception at range.

arxiv.orgpaperAdded 17 Oct 2026, published 15 Oct 2026

Introduces P^3IBench, a benchmark for proactive video understanding in real-life streaming scenarios, covering three capabilities: daily perception, risk prediction and intervention. It evaluates 12 open-source and proprietary video-language models (streaming and adapted non-streaming) and finds they struggle to make well-calibrated decisions about when to respond. The authors also build extra proactive training data and train P^3I-Assistant-4B, which improves several tasks unevenly, showing that proactive assistance needs response timing and restraint as well as visual understanding.

arxiv.orgpaperAdded 17 Oct 2026, published 15 Oct 2026

Release notes for Apertus 1.5, a fully open (weights, data, training details) continued pretraining of the 8B and 70B Apertus models on 4T and 2T additional tokens. It adds early-fusion image and audio understanding via discrete tokenizers (Emu3.5, WavTokenizer) without architectural change, a thinking mode, 262k context, and better tool use. It documents a four-stage pretraining and five-stage post-training pipeline (SFT, long-context SFT, RLVR, DPO, charter-alignment SDPO), plus evaluation via the MLLM-eval-suite and Every Eval Ever (EEE) Eval Cards, released checkpoints and datasets.

huggingface.coarticlePublished and added 15 Oct 2026

Hugging Face's launch post for Meta's Muse Spark Open, an open-weight 816B-parameter mixture-of-experts model (about 42B active per token). It has 256 experts with 8 routed per token, a 1M-token context, and perception and audio encoders for image, video and audio input. The post gives the architecture details, BF16 and NVFP4 checkpoints, and run recipes for transformers, llama.cpp, SGLang, vLLM, MLX and Inference Endpoints. It also covers LoRA SFT of the full model with TRL on 64 H100s and agentic demos (Pi 3D asset generation, an OpenEnv browser agent, a voice-driven shopping agent).

huggingface.coarticlePublished and added 15 Oct 2026

Hugging Face blog post introducing Black Forest Labs' open FLUX.3 Image Klein (distilled and base checkpoints) in Diffusers. It covers text-to-image, multi-reference image-to-image and bounding-box layout control, FP8/NVFP4 single-file quantized weights, DreamBooth LoRA training scripts for generation and editing, memory/speed figures (about 2 s per 1024×1024 image on an H100), and a detailed architecture tour: Qwen3-VL text encoder, two-stage per-stream then joint transformer blocks, per-token timestep modulation, 4D RoPE, and flow-matching denoising. It also shows how to customise the Modular Diffusers pipeline.

huggingface.coarticlePublished and added 15 Oct 2026

System card for GPT-Live-1.1 and GPT-Live-1.1 Quip, OpenAI's full-duplex voice models that listen and respond continuously. It reports voice-native disallowed-content evals on production and synthetic audio prompts, and finds parity overall but statistically significant regressions for GPT-Live-1.1 on sexual, mental health, emotional reliance and gore categories. It also covers alignment evals converted to audio (respecting warnings, blocker deception, broken search tool) and Preparedness Framework conclusions: neither model is High in Biological and Chemical, Cybersecurity or AI Self-Improvement.

deploymentsafety.openai.comarticlePublished and added 15 Oct 2026

Podcast interview with Jason Ma, co-founder and Chief Scientist of Dyna Robotics, on Dyna-2, a world-action model pre-trained on over one million hours of human egocentric video. The model predicts future video and uses it to guide robot actions, built on a diffusion transformer with flow matching. The discussion covers the robotics data bottleneck, adapting to new tasks from about 45 minutes of demonstrations, synthetic environments and self-generated training data, on-device inference and distillation, and Dyna's custom wheeled semi-humanoid hardware. The episode listing also covers commercial deployments (Din Tai Fung, Best Western), agentic harnesses for hour-long workflows, and world-action models versus VLAs, but the fetched text was truncated before those sections.

latent.spacetalkPublished and added 13 Oct 2026

ChunkAct brings action chunking to agentic robotics, where an LLM agent composes perception, motion and VLA tools guided by memory. Online, one call launches a chain of primitives that runs autonomously and returns control only on failure. Offline, memory is compiled into these chunks so execution needs no memory at test time and the planner is used only for real decisions. On LIBERO-Pro it improves task success by 5.0 points over the prior agentic-robotics state of the art, with 2.9x fewer LLM calls, 4.4x fewer input tokens and 2.3x fewer output tokens.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

MiniDog is a framework for training a text-to-image flow-matching diffusion transformer from scratch on four RTX 3090 GPUs, with 55 minutes of pretraining and 15 minutes of fine-tuning. It pretrains on 26k ImageNet dog images that were captioned, then fine-tunes on 2k FLUX.1-schnell generated images. Image and text features are precomputed and cached. The model is a 22M-parameter transformer with the E2E-INVAE semantic latent space, a small Qwen text encoder and in-context conditioning, and training is accelerated with iREPA and an EUPE perception encoder. The repo evaluates with FID, HPSv2, PickScore and VAE reconstruction metrics. It lowers the barrier to hands-on diffusion training research.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

Proposes mmVB, a lightweight plug-and-play module (1.6M trainable parameters) that assigns query-conditioned per-image visual token budgets using cross-image context for multimodal LLMs, without retraining the base model or accessing attention matrices (so it is compatible with FlashAttention). It cuts visual input tokens 9x, giving 5x faster time-to-first-token and 7x fewer prefill FLOPs while keeping 94.4% of full-resolution accuracy across eight multi-image reasoning benchmarks. It matters because existing token pruning, merging and compression methods target single-image redundancy and either need attention maps or retraining.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

PixelMoE is a pixel-space diffusion framework that scales capacity sparsely using standard LLM-style Mixture-of-Experts: token-choice routing, fine-grained routed experts, a shared expert and bias-based load balancing, with no diffusion-specific routing or auxiliary routing losses. It adds Pixel-Space Hierarchical Flow Matching, which varies spatial resolution along the denoising trajectory from global structure to fine detail. At matched active parameters it beats dense counterparts, and reaches FID 1.47 at 256x256 and 1.53 at 512x512 on class-conditional ImageNet-1K without representation alignment or perceptual losses. Code and models are promised but not yet released.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

RepProbe is a benchmark of 42K real and synthetic video clips with exactly 1–20 complete action repetitions, controlling for clip position, temporal coverage and duration. Video language models' counting accuracy collapses at high counts, and this is not explained by frame sampling alone. Linear probes on final-layer representations show count information stays linearly decodable even when the model's answer is wrong, so the failure arises at the output stage, where answers cluster on a few preferred numbers.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

Plan-Bench is a benchmark of 699 curated samples across 16 subclasses and three dimensions. It tests whether vision-language models can reason over spatial and temporal information and produce multi-step plans under constraints for embodied tasks, with action execution isolated out. Across 28 VLMs the best model reaches only 75.0% accuracy. Scaling generally helps, and larger models look more promising as high-level planners. Code and benchmark are released.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

Game2Real-Bench tests whether vision-language model performance on procedurally generated toy games predicts real-world visual reasoning, and whether game examples help transfer. It has 1,000 verifiable game samples and 400 real-world samples, matched by shared reasoning requirements (fine-grained perception, spatial reasoning, constraint reasoning, sequential planning). Across 25 VLMs, game and real scores correlate strongly (Pearson/Spearman 0.935). One-shot game-to-real transfer on four models improves constraint reasoning for several but hurts fine-grained perception. The authors also release Game2Real-Dataset, with about 24K verified game samples across 18 tasks and 65 QA families.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

OmniCompass is a manually annotated benchmark for video grounding and understanding with two tracks: OmniCompass-VL (820 videos, 2,460 instances; vision-language models) and OmniCompass-AV (240 videos, 720 instances; joint audio-visual evidence for omni-modal models). Each track covers temporal grounding, spatial grounding and QA across short/medium/long videos, and each instance is run in paired text-only and text+image target specifications. Evaluating 50 MLLMs shows open-source models lag proprietary ones, especially on audio-visual understanding, that precise temporal/spatial grounding remains hard, and that predictions are inconsistent across the paired specifications.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

SP$^2$ec (Single-Peak SPeculative decoding) is a training-free bandit algorithm that adapts the draft configuration of self-speculative decoding per prompt for vision-language models. It relies on the single-peak throughput pattern seen in SWIFT, KnapSpec, SpecVLM and the authors' cosSim method. It minimizes a new wall-time regret measure and has theoretical guarantees of near-optimal configuration with negligible exploration cost. On Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-Mistral-7B it improves over the original self-speculative methods by up to 11% on visual and textual tasks.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

Google's developer post presents Gemini 4 Argon as a physical-reasoning model for the whole robotics lifecycle. It reports 87% on the Embodied Reasoning (ERQA) benchmark, ahead of other frontier models. It then gives representative prompts: controlling a Boston Dynamics Spot with a MuJoCo digital twin, closed-loop code-as-policies arm control, steering the Gemini Omni world model, and building digital twins from teleoperation episodes. Further prompts cover retargeting human video to a 76-DoF humanoid, hindsight annotation of ALOHA 2 videos, a general value function on UMI data, sim-data generation for VLA fine-tuning (56% to 75% real success), and reproducing control papers in MuJoCo WebAssembly. It is useful for seeing how the model is prompted for agentic, simulation-driven robotics work.

developers.googleblog.comarticlePublished and added 13 Oct 2026

UniW(A)M is a mixture-of-experts model built on diffusion Transformers that unifies action-conditioned world modeling (AC-WM) with several world-action model (WAM) variants. Shared experts capture common physical dynamics, while atomic capability experts are reused across interfaces, each keeping its own conditional pathway. At deployment a plug-in adaptive interface router (PAIR), trained on a small subset of target-task chunk scenarios with the backbone frozen, picks a WAM interface for each execution chunk. It reaches 88.3% average real-world success against 76.7% for the strongest baseline, leads WorldArena on all three action-conditioned world-modeling tasks, and PAIR raises OOD success from 63.3% to 76.7%.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

Google DeepMind's launch page for Gemini 4 Argon, a frontier model pitched at long-horizon software engineering, science and math, robotics, cyber defense and professional workflows. Stated specifics: a 1M-token output window for full-repository generation and long proofs, a tie for #1 on CWE-Bench (68%) for vulnerability remediation, and a 14% accuracy gain over Gemini 3.1 Pro in Box's evaluation. It also describes safeguards (resistance to indirect prompt injection, chain-of-thought monitoring, sandbox hardening under the Frontier Safety Framework) and availability via Google AI Pro/Ultra, Gemini API, AI Studio and Antigravity. Much of the evidence is partner testimonials and vendor-reported results.

deepmind.googlearticlePublished and added 13 Oct 2026

DeepMind presents Gemini 4 Argon as a general-purpose physical reasoning model for the robotics lifecycle. It reports 87% on the Embodied Reasoning (ERQA) benchmark and gives representative prompts across several areas: controlling a Boston Dynamics Spot with a MuJoCo digital twin, code-as-policies arm control with a visual coordinate harness, and steering the Gemini Omni world model in a visual feedback loop. Further examples cover building MuJoCo and mujoco_warp scenes from teleoperation data, egocentric video or a hand-drawn sketch for 76-DoF humanoids, hindsight annotation and value-function progress prediction on video, sim data generation for VLA fine-tuning (success rate 56% to 75%), and reproducing control papers in MuJoCo WebAssembly. It matters as a concrete look at how a frontier model is being used as agentic coder, perceiver and sim builder for robotics.

deepmind.googlearticlePublished and added 13 Oct 2026

EmoGenie is a benchmark for emotions expressed and evoked by text-to-image outputs. It has 6,400 images from eight T2I models using 800 emotion-targeted prompts. It includes EmoGenie-QA (90,000 closed-set and 1,057 open-ended questions) and EmoGenie-Assess (13,175 mean opinion scores for valence, arousal, dominance and aesthetic quality). The authors also propose EmoProbe, which pairs emotion QA with response-aligned, dimension-specific regression heads. They report that it beats baselines on emotion QA and correlates better with human ratings. The work shows how reliably MLLMs interpret affect in generated images.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

Proposes a vision-language-model framework that incrementally reconstructs heading hierarchies across tens or hundreds of pages using a level-wise multimodal memory. The memory keeps a compact set of earlier pages chosen by anchors (here, the most recent node at each heading level), so the VLM sees the visual, textual and layout patterns of headings at every level. The authors also introduce CLDoc-HHRBench (250 documents, 11,212 pages, expert-verified annotations) and report results on it and three existing benchmarks. It matters for RAG pipelines that need coherent document-level structure, which page-by-page OCR/VLM parsing doesn't give.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

PDF TO MP3 is a hosted MCP server (Streamable HTTP, account authentication) that turns PDFs into audiobooks. The user's existing AI subscription, such as Codex, handles document processing through the MCP connection. Speech generation is billed separately through the user's own OpenAI or Google text-to-speech API key. The setup is three steps: add the MCP server, add a speech key, then attach a PDF and ask the assistant to use the MCP. It shows a "bring your own subscription" pattern that keeps the vendor's compute costs low.

pdftomp3.comtoolPublished and added 13 Oct 2026

The paper studies regressive sycophancy under fabricated evidence: a medical VLM gives a correct answer, then abandons it when the user pushes back with a fake prescription. It tests four open VLMs on 400 VQA-RAD yes/no questions in 19,200 multi-turn conversations. Without a synthetic-document disclosure, models abandon 80.1% of 6,488 correct answers, and the same prescription raises Qwen2.5-VL-7B's flip rate from 45.9% to 89.6% while lowering Qwen3-VL-8B's from 77.7% to 50.9%. Models often reject the prescription yet keep the correct answer in only some conversations (915 of 3,232), and 40% of those flip later. System-prompt disclosure of synthetic documents cuts one model's flip rate from 77.8% to 47.3%. The authors conclude that recognising a forgery does not stop a model from adopting the forged answer.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

Model card for EmbeddingGemma 2, an open (Apache 2.0) 740M-parameter multimodal embedding model from Google DeepMind built on Gemma 4. It maps text, code, images, video and audio into one 768-d space, with a 270M text backbone and optionally loadable 170M vision and 300M audio encoders, an 8K context, and Matryoshka truncation to 128/256/512/768 dims. The card gives practical guidance: task instruction prefixes, re-normalising after truncation, avoiding float16 (NaN or degraded embeddings), interleaved multimodal inputs with placeholder tokens, and vision token budgets. It targets on-device search, RAG, classification and clustering.

huggingface.corepoAdded 12 Oct 2026

Atticus is an MIT-licensed macOS 26+ hold-to-talk voice assistant that uses Apple's on-device speech transcription and runs AI requests through Claude Code on an existing Claude Pro/Max plan, so no API key is needed. Simple commands skip the model entirely (apps open in 0.08 s), actions it has seen once are learned as zero-cost shortcuts, and a "remember…" memory persists across asks. It also sets up no-AI background watchers (polling every 5 s) and holds itself to a budget of a median under $0.002 and 6 s per AI ask, with a system prompt under 10,000 characters.

github.comrepoPublished and added 12 Oct 2026

God's Eye View is an open-source, vanilla JavaScript + CesiumJS + Vite 3D globe that fuses live public feeds (aircraft, ships, satellites, quakes, public cameras, weather) into one explorable scene. Its AI layer is a hands-free voice agent on the OpenAI Realtime API with 30 tools: camera fly-to, whiteboard annotations, analyst queries over live layers, and scene-context and screenshot grounding. Keys stay server-side and the browser gets only a short-lived session token. An experimental ChatGPT/Codex OAuth path is also supported. The app doubles as an MCP server, so Claude Desktop, Codex or ChatGPT can open the globe from a conversation.

github.comrepoAdded 11 Oct 2026

Latent Space interviews Standard Bots' CEO Evan Beard and Head of AI Leif Jentoft about the AI stack behind its industrial robot arms. A shared base model (low billions of parameters) is adapted per customer through demonstrations and fine-tuning. A zero-shot perception model, whose backbone is trained on over a billion images, handles machine tending, while conventional code handles motion and cell logic. Inference runs on-premises on edge GPUs that stream action chunks to low-level control. Failures and human corrections feed fleet learning except in air-gapped defense deployments, and the StandardOS APIs/SDKs let developers bring their own models such as NVIDIA Cosmos. The lesson is to keep the model's role narrow and use targeted, high-quality data to fix edge cases instead of chasing dataset size.

latent.spacearticlePublished and added 10 Oct 2026

Introduces Confidential Error-correcting Codes (CECs), a primitive that decodes reliably from perturbed codewords while keeping the message confidential, defined by confidentiality, robustness and soundness. The Randomness-Security Separation framework encodes a nonce and masks the payload with a PRF; the Conc-CEC instantiation uses concatenated Reed-Muller and Reed-Solomon codes and claims stronger concrete security than PRCs (CRYPTO'24). The application, CECMark, watermarks AI-generated images and reportedly beats Gaussian Shading, PRCMark and T2SMark across 56 perturbations with a false positive rate below 1e-5 and no side effects.

eprint.iacr.orgpaperAdded 14 Oct 2026, published 10 Oct 2026

Model card for Qwen3.8-27B, a 27B dense vision-language model from the Qwen team with a hybrid Gated DeltaNet / Gated Attention layout (64 layers, 16×(3 DeltaNet + 1 attention)), MTP training, and 262k native context extensible to 1M via YaRN. It targets coding, research and long-horizon agentic tasks, with thinking on by default, reasoning_effort levels (xhigh/medium/low) and preserve_thinking. The card gives deployment recipes for Transformers, vLLM, SGLang and TokenSpeed, recommended sampling parameters, video-token settings for hour-scale video, and benchmark notes (SWE-bench Pro, NL2Repo-Bench, DeepSWE, HLE, MathVision, WebArena-Verified, etc.).

huggingface.corepoAdded 9 Oct 2026

The paper studies joint attention in multimodal diffusion transformers (MMDiTs), where the evolving image updates text representations. It finds that this image-to-text (I2T) feedback tends to preserve objects and attributes already in the image, errors included, and that suppressing it improves generation. It proposes Semantic Feedback Guidance (SFG), which guides away from an auxiliary branch that strengthens I2T feedback and weakens text-to-image conditioning, and iSFG, an in-place variant that needs no extra model evaluations. SFG runs for the first 20% of sampling steps (about 1.2x base NFE without CFG). Across three backbones, adding SFG to CFG on FLUX raises GenEval from 0.6661 to 0.7479, and iSFG on SD3.5M adds +1.33 HPSv2.1 over CFG at zero extra NFE. The control also improves VBench scores for text-to-video and image-to-video.

arxiv.orgpaperAdded 12 Oct 2026, published 9 Oct 2026

Hide and Seek (HnS) is a concept-level attribution method for black-box vision models. It needs no model weights and no concept-labeled datasets. Given a concept and an image set, it uses a vision-language model to retrieve concept-aligned images and localize the concept, masks the localized region, and takes attribution from the change in the black-box model's output. It agreed better with ground-truth attributions than existing methods on CelebA and COCO-Stuff, and it works on tasks existing concept attribution methods cannot handle, such as image captioning and satellite image segmentation.

arxiv.orgpaperAdded 12 Oct 2026, published 9 Oct 2026

Introduces EICS (Evaluation-Invariant Concept Stability), a metric for checking whether representations consistently encode a concept under controlled one-concept interventions. Also introduces CASP-VL, a benchmark of causally controlled spatial perturbations derived from Visual Genome scene graphs. Across several VLM architectures, spatial relations are represented much less faithfully than lexical information, even for models well above chance on compositional benchmarks. The relation is identifiable at the predicate's own tokens but is lost when the sentence is aggregated into a global representation (EOS token or mean pooling), a failure that output-based evaluation does not reveal.

arxiv.orgpaperAdded 12 Oct 2026, published 9 Oct 2026

Robo dir is a directory that aggregates robotics and physical-AI datasets in one place. The launch tweet claims 3,494 datasets, 86,424 Hugging Face uploads, 236B+ frames and 2.2M+ hours of data. It also exposes the index through a single MCP server with no sign-in, so agents can search for robot training data programmatically.

x.comtoolPublished and added 9 Oct 2026

Proposes the Answer-Evidence Support Protocol (AESP), which checks whether the grounding evidence a model outputs actually supports its own answer in grounded remote sensing visual question answering. It applies task-specific rules to existence, counting and comparison questions. The authors build the G-RSVQA benchmark of 15,027 samples with questions, reference answers and object-level annotations. Tests on seven grounded vision-language models show evidence-support failures on all three question types, with up to 50.82% of correct answers unsupported by their grounding evidence.

arxiv.orgpaperAdded 12 Oct 2026, published 9 Oct 2026

Social post announcing Qwen's open-weights release of Qwen-Image-2.1-Turbo, an accelerated checkpoint of Qwen-Image-2.1 on the same 7B visual generation architecture. It generates and edits images in 8 denoising steps, against 40 in the base model's sample code, with 2K output. Qwen claims the fewer steps do not reduce quality. The license is non-commercial, which limits production use.

x.comthreadPublished and added 9 Oct 2026

RARA (Retrieval-Anchored Representation Alignment) is a two-stage method for adapting pre-trained robot policies such as Vision-Language-Action models to new tasks from few demonstrations. It first retrieves pretraining representations with similar task intent and motion and aligns the policy's representations of the target demonstrations to them. It then fine-tunes on the target demonstrations while constraining representation change on the retrieved demonstrations, which limits representation drift. Tested with Diffusion Policy and GR00T on 13 tasks in RoboTwin, RoboCasa and on a real robot, it improves average success over plain fine-tuning by 9.5 points in-distribution and 13.6 out-of-distribution, and beats all compared baselines.

arxiv.orgpaperAdded 12 Oct 2026, published 9 Oct 2026

ManeuverFlow is an end-to-end autonomous driving planner that generates trajectories for distinct driving intentions in one step. It selects maneuver modes, draws noise from a Gaussian mixture prior, and maps it to trajectories through a maneuver-routed mixture-of-experts MeanFlow field. Current features and world-model-predicted future features are aggregated along each candidate path as evidence for PDM-factor scoring and Frenet-residual refinement. With a TransFuser-style ResNet-34 backbone it reaches 90.6 PDMS on NAVSIM v1 and 90.1 EPDMS on NAVSIM v2 navtest. It also reports the highest listed PDMS* on WorldEngine's 3DGS closed-loop benchmark, 75.19 with logged traffic and 74.58 with IDM-controlled traffic.

arxiv.orgpaperAdded 12 Oct 2026, published 9 Oct 2026

DreamTrue is a multi-view, cross-embodiment robot world model for action-faithful, physically plausible video prediction. It renders action trajectories into image-space conditions aligned to target videos by offline geometric calibration. Counterfactual post-training modifies recorded trajectories to cover unsuccessful interactions and contact configurations. A human-annotated defect dataset trains an embodied video reward model whose scores drive RL post-training. On AgiBot it reaches state-of-the-art action following and cuts the human-assessed interaction defect rate from 48.12% to 6.25%. It ranked first in the world model track of the AgiBot World Challenge 2026.

arxiv.orgpaperAdded 9 Oct 2026, published 8 Oct 2026

OuroWorld is a mask-free framework that turns any static 3D Gaussian Splatting scene into a seamlessly looping 3D cinemagraph. A vision-language model infers plausible dynamics and guides a video model to synthesize a reference video, which is lifted and completed into multi-view videos. Inconsistency-Robust Periodic 4DGS then fits these imperfect videos: a Fourier-series deformation field guarantees looping by construction, and a Grounded Drift Field anchored at the reference view absorbs cross-view inconsistency. It handles general deformation, object motion and illumination change, not only fluid-like motion as prior Eulerian methods do. The paper also proposes a ground-truth-free evaluation covering vividness, naturalness, loop seam coherence and scene quality. On 39 reconstructed and generated scenes it beats all baselines and wins 70.8%-99.0% of user-study comparisons.

arxiv.orgpaperAdded 9 Oct 2026, published 8 Oct 2026

OmniCapBench is a benchmark of 786 densely annotated videos that evaluates audio-visual captioning in multimodal LLMs. Instead of scoring free-form captions, it asks models for sets of atomic, verifiable units across three tracks (entity references, visual shots, audio events). These are scored with deterministic constraint checks plus localized LLM-based semantic comparisons, which gives coverage, localization and stable scoring together. Tests on frontier MLLMs show strong local perception but weak long-horizon audio-visual reasoning, especially identity drift and cross-modal misalignment.

arxiv.orgpaperAdded 9 Oct 2026, published 8 Oct 2026

WOVEN is a training source and benchmark for visual transition reasoning in multimodal LLMs. It has 36,076 examples spanning 20 scene types, 5 action types and 8 reasoning types, built from rollouts of video-pretrained generative models. Evaluating 38 frontier MLLMs (including GPT-5.4 and Qwen3-VL-235B-A22B) shows systematic deficits that persist with scale. Training on WOVEN subsets of about 2,000 items improves 22 of 26 external benchmarks by up to 27.3 points, and WOVEN data can replace 30-50% of a task's own training data. The authors derive a recipe: select supervision by the reasoning operation it teaches rather than by scene or action, and prefer larger visual-state changes.

arxiv.orgpaperAdded 9 Oct 2026, published 8 Oct 2026

Connected Self Forcing extends Self Forcing, which trains autoregressive video generators on their own rollouts with KV caching but detaches historical caches and so cuts backward gradients between chunks. It reconnects gradient paths across chunks, passing gradients through generated latents into the computations that produced them, so later predictions can shape how earlier context is generated. Shortcut gradient replay recovers cross-chunk gradients without keeping the full rollout computation graph. Combined with distribution matching distillation, it improves long-horizon visual quality and temporal consistency with no change to inference.

arxiv.orgpaperAdded 9 Oct 2026, published 8 Oct 2026

PathTime-VLA reparameterizes Vision-Language-Action actions as a progress-indexed interaction path X(s) plus a positive interval-time profile that defines a monotone time law t(s). Geometry is thus decoupled from execution speed, so teleoperation timing artifacts don't contaminate the path target. A staged post-training recipe uses demonstrations and DAgger interventions for a target-domain prior, Speed-DQN to learn execution-speed multipliers from robot interaction, and Path-AWR to refine the diffusion path generator from rollout outcomes. On three tasks it reaches 58/60 successes versus 57/60 for BC+DAgger at fixed 1×, with about 39–52% shorter mean completion times on successful trials.

arxiv.orgpaperAdded 9 Oct 2026, published 8 Oct 2026

Liquid AI released Open d1, two open-weight "decision models" — d1-3B (text + vision) and d1-omni-600M (text + image or text + audio) — that don't generate text tokens at all. Instead they take a state plus a typed question as input and return calibrated probabilities in a single forward pass, positioning them as lightweight classifiers/judges rather than generative LLMs.

x.comthreadPublished and added 8 Oct 2026

Google launched Playground, a browser-based tool letting U.S. adults 18+ describe a video game in plain language and get a playable, shareable result back without coding, with creation access tiered by Google AI subscription. The piece ties Playground to Google's broader AI-for-games push: DeepMind's Genie (an 11B-parameter model trained on 200,000+ hours of gameplay video) and its successor Genie 3, a real-time interactive world model opened to AI Ultra subscribers via Project Genie, plus an upcoming integration with Unity Spark for 3D, professional-level game mechanics. Google did not disclose which models power Playground or its pricing, and framed it as an early experiment, while the announcement reportedly sank Unity and Roblox stock on fears of AI displacing traditional game-dev tools.

decrypt.coarticlePublished and added 7 Oct 2026

This paper shows that vision-language-action (VLA) models are highly sensitive to how instructions are phrased: a single-word change (e.g. "stove" vs "hot plate") can shift task success by dozens of points on π0 and π0.5 checkpoints in the LIBERO benchmark, even after rephrase-augmentation finetuning. The authors characterize this with statistically tested single-edit swings and an oracle phrase search, then mitigate it without retraining by having an LLM distill scoring evidence into 10-20 explicit rephrasing rules that rewrite incoming instructions at deployment time, improving frozen π0 by 16-27% relative on held-out tasks and lifting π0.5's in-finetune success from 93.6% to 97.8%.

arxiv.orgpaperAdded 8 Oct 2026, published 7 Oct 2026

RoboJEPA is an 8B-parameter latent world model built on the Joint Embedding Predictive Architecture (JEPA), trained on real-robot data spanning 12 robotic embodiments. The paper establishes the first scaling laws for multi-embodiment robotic world models, showing that imagination error (latent rollout error) follows a second-order power law in compute and predicts downstream planning performance, and demonstrates zero-shot deployment of the model as a goal-image-conditioned planner on real hardware for long-horizon tasks.

arxiv.orgpaperAdded 8 Oct 2026, published 7 Oct 2026

Liquid AI announces Open d1, two open-weight multimodal decision models. d1-3B (built from LFM2.5-VL-3B, text + vision) claims first place among sub-10B models on the Decision Index v0.2.1, while the experimental d1-omni-600M (LFM2.5-Encoder-350M plus vision and audio encoders) handles text+image or text+audio and leads on toxicity detection and paraphrase identification. Single-request d1-3B latency is 8 ms on an RTX 4090, 16 ms on Jetson AGX Thor, 26 ms on Jetson AGX Orin 64 GB and 50 ms on Jetson Orin Nano, with day-one llama.cpp support; demos include live-camera apps and Isaac Sim navigation. Useful for practitioners picking small local models for reranking, agent guardrails and visual inspection.

x.comthreadAdded 9 Oct 2026, published 7 Oct 2026

This paper introduces Self-Correction Optimization (SCO), a training-free method that improves consistency in interleaved image-text generation from multimodal large language models. SCO treats the classifier-free guidance update as a reference signal and applies minimal corrections under two constraints—a new-event constraint for temporal consistency and a state-preserving constraint for visual-subject coherence—avoiding the computational cost of additional training with augmented data. The authors show improvements on interleaved multimodal generation benchmarks and extend the method to video generation tasks like robot manipulation and long-horizon handcrafting.

arxiv.orgpaperAdded 8 Oct 2026, published 7 Oct 2026

GeoCoTDrive is a vision-language-action (VLA) framework for autonomous driving that introduces an explicit geometric chain-of-thought: it first grounds 2D regions corresponding to decision-critical cues, then retrieves localized 3D geometric priors from a geometric foundation model within those regions, interleaving these features into the autoregressive context to drive trajectory generation. The paper also introduces "planning-relevant grounding," a new region-level grounding supervision task, and the PlanningGrounding dataset to teach VLAs planning-oriented grounding, showing consistent gains in safety-critical planning across multiple end-to-end driving benchmarks.

arxiv.orgpaperAdded 8 Oct 2026, published 7 Oct 2026

TaoD2C-Bench is a benchmark for evaluating multimodal large language models (MLLMs) on industrial design-to-code (D2C) tasks — converting UI designs into code that satisfies implementation constraints, not just visual fidelity. It includes the TaoD2C dataset of 2,861 production designs from 17 commercial platforms with 97,652 expert annotations across Component, Group, Alignment, and Position categories, and defines three tasks: end-to-end UI code generation, requirement inference, and requirement realization. Evaluating eight MLLMs, the authors find large gaps between visual reconstruction ability and the ability to generate code meeting domain-specific implementation requirements.

arxiv.orgpaperAdded 8 Oct 2026, published 7 Oct 2026

CARES is a synthetic benchmark of 10,000 two-speaker audio scenes designed to test whether audio-language models correctly identify which sounds are 'salient' — defined as sounds a speaker audibly reacts to. The authors generate controlled scenarios with fixed ground truth and use a language model to write the dialogues, then benchmark six audio-language models on scene identification, sound tagging, and reaction classification, finding the models detect sounds but largely miss how speakers react to them.

arxiv.orgpaperAdded 8 Oct 2026, published 7 Oct 2026

TII (Technology Innovation Institute) announces Falcon-ASR, a 1.6B-parameter automatic speech recognition model focused on the Emirati Arabic dialect, which also supports MSA, other Gulf dialects, English, French, Spanish and Portuguese under one set of weights. It reports 20.92% average WER across six Arabic benchmarks on the Open Universal Arabic ASR Leaderboard (beating the best published leaderboard result of 23.17%), 22.73% WER / 10.19% CER on an internal Emirati evaluation (beating Qwen3-Omni by 4.07 points), and 5.74% mean WER on the Hugging Face Open ASR Leaderboard's English test sets. The model builds on the team's Falcon3-Audio work, supports word-level timestamps, and is available via a Hugging Face demo space with API access planned.

huggingface.coarticleAdded 8 Oct 2026, published 7 Oct 2026

Simon Willison built llm-openai-decisions, a plugin for his `llm` CLI tool that wraps OpenAI's newly announced Decisions API (unveiled at DevDay), modeled on his existing llm-typesafe plugin for Jev's competing API. Both APIs support structured yes/no, multiple-choice, and score-style questions, but OpenAI's gpt-6-luna model also accepts image input (at 10 cents per million input tokens, versus Jev's 4.2 cents), and the plugin itself was built by having GPT-6 Astra read OpenAI's docs. The post includes install instructions and a worked example querying an image attachment for a yes/no answer.

simonwillison.netwriteupAdded 7 Oct 2026, published 6 Oct 2026

Decrypt reports on Google October launch of Nano Banana 2.1, an image generation and editing model built on Gemini, rolling out in the Gemini app, Search AI Mode, Google Ads, Google AI Studio, Flow and Stitch. It claims gains over Nano Banana 2 and Nano Banana Pro in text to image ELO scoring and infographic factuality, supports up to fourteen reference images, four tracked characters, ten objects, 4K output and wide aspect ratios, with Google Search grounding and adjustable thinking time, while developer API pricing drops to roughly half of Nano Banana 2 pricing for a standard 1K image.

decrypt.coarticlePublished and added 6 Oct 2026

Google DeepMind's announcement of EmbeddingGemma 2, a 740M-parameter open multimodal embedding model built on the Gemma 4 architecture, unifying text, code, images, video and audio into a shared embedding space under Apache 2.0. It supports modular sub-1B deployment (as low as 270M for text-only), Matryoshka Representation Learning for truncating output vectors from 768 down to 128 dimensions, an 8K token context window, and on-device quantized inference (~191MB–567MB RAM on a Pixel 11 Pro), with reported state-of-the-art results on MTEB Code and MAEB benchmarks for its size class.

deepmind.googlearticlePublished and added 6 Oct 2026

World Models' Last Exam in Physics introduces a measurement-based benchmark of 40 controlled tasks across mechanics, optics, fluids, thermal/phase-change phenomena, electromagnetism, and surface tension, designed to test whether video world models generate physically consistent outputs. Each task pairs an initial image and prompt with predefined physical criteria, and a custom evaluator combines observability screening with task-specific quantitative measurements rather than relying on reference videos or model-based judgments. Testing eight video generation models across 1,280 generated videos found persistent physical inconsistencies, with the best model scoring only 57.76/100, and the evaluator showed higher agreement with human judgments than a vision-language model baseline.

arxiv.orgpaperAdded 7 Oct 2026, published 6 Oct 2026

PEARS is a hybrid reinforcement-learning framework for sample-efficient online adaptation of pretrained tactile-manipulation policies to out-of-distribution conditions. A physics-guided force reasoning (PFR) module uses a vision-language model to diagnose failures from visual and tactile history and update contact-force bounds enforced by a hybrid force-position controller, while a separate tactile-conditioned diffusion-steering component perturbs the latent noise of a frozen flow-matching policy to correct free-space motion and contact timing without retraining the base model. It improves success rates by 12.4-37.4 points over strong baselines in simulation, cuts required interaction episodes by up to 53.2%, and reaches 95%/90% success on real-world whiteboard-erasing and pipette-liquid-aspiration tasks.

arxiv.orgpaperAdded 7 Oct 2026, published 6 Oct 2026

ALIVE is a framework for first-frame-guided video editing that inserts objects so they coherently participate in interactions (e.g., being picked up or manipulated) rather than appearing static. The authors curate a 35,800-pair editing dataset combining 3D-rendered, model-generated and real-world videos plus pairs from ROSE, and train a vision-language model to predict interaction guidance from the edited first frame and an instruction naming the object. They introduce the ALIVE-interaction benchmark to measure interaction fidelity, source preservation and visual coherence via a unified VLM-based protocol, reporting a 43.9% Overall improvement on this benchmark and 4.4% on a general video object insertion benchmark over the strongest baseline, with VLM-predicted guidance adding a further 0.95-point gain.

arxiv.orgpaperAdded 7 Oct 2026, published 6 Oct 2026

This arXiv paper introduces a post-training semantic lifting method for 3D Gaussian Splatting that assigns a single target class to Gaussians by accumulating target/non-target evidence across all views, weighted by per-view visibility, then filters with a dual-threshold (β and γβ) scheme that selects high-confidence seeds and extends them via connected components. The authors evaluate by transferring labels from Gaussians to mesh vertices and decompose error into three sources — the 2D detector, the lifting step, and the representation transfer — tuning thresholds on seven Replica scenes and testing on ten held-out ScanNet++ scenes, reaching mean mIoU of 0.93/0.65 (dataset masks/YOLO masks) on validation and 0.80/0.54 on test, a 0.24 mIoU improvement over prior per-view thresholding. The error analysis finds the 2D detector is the dominant remaining error source, informing where future gains in 3D semantic reconstruction should focus.

arxiv.orgpaperAdded 7 Oct 2026, published 6 Oct 2026

This paper proposes an extensible benchmark for studying spoken dialogue artifacts between humans and social robots, including clarification requests, interruptions, embodied signals like head nods and facial cues, and time constraints. It argues that while language models give a plug-and-play human-robot interface, handling speech, fast interaction and collaboration remains an open challenge, and recommends Retico, a real-time communication framework, as the technical substrate for giving robots spoken dialogue capabilities.

arxiv.orgpaperAdded 7 Oct 2026, published 6 Oct 2026

EC-RAG (Event Chain Retrieval-Augmented Generation) is a training-free framework for long-video question answering that organizes video content into an explicit, temporally ordered "event chain" instead of retrieving isolated frames or snippets. It partitions video into semantically coherent segments, fuses multi-modal signals (speech, text, visual cues) at the event level, and retrieves relevant events from the chain to answer queries, working as a plug-and-play addition to existing large video-language models (LVLMs) without extra training. Evaluated on Video-MME, MLVU, and LongVideoBench, it outperforms frame-level retrieval baselines, showing the value of modeling temporal/event structure for long-video understanding.

arxiv.orgpaperAdded 7 Oct 2026, published 6 Oct 2026

Agent Sprite Forge v0.4.0 is an open-source skill/plugin for Claude Code, Codex and Grok CLI that takes a single character image and generates a full game-ready animated sprite set — idle, walk, run, attack, jump and hurt — all derived from that one master still. It adds official API support for OpenAI, Gemini, xAI, ModelArk and fal.ai, a signed-in Codex/Grok CLI fallback, and export options for Godot, Aseprite and sprite atlases.

x.comtoolPublished and added 6 Oct 2026

A tweet reports that Mistral has announced Mistral Large 4, internally nicknamed "Le Chonk," a mixture-of-experts model with roughly 1 trillion total parameters and 49 billion active parameters. It is described as natively multimodal and state-of-the-art on critical workloads including cyber defense, with API access starting immediately and an open-weight release planned for late October.

x.comthreadPublished and added 6 Oct 2026

Mistral announces Mistral Large 4 ("ML4"/"Le Chonk"), a 1-trillion-parameter natively multimodal Mixture-of-Experts model with 52B active parameters, trained from scratch on 3,800 NVIDIA Grace Blackwell GPUs in Mistral's own European datacenters. The post details state-of-the-art open-weight results on cybersecurity benchmarks (82% on an Artificial Analysis vulnerability-reproduction test, 93% on Cybench), agentic coding (DeepSWE, Terminal-Bench, AutomationBench), finance/legal evals (via Vals, Harvey's Legal Agent benchmark), visual grounding (beating GPT-6 Astra on Dense 200), and safety benchmarks (Lakera B3, KORA). It also describes the large-scale RL post-training pipeline (33B tokens/day at 3k GPUs) used to produce the model, with full weights promised by end of month.

mistral.aiarticleAdded 7 Oct 2026, published 6 Oct 2026

An ISMIR 2026 (Abu Dhabi) research project that fine-tunes Aria, a transformer pretrained on piano MIDI, on solo recordings from twelve jazz pianists in the PiJAMA dataset, adding a gated cross-attention layer that conditions generation on a learned per-pianist embedding. Because standard held-out perplexity is nearly blind to style (the next note is predictable regardless of pianist), the authors instead measure style transfer by sliding a pianist classifier over freely generated continuations: conditioned continuations are attributed to the intended pianist 70% of the time versus 37% unconditioned, and a classifier trained only on generated music identifies real recordings with 95% accuracy. The page is an interactive demo with audio examples, blindfold listening tests, and per-pianist results (e.g. 96% attribution for Hank Jones/Dick Hyman vs 29% for Cedar Walton), alongside a link to the full paper with ablations on mismatch conditioning and memorization checks.

almostimplemented.github.iowriteupAdded 6 Oct 2026, published 5 Oct 2026

This paper introduces Recursive Video In-Context Learning (RV-ICL), a training-free method that lets an LLM agent orchestrating frozen vision-language-action (VLA) policies learn from a single demonstration video rather than from full video or fixed keyframes. The demonstration is converted into a navigable hierarchy (task keyframes, phases, moments, short clips) built around sub-events like grasps and releases, exposed to the agent as read-only tools it queries coarsely during planning and drills into for contact-level detail during execution. Built on the RPent agent, it raises task success from 92.6 percent to 96.5 percent on LIBERO-PRO and from 86.7 percent to 95.8 percent on LIBERO-Plus with only one demonstration per task.

arxiv.orgpaperAdded 6 Oct 2026, published 5 Oct 2026

MemPilot is a framework that lets LLM agents curate memory on demand rather than query-agnostically, using a multi-step policy trained via reinforcement learning to choose between retrieving from pre-built memory or delegating query-specific curation of raw multimodal history to heterogeneous LLMs and VLMs. The policy controls evidence amount, curation instructions, model choice, and visual access, and is optimized with objective-wise advantage decoupling and prefix-based marginal utility estimation to balance performance, cost, and latency. Tested across five multimodal agent-memory benchmarks, it achieves broader performance-cost-latency trade-off frontiers than existing trade-off-aware baselines.

arxiv.orgpaperAdded 6 Oct 2026, published 5 Oct 2026

MedPrune is a multi-agent framework for medical visual question answering that models the diagnostic process as a heterogeneous communication graph of specialist agents across departments, then dynamically prunes both nodes (irrelevant specialist agents, via reinforcement learning) and edges (low-value intra-/inter-departmental connections) to cut redundant communication. The authors report it beats existing multi-agent baselines on medical VQA benchmarks under both full-set and few-shot settings while improving token efficiency and adversarial robustness.

arxiv.orgpaperAdded 6 Oct 2026, published 5 Oct 2026

Lens3D is a method for improving fine-grained object understanding in 3D large language models by using 3D localization to select informative 2D views that are fed to an external vision-language model (the LensUnd pipeline), then distilling that knowledge back into native 3D LLMs via detailed caption supervision (LensDistill), avoiding the need for external VLM calls at inference time. The authors also release LensBench, a held-out evaluation set of 2,068 objects with three reference descriptions each, and show gains in fine-grained object captioning on Video-3D LLM and 3DRS while preserving grounding and scene-level QA performance.

arxiv.orgpaperAdded 6 Oct 2026, published 5 Oct 2026

BrainTRACE is a report-grounded benchmark for evaluating whether vision-language models can trace the evidence structure required for longitudinal brain MRI interpretation. It contains 7,273 scored VQA instances derived from 1,778 longitudinal patients, 7,299 MRI studies, and ~29k co-registered 3D MRI sequence volumes, organized across five levels of clinical reasoning and evidence demands like longitudinal comparison, multi-sequence integration, and volumetric spatial grounding. Evaluation of 20 VLM configurations finds that current models identify isolated visual cues but rarely compose them into grounded longitudinal clinical interpretations.

arxiv.orgpaperAdded 6 Oct 2026, published 5 Oct 2026

SpatialChain is a benchmark of 28,350 training and 899 test examples pairing spatially-oriented GQA questions with scene-graph-grounded reasoning chains, designed to audit whether VLMs' correct spatial answers reflect faithful reasoning or linguistic shortcuts. Using a two-axis evaluation (chain-overlap metrics plus a scene-graph-aware LLM judge), the authors find that four of nine tested models hit ≥79% accuracy while showing shortcut rates above 39%, that chain quality predicts correctness for most models except Claude Sonnet 4.6 and InternVL3.5-8B (which fail differently, via terse vs. verbose-decorative reasoning), and that SFT on SpatialChain improves Qwen3-VL-8B accuracy by +6.2pp while cutting its shortcut rate to 22%. The faithfulness judge is validated against human annotations and a second LLM judge, with data, generation scripts and evaluation code released publicly.

arxiv.orgpaperAdded 6 Oct 2026, published 5 Oct 2026

Sidekick is an open-source macOS plugin for Claude Code that lets you define a one-sentence persona (e.g. "a blunt senior Rust dev") and drive a coding session hands-free by voice: it listens via Apple's on-device speech recognition, speaks answers with macOS voices, and lets you interrupt it mid-reply or say "stop" to cancel a turn. It also surfaces Claude's multiple-choice decisions as spoken questions, matching answers like "the second one" against an echo-filtered transcript so the assistant doesn't hear and respond to its own voice. Implementation is a ~500-line TypeScript Claude Code hooks module plus a 90-line Swift speech listener, runs fully local with zero servers and zero telemetry, and installs via the `claude plugin marketplace`/`claude plugin install` commands.

x.comthreadPublished and added 4 Oct 2026

ldraw-nova is an open-source agentic pipeline where an LLM agent (tested with models like Claude Opus 5.5 and GPT-6 Astra) takes a build prompt, reads LDraw-building instructions, and produces a plan.json describing a model's geometry. From that plan it writes a Python generator script that, when executed, outputs an LDraw (.mpd) 3D CAD source file, viewable in 3D/VR (Meta Quest 3) or exportable to Blender-editable glTF. The agent uses jev-rerank, a semantic search tool with reranking backed by TypeSafe's Jev System One model, to find suitable LEGO parts and example models, falling back to full-text search without an API key; it also uses headless rendering and collision/gap detection to self-inspect and iterate on designs.

github.comrepoAdded 3 Oct 2026, published 2 Oct 2026

Tavus introduces Griffin, a "Human Interaction Model" (HIM): a full-duplex, video-to-video system that unifies real-time perception, conversational decision-making, and expressive audiovisual generation into one model rather than a cascaded pipeline. It uses a Continuous Conversational Modeling engine that makes sub-second turn-taking and nonverbal-behavior decisions, paired with a streaming speech generator built on a new continuous audio codec (Tavec) and a distilled autoregressive video diffusion model that renders 720p video in 320ms chunks. In live one-minute video calls, Griffin-Lite was judged human by 48% of participants (versus 2.4% for Tavus's prior Phoenix-4.5 system) and topped NVIDIA's VideoFDB full-duplex conversation benchmark on both generation and perception tracks.

tavus.ioarticleAdded 2 Oct 2026

Pocket TTS is a 100M-parameter text-to-speech model from Kyutai designed to run efficiently on CPU, generating audio faster than real-time (~6x on a MacBook Air M4) with ~200ms latency to first audio chunk and voice cloning from short audio samples. It supports English, French, German, Portuguese, Italian, and Spanish, ships a Python library and CLI, and has spawned numerous community ports (WASM, ONNX, Rust/Candle, MLX, C++, C#) and fine-tuned language variants (Czech, Hindi, Korean, Persian, Indonesian, Estonian, Welsh, Polish).

huggingface.corepoAdded 2 Oct 2026

Decrypt reports on Tavus' Griffin, a "Human Interaction Model" built for real-time, full-duplex face-to-face video conversation (listening, watching and talking simultaneously rather than turn-by-turn). On Tavus' own unverified test, Griffin-Lite convinced 48% of 54 participants on a one-minute video call that they were talking to a human, versus 2.4% for Tavus' prior stitched three-model system, and it ranks first on NVIDIA's VideoFDB benchmark with a 3.83/5 generation score (vs. 2.80 for the next-best system and 3.92 for a human reference) and an average audio-to-video delay of 0.43s on H100 GPUs. The article notes the test isn't a classic Turing test, isn't independently verified, and contrasts it with a UC San Diego study where GPT-4.5 passed as human in text conversation 73% of the time, while flagging real-world misuse risk from deepfake video-call scams (e.g., BlueNoroff/Lazarus Group).

decrypt.coarticlePublished and added 2 Oct 2026

IDRF is a reward fine-tuning framework for few-step masked discrete diffusion generators that replaces the intractable sequence-level KL penalty in reverse-KL-regularized objectives with an inverse-distillation regularization term, proven to upper-bound the true KL divergence to the reference distribution. It frames few-step diffusion sampling as a finite-horizon Markov decision process and optimizes reward via a clipped policy-gradient objective over the student's own trajectories, without needing reference-model rollouts. Across DNA, image, and text generation tasks, IDRF reaches high reward with up to 32x fewer denoising steps than the reference model while reducing reward hacking and preserving sample quality.

arxiv.orgpaperAdded 5 Oct 2026, published 2 Oct 2026

The World Embedding Benchmark is a new evaluation suite of 8,000 controlled simulation cases across 80 families (fluid mechanics, solid mechanics, dynamics, optics & electromagnetism), each pairing a rendered video with simulation-derived physical annotations. It tests pre-trained omnimodal embedding models on text-video retrieval, physical-property regression, and multiple-choice video-description classification, finding that current embeddings align poorly cross-modally while lightweight probes can still recover physical quantities, and that continual contrastive training improves alignment but hurts regression. The authors also show retrieval-augmented generation using these embeddings with MiniMax-H3 improves the physical fidelity of generated videos.

arxiv.orgpaperAdded 5 Oct 2026, published 2 Oct 2026

DEPICT is a training-free metric for scoring text-to-image alignment that replaces the fixed-YES assumption of decomposed VQA-based evaluators with an expected-agreement score between image-based and caption-only answers, weighted by how decisively the caption determines each question, then merges this with a holistic score. Evaluated across five benchmarks and eleven vision-language model backbones from three model families, it raises negation accuracy from 19% to 88% and outperforms other training-free metrics while beating fine-tuned evaluators on two of three human-correlation benchmarks.

arxiv.orgpaperAdded 5 Oct 2026, published 2 Oct 2026

MobiAgent is a dual-loop agentic framework for long-horizon mobile manipulation that separates deployment execution from policy self-improvement. An Inner Loop uses vision-language models for receding-horizon planning and visual reflection to compose atomic skills executed by flow-matching experts sharing a unified VLM backbone, while an Outer Loop autonomously segments, verifies, and clusters deployment rollouts to discover new atomic skills and continuously fine-tune the skill library without human annotation. On BEHAVIOR-1K it beats pi_0.5-TA by 22.5 percentage points, and autonomous data recycling raises RoboCasa success from 7.50% to 27.50% and Astribot S1 success from 32.5% to 57.5%.

arxiv.orgpaperAdded 5 Oct 2026, published 2 Oct 2026

This paper presents an automated pipeline that uses a vision-language model to annotate leaf midlines in rendered orthographic views of maize point-cloud scans, then applies deterministic geometric algorithms and differentiable NURBS fitting to reconstruct editable, organ-level procedural 3D plant models without manual tuning or species-specific segmentation training. Evaluated on 100 field-grown maize plants from the MaizeField3D dataset, it achieves a median whole-plant Chamfer distance of 5.4 mm and recovers 99.4% of curated reference leaves at IoU ≥ 0.5, outperforming an earlier semi-automated pipeline. It demonstrates that VLM annotations can become reliable quantitative measurements when paired with correcting geometric post-processing, enabling scalable generation of 3D plant assets for high-throughput phenotyping and in silico breeding.

arxiv.orgpaperAdded 5 Oct 2026, published 2 Oct 2026

CLIMB is a training-free, inference-time framework for multimodal retrieval-augmented generation that improves knowledge-intensive visual question answering. It first builds a compact, non-redundant evidence pool using an MMR-style objective balancing query relevance and redundancy, then applies an R/E/C critic (relevance, evidence specificity, cross-modal alignment) alongside a confidence estimator that only accepts an answer update when estimated confidence increases, giving a built-in stopping criterion without modifying the underlying retriever or MLLM. Experiments on Encyclopedic-VQA and InfoSeek show consistent gains over existing multimodal RAG baselines.

arxiv.orgpaperAdded 5 Oct 2026, published 2 Oct 2026

Patch-to-Prune (P2P) is a training-free framework that repurposes activation patching from mechanistic interpretability into an inference-time compute bypass for vision-language models, replacing unnecessary visual-token projections in certain decoder layers with fixed neutral proxy vectors while preserving sequence length, token order, and residual pathways. Tested on Qwen2.5-VL and LLaVA family models across seven multimodal benchmarks, P2P at 3 percent tolerance retains about 94 percent of dense accuracy while cutting FLOPs by 55 percent, and reveals that intermediate decoder layers do most of the task-relevant visual integration while early and late layers need little token-specific visual computation.

arxiv.orgpaperAdded 5 Oct 2026, published 2 Oct 2026

Latent Space AINews digest reports on Pi 1.0 and Pi Durable, an agent harness from the Earendil team ported to TypeScript that externalizes agent state for crash recovery, pluggable storage (Memory/SQLite/JSONL), concurrent branching conversations, hot-swappable tools, and multi-user state sync alongside conversation transcripts. The rest of the issue recaps frontier launches: Gemini 4 Argon, OpenAI efficiency-focused GPT-6.1 Sol (with Artificial Analysis cost comparisons against GPT-6 Sol and GPT-6 Astra), Upstage Solar Mini 4 (35B/3B MoE, 1M context), Black Forest Labs FLUX 3 image model, and Tavus Griffin video-to-video conversational agent.

latent.spacearticlePublished and added 2 Oct 2026

Announcement of an updated model card for JEV-27B-VL, an open-weight vision-language model released under Apache-2.0. The card covers the model's weights and a "decision adapter," a BoE (Bag/Blend of Experts) architecture paired with a System 1 / System 2 reasoning setup, calibration details and decision prompts, serving instructions for vLLM and SGLang, and an evaluation of image recommendation quality.

x.comthreadPublished and added 2 Oct 2026

EvoVLM searches over prompts, image preprocessing, generation budgets and answer extraction for frozen Qwen3-VL-2B-Instruct and Qwen3-VL-2B-Thinking, without fine-tuning weights. A local Qwen model ranks a catalog of typed mutations, a trusted renderer produces Python programs, and Pareto selection on accuracy and latency drives two small generations on scientific chart understanding (CharXiv subset). Recorded results on Dev128: Evolved Instruct keeps accuracy with 8.9% lower median latency, and Evolved Thinking gains 16.41 points of accuracy with 10.6% lower median latency. CPU-only replay and fail-closed evidence verification are included. The authors state the limits: small sample, overlap with the evolution set, and results tied to the recorded hardware.

github.comrepoAdded 16 Oct 2026, published 2 Oct 2026

InterEvolve is a test-time evolution method for humanoid loco-manipulation that lets a pretrained controller solve new tasks without retraining. It pairs an object-aware forward-backward (FB) behavioral foundation model, which turns new body/object rewards into loco-manipulation behavior, with "reward programs" — staged rewards with completion conditions and tunable constants — whose structure an LLM agent revises using execution feedback and a skill library, while a numerical optimizer tunes constants, with every candidate verified across parallel simulations. Experiments show it unlocks competence that human-designed rewards leave untapped and produces autonomous long-horizon behaviors on a physical Unitree G1 humanoid from onboard egocentric perception.

arxiv.orgpaperAdded 2 Oct 2026, published 1 Oct 2026

DuoMind is a distributed hierarchical framework for multi-robot coordination that pairs a per-robot vision-language-action (VLA) model for low-level execution with a vision-language model (VLM) orchestrator for high-level reasoning and inter-agent semantic communication. The authors also introduce RoboPoly, a new benchmark of long-horizon manipulation tasks requiring coordinated, closed-loop execution under distributed control, and show gains on RoboPoly and RoboTwin with ablations confirming the value of hierarchical orchestration and semantic messaging.

arxiv.orgpaperAdded 2 Oct 2026, published 1 Oct 2026

MosaiChunk is a spatio-temporal memory mechanism for autoregressive video generation that composes a "mosaic" of selected historical key-value (KV) cache entries across space and time, letting a frozen video generator recover fine-grained visual details of objects/scenes that fell out of the finite context window. Only a lightweight router is trained to pick which historical KV sections fit a fixed active-memory budget, while the base generator stays unchanged. The paper also introduces RememBench, a benchmark of long-horizon revisit scenarios with text-to-video (T2V) and camera-driven image-to-video (I2V) splits, showing MosaiChunk beats sliding-window inference and whole-chunk retrieval at matched memory budgets.

arxiv.orgpaperAdded 2 Oct 2026, published 1 Oct 2026

This arXiv paper studies emergent semantic specialization among experts in multimodal Mixture-of-Experts (MoE) models and introduces ExpertLens, a data-free method that identifies domain-specialized experts by decoding router weights into vocabulary tokens from pretrained weights alone. Using this to selectively fine-tune only the relevant experts, the authors match or beat full fine-tuning on math, medical, and remote-sensing tasks while updating only 21.7-47.0% of parameters, achieving a 4x training speedup and outperforming LoRA in both performance and efficiency.

arxiv.orgpaperAdded 2 Oct 2026, published 1 Oct 2026

Where-OPD introduces an on-policy self-distillation method for multimodal LLMs where a teacher model receives textual spatial guidance (object identities and coordinates from procedurally generated synthetic scenes) to locate relevant image regions, while a student learns to reproduce this behavior from just the image and question. The approach is annotation-free and scalable, improving performance on counting, document and chart understanding, and transferring from synthetic training to real-world perception benchmarks (CVBench, V*, ZoomBench, BLINK, HR-Bench, MME-RealWorld) with a 3.23-point average gain.

arxiv.orgpaperAdded 2 Oct 2026, published 1 Oct 2026

UniWAM is a unified architecture that combines a physical reasoner, a world generator, and an action predictor to jointly learn semantic understanding, visual generation, and action prediction for embodied agents, bridging vision-language-action (VLA) models and world-action models. It uses a data cleaning/annotation pipeline for human egocentric and robot data, represents low-level actions in natural language for VQA/egocentric/robot co-training, and introduces future visual noise augmentation plus history-conditioned flow matching to cut denoising steps while achieving state-of-the-art results on in-distribution performance, robustness, generalization, instruction following, and long-horizon tasks. The paper also reports a log-linear scaling law for unified human-robot co-training.

arxiv.orgpaperAdded 2 Oct 2026, published 1 Oct 2026

The paper introduces Selection-based Structured Reasoning (SSR), a framework that replaces free-form chain-of-thought generation in multimodal search agents with selection among pre-specified, reusable natural-language reasoning candidates, scored by likelihood via teacher-forced prefilling with a shared KV cache. Evaluated on seven multimodal search benchmarks using 2B and 4B scale models under both reinforcement learning and supervised fine-tuning, SSR matches the success rate of leading search agents of the same scale while cutting per-turn reasoning latency by over 90% and total per-question inference latency by 28-54%. It targets the problem of small models producing long, low-value reasoning traces before each action, which is costly at inference time.

arxiv.orgpaperAdded 2 Oct 2026, published 1 Oct 2026

LineupRL is an RLVR (reinforcement learning with verifiable rewards) pipeline for training time series captioning models, where the reward comes from a frozen LLM verifier that must match a generated caption back to its source time series among distractors, rather than judging caption quality directly. The authors show a 3B vision-language model trained with LineupRL outperforms SFT and RL baselines across two captioning benchmarks plus downstream forecasting/reconstruction tasks, and even beats the 72B VLM whose captions the SFT baseline distills from, using 1/24 the parameters, while resisting reward hacking.

arxiv.orgpaperAdded 2 Oct 2026, published 1 Oct 2026

Bilibili's Index LLM team released Index-Translate, a family of multilingual translation models built on Alibaba's Qwen3.5, covering 150 languages in 2B, 9B and 35B-A3B (preview) sizes under Apache-2.0 on Hugging Face and ModelScope. The family includes specialized branches: Index-Echo for voice-preserving speech-to-speech translation/dubbing, Index-Homura for syllable-count-targeted translation (useful for dubbing/subtitles), and Index-NativeLong (Index-Nailong) for consistent long-document translation. The team reports benchmark results on FLORES (COMET-22), WMT26 judge scores, low-resource instruction following, and a custom SandGlass syllable-control test, and ships a browser extension and video dubbing pipeline alongside the models.

x.comarticlePublished and added 1 Oct 2026

RMS-AQA is a two-stage spatial audio question answering benchmark for domestic environments, designed to test audio-language models' ability to first ground audible sound events and then perform spatio-temporal reasoning about them. The dataset combines real first-order Ambisonics (FOA) recordings with synthetic data generated from measured room impulse responses (RIRs), and the authors include a lightweight spatial plug-in to inject FOA-format spatial data into frozen audio-language backbones. Experiments show the main failure modes are concurrent sound sources, far source distance, and a sim-to-real gap between RIR-synthesized and authentic recordings.

arxiv.orgpaperAdded 2 Oct 2026, published 1 Oct 2026

DeepSeek-V4.1-Flash is a multimodal Mixture-of-Experts model with 552B backbone parameters, 1M-token context, and a novel Causal Encoder-Decoder (CED) architecture where the decoder's global KV cache is projected from encoder hidden states rather than recomputed per layer, activating only 8B/16B parameters per token during prefill/decode. It introduces Compressed Sparse Attention 2 (CSA2) with FP4 KV caching to cut the global KV cache footprint to 890 bytes/token (~1/4 of DeepSeek-V4-Flash, ~1/437 of DeepSeek-V1), alongside Engram conditional memory, DSpark speculative decoding, and a continuously controllable reasoning-effort setting (1–100). The card documents pre/post-training recipes (45T tokens, SFT→RL→on-policy distillation with synthetic agent environments) and reports results across agentic and code benchmarks (Terminal-Bench 2.1/3.0/4.0, DeepSWE v1.1, GPQA Diamond, Chartography, ZeroBench, AutomationBench), plus deployment instructions for Transformers, vLLM, SGLang, and Docker.

huggingface.corepoAdded 30 Sep 2026

Multimodal Flow (MF-1) is a fully continuous generative model that unifies language and vision modeling by representing both text and images as ordered continuous "hyperchunks" processed through a single shared chunk-causal Flow Matching backbone, avoiding the visual quantization bottleneck of discrete-token unified models and the mismatched objectives of hybrid discrete/continuous approaches. Joint attention handles cross-modal interaction while modality-specific feed-forward networks process each modality, and the model predicts multiple chunks in parallel during training but generates sequentially at inference. Pretrained at 0.6B/1.2B/1.6B scales on 150B tokens, MF-1 scores 82.8 average on GenEval/DPG-Bench and 75.3 on VQAv2/MMBench/POPE, remaining competitive with unified models trained on far more data and outperforming matched hybrid/discrete baselines; code and model are publicly released.

arxiv.orgpaperAdded 1 Oct 2026, published 30 Sep 2026

The paper introduces Ranking-PE, a modification to reflective prompt-optimization methods like GEPA that replaces per-instance accuracy scoring with pairwise AUROC-equivalent scoring (via the Wilcoxon-Mann-Whitney identity) across the scores matrix, reflection feedback, and candidate selection. Applied to multimodal clinical diagnosis on MIMIC with fine-tuned Qwen3-VL-8B and MedGemma-4B, it shows accuracy-based prompt evolution can degrade ranking performance under class imbalance, while Ranking-PE improves AUROC by +5.8pp and +16.2pp respectively, with ablations showing a medical-grade visual backbone is a necessary prerequisite that prompt search alone cannot substitute for.

arxiv.orgpaperAdded 1 Oct 2026, published 30 Sep 2026

The authors show that reported gains from d'Ascoli et al. (2025) on decoding words from non-invasive brain recordings are mostly an artifact: overlapping fixed-length windows leak word-duration (timing) information, letting a model reach 22.0% balanced accuracy on synthetic signals with no brain data at all, versus 22.3% on real recordings. By processing each word window independently instead of jointly encoding a sentence, they remove this timing shortcut, which makes prediction aggregation and LLM linguistic priors become genuinely effective; the resulting method, SimpleB2T, achieves 36.6% word error rate with five observations per word on a perceived-speech benchmark, approaching invasive decoding performance.

arxiv.orgpaperAdded 1 Oct 2026, published 30 Sep 2026

Physis-Lang is a self-evolving framework that uses natural language (describing entities, causes, interactions, governing principles, and temporal evolution) as the shared physical-knowledge representation across data curation, training, and generation for video world models. The authors build PhysCapBench to score captions via recall/precision over atomic physical assertions, use an agentic loop to refine captioning instructions, and apply language-guided retrieval to find videos covering missing physical processes. Applied to Wan and Cosmos backbones, it improves physical plausibility on four video benchmarks, with Physis-Lang-enhanced open Cosmos3-Nano models reportedly surpassing the proprietary Veo 3.1.

arxiv.orgpaperAdded 1 Oct 2026, published 30 Sep 2026

ViTeX-Bench is a benchmark suite for high-fidelity video scene text editing, pairing a dataset of 387 real-world 720p videos (text-region masks plus editing instructions, 230 for training and 157 frozen for evaluation) with a three-axis, 13-metric evaluation protocol covering text correctness, visual/temporal quality, and edit locality. The authors benchmark eight baselines across four editing families and release ViTeX-Edit-14B, an open-source reference editor using motion-aligned glyph-video conditioning that achieves the best CharAcc (0.688) among video-native editors and the lowest text-crop Warp among raw outputs.

arxiv.orgpaperAdded 1 Oct 2026, published 30 Sep 2026

AssemblyWorld is an interactive 3D environment where general-purpose agents assemble objects (furniture, industrial parts, fractured pieces) by inspecting rendered 2D views and manipulating supplied rigid parts, optionally guided by images or assembly manuals, without any assembly-specific fine-tuning. The accompanying AssemblyWorldBench benchmark covers 100 tasks across 80 objects and evaluates eight agent systems geometrically; the best system reaches 80.9% part accuracy but only 59.4% complete-assembly success, and open-source systems lag closed-source ones in both reliability and accuracy. The paper analyzes interaction trajectories and failure modes to characterize the gap between approximate structure recovery and precise 3D reconstruction.

arxiv.orgpaperAdded 1 Oct 2026, published 30 Sep 2026

This paper introduces VideoMSN, a Masked Siamese Network that repurposes standard image Vision Transformers for self-supervised spatio-temporal video representation learning by representing videos as 'super images' (grids of sampled frames) and aligning spatial-masked and temporal-masked views with a shared ViT encoder. Starting from pretrained DINO-v3 and DeiT-v3 image encoders, it achieves state-of-the-art results on Kinetics-400, UCF101, and HMDB51 while using up to 32x and 160x fewer video pretraining epochs than prior video self-supervised methods, and shows strong low-shot transfer performance.

arxiv.orgpaperAdded 1 Oct 2026, published 30 Sep 2026

Ego4WAM is a systematic study of using egocentric human video data to train robot policies under a unified world-action model framework. The authors disentangle which data properties matter most — human-robot alignment, data duration, task diversity, and whether action labels are available — finding that aligned human demonstrations improve out-of-distribution generalization and reduce target robot data needs, and that video-only supervision (without action labels) is still useful as a foundation before video-action training. Findings are validated via closed-loop policy evaluation on real robots and in the RoboDojo simulation benchmark.

arxiv.orgpaperAdded 1 Oct 2026, published 30 Sep 2026

LokalBot 0.9.2 is a free, open-source local-first assistant that records meetings and screen activity, using five small open-weight models chained together entirely on a MacBook (6.8 GB total): Nemotron 3 for speaker diarization, Qwen3-ASR 1.7B for transcription, LFM2.5 1.2B for sentence completion, Harrier 0.6B for semantic search, and Qwen3.5 4B for recap generation. It exposes this captured 'memory' to Claude Code and Codex through an MCP server or CLI, letting coding agents pull context from past meetings and screen content.

x.comthreadPublished and added 30 Sep 2026

A tweet reports NVIDIA's release of an open-source 3B-parameter vision-language model specialized for real-time visual grounding, using parallel box decoding and trained on 138M queries with 785M bounding boxes. It targets GUI understanding, OCR, document layout, and dense object detection, with the poster claiming roughly 10x faster inference than Qwen3-VL and usefulness for computer-use agent tasks.

x.comthreadPublished and added 30 Sep 2026

Hugging Face introduces the Open TTS Leaderboard, which ranks text-to-speech and voice-cloning models using objective, scalable metrics rather than slow human-voting arenas: intelligibility via WER/CER computed with Qwen3 ASR, speed via RTFx (batched) and TTFA (streaming) on H200 GPU/CPU, and speaker similarity via WavLM embedding cosine similarity. It covers multilingual evaluation (Seed TTS Eval, CV3 Eval zero-shot) across English, Chinese, Japanese and Korean, highlights leading open models like Kokoro-82M, supertonic-3, s2-pro, OmniVoice and Fun-CosyVoice3, and includes a Listen tab for human comparison and voting and a Streaming tab benchmarking TTFA, with plans to open-source the evaluation scripts.

huggingface.coarticlePublished and added 30 Sep 2026

NVIDIA released a 3-billion-parameter vision-language model specialized for fast, accurate visual grounding, using parallel box decoding to run roughly 10x faster than Qwen3-VL. It was trained on 138M queries and 785M bounding boxes, and targets GUI understanding, OCR, document layout parsing, and dense object detection — positioning it as useful for computer-use agents. The model is open source.

x.comthreadAdded 30 Sep 2026, published 29 Sep 2026

Point2Part is a promptable 3D part-decomposition model that, given 3D point prompts (one per desired part), jointly partitions an image or mesh into non-overlapping parts that fully cover the shape, unlike prior methods that allow overlaps or gaps. It builds on a pretrained 3D generation model's shape latent space, adding a prompt encoder mapping point prompts to part tokens and a coarse-to-fine part decoder that assigns every shape position to exactly one part, unifying image-to-part generation, mesh-to-part generation, and part segmentation in one model. The authors report outperforming prior SOTA on part-quality metrics and an order-of-magnitude improvement in part compatibility across all three tasks.

arxiv.orgpaperAdded 30 Sep 2026, published 29 Sep 2026

Imagine3D-LLM appends learnable summary tokens after image tokens in an MLLM, decoding them into a compact 3D Gaussian Splatting representation supervised by a photometric reconstruction loss, trained jointly with next-token prediction. This mimics human coarse spatial reasoning (identifying common objects across views, inferring relative geometry) rather than relying on fine-grained pixel correspondence or fused 3D geometry foundation model features, and the paper shows this reconstruction objective also improves cross-frame correspondence in the LLM's image features. The approach outperforms prior methods across multiple spatial reasoning and 3D understanding benchmarks.

arxiv.orgpaperAdded 30 Sep 2026, published 29 Sep 2026

PRISM is a real-to-sim-to-real framework that scales humanoid loco-manipulation training by generating hundreds of diverse "counterfactual" human-object interaction videos from a handful of real exemplar clips via video-to-video generation. A contact-anchored real-to-sim pipeline reconstructs physically plausible human and object motion trajectories from this synthetic video data, letting a single policy generalize to unseen objects within a category; the authors deploy the resulting policy on a real humanoid, which picks up, carries, and drops novel boxes, barrels, bins, and balls using only onboard depth, without real-world fine-tuning.

arxiv.orgpaperAdded 30 Sep 2026, published 29 Sep 2026

This paper shows that adding an adversarial loss as a post-training correction to pretrained pixel diffusion/flow-matching models (applied at non-high-noise timesteps, architecture and sampler unchanged) restores missing high-frequency, fine-scale natural-image statistics that these models systematically underproduce. Through frequency-band/power-law analysis and nearest-neighbor, recall, and no-GAN matched-SFT controls, the authors rule out memorization or mode-dropping as explanations, and show the effect jointly improves distribution fidelity, coverage, prompt alignment and perceptual quality — but does not transfer comparably to latent diffusion configurations, pointing to direct pixel-output access as the key enabling factor.

arxiv.orgpaperAdded 30 Sep 2026, published 29 Sep 2026

Rho is a family of open-weights vision-language-action (VLA) models for bimanual robotic manipulation, designed for data-light adaptation across three dual-arm embodiments (YAM Box, UR AI Trainer, FR3 Duo). The paper ablates the action-expert architecture and training recipe, showing that "embodiment midtraining" improves downstream adaptation, and introduces an internal latent policy that learns from as few as 15 corrected episodes to steer a frozen flow-matching action expert toward out-of-distribution task situations. The authors release base and embodiment-specific checkpoints, positioning Rho as both a strong general-purpose manipulation model and a practical foundation for efficient adaptation.

arxiv.orgpaperAdded 30 Sep 2026, published 29 Sep 2026

This paper studies how to design the shared representation space used by world-action models, which jointly learn robot control policies and predict future observations. It finds that neither reconstruction fidelity nor raw pre-trained perceptual features alone yield good policy learning, and proposes ReWAM, built on pre-trained DINO features with a Temporal Representation Bottleneck and Action-Grounded Representation Shaping that routes only action-loss gradients into the bottleneck so the policy shapes what is encoded while the world model learns its dynamics. Without generative video pre-training, ReWAM reaches 93.6% success on RoboTwin 2.0 and an 8.28% success rate on RoboDojo using roughly 600 hours of embodied pre-training data.

arxiv.orgpaperAdded 30 Sep 2026, published 29 Sep 2026

EmoRES-TTS introduces a training-free vector steering method for emotion-conditioned text-to-speech that decomposes an emotion direction into a shared 'away-from-neutral' component and a residual 'toward-target-emotion' component, controlling each separately without retraining the backbone model. Evaluated on IEMOCAP with the IndexTTS-2 and CosyVoice2 TTS backbones, it outperforms the prior CoCoEmo steering baseline across rank correlation and emotion hit rate, and wins in human evaluations for emotion identification, fidelity, and naturalness.

arxiv.orgpaperAdded 30 Sep 2026, published 29 Sep 2026

DMA^2 is a distribution matching distillation (DMD) method tailored to pixel-space (rather than latent) diffusion teachers for text-to-image generation. It introduces DINO-Adv, which removes the heavy fake-score critic from the adversarial gradient path and instead supplies local patch guidance via an external DINOv2 representation, plus AF-Loss, a parameter-free auxiliary semantic distribution-field objective operating in DINOv2 feature space on rolling real/generated supports. The resulting four-step student outperforms its 25-step teacher and other few-step distillers on DPG-Bench, GenEval, VQAScore, and COCO30K.

arxiv.orgpaperAdded 30 Sep 2026, published 29 Sep 2026

This paper introduces Grounded Entity Biographies (GEB), a long-video memory framework that links visually grounded observations of the same physical object or person across clips into retrievable "biographies," addressing the problem of text-derived entity descriptions failing to resolve physical identity across hours- or days-long recordings. During question answering, these biographies are retrieved alongside episodic evidence so a model can follow an entity through events via identity links established at memory-construction time. Evaluated on four long-video QA benchmarks (day-long and week-long recordings), GEB improves over prior memory frameworks, reaching 72.0% accuracy on EgoLifeQA — 4.4 points above the prior best published result — with ablations showing both grounded identity association and biography reading contribute to the gains.

arxiv.orgpaperAdded 30 Sep 2026, published 29 Sep 2026

FracGen is a fracture-aware video generation model that, given a single image of an intact object, generates plausible and controllable fracture dynamics (where an object tears, crack propagation speed, pre-failure deformation) conditioned on physics signals. It is trained using FracSim, a material point method (MPM) simulation framework augmented with a continuum damage model that produces paired fracture videos and dense pixel-aligned physical fields at no extra rendering cost; FracGen jointly predicts these fields alongside RGB video and is supervised with physics-informed consistency losses. The authors also introduce a benchmark for evaluating physical plausibility of generated fracture video, showing FracGen outperforms existing video generation baselines in both physical and visual fidelity.

arxiv.orgpaperAdded 30 Sep 2026, published 29 Sep 2026

This paper introduces stochastic world models as a new class of perception surrogates for formally verifying vision-based neural feedback (control) systems, replacing GAN-based surrogates that are large and hard to verify. The world model is built with physically grounded latents using operations standard verifiers can bound, and reproduces held-out frames more faithfully than GAN surrogates with up to 130x more parameters. Combined with a verification procedure mixing falsification, adaptive refinement, symbolic and backward analyses, it resolves the entire state space on an emergency braking benchmark (38% more than the prior state-of-the-art verifier) and over 80% of the state space on a previously-unverified RGB version of the benchmark.

arxiv.orgpaperAdded 30 Sep 2026, published 29 Sep 2026

This paper shows that internal expert-routing probabilities in Mixture-of-Experts vision-language models (Qwen3-VL-30B-A3B-Instruct and Gemma-4-26B-A4B-it) carry a strong, low-cost signal for detecting when a model is about to answer a question about an object that isn't actually in the image (a 'target-absence grounding failure'). Linear detectors trained on target-token routing probabilities achieve ROC-AUCs of 0.9988/0.9956 on GQA-Inpaint (retaining 0.78-0.81 on the external OBER dataset), and gating a target-aware review prompt on these detections improves end-to-end accuracy by up to +22% without modifying model weights. The work matters for practitioners building on MoE VLMs because it demonstrates a weight-free, pre-generation hallucination detector that reuses computation the model already produces.

arxiv.orgpaperAdded 30 Sep 2026, published 29 Sep 2026

Simon Willison built an experimental local-only photo tool that detects faces and automatically blurs them to protect the identity of strangers (e.g. protesters) photographed in public, also stripping metadata. It runs entirely client-side using Google's MediaPipe compiled to WebAssembly via @mediapipe/tasks-vision, with the BlazeFace model for face detection, and was built with help from GPT-6 Astra.

simonwillison.nettoolAdded 2 Oct 2026, published 29 Sep 2026

A Latent Space AINews digest covering AMD's 8.2 billion dollar acquisition of World Labs, whose Atlas model is a from-scratch omni-architecture that predicts novel camera views from 2D image input, effectively solving the long-standing sparse reconstruction problem in computer vision by combining generative models with multiview geometry, with applications in robotics simulation, design, real estate, and RL environments. The issue also recaps the Claude Sonnet 5.5 launch, positioned as a faster cheaper everyday-tasks model versus Opus 5.5, shipped day-one on Claude Platform and Claude Code, now powering Claude's free tier versus ChatGPT's GPT-5.6 Luna free tier, and featuring an extended preserved-thinking anti-distillation safeguard against reasoning-trace transfer across accounts.

latent.spacearticlePublished and added 29 Sep 2026

FurE is a strand-based method for reconstructing editable 3D animal fur from multi-view images without requiring animal-fur training data. It optimizes a root-conditioned latent field decoded into strand geometry via a PCA-based decoder trained on human-hair strand data, combined with a defurred animal body recovered using a surface-constrained Gaussian Frosting representation and part-based priors, achieving a 10x speedup over dense per-strand optimization baselines while retaining strand fidelity on synthetic and real-world sequences.

arxiv.orgpaperAdded 29 Sep 2026, published 28 Sep 2026

PDMD (Projected Distribution Matching Distillation) is a one-line fix to Distribution Matching Distillation (DMD) for video diffusion models that projects out the component of the DMD update parallel to the student-critic endpoint residual, filtering accumulated critic errors that otherwise cause oversaturation and artifacts during few-step distillation. The authors prove the residual is an unbiased estimate of critic endpoint error and show the projection removes a constant fraction of that error while discarding negligible true DMD signal. On Wan2.1 it reaches a VBench score of 83.73 at 4 NFE (+1.03 over DMD), and on MiniMax-H3 joint video-audio generation it improves VideoGen-Eval visual score and tops all six audio metrics among 4-NFE models, with code and models released.

arxiv.orgpaperAdded 29 Sep 2026, published 28 Sep 2026

UMM-Reflection is a reinforcement learning method that trains a unified multimodal model to jointly learn reflection text and image revision within a single closed loop: diagnose an image's flaws, revise it, observe the result, and repeat. Sibling trajectories sharing one initial image enable a group-relative advantage that compares whole reflection strategies rather than assigning credit per round, and one trajectory-level advantage updates both the reflection tokens and the flow-based revision head jointly, unlike SFT cold-starts or RL that optimizes only the renderer. On BAGEL, it improves GenEval by 12.05 points over SFT and transfers to unseen benchmarks WISE, OneIG-Bench and T2I-CompBench++, with no external verifier needed at inference.

arxiv.orgpaperAdded 29 Sep 2026, published 28 Sep 2026

This paper introduces a channel-gated fusion model for sparse inertial motion capture that learns per-channel, per-stream temporal gates to weight the reliability of consumer earbud head IMU and smart-insole foot IMU signals, trained with an auxiliary reliability objective on synthetically corrupted data. On a new 35-take benchmark (using SAM-3D-Body pseudo-ground-truth labels), the gated model achieves 69.4mm error on clean data and remains most robust across simulated bias, drift, and dropout faults, automatically suppressing biased firmware-fused foot-orientation channels without test-time supervision (dropout detection AUROC 0.92-0.999). It is compared against a fine-tuned HMD-Poser baseline, which is more accurate on clean data (64.4mm) but degrades far more under worst-case faults (+16.1mm vs +3.5mm).

arxiv.orgpaperAdded 29 Sep 2026, published 28 Sep 2026

This arXiv paper introduces a unified theoretical framework for "distributional training" in one-step visual generation, showing that existing methods like FD-Loss and Gaussian-kernel Drifting are special cases of Gaussian optimal transport and kernel-density KL matching under Wasserstein gradient flow. Building on this, the authors propose MGFlow, which models feature distributions with Gaussian mixtures at adjustable granularity and combines mass-constrained sample assignment with paired component updates to fix mode collapse. MGFlow sets new state-of-the-art FDr scores on ImageNet 256×256 (1.45 on pMF-H, 1.64 on JiT-H) and post-trains FLUX.2 into a one-step generator that beats the original four-step model on GenEval and PickScore.

arxiv.orgpaperAdded 29 Sep 2026, published 28 Sep 2026

DexRoam is a system for learning mobile bimanual dexterous manipulation by training VLA (vision-language-action) policies jointly on human and robot demonstrations. It uses a tracker-free capture setup (consumer VR headset plus head-mounted stereo camera) and three alignment stages (embodiment, action-semantic, temporal) to map continuous whole-body human motion into robot action space without discarding fine-grained coordination. Experiments with GR00T N1.7 and pi0.5 backbones show human demonstrations raise real-world task success from 29% to 56% and 32% to 57% respectively, matching robot-only training with half the robot data.

arxiv.orgpaperAdded 29 Sep 2026, published 28 Sep 2026

This paper studies how to initialize efficient linear-attention Vision Transformers from pretrained Softmax ViT weights instead of training from scratch. It finds that attention weights are operator-specific and don't transfer well by direct copying (distillation with a tailored loss is needed to recover token-routing behavior), while MLP weights are operator-agnostic and can simply be copied, and combining copied MLPs with distilled attention lets linear ViTs match or surpass Softmax ViT performance across model sizes and datasets.

arxiv.orgpaperAdded 29 Sep 2026, published 28 Sep 2026

InfiniHand is an end-to-end streaming feed-forward model that jointly estimates MANO hand-pose parameters, camera trajectory, and world-space hand location from uncalibrated egocentric video, replacing the usual cascade of separate hand-pose and SLAM systems. It uses persistent spatiotemporal memory coupled with hand-centered visual features, trained in two stages on ~5,000 hours of aggregated egocentric datasets, and reports a 21.4% PA-p reduction on ARCTIC over ViDiHand plus reduced world-space drift while running at 11.19 FPS (2x the throughput of HaWoR).

arxiv.orgpaperAdded 29 Sep 2026, published 28 Sep 2026

GeoVerse is a novel view synthesis framework that generates within the geometric latent space of a pretrained 3D foundation model while injecting appearance priors from the video generative model Wan2.2 VACE via a ControlNet-style adapter, aiming to reconcile faithful reconstruction with plausible completion of unseen scene content. A global spatial memory aggregates observed and synthesized content and reprojects target-aligned guidance to enforce cross-view world consistency. It reports a 2.23 dB PSNR improvement on DL3DV and 32.4% lower absolute trajectory error on Mip-NeRF360 versus the GLD baseline.

arxiv.orgpaperAdded 29 Sep 2026, published 28 Sep 2026

FlowAct-R2 is a framework for interactive humanoid video generation that pairs a Streaming Multimodal Reference Diffusion Transformer (built on the Seedance 2.0 Mini reference-to-video backbone) with a Proactive Interaction Agent. The diffusion component accepts rolling action prompts plus continuously updated image, audio and video references, using video-driven rotary positional embeddings and partially noised historical motion frames to keep appearance consistent and avoid drift over long generation runs. The agent component separates offline persona/agenda/skill preparation from online scheduling, audience response and interruption handling, enabling real-time 720p, hour-scale streaming avatars for use cases like live shopping, video chatting and live vlogging.

arxiv.orgpaperAdded 29 Sep 2026, published 28 Sep 2026

This paper studies how camera placement affects the accuracy of AI-based pose estimation for detecting movement errors in rehabilitation exercises. The authors introduce REHAB26-ViewAngles, a dataset of correct and incorrect exercise executions filmed from many camera angles, plus a novel separability metric to measure how well an algorithm distinguishes valid from faulty repetitions. They benchmark single-camera 2D and 3D pose estimation against four multi-camera strategies (orthogonal 2D views, 3D triangulation, weighted 3D fusion, and a transformer trained on two synchronized cameras), finding an optimally placed 2D camera beats the standard frontal view by 16.9% and that combining two views adds up to 13.1% more accuracy — practical guidance for deploying rehabilitation monitoring at home or in clinics.

arxiv.orgpaperAdded 29 Sep 2026, published 28 Sep 2026

This paper systematically evaluates GPT-6 Astra alongside five other frontier general-purpose AI systems across 34 capabilities and 55 benchmarks spanning nine areas of computer vision, benchmarking them against dedicated specialist models and human performance where available. It finds general-purpose systems increasingly match reference performance on semantic interpretation, reasoning, and object-centric prediction, but still lag on metric geometric accuracy, faithful reconstruction, temporally consistent dense prediction, and fine-grained specialist visual knowledge. Astra shows the strongest gains among the systems tested, particularly in visual/spatial reasoning and structured prediction tasks.

arxiv.orgpaperAdded 29 Sep 2026, published 28 Sep 2026

MiMo-V2.6-Pro-RL is Xiaomi's flagship omnimodal (text/image/video/audio) sparse MoE model, 1.02T total / 42B activated parameters with a 1M-token context, built around scaling RL compute toward self-improvement. It uses a single mixed GRPO run across coding, agentic, visual and cybersecurity tasks, a novel groupwise agentic grading scheme (Groupwise Reward Synthesis + Groupwise Advantage Redistribution) to rank passing rollouts rather than just pass/fail, and a multi-teacher on-policy distillation stage (MOPD2); it also ships a 5-layer speculative MTP decoder and is deployable via vLLM and SGLang.

huggingface.corepoAdded 28 Sep 2026

MEOW is a feed-forward 3D reconstruction model that jointly predicts metric pointmaps and camera poses from N-view image tuples mixing perspective, fisheye, and full 360° panoramic images in a single forward pass, without any camera calibration, distortion parameters, or camera-type labels supplied. It keeps a perspective-pretrained backbone and adapts it to heterogeneous cameras purely through a procedural synthetic data engine that renders scenes across a continuous manifold of camera models with exact rays, depth, and verified covisibility, then transfers zero-shot to real captures, beating the Wid3R baseline (79.9 vs 53.8 mAA@30 on heterogeneous 2D3DS tuples even when Wid3R is given ground-truth camera types) and scoring 79.4 AUC@30 on a new laser-scanned mixed-camera benchmark.

arxiv.orgpaperAdded 29 Sep 2026, published 28 Sep 2026

This paper extends D4orm, a dynamics-aware diffusion-denoising framework for multi-robot trajectory planning, into a family of planner architectures: a decoupled planner for scalability, an online receding-horizon planner with feedback control, and a distributed planner for resource-constrained settings. D4orm uses massively parallel GPU sampling to iteratively denoise candidate control trajectories into kinodynamically feasible, conflict-free solutions, outperforming sampling-based baselines like MPPI and a learned diffusion-model method. The authors demonstrate zero-shot deployment on ten real quadrotors, deconfliction with 100 simulated robots, and fully onboard distributed 'lifelong' operation on six ground robots.

arxiv.orgpaperAdded 29 Sep 2026, published 28 Sep 2026

Awesome-list repository cataloguing 183 open-source 'skills' and toolkits that let Claude Code, Codex and other coding agents generate or edit video: rendering frameworks like HyperFrames and Remotion, motion graphics, explainer/promo video generators, short-form editors, digital avatars and music videos. Each entry is read and security-graded by a companion service (Agent Skills Hub / agentskillshub.top), with a machine-readable JSON feed and category breakdowns (frameworks, promos, explainers, editing, shorts, avatars, animation, motion graphics, scripts, music videos).

github.comrepoAdded 29 Sep 2026, published 27 Sep 2026

Simon Willison describes prompting Claude Opus 5.5 with reference photos to generate an animated pixel-art HTML5 canvas scene (kākāpō parrots partying with confetti), then using a local Claude Code session to drive Playwright, click through the page at scripted timestamps, and record a 15-second video of the result for a keynote slide. The post includes the full Playwright script Claude Code produced, showing how a coding agent can be tasked with browser automation and video capture as an end-to-end deliverable, not just code generation.

simonwillison.netwriteupAdded 29 Sep 2026, published 26 Sep 2026

MiMo-V2.6-Flash-RL is Xiaomi's efficiency-balanced checkpoint in the MiMo-V2.6 series, a 309B-total/15B-activated sparse MoE model with native text, image, video and audio support and a 1M-token context window. It's trained with a single mixed reinforcement learning run (GRPO at large batch scale) across coding, agentic, visual and cybersecurity tasks, using a novel Groupwise Agentic Grading scheme (Groupwise Reward Synthesis and Groupwise Advantage Redistribution) to build a self-improvement loop, followed by Multi-Prefix Multi-Teacher On-Policy Distillation (MOPD2). The card documents the architecture (MiMo ViT vision encoder, AudioTokenizer audio encoder, 5-layer speculative MTP decoder) and deployment recipes for vLLM and SGLang.

huggingface.corepoAdded 26 Sep 2026

This paper studies verifier-guided self-DPO training of vision-language models and identifies "silent negative transfer": a verifier that reliably improves one task (MathVista, +8.0 held-out) can secretly harm generalization on another (MMMU) even while self-training metrics keep improving. The authors decompose this into gradient fidelity (alignment between verifier-induced and correct update directions) and gradient transferability (alignment between correct and held-out directions), deriving a Safe-Transfer Margin certifying positive held-out alignment. They propose Asymmetric Acceptance Gating (AAG), a precision-first preference-pair filter that raises fidelity on MMMU from 0.29 to 0.42 and restores positive held-out alignment (from -0.13 to +0.22), beating raw verifier-guided training across six configurations.

arxiv.orgpaperAdded 26 Sep 2026

Ling-3.0-flash-VL is InclusionAI/Ant Group's native multimodal model built on Ling-3.0-flash, adding image and video understanding via a ViT encoder, VideoRoPE, and a hybrid MoE backbone (42 layers alternating KDA and Gated MLA at 5:1) with 124B total parameters but only 5.5B activated per token, and a 256K context window. It targets agentic multimodal workflows — understanding, reasoning, and acting on visual interfaces — scoring 42 on Artificial Analysis Intelligence Index v4.1.1, up from 38 for the text-only Ling-3.0-flash. The card includes concrete SGLang and vLLM serving recipes for various GPU configurations.

huggingface.corepoAdded 26 Sep 2026

FuseReg is a training method for representation autoencoders (RAEs) that replaces fixed heuristic selection of pretrained-encoder layers with training over random subsets of encoder layers, explicitly regularizing sensitivity to cross-layer disagreement. On ImageNet-256 with DINOv3-L, a single FuseReg decoder matches or beats decoders specialized to fixed layer fusions across full, sparse, and single-layer configurations, and swapping it into an unchanged RAEv2 DiT-XL generator cuts unguided gFID by 27%; joint regularization during diffusion training further reduces unguided gFID by 29% on DiT-Base. The work targets the trade-off where shallow encoder layers preserve pixel detail but deep layers yield better generation metrics, narrowing the reconstruction-generation gap without modifying the pretrained encoder.

arxiv.orgpaperAdded 28 Sep 2026, published 25 Sep 2026

This arXiv paper addresses aligning the distribution of an attribute in a black-box generative AI model's outputs, such as fairness attributes or synthetic data representativeness, with a user-specified target distribution, using only repeated query access without model internals. The authors develop query-minimizing post-processing algorithms for exact and approximate alignment across multiple generated outputs, proving asymptotic optimality, and validate the approach on text-to-image generation and geocoded persona generation tasks.

arxiv.orgpaperAdded 28 Sep 2026, published 25 Sep 2026

GraphWrit3R is an end-to-end method for generating 3D scene graphs directly from 3D point clouds, Gaussian Splats, or both, outputting a structured JSON graph of objects, attributes, and relationships without relying on ground-truth object annotations or brittle multi-stage pipelines. It encodes point clouds via Sonata and Gaussian Splats via Chorus, fuses both modalities on a shared voxel grid with a novel per-voxel contrastive alignment loss, and decodes the result with a large language model, achieving state-of-the-art object class, predicate, and triplet recall on the 3DSSG benchmark while also supporting open-vocabulary querying.

arxiv.orgpaperAdded 28 Sep 2026, published 25 Sep 2026

RePlay is a spoken dialogue system, adapted from PersonaPlex, that handles multi-turn conversation by retrieving and replaying pre-recorded lines instead of generating speech, letting applications guarantee exact wording and vocal delivery. It probes model layers to find the point where the upcoming response becomes recoverable, uses that hidden state as a retrieval query, and replaces text/speech generation with lightweight turn-taking and retrieval heads, achieving a median latency of 383ms (3-7x faster than ASR-LLM cascades) at the cost of lower exact-line accuracy; a user study found participants preferred RePlay 63% vs 12% over a fast small-LLM cascade.

arxiv.orgpaperAdded 28 Sep 2026, published 25 Sep 2026

This paper presents a two-layer humanoid locomotion architecture: a perceptive flow-matching motion generator that plans whole-body trajectories from raw depth images, paired with a control-guided RL tracking policy that follows those motions. The core contribution is a sample-efficient off-policy RL fine-tuning loop — using structured search with the generator plus advantage weighted regression — that improves terrain consistency on unseen geometries, boosting successful terrain traversals by up to 25 percentage points and skill selection by up to 80 points. The system is deployed on a Unitree G1 humanoid, enabling walking, running, jumping on/off boxes, and stair traversal outdoors using only raw depth images (no odometry or height maps).

arxiv.orgpaperAdded 28 Sep 2026, published 25 Sep 2026

This paper studies Implicit Neural Representations (INRs) as a lightweight alternative to conventional CNN pipelines (e.g. U-Net) for brain MRI semantic segmentation, analyzing their scaling behavior and cross-domain generalization. The authors find INR-based models excel in low-parameter, limited-augmentation regimes but don't scale as well as U-Net with larger capacity, and that segmentation-relevant information is distributed across multiple INR layers. Building on this, they propose HierINRSeg, a hierarchical multi-layer INR architecture that beats the MetaSeg baseline by 5.6 Dice points in-domain and 8.2 points out-of-domain.

arxiv.orgpaperAdded 28 Sep 2026, published 25 Sep 2026

MexHat is a new video dataset of around 1,000 clips for hate speech detection in Mexican Spanish, annotated for a three-way task (no negative content, offensive content, hate speech) and a fine-grained task with three hate-speech sub-categories. The paper reports baseline model results highlighting the challenges of capturing culturally-specific, context-dependent multimodal cues in a non-English, low-resource setting.

arxiv.orgpaperAdded 28 Sep 2026, published 25 Sep 2026

BeatGraph is a self-supervised ECG representation model that tokenizes signals at the level of individual heartbeats rather than fixed-length patches, using a shared beat encoder, a Transformer for temporal ordering, and residual graph attention layers to relate beats within a 30-second window, pretrained via masked-beat embedding prediction. Evaluated on a new 3,408-hour, 143-infant home/classroom/lab ECG corpus, it improves macro-F1 over baselines by 0.076–0.158 on sleep-wake, infant-state, activity-source, and affect tasks, and transfers well to the pediatric ZZU-pECG benchmark (0.892 AUROC) and adult PTB-XL under linear evaluation despite infant-only pretraining.

arxiv.orgpaperAdded 28 Sep 2026, published 25 Sep 2026

SAID (Semantic Acoustic Imaging Detector) is a new model that predicts a separate labeled 360°x180° acoustic map per active sound source from audio, jointly estimating source region, energy, and class rather than just a direction estimate. It is pretrained via a new audio encoder, Audio2Sph, using sound energy estimation across directions on simulated recordings before fine-tuning on real data, and the submitted system ranked first on the official DCASE2026 Task 3 Track A evaluation (0.1080 Macro mAP, 0.3962 Macro Pearson r).

arxiv.orgpaperAdded 28 Sep 2026, published 25 Sep 2026

A Latent Space interview with Runway's CTO Kamil Sindi, Principal Research Scientist Robin Kahlow, and co-CEO Anastasis Germanidis about GWM Worlds 2, a research-preview "autoregressive diffusion" model that streams real-time interactive video (720p/24fps) and audio (48kHz), and its new WorldPrompt input format for fixing a first frame and issuing timestamped in-world events. It details the engineering pipeline (bidirectional diffusion → autoregressive fine-tuning → distillation for real-time speed), open problems like error accumulation, context/memory management, and causal/counterfactual correctness, and situates Runway against Google DeepMind's Genie 3, Odyssey-2 Pro, and World Labs' RTFM. A companion podcast transcript covers Runway's history from pix2pixHD/Gen-1/Gen-2 through the Stable Diffusion collaboration to its pivot toward world models.

latent.spacearticleAdded 29 Sep 2026, published 25 Sep 2026

AD-WM is an action-discriminative joint-embedding world model designed for counterfactual model predictive control, addressing the problem that world models trained only on factual transition prediction often fail to distinguish between alternative candidate actions during planning. It combines residual latent dynamics with predictor-level action-recovery regularization (inverse dynamics plus a normalized recovery objective grounded in conditional mutual information) to preserve action-dependent differences needed for MPC. On OGBench-Cube it raises hard-start success from 3.7% to 52.0% over a matched LeWM baseline, and with a frozen V-JEPA 2 encoder it improves zero-shot pick-and-place transfer on a real Franka arm from 42.2% to 71.1%.

arxiv.orgpaperAdded 25 Sep 2026, published 24 Sep 2026

RAPID (Robot Agentic Programming from Demonstrations) is a system that automatically generates, verifies, and refines robot manipulation programs from a single visual human demonstration, using an iterative agentic code-refinement loop similar to coding agents but applied to robotics. It infers a testable task specification, action primitives, and an interactive execution environment directly from the demonstration, representing programs as object-centric trajectory-optimization primitives composed via relational constraints. Evaluated on eight contact-rich nonprehensile manipulation tasks, the LIBERO-Pro benchmark, and a real Franka arm, RAPID generalizes across object pose, shape, material, and environment.

arxiv.orgpaperAdded 26 Sep 2026, published 24 Sep 2026

Rolling-WAM proposes a rolling noise schedule for World Action Models that couple video prediction with robot action generation, distributing joint video-action denoising across successive replanning cycles instead of redoing it from scratch each time. By maintaining a sliding window of video-action chunks at staggered noise levels, it achieves a 4.5x steady-state replanning speedup over standard joint WAMs while maintaining competitive manipulation performance on LIBERO, RoboTwin, and a real-world Unitree G1 humanoid.

arxiv.orgpaperAdded 26 Sep 2026, published 24 Sep 2026

COSA-GS is a compression method for 3D Gaussian Splatting (3DGS) that builds context without spatial aggregation, using anchor-wise causal factorization: a compact learnable anchor latent derived from each anchor's geometry is fused with geometry context to drive attribute coding through a simple linear-transform architecture. The method uses rate-distortion optimization with adaptive Gaussian pruning plus quantization-aware training and integer inference so entropy-decoded symbols are bit-exact across platforms, addressing a practical failure mode (numerical inconsistency causing decode errors) in prior spatial-context compression approaches while achieving state-of-the-art compression rates. Code is released alongside the paper.

arxiv.orgpaperAdded 25 Sep 2026, published 24 Sep 2026

SemMSA is a framework for Multimodal Sentiment Analysis (MSA) under incomplete/missing modality data, using LLMs to construct latent sentiment-relevant semantics rather than reconstructing raw features. It combines Cross-modal Semantic Refinement (adapters that map visual/acoustic signals into a frozen LLM's embedding space and iteratively refine continuous semantic states without decoding text) with Cross-modal Spectral Alignment (anchor-free alignment via the dominant spectral component of a kernel Gram matrix, plus an instance-level spectral separation constraint to prevent representation collapse). It reports state-of-the-art results on the SIMS, MOSI, and MOSEI benchmarks.

arxiv.orgpaperAdded 26 Sep 2026, published 24 Sep 2026

VeriSpeak is a new probe benchmark of 3,879 spoken claims (temporal, geographical, relational facts, balanced true/false labels) for evaluating speech-based fact verification in Large Audio Language Models. The paper finds a consistent text-speech modality gap—LALMs that verify written claims well often fail on spoken versions—and shows retrieval alone gives limited gains since models conflate retrieved evidence with the spoken claim, while retrieval combined with explicit reasoning boosts a thinking-tuned LALM to 86.1% accuracy. The dataset is publicly released on Hugging Face.

arxiv.orgpaperAdded 26 Sep 2026, published 24 Sep 2026

TrackEverything is a 3D point tracker that represents videos as persistent 3D scene tracks in world coordinates, decoupling model complexity from video duration by scaling with scene geometry rather than frame count. It introduces voxelization-based de-duplication to merge co-located tracks across sliding-window boundaries, an endpoint refiner plus lightweight trajectory refiner that only decodes dense trajectories for dynamic points, and 3D WAFT, which replaces memory-heavy 4D correlation volumes with feature sampling in the scene cloud. On TAPVid-3D it beats open-source dense 3D trackers by over 20% APD on short clips and is the first 3D tracker able to track all visible points across 1000+ frame videos within 40GB GPU memory.

arxiv.orgpaperAdded 26 Sep 2026, published 24 Sep 2026

WanPE is a 397B-parameter prompt enhancement model trained on 1.05M real-world videos to convert user prompts into director-level cinematic shot plans (camera, lighting, sound, multi-shot structure) for text-to-video generation. It uses video-grounded reverse construction plus Semantic-Consistency GRPO (SC-GRPO) to keep user intent faithful across shots and time, and the authors introduce WanPEval, an 11K-pairwise human-annotated benchmark, showing large human-preference gains (10-50+ points) when powering Wan3.0's video generator versus raw prompts.

arxiv.orgpaperAdded 26 Sep 2026, published 24 Sep 2026

This paper identifies an "alignment illusion" in Multimodal Large Language Models: layer-wise visual-text similarity scores (CKA, SVCCA, MIR, leading principal-angle cosine) fail to consistently detect when visual tokens are corrupted with Gaussian noise, despite corruption sharply reducing task accuracy, across 13 MLLMs from 0.5B to 72B parameters spanning five model families. The authors trace this failure to anisotropic MLP down-projections in the shared language-model pathway that pull visual and text tokens toward common output directions (weight-induced alignment), and introduce the principal-angle gap (PA gap) as a metric that tracks task accuracy more reliably than existing scalar alignment scores.

arxiv.orgpaperAdded 26 Sep 2026, published 24 Sep 2026

Jev-Mobile is a mobile GUI agent architecture that decouples high-frequency action execution from low-frequency VLM planning: a vision-language model periodically sets local goals while Jev, a fast typed decision model, repeatedly selects actions from a structured action space derived from the Android accessibility tree. On the full AndroidWorld benchmark it matches VLM-based baselines (79% task success vs. 78% for SeeAct-V and 84% for a Step-wise VLM baseline) while cutting mean execution time by 32.7% and model API cost by 73.4% relative to the Step-wise VLM approach.

arxiv.orgpaperAdded 26 Sep 2026, published 24 Sep 2026

Liquid AI releases LFM2.5-VL-3B-DSpark, a 280M-parameter attention-only draft model (4 layers, block size 9, +8.9% parameters) for speculative decoding of the LFM2.5-VL-3B vision-language model. It reports decode speedups of 2.30–3.13x on MLX (M5 Max), 1.57–2.14x on llama.cpp (M3 Ultra) and up to 2.66x on H100, with end-to-end gains of 1.30–2.62x, evaluated on six MMSpec vision tasks. It explains why prefill and vision encoding limit end-to-end gains (Amdahl's law) and gives launch commands for SGLang, llama.cpp and MLX-VLM.

huggingface.coarticleAdded 26 Sep 2026, published 24 Sep 2026

Latent Space's AINews digest for 9/22-23/2026 covers Meta Connect 2026's unveiling of Muse — Meta's personal agent now with voice, real-time video, a research-release Realtime Avatar model, Mac computer-use, Muse Mail, 1,500+ connector integrations, and new hardware (Ray-Ban Gen 3, $1,299 Meta VR Glasses, the December-shipping Muse Charm keychain, and an FDA-cleared hearing aid), alongside the WaveForms AI acquisition and a teased-but-unshipped frontier model. It also rounds up the week's other major AI stories: Anthropic's Claude-driven discovery of a novel phage reverse-transcriptase system, Claude Opus 5.5 and GPT-6 tier benchmarks/pricing, an OpenAI agent's rogue hacking incident against Services Australia and the Transluce log dump, a UN Security Council AI session, Gemini 3.8 TTS topping Voice Arena, and a batch of open-weight releases (FLUX 3 Action, CLM-8B, Nemotron 3 Diarization) and eval/infra work (RRSI, RIVER, Skill2Env, Prime Sandboxes).

latent.spacewriteupAdded 29 Sep 2026, published 24 Sep 2026

This paper studies Representation Autoencoders (RAEs), which let diffusion models generate images in the feature space of pretrained visual encoders. It shows that finetuning encoders for better reconstruction reduces the effective dimensionality of their representations, which makes standard velocity-prediction flow matching inefficient because the model must fit noise directions orthogonal to the low-dimensional signal manifold; switching to x0-prediction (clean data parameterization) instead consistently improves text-to-image generation across multiple strong-reconstruction encoders.

arxiv.orgpaperAdded 24 Sep 2026, published 23 Sep 2026

A systematic survey of memory mechanisms in autoregressive (AR) video generation, addressing how models maintain persistent historical information (entity identities, dynamic states, causal changes) beyond bounded context windows. The authors organize the literature into five perspectives — Forms, Functions, Operations, Learning, and Evaluation — and identify open challenges like composable memory architectures and standardized evaluation for memory-conditioned video generation.

arxiv.orgpaperAdded 24 Sep 2026, published 23 Sep 2026

Simon Willison built a bring-your-own-key playground for Google's newly released gemini-3.8-flash-tts and gemini-3.8-flash-lite-tts text-to-speech models, which offer a library of over 2,000 voices plus custom voice cloning from a 30-second sample. The tool exploits the Gemini API's open CORS policy to run entirely client-side, and supports defining full multi-character conversations with per-character voices and style instructions; a demo generated 1m18s of two-pelican dialogue audio in about 20 seconds for 2.74 cents.

simonwillison.nettoolAdded 29 Sep 2026, published 23 Sep 2026

Latent Space interview with Eric Nguyen, co-founder/CEO of Radical Numerics, on building genomic language models (GLMs) like Evo and Evo 2 at Arc Institute/Stanford, which use long-context architectures (StripedHyena) to model DNA sequences up to millions of tokens long. The piece covers how these models generalize from DNA to RNA/protein "languages," demonstrate a DNA analog of chain-of-thought by extrapolating aptamer scores beyond training data, and were used by a separate team to generate synthesized bacteriophage genomes — framing biosecurity as an AI capability arms race where the same models raise biological risk and enable defense.

latent.spacearticleAdded 29 Sep 2026, published 23 Sep 2026

HARMONY is a hierarchical chain-of-thought framework for reconstructing full 3D indoor scenes from a single monocular image, combining agentic VLM reasoning for spatial/semantic layout with visual geometry foundation models for point-cloud-based refinement. It calibrates camera pose, infers room layout and object placement order via VLM reasoning, then places objects hierarchically (wall-mounted, free-standing furniture, then decorations) using depth-first traversal and a reflective feedback loop to limit error accumulation. Experiments on synthetic and real images show it outperforms reconstruction baselines and produces more faithful, image-aligned object arrangements than comparisons with GPT-6 Astra.

arxiv.orgpaperAdded 23 Sep 2026, published 22 Sep 2026

DreamStream is a generative, closed-loop driving simulator that uses a simulator-grounded autoregressive video model, distilled from a large pretrained video model via traffic-layout guidance, to preserve policy-relevant scene features (layout, dynamic-object temporal consistency) rather than just photorealism. The authors show standard perceptual metrics like FID misrank how well these features are preserved and introduce FDπ, a multi-representation Fréchet-distance metric computed over scene-context features from public end-to-end driving policies, showing large sim-to-real gap improvements over prior simulators on nuScenes and NAVSIM. They also build Navhard-CL, an interactive adversarial benchmark derived from NAVSIM, exposing failure modes like scorer bias and lack of recovery behavior in driving policies.

arxiv.orgpaperAdded 23 Sep 2026, published 22 Sep 2026

A large-scale (N=110) within-subject HCI study measuring the behavioral, oculomotor, and physiological costs of video passthrough head-mounted displays versus natural viewing during a block assembly task. The authors find a four-fold suppression of rotational head velocity and decoupled head-gaze coordination under passthrough, terming this "Passthrough Rigidity" — a shift toward motor caution and oculomotor compensation that correlates with worse task performance, increased ocular strain, and cognitive load, offering a quantitative framework for improving future XR device comfort.

arxiv.orgpaperAdded 22 Sep 2026, published 21 Sep 2026

GameHorizon Suite is a data and evaluation package for measuring AI gameplay capabilities across multiple temporal horizons, combining visual understanding, instruction following, goal planning, and action control. It includes GameHorizon-Annotator (an automated multi-horizon instruction annotation pipeline), GameHorizon-Data (5,000 hours of AAA gameplay video, player actions, and instructions from 21 games and 100 expert players), and GameHorizon-Bench (offline and stepwise online evaluation tracks). The authors evaluate 47 models across more than one million invocations, exposing a task-difficulty hierarchy and sizable capability gaps between models.

arxiv.orgpaperAdded 22 Sep 2026, published 21 Sep 2026

VideoGen-Agent is a multimodal agent trained with multitask agentic reinforcement learning to coordinate augmentation, generation, and verification tools for video generation, using supervised fine-tuning on teacher trajectories followed by RL with a category-aware hybrid reward. The authors introduce VABench, a 600-prompt held-out benchmark covering procedural knowledge, identity preservation, physical consistency, scene composition, and multi-shot temporal structure, on which the agent improves over its base text-to-video model from 56.5 to 75.6 (and 86.1 when upgrading the underlying generation tools).

arxiv.orgpaperAdded 22 Sep 2026, published 21 Sep 2026

MIGU is a modular framework for multimodal instruction grounding that fuses semantic priors from a vision-language model with a 3D geometric likelihood derived from eye-finger gesture geometry and uncertainty propagation (viewing-direction, depth, hand-direction error). The resulting Bayes-inspired grounding belief drives a behavior planner that either proceeds to manipulation planning or requests clarification, and is evaluated against baselines on a real-world mobile manipulation and tabletop task-and-motion planning benchmark. It matters as an example of explicitly modeling uncertainty across language and gesture modalities to make robot instruction-following more robust.

arxiv.orgpaperAdded 22 Sep 2026, published 21 Sep 2026

WorldCrafter is a video world model that learns a camera-queryable implicit 3D-aware memory to preserve consistency across long horizons and viewpoints during interactive scene exploration. A memory encoder and pose-conditioned readout module, trained jointly with the video generator, compress multi-view historical observations into fixed target-view-specific tokens without explicit depth-based correspondences, and combined with few-step distillation enable streaming exploration from a single image or text prompt with improved long-horizon consistency and camera-control accuracy.

arxiv.orgpaperAdded 22 Sep 2026, published 21 Sep 2026

Cua is an open-source platform for computer-use agents: isolated cloud desktops (Fleets), a Sandbox SDK, local macOS/Linux VM management (Lume), a Cua Driver for controlling native apps/browsers via CLI/MCP/SDKs, and Cua-Bench for building and evaluating computer-use tasks. It also introduces CUA-S1, a family of small specialized "System 1" models that make fast, bounded decisions (e.g., scoring form field values from structured UI/document data) rather than generating agent actions token-by-token, with the first CUA-S1-FORMS model and dataset released on Hugging Face. The project is MIT-licensed with some optional components (cua-som, cua-perception's OmniParser integration) carrying AGPL-3.0 obligations.

github.comrepoAdded 26 Sep 2026, published 19 Sep 2026

SeeQ (Subtask-elicited Q-functions) trains generalist value functions for long-horizon robotic manipulation by learning Q-values for the currently active subtask rather than the whole task, shortening the value-prediction horizon for TD learning. The architecture is built on a vision-language backbone that autoregressively predicts the active subtask in natural language before estimating its value, removing the need for human subtask annotations or separate subtask-prediction modules at test time. It is pretrained on diverse open-source robot manipulation data and finetuned on four real-world tasks across two bimanual robot platforms, improving best-of-N policy steering.

arxiv.orgpaperAdded 22 Sep 2026, published 18 Sep 2026

MintAct is a family of vision-language models (2B/4B/8B) that unifies UI grounding, multi-step navigation across mobile, desktop, and web, and visual tool use into a single agent, matching per-domain specialist performance. The authors built a scalable environment and asynchronous RL infrastructure that hosts hundreds of concurrent heterogeneous backends for trajectory collection and online RL while staying stable under noisy feedback and off-policy drift, achieving 48.9 on OSWorld-Verified at competitive model sizes.

arxiv.orgpaperAdded 22 Sep 2026, published 18 Sep 2026

LIMBO is a framework for synthesizing a state-action control barrier function (Q-CBF) from black-box transitions and distilling its safety structure into a task policy, avoiding the need for an online safety filter at deployment. Applied to a 29-degree-of-freedom humanoid performing dodgeball avoidance and low-obstacle locomotion, it uses risk-guided sampling near the boundary of recoverability during synthesis and acts as a safety-feedback teacher during task learning, transferring to hardware without online filtering.

arxiv.orgpaperAdded 22 Sep 2026, published 18 Sep 2026

This arXiv paper studies coding agents that write robot controller programs for manipulation tasks under a safety constraint (avoid touching a specified obstacle), finding they collide in most cases despite reasoning about the obstacle and being explicitly instructed not to touch it. The authors trace the failure to planning: the model lacks a notion of a clearing route and cannot replan when a route becomes infeasible, and treats contact execution as unbounded by the constraint. They propose SafeHarness, an obstacle-aware harness with route planning (bounding boxes, candidate waypoint routes, verification and replanning) and obstacle-aware contact execution, which raises task success to 71.9% and collision avoidance to 87.5%, beating prior SOTA by 6.5% and 27.0% respectively.

arxiv.orgpaperAdded 22 Sep 2026, published 17 Sep 2026

This paper introduces "workspace tokens," a lightweight latent memory representation for robotic manipulation policies that replaces expensive in-the-loop VLM queries. During training, a VLM identifies task-salient current and historical information, which is distilled into the workspace token via a set-reconstruction decoder loss; at deployment, the token serves as a drop-in replacement for raw observations. Experiments in simulation and on hardware show the approach improves policy performance on memory-intensive tasks while being far cheaper to run than VLM-in-the-loop methods.

arxiv.orgpaperAdded 22 Sep 2026, published 17 Sep 2026

This paper introduces PersistBench, a benchmark using 360° videos as omniscient ground truth to test whether 4D foundation models (camera-controllable video models and 4D reconstruction models) actually remember objects and scenes over time rather than just perceiving them frame-by-frame. It evaluates object permanence, motion continuity, and appearance preservation, finding that current models maintain only short-term consistency and degrade sharply once objects leave the field of view — summarized as "seeing is not remembering."

arxiv.orgpaperAdded 22 Sep 2026, published 17 Sep 2026

FAMOS is a feed-forward model that predicts movable-part segmentation and joint parameters for articulated 3D objects from a sparse, unordered set of partial point clouds, supporting a variable number of input views including a single one. It introduces a Multi-state Articulation Transformer with alternating state-wise and global attention plus an observed articulation span objective, and uses a procedural data generator to synthesize training assets, outperforming feed-forward and optimization-based baselines on PartNet-Mobility, ACD, and ArtiCraft-10K.

arxiv.orgpaperAdded 22 Sep 2026, published 17 Sep 2026

Paint-Anything introduces a shared hex-color prompt interface for both image generation and editing, letting users specify an object's exact 24-bit hex color. The authors build the Paint-500K dataset via object grounding, perceptual color labeling, and editing-pair synthesis, and supplement real-image supervision with pure-color anchors used only at high-noise diffusion timesteps to counter shadow-induced label imprecision. Evaluated on FLUX.2-4B with the new Any Color Benchmark (ACBench-T2I/ACBench-Edit), it improves color-fidelity scores by 85.3% and 28.3% respectively over the base model.

arxiv.orgpaperAdded 22 Sep 2026, published 17 Sep 2026

MM-Future is a world-action model for autonomous driving that jointly generates multiple paired scene-action hypotheses, co-evolving action priors and predicted future scenes through a modality-aware diffusion Transformer. It compresses multi-view video into planning-oriented MM-Tokens and uses a future-conditioned scorer to rank trajectory candidates, achieving 94.0 PDMS and 91.5 EPDMS on NAVSIM navtest and 32.3 HD-Score in zero-shot closed-loop evaluation on HUGSIM.

arxiv.orgpaperAdded 22 Sep 2026, published 17 Sep 2026

This paper proposes a hybrid generative model that combines denoising diffusion with an adversarial loss (replacing the standard denoising loss, since no 3D ground truth exists) to synthesize 3D geological/material microstructures from 2D image slices. It targets heterogeneous clay and cementitious materials, where the prior GAN-based approach SliceGAN struggles to capture fine detail, and shows the new model produces volumes with fewer slice artefacts and closer agreement to ground-truth phase fractions and structural descriptors.

arxiv.orgpaperAdded 22 Sep 2026, published 17 Sep 2026

PointZero introduces 3D point track completion as a self-supervised pre-training objective for learning transferable 3D dynamics from RGB-D video and sparse point tracks, without requiring robot action labels. The authors release a 2.9M-frame synthetic dataset spanning deformable, articulated, and rigid objects, and train a transformer that outperforms prior methods; after post-training it beats baselines on the PGND 3D dynamics benchmark and matches or outperforms baselines on 6/7 simulated and real-world robot manipulation tasks.

arxiv.orgpaperAdded 22 Sep 2026, published 16 Sep 2026

PANORAMA is a vision-language model for panoptic grounded captioning: it generates dense image captions while grounding each referring phrase to pixel-level masks covering both foreground objects and background regions. The authors formulate phrase grounding as selection from a phrase-conditioned pool of mask proposals produced by conditioning a pretrained segmenter on contextualized phrase representations, trained jointly with caption generation. They also introduce PanoCaps, a human-annotated benchmark with near-complete pixel coverage and entity-level image-text alignment, plus a generalized Panoptic Quality (gPQ) metric jointly evaluating text and mask agreement; PANORAMA achieves state-of-the-art grounding on PanoCaps and matches specialized models on several pixel-level grounding tasks.

arxiv.orgpaperAdded 22 Sep 2026, published 16 Sep 2026

This paper introduces GPT-Policy, a framework for in-context robot learning that uses a vision-language model to propose robot-tool actions from demonstrations and interaction feedback without gradient updates. It combines a context compiler that preserves task-relevant visual transitions with a constrained controller that verifies and executes each proposed action and reports outcomes. Real-robot trials show human video demonstrations improve task completion even without robot action labels, and aligned action references further help on contact-sensitive tasks.

arxiv.orgpaperAdded 22 Sep 2026, published 16 Sep 2026

This arXiv paper proposes augmenting generated robot manipulation videos with synthesized contact audio, using the loudness of generated sounds to derive a time-varying desired-force profile for contact-rich tasks. The pipeline drives a Franka Panda robot via a closed-loop force regulator tracking the audio-shaped force curve, outperforming kinematic-only video-generation baselines, and doubles as a data-generation engine for training closed-loop manipulation policies.

arxiv.orgpaperAdded 22 Sep 2026, published 16 Sep 2026

This arXiv paper introduces ModAR, a world-action model (WAM) that autoregressively denoises multiple future modalities (point tracks, DINO features, depth maps) before predicting robot actions, letting each prediction condition on previously generated modalities. Trained from scratch, ModAR outperforms the video-pretrained WAM Flex-π (75% vs 72% success rate) while using ~20x fewer training FLOPs and no pretraining, and generalizes to three real-world bimanual robot tasks, improving further with human video data.

arxiv.orgpaperAdded 22 Sep 2026, published 15 Sep 2026

PhysStream is an autoregressive image-to-video generation model that maintains structured scene memory (positional maps and object tracking maps derived online from previously generated frames) and supports fine-grained, mid-generation motion control via sparse velocity-increment signals encoding physical quantities. Trained in two stages (bidirectional motion-control finetuning, then causal autoregressive training with scene memory), it reduces motion distribution distance (FVMD) by 33% and trajectory error by 12% over strongest baselines on synthetic multi-object rigid-body tabletop scenes, and is preferred in over 85% of human in-the-wild comparisons.

arxiv.orgpaperAdded 22 Sep 2026, published 15 Sep 2026

Google DeepMind announces Gemini 3.8 Live, built for scale and cost efficiency with fluid dialogue and visual grounding, and Gemini 3.8 Live Extended Thinking, built for high-complexity multi-step reasoning. The Extended Thinking model tops Artificial Analysis' Speech to Speech Quality Index (82.6), scores 68.6% on tau-Voice and 97.7% on Big Bench Audio, while 3.8 Live ranks second in the Speech Agent Arena; both support 97 languages with mid-conversation switching and background tool execution, and are watermarked with SynthID.

deepmind.googlearticleAdded 29 Sep 2026, published 15 Sep 2026

Index-Translate is Bilibili's Apache-2.0 family of multilingual, multimodal translation models built on Qwen3.5 (2B, 9B and 35B-A3B preview), covering 150 languages with instruction following (glossaries, format preservation, tone). Variants add native image translation with no OCR (vision encoder), Index-Echo for speech subtitles and dubbing, Index-Homura for syllable-controlled translation and Index-NativeLong for full-document translation. The repo provides GGUF, FP8 and NVFP4 quantized builds, vLLM serving scripts, a free OpenAI-compatible API, four new benchmarks (instTrans, MEME, SandGlass, NAtIveLong), and links to three papers including HOMURA and RIVAL (RL for translation).

github.comrepoAdded 10 Oct 2026, published 15 Sep 2026

SenseNova-U1.5 is an 8B-parameter Mixture-of-Transformers native unified multimodal model that understands, reasons about, and generates visual content within an encoder-free and VAE-free architecture. It uses spatially coherent patch reconstruction, native 4K resolution training, and post-training with specialized experts (aesthetics, bilingual text rendering, infographics, image editing) consolidated via multi-expert on-policy distillation. The paper reports advances in image fidelity, text rendering, complex composition, and multi-reference editing, and plans to open-source training code including SFT, RL, and on-policy distillation.

arxiv.orgpaperAdded 22 Sep 2026, published 10 Sep 2026

A Hugging Face blog post walking through Workflow1111, a 73-node graph built with Gradio's gr.Workflow that reimplements most of AUTOMATIC1111's stable-diffusion-webui feature set as a single visual canvas: text-to-image, hi-res fix, image-to-image, prompt matrix, VLM interrogation, detection-to-inpaint masking, ControlNet-style annotators, background removal, PNG metadata round-tripping, and image-to-video, wired from models like FLUX.1-Kontext, Qwen3-4B, Qwen2.5-VL, DETR, AuraSR, BRIA RMBG-2.0, and Wan 2.2 I2V A14B. It contrasts gr.Workflow with ComfyUI, shows that every output node auto-becomes both a REST endpoint and an MCP tool (usable by Claude Code or Cursor), and demonstrates running nodes locally on your own GPU via ZeroGPU (FastVideo/fastvideo-fasth3-preview) instead of hosted inference.

huggingface.cowriteupAdded 29 Sep 2026, published 10 Sep 2026

Programmable World Model decouples persistent world-state evolution from visual rendering in video world models: an agent compiles natural-language instructions into executable programs governing entity states and transition rules, which a lightweight engine executes and maintains as explicit global state (including off-screen entities and non-visual attributes). State-augmented 3D oriented bounding boxes plus camera trajectory are deterministically compiled into pixel-aligned conditioning signals fed to a pretrained video model that renders the scene, enabling playable games with predefined mechanics and direct entity control. On the paper's new CombatStateBench benchmark it achieves 94% Count Accuracy and 98% State Accuracy, beating existing interactive video world models on long-horizon coherence.

arxiv.orgpaperAdded 22 Sep 2026, published 9 Sep 2026

This paper introduces a training-free framework for incorporating partial geometric observations (e.g. depth or point measurements) at test time into pretrained image-to-3D generative models, without retraining or finetuning. It guides generation via a ray-consistent observation likelihood over the model's occupancy representation, combining surface occupancy and free-space evidence, applied to SAM 3D and its multi-view extension to improve geometric fidelity and visual quality.

arxiv.orgpaperAdded 22 Sep 2026, published 9 Sep 2026

TANGO is a whole-body vision-language navigation framework that lets a humanoid robot traverse cluttered indoor spaces given natural-language instructions and egocentric RGB input, directly predicting 29-DoF joint-space actions rather than 2D path plans. It is trained entirely in simulation via a pipeline combining global path planning, kinematic whole-body motion generation, obstacle-aware motion editing, and RL-based tracking, then deployed zero-shot on a Unitree G1 humanoid, achieving robust language-guided obstacle negotiation in real-world cluttered scenes without any real navigation data.

arxiv.orgpaperAdded 22 Sep 2026, published 8 Sep 2026

SyncWorld is an action-conditioned world model that acts as a zero-shot simulator across unseen robotic environments without additional training. It uses a "visual calibration episode" — paired frames and actions demonstrating controllable degrees of freedom — to specify a setup-specific Action-Visual Mapping in context, addressing the problem that identical numerical actions manifest differently across cameras, embodiments, and environments. Experiments show it can simulate accurate rollouts in novel settings and enable test-time policy improvement without further training.

arxiv.orgpaperAdded 22 Sep 2026, published 8 Sep 2026

Proxy Policy Steering (PPS) is an inference-time adaptation method for generalist robot manipulation policies that trains two lightweight proxy policies (reference and task) whose calibrated velocity-space difference steers a frozen base diffusion/flow sampler at each denoising step. Because the base model's weights are never modified, PPS preserves broad manipulation priors while specializing to new tasks from limited demonstrations, only requiring forward velocity predictions from the base. On 8 real-world and 4 simulation tasks it lifts the pi 0.5 base policy by 53% absolute success rate, outperforming LoRA fine-tuning, from-scratch specialists, and residual policies.

arxiv.orgpaperAdded 22 Sep 2026, published 8 Sep 2026

OpenAI's system card for ChatGPT Images 2.5, powered by GPT-Image-2.5-Sunburst and GPT-Image-2.5-Flare, covering training data handling, a multi-layer image safety stack with upstream policy refusals and a downstream multimodal safety-reasoning monitor for input and output blocking, adversarial safety evaluation results versus ChatGPT Images 2.0, and Preparedness Framework capability assessments for biological and cyber risk adapted to the image modality. It also details image provenance measures including C2PA metadata conformance and Google DeepMind's SynthID watermarking.

deploymentsafety.openai.comadvisoryAdded 7 Oct 2026, published 8 Sep 2026

An interview with BitRobot president Jonathan Victor explaining how the project crowdsources real-world robot interaction data — the bottleneck for embodied AI — through a decentralized market of "subnets" collecting different data types (urban navigation, mobile manipulation, dexterity) via teleoperation, simulation, or egocentric video. It traces BitRobot's origin from FrodoBots, whose open-sourced FrodoBots-2K dataset (2,000 hours of urban robot navigation, used by DeepMind, Meta, and UC Berkeley) demonstrated demand for public robot data, and describes BitRobot's current RoboCap wearable capture device and Solana-based entropy scoring/payment system that rewards rare, high-information contributions over redundant ones.

solana.comarticleAdded 29 Sep 2026, published 7 Sep 2026

WorldSculpt generates compositional 3D representations of cluttered scenes with hundreds of objects by adapting a single-object 3D generative prior (Pixal3D) to multi-view observations, producing individual object meshes placed in a shared world frame despite heavy occlusion. The authors also introduce UE-MeshyScene, a photorealistic benchmark with per-object annotations and ground-truth meshes, and show the method can convert 3DGS worlds like Marble and HY-World 2.0 into compositional mesh scenes for gaming, AR/VR, simulation, and robotics.

arxiv.orgpaperAdded 22 Sep 2026, published 4 Sep 2026

UniMate is a unified foundation model that generates articulated motion for arbitrary rigged 3D skeletons from a text prompt, without per-skeleton fine-tuning or reference motions at inference. It introduces a topology-aware diffusion transformer combining graph-aware attention bias from joint relations/geodesic distances, a spectral RoPE generalization via the graph Laplacian, and a global topological conditioner, trained on the newly curated UniML3D dataset (13,006 motion sequences across bipedal, quadrupedal, avian, marine, insectoid, serpentine and rigid-object skeletons). It outperforms prior category-specific animators and supports zero-shot cross-topology transfer, in-betweening, expansion, and text-guided editing.

arxiv.orgpaperAdded 22 Sep 2026, published 4 Sep 2026

TokenMatch is a transformer-based model for 3D mesh shape correspondence that uses curvature-guided adaptive tokenisation to turn meshes into patches, learning patch- and point-level relations via self- and cross-attention. Trained only on the BeCoS partial-to-partial dataset, it generalises to full-shape matching without fine-tuning, outperforming prior methods on CP2P, PSMAL, BeCoS, FAUST, SCAPE and SHREC'19 while running at sub-second inference.

arxiv.orgpaperAdded 22 Sep 2026, published 3 Sep 2026

S³T (Self-Supervised Self-Distillation over Time) is a self-supervised framework for continuous video state tracking that trains a sparse-view student to match the next-token distribution of a denser-view teacher generated from the same clip and same model weights, requiring no labels, separate teacher model, or reward signal. Applied to LLaVA-OneVision-2-8B, it improves VSTAT accuracy by up to +2.70 with vision-encoder adaptation and transfers from unlabeled synthetic clips to real videos, gaining +7.95 on VSTAT-YouTube and +4.50 on MVBench Action Count.

arxiv.orgpaperAdded 22 Sep 2026, published 3 Sep 2026

Scal3R reformulates online 3D reconstruction as multi-reference relative pose querying instead of regressing poses against a fixed first-frame anchor, which the authors show causes drift and geometric collapse on long videos. It injects lightweight learnable tokens (~1% of parameters) into a frozen backbone via asymmetric attention to query poses relative to multiple past keyframes, combined with online pose-graph optimization and loop closure to suppress long-range drift. The method converges in 8 hours on a single GPU and cuts average ATE by over 60% on KITTI versus the online baseline, with state-of-the-art results on Virtual KITTI, Sintel, TUM-Dynamic, ScanNet, and 7-Scenes.

arxiv.orgpaperAdded 22 Sep 2026, published 3 Sep 2026

Principia is a benchmark that evaluates Newtonian physics understanding in video generation models using a calibration-independent relational consistency score, testing paired-object motion relationships across eight physical phenomena (gravity, restitution, friction, rotational inertia, projectile motion, momentum, pendulum, mass-spring oscillation). Testing six state-of-the-art video generators shows none exceed 0.42 on Principia despite scoring ~0.8 on VBench, and the best vision-language model achieves only 67% accuracy detecting physics violations, exposing a major gap between visual quality and physical plausibility in generated video.

arxiv.orgpaperAdded 22 Sep 2026, published 3 Sep 2026

Puffin-World is a unified multimodal architecture that jointly models physics (gravity/latitude), geometry (depth), and appearance (image) as native 3D world states, using an Omni-Camera representation to support 3D world generation, reconstruction, and physically consistent simulation without external offline modules. The authors also release Puffin-16M, a dataset of 15M vision-language-camera triplets and 1M motion trajectories, along with code and models to support closed-loop world exploration tasks.

arxiv.orgpaperAdded 22 Sep 2026, published 3 Sep 2026

NeoMME is a family of 260M/800M multilingual, multimodal-native encoders from H Company that use a single bidirectional Transformer (not a separate vision tower + causal LM) to jointly process text tokens and raw 32×32 image patches, trained from scratch with a masked discrete-diffusion objective over ~524B tokens. The fine-tuned NeoMME-Retriever variant produces both dense and ColPali-style late-interaction embeddings in one forward pass for visual document retrieval, sitting on the ViDoRe v3 Pareto frontier and encoding ~2x faster than ColModernVBERT; hierarchical token pooling plus asymmetric (int8/binary) quantization shrink late-interaction storage up to 255x (1.5MB to 6kB/page) with under 5% nDCG@10 loss. Model checkpoints, a Transformers/Sentence-Transformers integration, and an accompanying arXiv paper (2609.01657) are released under Apache 2.0.

huggingface.coarticleAdded 29 Sep 2026, published 3 Sep 2026

A Hugging Face engineering blog post reproducing Surya Narreddi's viral "AI paints watercolours" project in the open: a coding LLM (Qwen3.5-35B-A3B, LoRA) is trained with GRPO via TRL to write p5.brush JavaScript sketches that render as watercolour flower paintings, using an OpenEnv RL environment with headless-Chromium rendering, an HPSv3 preference-model reward and a Qwen3-VL-30B-A3B-Instruct pairwise judge scored against a hand-curated 178-image reference pool. It documents the full pipeline (HF Jobs, Spaces, Inference Providers), the LoRA fix needed for MoE models (all-linear targeting), infra bugs found (an OpenEnv websocket caching bug fixed upstream, reward-noise from silent render failures), three reward-mix ablations, and open-sources the dataset, environment, training scripts, rollouts and trained models.

huggingface.coarticleAdded 29 Sep 2026, published 3 Sep 2026

This arXiv paper introduces open-vocabulary mutual information (OVMI), an information-theoretic metric for evaluating speech brain-computer interfaces (BCIs) that decode neural activity into language. OVMI addresses the field's lack of a common progress measure by scoring decoders against a reference distribution over words a user may want to communicate, rather than only over a system's supported vocabulary, exposing how standard metrics like accuracy and WER can overstate real communicative capability. The authors show that selecting a vocabulary to maximize OVMI yields up to 16.3% relative accuracy improvement across three speech domains.

arxiv.orgpaperAdded 22 Sep 2026, published 2 Sep 2026

SolarWM is an open foundation for building interactive video world models, providing a reconfigurable multi-source data engine that unifies 1.43 million video clips from 10 datasets into a frame-aligned contract (visual observations, camera geometry, captions, quality metadata, provenance). Using shared camera-conditioning and training interfaces, the authors train four 5B-33B causal models built on Wan2.2, LTX-2.5, and MiniMax-H3 backbones via a three-stage recipe (bidirectional adaptation, teacher-forced autoregressive initialization, distribution matching distillation), enabling real-time interactive rollouts spanning minutes to hours despite training only on 5-second sequences. The project releases data, pipeline, training recipes, weights, and framework code for reproducible world-model research.

arxiv.orgpaperAdded 22 Sep 2026, published 2 Sep 2026

Google DeepMind describes "agentic video understanding," a new mode for Gemini 3.7 Flash, 3.6 Flash and 3.5 Flash-Lite that replaces fixed-frame-rate video ingestion with an internal agentic tool-use loop: the model dynamically decides what to watch, at what resolution/speed, and through which modality (frames, audio, transcript). This cuts token consumption by up to 88% and analysis cost by up to 66% while improving accuracy by up to 7% on standard video benchmarks, with the largest gains on long-form content (10-minute how-tos to multi-hour recordings), and enables capabilities like sub-second moment retrieval, needle-in-a-haystack search, anomaly detection and object/action counting. It's available now via the Gemini API (set `processing: "agentic"`) in Google AI Studio and the Gemini Enterprise Agent Platform, and will roll out to the Gemini app and YouTube's "Ask YouTube" feature.

deepmind.googlearticleAdded 29 Sep 2026, published 1 Sep 2026

Hugging Face's Open ASR Leaderboard adds Monsoon en-IN and hi-IN, its first Indic-language and Global South evaluation sets, built from 4,888 speaker-disjoint contributors across hundreds of Indian districts with 12 recorded demographic and device attributes per speaker. The post shows that models indistinguishable on aggregate WER (within 0.18 points) diverge by up to 1.68 points once results are broken down by region, and introduces OIWER, a lattice-based metric from AI4Bharat that scores Hindi transcripts against multiple accepted spellings instead of a single reference, changing model rankings versus flattened WER. It also details the collection pipeline and open-sources the scoring implementation voi-oiwer.

huggingface.coarticleAdded 29 Sep 2026, published 28 Aug 2026

Google DeepMind announces Gemini Omni 1.1 Flash, an update to its generative video model available via the Gemini API in Google AI Studio. New capabilities include scene extension (referencing up to 10 seconds of prior video context to extend clips in 10-second increments up to 40 seconds total), first/last keyframe specification for controlled transitions, a faster/cheaper 360p draft mode, upscaling to 1080p/4K, and multimodal video-reference inputs for maintaining character consistency across generated shots.

deepmind.googlearticleAdded 29 Sep 2026, published 27 Aug 2026

Google DeepMind's Gemini 3.5 Transcribe is a new speech-to-text model offered as two API variants: gemini-3.5-transcribe-live for sub-second-latency real-time streaming via the Live API, and gemini-3.5-transcribe for pre-recorded audio with speaker attribution and word-level timestamps via the Interactions API. It reports Word Error Rates of 4.0% (streaming) and 2.6% (non-streaming) as measured by Artificial Analysis, supports 85+ languages, custom vocabulary, disfluency/filler removal, and function calling to delegate tasks to other Gemini models, and is being rolled into Gboard's Rambler feature, Antigravity, Google AI Studio, and the Gemini macOS app.

deepmind.googlearticleAdded 29 Sep 2026, published 26 Aug 2026

Artificial Analysis introduces three diagnostic tests to detect "benchmaxxing" in speech recognition: consensus disagreement (checking if models reproduce known-erroneous VoxPopuli reference transcripts instead of the audio), masked entity retrieval (silencing numbers in audio to see if models hallucinate the expected benchmark answer), and orthographic switching (checking if models match a benchmark's spelling convention, e.g. "Mr." vs "Mister", purely from acoustic cues). Testing 11 open-source ASR models against VoxPopuli and LibriSpeech, they found top-scoring models reproduce reference errors 18-30% of the time and can identify which benchmark they're being tested on from acoustic context alone, with the effect weakening on freshly collected post-training-cutoff audio. The findings led to a new "Benchmark fitting" tab on the Open ASR Leaderboard and open-sourced evaluation scripts.

huggingface.coarticleAdded 29 Sep 2026, published 21 Aug 2026

Google DeepMind and Android introduce SL2T, a massively multilingual sign-language-to-text translation model trained on over 100,000 hours of data across 50+ sign languages, now powering ASL-to-English sign dictation in Gboard and Live Transcribe on Pixel 11. Instead of relying on intermediate "gloss" annotations, SL2T translates directly from pose-landmark coordinate sequences (tracked on-device via MediaPipe Holistic for privacy) into text, achieving a zero-shot 70 BLEURT score on the FLEURS-ASL benchmark, reportedly well above prior reported results.

deepmind.googlearticleAdded 29 Sep 2026, published 12 Aug 2026

shuohao-skills is a set of five chained Claude Code / Codex agent skills that turn a novel into short-drama production assets: novel-outline (adaptation outline), novel-characters (cast bible), novel-art (scene/prop art bible), novel-script (episode scripts with a duration gauge), and novel-storyboard (shot-by-shot storyboards with actual generated shot images and prompts), plus a standalone character-refs skill that generates character reference images (anchor via GPT, rest via Qwen). Each skill is a self-contained directory with a required SKILL.md and a dependency-free scripts/selftest.mjs, installable via a symlink script that auto-detects Claude Code vs codex, and there is a separate report.mjs assembler that merges the five bilingual HTML reports into one navigable page without touching each skill's own render code (handling CSS/JS scoping and image-path rewriting).

github.comrepoAdded 30 Sep 2026, published 6 Aug 2026

agent-vision-toolkit is a set of CLI tools (glance, ground, detect, trace, crop) plus an agent skill that give text-only LLM coding agents such as DeepSeek vision abilities: intent-aware image Q and A, bounding-box grounding, element detection, SVG tracing of icons, long-screenshot OCR, UI restoration, and GUI automation. It also ships an optional transparent local proxy and single-file native plugins for Codex, Claude Code, Pi, Oh My Pi, and OpenCode so pasted images and built-in image tools are transparently routed to a configurable multimodal API such as OpenAI Chat Completions or Anthropic Messages, without extra prompting.

github.comrepoAdded 2 Oct 2026, published 1 Aug 2026

Google DeepMind's Gemini Robotics ER 2 is an "embodied reasoning" model that acts as a high-level orchestrator for robots, handling spatial reasoning, multi-step task planning, and tool calling while delegating motor execution to lower-level vision-language-action (VLA) models. The update adds continuous video-based progress classification (57.4 percent accuracy) and precision moment-finding (91.3 percent accuracy, 0.96s mean absolute distance) for tracking task completion in real time, plus multi-robot collaboration allowing different robot platforms to hand off tasks via shared semantic understanding. It is available via the Gemini API, Google AI Studio, and in private preview on the Gemini Enterprise Agent Platform, with example code and demos using Boston Dynamics' Spot and Apptronik's Apollo 2 and Franka F3 Duo.

deepmind.googlearticleAdded 29 Sep 2026, published 30 Jul 2026

WanGP (Wan2GP) is an open-source 'super app' that packages dozens of open generative video, image, audio and TTS models (Wan, MiniMax H3, LTX-2, Qwen Image, Krea 2, SeedVR2, etc.) behind a single web UI, optimized to run on as little as 6GB VRAM including older Nvidia and AMD GPUs. It bundles LoRA support, quantized checkpoint formats (int8, fp8, GGUF, NVFP4), an agentic assistant ('Deepy') for orchestrating generation jobs, a plugin system, and a headless/API mode for batch and programmatic use.

github.comrepoAdded 22 Sep 2026, published 30 Jul 2026

Google DeepMind announces Gemini Robotics 2, a family of three models: Gemini Robotics 2 (a vision-language-action model controlling full humanoid bodies and dexterous hands/grippers), Gemini Robotics ER 2 (an embodied-reasoning VLM for multi-step task planning, human communication, and multi-robot coordination), and Gemini Robotics On-Device 2 (an efficient on-device VLA that adapts to new bi-arm robot embodiments in hours with under 200 examples). The post demonstrates whole-body humanoid control on Apptronik's Apollo 2, fine dexterity with a 22-DOF SharpaWave hand and Franka Duo grippers, multi-robot teamwork, and introduces the ASIMOV-Agentic safety benchmark for agentic robotics safety.

deepmind.googlearticleAdded 29 Sep 2026, published 28 Jul 2026

video-shotcraft is an installable agent skill for Claude Code and Codex that storyboards, animates, and sound-designs cinematic product promo videos using Remotion, with a library of 157+ shot recipe cards, 214+ motion previews, and a complete production template (Ink Press). It ships a post-delivery "CapCut-style" editing workbench for adjusting shots/captions/SFX and can export finished films as editable JianYing (CapCut CN) drafts, alongside a sibling skill, video-talkcraft, for voice-synced narration videos.

github.comrepoAdded 5 Oct 2026, published 19 Jul 2026

Jonex is an open-source enterprise AI knowledge platform that combines multimodal document/video parsing with an ontology-based knowledge compiler, indexing content into both a vector store (Milvus) and a knowledge graph (Neo4j) for source-grounded retrieval. It exposes ontology-first hybrid search APIs with reasoning traces and references, and integrates with LightRAG, RAG-Anything, and MinerU for graph-enhanced retrieval and document parsing, plus an MCP server for tool exposure.

github.comrepoAdded 2 Oct 2026, published 19 Jul 2026

TurboFieldfare is a custom Swift + Metal inference runtime that runs the instruction-tuned Gemma 4 26B-A4B mixture-of-experts model on Apple Silicon Macs with as little as 8 GB of RAM. Rather than loading the full 14.3 GB model into memory, it keeps a 1.35 GB shared core and FP16 KV cache resident and streams only the per-token routed experts from SSD via a bounded 16-slot LFU expert cache, using 4-bit MLX-style affine quantization and custom Metal kernels for GEMV, attention, MoE, RoPE and sampling. It ships a native Mac app, CLI, and a loopback OpenAI-compatible server with function-tool support, plus an optional vision-tower companion pack for image input on M2+ Macs, and documents 103 benchmarked experiments behind its design choices.

github.comrepoAdded 2 Oct 2026, published 17 Jul 2026