🎨 Multimodal327

RoboAPI is a public API from PokeBot, powered by its world-action model Poke-Aurora, that lets robot arms do new tasks from natural-language instructions with no task-specific data collection or training. A user connects a supported arm with a wrist-mounted camera, gives a task such as "clean the table", and receives the next moves in near real time. The post gives no benchmarks, supported hardware list or pricing.

x.comthreadAdded 1 Nov 2026, published 31 Oct 2026

Hyperflow H3 is Video Rebirth's 8-step LoRA for MiniMax H3, released in September 2026 with open weights. Artificial Analysis ranks it #5 on AA-Video-I2V v2.1 and #3 on AA-Video-Editing v1.1, ahead of MiniMax H3 (768p). It costs $9.00 per minute for Image to Video and $18.00 for Video Editing at 1088p, under half of Dreamina Seedance 2.5 ($34.12 and $40.82). The thread gives per-capability and per-use-case breakdowns.

x.comthreadAdded 31 Oct 2026, published 30 Oct 2026

MonsterHouse is a Halloween interactive projection built on the NVIDIA Nemotron Voice Agent: Nemotron Speech ASR, Nemotron 3.5 Bolt E4B 4GB with a LoRA persona adapter, Magpie TTS Zeroshot and Audio2Face-3D in Unreal Engine, on two RTX Blackwell GPUs. Median end-of-speech-to-audio is 1.70 seconds live and 0.72 seconds with cached responses. The post gives a build path and tips on caching and state handling.

huggingface.cowriteupPublished and added 30 Oct 2026

Overtone is a caption format (.overtone.json), a CLI and a video.js plugin that store each spoken word once and cut captions separately per screen shape (16:9, 9:16 or custom). The transcribe pipeline combines gpt-transcribe, Azure AI Speech and optional ElevenLabs Scribe, resolves disagreements with an LLM vote, and uses an LLM editor for corrections. A small public benchmark reports 5.3% word error rate and $0.023–0.025 per audio minute.

github.comrepoAdded 30 Oct 2026, published 29 Oct 2026

RADIOlogy is a training-free, single-pass zero-shot anomaly detection method that uses multi-head attention pooling in vision-language models. It scores anomalies by contrasting average similarity between vision features and sets of normal and abnormal prompts, with no windowed execution. RADIO agglomerative features drive a dense refinement pipeline for pixel-level segmentation, and the method is applied to medical images.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

PhysCount is a benchmark of 546 real-world images and 4,595 human-written questions in 651 target-specific groups. It tests whether multimodal LLMs can separate visible appearances (direct view, reflections, glass) from the physical entities behind them. Across 40 MLLMs, counting appearances is much easier than linking them to entities, and the best model answers every question correctly in only 13.36% of groups.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

DecoyCount is a benchmark from Elorian and Mercor with 1,660 items that tests whether models count objects or fall back on visual shortcuts. Frontier models scored below 50%: GPT-6 Astra led with 43.17%, ahead of Gemini-3.8-flash (32.71%), Qwen3.8-max (28.31%) and Claude-Fable-5.1 (25.22%). CEO Andrew Dai says Elorian's Glimpse model scored 50.48% in a single internal run without tools, and contrasts visual reasoning with world models such as Atlas and Genie 4.

latent.spacearticleAdded 28 Oct 2026, published 29 Oct 2026

A five-task benchmark tests multimodal LLM color judgment against CIE colorimetric ground truth: AWB pair judgment, skin-tone bias direction, CCT estimation, Delta E 2000 pair ordering, and color naming under changing illuminants. Eight endpoints were tested. On ColorChecker-24 stimuli, closed frontier models reach CCT MAE of 715-1056 K versus 1358-2133 K for open-weight models, and a non-ColorChecker control shows this gap is partly template-dependent.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

Granite Switch 4.1.1 is a backwards-compatible patch release of IBM's modular Switch architecture, which configures a language model with specialized, cache-efficient adapters on one endpoint. It updates the Granite Speech ASR model (10 languages, native punctuation and capitalization) and the Decision adapter in granitelib-core, measured on the Jev Decision Index test set. Only the new adapters were retrained; the base model is unchanged. IBM plans tutorials and an Adapter SDK for third-party adapters.

huggingface.coarticlePublished and added 29 Oct 2026

Nemotron 3.5 Transcribe is a free open-weight speech-to-text model from NVIDIA covering 40 languages, including mixed-language speech. The post says it ties for first among open models with a 2.8% error rate. It says the model needs a large GPU such as an H100 or RTX 5090, so it will not run on a laptop. The post gives no benchmark name or model card link.

x.comthreadAdded 30 Oct 2026, published 29 Oct 2026

M-PFT is an angular-margin method for prompt tuning of vision-language models that fixes base-to-novel dual miscalibration (underconfidence on base classes, overconfidence on novel classes). It adds two losses: minimizing the maximum cosine similarity between tunable class embeddings, and minimizing angular distance to each zero-shot embedding. Tests on 9 prompt-tuning methods and 15 datasets show lower novel-class calibration error at comparable accuracy and negligible overhead.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

MedCBR is a two-stage alignment framework for medical vision language models. Difficulty Weighted Boundary Preference Optimization (DBPO) trains on counterfactual preference pairs at the model's capability frontier. Entropy Gated Grounded Policy Optimization (EGPO) then applies distilled clinical rubrics according to prediction entropy. It reaches 61.2% average accuracy on six medical benchmarks, with the largest gains on differential diagnosis, and the two stages combine super-additively.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

Nemotron 3.5 Transcribe is an open-weight, promptable NVIDIA mixture-of-experts model with 30B total and about 4B active parameters. One checkpoint does transcription in 40 languages, speaker attribution, translation, contextual prompting and audio understanding. The post says it ranked first in VoiceArena's comparison of 14 diarization plus ASR systems. The fetched text is cut off and gives no license or download details.

x.comthreadAdded 30 Oct 2026, published 29 Oct 2026

AnySpeechRecognizer and AnyDecisionModel are two Swift packages from Hugging Face. The first mirrors Apple's SpeechAnalyzer API with swappable engines (NVIDIA Parakeet, Alibaba Qwen3-ASR on Core ML) and adds Sortformer speaker diarization. The second returns typed yes/no, choice and score answers with probabilities via MLX, Core AI, llama.cpp, Jev or Ollama. The post gives M4 Max timings, model sizes and calibration options.

huggingface.coarticlePublished and added 29 Oct 2026

This paper studies whether vision-language models can infer how long a person stays somewhere (living, staying, or leaving) from geotagged social media posts. The authors build a new corpus of geotagged tweets co-annotated by humans and LLMs. State-of-the-art models struggle even with multi-turn Chain-of-Thought prompting, often missing contextual cues and subtle hashtags while over-relying on prominent visual details.

arxiv.orgpaperAdded 30 Oct 2026, published 28 Oct 2026

inkGround is a benchmark for stroke trajectory recovery from full-page, camera-captured handwriting, with 537 page images (about 18.7k words, 163k strokes) and pseudo-label references filtered by OCR agreement and human raters. Models get a page image and a target word, then output ordered stroke points. Seven VLMs were scored with DTW, Chamfer distance, readability and style similarity. Closed-source models, notably Gemini-3.1-Pro, did best, and the ranking held on IAM-OnDB.

arxiv.orgpaperAdded 30 Oct 2026, published 28 Oct 2026

AskStream is a multi-turn benchmark for streaming video understanding that adds a third proactive action, ask, alongside wait and answer, for cases where video or user information is incomplete. It has 438 videos, 2,714 manually annotated frame-level time windows (wait, ask, answer), four task categories and eight subtasks. The strongest model reaches only 52.18% on the proposed Ask-F1 metric, in both multiple-choice and open-ended settings.

arxiv.orgpaperAdded 30 Oct 2026, published 28 Oct 2026

QbD-VLA is a mixed-precision post-training quantization method for vision-language-action models. It uses task success from short closed-loop evaluations to assign weight and activation bit widths, accounting for interactions between components. On four VLA models across all four LIBERO suites it holds task performance at about 4-bit weights and under 6-bit activations. The gain on SIMPLER is small. On a Franka Research 3 arm, X-VLA shows no detected loss, but π0.5 fails.

arxiv.orgpaperAdded 30 Oct 2026, published 28 Oct 2026

Nemotron 3.5 Transcribe is an open-weight, promptable 30B mixture-of-experts model from NVIDIA with about 4B active parameters, built for recorded audio. One checkpoint transcribes 40 languages, handles recordings up to an hour, labels up to eight speakers, translates, and answers questions about English audio. Weights and a demo are on Hugging Face. The post gives no benchmarks.

x.comthreadAdded 29 Oct 2026, published 28 Oct 2026

Nemotron 3.5 Transcribe is an open-weight, promptable speech model that handles transcription in 40 languages, translation, diarization for up to eight speakers, and audio Q&A. It has about 4B active of 30B parameters and tops open-weight ASR and diarization leaderboards. The post also covers the Granary v2 curation pipeline (463,062 hours, 48 languages) and a medical fine-tuning case study that cut held-out WER by 45.8%.

huggingface.coarticlePublished and added 28 Oct 2026

This paper studies data scaling laws in stereo matching and releases XStereo, a realistic synthetic dataset of 200 thousand samples. In-domain training on five large datasets shows two regimes: a standard power law, then saturation, with a transition point roughly fixed for a given model. Cross-domain performance depends only weakly on source data scale, and distribution and coverage diversity reduce domain gaps more.

arxiv.orgpaperAdded 30 Oct 2026, published 28 Oct 2026

The paper studies detection and temporal localization of fraudulent inducements (prompts to follow, join chat groups, click links) in internet finance short videos. It compares OCR+LLM, OCR+MLLM and a zero-shot MLLM, and builds a dataset of 2,419 videos (66.49 hours, 3,194 events, four categories). OCR+MLLM reaches mIoU 26.52 versus 14.49 for keyframe OCR+LLM, with 67.0 percent macro recall, using a recall-weighted F2 localization reward.

arxiv.orgpaperAdded 30 Oct 2026, published 28 Oct 2026

This paper proposes an fMLLR-based compensation mechanism that cuts the fine-tuning data needed in transfer learning for speech emotion recognition under train-test acoustic mismatch. An autoencoder reduces trainable parameters, and an affine transform in the encoder latent space models rotations and shifts. In a human-robot interaction task, mean CCC rose 16.8% with 12.9% of the training set and 14.4% with 50%.

arxiv.orgpaperAdded 30 Oct 2026, published 27 Oct 2026

Know, then Draw is a learning paradigm for visual generation that uses autoregressive pretraining to acquire semantic knowledge and diffusion post-training for high-fidelity rendering. Controlled comparisons find autoregression the stronger pretraining objective, and semantic quality matters more than AR-only generation quality. This motivates a purely semantic tokenizer trained with language supervision only. A 3B text-to-image model trained in 6,600 H100 GPU-hours scores 86 on GenEval and 87 on DPG-Bench.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

This paper presents an annotated egocentric tennis dataset (egocentric video synced with five allocentric views, global positions, contact times, stroke types) and evaluates image encoders, video encoders and vision-language models on seven prediction tasks. Best models reach 0.14 m player and 0.54 m ball localization error and 0.14 s contact-time error. Depth input helps 3D ball localization, and longer histories do not help forecasting. A conditional diffusion model generates whole-body tennis motion in real time.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

VideoLVR is a method that lets video multimodal LLMs build latent visual evidence from local regions at relevant moments and reason over it interleaved with text tokens. A visual token prediction objective, defined over a vocabulary derived from the input video, supervises the latents. It extends to reinforcement learning with sampled visual tokens as actions and final-answer rewards, and improves fine-grained video benchmarks without much context growth.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

This paper defines a character naming task for films: given a name spoken in dialogue and its video context, the model must pick which visible character bears that name. It provides a dataset with annotations for character naming and first appearances, a video-and-dialogue model, and a comparison with proprietary multimodal LLMs. It also proposes a progressive framework that builds a character bank as the film plays, without cast lists or actor photographs.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

LFM3 is Liquid AI's open-weight family of six ~2B-parameter models (text, vision-language, audio, omni and two d1 decision models), all branched from LFM3-1.7B-Base. The base model uses a tri-hybrid backbone of 13 gated short convolutions, 5 sliding-window attention layers and 6 global attention layers, pre-trained on 22T tokens with a 256K context. The post gives transformers snippets, benchmark comparisons against LFM2.5 and Qwen3.5, and a robotics fine-tuning example.

huggingface.coarticlePublished and added 27 Oct 2026

LFM3 is Liquid AI's new family of open-weight multimodal models, six models of about 2B parameters branched from LFM3-1.7B-Base (22T pre-training tokens, 256K context). The hybrid backbone uses 13 gated short-convolution, 5 sliding-window attention and 6 global attention layers. Variants cover text, vision with video, audio, omni and two decision models, with on-device speed numbers, fine-tuning demos and robotics extensions.

liquid.aiarticleAdded 28 Oct 2026, published 27 Oct 2026

Genie 4 is Google DeepMind's real-time interactive world model, announced as a research preview for trusted testers. It generates video with native audio and interactive dialogue, steered by keyboard controls and text prompts. Gemini 3.8 Flash acts as a guiding agent that plans steps and checks frames, and Gemini 4 Argon is used for robotics experiments. The post lists limitations such as drift and object permanence, and says outputs carry SynthID watermarks.

deepmind.googlearticlePublished and added 27 Oct 2026

Genie 4 is Google DeepMind's world model for real-time interactive simulation, with native audio generated on the fly (ambient sound, music, dialogue). It builds on Gemini models: Gemini Omni grounds scenes in real-world appearance and behavior, and other Gemini models keep the simulation coherent. Users steer scenes by natural language. Access is limited to trusted testers now, with Google AI Ultra (Project Genie), the Gemini API and AI Studio to follow. The post gives no architecture or benchmark detail.

blog.googlearticleAdded 30 Oct 2026, published 27 Oct 2026

This paper studies how multimodal LLMs refuse questions that an image cannot answer, using the MoHoBench benchmark on three MLLMs. It varies the meaning and placement of refusal instructions, compares supervised and preference-based post-training, and intervenes on a refusal-related activation direction. Instructions change refusal at the sample level in ways aggregate rates hide. Post-training raises refusal most but can mix refusal with attempted answers. Activation intervention gives smaller, transferable changes.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

heditor is an open-source (MIT) agent skill from pycodebr that has Claude Code, Codex, Cursor or OpenCode edit talking-head videos. It transcribes with faster-whisper, picks a 30-70s segment, cuts frame-accurately, builds HTML + GSAP motion graphics, mixes audio at -14 LUFS, and runs QA with vision-reviewed previews. It offers 42 named styles and a long-form mode with chapters, SRT/VTT and lower thirds. Install: npx skills add pycodebr/heditor -g.

x.comthreadAdded 28 Oct 2026, published 27 Oct 2026

Hi3D v1.0 (formerly Sparc3D) is an open-sourced image-to-3D model from Hi3D, which runs a closed v3.0 refining at 2048 cubes per side. In a test with 13 professionals in 3D algorithms, 562 judgments against eight other image-to-3D models favoured it 83.5% of the time. The thread also covers the Hi3D Agent and an MCP server for other agents. Back and bottom views were checked, where TRELLIS.2 collapsed a winged figure.

x.comthreadAdded 28 Oct 2026, published 27 Oct 2026

AxisQ is a post-training quantization framework for camouflaged object detection (COD) that allocates activation bits by feature depth. It uses the channel domain for shallow features and the local-frequency domain for deep features, under a fixed per-layer average bit budget. It adds a boundary-aware Hessian approximation for reconstruction. On CAMO at W2A2, weighted F-measure improves 12.7% for CFRN and 33.3% for CamoFormer-P over the strongest baselines.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

GPT-6.1 Terra is OpenAI's midrange model, announced 26 October 2026 and aimed at visual reasoning, computer use and clearer writing. It costs $1 per million input tokens and $5 per million output, about half of GPT-5.6 Terra and GPT-6.1 Sol and roughly 10x GPT-6 Luna. Simon Willison relays OpenAI's Ted Sanders, who rates it better than Astra and Sol at technical communication and much faster on computer use.

simonwillison.netarticleAdded 27 Oct 2026, published 26 Oct 2026

This article reports a study submitted October 23, 2026 by Qiangong Zhou, Jiafeng Liang, Yimin Dong, Yang He and Hongxiang Li. They added Test-Time Training (fast-weight memory) to the vision-language-action models InternVLA-A1 and Pi0.5, trained on paired human demonstrations and robot trajectories. Shorter videos, full-trajectory gradient connectivity and contrastive regularization gave no clear gain in demonstration-guided target selection at the tested scale.

en.cryptonomist.charticlePublished and added 26 Oct 2026

RoSA is a finetuning framework from Qiuyu Kong and three co-authors for long-text image-text retrieval. It targets "saliency collapse", where models over-rely on text tied to prominent visual features and miss contextual information. RoSA splits pairs into salient and contextual views using object-level saliency estimation and LLM text decomposition, then trains on both with plug-and-play auxiliary alignment objectives. Code and checkpoints are public.

en.cryptonomist.charticlePublished and added 26 Oct 2026

A dermatologist tested GPT-6 Astra and Claude Fable 5.1 on 200 biopsy-proven lesions from MILK10k, giving each a clinical and a dermoscopic image. A CNN (Inception v3, fine-tuned for about twenty minutes on 68,011 ISIC images) named the right diagnosis for 62% of lesions, against 42% for both LLMs. The post lists caveats (small sample, single run, possible contamination) and links the cases, prompts, replies and scoring code.

lynchlab.netwriteupAdded 26 Oct 2026, published 25 Oct 2026

A stage-aware visual attribution framework for Vision-Language Navigation (VLN) that explains a robot's action choices. It identifies "instruction-stage insensitivity", where existing explainers give similar outputs across navigation stages. The method builds action-conditioned temporal instruction-visual attributions, aggregates relevant historical evidence by stage, and applies an instruction-guided spatial filter. It improves attribution quality across several VLN policies and benchmarks, helps diagnose failure cases, and was tested on a Unitree G1 robot.

arxiv.orgpaperAdded 27 Oct 2026, published 25 Oct 2026

This paper presents a pipeline that uses persona-conditioned vision language models to surface interpretive ambiguity in visualizations. Topic-relevant personas elicit candidate takeaways, and a Target-Pattern-Scope framework lets LLM judges label statement pairs as Compatible, Disjoint or Opposing, which are summarized in an ambiguity profile. The authors benchmark VLMs on chart comprehension per chart type, check labeling against human annotations, and run case studies on public visualizations and a redesign.

arxiv.orgpaperAdded 27 Oct 2026, published 25 Oct 2026

InteractiveWorld is an environment for closed-loop interaction between an AI assistant and a simulated human agent who sees the scene, follows instructions and acts independently. InteractiveBench, built on it, has 120 multi-turn tasks in six categories scored by verifiable, state-based rewards. Across 12 frontier MLLMs, the best model, Claude Fable 5.1, reaches at most 20% on long-horizon Guidance and 25% on Planning tasks.

arxiv.orgpaperAdded 27 Oct 2026, published 25 Oct 2026

Dessa is a voice AI model from Block, trained from scratch on human speech only, with no text stage. It listens while speaking, so it handles interruptions, laughter and tone changes. Block says it trained on one Nvidia GB200 NVL72 system and that it runs locally on a MacBook Pro with M4 Max or better. Model size, benchmarks and release date are undisclosed. Early access goes to a small group of AI safety researchers. Reactions split between praise and scam fears.

coindesk.comarticlePublished and added 24 Oct 2026

OWLS (Open-world Language-conditioned Speed Fields) plans directly over a vision-language model's dense feature map. A text prompt describing traversable terrain scores every pixel of one camera image. The scores are projected onto the ground, adapted with two morphological operations, and used as the speed field of a minimum-time planner. It needs no training, terrain labels, map or extra sensors. It was tested on a physical rover, on simulated Mars and Moon rovers over 200 procedural terrains, and on real planetary imagery.

arxiv.orgpaperAdded 26 Oct 2026, published 23 Oct 2026

DeepFaune-UK is a fine-tune of the open DeepFaune camera-trap classifier on 28 UK classes, paired with MegaDetector v6. On 20,212 MammalWeb images it reached 88.8% strict accuracy against 56.5% (DeepFaune v1.5.0) and 59.0% (SpeciesNet). Re-applying the recipe to the DINOv3-based classifier gave 90.6%. Fine-tuning on about 150 crops per class matched full-data out-of-distribution accuracy, and a DINOv2 run took about 22 minutes on under 4 GB of GPU memory.

arxiv.orgpaperAdded 26 Oct 2026, published 23 Oct 2026

UAV-SwarmAgent is a hierarchical multi-agent system where an operator gives one natural-language sentence and a ground-station agent writes and runs code that produces per-UAV routes. A vision-language model grounds targets in an aerial map, and worker agents verify incidents with a compact onboard model. In simulation it located the incident in 244 of 255 missions (95.7%), and flight time followed an Amdahl-type law with speedup bounded at 1.52x for one to five UAVs.

arxiv.orgpaperAdded 26 Oct 2026, published 23 Oct 2026

Signal-Aware Adaptive Fusion (SAAF) is a training-free zero-shot video anomaly detection framework that cuts the compute cost of vision-language models. It combines a dual-signal sampler (frame-to-frame delta and rolling-context drift in embedding space, OR-gated with EMA thresholds), a threshold-gated cascade from frame-level to video-level analysis, and signal-aware score fusion. It reports AUC-ROC of 92.07% on XD-Violence, 86.70% on MSAD and 80.54% on ShanghaiTech, at 2.2x to 2.6x the throughput of the prior best zero-shot method.

arxiv.orgpaperAdded 26 Oct 2026, published 23 Oct 2026

MART-GRPO is a variant of group-relative policy optimization for vision-language-action (VLA) reinforcement learning. It uses a LoRA-fine-tuned VLM to find the last visible error without recovery in a failed rollout video, keeps failure feedback up to that point, and drops it afterward. Martingale stopping conditions show this preserves the expected GRPO gradient and can reduce covariance. With OpenVLA-OFT on RoboTwin 2.0, it raises success rates over full-trajectory GRPO across multiple tasks.

arxiv.orgpaperAdded 26 Oct 2026, published 23 Oct 2026

4DMemory is a benchmark suite for long-horizon spatiotemporal memory: the LongVideo dataset (about 900 hours of video, over 5,000 manually reviewed questions) and 4DMemoryBench. It also presents the 4DMemory Harness, combining four skills, 22+ tool models and a model-specific RAG system. Across 13 models, the harness improved every one by over 50%, and the same models scored over 10% lower than on a previous benchmark.

arxiv.orgpaperAdded 26 Oct 2026, published 23 Oct 2026

MAST-VLA is a vision-language-action framework for multi-skill robot manipulation that conditions on both the skill to execute and where and how to contact objects. It adds skill-conditioned LoRA Mixture-of-Experts adapters to the LLM backbone and turns a predicted affordance heatmap into a patch-level spatial prior for the action head. It reaches an Avg. Len of 4.65 on CALVIN ABC→D and was tested on a real robot.

arxiv.orgpaperAdded 26 Oct 2026, published 23 Oct 2026

Sensorimotor Alignment (S^2) is a training-free offline metric that measures how well an embodied model's sensory representation aligns with the geometry of physical motion. It extends the Platonic Representation Hypothesis to motion trajectories. S^2 correlates positively with downstream robot task success across embodied model families, helps pick promising VLM backbones before fine-tuning into VLAs, and explains behaviours seen in VLAs and world action models.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026

F2G-Pose is a direct, single-pass framework that predicts category-level object pose, metric size and camera-frame completed shape from one RGB-D image. It lifts frozen RADIO foundation features onto partial point clouds, tokenizes local geometry with DGCNN, and fuses it with a Mixture-of-Experts Transformer. It trains only on synthetic SOPE data, leads on SOPE and ROPE at 28 FPS, and is tested on HouseCat6D, HANDAL and real manipulation.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026

Progress Probes (P^2) trains linear probes on a vision-language-action model's own activations, using demonstrations, to estimate task progress, then flags executions that deviate from expected progress. Tested on the open-source pi0.5 across simulated and real-world manipulation tasks, it matches or beats VLM-based monitors and failure-data-trained methods at failure detection, without an external foundation model.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026

TTR-Compressor is a sequence compression method that treats input tokens as a test-time regression problem. The regression is learned end-to-end by backpropagation, and its coefficients are solved in closed form in the forward pass to give the compressed representation. As a block-wise sparse attention compressor it improves RULER over attention-based compressors by a large margin. As a visual token merger for multimodal LLMs it beats the MLP merger and matches it with 55.6% fewer visual tokens.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026

CellTrack-Bench is a benchmark that tests vision-language models on microscopic cell tracking and compares them with specialised trackers. It has 1,537 problems covering 9,041 cell-linking decisions, drawn from 15 cell-tracking datasets across four imaging modalities, with a hard subset built from failures shared by distinct algorithmic trackers. The authors compare environment harnesses, benchmark 9 closed-source and 13 open-weight models, and analyse failure modes.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026

DuoCook is an Overcooked-inspired benchmark in which one vision-language model controls two cooks from top-down views in Unreal Engine environments built with coding agents. It has 20 maps in five categories covering equipment contention, changing terrain and handoff dependencies. Four VLMs were compared; the best, gpt-6-astra, reached 26.2% mean order completion. A notebook memory ablation had mixed effects: positive for Claude and Qwen, negative for GPT and Gemini.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026

OlmoEarth v1.3 is an open satellite embedding model from Ai2 that reads a year of Sentinel-1, Sentinel-2 and Landsat data and outputs a 128-dimensional vector per 10 m pixel (the first 64 dimensions also work alone). Its embeddings beat AlphaEarth Foundations and Tessera v2 on 20 of 27 evaluations. Precomputed global embeddings, a browser Embeddings Explorer and pre-training cost (about $2,720 of H100 time) are covered.

huggingface.coarticlePublished and added 22 Oct 2026

Rate-Controlled Refresh (RCR) is a KV cache re-selection method for multimodal LLMs in long-form generation. Attention drift decides when and where to re-select visual tokens, and a per-sample controller sets how often. RCR beats InfiniGen by 1.6-3.8 ROUGE-L points on five of six model-task settings and stays within 1.7 points of exact per-step re-selection. On video description, where drift is homogeneous, it no longer beats a fixed threshold.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026

Forest Typology (ForTy) 2020 is a global 10-meter map from Google DeepMind, Google Research, WRI and IIASA. It sorts land into six FAO-based classes: primary, naturally regenerating, planted, plantation, tree crops and agroforestry, and other land. The model uses AlphaEarth Foundations embeddings and about 1.7 million labeled locations. It was 90% accurate on forest type across 9,000+ expert-checked sites. An early 2017 to 2025 time series is on Earth Engine.

deepmind.googlearticlePublished and added 22 Oct 2026

OnKey is a free, MIT-licensed singing trainer for Apple Silicon Macs. After a song ends in Spotify, Apple Music or browser YouTube, it separates the lead vocal with UVR's MDX-Net Karaoke 2, tracks pitch with CREPE and turns it into syllable-level notes with the GAME model. On later plays it scrolls the notes, draws the user's live pitch and scores each line. Analysis runs on-device, and the repo explains the build steps and network requests.

github.comrepoAdded 24 Oct 2026, published 22 Oct 2026

TRIAD is a benchmark for omni-modal ambiguity resolution. It has 327 hand-written text-image-audio queries in 86 scene groups across 18 ambiguity categories, and removing any one modality makes the answer non-unique. Across 14 omni-modal systems, the best model trails humans by over 20 points in group accuracy. Without answer options, model group accuracy falls to single digits, and leave-one-out tests show audio cues are often underused.

arxiv.orgpaperAdded 23 Oct 2026, published 22 Oct 2026

Matrix-Game 4.0 is a system that separates game logic from visual rendering for playable generated worlds. A lightweight engine holds world state and rules, a multimodal coding agent writes game code and scene captions, and a distilled autoregressive video model turns low-poly proxies into detailed frames. It runs 480p at 60 FPS on one RTX 5090 with 112 ms key-press-to-display latency. Code, weights and assets are released.

huggingface.copaperAdded 22 Oct 2026

XiaoZhi ESP32 is an open-source, MIT-licensed firmware stack that turns ESP32 boards into MCP-based voice chatbots using models such as Qwen and DeepSeek. It streams Opus audio over WebSocket or MQTT + UDP through ASR, LLM and TTS, adds offline ESP-SR wake words, and uses device-side MCP to drive speakers, LEDs, servos and GPIO. It supports 138 board directories and builds with ESP-IDF 6.0.1 or newer.

x.comthreadAdded 23 Oct 2026, published 21 Oct 2026

Granite Speech 5.0 is IBM's update to its open speech model collection: the new granite-speech-5.0-2b, a multilingual granite-speech-5.0-470m-turboctc, and a revised granite-speech-4.1-2b-plus. The 2B model ranks #2 among open-weight models on average English WER (5.06%) on the Open ASR Leaderboard. Turbo CTC reaches 19,870 RTFx, and the 4.1 Plus revision cuts long-form WER 24%. All are Apache 2.0.

huggingface.coarticlePublished and added 21 Oct 2026

GPT Voice is a free, open-source Mac app by Vishesh Raheja that brings ChatGPT's dictation to any macOS app. A shortcut starts recording, and the transcript is pasted where the user is typing. It uses the ChatGPT Pro subscription, so it needs no API key or extra subscription. The author built it after Handy handled coding prompts poorly and they did not want to pay for Wispr Flow. The thread gives no details on local versus API transcription or latency.

x.comthreadAdded 22 Oct 2026, published 21 Oct 2026

Cosmos3-H-Dreams is NVIDIA's real-time generative surgical simulator, trained on clinical cholecystectomy video and kinematics from the CMR Surgical Versius platform. It builds on the 16B Cosmos3-H-Embodied-Nano, distills a bidirectional teacher into four-step and two-step causal students with Self-Forcing, and streams through FlashDreams on one RTX PRO 6000 GPU. Checkpoints and a recipe for custom data are released.

huggingface.coarticlePublished and added 20 Oct 2026

Krea 3 is an image generation and editing model from Krea. A post on X says its "Fast" variant scores 1125 Elo on the Artificial Analysis image editing leaderboard, against 1124 for Google's Nano Banana 2, while producing 10 images in the time Google produces 2. It also says the "Ultra" variant ranks in the top 4, behind only OpenAI models. The post gives no methodology or model details.

x.comthreadAdded 21 Oct 2026, published 20 Oct 2026

UniWALM is a unified world-action-language model that learns video, action and language in one mixture-of-transformers backbone with per-modality diffusion heads that co-denoise together. Generated subtask and textual memory are fed back on the next step, so no external planner or memory bank is needed. It adds Flow Matching with Correction (FMC) for recovery from perturbed actions. It reaches 34.5% on the memory-dependent RMBench subset, above Mem-0's 28.5%, and lifts real-world Find-the-Item success from 11.1% to 61.1%.

arxiv.orgpaperAdded 22 Oct 2026, published 20 Oct 2026

InterWebCoder is a framework for interactive webpage code generation by multimodal LLMs. It synthesizes self-contained webpages with executable test scripts from recorded interaction trajectories, repairs failures using runtime evidence, then trains with test-ID-weighted SFT and GRPO using test pass rates and visual similarity as rewards. The pipeline yields 1,736 verified samples from 2,058 tasks. A 9B model scores 48.54 on IWR-Bench and 48.42 on Vision2Web, and reward latency drops over 97%.

arxiv.orgpaperAdded 22 Oct 2026, published 20 Oct 2026

AnomalyAgent is a post-training fusion framework for visual anomaly detection that ensembles pretrained detectors. It aligns scores and anomaly maps, summarizes disagreement, and treats fusion as a choice among score actions and map actions. A constrained vision-language model inspects high-conflict samples and picks the actions. It beats equal-weight fusion and gets the best mean image AUROC and pixel AP among compared fusion methods on MVTec AD and VisA.

arxiv.orgpaperAdded 22 Oct 2026, published 20 Oct 2026

Code2Video-Compass is a benchmark of 2,104 tasks for code-based video generation, where multimodal LLMs write executable programs that produce complete videos from text (T2V), image (I2V) or video (V2V) inputs. Tasks span 10 categories and 24 subtypes. Scoring uses 12 objective metrics and five MLLM-judge metrics. Across 15 proprietary and open-weight models from nine families, performance falls as construction, dynamic animation and state-dependency complexity rise.

arxiv.orgpaperAdded 22 Oct 2026, published 20 Oct 2026

DIVE is a benchmark for full-duplex interaction with visual engagement in continuous audio/video streams. It has 525 human-annotated samples across six tasks in three scenarios: proactive intervention, continuous engagement and interaction coordination. Evaluated real-time models reach sub-second average latency, but window response rates stay at or below 53.77%, and adapting ongoing responses to new information is hardest.

arxiv.orgpaperAdded 22 Oct 2026, published 20 Oct 2026

Jev Cards is a browser picture quiz where a player competes against Jev 1.14, a vision decision model. The 490 cards cover fruit, flags, maps, chess and illusions. Jev reads the image and question and returns a probability for every option, or a box for "tap the place" cards, in one call of about 0.25 seconds. It never writes text. The page reports 5,256 wins in 6,820 games.

jevcards.comtoolAdded 21 Oct 2026, published 20 Oct 2026

This paper presents a multimodal engineering GraphRAG framework that links post-wind-event damage evidence from images and reconnaissance reports to building-condition states, recovery actions and downtime. Evidence is stored as a provenance-preserving, component-level representation, and graph-aligned retrieval uses compatibility, contradiction, evidence-precedence and abstention checks. Demonstrated on tornado-affected low-rise homes, it beats BaselineRAG and Structured Baseline RAG, and ablations test each component.

arxiv.orgpaperAdded 22 Oct 2026, published 20 Oct 2026

OpenVLHarness is a multimodal reasoning harness for MLLMs built on a capability layer (task-level tool interfaces plus evidence rendering) and persistent multimodal memory that stores images, structured data and web references as addressable artifacts. A coding interface runs generated programs over stored data. Across 22 benchmarks and eight backbones it adds 9.7-12.6 points over no-tool baselines, and GPT-6 Luna at medium effort beats Codex and Visual Sketchpad at maximum effort using at most 37% of their tokens.

arxiv.orgpaperAdded 21 Oct 2026, published 19 Oct 2026

The paper introduces PhysRSI-eval, a five-axis metric for physical recursive self-improvement: first-attempt performance, efficiency, success, budget and improvement from failure. Several VLMs are tested on 18 manipulation tasks under a fixed interaction budget. It proposes the Physical Barrel Hypothesis and studies observational scaling. Workspace interventions show that removing history lowers late-round success, while frozen workspaces still help on unseen layouts.

arxiv.orgpaperAdded 21 Oct 2026, published 19 Oct 2026

This preprint grades evidence for pathology vision-language models and tests four claims with controlled probes on released checkpoints and public data. Stage-redacted report embeddings raise pooled C-index by 0.019 (progression-free interval) and 0.016 (overall survival) across 1,633 TCGA patients. Zero-shot prompting matches a linear probe. SlideChat scores 74.82% with the slide and 74.41% without it. Paraphrasing flips 8-23% of answers.

arxiv.orgpaperAdded 21 Oct 2026, published 19 Oct 2026

BrainLoRA is a parameter-efficient fine-tuning method that adapts multimodal backbones of up to 104B parameters to predict neural responses. It predicts multiple timepoints in one causal pass instead of sliding windows, and a multi-LoRA kernel trains many adapters on a shared backbone. Across nine backbones it improves encoding, and a 200+ model ensemble ranks first on the Algonauts post-challenge leaderboard.

arxiv.orgpaperAdded 21 Oct 2026, published 19 Oct 2026

Mise-en-Video (MeV) is a multi-shot video generation framework where creators stage shots in 3D space and then generate the matching video. It combines hierarchical attention for multi-level narrative consistency with 3D-aware layout guidance for per-shot spatial control. The authors built a 160K-video dataset with hierarchical multi-shot texts and oriented 3D bounding boxes, and report gains over baselines in quality, consistency and subject controllability.

arxiv.orgpaperAdded 21 Oct 2026, published 19 Oct 2026

HAF-VLA is a hierarchical action-flow framework that adapts a pretrained flow-matching vision-language-action model to humanoid whole-body loco-manipulation. It denoises a shared action space in stages: locomotion and head commands first, then waist refinement, then full-body manipulation. Learnable stage embeddings and cross-stage KV-cache conditioning carry dependencies between stages. The authors report gains over existing models on six real-world tasks.

arxiv.orgpaperAdded 21 Oct 2026, published 19 Oct 2026

Model card for Meta Superintelligence Labs' Muse Spark Open, an open-weights, natively multimodal model with 816B total and 42B active parameters and a 1M-token context window. It targets coding and agentic work (tool use, long-horizon tasks, multimodal reasoning, synthetic data generation, LLM-as-judge). It ships in BF16 and NVFP4, needs about 700GB VRAM (NVFP4) or 2TB (BF16), runs on SGLang, vLLM, Transformers, TensorRT-LLM, Unsloth, llama.cpp and MLX, and supports a reasoning_effort setting. The Muse Model License requires "Built with Muse Spark Open" attribution and a separate agreement for Model-as-a-Service above $20M revenue. The card reports safety results against Kimi K3 on cyber, chem/bio and loss-of-control evals.

huggingface.corepoAdded 17 Oct 2026

FedRAPP is a personalized aggregation framework for Source-Free Federated Test-Time Prompt Tuning (SF-FTTPT), where clients adapt CLIP soft prompts on unlabeled, non-IID test streams without source data. It combines reliability-aware client assessment, a server-side exponential moving-average prompt memory and similarity-guided peer aggregation. Across nine CLIP benchmark datasets and ImageNet variants it beats standard federated aggregation baselines while keeping prompt-only efficiency.

arxiv.orgpaperAdded 20 Oct 2026, published 16 Oct 2026

FlexPart is a framework for text-controlled 3D mesh segmentation that assigns discrete part IDs to mesh faces through a view-independent part-identity field. It uses a control prompt of part names with location and shape clauses, trained on a VLM-built multi-granularity dataset. On FlexPartBench it raises average mIoU by 22% over the strongest baseline, and by 15% with planner-completed prompts.

arxiv.orgpaperAdded 20 Oct 2026, published 16 Oct 2026

LILIE (Language-Informed and Lossless Image Encoder) is an invertible image transform built from a normalizing flow transformer, trained jointly with a vision-language model via next token prediction so captions can be recovered from noised features. The encoder supplies a language-informed loss for text-to-image generation. JiT models trained with it cut gFD from 66.84 to 53.01 against a pixel-only baseline on the GPIC dataset. Code, encoders and models are released.

arxiv.orgpaperAdded 20 Oct 2026, published 16 Oct 2026

This paper presents a multimodal retrieval-augmented generation framework for Medical Visual Question Answering. A Gated Multimodal Unit learns a joint query representation from the medical image and the clinical question, so evidence retrieval uses both modalities while the Med-LVLM stays frozen. On benchmark datasets it consistently beats a reference Med-LVLM and stays competitive with state-of-the-art methods. The page gives no dataset names or numbers.

arxiv.orgpaperAdded 20 Oct 2026, published 16 Oct 2026

A tweet from AmingIn_AI describes a macOS screenshot-annotation tool. On release it saves the screenshot, a crop and the user's marks locally, then copies a 127.0.0.1 link. Claude Code or Codex on the same Mac can fetch that link and receive the marked image plus app/window info, and nothing is uploaded. It shows a local, privacy-preserving way to hand visual context to coding agents without MCP or cloud upload.

x.comthreadAdded 17 Oct 2026, published 16 Oct 2026

MUB (Music Understanding Benchmark) is a set of 500 identity-masked recordings with existing annotations for tempo, key, time signature and structural intervals. It covers all 24 keys, tempos of 47 to 180 BPM and 4,676 intervals, and strips titles, artists and platform IDs from inputs. Baselines reach 53.6% tempo accuracy (DSP) and 98.4% time-signature accuracy (constant 4/4). No multimodal LLM has been tested yet, and rights clearance is unresolved.

arxiv.orgpaperAdded 20 Oct 2026, published 16 Oct 2026

MOSS-Image is an open-source text-to-image model family from OpenMOSS with a 5B backbone, released October 2026 as MOSS-Image-Base and MOSS-Image-SFT. It supports JSON layout control with bounding boxes, text rendering inside images, and prompts up to 4K tokens. The repo gives a Diffusers pipeline, a Gradio app, a ComfyUI plugin, LoRA training guidance, a prompt-enhancer skill and day-0 SGLang and vLLM-Omni support.

github.comrepoAdded 20 Oct 2026, published 16 Oct 2026

IBM announces Granite-Switch-4.1-8B and -30B. Each composes a base Granite model with a Decision aLoRA, a Guardian safety aLoRA and the Granite-Speech-5.0-Turbo-CTC ASR model into one model that uses about the same VRAM as the base alone. Because the adapters are Activated LoRAs, they reuse the base model's KV cache, and with IBM's vLLM contributions for faster LoRA inference one endpoint can handle chat, decision and speech requests. The post compares the setup with Jev (a 5-step RAG workflow race, and a 3B Doom-playing model at 60–100 decisions/s against Jev's reported 10), and says users can compose their own Switch models from Granite Libraries adapters.

huggingface.coarticlePublished and added 15 Oct 2026

Social post reporting Meta's release of Muse Spark Open, described as the most capable US open-weight model: 816B total and 42B active parameters, a 1M-token context, and text, image, video and audio input in one model. It is pitched at coding and agentic work and tuned to run on two M5 Ultra Mac Studios, with weights available at release. The poster notes that in Meta's own chart, China's Kimi K3 and GLM 5.3 still beat it on coding.

x.comthreadAdded 16 Oct 2026, published 15 Oct 2026

A tweet relaying OpenAI's announcement of GPT-Live-1.1, an update to the voice model behind ChatGPT voice mode. It reduces how often the model says "checking", adds new voices and improves accents, including new voices in French, German, Hindi, Mexican Spanish, Korean and Japanese. It is rolling out to all users in ChatGPT, Codex and "dots", with API access coming soon. The poster frames it as a response to a new xAI/SpaceXSI Grok Voice release.

x.comthreadAdded 16 Oct 2026, published 15 Oct 2026

P^3-WAM is a unified world action model for egocentric dexterous manipulation with three modes: Perceive (reconstruct articulated hand motion and camera trajectories from video), Predict (generate visual futures under prescribed actions) and Plan (jointly generate action trajectories and their anticipated visual outcomes from an initial image, a language instruction and an optional visual goal). It adds a compact structured 3D action representation to a pretrained video diffusion transformer through a mixture-of-transformers design with shared base weights, modality-specific adaptation and bidirectional visual-action attention. Randomized modality visibility lets video and 3D actions act as either context or target. Unified training beats single-mode training and outperforms specialized baselines on 3D hand reconstruction and action-conditioned video prediction across four datasets, and gives better instruction adherence than video-generation baselines.

arxiv.orgpaperAdded 17 Oct 2026, published 15 Oct 2026

The paper introduces 3DVerifierBench, a 10-task benchmark for generative verification of 3D content. It scores accept/reject decisions, rejection explanations, and responses to controlled changes in specifications and candidates. A human study shows a 21.6-point gap to the strongest tested model, GPT-6 Astra. The authors also build a pipeline of multi-view fact extraction, controlled 3D editing and native annotations. It yields 14,126 training examples with contradiction explanations and linked counterfactuals. Verification-guided selection raises complete factual success from 26.7% to 90.0% with oracle selection over sixteen candidates, and human-guided recursive refinement satisfies all seven requirements for every input in an eight-input study.

arxiv.orgpaperAdded 17 Oct 2026, published 15 Oct 2026

DK-VLM adds Dual-Kernel Attention to a vision-language model. A semantic kernel keeps the pretrained attention and positional encoding. A separate geometric kernel uses estimated 3D points and camera poses to modulate attention through point displacements and relative view transformations, and a learned gate fuses the two outputs. Geometry therefore enters attention as position, not as fused 3D reconstruction features. The paper also adds Reference-Frame Composition (RC) supervision, which gives stepwise turn and heading targets derived from verified single-step annotations, to train multi-step spatial reasoning as the observer moves. DK-VLM-8B reports the highest average score across six spatial benchmarks among the compared methods, with 74.33 on VSI-Bench and 58.28 on ReVSI-32, and it preserves general video understanding.

arxiv.orgpaperAdded 17 Oct 2026, published 15 Oct 2026

Velox is a training-free attention router for chunked video-diffusion world models. It uses camera frustum slabs with depth intervals from streaming geometry to pick which historical token blocks to attend to, instead of letting the KV cache grow with rollout length. Frustum overlap softly modulates each layer's model-derived block importance, and uncovered regions keep the model's own routing. The fixed block granularity suits FlashAttention-style tiled execution, and slab construction runs asynchronously with denoising on the same H200. At chunk 48 it speeds up attention 4.36× and denoising 2.43×, and on the 158-case WBench navigation split it stays within 0.15 average points of dense LingBot-Fast-Cam.

arxiv.orgpaperAdded 17 Oct 2026, published 15 Oct 2026

Proposes SCOPED (Skill-Decoupled On-Policy Expert Distillation), which trains capability-specific RL experts for vision-language models. It extracts explicit skills from them through reward-guided reflection. During on-policy distillation the skills guide expert supervision but are hidden from the unified student, so the student internalizes the procedural knowledge and needs no explicit skills at inference. Across eight multimodal benchmarks it beats the backbone and mixed-capability RL. Analyses show cross-capability gradient interference in joint RL, and that skill augmentation gives more consistent expert trajectories, more attention to task-relevant visual evidence, and better preservation of expertise.

arxiv.orgpaperAdded 17 Oct 2026, published 15 Oct 2026

Presents a modular, fully onboard UAV navigation pipeline that closes the "human-aware safety gap" of geometry-only planners. A shared BLIP encoder drives captioning and text-conditioned segmentation, which is fused with depth for 3D obstacle localization. Qwen-2.5-0.5B, stabilized with RAG and grammar-constrained JSON decoding, infers per-category risk once per class and caches the verdict. Perception runs at 10 Hz on a Jetson Orin NX, and fallback LLM queries take about 0.8 s off the control path. In Gazebo it gives roughly 3x fewer collisions than a geometric planner and raises semantics-aware landing success from 34% to 96%. The 224-pixel encoder input limits small-object perception at range.

arxiv.orgpaperAdded 17 Oct 2026, published 15 Oct 2026

Introduces P^3IBench, a benchmark for proactive video understanding in real-life streaming scenarios, covering three capabilities: daily perception, risk prediction and intervention. It evaluates 12 open-source and proprietary video-language models (streaming and adapted non-streaming) and finds they struggle to make well-calibrated decisions about when to respond. The authors also build extra proactive training data and train P^3I-Assistant-4B, which improves several tasks unevenly, showing that proactive assistance needs response timing and restraint as well as visual understanding.

arxiv.orgpaperAdded 17 Oct 2026, published 15 Oct 2026

Release notes for Apertus 1.5, a fully open (weights, data, training details) continued pretraining of the 8B and 70B Apertus models on 4T and 2T additional tokens. It adds early-fusion image and audio understanding via discrete tokenizers (Emu3.5, WavTokenizer) without architectural change, a thinking mode, 262k context, and better tool use. It documents a four-stage pretraining and five-stage post-training pipeline (SFT, long-context SFT, RLVR, DPO, charter-alignment SDPO), plus evaluation via the MLLM-eval-suite and Every Eval Ever (EEE) Eval Cards, released checkpoints and datasets.

huggingface.coarticlePublished and added 15 Oct 2026

Hugging Face's launch post for Meta's Muse Spark Open, an open-weight 816B-parameter mixture-of-experts model (about 42B active per token). It has 256 experts with 8 routed per token, a 1M-token context, and perception and audio encoders for image, video and audio input. The post gives the architecture details, BF16 and NVFP4 checkpoints, and run recipes for transformers, llama.cpp, SGLang, vLLM, MLX and Inference Endpoints. It also covers LoRA SFT of the full model with TRL on 64 H100s and agentic demos (Pi 3D asset generation, an OpenEnv browser agent, a voice-driven shopping agent).

huggingface.coarticlePublished and added 15 Oct 2026

Hugging Face blog post introducing Black Forest Labs' open FLUX.3 Image Klein (distilled and base checkpoints) in Diffusers. It covers text-to-image, multi-reference image-to-image and bounding-box layout control, FP8/NVFP4 single-file quantized weights, DreamBooth LoRA training scripts for generation and editing, memory/speed figures (about 2 s per 1024×1024 image on an H100), and a detailed architecture tour: Qwen3-VL text encoder, two-stage per-stream then joint transformer blocks, per-token timestep modulation, 4D RoPE, and flow-matching denoising. It also shows how to customise the Modular Diffusers pipeline.

huggingface.coarticlePublished and added 15 Oct 2026

System card for GPT-Live-1.1 and GPT-Live-1.1 Quip, OpenAI's full-duplex voice models that listen and respond continuously. It reports voice-native disallowed-content evals on production and synthetic audio prompts, and finds parity overall but statistically significant regressions for GPT-Live-1.1 on sexual, mental health, emotional reliance and gore categories. It also covers alignment evals converted to audio (respecting warnings, blocker deception, broken search tool) and Preparedness Framework conclusions: neither model is High in Biological and Chemical, Cybersecurity or AI Self-Improvement.

deploymentsafety.openai.comarticlePublished and added 15 Oct 2026

Podcast interview with Jason Ma, co-founder and Chief Scientist of Dyna Robotics, on Dyna-2, a world-action model pre-trained on over one million hours of human egocentric video. The model predicts future video and uses it to guide robot actions, built on a diffusion transformer with flow matching. The discussion covers the robotics data bottleneck, adapting to new tasks from about 45 minutes of demonstrations, synthetic environments and self-generated training data, on-device inference and distillation, and Dyna's custom wheeled semi-humanoid hardware. The episode listing also covers commercial deployments (Din Tai Fung, Best Western), agentic harnesses for hour-long workflows, and world-action models versus VLAs, but the fetched text was truncated before those sections.

latent.spacetalkPublished and added 13 Oct 2026

ChunkAct brings action chunking to agentic robotics, where an LLM agent composes perception, motion and VLA tools guided by memory. Online, one call launches a chain of primitives that runs autonomously and returns control only on failure. Offline, memory is compiled into these chunks so execution needs no memory at test time and the planner is used only for real decisions. On LIBERO-Pro it improves task success by 5.0 points over the prior agentic-robotics state of the art, with 2.9x fewer LLM calls, 4.4x fewer input tokens and 2.3x fewer output tokens.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

MiniDog is a framework for training a text-to-image flow-matching diffusion transformer from scratch on four RTX 3090 GPUs, with 55 minutes of pretraining and 15 minutes of fine-tuning. It pretrains on 26k ImageNet dog images that were captioned, then fine-tunes on 2k FLUX.1-schnell generated images. Image and text features are precomputed and cached. The model is a 22M-parameter transformer with the E2E-INVAE semantic latent space, a small Qwen text encoder and in-context conditioning, and training is accelerated with iREPA and an EUPE perception encoder. The repo evaluates with FID, HPSv2, PickScore and VAE reconstruction metrics. It lowers the barrier to hands-on diffusion training research.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

Proposes mmVB, a lightweight plug-and-play module (1.6M trainable parameters) that assigns query-conditioned per-image visual token budgets using cross-image context for multimodal LLMs, without retraining the base model or accessing attention matrices (so it is compatible with FlashAttention). It cuts visual input tokens 9x, giving 5x faster time-to-first-token and 7x fewer prefill FLOPs while keeping 94.4% of full-resolution accuracy across eight multi-image reasoning benchmarks. It matters because existing token pruning, merging and compression methods target single-image redundancy and either need attention maps or retraining.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

PixelMoE is a pixel-space diffusion framework that scales capacity sparsely using standard LLM-style Mixture-of-Experts: token-choice routing, fine-grained routed experts, a shared expert and bias-based load balancing, with no diffusion-specific routing or auxiliary routing losses. It adds Pixel-Space Hierarchical Flow Matching, which varies spatial resolution along the denoising trajectory from global structure to fine detail. At matched active parameters it beats dense counterparts, and reaches FID 1.47 at 256x256 and 1.53 at 512x512 on class-conditional ImageNet-1K without representation alignment or perceptual losses. Code and models are promised but not yet released.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

RepProbe is a benchmark of 42K real and synthetic video clips with exactly 1–20 complete action repetitions, controlling for clip position, temporal coverage and duration. Video language models' counting accuracy collapses at high counts, and this is not explained by frame sampling alone. Linear probes on final-layer representations show count information stays linearly decodable even when the model's answer is wrong, so the failure arises at the output stage, where answers cluster on a few preferred numbers.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

Plan-Bench is a benchmark of 699 curated samples across 16 subclasses and three dimensions. It tests whether vision-language models can reason over spatial and temporal information and produce multi-step plans under constraints for embodied tasks, with action execution isolated out. Across 28 VLMs the best model reaches only 75.0% accuracy. Scaling generally helps, and larger models look more promising as high-level planners. Code and benchmark are released.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

Game2Real-Bench tests whether vision-language model performance on procedurally generated toy games predicts real-world visual reasoning, and whether game examples help transfer. It has 1,000 verifiable game samples and 400 real-world samples, matched by shared reasoning requirements (fine-grained perception, spatial reasoning, constraint reasoning, sequential planning). Across 25 VLMs, game and real scores correlate strongly (Pearson/Spearman 0.935). One-shot game-to-real transfer on four models improves constraint reasoning for several but hurts fine-grained perception. The authors also release Game2Real-Dataset, with about 24K verified game samples across 18 tasks and 65 QA families.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

OmniCompass is a manually annotated benchmark for video grounding and understanding with two tracks: OmniCompass-VL (820 videos, 2,460 instances; vision-language models) and OmniCompass-AV (240 videos, 720 instances; joint audio-visual evidence for omni-modal models). Each track covers temporal grounding, spatial grounding and QA across short/medium/long videos, and each instance is run in paired text-only and text+image target specifications. Evaluating 50 MLLMs shows open-source models lag proprietary ones, especially on audio-visual understanding, that precise temporal/spatial grounding remains hard, and that predictions are inconsistent across the paired specifications.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

SP$^2$ec (Single-Peak SPeculative decoding) is a training-free bandit algorithm that adapts the draft configuration of self-speculative decoding per prompt for vision-language models. It relies on the single-peak throughput pattern seen in SWIFT, KnapSpec, SpecVLM and the authors' cosSim method. It minimizes a new wall-time regret measure and has theoretical guarantees of near-optimal configuration with negligible exploration cost. On Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-Mistral-7B it improves over the original self-speculative methods by up to 11% on visual and textual tasks.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

Google's developer post presents Gemini 4 Argon as a physical-reasoning model for the whole robotics lifecycle. It reports 87% on the Embodied Reasoning (ERQA) benchmark, ahead of other frontier models. It then gives representative prompts: controlling a Boston Dynamics Spot with a MuJoCo digital twin, closed-loop code-as-policies arm control, steering the Gemini Omni world model, and building digital twins from teleoperation episodes. Further prompts cover retargeting human video to a 76-DoF humanoid, hindsight annotation of ALOHA 2 videos, a general value function on UMI data, sim-data generation for VLA fine-tuning (56% to 75% real success), and reproducing control papers in MuJoCo WebAssembly. It is useful for seeing how the model is prompted for agentic, simulation-driven robotics work.

developers.googleblog.comarticlePublished and added 13 Oct 2026

UniW(A)M is a mixture-of-experts model built on diffusion Transformers that unifies action-conditioned world modeling (AC-WM) with several world-action model (WAM) variants. Shared experts capture common physical dynamics, while atomic capability experts are reused across interfaces, each keeping its own conditional pathway. At deployment a plug-in adaptive interface router (PAIR), trained on a small subset of target-task chunk scenarios with the backbone frozen, picks a WAM interface for each execution chunk. It reaches 88.3% average real-world success against 76.7% for the strongest baseline, leads WorldArena on all three action-conditioned world-modeling tasks, and PAIR raises OOD success from 63.3% to 76.7%.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

Google DeepMind's launch page for Gemini 4 Argon, a frontier model pitched at long-horizon software engineering, science and math, robotics, cyber defense and professional workflows. Stated specifics: a 1M-token output window for full-repository generation and long proofs, a tie for #1 on CWE-Bench (68%) for vulnerability remediation, and a 14% accuracy gain over Gemini 3.1 Pro in Box's evaluation. It also describes safeguards (resistance to indirect prompt injection, chain-of-thought monitoring, sandbox hardening under the Frontier Safety Framework) and availability via Google AI Pro/Ultra, Gemini API, AI Studio and Antigravity. Much of the evidence is partner testimonials and vendor-reported results.

deepmind.googlearticlePublished and added 13 Oct 2026

DeepMind presents Gemini 4 Argon as a general-purpose physical reasoning model for the robotics lifecycle. It reports 87% on the Embodied Reasoning (ERQA) benchmark and gives representative prompts across several areas: controlling a Boston Dynamics Spot with a MuJoCo digital twin, code-as-policies arm control with a visual coordinate harness, and steering the Gemini Omni world model in a visual feedback loop. Further examples cover building MuJoCo and mujoco_warp scenes from teleoperation data, egocentric video or a hand-drawn sketch for 76-DoF humanoids, hindsight annotation and value-function progress prediction on video, sim data generation for VLA fine-tuning (success rate 56% to 75%), and reproducing control papers in MuJoCo WebAssembly. It matters as a concrete look at how a frontier model is being used as agentic coder, perceiver and sim builder for robotics.

deepmind.googlearticlePublished and added 13 Oct 2026

EmoGenie is a benchmark for emotions expressed and evoked by text-to-image outputs. It has 6,400 images from eight T2I models using 800 emotion-targeted prompts. It includes EmoGenie-QA (90,000 closed-set and 1,057 open-ended questions) and EmoGenie-Assess (13,175 mean opinion scores for valence, arousal, dominance and aesthetic quality). The authors also propose EmoProbe, which pairs emotion QA with response-aligned, dimension-specific regression heads. They report that it beats baselines on emotion QA and correlates better with human ratings. The work shows how reliably MLLMs interpret affect in generated images.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

Proposes a vision-language-model framework that incrementally reconstructs heading hierarchies across tens or hundreds of pages using a level-wise multimodal memory. The memory keeps a compact set of earlier pages chosen by anchors (here, the most recent node at each heading level), so the VLM sees the visual, textual and layout patterns of headings at every level. The authors also introduce CLDoc-HHRBench (250 documents, 11,212 pages, expert-verified annotations) and report results on it and three existing benchmarks. It matters for RAG pipelines that need coherent document-level structure, which page-by-page OCR/VLM parsing doesn't give.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026

PDF TO MP3 is a hosted MCP server (Streamable HTTP, account authentication) that turns PDFs into audiobooks. The user's existing AI subscription, such as Codex, handles document processing through the MCP connection. Speech generation is billed separately through the user's own OpenAI or Google text-to-speech API key. The setup is three steps: add the MCP server, add a speech key, then attach a PDF and ask the assistant to use the MCP. It shows a "bring your own subscription" pattern that keeps the vendor's compute costs low.

pdftomp3.comtoolPublished and added 13 Oct 2026

The paper studies regressive sycophancy under fabricated evidence: a medical VLM gives a correct answer, then abandons it when the user pushes back with a fake prescription. It tests four open VLMs on 400 VQA-RAD yes/no questions in 19,200 multi-turn conversations. Without a synthetic-document disclosure, models abandon 80.1% of 6,488 correct answers, and the same prescription raises Qwen2.5-VL-7B's flip rate from 45.9% to 89.6% while lowering Qwen3-VL-8B's from 77.7% to 50.9%. Models often reject the prescription yet keep the correct answer in only some conversations (915 of 3,232), and 40% of those flip later. System-prompt disclosure of synthetic documents cuts one model's flip rate from 77.8% to 47.3%. The authors conclude that recognising a forgery does not stop a model from adopting the forged answer.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026