JADE (Joint Action DEnoiser) is a diffusion-based vision-language-action architecture in which the VLM reads the current noisy action latents together with images and instructions at every denoising step, and guides the action expert through gated cross-attention. The VLM is also trained to recover clean action latents, and an adaptive step-size schedule cuts sampling steps. On RoboDojo it averages 12.67% success, the highest among ฯ0.5-based methods, and it adds 20.8 points over ฯ0.5 on four real bimanual tasks.
vision-language-action modelvision-language modeldiffusion model
JADEฯ0.5RoboDojo+2
research, multimodalยท Added 3 Nov 2026, published 31 Oct 2026
The paper audits six situational-safety benchmarks for multimodal LLMs and proposes the Benchmark Card, seven measured fields that show whether a SAFE/UNSAFE label score comes from the scene, the request or the benchmark files. MSSBench chat leaks its label through image size, which alone reaches 82.3% accuracy. The Spot-Stop decomposition splits ranking-versus-labelling gaps across fourteen open MLLMs, and label-free recentering improves ranking on VSCBench for 13 of 14 models.
vision-language modelLLM
MSSBenchMOSSBenchHoliSafe+2
evals, ai-safety, multimodal, researchยท Added 3 Nov 2026, published 31 Oct 2026
RADIOlogy is a training-free, single-pass zero-shot anomaly detection method that uses multi-head attention pooling in vision-language models. It scores anomalies by contrasting average similarity between vision features and sets of normal and abnormal prompts, with no windowed execution. RADIO agglomerative features drive a dense refinement pipeline for pixel-level segmentation, and the method is applied to medical images.
CLIPRADIOvision-language model
RADIOlogyarXiv
research, multimodalยท Added 31 Oct 2026, published 29 Oct 2026
PhysCount is a benchmark of 546 real-world images and 4,595 human-written questions in 651 target-specific groups. It tests whether multimodal LLMs can separate visible appearances (direct view, reflections, glass) from the physical entities behind them. Across 40 MLLMs, counting appearances is much easier than linking them to entities, and the best model answers every question correctly in only 13.36% of groups.
vision-language model
PhysCountarXiv
evals, multimodal, researchยท Added 31 Oct 2026, published 29 Oct 2026
inkGround is a benchmark for stroke trajectory recovery from full-page, camera-captured handwriting, with 537 page images (about 18.7k words, 163k strokes) and pseudo-label references filtered by OCR agreement and human raters. Models get a page image and a target word, then output ordered stroke points. Seven VLMs were scored with DTW, Chamfer distance, readability and style similarity. Closed-source models, notably Gemini-3.1-Pro, did best, and the ranking held on IAM-OnDB.
vision-language model
inkGroundIAM-OnDBGemini-3.1-Pro+1
evals, multimodal, researchยท Added 30 Oct 2026, published 28 Oct 2026
Beyond the Event Horizon of Keccak extends a 2011 observation that a path in the Keccak sponge from a zero-capacity state back to one yields infinite second-preimage families. Three theorems cover loops from any capacity value x, k-block loops and cycles, and Joux-style multicollisions with search cost independent of k. Verified examples run on Keccak-f[200] and Keccak-f[1600] with c=32; it also proposes bridge-based protocols.
Know, then Draw is a learning paradigm for visual generation that uses autoregressive pretraining to acquire semantic knowledge and diffusion post-training for high-fidelity rendering. Controlled comparisons find autoregression the stronger pretraining objective, and semantic quality matters more than AR-only generation quality. This motivates a purely semantic tokenizer trained with language supervision only. A 3B text-to-image model trained in 6,600 H100 GPU-hours scores 86 on GenEval and 87 on DPG-Bench.
diffusion modelTransformer
arXivGenEvalDPG-Bench+1
research, multimodal, trainingยท Added 28 Oct 2026, published 27 Oct 2026
This paper presents an annotated egocentric tennis dataset (egocentric video synced with five allocentric views, global positions, contact times, stroke types) and evaluates image encoders, video encoders and vision-language models on seven prediction tasks. Best models reach 0.14 m player and 0.54 m ball localization error and 0.14 s contact-time error. Depth input helps 3D ball localization, and longer histories do not help forecasting. A conditional diffusion model generates whole-body tennis motion in real time.
vision-language modeldiffusion modelTransformer
arXiv
research, multimodal, evalsยท Added 28 Oct 2026, published 27 Oct 2026
This paper studies how multimodal LLMs refuse questions that an image cannot answer, using the MoHoBench benchmark on three MLLMs. It varies the meaning and placement of refusal instructions, compares supervised and preference-based post-training, and intervenes on a refusal-related activation direction. Instructions change refusal at the sample level in ways aggregate rates hide. Post-training raises refusal most but can mix refusal with attempted answers. Activation intervention gives smaller, transferable changes.
vision-language modelLLM
MoHoBencharXiv
ai-safety, multimodal, researchยท Added 28 Oct 2026, published 27 Oct 2026
Falko Strenzke's ePrint 2026/2635 presents a multi-target second-preimage attack on message hashing in HSS/LMS and XMSS with SHA-256 and SHA-256/192. It herds chaining states from many collected signatures into one state, defeating the key and leaf-index prefix. With 2^64 signatures, SHA-256 cost falls from 2^256 to about 2^166 (2^156 at maximal message length). SHAKE256 parameter sets are unaffected.
OWLS (Open-world Language-conditioned Speed Fields) plans directly over a vision-language model's dense feature map. A text prompt describing traversable terrain scores every pixel of one camera image. The scores are projected onto the ground, adapted with two morphological operations, and used as the speed field of a minimum-time planner. It needs no training, terrain labels, map or extra sensors. It was tested on a physical rover, on simulated Mars and Moon rovers over 200 procedural terrains, and on real planetary imagery.
vision-language model
OWLSarXiv
research, multimodalยท Added 26 Oct 2026, published 23 Oct 2026
DeepFaune-UK is a fine-tune of the open DeepFaune camera-trap classifier on 28 UK classes, paired with MegaDetector v6. On 20,212 MammalWeb images it reached 88.8% strict accuracy against 56.5% (DeepFaune v1.5.0) and 59.0% (SpeciesNet). Re-applying the recipe to the DINOv3-based classifier gave 90.6%. Fine-tuning on about 150 crops per class matched full-data out-of-distribution accuracy, and a DINOv2 run took about 22 minutes on under 4 GB of GPU memory.
DeepFauneMegaDetectorSpeciesNet+2
DeepFaune-UKMammalWeb
research, training, multimodalยท Added 26 Oct 2026, published 23 Oct 2026
UAV-SwarmAgent is a hierarchical multi-agent system where an operator gives one natural-language sentence and a ground-station agent writes and runs code that produces per-UAV routes. A vision-language model grounds targets in an aerial map, and worker agents verify incidents with a compact onboard model. In simulation it located the incident in 244 of 255 missions (95.7%), and flight time followed an Amdahl-type law with speedup bounded at 1.52x for one to five UAVs.
vision-language modelLLM
UAV-SwarmAgentarXiv
research, agents, multimodalยท Added 26 Oct 2026, published 23 Oct 2026
Decode-time provenance binding is a defense for Model Context Protocol agent pipelines against cross-modal trust laundering, where instructions hidden in an email-signature image via LSB, DCT or metadata steganography get decoded by a tool. It combines HMAC-authenticated trust labels, a provenance ledger and context-taint tracking to block critical tool calls. On a real MCP testbed with 24 adaptive payloads, attack success fell from 100% undefended (62.5% for keyword and regex filters) to 0%.
F2G-Pose is a direct, single-pass framework that predicts category-level object pose, metric size and camera-frame completed shape from one RGB-D image. It lifts frozen RADIO foundation features onto partial point clouds, tokenizes local geometry with DGCNN, and fuses it with a Mixture-of-Experts Transformer. It trains only on synthetic SOPE data, leads on SOPE and ROPE at 28 FPS, and is tested on HouseCat6D, HANDAL and real manipulation.
TransformerMixture of ExpertsDGCNN+2
F2G-PoseSOPEROPE+4
research, multimodalยท Added 24 Oct 2026, published 22 Oct 2026
TRIAD is a benchmark for omni-modal ambiguity resolution. It has 327 hand-written text-image-audio queries in 86 scene groups across 18 ambiguity categories, and removing any one modality makes the answer non-unique. Across 14 omni-modal systems, the best model trails humans by over 20 points in group accuracy. Without answer options, model group accuracy falls to single digits, and leave-one-out tests show audio cues are often underused.
multimodal large language modelomni-modal LLM
TRIADarXiv
evals, multimodal, researchยท Added 23 Oct 2026, published 22 Oct 2026
MPIDetector is a detection framework for multi-modal prompt injection that uses few-shot in-context learning so an LLM can reason about the semantic intent of text and image inputs, without fine-tuning. The authors built a benchmark of eight textual and seven visual attack categories plus benign sets. They report up to 90.91% accuracy on text, 99.28% on images, and up to 99.80% detection on 500 obfuscated WebInject web-agent injections, beating baselines.
GeminiQwenLLM
prompt injectionWebInjectMPIDetector
AI Securityยท Added 24 Oct 2026, published 21 Oct 2026
AnomalyAgent is a post-training fusion framework for visual anomaly detection that ensembles pretrained detectors. It aligns scores and anomaly maps, summarizes disagreement, and treats fusion as a choice among score actions and map actions. A constrained vision-language model inspects high-conflict samples and picks the actions. It beats equal-weight fusion and gets the best mean image AUROC and pixel AP among compared fusion methods on MVTec AD and VisA.
vision-language model
AnomalyAgentMVTec ADVisA dataset+2
research, multimodal, agentsยท Added 22 Oct 2026, published 20 Oct 2026
Code2Video-Compass is a benchmark of 2,104 tasks for code-based video generation, where multimodal LLMs write executable programs that produce complete videos from text (T2V), image (I2V) or video (V2V) inputs. Tasks span 10 categories and 24 subtypes. Scoring uses 12 objective metrics and five MLLM-judge metrics. Across 15 proprietary and open-weight models from nine families, performance falls as construction, dynamic animation and state-dependency complexity rise.
vision-language modelLLM
Code2Video-CompassarXiv
evals, multimodal, researchยท Added 22 Oct 2026, published 20 Oct 2026
This paper presents a multimodal engineering GraphRAG framework that links post-wind-event damage evidence from images and reconnaissance reports to building-condition states, recovery actions and downtime. Evidence is stored as a provenance-preserving, component-level representation, and graph-aligned retrieval uses compatibility, contradiction, evidence-precedence and abstention checks. Demonstrated on tornado-affected low-rise homes, it beats BaselineRAG and Structured Baseline RAG, and ablations test each component.
RAGGraphRAG
BaselineRAGStructured Baseline RAGarXiv
research, llm-apps, multimodalยท Added 22 Oct 2026, published 20 Oct 2026
OpenVLHarness is a multimodal reasoning harness for MLLMs built on a capability layer (task-level tool interfaces plus evidence rendering) and persistent multimodal memory that stores images, structured data and web references as addressable artifacts. A coding interface runs generated programs over stored data. Across 22 benchmarks and eight backbones it adds 9.7-12.6 points over no-tool baselines, and GPT-6 Luna at medium effort beats Codex and Visual Sketchpad at maximum effort using at most 37% of their tokens.
vision-language modelQwen3-VLCodex+1
OpenVLHarnessKimi K3GPT-6 Astra+3
agents, multimodal, researchยท Added 21 Oct 2026, published 19 Oct 2026
FedRAPP is a personalized aggregation framework for Source-Free Federated Test-Time Prompt Tuning (SF-FTTPT), where clients adapt CLIP soft prompts on unlabeled, non-IID test streams without source data. It combines reliability-aware client assessment, a server-side exponential moving-average prompt memory and similarity-guided peer aggregation. Across nine CLIP benchmark datasets and ImageNet variants it beats standard federated aggregation baselines while keeping prompt-only efficiency.
CLIPvision-language model
FedRAPPImageNetarXiv
research, multimodalยท Added 20 Oct 2026, published 16 Oct 2026
LILIE (Language-Informed and Lossless Image Encoder) is an invertible image transform built from a normalizing flow transformer, trained jointly with a vision-language model via next token prediction so captions can be recovered from noised features. The encoder supplies a language-informed loss for text-to-image generation. JiT models trained with it cut gFD from 66.84 to 53.01 against a pixel-only baseline on the GPIC dataset. Code, encoders and models are released.
normalizing flowTransformervision-language model
LILIEJiTGPIC+1
research, multimodalยท Added 20 Oct 2026, published 16 Oct 2026
This paper presents a multimodal retrieval-augmented generation framework for Medical Visual Question Answering. A Gated Multimodal Unit learns a joint query representation from the medical image and the clinical question, so evidence retrieval uses both modalities while the Med-LVLM stays frozen. On benchmark datasets it consistently beats a reference Med-LVLM and stays competitive with state-of-the-art methods. The page gives no dataset names or numbers.
RAGvision-language model
arXiv
research, llm-apps, multimodalยท Added 20 Oct 2026, published 16 Oct 2026
P^3-WAM is a unified world action model for egocentric dexterous manipulation with three modes: Perceive (reconstruct articulated hand motion and camera trajectories from video), Predict (generate visual futures under prescribed actions) and Plan (jointly generate action trajectories and their anticipated visual outcomes from an initial image, a language instruction and an optional visual goal). It adds a compact structured 3D action representation to a pretrained video diffusion transformer through a mixture-of-transformers design with shared base weights, modality-specific adaptation and bidirectional visual-action attention. Randomized modality visibility lets video and 3D actions act as either context or target. Unified training beats single-mode training and outperforms specialized baselines on 3D hand reconstruction and action-conditioned video prediction across four datasets, and gives better instruction adherence than video-generation baselines.
diffusion modelTransformervision-language model
P^3-WAMarXiv
research, multimodalยท Added 17 Oct 2026, published 15 Oct 2026
GhostFaceMPC is presented as the first end-to-end face recognition system that runs both the deep CNN feature extractor and the matching step under secure multi-party computation. Raw images and biometric embeddings are never exposed at any stage, from feature extraction through storage to matching. The authors contribute an MPC-friendly quantization of the GhostFaceNet model and circuit optimizations that make this practical, and they release code on GitHub. It is relevant as an example of making CNN inference feasible under cryptographic constraints.
GhostFaceNetCNNsecure multi-party computation+1
GhostFaceMPC
research, inferenceยท Published and added 15 Oct 2026
GhostFaceMPC is presented as the first end-to-end face recognition system that runs a deep CNN feature extractor (GhostFaceNet) and the matching step under secure multi-party computation. Raw images and biometric embeddings are never exposed during extraction, storage or matching. The authors contribute an MPC-friendly quantization of GhostFaceNet and circuit optimizations that make the approach practical, and they release code at crypto-unict/GhostFaceMPC.
MiniDog is a framework for training a text-to-image flow-matching diffusion transformer from scratch on four RTX 3090 GPUs, with 55 minutes of pretraining and 15 minutes of fine-tuning. It pretrains on 26k ImageNet dog images that were captioned, then fine-tunes on 2k FLUX.1-schnell generated images. Image and text features are precomputed and cached. The model is a 22M-parameter transformer with the E2E-INVAE semantic latent space, a small Qwen text encoder and in-context conditioning, and training is accelerated with iREPA and an EUPE perception encoder. The repo evaluates with FID, HPSv2, PickScore and VAE reconstruction metrics. It lowers the barrier to hands-on diffusion training research.
flow matchingdiffusion modelTransformer+1
MiniDogImageNetFLUX.1-schnell+7
research, training, multimodalยท Added 14 Oct 2026, published 13 Oct 2026
Proposes mmVB, a lightweight plug-and-play module (1.6M trainable parameters) that assigns query-conditioned per-image visual token budgets using cross-image context for multimodal LLMs, without retraining the base model or accessing attention matrices (so it is compatible with FlashAttention). It cuts visual input tokens 9x, giving 5x faster time-to-first-token and 7x fewer prefill FLOPs while keeping 94.4% of full-resolution accuracy across eight multi-image reasoning benchmarks. It matters because existing token pruning, merging and compression methods target single-image redundancy and either need attention maps or retraining.
FlashAttentionvision-language model
mmVB
research, multimodal, inferenceยท Added 14 Oct 2026, published 13 Oct 2026
PixelMoE is a pixel-space diffusion framework that scales capacity sparsely using standard LLM-style Mixture-of-Experts: token-choice routing, fine-grained routed experts, a shared expert and bias-based load balancing, with no diffusion-specific routing or auxiliary routing losses. It adds Pixel-Space Hierarchical Flow Matching, which varies spatial resolution along the denoising trajectory from global structure to fine detail. At matched active parameters it beats dense counterparts, and reaches FID 1.47 at 256x256 and 1.53 at 512x512 on class-conditional ImageNet-1K without representation alignment or perceptual losses. Code and models are promised but not yet released.
Mixture of Expertsdiffusion modelflow matching+1
PixelMoEImageNet-1KarXiv
research, multimodalยท Added 14 Oct 2026, published 13 Oct 2026
arxiv.orgpaper๐ง AI
