Skip to content

Back to the catalog

JFrog: PixelSmash
jfrog.comwriteup

JFrog: PixelSmash

JFrog Security Research discloses CVE-2026-8461 ("PixelSmash"), a critical heap out-of-bounds write in FFmpeg's MagicYUV decoder caused by a rounding mismatch between chroma-plane allocation and slice-height calculation. The team escalates the OOB write into reliable remote code execution by hijacking the AVBuffer.free function pointer to call system(), demonstrating full RCE via a 50KB crafted AVI against a Jellyfin media server (through automatic library/ffprobe scanning) and a Nextcloud instance (via video thumbnail preview generation), plus DoS crashes across mpv, Kodi, OBS Studio, Immich, PhotoPrism and vLLM. The post frames the bug as a software supply-chain issue since libavcodec is a silent transitive dependency in hundreds of downstream media applications, and covers detection, patching, and build-time workarounds.

TL;DR JFrog details CVE-2026-8461, a critical FFmpeg MagicYUV decoder heap overflow escalated to RCE against Jellyfin and Nextcloud via a single crafted video file.
Document

What it is

PixelSmash is CVE-2026-8461, a heap out-of-bounds write in FFmpeg's MagicYUV decoder (CVSS 8.8), disclosed by JFrog Security Research. The summary covers the RCE demonstrations; the blog additionally supplies a fixed version, a detection command, two workarounds, and the specific attack surface — which is why this document exists.

How it works

The bug is triggered by malformed slice_height values in a crafted MagicYUV bitstream, causing an out-of-bounds write into a heap-allocated frame buffer. A valid delivery container is AVI, MKV, or MOV. JFrog escalated the crash to reliable RCE; the patch rejects the malformed slice heights, and the page notes the reference MagicYUV encoder always emits aligned slice heights, so the check only rejects malicious input.

Preconditions / Affected

The MagicYUV decoder is enabled by default in every upstream FFmpeg build and every distro package tested (Ubuntu, Debian, Fedora, Arch, Alpine) before version 9.0. Exploitation requires only delivering a media file to something that decodes video via libavcodec — no authentication, privileges, or prior access. Trigger paths include a user opening or even just browsing a folder containing the file (thumbnail generators used by GNOME/KDE/XFCE), server-side auto-processing (Jellyfin, Emby, Nextcloud, Immich, plus chat platforms Slack/Discord/Telegram and cloud transcoders AWS MediaConvert/Cloudflare Stream), and NAS/smart-TV appliances (Synology, QNAP).

What to do

Upgrade FFmpeg to the fixed version (9.0 or later). Detect the vulnerable decoder:

ffmpeg -decoders 2>/dev/null | grep magicyuv

Output containing VFS..D magicyuv means the build is vulnerable.

If you cannot upgrade immediately, disable the decoder at build time:

./configure --disable-decoder=magicyuv [your other flags]
make && make install

Or apply the minimal 7-line patch to libavcodec/magicyuv.c:

     if (s->slice_height <= 0 || s->slice_height > INT_MAX -  avctx->coded_height) {
         av_log(avctx, AV_LOG_ERROR, "invalid slice height: %d\n",    s->slice_height);
         return AVERROR_INVALIDDATA;
     }
+    if ((s->slice_height >> s->vshift[1]) <= s->interlaced) {
+        av_log(avctx, AV_LOG_ERROR, "impossible slice height\n");
+        return AVERROR_INVALIDDATA;
+    }
+    if ((avctx->coded_height % s->slice_height) && ((avctx->coded_height % +        s->slice_height) >> s->vshift[1]) <= s->interlaced) {
+        av_log(avctx, AV_LOG_ERROR, "impossible height\n");
+        return AVERROR_INVALIDDATA;

Caveats

The downstream projects (Jellyfin, mpv, Nextcloud, Immich, OBS, vLLM, and others) did not introduce the bug and, per JFrog, "most have no mechanism to detect or mitigate it independently" — so patching a single application is insufficient if other bundled/linked FFmpeg copies exist on the same host. Full RCE was demonstrated against Jellyfin and Nextcloud; the other listed products were confirmed to crash, not proven exploitable to RCE.

CVEs named here

  • CVE-2026-8461

Technologies named here

  • FFmpeg
  • GDB
  • AddressSanitizer
  • strace
  • Nextcloud
  • JFrog Security Research Center
  • OBS Studio
  • vLLM
  • Emby (crash confirmed)
  • ffmpegthumbnailer (crash confirmed, used by GNOME, KDE, XFCE)
  • ffprobe
  • Immich (crash confirmed)
  • Jellyfin
  • Jellyfin 10.11.9 (RCE demonstrated via library scan)
  • jellyfin-ffmpeg 7.1.3 (Jellyfin's bundled FFmpeg, RCE demonstrated)
  • Kodi (crash confirmed)
  • MagicYUV
  • mpv (crash confirmed)
  • PhotoPrism (crash confirmed)
  • Plex (builds FFmpeg with a minimal decoder allow-list, used as a defensive comparison)

Techniques named here

  • Heap-based Buffer Overflow
  • Reverse shell
  • Function pointer overwrite (AVBuffer.free hijack)
  • Heap layout calibration
  • Zero-click delivery via monitored watch folder (torrent download)

Concepts named here

  • ASLR
  • Cost amplification in cloud pipelines
  • Decoder allow-listing
  • Silent heap corruption
  • Software supply chain vulnerability