Introducing AnySpeechRecognizer and AnyDecisionModel: Open Models for Speech and Decisions on Apple Platforms

What it claims

AnySpeechRecognizer re-implements Apple's SpeechAnalyzer API from macOS 26 and iOS 26: SpeechAnalyzer, SpeechTranscriber, DictationTranscriber, SpeechDetector and AssetInventory. Apple's engine is the default, so switching from import Speech to import AnySpeechRecognizer is enough for code that uses the shared API. It offers these alternative engines:

• Parakeet v3 (NVIDIA TDT 0.6B v3): 25 European languages with automatic language detection.
• Parakeet Ultra: Moondream's post-trained v3, which was more accurate in most of the authors' measurements.
• Parakeet Unified: English-only.
• Qwen3-ASR (0.6B and 1.7B): 30 languages and 22 Chinese dialects, including Mandarin, Cantonese, Japanese and Korean, which Parakeet does not cover.

AnyDecisionModel reads answers from model probabilities rather than parsing generated text.

How it works

The Parakeet models run on the package's own Core ML runtime. It supports file and live input, volatile results, word timing and word confidence. Qwen3-ASR runs on the GPU and returns one final result per segment of up to 40 seconds, with no volatile results, word timing or confidence. The authors position it for files, not live input. An optional 8-second Parakeet encoder for live input cuts live analysis time by about a third, at a somewhat higher word error rate. Each Hub repository records its source checkpoint and a per-file checksum, and the package pins a fixed revision. The conversion workflows mostly follow FluidInference's mobius scripts.

SpeakerDiarizer runs NVIDIA Streaming Sortformer (v2 by default; v2.1 has more meeting training data) alongside an Apple or Parakeet transcriber on the same timeline. It tracks up to four speakers, including overlapping speech. SpeakerTimeline assigns each timed word to a speaker.

In AnyDecisionModel, a DecisionSession holds one text or JSON state. Questions are independent of each other, and decide(_:) batches them. The next-token backends give each allowed answer a one-token label and read the label probabilities from one forward pass. Labels are "yes"/"no", A to Z, or 0 to 9, so a choice can have at most 26 options and a score at most 10 levels. MLXHeadDecisionModel instead reads Cloudflare Clef's trained decision head (head: .jointSchema), which has no option limit.

Several features help check the probabilities:

• Calibration: applies a temperature to every answer and a bias to yes/no answers.
• rotationDebiasing: asks a choice question once per rotation of the options and averages the results, at one prompt evaluation per rotation. It reduces position bias.
• Confidence: based on entropy, so it measures how concentrated the probabilities are, not whether the answer is right.
• Diagnostics: report how much probability fell on the allowed answers.

Evidence

All measurements come from an M4 Max Mac Studio on macOS 27.0.1:

• Parakeet speed: one hour of audio takes about 20 to 40 seconds from a file and 1.5 to 2.5 minutes live, using about 100 MB of memory after Core ML compiles the model.
• Qwen3-ASR: much slower, and needs a few gigabytes of memory.
• Meetings: on AMI meeting recordings, every engine tested had a word error rate above 23%.
• Missing table: the LibriSpeech word error rate table (3 to 8 minute files) is not in the fetched text.

For decisions, the Emoji Magnet demo asks 343 yes/no questions per phrase:

• Hosted Jev: answers all 343 in one request in about 0.5 seconds.
• Local MLX with Qwen3 4B: uses prefix caching and answers 16 questions per forward pass. Each group of 16 takes about 0.45 seconds after the model loads, and a full pass takes about 10 seconds.

How to use it

dependencies: [
    .package(url: "https://github.com/mattt/AnySpeechRecognizer", from: "0.1.0"),
    .package(
        url: "https://github.com/mattt/AnyDecisionModel",
        from: "0.6.0",
        traits: ["MLX"]
    ),
]

Requirements:

• Both packages: Swift 6.2 or later and macOS 26 or iOS 26 or later.
• Apple silicon: needed for the Parakeet, Sortformer and MLX runtimes.
• MLX build: needs Xcode's Swift 6.4 or later and the Metal toolchain.
• Core AI: needs Xcode 27 and an OS 27 runtime.

Without traits, AnyDecisionModel has no third-party dependencies and includes the Jev and Ollama backends, which also build on Linux. The MLX, CoreAI and Llama traits enable the local backends. Ollama requires version 0.35.0 or later, or 0.40.5 or later for cloud models.

Download sizes:

• Each Parakeet: about 600 MB, plus about 590 MB for the live encoder.
• Qwen3-ASR: 0.9 to 2.4 GB.
• Sortformer: about 250 MB.
• Clef-Flash 4-bit (9B): 6.2 GB. Its model card recommends at least 16 GB of memory.

Files go in the standard Hugging Face cache, which unsandboxed Mac apps share with Python tools. The hear command-line tool needs --backend apple, --backend parakeet or --backend qwen. Running hear prepare with the same options installs the assets, and hear evaluate scores your own files.

Limits

The speech package has been tested only on macOS 27, not on an iPhone. Both packages are at early versions (0.1.0 and 0.6.0). AnyDecisionModel's CoreAI trait cannot be combined with AnySpeechRecognizer, because coreai-kit 0.7.3 requires swift-huggingface below 1.0 and AnySpeechRecognizer requires 1.x. The Clef backend accepts only text and JSON, although Clef itself reads images and video. The authors note that raw language-model probabilities are often overconfident, and they advise fitting calibration and confidence thresholds on separate datasets of your own. Benchmarks cover one machine and a few datasets. The page states no licences for the converted models.