Fast Accuracy-Preserving RNNT Beam Search on Apple Silicon

Collective Library edition. This is the complete technical report. Private filesystem paths, internal run identifiers, campaign-control notes, and repository navigation were removed. Technical claims, code, measurements, evidence labels, citations, corrections, and falsification criteria are preserved.

Date: 2026-09-06
Status: Ingested external field guide; Argmax claims normalized against public primary sources, the MIT-licensed local Parakeet v2 and v3 Core ML artifacts inspected directly, Core ML APIs checked through current coremltools documentation, and malsd_batch semantics corrected against pinned NeMo Speech source

Executive Answer

Argmax publicly demonstrates a fast Parakeet TDT deployment, not a fast beam decoder for the reference implementation's Parakeet Unified RNNT. Argmax reports more than 100 seconds of audio transcribed per wall-clock second on an 8 GB M1 MacBook Air, 10+ TFLOPS on the Apple Neural Engine for Parakeet v2, and 15+ TFLOPS for a ten-minute Parakeet v3 workload on an iPhone 16 Pro. It also reports a 476 MB compressed v2 package and a 494 MB v3 package. Those are vendor measurements of an encoder-heavy TDT system. Public material does not identify its search algorithm or beam width, but the inspectable MIT-licensed v2 and v3 artifacts now settle the model-side implementation: four static Core ML programs, 6-bit-palettized encoder weights, ANE-friendly rank-4 encoder tensors, both joint projections hoisted out of the recurrent joint call, explicit FP16 LSTM state, and a batch-one host-driven search boundary.

TDT contributes real but bounded architectural leverage. Its duration head can jump over encoder frames; the TDT paper reports up to 2.82× faster ASR inference than a conventional transducer on its experiments.[4] That cannot explain a 100× speed factor by itself, and it cannot be copied into frozen Unified RNNT weights. The artifact shows the transferable model split and compression choices. The SDK's host search algorithm and benchmark inclusion boundary remain undisclosed.

For the reference implementation's RNNT, the shortest credible exact path is:

1. Hoist the encoder and predictor projections out of the joint inner loop. NeMo's current malsd_batch does this already; the reference implementation's current JointDecisionSingleStep recomputes both projections on every call.
2. Export fixed-batch FP32 predictor and projected-joint graphs for the selected beam width, initially width 8. Never dispatch one Core ML prediction per hypothesis.
3. Reproduce the pinned malsd_batch state machine exactly: top-K per hypothesis, global K-of-K² pruning, max-symbol blank forcing, pointer-based history, post-prune duplicate recombination by log-sum-exp, blank-state preservation, and final score normalization.
4. Keep state tensors explicit. MLState is a poor first fit for branching hypotheses; output backings are an optional measured optimization, not a zero-copy guarantee.
5. If two Core ML calls per alignment step miss the budget, test one fixed-shape fused transition-plus-joint graph or a native CPU recurrent kernel. Do not start with custom kernels.

The most important new implementation observation is not an exotic beam algorithm. It is projection hoisting: the current runtime repeatedly pays work that depends only on the encoder frame or current token prefix. Remove that first.

Evidence Labels

1. What Argmax Demonstrably Did

1.1. Public performance claims

The raw report asserted a roughly 66 MB memory footprint. No supporting Argmax primary source was found, and it conflicts with the public 476/494 MB package sizes if read as model residency. It is excluded.

1.2. Architecture is part of the speed result

Parakeet v2 and v3 are TDT models. A TDT predicts a token distribution and a duration distribution, then advances by the selected duration instead of necessarily visiting the next encoder frame. The original paper reports up to 2.82× faster ASR inference than its conventional transducer baselines.[4] NVIDIA's explanation is concrete: long predicted durations skip blank-heavy regions.[6]

Three conclusions follow:

• TDT frame skipping is not an RNNT beam optimization.
• The paper's “up to 2.82×” does not explain Argmax's full speed factor.
• A 0.6B encoder at high ANE utilization can dominate the public throughput number while saying little about the recurrent decoder.

1.3. What the MIT-licensed v2 and v3 artifacts reveal

Inspectable Argmax artifact. Both inspected LICENSE_NOTICE.txt files now identify the weights as MIT licensed and permit reverse engineering and modification. Each variant is four compiled ML Programs with readable metadata.json and MIL. All eight programs target Core ML specification version 8 with iOS 17/macOS 14 availability. The v3 conversion metadata records TorchScript conversion with PyTorch 2.5.0 and coremltools 9.0b1 on 2025-08-14; v2's decoder, joint, and mel graphs record coremltools 8.2 and PyTorch 2.6.0, while its encoder records coremltools 8.3.0 and PyTorch 2.5.0.

The two variants share the same split and the same 15-second, 16 kHz window (audio [240000] → mel [1,1,1501,128] → encoder frames [1,1024,1,188] plus projected encoder [1,640,1,188]). They differ in vocabulary size, joint width, and how aggressively the frontend is palettized:

The v3 dimensions are strong structural evidence for a TDT joint: 8,193 embedding rows versus 8,198 joint outputs—five extra values consistent with duration choices. v2 is the same transducer split with a small-vocab joint: 1,025 embedding rows versus 1,030 joint outputs, also five extra classes. Neither artifact includes a tokenizer or decoder configuration, so exact blank, token, and duration semantics—and how the host separates or renormalizes their scores—remain unconfirmed.

The useful reverse-engineered facts are:

1. Projection hoisting is real on both variants. Argmax computes the encoder projection once for the whole utterance and the predictor projection once per decoder update. Its recurrent joint graph never repeats either 640-wide projection.
2. The encoder is shaped for Core ML's image-oriented execution path. Every major encoder tensor is rank 4, with sequence in the last dimension; dense channel projections are represented as 1×1 convolutions.
3. Compression is concentrated in the heavy path. Both encoders use 64-entry lookup tables—6-bit palettization. v3 also palettizes the mel frontend; v2 leaves mel dense FP16. The small recurrent decoder and joint stay plain FP16 on both.
4. The Core ML graphs contain no search. There is no argmax, topk, beam state, pruning, or prefix operation. MultimodalLogits returns the full vector to host code.
5. The decoder interface is batch one. The artifacts expose no hypothesis-batch dimension, so they are not evidence of a batched Core ML beam implementation.
6. The advertised file throughput uses a fixed 15-second model window. Long-file speed can amortize model load and host setup over many windows; it is a different boundary from the reference implementation's 30-second end-of-audio finalization gate.

These facts explain a substantial part of the speed without inventing an SDK algorithm: compact accelerator-friendly encoding, projections removed from the recurrent hot loop, a small FP16 host-driven decoder boundary, and TDT frame skipping.

1.4. What remains unknown

Public evidence does not settle:

• greedy, beam, or another TDT search;
• beam width, pruning, score normalization, or maximum symbols;
• native Swift, C++, Metal, or mixed host bookkeeping around the four Core ML programs;
• whether tokenization, punctuation, timestamps, and segmentation are inside every speed number;
• cold load and specialization cost;
• per-component latency and memory.

Do not replace those unknowns with a story. Inspect the model graph for model-side facts; measure the SDK end to end only when the same audio, device, output contract, and timing boundary can be used.

2. The Actual malsd_batch Target

2.1. Pin implementation, not algorithm name

The selected quality result uses NeMo malsd_batch, but an algorithm name is not a conformance specification. In pinned NeMo Speech 3.1.0+95f92737cf, ModifiedALSDBatchedRNNTComputer states two modifications to the paper:

• it does not support prediction-network caching;
• it does not estimate target length; it limits label expansions per frame.[7]

The raw report described a trie-based predictor cache as mandatory. That is incorrect for the pinned target. BatchedBeamHyps stores dense tensors of current labels and predecessor beam pointers, plus transcript hashes used for duplicate detection. The history is tree-like, but the MALSD computer does not cache predictor results by prefix.[8]

2.2. One pinned iteration

For batch B, beam K, vocabulary-plus-blank V, and projected width 640, the source performs:

project encoder once                                      [B,T,640]
initialize predictor for B*K hypotheses                  [B*K,1,640]

while any hypothesis is active:
    gather one projected encoder frame per hypothesis    [B*K,1,640]
    joint-after-projection                                [B*K,1,1,V]
    log_softmax in the active floating dtype              [B,K,V]
    top K labels for each hypothesis                      [B,K,K]
    add previous hypothesis scores                        [B,K,K]
    force blank where max-symbol limit fired
    select global top K from K*K candidates               [B,K]
    append labels and predecessor pointers
    recombine identical surviving transcripts by log-sum-exp
    gather parent predictor states
    run predictor for all B*K selected labels
    restore previous output/state for blank transitions
    advance acoustic time only on blank

This ordering matters. NeMo prunes K² candidates to K before recombining identical surviving hypotheses. Recombination sets all but the highest-scoring representative inactive and log-sum-exp merges their scores; it does not refill the vacated beam slots from discarded candidates.[7][8] An implementation that merges before pruning is a different decoder even if it looks mathematically cleaner.

2.3. Equality and recombination

For RNNT, pinned BatchedBeamHyps.recombine_hyps_() considers hypotheses equal when all three match:

• transcript hash;
• last label;
• non-blank transcript length.

It computes torch.logsumexp over equal scores and keeps the slot that previously held the highest score.[8] Hash collision is theoretically possible. A Swift port should either reproduce the hash and verify tokens on collision, or compare pointer-chain tokens directly for the at-most-eight survivors. With K=8, collision-safe verification is cheap.

2.4. Configuration fields that can change output

Freeze these in every receipt:

BeamRNNTInfer and BeamBatchedRNNTInfer are different code paths. Beam size one is also not the campaign's greedy_batch; NeMo warns that cached beam-one greedy may differ from the dedicated greedy decoder.[9]

2.5. Test the state machine with a scorer, not a static logit tensor

An RNNT does not have one prefix-independent [T,V] logit matrix. Its joint logits depend on acoustic time and predictor state, which depends on the emitted prefix. The raw report's proposed “extract raw log_probs once, then feed both decoders” test cannot establish RNNT beam parity.

Use a deterministic toy scorer instead:

class ToyRnntScorer:
    def log_probs(self, time_index: int, token_prefix: tuple[int, ...]) -> list[float]:
        return self.table[(time_index, token_prefix)]

Run the pinned Python state machine and the Swift state machine against the same table. Include fixtures for:

• blank versus label ties;
• two alignments reaching the same transcript;
• recombination after pruning;
• max-symbol blank forcing;
• an inactive final hypothesis retained in slot zero;
• final score normalization changing the winner;
• hash collision fallback.

Only after the toy state machine matches should model numerics enter the experiment.

3. Delete Repeated Projection Work First

3.1. Current the reference implementation hot path

Implementation artifact. scripts/coreml/components.py currently implements a full joint wrapper:

enc_proj = self.module.enc(encoder_outputs)
dec_proj = self.module.pred(decoder_outputs)
x = enc_proj.unsqueeze(2) + dec_proj.unsqueeze(1)
x = self.module.joint_net(x)

JointDecisionSingleStep invokes that wrapper for each decoder decision. Therefore:

• the same encoder frame is projected from 1,024 to 640 for every label expansion at that frame;
• the same predictor output is projected from 640 to 640 until its prefix changes;
• the graph computes softmax, argmax, and top 64 for a greedy loop that needs less;
• the interfaces are batch one.

This is acceptable as a first correct exporter. It is the wrong boundary for a fast beam loop.

3.2. NeMo's source already shows the split

Pinned malsd_batch calls joint.project_encoder() once before its loop and joint.project_prednet() after predictor updates. The loop calls only joint_after_projection().[7] This is exact at the algorithm level and transfers directly to Core ML.

The target graph split is:

audio -> mel -> encoder + encoder projection once
                    |
                    +--> projected encoder [1,T,640]

selected tokens + parent states
    -> batched predictor + predictor projection [K,640] + new states

projected encoder frames [K,640] + projected predictor [K,640]
    -> add -> ReLU -> final linear -> log-softmax -> top K + blank score

Moving operations across Core ML graph boundaries can change floating-point fusion and accumulation. “Same PyTorch operators” is not a numerical parity result. Compare component outputs and transcript identities before calling the split exact.

3.3. Minimal fixed-batch export shape

Start with one model per selected beam width. Separate fixed models are simpler to compile, profile, and reject than a multifunction or flexible-shape package.

class ProjectedPredictor(torch.nn.Module):
    def __init__(self, decoder, joint):
        super().__init__()
        self.decoder = decoder
        self.pred_projection = joint.pred

    def forward(self, tokens, lengths, h, c):
        y, _, (h_new, c_new) = self.decoder(
            targets=tokens.long(),
            target_length=lengths.long(),
            states=[h, c],
        )
        projected = self.pred_projection(y.transpose(1, 2)).squeeze(1)
        return projected, h_new, c_new

class ProjectedJointTopK(torch.nn.Module):
    def __init__(self, joint, beam_size: int, blank_idx: int):
        super().__init__()
        self.activation = joint.joint_net[0]
        self.output = joint.joint_net[2]
        self.beam_size = beam_size
        self.blank_idx = blank_idx

    def forward(self, encoder_projected, predictor_projected):
        logits = self.output(self.activation(encoder_projected + predictor_projected))
        log_probs = torch.log_softmax(logits, dim=-1, dtype=torch.float32)
        scores, ids = torch.topk(log_probs, k=self.beam_size, dim=-1)
        return scores, ids.to(torch.int32), log_probs[:, self.blank_idx]

For beam 8, trace fixed inputs:

inputs = [
    ct.TensorType(name="tokens", shape=(8, 1), dtype=np.int32),
    ct.TensorType(name="lengths", shape=(8,), dtype=np.int32),
    ct.TensorType(name="h_in", shape=(2, 8, 640), dtype=np.float32),
    ct.TensorType(name="c_in", shape=(2, 8, 640), dtype=np.float32),
]

predictor = ct.convert(
    traced_predictor,
    convert_to="mlprogram",
    inputs=inputs,
    compute_precision=ct.precision.FLOAT32,
    minimum_deployment_target=ct.target.iOS17,
)

Current coremltools also supports ct.EnumeratedShapes directly on ct.TensorType for a finite shape set.[10] Do not add it initially. Coupled token and state batch dimensions are easier to reason about as separate fixed packages, and only the development winner needs to ship.

3.4. Keep top-K inside the joint graph

Returning [K,1,033] FP32 log probabilities moves about 33 KB per iteration at beam 8. Returning K top scores, K token IDs, and one blank score per hypothesis moves hundreds of bytes. The existing exporter already proves Core ML can compile topk.

Top-K inside the graph is therefore the first arm to test. Preserve:

• log_softmax before topk;
• the same FP32 arithmetic target;
• blank score even when blank is not in top K;
• deterministic tie handling checked on fixtures.

A native partial selector remains a fallback if Core ML's topk placement or latency is bad. Do not full-sort 8,264 values per iteration, and do not assume a named Accelerate routine is faster without a microbenchmark.

4. The Native Beam Loop

4.1. Fixed storage

At beam 8, use fixed-capacity storage:

scores                         Float[K]
time_indices                   Int32[K]
last_labels                    Int32[K]
nonblank_lengths               Int32[K]
symbols_at_current_time        Int32[K]
h, c                           Float[2,K,640]
predictor_projected            Float[K,640]
history token arena            Int32[max_steps,K]
history predecessor arena      UInt8[max_steps,K]
transcript hashes              UInt64[K]

The predecessor arena avoids copying token arrays when a hypothesis branches. Reconstruct only final transcripts. With eight survivors, recombination can compare hash, last label, and length first, then walk predecessor chains only on a potential match.

4.2. One model call per graph, not per hypothesis

The safe first port mirrors NeMo:

• one batched projected-joint call per alignment iteration;
• one batched predictor call after pruning;
• restore parent output/state for blank or inactive transitions;
• all gathers, score arithmetic, K² pruning, history updates, and recombination in one native loop.

Serially calling a batch-one decoder eight times is not a beam implementation worth benchmarking. It multiplies feature-provider construction, scheduling, output allocation, and dispatch.

4.3. Explicit states before MLState

Core ML stateful models use ct.StateType and require iOS 18/macOS 15-era state APIs.[11] A state object is naturally a mutable linear history. Beam search branches and reorders eight histories every iteration. Cloning opaque MLState objects is not the simple path, and using one shared state would serialize or corrupt branches.

Keep [2,K,640] hidden and cell tensors explicit until a physical experiment proves a stateful model helps. This preserves the iOS 17/macOS 14 floor and makes gather/scatter observable.

4.4. Output backings are a proposal

MLPredictionOptions.outputBackings can propose client-allocated MLMultiArray or pixel-buffer storage for named outputs. Core ML may ignore the proposal; Apple requires checking object identity. It is unsupported for batch prediction, though a normal prediction whose tensor has a batch dimension is not the batch-prediction API.[12]

let options = MLPredictionOptions()
options.outputBackings = [
    "h_out": hBacking,
    "c_out": cBacking,
    "top_scores": scoreBacking,
    "top_ids": idBacking,
]

let result = try model.prediction(from: input, options: options)
let usedHBacking =
    result.featureValue(for: "h_out")?.multiArrayValue === hBacking

Record usedHBacking rather than claiming zero-copy. Avoid touching a pixel-buffer-backed array's data pointer before prediction; doing so can force synchronization.[12]

4.5. A one-dispatch exact candidate

If the two-call loop misses latency, test a fixed-shape graph that accepts:

• selected labels;
• parent hidden/cell states;
• parent projected predictor output;
• a blank/inactive preserve mask;
• selected projected encoder frames.

Inside one graph:

1. run the predictor on all K labels;
2. select parent versus new state/output by the preserve mask;
3. run predictor projection;
4. run projected joint, FP32 log-softmax, top-K, and blank extraction.

This reduces the steady-state path to one Core ML dispatch per alignment iteration. It is algorithmically equivalent to NeMo's run-then-restore sequence. Core ML may fuse or round differently, so token-level parity—not operator inspection—decides whether it is acceptable.

5. Runtime Choices

There is no source-backed universal winner. The encoder and recurrent loop are different workloads. The ANE residency guide shows why placement and speed must be measured separately on each device. The small-recurrent-graph dispatch guide covers the per-call dispatch floor by compute unit and the measured FP32-versus-FP16 CPU cost behind the "FP32 is unavailable on ANE" row above.

6. Exact and Approximate Optimizations

An adaptive beam can save useful work, but only after an exact beam exists as an oracle. Cheap confidence signals include top-1/top-2 margin, blank probability, entropy, active-beam score gap, and beam agreement. A confident greedy path can still hide a deletion that beam repairs, so no signal is safe by intuition.

Two-pass greedy-then-beam improves perceived partial latency only if greedy text is shown immediately. It does not reduce end-of-audio finalization latency unless beam is skipped or pruned.

7. Smallest Experiments That Settle the Design

Experiment 0: Finish the quality rank

Score greedy and beams 3, 4, and 8 on the sealed 4,182-row development role. Select a width before exporting several beam packages. The 500-clip rank is evidence for that slice, not authority to ship width 8.

Experiment 1: Count the actual work

Instrument pinned NeMo on probe50:

• alignment iterations per clip;
• joint calls;
• predictor calls;
• blank/non-blank transitions;
• active-beam count;
• duplicate recombinations;
• forced blanks at max_symbols=10;
• K² candidate selections;
• final token count.

This converts the 400 ms target into a per-iteration budget.

Experiment 2: Hoist projections before building beam

Export batch-one FP32 projected encoder, projected predictor, and projected joint. On a public ladder:

• compare projected tensors to NeMo;
• compare greedy tokens to the current unhoisted graph;
• measure isolated p50/p95 and end-to-end greedy p95;
• inspect CPU/GPU/ANE placement.

Kill: no latency reduction, any unexplained top-1 change, or a larger package with no measured benefit.

Experiment 3: Fixed batch-8 microbenchmark

Use deterministic inputs and one process per configuration. Measure:

• predictor B=1 versus B=8;
• projected joint B=1 versus B=8;
• full logits versus in-graph top-K outputs;
• fresh versus reused feature providers;
• default outputs versus verified output backings;
• separate versus fused transition-plus-joint.

Run on M1 8 GB, M1 16 GB, and iPhone 12 Pro. The M2 Ultra can debug, not certify.

Experiment 4: State-machine parity

Pass toy scorer fixtures through pinned Python and Swift. Compare after every iteration:

• selected parent indices;
• labels;
• unnormalized scores;
• normalized final scores;
• time indices;
• max-symbol counters;
• active masks;
• recombination groups;
• predecessor pointers.

Kill: first unexplained mismatch.

Experiment 5: Component and probe50 parity

Use frozen real encoder outputs. Compare NeMo FP32 projections to Core ML FP32, then compare a Python host loop and Swift host loop over the same Core ML components. This separates graph numerics from beam bookkeeping.

Kill: first Swift-versus-Python Core ML hypothesis mismatch. NeMo-versus-Core ML differences are reported separately because existing FP32 conversion is already 496/500 exact, not 500/500.

Experiment 6: Quality and latency escalation

Run the exact candidate on parity500, then the full development role, with the untouched greedy baseline retained. Interleave latency arms on each device after warmup.

Required receipt fields:

{
  "model_revision": "...",
  "package_digests": {},
  "decoder_config": {
    "strategy": "malsd_batch",
    "beam_size": 8,
    "max_symbols": 10,
    "score_norm": true,
    "dtype": "fp32"
  },
  "host": {
    "hardware": "...",
    "os_build": "...",
    "thermal_state": "..."
  },
  "latency_ms": {
    "preprocess_p50": 0,
    "encoder_p50": 0,
    "predictor_p50": 0,
    "joint_p50": 0,
    "bookkeeping_p50": 0,
    "finalization_p50": 0,
    "finalization_p95": 0
  },
  "counts": {
    "alignment_iterations": 0,
    "predictor_calls": 0,
    "joint_calls": 0,
    "recombinations": 0,
    "bytes_copied": 0,
    "allocations": 0
  },
  "quality": {
    "raw_wer": 0,
    "greedy_minus_candidate_points": 0,
    "paired_interval": [0, 0],
    "exact_hypotheses": 0
  }
}

Experiment 7: Approximation only if exact misses

Freeze the exact beam as teacher. Sweep one approximation at a time on probe50, then parity500. Reject any arm that loses a preregistered material fraction of beam's measured gain over greedy. Never tune and report on the same full-development role without recording that reuse.

8. Failure Modes

• Calling Core ML once per hypothesis. Batch is the first requirement, not a later optimization.
• Recomputing both joint projections. Encoder projection is once per utterance; predictor projection is once per changed prefix.
• Calling a static [T,V] tensor an RNNT lattice. Joint scores depend on predictor prefix.
• Merging before pruning. Pinned NeMo prunes K² to K, then recombines survivors.
• Implementing a predictor cache and calling it exact NeMo parity. The pinned MALSD implementation explicitly does not cache predictor outputs.
• Treating beam one as greedy_batch. NeMo says they can differ.
• Using MLState because it sounds zero-copy. Branching and beam reordering are the actual state problem.
• Claiming output backings were used. Check object identity and measure allocations.
• Returning all 1,033 logits without measuring transfer. In-graph top-K plus blank is the smaller contract.
• Full-sorting the vocabulary for eight winners. Use graph top-K or measured partial selection.
• Assuming FP16 is harmless. the reference implementation measured 25 of 500 transcript changes from FP16 CPU components.
• Inferring Core ML beam cost from NeMo CPU wall time. The shared PyTorch encoder dilutes the decoder ratio.
• Inferring an M1 or iPhone result from a 60-core M2 Ultra GPU. Device ordering is not portable.
• Calling Argmax's TDT speed a beam benchmark. Public sources do not disclose a beam.

9. Ranked Optimization Ladder

The simple stopping rule is physical: do not write rung 4 until rung 3's batch-8 microbenchmark can plausibly fit inside the per-iteration budget; do not write a custom kernel until a trace proves dispatch, not math or the encoder, is the limiter.

10. Ingest Corrections to the Raw Report

• Removed the unsupported 66 MB Argmax memory claim.
• Replaced “Argmax speed is primarily TDT” with the narrower evidence: TDT supplies up to 2.82× in its paper; the remaining system-speed attribution is not established.
• Removed speculation that Argmax uses adaptive beam, custom Metal, or C++.
• Corrected BatchedBeamHyps: dense tensors, predecessor pointers, and hashes; no predictor prefix cache in pinned MALSD.
• Corrected recombination ordering to post-prune, matching source.
• Removed the false static-logit RNNT parity recipe and replaced it with a prefix-aware toy scorer.
• Replaced deprecated neural-network flexible-shape mutation with current ct.EnumeratedShapes guidance, while recommending fixed packages first.
• Corrected ct.ComputeUnits.ALL to singular ct.ComputeUnit.
• Removed the unsupported claim that Core ML compute plans report bnns, mps_graph, or ane backend names; they report preferred/supported compute-device objects.
• Downgraded MLState and output backings from “mandatory zero-copy” to measured options with documented constraints.
• Removed the unverified MLX 2 GB and 2.3 ms dispatch constants.
• Removed the proposed vDSP_vsorti top-K snippet; a full sort is not the right default and the API/performance claim was not established.

Sources

1. Argmax, “Nvidia Frontier Speech Models on Argmax SDK”, 2025-06-19.
2. Argmax, “Managing Model Files”, accessed 2026-09-06.
3. Argmax, “iPhone 17: On-device inference benchmarks”, 2025-09-21.
4. Xu et al., “Efficient Sequence Transduction by Jointly Predicting Tokens and Durations”, ICML 2023.
5. Argmax, “Argmax Pro SDK”, 2025-07-24.
6. NVIDIA, “Turbocharge ASR Accuracy and Speed with NVIDIA NeMo Parakeet-TDT”, 2024-04-18.
7. NVIDIA NeMo Speech, rnnt_malsd_batched_computer.py, pinned revision 95f92737cf.
8. NVIDIA NeMo Speech, batched_beam_decoding_utils.py, pinned revision 95f92737cf.
9. NVIDIA NeMo Speech, rnnt_decoding.py, pinned revision 95f92737cf.
10. Apple coremltools, “Flexible Input Shapes”, current documentation checked through Context7 on 2026-09-06.
11. Apple coremltools, “Stateful Models”, current documentation checked through Context7 on 2026-09-06.
12. Apple, MLPredictionOptions.outputBackings, accessed 2026-09-06.
13. Saon, Tüske, and Audhkhasi, “Alignment-Length Synchronous Decoding for RNN Transducer”, ICASSP 2020.