Repository navigation
Add buffered streaming for the Nemotron unified ASR checkpoint - #109
Merged
localai-org-maint-bot merged 10 commits intoOct 30, 2026
Merged
localai-org-maint-bot merged 10 commits into
localai-org-maint-bot merged 10 commits into
Conversation
The unified checkpoints (att_context_style chunked_limited_with_rc) count the attention chunk grid from the first frame of the encoder input, not from the start of the stream. The buffered streaming driver re-encodes a window at each step, so the mask has to follow that window. build_rc_mask fills the additive [T, T] mask with the rule from NeMo ConformerEncoder._create_masks: query i sees key j when max(0, q*C - L) <= j <= min(T-1, q*C + C - 1 + R), q = i / C, and the key is inside the valid length. The grid is not re-aligned to the left context, so a 70-frame left context with 13-frame chunks is misaligned on purpose. Padded query rows stay unmasked, as in the offline path. RcMask is passed through ConformerLayer, RelPosAttention (dense and batched builders) and a new Encoder::forward_rc. The default (no mask) keeps the existing graphs. forward_rc uses full attention, because the window mask replaces the local-window path. test_rc_mask checks the mask against NeMo's own output for the unified presets, the 1.04 s misaligned grid, warm-up windows and a partial last window. The fixture comes from scripts/gen_rc_mask_fixture.py, pinned to NeMo cd5e4fbd36ba. Assisted-by: Claude:claude-haiku-5-5 [Claude Code]
The unified checkpoints stream by re-encoding a window of audio at each step: [left | chunk | right] in 80 ms frames. Each step recomputes the log-mel over the window (normalized over that window), runs the encoder under the window-relative RC mask, applies the language prompt, and decodes only the chunk frames with the RNN-T state carried across steps. Committed tokens never change. BufferedSchedule is the step schedule of NeMo's streaming example (speech_to_text_streaming_infer_rnnt.py, batch 1) with no model. A step runs once more than its right edge has arrived, so the step that reaches the end of the audio is always the last one, and it is run by finalize. The context arithmetic is a port of NeMo's ContextSize, in samples, so the last step trims the left context by samples, not frames. BufferedStreamingSession wires the schedule to the mel frontend, the encoder and the RNN-T decoder. It keeps only the audio that a later window can still read. test_buffered_schedule checks the schedule without a checkpoint: the edges match NeMo's clamped sequence, every feed size gives the same steps, and only the last step is last. test_unified_stream compares each step with a NeMo reference (context, window, encoder length, decoded frames, tokens, encoder slice) and the final text. It needs the checkpoint and a baseline from scripts/gen_unified_stream_baseline.py. Assisted-by: Claude:claude-haiku-5-5 [Claude Code]
Expose the buffered streaming driver of unified checkpoints through the C API (ABI v11, additive) and parakeet-cli. C API: parakeet_capi_stream_begin, _begin_lang and the feed, finalize, drain and JSON calls now accept a unified model and run a buffered stream with the default window. The new calls are parakeet_capi_stream_kind and _caps (NONE, CACHE_AWARE, BUFFERED), parakeet_stream_opts with parakeet_capi_stream_begin_opts (language, preset index or custom L/C/R frames, PARAKEET_STREAM_PRESET_DEFAULT and _CUSTOM), parakeet_capi_stream_presets_json, parakeet_capi_stream_info_json and parakeet_capi_stream_reset. A buffered stream raises no EOU or EOB events. Cache-aware streams behave as before. The ABI bump updates the version assertion in four existing tests from 10 to 11. CLI: parakeet-cli transcribe --stream on a unified model takes --stream-latency SEC (matches a preset) or --stream-context L,C,R in ms, and --feed-ms N for the block size. parakeet-cli info lists the presets. The window and feed flags are refused on other models. test_capi_stream_unified checks the stream kind, the presets, the info JSON, identical text for feed sizes 1 to 7777 samples, every preset, and the refusals. test_unified_stream now gates each step's encoder slice at 1e-5 (steady state) and 1e-3 (warm-up). The warm-up gap comes from the mel front end (PR1's known difference), not the encoder: NeMo's mel fed to our encoder brings it to 2e-5 or less. Tokens remain the hard gate. Measured numbers are in docs/parity.md. Tentative lookahead text is not included. Assisted-by: Claude:claude-haiku-5-5 [Claude Code]
take_new_text cut the transcript at a byte offset taken from the previous call. A transcript that changed under the emitted bytes, or a sequence that ended in the middle of a UTF-8 character, produced a wrong or split slice. take_text_delta keeps the emitted text and compares it with the current transcript. It returns nothing while the emitted text is no longer a prefix of the transcript. It holds back a trailing incomplete UTF-8 sequence until it completes, or until finalize. The session stores the emitted text in emitted_ in place of the offset. test_buffered_session_pure checks the delta with a fake transcript sequence: growing text, two- and four-byte characters split across steps, flush at end of stream, revision and truncation. Assisted-by: Claude:claude-haiku-5-5 [Claude Code]
feed runs the schedule's ready steps in order, but the schedule advances before the steps run. When a step threw, the remaining planned steps were dropped and a later feed carried on with the audio and the context out of step. StepFailureLatch wraps feed and finalize. If the wrapped call throws, the latch is set and the exception is rethrown. Later feed, finalize, take_new_text and drain_words throw std::runtime_error until reset(), which clears the latch. The C API already turns the exception into a NULL return with parakeet_capi_last_error set, so the error reaches the caller. test_buffered_session_pure checks the latch with a throwing function: the first throw is rethrown, later calls throw without running their function, and clear() restores the stream. Assisted-by: Claude:claude-haiku-5-5 [Claude Code]
parakeet_capi_stream_info_json and parakeet_capi_stream_presets_json formatted their fractional numbers with snprintf. %f follows LC_NUMERIC, so under a comma-decimal locale the JSON carried "0,080000" and was not valid for a parser that expects a dot. buffered_info_json, buffered_preset_json and format_fixed build the same JSON. format_fixed writes through an ostringstream imbued with std::locale::classic(), which is available on every supported toolchain, where std::to_chars for floating point is not. The output text is unchanged under the C locale. The cache-aware info JSON uses format_fixed too. test_buffered_locale checks the JSON under the C locale. It then tries the comma-decimal locales it can find and checks the same JSON under the first one installed. Without one the test exits with 77 (skipped). The rest of the parakeet_capi_*_json family still uses snprintf and is not changed by this commit. Assisted-by: Claude:claude-haiku-5-5 [Claude Code]
The window is left + chunk + right in 80 ms frames. Nothing bounded it: the custom preset took any int, the sum was formed in int, and the CLI cast a 64-bit millisecond count to int after dividing by 80. check_buffered_window validates a window in 64-bit arithmetic: left >= 0, chunk >= 1, right >= 0, and every part and their sum at most 4000 frames (320 s). It returns the reason for a refusal, or an empty string. - BufferedStreamingSession checks the window in its first member initializer, so an invalid window throws std::invalid_argument before any model part reads a weight. - parakeet_capi_stream_begin_opts checks the custom and preset windows and sets parakeet_capi_last_error on a refusal; it returns NULL. - parakeet-cli transcribe --stream-context parses with 64-bit values, checks the frames and exits with 2 and a message on a refusal. The --stream-latency path avoids an out-of-range lround. The public header comment states the limit. The ABI and the signatures are unchanged. test_buffered_session_pure checks the rule with valid, negative, zero, over-limit and INT_MAX and INT64_MAX inputs. test_unified_synthetic adds the constructor and C API refusals on a KV-only unified file, so no checkpoint is needed. Assisted-by: Claude:claude-haiku-5-5 [Claude Code]
The shorter buffered presets re-encode the whole window at every step, so on CPU only the default 1.12 s preset keeps up with real time. The GPU case is left to measurement, and the section says so. It also states the window limits (4000 frames in total, chunk at least 1) that the C API enforces. Assisted-by: Claude:claude-haiku-5-5 [Claude Code]
Stage 5 expected parakeet_capi_stream_begin to refuse a unified checkpoint. Since the buffered stream landed, the model opens a buffered stream on the default window. Stage 5 now checks that parakeet_capi_stream_kind reports PARAKEET_STREAM_KIND_BUFFERED and that stream_begin_lang returns a stream, then frees it. Assisted-by: Claude:claude-haiku-5-5 [Claude Code]
parakeet_capi_stream_begin said it returns NULL for any model that is not cache-aware. Unified models now open a buffered stream on the default window, so the comment now says which models start which stream and that NULL is returned only when the model supports neither. Assisted-by: Claude:claude-haiku-5-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Buffered streaming for the unified checkpoint
nvidia/nemotron-labs-unified-asr-0.6b(
att_context_style = chunked_limited_with_rc), on top of #108 (offline support)._create_masks(both the dense and batched builders, and a newEncoder::forward_rc).pk::BufferedStreamingSession: each step re-runs the model on theraw audio window
[left | chunk | right]. The log-mel is normalized over the window, theencoder runs under the window mask, the language prompt is applied, and only the chunk
frames are decoded with the RNN-T state carried across steps. Committed tokens never change.
The step schedule is a model-free class
BufferedSchedule, a port of NeMo's streaming loopand its
ContextSizearithmetic in samples.C API v11
parakeet_capi_stream_begin/_begin_langand the feed, finalize, drain and JSON callsnow accept a unified model and run a buffered stream on the default window. Cache-aware
streams behave as before.
parakeet_capi_stream_kindand_caps(NONE, CACHE_AWARE, BUFFERED),parakeet_stream_optswithparakeet_capi_stream_begin_opts(language, preset index orcustom L/C/R frames),
parakeet_capi_stream_presets_json,parakeet_capi_stream_info_json,and
parakeet_capi_stream_reset.<EOU>/<EOB>events.CLI:
parakeet-cli transcribe --streamon a unified model takes--stream-latency SEC(matches a preset),
--stream-context L,C,R(ms) and--feed-ms N.parakeet-cli infolists the presets. The window flags are refused on other models.
Parity
Checked against NeMo's streaming loop (
cd5e4fbd36ba, B = 1, float32, CPU). Every step ofevery preset matches on the context, window length, encoder length, decoded frames and
tokens. The final tokens and text are identical on both clips.
Encoder-slice finding (not an encoder or mask bug)
The encoder-slice difference is largest while the left context is still growing (the first
few dozen steps). Steady-state steps match to about 7e-7.
Cause: the log-mel front end. Our window mel differs from NeMo's by 1.2e-4 to 2.3e-4 during
warm-up, and about 5e-5 at steady state. This is the mel difference already recorded in
PR1, from computing the mel in double precision where NeMo uses float32. Per-feature
normalization over the few frames of a short window amplifies it. The check: when NeMo's own
mel is fed to our encoder, the slice difference drops to 2.6e-6 to 1.9e-5 in warm-up and
7e-7 at steady state. The window sample counts and encoder lengths match NeMo at every
checked step, so the window handling, mask and positional encoding agree.
The test gates each step's slice at 1e-5 (steady state) and 1e-3 (warm-up). Tokens are the
hard gate. Fixing the mel precision would change the shared front end for every model, so
it is left for a separate PR.
Timing (measured under load, CPU only)
German clip (17.9 s), f32, 12 threads on a laptop CPU (Ryzen 5 7430U). The machine was
shared with other jobs:
uptimeload average was 15 to 25 during the runs, so thesenumbers are inflated. Step time is the wall time minus the 7.0 s model load, divided by
the steps. Load is step time divided by the chunk duration.
Each step runs about 80 encoder frames whatever the preset, so the time per step is nearly
flat. On this CPU only the 1.12 s preset keeps up with real time. GPU numbers are not
included.
Not included
Verification
ctest --test-dir build -LE model: 58 of 58 pass (includestest_rc_maskagainst a NeMomask fixture, and
test_buffered_schedule, which checks feed-size invariance of the steps).PARAKEET_TEST_GGUF_UNIFIED,PARAKEET_TEST_BASELINE_UNIFIED_STREAM,PARAKEET_TEST_UNIFIED_STREAM_WAV:test_unified_stream(per-step parity, all presets);PARAKEET_TEST_GGUF_UNIFIED:test_capi_stream_unified.The NeMo baselines come from
scripts/gen_unified_stream_baseline.py.tests/fixtures/speech.wav: the output ofparakeet-cliis byte-identical to the build oforigin/master(4fbf4d9) fortdt_ctc-110m-f16(plain,--timestamps,--json) andctc-0.6b-f16(--json).🤖 Generated with Claude Code