[Bharath2]

Problem

Three things go wrong with Nemotron 3.5 ASR streaming on master (4fbf4d9):

1. --stream crashes on CUDA.
   • parakeet-cli transcribe --stream stops right after the backend starts, with an access violation
     (0xC0000005 on Windows). It happens with every model file I tried, including
     nemotron-3.5-asr-streaming-0.6b-q8_0.gguf and -q6_k.gguf from mudler/parakeet-cpp-gguf.
   • Offline transcription on the same build works.
   • The cause: MelKernel takes a tensor pointer before ensure_weights_realized(). On a device backend,
     realizing swaps the name-to-tensor map to device tensors and frees the host context, so the pointer dangles.
   • StreamingDiarization has the same order for learnable_sil_emb: it takes the tensor, then realizes.
   • perf: faster GPU decode: one SSM_CONV for the depthwise conv, less host work per step #102 notes the same MelKernel read, and says Model::load realizes the weights first. The --stream path
     still crashes on master, and realizing inside MelKernel fixes it.
2. Language tags in streamed text and word timestamps.
   • Offline decode emits raw language tag tokens (<en-US>) in transcript text #40 removed the tags from the offline text. The streaming session still appends them, so the streamed text has
     <en-US> after every sentence.
   • Word timestamps, offline and streaming, list the tag as a word: 5.76-5.84 <en-US> (1.00).
3. Two spaces between sentences in the offline text, as in launch. We're.
   • The model emits a bare space piece before each tag.
   • Once the tag is stripped, that space and the next word's own space both remain.

Fix

• mel.cpp: realize the weights before taking any tensor pointer.
• diarization_streaming.cpp: move the existing ensure_weights_realized() call before ml.tensor(...).
• tokenizer:
  • is_special_piece() is factored out of strip_special_tokens(), with the same rule (<...> or [...]);
  • detokenize() no longer emits two spaces in a row.
• StreamingSession: special pieces no longer go into the streamed text or the word tokens.
• group_words(): drops special pieces, and the bare space piece right before one, so a tag never becomes a word.
  This covers offline and streaming.
• Docs: AGENTS.md (the test list and the streaming semantics) and docs/capi.md now say the streamed text drops
  language tags as well as <EOU>/<EOB>.

Text without tags or double spaces comes out unchanged.

Measurement

Setup:

• Build: master 4fbf4d9, Release, Windows 11, MSVC. One CUDA 12.8 build (sm_86) and one CPU (AVX2) build.
• GPU: RTX 3060 Laptop (6 GB).
• Models: nemotron-3.5-asr-streaming-0.6b-q8_0.gguf and -q6_k.gguf from mudler/parakeet-cpp-gguf at
  741158a.
• Input: 20 English dictation clips (Piper TTS, 16 kHz, 9.7 to 37.3 s each, 546 s in all), with --lang en-US.
• WER: lowercase, punctuation removed, the same scorer before and after.

CUDA build, 20 clips:

	Before	After
--stream runs that crash, q8_0 / q6_k	20 / 20, 20 / 20	0 / 20, 0 / 20
--stream WER, q8_0 / q6_k	no output	5.14% / 5.14%
--stream: <en-US> tags in the text	no output	0
Offline: double spaces in the text, q8_0 / q6_k	60 / 63	0 / 0
Offline WER, q8_0 / q6_k	4.98% / 5.14%	4.98% / 5.14%

Notes:

• The offline text is otherwise unchanged: it is identical before and after apart from whitespace, in 20 of 20
  clips for both models.
• Streaming on the CPU build: it worked before the fix, but its text carried 47 (q8_0) and 46 (q6_k) <en-US>
  tags in 10 clips.

Tests

• New test_stream_special_pieces, which needs no model. It checks two things:

  • detokenize() must not double the space a removed tag leaves;
  • group_words() must not turn a tag, or the space piece before it, into a word.

  On master it fails with text 'The sun. It', expected 'The sun. It' and
  words 'The|sun.|<en-US>|It', expected 'The|sun.|It'. It passes with this change.

• ctest -LE model on the CPU build (CED and voice-detect off): 51 of 54 pass, including
  test_special_token_filter. The other three fail the same way on master; they are Windows issues these files
  don't touch:

  • test_backend_device and test_bundle don't compile with MSVC (setenv, unistd.h);
  • test_audio_io stops with a numerical exception.

• The model tests were not run. The CLI runs above cover the streaming and offline paths with real models.

Limits

• The diarization change follows the same pattern but was not run: no diarization model was used in these runs.
• test_streaming_eou_reset was not run: it needs the realtime EOU model and its NeMo baseline.
  • This change leaves the <EOU>/<EOB> handling as it is.
  • But detokenize() now never emits two spaces in a row, and that rule applies to that model's text too.
• Windows only. The crash was seen on CUDA only; CPU streaming did not crash before the fix.
• English clips only.