All pull requests

Pull requests

• perf(bitcannon): add RVV UTF-8 character classification

  #2677·xiazhuozhao opened Nov 11, 2026·

  ·

• fix(bpe): fuse unk tokens on models with affixes

  #2676·Iamsohungrynow opened Nov 11, 2026·

  ·

• fix(vocab): add spare slots to small perfect hash tables

  #2675·Iamsohungrynow opened Nov 11, 2026·

  ·

• fix(convert): preserve input vocabulary order

  #2674·YeexiaoZheng opened Nov 11, 2026·

  ·

• fix(bpe): allow incomplete byte fallback vocabularies

  #2673·rleungx opened Nov 10, 2026·

  ·
  1

• Fix FalconMamba tokenizer conversion with nonterminal ByteLevel

  #2672·voropaevv opened Nov 10, 2026·

  ·
  1

• fix(regex): refuse overlapping regex matches

  #2671·SBrandeis opened Nov 9, 2026Collaborator·

  ·
  1

• chore(python): drop support for python 3.10

  #2668·SBrandeis opened Nov 9, 2026Collaborator·

  ·

• feat(python): expose the ByteLevel normalizer alphabet

  #2662·Mist-wu opened Nov 9, 2026·

  ·

• fix(parallelism): release the pool lock around fork()

  #2656·McPatate opened Nov 6, 2026Member·1/9·

  ·
  2

• docs: fix typo in README

  #2654·dheiver2 opened Nov 6, 2026·

  ·

• perf(bitcannon): add RVV literal matching

  #2650·xiazhuozhao opened Nov 4, 2026·

  ·
  7

• perf(parallel): only cut sequences that could hold up the batch

  #2649·McPatate opened Nov 4, 2026Member·

  ·
  1

• perf(parallelism): read the cached pool without a lock

  #2648·McPatate opened Nov 4, 2026Member·9/9·

  ·
  1

• refactor(parallelism)!: move the MaybeParallel* iterators to tk-train

  #2647·McPatate opened Nov 4, 2026Member·8/9·

  ·
  1

• feat(python): add encode_iter to stream a batch's encodings

  #2644·McPatate opened Nov 3, 2026Member·

  ·
  1

• feat(python): add set_max_threads / get_max_threads and a per-call max_threads

  #2643·McPatate opened Nov 3, 2026Member·7/9·

  ·
  1

• feat(parallelism)!: stop reading TOKENIZERS_PARALLELISM

  #2642·McPatate opened Nov 3, 2026Member·4/9·

  ·
  1

• perf(parallel): cut around normalized added tokens

  #2641·McPatate opened Nov 3, 2026Member·3/3·

  ·
  1

• feat(parallelism): default forked children to one thread

  #2640·McPatate opened Nov 3, 2026Member·6/9·

  ·
  1

• feat(parallel): add per-call EncodeOptions::max_threads

  #2639·McPatate opened Nov 3, 2026Member·3/9·

  ·
  1

• perf(parallel): reassemble chunked sequences in one copy

  #2638·McPatate opened Nov 3, 2026Member·

  ·
  1

• perf(parallel): split large inputs at safe pre-token boundaries

  #2637·McPatate opened Nov 3, 2026Member·2/3·

  ·
  1

• feat(bitcannon): add safe cut points after letter runs

  #2636·McPatate opened Nov 3, 2026Member·1/3·

  ·
  1

• fix(parallel): error instead of hanging when a handle crosses a fork

  #2635·McPatate opened Nov 3, 2026Member·5/9·

  ·
  1