2 Commits

Author	SHA1	Message	Date
Tarry Singh	4294a4d08f	Bump hftokenizers dependency versions in Bazel and Cargo lockfiles (MODULE.bazel.lock, Cargo.toml, Cargo.lock)	2025-04-04 12:54:33 +00:00
Tarry Singh	959bc48c42	Add HuggingFace tokenizer bindings and SentencePiece integration; update BUILD files, async utilities, and FFI modules to support the new tokenizers.	2024-02-28 15:47:37 +00:00