Radix/zml/tokenizer/hftokenizers/Cargo.toml

[package]
name = "zml_tokenizer_hftokenizers"
version = "0.1.0"
edition = "2021"

[dependencies]
tokenizers = { version = "0.21.4", default-features = false, features = [
    # We don't need/want the progressbar feature
    "onig",
    "esaxx_fast",
] }

[lib]
name = "zml_tokenizer_hftokenizers"
path = "hftokenizers.rs"
Add HuggingFace tokenizer bindings and SentencePiece integration; update BUILD files, async utilities, and FFI modules to support the new tokenizers. 2024-02-28 15:47:37 +00:00			`[package]`
			`name = "zml_tokenizer_hftokenizers"`
			`version = "0.1.0"`
			`edition = "2021"`

			`[dependencies]`
third_party/xla: de-modularize xla This PR intends to remove XLA as a bzlmod and transfer it as a non bzlmod dep. This is because this module will never be upstreamed as is, so keep it private. Also, we fetch llvm-raw and stablehlo from it, which is fine. While there, dummify the various local_config XLA symbols to please the imports, as we don't use those parts in ZML itself. Closes 2025-06-23 09:13:43 +00:00			`tokenizers = { version = "0.21.4", default-features = false, features = [`
			`# We don't need/want the progressbar feature`
			`"onig",`
			`"esaxx_fast",`
			`] }`
Add HuggingFace tokenizer bindings and SentencePiece integration; update BUILD files, async utilities, and FFI modules to support the new tokenizers. 2024-02-28 15:47:37 +00:00
			`[lib]`
			`name = "zml_tokenizer_hftokenizers"`
			`path = "hftokenizers.rs"`