Radix/zml/tokenizer/hftokenizers
Tarry Singh fe56f03f5d third_party/xla: de-modularize xla
This PR intends to remove XLA as a bzlmod and transfer it as a non
bzlmod dep.

This is because this module will never be upstreamed as is, so keep it
private.

Also, we fetch llvm-raw and stablehlo from it, which is fine.

While there, dummify the various local_config XLA symbols to please the
imports, as we don't use those parts in ZML itself.

Closes
2025-06-23 09:13:43 +00:00
..
BUILD.bazel Remove examples workspace and clean up related Bazel BUILD/MODULE files and Zig build scripts. 2025-06-19 09:30:29 +00:00
Cargo.lock third_party/xla: de-modularize xla 2025-06-23 09:13:43 +00:00
Cargo.toml third_party/xla: de-modularize xla 2025-06-23 09:13:43 +00:00
hftokenizers.h Add HuggingFace tokenizer bindings and SentencePiece integration; update BUILD files, async utilities, and FFI modules to support the new tokenizers. 2024-02-28 15:47:37 +00:00
hftokenizers.rs Add HuggingFace tokenizer bindings and SentencePiece integration; update BUILD files, async utilities, and FFI modules to support the new tokenizers. 2024-02-28 15:47:37 +00:00
hftokenizers.zig Integrate TinyLlama support, restore the homemade tokenizer, and align Zig API naming across stdx and zml tokenizer modules. 2024-04-05 15:07:29 +00:00