2 Commits

Author	SHA1	Message	Date
Tarry Singh	05d23beb23	Add Normalizer.fromHfJson to read HuggingFace tokenizer JSON and map to internal options, including a configurable magic space token and a debug flag for token merges. Adjust default handling of extra whitespaces to align with HF defaults.	2023-03-29 16:10:29 +00:00
Tarry Singh	266da6d4be	Add initial Bazel build configuration, async runtime implementation, and core MLIR dialect definitions for ZML.	2023-01-02 14:28:25 +00:00