|
|
9b7eea8ac2
|
Add stdx utilities and rework async signature inference; tidy executable logging.
|
2023-06-21 14:45:14 +00:00 |
|
|
|
05d23beb23
|
Add Normalizer.fromHfJson to read HuggingFace tokenizer JSON and map to internal options, including a configurable magic space token and a debug flag for token merges. Adjust default handling of extra whitespaces to align with HF defaults.
|
2023-03-29 16:10:29 +00:00 |
|
|
|
ecf52ad724
|
zml.tokenizer: Implement proper byte fallback support by converting hex byte strings (e.g., “<0x40>”) to their characters and splitting unknown UTF‑8 codepoints into bytes, fixing tokenization.
|
2023-02-28 14:40:25 +00:00 |
|
|
|
ebdb8db213
|
zml/tests: re‑enable all Zig tests, fix precision issue by switching to f32, and add refAllDecls to ensure all declarations are tested
|
2023-01-23 16:28:19 +00:00 |
|
|
|
266da6d4be
|
Add initial Bazel build configuration, async runtime implementation, and core MLIR dialect definitions for ZML.
|
2023-01-02 14:28:25 +00:00 |
|
