|
|
d4db5ccc6b
|
Integrate TinyLlama support, restore the homemade tokenizer, and align Zig API naming across stdx and zml tokenizer modules.
|
2024-04-05 15:07:29 +00:00 |
|
|
|
959bc48c42
|
Add HuggingFace tokenizer bindings and SentencePiece integration; update BUILD files, async utilities, and FFI modules to support the new tokenizers.
|
2024-02-28 15:47:37 +00:00 |
|
|
|
c109b12e1b
|
Various minor fixes: rewrite tinyllama tokenizer newline token, prevent HostBuffer.isContiguous false trigger on 1‑dim axes, improve HostBuffer.slice1d error messages, simplify module.zig output to show .mlir file path, correct setFlags handling of comptime int/float, make tokenizer.zig return <oob> for out‑of‑range detokenization, and speed up Buffer.constant creation up to 2.5 GB/s on CUDA.
|
2024-02-19 12:34:18 +00:00 |
|
|
|
b8a0aaee5a
|
Update tokenizer to handle byte_fallback for Llama3 GPT2 vocab and add a Llama3‑specific normalizer; adjust tinyllama.zig and hostbuffer.zig to use the new tokenization logic.
|
2024-02-05 15:22:44 +00:00 |
|
|
|
9b7eea8ac2
|
Add stdx utilities and rework async signature inference; tidy executable logging.
|
2023-06-21 14:45:14 +00:00 |
|
|
|
89cf2233d3
|
zml/aio: enable reading metadata from index.json for sharded safetensor files, allowing metadata storage alongside model config.
|
2023-05-23 15:06:59 +00:00 |
|
|
|
11006ca08d
|
Refactor torch module: merge PickleData into Parser as torch.File, rename value file to py_object.zig, use buffered reader for pickle and zip headers, adjust intermediate result handling, simplify Python dict representation, separate kwargs from args, and add extensive tests for long integers, protocol 0, zipped pickle, and a complex PyTorch Conv2d case; also streamline BufferStore initialization.
|
2023-04-20 15:43:18 +00:00 |
|
|
|
8e43a45a3c
|
Add event waiting when invoking a module and improve multi‑device sharding handling.
|
2023-04-11 11:32:09 +00:00 |
|
|
|
0189b71070
|
Rename zml.aio.Value to zml.aio.Metadata, simplify its type variants, and update torch pickle/eval APIs accordingly.
|
2023-04-07 16:45:58 +00:00 |
|
|
|
e25f70d923
|
Rename and simplify modules in zml/aio/torch: replace redundant qualified names, remove generic utilities, inline code, reorder functions for top‑to‑bottom readability, and extract parsing logic into parseTensor and parseStorage functions.
|
2023-04-04 17:20:53 +00:00 |
|
|
|
05d23beb23
|
Add Normalizer.fromHfJson to read HuggingFace tokenizer JSON and map to internal options, including a configurable magic space token and a debug flag for token merges. Adjust default handling of extra whitespaces to align with HF defaults.
|
2023-03-29 16:10:29 +00:00 |
|
|
|
ef922e3aea
|
Fix empty JSON array handling in safetensor metadata loader and refactor torch loader (make ops slices const and improve readability).
|
2023-03-28 16:17:00 +00:00 |
|
|
|
ecf52ad724
|
zml.tokenizer: Implement proper byte fallback support by converting hex byte strings (e.g., “<0x40>”) to their characters and splitting unknown UTF‑8 codepoints into bytes, fixing tokenization.
|
2023-02-28 14:40:25 +00:00 |
|
|
|
be6328813d
|
zml: clean up dead and commented code; note that copyslice is currently broken and pending reimplementation
|
2023-02-08 17:13:47 +00:00 |
|
|
|
897786e440
|
aio: correct refAllDecls handling for yaml and nemo modules
|
2023-01-31 11:58:58 +00:00 |
|
|
|
7dcd8b516c
|
zml/nn: fix resize implementations (resizeBilinear and resizeBicubic) and expand refAllDecl usage; all tests pass
|
2023-01-27 14:35:11 +00:00 |
|
|
|
5e1688cbfd
|
aio: refactor PyTorch model parsing for better readability and optimize slice handling
|
2023-01-25 12:16:27 +00:00 |
|
|
|
ebdb8db213
|
zml/tests: re‑enable all Zig tests, fix precision issue by switching to f32, and add refAllDecls to ensure all declarations are tested
|
2023-01-23 16:28:19 +00:00 |
|
|
|
266da6d4be
|
Add initial Bazel build configuration, async runtime implementation, and core MLIR dialect definitions for ZML.
|
2023-01-02 14:28:25 +00:00 |
|
