|
|
d4db5ccc6b
|
Integrate TinyLlama support, restore the homemade tokenizer, and align Zig API naming across stdx and zml tokenizer modules.
|
2024-04-05 15:07:29 +00:00 |
|
|
|
c109b12e1b
|
Various minor fixes: rewrite tinyllama tokenizer newline token, prevent HostBuffer.isContiguous false trigger on 1‑dim axes, improve HostBuffer.slice1d error messages, simplify module.zig output to show .mlir file path, correct setFlags handling of comptime int/float, make tokenizer.zig return <oob> for out‑of‑range detokenization, and speed up Buffer.constant creation up to 2.5 GB/s on CUDA.
|
2024-02-19 12:34:18 +00:00 |
|
|
|
b8a0aaee5a
|
Update tokenizer to handle byte_fallback for Llama3 GPT2 vocab and add a Llama3‑specific normalizer; adjust tinyllama.zig and hostbuffer.zig to use the new tokenization logic.
|
2024-02-05 15:22:44 +00:00 |
|
|
|
9b7eea8ac2
|
Add stdx utilities and rework async signature inference; tidy executable logging.
|
2023-06-21 14:45:14 +00:00 |
|
|
|
0189b71070
|
Rename zml.aio.Value to zml.aio.Metadata, simplify its type variants, and update torch pickle/eval APIs accordingly.
|
2023-04-07 16:45:58 +00:00 |
|
|
|
266da6d4be
|
Add initial Bazel build configuration, async runtime implementation, and core MLIR dialect definitions for ZML.
|
2023-01-02 14:28:25 +00:00 |
|
