When I wrote the first Quaternion-Augmented TurboQuant post, QATQ was still mostly a question.
It had a good name, a strange little spark of mathematics behind it, and a premise that felt just elegant enough to be suspicious: could quaternion structure make Google’s TurboQuant-style KV cache compression even more aggressive?
That question mattered because the KV cache is not some decorative internal detail of modern language models. It is the working memory that grows as a model reads and generates. Every new token leans on the key and value projections of the tokens that came before it. As context length grows, that cache becomes one of the most consequential pieces of state in the whole system.
For local inference, that can mean memory pressure. For runtime migration, it can mean the difference between a state that can be moved and a state that stays trapped inside one process, on one machine, in one runtime.
The first post was deliberately speculative. It asked whether four-dimensional quaternion rotations could expose enough structure inside attention tensors to make long-context local inference more practical on consumer hardware. It was the kind of idea that makes you open too many tabs, sketch matrices in the margin, and wonder whether a mathematical trick might one day become infrastructure.
QATQ has now crossed a different line.
It has a real v0.1.1 release. It has a public GitHub repository, a technical whitepaper, release automation, cross-platform binaries, signed and notarised macOS archives, public fixtures, benchmark gates, fuzzing scaffolds, and a Rust crate shape ready for downstream integration.
The project is no longer only asking whether Quaternion-Augmented TurboQuant could work. It is beginning to show what kind of product that idea wants to become.
You can inspect the project directly on GitHub, and read the current technical explanation in the QATQ whitepaper.
The Pressure Was Real
The original paper was right about the pressure point.
KV caches are expensive, structured, and central to modern LLM execution. They grow with context length, layer count, head structure, and dtype. If you care about long-context local inference, moving live model state between machines, or making AI runtimes less rigid, you eventually run into the same question: what do you do with all that state?
The first QATQ idea brought two threads together:
- TurboQuant-style training-free compression, using rotation, scalar quantisation, and compact residual information.
- Quaternion structure, grouping channels into four-component lanes so transforms can operate on richer local geometry than isolated scalar values.
The combination still matters. But the path from paper to product changed the claim.
Exact First, Lossy Later
The speculative paper was mostly about lossy inference compression. The dream was smaller live KV caches with no meaningful loss in model quality.
The production release starts somewhere more disciplined: exact storage and transfer of exported tensors.
That pivot came from a blunt infrastructure question: did the bytes come back exactly?
If QATQ changes tensor values, even in a way that seems harmless, then it inherits a much heavier burden. It has to prove perplexity, task behaviour, long-context behaviour, runtime acceptance, and downstream quality across a serious matrix of models and prompts.
Exact compression gives the project a cleaner first contract:
- accept exported tensor bytes;
- compress them with tensor-aware exact strategies;
- decode them bit-for-bit;
- verify the result;
- store the payload raw when compression is not the honest answer.
That is less glamorous than claiming universal live VRAM reduction. It is also a much better foundation for a real tool.
The Quaternion Part Had To Earn Its Keep
There was a slightly awkward moment in the project where it looked as if the name QATQ might become decorative.
Some of the strongest early wins came from byte-plane transforms and zstd entropy coding. Those are useful techniques, but they do not exactly shout Quaternion-Augmented TurboQuant from the rooftops. If the best version of the project simply wrapped tensor bytes in byte-plane zstd, then the name would have been a fossil from a more romantic design phase.
So the rule became simple: the quaternion transform only belonged in the exact codec if it could justify itself.
It had to be exactly reversible. It had to reduce entropy on real tensor payloads. It had to cost less metadata than it saved. And it had to beat simpler byte-plane strategies when the data supported it.
That is the version that made it into QATQ.
The qatq-exact path now includes a reversible quaternion-chain zstd candidate. It is not forced. It is not a branding tax. It competes against raw bits, byte-run strategies, byte-plane strategies, byte-plane zstd, packed RLE, delta-XOR residuals, and other exact candidates. The codec chooses it only when it wins.
That feels like a healthier relationship between the mathematics and the product. The quaternion idea is no longer a speculative ornament. It is an exact compression strategy with a job to do.
What v0.1.1 Actually Ships
QATQ v0.1.1 is best understood as an exact tensor-aware codec toolkit for exported LLM KV caches and runtime migration artefacts.
The core release surface is:
qatq-exact, the primary exact codec path;QATC, a sequential container for large tensor chunks;- native exact support for
f32,f16, andbf16tensor bytes; - a CLI for encoding, decoding, fixture generation, and verification;
- Rust library APIs for downstream integration;
- benchmark utilities for public fixtures and runtime-exported tensors.
The lossless claim is intentionally scoped. QATQ claims bit-identical recovery for the exact codec and the QATC container. The older lossy TurboQuant-style and Phase 1 paths remain useful research comparators, but they are not the launch claim.
That distinction matters. A project should not borrow credibility from a speculative paper unless the released implementation proves the specific thing it claims.
Release Automation Turned Assumptions Into Evidence
The release process forced QATQ to become more than a promising local experiment.
It now has deterministic public fixtures and a public benchmark corpus. It has a fixture audit. It has compression gates that compare QATQ against raw, zstd, and lz4 on public compression-positive fixtures. It has deterministic KV stress testing across thousands of cases. It has fuzzing scaffolds and scheduled coverage and supply-chain workflows.
It also has release automation that builds binaries for Linux, macOS, and Windows, generates checksums, creates installers, and publishes GitHub Releases from annotated SemVer tags.
The first release was not frictionless, which is part of the point. The automation caught real workflow bugs:
- the tag-on-master guard initially could not authenticate its fetch;
- the macOS notarisation step used Bash
mapfile, which does not exist on the default Bash shipped on GitHub macOS runners.
Both were fixed before the release completed.
That is why release automation matters. It turns assumptions into logs. It gives the project something more durable than local confidence.
The final v0.1.1 release completed successfully with cross-platform archives, checksums, a source tarball, installers, and signed and notarised macOS artefacts.
PermeantOS Made It Less Abstract
The most important external pressure came from live migration work.
PermeantOS needed a way to move model runtime state across machines. In one AWS proof path, a local MLX runtime migrated a 1,920-token live state to AWS vLLM over production WSS/mTLS, with a complex Agent Memory Graph attached. The target continued exactly at 16, 32, 64, and 128 tokens, exported the runtime boundary back to MLX, and passed the return-home continuation proof.
That is a very different kind of test from a chart on its own. It is a working system asking whether the codec helps with a real handoff.
The first QATQ-compatible path proved exact transfer, but it was larger than raw. That was humbling. It was also useful, because it forced the project to separate the story it wanted to tell from the evidence it actually had.
The later standalone QATQ path changed the result. QATQ compressed the same streamed block artefacts below raw, zstd, and lz4 while preserving the exact live migration behaviour.
That does not prove universal superiority. It does prove that QATQ had become relevant to a real runtime migration workflow, not only to a toy benchmark.
That is the kind of evidence I care about most: a working system that needed the codec, used it, and validated the handoff.
What It Is Not Claiming Yet
QATQ is not claiming live GPU VRAM reduction yet.
That would require deeper runtime integration: paging or offloading KV tensors inside a live inference engine, decoding them under generation latency pressure, and proving that the memory saved is worth the performance trade-off. That remains an experimental roadmap goal, not part of the v0.1.1 product claim.
QATQ is also not claiming that its lossy paths beat TurboQuant on model quality. The project includes TurboQuant-style comparators and quaternion-inspired research paths, but the production release is exact. Any stronger lossy claim needs end-to-end model and task benchmarks across a broader matrix.
The clean claim is narrower and better:
QATQ is a Rust codec for exact tensor-aware compression of exported LLM KV caches and migration artefacts, with a reversible quaternion-backed strategy that is selected when it actually wins.
Why Release It Now
I open-sourced QATQ now because it has crossed the line from idea to inspectable artefact.
It is not finished. It is not the final answer to KV cache compression. But it has enough release discipline to be useful to other engineers:
- the repository can be built and tested from source;
- the public fixtures can be regenerated and verified;
- the release artefacts are reproducible through GitHub Actions;
- the whitepaper explains both the original foundation and the current claim boundary;
- the code is available for runtime and migration projects that want an exact codec rather than a hand-wavy compression story.
That feels like the right moment to invite scrutiny.
The GitHub repository is here: github.com/kabudu/qatq.
The technical whitepaper is here: QATQ Technical Whitepaper.
The first release is here: QATQ v0.1.1.
What Comes Next
The next phase is product hardening around real integrations.
QATQ needs broader runtime-exported KV benchmarks across more models, prompts, context lengths, dtypes, and chunk sizes. It needs the llama.cpp exporter path to become less patch-like, or at least clearly version-pinned as an adapter. It needs longer fuzz runs and more hostile container-input review. It needs crates.io publication only when API names and downstream integration expectations are settled.
Most of all, it needs to stay honest about the difference between storage and live memory.
Storage and transfer compression are here now. Live VRAM reduction is the research frontier.
That is a good place for a project to be: useful enough to release, precise enough to trust, and still strange enough to keep pulling the thread.
The first post asked whether quaternions and TurboQuant could dance together.
The answer, so far, is more interesting than I expected.
They can, but only after the product makes them prove every step.