Platforms

Kvad is developed and measured on an M5 Pro with 48 GB. That is the machine the numbers in this documentation come from, unless a number says otherwise.

#Support

Intel Macs and Windows have no build.

There is no CUDA build. The GPU engine is written for Metal, and its fastest kernels are specific to Apple's GPUs.

#Apple silicon

Any M-series Mac runs everything. What changes between them is speed and how large a model fits.

M5 and later. The M5's GPU has matrix units that ordinary Metal shaders never reach. Kvad has its own kernels for them, used for dense and quantised matrix products in image models, video models and language-model prefill. They need the Apple10 GPU family, and Kvad checks for it when it loads a model. Apple silicon has the measurements.

M1 to M4. The same models run through the standard Metal kernels. Nothing is switched off, and nothing needs configuring: the accelerated path is simply not taken.

To turn the M5 kernels off, for a comparison of your own:

KVAD_GPU_MPP=0 kvad serve

#Memory

Unified memory is what decides which models you can run. As a rule, a model needs its weights at the precision you load it at, plus a KV cache that grows with the conversation.

The server will not load a model that does not fit beside the ones already in memory. It says what is holding the memory instead, and never unloads something to make room. kvad search marks each result with the precision at which it fits this machine.

#Linux

The hand-written CPU engine is the same code on both systems. On arm64 it uses NEON, the dot-product extension and i8mm where the processor has them. A Linux build is what the release workflow produces; it is tested less than the Mac build, because nobody develops on it.