yzma

Go with your own intelligence - local inference in Go using llama.cpp including hardware acceleration

Get Started Try it in your browser See the code

yzma lets you write Go applications that use llama.cpp for local inference.

Your models run in the same process as your program. You don’t need a model server or a C compiler. And you can use the hardware acceleration that your machine has.

Accelerate

Use CUDA, Metal, Vulkan, ROCm, or WebGPU for maximum performance.

See

Run Vision Language Models on images, audio, and video.

Run anywhere

Run on Linux, macOS, Windows, or in a browser with WebAssembly.

See it run, now

A model in your browser. No install, no signup, and no server.

The page downloads the model once and then runs it on your machine. Chrome and Edge use the GPU with WebGPU.

Try it in your browser

Devices On The Edge

These small devices are tested and benchmarked for inference at the edge.

Arduino UNO Q

Run models on the Linux side of this dual-function board using CPU.

Jetson Orin Nano

Use the onboard GPU with CUDA or Vulkan.

Raspberry Pi

Run a model with the CPU on this popular board.

Is it Fast? Yes.

yzma calls llama.cpp in the same process. There is no server between your program and the model.

A desktop class GPU can process more than 850 tokens a second. An edge device like Arduino UNO Q can process more than 32 tokens a second with its CPU alone.

See the benchmarks