yzma

Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Overview

yzma calls llama.cpp in the same process as your Go program. There is no model server and there is no network connection.

flowchart TD
    subgraph app["Your Go program"]
        code["Application code"]
    end
    subgraph yzma
        llama["pkg/llama"]
        mtmd["pkg/mtmd"]
        loader["pkg/loader"]
    end
    subgraph runtime["Run time"]
        purego["purego"]
        ffi["libffi"]
    end
    subgraph native["llama.cpp shared libraries"]
        libllama["libllama"]
        libmtmd["libmtmd"]
        ggml["ggml backends"]
    end
    subgraph hw["Hardware"]
        cpu["CPU"]
        gpu["GPU"]
    end

    code --> llama
    code --> mtmd
    llama --> loader
    mtmd --> loader
    loader --> purego
    loader --> ffi
    purego --> libllama
    purego --> libmtmd
    ffi --> libllama
    ffi --> libmtmd
    libllama --> ggml
    libmtmd --> ggml
    ggml --> cpu
    ggml --> gpu

No CGo

Most Go bindings for a C library use CGo. CGo needs a C compiler, and it makes cross compilation hard.

Command line

Mon, 01 Jan 0001 00:00:00 +0000

Install the command

go install github.com/hybridgroup/yzma@latest

The commands

NAME:
   yzma - YZMA command line tool

USAGE:
   yzma [global options] command [command options]

COMMANDS:
   install  Install llama.cpp libraries used by yzma
   verify   Check the installed llama.cpp libraries against their published digests
   system   Show llama.cpp system information
   llama    Show most recent llama.cpp version
   model    Manage models
   version  Show yzma version
   info     Show yzma version
   help, h  Shows a list of commands or help for one command

yzma install

Downloads the prebuilt llama.cpp libraries for this machine.

Hello world

Mon, 01 Jan 0001 00:00:00 +0000

This is the smallest yzma program.

Before you start

Install yzma and the llama.cpp libraries. See Quick install. Then set YZMA_LIB.

Download the model:

yzma model get -u https://huggingface.co/QuantFactory/SmolLM2-135M-GGUF/resolve/main/SmolLM2-135M.Q4_K_M.gguf

Run the example

$ go run ./examples/hello/


"Yes, I'm ready to go."

The complete program is on the Your first program page.

The steps

1. Open the libraries.

llama.Load(libPath)
llama.LogSet(llama.LogSilent())
llama.Init()

LogSilent stops the messages that llama.cpp prints. Remove that line to see what the library does.

Models

Mon, 01 Jan 0001 00:00:00 +0000

yzma uses models in the GGUF format that llama.cpp supports. There are more than 201,000 of them on Hugging Face:

https://huggingface.co/models?library=gguf&sort=trending

Set the YZMA_LIB environment variable before you run any of these commands.

Vision Language Models (VLM)

A Vision Language Model takes an image and text, and it gives text back. Each one needs a model file and a projector file.

Qwen3-VL-2B-Instruct

https://huggingface.co/bartowski/Qwen_Qwen3-VL-2B-Instruct-GGUF

Good quality and a good size for most work. This is the model to start with.

Tools and frameworks

Mon, 01 Jan 0001 00:00:00 +0000

Kronk

Kronk gives a high level API that feels like an OpenAI compatible API.

https://github.com/ardanlabs/kronk

Fantasy

Fantasy lets you build AI agents with Go. It works with many providers and many models through one API. It supports Kronk, which is built on yzma.

https://github.com/charmbracelet/fantasy

Applications

Mon, 01 Jan 0001 00:00:00 +0000

wasmVision

wasmVision is a computer vision processing engine with advanced algorithms and vision models for machine learning.

https://github.com/wasmvision/wasmvision

NornicDB

NornicDB is a graph database for AI agents and knowledge systems.

https://github.com/orneryd/NornicDB

Crush

Crush gives agentic coding with style. It is built on Fantasy.

https://github.com/charmbracelet/crush

floop

floop gives a memory that lasts to AI coding agents.

https://github.com/nvandessel/floop

ImmyGo

ImmyGo is a Go user interface framework that puts AI first. Describe what you want, and ImmyGo builds it. It is built on Gio, and it uses yzma, Ollama, and the Anthropic API for its local AI.

Chat

Mon, 01 Jan 0001 00:00:00 +0000

The hello program sends one prompt. A chat program keeps the conversation, so the model remembers what came before.

Before you start

Download a model that follows an instruction:

yzma model get -u https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-fp16.gguf

Run the example

$ go run ./examples/chat/ -model ./models/qwen2.5-0.5b-instruct-fp16.gguf
Enter prompt: Are you ready to go?

Yes, I'm ready to go! What would you like to do?

Enter prompt: Let's go to the zoo


Great! Let's go to the zoo. What would you like to see?

Enter prompt: I want to feed the llama


Sure! Let's go to the zoo and feed the llama. What kind of llama are you interested in feeding?

See the code.

Chat templates

Mon, 01 Jan 0001 00:00:00 +0000

An instruction model expects a marker before each message. The markers say which part is the system prompt, which part the user said, and where the model must write. That shape is the chat template.

Each model family has its own template. If you send the wrong shape, the model gives a poor answer or no answer.

The template in the model file

Most GGUF files hold their own template.

Download models

Mon, 01 Jan 0001 00:00:00 +0000

yzma uses models in the GGUF format that llama.cpp supports. There are more than 201,000 GGUF models on Hugging Face:

https://huggingface.co/models?library=gguf&sort=trending

Use the yzma command

The yzma command downloads a model for you:

yzma model get -u https://huggingface.co/QuantFactory/SmolLM2-135M-GGUF/resolve/main/SmolLM2-135M.Q4_K_M.gguf

The command puts the file in the default models directory. A Go program finds that directory with download.DefaultModelsDir().

Choose a model

A larger model gives better answers, but it needs more memory and more time. Start with a small model.

Environment variables

Mon, 01 Jan 0001 00:00:00 +0000

For your programs

YZMA_LIB

The directory that holds the llama.cpp shared library files. This is the only variable that a normal program needs.

export YZMA_LIB=/path/to/lib

set YZMA_LIB=C:\yzma\lib

The yzma install command and the yzma verify command also read it, so the --lib flag is not necessary when it is set.

macOS

Mon, 01 Jan 0001 00:00:00 +0000

macOS on Apple silicon uses the Metal GPU. You do not need to install a driver.

Decide where you want to put the library files, then run this command:

yzma install --lib /path/to/lib

The yzma install command prints instructions for your system when it finishes. Follow them to complete the installation.

Then set the YZMA_LIB environment variable:

export YZMA_LIB=/path/to/lib

Next steps

Go to Download models.

Models

Mon, 01 Jan 0001 00:00:00 +0000

GGUF

yzma uses models in the GGUF format. This is the format that llama.cpp reads. A GGUF file holds the weights of the model and the metadata about it, such as the name, the size of the context, and the chat template.

There are more than 201,000 GGUF models on Hugging Face:

https://huggingface.co/models?library=gguf&sort=trending

Read the metadata of a file with the modelinfo example program.

Quantization

The name of a model file usually ends with a quantization code, such as Q4_K_M or Q8_0 or fp16.

Hardware acceleration

Mon, 01 Jan 0001 00:00:00 +0000

llama.cpp has a backend for each kind of processor. yzma installs the backend that you name, and llama.cpp uses it.

What each platform supports

Operating system	CPU	GPU
Linux	amd64, arm64	CUDA, Vulkan, HIP, ROCm, SYCL
macOS	arm64	Metal
Windows	amd64	CUDA, Vulkan, HIP, SYCL, OpenCL

A browser is also a target:

Target	CPU	GPU
Browser	wasm32 SIMD, one or more threads	WebGPU

Which backend to choose

Backend	Use it when
CPU	You have no GPU, or the model is small.
CUDA	You have an NVIDIA GPU. This is the fastest choice on NVIDIA hardware.
Metal	You have a Mac with Apple silicon. No installation is necessary.
ROCm	You have an AMD GPU and the ROCm 7.2 drivers.
Vulkan	You have a GPU but no vendor driver stack. Vulkan works on many cards.
WebGPU	Your program runs in a browser.

Name the backend with the --processor flag:

Your first program

Mon, 01 Jan 0001 00:00:00 +0000

This page shows the smallest yzma program. It sends a prompt to a model and prints the answer.

Before you start

Install yzma and the llama.cpp libraries. See Quick install.
Set the YZMA_LIB environment variable.
Download the model:

yzma model get -u https://huggingface.co/QuantFactory/SmolLM2-135M-GGUF/resolve/main/SmolLM2-135M.Q4_K_M.gguf

The program

package main

import (
	"fmt"
	"os"
	"path/filepath"

	"github.com/hybridgroup/yzma/pkg/download"
	"github.com/hybridgroup/yzma/pkg/llama"
)

var (
	modelFile            = "SmolLM2-135M.Q4_K_M.gguf"
	prompt               = "Are you ready to go?"
	libPath              = os.Getenv("YZMA_LIB")
	responseLength int32 = 12
)

func main() {
	llama.Load(libPath)
	llama.LogSet(llama.LogSilent())

	llama.Init()

	model, _ := llama.ModelLoadFromFile(filepath.Join(download.DefaultModelsDir(), modelFile), llama.ModelDefaultParams())
	ctx, _ := llama.InitFromModel(model, llama.ContextDefaultParams())

	vocab := llama.ModelGetVocab(model)

	tokens := llama.Tokenize(vocab, prompt, true, false)

	batch := llama.BatchGetOne(tokens)

	sampler := llama.SamplerChainInit(llama.SamplerChainDefaultParams())
	llama.SamplerChainAdd(sampler, llama.SamplerInitGreedy())

	for pos := int32(0); pos < responseLength; pos += batch.NTokens {
		llama.Decode(ctx, batch)
		token := llama.SamplerSample(sampler, ctx, -1)

		if llama.VocabIsEOG(vocab, token) {
			fmt.Println()
			break
		}

		buf := make([]byte, 36)
		len := llama.TokenToPiece(vocab, token, buf, 0, true)

		fmt.Print(string(buf[:len]))

		batch = llama.BatchGetOne([]llama.Token{token})
	}

	fmt.Println()
}

Run it

$ go run ./examples/hello/


"Yes, I'm ready to go."

No C compiler is necessary. The normal go run command is sufficient.

Linux

Mon, 01 Jan 0001 00:00:00 +0000

Linux runs on amd64 and on arm64. Choose the tab for the processor that you want to use.

Decide where you want to put the library files, then run this command:

yzma install --lib /path/to/lib

To use an NVIDIA GPU, first install the CUDA drivers. See the CUDA installation guide.

Platforms

Mon, 01 Jan 0001 00:00:00 +0000

Operating systems and processors

Operating system	CPU	GPU
Linux	amd64, arm64	CUDA, Vulkan, HIP, ROCm, SYCL
macOS	arm64	Metal
Windows	amd64	CUDA, Vulkan, HIP, SYCL, OpenCL

A browser is also a target:

Target	CPU	GPU
Browser	wasm32 SIMD, one or more threads	WebGPU

In a browser the API is the smaller one of pkg/llamawasm. It has no audio, no video, no LoRA adapters, no saved state, and no quantization.

Boards

Board	Notes
Raspberry Pi 4 and 5	The 64 bit Raspberry Pi OS. Install with `--os trixie` or `--os bookworm`.
NVIDIA Jetson Orin	Install Jetpack first. CUDA or Vulkan.
Arduino UNO Q	Install with `--os trixie`.

Go version

yzma needs Go 1.26 or later.

Sampling

Mon, 01 Jan 0001 00:00:00 +0000

A model does not give one token. It gives a score for every token in the vocabulary. The sampler chain turns those scores into one token.

The chain

Each sampler in the chain changes the scores, or it takes a token. The samplers run in the order that you add them.

sampler := llama.SamplerChainInit(llama.SamplerChainDefaultParams())
llama.SamplerChainAdd(sampler, llama.SamplerInitTopK(40))
llama.SamplerChainAdd(sampler, llama.SamplerInitTopP(0.95, 1))
llama.SamplerChainAdd(sampler, llama.SamplerInitMinP(0.05, 1))
llama.SamplerChainAdd(sampler, llama.SamplerInitTemp(0.8))
llama.SamplerChainAdd(sampler, llama.SamplerInitDist(llama.DefaultSeed))
defer llama.SamplerFree(sampler)

Put the sampler that takes the token last. That is SamplerInitDist or SamplerInitGreedy.

Examples and tutorials

Mon, 01 Jan 0001 00:00:00 +0000

Running local LLMs and VLMs on the Arduino UNO Q with yzma

Marc Pous (@mpous) shows how to run local LLMs and VLMs on the Arduino UNO Q with yzma and llama.cpp.

https://projecthub.arduino.cc/marc-edgeimpulse/running-local-llms-and-vlms-on-the-arduino-uno-q-with-yzma-74e288

Ardan Labs ai-training

Examples that show Go developers how to use AI in their products.

https://github.com/ardanlabs/ai-training

Captions With Attitude

Captions in your browser, from your webcam, made by a Vision Language Model. The Go program runs on your own machine with llama.cpp.

Vision

Mon, 01 Jan 0001 00:00:00 +0000

A Vision Language Model reads an image and text together. yzma sends both with the pkg/mtmd package.

Before you start

A VLM needs two files. Download both:

yzma model get -u https://huggingface.co/ggml-org/Qwen2.5-VL-3B-Instruct-GGUF/resolve/main/Qwen2.5-VL-3B-Instruct-Q8_0.gguf
yzma model get -u https://huggingface.co/ggml-org/Qwen2.5-VL-3B-Instruct-GGUF/resolve/main/mmproj-Qwen2.5-VL-3B-Instruct-Q8_0.gguf

The file with mmproj in the name is the projector. It turns the image into tokens that the model reads.

Run the example

$ go run ./examples/vlm/ -model ~/models/Qwen2.5-VL-3B-Instruct-Q8_0.gguf -mmproj ~/models/mmproj-Qwen2.5-VL-3B-Instruct-Q8_0.gguf -image ./images/domestic_llama.jpg -p "What is in this picture?"

The image features a white llama standing in a fenced-in area, possibly a zoo or a farm. The llama is positioned in the center of the image, with its body facing the right side. The fenced area is surrounded by trees, creating a natural environment for the llama.

See the code.

Embeddings

Mon, 01 Jan 0001 00:00:00 +0000

An embedding is a vector of numbers that stands for a piece of text. Two pieces of text with a similar meaning give two similar vectors. Use embeddings for search, for grouping, and for comparison.

Before you start

Download a model:

yzma model get -u https://huggingface.co/QuantFactory/SmolLM-135M-GGUF/resolve/main/SmolLM-135M.Q2_K.gguf

Run the example

$ go run ./examples/embeddings/ -model ~/models/SmolLM-135M.Q2_K.gguf -p "Hello World"
-0.009294 -0.003438 0.004629 -0.005551 0.003048 ...

See the code.

The steps

1. Turn on embeddings in the context parameters.

Examples

Mon, 01 Jan 0001 00:00:00 +0000

The examples directory holds these programs. Each one has its own README.

Native

Example	What it shows
`hello`	The smallest program. Loads a model and prints an answer.
`chat`	An interactive chat that keeps the conversation.
`vlm`	Sends an image and a prompt to a Vision Language Model.
`describe`	Describes an image from a URL or a local file. Installs as a command.
`embeddings`	Turns text into a vector.
`tooluse`	One round of tool calling.
`multitool`	Tool calling in a loop, for a problem with steps.
`modelinfo`	Prints the description, the size, and the metadata of a model.
`systeminfo`	Lists the devices that `llama.cpp` found.
`installer`	Installs the `llama.cpp` libraries from Go code.
`resolver`	Installs from your own mirror with a custom resolver.

Browser

Example	What it shows
`wasm/chat`	A chat page.
`wasm/vlm`	A page that asks a question about an image.
`wasm/tools`	A page where the model calls tools.

Run one

Clone the repository, set YZMA_LIB, and download a model.

Blog posts and video

Mon, 01 Jan 0001 00:00:00 +0000

Understanding Go AI Inference: Yzma

Jesús Espino

https://internals-for-interns.com/posts/yzma/

Understanding Go AI Inference: What is Inference?

Jesús Espino

https://internals-for-interns.com/posts/go-ai-inference-what-is-inference/

Free Yourself From The Copilot Tax

Bill Kennedy

https://www.kronkai.com/blog/free-yourself-from-the-copilot-tax

Building Gemma 4 Local-Powered LLM Apps with Go and Yzma

Vladimir Vivien

https://medium.com/@vladimirvivien/building-gemma-4-local-powered-llm-apps-with-go-and-yzma-6bc43d48ee4e

yzma 1.7: Perform Local Inference with llama.cpp

Golang Weekly

https://golangweekly.com/issues/588

Go Lightning talks at FOSDEM 2026

Bill Kennedy

https://cuddly.tube/w/mBtDbXQLdeHQh1PqLmeV99?start=7m58s

Go in the AI/ML Landscape: A Practical Guide

Vladimir Vivien

https://medium.com/@vladimirvivien/go-in-the-ai-ml-landscape-a-practical-guide-d36d44f360d2

Multimodal input

Mon, 01 Jan 0001 00:00:00 +0000

pkg/mtmd sends more than text to a model. It handles images, audio, and video.

Load the projector

A multimodal model needs a projector file. The projector turns the image, the sound, or the video into tokens.

mtmd.Load(libPath)

mctxParams := mtmd.ContextParamsDefault()
mtmdCtx, err := mtmd.InitFromFile(projFile, model, mctxParams)
defer mtmd.Free(mtmdCtx)

Check what the model accepts

mtmd.SupportVision(mtmdCtx)
mtmd.SupportAudio(mtmdCtx)
mtmd.SupportVideo(mtmdCtx)

Each one gives a boolean. Check before you send the input.

Images

Read an image from a file:

bitmap := mtmd.BitmapInitFromFile(mtmdCtx, imageFile, false, mtmd.InitOptDefault())
defer mtmd.BitmapFree(bitmap.Bitmap)

Read an image from memory:

Next steps

Mon, 01 Jan 0001 00:00:00 +0000

Your first program works. Here is what to do next.

Learn the parts

Architecture shows how yzma calls llama.cpp.
Packages shows what each Go package does.
Hardware acceleration shows which processor to use.

Write more programs

Chat keeps a conversation.
Vision asks a question about an image.
Embeddings turns text into a vector.
Tool calling lets a model call your Go functions.
Browser runs the same program in a web page.

Read the reference

Command line lists every yzma command.
Environment variables lists every setting.
Examples lists the example programs in the repository.
The Go API is on pkg.go.dev.

See what other people build

The Projects section shows the tools, the applications, and the articles that use yzma.

Packages

Mon, 01 Jan 0001 00:00:00 +0000

yzma is a set of Go packages. Most programs use two or three of them.

Package	What it does
`pkg/llama`	The main bindings. Models, contexts, tokens, batches, samplers, and the memory of a conversation.
`pkg/mtmd`	Multimodal input. Images, audio, and video.
`pkg/download`	Downloads the `llama.cpp` libraries and the models. Checks the digests.
`pkg/message`	Chat messages and tool calls. Reads the tool call format of each model family.
`pkg/template`	Jinja chat templates.
`pkg/llamawasm`	The browser version of `pkg/llama`. Build tag `js && wasm`.
`pkg/vlm`	An experimental high level type for Vision Language Models.
`pkg/loader`	Opens a shared library and prepares a function call.
`pkg/utils`	Converts a Go string to a C string and back.

pkg/llama

This is the package that most programs import. It gives about 250 functions, in these groups.

Windows

Mon, 01 Jan 0001 00:00:00 +0000

Choose the tab for the processor that you want to use.

Decide where you want to put the library files, then run this command:

yzma install --lib C:\path\to\lib

To use an NVIDIA GPU, first install the CUDA drivers. See the CUDA installation guide.

Packages

Mon, 01 Jan 0001 00:00:00 +0000

The complete API is on pkg.go.dev.

Package	Import path	What it does
llama	`pkg/llama`	The main bindings for `llama.cpp`.
mtmd	`pkg/mtmd`	Images, audio, and video.
download	`pkg/download`	Installs the libraries and the models.
message	`pkg/message`	Chat messages and tool calls.
template	`pkg/template`	Jinja chat templates.
llamawasm	`pkg/llamawasm`	The browser version. Build tag `js && wasm`.
vlm	`pkg/vlm`	An experimental high level type for Vision Language Models.
loader	`pkg/loader`	Opens a shared library and prepares a call.
utils	`pkg/utils`	Converts a Go string to a C string and back.

See Packages for what each one is for.

Raspberry Pi

Mon, 01 Jan 0001 00:00:00 +0000

yzma runs on a Raspberry Pi 4 and on a Raspberry Pi 5. You need the 64 bit version of the Raspberry Pi OS.

The Raspberry Pi OS uses its own library versions, so you must name the operating system with the --os flag.

Use this command for the latest version of the Raspberry Pi OS:

Tool calling

Mon, 01 Jan 0001 00:00:00 +0000

Tool calling lets a model ask your program to run a function. The model writes a request, your code runs the function, and you send the result back.

See the tutorial for a complete program. This page holds the details.

The parts

pkg/message holds these types.

Type	What it is
`ToolDefinition`	The description of a tool that you give the model.
`ToolFunctionDefinition`	The name, the description, and the parameters of a function.
`ToolCall`	A request from the model to run one function.
`ToolFunction`	The name and the arguments of that request.
`Tool`	A message that holds tool calls, and the text that came with them.
`ToolResponse`	The result that you send back to the model.
`Chat`	A normal message with a role and content.

Read the tool calls

toolCalls := message.ParseToolCalls(response)

ParseToolCalls finds the format without help. It reads every format that the table below lists.

Tool calling

Mon, 01 Jan 0001 00:00:00 +0000

A model cannot do arithmetic well and it cannot read the clock. Tool calling solves this. You tell the model which functions it has, the model asks for one, and your Go code runs it.

Before you start

Use a model that supports tool calling:

yzma model get -u https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-fp16.gguf

Run the example

$ go run ./examples/tooluse/ -model ~/models/qwen2.5-0.5b-instruct-fp16.gguf
=== Tool Calling Example ===

User: What is 15 + 27?

Assistant: 15 + 27 = 42

The result of adding 15 and 27 is 42.

Ask your own question:

WebAssembly

Mon, 01 Jan 0001 00:00:00 +0000

yzma runs in a browser. The model stays on the machine of the reader, and no server does the work.

Two modules

There are two WebAssembly modules in the page.

flowchart TD
    page["page (index.html)"]
    worker["Web Worker"]
    go["Go program (TinyGo)"]
    cpp["llama.cpp module (Emscripten)"]

    page -->|postMessage| worker
    worker --> go
    worker --> cpp
    go -->|through JavaScript| cpp

On a native platform yzma calls llama.cpp with libffi and opens the shared libraries at run time. A WebAssembly module has no dlopen and no libffi, and TinyGo cannot compile the C++ of llama.cpp.

Browser

Mon, 01 Jan 0001 00:00:00 +0000

yzma runs in a browser. The model stays on the machine of the reader, and no server does the work.

Before you start

You need TinyGo 0.42.0 or later, and a clone of the yzma repository.

git clone https://github.com/hybridgroup/yzma.git
cd yzma

Get the WebAssembly build

make download-llama.cpp-wasm

This is the same as:

yzma install --lib ./build/wasm --os wasm

It downloads all three builds. There is a build for WebGPU, a build with more than one thread, and a build with one thread.

NVIDIA Jetson Orin

Mon, 01 Jan 0001 00:00:00 +0000

yzma runs on the NVIDIA Jetson Orin. Install the latest version of the Jetpack software for your device before you start. Jetpack gives you the GPU drivers.

Decide where you want to put the library files, then run this command:

yzma install --lib /path/to/lib --processor cuda

Vulkan on the Jetson Orin needs a newer GLIBC. Upgrade the shared libraries first:

Install from your program

Mon, 01 Jan 0001 00:00:00 +0000

Your application can install the llama.cpp libraries itself. Then the person who uses it does not have to run the yzma command.

pkg/download does this work.

Install

import (
	"context"
	"runtime"

	"github.com/hybridgroup/yzma/pkg/download"
)

target := download.Target{
	Arch:      download.MustParseArch(runtime.GOARCH),
	OS:        download.MustParseOS(runtime.GOOS),
	Processor: download.CUDA,
	Version:   "latest",
}

err := download.Install(context.Background(), target, libPath, download.ProgressTracker, nil)

The last argument is the resolver. A nil value uses the built in table. See Custom resolvers.

See the example code.

The target

Field	What it holds
`Arch`	`download.AMD64` or `download.ARM64`.
`OS`	`download.Linux`, `download.Darwin`, `download.Windows`, `download.Bookworm`, `download.Trixie`, or `download.Wasm`.
`Processor`	`download.CPU`, `download.CUDA`, `download.Metal`, `download.Vulkan`, `download.ROCm`, or `download.WebGPU`.
`Version`	The release tag of `llama.cpp`, or `"latest"`, or an empty string.

Find the processor

yzma can find CUDA and ROCm on the machine.

Roadmap

Mon, 01 Jan 0001 00:00:00 +0000

yzma covers more than 96 percent of the llama.cpp functionality.

The complete list changes with each release, so it stays in the repository:

ROADMAP.md

That page gives a table for each area. Each row names a llama.cpp function and gives two columns. One column is the state of the wrapper in yzma. The other column is the state of the same function in a browser.

The areas

Backend, Model, Vocab, Context, Backend Sampling, Memory, Batch, Sampling, Logging, Performance, Chat, State, LoRA, and mtmd.

Verification

Mon, 01 Jan 0001 00:00:00 +0000

yzma install downloads shared libraries and then runs them in your process. Thus yzma checks each file before it writes anything.

Digests

Install reads the SHA-256 of each asset and compares it with the expected value. The expected values come from the manifest that llama-cpp-builder publishes for each release tag.

The manifest is an asset of the release, and there is a copy beside the version files:

https://github.com/hybridgroup/llama-cpp-builder/releases/download/b10783/b10783.json
https://hybridgroup.github.io/llama-cpp-builder/digests/b10783.json

Both hold the same bytes. yzma reads the release asset first. It falls back to the copy.

Arduino UNO Q

Mon, 01 Jan 0001 00:00:00 +0000

yzma runs on the Arduino UNO Q board. The board runs a Debian based Linux, so the installation uses the trixie operating system name.

yzma install --lib /path/to/lib --processor cpu --os trixie

Then set the YZMA_LIB environment variable:

export YZMA_LIB=/path/to/lib

The board has a small amount of memory. Use a small model. See Models.

Next steps

Go to Download models.

Benchmarks

Mon, 01 Jan 0001 00:00:00 +0000

yzma is fast because it calls llama.cpp in the same process. There is no external server.

The complete measurements change with each release, so they stay in the repository:

BENCHMARKS.md

That page has results for Linux with a CPU on amd64 and arm64, Linux with CUDA, ROCm, and Vulkan, macOS with Metal, and Windows with a CPU, CUDA, and Vulkan. Each group has text model results and multimodal model results.

Custom resolvers

Mon, 01 Jan 0001 00:00:00 +0000

A Resolver reports the files to install for a target. Install downloads them in the order that the resolver gives.

Write a resolver when you need a build that the built in table does not name. For example:

An internal mirror or an artifact server.
A file:// path on a machine with no network.
Your own build of llama.cpp.
A CUDA major version that is not the default.

Write one

resolver := download.ResolverFunc(func(t download.Target) ([]string, error) {
	if t.OS == download.Linux && t.Arch == download.AMD64 && t.Processor == download.CUDA {
		return []string{"https://mirror.example.com/llama/" + t.Version + "-cuda12-x64.tar.gz"}, nil
	}

	// Anything that you do not care about falls through to the built in table.
	return download.DefaultResolver.Resolve(t)
})

err := download.Install(context.Background(), target, libPath, download.ProgressTracker, resolver)

See the example code.

Browser

Mon, 01 Jan 0001 00:00:00 +0000

yzma runs in a browser with the WebAssembly build of llama.cpp.

yzma install --lib /path/to/web --os wasm

This command downloads all three builds. There is a build for WebGPU, a build with more than one thread, and a build with one thread. The JavaScript glue selects the best build that the page can run.

Build	What the browser must have
WebGPU	WebGPU with f16 shaders, and JSPI. Chrome or Edge 137 and later, or Firefox 153 and later with two settings in `about:config`.
More threads	`SharedArrayBuffer`, so the page must send the COOP header and the COEP header.
One thread	Nothing. It works in every browser.

A browser with no WebGPU still works. It runs on the CPU.

llama.cpp versions

Mon, 01 Jan 0001 00:00:00 +0000

Sometimes a change in llama.cpp breaks yzma. This page gives the known compatible versions.

For the current list, see the README.

Tagged releases

llama.cpp	yzma
v0.3.0	v1.25.0
v0.4.0	v1.26.0 to v1.26.1
v0.4.1	v1.27.0

A tagged release of yzma installs its own llama.cpp release by default. Thus yzma install with no --version flag gets the version in this table.

Use -version latest to get the most recent nightly build instead. A build from the main branch always uses the most recent nightly build.

Verify an installation

Mon, 01 Jan 0001 00:00:00 +0000

yzma install downloads shared libraries and your program then runs them. Thus yzma checks each file.

See Verification for how the checks work. This page shows the commands and the Go code.

Check an installation

yzma verify --lib /path/to/lib

Flag	What it does
`--lib`, `-l`	The directory with the libraries. Also reads `YZMA_LIB`.
`--version`, `-v`	The `llama.cpp` version that must be there.
`--strict`	Also fails when the directory holds a file that this install did not put there.
`--json`	Writes the report as JSON.

The check reads the manifest that the install kept, so it needs no network.

Build for a browser

Mon, 01 Jan 0001 00:00:00 +0000

This page holds the details for a program that runs in a browser. See the tutorial to build and run one, and WebAssembly for how the parts fit together.

Select the build

yzma-loader.js selects the best build that the browser can run.

Build	What the browser must have
`yzma_wasm_webgpu`	WebGPU with f16 shaders, and JSPI. Chrome and Edge 137 or later.
`yzma_wasm_mt`	`SharedArrayBuffer`, so a page with the COOP header and the COEP header.
`yzma_wasm`	Nothing. It works in every browser.

A page can set the choice with globalThis.yzmaMode. The values are auto, which is the default, webgpu, and cpu. A page URL also accepts ?mode=cpu or ?mode=webgpu.

Manual installation

Mon, 01 Jan 0001 00:00:00 +0000

You do not have to use the yzma install command. You can download and extract the library files yourself.

Where to get the files

Most of the prebuilt llama.cpp binaries are here:

https://github.com/ggml-org/llama.cpp/releases

We build the Ubuntu arm64 CUDA and Vulkan binaries, and the WebAssembly builds. They are here:

https://github.com/hybridgroup/llama-cpp-builder/releases

Extract the files

Extract the library files into a directory on your machine. The file extension depends on the operating system.

Cross compiling

Mon, 01 Jan 0001 00:00:00 +0000

yzma uses no CGo, so cross compilation is the normal Go workflow. Set GOOS and GOARCH, and build.

GOOS=linux GOARCH=arm64 go build -o myapp-arm64 .

You need no C compiler and no cross toolchain.

Why this works

A CGo program must compile C code for the target, so it needs a cross toolchain for each target. yzma calls llama.cpp through purego and libffi. Nothing about llama.cpp is part of your binary, so the Go compiler alone is sufficient.