Guides on yzma

Models

Mon, 01 Jan 0001 00:00:00 +0000

yzma uses models in the GGUF format that llama.cpp supports. There are more than 201,000 of them on Hugging Face:

https://huggingface.co/models?library=gguf&sort=trending

Set the YZMA_LIB environment variable before you run any of these commands.

Vision Language Models (VLM)

A Vision Language Model takes an image and text, and it gives text back. Each one needs a model file and a projector file.

Qwen3-VL-2B-Instruct

https://huggingface.co/bartowski/Qwen_Qwen3-VL-2B-Instruct-GGUF

Good quality and a good size for most work. This is the model to start with.

Chat templates

Mon, 01 Jan 0001 00:00:00 +0000

An instruction model expects a marker before each message. The markers say which part is the system prompt, which part the user said, and where the model must write. That shape is the chat template.

Each model family has its own template. If you send the wrong shape, the model gives a poor answer or no answer.

The template in the model file

Most GGUF files hold their own template.

Sampling

Mon, 01 Jan 0001 00:00:00 +0000

A model does not give one token. It gives a score for every token in the vocabulary. The sampler chain turns those scores into one token.

The chain

Each sampler in the chain changes the scores, or it takes a token. The samplers run in the order that you add them.

sampler := llama.SamplerChainInit(llama.SamplerChainDefaultParams())
llama.SamplerChainAdd(sampler, llama.SamplerInitTopK(40))
llama.SamplerChainAdd(sampler, llama.SamplerInitTopP(0.95, 1))
llama.SamplerChainAdd(sampler, llama.SamplerInitMinP(0.05, 1))
llama.SamplerChainAdd(sampler, llama.SamplerInitTemp(0.8))
llama.SamplerChainAdd(sampler, llama.SamplerInitDist(llama.DefaultSeed))
defer llama.SamplerFree(sampler)

Put the sampler that takes the token last. That is SamplerInitDist or SamplerInitGreedy.

Multimodal input

Mon, 01 Jan 0001 00:00:00 +0000

pkg/mtmd sends more than text to a model. It handles images, audio, and video.

Load the projector

A multimodal model needs a projector file. The projector turns the image, the sound, or the video into tokens.

mtmd.Load(libPath)

mctxParams := mtmd.ContextParamsDefault()
mtmdCtx, err := mtmd.InitFromFile(projFile, model, mctxParams)
defer mtmd.Free(mtmdCtx)

Check what the model accepts

mtmd.SupportVision(mtmdCtx)
mtmd.SupportAudio(mtmdCtx)
mtmd.SupportVideo(mtmdCtx)

Each one gives a boolean. Check before you send the input.

Images

Read an image from a file:

bitmap := mtmd.BitmapInitFromFile(mtmdCtx, imageFile, false, mtmd.InitOptDefault())
defer mtmd.BitmapFree(bitmap.Bitmap)

Read an image from memory:

Tool calling

Mon, 01 Jan 0001 00:00:00 +0000

Tool calling lets a model ask your program to run a function. The model writes a request, your code runs the function, and you send the result back.

See the tutorial for a complete program. This page holds the details.

The parts

pkg/message holds these types.

Type	What it is
`ToolDefinition`	The description of a tool that you give the model.
`ToolFunctionDefinition`	The name, the description, and the parameters of a function.
`ToolCall`	A request from the model to run one function.
`ToolFunction`	The name and the arguments of that request.
`Tool`	A message that holds tool calls, and the text that came with them.
`ToolResponse`	The result that you send back to the model.
`Chat`	A normal message with a role and content.

Read the tool calls

toolCalls := message.ParseToolCalls(response)

ParseToolCalls finds the format without help. It reads every format that the table below lists.

Install from your program

Mon, 01 Jan 0001 00:00:00 +0000

Your application can install the llama.cpp libraries itself. Then the person who uses it does not have to run the yzma command.

pkg/download does this work.

Install

import (
	"context"
	"runtime"

	"github.com/hybridgroup/yzma/pkg/download"
)

target := download.Target{
	Arch:      download.MustParseArch(runtime.GOARCH),
	OS:        download.MustParseOS(runtime.GOOS),
	Processor: download.CUDA,
	Version:   "latest",
}

err := download.Install(context.Background(), target, libPath, download.ProgressTracker, nil)

The last argument is the resolver. A nil value uses the built in table. See Custom resolvers.

See the example code.

The target

Field	What it holds
`Arch`	`download.AMD64` or `download.ARM64`.
`OS`	`download.Linux`, `download.Darwin`, `download.Windows`, `download.Bookworm`, `download.Trixie`, or `download.Wasm`.
`Processor`	`download.CPU`, `download.CUDA`, `download.Metal`, `download.Vulkan`, `download.ROCm`, or `download.WebGPU`.
`Version`	The release tag of `llama.cpp`, or `"latest"`, or an empty string.

Find the processor

yzma can find CUDA and ROCm on the machine.

Custom resolvers

Mon, 01 Jan 0001 00:00:00 +0000

A Resolver reports the files to install for a target. Install downloads them in the order that the resolver gives.

Write a resolver when you need a build that the built in table does not name. For example:

An internal mirror or an artifact server.
A file:// path on a machine with no network.
Your own build of llama.cpp.
A CUDA major version that is not the default.

Write one

resolver := download.ResolverFunc(func(t download.Target) ([]string, error) {
	if t.OS == download.Linux && t.Arch == download.AMD64 && t.Processor == download.CUDA {
		return []string{"https://mirror.example.com/llama/" + t.Version + "-cuda12-x64.tar.gz"}, nil
	}

	// Anything that you do not care about falls through to the built in table.
	return download.DefaultResolver.Resolve(t)
})

err := download.Install(context.Background(), target, libPath, download.ProgressTracker, resolver)

See the example code.

Verify an installation

Mon, 01 Jan 0001 00:00:00 +0000

yzma install downloads shared libraries and your program then runs them. Thus yzma checks each file.

See Verification for how the checks work. This page shows the commands and the Go code.

Check an installation

yzma verify --lib /path/to/lib

Flag	What it does
`--lib`, `-l`	The directory with the libraries. Also reads `YZMA_LIB`.
`--version`, `-v`	The `llama.cpp` version that must be there.
`--strict`	Also fails when the directory holds a file that this install did not put there.
`--json`	Writes the report as JSON.

The check reads the manifest that the install kept, so it needs no network.

Build for a browser

Mon, 01 Jan 0001 00:00:00 +0000

This page holds the details for a program that runs in a browser. See the tutorial to build and run one, and WebAssembly for how the parts fit together.

Select the build

yzma-loader.js selects the best build that the browser can run.

Build	What the browser must have
`yzma_wasm_webgpu`	WebGPU with f16 shaders, and JSPI. Chrome and Edge 137 or later.
`yzma_wasm_mt`	`SharedArrayBuffer`, so a page with the COOP header and the COEP header.
`yzma_wasm`	Nothing. It works in every browser.

A page can set the choice with globalThis.yzmaMode. The values are auto, which is the default, webgpu, and cpu. A page URL also accepts ?mode=cpu or ?mode=webgpu.

Cross compiling

Mon, 01 Jan 0001 00:00:00 +0000

yzma uses no CGo, so cross compilation is the normal Go workflow. Set GOOS and GOARCH, and build.

GOOS=linux GOARCH=arm64 go build -o myapp-arm64 .

You need no C compiler and no cross toolchain.

Why this works

A CGo program must compile C code for the target, so it needs a cross toolchain for each target. yzma calls llama.cpp through purego and libffi. Nothing about llama.cpp is part of your binary, so the Go compiler alone is sufficient.