Concepts on yzma

Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Overview

yzma calls llama.cpp in the same process as your Go program. There is no model server and there is no network connection.

flowchart TD
    subgraph app["Your Go program"]
        code["Application code"]
    end
    subgraph yzma
        llama["pkg/llama"]
        mtmd["pkg/mtmd"]
        loader["pkg/loader"]
    end
    subgraph runtime["Run time"]
        purego["purego"]
        ffi["libffi"]
    end
    subgraph native["llama.cpp shared libraries"]
        libllama["libllama"]
        libmtmd["libmtmd"]
        ggml["ggml backends"]
    end
    subgraph hw["Hardware"]
        cpu["CPU"]
        gpu["GPU"]
    end

    code --> llama
    code --> mtmd
    llama --> loader
    mtmd --> loader
    loader --> purego
    loader --> ffi
    purego --> libllama
    purego --> libmtmd
    ffi --> libllama
    ffi --> libmtmd
    libllama --> ggml
    libmtmd --> ggml
    ggml --> cpu
    ggml --> gpu

No CGo

Most Go bindings for a C library use CGo. CGo needs a C compiler, and it makes cross compilation hard.

Models

Mon, 01 Jan 0001 00:00:00 +0000

GGUF

yzma uses models in the GGUF format. This is the format that llama.cpp reads. A GGUF file holds the weights of the model and the metadata about it, such as the name, the size of the context, and the chat template.

There are more than 201,000 GGUF models on Hugging Face:

https://huggingface.co/models?library=gguf&sort=trending

Read the metadata of a file with the modelinfo example program.

Quantization

The name of a model file usually ends with a quantization code, such as Q4_K_M or Q8_0 or fp16.

Hardware acceleration

Mon, 01 Jan 0001 00:00:00 +0000

llama.cpp has a backend for each kind of processor. yzma installs the backend that you name, and llama.cpp uses it.

What each platform supports

Operating system	CPU	GPU
Linux	amd64, arm64	CUDA, Vulkan, HIP, ROCm, SYCL
macOS	arm64	Metal
Windows	amd64	CUDA, Vulkan, HIP, SYCL, OpenCL

A browser is also a target:

Target	CPU	GPU
Browser	wasm32 SIMD, one or more threads	WebGPU

Which backend to choose

Backend	Use it when
CPU	You have no GPU, or the model is small.
CUDA	You have an NVIDIA GPU. This is the fastest choice on NVIDIA hardware.
Metal	You have a Mac with Apple silicon. No installation is necessary.
ROCm	You have an AMD GPU and the ROCm 7.2 drivers.
Vulkan	You have a GPU but no vendor driver stack. Vulkan works on many cards.
WebGPU	Your program runs in a browser.

Name the backend with the --processor flag:

Packages

Mon, 01 Jan 0001 00:00:00 +0000

yzma is a set of Go packages. Most programs use two or three of them.

Package	What it does
`pkg/llama`	The main bindings. Models, contexts, tokens, batches, samplers, and the memory of a conversation.
`pkg/mtmd`	Multimodal input. Images, audio, and video.
`pkg/download`	Downloads the `llama.cpp` libraries and the models. Checks the digests.
`pkg/message`	Chat messages and tool calls. Reads the tool call format of each model family.
`pkg/template`	Jinja chat templates.
`pkg/llamawasm`	The browser version of `pkg/llama`. Build tag `js && wasm`.
`pkg/vlm`	An experimental high level type for Vision Language Models.
`pkg/loader`	Opens a shared library and prepares a function call.
`pkg/utils`	Converts a Go string to a C string and back.

pkg/llama

This is the package that most programs import. It gives about 250 functions, in these groups.

WebAssembly

Mon, 01 Jan 0001 00:00:00 +0000

yzma runs in a browser. The model stays on the machine of the reader, and no server does the work.

Two modules

There are two WebAssembly modules in the page.

flowchart TD
    page["page (index.html)"]
    worker["Web Worker"]
    go["Go program (TinyGo)"]
    cpp["llama.cpp module (Emscripten)"]

    page -->|postMessage| worker
    worker --> go
    worker --> cpp
    go -->|through JavaScript| cpp

On a native platform yzma calls llama.cpp with libffi and opens the shared libraries at run time. A WebAssembly module has no dlopen and no libffi, and TinyGo cannot compile the C++ of llama.cpp.

Verification

Mon, 01 Jan 0001 00:00:00 +0000

yzma install downloads shared libraries and then runs them in your process. Thus yzma checks each file before it writes anything.

Digests

Install reads the SHA-256 of each asset and compares it with the expected value. The expected values come from the manifest that llama-cpp-builder publishes for each release tag.

The manifest is an asset of the release, and there is a copy beside the version files:

https://github.com/hybridgroup/llama-cpp-builder/releases/download/b10783/b10783.json
https://hybridgroup.github.io/llama-cpp-builder/digests/b10783.json

Both hold the same bytes. yzma reads the release asset first. It falls back to the copy.