Getting Started on yzma

Download models

Mon, 01 Jan 0001 00:00:00 +0000

yzma uses models in the GGUF format that llama.cpp supports. There are more than 201,000 GGUF models on Hugging Face:

https://huggingface.co/models?library=gguf&sort=trending

Use the yzma command

The yzma command downloads a model for you:

yzma model get -u https://huggingface.co/QuantFactory/SmolLM2-135M-GGUF/resolve/main/SmolLM2-135M.Q4_K_M.gguf

The command puts the file in the default models directory. A Go program finds that directory with download.DefaultModelsDir().

Choose a model

A larger model gives better answers, but it needs more memory and more time. Start with a small model.

Your first program

Mon, 01 Jan 0001 00:00:00 +0000

This page shows the smallest yzma program. It sends a prompt to a model and prints the answer.

Before you start

Install yzma and the llama.cpp libraries. See Quick install.
Set the YZMA_LIB environment variable.
Download the model:

yzma model get -u https://huggingface.co/QuantFactory/SmolLM2-135M-GGUF/resolve/main/SmolLM2-135M.Q4_K_M.gguf

The program

package main

import (
	"fmt"
	"os"
	"path/filepath"

	"github.com/hybridgroup/yzma/pkg/download"
	"github.com/hybridgroup/yzma/pkg/llama"
)

var (
	modelFile            = "SmolLM2-135M.Q4_K_M.gguf"
	prompt               = "Are you ready to go?"
	libPath              = os.Getenv("YZMA_LIB")
	responseLength int32 = 12
)

func main() {
	llama.Load(libPath)
	llama.LogSet(llama.LogSilent())

	llama.Init()

	model, _ := llama.ModelLoadFromFile(filepath.Join(download.DefaultModelsDir(), modelFile), llama.ModelDefaultParams())
	ctx, _ := llama.InitFromModel(model, llama.ContextDefaultParams())

	vocab := llama.ModelGetVocab(model)

	tokens := llama.Tokenize(vocab, prompt, true, false)

	batch := llama.BatchGetOne(tokens)

	sampler := llama.SamplerChainInit(llama.SamplerChainDefaultParams())
	llama.SamplerChainAdd(sampler, llama.SamplerInitGreedy())

	for pos := int32(0); pos < responseLength; pos += batch.NTokens {
		llama.Decode(ctx, batch)
		token := llama.SamplerSample(sampler, ctx, -1)

		if llama.VocabIsEOG(vocab, token) {
			fmt.Println()
			break
		}

		buf := make([]byte, 36)
		len := llama.TokenToPiece(vocab, token, buf, 0, true)

		fmt.Print(string(buf[:len]))

		batch = llama.BatchGetOne([]llama.Token{token})
	}

	fmt.Println()
}

Run it

$ go run ./examples/hello/


"Yes, I'm ready to go."

No C compiler is necessary. The normal go run command is sufficient.

Next steps

Mon, 01 Jan 0001 00:00:00 +0000

Your first program works. Here is what to do next.

Learn the parts

Architecture shows how yzma calls llama.cpp.
Packages shows what each Go package does.
Hardware acceleration shows which processor to use.

Write more programs

Chat keeps a conversation.
Vision asks a question about an image.
Embeddings turns text into a vector.
Tool calling lets a model call your Go functions.
Browser runs the same program in a web page.

Read the reference

Command line lists every yzma command.
Environment variables lists every setting.
Examples lists the example programs in the repository.
The Go API is on pkg.go.dev.

See what other people build

The Projects section shows the tools, the applications, and the articles that use yzma.