Hello world

This is the smallest yzma program.

Before you start

Install yzma and the llama.cpp libraries. See Quick install. Then set YZMA_LIB.

Download the model:

yzma model get -u https://huggingface.co/QuantFactory/SmolLM2-135M-GGUF/resolve/main/SmolLM2-135M.Q4_K_M.gguf

Run the example

$ go run ./examples/hello/


"Yes, I'm ready to go."

The complete program is on the Your first program page.

The steps

1. Open the libraries.

llama.Load(libPath)
llama.LogSet(llama.LogSilent())
llama.Init()

LogSilent hides the messages that llama.cpp prints. Remove that line to see what the library does.

Load and Init each return an error. This program ignores them to stay short. A real program checks them.

2. Load the model and create a context.

model, _ := llama.ModelLoadFromFile(path, llama.ModelDefaultParams())
ctx, _ := llama.InitFromModel(model, llama.ContextDefaultParams())

The model holds the weights. The context holds the state of one conversation.

3. Turn the prompt into tokens.

vocab := llama.ModelGetVocab(model)
tokens := llama.Tokenize(vocab, prompt, true, false)
batch := llama.BatchGetOne(tokens)

The third argument of Tokenize adds the beginning of sequence token.

4. Make a sampler chain.

sampler := llama.SamplerChainInit(llama.SamplerChainDefaultParams())
llama.SamplerChainAdd(sampler, llama.SamplerInitGreedy())

A greedy sampler always takes the token with the highest score. So the answer is the same every time. See Sampling for the other samplers.

5. Run the loop.

for pos := int32(0); pos < responseLength; pos += batch.NTokens {
	llama.Decode(ctx, batch)
	token := llama.SamplerSample(sampler, ctx, -1)

	if llama.VocabIsEOG(vocab, token) {
		break
	}

	buf := make([]byte, 36)
	len := llama.TokenToPiece(vocab, token, buf, 0, true)
	fmt.Print(string(buf[:len]))

	batch = llama.BatchGetOne([]llama.Token{token})
}

Decode runs the model. SamplerSample takes the next token. TokenToPiece turns that token into text. The new token becomes the next batch.

VocabIsEOG reports the end of generation token. The model returns this token when it has no more to say.

Next steps

Go to Chat to keep a conversation.