Install from your program

Your application can install the llama.cpp libraries itself. Then the person who uses it does not have to run the yzma command.

pkg/download does this work.

Install

import (
	"context"
	"runtime"

	"github.com/hybridgroup/yzma/pkg/download"
)

target := download.Target{
	Arch:      download.MustParseArch(runtime.GOARCH),
	OS:        download.MustParseOS(runtime.GOOS),
	Processor: download.CUDA,
	Version:   "latest",
}

err := download.Install(context.Background(), target, libPath, download.ProgressTracker, nil)

The last argument is the resolver. A nil value uses the built in table. See Custom resolvers.

See the example code.

The target

Find the processor

yzma can find CUDA and ROCm on the machine.

switch {
case download.HasCUDA():
	target.Processor = download.CUDA
case download.HasROCm():
	target.Processor = download.ROCm
default:
	target.Processor = download.CPU
}

The version

• An empty string takes download.DefaultVersion. That is the llama.cpp release that this yzma release was tested with. A development build of yzma leaves that empty, so an empty string then gets the most recent nightly build.
• "latest" always gets the most recent nightly build.
• A tag such as b10783 gets that release.

Install resolves the version to a release tag before it calls the resolver, so the resolver always sees a concrete version.

LlamaLatestVersion and LlamaNightlyTag read the current tags.

Do not install twice

if download.AlreadyInstalled(libPath, target) {
	return nil
}

Show the progress

download.ProgressTracker prints the progress to the terminal. DefaultProgressTracker makes a new one. Pass nil for no output. Write your own tracker to show the progress in your own user interface.

Download a model

err := download.GetModel(modelURL, download.DefaultModelsDir())

GetModelWithProgress and GetModelWithContext give you a tracker and a context.

Check the files

Install checks the SHA-256 of each asset before it writes anything. See Verify an installation.

Ship the libraries with your application

You do not have to download at all. Put the library files beside your program and give the path to llama.Load:

llama.Load(filepath.Join(exeDir, "lib"))

This gives you a program that works with no network. The cost is a larger download for the application, and you must build one package for each platform.