Skip to main content

​
What it’s good for

Full privacy — no data leaves your machine. Run any GGUF model with direct control over quantization, context size, and GPU layers. Best for users who want maximum control over their inference setup.

​
Requirements

  • llama.cpp built with llama-server
  • A GGUF model file downloaded to your machine
  • Sufficient RAM/VRAM for your chosen model and context size
Install llama.cpp:

​
Configure in Spaceduck

​
Chat

1

Start llama-server

-ngl 99 offloads all layers to GPU. Reduce this number if you run out of VRAM.
2

Configure Spaceduck

In Settings > Chat:
  • Provider: llama.cpp
  • Base URL: http://127.0.0.1:8080/v1
  • Model: leave empty (llama-server uses the loaded model)
Or via CLI:
3

Verify

You should see your loaded model in the response.

​
Embeddings

llama.cpp runs embeddings on a separate server instance with the --embeddings flag. This is the two-server pattern.
1

Start embedding server

The -hf flag downloads the model from Hugging Face automatically. You can also use -m /path/to/model.gguf for a local file.
2

Configure Spaceduck

In Settings > Memory:
  • Toggle Semantic recall on
  • Provider: llama.cpp
  • Server URL: http://127.0.0.1:8081/v1
  • Model: leave empty
  • Dimensions: 768 (for nomic-embed-text-v1.5)
Or via CLI:
3

Verify

You should get back a JSON response with an embedding array.In Settings > Memory, click the Test button — you should see a green status indicator.

​
Test and troubleshoot

If responses look wrong (e.g., raw tokens, repeated text), the most common cause is a missing chat template. Add --chat-template chatml or the appropriate template for your model to the llama-server command.