How to Swap the GGUF Model in SynapCores (Native + Docker)

Updated for SynapCores CE v1.14.3-ce. Every command below was executed verbatim against that build; the outputs shown are the real ones. An earlier version of this article documented the pre-v1.8 filename-based model loader — that mechanism no longer exists. See What changed below if you followed the old guide.

SynapCores CE runs its own in-process inference engine (llama.cpp), so GENERATE(), EMBED(), and AGENT_RUN() work with no API keys, no Ollama daemon, and no cloud dependency. This guide shows how to change which model it uses.

What changed (if you followed the older version of this guide)

The old flow was: download a .gguf from Hugging Face, drop it in models/text/, and set model to the filename minus the extension.

That is gone. Since v1.8.0 the engine has a model registry: models are referenced by registry name (qwen2.5-coder:7b), pulled as a manifest plus content-addressed blobs, and stored under <data_dir>/models/. The GGUF, its chat template, its params, and (for vision models) the projector sidecar are fetched and kept together — which is what makes a swap a one-line config change instead of a hunt for a matching prompt template.

Nothing is baked into the binary or the Docker image. A fresh install ships zero models and fetches what it needs on first use:

$ docker run --rm --entrypoint /opt/synapcores/synapcores \
    synapcores/community:latest models list
NAME  ARCH  SIZE  PULLED

The defaults

The shipped gateway.toml looks like this:

[query.ai_service]
provider        = "local"
model           = "qwen2.5-coder:7b"
embedding_model = "all-minilm:latest"
auto_pull       = true

There is one provider and several model slots:

auto_pull = true means you can skip the CLI entirely: set a name, restart, and the engine fetches it on the first call that needs it. The CLI exists so you can do that download deliberately — before a demo, on a build host, or behind a proxy — instead of on the first user request.

Managing models with the CLI

This is the part that replaces the old download-and-drop workflow.

synapcores pull

$ synapcores pull qwen2.5:0.5b
pulling library/qwen2.5:0.5b
verifying sha256... ok
writing manifest sidecar... ok
installed library/qwen2.5:0.5b (architecture=qwen2, size=379.4 MiB, digest=sha256:a8b0c5157701)

Accepted name forms — the same ones gateway.toml takes:

qwen2.5:0.5b                           library + default registry
library/all-minilm:latest              explicit namespace
user/model:tag                         third-party namespace
registry.example.com/user/model:tag    fully-qualified

Three properties worth knowing, because they change how you script it:

• Resumable. An interrupted download leaves a .partial file; re-running pull resumes from the byte offset already on disk via a range request, and the streaming hash verifier picks up mid-stream, so the end-to-end checksum is still correct.
• Verified. Every blob is sha256-checked against the digest the registry advertised. A mismatch quarantines the partial as <sha>.bad and exits non-zero — a corrupt download is never silently overwritten by a retry.
• Idempotent. Pulling a model you already have returns immediately when the local manifest digest matches the registry's. Safe in a provisioning script that runs on every boot.

synapcores models list

$ synapcores models list
NAME                       ARCH   SIZE       PULLED
library/all-minilm:latest  bert   43.8 MiB   1 day ago
library/qwen2.5-coder:7b   qwen2  4.4 GiB    1 day ago
library/qwen2.5:0.5b       qwen2  379.4 MiB  0 seconds ago

synapcores models delete

$ synapcores models delete qwen2.5:0.5b
deleted library/qwen2.5:0.5b (reclaimed 379.4 MiB)

Deleting removes the sidecar, then garbage-collects any blob no longer referenced by another installed model — blobs shared with a model you kept survive. It is idempotent, so scripted cleanup never fails on a double-delete:

$ synapcores models delete not-a-real-model:tag
model 'library/not-a-real-model:tag' is not installed
$ echo $?
0

Which data directory does the CLI write to?

Models land under the gateway's data_dir. When you invoke the CLI with --config, it reads data_dir from that file. Without --config, it uses $AIDB_DATA_DIR:

# Explicit config — writes to the data_dir in that file
synapcores --config /etc/synapcores/gateway.toml pull qwen2.5-coder:7b

# No config — writes to $AIDB_DATA_DIR
AIDB_DATA_DIR=/var/lib/synapcores synapcores pull qwen2.5-coder:7b

Get this wrong and the pull succeeds into a directory the gateway never reads — the classic "I downloaded it but the engine still says model not found".

Swap on a native install

The whole swap is: pull the model, point gateway.toml at it, restart.

Step 1 — Pull the model

sudo -u synapcores /opt/synapcores/synapcores \
  --config /etc/synapcores/gateway.toml pull qwen2.5:0.5b

Passing --config matters: it makes the CLI write into the same data_dir the gateway reads.

Step 2 — Point the config at it

[query.ai_service]
provider        = "local"
model           = "qwen2.5:0.5b"     # was "qwen2.5-coder:7b"
embedding_model = "all-minilm:latest"
auto_pull       = true

Leave embedding_model alone unless you are deliberately changing embeddings. Changing it invalidates every vector already stored — existing embeddings were produced by the old model and are not comparable to new ones. Re-embed if you do change it.

Step 3 — Restart

sudo systemctl restart synapcores

Step 4 — Verify the running engine actually uses it

TOKEN=$(curl -fsS -X POST http://127.0.0.1:8080/v1/auth/login \
  -H 'Content-Type: application/json' \
  -d '{"username":"admin","password":"<your-admin-password>"}' \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["access_token"])')

curl -sS --max-time 300 -X POST http://127.0.0.1:8080/v1/query/execute \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"sql":"SELECT GENERATE('"'"'Reply with exactly: SWAP_OK'"'"') AS answer"}'

Verified on v1.14.3-ce after swapping qwen2.5-coder:7b → qwen2.5:0.5b:

{"data":{"rows":[["SWAP_OK"]],"rows_affected":1}}

models list is the fast sanity check that the file is where the engine will look for it; the GENERATE round-trip is the proof that it loaded.

Swap on the Docker image

Nothing is baked into the image, and models live in the data volume, not the image layer. So the swap is a pull into the volume plus a config mount — you never rebuild or bind-mount a .gguf.

Step 1 — Get the shipped config

mkdir -p ./model-swap && cd ./model-swap

docker run --rm --entrypoint cat \
  synapcores/community:latest \
  /etc/synapcores/gateway.toml > gateway.toml

Step 2 — Pull the model into the data volume

The CLI lives at /opt/synapcores/synapcores inside the image, and the image's AIDB_DATA_DIR is /opt/synapcores/aidb_data:

docker volume create synapcores-data

docker run --rm \
  -v synapcores-data:/opt/synapcores/aidb_data \
  --entrypoint /opt/synapcores/synapcores \
  synapcores/community:latest pull qwen2.5:0.5b

Confirm it landed in the volume:

docker run --rm \
  -v synapcores-data:/opt/synapcores/aidb_data \
  --entrypoint /opt/synapcores/synapcores \
  synapcores/community:latest models list

Step 3 — Edit the config and run

sed -i 's|^model  *=.*|model           = "qwen2.5:0.5b"|' gateway.toml
grep -A 4 '\[query.ai_service\]' gateway.toml

docker run -d --name synapcores \
  -p 8080:8080 \
  -e AIDB_ACCEPT_LICENSE=1 \
  -v synapcores-data:/opt/synapcores/aidb_data \
  -v "$PWD/gateway.toml":/etc/synapcores/gateway.toml:ro \
  synapcores/community:latest

Because the model is in the volume, it survives container replacement — an upgrade re-uses the models you already pulled instead of re-downloading gigabytes.

Step 4 — Verify

PASS=$(docker logs synapcores 2>&1 | sed 's/\x1b\[[0-9;]*m//g' \
  | grep -oP 'password:\s*\K\S+' | head -1)

TOKEN=$(curl -fsS -X POST http://127.0.0.1:8080/v1/auth/login \
  -H 'Content-Type: application/json' \
  -d "{\"username\":\"admin\",\"password\":\"$PASS\"}" \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["access_token"])')

curl -sS --max-time 300 -X POST http://127.0.0.1:8080/v1/query/execute \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"sql":"SELECT GENERATE('"'"'Reply with exactly: SWAP_OK'"'"') AS answer"}'

Gotcha: the AI Chat panel uses a different model

[query.ai_service] governs SQL only — GENERATE, EMBED, NL2SQL, AGENT_RUN. The AI Chat panel in the UI has its own registry, and changing [query.ai_service].model does not change what the chat panel uses.

Built-in chat models, pre-matched to a working chat template:

To change the chat default you need an ai_chat.toml and must point the gateway at it with AI_CHAT_CONFIG_PATH. If you swapped the SQL model and the chat panel seems unchanged, this is why — it is working as designed.

Picking a model

Quantization is baked into the registry tag, so you choose it by picking the tag rather than by hunting for a file. As a rough guide for a 7B:

Divide by ~6 for a 1B, ~2.5 for a 3B. Q4_K_M is the right default — under 1% accuracy loss versus FP16 on most benchmarks at a third of the size.

Architectures the embedded runtime supports (whatever llama.cpp upstream does):

llama / llama2 / llama3.x      mistral / mixtral      gemma / gemma2 / gemma3
phi2 / phi3 / phi3.5           qwen / qwen2 / qwen2.5 / qwen3
deepseek (v2 / v3)             yi      falcon         granite (incl. 3.x / MoE)
starcoder / starcoder2         codellama              tinyllama       stablelm

A smaller model is not only faster — on a CPU-only host it is often the difference between an agent loop that finishes and one that times out. The 0.5B swap above answers in well under a second on a 6-core desktop.

Using a cloud model instead

If you would rather not run inference locally, the same [query.ai_service] block points at a hosted provider. provider accepts local, ollama, openai, anthropic, and gemini:

[query.ai_service]
provider        = "anthropic"
model           = "claude-opus-5"
embedding_model = "all-minilm"     # Anthropic has no embeddings endpoint —
                                    # EMBED() routes to the bundled local model

The API key comes from ANTHROPIC_API_KEY (or OPENAI_API_KEY, GEMINI_API_KEY / GOOGLE_API_KEY) so it need not be written to disk.

There is one provider and one base URL per instance. If you need a different vendor per modality, put an OpenAI-compatible router in front of the gateway, or run a second instance.

Cloud providers for SQL functions require v1.14.3-ce or newer. In earlier builds provider = "anthropic" was wired for the REST endpoints but not for SQL, so GENERATE() returned an empty string.