Skip to content

On a laptop or a Mac

The default backbone, callbus-emb-v0.2, is an 8B model, and Docker serves it with vLLM on a GPU of 16 GB (8 GB in FP8). Without such a GPU, Ollama runs GGUF builds of the same model (yasinyaman/callbus-emb-8b-GGUF, its card), at nearly the same quality and more slowly, and of smaller Qwen3-Embedding models, which are faster and score lower.

Pick a model

Measured on a laptop with an RTX 3050 Ti (4 GB) and 14 GB of RAM, Ollama 0.35.1, num_ctx 8192. The benchmarks run with instructions, as in the README table:

Model (Ollama name) GGUF LiveMCPBench NDCG@10 MCP-Zero top-1 ToolRet NDCG@10 (cat-macro) One new search 525 tools embedded in
callbus-emb-v0.2-q4_k_m 5.0 GB 56.22 87.07 59.51 (54.53) 0.67 s 5.4 min
callbus-emb-v0.2-q8_0 8.7 GB 55.41 88.72 58.94 (54.38) 0.78 s 8.8 min
qwen3-emb-4b-q4_k_m 2.5 GB 48.71 79.19 50.41 (44.99) 0.33 s 2.8 min
qwen3-emb-0.6b-q8_0 0.6 GB 49.21 79.98 48.54 (43.14) 0.05 s 29 s
vLLM, bf16, on a GPU 55.74 88.57 58.90 (54.36)
  • Take Q8_0 if 9 GB fits, Q4_K_M if it does not. Q8_0 is level with bf16 on all three sets, with nothing to separate them on paired tests (MCP-Zero top-1 88.72 against 88.57, p 0.48; ToolRet 58.94 against 58.90, p 0.31). Q4_K_M costs 1.5 points of top-1 on MCP-Zero's 2,792 agent-written requests (87.07; 9 requests up, 51 down, p < 0.0001). The other sets do not see it: on ToolRet, where requests carry instructions and tools long documentation, Q4_K_M is 0.6 points above bf16 (p 0.0001, level by category), and LiveMCPBench's 94 tasks are too few to separate anything. Coarser weights cost most on short requests, which is what an agent writes.
  • The two callbus-emb-v0.2 rows' quality columns are the published files (yasinyaman/callbus-emb-8b-GGUF), scored through Ollama on the machine that built them; every timing column is this laptop. Quality does not depend on the hardware, timings do.
  • Qwen3-Embedding-0.6B and 4B are faster, and score about 7 points lower on LiveMCPBench, 9 to 11 lower on ToolRet and 9 lower at top-1 on MCP-Zero. They are still well ahead of BM25 (ToolRet 39.27, 36.41 cat-macro). The 0.6B's ToolRet row is the f16 build, which scores like Q8_0 on LiveMCPBench (49.12). Where they really fall behind is tasks that need more than one tool: on generated two-tool tasks over a GitHub + Stripe catalogue they score 72.3 and 71.0 against the default's 83.0, while on requests for a single tool they are within 1 to 2 points (90.1 and 91.2 against 92.5). Four times the size buys nothing here — the 4B trails the 0.6B on both MCP-Zero and the two-tool tasks.
  • One new search is the median time of a search for a request the server has not seen before: one request at a time, on LiveMCPBench, embedding included. A request seen before is answered from the embedding cache in a few milliseconds, whatever the model.
  • On a 4 GB GPU the 8B builds run mostly on the CPU. A catalogue is embedded once: ingest stores the vectors and a restart reuses them. On this laptop the first ingest embeds about 100 tools a minute with the 8B, 200 with the 4B and 700 with the 0.6B, so 2,000 tools take about 20, 10 and 3 minutes.

Set up

# the LoRA backbone as GGUF; hf: pip install huggingface_hub
hf download yasinyaman/callbus-emb-8b-GGUF callbus-emb-8b-v0.2-Q4_K_M.gguf --revision v0.2 --local-dir models
# ... -Q8_0.gguf instead (8.7 GB) where there is room: level with bf16 on MCP-Zero's short requests
printf 'FROM ./models/callbus-emb-8b-v0.2-Q4_K_M.gguf\nPARAMETER num_ctx 8192\n' > Modelfile
ollama create callbus-emb-v0.2-q4_k_m -f Modelfile

export CALLBUS_EMB_URL=http://127.0.0.1:11434/v1 CALLBUS_EMB_MODEL=callbus-emb-v0.2-q4_k_m
callbus ingest mcp --server time="uvx mcp-server-time" --out tools/ --emb-url $CALLBUS_EMB_URL
callbus serve --data tools/

The small models come from Qwen's own GGUF repositories, in the same way:

hf download Qwen/Qwen3-Embedding-0.6B-GGUF Qwen3-Embedding-0.6B-Q8_0.gguf --local-dir models
printf 'FROM ./models/Qwen3-Embedding-0.6B-Q8_0.gguf\nPARAMETER num_ctx 8192\n' > Modelfile
ollama create qwen3-emb-0.6b-q8_0 -f Modelfile

Good to know

  • Context. Ollama ignores vLLM's truncate_prompt_tokens (--truncate). It cuts a longer input at num_ctx instead, keeping the start, as vLLM does, so set num_ctx to 8192 as above. Ollama's own qwen3-embedding:0.6b and :4b work too, but with a context of 4096.
  • Heads. The packaged heads take the vectors of Qwen3-Embedding-8B (4096 numbers). The names above skip them. For a model under another name, pass --clm-ckpt none to search and serve.
  • Names. The embedding cache is keyed by the model's name, so give each build its own name (the quantization in it, as above). Two builds under one name would share vectors that differ.
  • llama.cpp's own server (llama-server --embeddings) works for short texts. In embedding mode it keeps an output row for every input token, about 0.6 MB per token for Qwen3, so an 8192-token input needs about 5 GB of GPU memory. It also ignores truncation. In our test, a longer input than fit crashed the server instead of returning an error. Ollama plans its memory and offloads to the CPU instead.
  • Apple Silicon. Ollama runs these models on the Mac's GPU. We have not measured them there.