unsloth-orchestrator

Thin control plane for a 2-machine Unsloth Studio fleet on Fedora CoreOS.

Most of what this repo does is configure off-the-shelf parts (Unsloth Studio, LiteLLM, Caddy, oauth2-proxy, PocketID) into a coherent shape. The Orchestrator binary is small: an operator landing page, a unix-socket admin CLI, and a reconciler that keeps LiteLLM's config in sync with the operator's intent.

Read first: CONTEXT.md for vocabulary, docs/architecture.md for the picture, and docs/adr/ for the decisions.

Topology in one sentence

Clients hit LiteLLM with an sk-litellm-… key → LiteLLM proxies to one of the two Workers using an sk-unsloth-… key the operator pasted earlier → the Worker is a single off-the-shelf unsloth/unsloth container with its full UI accessible at worker-<n>.<domain> (via Caddy + PocketID).

Repo layout

CONTEXT.md                 — vocabulary
docs/
  architecture.md          — diagram + request flows + state map
  adr/                     — design decisions (read 0001 first)
  quadlet-patterns.md      — Fedora CoreOS / Quadlet conventions
quadlet/
  orchestrator.container   — Orchestrator daemon (control plane)
  orchestrator-data.volume — its SQLite
  unsloth-studio.container — runs on each Worker
  unsloth-studio-home.volume
  huggingface-cache.volume
  litellm.container
  litellm-postgres.container
  litellm-postgres-data.volume
  litellm/config.yaml      — static bootstrap (general_settings only)
  oauth2-proxy.container   — template (skip if you already have one)
  caddy/orchestrator.caddyfile.snippet
cmd/
  orchestrator/main.go     — daemon entrypoint
  unsloth-orch/main.go     — operator CLI
internal/
  config, store, server, litellm
scripts/
  sops-bootstrap.sh        — first-install secret provisioning
Containerfile              — multi-stage build for the daemon + CLI
upstream-issues/           — tracker for unslothai/unsloth#5031

Deploy

Three Fedora CoreOS machines: one control plane (no GPU needed), two Workers (one GPU each). All three on a shared Tailscale (or other Overlay) network. Caddy + PocketID already deployed on the control plane.

1. Provision secrets

On any host with sops and age:

SOPS_AGE_RECIPIENT="age1..." ./scripts/sops-bootstrap.sh

This generates /etc/unsloth/{orchestrator,litellm,litellm-postgres,worker}.enc.yaml. Copy worker.enc.yaml to both Workers; copy the rest to the control plane. The matching age private key goes in /etc/unsloth/secrets-age.key (mode 0400) on every machine.

2. Build and ship the orchestrator image

On a build host (or directly on the control plane):

podman build -t localhost/unsloth-orchestrator:v0.1 .

Pin by digest in quadlet/orchestrator.container after the first build.

3. Install Quadlets

On the control plane, copy the orchestrator + LiteLLM + (optionally) oauth2-proxy quadlets into /etc/containers/systemd/:

/etc/containers/systemd/orchestrator.container
/etc/containers/systemd/orchestrator-data.volume
/etc/containers/systemd/litellm.container
/etc/containers/systemd/litellm-postgres.container
/etc/containers/systemd/litellm-postgres-data.volume

Drop the static quadlet/litellm/config.yaml at /etc/litellm/config.yaml on the control plane. It carries only general_settings + litellm_settings — the model list lives in LiteLLM's Postgres and is owned by the Orchestrator via the admin API.

On each Worker, copy the Unsloth quadlets:

/etc/containers/systemd/unsloth-studio.container
/etc/containers/systemd/unsloth-studio-home.volume
/etc/containers/systemd/huggingface-cache.volume

Then systemctl daemon-reload && systemctl enable --now everything.

4. Wire Caddy

Merge the contents of quadlet/caddy/orchestrator.caddyfile.snippet into your existing Caddyfile (or import it). Substitute your real domain for unsloth.example.

Reload Caddy.

5. Register Workers + load models

ssh control-plane sudo unsloth-orch worker add worker-1 worker-1.overlay
ssh control-plane sudo unsloth-orch worker add worker-2 worker-2.overlay

Visit https://worker-1.<domain>/, log into Unsloth (the bootstrap password from step 1), pick a model from the model browser, mint a UI API key (Settings → API → Create). Copy the sk-unsloth-… token. Then:

ssh control-plane sudo unsloth-orch worker set-key worker-1 sk-unsloth-...

Repeat for worker-2. The Orchestrator's reconciler POSTs the new model entry to LiteLLM's admin API (/model/new → Postgres) within 30 seconds. No LiteLLM restart needed; the change takes effect on the next request.

6. Mint a Client key for yourself

ssh control-plane sudo unsloth-orch key create my-laptop

The CLI prints sk-litellm-… once. Point your client at https://litellm.<domain>/v1 with that token.

Operating

• Add a model to a Worker: open the Worker's proxied UI, load it from the model browser. Nothing on the orchestrator side to change — LiteLLM's wildcard routing picks it up on the next /v1/models call.
• Swap which model a Worker is serving: same — UI does it.
• Rotate a Worker key: mint a new one in the Worker's UI, then unsloth-orch worker set-key <name> <new-key>. Reconciler pushes the new key to LiteLLM within 30s; no restart.
• Revoke a Client key: unsloth-orch key revoke <name>.
• Check sync health: the shell UI shows a red banner if the reconciler is failing, plus last_sync_at. CLI: unsloth-orch status.

For incident response (LiteLLM down, key rotation, backups, host reinstall, SOPS-age key compromise) see docs/runbook.md.

Development

The daemon supports an insecure dev mode that fakes X-Forwarded-User when oauth2-proxy isn't in the path. Never use in production.

ORCHESTRATOR_DEV_INSECURE=1 \
ORCHESTRATOR_STATE_DB=/tmp/state.db \
ORCHESTRATOR_ADMIN_SOCKET=/tmp/admin.sock \
ORCHESTRATOR_LITELLM_URL=http://127.0.0.1:4000 \
LITELLM_MASTER_KEY=dev \
go run ./cmd/orchestrator

Upstream issues tracked

• unslothai/unsloth#5031 — option to disable Unsloth's UI auth in trusted-network deployments; would let us remove the double-login UX. See upstream-issues/unslothai-unsloth-5031-disable-auth.md.