HTTP API¶
The daemon is the whole control plane, and HTTP is its only interface. The Python SDK and the sky CLI are clients of this API and get no privileged path into it — anything they do, you can do with httpie.
The surface is small on purpose: 31 paths under /v1. Most of what you need to know is not the routes but the four conventions that apply across all of them.
Two families of resource¶
The split is what makes the rest of the API predictable.
Declarative resources — compute and node — carry spec (what you asked for) and status (what has been observed). PATCH only ever touches spec; status is written by the reconciler and never by a client. There is deliberately no operation or job resource to poll: progress is generation against status.observed_generation.
Imperative resources — task — are append-only facts with exactly one terminal outcome. Its executions are the physical attempts. A retry creates another execution, never another task, which is what lets an SDK Future keep a stable handle across a node that died under it.
compute ──< generation one frozen definition per revision of the spec
└─< node one machine, ranked
task ──< execution one physical attempt, ordinal-counted
provider ─< offer a cached hardware catalog
function ── blob code, arguments and results, addressed by hash
Optimistic concurrency¶
Every declarative resource carries a revision, served as an ETag:
A write that changes one must say which revision it expected:
spec takes nodes, and image on a compute created with Image(mutable=True). A mutable image changes pip, pip_indexes, includes, excludes and includes_sha256, and nothing else. A different value for any other field, or any image change on a compute whose image is fixed, is refused with 422 image_fixed, naming the field. A nodes change on a compute running a collective plugin is refused with 422 compute_not_resizable: its process group was formed with the ranks it started with.
Each node reports image in its resource, the digest of the image it materialized, and it is null until the node first reaches ready. A ready node whose digest differs from the compute's spec.image is refreshed; the refresh shows as node.bootstrapping followed by node.ready, and status.observed_generation catches up with generation once every node reports the current digest. No event type is added for it.
If the stored revision has moved on, the write is refused with 412 and revision_conflict. That error is retryable: re-read, re-apply, re-send. Every successful write bumps the revision.
This is what keeps two clients — your script and a sky compute delete in another terminal — from silently overwriting each other's intent.
Idempotency¶
Any request that creates something takes an Idempotency-Key:
The daemon stores the key alongside a fingerprint of the request. The same key with the same request is a retry, and returns the original resource rather than creating a second. The same key with a different request is a bug on the caller's side, and gets 409 idempotency_conflict instead of a resource nobody asked for.
This matters more than it looks. The client cannot tell a lost response from a lost request, so without it every network blip during provisioning risks paying for a second cluster.
Errors¶
Every failure is the same JSON object, whatever produced it:
{
"code": "revision_conflict",
"message": "compute cmp_7f3a1c is at revision 9",
"retryable": true,
"request_id": "...",
"details": {"if_match": "7"}
}
code is a closed set, so a client matches on it rather than parsing prose. retryable says whether trying again can plausibly work — it is a property of the error, not a guess for the caller to make. Which of these a given route can answer with is declared on the route, so a generated client knows before it calls.
| Code | Status | Retryable | Meaning |
|---|---|---|---|
not_found |
404 | no | No such resource |
revision_conflict |
412 | yes | If-Match did not match the stored revision |
idempotency_conflict |
409 | no | Key reused with a different request |
lease_held |
409 | yes | Another process owns this compute |
compute_not_accepting |
422 | no | The compute is deleting or failed |
compute_not_resizable |
422 | no | The compute runs a collective, and its ranks are frozen |
image_fixed |
422 | no | The compute's image is not mutable, or the change is to a field a mutable image cannot change in place |
unsupported_provider |
422 | no | No adapter registered for that kind |
unsupported_plugin |
422 | no | No plugin registered under that kind |
hash_mismatch |
400 | no | Uploaded content does not hash to its name |
task_failed |
409 | no | Reading the result of a task that failed |
task_indeterminate |
409 | no | Contact was lost after code may have run |
duplication_not_acknowledged |
409 | no | Retrying an indeterminate task without accepting it may run twice |
capability_mismatch |
422 | no | The provider cannot do what the spec asks — volumes, clustering |
source_rejected |
422 | no | Text registered as a function does not parse, or defines no function by that name |
release_pending |
— | yes | Only in status.last_error: every machine is gone and the provider would not release the binding yet |
reconcile_failed |
— | yes | Only in status.last_error: a reconcile pass broke on an error with no code of its own |
Credentials never appear in a compute spec: they belong on a provider row, which no read path selects and the API never returns, and a spec names that row by kind and name. A spec is served back, which is why it has no field a credential could go in.
Content-addressed payloads¶
Code, arguments, and results never travel in a task body. They are uploaded as blobs named by their SHA-256 and referenced by that name:
$ http PUT :17590/v1/blobs/<sha256> < payload.bin
$ http PUT :17590/v1/functions/<sha256> codec=cloudpickle-lz4
$ http POST :17590/v1/tasks function=<sha256> args_sha256=<sha256> compute=cmp_7f3a1c
The same argument broadcast to a hundred nodes is stored once. A PUT of content already present is a no-op, so a client can skip the upload entirely by trying the task first. Results come back the same way, which is why reading one twice does not consume it.
Calling without an interpreter¶
Those blobs are cloudpickle, which only Python can write. A caller that has none — the browser console, or anything speaking HTTP and nothing more — says the same two things in the only notation it has, and the daemon encodes them at the edge:
$ http POST :17590/v1/functions name=area source='import math
def area(radius):
return math.pi * radius**2
'
$ http POST :17590/v1/tasks function=<sha256> compute=cmp_7f3a1c dispatch=one rank:=0 call:='{"args": [2.0]}'
POST /v1/functions takes a module and the name of the function in it to call. The daemon does not run a line of it: the text is captured in a callable, that callable is pickled, and what is registered is the same blob PUT would have taken — compiled on the machine, once per call. Text that does not parse, or that defines no function under that name, is refused with source_rejected rather than discovered twenty minutes later on a machine that is already costing money.
call is the third way a task can say its arguments, beside args_inline and args_sha256. A list is taken in order, an object by name, and what fits is what JSON has — an argument that is a dataframe is an argument that has to be built where Python is.
A function registered this way is read back with its source. A pickle has no text in it, so the SDK sends the text of one it uploads right after the pickle, to PUT /v1/functions/{sha256}/excerpt: the function, with the imports, constants, functions and classes of its module that it uses, in the order of the file. It is read where the function was defined, stored as sent and never run — and a function defined where there was no file to read, in a REPL or an exec, has none.
One function, many uploads¶
Content-addressing names an upload, and an upload changes for reasons that have nothing to do with the code: the function moved down its file, or a run captured a different id. So uploads are grouped. lineage is one function — the same name and qualname in the same file — and version counts the changes to its code along it, in order:
$ http :17590/v1/functions latest==true # one row per function, at its newest upload
$ http :17590/v1/functions lineage==<lineage> # every upload of one function, newest first
An upload whose code matches the one before it is the same version, whatever else about its bytes moved. Going back to older code is a new version rather than the old number again, so the newest upload of a function always carries its highest version — which is what running "the latest" means. A task still names the exact upload it ran.
The name, the file and the shape of the code are read off the pickle without unpickling it — the daemon never runs a function to find out what it is.
rank picks the machine. Without it, dispatch: one takes any node with a slot going spare; with it, the task waits for that one rather than settling for another.
Watching instead of polling¶
There is no status endpoint to poll on a timer. GET /v1/events is a Server-Sent Events stream carrying every lifecycle transition, node output, and task outcome, with replay from Last-Event-ID. That is how the SDK learns a compute is ready and how sky log prints a bootstrap it was not around to watch.
Each message's data: is one of eight payload types, discriminated by a type field inside the payload:
{"type": "node.console", "compute": "cmp_7f3a1c", "node": "nod_2b91", "content": "epoch 3 loss 0.214"}
The frame's event: name is finer than that tag — ten node states share node.state, four task outcomes share task.state — and it is what the types filter matches on. The tag is what lets a payload be decoded on its own, once it has been written down, exported, or replayed out of the stream.
See Events for the stream's filters, replay semantics, and behaviour with slow consumers.
A terminal on a machine¶
Three routes carry a live connection rather than a document, and all three reach a machine the daemon holds an SSH link to — which is every machine that has answered SSH, not only the ones whose bootstrap finished. Watching one install its driver is most of what a terminal is for.
The pair is for a client that cannot hold a socket:
$ http POST ':17590/v1/computes/cmp_7f3a1c/shell/up?cid=s1&node=0' < keystrokes # the keyboard
$ http GET ':17590/v1/computes/cmp_7f3a1c/shell/down?cid=s1' # what it paints
Two half-duplex streams, tied by a cid the caller mints, because HTTP/1.1 will not carry a request body that is still being written alongside the response to it. /forward/up and /forward/down are the same shape around a node port instead of a pty. Neither is resumable: a dropped stream is a dead session.
The socket is for a client that can, which in practice means a browser — fetch cannot stream a request body over HTTP/1.1 at all:
Binary frames are the terminal, both ways. Text frames up are the screen's new shape, {"columns": 132, "rows": 50}, which the pair has nowhere to put — it carries the size once, in the query that opens it. A session that cannot be opened is refused with the same Error object every other route answers with, sent as a text frame and followed by a close with code 4409; the handshake itself is accepted first, because a browser is told nothing about a rejected upgrade.
This is the one route the OpenAPI document below does not describe, OpenAPI having no notion of a WebSocket.
Health¶
$ http GET :17590/v1/health/live # the process is up
$ http GET :17590/v1/health/ready # it can serve
$ http GET :17590/v1/health/dependencies # what it depends on, and their state
sky server start waits on live; sky config validate checks ready.
The full specification¶
Every route above bar the socket — with its parameters, request bodies, response shapes and schemas — is in the OpenAPI document, browsable in full:
It is generated from the running application by scripts/gen_openapi.py (task docs:openapi), so it cannot drift from the controllers. The raw document is at openapi.json if you would rather point a code generator at it.
Next steps¶
- API explorer — the full OpenAPI document, rendered
- Persistence — what the daemon writes down, and what survives a restart
- Architecture — the components behind these routes
- Events — the SSE stream and its replay semantics
- CLI — the command-line client for this API