Oxide Wiki

Thus far this page was auto-generated with Cursor, will put my git info in here soon.

A working reference for Oxide Computer Company racks: hardware, control plane, and the operator/user surfaces you actually touch.

What it is

Oxide sells a rack-scale computer: servers, storage, networking, and a cloud control plane that ship as one product. The pitch is on-prem infrastructure that behaves like a public cloud API instead of a pile of vendor SKUs you assemble yourself.

Host software runs on illumos (Oxide's Helios distribution). Virtual machines run under the Propolis hypervisor. The control plane is Omicron.

Rack hardware

A rack is self-contained. Typical first-generation capacity is up to 32 compute sleds, two switches, and two power shelves. Second-generation racks go up to 24 sleds with higher per-sled density.

Boards

  • Compute sleds (Gimlet / Cosmo by generation). Each sled has a host CPU for customer and control-plane work, plus a service processor (SP) with a root of trust for power, thermals, serial console, and firmware.
  • Sidecar switches. Two Tofino ASICs switch control-plane and data-plane traffic. Each sled has a backplane link to both switches. There is no host CPU in the Sidecar; the ASIC is PCIe-attached to an adjacent sled.
  • Scrimlets are the sleds that own those PCIe switch connections and run switch-zone software.
  • Power shelves with a Power Shelf Controller (PSC) for rectifier monitoring, presence, and on/off control.

Every board type has an SP on the management network. SPs stay reachable even when the host OS is down.

Control plane (Omicron)

The data plane is CPU, storage, and networking given to users. The control plane is how you provision, configure, monitor, and repair that.

Design goals (availability of any two hardware/software failures, durable user resources, strong consistency in the API namespace) live in the Omicron architecture notes.

Tied to hardware

  • Sled agent — instances, storage, networking, and hardware inventory on that sled.
  • Dendrite — switch configuration (forwarding, NAT, and related tables).
  • Management Gateway Service (MGS) — APIs to every SP in the rack.
  • Wicket / wicketd — technician-port TUI for first boot and support, before customer networking is up.
  • NTP — boundary NTP on the Scrimlets, internal NTP on other sleds.

Rack-wide services

  • Nexus — operator and customer APIs, web console assets, inventory, alerts, recovery work. Stateless enough to scale; durable state is elsewhere.
  • CockroachDB — replicated control-plane database.
  • ClickHouse + Oximeter — metrics collection and query.
  • Internal DNS — how control-plane components find each other.
  • External DNS — customer-facing names for API and console.

Most of those services run in illumos zones and can be replicated across sleds.

User and operator model

  • Silo — isolation boundary. Identity (local users or SAML IdP), quotas, and who can see what.
  • Project — workspace inside a silo for instances, disks, images, and VPCs.
  • Instance — a VM with disks, NICs, SSH keys, and optional floating IPs.
  • VPC — virtual network, subnets, firewall rules, and internet gateways.
  • Images / snapshots — boot media and point-in-time disk copies.

Rack bring-up finishes by assigning external instance IPs, creating a silo, and making a silo administrator (with or without an identity provider).

Surfaces you use

  • Web console — a static React client served by Nexus. It is an API client, not a second control plane. Preview: console-preview.oxide.computer.
  • HTTP API — the source of truth. Console, CLI, and Terraform all talk to it.
  • oxide CLI — same API from a terminal.
  • Wicket — physical technician port when the rack is dark or networking is not configured yet.

Quick mental model

  1. SPs and the management network keep the metal reachable.
  2. Sled agents and switch software own local resources.
  3. Nexus + CockroachDB are the rack's brain and memory.
  4. Users never talk to sleds directly; they talk to Nexus (console, API, or CLI).

Related project notes on this site: Grafana Wiki.