Architecture

CosmicAC is a self-hosted platform that runs GPU workloads on your Kubernetes cluster. Its components run outside the cluster and reach it through the Kubernetes API, so you keep operating the cluster while CosmicAC runs workloads on it.

For how each job type runs, see GPU Container architecture and Managed Inference architecture. For deployment steps, see Installation.

Deployment architecture

Setting up your cluster is separate from deploying CosmicAC. You bring a Kubernetes cluster that already has its GPU nodes and KubeVirt configured. The CosmicAC components then connect to that cluster and run your workloads on it.

CosmicAC provisions a GPU virtual machine instance (VMI), a KubeVirt resource, for every workload. The Containerized Data Importer (CDI) imports the VMI root disk from a registry image. The VMI claims one or more whole GPUs through PCI passthrough, and a multi-node workload also claims InfiniBand.

Your cluster's requirements fall into two tiers.

• Base platform: the Kubernetes, GPU, virtualization, storage, and registry foundation that every workload needs.
• Overlay networking: an add-on tier that connects a workload's nodes over an OVS/VXLAN subcluster, and provides a WireGuard gateway for workloads that need an isolated network.

A multi-node vLLM Managed Inference replica needs the subcluster. See Requirements.

The following diagram shows the Kubernetes resources that cosmicac-wrk-server-k8s-nvidia creates for a job, how the VMI claims its GPUs, and where the overlay network attaches.

Racks, nodes, and GPUs

Your GPU capacity has three nested levels.

• Rack: the GPU capacity behind one worker, listed under a single location. Despite the name, a rack is an entry in your CosmicAC network, not physical hardware. The worker reports the nodes and GPUs of the cluster it connects to, and cosmicac-wrk-ork tracks them as available capacity.
• Node: a machine in your cluster that holds a pool of GPUs. More than one job can run on the same node, each claiming different GPUs from its pool.
• GPU: a single accelerator card, such as an H100. A job claims one or more whole GPUs through passthrough.

A default deployment registers one rack during task bootstrap. To register, list, or remove racks yourself, see Manage racks in your CosmicAC network.

When you create a job, CosmicAC finds a rack in the location you chose, then reserves GPUs on nodes in that rack. Every VMI runs inside a Kubernetes pod on a node. Because a node holds a pool of GPUs, one node can host many jobs at once, each in its own pod.

For a Managed Inference Job, the replica GPU count determines how many nodes it takes.

• Single node: the replica's GPU count fits one node, so it runs in a single VMI.
• Multi-node: the replica's GPU count is more than one node holds, so it runs one VMI on each node.

Only a vLLM Managed Inference replica can be multi-node. It takes its nodes from one rack, and each of those nodes contributes the same number of GPUs. CosmicAC joins the replica's VMIs over InfiniBand and an overlay network.

Every other job runs on one node. If you request more GPUs for a GPU Container Job than a node holds, CosmicAC fails the job.

CosmicAC components

Most CosmicAC components run on your host machine, outside your Kubernetes cluster. The CLI runs on your workstation, and the per-job agents run inside each job's VMI.

Caddy proxies /api to cosmicac-app-node, /inference to cosmicac-proxy-inference, and /monitor to cosmicac-wrk-monitor. Every other path serves the web interface.

Holepunch stack

Inside CosmicAC, the components connect to each other over the Holepunch peer-to-peer stack rather than through a central server. Components address each other directly, so there is no central broker to route, bottleneck, or expose internal traffic.

Next steps

• GPU Container architecture
• Managed Inference architecture
• Observability architecture
• Installation