Managed Inference architecture
How a Managed Inference Job runs on your cluster, and how CosmicAC authenticates and routes an inference request.
A Managed Inference Job runs an open source model inside a KubeVirt virtual machine instance (VMI), using vLLM for language models or Parakeet for speech-to-text. cosmicac-proxy-inference exposes that model as an OpenAI-compatible endpoint, authenticates requests, and balances load. You reach the model through that endpoint from any OpenAI-compatible client, or by running inference directly with cosmicac-cli.
The following diagram shows how a job request reaches your cluster and how an inference request reaches the model. The example is a multi-node replica on two nodes, though a replica that fits on one node runs a single VMI.
The inference agent
Every VMI in a replica runs cosmicac-wrk-agent-inference, the inference agent. The agent runs the model server in a second container built from the vLLM or Parakeet runtime image, and restarts that container if it exits.
The agent isn't the model server. Every VMI runs one of each, and because the agent owns the server's container, it reads what the server prints and forwards it into the job's Application logs.
How a job starts
When you create a Managed Inference Job from cosmicac-ui or cosmicac-cli, cosmicac-app-node authenticates the request and forwards it to cosmicac-wrk-ork. The orchestrator allocates GPUs for each replica, then hands the job to cosmicac-wrk-server-k8s-nvidia. That worker creates the job's Kubernetes resources through your cluster's API server, and Kubernetes runs each replica as a pod holding a VMI.
A multi-node replica runs one pod and one VMI on each of its nodes. The first VMI runs the model server and answers requests, and the others add their GPUs to that server. Ray forms the cluster over the overlay network, and the NVIDIA Collective Communications Library (NCCL) carries the GPU-to-GPU traffic over InfiniBand.
As a replica starts, the inference agent on its serving node registers in the DHT table, so cosmicac-proxy-inference can discover the replica.
How CosmicAC serves a request
Serving traffic follows a separate path from job creation. You send a request to the inference endpoint from any OpenAI-compatible client, or run inference from cosmicac-cli. cosmicac-proxy-inference checks the API key when the endpoint requires one, searches the DHT table by topic to discover a model server, and balances load across the running servers. The inference agent then runs the request on the job's model server and returns the response.