Replicas
What a replica is, why you run more than one, and when a replica spans more than one node.
A replica is one running copy of the job's model. All replicas serve requests through a single endpoint. When you create a Managed Inference Job, you choose how many replicas to run.
Running more than one replica gives the endpoint two benefits.
- Capacity: more replicas serve more traffic, because CosmicAC balances requests across every healthy replica.
- Resilience: if a replica fails, the endpoint keeps serving on the remaining replicas at reduced capacity.
A round-robin load balancer spreads requests across the healthy replicas, and drops an unhealthy one from rotation. It holds no sticky sessions, so consecutive requests from one caller can land on different replicas.
CosmicAC doesn't scale the count automatically, and a job keeps the count it started with. The endpoint's model health reflects the health of its replicas.
Multi-node replicas
A replica claims the GPUs that its model needs. When a model needs more GPUs than one of your nodes holds, the replica runs on more than one node from one rack, where each node runs one VMI and contributes an equal number of GPUs. Only a vLLM replica can be multi-node, and a Parakeet replica always runs on one node.
Those VMIs serve as one replica. The first VMI runs the model server and answers requests from the load balancer, and the others add their GPUs to that server. CosmicAC connects them over InfiniBand and an overlay network, so a multi-node replica needs the overlay networking add-ons on your cluster. See Requirements.
You don't choose the node count. Replicas and nodes scale independently, so the replica count sets how many copies of the model serve the endpoint, and the model's GPU count sets how many nodes each copy takes. You set a GPU count for one replica, and that count must match the per_replica_gpu_count in the model master. See GPU configuration.
To see the node count and node names for each replica, run cosmicac jobs detail.