Managed Inference Job
What a Managed Inference Job is, when to use one, and how you call it.
A Managed Inference Job runs an open source model inside a KubeVirt virtual machine instance (VMI) on your cluster and serves it behind an OpenAI-compatible API. The job uses one of two runtimes, vLLM for language models or Parakeet for speech-to-text. You send requests the same way you would to any OpenAI-compatible endpoint.
When to use one
A Managed Inference Job fits when you want to call a model over an API. You pick an open source model, and CosmicAC serves it.
If you instead want direct control of a GPU to run your code, a GPU Container Job is the better fit. It gives you a machine and a shell, and you set up the environment yourself.
What you get
- OpenAI-compatible API: your existing clients and SDKs work without changes.
- Open source models: served on your cluster with vLLM or Parakeet.
- Managed serving: you don't set up the runtime or GPU environment yourself, because CosmicAC provisions and runs the model server.
Runtimes
A Managed Inference Job uses one of two runtimes.
- vLLM: serves open source language models behind an OpenAI-compatible chat endpoint.
- Parakeet: serves the NVIDIA Parakeet speech-to-text model
nvidia/parakeet-tdt-0.6b-v3behind an OpenAI-compatible transcription endpoint.
Supported models
CosmicAC ships tuned serving parameters for a set of vLLM models. For that list and each model's values, see Recommended model parameters.
Those models aren't a limit. A vLLM job serves any model vLLM itself supports, once the model has a model master, which stores its runtime image and default serving parameters. For a model with a published vLLM recipe, CosmicAC creates the model master from that recipe rather than asking you for each value. See Add a model master from a vLLM recipe. For any other model vLLM supports, add the model master manually.
A Parakeet job serves one speech-to-text model, nvidia/parakeet-tdt-0.6b-v3. Its model master stores no runtime image, data type, quantisation, tensor parallel, or reasoning parser. CosmicAC supplies the speech-to-text runtime image from the job type.
How it works
When you create a Managed Inference Job, CosmicAC allocates the GPUs that each replica needs. CosmicAC then provisions the VMIs that serve the model with the chosen runtime. A replica that fits on one node runs a single VMI, and a multi-node replica runs one VMI on each node. After the model is serving, you call it through the OpenAI-compatible endpoint. The endpoint authenticates your requests and routes them to the running model.
Restarting the job replaces its VMIs but keeps its storage and resources. Deleting the job removes its VMIs, resources, and storage.
For how a request flows through the CosmicAC components, see Architecture.
How you connect
You call the model in two ways.
- OpenAI-compatible client: send requests over HTTP.
- CosmicAC CLI: run inference on a vLLM endpoint directly from the terminal.
Both authenticate with an API key. See Create an API key.
To connect a client to a vLLM endpoint, see Connect to a vLLM Managed Inference endpoint. For a Parakeet endpoint, see Transcribe audio.
Guides
Create a job
Create a vLLM Managed Inference Job with the CLI
Serve a language model from the terminal.
Create a vLLM Managed Inference Job in the web interface
Serve a language model from the CosmicAC web interface.
Create a Parakeet Managed Inference Job with the CLI
Serve the speech-to-text model from the terminal.
Create a Parakeet Managed Inference Job in the web interface
Serve the speech-to-text model from the CosmicAC web interface.
Call an endpoint
Create an API key with the CLI
Authenticate your requests to an endpoint.
Connect to a vLLM Managed Inference endpoint
Send a chat completion request from a client or the CLI.
Transcribe audio with a Parakeet endpoint
Send audio from the CLI or curl and read the transcript.
Check the health of a Managed Inference endpoint
Read the status, success rate, and latency of an endpoint.