Model health
What Healthy, Degraded, and Down mean for a Managed Inference endpoint, and how CosmicAC measures them.
Model health measures how well a Managed Inference Job's replicas respond to inference requests. A job reports it alongside its status.
CosmicAC rates each replica from its health probes and its recent request success, then combines those ratings into one value for the endpoint.
- Healthy: every replica is passing its health checks and serving requests successfully. A replica awaiting its first probe also counts as healthy.
- Degraded: some replicas are unhealthy, but not every replica is down. A replica becomes degraded when it starts failing its health checks, or when its recent success rate falls below the configured threshold even though its health check passes. The endpoint stays up and serves on its healthy replicas.
- Down: every replica is down, or the endpoint has no registered replicas. A replica becomes down when it keeps failing its health checks for longer than the configured threshold, or when its recent success rate reaches zero.
CosmicAC reads both thresholds from the cosmicac-proxy-inference service's configuration. To change them, see Model health settings.
How CosmicAC measures model health
cosmicac-proxy-inference collects these measurements rather than querying the model server for them. The proxy measures the latency of every request it forwards, records whether the request succeeded, and writes the result to a store on disk. Because that store is outside the model server, restarting the proxy or the model server leaves the measurements intact.
These measurements include the health probes that the proxy sends, not only the requests your users make. An endpoint that serves no user requests still records traffic.
How model health differs from job status
Model health is separate from the job's status. A job can be Running while its endpoint reports Degraded, because the job is up but one replica serves poorly.