Check Managed Inference endpoint health
Check the status, success rate, and latency of a Managed Inference endpoint from the web interface or the CLI.
Check the health and performance of a Managed Inference endpoint from the web interface or the CLI.
Prerequisites
You need the following before you start:
- A running Managed Inference Job with a serving endpoint. See Create a Managed Inference Job.
- Access to the CosmicAC web interface, for the web method.
- The CosmicAC CLI installed and configured, for the CLI method. See Install the CLI.
Steps
Run the health check
Use the web interface or the CLI. Both methods report every deployed endpoint.
In the left sidebar, open Model Health, then click Run health check.
Read the results
Each endpoint reports its health and performance:
- Status shows
Healthy,Degraded, orDown. For what each value means, see Model health. - Success rate shows the share of requests that succeeded.
- Traffic shows the total requests handled.
- Failures shows how many of those requests failed.
- Avg response shows the average response time in milliseconds. The CLI prints it as Latency (avg).
- Last health check shows when CosmicAC last probed the endpoint. The CLI doesn't print it.
- Last updated shows when CosmicAC generated the reading. The CLI prints it as Timestamp.
The web interface and the CLI cover different time ranges, so the same endpoint can report different numbers in each. In the web interface, choose a range at the top of the Model Health page, next to Run health check. The choices are 1H, 6H, 24H, 7D, and 30D. By default, the page shows 7D. The CLI has no range option and always reports the last hour.
These figures count the health probes that CosmicAC sends, not only the requests your users make. A probe is itself a small inference request, so an endpoint that serves no user traffic still records traffic here. How much depends on your probe interval, which is five minutes by default. To change it, see Model health settings.
The CLI also prints a per-replica table, with the replica ID, status, traffic, failures, and average latency for each replica. Use it to find an unhealthy replica. The web interface shows a replica health count on each endpoint card.
CosmicAC measures these metrics itself. The inference proxy times every request it forwards to the model server, records whether it succeeded, and writes the result to a store on disk. The metrics describe how the model server behaved, but CosmicAC doesn't read them from it, so they survive a restart of either one.
To chart these metrics, alert on them, or combine them with metrics from outside CosmicAC, run an observability stack such as Prometheus.
Logs are a separate system. CosmicAC collects logs for Managed Inference Jobs only when K8S_LOG_STORE_ENABLED is true. See Observability and logs.