A Kubernetes Pod can have a running process while the application it contains is unable to serve customers. Health probes help the kubelet and Service routing system respond to different states, but an incorrectly designed probe can make an outage worse. This article focuses on operating decisions: when to restart, when to remove an endpoint from normal traffic, how to protect slow startup and what evidence should guide troubleshooting. It is deliberately distinct from a basic installation walkthrough.
Know which question each probe answers
A liveness probe asks whether a running container should be restarted because its process is unable to make progress. A readiness probe asks whether a container should receive normal Service traffic. A startup probe provides a separate window for applications that require substantial initialization time. These are related but different decisions. A temporary backend outage may make a frontend unable to answer customer requests without making its own process irrecoverably broken. In that case, marking the Pod not ready may be appropriate, while repeatedly restarting it could increase load and prevent recovery.
The kubelet periodically evaluates the configured probe mechanism, which may use an HTTP request, TCP socket, executable command, or a supported gRPC health interface. The probe’s success or failure is not a universal measurement of user satisfaction; it only represents the chosen test. Design a probe endpoint around the specific response Kubernetes needs. An HTTP status that remains successful while the application is deadlocked will mislead liveness; a readiness endpoint that checks noncritical dependencies too aggressively can withdraw healthy capacity during an unrelated outage. The Kubernetes cluster architecture context helps relate the kubelet, Pod conditions and Services.
Understand readiness at the Service boundary
When a readiness check fails, Kubernetes marks the relevant container unready and the Pod stops being a normal ready endpoint for matching Services. The process may remain running. This allows a slow-loading application to finish initialization before traffic is directed to it, and it can temporarily remove an overloaded instance from rotation. Readiness checks continue throughout a container’s lifecycle, so an instance that later recovers can return to service. An accepted route through an ingress controller does not override the absence of ready backend endpoints.
Choose readiness behavior from real application needs. A web process might need a configuration snapshot and active connection pool before it can accept work. A worker may need an initialized queue connection, but checking whether every independent third-party service in the world responds can make readiness brittle. Set thresholds and periods based on normal transient conditions rather than one arbitrarily timed request. Monitor the effective endpoints and measure whether traffic stops reaching instances that report unready. A sudden loss of ready capacity can be as disruptive as the fault that caused it.
Make liveness a conservative recovery mechanism
Liveness is useful for failures from which a process cannot recover without restarting, such as a persistent deadlock. It is not a general-purpose indicator of every unavailable dependency. If an application is waiting for a temporary database interruption to pass, restarting it repeatedly may create a storm of reconnections and lose useful diagnostic context. A liveness endpoint should be inexpensive, deterministic and focused on the process’s own ability to function. Do not use a heavyweight end-to-end transaction as a liveness check without understanding the effect of an intermittent downstream failure.
The kubelet uses the configured failure thresholds and timing rules when deciding to restart. A check that runs too frequently under load can generate false failures. An overgenerous threshold may delay the restart of a genuinely stuck process. Test both a recoverable temporary fault and a simulated persistent deadlock in an isolated environment. Observe whether a failing readiness check removes the Pod from traffic before liveness eventually causes a restart. The distinction is important when a deployment has only a few replicas and every replacement takes several minutes to warm up.
Use startup probes to protect slow initialization
Some containers perform migrations, load large datasets, build caches or initialize runtime components before serving requests. A liveness probe tuned for normal steady-state behavior can kill such a container before it starts. A startup probe provides a controlled initialization window: while it has not succeeded, the kubelet suppresses regular readiness and liveness checks for that container. Once startup succeeds, normal checks take over. This makes a startup probe better suited to slow but expected initialization than simply adding a long delay to every liveness check.
Calculate an initialization allowance based on observed startup times and worst-case conditions. A high failureThreshold multiplied by periodSeconds permits a longer startup window, but it should not be infinite. Test what happens when startup never completes. A failed migration or invalid secret should eventually produce useful diagnostics, not leave a broken Pod in an ambiguous state forever. Review application logs and Pod events when a startup probe fails. A ready-state assertion only becomes trustworthy when the application has genuinely completed the setup on which user requests depend.
Choose probe mechanics that represent the failure
HTTP probes can validate an endpoint that reports application readiness, liveness or startup. TCP probes mainly test whether a connection can be established to the configured port; they cannot establish that a business transaction will succeed. Exec probes run a command inside a container and can be useful where no network health endpoint exists, but they may impose overhead and require appropriate executable permissions. gRPC health checks are available where the application and Kubernetes configuration support them. The method should match the application, not be selected solely because a sample manifest happened to use it.
A reliable health endpoint should respond quickly and not depend on an expensive query. Avoid returning sensitive configuration details in probe responses. Do not make health checks mutable business operations. Test how the endpoint behaves under load, during dependency failures and after deployment. A status code that is consistently green is not informative if it remains green while the main request handler is blocked. Conversely, a probe that reports failure under routine background work may generate needless restarts. Document the intended observable state and the exact response that indicates success.
Connect probes to rollout and disruption strategy
A Deployment rollout depends on replicas becoming ready so that the system can replace old Pods without removing too much healthy capacity. A ready signal that turns green before an application is usable may produce a burst of errors immediately after an update. A signal that remains false because it requires an unavailable optional service can stall deployment even though the application could handle core requests. Review readiness, maxUnavailable, maxSurge and application shutdown behavior together. A PodDisruptionBudget constrains some voluntary disruptions, but it cannot create capacity or prevent every failure.
During a rolling update, observe the actual number of ready endpoints rather than only the Pod phase or Deployment status summary. Change a readiness endpoint intentionally in a nonproduction cluster and see whether the old replica remains until replacement capacity is usable. Test graceful termination while in-flight requests are active, since probe configuration alone does not guarantee a safe drain. The Kubernetes readiness and liveness probes material can deepen the configuration mechanics, but this article emphasizes how health signals affect deployments and incident response.
Troubleshoot events, endpoints and container restarts
When a workload stops serving traffic, inspect Pod conditions, restart counts, container logs, Kubernetes events, Service selectors and EndpointSlice state. A high restart count suggests repeated process termination or liveness failure, while Pods that remain Running but unready point toward different causes. Compare probe timestamps with application start logs, dependency status and recent configuration changes. A failure that begins only after a new image is deployed may be a changed health endpoint or slower initialization rather than a cluster-wide networking defect.
Do not change several timeouts at once and then assume the problem has been fixed. Form a hypothesis, collect a small set of observations, apply one controlled adjustment and see whether the expected result occurs. A temporary improvement under light load may conceal a recurring problem at peak traffic. Preserve a working deployment revision for rollback. If a readiness failure removes every endpoint, investigate whether the check is tied to a shared dependency. If liveness restarts every instance together, consider whether the restart is contributing to the outage rather than curing it.
Practice a reversible failure lab
In a disposable cluster, deploy a small HTTP application with an explicit startup endpoint and separate readiness and liveness endpoints. Begin with two replicas and verify that both receive test traffic. Make readiness fail on one Pod and confirm its endpoint is removed without an immediate forced restart. Restore readiness and verify that traffic resumes. Then simulate an unrecoverable process state that should fail liveness, and observe the restart and Pod events. Finally, introduce a slow startup delay and confirm that a configured startup probe permits initial loading without a liveness restart loop.
Record expected and observed results, including timings, status transitions and the effect on users. Shut down test resources after practice and review the costs of any cloud cluster, load balancer or log storage used. A useful health check is not simply an endpoint returning 200; it represents an explicit decision about restarting a process, accepting traffic, or permitting initialization to continue. When those decisions are tied to service behavior and tested failure modes, probes help Kubernetes improve availability instead of becoming another source of cascading failures.