Health Check vs Readiness vs Liveness Probe in Kubernetes: Don't Let Traffic Land on a Dead Pod

Every pod is green and Running, yet users get a 502 Bad Gateway? Or all your pods restart at once because the database got slow? The wrong probe is usually behind it.

Read in Indonesian

Have you ever had a Kubernetes deployment that looked like a success? Every pod sits in Running, a nice healthy green. Then a user opens the app and gets a 502 Bad Gateway in the face.

Or the scarier version. The database server slows down for a moment. Suddenly Kubernetes restarts every backend pod at once, and you have a cascading failure on your hands. 😅

This classic problem almost always has one root. A misunderstanding about health checks in Kubernetes.

Many junior developers assume one simple /health endpoint is enough. Kubernetes actually gives you three different probe mechanisms. Their jobs and consequences are nothing alike. There is the startup probe, the liveness probe, and the readiness probe.

Let's take apart how each probe works. And how to wire them correctly, so the app holds up in production.

Key takeaways

  • The three probe types in Kubernetes have separate jobs: a startup probe for early initialization, a liveness probe for deadlocks and crashes, and a readiness probe for whether the app can take traffic.
  • A liveness probe that checks external dependencies (database, Redis) is a death trap, because it can trigger a mass restart and a cascading failure.
  • A /healthz endpoint for liveness and a /ready endpoint for readiness should be split in your backend, whether it runs on FastAPI, Go, or Node.js.
  • initialDelaySeconds, periodSeconds, timeoutSeconds, and failureThreshold need tuning so a brief CPU spike does not fire a false-positive restart.

The Three Probes in Kubernetes

To make it easy, picture a fast-food restaurant. There is a cook, a kitchen, and a manager. Each probe has the role of one of them.

The startup probe checks whether the app's initialization has finished. Some apps take a long time to really boot. Think of one running a heavy framework like Java Spring Boot. Or a backend that loads a machine learning model and runs database migrations. That can take tens of seconds, up to minutes.

While the startup probe has not succeeded (Success), Kubernetes disables the liveness and readiness checks. The reason is obvious. The app should not get killed by a liveness probe before it ever finishes waking up.

The liveness probe answers one question. Is the app process still alive and healthy? Or has it hit a deadlock and hung completely? If the liveness probe fails (Failed) past its tolerance (failureThreshold), kubelet kills the container and restarts it. Think of it this way. If the cook passes out in the kitchen, the restaurant manager replaces him with a new cook.

The readiness probe answers a different question. Can the app take traffic from users right now? A pod might be perfectly alive. But it cannot process a request at the moment. Maybe the local in-memory cache is being refreshed. Maybe the database connection pool is full. Maybe the app is chewing through a heavy batch queue.

If the readiness probe fails, Kubernetes does not kill that pod. It only pulls the pod out of the Service endpoint list, that is, the Load Balancer. User traffic stops flowing to that pod until the probe succeeds again.

The Biggest Trap: Checking the Database in a Liveness Probe

This is the most dangerous anti-pattern you will meet in the field:

PYTHON
@app.get("/healthz")  # THE WRONG WAY TO DO A LIVENESS PROBE!
async def health_check(db: Session = Depends(get_db)):
    # NEVER DO THIS IN A LIVENESS PROBE!
    await db.execute(text("SELECT 1"))
    return {"status": "ok"}

Now imagine your PostgreSQL database suddenly gets overloaded. CPU at 100%. The SELECT 1 query times out after more than 2 seconds. What happens when the endpoint above is wired as a liveness probe?

  1. The liveness probe on Pod 1 times out. Kubernetes kills Pod 1 and builds a new pod.
  2. The liveness probe on Pod 2 times out. Kubernetes kills Pod 2.
  3. And so on, until every pod restarts at the same time!
  4. The new pods then flood the database with fresh connection pools. That is a connection spike. The database gets even sicker. 😅

So here is the golden rule. A liveness probe only checks the app's own internal state. Whether the event loop is not blocking, or whether the web server still responds. Never check external dependencies like a database, Redis, or a third-party service. The readiness probe is the right place for that. There you can verify that the database connection is ready before taking a user's business request.

Split Your Health Check Endpoints

The cleanest approach is to separate the endpoints in your backend:

PYTHON
from fastapi import FastAPI, status, Response

app = FastAPI()

is_app_ready = False

@app.on_event("startup")
async def startup_event():
    # Initialize connections, load models, warm the cache...
    global is_app_ready
    await init_database_connection()
    is_app_ready = True

@app.get("/healthz", status_code=status.HTTP_200_OK)  # 1. liveness: is FastAPI still breathing?
async def liveness():
    return {"status": "alive"}

@app.get("/ready")  # 2. readiness: are the dependencies ready for traffic?
async def readiness(response: Response):
    global is_app_ready
    if not is_app_ready or not await check_db_health():
        response.status_code = status.HTTP_503_SERVICE_UNAVAILABLE
        return {"status": "unavailable"}
    return {"status": "ready"}

The Ideal Kubernetes Manifest

Here is a sample Kubernetes Deployment. It combines all three probes in harmony. I added a selector and labels as well, so the manifest is actually valid.

YAML
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: payment-service
  template:
    metadata:
      labels:
        app: payment-service
    spec:
      containers:
      - name: payment-api
        image: payment-service:v1.2.0
        ports:
        - containerPort: 8000

        # 1. Startup Probe: give the app room to breathe on cold start
        startupProbe:
          httpGet:
            path: /healthz
            port: 8000
          failureThreshold: 30
          periodSeconds: 2 # Total boot wait: up to 60 seconds

        # 2. Liveness Probe: kill the container only on a total hang
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8000
          periodSeconds: 10
          timeoutSeconds: 2
          failureThreshold: 3 # Tolerate 3 consecutive failures

        # 3. Readiness Probe: pull traffic when the app is overloaded or the DB drops
        readinessProbe:
          httpGet:
            path: /ready
            port: 8000
          periodSeconds: 5
          timeoutSeconds: 2
          failureThreshold: 2
          successThreshold: 1 # Send traffic immediately once healthy again

Look at periodSeconds and failureThreshold. Do not set timeoutSeconds too aggressively, say 100 ms. During a brief CPU spike, a probe request can get delayed. That triggers a restart you did not need, a false-positive restart.

So How Do You Keep Pods from Dying on the Job?

By keeping the liveness and readiness roles apart, with discipline, you get three things. First, you avoid the nightmare of a cascading restart when database infrastructure struggles. Second, users never get a 502/504. Traffic only goes to pods that are genuinely ready to serve. Third, rolling deployment and HPA (Horizontal Pod Autoscaler) auto-scaling run far more smoothly, with no downtime.

Are the probes in your Kubernetes cluster configured correctly today? Go audit your manifests now.

Happy Kubernetes-ing, and may your pods never restart for no reason! 👋

Inva

Writer

Get better at shipping code.

Join the other developers getting tech updates, tips, and relaxed tutorials in their inbox every week.

Stay in the loop

Tech updates, coding tips, and relaxed insights straight to your inbox every week.

No spam. Unsubscribe anytime.

Read next

See all
Struktur DataWhat a Linked List Actually Is

A linked list is a linear chain of nodes, each holding data and a pointer to the next node. Here is what that chain looks like, what it costs next to an array, and the same minimal implementation in Java, Python, and JavaScript.

For developers who learn by struggling

© 2026 Inva.dev.