Problem#

Pods in dev-webs were stuck at 1/2 Running. Readiness probes were failing with context deadline exceeded.

Symptoms#

kubectl get pods -n dev-webs
# NAME                  READY   STATUS    RESTARTS
# dev-api-gateway-xxx   1/2     Running   0

kubectl describe pod dev-api-gateway-xxx -n dev-webs
# Readiness probe failed: Get "http://10.x.x.x:8085/actuator/health/readiness": context deadline exceeded

Root Cause#

The app container started before the Istio sidecar (envoy-proxy) was fully initialized. When the app tried to connect to external services (DB, Redis, etc.), the sidecar wasn’t ready to handle traffic yet, causing the connections to fail.

[Pod startup]
├── istio-init (completed)
├── app container (starts immediately) <- attempts DB connection
└── istio-proxy (still starting up) <- cannot handle traffic yet!

Result: app cannot connect to DB -> health check fails -> CrashLoopBackOff

Fix#

Add the holdApplicationUntilProxyStarts annotation to deployment.yaml:

# common-charts/apps/java-service/templates/deployment.yaml
template:
  metadata:
    annotations:
      sidecar.istio.io/inject: "{{ .Values.istio.sidecar }}"
      {{- if .Values.istio.sidecar }}
      proxy.istio.io/config: '{"holdApplicationUntilProxyStarts": true}'
      {{- end }}

How It Works#

[Pod startup - holdApplicationUntilProxyStarts: true]
├── istio-init (completed)
├── istio-proxy (starting, waiting until ready)
│   └── [sidecar ready!]
└── app container (starts only after sidecar is ready) <- DB connection now works

kubelet waits for the istio-proxy postStart hook to complete before starting the app container.

Notes#

  • Requires Istio 1.7+
  • Slightly increases pod startup time (waiting for sidecar readiness)
  • No impact on pods with Istio sidecar injection disabled
  • common-charts/apps/java-service/templates/deployment.yaml
  • common-charts/apps/ai-service/templates/deployment.yaml