Kubernetes Container Probes
External
- https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes
- https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes
- https://medium.com/swlh/fantastic-probes-and-how-to-configure-them-fef7e030bd2f
Internal
TODO
- Merge and deplete OpenShift Container Probes.
Overview
A probe is a diagnostic performed periodically by the kubelet on a container. To perform the diagnostic, the kubelet calls a handler, that must be declared and implemented by the container. Each probe has one of these results:
- success - the container passed the diagnostic
- failure - the container failed the diagnostic
- unknown - the diagnostic itself filed so no action should be taken.
There are three kinds of probes: startup, liveness and readiness.
Handlers
A handler is a piece of logic declared and implemented by the container, which is used by Kubernetes control mechanism to draw conclusions about the state the container is in There are three types of handlers, described below. Any of these handlers can be used to perform startup, liveness and readiness checks.
ExecAction
The exec action (declared as "exec:") executes a specified command inside the container. The diagnostic is considered successful if the command exits with a status code of 0.
HTTPGetAction
Performs an HTTP GET request against the container’s IP address on a specified port and path. The diagnostic is considered successful if the response has a status code greater than or equal to 200 and less than 400
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 180
timeoutSeconds: 30
periodSeconds: 25
TCPSocketAction
Performs a TCP check against the container’s IP address on a specified port. The diagnostic is considered successful if the connection is successfully established.
Container and Pod Startup Check
The startup check is performed by a startup probe. Startup probes have been introduced in Kubernetes 1.16. The probe indicates whether the application within the container is started. If a startup probe is not provided, the default result is "success". If a startup probe is provided, all other probes are disabled until the startup probe succeeds. If the startup probe fails, the container is killed and it is subject to its restart policy. TODO: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#when-should-you-use-a-startup-probe
Container and Pod Liveness Check
The liveness check is performed by a liveness probe. The probe indicates whether the container is running. If a liveness probe is not provided, the default is "success". If a liveness probe is provided and it fails, the container will be killed and then subjected to its restart policy. (Not the pod? How about atomicity?)
Relationship between killing pods and containers - research needed. TODO: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#when-should-you-use-a-liveness-probe
Container and Pod Readiness Check
Containers IP address and port pairs (endpoints) are added to a service's Endpoints list and forwarded traffic to if the service selector matches the pod labels and the pod is "ready", meaning that all pod's containers that expose ports with the service are ready. Usually a pod exposes just one container, so "ready pod" and "ready container" are in this case equivalent. The situation when a pod exposes multiple containers is addressed in detail in the Readiness and Multiple Containers per Pod section.
A ready container is a container for which the readiness probe executes successfully. Once a container endpoint is added to the Endpoints instance, the corresponding readiness probe is invoked periodically and the endpoint stays in the Endpoints list as long as the probe executes successfully.
The endpoint is removed from the list on readiness probe failure. Technically, the pod's IP address is not actually removed from Endpoints, but listed in the "notReadyAddresses" list. It does not, however, show up anymore in 'kubectl get ep <name>' output, and the backing pod does not receive traffic anymore.
If the probe starts succeeding again, the pod is put again in traffic.
If there is more than one container per pod and at least one readiness probe fails, the endpoints for all containers in the pod are removed (technically, the pod's IP address is listed in the "notReadyAddresses" list).
The notion of being "ready" is something that is specific to each container. For example, in the initialization phase of the pod, its traffic-serving container may need time to load either configuration or data, or it may need to perform a warm-up procedure to prevent the first user request from taking too long and affecting user experience. The readiness probe should be designed in such a way that it start succeeding only after initialization.
Container that serve load in production should always define a readiness probe.
Readiness probe should not be used for orderly taking pods out of load. That should be done by either deleting the pod, or defining a label "enabled=true" or similar that can be switched on or off.
Readiness Probe Operations
If the container does not provide a readiness probe, the default diagnostic result is "success".
If a probe is declared, the default state of readiness before the initial delay is "failure". The initial delay and probe timing arithmetic is explained in the Probe Template section.
The pod's readiness state is displayed in the output of kubectl get pod command. A pod with one container displays a report similar to:
NAME READY STATUS RESTARTS AGE
cassandra-0 0/1 Running 0 23s
A pod with two containers displays a report similar to:
NAME READY STATUS RESTARTS AGE
httpd 1/2 Running 0 31m
If the readiness probe fails (or at least one of the readiness probes fails for multi-container pods), the pod is removed from the Endpoints list. If the pod then becomes ready again, it is re-added.
Unlike a liveness probe, if a container fails the readiness check, it will not be killed or restarted.
Note that the container may put itself into a unready state regardless of whether the readiness probe exists. The Pod remains in the unready state while it waits for the containers in the pod to stop.
Readiness and Multiple Containers per Pod
If a pod defines multiple containers, each container may declare its own readiness probe. The pod is considered ready when all of its containers are ready. If at least one container is not ready, even if all others are ready, the pod will not count as "ready" and it will not be added to, or it will be removed from the service Endpoints.
Manual Readiness Probe Example
readinessProbe:
exec:
command:
- ls
- /tmp/ready
initialDelaySeconds: 1
periodSeconds: 1
successThreshold: 1
failureThreshold: 1
timeoutSeconds: 1
Probe Template
The probe templates are sub-trees in the pod manifest.
kind: Pod
spec:
containers:
- name: ...
readinessProbe|livenessProbe:
exec:
Example:
readinessProbe|livenessProbe:
exec:
command:
- /bin/sh
- -c
- nodetool status | grep -E "^UN\s+${POD_IP}"
initialDelaySeconds: 90
periodSeconds: 30
successThreshold: 1
failureThreshold: 3
timeoutSeconds: 5
Elements
Also see Readiness Probe Operations above.
initialDelaySeconds
Specifies the number of seconds after the container has started before the probe is executed for the first time. After the initial delay, the probe is invoked periodically, with a periodicity of periodSeconds seconds.
periodSeconds
How often (in seconds) to perform the probe. Default to 10 seconds. Minimum value is 1. If the probe executes successfully, the next invocation will be executed in periodSeconds seconds.
timeoutSeconds
Number of seconds after which the probe times out. Defaults to 1 second. Minimum value is 1. What happens on timeout?
failureThreshold
Minimum consecutive failures for the probe to be considered failed after having succeeded. Defaults to 3. Minimum value is 1.
successThreshold
Minimum consecutive successes for the probe to be considered successful after having failed. Defaults to 1. Must be 1 for liveness. Minimum value is 1.