Kubernetes Container Probes: Difference between revisions
| Line 63: | Line 63: | ||
=Container Readiness Check= | =Container Readiness Check= | ||
Pods are added to a [[Kubernetes_Service_Concepts#Service_.28ClusterIP_Service.29|service]]'s [[Kubernetes_Service_Concepts#Endpoints|Endpoints]] list and forwarded traffic if there is a selector match '''and''' the pod readiness probe executes successfully. | Pods are added to a [[Kubernetes_Service_Concepts#Service_.28ClusterIP_Service.29|service]]'s [[Kubernetes_Service_Concepts#Endpoints|Endpoints]] list and forwarded traffic if there is a selector match '''and''' the pod readiness probe executes successfully. Once a pod is added to the Endpoints instance, the readiness probe is invoked repeatedly and the pod stays in the Endpoints list as long as the probe executes successfully. The pod is removed from the list on probe failure, but it can be added again if the probe starts succeeding again. | ||
Revision as of 03:38, 22 September 2020
External
- https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes
- https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes
- https://medium.com/swlh/fantastic-probes-and-how-to-configure-them-fef7e030bd2f
Internal
Overview
A probe is a diagnostic performed periodically by the kubelet on a container. To perform the diagnostic, the kubelet calls a handler, that must be declared and implemented by the container. Each probe has one of these results:
- success - the container passed the diagnostic
- failure - the container failed the diagnostic
- unknown - the diagnostic itself filed so no action should be taken.
There are three kinds of probes: startup, liveness and readiness.
Handlers
A handler is a piece of logic declared and implemented by the container, which is used by Kubernetes control mechanism to draw conclusions about the state the container is in There are three types of handlers, described below. Any of these handlers can be used to perform startup, liveness and readiness checks.
ExecAction
The exec action (declared as "exec:") executes a specified command inside the container. The diagnostic is considered successful if the command exits with a status code of 0.
TCPSocketAction
Performs a TCP check against the container’s IP address on a specified port. The diagnostic is considered successful if the port is open.
HTTPGetAction
Performs an HTTP GET request against the container’s IP address on a specified port and path. The diagnostic is considered successful if the response has a status code greater than or equal to 200 and less than 400
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 180
timeoutSeconds: 30
periodSeconds: 25
Container Startup Check
The startup check is performed by a startup probe. Startup probes have been introduced in Kubernetes 1.16. The probe indicates whether the application within the container is started. If a startup probe is not provided, the default result is "success". If a startup probe is provided, all other probes are disabled until the startup probe succeeds. If the startup probe fails, the container is killed and it is subject to its restart policy.
Container Liveness Check
The liveness check is performed by a liveness probe. The probe indicates whether the container is running. If a liveness probe is not provided, the default is "success". If a liveness probe is provided and it fails, the container will be killed and then subjected to its restart policy. (Not the pod? How about atomicity?)
Relationship between killing pods and containers - research needed.
Container Readiness Check
Pods are added to a service's Endpoints list and forwarded traffic if there is a selector match and the pod readiness probe executes successfully. Once a pod is added to the Endpoints instance, the readiness probe is invoked repeatedly and the pod stays in the Endpoints list as long as the probe executes successfully. The pod is removed from the list on probe failure, but it can be added again if the probe starts succeeding again.
A container IP address is added to the service's Endpoints list only if it is ready to serve traffic, as reflected by the result of its readiness probe. The probe is executed periodically and the container is taken out of the list if the probe fails - and possibly re-added if the probe starts succeeding again. More details about readiness probes:
The readiness check is performed by a readiness probe. If the container does not provide a readiness probe, the default diagnostic result is "success". The result of the container readiness check is used by Kubernetes to know when the container is ready to accept traffic - serve requests. The presence of a readiness probe allows to start the container without sending traffic into it, until it is ready. The pod is considered ready when all of its containers are ready.
The readiness check is used by services to decide whether to send traffic into the pod or not. If the probe fails, the container will be removed from service endpoints.
The default state of readiness before the initial delay is "failure".
Clarify the relationship between container and pod readiness.
The pod's readiness state is displayed in the output of kubectl get pod command:
NAME READY STATUS RESTARTS AGE cassandra-0 0/1 Running 0 23s
Note that the container may put itself into a unready state regardless of whether the readiness probe exists. The Pod remains in the unready state while it waits for the containers in the pod to stop.
Probe Template
kind: Pod
spec:
containers:
- name: ...
readinessProbe|livenessProbe:
exec:
Example:
readinessProbe|livenessProbe:
exec:
command:
- /bin/sh
- -c
- nodetool status | grep -E "^UN\s+${POD_IP}"
initialDelaySeconds: 90
periodSeconds: 30
successThreshold: 1
failureThreshold: 3
timeoutSeconds: 5
Elements
initialDelaySeconds
Specifies the number of seconds after the container has started before the probe is initiated.
periodSeconds
How often (in seconds) to perform the probe. Default to 10 seconds. Minimum value is 1.
timeoutSeconds
Number of seconds after which the probe times out. Defaults to 1 second. Minimum value is 1. What happens on timeout?
failureThreshold
Minimum consecutive failures for the probe to be considered failed after having succeeded. Defaults to 3. Minimum value is 1.
successThreshold
Minimum consecutive successes for the probe to be considered successful after having failed. Defaults to 1. Must be 1 for liveness. Minimum value is 1.
TODO
- Merge and deplete OpenShift Container Probes.