Kubernetes Cluster Configuration Concepts

From NovaOrdis Knowledge Base
Jump to navigation Jump to search

Internal

Overview

Containerized applications can be configured with:

  • command-line arguments, hardcoded into the container's manifest.
  • environment variables, hardcoded into the container's manifest.
  • mounting configuration files into containers through a volume.
  • ConfigMaps.
  • Secrets.

ConfigMap

ConfigMap Overview

ConfigMaps are containers for storing configuration data. A ConfigMap is a key/value map, where the values range from short literals to full files. Other objects, for example pods, can access the data in a ConfigMap. ConfigMap data can be projected into a pod as an environment variable or as a file, part of a volume mount of a configMap volume. For very specific use cases, the ConfigMap can be read via the API Server API calls. This mechanism decouples configuration lifecycle from the pod (pod manifest and container images) lifecycle. ConfigMaps can be created from a manifest file, or imperatively on command line, by specifying finely grained values of individual key value pairs or coarsely grained entire files.

ConfigMaps reside in a specific namespace. A ConfigMap can only be referenced by pods residing in the same namespace

ConfigMap Details

ConfigMap Keys

ConfigMap keys must be a valid DNS subdomain - they may contain only alphanumeric characters, dashes, underscores and dots. They may optionally include a leading dot.

ConfigMap Projection

As Environment Variables

An individual key can be projected as environment variable as follows: 

apiVersion: v1
kind: Pod
...
spec:
  containers:
  - name: ...
    ...
    env:
      - name: COLOR
        valueFrom:
          configMapKeyRef:
            name: example
            key: color

In the example above, "example" is the name of the ConfigMap that contains a "color" key. The value will be projected as a COLOR environment variable.

If the ConfigMap referred from a container definition does not exist, the pod (and other containers in the pod) will start normally, except the container that refers the non-existent ConfigMap.


If the ConfigMap is created after the pod is scheduled, the container referring it will start, without having to recreate the pod.

The container can be configured to start in absence of the ConfigMap by setting configMapKeyRef.optional: true.

All keys can be projected as environment variable at once as follows: 

apiVersion: v1
kind: Pod
...
spec:
  containers:
  - name: ...
    ...
    envFrom:
      - prefix: PROJECTED_
        configMapRef:
          name: example

In the example above, the ConfigMap name is "example". Assuming it contains two keys ("color" and "shape"), the values will be projected in the pod as: 

PROJECTED_color=red
PROJECTED_shape=square

Note that prefix is optional, if missing, the name of the environment variable will be the same as the name of the key.

Projecting both Individual Keys and All Keys

"envFrom" and "env" are not mutually exclusive, both can be used for the same container, and the environment variable space will contain the contributions from all.

As Files

The values of a ConfigMap can be projected into pods as content of "configMap" volumes. A "configMap" volume will expose each entry of the ConfigMap as an individual file. The process running inside the container can get the value by reading the file. This method is best suited for passing large config files, but it works for short values also. 

apiVersion: v1
kind: Pod
...
spec:
  containers:
  - name: ...
    ...
    volumeMounts:
    - name: config-volume
      mountPath: /etc/my-config-dir
      readOnly: true
  volumes:
  - name: config-volume
    configMap:
      name: example

In the above example, if the ConfigMap contains two files (A.txt and B.txt), their content will be available in the container in the /etc/my-config-dir directory.


If /etc/my-config-dir exists in the container's filesystem, and it contains files, they will be hidden by the mount operation, unless subPath option is used.

Only select entries of a ConfigMap, as opposite to all of them, can be projected with the following syntax - which only affects the volume definition. When specifying individual entries, the filename of each individual entry should be set with path. The key can be the same as the path.


If a key that does not exist in ConfigMap is specified in the "items" array, the pod will stay in "ContainerCreating" status, with a "FailedMount" warning.

apiVersion: v1
kind: Pod
...
spec:
  ...
  volumes:
  - name: config-volume
    configMap:
      name: example
      items:
      - key: A.txt
        path: Projected-A.txt

The files can be mounted with specific options using defaultMode.

ConfigMap Updates

ConfigMap state in case of a ConfigMap projected as a volume will be updated in the pod when the backing ConfigMap changes - after a certain delay - without having to recreate the pod or the container. However, this is not true anymore if individual entries in a ConfigMap are projected with subPath (see below). If the ConfigMap projects multiple files, all files are updated atomically, via a mechanism that involves symbolic links.

subPath

Normally, mounting a configMap volume hides all files in the mount point directory, unless subPath option is used. subPath is a syntax that allows mounting of just one file inside a mount point, without hiding other files in that mount point directory. The files in the target mount point directory are still accessible, and only the ConfigMap key specified by subPath is mounted. 

apiVersion: v1
kind: Pod
...
spec:
  containers:
  - name: ...
    ...
    volumeMounts:
    - name: config-volume
      mountPath: /etc/security/my-A.txt
      subPath: A.txt
  volumes:
  - name: config-volume
    configMap:
      name: example

subPath-mounted files will not be updated when the backing ConfigMap changes, unlike their non-subPath counterparts.

ConfigMap Operations

Secrets

https://kubernetes.io/docs/concepts/configuration/secret/

Secret Overview

A secret is a Kubernetes mechanism that can be used to expose security-sensitive data to pods as environment variables or as files, projected into container as a volume mount of a secret volume. The most common method to create a secret is to write the content into a file, then use the file as argument to the Kubernetes API to create the secret resource. It is also possible to create a secret entirely from a manifest.

Secret Manifest

Secret Manifest

Secret Details

A secret is a mechanism, backed by the Secret Kubernetes API resource, that allows applications running on a Kubernetes cluster to safely manage, store and access security-sensitive information such as passwords, OAuth tokens and ssh keys. This mechanism provides a better alternative to placing that information in a container image or in the pod metadata. An individual secret contains a small amount of data, limited to 1 MiB - this is to discourage creation of very large secrets that would exhaust API server and kubelet memory. Entire multi-line configuration files can be exposed as secrets.

A Secret instance contains two maps: the data map, which is used to store arbitrary key/value pairs, where the values are base64-encoded string, and stringData map, which is a field provided for convenience that allows to provide secret data as unencoded fields. "stringData" field allows putting a non-base64 encoded string directly into the secret, and the string will be encoded by Kubernetes when the Secret is created or updated. When a secret instance is queried, only the data map is exposed. It is valid to declare an "empty" secret - a secret without any data or stringData map elements. If such a secret is projected a

A secret is namespaced and it can only be referenced by pods in the same namespace.

Multiple pods can reference the same secret. A pod must explicitly reference a secret in its manifest to access it. If that does not happen, the system will not initialize the infrastructure that exposes the information to the pod.

Both users and the system can create secrets. For example, service accounts automatically create secrets which contain credentials for accessing the API server and they automatically expose these secrets to the pods. Using these secrets is the recommended method to securely access the API server.

A secret needs to be created before a pod that depends on it is scheduled: once a pod is scheduled, the kubelet will try to fetch the secret value. If the secret cannot be fetched because it does not exist, the kubelet will periodically retry and will report the occurrence as an event, and eventually time out. While in that state, the pod reports a "ContainerCreating" status, and kubectl describe provides more details: 

Events:
  Type     Reason       Age                   From                     Message
  ----     ------       ----                  ----                     -------
  Warning  FailedMount  18s (x10 over 4m28s)  kubelet, docker-desktop  MountVolume.SetUp failed for volume "secret-volume" : secret "blue" not found
  Warning  FailedMount  7s (x2 over 2m25s)    kubelet, docker-desktop  Unable to mount volumes for pod "httpd_default(3dfc3194-c812-4353-a760-70b80abf6a1f)": 
                                                                       timeout expired waiting for volumes to attach or mount for pod "default"/"httpd". 
                                                                       list of unmounted volumes=[secret-volume]. 
                                                                       list of unattached volumes=[secret-volume default-token-dxrf8]

Once the secret is fetched, the kubelet will create and mount a volume containing it. None of the pod’s containers will start until all the pod’s volumes are mounted.

The behavior on secret update or deletion depends on how the secret is projected in the pod: if a secret is projected as file, the projection behaved differently on update or removal than in the case of environment variables.

Secret Types

Opaque

kubernetes.io/service-account-token

Accessible on pods as /var/run/secrets/kubernetes.io/serviceaccount.

imagePullSecrets

Image pull secrets are essentially a combination of registry, username, and password. TODO:

Also see:

Container Image Pull Concepts

Secret Projection

Secrets are consumed by applications, and they can are projected into pods in two ways: as files and as environment variables. However, secrets can also be used by other parts of the system, without being directly exposed to pods.

Secrets Projected as Files

One option is to project secrets as files in dedicated pod volumes. When the pod deploys, each key/value entry of the secret's data map is exposed as an individual file under the volume associated with the secret.

File Names and Paths

The file name is given by the key and the file content is filled with decoded secret value corresponding to the key. The key, and implicitly the file name, can contain dashes and dots. By default, each key is projected as a file in the volume root. However, specific keys can be mapped onto arbitrary relative paths in the volume, and the permissions under which corresponding files are exposed can be configured individually. Many files (key/value pairs) can be packaged into one secret, or many secrets can be used, whichever is most convenient.

If the key name starts with a dot, it will be projected as a "hidden file". This means that when the secret is projected as file, the file created for that key will cary the name of the key, and since the name of the key starts with a dot, it is usually handled as a "hidden file" by most OSes. For specific configuration details, and advanced behavior see Consume a Secret as a File.

TODO: It seems that if the directory corresponding to the pod volume mount point does not exist, and the mount point directory is created, the default permissions are: drwxrwxrwt 3 root root 100 Jan 29 06:47. How can that be changed to something more restrictive via configuration?

File Projection on Secret Update

When a secret already projected in a volume is updated, the projected keys are eventually updated as well. The kubelet periodically checks whether the mounted secret is fresh. However, it is using its local cache for getting the current value of the secret, so depending on the cache configuration, the propagation delay of the fresh value to the pod can be as long as kubelet sync period + cache propagation delay, which depends on the cache type. Note that a secret propagated via a subPath volume mount will not receive updates. For more details on kubelet cache see:

kubelet Cache

File Projection on Secret Removal

If the secret is deleted while being projected in a pod as a file, the pod will keep running, and the last version of the secret will remain available. However, a warning event will appear: 

Warning  FailedMount  28s (x9 over 2m36s)  kubelet, docker-desktop  MountVolume.SetUp failed for volume "secret-volume" : secret "blue" not found

Obviously, the pod will fail to start, if restarted, as described above.

Secrets Projected as Environment Variables

Another option would be to project secrets as environment variables in the pod's container environments. This option does not require declaring and mounting a volume, but just simply referencing a secret's key - each key/value pair declared in the secret is projected as an individual environment variable. The secret's keys appear as normal environment variables containing the base-64 decoded values of the secret value, inside the container's environment.

If a key is not a valid environment variable, that key will be skipped, and an event generated, but the pod will be allowed to start.

Environment Variable Projection on Secret Update

Environment Variable Projection on Secret Removal

Secrets Operations

Secrets TODO

Retrieved from "https://kb.novaordis.com/index.php?title=Kubernetes_Cluster_Configuration_Concepts&oldid=72798"