Kubernetes Cluster Configuration Concepts

External

• https://12factor.net/config

Internal

• Kubernetes Concepts
• Kubernetes Storage Concepts
• Go Configuration

Overview

Mixing code and configuration is an anti-pattern in the context of continuous delivery, where the application image is created once and then moves unaltered through various stages of the deployment pipeline until reaches production. Configuration does not belong in the image. It must be externalized and maintained separately. This approach promotes sharing ion immutable application container images.

Configuration can be maintained separately as:

• Application command-line arguments, hardcoded into the container's manifest.
• Environment variables, hardcoded into the container's manifest or injected at runtime. A discussion about suitability of environment variables to configure Kubernetes-deployed applications, with advantages and disadvantages, is available in Kubernetes Patterns | Environment Variable-Based Configuration.
• Configuration files exposed to container via a mounted volume.
• ConfigMaps.
• Secrets.
• Combinations of the above.

Default Configuration

Default configuration values play a central role in the "convention over configuration" paradigm. For the most common situations, properly chosen default configuration values reduce friction and increase operational efficiency.

However, there are situations when an improperly chosen default value, or a shifting usage pattern, require changing the default value, and this is a difficult task, akin to changing code (default values are hardcoded, and changing them requires a rebuild). Then, people are usually surprised by the new behavior. The change has to be communicated as it would be an API change, and the application will require a semver major version release.

If no natural default value emerges, it's better to leave it to external configuration and get the application to throw an error if the configuration is missing. This will break the application early and prominently, instead of causing to silently fail somewhere else.

If you are not 90% that the default value is reasonable, avoid default values.

Database URLs and passwords are obvious candidates for not providing default values.

Configuration Patterns

Kubernetes Patterns

Environment Variable-Based Configuration

Kubernetes Patterns | Environment Variable-Based Configuration

Configuration Resource

Kubernetes Patterns | Configuration Resource

Immutable Configuration

Kubernetes Patterns | Immutable Configuration

Configuration Template

Kubernetes Patterns | Configuration Template

ConfigMap

https://kubernetes.io/docs/concepts/configuration/configmap/

https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/

ConfigMap Overview

ConfigMaps is a Kubernetes resource dedicated to storing configuration data that allow decoupling an application configuration cycle from the application's build cycle.

A ConfigMap is a key/value map, where the values range from short literals to full files. Other Kubernetes resources, like pods, can access the data in a ConfigMap. ConfigMap data can be projected into a pod as environment variables or as files, in the form of a volume mount of a configMap volume. For very specific use cases, the ConfigMap can be read via the API Server API calls.

Exposing configuration as ConfigMap decouples configuration lifecycle from the pod (pod manifest and container images) lifecycle.

ConfigMaps can be created from a manifest file, or imperatively on command line, by specifying finely grained values of individual key value pairs or coarsely grained entire files.

ConfigMaps are namespace-scoped. They reside in a specific namespace and can only be referenced by pods deployed in the same namespace. Unlike most Kubernetes resources, that have a spec, ConfigMaps has a data and binaryData fields.

ConfigMap Manifest

ConfigMap Manifest

ConfigMap Details

ConfigMap Keys

ConfigMap keys must be a valid DNS subdomain. They may contain only alphanumeric characters, dashes, underscores and dots. They may optionally include a leading dot.

ConfigMap Projection

As Environment Variables

An individual ConfigMap key can be projected as environment variable for a pod containers if the pod manifest is configured as follows:

apiVersion: v1
kind: Pod
...
spec:
  containers:
  - name: ...
    ...
    env:
      - name: COLOR
        valueFrom:
          configMapKeyRef:
            name: example
            key: COLOR

In the example above, "example" is the name of the ConfigMap that contains a "COLOR" key. The value will be projected as a COLOR environment variable. It is recommended to use all upper cap characters for the ConfigMap keys intended to be projected as environment variables. Any key that cannot be used as an environment variable name (example: illeg.al) is ignored. If the same environment variable is defined in a ConfigMap and set directly as an env array element, the value set in the array takes precedence.

If the ConfigMap referred from a container definition does not exist, the pod and its containers will start normally, except the container that refers the non-existent ConfigMap.

If the ConfigMap is created after the pod is scheduled, the container referring it will start, without having to recreate the pod.

The container can be configured to start in absence of the ConfigMap by setting configMapKeyRef.optional: true.

All keys can be projected as environment variable at once as follows:

apiVersion: v1
kind: Pod
...
spec:
  containers:
  - name: ...
    ...
    envFrom:
      - prefix: PROJECTED_
        configMapRef:
          name: example

In the example above, the ConfigMap name is "example". Assuming it contains two keys ("COLOR" and "SHAPE"), the values will be projected in the pod as:

PROJECTED_COLOR=red
PROJECTED_SHAPE=square

Note that prefix is optional, if missing, the name of the environment variable will be the same as the name of the key.

Changes of ConfigMap entries projected as environment variables do not reflect in the values of the environment variables injected in the container environment, because environment variables cannot be changed after the process has been started.

Projecting both Individual Keys and All Keys

envFrom and env are not mutually exclusive, both can be used for the same container, and the environment variable space will contain the contributions from all.

As Files

The values of a ConfigMap can be projected into pods as content of "configMap" volumes. A "configMap" volume will expose each entry of the ConfigMap as an individual file. The process running inside the container can get the value by reading the file. This method is best suited for passing large config files, but it works for short values also.

It is recommended to proper file name capitalization for those keys intended to be projected as files.

apiVersion: v1
kind: Pod
...
spec:
  containers:
  - name: ...
    ...
    volumeMounts:
    - name: config-volume
      mountPath: /etc/my-config-dir
      readOnly: true
  volumes:
  - name: config-volume
    configMap:
      name: example

In the above example, if the ConfigMap contains two files (A.txt and B.txt), their content will be available in the container in the /etc/my-config-dir directory.

If /etc/my-config-dir exists in the container's filesystem, and it contains files, they will be hidden by the mount operation, unless subPath option is used.

Only select entries of a ConfigMap, as opposite to all of them, can be projected using the items syntax. When specifying individual entries, the filename of each individual entry should be set with path. The key can be the same as the path. Individual permissions under which the file is made available can be specified with mode.

If a key that does not exist in ConfigMap is specified in the items array, the pod will stay in "ContainerCreating" status, with a "FailedMount" warning.

apiVersion: v1
kind: Pod
...
spec:
  ...
  volumes:
  - name: config-volume
    configMap:
      name: example
      items:
      - key: A.txt
        path: Projected-A.txt
        mode: 0400

The files can be mounted with specific options using defaultMode.

ConfigMap Updates

For a ConfigMap projected as a volume, the volume content will change and the new content will be exposed to the pod when the backing ConfigMap changes, without having to recreate the pod or the container. The change propagates with a delay. However, this is not true anymore if individual entries in a ConfigMap are projected with subPath (see below). If the ConfigMap projects multiple files, all files are updated atomically, via a mechanism that involves symbolic links.

subPath

Normally, mounting a ConfigMap volume hides all files in the mount point directory, unless subPath option is used. subPath is a syntax that allows mounting of just one file inside a mount point, without hiding other files in that mount point directory. The files in the target mount point directory are still accessible, and only the ConfigMap key specified by subPath is mounted.

apiVersion: v1
kind: Pod
...
spec:
  containers:
  - name: ...
    ...
    volumeMounts:
    - name: config-volume
      mountPath: /etc/security/my-A.txt
      subPath: A.txt
  volumes:
  - name: config-volume
    configMap:
      name: example

subPath-mounted files will not be updated when the backing ConfigMap changes, unlike their non-subPath counterparts.

Immutable ConfigMaps

The immutable field, available since 1.21, allows configuring the resource to prevent update after creation.

ConfigMap Operations

• Create a ConfigMap with CLI by Specifying Key/Value Pairs
• Create a ConfigMap with CLI by Specifying the Content of a File
• Edit an Already Deployed ConfigMap

Secrets

https://kubernetes.io/docs/concepts/configuration/secret/

https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/

Secret Overview

A secret is a Kubernetes resource dedicated to storing and exposing security-sensitive data to pods as environment variables or as files, projected into container as a volume mount of a secret volume.

Secrets allow decoupling an application configuration cycle from the application's build cycle.

The most common method to create a secret is to write the content into a file, then use the file as argument to the Kubernetes API to create the secret resource. It is also possible to create a secret entirely from a manifest.

Secrets differ from ConfigMap in that their data is base64-encoded. Additionally, a Secret is distributed only to nodes running Pods that need access to the Secret. On the nodes, Secrets are stored in memory in a tmpfs and never written to physical storage. They are removed when the Pod is removed. In etcd, the Secrets can be stored in encrypted form. For more details see:

https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/

Secret Manifest

Secret Manifest

Secret Details

A secret is a mechanism, backed by the Secret Kubernetes API resource, that allows applications running on a Kubernetes cluster to safely manage, store and access security-sensitive information such as passwords, OAuth tokens and ssh keys. This mechanism provides a better alternative to placing that information in a container image or in the pod metadata. An individual secret contains a small amount of data, limited to 1 MiB. This limitation exists to discourage creation of very large secrets that would exhaust API server and kubelet memory. Entire multi-line configuration files can be exposed as secrets.

A Secret instance contains two maps: the data map, which is used to store arbitrary key/value pairs, where the values are base64-encoded string, and stringData map, which is a convenience field that allows to provide secret data as unencoded fields. The stringData field allows placing a non-base64 encoded string directly into the secret, and the string will be encoded by Kubernetes when the Secret is created or updated. When a secret instance is queried, only the data map is exposed. It is valid to declare an "empty" secret. An empty secrete does not contain any data or stringData map elements. If such a secret is projected as file, the mountPath directory is created, if it did not exist already, but no files show up in it.

A secret is namespaced and it can only be referenced by pods in the same namespace.

Multiple pods can reference the same secret. A pod must explicitly reference a secret in its manifest to access it. If that does not happen, the system will not initialize the infrastructure that exposes the information to the pod.

Both users and the system can create secrets. For example, service accounts automatically create secrets which contain credentials for accessing the API server and they automatically expose these secrets to the pods. Using these secrets is the recommended method to securely access the API server.

A secret needs to be created before a pod that depends on it is scheduled. Once a pod is scheduled, the kubelet will try to fetch the secret value. If the secret cannot be fetched because it does not exist, the kubelet will periodically retry and will report the occurrence as an event, and eventually time out. While in that state, the pod reports a "ContainerCreating" status, and kubectl describe provides more details:

Events:
 Type     Reason       Age                   From                     Message
 ----     ------       ----                  ----                     -------
 Warning  FailedMount  18s (x10 over 4m28s)  kubelet, docker-desktop  MountVolume.SetUp failed for volume "secret-volume" : secret "blue" not found
 Warning  FailedMount  7s (x2 over 2m25s)    kubelet, docker-desktop  Unable to mount volumes for pod "httpd_default(3dfc3194-c812-4353-a760-70b80abf6a1f)": 
                                                                      timeout expired waiting for volumes to attach or mount for pod "default"/"httpd". 
                                                                      list of unmounted volumes=[secret-volume]. 
                                                                      list of unattached volumes=[secret-volume default-token-dxrf8]

Once the secret is fetched, the kubelet will create and mount a volume containing it. None of the pod’s containers will start until all the pod’s volumes are mounted.

The behavior on secret update or deletion depends on how the secret is projected in the pod. If a secret is projected as file, the projection behaves differently on update or removal than in the case of environment variables.

Secret Projection in Pods

In most cases, secrets are consumed by applications deployed in pods, so the secret needs to be projected into the pod. There are two ways to project a secret into a pod:

• As files in a volume mounted by one or more of the pod's containers.
• As container environment variables.

However, secrets can also be used by other parts of the system, without being directly exposed to pods. An often encountered example involves secrets used by the kubelet when pulling images for the pod.

Secrets Projected as Files

One option is to project secrets as files in dedicated pod volumes mounted by one or more of the pod's containers. When the pod deploys, each key/value entry of the secret's data map is exposed as an individual file under the volume associated with the secret. It is valid to declare an empty secret, with no data entries, and in this case no corresponding files will be created.

File Names and Paths

The file name is given by the key and the file content is filled with decoded secret value corresponding to the key. The key, and implicitly the file name, can contain dashes and dots. By default, each key is projected as a file in the volume root. However, specific keys can be mapped onto arbitrary relative paths in the volume, and the permissions under which corresponding files are exposed can be configured individually. Many files (key/value pairs) can be packaged into one secret, or many secrets can be used, whichever is most convenient.

If the key name starts with a dot, it will be projected as a "hidden file". This means that when the secret is projected as file, the file created for that key will cary the name of the key, and since the name of the key starts with a dot, it is usually handled as a "hidden file" by most OSes. For specific configuration details, and advanced behavior see Consume a Secret as a File.

TODO: It seems that if the directory corresponding to the pod volume mount point does not exist, and the mount point directory is created, the default permissions are: drwxrwxrwt 3 root root 100 Jan 29 06:47. How can that be changed to something more restrictive via configuration?

File Projection on Secret Update

When a secret already projected in a volume is updated, the projected keys are eventually updated as well. The kubelet periodically checks whether the mounted secret is fresh. However, it is using its local cache for getting the current value of the secret, so depending on the cache configuration, the propagation delay of the fresh value to the pod can be as long as kubelet sync period + cache propagation delay, which depends on the cache type. Note that a secret propagated via a subPath volume mount will not receive updates. For more details on kubelet cache see:

kubelet Cache

File Projection on Secret Removal

If the secret is deleted while being projected in a pod as a file, the pod will keep running, and the last version of the secret will remain available. However, a warning event will appear:

Warning  FailedMount  28s (x9 over 2m36s)  kubelet, docker-desktop  MountVolume.SetUp failed for volume "secret-volume" : secret "blue" not found

The pod will fail to start, if restarted, as described above.

Secrets Projected as Environment Variables

Another option would be to project secrets as environment variables in the pod's container environments. This option does not require declaring and mounting a volume, but just simply referencing a secret's key - each key/value pair declared in the secret is projected as an individual environment variable. The secret's keys appear as normal environment variables containing the base-64 decoded values of the secret value, inside the container's environment.

If a key is not a valid environment variable, that key will be skipped, and an event generated, but the pod will be allowed to start.

Secrets Required to Pull Images for Pods

https://kubernetes.io/docs/concepts/configuration/secret/#using-imagepullsecrets

https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod

In most cases, secrets are required by applications deployed in pods, so they are projected into the pods. However, an often encountered situation when this is NOT the case is when we expose secrets to the kubelet so it can download images required by the pods.

The secrets required by the kubelet to pull images can be provided in two ways:

• in the the pod manifest
• in a service account

imagePullSecrets Field in the Pod Manifest

The pod manifest can be used to list secrets required by the kubelet to download images. Those secrets are listed under the optional imagePullSecret field of the pod manifest:

Pod Manifest

imagePullSecrets element contains an optional list of secret names to use for pulling any of the images used by the pod described by the manifest. The secrets referred from this list must be deployed and exist in the same namespace. If provided, these secrets will be passed to individual puller implementations for them to use:

apiVersion: v1
kind: Pod
spec:  
  imagePullSecrets:
    - name: myPullSecret1
    - name: myPullSecret2
    - ...

In the case of docker, only DockerConfig type secrets are honored.

For a summary of various image pull concepts see:

Kubernetes Container Image Pull Concepts | Private Registries

imagePullSecrets in a Service Account

https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account

Secret Types

https://kubernetes.io/docs/concepts/configuration/secret/#secret-types

The secret type is configured with the "type" top-level key:

apiVersion: v1
kind: Secret
type: <secret-type>

Opaque

type: Opaque

kubernetes.io/service-account-token

type: kubernetes.io/service-account-token

Accessible on pods as /var/run/secrets/kubernetes.io/serviceaccount. For more details see:

Kubernetes Security Concepts | Service Accounts Credentials (service-account-token)

kubernetes.io/basic-auth

type: kubernetes.io/basic-auth

kubernetes.io/ssh-auth

type: kubernetes.io/ssh-auth

kubernetes.io/tls

type: kubernetes.io/tls

bootstrap.kubernetes.io/token

type: bootstrap.kubernetes.io/token

Docker Secrets

https://kubernetes.io/docs/concepts/configuration/secret/#docker-config-secrets

There are two types of Docker-specific secrets that can be used to pull images from authenticated Docker registries:

• kubernetes.io/dockerconfigjson
• kubernetes.io/dockercfg (legacy)

kubernetes.io/dockerconfigjson

The kubernetes.io/dockerconfigjson Secret type is designed for storing serialized JSON content that follows the same format rules as ~/.docker/config.json, which is the new format for ~/.dockercfg. When using this Secret type, the data field must contain a .dockerconfigjson key, whose value is the base64-encoded ~/.docker/config.json value.

apiVersion: v1
kind: Secret
metadata:
  name: someDockerConfigJsonSecret
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: 'faJ...eZ0=' # base64 encoded ~/.docker/config.json file

Once deployed, this Secret projects as a serialized ~/.docker/config.json file.

kubectl can be used to create the secret, including the properly formatted ~/.docker/config.json file content, with the following command:

kubectl create secret docker-registry my-secret \
  --docker-server=https://someregistry.somedomain.io \
  --docker-username=somename \
  --docker-password=somepasswd \
  --docker-email=somename@acme.com

For in-line help:

kubectl create secret docker-registry --help

The command will create the following secret:

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: eyJhdXRocyI6eyJodHRwczovL3NvbWVyZWdpc3RyeS5zb21lZG9tYWluLmlvIjp7InVzZXJuYW1lIjoic29tZW5hbWUiLCJwYXNzd29yZCI6InNvbWVwYXNzd2QiLCJhdXRoIjoiYzI5dFpXNWhiV1U2YzI5dFpYQmhjM04zWkE9PSJ9fX0=

The base64-encoded .dockerconfigjson can be decoded at:

{
  "auths": {
    "https://someregistry.somedomain.io": {
      "username":"somename",
      "password":"somepasswd",
      "auth":"c29tZW5hbWU6c29tZXBhc3N3ZA=="
    }
  }
}

kubernetes.io/dockercfg

type: kubernetes.io/dockercfg

This is a legacy format, use kubernetes.io/dockerconfigjson if possible.

Immutable Secrets

https://kubernetes.io/docs/concepts/configuration/secret/#secret-immutable

The immutable field, available since 1.21, allows configuring the resource to prevent update after creation.

Secrets Operations

• Create a Secret with kubectl CLI
• Create a Secret from a manifest
• Consume a Secret as a File
  • Projection of Keys to Specific Paths with Specific Permissions
• Consume a Secret as an Environment Variable
• Update a Secret
• Delete a Secret

Secrets TODO

• Automatic mounting of manually creates secrets. Manually created secrets, like those for accessing a GitHub account can be automatically attached to pods based on their service account. https://kubernetes.io/docs/tasks/inject-data-application/podpreset/
• Pod with ssh keys: https://kubernetes.io/docs/concepts/configuration/secret/#use-case-pod-with-ssh-keys
• Generating Secrets https://github.com/kubernetes-sigs/kustomize/blob/master/examples/secretGeneratorPlugin.md

Downward API

Downward API