Kubernetes Cluster Configuration Concepts

Internal

• Kubernetes Concepts

Overview

Containerized applications can be configured with:

• command-line arguments, hardcoded into the container's manifest
• environment variables, hardcoded into the container's manifest
• mounting configuration files into containers through a volume
• ConfigMaps
• Secrets

ConfigMap

ConfigMap Overview

ConfigMaps are containers for storing configuration data. A ConfigMap is a key/value map, where the values range from short literals to full files. Other objects, for example pods, can access the data in a ConfigMap. ConfigMap data can be projected into a pod as an environment variable or as a file, part of a special volume mount. For very specific use cases, the ConfigMap can be read via the API Server API calls. This mechanism decouples configuration lifecycle from the pod (pod manifest and container images) lifecycle. ConfigMaps can be created from a manifest file, or imperatively on command line, by specifying finely grained values of individual key value pairs or coarsely grained entire files.

ConfigMap Details

ConfigMap Keys

ConfigMap keys must be a valid DNS subdomain - they may contain only alphanumeric characters, dashes, underscores and dots. They may optionally include a leading dot.

ConfigMap Projection

As Environment Variables

An individual key can be projected as environment variable as follows:

apiVersion: v1
kind: Pod
...
spec:
  containers:
  - name: ...
    ...
    env:
      - name: COLOR
        valueFrom:
          configMapKeyRef:
            name: example
            key: color

In the example above, "example" is the name of the ConfigMap that contains a "color" key. The value will be projected as a COLOR environment variable.

If the ConfigMap referred from a container definition does not exist, the pod (and other containers in the pod) will start normally, except the container that refers the non-existent ConfigMap.

If the ConfigMap is created after the pod is scheduled, the container referring it will start, without having to recreate the pod.

The container can be configured to start in absence of the ConfigMap by setting configMapKeyRef.optional: true.

All keys can be projected as environment variable at once as follows:

apiVersion: v1
kind: Pod
...
spec:
  containers:
  - name: ...
    ...
    envFrom:
      - prefix: PROJECTED_
        configMapRef:
          name: example

In the example above, the ConfigMap name is "example". Assuming it contains two keys ("color" and "shape"), the values will be projected in the pod as:

PROJECTED_color=red
PROJECTED_shape=square

Note that prefix is optional, if missing, the name of the environment variable will be the same as the name of the key.

As Files

The values of a ConfigMap can be projected into pods as content of "configMap" volumes. A "configMap" volume will expose each entry of the ConfigMap as an individual file. The process running inside the container can get the value by reading the file. This method is best suited for passing large config files, but it works for short values also.

ConfigMap Operations

• Create a ConfigMap with CLI by Specifying Key/Value Pairs
• Create a ConfigMap with CLI by Specifying the Content of a File

Secrets

Secrets

Secret Overview

A secret is a Kubernetes mechanism that can be used to expose security-sensitive data to pods as environment variables or as files. The most common method to create a secret is to write the content into a file, then use the file as argument to the Kubernetes API to create the secret resource. It is also possible to create a secret entirely from a manifest.

Secret Details

A secret is a mechanism, backed by the Secret Kubernetes API resource, that allows applications running on a Kubernetes cluster to safely manage, store and access security-sensitive information such as passwords, OAuth tokens and ssh keys. This mechanism provides a better alternative to placing that information in a container image or in the pod metadata. An individual secret contains a small amount of data, limited to 1 MiB - this is to discourage creation of very large secrets that would exhaust API server and kubelet memory. Entire multi-line configuration files can be exposed as secrets.

A Secret instance contains two maps: the data map, which is used to store arbitrary key/value pairs, where the values are base64-encoded string, and stringData map, which is a field provided for convenience that allows to provide secret data as unencoded fields. "stringData" field allows putting a non-base64 encoded string directly into the secret, and the string will be encoded by Kubernetes when the Secret is created or updated. When a secret instance is queried, only the data map is exposed.

A secrets exists in a namespace and they can only be referenced by pods in the same namespace. Multiple pods can reference the same secret. A pod must explicitly reference a secret in its manifest to access it. If that does not happen, the system will not initialize the infrastructure that exposes the information to the pod.

Both users and the system can create secrets. For example, service accounts automatically create secrets which contain credentials for accessing the API server and they automatically expose these secrets to the pods. Using these secrets is the recommended method to securely access the API server.

Secret Projection

Secrets are consumed by applications, and they can are projected into pods in two ways: as files and as environment variables. However, secrets can also be used by other parts of the system, without being directly exposed to pods. A secret needs to be created before a pod that depends on it is scheduled: once a pod is scheduled, the kubelet will try to fetch the secret value. If the secret cannot be fetched because it does not exist, kubelet will periodically retry and will report the occurrence as an event. Once the secret is fetched, the kubelet will create and mount a volume containing it. None of the pod’s containers will start until all the pod’s volumes are mounted.

Secrets Projected as Files

One option is to project secrets as files in dedicated volumes mounted in the pod. When the pod deploys, each key/value entry of the secret's data map is exposed as an individual file under the volume associated with the secret. The the file name is given by the key and the file content is filled with decoded secret value corresponding to the key. For specific configuration details, see Consume a Secret as a File. By default, each key is projected as a file in the volume root. However, specific keys can be mapped onto arbitrary relative paths in the volume, and the permissions under which corresponding files are exposed can be configured individually. For more details, see Projection of Keys to Specific Paths with Specific Permissions. Many files (key/value pairs) can be packaged into one secret, or many secrets can be used, whichever is most convenient.

If the key name starts with a dot, it will be projected as a "hidden file".

When a secret already projected in a volume is updated, the projected keys are eventually updated as well. The kubelet periodically checks whether the mounted secret is fresh. However, it is using its local cache for getting the current value of the secret, so depending on the cache configuration, the propagation delay of the fresh value to the pod can be as long as kubelet sync period + cache propagation delay, which depends on the cache type. Note that a secret propagated via a subPath volume mount will not receive updates. For more details on kubelet cache see:

kubelet Cache

Secrets Projected as Environment Variables

Another option would be to project secrets as environment variables in the pod's container environments. This option does not require declaring and mounting a volume, but just simply referencing a secret's key - each key/value pair declared in the secret is projected as an individual environment variable. The secret's keys appear as normal environment variables containing the base-64 decoded values of the secret value, inside the container's environment.

If a key is not a valid environment variable, that key will be skipped, and an event generated, but the pod will be allowed to start.

Secret Types

kubernetes.io/service-account-token

Accessible on pods as /var/run/secrets/kubernetes.io/serviceaccount.

imagePullSecrets

Image pull secrets are essentially a combination of registry, username, and password. TODO:

• https://kubernetes.io/docs/concepts/configuration/secret/#using-imagepullsecrets
• https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
• https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account

Secrets Operations

• Create a Secret with kubectl CLI
• Create a Secret from a manifest
• Consume a Secret as a File
  • Projection of Keys to Specific Paths with Specific Permissions
• Consume a Secret as an Environment Variable

Secrets TODO

• Automatic mounting of manually creates secrets. Manually created secrets, like those for accessing a GitHub account can be automatically attached to pods based on their service account. https://kubernetes.io/docs/tasks/inject-data-application/podpreset/
• Pod with ssh keys: https://kubernetes.io/docs/concepts/configuration/secret/#use-case-pod-with-ssh-keys
• Generating Secrets https://github.com/kubernetes-sigs/kustomize/blob/master/examples/secretGeneratorPlugin.md