Kubernetes Storage Concepts
Internal
Overview
Kubernetes has a mature and feature-rich subsystem called the persistent volume subsystem, which exposes external storage to applications.
Volumes
Regardless of where it comes from, external storage is exposed to pods in the form of volumes.
A Kubernetes volume has the same lifetime as the pod that encloses it. The volume outlives any containers that run within the pod, and data is preserved across container restarts. However, when a pod ceases to exist, the volume will too cease to exist. A pod can use multiple volumes, at the same time. Conceptually, a pod volume is just a directory, which is accessible to the containers in the pod. However, the actual backing medium of the directory, and its contents are determined by the particular volume type used. More details on how volumes and volume mounts are declared in the pod manifests are available in:
Also see difference between a (pod) volume and a Persistent volume.
Volume Types
configMap
secret
downwardAPI
emptyDir
An emptyDir volume is erased when the pod is removed.
hostPath
local
nfs
persistentVolumeClaim
projected
awsElasticBlockStore
An awsElasticBlockStore volume mounts an Amazon Elastic Block Store volume into the pod. When the pod is removed, the contents of the ESB volumes are preserved, and the ESB volume is merely unmounted. This means it can be pre-populated with data, which can be handed off to pods.
azureDisk
Volumes can be of several types:
- Persistent volume claim
- host path
TODO: Also see: Regardless of where it comes from, storage is exposed to the Kubernetes cluster in the form of volumes. Secrets may be exposed as files in dedicated volumes mounted in the pod.
Storage Providers
Storage is made available to a Kubernetes cluster by storage providers. The Kubernetes persistent volume subsystem supports, among others:
- iSCSI volumes
- SMB
- NFS volumes
- Enterprise storage arrays from vendors like EMC and NetApp
- object storage blobs
- Amazon Elastic Block Store block devices
- Azure File resources (AzureDisk).
- GCE Persistent Disks
- GlusterFS volumes
Each storage provider has its own plugin that handles the details of exposing the storage to the Kubernetes cluster.
Storage Plugins
The terms "storage plugin" and "provisioner" can be used interchangeably. "Provisioner" is used especially when dynamic provisioning is involved. "Driver" is another equivalent term for storage plugin.
Old storage plugins used to be implemented as part of the main Kubernetes code tree (in-tree), which raised a series of problems, such as that all had to be open-source and their release cycle was tied to the Kubernetes release cycle. Newer plugins are based on the Container Storage Interface (CSI) and can be developed out-of-tree.
Plugin Types
kuberentes.io/aws-ebs
kuberentes.io/gce-pd
Container Storage Interface (CSI)
Container Storage Interface (CSI) is an open standard that provides a clean interface for storage plugins and abstracts the internal Kubernetes storage details. CSI provides means so the external storage can be leveraged in a uniform way across multiple container orchestrators - not only Kubernetes.
API Resources
The persistent volume subsystem consists of the following three API resource types that allow applications to consume storage: persistent volumes, persistent volume claims and storage classes:
Persistent Volume (PV)
The persistent volume is the API resource that maps onto external storage assets and makes them accessible to the Kubernetes cluster and to applications. Each persistent volume is an object in the Kubernetes cluster that maps back to a specific storage asset (LUN, share, blob, etc.). A single external storage asset can only be used by a single persistent volume.
A pod can use a persistent volume by indicating a persistent volume claim (see below) whose access mode, storage class name and capacity match that of the persistent volume. The pod cannot specify a persistent volume directly, the match is intermediated by the Kubernetes cluster.
Difference between a Pod Volume and a Persistent Volume
Persistent Volume Manifest
Access Mode
A persistent volume can be bound in only one mode - it is not possible for a persistent volume to have a persistent volume claim bound to it in ROM mode and another persistent volume claim bound in RWM mode.
ReadWriteOnce (RWO)
This mode defines a persistent volume that can only be bound in read/write mode by a single persistent volume claim. An attempt to bind it via multiple persistent volume claims will fail. In general, block storage normally only supports RWO.
ReadWriteMany (RWM)
This mode defines a persistent volume that can be bound in read/write mode by multiple persistent volume claims. In general, file storage and object storage support RWM.
ReadOnlyMany (ROM)
This mode defines a persistent volume that can be bound in read only mode by multiple persistent volume claims.
Reclaim Policy
The reclaim policy tells Kubernetes what to do with a persistent volume when its persistent volume claim has been released.
Delete
This policy deletes the persistent volume and the underlying associated external storage resource, on the external storage system. This is the default policy for volumes that are created dynamically via a storage class.
Retain
This policy keeps the persistent volume in the cluster, as well as the underlying associated external storage resource, on the external storage system. However, it will prevent another persistent volume claim from using the persistent volume.
To reuse the space associated with a retained persistent volume, the persistent volume should be manually deleted, the underlying external storage reformatted and then the persistent volume should be recreated.
Storage Class Name
Capacity
The capacity, expressed in the persistent volume manifest, can be less than the actual underlying physical storage, but cannot be more.
Persistent Volume Claim (PVC)
Pods do not act directly on persistent volumes, they need something called persistent volume claim, which is an API resource object that is bound to the persistent volume the pod wants to use. A persistent volume claim is similar to a ticket that authorizes a pod to use a certain persistent volume. Once an application has a persistent volume claim, it can mount the respective volume into its pod.
Persistent Volume Claim Manifest
Storage Class (SC)
A storage class is an API resource that allows the definition of a class or tier of storage, from which an application can then dynamically request storage. Different classes might map to quality-of-service levels, or to backup policies, or to arbitrary policies defined by the cluster administrators. Obviously, the type of storage classes that can be defined depends on the types of external storage the Kubernetes cluster has access to. A pod can use a dynamically-provisioned persistent volume from a specific storage class by using a persistent volume claim that references that storage class by name. The persistent volume that will provide the storage does not need to be created or declared: the storage class creates the persistent volume dynamically. Once deployed, the storage class watches the API server for new PVC objects that reference its name. When a matching persistent volume claim appears, the storage class dynamically creates the required persistent volume.
The storage class resources are defined in the storage.k8s.io/v1 API group. Each storage class object relates to a single provisioner. StorageClass objects are immutable, they cannot be modified once deployed.
Storage Class Manifest
Default Storage Class
For the time being, the default storage class is set via annotations. If the cluster has a default storage class, a pod can be deployed using just a persistent volume claim - the storage class does not need to be manually created.