Kubernetes Storage Concepts

Internal

• Kubernetes Concepts

Overview

Kubernetes has a mature and feature-rich subsystem called the persistent volume subsystem. Regardless of where it comes from, storage is exposed to the Kubernetes cluster in the form of a volume.

Storage Providers

Storage is exposed to a Kubernetes cluster by storage providers.

The Kubernetes persistent volume subsystem supports, among others:

• iSCSI volumes
• SMB
• Enterprise storage arrays from vendors like EMC and NetApp
• NFS volumes
• object storage blobs
• Amazon Elastic Block Store block devices
• Azure File resources (AzureDisk).
• GCE Persistent Disks

Each storage provider has its own plugin that handles the details of exposing the storage to the Kubernetes cluster.

Storage Plugins

The terms "storage plugin" and "provisioner" are used interchangeably. "Provisioner" is used especially when dynamic provisioning is involved. "Driver" is a term that is also used here.

Old storage plugins used to be implemented as part of the main Kubernetes code tree (in-tree), which raised a series of problems, such as that all had to be open-source and their release cycle was tied to the Kubernetes release cycle.

Newer plugins are based on the Container Storage Interface (CSI) and can be developed out-of-tree.

Plugin Types

The plugins are also referred to as "provisioners":

Container Storage Interface (CSI)

Container Storage Interface (CSI) is an open standard that provides a clean interface for storage plugins and abstracts the internal Kubernetes storage details. CSI provides means so the external storage can be leveraged in a uniform way across multiple container orchestrators (not only Kubernetes).

API Resources

The persistent volume subsystem consists of the following three API resource types that allow applications to consume storage: persistent volumes, persistent volume claims and storage classes:

Persistent Volume (PV)

The persistent volume is the API resource that maps onto external storage assets and makes them accessible to the Kubernetes cluster, and to applications. Each persistent volume is an object in the Kubernetes cluster that maps back to a specific storage asset (LUN, share, blob, etc.). A single external storage asset can only be used by a single persistent volume.

A pod can use a persistent volume by indicating a persistent volume claim (see below) whose access mode, storage class name and capacity match that of the persistent volume. The pod cannot specify a persistent volume directly, the match is intermediated by the Kubernetes cluster.

Persistent Volume Manifest

Persistent Volume Manifest

Access Mode

A persistent volume can be bound in only one mode - it is not possible for a persistent volume to have a persistent volume claim bound to it in ROM mode and another persistent volume claim bound in RWM mode.

ReadWriteOnce (RWO)

This mode defines a persistent volume that can only be bound in read/write mode by a single persistent volume claim. An attempt to bind it via multiple persistent volume claims will fail. In general, block storage normally only supports RWO.

ReadWriteMany (RWM)

This mode defines a persistent volume that can be bound in read/write mode by multiple persistent volume claims. In general, file storage and object storage support RWM.

ReadOnlyMany (ROM)

This mode defines a persistent volume that can be bound in read only mode by multiple persistent volume claims.

Reclaim Policy

The reclaim policy tells Kubernetes what to do with a persistent volume when its persistent volume claim has been released.

Delete

This policy deletes the persistent volume and the underlying associated external storage resource, on the external storage system. This is the default policy for volumes that are created dynamically via a storage class.

Retain

This policy keeps the persistent volume in the cluster, as well as the underlying associated external storage resource, on the external storage system. However, it will prevent another persistent volume claim from using the persistent volume.

To reuse the space associated with a retained persistent volume, the persistent volume should be manually deleted, the underlying external storage reformatted and then the persistent volume should be recreated.

Capacity

The capacity, expressed in the persistent volume manifest, can be less than the actual underlying physical storage, but cannot be more.

Persistent Volume Claim (PVC)

Pods do not act directly on persistent volumes, they need something called persistent volume claim, which is an API resource object that is bound to the persistent volume the pod wants to use.A persistent volume claim is similar to a ticket that authorizes a pod to use a certain persistent volume. Once an application has a persistent volume claim, it can mount the respective volume into its pod.

Persistent Volume Claim Manifest

Persistent Volume Claim Manifest

Storage Class (SC)

A storage class is an API resource that allows the definition of a class or tier of storage. Obviously, the type of storage classes that can be defined depends on the types of external storage the Kubernetes cluster has access to. A pod can use a dynamically-provisioned persistent volume from a specific storage class by using a persistent volume claim that references that storage class by name. The persistent volume does not need to be created or declared: the storage class creates the persistent volume dynamically. Once deployed, the storage class watches the API server for new PVC objects that reference its name. When a matching persistent volume claim appears, the storage class dynamically creates the required persistent volume.

The storage class resources are defined in the storage.k8s.io/v1 API group. Each storage class object relates to a single provisioner. StorageClass objects are immutable, they cannot be modified once deployed.

Storage Class Manifest

Storage Class Manifest

Default Storage Class

For the time being, the default storage class is set via annotations. If the cluster has a default storage class, a pod can be deployed using just a persistent volume claim - the storage class does not need to be manually created.

Volume

Secrets may be exposed as files in dedicated volumes mounted in the pod.

Volume Type

• hostPath
• configMap

Dynamic Volume Provisioning

Dynamic volume provisioner https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/

Persistence Operations

Persistence Operations

Organizatorium

• GlusterFS Kubernetes