Docker device-mapper Storage Backend

External

• https://docs.docker.com/storage/storagedriver/
• https://docs.docker.com/storage/storagedriver/device-mapper-driver/
• https://docs.docker.com/engine/reference/commandline/dockerd/#options-per-storage-driver

Internal

• Docker Storage Concepts - Storage Backend
• Device Mapper

Overview

The Docker device-mapper Storage Backend is built upon the device mapper framework. The devicemapper driver expects access to a dedicated block device - meaning that only the Docker server process should have exclusive access to it. The driver will operate at block level, rather than file level. Because of that, this type of storage usually performs better than using a filesystem accessed via the operating system. Once configured with a devicemapper driver, the docker server cannot be reconfigured with another driver unless the containers already stored within it are discarded (or saved with docker save and re-imported).

devicemapper Mode

The devicemapper driver can be configured to run in one of the loop-lvm or direct-lvm modes.

loop-lvm mode uses a loopback device for storage, but this is a configuration that is only appropriate for testing. Loopback devices are slow, resource-intensive, and they can introduce race conditions. The only upside is that the setup is easier, so they may be used for testing. For the procedure of configuring loop-lvm mode, see https://docs.docker.com/storage/storagedriver/device-mapper-driver/#configure-loop-lvm-mode-for-testing.

direct-lvm uses an actual uses block device to create the thin pool. This mode is faster than using loopback devices, uses system resources more efficiently, and block devices can grow as needed.

Thin Pool

Thin Provisioning

Info

At runtime, general statistics on the devicemapper storage driver can be obtained with:

docker info
Storage Driver: devicemapper
 Pool Name: docker_vg-container--thinpool
 Pool Blocksize: 524.3 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file:
 Metadata file:
 Data Space Used: 16.81 GB
 Data Space Total: 53.57 GB
 Data Space Available: 36.76 GB
 Metadata Space Used: 3.338 MB
 Metadata Space Total: 54.53 MB
 Metadata Space Available: 51.19 MB
 Thin Pool Minimum Free Space: 5.357 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.140-RHEL7 (2017-05-03)
 ...

Installation

Devicemapper direct-lvm Installation

https://docs.docker.com/storage/storagedriver/device-mapper-driver/#configure-direct-lvm-mode-for-production

This procedure is part of the Docker installation procedure.

Production systems using the devicemapper storage driver must use direct-lvm mode, which uses block devices to create a thin pool.

If this is an initial Docker installation, and just one block device will be used for storage, Docker can be configured to initialize and manage the single block device, without additional manual configuration.

The block device to be used by the storage backend must be configured externally. Examples:

• Add block storage to libvirt guests
• Add block storage to VMware Fusion guests

After the device is configured and attached to the guest, add the following to daemon.json:

{
  "storage-driver": "devicemapper",
  "storage-opts": [
    "dm.directlvm_device=/dev/vdb",
    "dm.thinp_percent=95",
    "dm.thinp_metapercent=1",
    "dm.thinp_autoextend_threshold=80",
    "dm.thinp_autoextend_percent=20",
    "dm.directlvm_device_force=false"
  ]
}

Then restart docker:

systemctl stop docker
systemctl start docker

The initialization results can be verified with docker info. The result should be similar with the output shown above.

The newly created thin pool device can be inspected with pvs/vgs/lvs:

pvs
  PV         VG     Fmt  Attr PSize  PFree
  /dev/vdb   docker lvm2 a--  46.56g <1.40g

vgs
  VG     #PV #LV #SN Attr   VSize  VFree
  docker   1   1   0 wz--n- 46.56g <1.40g

lvs
  LV       VG     Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  thinpool docker twi-a-t--- 44.23g             0.04   0.02

Base Device Size

https://www.projectatomic.io/blog/2016/03/daemon_option_basedevicesize/

Base size is the maximum size that a image, and containers based on it, can grow to. Essentially, the size of the root filesystem of the container.

View Base Device Size

The docker engine-wide base image size is reported with docker info (see above). The container-specific base image size is reported with docker inspect {{.GraphDriver.Data.DeviceSize}}:

...
"GraphDriver": {
   "Name": "devicemapper",
   "Data": {
       "DeviceId": "3315",
       "DeviceName": "docker-253:3-12968682-7827556d545926857d669f5f9bb7fd69083e7e98636ff3f060d352223f4918ab",
       "DeviceSize": "10737418240"
   }
}
...

At runtime, the same amount can be viewed by attaching to the container and execute:

df -h
Filesystem                               Size  Used Avail Use% Mounted on
/dev/mapper/docker-253:3-12968682-....   10G  658M  9.4G   7% /

Modify Base Device Size

The docker engine instance wide value set with:

--storage-opt dm.basesize=20G

on docker engine's command line.

For a specific container, it can be set with:

docker run --storage-opt size=60

Apparently it can only be expanded, not shrunk - needs experimentation. Existing images will retain the old base device size, unless they are rmi/pull.

TODO

Device-mapper driver metadata is stored in /var/lib/docker/devicemapper/metadata. Docker state can be reset by stopping docker and by removing /var/lib/docker/devicemapper/metadata directory. This will destroy existing docker images and containers and all the docker metadata.

The thin pool.

Base device.

/var/lib/docker/devicemapper/metadata/base
{"device_id":1,"size":10737418240,"transaction_id":1,"initialized":true,"deleted":false}

What is metadata in this context? The data that is maintained in /var/lib/docker/devicemapper/metadata?

The concept of a snapshot of an image.

The concept of a devicemapper snapshot.

There seems to be a base device for each running container.

How are images and container stored.

A container in this context is a writable layer.

Layer.

Thin pool block and thin pool block size.