Docker device-mapper Storage Backend: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
 
(54 intermediate revisions by the same user not shown)
Line 1: Line 1:
=External=
=External=


* https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/
* https://docs.docker.com/storage/storagedriver/
* https://docs.docker.com/storage/storagedriver/device-mapper-driver/
* https://docs.docker.com/storage/storagedriver/device-mapper-driver/
* https://docs.docker.com/engine/reference/commandline/dockerd/#options-per-storage-driver


=Internal=
=Internal=


* [[Docker_Concepts#Storage_Backend|Docker Concepts - Storage Backend]]
* [[Docker_Storage_Concepts#devicemapper_Storage_Driver|Docker Storage Concepts - Storage Backend]]
* [[Device Mapper Concepts#Overview|Device Mapper]]


=Overview=
=Overview=


The Docker device-mapper Storage Backend is built upon the [[Device Mapper Concepts#Overview|device mapper framework]]. The devicemapper driver expects access to a dedicated block device - meaning that only the Docker server process should have exclusive access to it. The driver will operate at block level, rather than file level. Because of that, this type of storage usually performs better than using a filesystem accessed via the operating system. Once configured with a devicemapper driver, the docker server cannot be reconfigured with another driver unless the containers already stored within it are discarded (or saved with [[docker save]] and re-imported).


Device-mapper driver metadata is stored in /var/lib/docker/devicemapper/metadata. Docker state can be reset by stopping docker and by removing /var/lib/docker/devicemapper/metadata directory. This will destroy existing docker images and containers and all the docker metadata.
==devicemapper Mode==
 
The devicemapper driver can be configured to run in one of the [[#loop-lvm|loop-lvm]] or [[#direct-lvm|direct-lvm]] modes.
 
<span id='loop-lvm'></span>'''loop-lvm''' mode uses a [[Linux_7_Storage_Concepts#Loopback_Device|loopback device]] for storage, but this is a configuration that is only appropriate for testing. Loopback devices are slow, resource-intensive, and they can introduce race conditions. The only upside is that the setup is easier, so they may be used for testing. For the procedure of configuring loop-lvm mode, see https://docs.docker.com/storage/storagedriver/device-mapper-driver/#configure-loop-lvm-mode-for-testing.
 
<span id='direct-lvm'></span>'''direct-lvm''' uses an actual uses block device to create the [[#Thin_Pool|thin pool]]. This mode is faster than using loopback devices, uses system resources more efficiently, and block devices can grow as needed.
 
==Thin Pool==
 
{{Internal|Virtualization_Concepts#Thin_Provisioning|Thin Provisioning}}


=Info=
=Info=
Line 21: Line 34:
   Pool Name: docker_vg-container--thinpool
   Pool Name: docker_vg-container--thinpool
   Pool Blocksize: 524.3 kB
   Pool Blocksize: 524.3 kB
   [[#Base_Device_Size|Base Device Size]]: 10.74 GB
   <span id='docker_info_based_device_size'></span>[[#Base_Device_Size|Base Device Size]]: 10.74 GB
   Backing Filesystem: xfs
   Backing Filesystem: xfs
   Data file:
   Data file:
Line 39: Line 52:
   ...
   ...


==Base Device Size==
=Installation=
 
==Devicemapper direct-lvm Installation==
 
{{External|https://docs.docker.com/storage/storagedriver/device-mapper-driver/#configure-direct-lvm-mode-for-production}}
 
This procedure is part of the [[Docker_Linux_Installation#devicemapper_Storage_Setup|Docker installation procedure]].
 
Production systems using the devicemapper storage driver must use direct-lvm mode, which uses block devices to create a thin pool.
 
If this is an initial Docker installation, and just one block device will be used for storage, Docker can be configured to initialize and manage the single block device, without additional manual configuration.
 
The block device to be used by the storage backend must be configured externally. Examples:
* [[Linux Virtualization Adding Storage to Guests|Add block storage to libvirt guests]]
* [[VMware_Fusion_Operations#Add_Block_Storage_to_Guest|Add block storage to VMware Fusion guests]]
 
After the device is configured and attached to the guest, add the following to [[daemon.json]]:
 
<syntaxhighlight lang='json'>
{
  "storage-driver": "devicemapper",
  "storage-opts": [
    "dm.directlvm_device=/dev/vdb",
    "dm.thinp_percent=95",
    "dm.thinp_metapercent=1",
    "dm.thinp_autoextend_threshold=80",
    "dm.thinp_autoextend_percent=20",
    "dm.directlvm_device_force=false"
  ]
}
</syntaxhighlight>
 
Then restart docker:
 
systemctl stop docker
systemctl start docker
 
The initialization results can be verified with [[docker info]]. The result should be similar with the output shown [[#Info|above]].
 
The newly created thin pool device can be inspected with pvs/vgs/lvs:
 
<syntaxhighlight lang='bash'>
pvs
  PV        VG    Fmt  Attr PSize  PFree
  /dev/vdb  docker lvm2 a--  46.56g <1.40g
 
vgs
  VG    #PV #LV #SN Attr  VSize  VFree
  docker  1  1  0 wz--n- 46.56g <1.40g
 
lvs
  LV      VG    Attr      LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  thinpool docker twi-a-t--- 44.23g            0.04  0.02
</syntaxhighlight>
 
=Base Device Size=


{{External|https://www.projectatomic.io/blog/2016/03/daemon_option_basedevicesize/}}
{{External|https://www.projectatomic.io/blog/2016/03/daemon_option_basedevicesize/}}
Line 45: Line 113:
Base size is the maximum size that a image, and containers based on it, can grow to. Essentially, the size of the root filesystem of the container.
Base size is the maximum size that a image, and containers based on it, can grow to. Essentially, the size of the root filesystem of the container.


Reported with [[docker info]] and also with [[docker inspect]] &#123;{.GraphDriver.Data.DeviceSize}}:
==View Base Device Size==
 
The docker engine-wide base image size is reported with [[#docker_info_based_device_size|docker info]] (see above). The container-specific base image size is reported with [[docker inspect]] &#123;{.GraphDriver.Data.DeviceSize}}:


  ...
  ...
Line 64: Line 134:
  /dev/mapper/docker-253:3-12968682-....  '''10G'''  658M  9.4G  7% /
  /dev/mapper/docker-253:3-12968682-....  '''10G'''  658M  9.4G  7% /


Set with:
==Modify Base Device Size==
 
The docker engine instance wide value set with:


  --storage-opt dm.basesize=20G
  --storage-opt dm.basesize=20G


Apparently it can only be expanded, not shrunk - needs experimentation.
on [[Docker_Server_Configuration#Overview|docker engine's command line]].
 
For a specific container, it can be set with:
 
[[Docker_run#--storage-opt_list|docker run --storage-opt size=60]]
 
Apparently it can only be expanded, not shrunk - needs experimentation. Existing images will retain the old base device size, unless they are rmi/pull.
 
=TODO=
 
Device-mapper driver metadata is stored in /var/lib/docker/devicemapper/metadata. Docker state can be reset by stopping docker and by removing /var/lib/docker/devicemapper/metadata directory. This will destroy existing docker images and containers and all the docker metadata.
 
The thin pool.
 
Base device.
 
/var/lib/docker/devicemapper/metadata/base
{"device_id":1,"size":10737418240,"transaction_id":1,"initialized":true,"deleted":false}
 
What is metadata in this context? The data that is maintained in /var/lib/docker/devicemapper/metadata?
 
The concept of a snapshot of an image.
 
The concept of a devicemapper snapshot.
 
<font color=red>There seems to be a base device for each running container.</font>


Existing images will retain the old base device size, unless they are rmi/pull.
How are images and container stored.


https://github.com/moby/moby/pull/19367 seems to suggest there's a per-container option to set the base size of the image, on a per-container instance basis.
A container in this context is a writable layer.


Layer.


See [[Docker_run#--storage-opt_list|docker run --storage-opt]].
Thin pool block and thin pool block size.

Latest revision as of 18:50, 15 December 2018

External

Internal

Overview

The Docker device-mapper Storage Backend is built upon the device mapper framework. The devicemapper driver expects access to a dedicated block device - meaning that only the Docker server process should have exclusive access to it. The driver will operate at block level, rather than file level. Because of that, this type of storage usually performs better than using a filesystem accessed via the operating system. Once configured with a devicemapper driver, the docker server cannot be reconfigured with another driver unless the containers already stored within it are discarded (or saved with docker save and re-imported).

devicemapper Mode

The devicemapper driver can be configured to run in one of the loop-lvm or direct-lvm modes.

loop-lvm mode uses a loopback device for storage, but this is a configuration that is only appropriate for testing. Loopback devices are slow, resource-intensive, and they can introduce race conditions. The only upside is that the setup is easier, so they may be used for testing. For the procedure of configuring loop-lvm mode, see https://docs.docker.com/storage/storagedriver/device-mapper-driver/#configure-loop-lvm-mode-for-testing.

direct-lvm uses an actual uses block device to create the thin pool. This mode is faster than using loopback devices, uses system resources more efficiently, and block devices can grow as needed.

Thin Pool

Thin Provisioning

Info

At runtime, general statistics on the devicemapper storage driver can be obtained with:

docker info
Storage Driver: devicemapper
 Pool Name: docker_vg-container--thinpool
 Pool Blocksize: 524.3 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file:
 Metadata file:
 Data Space Used: 16.81 GB
 Data Space Total: 53.57 GB
 Data Space Available: 36.76 GB
 Metadata Space Used: 3.338 MB
 Metadata Space Total: 54.53 MB
 Metadata Space Available: 51.19 MB
 Thin Pool Minimum Free Space: 5.357 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.140-RHEL7 (2017-05-03)
 ...

Installation

Devicemapper direct-lvm Installation

https://docs.docker.com/storage/storagedriver/device-mapper-driver/#configure-direct-lvm-mode-for-production

This procedure is part of the Docker installation procedure.

Production systems using the devicemapper storage driver must use direct-lvm mode, which uses block devices to create a thin pool.

If this is an initial Docker installation, and just one block device will be used for storage, Docker can be configured to initialize and manage the single block device, without additional manual configuration.

The block device to be used by the storage backend must be configured externally. Examples:

After the device is configured and attached to the guest, add the following to daemon.json:

{
  "storage-driver": "devicemapper",
  "storage-opts": [
    "dm.directlvm_device=/dev/vdb",
    "dm.thinp_percent=95",
    "dm.thinp_metapercent=1",
    "dm.thinp_autoextend_threshold=80",
    "dm.thinp_autoextend_percent=20",
    "dm.directlvm_device_force=false"
  ]
}

Then restart docker:

systemctl stop docker
systemctl start docker

The initialization results can be verified with docker info. The result should be similar with the output shown above.

The newly created thin pool device can be inspected with pvs/vgs/lvs:

pvs
  PV         VG     Fmt  Attr PSize  PFree
  /dev/vdb   docker lvm2 a--  46.56g <1.40g

vgs
  VG     #PV #LV #SN Attr   VSize  VFree
  docker   1   1   0 wz--n- 46.56g <1.40g

lvs
  LV       VG     Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  thinpool docker twi-a-t--- 44.23g             0.04   0.02

Base Device Size

https://www.projectatomic.io/blog/2016/03/daemon_option_basedevicesize/

Base size is the maximum size that a image, and containers based on it, can grow to. Essentially, the size of the root filesystem of the container.

View Base Device Size

The docker engine-wide base image size is reported with docker info (see above). The container-specific base image size is reported with docker inspect {{.GraphDriver.Data.DeviceSize}}:

...
"GraphDriver": {
   "Name": "devicemapper",
   "Data": {
       "DeviceId": "3315",
       "DeviceName": "docker-253:3-12968682-7827556d545926857d669f5f9bb7fd69083e7e98636ff3f060d352223f4918ab",
       "DeviceSize": "10737418240"
   }
}
...

At runtime, the same amount can be viewed by attaching to the container and execute:

df -h
Filesystem                               Size  Used Avail Use% Mounted on
/dev/mapper/docker-253:3-12968682-....   10G  658M  9.4G   7% /

Modify Base Device Size

The docker engine instance wide value set with:

--storage-opt dm.basesize=20G

on docker engine's command line.

For a specific container, it can be set with:

docker run --storage-opt size=60

Apparently it can only be expanded, not shrunk - needs experimentation. Existing images will retain the old base device size, unless they are rmi/pull.

TODO

Device-mapper driver metadata is stored in /var/lib/docker/devicemapper/metadata. Docker state can be reset by stopping docker and by removing /var/lib/docker/devicemapper/metadata directory. This will destroy existing docker images and containers and all the docker metadata.

The thin pool.

Base device.

/var/lib/docker/devicemapper/metadata/base
{"device_id":1,"size":10737418240,"transaction_id":1,"initialized":true,"deleted":false}

What is metadata in this context? The data that is maintained in /var/lib/docker/devicemapper/metadata?

The concept of a snapshot of an image.

The concept of a devicemapper snapshot.

There seems to be a base device for each running container.

How are images and container stored.

A container in this context is a writable layer.

Layer.

Thin pool block and thin pool block size.