Linux cgroups

External

• https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/ch01
• https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/resource_management_guide/chap-introduction_to_control_groups
• https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt
• https://en.wikipedia.org/wiki/Cgroups
• http://man7.org/linux/man-pages/man7/cgroups.7.html
• https://docs.docker.com/config/containers/runmetrics/#control-groups

Internal

• Docker Concepts

Overview

cgroups is a Linux kernel feature that allows allocation of resources (CPU, system memory, network bandwidth, or a combination of these) among user-defined groups of processes running on the system. cgroups not only track groups of processes, but they also expose metrics about CPU, memory and block I/O usage.

cgroups are exposed through a pseudo-filesystem available at /sys/fs/cgroup (older systems expose it at /cgroup). The sub-directories of the cgroup pseudo-filesystem root correspond to different cgroups hierarchies: cpu, freezer, blkio.

This command returns a list of the cgroups that are mounted:

cat /proc/mounts | grep cgroup

The control groups subsystems known to the system are available in /proc/cgroups:

#subsys_name	hierarchy	num_cgroups	enabled
cpuset	6	13	1
cpu	4	89	1
cpuacct	4	89	1
memory	8	89	1
devices	3	83	1
freezer	10	13	1
net_cls	5	13	1
blkio	11	89	1
perf_event	2	13	1
hugetlb	9	13	1
pids	7	13	1
net_prio	5	13	1

cgroups are organized hierarchically, child cgroups inheriting certain attributes from their parent group. Many different hierarchies of cgroups can exist simultaneously on a system. Each hierarchy is attached to one or more subsystem, where a subsystem represents a single resource like CPU time or memory.

To figure out what cgroups a process belongs to, look at /proc/<pid>/cgroup: the cgroup is shown as a path relative to the root of the hierarchy mount point. "/" means the process has not been assigned to a group, while "/lxc/something" means the process is member of a container named "something".

cgroups can be configured via the cgconfig service.

cgroups Subsystems

These subsystems are also known as "controllers":

blkio

Sets limits on input/output access from and to block devices.

cpu

Uses the scheduler to provide cgroup tasks access to the CPU.

cpuacct

Generates automatic reports on CPU resources. Statistics are maintained in "cpuacct.stat", which contains the CPU usage accumulated by the processes of the group, broken down into user and system time. The times are expressed in ticks of 1/100th of a second, also called "user jiffies" There are USER_HZ "jiffies" per second, and on x86 systems, USER_HZ is 100. Historically, this mapped exactly to the number of scheduler "ticks" per second, but higher frequency scheduling and [tickless kernels] have made this number irrelevant.

system

More:

https://docs.docker.com/config/containers/runmetrics/#cpu-metrics-cpuacctstat

cpuset

Assigns individual CPUs and memory nodes to tasks in a cgroup.

memory

Memory metrics are found in the "memory" cgroup. To enable memory control group, add the following kernel command-line parameters:

cgroup_enable=memory swappacount=1

The metrics are available in "memory.stat".

More details:

https://docs.docker.com/config/containers/runmetrics/#metrics-from-cgroups-memory-cpu-block-io

net_cls

Tags network packets with a tag identifier (classid) that allow the Linux traffic controller (tc) to identify packets.

ns

The namespace subsystem.

Operations

The recommended location for cgroup hierarchies:

/sys/fs/cgroup