OpenShift 3.6 Generic Guest Template: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
No edit summary
 
(37 intermediate revisions by the same user not shown)
Line 2: Line 2:


* [[OpenShift_3.6_Installation#Guest_Template_Preparation|OpenShift 3.6 Installation]]
* [[OpenShift_3.6_Installation#Guest_Template_Preparation|OpenShift 3.6 Installation]]
=Overview=
The generic template contains all an OpenShift master or regular node, as well as the support node, need to function. It is built in top of the [[OpenShift_3.6_Basic_Guest_Template#Overview|basic template]].


=Clone the Basic Template=
=Clone the Basic Template=


Use the [[OpenShift_3.6_Basic_Guest_Template|basic template]] as a base image, and clone it, as described here:
Use the [[OpenShift_3.6_Basic_Guest_Template#Overview|basic template]] as a base image, and clone it, as described here:


{{Internal|Linux Virtualization Cloning a KVM Guest Virtual Machine|Cloning a Guest}}
{{Internal|Linux Virtualization Cloning a KVM Guest Virtual Machine|Cloning a Guest}}


The particular sequence follows:
The process will produce a ocp36.generic-template.
 
virsh shutdown ocp36.basic-template
 
virsh dumpxml ocp36.basic-template > /root/environments/ocp36/ocp36.basic-template.xml
cp ocp36.basic-template.xml ocp36.generic-template.xml
 
Adjust the configuration as described [[Linux_Virtualization_Cloning_a_KVM_Guest_Virtual_Machine#Adjust_the_Configuration|here]].
 
Clone the virtual machine image and create the new disk volumes, as described in [[Linux Virtualization Cloning a KVM Guest Virtual Machine|Cloning a Guest]].
 
virsh define /root/environments/ocp36/ocp36.generic-template.xml
 
virsh start --console ocp36.generic-template
 
Then [[Reconfigure Linux VM Guest Image|reconfigure Linux VM Guest Image]].


=Install OpenShift Binaries=
=Install OpenShift Binaries=
Line 88: Line 76:
  docker version
  docker version


==Docker Storage Setup==


<font color=red>Process: https://docs.docker.com/storage/storagedriver/device-mapper-driver/#configure-direct-lvm-mode-for-production</font>


Provision [[OpenShift_Concepts#Docker_Storage_in_OpenShift|storage for the Docker server]]. The [[Docker_Concepts#Loopback_Storage|default loopback storage]] is not appropriate for production, it should be replaced by a [[Linux_Logical_Volume_Management_Concepts#Thinly-Provisioned_Logical_Volumes_.28Thin_Volumes.29|thin-pool logical volume]]. Set up docker storage using Option A) described in the documentation: use an additional block device. In this case, the additional block device is provisioned as a 50GB dedicate volume group and exposed to the guest as the /dev/vdb block device.


/etc/sysconfig/docker-storage-setup:


<pre>
CONTAINER_THINPOOL=docker-pool
STORAGE_DRIVER=devicemapper
DEVS=/dev/vdb
CONTAINER_THINPOOL=container-thinpool
VG=docker_vg
ROOT_SIZE=50G
DATA_SIZE=100%FREE
MIN_DATA_SIZE=2G
#POOL_META_SIZE=16M
CHUNK_SIZE=512K
GROWPART=false
AUTO_EXTEND_POOL=yes
POOL_AUTOEXTEND_THRESHOLD=60
POOL_AUTOEXTEND_PERCENT=20
DEVICE_WAIT_TIMEOUT=60
WIPE_SIGNATURES=true
CONTAINER_ROOT_LV_SIZE=100%FREE
# CONTAINER_ROOT_LV_MOUNT_PATH="/var/lib/containers/container-runtime"
</pre>


{{Warn|Setting DATA_SIZE too small caused nodes not being able to start and OpenShift [[OpenShift Concepts#OutOfDisk|OutOfDisk events]].}}


If this is template reconfiguration, remove previous Docker metadata:


  rm -r /var/lib/docker
  cat /dev/null > /etc/sysconfig/docker-storage


The advanced installation procedure is supposed to update [[/etc/sysconfig/docker]] on nodes with OpenShift-specific configuration. The documentation says that the advanced installation procedure will add an "--insecure-registry" option, but that does not seem to be the case, so we add it manually in /etc/sysconfig/docker:
Execute:


<pre>
/usr/bin/docker-storage-setup
INSECURE_REGISTRY='--insecure-registry 172.30.0.0/16'
</pre>


The subnet value used to configure the insecure registry corresponds to the default value of the [[OpenShift_Concepts#The_Services_Subnet|services subnet]].
INFO: Volume group backing root filesystem could not be determined
INFO: Device node /dev/vdb1 exists.
  Physical volume "/dev/vdb1" successfully created.
  Volume group "docker_vg" successfully created
  Using default stripesize 64.00 KiB.
  Rounding up size to full physical extent 52.00 MiB
  Thin pool volume with chunk size 512.00 KiB can address at most 126.50 TiB of data.
  Logical volume "container-thinpool" created.
  Logical volume docker_vg/container-thinpool changed.


Provision [[OpenShift_Concepts#Docker_Storage_in_OpenShift|storage for the Docker server]]. The [[Docker_Concepts#Loopback_Storage|default loopback storage]] is not appropriate for production, it should be replaced by a [[Linux_Logical_Volume_Management_Concepts#Thinly-Provisioned_Logical_Volumes_.28Thin_Volumes.29|thin-pool logical volume]]. Follow https://docs.openshift.com/container-platform/3.5/install_config/install/host_preparation.html#configuring-docker-storage. Used Option A) "an additional block device". On VirtualBox or KVM, provision a new virtual disk and install it. At this stage, the size it is not important, as it will replaced with the actual storage when the nodes are built. Use 100 MB for the template.
lvs
* [[VirtualBox_Operations#Creating_and_Installing_a_new_Virtual_Disk|Creating and installing a new virtual disk on VirtualBox]]
  LV                VG        Attr      LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
* [[Virsh_vol-create-as|Creating a new logical volume]] on KVM, followed by attachment to the template. When creating the logical volume, name it "template-docker.storage", following the [[Linux_Virtualization_Naming_Conventions#Storage_Volume_Naming_Convention|storage volume naming conventions]].
  container-thinpool docker_vg twi-a-t--- 49.89g            0.00  0.11


KVM example (the template VM must be shut down prior to attaching the storage):
After setup, disable docker-storage-setup at startup, is not needed, storage already setup.


   virsh vol-create-as --pool main-storage-pool --name template-docker.raw --capacity 1024M
   systemctl disable docker-storage-setup
  virsh vol-list --pool main-storage-pool
   systemctl is-enabled docker-storage-setup
   virsh attach-disk template /main-storage-pool/template-docker.raw vdb --config


Restart the template VM, the new storage should be available as /dev/vdb.
==Docker and OpenShift Installation==


Then execute /usr/bin/docker-storage-setup with the base configuration read from  [[/usr/lib/docker-storage-setup/docker-storage-setup]] and custom configuration specified in /etc/sysconfig/docker-storage-setup, similarly to:
The OpenShift advanced installation procedure is supposed to update [[Docker_Server_Configuration#RedHat.2FCentos|/etc/sysconfig/docker]] on nodes with OpenShift-specific configuration. The documentation says that the advanced installation procedure will add an "--insecure-registry" option, but that does not seem to be the case, so based on previous installation experience, we learned to add it manually in /etc/sysconfig/docker:


  STORAGE_DRIVER=devicemapper
  OPTIONS=' --selinux-enabled --log-driver=journald <font color=teal>'''--insecure-registry 172.30.0.0/16'''</font>'
  DEVS=/dev/vdb
VG=docker_vg
# set to a little bit less than maximum amount of space available
DATA_SIZE=<b>1023M</b>
MIN_DATA_SIZE=1M


{{Warn|Setting DATA_SIZE too small caused nodes not being able to start and OpenShift [[OpenShift Concepts#OutOfDisk|OutOfDisk events]].}}
The subnet value used to configure the insecure registry corresponds to the default value of the [[OpenShift_Concepts#The_Services_Subnet|services subnet]].


Execute:
==Test Start Docker==


  /usr/bin/docker-storage-setup
Enable docker to start at boot:


Under some circumstances, /usr/bin/docker-storage-setup fails with:
systemctl enable docker
systemctl start docker


<pre>
Reboot the system and then check [[Docker Server Runtime]].
[...]
end of partition 1 has impossible value for cylinders: 65 (should be in 0-64)
sfdisk: I don't like these partitions - nothing changed.
(If you really want this, use the --force option.)
</pre>
 
If this happens, use the patched docker-storage-setup available here: https://github.com/NovaOrdis/playground/blob/master/openshift/3.5/patches/node/usr/bin/docker-storage-setup


Before running it, remove any logical volume, volume group and physical volume that may have been created and leave an empty /dev/vdb1 partition. Then run
=Optimizing Docker Storage=


  /usr/bin/docker-storage-setup --force
{{External|https://docs.openshift.com/container-platform/3.6/scaling_performance/optimizing_storage.html#optimizing-storage}}


After the script completes successfully, it creates a logical volume with an XFS filesystem mounted on docker root directory /var/lib/docker and the Docker storage configuration file /etc/sysconfig/docker-storage. The thin pool to be used by Docker should be visible in [[lvs]]:
==Prevent Docker to Start at Boot==


<pre>
We do not want the template to start docker at boot, because in some cases we may want to modify the storage depending on the node's particularities.
# lvs


  LV          VG        Attr      LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
Also we do not want to automatically start, because we may want to configure new storage manually.
  docker-pool docker_vg twi-a-t--- 500.00m            0.00  0.88
  root        main_vg  -wi-ao----  7.00g
</pre>


<span id='Ny62gV'></span>Alternatively, you can follow the manual procedure of provisioning Docker storage on a dedicated block device:
  systemctl disable docker
 
  systemctl is-enabled docker
{{Internal|Provision Docker Storage on a Dedicated Block Device|Provision Docker Storage on a Dedicated Block Device}}
 
Disable docker-storage-setup, is not needed, storage already setup.


   systemctl disable docker-storage-setup
   systemctl disable docker-storage-setup
   systemctl is-enabled docker-storage-setup
   systemctl is-enabled docker-storage-setup
Enable Docker at boot and start it.
  systemctl enable docker
  systemctl start docker
 
  systemctl status docker
Reboot the system and then check [[Docker Server Runtime]].
<font color=red>TODO: parse and NOKB this: https://docs.openshift.com/container-platform/3.5/scaling_performance/optimizing_storage.html#optimizing-storage</font>
Generic Docker installation instructions [[Docker_Installation#Prerequisites|Docker Installation]].

Latest revision as of 20:37, 20 April 2018

Internal

Overview

The generic template contains all an OpenShift master or regular node, as well as the support node, need to function. It is built in top of the basic template.

Clone the Basic Template

Use the basic template as a base image, and clone it, as described here:

Cloning a Guest

The process will produce a ocp36.generic-template.

Install OpenShift Binaries

Attach the Node to the OpenShift Subscription

https://docs.openshift.com/container-platform/3.6/install_config/install/host_preparation.html#host-registration

More details about subscription management:

RedHat Subscription Manager

The node, being based on the basic template, should have been already registered with the subscription manager: 

subscription-manager status

subscription-manager identity

subscription-manager list --available --matches '*OpenShift*'

Identify the OpenShift pool ID: 

subscription-manager attach --pool=<pool-id>

Manage Repositories and Install Utilities

Once attached to the pool, all repositories must be first disabled, and then only select repositories are re-enabled: 

subscription-manager repos --disable="*"

subscription-manager repos --list-enabled
yum repolist

If there are any enabled repositories, disable them with: 

yum-config-manager --disable <repo_id>

Enable only the repositories required by OpenShift Container Platform 3.6: 

subscription-manager repos --enable="rhel-7-server-rpms" --enable="rhel-7-server-extras-rpms" --enable="rhel-7-server-ose-3.6-rpms" --enable="rhel-7-fast-datapath-rpms"

subscription-manager repos --list-enabled
yum repolist

yum update -y

Install atomic-openshift-utils, which contains OpenShift installer utilities and other tools required by the installation process: 

yum install atomic-openshift-utils

At this point, no OpenShift binaries, except installation utilities, are installed. The advanced installer knows how to override this and it will install the binaries as expected, without any further intervention.

Install Docker

https://docs.openshift.com/container-platform/3.6/install_config/install/host_preparation.html#installing-docker

Install Docker on the generic template. On the support host, it will simply not be activated. Docker is also required on masters too, but the image storage allocated to it can be made much smaller than on the nodes and infranodes. The binaries must be installed from the rhel-7-server-ose-3.*-rpms repository and have it running before installing OpenShift.

OpenShift 3.6 requires Docker 1.12. 

yum install docker
docker version

Docker Storage Setup

Process: https://docs.docker.com/storage/storagedriver/device-mapper-driver/#configure-direct-lvm-mode-for-production

Provision storage for the Docker server. The default loopback storage is not appropriate for production, it should be replaced by a thin-pool logical volume. Set up docker storage using Option A) described in the documentation: use an additional block device. In this case, the additional block device is provisioned as a 50GB dedicate volume group and exposed to the guest as the /dev/vdb block device.

/etc/sysconfig/docker-storage-setup: 

CONTAINER_THINPOOL=docker-pool
STORAGE_DRIVER=devicemapper
DEVS=/dev/vdb
CONTAINER_THINPOOL=container-thinpool
VG=docker_vg
ROOT_SIZE=50G
DATA_SIZE=100%FREE
MIN_DATA_SIZE=2G
#POOL_META_SIZE=16M
CHUNK_SIZE=512K
GROWPART=false
AUTO_EXTEND_POOL=yes
POOL_AUTOEXTEND_THRESHOLD=60
POOL_AUTOEXTEND_PERCENT=20
DEVICE_WAIT_TIMEOUT=60
WIPE_SIGNATURES=true
CONTAINER_ROOT_LV_SIZE=100%FREE
# CONTAINER_ROOT_LV_MOUNT_PATH="/var/lib/containers/container-runtime"

Setting DATA_SIZE too small caused nodes not being able to start and OpenShift OutOfDisk events.

If this is template reconfiguration, remove previous Docker metadata: 

 rm -r /var/lib/docker
 cat /dev/null > /etc/sysconfig/docker-storage

Execute: 

/usr/bin/docker-storage-setup

INFO: Volume group backing root filesystem could not be determined
INFO: Device node /dev/vdb1 exists.
 Physical volume "/dev/vdb1" successfully created.
 Volume group "docker_vg" successfully created
 Using default stripesize 64.00 KiB.
 Rounding up size to full physical extent 52.00 MiB
 Thin pool volume with chunk size 512.00 KiB can address at most 126.50 TiB of data.
 Logical volume "container-thinpool" created.
 Logical volume docker_vg/container-thinpool changed.

lvs
 LV                 VG        Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
 container-thinpool docker_vg twi-a-t--- 49.89g             0.00   0.11

After setup, disable docker-storage-setup at startup, is not needed, storage already setup. 

 systemctl disable docker-storage-setup
 systemctl is-enabled docker-storage-setup

Docker and OpenShift Installation

The OpenShift advanced installation procedure is supposed to update /etc/sysconfig/docker on nodes with OpenShift-specific configuration. The documentation says that the advanced installation procedure will add an "--insecure-registry" option, but that does not seem to be the case, so based on previous installation experience, we learned to add it manually in /etc/sysconfig/docker: 

OPTIONS=' --selinux-enabled  --log-driver=journald --insecure-registry 172.30.0.0/16'

The subnet value used to configure the insecure registry corresponds to the default value of the services subnet.

Test Start Docker

Enable docker to start at boot: 

systemctl enable docker
systemctl start docker

Reboot the system and then check Docker Server Runtime.

Optimizing Docker Storage

https://docs.openshift.com/container-platform/3.6/scaling_performance/optimizing_storage.html#optimizing-storage

Prevent Docker to Start at Boot

We do not want the template to start docker at boot, because in some cases we may want to modify the storage depending on the node's particularities.

Also we do not want to automatically start, because we may want to configure new storage manually. 

 systemctl disable docker
 systemctl is-enabled docker

 systemctl disable docker-storage-setup
 systemctl is-enabled docker-storage-setup
Retrieved from "https://kb.novaordis.com/index.php?title=OpenShift_3.6_Generic_Guest_Template&oldid=41114"