OpenShift 3.5 Installation

From NovaOrdis Knowledge Base
Jump to navigation Jump to search

External

Internal

Overview

There are two installation methods: quick install, which uses a CLI tool available in the "atomic-openshift-utils" package, which, in turn, uses Ansible in the background, and advanced install. The advanced install assumes familiarity with Ansible. This document covers advance install.

Prerequisites

External DNS Setup

An external DNS server is required.

Procedure to configure a bind server:

Set up a bind Server

The DNS server will need to be capable to support wildcard sub-domains. For the configuration procedure see:

Declare a Wildcard Subdomain

Minimum Hardware Requirements

https://docs.openshift.com/container-platform/3.5/install_config/install/prerequisites.html#hardware

A full RHEL7.3/Master installation requires 121 MB in /boot and 1.7GB in /.

O/S Requirements and Configuration

https://docs.openshift.com/container-platform/3.5/install_config/install/prerequisites.html
https://docs.openshift.com/container-platform/3.5/install_config/install/host_preparation.html

Install RHEL 7.3 in "minimal" installation mode. A possible way to install it is on top of a VirtualBox virtual machine, as described here.

OpenShift requires NetworkManager on all nodes (see https://docs.openshift.com/container-platform/3.5/install_config/install/prerequisites.html#prereq-networkmanager). Using NetworkManager, assign a static IP address to the interface to be used by the OpenShift cluster, as described here: adding a Static Ethernet Connection with NetworkManager.

Turn off sshd client name DNS verification.

Attach the node to the subscription, using subscription manager, as described here: registering a RHEL System with subscription manager then follow the specific subscription steps required by OpenShift: https://docs.openshift.com/container-platform/3.5/install_config/install/host_preparation.html#host-registration

Sequence of steps - the goal of these steps is to configure the following supported repositories on the system: "rhel-7-server-rpms", "rhel-7-server-extras-rpms", "rhel-7-server-ose-3.5-rpms", "rhel-7-fast-datapath-rpms": 

subscription-manager register
subscription-manager list --available --matches '*OpenShift*'
subscription-manager attach --pool=<pool-id> --quantity=1
subscription-manager repos --disable="*"
subscription-manager repos --list-enabled
yum repolist
yum-config-manager --disable <repo_id>
subscription-manager repos --enable="rhel-7-server-rpms" --enable="rhel-7-server-extras-rpms" --enable="rhel-7-server-ose-3.5-rpms" --enable="rhel-7-fast-datapath-rpms"
subscription-manager repos --list-enabled
yum repolist

Install base packages (https://docs.openshift.com/container-platform/3.5/install_config/install/host_preparation.html#installing-base-packages): 

yum install wget git net-tools bind-utils iptables-services bridge-utils bash-completion
yum update -y
yum install atomic-openshift-utils

Prevent accidental upgrades of OpenShiift and Docker, by installing "excluder" packages. The *-excluder packages add entries to the "exclude" directive in the host’s /etc/yum.conf file when installed. Those entries can be removed later when we explicitly want to upgrade OpenShift or Docker. More details in yum Exclusion. 

yum install atomic-openshift-excluder atomic-openshift-docker-excluder

If later we need to upgrade, we must run the following command: 

atomic-openshift-excluder unexclude

Reboot the system to make sure it starts correctly after package installation: 

systemctl reboot

Configure the DNS client to use the DNS server that was installed as part of the procedure. See Manual /etc/resolv.conf Configuration and https://docs.openshift.com/container-platform/3.5/install_config/install/prerequisites.html#prereq-dns

Make sure SELinux is enabled on all hosts. If is not, enable SELinux and make sure SELINUXTYPE is "targeted" in /etc/selinux/config. 

sestatus

Docker Installation

https://docs.openshift.com/container-platform/3.5/install_config/install/host_preparation.html#installing-docker

Install Docker. Docker is technically not required on masters, but it is easier to create a uniform image and only disable docker on masters. The binaries must be installed from the rhel-7-server-ose-3.*-rpms repository and have it running before installing OpenShift.

OpenShift 3.5 requires Docker 1.12. 

yum install docker
docker version

The advanced installation procedure will update /etc/sysconfig/docker on nodes with OpenShift-specific configuration.

Provision storage for the Docker server. The default loopback storage is not appropriate for production, it should be replaced by a thin-pool logical volume. Follow https://docs.openshift.com/container-platform/3.5/install_config/install/host_preparation.html#configuring-docker-storage. Used Option A) "an additional block device". On VirtualBox, provision a new virtual disk of appropriate size and configure it as Docker storage backend.

The procedure consists in executing /usr/bin/docker-storage-setup with the base configuration read from /usr/lib/docker-storage-setup/docker-storage-setup and custom configuration specified in /etc/sysconfig/docker-storage-setup, similarly to: 

STORAGE_DRIVER=devicemapper
DEVS=/dev/sdb
VG=docker_vg
DATA_SIZE=500M
MIN_DATA_SIZE=1M

Under some circumstances, /usr/bin/docker-storage-setup fails with: 

[...]
end of partition 1 has impossible value for cylinders: 65 (should be in 0-64)
sfdisk: I don't like these partitions - nothing changed.
(If you really want this, use the --force option.)

If that happens, follow the manual procedure of provisioning Docker storage on a dedicated block device:

Provision Docker Storage on a Dedicated Block Device

After the script completes successfully, it creates a logical volume with an XFS filesystem mounted on docker root directory /var/lib/docker and the Docker storage configuration file /etc/sysconfig/docker-storage. The thin pool to be used by Docker should be visible in lvs: 

# lvs

  LV          VG        Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  docker-pool docker_vg twi-a-t--- 500.00m             0.00   0.88
  root        main_vg   -wi-ao----   7.00g

Disable docker-storage-setup, is not needed, storage already setup. 

systemctl disable docker-storage-setup

Enable Docker at boot and start it. 

systemctl enable docker
systemctl start docker

Reboot the system and then check Docker Server Runtime.

Generic Docker installation instructions Docker Installation.

Installation User

Create an installation user that can log in remotely from support.openshift35.local, which is the host that will drive installation. Conventionally we name that user "ansible" and it must be able to passwordlessly ssh into itself and all other environment nodes, and do passwordless sudo. The user will be part of the shared image. 

groupadd -g 1200 ansible
useradd -m -g ansible -u 1200 ansible

Configure public/private key authentication and install the public key into its own authorized_keys file.

Allow passwordless sudo to "ansible".

Firewall Configuration

Turn off firewalld and configure the iptables service. 

systemctl stop firewalld
systemctl disable firewalld
systemctl is-enabled firewalld

OpenShift needs iptables running: 

systemctl enable iptables
systemctl start iptables

NFS Client

Install NFS client dependencies.

Miscellaneous

Cloud-Provider Specific Configuration

Special Configuration

NFS Server

The NFS server will serve storage for persistent volumes, metrics and logging.

Build a dedicated storage device to be shared via NFS, but do not export individual volumes; the OpenShift installer will do that automatically. At this stage, make sure the storage is mounted, NFS server binaries are installed, and iptables is correctly configured to allow NFS to serve storage. A summary of these steps is presented in Provisioning and Installing a New Filesystem on a New Logical Volume and NFS Server Installation. All steps of the procedure must be executed, less the export part.

Disable Docker.

Remove the docker_vg volume group and remove the dedicated Docker storage.

Common Image Post-Processing

docker version

lvs

From support.openshift35.local, try: 

ssh -i ~ansible/.ssh/id_rsa ansible@master2

OpenShift Advanced Installation

https://docs.openshift.com/container-platform/latest/install_config/install/advanced_install.html#install-config-install-advanced-install
https://github.com/openshift/openshift-ansible/blob/master/inventory/byo/hosts.ose.example

The Support Node

Execute the installation as "ansible" from the support node. 

cd /usr/share/ansible
chgrp -R ansible /usr/share/ansible
chmod -R g+w /usr/share/ansible

The support node needs at least 1 GB or RAM to run the installation process.

Configure Ansible Inventory File

The default Ansible inventory file is /etc/ansible/hosts. It is used by the Ansible playbook to install the OpenShift environment. The inventory file describes the configuration and the topology of the OpenShift cluster. Start from a template like https://github.com/NovaOrdis/playground/blob/master/openshift/3.5/hosts and customize it to match the environment.

If the target nodes have multiple network interfaces, and the network interface used to cluster OpenShift is NOT associated with the default route, modify the inventory file as follows: 

...
openshift_set_node_ip=true

...
[nodes]
master1.openshift35.local openshift_ip=172.23.0.4 ...

Patch Ansible Logic

External DNS Server Support

OpenShift 3.5 installer did not handle 'openshift_dns_ip' properly, dnsmasq/NewrokManager runtime ignored it. In order to fix it, has to use the following:

Pre-Flight

On the support node, as "ansible": 

ansible all -m ping
ansible nodes -m shell -a "docker version"
ansible nodes -m shell -a "nslookup something.apps.openshift35.external"

Running the Advanced Installation

ansible-playbook -vvv /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml

For verbose installation use -vvv or -vvvv.

To use a different inventory file than /etc/ansible/hosts, run: 

ansible-playbook -vvv -i /custom/path/to/inventory/file /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml

Output of a Successful Run

PLAY RECAP *********************************************************************
infranode1.rdu20.internal  : ok=217  changed=41   unreachable=0    failed=0
infranode2.rdu20.internal  : ok=217  changed=41   unreachable=0    failed=0
loadbalancer1.rdu20.internal : ok=73   changed=10   unreachable=0    failed=0
localhost                  : ok=12   changed=0    unreachable=0    failed=0
master1.rdu20.internal     : ok=1035 changed=267  unreachable=0    failed=0
master2.rdu20.internal     : ok=440  changed=112  unreachable=0    failed=0
master3.rdu20.internal     : ok=440  changed=112  unreachable=0    failed=0
node1.rdu20.internal       : ok=217  changed=41   unreachable=0    failed=0
node2.rdu20.internal       : ok=217  changed=41   unreachable=0    failed=0
support3.rdu20.internal    : ok=77   changed=11   unreachable=0    failed=0

Verifying the Installation

OpenShift Installation Validation

Uninstallation

In case the installation procedure ran into problems, troubleshoot and before re-starting the installation procedure, uninstall: 

ansible-playbook [-i /custom/path/to/inventory/file] /usr/share/ansible/openshift-ansible/playbooks/adhoc/uninstall.yml

OpenShift Environment Post-Install

  • Deploy the Integrated Docker Registry
  • Deploy the HAProxy Router
  • Load Image Streams
  • Load Templates

TODO

DNS Configuration

After setup, the DNS server needs to be configured to resolve a public wildcard DNS entry to the public IP address of the node that executes the default router, by adding an A record, with a low TTL. If the environment has multiple routers, an external load balancer is required. 

*.myapp.example.com. 300 IN  A 1.2.3.4

Also see https://docs.openshift.com/container-platform/3.5/install_config/install/prerequisites.html#wildcard-dns-prereq

  • Verify that /etc/sysconfig/docker on all nodes contains --selinux-enabled and --insecure-registry 172.30.0.0/16
  • Most of the ports will be configured in iptables by the installer, automatically - verify that. OpenShift Ports.

Optional Post-Install Tasks

Stand-alone Registry

Stand-alone Registry - Concepts
https://docs.openshift.com/container-platform/3.5/install_config/install/stand_alone_registry.html#registry-installation-methods
Retrieved from "https://kb.novaordis.com/index.php?title=OpenShift_3.5_Installation&oldid=23185"