OpenShift CI/CD Concepts
External
- https://docs.openshift.com/container-platform/latest/using_images/other_images/jenkins.html
- https://blog.openshift.com/cicd-with-openshift/, youtu.be demos: 65BnTLcDAJI, wSFyg6Etwx8
Internal
Overview
OpenShift provides a certified Jenkins container for building Continuous Delivery pipelines, which can be deployed as a pod in OpenShift. OpenShift comes with standard Jenkins templates, which are recommended to be used when implementing CI/CD pipelines.
Once deployed, Jenkins spins up two pods: the main "jenkins" pod which contains the functionality that drives the build logic, and a "jenkins-jnlp" pod that is an auxiliary service used for master/slave communication. Why doesn't the CI/CD project have a jenkins-jnlp pod? The service exists, maybe the pod was not needed yet? There is no container template for jenkins-jnlp. The template also creates a "jenkins" service account used by the Jenkins pod to authenticate. The service account is automatically bound to the "edit" role in the target project.
The Jenkins pod can be collocated with the application it is intended to build, within the same project, or it can inhabit its own dedicated project and can provide CI/CD services to other projects, which share the common instance. The Collocated Jenkins section addresses aspects that are specific to a Jenkins instance that lives in the same OpenShift project with the application it services. The Shared Jenkins section describes the particularities of the situation when a single Jenkins instance is shared by multiple projects.
The Jenkins instance can be left with a default build environment, which is completely oblivious of OpenShift, but in this case it will attempt to run a build inside de pod and it will fail due to lack of Maven or other utilities. The alternative is to use a Kubernetes plugin build environment, which dynamically provisions Jenkins slaves on OpenShift nodes. More details are available in the Kubernetes Plugin section.
Jenkins
Jenkins is used by OpenShift to provide the execution engine behind OpenShift pipeline builds.
Jenkins instance(s) is/are deployed as a pod. The Jenkins pod can be deployed within the project that needs CI/CD services, or it can be deployed in a shared project, accessed by all projects that need CI/CD services. The system-wide Jenkins instance must be declared in master-config.yaml.
OpenShift Jenkins Docker Image
The Jenkins image intended to use with OpenShift is developed under this repository https://github.com/openshift/jenkins. The repository contains the Dockerfiles that can be used to build the image. The images are pushed to DockerHub as openshift/jenkins-2-centos7, openshift/jenkins-slave-base-centos7, openshift/jenkins-slave-maven-centos7, and openshift/jenkins-slave-nodejs-centos7 and OpenShift Container Platform integrates them with the enterprise product.
Jenkins Heap Sizing
The heap sizing is calculated by the container's CMD script (/usr/libexec/s2i/run) based on the OpenShift memory limit set in the deployment configuration, which propagates to the container as /sys/fs/cgroup/memory/memory.limit_in_bytes, and CONTAINER_HEAP_PERCENT.
If the JVM_ARCH environment variable is set at installation from template, the memory calculated as mentioned above. If not, and the memory of the container is less than 4GB, meaning max heap of 2GB, 32bit is forced to save space.
The template comes with a small metaspace, which can be increased with "JAVA_GC_OPTS="-XX:+UseParallelGC -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MaxMetaspaceSize=384m"
OpenShift Jenkins Templates
OpenShift comes with two standard Jenkins templates ("jerkins-ephemeral" and "jenkins-persistent"), both available in the "openshift" namespace:
oc -n openshift get templates | grep jenkins jenkins-ephemeral Jenkins service, without persistent storage.... jenkins-persistent Jenkins service, with persistent storage....
They can be used to deploy an "ephemeral" or a "persistent" Jenkins. The "persistent" Jenkins will save its state between the pod restarts, and needs a persistent volume with sufficient storage space to be available at the time of the installation.
Build Environment
OpenShift Jenkins image comes pre-preconfigured to allow access to a Kubernetes CLI (kubectl) build environment, which uses the Kubernetes plugin to provision Kubernetes-based Jenkins agents.
For more details about Jenkins build environment, see:
Pipeline
An OpenShift Jenkins pipeline is a Jenkins Pipeline .... TODO. OpenShift Jenkins pipeline are an essential component of the OpenShift pipeline builds.
TODO
Pipelines can be defined in-line in the Jenkins job, or in the source code repository, in a file named "Jenkinsfile". The pipelines are written in Groovy. Jenkins drives the pipeline, but it uses OpenShift to perform the build and the deployment, by invoking OpenShift API via "openshiftBuild", "openshiftVerifyBuild", "openshiftDeploy", "openshiftVerifyDeployment", "openshiftVerifyService". Used by the pipeline build.
The pipeline definition starts with a node() statement.
node('maven') {
...
}
defines a pipeline that runs inside of a Maven pod.
node {
...
}
defines a pipeline that executes in a Jenkins pod.
Possible node values: "maven", "nodejs", "custom".
A pipeline consists of stages, which have graphical representation in Jenkins or OpenShift UI. A stage can be used to group tasks, and they should be scoped with { and }.
Pipeline Definition
node {
stage ("Build") {
...
openshiftBuild
apiURL: 'https://openshift.default.svc.cluster.local',
authToken: ,
bldCfg: 'hello-nodejs',
buildName: ,
checkForTriggeredDeployments: 'false',
commitID: ,
namespace: ,
showBuildLogs: 'false',
verbose: 'false',
waitTime:
openshiftVerifyBuild
apiURL: 'https://openshift.default.svc.cluster.local',
authToken: ,
bldCfg: 'hello-nodejs',
checkForTriggeredDeployments:
'false', namespace: ,
verbose: 'false'
openshiftTag
...
}
stage ("Deploy") {
...
openshiftDeploy
apiURL: 'https://openshift.default.svc.cluster.local',
authToken: ,
depCfg: 'hello-nodejs',
namespace: ,
verbose: 'false',
waitTime:
openshiftVerifyDeployment
apiURL: 'https://openshift.default.svc.cluster.local',
authToken: ,
depCfg: 'hello-nodejs',
namespace: ,
replicaCount: '1',
verbose: 'false',
verifyReplicaCount: 'false',
waitTime:
}
stage ("Verify") {
...
openshiftVerifyService
apiURL: 'https://openshift.default.svc.cluster.local',
authToken: ,
namespace: ,
svcName: 'hello-nodejs',
verbose: 'false'
}
}
Kubernetes Plugin
The Kubernetes plug-in is a Jenkins extension that uses a Kubernetes cluster to dynamically provision Jenkins agents, use them to run builds and then dispose of them. It does do by using Kubernetes scheduling mechanism to optimize the load. The Jenkins images that ship with OpenShift comes with the Kubernetes plugin pre-installed.
An example of how to configure and use the Kubernetes plugin is available here:
TODO: deplete https://docs.openshift.org/latest/using_images/other_images/jenkins.html#using-the-jenkins-kubernetes-plug-in-to-run-jobs
OpenShift Jenkins Plugin
The OpenShift Jenkins Plugin is another Jenkins extension that exposes OpenShift-specific build and post-build actions in Jenkins:
A step-by-step example of how to use the OpenShift Jenkins Plugin is available here:
OpenShift Pipeline Plugin
Collocated Jenkins
This section addresses aspects that are specific to a Jenkins instance that lives in the same OpenShift project with the application it services.
This is an example of a project that sets up a collocated Jenkins instance and uses it to build and deploy an application:
This section describes the particularities of the situation when a single Jenkins instance is shared by multiple projects.
TODO: #Cross-Project_Access.
Security Considerations
Jenkins components need to access the OpenShift API exposed by the master for various operations: to access container images, to trigger a build, to check the status of a build, etc. so special privileges need to be assigned to the service account associated with the Jenkins deployment. By default, Jenkins authenticates to the API using the "system:serviceaccount:<project-name>:default service account, where <project-name> is the name of the project the Jenkins pod runs in. The service account must be granted the "edit" role. "default" is a generic account, so in some cases it may be better to created a dedicated "jenkins" service account, to be used by the Jenkins processes: "system:serviceaccount:<project-name>:jenkins. The standard Jenkins templates that come with OpenShift create the "jenkins" service account and give it the appropriate privileges automatically.
- TODO: OAuth authentication: https://docs.openshift.com/container-platform/latest/using_images/other_images/jenkins.html#jenkins-openshift-oauth-authentication
- TODO: Standard authentication: https://docs.openshift.com/container-platform/latest/using_images/other_images/jenkins.html#jenkins-jenkins-standard-authentication
Cross-Project Access
If Jenkins is configured to perform CI/CD services for other projects, the service account associated with the Jenkins pods in the Jenkins project must be granted elevated privileges in the client projects:
oc policy add-role-to-user edit system:serviceaccount:<jenkins-project-name>:jenkins -n <client-project>
To list the roles associated with a service account, use oc get rolebindings or oc describe policyBindings.
- TODO: Cross-project access: https://docs.openshift.com/container-platform/latest/using_images/other_images/jenkins.html#jenkins-cross-project-access
OpenShift Login Plugin
Authentication is managed by default by the OpenShift Login plugin, when the access is done via a public route. If you intend to log into Jenkins via the service IP, you will need to annotate the Jenkins service account with a redirect URL so that the OAuth server's whitelist is updated and it allows the login to complete.
oc annotate sa/jenkins serviceaccounts.openshift.io/oauth-redirecturi.1=http://<jenkins_service_ip:jenkins_service_port>/securityRealm/finishLogin --overwrite
TO DEPLETE
Deplete anything under this.
When necessary, it scales the pipeline execution by on-demand provisioning of multiple Jenkins containers, allowing Jenkins to run many jobs in parallel. OpenShift can integrate with Git-based source code repositories. When a developer pushes code to the GIt repository, Jenkins pulls the code and performs a project build. After the build is complete, Jenkins invokes the OpenShift master node API to initiate a "source-to-image" (S2I) build. As part of the SI process, the latest project artifact is downloaded from Jenkins and included in the image that runs in the OpenShift node.
Jenkins Pods and Projects
Jenkins infrastructure can run in an arbitrary project, on a per-project basis, but a more common setup is to create a dedicated CI/CD project and configure it to perform CI/CD services for other "client" projects.
Jenkins slave pods - they are configured in Jenkins system configuration as Kubernetes pods. Jenkins executes builds in separated pods in the same project it belongs. This is possible due to the Jenkins Kubernetes Plug-in. However, the builds do not use Jenkins pod resources. The slave pods included with OpenShift are the base pod to build custom slave pods, Node.js and Maven ("jenkins-slave-maven-rhel7").
Auxiliary Tools
Jenkinsfile
Jenkinsfile contains the definition of the pipeline that drives an OpenShift pipeline build, written in Groovy. It can be injected into a pipeline build configuration in two ways: inline in the build configuration and as a file in the source repository.
Inline Jenkinsfile
The Jenkins pipeline logic can be specified in-line in the build configuration as follows:
apiVersion: "v1"
kind: "BuildConfig"
metadata:
name: "my-pipeline-build-configuration"
spec:
strategy:
type: JenkinsPipeline
jenkinsPipelineStrategy:
jenkinsfile: |
//
// in-line pipeline code
//
...
Jenkinsfile in Source Repository
apiVersion: "v1"
kind: "BuildConfig"
metadata:
name: "my-pipeline-build-configuration"
spec:
source:
git:
uri: "https://....git"
strategy:
type: JenkinsPipeline
jenkinsPipelineStrategy:
jenkinsfilePath: "jenkinsfile/path/relative/to/the/root/of/the/repository"