Jenkins Concepts

External

• https://jenkins.io/doc/
• https://jenkins.io/doc/book/glossary/

Internal

• Jenkins
• Jenkins Pipeline Syntax
• Groovy

Overview

Jenkins is a Continuous Integration engine that can be also adapted to drive Continuous Delivery processes.

Instance

The Jenkins instance URL and name can be retrieved from the JENKINS_URL environment variable.

Node

A machine which is part of the Jenkins environment and capable of executing pipelines or projects. Both the masters and agents are considered to be nodes. To view the list of nodes, go to Manage Jenkins → Manage Nodes. The master(s) and the worker nodes are listed there.

Master

A master is a central, coordinating process that stores configuration, loads plugins, and renders the various user interfaces for Jenkins.

The master runs the Java pipeline code, and this model may introduce some confusion when the Java pipeline code handles files residing on nodes using the File and Path APIs. The problem is explained in detail below in the Pipeline and Files section.

To get the master host name from a pipeline:

Jenkins.getInstance().getComputer('').getHostName()

Agent

An agent, also known as a "build agent", is the runtime that executes the build. An agent can be a host, virtual machine or a container. Agents to execute a build may be selected with a label expression, specified in the job configuration. The agent connects to a master and executes tasks when directed by master.

Label

Agents are grouped using labels, which is user-defined text indicating similar functionality or capability. Also see:

Label Expressions

Agent Cloud

A cloud is a system configuration that provides dynamic agent provisioning and allocations. Example of clouds are Amazon EC2 cloud, Amazon EC2 Container Service cloud, Azure VM agents, etc.

Amazon EC2 Cloud

The cloud of agents consists of Amazon EC2 instances, provisioned and managed by the "Amazon EC2" plugin. For more details, see:

Amazon EC2 Plugin

Amazon EC2 Container Service Cloud

The cloud of agents consists of ECS containers, deployed in an ECS cluster. The agent cloud is managed by the (incorrectly named) "Amazon EC2 Container Service" plugin. For more details, see:

Amazon EC2 Container Service plugin

Worker

Equivalent with agent, the master needs workers to execute specific parts of a build pipeline.

Docker Plugin Agent

Docker Plugin Agent

Node JSON Representation

Jenkins BlueOcean Node JSON Representation

Executor

An executor is a slot for execution of work defined by a pipeline or project on a node. A node may have zero or more executors configured which corresponds to how many concurrent projects or pipelines are able to execute on that node. An executor can be thought as a single “process ID” or as the basic unit of resource that Jenkins executes on the node (in most cases the node is an agent) to run a build. An executor allows a build to run on an agent.

Items

Pipelines, projects and folders are items.

Folder

Pipelines and projects can be organized in folders, which is an organizational container for items, similar to folders on a filesystem.

Project (Freestyle Project)

A project is a user-configured description of work which Jenkins should perform, such as building a piece of software, etc. Projects are a replacement for the deprecated term "job".

GitHub Repository

https://wiki.jenkins.io/display/JENKINS/GitHub+Branch+Source+Plugin

An item that watches a GitHub repository and triggers actions on repository changes. Not found under "New Items" in the UI. A new one can be created by copying an existing GitHub Repository by going to New Item → Copy from .

GitHub Repository Configuration

Job

A job is a deprecated term, synonymous with Project. A single execution of a job results in a build.

Jobs can be created from the UI and they show up on the dashboard, in the main table. A job state is maintained in $JENKINS_HOME/jobs/<job-name>. The job directory contains a config.xml and a "builds" subdirectory, which contains build state. The state include links to "lastFailedBuild", "lastStableBuild", "lastSuccessfulBuild", "lastUnstableBuild" and "lastUnsuccessfulBuild".

A job can be configured with Jenkins -> job -> Configure.

A job's configuration contains the following:

• General Configuration
  • Label Expression
• Source Code Management configuration
• Build Triggers
• Build Environment
• Build definition - usually a shell command.
• Post-build actions.

Each job has its own security context.

Build (Run)

A build is the result of a single execution of a project (job). The build belongs to the project (job).

Builds have monotonically increasing IDs (#33). They are also sometimes called "runs".

For a multi-branch pipeline, there are multiple series of runs with monotonically increasing run IDs, each series being associated associated with a branch of the multi-branch pipeline. The JSON representation of a specific multi-branch pipeline run can be obtained via REST with a GET invocation into https://<jenkins-rest-base-url>/pipelines/<pipeline-name>/branches/<branch-name>/runs/<run-id>/

Build (Run) JSON Representation

Jenkins BlueOcean Build (Run) JSON Representation

Build Artifact

An artifact is an immutable file generated during a build or a pipeline run. The artifact is archived onto the master for later retrieval by users. Usage of artifacts can be tracked globally across multiple pipeline or projects via the artifact's fingerprint. Jenkins provides API to manage artifacts (archiveArtifacts, etc.)

Publisher

A publisher is part of a build that publishes reports, sends notifications, etc. after the completion of all configured steps.

Test Results

After executing a pipeline stage that runs JUnit tests and produces JUnit test results in XML format, the test results are collected with:

stage.junit '**/TEST-*.xml'

junit step cues Jenkins to ingest the test results, process them and provide historical test result trends, a web UI for viewing test reports, tracking failures, etc.

On a typical agent, the tests are located under /home/ec2-user/jenkins-agent/workspace/<branch-name>/<git-repository-name>/**/target/*-reports/**/TEST-*.xml.

Test JSON Representation

Jenkins BlueOcean Test JSON Representation

Aborting a Build

https://wiki.jenkins.io/display/JENKINS/Aborting+a+build

Pipeline

https://jenkins.io/doc/book/pipeline/overview/

A user-defined model of software continuous delivery pipeline.

Pipeline Overview

A Jenkins pipeline is a suite of plugins that implement a continuous integration pipeline. A pipeline orchestrates long-running activities that can span multiple build agents. Pipelines are expressed in Jenkins via the Pipeline DSL syntax (see below). The definition is written into a Jenkinsfile, or declared in the Jenkins UI. The pipeline code defines the entire build process, which typically includes stages for building the application, testing it and delivering it. An indented rendering of the pipeline execution is available by navigating to a build then picking "Pipeline Steps" from the left menu. A graphic rendering is available by picking "Open Blue Ocean" from the same menu. Pipelines are formerly known as workflows.

The list of pipelines for a Jenkins instance can be obtained via REST with a GET invocation into https://<jenkins-rest-base-url>/pipelines/.

When integrated with OpenShift, the Jenkins Pipeline functionality provides the engine for OpenShift Pipeline builds. More details about OpenShift Jenkins pipeline are available here OpenShift Jenkins Pipelines.

Pipeline Types

(Simple) Pipeline

Simple Pipeline Configuration

Simple Pipeline JSON Representation

Jenkins BlueOcean Simple Pipeline JSON Representation

Multibranch Pipeline

A set of pipeline projects according to detected branches in an SCM repository.

Multibranch Pipeline JSON Representation

Jenkins BlueOcean Multibranch Pipeline JSON Representation

Multibranch Pipeline with Defaults

Extends the Multibranch pipeline plugin.

Pipeline Attributes

Name

It is the same as the project name.

It is best if it does not contain spaces, otherwise the workspace name, which includes the pipeline name, will contains spaces, the corresponding directory name will also contain space and this will most likely break scripts.

If you want to use names with spaces for display purposes, use display name.

Display Name

Can be set as follows: Pipeline page → Configure → General → Advanced Project Options → Click Advanced → Display Name.

Jenkinsfile

https://jenkins.io/doc/book/pipeline/jenkinsfile/

Jenkinsfile contains the definition of the pipeline, written either in Scripted Pipeline or Declarative Pipeline syntax. It is usually committed in the project source code repository - this allows the specification of the CD pipeline to be treated as part of the application, and thus be versioned and reviewed like any other application piece of code. It is best if the Jenkinsfile is thought of, and used as the single source of truth for the pipeline it configures. The Jenkinsfile is not a replacement for an existing build tool such as Maven or Gradle, but rather can be viewed as a glue layer to bind the multiple phases of a project’s development lifecycle (build, test, deploy, etc) together.

Jenkinsfile uses rules identical to Groovy for string interpolation.

A Jenkinsfile can be written with two types of syntax: declarative (newer and recommended) and scripted.

Jenkinsfile can retrieve pipeline parameter values.

Pipeline Syntax

Pipeline Syntax

Pipeline Global Variables

Global variables are available to the Pipeline directly, not as steps. They expose methods and variables to be accessed from the Pipeline script, Jenkinsfile. The following global variables are available:

env (Pipeline Environment Variables)

Pipeline Environment Variables

params (Pipeline Parameters)

Jenkins Pipeline Parameters

currentBuild

currentBuild

pipeline

https://github.com/jenkinsci/pipeline-model-definition-plugin/wiki/Getting-Started

Pipeline Node

A node is a machine. It is part of the Jenkins environment and it is capable of executing a pipeline. More details about Jenkins node are available in the "Node" section.

Stage

A stage is part of pipeline, used for defining a conceptually distinct subset of tasks performed through the pipeline (e.g. "Build", "Test", "Deploy"). The logical stage is declared using the stage pipeline step. For more details see:

Jenkins Pipeline Syntax | stage

Step

A step is a single task, that defines what Jenkins should do do at a particular point in time - a step in the process. Steps can be implemented as Groovy classes.

Pipeline Syntax, Writing a Jenkins Pipeline

Jenkins Pipeline Syntax

Writing a Jenkins Pipeline

Pipeline Execution

Jenkins Pipeline Execution

Pipeline Result

See:

Build Result

Pipeline Failure

Declarative Pipeline Failure Handling

Declarative Pipeline Failure Handling

Scripted Pipeline Failure Handling

Scripted Pipeline Failure Handling

Pipeline and Files

https://support.cloudbees.com/hc/en-us/articles/230922128-Pipeline-Using-java-io-File-in-a-Pipeline-description

The Java pipeline code runs on the master, which connect remotely to various nodes that contribute to the pipeline work. This model may introduce some confusion when the Java pipeline code handles files residing on nodes using the File and Path APIs. The File and Path API sections that deal with file or directory existence on disk will interact with the master's filesystem, which does not contain the files, as they're residing on nodes. To address these issues, the DSL introduces several steps that interact with the files on the node:

• readFile
• writeFile
• fileExists
• touch

A good practice is to separate logic that deals with files on master (and thus is allowed to use java.nio.files APIs) and files on nodes (and thus limited to use pipeline.readFile(), etc) in different methods.

Trigger

A trigger is a criterion for triggering a new pipeline run or a build.

Downstream Pipeline/Project

A downstream pipeline or project is an pipeline/project that is triggered as part of the execution of a separate pipeline or project.

Upstream Pipeline/Project

An upstream pipeline or project triggers downstream pipeline or project as part of its execution.

View

Shows as a tab on the entry page. Available choice:

• Build Pipeline View. Show the jobs in a build pipeline view. The complete pipeline of jobs that a version propagates through are shown as a row in the view.
• Delivery Pipeline View. Continuous Delivery pipelines, aimed for visualization on information radiators. Shows one or more delivery pipeline instances, based on traditional Jenkins jobs with upstream/downstream dependencies.
• Delivery Pipeline View for Jenkins Pipelines. Shows one or more delivery pipeline instances, based on Jenkins pipelines (created using the Pipeline or Workflow plugin).
• List View. Shows items in a simple list format. You can choose which jobs are to be displayed in which view.
• MultiJob View. Shows items in a simple list format. You can choose which jobs are to be displayed in which view.
• My View. This view automatically displays all the jobs that the current user has an access to.
• Pipeline Aggregator View. Full screen dashboard that will aggregate the pipelines showing the stages and the commiters.

The most common is List View.

Workspace

A workspace is a disposable directory on the file system of a node where work can be done by a pipeline or project. Workspaces are typically left in place after a build or pipeline run completes unless specific workspace cleanup policies have been put in place on the master. Workloads running on a slave node (via the node wrapper) get a fresh/empty workspace directory upon entry into the node wrapper. This may require using (S3-based) stash/unstash for large directories that need to be made available for slave node execution.

Build Environment

OpenShift Jenkins Build Environment

Distributed Build

https://wiki.jenkins.io/display/JENKINS/Distributed+builds

Core

The core (jenkins.war) is the primary Jenkins application that provides the basic web UI, configuration and foundation upon which plugins can be built.

Jenkins Core Libraries

https://jenkins.io/doc/developer/plugin-development/dependencies-and-class-loading/

https://updates.jenkins-ci.org/download/war/

These are libraries are useful when writing code to be execute inside Jenkins.

Plugin

https://wiki.jenkins.io/display/JENKINS/Plugin+tutorial

A plugin is an extension to Jenkins functionality provided separately from Jenkins core. The list of installed plugins can be obtained on the console Jenkins -> Manage Jenkins -> Manage Plugins -> Installed.

Interesting Jenkins Plugins

• Task Scanner https://wiki.jenkins.io/pages/viewpage.action?pageId=2916549
• Docker
• Credentials
• Amazon EC2
• Amazon EC2 Container Service

Server Administration Concepts

Instance State

A Jenkins instance state consists of the following:

• The WAR file
• Plugins.
• Settings
• Build logs
• Artifacts archives
• Jobs
• Various temporary files associated with running the jobs, including repository clones: "jenkins_home/workspace/test-build/.git/..."

Everything mentioned above is stored under $JENKINS_HOME. It is sufficient to archive this directory to make a back up of the instance. Similarly, restoring the data is just replacing the contents of the JENKINS_HOME directory from a back up.

The WAR file is stored under ${JENKINS_HOME}/war. The location can be overridden with --webroot=<new-location>. The WAR is expanded during the startup sequence.

The plugins are stored under ${JENKINS_HOME}/plugins. The location can be overridden with --pluginroot =<new-location>.

Jenkins on Docker

Running Jenkins as a Docker Container

Running Jenkins as a Docker Container

Running Build Agents as Docker Containers

Jenkins Docker Plugin

CLI

Jenkins CLI

Security

Security Concepts

IntelliJ and Jenkins

IntelliJ and Jenkins

Blue Ocean REST API

https://github.com/jenkinsci/blueocean-plugin/tree/master/blueocean-rest

Jenkins REST Base URL

https://<host>/<instance-name>/blue/rest/organizations/jenkins

Trailing Slash

All URLs must end in a trailing slash to return resources, otherwise they will silently "succeed" but return nothing.

Pagination and Number of Records

Apparently by default only maximum 100 records are returned by default for the node API. To modify this:

.../?start=0&limit=1000

Reference: https://github.com/jenkinsci/blueocean-plugin/pull/486/files#r77956164

Extending Jenkins with Shared Libraries

https://www.jenkins.io/doc/book/pipeline/shared-libraries/

@Library('my-library') _

Use a tag or a branch (relative to the repository that contains the library code):

@Library('my-library@task/my-task-branch') _

CPS

• https://www.jenkins.io/doc/book/pipeline/cps-method-mismatches/
• https://www.devopscat.tech/2018/10/what-is-noncps-in-jenkins-pipeline/
• https://plugins.jenkins.io/workflow-cps/

@NonCPS

dependencies {
  implementation 'com.cloudbees:groovy-cps:1.31'
}

⚠️ CPS-transformed code may call non-CPS-transformed code or other CPS-transformed code, and non-CPS-transformed code may call other non-CPS-transformed code, but non-CPS-transformed code may not call CPS-transformed code. If non-CPS-transformed code calls CPS-transformed code, the CPS interpreter is unable to operate correctly, resulting in incorrect and often confusing results.

Non-CPS-transformed code may not call pipeline steps. The following won't work:

@NonCPS
def something() {
 node(arch) {
   sh 'something'
 }
}