Bamboo Operations: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
 
(39 intermediate revisions by the same user not shown)
Line 49: Line 49:
Save Repository.
Save Repository.


=Create a Typical Task=
=<span id='Create_a_Typical_Task'></span>Configure a Job=


This is how you create a typical [[Bamboo_Concepts#Task|task]] for the [[Bamboo_Concepts#Default_Job|Default job]].
A plan comes with a "[[Bamboo_Concepts#Default_Job|Default job]]", with a "Source Code Checkout" first task, which can be customized as shown [[#Source_Code_Checkout|below]]. More [[Bamboo_Concepts#Task|tasks]] can also be added:


====Source Code Checkout====
==Source Code Checkout==


Tasks -> Add task -> Source Control -> Source Code Checkout.  
Tasks -> Add task -> Source Control -> Source Code Checkout.  
Line 65: Line 65:
Force clean build - typically not selected. Removes the source directory and checks it out again prior to each build. This may significantly increase build times.  
Force clean build - typically not selected. Removes the source directory and checks it out again prior to each build. This may significantly increase build times.  


====Command====
==Build Command==


Tasks -> Add task -> Command.  
Tasks -> Add task -> Command.  
The configuration of the build command is different in the case of library and of a service that deploys as a Docker container.
===Building a Library===


Task description: "Gradle build and publish command"
Task description: "Gradle build and publish command"
Line 77: Line 81:
  clean build publish
  clean build publish


<span id='Task_Environment_Variable_Support'>'''Task_Environment Variables Support'''. Environment variables: This allows setting [[Bamboo_Concepts#Task_Environment_Variables|task environment variables]], which will propagate to the task environment. The syntax is presented below:
===Building a Service===
 
The service does not. publish a JAR in the Maven repository, it builds a container and push it into a Docker registry instead.
 
Task description: "Gradle docker build"
 
Executable: Gradle4
 
Argument:
 
-Dorg.gradle.project.BAMBOO_BUILD_NUMBER=${[[Bamboo_Concepts#buildNumber|bamboo.buildNumber]]} clean docker
 
For details on how Gradle builds Docker images, see: {{Internal|Com.palantir.docker|com.palantir.docker Gradle Plugin}}
 
The service build needs an additional docker build task:
 
===<span id='Docker_Build_Script'></span>Docker Image Registry Push Script===
 
This is only required for service builds:
 
Script task.
 
Task description: "Docker Image Registry Push Script"
 
Interpreter: Shell
 
Script location: Inline
 
Script body:
 
<syntaxhighlight lang='bash'>
#!/usr/bin/env bash
 
MAVEN_GROUP_NAME=$1
APPLICATION_NAME=$2
GIT_BRANCH=$3
REVISION_NUMBER=$4
BAMBOO_BUILD_NUMBER=$5
 
NAMESPACE=${MAVEN_GROUP_NAME}
AWS_ACCOUNT_ID=673499572719
REGISTRY_URL=${AWS_ACCOUNT_ID}.dkr.ecr.us-west-2.amazonaws.com
 
echo "MAVEN_GROUP_NAME=${MAVEN_GROUP_NAME}"
echo "APPLICATION_NAME=${APPLICATION_NAME}"
echo "GIT_BRANCH=${GIT_BRANCH}"
echo "REVISION_NUMBER=${REVISION_NUMBER}"
echo "BAMBOO_BUILD_NUMBER=${BAMBOO_BUILD_NUMBER}"
echo "NAMESPACE=${NAMESPACE}"
echo "AWS_ACCOUNT_ID=${AWS_ACCOUNT_ID}"
echo "REGISTRY_URL=${REGISTRY_URL}"
 
echo "logging into ECR for AWS ID ${AWS_ACCOUNT_ID} ..."
 
$(aws ecr get-login --no-include-email --region us-west-2 --registry-ids ${AWS_ACCOUNT_ID})
 
#
# debugging
#
docker images
 
#
# clean the repository
#
# docker rmi --force $(docker images --format "{{.ID}}")
 
source_image_tag=${NAMESPACE}/${APPLICATION_NAME}:latest
target_image_tag=${REGISTRY_URL}/${NAMESPACE}/${APPLICATION_NAME}:${GIT_BRANCH}-${BAMBOO_BUILD_NUMBER}
target_image_tag_latest=${REGISTRY_URL}/${NAMESPACE}/${APPLICATION_NAME}:latest
 
echo "tagging the latest image ${source_image_tag} as ${target_image_tag} ..."
 
docker tag ${source_image_tag} ${target_image_tag} || { echo "docker tag failed" 1>&2; exit 1; }
 
echo "pushing image ${target_image_tag} ..."
 
docker push ${target_image_tag} || { echo "docker push failed" 1>&2; exit 1; }
 
echo "tagging the latest image ${source_image_tag} as ${target_image_tag_latest} ..."
 
docker tag ${source_image_tag} ${target_image_tag_latest} || { echo "docker tag failed" 1>&2; exit 1; }
 
echo "pushing image ${target_image_tag_latest} ..."
 
docker push ${target_image_tag_latest} || { echo "docker push failed" 1>&2; exit 1; }
</syntaxhighlight>
 
Argument:
playground.aws.example myapp ${bamboo.repository.git.branch} ${bamboo.repository.revision.number} ${[[Bamboo_Concepts#buildNumber|bamboo.buildNumber]]}
 
{{Warn|The AWS user under whos identity Bamboo runs must have "ecr:InitiateLayerUpload" permission with the target ECR registry, otherwise Docker push fails.}}
 
For more details, see: {{Internal|#AWS_Identity|Bamboo AWS Identity}}
 
===Task Environment Variables Support===
 
Environment variables: This allows setting [[Bamboo_Concepts#Task_Environment_Variables|task environment variables]], which will propagate to the task environment. The syntax is presented below:
<syntaxhighlight lang='java'>
<syntaxhighlight lang='java'>
MY_TEST_ENVIRONMENT_VARIABLE="something"
MY_TEST_ENVIRONMENT_VARIABLE="something"
Line 84: Line 185:
If, for example, we are building and testing Spring applications, that value will [[Spring_Property_Injection_Concepts#Environment_Variables|automatically propagate]] to the Spring runtime as "my.test.enviornment.variable" and can be used in [[Spring_Property_Injection_Concepts#Property_Placeholders|placeholders]].
If, for example, we are building and testing Spring applications, that value will [[Spring_Property_Injection_Concepts#Environment_Variables|automatically propagate]] to the Spring runtime as "my.test.enviornment.variable" and can be used in [[Spring_Property_Injection_Concepts#Property_Placeholders|placeholders]].


====JUnit Parser====
Multiple variables should be separated with spaces. Parameters with spaces must be quoted (e.g ANT_OPTS="-Xms200m -Xmx700m").
 
==JUnit Parser==


Tasks -> Add task -> JUnit Parser.  
Tasks -> Add task -> JUnit Parser.  
Line 95: Line 198:
<font color=darkgray>This value seems different from default. It works, but why different?</font>
<font color=darkgray>This value seems different from default. It works, but why different?</font>


====Command====
==Clover File Converter Command==


Tasks -> Add task -> Command.  
Tasks -> Add task -> Command.  
Line 109: Line 212:
</syntaxhighlight>
</syntaxhighlight>


====Command====
==Sonarqube Command==


Tasks -> Add task -> Command.  
Tasks -> Add task -> Command.  
Line 120: Line 223:


<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
sonarqube  -Dsonar.branch.name=${bamboo.repository.git.branch} -Dsonar.projectKey=up.plat-svc.cryptm  -Dsonar.organization=test-inc  -Dsonar.host.url=https://sonarcloud.io  -Dsonar.login=...
sonarqube  -Dsonar.branch.name=${bamboo.repository.git.branch} -Dsonar.projectKey=up.plat-svc.cryptm  -Dsonar.organization=test-inc  -Dsonar.host.url=https://sonarcloud.io  -Dsonar.login=... -Dorg.gradle.project.BAMBOO_BUILD_NUMBER=${bamboo.buildNumber}
</syntaxhighlight>
</syntaxhighlight>
{{Warn|If the build.gradle declares project properties that are provided to gradle in command line, make sure the corresponding command-line options are present in the above command.}}


=Variables=
=Variables=
Line 145: Line 251:
=Deployment Operations=
=Deployment Operations=


[[Bamboo_Concepts#Deployment_Operations|Deployment projects]].
[[Bamboo_Concepts#Deployment_Operations|Deployment projects]] operations:
 


==Configure a Deployment Project==
==Configure a Deployment Project==
Line 152: Line 257:
1. Select  a build plan.
1. Select  a build plan.


2. Edit tasks
===Edit tasks===
 
====Clean working directory====
 
"Clean working directory" task.


* "Clean working directory" task.
====Deploy Image in AWS Script====
* Script:


Interpreter: Shell
Interpreter: Shell
Line 164: Line 272:
#!/usr/bin/env bash
#!/usr/bin/env bash


MAVEN_ARTIFACT_NAME=$1
MAVEN_GROUP_NAME=$1
DEPLOYMENT_ENVIRONMENT_NAME=$2
APPLICATION_NAME=$2
TARGET_AWS_CLUSTER=$3
DEPLOYMENT_ENVIRONMENT_NAME=$3
BAMBOO_RELEASE_NAME=$4
TARGET_AWS_CLUSTER=$4
BAMBOO_RELEASE_NAME=$5
REGISTRY_URL=144446676909.dkr.ecr.us-west-2.amazonaws.com


REGISTRY_URL=144446676909.dkr.ecr.us-west-2.amazonaws.com
echo "MAVEN_GROUP_NAME: ${MAVEN_GROUP_NAME}"
MAVEN_GROUP_NAME=playground.aws.example
echo "APPLICATION_NAME: ${APPLICATION_NAME}"
echo "DEPLOYMENT_ENVIRONMENT_NAME: ${DEPLOYMENT_ENVIRONMENT_NAME}"
echo "TARGET_AWS_CLUSTER: ${TARGET_AWS_CLUSTER}"
echo "BAMBOO_RELEASE_NAME: ${BAMBOO_RELEASE_NAME}"
echo "REGISTRY_URL: ${REGISTRY_URL}"


echo "logging into ECR ..."
echo "logging into ECR ..."
Line 176: Line 290:
$(aws ecr get-login --no-include-email --region us-west-2)
$(aws ecr get-login --no-include-email --region us-west-2)


source_image=${REGISTRY_URL}/${MAVEN_GROUP_NAME}/${MAVEN_ARTIFACT_NAME}
source_image_tag=${REGISTRY_URL}/${MAVEN_GROUP_NAME}/${APPLICATION_NAME}:${BAMBOO_RELEASE_NAME}
source_tag=${BAMBOO_RELEASE_NAME}
target_image_tag=${REGISTRY_URL}/${MAVEN_GROUP_NAME}/${APPLICATION_NAME}:${DEPLOYMENT_ENVIRONMENT_NAME}
target_image=${REGISTRY_URL}/${MAVEN_GROUP_NAME}/${MAVEN_ARTIFACT_NAME}
target_tag=${DEPLOYMENT_ENVIRONMENT_NAME}


echo "tagging ${source_image}:${source_tag} as ${target_image}:${target_tag} ..."
echo "tagging ${source_image_tag} as ${target_image_tag} ..."


docker tag ${source_image}:${source_tag}  ${target_image}:${target_tag}
docker tag ${source_image_tag} ${target_image_tag}


echo "pushing ${target_image}:${target_tag} ..."
echo "pushing ${target_image_tag} ..."


docker push ${target_image}:${target_tag}
docker push ${target_image_tag}


echo "updating AWS service ${APPLICATION_NAME} on cluster ${TARGET_AWS_CLUSTER} ..."
echo "updating AWS service ${APPLICATION_NAME} on cluster ${TARGET_AWS_CLUSTER} ..."
Line 196: Line 308:
Arguments:  
Arguments:  


  myapp dev01 cluster-dev01 ${bamboo.deploy.release}
  com.playground myapp dev01 cluster-dev01 ${bamboo.deploy.release}
 
where ${bamboo.deploy.release} is substituted by Bamboo at runtime with [[Bamboo_Concepts#Release_Name|release name]], which usually is similar to "develop-699".
 
Note that the image to be deployed in the target cluster, in this case 144446676909.dkr.ecr.us-west-2.amazonaws.com/playground.aws.example/myapp:develop-699, must be previously built by the Bamboo as part of a [[Bamboo_Concepts#Build_Plan|build plan]].
 
A typical build plan driven by Gradle ends up in a command similar to:
 
gradle -Dorg.gradle.project.BAMBOONUM=699 clean docker
 
Gradle can use a Docker plugin such as [[com.palantir.docker]].
 
=AWS Identity=
 
<font color=darkgray>Figure out how to configure the identity of the AWS user running the Bamboo instance.</font>

Latest revision as of 09:41, 6 February 2019

Internal

Plan Operations

Create a Plan

A plan can be created and configured any time after creation of the enclosing the project:

Projects -> the project in question -> Upper Blue Menu -> Create -> Create plan.

The "Configure Plan" view is exposed.

Configure a Plan

Projects -> the project in question -> Select the Plan -> Actions drop down -> Configure plan.

Configuration options when creating the plan:

  • Project
  • Plan name
  • Plan key
  • Plan description
  • Plan access - allow all users to view this plan
  • Link repository to new build plan -> Link new repository. The process is similar to the one described in the "Repository Operations" section.

The repository can be specified during the plan creation process, or later.

Repository Operations

To add a repository to a plan:

Project -> Plan -> Actions: Configure plan -> "Repositories" tab -> Add Repository -> "Git" repository (see difference between Git and GitHub repositories)

Display name: up.plat-svc.cryptm. Use the name of repository.

Repository URL: git@github.com:test-inc/up.plat-svc.cryptm.git

Authentication type: SSH private key

Use shared credentials, shared credentials: Github Bamboo User SSH

Branch: develop (if nothing is specified, it assumes "master")

Test Connection

Repository access: Allow all users to reuse the configuration of this repository.

Save Repository.

Configure a Job

A plan comes with a "Default job", with a "Source Code Checkout" first task, which can be customized as shown below. More tasks can also be added:

Source Code Checkout

Tasks -> Add task -> Source Control -> Source Code Checkout.

Task description: "GitHub checkout"

Repository: pick from pre-configured, see repository.

Checkout Directory: this is optional, to specify an alternative to the default directory.

Force clean build - typically not selected. Removes the source directory and checks it out again prior to each build. This may significantly increase build times.

Build Command

Tasks -> Add task -> Command.

The configuration of the build command is different in the case of library and of a service that deploys as a Docker container.

Building a Library

Task description: "Gradle build and publish command"

Executable: Gradle4

Argument:

clean build publish

Building a Service

The service does not. publish a JAR in the Maven repository, it builds a container and push it into a Docker registry instead.

Task description: "Gradle docker build"

Executable: Gradle4

Argument:

-Dorg.gradle.project.BAMBOO_BUILD_NUMBER=${bamboo.buildNumber} clean docker

For details on how Gradle builds Docker images, see:

com.palantir.docker Gradle Plugin

The service build needs an additional docker build task:

Docker Image Registry Push Script

This is only required for service builds:

Script task.

Task description: "Docker Image Registry Push Script"

Interpreter: Shell

Script location: Inline

Script body:

#!/usr/bin/env bash

MAVEN_GROUP_NAME=$1
APPLICATION_NAME=$2
GIT_BRANCH=$3
REVISION_NUMBER=$4
BAMBOO_BUILD_NUMBER=$5

NAMESPACE=${MAVEN_GROUP_NAME}
AWS_ACCOUNT_ID=673499572719
REGISTRY_URL=${AWS_ACCOUNT_ID}.dkr.ecr.us-west-2.amazonaws.com

echo "MAVEN_GROUP_NAME=${MAVEN_GROUP_NAME}"
echo "APPLICATION_NAME=${APPLICATION_NAME}"
echo "GIT_BRANCH=${GIT_BRANCH}"
echo "REVISION_NUMBER=${REVISION_NUMBER}"
echo "BAMBOO_BUILD_NUMBER=${BAMBOO_BUILD_NUMBER}"
echo "NAMESPACE=${NAMESPACE}"
echo "AWS_ACCOUNT_ID=${AWS_ACCOUNT_ID}"
echo "REGISTRY_URL=${REGISTRY_URL}"

echo "logging into ECR for AWS ID ${AWS_ACCOUNT_ID} ..."

$(aws ecr get-login --no-include-email --region us-west-2 --registry-ids ${AWS_ACCOUNT_ID})

#
# debugging
#
docker images

#
# clean the repository
#
# docker rmi --force $(docker images --format "{{.ID}}")

source_image_tag=${NAMESPACE}/${APPLICATION_NAME}:latest
target_image_tag=${REGISTRY_URL}/${NAMESPACE}/${APPLICATION_NAME}:${GIT_BRANCH}-${BAMBOO_BUILD_NUMBER}
target_image_tag_latest=${REGISTRY_URL}/${NAMESPACE}/${APPLICATION_NAME}:latest

echo "tagging the latest image ${source_image_tag} as ${target_image_tag} ..."

docker tag ${source_image_tag} ${target_image_tag} || { echo "docker tag failed" 1>&2; exit 1; }

echo "pushing image ${target_image_tag} ..."

docker push ${target_image_tag} || { echo "docker push failed" 1>&2; exit 1; }

echo "tagging the latest image ${source_image_tag} as ${target_image_tag_latest} ..."

docker tag ${source_image_tag} ${target_image_tag_latest} || { echo "docker tag failed" 1>&2; exit 1; }

echo "pushing image ${target_image_tag_latest} ..."

docker push ${target_image_tag_latest} || { echo "docker push failed" 1>&2; exit 1; }

Argument:

playground.aws.example myapp ${bamboo.repository.git.branch} ${bamboo.repository.revision.number} ${bamboo.buildNumber}

The AWS user under whos identity Bamboo runs must have "ecr:InitiateLayerUpload" permission with the target ECR registry, otherwise Docker push fails.

For more details, see:

Bamboo AWS Identity

Task Environment Variables Support

Environment variables: This allows setting task environment variables, which will propagate to the task environment. The syntax is presented below:

MY_TEST_ENVIRONMENT_VARIABLE="something"

If, for example, we are building and testing Spring applications, that value will automatically propagate to the Spring runtime as "my.test.enviornment.variable" and can be used in placeholders.

Multiple variables should be separated with spaces. Parameters with spaces must be quoted (e.g ANT_OPTS="-Xms200m -Xmx700m").

JUnit Parser

Tasks -> Add task -> JUnit Parser.

Task description: "Parses JUnit test results"

Specify custom result directories:

**/test-results/test/*.xml

This value seems different from default. It works, but why different?

Clover File Converter Command

Tasks -> Add task -> Command.

Task description: "Clover file converter"

Executable: CloverConverter

Argument:

clean build publish

Sonarqube Command

Tasks -> Add task -> Command.

Task description: "SonarQube"

Executable: Gradle4,

Argument:

sonarqube  -Dsonar.branch.name=${bamboo.repository.git.branch} -Dsonar.projectKey=up.plat-svc.cryptm  -Dsonar.organization=test-inc   -Dsonar.host.url=https://sonarcloud.io  -Dsonar.login=... -Dorg.gradle.project.BAMBOO_BUILD_NUMBER=${bamboo.buildNumber}



If the build.gradle declares project properties that are provided to gradle in command line, make sure the corresponding command-line options are present in the above command.

Variables

Variables operations.

Define a Plan Variable

TODO: https://confluence.atlassian.com/bamboo/defining-plan-variables-289276859.html

Define a Global Variable

https://confluence.atlassian.com/bamboo/defining-global-variables-289277112.html

This procedure defines a global variable:

Go to the Cog Wheel on the front page -> Global Variables -> Add.

Note that a MY_VARIABLE defined here will be exposed as bamboo_MY_VARIABLE in the tasks' shells. For more details see:

Global Variables

Deployment Operations

Deployment projects operations:

Configure a Deployment Project

1. Select a build plan.

Edit tasks

Clean working directory

"Clean working directory" task.

Deploy Image in AWS Script

Interpreter: Shell

Script location: Inline

#!/usr/bin/env bash

MAVEN_GROUP_NAME=$1
APPLICATION_NAME=$2
DEPLOYMENT_ENVIRONMENT_NAME=$3
TARGET_AWS_CLUSTER=$4
BAMBOO_RELEASE_NAME=$5
REGISTRY_URL=144446676909.dkr.ecr.us-west-2.amazonaws.com

echo "MAVEN_GROUP_NAME: ${MAVEN_GROUP_NAME}"
echo "APPLICATION_NAME: ${APPLICATION_NAME}"
echo "DEPLOYMENT_ENVIRONMENT_NAME: ${DEPLOYMENT_ENVIRONMENT_NAME}"
echo "TARGET_AWS_CLUSTER: ${TARGET_AWS_CLUSTER}"
echo "BAMBOO_RELEASE_NAME: ${BAMBOO_RELEASE_NAME}"
echo "REGISTRY_URL: ${REGISTRY_URL}"

echo "logging into ECR ..."

$(aws ecr get-login --no-include-email --region us-west-2)

source_image_tag=${REGISTRY_URL}/${MAVEN_GROUP_NAME}/${APPLICATION_NAME}:${BAMBOO_RELEASE_NAME}
target_image_tag=${REGISTRY_URL}/${MAVEN_GROUP_NAME}/${APPLICATION_NAME}:${DEPLOYMENT_ENVIRONMENT_NAME}

echo "tagging ${source_image_tag} as ${target_image_tag} ..."

docker tag ${source_image_tag} ${target_image_tag}

echo "pushing ${target_image_tag} ..."

docker push ${target_image_tag}

echo "updating AWS service ${APPLICATION_NAME} on cluster ${TARGET_AWS_CLUSTER} ..."

aws ecs update-service --cluster ${TARGET_AWS_CLUSTER} --service ${APPLICATION_NAME} --force-new-deployment

Arguments:

com.playground myapp dev01 cluster-dev01 ${bamboo.deploy.release}

where ${bamboo.deploy.release} is substituted by Bamboo at runtime with release name, which usually is similar to "develop-699".

Note that the image to be deployed in the target cluster, in this case 144446676909.dkr.ecr.us-west-2.amazonaws.com/playground.aws.example/myapp:develop-699, must be previously built by the Bamboo as part of a build plan.

A typical build plan driven by Gradle ends up in a command similar to:

gradle -Dorg.gradle.project.BAMBOONUM=699 clean docker

Gradle can use a Docker plugin such as com.palantir.docker.

AWS Identity

Figure out how to configure the identity of the AWS user running the Bamboo instance.