AWS CodeBuild Operations: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
 
(34 intermediate revisions by the same user not shown)
Line 7: Line 7:
==Project configuration==
==Project configuration==


Select [[AWS_CodeBuild_Concepts#Build_Badge|Build badge]].
====Project name====
 
====Description====
 
====Build badge====
For more details about the build badge, see Select [[AWS_CodeBuild_Concepts#Build_Badge|AWS CodeBuild Concepts - Build Badge]].
 
====Tags====


==Source==
==Source==


Source provider: GitHub
====Source provider====
 
GitHub
 
====Repository====


Repository in my GitHub account.
Repository in my GitHub account.
====GitHub repository====
====Connection Status====


Additional configuration
Additional configuration


Git clone depth: 1
====Git clone depth====
 
Clone depth 1
 
====Build Status====


==Primary source webhook events==
==Primary source webhook events==
Line 25: Line 44:
==Environment==
==Environment==


Environment image: Managed image
====Environment image====
 
The options are "Managed image" or "Custom image". A managed image is always preferable if it provides all the build functionality you need. However, if you need extra functionality, you can build and use a custom image.
 
'''Managed image configuration''':


Operating system: Ubuntu
Operating system: Ubuntu
Line 35: Line 58:
{{Warn|The Java runtime may have an obsolete Gradle version. If that is the case, [[Gradle_Concepts#The_Gradle_Wrapper|initialize your project with gradlew]] and configure the buildspec file to use ./gradlew to build the project.}}
{{Warn|The Java runtime may have an obsolete Gradle version. If that is the case, [[Gradle_Concepts#The_Gradle_Wrapper|initialize your project with gradlew]] and configure the buildspec file to use ./gradlew to build the project.}}


Privileged: Enable this flag if you want to build Docker images or want your builds to get [[AWS_CodeBuild_Concepts#Privileged_Build|elevated privileges]]
'''Custom image configuration''':
 
Environment type: Linux
 
Image registry: Amazon ECR
 
ECR account: My ECR account
 
Amazon ECR Repository: "com.example/unity/unity-build-image"


Service role:
Amazon ECR image: latest


For the implications on the build process of correctly configuring the service role, see: {{Internal|AWS_CodeBuild_Concepts#Service_Role|CodeBuild Service Role}}
Image pull credentials: Project service role. This is the service role associated with this build project.
 
====Privileged====
 
This flag should be enabled if the build project builds Docker images. More details about elevated privileges: [[AWS_CodeBuild_Concepts#Privileged_Build|AWS CodeBuild Concepts - Elevated Privileges]].
 
====Service role====
 
New service role. As a name pattern, use ''project-name''-build-service-role. Eg. "themyscira-unity-build-service-role". The role will require specific permissions, depending on what the build is supposed to do. For more details on what permission the role may need, and how to grant them, see: {{Internal|AWS_CodeBuild_Concepts#Service_Role|CodeBuild Service Role}}


<span id='Allow_AWS_CodeBuild_to_modify_this_service_role_so_it_can_be_used_with_this_build_project'></span>Allow AWS CodeBuild to modify this service role so it can be used with this build project. If selected, the console configuration code updates the role so it contains the appropriate policies.
<span id='Allow_AWS_CodeBuild_to_modify_this_service_role_so_it_can_be_used_with_this_build_project'></span>Allow AWS CodeBuild to modify this service role so it can be used with this build project. If selected, the console configuration code updates the role so it contains the appropriate policies.
Line 49: Line 88:
unselect "Allow AWS CodeBuild to modify ..."
unselect "Allow AWS CodeBuild to modify ..."


Environment variables:
====Timeout====
 
====Queued timeout====
 
====Certificate====
 
====VPC====
 
====Compute====
 
====Environment variables====


For more details see: {{Internal|AWS_CodeBuild_Concepts#Environment_Variables|CodeBuild Concepts - Environment Variables}}
For more details see: {{Internal|AWS_CodeBuild_Concepts#Environment_Variables|CodeBuild Concepts - Environment Variables}}
Line 71: Line 120:


==Buildspec==
==Buildspec==
====Build specifications====


Use a buildspec file.
Use a buildspec file.


Specify name of the file, if not buildspec.yml,
Specify name of the file, if not buildspec.yml. The default is a buildspec.yml in the source code root directory.
 
For gradle modules, the following works:
 
./themyscira/buildspec.yml


==Artifacts==
==Artifacts==
Line 80: Line 135:
==Logs==
==Logs==


CloudWatch
====CloudWatch====


Select CloudWatch logs.
Select CloudWatch logs.
Line 86: Line 141:
Group Name: /playground
Group Name: /playground


Stream Name: ops-unity-build
Stream Name: themyscira-unity-build


You must enable CloudWatch logging if you want any kind of build logs, including the "console" log.
You must enable CloudWatch logging if you want any kind of build logs, including the "console" log.
Line 101: Line 156:


Start Build
Start Build
=Build an AWS CodeBuild Docker Image=
{{External|https://docs.aws.amazon.com/codebuild/latest/userguide/sample-docker-custom-image.html}}
{{External|https://docs.aws.amazon.com/codebuild/latest/APIReference/API_ProjectEnvironment.html}}
This procedure documents the process of creating a custom [[AWS_CodeBuild_Concepts#Environment_Image|environment image]].
git clone git@github.com:aws/aws-codebuild-docker-images.git
cd aws-codebuild-docker-images/ubuntu/java/openjdk-8
docker build -t aws/codebuild/java:openjdk-8 .
docker images
REPOSITORY                                                                        TAG                IMAGE ID            CREATED            SIZE
aws/codebuild/java                                                                openjdk-8          5490a2e1223f        2 minutes ago      1.6GB
{{Warn|If custom build images need to be able to interact with Docker - for creating Docker images, for example -, the Docker server [[#Amend_the_Build_Specification_to_Explicitly_Start_the_Docker_Daemon|must be started manually in the build specification]], as shown below.}}
=Amend the Build Specification to Explicitly Start the Docker Daemon=
In case of custom environment images, even those based on an AWS CodeBuild image, built as shown [[#Build_an_AWS_CodeBuild_Docker_Image|above]], the docker daemon must be started explicitly in the build specification. This has to be done even if the "[[AWS_CodeBuild_Concepts#Privileged_Build|privileged build]]" flag is enabled, the docker daemon must be started explicitly in build spec - it is not started automatically as in the case of AWS-provided environment images.
This sequence works for Ubuntu-based images:
<syntaxhighlight lang='yaml'>
version: 0.2
...
phases:
  install:
    commands:
      - nohup /usr/local/bin/dockerd --host=unix:///var/run/docker.sock --host=tcp://127.0.0.1:2375 --storage-driver=overlay2&
      - timeout 15 sh -c "until docker info; do echo .; sleep 1; done"
  build:
...
</syntaxhighlight>


=Troubleshooting=
=Troubleshooting=


{{External|https://docs.aws.amazon.com/codebuild/latest/userguide/troubleshooting.html}}
Generic troubleshooting advice: {{External|https://docs.aws.amazon.com/codebuild/latest/userguide/troubleshooting.html}}
 
==Failed Status==
 
For more details, go to: Build Project -> Build History -> Build Run -> Phase details
 
==ECR AccessDeniedException==
 
An error occurred (AccessDeniedException) when calling the GetAuthorizationToken operation: User: arn:aws:iam::673499572719:user/codebuild-p2 is not authorized to perform: ecr:GetAuthorizationToken on resource: *
 
Add ecr:GetAuthorizationToken for the user in question to the build role. These are extremely lax permissions, it could be further tightened up:
 
<syntaxhighlight lang='javascript'>
{
...
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "ecr:*",
"Resource": "*"
}
</syntaxhighlight>

Latest revision as of 04:19, 15 March 2019

Internal

Create a Build Project

Project configuration

Project name

Description

Build badge

For more details about the build badge, see Select AWS CodeBuild Concepts - Build Badge.

Tags

Source

Source provider

GitHub

Repository

Repository in my GitHub account.

GitHub repository

Connection Status

Additional configuration

Git clone depth

Clone depth 1

Build Status

Primary source webhook events

Webhook - Rebuild every time a code change is pushed to this repository. Enable and experiment with that.

Environment

Environment image

The options are "Managed image" or "Custom image". A managed image is always preferable if it provides all the build functionality you need. However, if you need extra functionality, you can build and use a custom image.

Managed image configuration:

Operating system: Ubuntu

Runtime: Java

Runtime version: aws/codebuild/java:openjdk-8


The Java runtime may have an obsolete Gradle version. If that is the case, initialize your project with gradlew and configure the buildspec file to use ./gradlew to build the project.

Custom image configuration:

Environment type: Linux

Image registry: Amazon ECR

ECR account: My ECR account

Amazon ECR Repository: "com.example/unity/unity-build-image"

Amazon ECR image: latest

Image pull credentials: Project service role. This is the service role associated with this build project.

Privileged

This flag should be enabled if the build project builds Docker images. More details about elevated privileges: AWS CodeBuild Concepts - Elevated Privileges.

Service role

New service role. As a name pattern, use project-name-build-service-role. Eg. "themyscira-unity-build-service-role". The role will require specific permissions, depending on what the build is supposed to do. For more details on what permission the role may need, and how to grant them, see:

CodeBuild Service Role

Allow AWS CodeBuild to modify this service role so it can be used with this build project. If selected, the console configuration code updates the role so it contains the appropriate policies.

If you get:

The policy's default version was not created by enhanced zero click role creation or was not the most recent version created by enhanced zero click role creation.

unselect "Allow AWS CodeBuild to modify ..."

Timeout

Queued timeout

Certificate

VPC

Compute

Environment variables

For more details see:

CodeBuild Concepts - Environment Variables

No user environment variables can start with CODEBUILD_

Name: CODEBUILD_SOURCE_VERSION, Value: "unity" (for GitHub, this is the branch name to be uses)

For personal account:

  • Name: AWS_ACCESS_KEY Value: ...
  • Name: SK Value: TBD

For shared build account:

  • Name: AWS_ACCESS_KEY_ID
  • Name: AWS_SECRET_ACCESS_KEY

Additional Configuration

Timeout: 20 minutes.

Buildspec

Build specifications

Use a buildspec file.

Specify name of the file, if not buildspec.yml. The default is a buildspec.yml in the source code root directory.

For gradle modules, the following works:

./themyscira/buildspec.yml

Artifacts

Logs

CloudWatch

Select CloudWatch logs.

Group Name: /playground

Stream Name: themyscira-unity-build

You must enable CloudWatch logging if you want any kind of build logs, including the "console" log.

Run a Build Project

Run Build

Source version: unity

Environment variables override:

SK:

Start Build

Build an AWS CodeBuild Docker Image

https://docs.aws.amazon.com/codebuild/latest/userguide/sample-docker-custom-image.html
https://docs.aws.amazon.com/codebuild/latest/APIReference/API_ProjectEnvironment.html

This procedure documents the process of creating a custom environment image.

git clone git@github.com:aws/aws-codebuild-docker-images.git
cd aws-codebuild-docker-images/ubuntu/java/openjdk-8 
docker build -t aws/codebuild/java:openjdk-8 .

docker images
REPOSITORY                                                                        TAG                 IMAGE ID            CREATED             SIZE
aws/codebuild/java                                                                openjdk-8           5490a2e1223f        2 minutes ago       1.6GB

If custom build images need to be able to interact with Docker - for creating Docker images, for example -, the Docker server must be started manually in the build specification, as shown below.

Amend the Build Specification to Explicitly Start the Docker Daemon

In case of custom environment images, even those based on an AWS CodeBuild image, built as shown above, the docker daemon must be started explicitly in the build specification. This has to be done even if the "privileged build" flag is enabled, the docker daemon must be started explicitly in build spec - it is not started automatically as in the case of AWS-provided environment images.

This sequence works for Ubuntu-based images:

version: 0.2
...
phases:
  install:
    commands:
      - nohup /usr/local/bin/dockerd --host=unix:///var/run/docker.sock --host=tcp://127.0.0.1:2375 --storage-driver=overlay2&
      - timeout 15 sh -c "until docker info; do echo .; sleep 1; done"
  build:
...

Troubleshooting

Generic troubleshooting advice:

https://docs.aws.amazon.com/codebuild/latest/userguide/troubleshooting.html

Failed Status

For more details, go to: Build Project -> Build History -> Build Run -> Phase details

ECR AccessDeniedException

An error occurred (AccessDeniedException) when calling the GetAuthorizationToken operation: User: arn:aws:iam::673499572719:user/codebuild-p2 is not authorized to perform: ecr:GetAuthorizationToken on resource: *

Add ecr:GetAuthorizationToken for the user in question to the build role. These are extremely lax permissions, it could be further tightened up:

{
 ...
 "Sid": "VisualEditor1",
 "Effect": "Allow",
 "Action": "ecr:*",
 "Resource": "*"
}