Nort Concepts

Internal

• nort User Manual

Overview

nort (Nova Ordis Release Tool) is a tool that implements a specific software release workflow. It coordinates Maven, git, GitHub, and interacts with various other resources associated with a software project release cycle. In this respect, it is opinionated. It enforces its opinions by driving various resources and tools (Maven, GitHub, artifact repositories, target systems upon the publicly-consumable artifacts are deployed, etc.) to interact in specific ways. nort is driven by a very simple set of top-level commands: "release", "build", "publish" and "install", which trigger sequence of operations described below. Those sequences of operations involve relatively complex - and distributed - interactions. In most cases, you will only need to use "release". One of the most useful features is the "rollback" functionality: if one of the last steps in a complex release sequence fails, nort rollbacks the effects of all previously executed steps, leaving the work area and various dependent resource in the same state they were before the failed release process started. The rollback feature is implicit, and automatic. For more details see the "Rollback" section.

Dot Version

A dot version is a label that marks a specific point in time in the evolution of a software project. It contains major, minor and patch components: "1.2.3". It does not contain a "SNAPSHOT" fragment. A dot version associated with an artifact indicates that the artifact was built based on source code that passed all the tests and it is ready for public consumption. The copy of the source tree of a project under active development in a work area is never associated with a dot version, as the project is "open for changes". For more details about this convention, see "Current Version" section.

Snapshot Version

A snapshot version is a label that marks a specific point in time in the evolution of a software project, and it primarily indicates that, at that time, the functionality provided by the project is not ready for public consumption. In other words, it says that source code tree used to build the artifact from was under development at the time the build was performed. The "SNAPSHOT" level designates a snapshot version. Example: "1.2.3-SNAPSHOT-5".

A snapshot version is considered to be preceding the corresponding dot version. For example "1.2.3-SHAPSHOT-5" precedes "1.2.3", or it is older than "1.2.3", because we first work on a snapshot of a specific version before we release that version.

A snapshot-versioned artifact should never be used outside the development process. Snapshot-versioned artifacts may be routinely released, but they must only be used when dependent projects under development, themselves labeled with a snapshot version, require new functionality or defect fixes from their dependencies. In that situation, the dependency releases a snapshot release, the dependent updates its metadata to point to the newly released dependency snapshot release, and its development continues. When the dependent is ready for a dot release, first the dependencies are "dot released" and then the dependent is "dot released". nort provides logic to do the underlying bookkeeping and automate these workflows.

Current Version

The current version is the version designating the code base in the work area of a project. For Maven-built projects, it is the <version> information written into the POM file present in the work area.

Under the conventions enforced by nort, the current version is always a snapshot version, as projects are continuously evolving and they are in principle open for new functionality and defect fixes.

A version becomes a dot version during the release process. However, a dot version never lasts in a work area for more than a brief time interval. The dot version is used to build the corresponding artifacts (sources and binaries), which are then published in their corresponding repositories, as part of the release process, and possibly even deployed "in production" - in places where they are used. If all goes well, the current version is immediately upgraded to the next logical SNAPSHOT. If something goes wrong, the current version remains set to the SNAPSHOT that failed to become a dot version, until the developer fixes the problem and repeats, successfully, the release process.

The "current.version" Runtime Variable

The current version is exposed to the runtime as "current.version" runtime variables. For more information about the runtime variables, see the "Runtime Variables" section.

Release

A release consists in a sequence of operations - interactions with the local project work area, the build system metadata, and various external source code and binary repositories - which end with the generation of a set of release artifacts. Depending of the release type and the options used when the release was triggered, the set of released artifacts are different, and end up in different places. For more details, see the "sequences" section.

Release Types

Custom

For syntax details see the release command.

State Change

A release sequence may or may not change the state of the project. By "changing the state" we understand modification of project metadata (such as incrementing the version information in the project descriptor or pom.xml), modification of the code base as result of the release sequence, installation of various artifacts in various repositories. Some actions, even if they are executed and modify state on disk, do not count as "changing state". For example, the fact that we ran tests or we modified files in the target directory does not count as state change.

Sequences

The Release Sequence

The "release" sequence is the most complex, and all-encompassing. The full release sequence involves release qualification, which is aimed at failing fast if needed, project metadata updates, a build sequence, an publish sequence and a installation sequence.

The user can instruct nort to end the release sequence at a certain intermediate point, to skip certain steps (if --no-install is used, the release will perform the entire release sequence without installation), or start it from an intermediate point. For example, an already published artifact can be installed (or deployed) in production by triggering the install sequence.

The Qualification Sequence

The qualification sequence is aimed at establishing quickly if all conditions for a successful release are in place. We try to fail as fast as we can, and not wait until later in the process. During the qualification sequence, nort performs any metadata changes required by the version being released and insures the code base is ready for release.

The qualification sequence consists in the following steps:

1. Check Pre-Conditions

• Make sure the current version is a snapshot version. The release fails if it encounters a dot version in the work area.
• If the project installs artifacts and it was configured with a way of obtaining the currently installed version, it attempts to obtain the currently installed version and the qualification sequence fails if the installed version is equal or newer than the version being released. This represent the enforcement of the Installation Version Constraints.
• Verify that the custom version, if we're doing a custom release, is newer than the current version.
• If the release we're building corresponds to a dot version (release type is major, minor or patch), or it is a custom label release, increment the project version metadata, so we can perform the qualification with the correct version. If we're performing a snapshot release, the correct version is already in place.

2. Run Tests

Execute all tests, unless --no-tests is specified on command line.

• If at least one test fails, report and fail. The work area is guaranteed to be left in the exact state it was found before the release sequence was initiated.
• If the tests pass, pass the control to the next sequence, which is usually the build sequence, or exit if there are no follow up sequence. This would be unusual.

The Build Sequence

The build sequence insures that the project artifacts corresponding to the version being released are built, and placed in a conventional staging section of the work area. For Maven-built projects, that is the ./target directory.

The build sequence is usually triggered as part of the release sequence. It can also be initiated on its own.

The build sequence consists in the following steps:

1. Build the artifacts (with or without running the tests, depending on whether the sequence before already ran them).
2. If the build fails, report and fail. The work area is guaranteed to be left in the exact state it was found before the release sequence was initiated.
3. Place the artifacts in a staging directory in the work area (usually the target directory).
4. Pass the control to the next sequence, which is usually the publish sequence or exit if there is no follow up sequence.

The Publish Sequence

As part of the publish sequence, the artifacts built by the build sequence are published in various code and artifact repositories. The results are similar to those of Maven's "install" and "deploy" phases, and in most cases, nort will indeed copy the artifacts into the local Maven repository and the configured remote hosted repository. However, the publish sequence does other things as well: it commits the code corresponding to the release into source repositories and tags the source repositories appropriately, it may send a release announcement, etc.

The publish sequence is usually triggered as part of the release sequence. Within the release sequence, it succeeds the build sequence. It can also be initiated on its own.

The publish sequence consists in the following steps:

1. Copy the artifacts built in the work area to the corresponding local artifact repository. Usually, JARs are installed under their corresponding artifact directories and binary releases are installed under their corresponding "release" modules in the local Maven repository. For more details about artifacts and what repositories are they installed into, see "Release Artifacts" section.
2. Optionally, copy the binary artifacts to the configured remote hosted repository. This step only happens if 1) the release is a dot release and 2) the POM for the project has a distribution management section and a correctly configured remote hosted repository. For more details on how to configure publishing to a remote artifact repository, see Publishing to Remote Artifact Repositories below.
3. Commit the changes to the local code repository (for git, that also needs "adding" the changes).
4. Tag the local repository. The default tag format is 'release-<version>' tag, unless the release is a snapshot, in which case the repository won't be tagged. The tag format can be configured using the "release.tag" configuration element. For the context of why this is useful see Multiple Release Streams in the Same Repository.
5. Optionally, it pushes the local repository changes to upstream repositories, unless the --no-push option is used. GitHub is an option.

TODO: review https://kb.novaordis.com/index.php/Public_Release_Procedure_for_Projects in the light of release sequence modifications.

Publishing to Remote Artifact Repositories

Nort will automatically publish dot releases to a remote artifact repository, if the remote artifact repository is configured in POM as follows:

<distributionManagement>
    <repository>
        <id>novaordis-nexus</id>
        <url>https://maven.apps.openshift.novaordis.io/repository/maven-releases/</url>
    </repository>
</distributionManagement>

For more details on how to configure distribution management, see:

pom.xml <distributionManagement>

There are situations when the remote repository is configured with self-signed certificate. The Java SSL layer Maven delegates to when establishing the connection will refuse to connect unless configured with a truststore that contains the remote server's certificate. Maven can be configured with the remote server's certificate by importing it into a local truststore, as described here:

Configure a Java HTTP Client to Accept Self-Signed Certificates

Then Nort must be configured to use that trustore, as described here:

Configure Nort with a local truststore

GitHub Public Release

TODO: implement functionality that will automate the https://github.com/NovaOrdis/novaordis-utilities/releases GitHub release.

Multiple Release Streams in the Same Repository

Because we have the option to customize the release tag format, we can enforce conventions that would allow us to maintain multiple and independent release streams in the same repository.

The Install Sequence

The install sequence takes the corresponding artifacts from the artifact repository and installs (deploys) them wherever they are supposed to be used. As an example, a command line utility can be installed on a local system under the /usr/local directory, or into an Amazon EC2 environment. A web application packaged as an EAR can be deployed on the production web site.

For a Java library, the installation sequence is a noop, as the previously run publish sequence already published it into the local repository. Also address the case where we want the JAR published into a remote repository. Should be this considered "publishing" or "installation"?

If --no-install option is used, the install sequence is skipped.

Embedded Installation Logic for Binary Distributions

For binary distributions, nort will assume they follow Embedded Installation Logic for Binary Distributions pattern and will attempt to execute the installation logic embedded within the binary distribution. This is the procedure to configure a project to automate installation with nort:

Configure the Project to be Installed with nort

Installation Version Constraints

Within the context of the versioning model enforced by nort, the artifact being released must never overwrite the current installed artifact during the installation process. If the version being released is equal or older than the installed version, the installation and consequently the release must fail. nort makes it easy to always create a new successive version (dot or snapshot). If there is a conflict, always create a newer version.

The Completion Sequence

The local workarea version metadata is tentatively incremented according to the rules described in the "Release Types" section.

Release Artifacts

A release may produce several kinds of artifacts.

Binary Distributions

Binary distributions are usually ZIP file containing everything an application needs in order to be installed on a new system. The binary distribution include libraries, wrapper scripts, release notes, documentation and possibly other files.

It is required that a binary distribution ZIP file contains just one top-level directory whose name follows the pattern: "<application-name>-<version>". nort will rely on this convention.

The recommended technique to build complex binary distributions is to add a dedicated "release" module to the project and let the Maven assembly plug-in to handle the process. The details are described in the "Building a Maven Complex Release Artifact" article. If a project has a binary distribution that is built as Maven assembly, nort will install it in the local Maven repository, under the "release" project module. Note that because <finalName> has no effect on the artifact installed in the repository, the name of the repository file will be "release-<version>.zip". Binary distributions may be built using a mechanism other than Maven - simply zipping files is a perfectly valid way of doing it - but irrespective to how the artifact is built, the publish sequence ends with "publishing" them them in some form of binary repository.

Variables

Runtime Variables

nort maintains a set of runtime variables, which are accessible as Java system properties in the JVM. These variables can also be used in the configuration file, and external command templates. They will be correctly de-referenced at the time of the execution. The runtime exposes both formats (dot-separated system properties and underscore separated shell variables).

• current.version, current_version

Environment Variables

nort will correctly de-reference the environment variables, if specified in configuration file values.

Rollback

One of the most useful features is the "rollback" functionality: if one of the last steps in a complex release sequence fails, nort rollbacks the effects of all previously executed steps, leaving the work area and various dependent resource in the same state they were before the failed release process started. The rollback feature is implicit, and automatic.

Per-Project Configuration

A project can be configured with nort metadata, which will be used during the release process. The configuration metadata is available under ${PROJECT_HOME}/.nort directory.

The default configuration file nort will look for in its configuration directory is project.yaml or project.yml. project.yaml and project.yml can be used interchangeably.

Example:

#
# release configuration 
#
qualification:
  os.command.to.get.installed.version: nort version

build:

publish:
  local.artifact.repository.root: ${M2}

install:
  installation.directory: ${RUNTIME_DIR}

Configuration elements supported so far:

• Qualification Sequence:
  • "os.command.to.get.installed.version": the OS command to be executed by nort to get the version of the currently installed artifact of the project. nort expects the version to be sent to stdout and it will apply heuristics to extract it. It will warn if this command is not configured.

• Install Sequence:
  • "installation.directory": The directory to unzip the installation artifact into. Once unzipped, the embedded installation logic will take over. Environment variable, such as ${RUNTIME_DIR} are supported.

New Project Initialization

The command that carries out a new project initialization is nort initialize.

TODO:

• Verify that the POM of the new project contains the definition of maven.build.timestamp.format and release_date, as required by "Nova Ordis Utilities Version Metadata Handling" instructions. If it does not, inject it.