Nort Concepts

Internal

• nort User Manual

Overview

nort (Nova Ordis Release Tool) is a tool that implements a specific software release workflow. It coordinates Maven, git, GitHub, and interacts with various other resources associated with a software project release cycle. In this respect, it is opinionated. It enforces its opinions by driving various resources and tools (Maven, GitHub, artifact repositories, target systems upon the publicly-consumable artifacts are deployed, etc.) to interact in specific ways. nort is driven by a very simple set of top-level commands: "release", "build", "publish" and "install", which trigger sequence of operations described below. Those sequences of operations involve relatively complex - and distributed - interactions. In most cases, you will only need to use "release". One of the most useful features is the "rollback" functionality: if one of the last steps in a complex release sequence fails, nort rollbacks the effects of all previously executed steps, leaving the work area and various dependent resource in the same state they were before the failed release process started. The rollback feature is implicit, and automatic. For more details see the "Rollback" section.

Dot Version

A dot version is a label that marks a specific point in time in the evolution of a software project. It contains major, minor and patch components: "1.2.3". It does not contain a "SNAPSHOT" fragment. A dot version associated with an artifact indicates that the artifact was built based on source code that passed all the tests and it is ready for public consumption. The copy of the source tree of a project under active development in a work area is never associated with a dot version, as the project is "open for changes". For more details about this convention, see "Current Version" section.

Snapshot Version

A snapshot version is a label that marks a specific point in time in the evolution of a software project, and it primarily indicates that, at that time, the functionality provided by the project is not ready for public consumption. In other words, it says that source code tree used to build the artifact from was under development at the time the build was performed. The "SNAPSHOT" level designates a snapshot version. Example: "1.2.3-SNAPSHOT-5".

A snapshot version is considered to be preceding the corresponding dot version. For example "1.2.3-SHAPSHOT-5" precedes "1.2.3", or it is older than "1.2.3", because we first work on a snapshot of a specific version before we release that version.

A snapshot-versioned artifact should never be used outside the development process. Snapshot-versioned artifacts may be routinely released, but they must only be used when dependent projects under development, themselves labeled with a snapshot version, require new functionality or defect fixes from their dependencies. In that situation, the dependency releases a snapshot release, the dependent updates its metadata to point to the newly released dependency snapshot release, and its development continues. When the dependent is ready for a dot release, first the dependencies are "dot released" and then the dependent is "dot released". nort provides logic to do the underlying bookkeeping and automate these workflows.

Current Version

The current version is the version designating the code base in the work area of a project. For Maven-built projects, it is the <version> information written into the POM file present in the work area.

Under the conventions enforced by nort, the current version is always a snapshot version, as projects are continuously evolving and they are in principle open for new functionality and defect fixes.

A version becomes a dot version during the release process. However, a dot version never lasts in a work area for more than a brief time interval. The dot version is used to build the corresponding artifacts (sources and binaries), which are then published in their corresponding repositories, as part of the release process, and possibly even deployed "in production" - in places where they are used. If all goes well, the current version is immediately upgraded to the next logical SNAPSHOT. If something goes wrong, the current version remains set to the SNAPSHOT that failed to become a dot version, until the developer fixes the problem and repeats, successfully, the release process.

The "current.version" Runtime Variable

The current version is exposed to the runtime as "current.version" runtime variables. For more information about the runtime variables, see the "Runtime Variables" section.

Release

A release consists in a sequence of operations - interactions with the local project work area, the build system metadata, and various external source code and binary repositories - which end with the generation of a set of release artifacts. Depending of the release type and the options used when the release was triggered, the set of released artifacts are different, and end up in different places. For more details, see the "sequences" section.

Release Types

Custom

For syntax details see the release command.

State Change

A release sequence may or may not change the state of the project. By "changing the state" we understand modification of project metadata (such as incrementing the version information in the project descriptor or pom.xml), modification of the code base as result of the release sequence, installation of various artifacts in various repositories. Some actions, even if they are executed and modify state on disk, do not count as "changing state". For example, the fact that we ran tests or we modified files in the target directory does not count as state change.

Sequences

The Release Sequence

The "release" sequence is the most complex, and all-encompassing. The full release sequence involves release qualification, which is aimed at failing fast if needed, project metadata updates, a build sequence, an publish sequence and a installation sequence. The user can instruct nort to end the process at a certain intermediate point, or start it from an intermediate point. For example, an already published artifact can be installed (or deployed) in production by triggering an install sequence.

The Qualification Sequence

The qualification sequence is aimed at establishing quickly if all conditions for a successful release are in place. We try to fail as fast as we can, and not wait until later in the process. During the qualification sequence, nort performs any metadata changes required by the version being released and insures the code base is ready for release.

The qualification sequence consists in the following steps:

1. Check Pre-Conditions

• Make sure the current version is a snapshot version. The release fails if it encounters a dot version in the work area.
• If the project installs artifacts and it was configured with a way of obtaining the currently installed version, it attempts to obtain the currently installed version and the qualification sequence fails if the installed version is equal or newer than the version being released. This represent the enforcement of the Installation Version Constraints.
• Verify that the custom version, if we're doing a custom release, is newer than the current version.
• If the release we're building corresponds to a dot version (release type is major, minor or patch), or it is a custom label release, increment the project version metadata, so we can perform the qualification with the correct version. If we're performing a snapshot release, the correct version is already in place.

2. Run Tests

Execute all tests.

• If at least one test fails, report and fail. The work area is guaranteed to be left in the exact state it was found before the release sequence was initiated.
• If the tests pass, pass the control to the next sequence, which is usually the build sequence, or exit if there are no follow up sequence. This would be unusual.

The Build Sequence

The build sequence insures that the project artifacts corresponding to the version being released are built, and placed in a conventional staging section of the work area. For Maven-built projects, that is the ./target directory.

The build sequence is usually triggered as part of the release sequence. It can also be initiated on its own. TODO: more details here.

The build sequence consists in the following steps:

1. Build the artifacts (with or without running the tests, depending on whether the sequence before already ran them).
2. If the build fails, report and fail. The work area is guaranteed to be left in the exact state it was found before the release sequence was initiated.
3. Place the artifacts in a staging directory in the work area (usually the target directory).
4. Pass the control to the next sequence, which is usually the publish sequence or exit if there is no follow up sequence.

The Publish Sequence

As part of the publish sequence, the artifacts built by the build sequence are published in various repositories. The results are similar to those of Maven's "install" phase, and in most cases, nort will indeed copy the artifacts into the local Maven repository. However, the publish sequence does other things as well: it commits the code corresponding to the release into source repositories, it may send a release announcement, etc.

The publish sequence is usually triggered as part of the release sequence. Within the release sequence, it succeeds the build sequence. It can also be initiated on its own. TODO: more details here.

The publish sequence consists in the following steps:

1. Move the artifacts from the work area to the corresponding artifact repositories. Usually, JARs are installed under their corresponding artifact directories and binary releases are installed under their corresponding "release" modules, both in the local Maven repository. For more details about artifacts and what repositories are they installed into, see "Release Artifacts" section.
2. Add the code in the local repository
3. Commit the changed, using ... log message.
4. Tag the local repository. Use 'release-<version>' tag, unless the release is a snapshot, in which case the repository won't be tagged.
5. Optionally, it pushes the local repository changes to upstream repositories. GitHub is an option.

TODO: review https://kb.novaordis.com/index.php/Public_Release_Procedure_for_Projects in the light of release sequence modifications.

GitHub Public Release

TODO: implement functionality that will automate the https://github.com/NovaOrdis/novaordis-utilities/releases GitHub release.

The Install Sequence

The install sequence takes the corresponding artifacts from the artifact repository and installs (or deploys) them wherever they are supposed to be used. As an example, a command line utility can be installed on a local system under the /usr/local directory, or into an Amazon EC2 environment. A web application packaged as an EAR can be deployed on the production web site.

For a Java library, the installation sequence is a noop, as the previously run publish sequence already published it into the local repository. Also address the case where we want the JAR published into a remote repository. Should be this considered "publishing" or "installation"?

Embedded Installation Logic for Binary Distributions

For binary distributions, nort will assume they follow Embedded Installation Logic for Binary Distributions pattern and will attempt to execute the installation logic embedded within the binary distribution. This is the procedure to configure a project to automate installation with nort:

Configure the Project to be Installed with nort

Installation Version Constraints

Within the context of the versioning model enforced by nort, the artifact being released must never overwrite the current installed artifact during the installation process. If the version being released is equal or older than the installed version, the installation and consequently the release must fail. nort makes it easy to always create a new successive version (dot or snapshot). If there is a conflict, always create a newer version.

The Completion Sequence

The local workarea version metadata is tentatively incremented according to the rules described in the "Release Types" section.

Release Artifacts

A release may produce several kinds of artifacts.

Binary Distributions

Binary distributions are usually ZIP file containing everything an application needs in order to be installed on a new system. The binary distribution include libraries, wrapper scripts, release notes, documentation and possibly other files.

It is required that a binary distribution ZIP file contains just one top-level directory whose name follows the pattern: "<application-name>-<version>". nort will rely on this convention.

The recommended technique to build complex binary distributions is to add a dedicated "release" module to the project and let the Maven assembly plug-in to handle the process. The details are described in the "Building a Maven Complex Release Artifact" article. If a project has a binary distribution that is built as Maven assembly, nort will install it in the local Maven repository, under the "release" project module. Note that because <finalName> has no effect on the artifact installed in the repository, the name of the repository file will be "release-<version>.zip". Binary distributions may be built using a mechanism other than Maven - simply zipping files is a perfectly valid way of doing it - but irrespective to how the artifact is built, the publish sequence ends with "publishing" them them in some form of binary repository.

Runtime Variables

nort maintains a set of runtime variables, which are accessible as Java system properties in the JVM. These variables can also be used in the configuration file, and external command templates. They will be correctly translated at the time of the execution. The runtime exposes both formats (dot-separated system properties and underscore separated shell variables).

• current.version, current_version

Rollback

One of the most useful features is the "rollback" functionality: if one of the last steps in a complex release sequence fails, nort rollbacks the effects of all previously executed steps, leaving the work area and various dependent resource in the same state they were before the failed release process started. The rollback feature is implicit, and automatic.

Per-Project Configuration

A project can be configured with nort metadata, which will be used during the release process. The configuration metadata is available under ${PROJECT_HOME}/.nort directory.

The default configuration file nort will look for in its configuration directory is project.yaml or project.yml. project.yaml and project.yml can be used interchangeably.

Example:

#
# release configuration 
#
qualification:
  os.command.to.get.installed.version: nort version

build:

publish:
  local.artifact.repository.root: ${M2}

install:
  installation.directory: ${RUNTIME_DIR}

Configuration elements supported so far:

• Qualification Sequence:
  • "os.command.to.get.installed.version": the OS command to be executed by nort to get the version of the currently installed artifact of the project. nort expects the version to be sent to stdout and it will apply heuristics to extract it. It will warn if this command is not configured.

• Install Sequence:
  • "installation.directory": The directory to unzip the installation artifact into. Once unzipped, the embedded installation logic will take over. Environment variable, such as ${RUNTIME_DIR} are supported.

New Project Initialization

The command that carries out a new project initialization is nort initialize.

TODO:

• Verify that the POM of the new project contains the definition of maven.build.timestamp.format and release_date, as required by "Nova Ordis Utilities Version Metadata Handling" instructions. If it does not, inject it.