Gradle Dependencies and Dependency Configurations TODEPLETE

External

• https://docs.gradle.org/current/userguide/dependency_management_for_java_projects.html

Internal

• Gradle Concepts

Overview

One of the most useful Gradle features is its dependency management capability. Dependency management is a technique for declaring, resolving and using dependencies required by a project in an automated fashion.

Gradle Dependency Concepts

Dependency

A dependency is an artifact the project needs. Dependencies are usually located in a repository. A dependency can be thought of as a pointer to another piece of software. Gradle supports several types of dependencies:

Module Dependency

Gradle Docs - Module Dependencies

The module dependency is the most common form of dependency. It refers to an artifact published by a Gradle module (or an equivalent of Gradle, such as Maven) in a repository. They are declared similar to:

dependencies {
    implementation '<group-id>:<artifact-id>:<version>'
 }

More details are available in the Declaring Module Dependencies section, below.

File Dependency

Gradle Docs - File Dependencies

A file dependency allows specification of a file present on an accessible filesystem as dependency, without first adding it to a repository. They are declared similar to:

dependencies {
    implementation files('libs/a.jar')
 }

More details are available in the Declaring File Dependencies section, below.

Project Dependency

Gradle Docs - Project Dependencies

A project dependency expresses the dependency on artifacts produced by other project that is part of the same multi-project build. The project dependency is declared similar to:

dependencies {
    implementation project(':subproject-A')
}

More details are available in the Declaring Project Dependencies section, below.

Gradle Distribution-Specific Dependency

https://docs.gradle.org/current/userguide/dependency_types.html#sub:api_dependencies

Dependency Configuration

Gradle groups dependencies in, and handles them as sets, referred to as dependency configurations. A dependency configuration is a named set of dependencies, grouped together for a specific goal. For example, some dependencies should be used during the compilation phase, whereas others need to be available during the testing phase, or at runtime. The concept is somewhat similar to the Maven dependency scope. Internally, a dependency configuration is implemented as a Configuration instance. Each configuration is identified by a unique name. Many Gradle plugins add pre-defined configurations to the project. For example, the Java plugin adds configurations to represent the various classpaths needed for compilation, testing, etc. A dependency configuration has three main purposes:

• To declare dependencies: plugins use dependency configurations to allow build authors to declare what other sub-projects or external artifacts are needed during the execution of tasks defined by the plugin.
• To resolve dependencies: plugins use dependency configurations to find and download the dependencies that are needed during the build
• To expose artifacts for consumption: plugins use dependency configurations to define what artifacts it generates. Configurations are using during the artifact publishing process.

Additionally to the dependency configurations introduced by plugins, custom configuration may be defined. A configuration can extend other configurations to form an configuration hierarchy. Child configurations inherit the whole set of dependencies declared for any of its parent configurations.

The dependency configurations of a project can displayed as shown in:

Dependency and Dependency Configuration Info

The dependency configurations of a project be configured in the project's build.gradle, within the configurations{...} script block, as shown below:

Configuring Dependency Configurations

Example of a typical set of dependency configuration for a Java project:

compile - Dependencies for source set 'main' (deprecated, use 'implementation ' instead).
implementation - Implementation only dependencies for source set 'main'.
runtime - Runtime dependencies for source set 'main' (deprecated, use 'runtimeOnly ' instead).
runtimeOnly - Runtime only dependencies for source set 'main'.
testCompile - Dependencies for source set 'test' (deprecated, use 'testImplementation ' instead).
testImplementation - Implementation only dependencies for source set 'test'.
annotationProcessor - Annotation processors and their dependencies for source set 'main'.
apiElements - API elements for main.
archives - Configuration for archive artifacts.
bootArchives - Configuration for Spring Boot archive artifacts.
compileClasspath - Compile classpath for source set 'main'.
compileOnly - Compile only dependencies for source set 'main'.
default - Configuration for default artifacts.
docker
jacocoAgent - The Jacoco agent to use to get coverage data.
jacocoAnt - The Jacoco ant tasks to use to get execute Gradle tasks.
runtimeClasspath - Runtime classpath of source set 'main'.
runtimeElements - Elements of runtime for main. 
testAnnotationProcessor - Annotation processors and their dependencies for source set 'test'.
testCompileClasspath - Compile classpath for source set 'test'.
testCompileOnly - Compile only dependencies for source set 'test'.
testRuntime - Runtime dependencies for source set 'test' (deprecated, use 'testRuntimeOnly ' instead).
testRuntimeClasspath - Runtime classpath of source set 'test'.
testRuntimeOnly - Runtime only dependencies for source set 'test'.

For more details on the dependency configurations introduced by the Java plugin, see:

Java Plugin - Dependency Configurations

Transitive Dependency

A transitive dependency is a dependency of a dependency of a module. The immediate dependencies are declared in a module's metadata, and the build system is supposed to automatically resolve the dependency graph.

TODO: https://docs.gradle.org/current/userguide/managing_transitive_dependencies.html.

Gradle offers facilities to inspect the transitive dependency graph. Transitive dependency resolution is enabled by default, but it can be turned off when the dependency is declared, as shown here.

Dynamic Version

An aggressive approach for consuming dependencies is to declare that a module depends on the latest version of a dependency, or the latest version of a version range. This approach is called dynamic versioning. By default, Gradle caches dynamic versions of dependencies for 24 hours. Within this time frame, Gradle does not try to resolve newer versions from the declared repositories. This threshold can be configured. The build.gradle syntax for dynamic versioning is:

dependencies {
    implementation 'org.slf4j:slf4j-api:1.2.+'
}

Changing Versions or Snapshots

Gradle Docs - Declaring Dependencies with Changing Version

What Maven refers to as snapshots are known to Gradle as changing versions. A changing version indicates that the respective feature is still under development and hasn't released a stable version for general availability yet. In both Maven and Gradle, changing versions contain the suffix -SNAPSHOT. A snapshot version always points at the latest artifact published.

A dependency on a changing version can be declared by performing both of the following configurations:

1. Listing the corresponding snapshot repository among the available repositories.
2. Declaring the dependency with the -SNAPSHOT suffix:

...

repositories {
    mavenCentral()
    ...
    maven {
        url = "s3://my-repo/snapshots"
        ...
    }
}
...
dependencies {
    ...
    implementation group: 'com.example', name: 'something', version: '1.0.0-SNAPSHOT'
    ...
}

The presence of "-SNAPSHOT" suffix in the dependency version string is sufficient to designate that dependency as "changing". Gradle will try to pull it according to its resolution policy, even if the dot section of the version string does not change.

It is possible to declare that a version is changing even if the version string does not contains "-SNAPSHOT". Gradle provides a "changing" keyword that can be used in the dependency declaration, as shown below. Note that this is not a common case and normally there should be no need to use "changing:". For more details see setChanging(true).

implementation group: 'com.example', name: 'something', version: '1.0.0', changing: true

By default, Gradle caches changing versions for 24 hours. Within this timeframe, Gradle does not contact any of the declared remote repositories for newer snapshot versions. This default setting might be too long for intense development, when a specific artifacts changes and it is being published frequently. The threshold can be configured, so remote repositories can be checked more frequently. For details, see Configuring Cached Snapshots Time to Live below. Alternatively, the cached snapshots can be forcedly updated with every execution of a build, using --refresh-dependencies on command line.

Dependency Resolution

At runtime, Gradle will attempt to locate declared dependencies needed for the task being executed in the repositories associated with the project. This process is called dependency resolution. The dependencies may need downloading form a removing repository, retrieval from a local repository or building another project, in case of a multi-project build whose sub-projects declare dependencies on another sub-projects. A specific dependency may find itself in the dependency graph because it was declared in the build script or it is dependency of a declared dependency - a transitive dependency. The dependency resolution works as follows:

• Given a dependency, Gradle attempts to resolve the dependency by searching for the module the dependency points at. Each repository is inspected in order. Depending on the type of repository, Gradle looks for the metadata file describing the module or directly for the default artifact files, usually a JAR.
  • If the dependency is declared as a dynamic version, Gradle will resolve this to the highest available concrete version in the repository.
  • If the module metadata is a POM file that has a parent POM, Gradle will recursively attempt to resolve each of the parent modules for the POM.
• Once each repository has been inspected for the module, Gradle will chose the "best" one using the following criteria:
  • For a dynamic version, a "higher" concrete value is preferred to a "lower" version.
  • Modules declared by a module metadata file are preferred over modules that have an artifact file only.
  • Modules from earlier repositories are preferred over modules in later repositories
  • When the dependency is declared by a concrete version and a module metadata file is found in a repository, there is no need to continue searching later repositories. The remainder of the process is short-circuited.
• All of the artifacts for the module are then requested from the same repository that was chosen in the process described above.
• If none of the dependency artifacts can be resolved, the build fails.
• Once a dependency is resolved, Gradle stores into a local cache, referred to as the dependency cache.

Under certain conditions, we may want to tweak the dependency resolution mechanism, and that is possible.

Resolution Rule

Gradle Docs - Customizing Dependency Resolution Behaivor

A resolution rule influences the behavior of how a dependency is resolved. Resolution rules are defined as part of the build logic.

Dependency Constraints

Gradle allows specifying conditions dependencies must meet to qualify as valid dependency for the project. These conditions are referred to as dependency constraints. A dependency constraint defines requirements that need to be met by a module to make it a valid result for a dependency resolution process. A recommended practice for large projects is to declare dependencies without versions and use dependency constraints for version declaration. The advantage is that dependency constrains allow you to manage versions of all dependencies, including transitive ones, in one place.

Dependency Configuration Container

Gradle Docs - Configuration Container

All dependency configurations associated with a project are maintained within a configuration container, which allows declaring and managing configurations. The configuration container is one of the project's containers. Dependency configuration is perform by declaring dependency details in the build.gradle dependencies{...} script block, which applies the configuration closure on the delegate configuration container instance.

Dependency and Dependency Configuration Info

The dependencies of a project and their association with various dependency configurations can be displayed with:

gradle dependencies 

The dependency configuration can also be displayed by accessing the configuration container of the project:

task displayconfigs {
    doFirst {
        println "project dependency configurations: "
        configurations.each { println '  ' + it.name }
    }
}

Declaring Dependencies

Dependencies for a project are declared in the associated build.gradle using the dedicated dependencies{...} script block. The dependencies() method executes the declared closure against the configuration container of this project, which is passed as the delegate object to the closure.

The syntax is:

dependencies {
    known-dependency-configuration-name dependency-coordinates-1, dependency-coordinates-2, ...
}

Note that the dependency configurations used to declare dependencies against must be known to the build, usually via the applied plugins. If the dependency configuration is not know, we get a build error.

Declaring Module Dependencies

The dependency coordinates for a module dependency, pulled from a repository, must be in the following format:

group-id:artifact-id:version

All declared dependencies must be available in a repository known to the project, so repositories must also be declared.

dependencies {
    api 'org.slf4j:slf4j-api:1.7.12'
    implementation 'com.google.guava:guava:23.0'
    ...
}

Dependencies can also be declared without a version, and dependency constraints can be used to manage the version, including the transitive versions.

dependencies {
    implementation 'org.slf4j:slf4j-api'
    constraints {
        implementation 'org.slf4j:slf4j-api:1.7.12'
    }
}

When declaring a dependency, a reason for the declaration can be provided in-line:

dependencies {
    implementation('org.slf4j:slf4j-api:1.7.12') {
        because 'because we like SLF4J'
    }
}

Alternative Dependency Declaration Format

dependencies {
    ...
    implementation group: 'com.example', name: 'something', version: '1.0.0-SNAPSHOT', changing: true
    ...
}

For details see above:

Declaring Dependencies on Changing Versions

Factoring Out Version Information

There are situations when multiple dependency declaration lines contain the same version information, for closely related, but different artifacts. In those situations, the version information can be factored out and declared as an extra property of the project, in a separated section. This practice improves the readability of the build file, and also simplifies reconfiguration in case of a version upgrade:

...
ext {
    slf4jVersion="1.7.23"
}
...
dependencies {
    implementation "org.slf4j:slf4j-api:${slf4jVersion}"
    implementation "org.slf4j:slf4j-log4j12:${slf4jVersion}"
}

Note that the ext closure is subject to placement restrictions, the extra property names are also subject to constraints, and when an extra property is used, the dependency coordinates should be declared in double quoted strings, to allow property resolution.

Declaring File Dependencies

Gradle Docs - Declaring a File Dependency

File dependencies are declared as follows:

dependencies {
    ...
    implementation files('libs/a.jar', 'libs/b.jar')
    implementation fileTree(dir: 'libs', include: '*.jar')
    ...
}

Declaring Project Dependencies

Gradle Docs - Declaring a Project Dependency

Dependencies between the sub-projects of the same multi-project build must be explicitly declared. If classes belonging to "subproject-B" depend on classes in "subproject-A", then in the build.gradle of subproject-B we must add:

dependencies {
    ...
    implementation project(':subproject-A')
    ...
}

More details:

Inter-Project Dependencies

Turning Off Transitive Dependency Resolution

Transitive dependency resolution can be turned off as follows:

dependencies {
    implementation('org.hibernate:hibernate:3.0.5') {
        transitive = false
    }
}

Forcing Source and Javadoc Download

Apparently, we need to use the "eclipse" or "idea" plugin and configure it to download sources and javadocs. For more details see:

Downloading Sources and Javadoc with Gradle IDEA Plugin

The alternative is to specify a classifier in the dependency declaration:

implementation group:'io.novaordis', name: 'example', version: '1.0.0', classifier: 'sources'
implementation group:'io.novaordis', name: 'example', version: '1.0.0', classifier: 'javadoc'

Equivalent configuration:

implementation 'io.novaordis:example:1.0.0:sources'
implementation 'io.novaordis:example:1.0.0:javadoc'

Configuring Dependency Configurations

The configurations{...} script block applies the declared closure to the dependency configurations of the project.

configurations.all {
    resolutionStrategy.cacheDynamicVersionsFor 10, 'minutes'
}

Alternative:

configurations.all {
    resolutionStrategy {
        cacheDynamicVersionsFor 10, 'minutes'
    }
}

configurations {
    testCompile.exclude group: 'io.example'
}

• TODO https://docs.gradle.org/current/dsl/org.gradle.api.artifacts.ConfigurationContainer.html
• TODO https://docs.gradle.org/current/dsl/org.gradle.api.artifacts.ResolutionStrategy.html

Configuring Cached Snapshots Time to Live

configurations.all {
    resolutionStrategy.cacheChangingModulesFor 4, 'hours'
}

To attempt to pull every time the latest version of a SNAPSHOT, this seems to work:

configurations.all {
    resolutionStrategy.cacheChangingModulesFor 0, 'seconds'
}

The Dependency Cache

https://docs.gradle.org/current/userguide/dependency_cache.html

Once a dependency is resolved as result of the dependency resolution, Gradle stores the associated artifacts in a local cache, known as the dependency cache. One of the goals of storing artifacts locally in the cache is to minimize the number of remote requests made during the dependency resolution process.

The dependency cache is located under $GRADLE_USER_HOME/caches (typically ~/.gradle/caches). The directory contains two storage types: file-based storage for downloaded artifacts and raw metadata files, and a binary store of resolved module meta-data. This cache is also known as the metadata cache. The metadata cache contains, among other things, the result of resolving dynamic versions to concrete versions, the resolved module metadata, including module artifacts and module dependencies, the resolved artifact metadata, including a pointer to the downloaded file in the file-based storage, the absence of a particular module or artifact, eliminating repeated attempts to access a resource that does not exist.

For a specific dependency version, the files are maintained under .gradle/caches/modules-2/files-* and the corresponding binary metadata is maintained in .gradle/caches/modules-2/metadata-*. For example, the binary artifacts for SLF4J API 1.7.25, where "org.slf4j" is the group ID and "slf4j-api" is the artifact ID, are stored as follows:

.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25
                                                         |
                                                         + df51c4a85dd6acf8b6cdc9323596766b3d577c28/slf4j-api-1.7.25.pom
                                                         |
                                                         + da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar
                                                         |
                                                         + 962153db4a9ea71b79d047dfd1b2a0d80d8f4739/slf4j-api-1.7.25-sources.jar

The corresponding binary metadata is stored as:

.gradle/caches/modules-2/metadata-2.58/descriptors/org.slf4j/slf4j-api/1.7.25
                                                                         |
                                                                         + 1dd858de07b774d6be9d3e38c5646087/descriptor.bin
                                                                         |
                                                                         + 33ddd5448b9e1258ce46034f20c899a8/descriptor.bin
                                                                         |
                                                                         + f8e6315c37eb56998f7a5ba08e30db71/descriptor.bin

Dependency resolution will fail if the required artifacts are not available in any repository specified in the build, even if the local cache has a copy of the artifact that was retrieved from a different repository.

Before downloading an artifact from a repository, Gradle tries to get its checksum first by attempting to download the SHA file associated with the artifact. If the checksum can be retrieved, and Gradle determines that a file with the same id and checksum exists in the cache. If the checksum cannot be retrieved, the artifact will be downloaded (and ignored if it matches an existing artifact).

The cache is periodically scanned (at most every 24 hours) for artifacts that have not been used for more than 30 days and obsolete artifacts are deleted, to ensure the cache does not grow indefinitely.

Forced Cache Update

Gradle can be told to ignore all cached entries for resolved modules and artifacts with

... --refresh-dependencies ...

command line option. When used, a fresh resolve will be performed against all configured repositories, with dynamic versions recalculated, modules refreshed, and artifacts downloaded. However, where possible Gradle will check if the previously downloaded artifacts are valid before downloading again. This is done by comparing published SHA1 values in the repository with the SHA1 values for existing downloaded artifacts. A new snapshot repository will produce the following log:

gradle clean build --refresh-dependencies
...
Download s3://my-repo/snapshots/com/example/my-artifact/1.0.0-SNAPSHOT/maven-metadata.xml
...
Download s3://my-repo/snapshots/com/example/my-artifact/1.0.0-SNAPSHOT/my-artifact-1.0.0-20181008.231746-18.jar

Low-Level Cache Manipulation

It is sometimes necessary to remove individual dependency from cache. TODO.