Go Modules: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
 
(36 intermediate revisions by the same user not shown)
Line 1: Line 1:
=External=
=External=
* https://go.dev/ref/mod
* https://go.dev/ref/mod
* https://go.dev/blog/using-go-modules
=Internal=
=Internal=
* [[Go_Language_Modularization#Modules|Go Modularization]]
* [[Go_Language_Modularization#Modules|Go Modularization]]
Line 13: Line 15:


=<span id='Module_Name'></span><span id='Module_Path'></span>Module Path (Module Name)=
=<span id='Module_Name'></span><span id='Module_Path'></span>Module Path (Module Name)=
The module path and module name are used interchangeably. The module path is specified when the module is initialized with:
The '''module path''' and '''module name''' are used interchangeably. The module path is the published location from which the module can be downloaded by [[#Go_Tool#Overview|<tt>go</tt> tool]], such as the module code’s repository location. For example, to download the module <code>golang.org/x/tools</code>, the <code>go</code> tool should go to the repository indicated by https://golang.org/x/tools. The process is described in [[#Finding_a_Repository_for_a_Module_Path|Finding a Repository for a Module Path]]. The module path serves as a unique identifier, when combined with the module’s version number. The module path serves as [[#Package_Import_Path|import path]] prefix for all the packages in the module. Packages in the standard library do not have a module path prefix.
<syntaxhighlight lang='bash'>
go mod init <module-path>
</syntaxhighlight>
For more details on <code>go.mod</code> initialization, see [[#Initialize_go.mod|<tt>go.mod</tt> Initialization]] below.


The module path serves as [[#Package_Import_Path|import path]] prefix for all the packages in the module. The module path reflects the published location, it indicates where the [[#Go_Tool#Overview|<tt>go</tt> tool]] command should look to download it. For example, to download the module <code>golang.org/x/tools</code>, the <code>go</code> tool should go to the repository indicated by https://golang.org/x/tools. Packages in the standard library do not have a module path prefix.
The module path consists of three parts: a repository root path, corresponding to the repository root directory, an optional module subdirectory, if the module is not stored directly in root, and a major version suffix (only for modules released at v2 or higher).


=<span id='Import_Path'></span>Package Import Path=
Example: <code>github.com/apache/yunikorn-core</code>, <code>example.com/monorepo/something/mymodule/v2</code>.
For packages distributed as part of a module, the package's [[Go_Packages#Import_Path|import path]] is the concatenation of the module's [[#Module_Path|module path]] and the package's subdirectory within the module. The [[#Multi-Package_Module_Layout|Multi-Package Module Layout]] example, provided below, contains packages with import paths <code>example.com/generic/a</code> and  <code>example.com/generic/other/b</code>. It is possible to declare a module that contains a single package, with identical module path and package import path. [[#Single-Package_Module|Single-Package Module]] is such an example.
=<tt>go.mod</tt>=
<code>go.mod</code> declares the [[#Module_Path|module path]], which is the [[Go_Packages#Import_Path|import path]] prefix for all [[Go_Packages#Overview|packages]] within the module. It also tracks the modules that provide dependencies to this module.


One way to create it is with <code>[[Go_Tool#mod|go mod init]]</code>:
Older Go versions used to require that the first path component have a dot, but this is not enforced anymore.
<syntaxhighlight lang='bash'>
go mod init <module-path>
</syntaxhighlight>
<code>go.mod</code> stays with the code, including in the source repository.


==Example==
The module path is specified when the module is initialized with <code>go mod init <module-path></code>. For more details on <code>go.mod</code> initialization, see: {{Internal|Go.mod#Initialize_go.mod|<tt>go.mod</tt> &#124; Initialize <tt>go.mod</tt>}}
<syntaxhighlight lang='go'>
module example.com/hello


go 1.21.0
<font color=darkkhaki>TODO: https://go.dev/doc/modules/managing-dependencies#naming_module</font>


require rsc.io/quote v1.5.2
=<span id='Import_Path'></span>Package Import Path=
 
For packages distributed as part of a module, the package's [[Go_Packages#Import_Path|import path]] is the concatenation of the module's [[#Module_Path|module path]] and the package's subdirectory within the module. The [[#Multi-Package_Module_Layout|Multi-Package Module Layout]] example, provided below, contains packages with import paths <code>example.com/generic/a</code> and  <code>example.com/generic/other/b</code>. It is possible to declare a module that contains a single package, with identical module path and package import path. [[#Single-Package_Module|Single-Package Module]] is such an example.
require (
=<tt>go.mod</tt>=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c // indirect
{{Internal|Go.mod|<tt>go.mod</tt>}}
rsc.io/sampler v1.3.0 // indirect
)
</syntaxhighlight>


=Declaring Modules=
=Declaring Modules=
This section assumes we want to write an <code>example.com/generic</code> module, which consists of two packages <code>a</code> and <code>b</code>. The <code>a</code> package lives in an <code>a</code> directory in the module root directory, which gives it a <code>example.com/generic/a</code> [[#Import_Path|import path]]. The <code>b</code> package lives in an <code>other/b</code> directory in the module root directory, which gives it a <code>example.com/generic/other/b</code> [[#Import_Path|import path]]. There is also a <code>main</code> package that provides the command line binary.
This section assumes we want to write an <code>example.com/generic</code> module, which consists of two packages <code>a</code> and <code>b</code>. The <code>a</code> package lives in an <code>a</code> directory in the module root directory, which gives it a <code>example.com/generic/a</code> [[#Import_Path|import path]]. The <code>b</code> package lives in an <code>other/b</code> directory in the module root directory, which gives it a <code>example.com/generic/other/b</code> [[#Import_Path|import path]]. There is also a <code>main</code> package that provides the command line binary.
==Initialize <tt>go.mod</tt>==
==Initialize <tt>go.mod</tt>==
In the [[#Module_Root_Directory|module root directory]], initialize the <code>[[Go.mod#Overview|go.mod]]</code> file.
{{Internal|Go.mod#Initialize_go.mod|Initialize <tt>go.mod</tt>}}
<syntaxhighlight lang='bash'>
 
go mod init <module-path>
</syntaxhighlight>
<syntaxhighlight lang='bash'>
go mod init example.com/generic
</syntaxhighlight>
==Module Layout==
==Module Layout==
<font color=darkkhaki>This section needs refactoring after reading "Managing module source" https://go.dev/doc/modules/managing-source</font>
<font color=darkkhaki>This section needs refactoring after reading
* https://go.dev/doc/modules/managing-dependencies
"Managing module source" https://go.dev/doc/modules/managing-source
* https://github.com/golang-standards/project-layout
* Think about the difference between project and module layout. Are there multi-module projects? Or usually a project is also a module?
</font>


<span id='Module_Root_Directory'></span>The '''module root directory''' contains the <code>[[Go.mod#Overview|go.mod]]</code> file and the package directories.  
<span id='Module_Root_Directory'></span>The '''module root directory''' contains the <code>[[Go.mod#Overview|go.mod]]</code> file and the package directories.  
Line 95: Line 81:


=Publishing Modules=
=Publishing Modules=
<font color=darkkhaki>This section needs refactoring after reading "Publishing a module" https://go.dev/doc/modules/publishing </font>
<font color=darkkhaki>This section needs refactoring after reading "Publishing a module" https://go.dev/doc/modules/publishing  


Publishing a module involves compiling, building and publishing the [[Go_Language#Object_File|object files]] corresponding to all or some of the module's packages, or, if the module contains a <code>main</code> package, building and publishing the executable file.
Also, reconcile with [[Go_Tool#install|go install]].
==<span id='Publishing_Locally'></span>Publishing Modules Locally==
</font>
Local module publishing is done with <code>[[Go_Tool#install|go install]]</code>, configured to support modules. For details on compiling and installing packages without module support see: {{Internal|Go_Packages#Publishing_Locally|Publishing Packages Locally}}
We assume that <code>go install</code> is configured in [[Go_Language_Modularization#Module-Aware_or_GOPATH_Mode|module-aware mode]].
 
===Single Package with No Internal Dependencies===
To compile and publish a single package within a module, execute:
<syntaxhighlight lang='bash'>
go install <package-import-path>
</syntaxhighlight>
where the package import path is concatenation of the module path and the package's relative import path within the module, as explained in [[#Package_Import_Path|Package Import Path]].
<syntaxhighlight lang='bash'>
go install example.com/generic/other/b
</syntaxhighlight>
<font color=darkkhaki>The command compiles and stores the object file in the [[Go_Build_Cache#Overview|build cache]], but it does not write in any user-level directory. Why?</font>
 
===Packages with Internal Dependencies===
If a module package <code>example.com/generic/a</code> depends on the package <code>example.com/generic/other/b</code> , the command
<syntaxhighlight lang='bash'>
go install example.com/generic/a
</syntaxhighlight>
will correctly locate and compile the source files for both <code>a</code> and <code>other/b</code> packages, and if deeper dependencies exist, all source files in the transitive dependency tree. If any of the dependencies fails to compile, the installation process will fail. However, only the [[Go_Language#Object_File|object file]].
 
<font color=darkkhaki>The command compiles and stores the object file in the [[Go_Build_Cache#Overview|build cache]], but it does not write in any user-level directory. Why?</font>
 
===Executables===
An executable is produced when the entire module is built with <code>go install</code> and the linker detects a <code>[[#The_.22main.22_Package|main]]</code> package among the packages being compiled.
<syntaxhighlight lang='bash'>
go install example.com/generic
</syntaxhighlight>
 
Assuming the example above has a <code>generic-cmd.go</code> source code that belongs to the <code>main</code> package and has a <code>main()</code> function, installing the module produces an executable and writes it in <code>~/go/bin</code> by default. If the environment variable  <code>[[Go_Environment_Variables#GOBIN|GOBIN]]</code> is set, the value takes precedence.
 
The name of the executable in this case is <code>generic</code> <font color=darkkhaki>(Why? It is not given by the source file name.)</font>
==Publishing Modules Remotely==
The typical way to publish a module is to share the source code in a version-controlled repository, with a [[#Module_Path|module path]] that reflects its published location. <font color=darkkhaki>More details required here.</font>


=Consuming Modules=
=Consuming Modules=
Line 166: Line 118:
* Same question for GoLand. Is the same place?
* Same question for GoLand. Is the same place?
* What does actually happen there? Are we downloading sources or binaries?</font>
* What does actually happen there? Are we downloading sources or binaries?</font>
===Finding a Repository for a Module Path===
{{External|https://go.dev/ref/mod#vcs-find}}
===Mapping Versions to Commits===
{{External|https://go.dev/ref/mod#vcs-version}}
===Mapping Pseudo-versions to Commits===
{{External|https://go.dev/ref/mod#vcs-pseudo}}
===Downloads===
The <code>go</code> command may download module source code and metadata [[#Direct_Download|directly]] from the repository that hosts it or from a [[#Proxy_Download|proxy]].
====Direct Download====
A direct download is when the <code>go</code> command downloads module source code and metadata directly from a version control repository. The alternative is a [[#Proxy_Download|proxy download]].
====Proxy Download====
{{External|https://go.dev/ref/mod#communicating-with-proxies}}
Downloading a module from a proxy is usually faster.
====Module Proxy Server====
A web server that implements the GOPROXY protocol. The <code>go</code> command downloads version information, <code>go.mod</code> files, and module zip files from module proxies.


=Managing Dependencies=
=Managing Dependencies=
In Go, dependencies are managed as modules that contain packages to be imported and used in your code. A well defined version of a module is uniquely identified by the [[#Module_Path_.28Module_Name.29|module path]], that can be used to actually download the module's source code, and a version.
<font color=darkkhaki>TODO: https://go.dev/doc/modules/managing-dependencies</font>
<font color=darkkhaki>TODO: https://go.dev/doc/modules/managing-dependencies</font>
=Developing and Publishing Modules=
=Developing and Publishing Modules=
<font color=darkkhaki>TODO: https://go.dev/doc/modules/developing</font>
<font color=darkkhaki>TODO: https://go.dev/doc/modules/developing</font>
Line 179: Line 150:
* Developing a major version update: https://go.dev/doc/modules/major-version
* Developing a major version update: https://go.dev/doc/modules/major-version
</font>
</font>
==Pseudo-version==
A version that encodes a revision identifier (such as a Git commit hash) and a timestamp from a version control system. For example, <code>v0.0.0-20191109021931-daa7c04131f5</code>. Used for compatibility with non-module repositories and in other situations when a tagged version is not available.
=Module Cache=
{{External|https://go.dev/ref/mod#module-cache}}
The '''module cache''' is a directory where the <code>go</code> command stores downloaded module files. The entire snapshot of the source tree is downloaded. The module cache is distinct from the [[Go_Build_Cache|build cache]], which contains compiled packages and other build artifacts. The default location of the module cache is <code>${[[Go_Environment_Variables#GOPATH|GOPATH]]}/pkg/mod</code>. The artifacts are stored in read-only mode to prevent accidental corruption.
To clean the module cache use <code>go clean -modcache</code>.
=Private Modules=
{{External|https://golang.org/ref/mod#private-modules}}
Also see: {{Internal|Go_Environment_Variables#GOPRIVATE,_GONOPROXY,_GONOSUMDB|<tt>GOPRIVATE</tt>,<tt>GONOPROXY</tt>,<tt>GONOSUMDB</tt>}}


=Other Module Subjects=
=Other Module Subjects=
==Module Proxy Server==
==Module Cache==
<font color=darkkhaki>Is the module cache the same as build cache?</font>
{{Internal|Go_Build_Cache|Build Cache}}
==Configuring Modules in GoLand==
==Configuring Modules in GoLand==
{{Internal|GoLand#Module-Aware_or_GOPATH_Mode|GoLand}}
{{Internal|GoLand#Module-Aware_or_GOPATH_Mode|GoLand}}

Latest revision as of 00:03, 10 January 2024

External

Internal

Overview

A module is a collection of related packages that are stored together under the same filesystem tree, released, versioned and distributed together. Modules have been introduced in Go 1.11. Packages, a lower-level namespacing, encapsulation and code sharing mechanism, are published as part of modules.

A module is stored as a directory that contains a go.mod file and subdirectories that contain the source code for the module's packages. The go.mod stores metadata such as dependencies needed by the module, required Go version, etc. If any of the subdirectories contains a go.mod file, that subdirectory represents an embedded module.

Modules may be downloaded directly from version control repositories, or from module proxy servers.

Module Path (Module Name)

The module path and module name are used interchangeably. The module path is the published location from which the module can be downloaded by go tool, such as the module code’s repository location. For example, to download the module golang.org/x/tools, the go tool should go to the repository indicated by https://golang.org/x/tools. The process is described in Finding a Repository for a Module Path. The module path serves as a unique identifier, when combined with the module’s version number. The module path serves as import path prefix for all the packages in the module. Packages in the standard library do not have a module path prefix.

The module path consists of three parts: a repository root path, corresponding to the repository root directory, an optional module subdirectory, if the module is not stored directly in root, and a major version suffix (only for modules released at v2 or higher).

Example: github.com/apache/yunikorn-core, example.com/monorepo/something/mymodule/v2.

Older Go versions used to require that the first path component have a dot, but this is not enforced anymore.

The module path is specified when the module is initialized with go mod init <module-path>. For more details on go.mod initialization, see:

go.mod | Initialize go.mod

TODO: https://go.dev/doc/modules/managing-dependencies#naming_module

Package Import Path

For packages distributed as part of a module, the package's import path is the concatenation of the module's module path and the package's subdirectory within the module. The Multi-Package Module Layout example, provided below, contains packages with import paths example.com/generic/a and example.com/generic/other/b. It is possible to declare a module that contains a single package, with identical module path and package import path. Single-Package Module is such an example.

go.mod

go.mod

Declaring Modules

This section assumes we want to write an example.com/generic module, which consists of two packages a and b. The a package lives in an a directory in the module root directory, which gives it a example.com/generic/a import path. The b package lives in an other/b directory in the module root directory, which gives it a example.com/generic/other/b import path. There is also a main package that provides the command line binary.

Initialize go.mod

Initialize go.mod

Module Layout

This section needs refactoring after reading

The module root directory contains the go.mod file and the package directories.

Single-Package Module

Simple, one-package modules, where the module path and the the package import path are the same, are supported. In the example below, the single-package module contains just a single a package. The module path example.com/a is the same as the package import path. The filesystem layout is:

.
├─ go.mod
└─ a.go

The source file declares inclusion in the a package:

  
package a

...

go.mod declares the module path and not much else:

  
module example.com/a
go 1.21.0

Multi-Package Module Layout

.
├── go.mod # Declares a module named example.com/generic
│
├── a
│   └─ a.go 
│
├── other
│    └─ b
│       └─ b.go
│
└── generic-cmd.go # part of the "main" package

Publishing Modules

This section needs refactoring after reading "Publishing a module" https://go.dev/doc/modules/publishing

Also, reconcile with go install.

Consuming Modules

A package published as part of module can be consumed from another consumer module by importing it from the consumer module source code with the import keyword followed by the package import path, which includes the module path. In simple cases, the module and package import paths can be the same:

package main

import (
  ...
  "example.com/a"
)

The go.mod of the consumer module must be updated with dependency tracking information. If the dependency is published and accessible via network, go.mod can be automatically updated with (after declaring imports appropriately in source files):

go mod tidy

However, for a local, unpublished modules whose code is available only on the local filesystem as a known relative path, the consumer module's go.mod can be edited manually as follows:

module example.com/consumer

[...]

require example.com/a v0.0.0-unpublished
replace example.com/a v0.0.0-unpublished => ../a  // relative path from the consumer module directory to the dependency module directory

Importing Packages from Remote Modules

Importing Precompiled Packages

Importing Packages from GitHub

  • Where does go mod store the precompiled packages?
  • Same question for GoLand. Is the same place?
  • What does actually happen there? Are we downloading sources or binaries?

Finding a Repository for a Module Path

https://go.dev/ref/mod#vcs-find

Mapping Versions to Commits

https://go.dev/ref/mod#vcs-version

Mapping Pseudo-versions to Commits

https://go.dev/ref/mod#vcs-pseudo

Downloads

The go command may download module source code and metadata directly from the repository that hosts it or from a proxy.

Direct Download

A direct download is when the go command downloads module source code and metadata directly from a version control repository. The alternative is a proxy download.

Proxy Download

https://go.dev/ref/mod#communicating-with-proxies

Downloading a module from a proxy is usually faster.

Module Proxy Server

A web server that implements the GOPROXY protocol. The go command downloads version information, go.mod files, and module zip files from module proxies.

Managing Dependencies

In Go, dependencies are managed as modules that contain packages to be imported and used in your code. A well defined version of a module is uniquely identified by the module path, that can be used to actually download the module's source code, and a version.

TODO: https://go.dev/doc/modules/managing-dependencies

Developing and Publishing Modules

TODO: https://go.dev/doc/modules/developing

Module Version

Modules evolve by publishing new versions.

TODO:

Pseudo-version

A version that encodes a revision identifier (such as a Git commit hash) and a timestamp from a version control system. For example, v0.0.0-20191109021931-daa7c04131f5. Used for compatibility with non-module repositories and in other situations when a tagged version is not available.

Module Cache

https://go.dev/ref/mod#module-cache

The module cache is a directory where the go command stores downloaded module files. The entire snapshot of the source tree is downloaded. The module cache is distinct from the build cache, which contains compiled packages and other build artifacts. The default location of the module cache is ${GOPATH}/pkg/mod. The artifacts are stored in read-only mode to prevent accidental corruption.

To clean the module cache use go clean -modcache.

Private Modules

https://golang.org/ref/mod#private-modules

Also see:

GOPRIVATE,GONOPROXY,GONOSUMDB

Other Module Subjects

Configuring Modules in GoLand

GoLand

TODO