Pulumi Concepts
External
- https://www.pulumi.com/docs/intro/concepts/
- https://www.pulumi.com/docs/guides/automation-api/concepts-terminology/
Internal
Overview
Pulumi is an Infrastructure as Code platform that allows using common programming languages, tools, and frameworks, to provision, update, and manage cloud infrastructure resources. Pulumi is one of the tools that can be used to manage generic Infrastructure as Code stacks. In Pulumi, Infrastructure resources are declared in programs. Programs reside in a project. The programs are instantiated as stacks in the infrastructure platform. A stack is an isolated and configurable instance of the program.
Architecture
Supported Runtimes and Programming Languages
Python
The runtime name is python.
TypeScript
The runtime name is nodejs.
Go
The runtime name is go.
.NET
The runtime name is dotnet.
Organization
Other organization is also referred to as the "owner".
The list of organizations available to the user in the backend is available with pulumi org ls.
- Relationship between a stack and an organization ("owner"). "New stack owner, some-org, does not match existing owner, o_feodorov."
Project
A project is a collection of code and a filesystem layout that supports declaring programs and stack configurations, and at the same time, a namespace mechanisms for stacks on the backend.
Local Project Representation
A project is a directory that contains one program and metadata on how to run the program, such as what runtime to use, where to look for the program, etc. The project definition metadata is maintained in Pulumi.yaml. The presence of Pulumi.yaml is the indication that directory is a Pulumi project.
The project's program may be applied to the infrastructure platform creating one or more independent resource sets, as defined in the program. An instantiation of a program is called a stack. The project may contain one or more stacks. Each stack is defined in the project by its stack settings file.
The project can be obtained programmatically. In Python, use get_project() function.
A project, a program and multiple stacks are tied together in a single execution context named workspace.
Creating a Project
Projects can be created with pulumi new command. The command creates the project metadata on the local filesystem and the first (and possibly the only one) stack, based on a template, and also modifies the backend, registering the newly created stack. At this point, no infrastructure resources are actually created in the infrastructure platform. The program and metadata can be read by the Pulumi CLI and applied to the infrastructure platform with the pulumi up command, updating the stack. Can valid projects be created by hand, without pulumi new, by manually crafting metadata and code?
Current Project
The current project is the project given by the nearest Pulumi.yaml file.
Project Name
The project name should represent the intent of the code within the project. For example, if the project is intended to create infrastructure for an application named "blue", a good project name would be "blue" or "blue-infra". Don't rely on the stack name (or names) to fully convey that semantics. The stack names should be mostly driven by the particular instantiation of the infrastructure resource set, such as "dev", or "staging" ("blue-dev" is a good name).
The project name is specified using the name attribute in Pulumi.yaml. It shows up in the Pulumi dashboard. It is used to aggregate the associated stacks and their resources underneath the project, under a simple hierarchy. Project names may only contain alphanumeric characters, hyphens, underscores and periods.
Workspace
A workspace is an execution context that contains a single project, a single program and multiple stacks. Workspaces manage the execution environment, providing utilities such as plugin installation, environment configuration ($PULUMI_HOME) and creation, deletion and listing of stacks. The workspace relies on Pulumi.yaml and stack settings files as intermediate format for project and stack settings. Pulumi maintains workspace local state in:
~
└─ .pulumi
└─ workspaces
└─ <project-name>-8698263d7571....4d46edf4e5bfeff-workspace.json
The content of the workspace JSON file is similar to:
{
"stack": "<org-name>/<project-name>/<stack-name>"
}
The value of the "stack" key indicates the active stack. The local state is initialized and updated by commands like pulumi stack init, etc.
What happens if the same program is checked out and executed from two different local directories?
Template
A template is a set of files, either built-in or exposed as a Git URL, which contain Python code and Pulumi metadata to bootstrap a project. An essential component of a template is the template section of Pulumi.yaml file. A list of available templates is presented executing pulumi new without any argument. An example is available here:
Intuitively, the right template can be invoked by using the name of the cloud and the language, example: aws-python.
GitHub Template Repository
If intending to access templates directly from a GitHub repository, the repository should be set with "Enable anonymous Git read access" (Settings → Danger Zone → Enable anonymous Git read access)
Project Layout
my-project ├─ README.md ├─ Pulumi.yaml # Project definition file ├─ Pulumi.red-stack.yaml # Stack settings file ├─ Pulumi.green-stack.yaml ├─ ... └─ ... # language-specific elements
Language-specific layouts:
Pulumi.yaml
Contains the project definition.
Deploying a Project
Deploying a project means applying the project's code with the active stack configuration to the infrastructure platform, thus creating or updating infrastructure resources. Deployment is initiated via CLI with the pulumi up command. Reconcile with What Happens when Code Is Applied to Platform?.
Project Operations
Projects on the Backend
Projects are not a first class concept on the backend, they only exist as a way to namespace stacks, and they are reflected in the stack full names. As such, there no command (yet?) that displays the project for a given organization, or for the current organization. To get a list of project for an organization, you can list all stacks, for all organizations, and then filter the results for the specific organization.
A project can be "deleted" by deleting all stacks that belong to it.
Program
A program contains code that describes how cloud infrastructure should be composed. It can be written in Python, TypeScript or Go. Infrastructure is declared by defining resource objects whose properties correspond to the desire state of the infrastructure. The properties are also used to express dependencies between resources, and can be exported outside the stack. It is recommended to group resource with common lifecycles together. Programs reside in projects. Automation API allows defining a Pulumi program as a function within an arbitrary codebase rather than a separated project, and use methods to get and set configuration parameters programmatically.
Local Program
A local program is a traditional Pulumi CLI-driven program with its own directory, a Pulumi.yaml file, along with a file that defines the Pulumi program. Automation API can be used to drive these programs, as well as the CLI.
Inline Program
Unlike traditional Pulumi programs, inline programs don’t require a separate package on disk, with its own file and a Pulumi.yaml. Inline programs are simply functions that can be authored in the same file as your Automation API program or be imported from anther package.
Paths in Program
When the program references resources in the local file system, their paths must be relative to the working directory. Not the project directory, in which the program is in?
Programming Model
Stack
A stack is a unit of deployment, an isolated, independently configurable instance of a Pulumi program, materialized as a set of infrastructure resources created by executing the program against the infrastructure platform. A stack is at the same time a state storage unit. The stack stores the “actual state” of the infrastructure resource set. Should the desired state, as expressed by the program, change, Pulumi compares the desired state with the actual state maintained by the stack, and if detects differences, it updates the infrastructure resources and the stack state. More details on the relationship between stacks and the Pulumi programming model are available in Pulumi Programming Model.
Every program is applied to the infrastructure as one, possibly more stacks. Stacks are commonly used to denote different phases of development, such as "development", "staging" and "production", or feature branches. A project can have an arbitrary number of stacks. By default, Pulumi creates a new empty stack per project when pulumi new is used, and a manually bootstrapped project starts with no stacks. Stacks can be created separately with pulumi stack init. Stack creation means creation of the associated stack settings file and the instantiation of the stack state representation in the Pulumi backend. Initially, a stack starts as "empty", meaning that is has no resource representations stored in the backend, but it can become a target of a deployment initiated with the pulumi update command. Upon the first, and subsequent deployment, the stack backend representation is populated with resources. The state built in memory, after executing the program, represents the "desired state" of the infrastructure, and Pulumi aims to bring the actual state in line with the desired state. Each stack has its own separate configuration, secretes, role-based controls (RBAC) and policies. Deep insight into its structure and components can be achieved by exporting the stack and examining the corresponding JSON file.
Stack Name
The stack name must be unique within a project and should express the intent behind a specific instantiation of the project infrastructure resource set, like "dev" or "testing". A pattern that seems reasonable is <project-name>-<function>, especially that in some circumstances, the project name is omitted from the stack name (like in <org-name>/<stack-name>). For more considerations on stack and project name, see project name. The stack name may only contain alphanumeric characters, hyphens, underscores and periods.
The stack name does now allow slashes, if attempting to use a stack name with slashes in it, the CLI complains: "Sorry, 'my-org/my-project/test1/test2' is not a valid stack name. could not parse stack name 'my-org/my-project/test1/test2'."
Fully Qualified Stack Name
A fully qualified stack name includes the organization and the project name: <org-name>/<project-name>/<stack-name>. If you are using Pulumi in your organization, and when new stacks are created, they will be created by default in your user account. To create the stack in the organization instead, name the stack using <org-name>/<stack-name>.
Stack URI
What is the semantics of a stack URI?
Empty Stack
An empty stack is a stack for which no infrastructure resources have been created yet in the infrastructure platform. Empty stacks are created when projects are initialized with pulumi new, or when pulumi stack init command is used. The opposite of an empty stack is a fully initialized stack. The stack is initialized as result of the pulumi up command.
Multiple Stacks per Project
When a project is created with pulumi new, the configuration of a stack is also automatically created. Additional stack configurations and stack backend state instances can be created for an existing project with the pulumi stack init command. Note that multiple stacks per project means that all the stacks share the same program (behavior) but have different configurations, corresponding to different stack settings files. Note that pulumi stack init does not fully manage the corresponding stack setting file, it does create it, but it leaves it into an incomplete state.
Current (Active, Default) Stack
When a project contains multiple stacks, one of them is active, or current, at any time. The current stack for a project can be displayed by running pulumi about from the project directory, or running pulumi stack ls, in which case the active stack will be marked with an asterisk. A stack can be set as active by running pulumi stack select. The active stack is maintained in the local workspace state, as the value of the "stack" key.
Stack Configuration
While each stack of a project has a similar resource set, it will differ from other stacks in its configuration. Pulumi offers a configuration system to manage such differences. The system provides different methods to introspect and set configuration, including CLI and the programming model. The key-value pairs for any given stack of the project are stored in the stack's settings file.
TO PROCESS: https://www.pulumi.com/docs/intro/concepts/config/
Stack Settings File
Each stack within a project has an associated file named Pulumi.<stack-name>.yaml that contains configuration, in form of key/value pairs, for the stack. The file resides in the project root directory. Secret values are encrypted, which, according to the Pulumi documentation, makes them safe to be checked in into a repository. This is debatable. The stack settings for ephemeral stacks are typically not checked into source control. A stack setting file can be created and managed with the pulumi config command.
Stack settings files are an implementation of the Infrastructure as Code Stack Configuration File pattern.
Configuration Management via CLI
Stack configuration can be read and updated with the pulumi config command.
Programmatic Configuration Management
Config.
TO DO
Configuration Keys and Namespace
Key space. TO DO
Restoring Configuration from Backend
TO DO: pulumi config refresh
Stack Inputs and Outputs
Stack References
Stack references are used by consumer stacks to get their dependencies. This is the Pulumi implementation of the Stack Data Lookup pattern: the stack reference allows access to the outputs of a provider stack from a consumer stack. To get values exposed by a producer stack, create an instance of StackReference representing the producer stack, from the consumer stack, and asynchronously access the values exported by producer via Outputs. The provider stack name must be fully qualified, including organization, project and stack name. Then, get the Output instance corresponding to the piece of data you want to access by invoking get_output() on the StackReference instance. The data must be previously exported by the provider stack with export(). Note that get_output() returns an Output instance. The actual data exported by the provider stack can be accessed asynchronously via apply() or other Output access patterns. Once a StackReference has been created, and exported data is accessed via Output instances, Pulumi understands the inter-stack dependency for scenarios like cascading updates.
# consumer stack code
import pulumi
provider = pulumi.StackReference("some-org/some-project/some-provider-stack")
output = provider.get_output("some_parameter"); # this is an Output instance
output.apply(lambda v: ... # do something with the value)
Stack Tags
Metadata can be attached to stacks, in form of tags. Each tags has a name and a value. A set of built-in tags are automatically assigned and updated each time a stack is updated. Tags are only supported with the Pulumi Service backend. To view a stack's tags, run pulumi stack tag ls. Tags are useful to group stacks in the console.
Custom tags can be applied with pulumi stack tag set <name> <value>. Custom tags should not be prefixed with pulumi:, gitHub:, or vcs: to avoid conflicting with built-in tags that are assigned and updated with fresh values each time a stack is updated.
Built-in Stack Tags
| Tag Name | Notes |
|---|---|
| pulumi:project | |
| pulumi:runtime | Example of a Python runtime section |
| pulumi:description | |
| gitHub:owner | |
| gitHub:repo | |
| vcs:owner | |
| vcs:repo | |
| vcs:kind |
Tag Operations
Stack Lifecycle
Create an Empty Stack
An empty stacks with the given name, ready for updates, can be created with pulumi stack init. The stack has no resources stored in the backend, but it can become a target of a deployment initiated with the pulumi update command. pulumi new command creates a default empty stack
Deploying a Stack
See Deploying a Project above.
Previewing a Stack
Refreshing a Stack
Destroying and Deleting a Stack
Destroying a stack means releasing and deleting infrastructure resources associated with the stack in the infrastructure platform. A stack is destroyed with pulumi destroy. The command uses the latest configuration values, rather than the ones that were last used when the program was deployed. Pulumi waits until all resources are shut down and deleted before it considers the destroy operation to be complete. The destroy operation deletes resources, but it does not delete the stack history or configuration.
Deleting a stack with pulumi stack rm means removing all stack history from the backend and the stack configuration file Pulumi.<stack-name>.yaml. A stack must be first destroyed, then deleted.
⚠️ Forcefully deleting a stack before destroying it may leave orphaned resources behind
Stack Export
A stack can be exported to see the raw data associated with the stack. The exported data can be manually modified, then imported to set the current state of the stack.
⚠️ Importing/exporting stacks in not a normal workflow, but an alternative workflow that subverts the usual way Pulumi manages resources and ensures immutable and repeatable infrastructure deployments. Importing an incorrect stack specification could lead to orphaning of cloud resources or the inability to make future updates to the stack.
Importing Infrastructure Resources into an Existing Stack
The import of external resources can be done from CLI with pulumi import.
Ephemeral Stack
Ephemeral stacks may have stack settings files.
Stack Operations
- List stacks
- Select a stack
- Display stack resources
- Display stack tags
- Create a stack
- Select a stack
- Remove (delete) a stack
- Export a stack
- Import a stack
Resource
State and Backends
Backend
The main job of a backend is to reliably store and manage the state of stacks, because Pulumi needs to understand the state of the infrastructure. Pulumi supports different backends: blob storage, locally, or in the Pulumi Service.
Among many other things, a backend provides dependencies to consumer stacks via the Stack Data Lookup pattern.
Name
URL
Authentication and Identity
The backend decides what a user can and cannot see, for example organizations, based on user's OD group configuration. A user's OD group list, as known by the Pulumi backend, is given by pulumi whoami --verbose.
Pulumi Service Backend
Pulumi SaaS:
Secrets
TO PROCESS: https://www.pulumi.com/docs/intro/concepts/secrets/
PULUMI_CONFIG_PASSPHRASE
PULUMI_CONFIG_PASSPHRASE_FILE
You can swap a passphrase out for a cloud provider key e.g. AWS KMS: https://www.pulumi.com/docs/reference/cli/pulumi_stack_change-secrets-provider/
Same shared access to the key for members of your org, and other stacks that use StackReferences, still applies as with the default passphrase approach.
TODO https://www.pulumi.com/learn/building-with-pulumi/secrets/
Logging
TO PROCESS: https://www.pulumi.com/docs/intro/concepts/logging/
Assets and Archives
TO PROCESS: https://www.pulumi.com/docs/intro/concepts/assets-archives/
Plugin
Plugin Operations
Function Serialization
TO PROCESS: https://www.pulumi.com/docs/intro/concepts/function-serialization/
Modularization
TO PROCESS: https://www.pulumi.com/blog/creating-and-reusing-cloud-components-using-package-managers/
Packages
Packages for various infrastructure providers are published in Pulumi registry.
Pulumi Registry
CLI
Auto-Naming
Automatically generates a random identifier and places it at the end of the given name, to avoid namespace collisions.
Automation API
Pulumi vs Terraform
Also see: