Pulumi Concepts: Difference between revisions
| Line 209: | Line 209: | ||
==Stack Lifecycle== | ==Stack Lifecycle== | ||
===Create an Empty Stack=== | |||
An empty stacks with the given name, ready for updates, can be created with <code>[[Pulumi_Operations#Create_Stack|pulumi stack init]]</code>. The stack has no resources stored in the backend, but it can become a target of a deployment initiated with the <code>pulumi update</code> command. <code>pulumi new</code> command creates a default empty stack | |||
===Deploying a Stack=== | ===Deploying a Stack=== | ||
See [[#Deploying_a_Project|Deploying a Project]] above. | See [[#Deploying_a_Project|Deploying a Project]] above. | ||
Revision as of 23:43, 24 January 2022
External
- https://www.pulumi.com/docs/intro/concepts/
- https://www.pulumi.com/docs/guides/automation-api/concepts-terminology/
Internal
Overview
Pulumi is an Infrastructure as Code platform that allows using common programming languages, tools, and frameworks, to provision, update, and manage cloud infrastructure resources. Pulumi is one of the tools that can be used to manage generic Infrastructure as Code stacks. In Pulumi, Infrastructure resources are declared in programs. Programs reside in a project. The programs are instantiated as stacks in the infrastructure platform. A stack is an isolated and configurable instance of the program.
Architecture
Supported Runtimes and Programming Languages
Python
The runtime name is python.
TypeScript
The runtime name is nodejs.
Go
The runtime name is go.
.NET
The runtime name is dotnet.
Organization
Other organization is also referred to as the "owner".
The list of organizations available to the user in the backend is available with pulumi org ls.
- Relationship between a stack and an organization ("owner"). "New stack owner, some-org, does not match existing owner, o_feodorov."
Project
A project is a collection of code and a filesystem layout that supports declaring programs and stack configurations, and at the same time, a namespace mechanisms for stacks on the backend.
Local Project Representation
A project is a directory that contains one program and metadata on how to run the program, such as what runtime to use, where to look for the program, etc. The project definition metadata is maintained in Pulumi.yaml. The presence of Pulumi.yaml is the indication that directory is a Pulumi project.
The project's program may be applied to the infrastructure platform creating one or more independent resource sets, as defined in the program. An instantiation of a program is called a stack. The project may contain one or more stacks. Each stack is defined in the project by its stack settings file.
The project can be obtained programmatically. In Python, use get_project() function.
A project, a program and multiple stacks are tied together in a single execution context named workspace.
Creating a Project
Projects can be created with pulumi new command. The command creates the project metadata on the local filesystem and the first (and possibly the only one) stack, based on a template, and also modifies the backend, registering the newly created stack. At this point, no infrastructure resources are actually created in the infrastructure platform. The program and metadata can be read by the Pulumi CLI and applied to the infrastructure platform with the pulumi up command, updating the stack. Can valid projects be created by hand, without pulumi new, by manually crafting metadata and code?
Current Project
The current project is the project given by the nearest Pulumi.yaml file.
Project Name
The project name should represent the intent of the code within the project. For example, if the project is intended to create infrastructure for an application named "blue", a good project name would be "blue" or "blue-infra". Don't rely on the stack name (or names) to fully convey that semantics. The stack names should be mostly driven by the particular instantiation of the infrastructure resource set, such as "dev", or "staging" ("blue-dev" is a good name).
The project name is specified using the name attribute in Pulumi.yaml. It shows up in the Pulumi dashboard. It is used to aggregate the associated stacks and their resources underneath the project, under a simple hierarchy. Project names may only contain alphanumeric characters, hyphens, underscores and periods.
Workspace
A workspace is an execution context that contains a single project, a single program and multiple stacks. Workspaces manage the execution environment, providing utilities such as plugin installation, environment configuration ($PULUMI_HOME) and creation, deletion and listing of stacks. The workspace relies on Pulumi.yaml and stack settings files as intermediate format for project and stack settings. Pulumi maintains workspace local state in:
~
└─ .pulumi
└─ workspaces
└─ <project-name>-8698263d7571....4d46edf4e5bfeff-workspace.json
The content of the workspace JSON file is similar to:
{
"stack": "<org-name>/<project-name>/<stack-name>"
}
Template
A template is a set of files, either built-in or exposed as a Git URL, which contain Python code and Pulumi metadata to bootstrap a project. An essential component of a template is the template section of Pulumi.yaml file. A list of available templates is presented executing pulumi new without any argument. An example is available here:
Intuitively, the right template can be invoked by using the name of the cloud and the language, example: aws-python.
GitHub Template Repository
If intending to access templates directly from a GitHub repository, the repository should be set with "Enable anonymous Git read access" (Settings → Danger Zone → Enable anonymous Git read access)
Project Layout
my-project ├─ README.md ├─ Pulumi.yaml # Project definition file ├─ Pulumi.red-stack.yaml # Stack settings file ├─ Pulumi.green-stack.yaml ├─ ... └─ ... # language-specific elements
Language-specific layouts:
Pulumi.yaml
Contains the project definition.
Deploying a Project
Deploying a project means applying the project's code with the active stack configuration to the infrastructure platform, thus creating or updating infrastructure resources. Deployment is initiated via CLI with the pulumi up command. Reconcile with What Happens when Code Is Applied to Platform?.
Project Operations
Projects on the Backend
Projects are not a first class concept on the backend, they only exist as a way to namespace stacks, and they are reflected in the stack full names. As such, there no command (yet?) that displays the project for a given organization, or for the current organization. To get a list of project for an organization, you can list all stacks, for all organizations, and then filter the results for the specific organization.
A project can be "deleted" by deleting all stacks that belong to it.
Program
A program contains code that describes how cloud infrastructure should be composed. It can be written in Python, TypeScript or Go. Infrastructure is declared by defining resource objects whose properties correspond to the desire state of the infrastructure. The properties are also used to express dependencies between resources, and can be exported outside the stack. It is recommended to group resource with common lifecycles together. Programs reside in projects. Automation API allows defining a Pulumi program as a function within an arbitrary codebase rather than a separated project, and use methods to get and set configuration parameters programmatically.
Paths in Program
When the program references resources in the local file system, their paths must be relative to the working directory. Not the project directory, in which the program is in?
Programming Model
The Pulumi programming model defines the core concepts in use when creating infrastructure as code programs. These concepts are made available in the Pulumi SDKs, that support Python, TypeScript and Go.
Stack
A stack is a unit of deployment, an isolated, independently configurable instance of a Pulumi program, materialized as a set of infrastructure resources created by executing the program against the infrastructure platform. Every program is applied to the infrastructure as one, possibly more stacks. Stacks are commonly used to denote different phases of development, such as "development", "staging" and "production", or feature branches. A project can have an arbitrary number of stacks. By default, Pulumi creates a new empty stack per project when pulumi new is used, and a manually bootstrapped project starts with no stacks. Stacks can be created separately with pulumi stack init. Stack creation means creation of the associated stack settings file and the instantiation of the stack state representation in the Pulumi backend. Initially, a stack starts as "empty", meaning that is has no resource representations stored in the backend, but it can become a target of a deployment initiated with the pulumi update command. Upon the first, and subsequent deployment, the stack backend representation is populated with resources. The state of a stack, as maintained by the backend, represents the "desired state" of the infrastructure, and Pulumi aims to bring the actual state in line with the desired state. Each stack has its own separate configuration, secretes, role-based controls (RBAC) and policies. Deep insight into its structure and components can be achieved by exporting the stack and examining the corresponding JSON file.
Stack Name
The stack name must be unique within a project and should express the intent behind a specific instantiation of the project infrastructure resource set, like "dev" or "testing". A pattern that seems reasonable is <project-name>-<function>, especially that in some circumstances, the project name is omitted from the stack name (like in <org-name>/<stack-name>. For more considerations on stack and project name, see project name. The stack name may only contain alphanumeric characters, hyphens, underscores and periods.
Fully Qualified Stack Name
A fully qualified stack name includes the organization and the project name: <org-name>/<project-name>/<stack-name>. If you are using Pulumi in your organization, and when new stacks are created, they will be created by default in your user account. To create the stack in the organization instead, name the stack using <org-name>/<stack-name>.
Stack URI
What is the semantics of a stack URI?
Multiple Stacks per Project
When a project is created with pulumi new, the configuration of a stack is also automatically created. Additional stack configurations can be created for an existing project with the pulumi stack init command. Note that multiple stacks per project means that all the stacks share the same program (behavior) but have different configurations, corresponding to different stack settings files. Note that pulumi stack init does not fully manage the corresponding stack setting file, it does create it, but it leaves it into an incomplete state.
Current (Active) Stack
When a project contains multiple stacks, one of them is active, or current, at any time. The current stack for a project can be displayed by running pulumi about in the project, or running pulumi stack ls, in which case the active stack will be marked with an asterisk. A stack can be set as active by running pulumi stack select.
Stack Configuration
While each stack of a project has a similar resource set, it will differ from other stacks in its configuration. Pulumi offers a configuration system to manage such differences. The system provides different methods to introspect and set configuration, including CLI and the programming model. The key-value pairs for any given stack of the project are stored in the stack's settings file.
TO PROCESS: https://www.pulumi.com/docs/intro/concepts/config/
Stack Settings File
Each stack within a project has an associated file named Pulumi.<stack-name>.yaml that contains configuration, in form of key/value pairs, for the stack. The file resides in the project root directory. Secret values are encrypted, which, according to the Pulumi documentation, makes them safe to be checked in into a repository. This is debatable. The stack settings for ephemeral stacks are typically not checked into source control. A stack setting file can be created and managed with the pulumi config command.
Stack settings files are an implementation of the Infrastructure as Code Stack Configuration File pattern.
Configuration Management via CLI
Stack configuration can be read and updated with the pulumi config command.
Programmatic Configuration Management
Config.
TO DO
Configuration Keys and Namespace
Key space. TO DO
Restoring Configuration from Backend
TO DO: pulumi config refresh
Stack Output (Exports)
A stack can be turned into a component that integrates with other components. The way Pulumi programs communicate information for external consumption and integration with other stacks is by using outputs, or exports. Outputs are values produced during an update and can be thought as programmatic results of a stack update, additional to the infrastructure resources created in the infrastructure platform. They are also known as "properties". They are maintained as part of the stack's state, by the backend service. If the stack was not deployed yet, no outputs are available, and the output values are resolved after pulumi up command completes. They can be retrieved via CLI with pulumi stack output <property-name>, displayed on the console or shared with other stacks, via stack references. Exposing an output is called "exporting". This is how it's done in Python:
import pulumi
...
pulumi.export("something", resource.someAttribute)
pulumi.export("color", "blue")
pulumi.export("details", {"shape": "square", "size": 10})
pulumi.export("whole_resource", resource)
Some resource properties are handled as secrets, and not displayed as part of the pulumi stack output unless the --show-secrets flag is used.
The value of a stack export can be a regular value, an Output or a Promise, effectively the same as an Input. They are JSON, though quotes are removed when exporting strings. An entire resource can be exported, and if that happens, it will be serialized as JSON. For example, if a SSM parameter is exported, it will be available as:
whole_resource {"allowed_pattern":"","arn":"arn:aws:ssm:us-west-2:999999999999:parameter/experimental/ovidiu/test2-382b9f1","data_type":"text","description":"","id":null,"key_id":"","name":"/experimental/ovidiu/test2-382b9f1","overwrite":null,"tags":{},"tags_all":{},"tier":"Standard","type":"String","urn":"urn:pulumi:ssm::blue::aws:ssm/parameter:Parameter::/experimental/ovidiu/test2","value":null,"version":1}
Stack References
Stack references are used by consumer stacks to get their dependencies. This is the Pulumi implementation of the Stack Data Lookup pattern: the stack reference allows access to the outputs of a provider stack from a consumer stack. To get values exposed by a producer stack, create an instance of StackReference representing the producer stack, from the consumer stack, and asynchronously access the values exported by producer via Outputs. The provider stack name must be fully qualified, including organization, project and stack name. Then, get the Output instance corresponding to the piece of data you want to access by invoking get_output() on the StackReference instance. The data must be previously exported by the provider stack with export(). Note that get_output() returns an Output instance. The actual data exported by the provider stack can be accessed asynchronously via apply() or other Output access patterns. Once a StackReference has been created, and exported data is accessed via Output instances, Pulumi understands the inter-stack dependency for scenarios like cascading updates.
# consumer stack code
import pulumi
provider = pulumi.StackReference("some-org/some-project/some-provider-stack")
output = provider.get_output("some_parameter"); # this is an Output instance
output.apply(lambda v: ... # do something with the value)
Stack Tags
Stacks have associated metadata as tags. Each tags has a name and a value. A set of built-in tags are automatically assigned and updated each time a stack is updated. Tags are only supported with the Pulumi Service backend. To view a stack's tags, run pulumi stack tag ls. Tags are useful to group stacks in the console.
Custom tags can be applied with pulumi stack tag set <name> <value>. Custom tags should not be prefixed with pulumi:, gitHub:, or vcs: to avoid conflicting with built-in tags that are assigned and updated with fresh values each time a stack is updated.
Built-in Tags
pulumi:project
pulumi:runtime
Example of a Python runtime section.
pulumi:description
gitHub:owner
gitHub:repo
vcs:owner
vcs:repo
vcs:kind
Stack Lifecycle
Create an Empty Stack
An empty stacks with the given name, ready for updates, can be created with pulumi stack init. The stack has no resources stored in the backend, but it can become a target of a deployment initiated with the pulumi update command. pulumi new command creates a default empty stack
Deploying a Stack
See Deploying a Project above.
Previewing a Stack
Refreshing a Stack
Destroying and Deleting a Stack
Destroying a stack means releasing and deleting resources associated with the stack with pulumi destroy. The command uses the latest configuration values, rather than the ones that were last used when the program was deployed. Pulumi waits until all resources are shut down and deleted before it considers the destroy operation to be complete.
Deleting a stack means removing all stack history from the backend and the stack configuration file Pulumi.<stack-name>.yaml. Use pulumi stack rm. A stack must be first destroyed, then deleted.
⚠️ Forcefully deleting a stack before destroying it may leave orphaned resources behind.
Stack Import/Export
A stack can be exported to see the raw data associated with the stack. The exported data can be manually modified, then imported to set the current state of the stack.
⚠️ Importing/exporting stacks in not a normal workflow, but an alternative workflow that subverts the usual way Pulumi manages resources and ensures immutable and repeatable infrastructure deployments. Importing an incorrect stack specification could lead to orphaning of cloud resources or the inability to make future updates to the stack.
Ephemeral Stack
Ephemeral stacks may have stack settings files.
Stack Operations
- List stacks
- Select a stack
- Display stack resources
- Display stack tags
- Create a stack
- Select a stack
- Remove (delete) a stack
- Export a stack
- Import a stack
Resource
Inputs and Outputs
State and Backends
Backend
The main job of a backend is to reliably store and manage the state of stacks, because Pulumi needs to understand the state of the infrastructure. Pulumi supports different backends: blob storage, locally, or in the Pulumi Service.
Among many other things, a backend provides dependencies to consumer stacks via the Stack Data Lookup pattern.
Name
URL
Authentication and Identity
The backend decides what a user can and cannot see, for example organizations, based on user's OD group configuration. A user's OD group list, as known by the Pulumi backend, is given by pulumi whoami --verbose.
Pulumi Service Backend
Pulumi SaaS:
Secrets
TO PROCESS: https://www.pulumi.com/docs/intro/concepts/secrets/
Logging
TO PROCESS: https://www.pulumi.com/docs/intro/concepts/logging/
Assets and Archives
TO PROCESS: https://www.pulumi.com/docs/intro/concepts/assets-archives/
Plugin
Plugin Operations
Function Serialization
TO PROCESS: https://www.pulumi.com/docs/intro/concepts/function-serialization/
Modularization
TO PROCESS: https://www.pulumi.com/blog/creating-and-reusing-cloud-components-using-package-managers/
Packages
Packages for various infrastructure providers are published in Pulumi registry.
Pulumi Registry
CLI
Auto-Naming
Automatically generates a random identifier and places it at the end of the given name, to avoid namespace collisions.
Automation API
Code Examples
Pulumi vs Terraform
Also see: