Pulumi Programming Model: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
 
(48 intermediate revisions by the same user not shown)
Line 4: Line 4:


=Overview=
=Overview=
In Pulumi, infrastructure resources are described programmatically in a [[Pulumi_Programming_Model#Program|program]], that gets executed when the <code>[[Pulumi_Operations#Project_Update_Dry_Run|pulumi preview]]</code> and <code>[[Pulumi_Operations#Project_Update|pulumi up]]</code> are run. At the end of the program execution, the '''desired state''' of the infrastructure is built in memory and it is compared with the actual state, pulled from the Pulumi backend, specifically from the state file associated with the active stack the program is being executed in the context of. Note that a stack's state file is a cache of the actual state of the infrastructure platform backend, and it can be synchronized with pulumi refresh.
In Pulumi, infrastructure resources are described programmatically in a [[Pulumi_Programming_Model#Program|program]], that gets executed when the <code>[[Pulumi_Operations#Project_Update_Dry_Run|pulumi preview]]</code> and <code>[[Pulumi_Operations#Project_Update|pulumi up]]</code> are run. At the end of the program execution, the '''desired state''' of the infrastructure is built in memory and it is compared with the '''actual state''', pulled from the Pulumi backend, specifically from the state file associated with the active stack the program is being executed in the context of.  


Depending on delta between the desired state and the actual state, the Pulumi client takes various actions: infrastructure resources are created, updated or deleted. It is the Pulumi client that interacts with the infrastructure platform, so proper access credentials must exist in the environment that runs the Pulumi client.
Before any interaction with the infrastructure platform could takes place, Pulumi needs to build a context to maintain a model of the infrastructure resources, as they exist in the infrastructure platform backend - the actual state. That is the [[Pulumi_Concepts#Stack|stack]]. A stack contains metadata and a state file, which has the actual state of the resources in the stack, as known at the moment. This is how [[#Creating_an_Empty_Stack|an empty stack is created]]. Note that the state file maintained by a stack is a cache of the actual state of the infrastructure platform backend, and it can be synchronized with <code>[[Pulumi_Operations#Refresh_Stack_State|pulumi refresh]]</code>. Depending on delta between the desired state and the actual state, the Pulumi client takes various actions: infrastructure resources are created, updated or deleted. It is the Pulumi client that interacts with the infrastructure platform, so proper access credentials must exist in the environment that runs the Pulumi client.
 
[[File:Pulumi_Programming_Model.png|834px]]


While the rest of this article uses Python as example, Pulumi offers SDKs for [[Python_Pulumi#Python_Pulumi_Programming_Model|Python]], [[TypeScript Pulumi#Overview|TypeScript]] and [[Go Pulumi#Overview|Go]].
While the rest of this article uses Python as example, Pulumi offers SDKs for [[Python_Pulumi#Python_Pulumi_Programming_Model|Python]], [[TypeScript Pulumi#Overview|TypeScript]] and [[Go Pulumi#Overview|Go]].


=Creating an Empty Stack=
=Creating an Empty Stack=
A project may contain several stacks out of which one is active.


Before any interaction with the infrastructure platform takes place, the Pulumi client needs to build a context that maintains a model of the infrastructure resources, as they exist in the infrastructure platform backend. That is the stack. A stack contains metadata and a state file, which carries the actual state of the resources in the stack, as known at the moment.
To list all stacks in the project:
<syntaxhighlight lang='bash'>
pulumi stack ls
</syntaxhighlight>
More details on listing stacks: {{Internal|Pulumi_Operations#List_Stacks|List Stacks}}


To create an empty stack in the project:
<syntaxhighlight lang='bash'>
pulumi stack init <org-name>/<stack-name>
</syntaxhighlight>
More details on creating empty stacks: {{Internal|Pulumi_Operations#Create_Stack|Create Stacks}}
The stack metadata in a raw format can be displayed with:
<syntaxhighlight lang='bash'>
pulumi stack export
</syntaxhighlight>
More details about exporting stacks: {{Internal|Pulumi_Operations#Export_a_Stack|Export a Stack}}


=Creating an Infrastructure Resource in Code=
=Creating an Infrastructure Resource in Code=
Starting with an empty stack, the simplest way to provision an infrastructure resource is to write the code that creates an instance of that resource, using the API exposed by the [[Pulumi_Architecture#Resource_Provider|dedicated resource provider]].
This is an example of a Python program, saved in a <code>__main__.py</code> in a Pulumi project:
<syntaxhighlight lang='py'>
import pulumi_datadog
dashboard_json = """
{
    "title": "Blue",
    "layout_type": "ordered",
    "reflow_type": "fixed",
    "widgets": [
        {
            "definition": {
                "color": "#4d4d4d",
                "font_size": "auto",
                "text": "This is some sample text.",
                "text_align": "left",
                "type": "free_text"
            },
            "layout": {
                "height": 1,
                "width": 2,
                "x": 0,
                "y": 0
            }
        }
    ]
}
"""
pulumi_datadog.DashboardJson("Dashboard-01", dashboard=dashboard_json)
</syntaxhighlight>
Upon the execution of the <code>pulumi up</code> command, the Pulumi runtime builds the desired state in memory and asks for the permission to perform the update:
<font size=-2>
pulumi up
<font color=magenta>Previewing update (aiml-dp/ovidiu-experimental-datadog-dashboard-from-json/main)</font>
      Type                            Name                                                  Plan
  <font color=green>+</font>  pulumi:pulumi:Stack            ovidiu-experimental-datadog-dashboard-from-json-main  <font color=green>create</font>
  <font color=green>+</font>  └─ datadog:index:DashboardJson  Dashboard-01                                          <font color=green>create</font>
<font color=magenta>Resources:</font>
    <font color=green>+ 2 to create</font>
<font color=cyan>Do you want to perform this update?</font>  [Use arrows to move, enter to select, type to filter]
  yes
> no
  details
</font>
If the permission is granted, Pulumi reconciles the desired state (one resource) and actual state (no resource) and crates the resource in the infrastructure platform backend. At the same time, the stack state is updated.


=Updating an Infrastructure Resource in Code=
=Updating an Infrastructure Resource in Code=
Upon the initial infrastructure resource creation, it state exists in three places: in the infrastructure platform itself, in the stack state file and in the program. To update the infrastructure resource from the program, update the code, and run <code>pulumi up</code> again. We change the JSON representation and update the color to dark blue:
<syntaxhighlight lang='py'>
import pulumi_datadog
dashboard_json = """
{
    "title": "Blue",
    ...
    "widgets": [
        {
            "definition": {
                "color": "#00008B",
              ...
            },
        ...
    ]
}
"""
pulumi_datadog.DashboardJson("Dashboard-01", dashboard=dashboard_json)
</syntaxhighlight>
<code>pulumi up</code> will build the desired state (which includes the dark blue color) in memory, will compare it with the stored actual state, and will detect differences:
<font size=-2>
pulumi up
<font color=magenta>Previewing update (aiml-dp/ovidiu-experimental-datadog-dashboard-from-json/main)</font>
      Type                            Name                                                  Plan      Info
      pulumi:pulumi:Stack            ovidiu-experimental-datadog-dashboard-from-json-main
  <font color=darkkhaki>~</font>  └─ datadog:index:DashboardJson  Dashboard-01                                          <font color=darkkhaki>update</font>    [diff: <font color=darkkhaki>~dashboard</font>]
Resources:
    <font color=darkkhaki>~ 1 to update</font>
    1 unchanged
<font color=cyan>Do you want to perform this update?</font>  [Use arrows to move, enter to select, type to filter]
  yes
> no
  details
</font>
If allowed, <code>pulumi up</code> will initiate infrastructure resource update. At the end of the process, all three states stored in code, stack and infrastructure platform will be reconciled.


=Importing (Adopting) an Infrastructure Resource=
=Importing (Adopting) an Infrastructure Resource=
An existing infrastructure resource can be imported (adopted) into an existing stack. This action allows it to be managed by Pulumi via the usual Pulumi workflows.


=Deleting an Infrastructure Resource in Code=
Assuming an empty stack and a "Green" dashboard that displays some green text in a widget, the import process consists of running the <code>pulumi import</code> and '''reconciling the state of the program with the infrastructure resource state'''.
 
To run <code>pulumi import</code>, the infrastructure platform resource ID is required.
 
<syntaxhighlight lang='bash'>
pulumi import datadog:index/dashboardJson:DashboardJson 'adopted-dashboard-01' aeu-qi8-d2c
</syntaxhighlight>
 
After import in the stack state, it is very important to also update the program with the same state. The exact code to use is provided by the <code>pulumi import</code> command at stdout:
<font size=-2>
...
<font color=magenta>Resources:</font>
    <font color=green>+ 1 created
    = 1 imported</font>
    <b>2 changes</b>
<font color=magenta>Duration: 2s</font>
Please copy the following code into your Pulumi application. Not doing so
will cause Pulumi to report that an update will happen on the next update command.
Please note that the imported resources are marked as protected. To destroy them
you will need to remove the `protect` option and run `pulumi update` *before*
the destroy will take effect.
import pulumi
import pulumi_datadog as datadog
adopted_dashboard_01 = datadog.DashboardJson("adopted-dashboard-01",
    dashboard="{\"description\":null,\"layout_type\":\"ordered\",\"notify_list\":[],\"reflow_type\":\"fixed\",\"restricted_roles\":[],\"template_variables\":[],\"title\":\"Green\",\"widgets\":[{\"definition\":{\"color\":\"#41c464\",\"font_size\":\"auto\",\"text\":\"This is some green text.\",\"text_align\":\"left\",\"type\":\"free_text\"},\"layout\":{\"height\":1,\"width\":2,\"x\":0,\"y\":0}}]}",
    url="/dashboard/aeu-qi8-d2c/green",
    opts=pulumi.ResourceOptions(protect=True))
</font>


In consequence, the program should be updated as follows:
<syntaxhighlight lang='py'>
import pulumi
import pulumi_datadog as datadog


I think there is a misunderstanding here. Apologies if the below info is redundant, but I think we can clear this up.
adopted_dashboard_01 = datadog.DashboardJson(
Pulumi uses a statefile to capture a “current” state of resources contained in the current stack, from a physical cloud provider perspective and a pulumi metadata perspective. The statefile is just that, a file or dumb storage that is the result of a pulumi up.
    "adopted-dashboard-01",
When you execute a pulumi up against a Pulumi program, Pulumi is taking the existing statefile (if one exists) and comparing it to the desired state, which is declared by the code in your Pulumi program.
    dashboard="{\"description\":null,\"layout_type\":\"ordered\",\"notify_list\":[],"
So what is happening here is, you’ve imported the resource into your Pulumi state and you’ve not included the resource definition (in code) in your Pulumi program. So when you run a pulumi up Pulumi looks at the statefile, sees the datadog dashboard and compares it with your code, which does not contain the resource definition and correctly determines it needs to delete the resource
              "\"reflow_type\":\"fixed\",\"restricted_roles\":[],\"template_variables\":[],"
The Pulumi programming model defines the core concepts in use when creating infrastructure as code [[#Program|programs]].  
              "\"title\":\"Green\",\"widgets\":[{\"definition\":{\"color\":\"#41c464\","
              "\"font_size\":\"auto\",\"text\":\"This is some green text.\","
              "\"text_align\":\"left\",\"type\":\"free_text\"},\"layout\":{\"height\":1,"
              "\"width\":2,\"x\":0,\"y\":0}}]}",
    url="/dashboard/aeu-qi8-d2c/green",
    opts=pulumi.ResourceOptions(protect=True))
</syntaxhighlight>


Start with Programming Model https://kb.novaordis.com/index.php/Pulumi_Concepts#Programming_Model as a central place to specify that while the “program” runs, the program builds a model of the infrastructure that is then synchronized with the infrastructure platform.
If the program is not updated as such, when <code>pulumi up</code> is executed the next time, the in-memory model will not contain the definition of "adopted-dashboard-01", which exists in the stack state and the infrastructure platform state, so Pulumi will attempt to delete it from the infrastructure platform.
Remove “Code Example” https://kb.novaordis.com/index.php/Pulumi_Concepts#Code_Examples  section work out https://kb.novaordis.com/index.php/Python_Pulumi_Code_Examples page into the “Programming Model”
Move the content of “Python Pulumi Code Examples” in-line into the “Programming Model” section and resolve the links pointing to the “Python Pulumi Code Examples”
Link to https://kb.novaordis.com/index.php/Pulumi_Datadog_DashboardJson#Internal Programming Model.
Describe in detail these two models (the second if we don’t want to change anything)


Note that using:
<syntaxhighlight lang='py'>
id = 'aeu-qi8-d2c'
datadog.DashboardJson.get('adopted-dashboard-01', id)
</syntaxhighlight>
is not a good idea. This function all will update the stack state and make 'adopted-dashboard-01' "external", which is not what we want.


For more details on importing individual resources into a stack see: {{Internal|Pulumi_Operations#Import_Individual_External_Resources_into_a_Stack|Import Individual External Resources into a Stack}}


Understand the difference between stack configuration and stack output.
=Deleting an Infrastructure Resource in Code=
To delete an infrastructure resource from code, simply remove from the program the API calls that instantiates that resource. Upon execution, that instance will be missing from the desired state, and Pulumi will remove it from the infrastructure platform and synchronize the stack state.
=Other Code Examples=
{{External|https://github.com/pulumi/examples}}
* [[Python Pulumi Working with Files|Working with Files]]
* [[Python Pulumi AWS SSM Parameter|AWS SSM Parameter]]
* [[Python Pulumi AWS S3|AWS S3]]
* [[Python Pulumi Kubernetes| Kubernetes]]
* [[Pulumi_Datadog_DashboardJson#Programming_Model|Datadog DashboardJson Programming Model]]

Latest revision as of 01:02, 15 April 2022

Internal

Overview

In Pulumi, infrastructure resources are described programmatically in a program, that gets executed when the pulumi preview and pulumi up are run. At the end of the program execution, the desired state of the infrastructure is built in memory and it is compared with the actual state, pulled from the Pulumi backend, specifically from the state file associated with the active stack the program is being executed in the context of.

Before any interaction with the infrastructure platform could takes place, Pulumi needs to build a context to maintain a model of the infrastructure resources, as they exist in the infrastructure platform backend - the actual state. That is the stack. A stack contains metadata and a state file, which has the actual state of the resources in the stack, as known at the moment. This is how an empty stack is created. Note that the state file maintained by a stack is a cache of the actual state of the infrastructure platform backend, and it can be synchronized with pulumi refresh. Depending on delta between the desired state and the actual state, the Pulumi client takes various actions: infrastructure resources are created, updated or deleted. It is the Pulumi client that interacts with the infrastructure platform, so proper access credentials must exist in the environment that runs the Pulumi client.

Pulumi Programming Model.png

While the rest of this article uses Python as example, Pulumi offers SDKs for Python, TypeScript and Go.

Creating an Empty Stack

A project may contain several stacks out of which one is active.

To list all stacks in the project:

pulumi stack ls

More details on listing stacks:

List Stacks

To create an empty stack in the project:

pulumi stack init <org-name>/<stack-name>

More details on creating empty stacks:

Create Stacks

The stack metadata in a raw format can be displayed with:

pulumi stack export

More details about exporting stacks:

Export a Stack

Creating an Infrastructure Resource in Code

Starting with an empty stack, the simplest way to provision an infrastructure resource is to write the code that creates an instance of that resource, using the API exposed by the dedicated resource provider.

This is an example of a Python program, saved in a __main__.py in a Pulumi project:

import pulumi_datadog

dashboard_json = """
{
    "title": "Blue",
    "layout_type": "ordered",
    "reflow_type": "fixed",
    "widgets": [
        {
            "definition": {
                "color": "#4d4d4d",
                "font_size": "auto",
                "text": "This is some sample text.",
                "text_align": "left",
                "type": "free_text"
            },
            "layout": {
                "height": 1,
                "width": 2,
                "x": 0,
                "y": 0
            }
        }
    ]
}
"""

pulumi_datadog.DashboardJson("Dashboard-01", dashboard=dashboard_json)

Upon the execution of the pulumi up command, the Pulumi runtime builds the desired state in memory and asks for the permission to perform the update:

pulumi up
Previewing update (aiml-dp/ovidiu-experimental-datadog-dashboard-from-json/main)

     Type                            Name                                                  Plan
 +   pulumi:pulumi:Stack             ovidiu-experimental-datadog-dashboard-from-json-main  create
 +   └─ datadog:index:DashboardJson  Dashboard-01                                          create

Resources:
    + 2 to create

Do you want to perform this update?  [Use arrows to move, enter to select, type to filter]
  yes
> no
  details

If the permission is granted, Pulumi reconciles the desired state (one resource) and actual state (no resource) and crates the resource in the infrastructure platform backend. At the same time, the stack state is updated.

Updating an Infrastructure Resource in Code

Upon the initial infrastructure resource creation, it state exists in three places: in the infrastructure platform itself, in the stack state file and in the program. To update the infrastructure resource from the program, update the code, and run pulumi up again. We change the JSON representation and update the color to dark blue:

import pulumi_datadog

dashboard_json = """
{
    "title": "Blue",
    ...
    "widgets": [
        {
            "definition": {
                "color": "#00008B",
               ...
            },
        ...
    ]
}
"""

pulumi_datadog.DashboardJson("Dashboard-01", dashboard=dashboard_json)

pulumi up will build the desired state (which includes the dark blue color) in memory, will compare it with the stored actual state, and will detect differences:

pulumi up
Previewing update (aiml-dp/ovidiu-experimental-datadog-dashboard-from-json/main)

     Type                            Name                                                  Plan       Info
     pulumi:pulumi:Stack             ovidiu-experimental-datadog-dashboard-from-json-main
 ~   └─ datadog:index:DashboardJson  Dashboard-01                                          update     [diff: ~dashboard]

Resources:
    ~ 1 to update
    1 unchanged

Do you want to perform this update?  [Use arrows to move, enter to select, type to filter]
  yes
> no
  details

If allowed, pulumi up will initiate infrastructure resource update. At the end of the process, all three states stored in code, stack and infrastructure platform will be reconciled.

Importing (Adopting) an Infrastructure Resource

An existing infrastructure resource can be imported (adopted) into an existing stack. This action allows it to be managed by Pulumi via the usual Pulumi workflows.

Assuming an empty stack and a "Green" dashboard that displays some green text in a widget, the import process consists of running the pulumi import and reconciling the state of the program with the infrastructure resource state.

To run pulumi import, the infrastructure platform resource ID is required.

pulumi import datadog:index/dashboardJson:DashboardJson 'adopted-dashboard-01' aeu-qi8-d2c

After import in the stack state, it is very important to also update the program with the same state. The exact code to use is provided by the pulumi import command at stdout:

...
Resources:
    + 1 created
    = 1 imported
    2 changes

Duration: 2s

Please copy the following code into your Pulumi application. Not doing so
will cause Pulumi to report that an update will happen on the next update command.

Please note that the imported resources are marked as protected. To destroy them
you will need to remove the `protect` option and run `pulumi update` *before*
the destroy will take effect.

import pulumi
import pulumi_datadog as datadog

adopted_dashboard_01 = datadog.DashboardJson("adopted-dashboard-01",
   dashboard="{\"description\":null,\"layout_type\":\"ordered\",\"notify_list\":[],\"reflow_type\":\"fixed\",\"restricted_roles\":[],\"template_variables\":[],\"title\":\"Green\",\"widgets\":[{\"definition\":{\"color\":\"#41c464\",\"font_size\":\"auto\",\"text\":\"This is some green text.\",\"text_align\":\"left\",\"type\":\"free_text\"},\"layout\":{\"height\":1,\"width\":2,\"x\":0,\"y\":0}}]}",
   url="/dashboard/aeu-qi8-d2c/green",
   opts=pulumi.ResourceOptions(protect=True))

In consequence, the program should be updated as follows:

import pulumi
import pulumi_datadog as datadog

adopted_dashboard_01 = datadog.DashboardJson(
    "adopted-dashboard-01",
    dashboard="{\"description\":null,\"layout_type\":\"ordered\",\"notify_list\":[],"
              "\"reflow_type\":\"fixed\",\"restricted_roles\":[],\"template_variables\":[],"
              "\"title\":\"Green\",\"widgets\":[{\"definition\":{\"color\":\"#41c464\","
              "\"font_size\":\"auto\",\"text\":\"This is some green text.\","
              "\"text_align\":\"left\",\"type\":\"free_text\"},\"layout\":{\"height\":1,"
              "\"width\":2,\"x\":0,\"y\":0}}]}",
    url="/dashboard/aeu-qi8-d2c/green",
    opts=pulumi.ResourceOptions(protect=True))

If the program is not updated as such, when pulumi up is executed the next time, the in-memory model will not contain the definition of "adopted-dashboard-01", which exists in the stack state and the infrastructure platform state, so Pulumi will attempt to delete it from the infrastructure platform.

Note that using:

id = 'aeu-qi8-d2c'
datadog.DashboardJson.get('adopted-dashboard-01', id)

is not a good idea. This function all will update the stack state and make 'adopted-dashboard-01' "external", which is not what we want.

For more details on importing individual resources into a stack see:

Import Individual External Resources into a Stack

Deleting an Infrastructure Resource in Code

To delete an infrastructure resource from code, simply remove from the program the API calls that instantiates that resource. Upon execution, that instance will be missing from the desired state, and Pulumi will remove it from the infrastructure platform and synchronize the stack state.

Other Code Examples

https://github.com/pulumi/examples