Blue-Green Deployments with Spinnaker: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
 
(43 intermediate revisions by the same user not shown)
Line 1: Line 1:
=External=
=External=
* https://blog.spinnaker.io/introducing-rollout-strategies-in-the-kubernetes-v2-provider-8bbffea109a
* https://spinnaker.io/docs/guides/user/kubernetes-v2/rollout-strategies/#redblack-rollouts
* https://spinnaker.io/docs/guides/user/kubernetes-v2/traffic-management/
* https://www.opsmx.com/blog/spinnaker-pipeline-blue-green-strategy-with-external-versioning-and-kubernetes-deployment-object/
* https://www.opsmx.com/blog/spinnaker-integration-with-istio/


=Internal=
=Internal=
Line 5: Line 10:
* [[Spinnaker#Subjects|Spinnaker]]
* [[Spinnaker#Subjects|Spinnaker]]
=Overview=
=Overview=
This article documents a pipeline that deploys a release in a Stage environment, waits for testing, the deploys the same release in Prod. The application is a Helm-packaged Kubernetes application, but some of the Kuberentes resources are created manually directly in Spinnaker.
This article documents a pipeline that deploys a release in a Stage environment, waits for testing, removes the release from Stage, and deploys the same release in Prod, while preserving the previous release in Prod. The Stage and Prod are serviced by two different namespaces ('of-stage' and 'of-prod'). The application is a Helm-packaged Kubernetes application, but some of the Kubernetes resources, such as the Ingress and the Service, are created manually directly in Spinnaker, one for each namespace. Also, ReplicaSets are used instead of Deployments, because Spinnaker does not handle Deployments well. It is fine if ReplicaSet is deployed as part of the Helm chart, instead of being created manually within the Spinnaker application.
 
=Prerequisites=
 
Create a Spinnaker Application as described here: {{Internal|Spinnaker_Create_and_Configure_an_Application#Create_an_Application|Create a Spinnaker Application}}
 
Create the "of-stage" and "of-prod" namespaces.


=Create Services=
=Create the Ingresses and Services=


Create two Services ("stage" and "prod") within the Spinnaker Application.
For each namespace ('of-stage' and 'of-prod'), create an Ingress and a Service each:


==<tt>of-stage</tt>==
Load Balancers → Create Load Balancer
Load Balancers → Create Load Balancer
Select the appropriate "account" (Kubernetes cluster)


Select the right "account" (Kubernetes cluster)
Use the following manifest (do not forget to adjust the namespace).


Manifest (do not forget to adjust the namespace):
===<tt>of-stage</tt> Ingress===
<syntaxhighlight lang='yaml'>
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: stage
  namespace: of-stage
  annotations:
    kubernetes.io/ingress.class: nginx
    ingress.beta.kubernetes.io/sni: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: 800m
    nginx.ingress.kubernetes.io/proxy-read-timeout: "200"
    nginx.ingress.kubernetes.io/upstream-hash-by: $binary_remote_addr
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
    - host: my-ingress.example.com
      http:
        paths:
          - path: /stage(/|$)(.*)
            pathType: ImplementationSpecific
            backend:
              service:
                name: stage
                port:
                  number: 8080
  tls:
    - hosts:
        - my-ingress.example.com
      secretName: my-secret
</syntaxhighlight>
===<tt>of-stage</tt> Service===
<syntaxhighlight lang='yaml'>
<syntaxhighlight lang='yaml'>
apiVersion: v1
apiVersion: v1
Line 21: Line 65:
metadata:
metadata:
   name: stage
   name: stage
   namespace: of02
   namespace: of-stage
spec:
spec:
   type: ClusterIP
   type: ClusterIP
   selector:
   selector:
     stage: 'true' # this label will be dynamically applied to the workload pods
     stage: 'true' # this label will be dynamically applied to the workload pods during deployment by Spinnaker
   ports:
   ports:
     - port: 8080
     - port: 8080
Line 31: Line 75:
       targetPort: 8080
       targetPort: 8080
</syntaxhighlight>
</syntaxhighlight>
==<tt>of-prod</tt>==
Load Balancers → Create Load Balancer
Select the appropriate "account" (Kubernetes cluster)
Use the following manifest (do not forget to adjust the namespace).
===<tt>of-prod</tt> Ingress===
<syntaxhighlight lang='yaml'>
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: prod
  namespace: of-prod
  annotations:
    kubernetes.io/ingress.class: nginx
    ingress.beta.kubernetes.io/sni: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: 800m
    nginx.ingress.kubernetes.io/proxy-read-timeout: "200"
    nginx.ingress.kubernetes.io/upstream-hash-by: $binary_remote_addr
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
    - host: my-ingress.example.com
      http:
        paths:
          - path: /prod(/|$)(.*)
            pathType: ImplementationSpecific
            backend:
              service:
                name: prod
                port:
                  number: 8080
  tls:
    - hosts:
        - my-ingress.example.com
      secretName: my-secret
</syntaxhighlight>
===<tt>of-prod</tt> Service===
<syntaxhighlight lang='yaml'>
<syntaxhighlight lang='yaml'>
apiVersion: v1
apiVersion: v1
Line 36: Line 118:
metadata:
metadata:
   name: prod
   name: prod
   namespace: of02
   namespace: of-prod
spec:
spec:
   type: ClusterIP
   type: ClusterIP
   selector:
   selector:
     prod: 'true' # this label will be dynamically applied to the workload pods
     prod: 'true' # this label will be dynamically applied to the workload pods during deployment by Spinnaker
   ports:
   ports:
     - port: 8080
     - port: 8080
Line 46: Line 128:
       targetPort: 8080
       targetPort: 8080
</syntaxhighlight>
</syntaxhighlight>
=Create or Import Required TLS Secrets=
=Create the Pipeline=
It will be a "provide image → deploy to stage → manual testing → manual judgement → deploy to prod" pipeline.
Name: "Stage - Manual Testing - Prod" (no "→" allowed in name)
==Provide Image Tag Stage==
Add stage → Evaluate Variables. Stage Name: "Provide Image Tag".
Variable name: "myapp_image_tag". Variable value: ${ parameters.myapp_image_tag }
Save Changes.
If the image tag is provided at execution as a variable, the pipeline must also be parameterized: Pipeline → Configure → Configuration → Parameters → Add parameter →
Name: myapp_image_tag
Label: no label
Required: check
Pin parameter: check
Description: "something"
==Helm Chart Rendering Stage==
Add stage → Bake (Manifest). Stage name: "Render Helm Chart". This stage will render the helm chart, apply the configuration overlay and overwrite the image tag. For more details, see: {{Internal|Spinnaker_Stage_Bake_(Manifest)#Overview|Bake (Manifest)}}


It's a good idea to test the pipeline execution immediately after this stage.


=Create the Pipeline=
==Stage Deployment Stage==
 
Add stage → Deploy (Manifest). Stage name: "Deploy in Stage". This stage will deploy the Helm chart and associate the workload with the "stage" service. For more details, see: {{Internal|Spinnaker_Stage_Deploy_(Manifest)#Overview|Deploy (Manifest)}}
 
<span id='Rollout_Strategy_Options'></span>At this stage, we do enable Rollout Strategy Options, so we can associate the workload with the "stage" service.


It will be a "deploy to stage → manual testing → manual judgement → deploy to prod" pipeline.
⚠️ If more than one service is used, ensure that the services do not share a common <code>metadata.labels</code> label key. If they do, there will be a "Service selector must have no label keys in common with target workload" error message when executing the stage.


Name: "Stage Manual Testing Prod"
Enable: "Spinnaker manages traffic based on your selected strategy" Service(s) Namespace → Service(s): "stage" Traffic: Send client requests to new pods → Strategy: "Highlander". ⚠️ "Highlander" rollout strategy will take care of removing previous stage releases.


=Organizatorium=
<font color=darkkhaki>
<font color=darkkhaki>
TO PROCESS:
* Link to detailed explanations of what happens for each of the rollout strategies from the point of view of 1) services 2) replicasets 3) pods.
* https://blog.spinnaker.io/introducing-rollout-strategies-in-the-kubernetes-v2-provider-8bbffea109a
* Understand the relationship between the replicasets and the Spinnaker cluster</font>
* https://spinnaker.io/docs/guides/user/kubernetes-v2/rollout-strategies/#redblack-rollouts
More details: {{Internal|Spinnaker_Stage_Deploy_(Manifest)#Rollout_Strategy_Options|Spinnaker Deployment Rollout Strategy Options}}
* https://spinnaker.io/docs/guides/user/kubernetes-v2/traffic-management/
 
* https://www.opsmx.com/blog/spinnaker-pipeline-blue-green-strategy-with-external-versioning-and-kubernetes-deployment-object/
==Manual Decision Stage==
* https://www.opsmx.com/blog/spinnaker-integration-with-istio/
 
Add stage → Manual Judgement. Stage name: "Wait on Stage Testing"
 
==Production Deployment Stage==
 
Add stage → Deploy (Manifest). Stage name: "Deploy in Prod". This stage will deploy the same Helm chart that was tested in Stage and associate the workload with the "prod" service. For more details, see: {{Internal|Spinnaker_Stage_Deploy_(Manifest)#Overview|Deploy (Manifest)}}
 
⚠️ If more than one service is used, ensure that the services do not share a common <code>metadata.labels</code> label key. If they do, there will be a "Service selector must have no label keys in common with target workload" error message when executing the stage.
 
Enable: "Spinnaker manages traffic based on your selected strategy" → Service(s) Namespace → Service(s): "prod" → Traffic: Send client requests to new pods → Strategy: "Red/Black". ⚠️ "Red/Black" rollout strategy will preserve previous production release.
 
<font color=darkkhaki>A certain number of ReplicaSets stay around, including the pods. How to manage them? They can be manually deleted from CLUSTERS → Select ReplicaSet to be deleted → ReplicaSet Actions → Delete.
</font>
</font>
More details: {{Internal|Spinnaker_Stage_Deploy_(Manifest)#Rollout_Strategy_Options|Spinnaker Deployment Rollout Strategy Options}}
=Rollback=
The rollback can be implemented either by creating a parallel pipeline based on [[Spinnaker_Stage_Undo_Rollout_(Manifest)|Undo Rollout (Manifest)]], or via a two-step UI procedure where the current "version" is disabled and the older "version" is enabled via UI.

Latest revision as of 01:46, 21 April 2022

External

Internal

Overview

This article documents a pipeline that deploys a release in a Stage environment, waits for testing, removes the release from Stage, and deploys the same release in Prod, while preserving the previous release in Prod. The Stage and Prod are serviced by two different namespaces ('of-stage' and 'of-prod'). The application is a Helm-packaged Kubernetes application, but some of the Kubernetes resources, such as the Ingress and the Service, are created manually directly in Spinnaker, one for each namespace. Also, ReplicaSets are used instead of Deployments, because Spinnaker does not handle Deployments well. It is fine if ReplicaSet is deployed as part of the Helm chart, instead of being created manually within the Spinnaker application.

Prerequisites

Create a Spinnaker Application as described here:

Create a Spinnaker Application

Create the "of-stage" and "of-prod" namespaces.

Create the Ingresses and Services

For each namespace ('of-stage' and 'of-prod'), create an Ingress and a Service each:

of-stage

Load Balancers → Create Load Balancer Select the appropriate "account" (Kubernetes cluster)

Use the following manifest (do not forget to adjust the namespace).

of-stage Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: stage
  namespace: of-stage
  annotations:
    kubernetes.io/ingress.class: nginx
    ingress.beta.kubernetes.io/sni: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: 800m
    nginx.ingress.kubernetes.io/proxy-read-timeout: "200"
    nginx.ingress.kubernetes.io/upstream-hash-by: $binary_remote_addr
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
    - host: my-ingress.example.com
      http:
        paths:
          - path: /stage(/|$)(.*)
            pathType: ImplementationSpecific
            backend:
              service:
                name: stage
                port:
                  number: 8080
  tls:
    - hosts:
        - my-ingress.example.com
      secretName: my-secret

of-stage Service

apiVersion: v1
kind: Service
metadata:
  name: stage
  namespace: of-stage
spec:
  type: ClusterIP
  selector:
    stage: 'true' # this label will be dynamically applied  to the workload pods during deployment by Spinnaker
  ports:
    - port: 8080
      name: http
      targetPort: 8080

of-prod

Load Balancers → Create Load Balancer Select the appropriate "account" (Kubernetes cluster)

Use the following manifest (do not forget to adjust the namespace).

of-prod Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: prod
  namespace: of-prod
  annotations:
    kubernetes.io/ingress.class: nginx
    ingress.beta.kubernetes.io/sni: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: 800m
    nginx.ingress.kubernetes.io/proxy-read-timeout: "200"
    nginx.ingress.kubernetes.io/upstream-hash-by: $binary_remote_addr
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
    - host: my-ingress.example.com
      http:
        paths:
          - path: /prod(/|$)(.*)
            pathType: ImplementationSpecific
            backend:
              service:
                name: prod
                port:
                  number: 8080
  tls:
    - hosts:
        - my-ingress.example.com
      secretName: my-secret

of-prod Service

apiVersion: v1
kind: Service
metadata:
  name: prod
  namespace: of-prod
spec:
  type: ClusterIP
  selector:
    prod: 'true' # this label will be dynamically applied  to the workload pods during deployment by Spinnaker
  ports:
    - port: 8080
      name: http
      targetPort: 8080

Create or Import Required TLS Secrets

Create the Pipeline

It will be a "provide image → deploy to stage → manual testing → manual judgement → deploy to prod" pipeline.

Name: "Stage - Manual Testing - Prod" (no "→" allowed in name)

Provide Image Tag Stage

Add stage → Evaluate Variables. Stage Name: "Provide Image Tag".

Variable name: "myapp_image_tag". Variable value: ${ parameters.myapp_image_tag }

Save Changes.

If the image tag is provided at execution as a variable, the pipeline must also be parameterized: Pipeline → Configure → Configuration → Parameters → Add parameter →

Name: myapp_image_tag

Label: no label

Required: check

Pin parameter: check

Description: "something"

Helm Chart Rendering Stage

Add stage → Bake (Manifest). Stage name: "Render Helm Chart". This stage will render the helm chart, apply the configuration overlay and overwrite the image tag. For more details, see:

Bake (Manifest)

It's a good idea to test the pipeline execution immediately after this stage.

Stage Deployment Stage

Add stage → Deploy (Manifest). Stage name: "Deploy in Stage". This stage will deploy the Helm chart and associate the workload with the "stage" service. For more details, see:

Deploy (Manifest)

At this stage, we do enable Rollout Strategy Options, so we can associate the workload with the "stage" service.

⚠️ If more than one service is used, ensure that the services do not share a common metadata.labels label key. If they do, there will be a "Service selector must have no label keys in common with target workload" error message when executing the stage.

Enable: "Spinnaker manages traffic based on your selected strategy" → Service(s) Namespace → Service(s): "stage" → Traffic: Send client requests to new pods → Strategy: "Highlander". ⚠️ "Highlander" rollout strategy will take care of removing previous stage releases.

  • Link to detailed explanations of what happens for each of the rollout strategies from the point of view of 1) services 2) replicasets 3) pods.
  • Understand the relationship between the replicasets and the Spinnaker cluster

More details:

Spinnaker Deployment Rollout Strategy Options

Manual Decision Stage

Add stage → Manual Judgement. Stage name: "Wait on Stage Testing"

Production Deployment Stage

Add stage → Deploy (Manifest). Stage name: "Deploy in Prod". This stage will deploy the same Helm chart that was tested in Stage and associate the workload with the "prod" service. For more details, see:

Deploy (Manifest)

⚠️ If more than one service is used, ensure that the services do not share a common metadata.labels label key. If they do, there will be a "Service selector must have no label keys in common with target workload" error message when executing the stage.

Enable: "Spinnaker manages traffic based on your selected strategy" → Service(s) Namespace → Service(s): "prod" → Traffic: Send client requests to new pods → Strategy: "Red/Black". ⚠️ "Red/Black" rollout strategy will preserve previous production release.

A certain number of ReplicaSets stay around, including the pods. How to manage them? They can be manually deleted from CLUSTERS → Select ReplicaSet to be deleted → ReplicaSet Actions → Delete.

More details:

Spinnaker Deployment Rollout Strategy Options

Rollback

The rollback can be implemented either by creating a parallel pipeline based on Undo Rollout (Manifest), or via a two-step UI procedure where the current "version" is disabled and the older "version" is enabled via UI.