Blue-Green Deployments with Spinnaker

External

• https://blog.spinnaker.io/introducing-rollout-strategies-in-the-kubernetes-v2-provider-8bbffea109a
• https://spinnaker.io/docs/guides/user/kubernetes-v2/rollout-strategies/#redblack-rollouts
• https://spinnaker.io/docs/guides/user/kubernetes-v2/traffic-management/
• https://www.opsmx.com/blog/spinnaker-pipeline-blue-green-strategy-with-external-versioning-and-kubernetes-deployment-object/
• https://www.opsmx.com/blog/spinnaker-integration-with-istio/

Internal

• Blue-Green Deployments
• Spinnaker

Overview

This article documents a pipeline that deploys a release in a Stage environment, waits for testing, removes the release from Stage, and deploys the same release in Prod, while preserving the previous release in Prod. The Stage and Prod are serviced by two different namespaces ('of-stage' and 'of-prod'). The application is a Helm-packaged Kubernetes application, but some of the Kubernetes resources, such as the Ingress and the Service, are created manually directly in Spinnaker, one for each namespace. Also, ReplicaSets are used instead of Deployments, because Spinnaker does not handle Deployments well. It is fine if ReplicaSet is deployed as part of the Helm chart, instead of being created manually within the Spinnaker application.

Prerequisites

Create a Spinnaker Application as described here:

Create a Spinnaker Application

Create the "of-stage" and "of-prod" namespaces.

Create the Ingresses and Services

For each namespace ('of-stage' and 'of-prod'), create an Ingress and a Service each:

of-stage

Load Balancers → Create Load Balancer Select the appropriate "account" (Kubernetes cluster)

Use the following manifest (do not forget to adjust the namespace).

of-stage Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: stage
  namespace: of-stage
  annotations:
    kubernetes.io/ingress.class: nginx
    ingress.beta.kubernetes.io/sni: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: 800m
    nginx.ingress.kubernetes.io/proxy-read-timeout: "200"
    nginx.ingress.kubernetes.io/upstream-hash-by: $binary_remote_addr
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
    - host: my-ingress.example.com
      http:
        paths:
          - path: /stage(/|$)(.*)
            pathType: ImplementationSpecific
            backend:
              service:
                name: stage
                port:
                  number: 8080
  tls:
    - hosts:
        - my-ingress.example.com
      secretName: my-secret

of-stage Service

apiVersion: v1
kind: Service
metadata:
  name: stage
  namespace: of-stage
spec:
  type: ClusterIP
  selector:
    stage: 'true' # this label will be dynamically applied  to the workload pods during deployment by Spinnaker
  ports:
    - port: 8080
      name: http
      targetPort: 8080

of-prod

Load Balancers → Create Load Balancer Select the appropriate "account" (Kubernetes cluster)

Use the following manifest (do not forget to adjust the namespace).

of-prod Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: prod
  namespace: of-prod
  annotations:
    kubernetes.io/ingress.class: nginx
    ingress.beta.kubernetes.io/sni: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: 800m
    nginx.ingress.kubernetes.io/proxy-read-timeout: "200"
    nginx.ingress.kubernetes.io/upstream-hash-by: $binary_remote_addr
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
    - host: my-ingress.example.com
      http:
        paths:
          - path: /prod(/|$)(.*)
            pathType: ImplementationSpecific
            backend:
              service:
                name: prod
                port:
                  number: 8080
  tls:
    - hosts:
        - my-ingress.example.com
      secretName: my-secret

of-prod Service

apiVersion: v1
kind: Service
metadata:
  name: prod
  namespace: of-prod
spec:
  type: ClusterIP
  selector:
    prod: 'true' # this label will be dynamically applied  to the workload pods during deployment by Spinnaker
  ports:
    - port: 8080
      name: http
      targetPort: 8080

Create the Pipeline

It will be a "provide image → deploy to stage → manual testing → manual judgement → deploy to prod" pipeline.

Name: "Stage - Manual Testing - Prod" (no "→" allowed in name)

Provide Image Tag Stage

Add stage → Evaluate Variables. Stage Name: "Provide Image Tag".

Variable name: "myapp_image_tag". Variable value: ${ parameters.myapp_image_tag }

Save Changes.

If the image tag is provided at execution as a variable, the pipeline must also be parameterized: Pipeline → Configure → Configuration → Parameters → Add parameter →

Name: myapp_image_tag

Label: no label

Required: check

Pin parameter: check

Description: "something"

Helm Chart Rendering Stage

Add stage → Bake (Manifest). Stage name: "Render Helm Chart". This stage will render the helm chart, apply the configuration overlay and overwrite the image tag. For more details, see:

Bake (Manifest)

It's a good idea to test the pipeline execution immediately after this stage.

Stage Deployment Stage

Add stage → Deploy (Manifest). Stage name: "Deploy in Stage". This stage will deploy the Helm chart and associate the workload with the "stage" service. For more details, see:

Deploy (Manifest)

At this stage, we do enable Rollout Strategy Options, so we can associate the workload with the "stage" service.

⚠️ If more than one service is used, ensure that the services do not share a common metadata.labels label key. If they do, there will be a "Service selector must have no label keys in common with target workload" error message when executing the stage.

Enable: "Spinnaker manages traffic based on your selected strategy" → Service(s) Namespace → Service(s): "stage" → Traffic: Send client requests to new pods → Strategy: "Highlander". ⚠️ "Highlander" rollout strategy will take care of removing previous stage releases.

• Link to detailed explanations of what happens for each of the rollout strategies from the point of view of 1) services 2) replicasets 3) pods.
• Understand the relationship between the replicasets and the Spinnaker cluster

More details:

Spinnaker Deployment Rollout Strategy Options

Manual Decision Stage

Add stage → Manual Judgement. Stage name: "Wait on Stage Testing"

Production Deployment Stage

Add stage → Deploy (Manifest). Stage name: "Deploy in Prod". This stage will deploy the same Helm chart that was tested in Stage and associate the workload with the "prod" service. For more details, see:

Deploy (Manifest)

⚠️ If more than one service is used, ensure that the services do not share a common metadata.labels label key. If they do, there will be a "Service selector must have no label keys in common with target workload" error message when executing the stage.

Enable: "Spinnaker manages traffic based on your selected strategy" → Service(s) Namespace → Service(s): "prod" → Traffic: Send client requests to new pods → Strategy: "Red/Black". ⚠️ "Red/Black" rollout strategy will preserve previous production release.

A certain number of ReplicaSets stay around, including the pods. How to manage them? They can be manually deleted from CLUSTERS → Select ReplicaSet to be deleted → ReplicaSet Actions → Delete.

More details:

Spinnaker Deployment Rollout Strategy Options

Rollback

The rollback can be implemented either by creating a parallel pipeline based on Undo Rollout (Manifest), or via a two-step UI procedure where the current "version" is disabled and the older "version" is enabled via UI.