Blue/Green Deployment (2): Implementing It on Kubernetes

Blue/Green on Kubernetes, built up through three stages - Service selector, Argo Rollouts, and Istio - and how to migrate the database alongside it.

TL;DR

Blue/Green on Kubernetes works with nothing but a Service selector, but automating the operational side is hard.
In production, reach for Argo Rollouts first so that promotion, verification, and rollback are automated.
Adding Istio extends this to gradual traffic shifting, traffic mirroring, and header-based routing.

Blue/Green on Kubernetes at a glance

The first decision is how traffic routing will be controlled. A realistic path is to grow through Service selector -> Argo Rollouts -> Istio as operational maturity increases.

Kubernetes provides declarative deployment management, but the default strategy is a rolling update. Blue/Green requires extra configuration.

Comparing the options

Service Selector: low complexity, manual promotion and rollback, best for learning and small deployments
Argo Rollouts: medium complexity, automated promotion and rollback, best for production
Istio: high complexity, automated promotion and rollback, best for large scale and complex routing
Flagger: medium complexity, automated promotion and rollback, best for GitOps environments (where a Git repository is the single source of truth for deployment state)

Plain Kubernetes

This approach fits learning and a quick Proof of Concept (PoC) best. The cost is that promotion checks, failure detection, and rollback triggers all have to be scripted by hand.

The most basic method is to change the Service selector to shift traffic.

Architecture

Blue deployment

yaml

# blue-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-blue
  labels:
    app: my-app
    color: blue
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
      color: blue
  template:
    metadata:
      labels:
        app: my-app
        color: blue
        version: v1.0.0
    spec:
      containers:
      - name: app
        image: my-app:1.0.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 20

Green deployment

The Green manifest has the same structure as Blue. Only five lines change: the identifying name and color label, and the version and image. replicas, resources, the probes, and the port are identical.

diff

# green-deployment.yaml - only these lines differ from blue-deployment.yaml
 metadata:
-  name: my-app-blue
+  name: my-app-green
   labels:
     app: my-app
-    color: blue
+    color: green
 spec:
   replicas: 3
   selector:
     matchLabels:
       app: my-app
-      color: blue
+      color: green
   template:
     metadata:
       labels:
         app: my-app
-        color: blue
-        version: v1.0.0
+        color: green
+        version: v1.1.0
     spec:
       containers:
       - name: app
-        image: my-app:1.0.0
+        image: my-app:1.1.0
         # ports, resources, readinessProbe, and livenessProbe are identical to Blue

The color label is what the Service selector matches on, so routing is decided entirely by this value being distinct between the two Deployments.

Service (the traffic router)

yaml

# service.yaml
apiVersion: v1
kind: Service
metadata:
  name: my-app
spec:
  type: ClusterIP
  selector:
    app: my-app
    color: blue  # change this to green to switch over
  ports:
    - name: http
      port: 80
      targetPort: 8080

Switchover script

bash

#!/bin/bash
# switch-traffic.sh
 
NAMESPACE=${NAMESPACE:-default}
SERVICE_NAME=${SERVICE_NAME:-my-app}
TARGET_COLOR=${1:-green}
 
echo "Switching traffic to $TARGET_COLOR..."
 
# check the current state
CURRENT_COLOR=$(kubectl get svc $SERVICE_NAME -n $NAMESPACE \
  -o jsonpath='{.spec.selector.color}')
echo "Current: $CURRENT_COLOR -> Target: $TARGET_COLOR"
 
# confirm the Green deployment is ready
READY_REPLICAS=$(kubectl get deployment my-app-$TARGET_COLOR -n $NAMESPACE \
  -o jsonpath='{.status.readyReplicas}')
DESIRED_REPLICAS=$(kubectl get deployment my-app-$TARGET_COLOR -n $NAMESPACE \
  -o jsonpath='{.spec.replicas}')
 
if [ "$READY_REPLICAS" != "$DESIRED_REPLICAS" ]; then
  echo "Error: Target deployment not ready ($READY_REPLICAS/$DESIRED_REPLICAS)"
  exit 1
fi
 
# switch traffic
kubectl patch svc $SERVICE_NAME -n $NAMESPACE \
  -p "{\"spec\":{\"selector\":{\"color\":\"$TARGET_COLOR\"}}}"
 
echo "Traffic switched to $TARGET_COLOR successfully!"
 
# check the state after the switch
kubectl get svc $SERVICE_NAME -n $NAMESPACE -o wide
kubectl get endpoints $SERVICE_NAME -n $NAMESPACE

Limits of the manual approach

Handling the selector switch by hand accumulates these problems.

Human error: a typo when changing the selector
Hard to automate: managing state inside a continuous integration / continuous delivery (CI/CD) pipeline gets complicated
No automated rollback: recovering from a problem needs manual intervention
No monitoring hookup: nothing can trigger an automatic rollback

Automating with Argo Rollouts

Deployment state is managed as a declarative resource, with approval gates and analysis steps built in. Compared to a manual switch this removes a class of human error and makes automatic rollback policies easy to express.

Argo Rollouts is a progressive delivery controller for Kubernetes that manages Blue/Green and canary deployments declaratively. Progressive delivery means shifting traffic to a new version in stages while verifying it.

Installation

bash

# install Argo Rollouts
kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts \
  -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml
 
# install the kubectl plugin (optional)
brew install argoproj/tap/kubectl-argo-rollouts  # macOS
# or
curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
chmod +x kubectl-argo-rollouts-linux-amd64
sudo mv kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts

The Rollout resource

yaml

# rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-app
spec:
  replicas: 4
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: app
        image: my-app:1.1.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
  strategy:
    blueGreen:
      # service receiving live production traffic
      activeService: my-app-active
      # service for previewing the new version
      previewService: my-app-preview
      # automatic promotion disabled (manual approval required)
      autoPromotionEnabled: false
      # how long the old ReplicaSet is kept after the switch
      scaleDownDelaySeconds: 30
      # replica count for the preview ReplicaSet
      previewReplicaCount: 2

Service configuration

yaml

# services.yaml
---
# Active service (current production)
apiVersion: v1
kind: Service
metadata:
  name: my-app-active
spec:
  selector:
    app: my-app
  ports:
    - port: 80
      targetPort: 8080
---
# Preview service (for testing the new version)
apiVersion: v1
kind: Service
metadata:
  name: my-app-preview
spec:
  selector:
    app: my-app
  ports:
    - port: 80
      targetPort: 8080

Deployment workflow

bash

# 1. check the current state
kubectl argo rollouts get rollout my-app -w
 
# 2. deploy the new version (change the image)
kubectl argo rollouts set image my-app app=my-app:1.2.0
 
# 3. test the preview environment
# run internal tests against the my-app-preview service
 
# 4. approve the traffic switch
kubectl argo rollouts promote my-app
 
# 5. roll back on failure
kubectl argo rollouts abort my-app
# or go back to the previous version
kubectl argo rollouts undo my-app

Checking state in the dashboard

bash

# start the Argo Rollouts dashboard
kubectl argo rollouts dashboard
 
# open http://localhost:3100 in a browser

Automated verification with Analysis

An AnalysisTemplate defines the metrics measured before and after the switch. successCondition is the pass threshold, here a 2xx response rate of at least 95%, and failureLimit is how many failures are tolerated. interval and count set the measurement period and the number of repetitions.

yaml

# analysis-template.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
  - name: service-name
  metrics:
  - name: success-rate
    interval: 30s
    count: 5
    successCondition: result[0] >= 0.95
    failureLimit: 3
    provider:
      prometheus:
        address: http://prometheus.monitoring:9090
        query: |
          sum(rate(http_requests_total{
            service="{{args.service-name}}",
            status=~"2.."
          }[5m])) /
          sum(rate(http_requests_total{
            service="{{args.service-name}}"
          }[5m]))

yaml

# rollout-with-analysis.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-app
spec:
  # ... (same settings as before)
  strategy:
    blueGreen:
      activeService: my-app-active
      previewService: my-app-preview
      autoPromotionEnabled: false
      # automatic analysis before the switch
      prePromotionAnalysis:
        templates:
        - templateName: success-rate
        args:
        - name: service-name
          value: my-app-preview
      # automatic analysis after the switch (rollback trigger)
      postPromotionAnalysis:
        templates:
        - templateName: success-rate
        args:
        - name: service-name
          value: my-app-active

Integrating with the Istio service mesh

Traffic can be controlled by percentage, and header-based branching and mirroring tests become available. In large environments that need advanced routing, this adds a lot of flexibility to Blue/Green operations.

Istio makes finer-grained traffic control possible.

Architecture

DestinationRule

yaml

# destination-rule.yaml
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: my-app
spec:
  host: my-app
  subsets:
  - name: blue
    labels:
      version: v1.0.0
  - name: green
    labels:
      version: v1.1.0

VirtualService (Blue/Green)

yaml

# virtual-service.yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: my-app
spec:
  hosts:
  - my-app
  - my-app.example.com
  gateways:
  - my-app-gateway
  http:
  # header-based routing (for testing)
  - match:
    - headers:
        x-canary:
          exact: "true"
    route:
    - destination:
        host: my-app
        subset: green
        port:
          number: 80
  # default traffic (to Blue)
  - route:
    - destination:
        host: my-app
        subset: blue
        port:
          number: 80
      weight: 100
    - destination:
        host: my-app
        subset: green
        port:
          number: 80
      weight: 0

Gradual switchover (Istio and Blue/Green hybrid)

yaml

# staged switchover example
# Step 1: 0% Green
- destination:
    host: my-app
    subset: blue
  weight: 100
- destination:
    host: my-app
    subset: green
  weight: 0
 
# Step 2: 10% Green (testing)
- destination:
    host: my-app
    subset: blue
  weight: 90
- destination:
    host: my-app
    subset: green
  weight: 10
 
# Step 3: 100% Green (switch complete)
- destination:
    host: my-app
    subset: blue
  weight: 0
- destination:
    host: my-app
    subset: green
  weight: 100

Traffic mirroring (shadow testing)

yaml

# copy real traffic to Green for testing (responses are discarded)
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: my-app
spec:
  hosts:
  - my-app
  http:
  - route:
    - destination:
        host: my-app
        subset: blue
      weight: 100
    mirror:
      host: my-app
      subset: green
    mirrorPercentage:
      value: 100.0

Database schema migration strategy

Under Blue/Green, two versions hit the database at the same time, so backward compatibility is the whole game. Following the Expand -> Migrate -> Contract order keeps the switch safe and preserves the ability to roll back.

The hardest part of Blue/Green deployment is changing the database schema, because both versions use the same database simultaneously.

The Expand and Contract pattern

A migration example, phase by phase

Phase 1: Expand

sql

-- add the new column (nullable)
ALTER TABLE users ADD COLUMN full_name VARCHAR(255);
 
-- set a default (compatible with existing data)
UPDATE users SET full_name = user_name WHERE full_name IS NULL;

python

# v2 application: supports both columns
class User:
    def get_display_name(self):
        # prefer the new column, fall back to the old one
        return self.full_name or self.user_name
 
    def save(self):
        # update both columns (backward compatibility)
        self.full_name = self.display_name
        self.user_name = self.display_name  # v1 compatibility

Phase 2: Migrate

python

# batch migration script
def migrate_user_names():
    users = User.objects.filter(full_name__isnull=True)
    for batch in chunked(users, 1000):
        for user in batch:
            user.full_name = user.user_name
        User.objects.bulk_update(batch, ['full_name'])

Phase 3: Contract

sql

-- run only after v1 is completely gone
-- and after a long enough monitoring period (for example two weeks)
ALTER TABLE users DROP COLUMN user_name;

Migration checklist

Step	Blue (v1)	Green (v2)	Rollback
1. Add the new column	ignores it	reads and writes	possible
2. Copy the data	ignores it	reads and writes	possible
3. Switch to v2	inactive	active	possible
4. Remove v1	deleted	active	risky
5. Drop the old column	inactive	active	impossible

Important: phase 5 cannot be rolled back, so run it only after a long enough monitoring period (one to two weeks at minimum).

Summary

Which Blue/Green implementation fits on Kubernetes depends on operational maturity. Switching the Service selector is enough for learning and small deployments. Production is safer with Argo Rollouts managing approval, analysis, and automatic rollback declaratively. Environments that need percentage-based control, header-based branching, or traffic mirroring add Istio on top.

Because both versions share one database, the schema has to follow Expand -> Migrate -> Contract to stay backward compatible and keep rollback on the table. The next post covers integrating Blue/Green with cloud environments and CI/CD pipelines.