Continuous Deployment to Google Cloud Platform with Drone CI: drone-gke and drone-gae

This article is based on engineering work originally published by the New York Times software development team, describing how they built and open-sourced Drone CI plugins for Google Cloud Platform deployments.

Background: Evaluating Google Cloud Platform at the New York Times

Over the course of roughly a year, the software development teams at The New York Times evaluated Google Cloud Platform (GCP) for use in future projects. Their newly formed Delivery and Site Reliability Engineering team surveyed a wide variety of software management techniques and tools, ultimately standardizing on two GCP compute environments—Google Container Engine (GKE) and Google App Engine (GAE)—combined with the open-source Drone continuous integration and delivery platform.

The result of this work was two open-source Drone plugins: drone-gke (for Google Container Engine) and drone-gae (for Google App Engine).

The Technology Stack

Google Container Engine (GKE)

Container Engine is Google’s managed Kubernetes orchestration platform. Kubernetes provides a declarative approach to managing containerized applications, with built-in automatic scaling and self-healing. By pooling compute resources, multiple isolated applications can run in the same cluster, maximizing resource utilization. It also encourages a common deployment standard across many independent engineering teams.

Google App Engine (GAE)

App Engine is Google’s serverless application platform, available since 2008. It scales automatically in response to traffic changes—particularly useful for publishers like the New York Times, where breaking news alerts or daily features such as the crossword puzzle can cause sharp, unpredictable traffic spikes.

Drone CI

Drone is an open-source CI/CD platform built on container technology. Each step of a build pipeline runs inside an ephemeral container, providing environment isolation and reproducibility. Drone’s plugin model—where plugins are simply Docker containers—makes it straightforward to extend. Its declarative YAML configuration is similar to commercial CI/CD services, enabling developers to ship features and bug fixes to production in minutes rather than on daily or weekly release schedules.

Why Build Custom Drone Plugins?

Google provides excellent command-line tools (gcloud, kubectl, appcfg) for interacting with its cloud services. The challenge was encapsulating these tools cleanly inside Drone pipelines to simplify the developer workflow. Because Drone plugins are Docker containers, the New York Times team could package the required Google Cloud SDK tooling once and reuse it across all their projects without requiring developers to install or configure anything locally.

drone-gke: Deploying to Google Container Engine

drone-gke wraps the gcloud and kubectl commands, allowing pipelines to orchestrate deployments to GKE and apply changes to existing Kubernetes clusters. A key feature is Kubernetes YAML templating: configuration files can include Go template variables that are populated at deploy time using Drone’s encrypted secrets and build-specific variables (such as the commit SHA).

Example: Full Pipeline Configuration (drone-gke)

The following .drone.yml example tests and builds a Go binary, publishes a container image to Google Container Registry (GCR), then deploys it to a GKE cluster:

# test and build our binary
build:
  image: golang:1.7

  environment:
    - GOPATH=/drone

  commands:
    - go get -t
    - go test -v -cover
    - CGO_ENABLED=0 go build -v -a

  when:
    event:
      - push
      - pull_request

# build and push our container to GCR
publish:
  gcr:
    storage_driver: overlay
    repo: my-gke-project/my-app
    tag: "$$COMMIT"
    token: >
      $$GOOGLE_CREDENTIALS

    when:
      event: push
      branch: master

# create and apply the Kubernetes configuration to GKE
deploy:
  gke:
    image: nytimes/drone-gke

    zone: us-central1-a
    cluster: my-k8s-cluster
    namespace: $$BRANCH
    token: >
      $$GOOGLE_CREDENTIALS

    vars:
      image: gcr.io/my-gke-project/my-app:$$COMMIT
      app: my-app
      env: dev
    secrets:
      api_token: $$API_TOKEN

    when:
      event: push
      branch: master

Example: Kubernetes Deployment and Service Manifests

The plugin renders these Go-templated Kubernetes manifests before applying them to the cluster. Template variables such as {{.image}}, {{.app}}, and {{.env}} are injected from the vars block in the Drone configuration above:

kind: Deployment
apiVersion: extensions/v1beta1

metadata:
  name: {{.app}}-{{.env}}

spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: {{.app}}
        env: {{.env}}
    spec:
      containers:
        - name: app
          image: {{.image}}
          ports:
            - containerPort: 8000
          env:
            - name: APP_NAME
              value: {{.app}}
            - name: API_TOKEN
              valueFrom:
                secretKeyRef:
                  name: secrets
                  key: api-token
---
kind: Service
apiVersion: v1

metadata:
  name: {{.app}}-{{.env}}

spec:
  type: LoadBalancer
  selector:
    app: {{.app}}
    env: {{.env}}
  ports:
    - port: 80
      targetPort: 8000
      protocol: TCPCode language: HTTP (http)

Example: Kubernetes Secrets Manifest

Sensitive values such as API tokens are injected via Drone’s secret management and rendered into a separate Kubernetes Secret manifest:

kind: Secret
apiVersion: v1

metadata:
  name: secrets

type: Opaque

data:
  api-token: {{.api_token}}Code language: HTTP (http)

drone-gae: Deploying to Google App Engine

drone-gae wraps the gcloud and appcfg commands, enabling deployments to both the App Engine standard environment (Go, PHP, Python) and the flexible environment (any language via a custom Docker image).

Example: Standard Environment Deployment (drone-gae)

This example deploys a new version of a Go service to App Engine’s standard environment, then migrates 100% of traffic to that version by setting it as the default:

deploy:
  # deploy new version to GAE
  gae:
    image: nytimes/drone-gae

    environment:
      - GOPATH=/drone

    action: update
    project: my-gae-project
    version: "$$COMMIT"
    token: >
      $$GOOGLE_CREDENTIALS

    when:
      event: push
      branch: master

  # set new version to 'default', which migrates 100% traffic
  gae:
    image: nytimes/drone-gae

    action: set_default_version
    project: my-gae-project
    version: "$$COMMIT"
    token: >
      $$GOOGLE_CREDENTIALS

    when:
      event: push
      branch: masterCode language: PHP (php)

Example: Flexible Environment Deployment (drone-gae)

Deploying to the App Engine flexible environment requires an additional step: building and publishing a Docker image to GCR first. The pipeline below tests and compiles the code, publishes the container, then deploys it to the flexible environment, stopping the previous version on completion:

# test and build our binary
build:
  image: your-dev/golang:1.7

  environment:
    - GOPATH=/drone

  commands:
    - go test -v -race ./...
    - go build -o api .

  when:
    event:
      - push
      - pull_request

# build and push our container to GCR
publish:
  gcr:
    storage_driver: overlay
    repo: my-gae-project/api
    tag: "$$COMMIT"
    token: >
       $$GOOGLE_CREDENTIALS

    when:
      branch: [develop, master]
      event: push

deploy:
  # deploy a new version using the docker image we just published
  # and stop any previous versions when complete
  gae:
    image: nytimes/drone-gae

    action: deploy
    project: my-gae-project
    flex_image: gcr.io/my-gae-project/api:$$COMMIT
    version: "$${COMMIT:0:10}"
    addl_flags:
     - --stop-previous-version
    token: >
      $$GOOGLE_CREDENTIALS

    when:
      event: push
      branch: developCode language: PHP (php)

Key Takeaways

• Plugin-as-container model: Drone plugins are Docker images, making them trivial to version, share, and reuse across teams without environment setup overhead.
• Secrets management: Drone’s encrypted secrets integrate directly with Kubernetes Secret manifests via Go templating, keeping sensitive values out of source control.
• Commit-based versioning: Using $$COMMIT as the image tag and App Engine version identifier creates a clear, auditable link between a Git commit and what is running in production.
• Staged traffic migration: The GAE plugin’s two-step approach—deploy a version, then promote it to default—allows controlled traffic migration without downtime.
• Flexible environment support: The same plugin handles both the standard and flexible App Engine runtimes, with the flexible path requiring an intermediate Docker image build step.

Getting the Plugins

Both drone-gke and drone-gae were open-sourced by The New York Times. At the time of publication, the Docker images were available as nytimes/drone-gke and nytimes/drone-gae on Docker Hub. The New York Times team encouraged the community to raise issues and submit pull requests to help harden the plugins.

Note: This article was originally written in January 2017. The Drone plugin API, GKE/GAE APIs, and the appcfg tool have evolved since then. Readers should consult current Drone and Google Cloud documentation before implementing these patterns in a modern pipeline.