Deploying Packer in GCP to build Windows Images using Cloud Scheduler and Cloud Build with Terraform

When recently working on a customer environment I was looking at ways to improve the startup time of their Windows Application servers. Their sophisticated Powershell startup script was taking about 15–20 minutes to execute fully and involved installing IIS, Azure DevOps & Stackdriver agents and the specialist application software. It was fairly clear to me that time could be saved by packing at least some of these elements into a base image. This then led me to consider deploying Packer on GCP. In this blog post I hope to discuss some of the steps taken to deploy this and how it hangs together.

Step 1: Deploy Packer Builder into GCR

I felt the obvious choice was to run Packer within Cloud Build using a Trigger defined in the Terraform. To make life even easier some kind people in the Cloud Builders Community have already packaged a Packer builder so the first step is simply to push this to the project GCR so Cloud Build can use it. After cloning down that repo change into the packer directory, then simply run:

gcloud builds submit .

Note: If you have any specific requirements for a certain GCR location, like this customer did, modify the cloudbuild.yaml file for the builder. The subsequent code references eu.gcr.io for this reason.

Step 2: Writing the Cloud Build Packer job YAML File

So the next step is to define the Cloud Build Packer job using YAML, there is an example provided in the Cloud Builders repo for inspiration but here is an anonymized copy of my resultant file:

steps:
  - name: eu.gcr.io/customer-app-${_ENV}/packer
    env:
      - PROJECT_ID=customer-app-${_ENV}
    args:
      - build
      - -force
      - -var
      - project_id=customer-app-${_ENV}
      - 'packer/app.json'
timeout: 1800s

In the above example I specified the timeout as 30 minutes as the GCP default (10 minutes) isn’t enough time for this job. The ${_ENV} value is a substitution that is passed into the Cloud Build job. In the example above this is to allow the same code to be run in multiple projects. Finally the packer/app.json is the location of the JSON file that Packer itself reads.

Step 3: Writing the Packer JSON File

Whilst I appreciate that as of v1.5 Packer can read HCL2, most documentation and examples are still in JSON (including the one I pinched and modified) so I opted to keep it this way:

{
    "builders": [
      {
        "type": "googlecompute",
        "project_id": "{{user `project_id`}}",
        "source_image_family": "windows-2016",
        "disk_size": "100",
        "disk_type": "pd-ssd",
        "machine_type": "n1-standard-2",
        "communicator": "winrm",
        "subnetwork": "app-vms",
        "tags": "packer-winrm",
        "winrm_username": "packer_user",
        "winrm_insecure": true,
        "winrm_use_ssl": true,
        "metadata": {
          "windows-startup-script-cmd": "winrm quickconfig -quiet & net user /add packer_user & net localgroup administrators packer_user /add & winrm set winrm/config/service/auth @{Basic=\"true\"}"
        },
        "zone": "europe-west2-a",
        "image_storage_locations": ["europe-west2"],
        "image_name": "app-{{timestamp}}",
        "image_family": "app-base"
      }
    ],
    "provisioners": [
      {
        "type": "powershell",
        "script": "packer/app.ps1"
      }
    ]
  }

Note that in the JSON I am parsing the powershell script and specifying a network tag of “packer-winrm”. The network tag is then used to target a specific WinRM firewall rule during the build process. The Terraform firewall code looks like this:

resource "google_compute_firewall" "packer-winrm" {
  name    = "packer-winrm"
  network = google_compute_network.vpc.name
allow {
    protocol = "tcp"
    ports    = ["5986"]
  }
source_ranges = ["0.0.0.0/0"]
target_tags = ["packer-winrm"]
}

Note: I can’t specify the source_ranges as Google doesn’t publish ranges for Cloud Build.

Step 4: Writing the Powershell

This step is only applies to Windows but I wanted to mention the requirement to pass a no_shutdown argument to the GCESysprep wrapper to prevent sysprep from restarting the image. Without this argument Packer fails the build as it assumes something has gone wrong.

##############
# INSTALL IIS 
##############
Write-Host "Installing IIS and all Sub Features"
Install-WindowsFeature -Name Web-Server -IncludeAllSubFeature
##############
#  GCESysprep
##############
GCESysprep -no_shutdown

Note: I found the GCESysprep documentation to be a little sparse so I would recommend reviewing the code in the Github Repo. This particular issue I found helped a great deal in finding a working solution.

Step 5: Writing Cloud Build Triggers

Normally with Cloud Build you are triggering off a PR or Push to a specific branch of a git repo. However in this case I don’t want to build an image every single time someone pushes to the repo. One solution to this would be to separate Packer into its own repo but in this case it was preferable to keep it in the repo with the infrastructure code. Therefore I created the trigger in a disabled state so it can only be manually triggered:

resource "google_cloudbuild_trigger" "app-packer" {
  provider = google-beta
project     = var.project_id
  name        = "app-packer"
  description = "Trigger to build Packer Image set to disabled as triggered via Cloud Scheduler"
  disabled    = true
github {
    owner = "customer-org"
    name  = "customer-app"
push {
      branch = var.git_branch
    }
  }
substitutions = {
    _ENV = var.env
  }
filename = "packer/packerbuild.yaml"
}

Note: You can also see above where I make the _ENV substitution based on a Terraform variable.

Step 6: Triggering Cloud Build with Cloud Scheduler

Before I begin this section I want to shout out Neil Kolban’s excellent post about Scheduling Cloud Builds which inspired this section. I won’t go over what he’s already covered but I will provide Terraform code for the Cloud Scheduler job:

resource "google_cloud_scheduler_job" "packer_build" {
  name        = "packer_build"
  description = "Trigger packer cloud build job via HTTP Post"
  schedule    = "45 23 * * 6"
http_target {
    uri = "https://cloudbuild.googleapis.com/v1/projects/${var.project_id}/triggers/app-packer:run"
    http_method = "POST"
    body = base64encode(<<-EOT
    {
      "branchName": "master"
    }
    EOT
    )
oauth_token {
        service_account_email = google_service_account.packer_scheduler.email
    }
  }
}

Step 7: Enjoy Cloud Scheduled Packer Builds :)

Now the Cloud Scheduler job will trigger Cloud Build which will go and package a Windows image using Packer before adding it to an image family from which it can be deployed.