Testing deployments

This section documents testing clusters and deployments.

Create a project in the “Engineering Projects” folder

Create a folder with your name here following our naming conventions. If you are an Support Engineer wishing to start your own project, ask for one to be created in the #dev-chat channel on slack. When you are done with the project, let #delivery know in the same thread you started in #cloud-devops so that they know it is safe to take it down.

How to manually start a test cluster in your test project in GCP

Go to your project
Click create a cluster.
Give it a name (a good convention is to prefix with your username).
Keep the defaults zonal and us-central1.
Set the default pool to 3 nodes.
Change machine type to n1-standard-16.
Click “Create Cluster”.
When cluster is ready, click connect and copy/paste the command and execute it (it looks something like gcloud container clusters get-credentials ....). Now kubectl is acting on this cluster.
Give yourself admin superpowers by executing:

kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user $(gcloud config get-value account)

Add a storage class by saving these contents

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: sourcegraph
  labels:
    deploy: sourcegraph-storage
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd # This configures SSDs (recommended).

into a file ‘sourcegraph.Storageclass.yaml’ and executing

kubectl apply -f sourcegraph.Storageclass.yaml

cd to your clone of deploy-sourcegraph and follow the remaining steps of the installation.

./kubectl-apply-all.sh

Recommendation: k9s is a nice command-line GUI tool for common kubectl operations. It shows the state of your cluster and offers keyboard short-cuts for all the common kubectl commands.

Once all the pods are running you can port-forward the frontend (or any other services you are interested in)

kubectl port-forward svc/sourcegraph-frontend 3080:30080

Please delete your test cluster when you are done testing. You may also consider deleting your project to ensure all resources tied to your account are cleaned up.

How to start a test cluster in your project on GCP with a script

The following script executes the same steps that were described in the previous section. Place the script in the root directory of your deploy-sourcegraph clone (also add it to your .git/info/exclude). Execute it from the repo root directory. It will spin up a test cluster in the namespace ns-sourcegraph.

export PROJECT=$(whoami)-testing
export PARENT_FOLDER=795981974432
gcloud projects create --folder=${PARENT_FOLDER} ${PROJECT}

#!/usr/bin/env bash

set -ex

if [  -d "generated-cluster" ]
then
    echo "Directory generated-cluster already exists. This script would override its contents. Please move it away."
    exit 1
fi

USER=$(whoami)

mkdir generated-cluster

./overlay-generate-cluster.sh namespaced generated-cluster

gcloud container clusters create "${USER}"-testing --zone us-central1-a --num-nodes 3 \
--machine-type n1-standard-16 --disk-type pd-ssd --project $PROJECT
gcloud container clusters get-credentials "${USER}"-testing --zone us-central1-a --project $PROJECT

kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user $(gcloud config get-value account)

kubectl apply -f base/sourcegraph.StorageClass.yaml

kubectl create namespace ns-sourcegraph

kubectl apply -n ns-sourcegraph --prune -l deploy=sourcegraph -f generated-cluster --recursive

timeout 5m kubectl -n ns-sourcegraph rollout status -w statefulset/indexed-search
timeout 5m kubectl -n ns-sourcegraph rollout status -w deployment/precise-code-intel-bundle-manager
timeout 5m kubectl -n ns-sourcegraph rollout status -w deployment/prometheus
timeout 5m kubectl -n ns-sourcegraph rollout status -w deployment/redis-cache
timeout 5m kubectl -n ns-sourcegraph rollout status -w deployment/redis-store
timeout 5m kubectl -n ns-sourcegraph rollout status -w statefulset/gitserver
timeout 5m kubectl -n ns-sourcegraph rollout status -w deployment/sourcegraph-frontend

kubectl -n ns-sourcegraph expose deployment sourcegraph-frontend --type=NodePort \
--name sourcegraph --type=LoadBalancer --port=3080 --target-port=3080

kubectl -n ns-sourcegraph describe service/sourcegraph

This script creates a load balancer and describes the exposed service. Look for the LoadBalancer Ingress IP and copy its value (if the IP hasn’t been assigned yet, wait a little and execute kubectl -n ns-sourcegraph describe service/sourcegraph until it appears).

You can paste the IP value into the following Caddyfile to have your new test cluster available at https://sourcegraph.test:3443

sourcegraph.test:3443
tls internal
reverse_proxy http://<INSERT LOAD BALANCER IP HERE>:3080

Again, please delete your test project when done. Click on the upper right corner next to your name and select “Delete Project”.

Recommendation: infra.app is a nice Kubernetes management app. It has some overlap with k9s but also complements it in some areas.

Building Docker images for a specific branch

If you need to build Docker images on Buildkite for testing purposes, e.g. you have a PR with a fix and want to deploy that fix to a test instance, you can push the branch to the special docker-images-patch and docker-images-patch-notest branches. Learn more about pipeline run types.

To request a build with to build images, you can use sg:

sg ci build [docker-images-patch|docker-images-patch-no-test]

Example: You want to build a new Docker image for frontend and gitserver based on your currently checked out branch. You would like to test gitserver as well, but the changes to frontend are trivial and don’t need to be tested again. The commands you would run are:

sg ci build docker-images-patch gitserver
sg ci build docker-images-patch-notest frontend

This will trigger two builds on Buildkite that will publish newly built Docker images. You can pull the image URL from the logs, and then use it to test changes locally.

When using the image, if you get an authentication issue similar to as follows:

Unable to find image 'us.gcr.io/sourcegraph-dev/server:8738645e6be92cd86da4126b340a9392dc8fc0c4_164858_candidate' locally
docker: Error response from daemon: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication.
See 'docker run --help'.

You can run gcloud auth configure-docker us.gcr.io to authenticate yourself.