Working with Kubernetes deployments

This section contains tips and advice for interacting with our Kubernetes deployments (most notably sourcegraph.com and k8s.sgdev.org).

How to set up access to Kubernetes

Make sure that you have been granted access to our Google Cloud project: https://console.developers.google.com/project/sourcegraph-dev?authuser=0. You may need to change authuser to the index of your sourcegraph.com Google account.
Install the gcloud command (CLI for interacting with the Google Cloud):
```
curl https://sdk.cloud.google.com | bash
```
Get authorization for your gcloud command:
```
gcloud auth login
```
Install the kubectl command (CLI for interacting with Kubernetes):
```
gcloud components install kubectl
```
Configure kubectl to point to the desired cluster using the appropriate gcloud container clusters get-credentials command listed in “Instances”.
Verify that you have access to kubernetes:
```
kubectl get pods --all-namespaces
```

Reauthenticate kubectl

If you see the following when running kubectl commands:

Unable to connect to the server: x509: certificate signed by unknown authority

Just run the appropriate gcloud container clusters get-credentials command listed at the top of this document again to reauthenticate.

Scaling Kubernetes clusters

Cluster scale should be managed via terraform. Please reference google_container_node_pool.primary_containerd_nodes.node_count this line in cloud’s terraform configuration to see where the number of nodes is configured for the cluster, and gke_num_nodes in the tfvars file to see the current number of nodes. For more details, see the terraform provider documentation.

Any changes to the cluster scale made via kubectl will eventually be overwritten by the values set in terraform.

Kubernetes backups

Snapshots of all Kubernetes resources are taken periodically and pushed to kube-backup.

These example commands are for the dot-com cluster where the Sourcegraph application is deployed to the prod namespace.

kubectl cheatsheet

List all pods	`kubectl get pods --namespace=prod -o=wide`
Describe the properties of a pod.	`kubectl --namespace=prod describe pod $POD_NAME`
Pull logs	`kubectl --namespace=prod logs $POD_NAME`
Get an interactive shell in a running pod container	`kubectl exec --namespace=prod -ti $POD_NAME -- /bin/sh`
Edit a “deployment” (such as changing environment variables).	`kubectl edit deployment --namespace=prod DEPLOYMENT_NAME` Note that the deployment name is not the pod name, and affects all pods running that deployment.
SSH into the VM running a pod.	Find the node ID from the NODE column of `kubectl get pods --namespace=prod -o=wide`. Go to the Google Compute Engine dashboard and click the “SSH” button in the top left to get the `gcloud` command to SSH into the node. `kubectl -n prod exec -it POD_NAME /bin/sh`
Kill a pod. All of our pods are part of a deployment, so the deployment will spin up a replacement pod automatically.	`kubectl delete --namespace=prod pod $POD_NAME`
Get a PostgreSQL client on the prod database.	`kubectl exec --namespace=prod -ti $PSQL_POD_ID -- psql -U sg`
List versions in production.	`kubectl -n prod get deploy -o jsonpath='{.items[*].spec.template.spec.containers[0].image} ' \| tr ' ' '\n' \| sort -u`
Get access to Jaeger locally.	`kubectl port-forward --namespace=prod svc/jaeger-query 16686`
Get access to debug server locally.	`kubectl port-forward $(kubectl get po --no-headers -l app=repo-updater \| cut -d ' ' -f 1) 6060`