Enabling executors

All managed instances will have executors enabled by default. #ce is responsible for providing the ARR to help us determine the compute resources we allocate to executors, learn more.

Deploy executors

Update terraform config

Open $CUSTOMER/terraform.tfvars and uncomment the executors block

$COMPANY is already substituted with actual value during instance creation time, so you only need to uncomment the tfvars block without changing anything else

-# enable_executors                   = true
-# executor_instance_tag              = "$COMPANY-executors"
-# executor_metrics_environment_label = "$COMPANY"
-# executor_min_replicas              = 0
-# executor_max_replicas              = 4
-# executor_notification_email        = "REDACTED" #wg-shipping-executors
+enable_executors                   = true
+executor_instance_tag              = "$COMPANY-executors"
+executor_metrics_environment_label = "$COMPANY"
+executor_min_replicas              = 0
+executor_max_replicas              = 4
+executor_notification_email        = "REDACTED" #wg-shipping-executors

Apply the terraform module

cd $CUSTOMER
terraform apply

Important: executors has to be set up on active instance, so if multiple VMs are running, use flag --deployment red|black in all mg commands below.

Add the executor token to the site configuration of the instance (note: this must be run in the $CUSTOMER directory)

mg executors set-token --token $(terraform output -raw executor_proxy_password)

Reload deployment config

Sync configuration

mg sync artifacts

Reload worker to catch up with the updated configuration

mg ssh-exec "cd /deployment/docker-compose && docker-compose up -d worker"

Confirm executors is actually working

Then run the command below, which will

  • increase the minimal replica of executors auto scaling group to 1
  • retry until either the executors are up or a 5 minute timeout has been reached, after which you should check configuration errors if it fails
  • scale auto scaling group back to 0
mg executors check

Wrapping up

Commit your changes and open a PR. Let #wg-shipping-executors know it is ready!

Troubleshooting

Ensure instance groups are there

$ gcloud compute instance-groups list --zones=$zone --project=$PROJECT
NAME                             LOCATION       SCOPE  NETWORK                MANAGED  INSTANCES
batches--sourcegraph-executor    us-central1-a  zone   sourcegraph-executors  Yes      0
codeintel--sourcegraph-executor  us-central1-a  zone   sourcegraph-executors  Yes      0

Ensure minNumReplicas is greater than 0

$ gcloud compute instance-groups managed describe batches--sourcegraph-executor --zone=$ZONE --project=$PROJECT --format=json | jq '.autoscaler.autoscalingPolicy'
{
  "minNumReplicas": 1,
}
$ gcloud compute instance-groups managed describe codeintel--sourcegraph-executor --zone=$ZONE --project=$PROJECT --format=json | jq '.autoscaler.autoscalingPolicy'
{
  "minNumReplicas": 1,
}

Ensure there is an active instance belong to one of the instance group (notes the batches--sourcegraph-executor-rqfs instance). Sometimes it may take GCP longer to spawn a new instance, so be patient. If no new instance is created for an unreasonable amount of time, consult GCP documentation for next step.

$ gcloud compute instances list --project=$PROJECT
NAME                                          ZONE           MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
batches--sourcegraph-executor-rqfs            us-central1-a  c2-standard-8  true         10.0.1.58    35.222.34.224   RUNNING
default-red-instance                          us-central1-a  n2-standard-8               10.2.0.3                     RUNNING
sourcegraph-executors-docker-registry-mirror  us-central1-a  n1-standard-2               10.0.1.2     35.239.105.148  RUNNING

If above all check out, visit the Compute Engine Console and check logs of the executor instance for more troubleshooting.