Environment

The code for this environment can be found in the following locations, depending on which part of the scale testing cluster you need to change.

cloud_sql_proxy -instances=sourcegraph-scaletesting:us-central1:sg-scaletesting-d2d240d290=tcp:5555

psql -h localhost -p 5555 -d sg -U 'dev-readonly'

Code hosts

In order to reproduce a customer scenario accurately, you’ll need access to the various code hosts that are available for testing purposes.

The following code hosts are available for testing:

Observability

In order to gather meaningful results of running tests against the scale testing instance, you can gather the following resources to help you come to a conclusion:

• Tracing: Selective tracing is enabled and traces are exported to GCP. You can see all traces by going to the GCP tracing dashboard. To use tracing all you have to add is &trace=1 to the url and the UI will show a View trace link, which takes you to the GCP dashboard for your particular trace.
• Sentry: scaletesting
• Infrastructure and Application logs: GKE logs are currently available for viewing in the Google Cloud Logs Explorer. See the official documentation for further information on how to use the logging platform.
• You can authenticate yourself to the dogfood cluster using the following command:

gcloud container clusters get-credentials dogfood --region us-central1 --project sourcegraph-dogfood

If you have already configured and authenticated to the cluster, you can of course interact directly with the deployment to observe logs or other behaviours. See the cheatsheet for useful kubectl commands.

Deploying code

To make changes to images, environment variables or other configuration, all updates should be made in values.yaml

First, authenticate yourself to the cluster using the following command:

gcloud container clusters get-credentials dogfood --region us-central1-f --project sourcegraph-dogfood

Then check that you have the Sourcegraph Helm chart installed:

helm repo add insiders https://helm.sourcegraph.com/insiders
helm repo update
helm search repo insiders --devel

Make changes to values.yaml by running the following script (this will update to latest main):

sg ops update-images -k helm ./helm/sourcegraph

Finally, merge your changes via a pull request, and run the following from the base of the deploy-sourcegraph-scaletesting repository:

helm repo update

# If you want to deploy using the same chart version:
SG_CHART_VERSION=$(helm history scaletesting -o json -n scaletesting | jq -r 'reverse | .[0].chart | sub("^sourcegraph-"; "")')
# If you want to deploy using a newer chart version ('helm search repo insiders --devel --versions' lists all available versions):
SG_CHART_VERSION=<new chart version>

helm upgrade --install --values ./helm/sourcegraph/values.yaml --version ${SG_CHART_VERSION} scaletesting insiders/sourcegraph -n scaletesting

kubectl get pod -n scaletesting --watch

sg live scaletesting

Scale the infrastructure down when not in use

To stop the devx compute instance when it is not in use, run the following:

gcloud compute instances stop devx --zone us-central1-a --project sourcegraph-scaletesting

or to start it:

gcloud compute instances start devx --zone us-central1-a --project sourcegraph-scaletesting

Pods in CrashLoopBackOff due to failed migration

Certain operations, such as changing images to a development tag, can cause the most recent migration to fail. This will prevent certain services from starting, making the instance unhealthy.

To resolve this, do the following:

1. Verify that the migration is incomplete by checking the schema_migrations table:
   • Open a Cloud SQL proxy connection with cloud_sql_proxy -instances=sourcegraph-scaletesting:us-central1:sg-scaletesting-d2d240d290=tcp:5449
   • Connect to the database with psql -h localhost -p 5449 -d sg -U sg (credentials)
   • Confirm that the last migration is incomplete by running SELECT * FROM schema_migrations;
2. Add the following to values.yaml in deploy-sourcegraph-scaletesting:

   migrator:
     args: [up, --ignore-single-pending-log=true]
   

3. Apply the values as described in Deploying code
4. Check the migrator logs in the frontend pod for a successful migration
5. Remove the args key/value from values.yaml and apply the values again! If the flag remains on the migrator pod, the database may be corrupted on subsequent deploys.

Instance start up and shut down

The instance can be started/shut down with the following commands:

• Start up: gcloud compute instances start ghe-scaletesting --zone=us-central1-f --project sourcegraph-scaletesting
• Shut down: gcloud compute instances stop ghe-scaletesting --zone=us-central1-f --project sourcegraph-scaletesting

When starting the instance, it could take up to 5 mins for the application to set up before fully operational. The UI will indicate when it is ready to be used.

Scaling the instance

The instance has been set up to handle 10k users based off of the recommended specs from GitHub.

It can also be scaled up or down with the following steps:

Note: Be cautious when scaling up, as we can only scale up and down CPU/Mem but not disk. To avoid over-provisioning, only increase disk size when needed.

Logging in

The instance has been set up with an admin account, please find the credentials in our 1Pass vault.

Data Safety

This instance has no guarantees regarding permanence, security, or stability. Do not add any sensitive or private data to this instance! Assume that this instance can be wiped or torn down at any given moment.

Over 100k repositories

The rctest.sgdev.org uses the following list of GitHub organizations to populate an instance with over 100k repositories:

• github.com/pld-linux (22k repos)
• github.com/londonappbrewery (28k repos)
• github.com/wp-plugins (52k repos)

In order to add them to a scaletesting instance, you can run the following command:

sg client codehost add-github \
  --display-name "repos-gh-100k" \
  --baseurl "https://scaletesting.sgdev.org" \
  --email "REDACTED" --password "REDACTED" \
  --github.token REDACTED \
  pld-linux londonappbrewery wp-plugins

Organization with 10k repositories with write access

See https://ghe.sgdev.org/testing which is a replica of https://github.com/londonappbrewery on our GitHub instance. They are owned by the testing user, who can write on those repos.

200k blank repos

• Single initial commit with README.md
• 198k repositories assigned to main-org: https://ghe-scaletesting.sgdev.org/main-org
  • 4750 teams have permissions on 27 or 28 repositories (teams main-org/teams/team-000000000 through main-org/teams/team-000004749)
  • 200 teams have permissions on 78 repositories (teams main-org/teams/team-000004750 through main-org/teams/team-000004949)
  • 50 teams have permissions on 985 repositories (teams main-org/teams/team-000004950 through main-org/teams/team-000004999)
  • 1000 repositories each have 3 users assigned as collaborators who are the only users with access (repos main-org/repo000000000 through main-org/repo000000999)
• 400 repositories each assigned to sub-org-000000000 through sub-org-000000004: https://ghe-scaletesting.sgdev.org/sub-org-000000000 and up
  • These have org-wide permissions for the members of each org (2k members per org)

See https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/dev/scaletesting/bulkrepocreate/README.md for the tool powering the creation.

Large binary files

A repository with large binary files (Ubuntu isos) is available at https://ghe.sgdev.org/scaletesting/large-binary-files

Large amount of commits

The following repositories are available to test against repositories with a massive amount of commits:

• github.com/sgtest/megarepo (>700k commits)
• gigarepo, served through git-combine (> 1.8M commits)

{
  // See the git-combine service and statefulset
  "url": "http://git-combine",
  // Do not change this. Sourcegraph uses this as a signal that url is 'src serve'.
  "repos": [
    "src-serve"
  ]
}

Perforce

A small tool named Synthforce has been created to generated synthetic repos (depots) in Perforce. For more details on how to use the tool to generate more synthetic files or history on a particular depot please see the README.

GitLab

Creating a scenario runner

While not implemented at this stage, creating a runner is very similar to how the end-to-end test runner is working on the main repo, you set up your client and reach the instance to perform your requests.

Depending on what you want to test, there are a few approaches you can take:

• You could simply work at the API level, by creating a GraphQL client
  • Example.
• You can work at the browser level, by mimicking what we do in smoke-tests and end-to-end tests.
  • Example and example