Managed instances are Docker Compose deployments only today. We do not currently offer Kubernetes managed instances.
These managed Docker Compose deployments can scale up to the largest GCP instance type available, n2-standard-128 with 128 CPU / 512 GB memory which is typically enough for most medium to large enterprises.
We do not offer Kubernetes managed instances today as this introduces some complexity for us in terms of ongoing maintenance and overhead, we may revisit this decision in the future.
For each type of Managed Instances (v1.0 and v.1.1), Sourcegraph maintains separate test environments:
for v1.0 - dev instance(all instances have been migrated to v1.1)
- for v1.1 - rctest instance
- for v1.1 - tpgi instance
- for v1.1 - s2 instance
Internal instances are created for various testing purposes:
- testing changes prior to the monthly upgrade on customer instances. upon a new release is made available, Cloud team will follow managed instances upgrade tracker to proceed with upgrade process.
- testing significant operational changes prior to applying to customer instances
- short-lived instances for product teams to test important product changes. Notes: any teammate may request a managed instance through our request process
This is a shared instance for the engineering organization to test unreleased versions, locally built images, or anything they would like to experiement with Managed Instances.
#cloud is responsible for the maintenance of infrastructure, including Cloud SQL and underlying VM. The team that is running the experiment is responsbile for keeping everyone updated on the experiement in #cloud and ensuring the application is working as intented. The team should consult the operation guide when interacting with the dev instance. (please backup the database and VM before doing anyting destructive)
Learn more from sourcegraph/customer#958
This is the internal Cloud dogfood instance for the entire company. #dev-experience is responsible for rolling out nightly builds on this instance and #cloud is responsible for the maintenance of infrastructure, including Cloud SQL and underlying VM.
All customer instances are considered part of the production environment and all changes applied to these customers should be well-tested in the test environment.
Upgrade process to new Sourcegraph version is also preceded with upgrading test instances - upgrade to v3.40.1.
Sourcegraph upgrades every test and customer instances according to SLA.
The release process is performed in steps:
- New version is released via release guild
- Github issue in Sourcegraph repository is open based on the managed instances upgrade template
- Github issue is labeled with
team/cloudand Cloud Team is automatically notified to perform Managed Instances upgrade. Label is part of the template.
- Cloud team performs upgrade of all instances in given order:
- for Instances with version v1.0
- for Instances with version v1.1
Sourcegraph managed instances are single-machine Docker-Compose deployments only. We do not offer Kubernetes managed instances, or multi-machine deployments, today.
With that said, Docker Compose deployments can scale up to the largest GCP instance type available, n1-standard-96 with 96 CPU & 360 GB memory, and are typically capable of supporting all but the largest of enterprises (around 25,000 repositories and 3,000 users are supported, based on what we have seen thus far.)
The main limitation of this model is that an underlying GCP infrastructure outage could result in downtime, i.e. is it not a HA deployment.
- Isolation: Each managed instance is created in an isolated GCP project with heavy gcloud access ACLs and network ACLs for security reasons.
- Admin access: Both the customer and Sourcegraph personnel will have access to an application-level admin account. Learn more about how we ensure secure access to your instance.
- VM/SSH access: Only Sourcegraph personnel will have access to the actual GCP VM, this is done securely through GCP IAP TCP proxy access only. Sourcegraph personnel can make changes or provide data from the VM upon request by the customer.
- Inbound network access: The customer may choose between having the deployment be accessible via the public internet and protected by their SSO provider, or for additional security have the deployment restricted to an allowlist of IP addresses only (such as their corporate VPN, etc.). Notes, in addition to the customer provided IP allowlist, traffic from well-known public code hosts (e.g. GitHub.com) is also permitted to access selected Sourcegraph endpoints to ensure functionality of certain features.
- Outbound network access: The Sourcegraph deployment will have unfettered egress TCP/ICMP access, and customers will need to allow the Sourcegraph deployment to contact their code host. This can be done by having their code-host be publicly accessible, or by allowing the static IP of the Sourcegraph deployment to access their code host.
- Web Application Firewall (WAF) protections: The Sourcegraph deployment, if open to the Internet, will be proxied through Cloudflare and leverage security features such as rate limiting and the Cloudflare WAF.
Access can be requested in #it-tech-ops WITH manager approval.
Every customer managed instance has alerts configured:
- uptime check - version v1.0 configured in dedicated GCP managed instance project
- uptime check - version v1.1 configured in dedicated GCP managed instance project
- instance performance metric alerts configured in scoped project for all managed instances
- application performance metrics - configured in customer intance site-config.json via
mg cliduring instance creation
- When alert is triggered, it is sent to Opsgenie channel:
On-call Cloud engineer has to decide, what is the alert type and if incident should be opened and follow the procedure to perform the incident. On-call Cloud engineer should use managed instances operations to check, assess and repair broken managed instance.
When alert is closed via incident resolution, post-mortem actions has to be assigned and performed.
Terraform is used to maintain all managed instances. You can find this configuration here: https://github.com/sourcegraph/deploy-sourcegraph-managed
All customer credentials, secrets, site configuration, app and user configuration—is stored in Postgres only (i.e. on the encrypted GCP disk). This allows customers to enter their access tokens, secrets, etc. directly into the app through the web UI without transferring them to us elsewhere.
Please review the Managed Instances v1.0 operations guide for instructions.
Managed Instances v1.1 documentation can be found here
Managed Instances v2.0 documentation can be found here