Managed instances are Docker Compose deployments today - we do not currently offer Kubernetes (or multi-node) managed instances. See known limitations for more context on scalability of Cloud.
For each type of Managed Instances (v1.0 and v.1.1), Sourcegraph maintains separate test environments:
for v1.0 - dev instance(all instances have been migrated to v1.1)
- for v1.1 - rctest instance
- for v1.1 - tpgi instance
- for v1.1 - s2 instance
- for v1.1 - research.sourcegraph.com
Internal instances are created for various testing purposes:
- testing changes prior to the monthly upgrade on customer instances. upon a new release is made available, Cloud team will follow managed instances upgrade tracker (this is created prior to monthly upgrade) to proceed with upgrade process.
- testing significant operational changes prior to applying to customer instances
- short-lived instances for product teams to test important product changes. Notes: any teammate may request a managed instance through our request process
This is a shared instance for the engineering organization to test unreleased versions, locally built images, or anything they would like to experiement with Managed Instances.
#cloud is responsible for the maintenance of infrastructure, including Cloud SQL and underlying VM. The team that is running the experiment is responsbile for keeping everyone updated on the experiement in #cloud and ensuring the application is working as intented. The team should consult the operation guide when interacting with the dev instance. (please backup the database and VM before doing anyting destructive)
Learn more from sourcegraph/customer#958
This is the internal Cloud dogfood instance for the entire company. #dev-experience is responsible for rolling out nightly builds on this instance and #cloud is responsible for the maintenance of infrastructure, including Cloud SQL and underlying VM.
Learn more from sourcegraph/customer#1221
All customer instances are considered part of the production environment and all changes applied to these customers should be well-tested in the test environment.
Upgrade process to new Sourcegraph version is also preceded with upgrading test instances - upgrade to v3.40.1.
Sourcegraph upgrades every test and customer instances according to SLA.
The release process is performed in steps:
- New version is released via release guild
- GitHub issue in Sourcegraph repository is open based on the managed instances upgrade template.
- Github issue is labeled with
team/cloudand Cloud Team is automatically notified to perform Managed Instances upgrade. Label is part of the template.
- Cloud team performs upgrade of all instances in given order:
- for Instances with version v1.0
- for Instances with version v1.1
Sourcegraph managed instances are single-machine Docker-Compose deployments only. We do not offer Kubernetes managed instances, or multi-machine deployments, today. The main limitation of the current model is that the underlying GCP infrastructure outage could result in downtime, i.e. is it not a highly-available deployment.
- Isolation: Each managed instance is created in an isolated GCP project with heavy gcloud access ACLs and network ACLs for security reasons.
- Admin access: Both the customer and Sourcegraph personnel will have access to an application-level admin account. Learn more about how we ensure secure access to your instance.
- VM/SSH access: Only Sourcegraph personnel will have access to the actual GCP VM, this is done securely through GCP IAP TCP proxy access only. Sourcegraph personnel can make changes or provide data from the VM upon request by the customer.
- Inbound network access: The customer may choose between having the deployment be accessible via the public internet and protected by their SSO provider, or for additional security have the deployment restricted to an allowlist of IP addresses only (such as their corporate VPN, etc.). Filtering of the IP allowlist is performed by our WAF provider, Cloudflare. Notes, in addition to the customer provided IP allowlist, traffic from well-known public code hosts (e.g. GitHub.com) is also permitted to access selected Sourcegraph endpoints to ensure functionality of certain features.
- Outbound network access: The Sourcegraph deployment will have unfettered egress TCP/ICMP access, and customers will need to allow the Sourcegraph deployment to contact their code host. This can be done by having their code-host be publicly accessible, or by allowing the static IP of the Sourcegraph deployment to access their code host.
- Web Application Firewall (WAF) protections: All managed instances are proxied through Cloudflare and leverage security features such as rate limiting and the Cloudflare WAF.
Access can be requested in #it-tech-ops WITH manager approval.
Every customer managed instance has alerts configured:
- uptime check - version v1.0 configured in dedicated GCP managed instance project
- uptime check - version v1.1 configured in dedicated GCP managed instance project
- instance performance metric alerts configured in scoped project for all managed instances
- application performance metrics - configured in customer intance site-config.json via
mg cliduring instance creation
- When alert is triggered, it is sent to Opsgenie channel:
On-call Cloud engineer has to decide, what is the alert type and if incident should be opened and follow the procedure to perform the incident. On-call Cloud engineer should use managed instances operations to check, assess and repair broken managed instance.
When alert is closed via incident resolution, post-mortem actions has to be assigned and performed.
Terraform is used to maintain all managed instances. You can find this configuration here: https://github.com/sourcegraph/deploy-sourcegraph-managed
All customer credentials, secrets, site configuration, app and user configuration—is stored in Postgres only (i.e. on the encrypted GCP disk). This allows customers to enter their access tokens, secrets, etc. directly into the app through the web UI without transferring them to us elsewhere.
Please review the Managed Instances v1.0 operations guide for instructions.
Managed Instances v1.1 documentation can be found here
Managed Instances v2.0 documentation can be found here