Our current on-call alerting system is run via Opsgenie. This system will send notifications via voice call, SMS, email, and Slack. Teams at Sourcegraph that have production systems where they need to alerted to potential issues should have an Opsgenie rotation.
Admins are able to view the current Opsgenie teams and create new teams. Engineering managers should have access to this page (notify the DevOps team if that is not the case).
See the official Opsgenie docs.
Ensure that the first escalation after the on-call engineer is to “Alert the entire team”. Ensure that the final escalation policy is set to “Send to the DevOps team” or “Send to all teams” bases on severity.
Opsgenie alerts on Cloud are configured in the following way:
- The site-config
- The Opsgenie team
- The ObservableOwner
(This process may change with #34861)