Debugging cheat sheet

The following document exists to help CS debug commonly encountered customer issues. It’s purpose is to simplify the troubleshooting process in the form of hinting first stops of where to explore when working to solve issues raised by our customers.

IssueErrorAssociated Logs / FilesAction / Notices
Access token creation failureN/AFrotend logs, Access token page in SG UI(look for any errors)N/A
Certificate signed by unknown authority errorN/AFrotend logs, Access token page in SG UI(look for any errors)N/A
Issues with the connection and authentication to GitHubN/AAccess token(permissions scopes)N/A
GitHub permissions not being reinforcedN/Acan’t sync GitHub permissions without using GH as the auth providerN/A
User is linked to multiple emailsN/ALook at “user_external_account” table in postgres db and make sure data is expected with no duplicatesN/A
IssueErrorAssociated Logs / FilesAction / Notices
Changeset not being appliedN/AChangeset Yaml file, personal access tokens or global service account token(if any)N/A
Unable to run Batch Changerun when running a ‘src batch preview’batch spec yaml file, Gitserver logs,docker logs sourcegraph-frontendN/AN/A
Src batch preview command sometimes works, sometimes results in 400 error.Check Personal Access Token scope, check, docker-compose logsN/AN/A
The UI not reflecting updated data of a changesetN/AN/AConfigure Webhooks to get the most accurate data regrding a changeset
IssueErrorAssociated Logs / FilesAction / Notices
LSIF uploads failingN/AFront end, precise code intel worker, grafanaN/A
Still showing semantic results after successful uploadN/Aprecise code intel workerN/A
Precise code intel worker OOMN/AFront end, precise code intel workerdescribe worker pod, request for upload size in postgres by running select upload_size from lsif_uploads order by upload_size limit 50;
Slowness during use of code-intelN/AN/AHigh precise-code-intel-worker usage
Precise-code-intel worker in CrashLoopBackOff statusN/Aprecise code intel worker logsN/A
frontend ponds are crash loopingN/Afrontend deployment yamlN/A
precise-code-intel worker OOMN/AN/AN/A
IssueErrorAssociated Logs / FilesAction / Notices
Permissions denied when cloning via SSHN/AN/ACopy .ssh directory into docker-compose folder and chmod 644 for all users to read ssh keys
Repositories not found errorsN/ARepo-updater logs, git server logs, frontend logs, grafana dashboard images for git server and index server, CPU/Memory limits(may be low)N/A
Repositories not cloneable due to authentication failure/ SSL certificate failureN/AGit server logs, Repo-updater logs. Pass GIT_SSL_NO_VERIFY to true with manual cloning of failing repoN/A
Repositories suddenly stop working timeout unable to get any results backN/Agitserver logs, frontend deployment yaml, GitLab ErrorsN/A
IssueErrorAssociated Logs / FilesAction / Notices
Install fresh database/migration/restore failureN/APGSQL LogsN/A
Liveness probe too many clients errorN/APgsql logs, repo and user count, max_connections configN/A
Unable to fetch new configuration errorN/APGSQL Logscheck max_connections config in postgres via SELECT
Signal killed and core dump errorsN/APGSQL Logscheck disk size consumption by running df -h when exec’d
IssueErrorAssociated Logs / FilesAction / Notices
sourcegraph url in web browser extension errorthe string doesn’t match the expected patternN/ACheck if user is using private browsing/incognito mode. Try on regular browsing
get extension via a proxy serverN/AN/ACheck if user has private extension registry, also check if src cli is installed, then suggest private registry option
slow codeintel request error leading to OOM errorsN/Afrontend logs, gitserver yaml filecheck for resources and request for increase
IssueErrorAssociated Logs / FilesAction / Notices
GRAFANA WARNING: Repo-updater: Less than 0 repos syncedN/AGrafana dashboard, repo-updater logs grepped for “error”,N/A
No data in GrafanaN/AN/ACheck permissions in the datasource folder in grafana service, check promethues targets in http://:9090
Logs not being accessibleLogs are not written to /var/log/grafana/grafana.logN/ACheck permissions in Promethus logs and docker logs
CSRF Token InvalidN/AN/ACheck site-admin/configuration page to confirm the alerts have been created
IssueErrorAssociated Logs / FilesAction / Notices
Search failure with TCP error connection refusedN/AFrontend logs, screenshot of error from SG UIN/A
Capped results for search queriesN/AN/AFlags passed in query, count:N flag
Archived repos showing up in regular search queriesN/ASite configconfirm customer token access scopes/ check site config for exclude configs
Connection closed before search complete errorN/AN/ACheck for flags passed in search query e.g. lang, type, count
IssueErrorAssociated Logs / FilesAction / Notices
Database corruptionN/AN/A"BEGIN; SET enable_indexscan = 'off'; SET enable_bitmapscan = 'off'; SELECT id FROM users WHERE username = 'USER' AND deleted_at IS NULL; SET enable_indexscan = 'on'; SET enable_bitmapscan = 'on'; SELECT id FROM users WHERE username = 'martin.sucha' AND deleted_at IS NULL; ROLLBACK;"
Sync issues after upgradingN/ASSH keyN/A