Managed Instance Private Code Hosts support

1. Create customer dedicated AWS account

playbook

2. Modify config.yaml with additional section:

spec:
  infrastructure:
    privateCodeHost:
      aws:
        accountID: <AWS_ACCOUNT_ID>
        highAvailable: false
        privateLinks:
          # Endpoint VPC Endpoint Service name
          - endpoint: com.amazonaws.vpce.<REGION>.<VPC_ENDPOINT_SERVICE_ID>
            # ports list of TCP ports to open for GKE in AWS Security Group, by default only 443
            ports:
              - 443
            # enabled only if customer uses Private DNS Domain, this can be enable only after customers accepts VPC Endpoint or has acceptance disabled
            privateDnsEnabled: true
        region: <AWS_REGION>
      enabled: true

3. Generate additional terraform stacks

mi2 generate cdktf
mi2 instance cdktf deploy -auto-approve

this will generate awsvpc and awsvpn stacks.

Creation process detail walkthrough:

1. AWS VPC is created in customer dedicated Sourcegraph AWS Account.
2. Cloud Router with GCP-side BGP ASN number is created in Managed Instance VPC.
3. AWS VPN Gateway with BGP AWS-side ASN number is created. It is attached to VPC and uses VPC route tables to propagate BPG sessions.
4. High Available GCP VPN Gateway is created in Managed Instance GCP VPC with 2 interfaces.
5. AWS Customer Gateway is created (uses GCP-side BGP ASN). For high available options 2 gateways are created.
6. AWS VPN connection is created with 2 tunnel options (every tunnel uses different 32B PreSharedKey). For high available options 2 VPC connections are created.
7. GCP External VPN Gateway is created with interfaces correspoding for each AWS VPN connection tunnel. For high available options External VPN Gateway has 4 interfaces, for non high-available 2.
8. For each External VPN Gateway interface GCP VPN Tunnel is created. It uses dedicated PreSharedKey (same in on AWS VPN Connection(s)).
9. For each GCP VPN Tunnel interface is added to Cloud Router. It uses IP range correspoding to AWS VPN Connection tunnel internal address.
10. For each AWS VPN Connection tunnel internal address BGP peer is added to Cloud Router.
11. AWS VPC Endpoint is created inside AWS VPC.

For more details, go to Google documentation

Additional steps for Private DNS Domain

1. Configure config.yaml

spec:
  infrastructure:
    gcp:
      gke:
        blue:
          machineType: XYZ
        green:
          additionalOauthScopes: # this might be deprecated soon, as scope is added by default
            - https://www.googleapis.com/auth/devstorage.read_only
          machineType: XYZ
    privateDNS:
      routes:
        - source: private.CUSTOMER.DOMAIN
          # use domain with region inside the URL, b/c it routes to every zone inside VPC
          destination: vpce-<ID>.vpce-svc-<HASH>.<REGION>.vpce.amazonaws.com
          # resolverIP: IP_OF_CUSTOM_RESOLVER - optional

2. Generate and apply cdktf resources:

Note: open PR and merge for VCS TFC changes to be applied.

mi2 generate cdktf
# modify GKE node pool and add artifact registry repository with permissions
mi2 instance tfc deploy -auto-approve

3. Generate dns-proxy sidecars

# generate sidecars and copy dns-proxy from control plane project to customer project artifact registry repository
mi2 generate kustomize
kustomize build --load-restrictor LoadRestrictionsNone --enable-helm kubernetes/ | kubectl --kubeconfig=$(mi2 instance kubeconfig) apply -f -

4. Open Pull Request.

(Customer) Reached out to the account team

Customer need to provide the following information

• The GCP region customer’s private code host or the entrypoint is in, e.g., us-central1. If it is not one of the Cloud supported region, please consult the Cloud team.
• The DNS name of the customer private code host, e.g., gitlab.internal.company.net. Notes this is the dns name that the customer users interact with daily. If the private code host has geo-specific dns name, please consult the Cloud team.

(Customer) Create a service attachment to expose the private code host

Customer should publish their services using PSC by follow GCP documentation. Below is a Terraform example:

Notes your firewall rule should permit ingress traffic from the proxy-only subnet to the private code host. It should also permit egress traffic from the private code host to the psc subnet.

The customer need to provide the service attachment id to Sourcegraph, e.g., projects/:id/regions/:region/serviceAttachments/:name.

### Private Service Connect

resource "google_compute_service_attachment" "self" {
  name        = "${var.name}-psc-${random_id.tf_prefix.hex}"
  description = "A service attachment to expose the internal load balancer for Private Service Connect"

  # the region has to the same as the consumer due to GCP limitation
  region      = var.region

  # optional, e.g., gitlab.internal.company.com
  domain_names          = [var.domain_root]

  enable_proxy_protocol = false
  # a psc subnet, or with a `purpose=PRIVATE_SERVICE_CONNECT` `google_compute_subnetwork` in Terraform
  nat_subnets           = [google_compute_subnetwork.psc.id]

  # an upstream forwarding rule to one of the following
  #
  # - regional internal proxy Network Load Balancer, google_compute_region_target_tcp_proxy
  #   with an INTERNAL_MANAGED load balancing scheme in the backend service
  # - regional internal Application Load Balancer, google_compute_region_target_http_proxy, google_compute_region_target_https_proxy,
  #   with a INTERNAL_MANAGED load balancing scheme in the backend service
  # - internal passthrough Network Load Balancer, google_compute_region_target_tcp_proxy,
  #   with an INTERNAL load balancing scheme in the backend service
  #
  # how to decide?
  #
  # - if the target deployment (MIG or NEG) does not terminate TLS, you should use an internal Application Load Balancer with TLS certificate configured
  # - if the target deployment (MIG or NEG) terminates TLS, you should use an internal passthrough Network Load Balancer on port 443
  #
  # notes in order to create an internal load balancer, GCP requires the creation of a proxy-only subnet within the network
  target_service        = google_compute_forwarding_rule.https.self_link

  connection_preference = "ACCEPT_MANUAL"
  # a maps of "project_id" => { "id": "$project_id", "limit": $limit }
  dynamic "consumer_accept_lists" {
    iterator = each
    for_each = var.authorized_consumer_projects
    content {
      project_id_or_num = each.value["id"]
      connection_limit  = each.value["limit"]
    }
  }
}

### Compute - Network (psc subnet)

resource "google_compute_subnetwork" "psc" {
  name    = "${var.name}-psc-${random_id.tf_prefix.hex}"
  region  = var.region
  network = google_compute_network.self.id

  purpose       = "PRIVATE_SERVICE_CONNECT"
  ip_cidr_range = var.psc_subnetwork_cidr
}

output "service_attachment_uri" {
  value = google_compute_service_attachment.self.id
}