Managed Instance Private Code Hosts support

WARNING: This is still work in progress!

Every v2.0 Cloud Instance is deployed in Google Cloud Platform.

Private Code Host is a code host deployed in a private network (for example AWS EC2 instance within VPC). To connect to this code host a user has to have access to the private network usually via VPC Peering, VPN, or tunneling.

This solution is recommended by Cloud Team, but only possible if customers’ code hosts can be accessible publicly and customer is able to allow incoming traffic from Sourcegraph-owned IP addresses. Outgoing traffic of Cloud instances goes through Cloud NAT with stable IPs. All IPs are reserved exclusively on a per customer basis. More informations about IP allowlist

AWS GCP site-to-site VPN solution (AWS code hosts only)

This option is for customers who want to connect to a private code host that is hosted on AWS infrastructure.

Architecture

AWS VPN architecture

Creation process

AWS GCP VPN extension connects an existing Cloud Instance with customer dedicated AWS account (maintained by Cloud Team).

1. Modify config.yaml with additional section:

spec:
  infrastructure:
    privateCodeHost:
      aws:
        accountID: <AWS_ACCOUNT_ID>
        highAvailable: true
        region: <AWS_REGION>
      enabled: true
      type: awsvpn

2. Generate additional terraform stacks

mi2 generate cdktf

this will generate awsvpc and awsvpn stacks.

Creation process detail walkthrough:

  1. AWS VPC is created in customer dedicated Sourcegraph AWS Account.
  2. Cloud Router with GCP-side BGP ASN number is created in Managed Instance VPC.
  3. AWS VPN Gateway with BGP AWS-side ASN number is created. It is attached to VPC and uses VPC route tables to propagate BPG sessions.
  4. High Available GCP VPN Gateway is created in Managed Instance GCP VPC with 2 interfaces.
  5. AWS Customer Gateway is created (uses GCP-side BGP ASN). For high available options 2 gateways are created.
  6. AWS VPN connection is created with 2 tunnel options (every tunnel uses different 32B PreSharedKey). For high available options 2 VPC connections are created.
  7. GCP External VPN Gateway is created with interfaces correspoding for each AWS VPN connection tunnel. For high available options External VPN Gateway has 4 interfaces, for non high-available 2.
  8. For each External VPN Gateway interface GCP VPN Tunnel is created. It uses dedicated PreSharedKey (same in on AWS VPN Connection(s)).
  9. For each GCP VPN Tunnel interface is added to Cloud Router. It uses IP range correspoding to AWS VPN Connection tunnel internal address.
  10. For each AWS VPN Connection tunnel internal address BGP peer is added to Cloud Router.

For more details, go to Google documentation

VPN Verification

For each customer using private code host, additional section to our generated operation dashboard is added. Upon enabling the private code host support, follow the process to update the dashboard

AWS Private Link is a secure method to expose a single endpoint (i.e. code host) from AWS VPC to specific AWS Principals (IAM in specific AWS VPC). This solution ensures code the host is not exposed from AWS and only a given principal can access it.

When a customer exposes the code host via VPC Endpoint Service, the Cloud Instance needs to create an AWS VPC Endpoint to expose connections inside the Sourcegraph Managed AWS VPC.

  1. Add addtional field to privateCodeHost section
spec:
  infrastructure:
    privateCodeHost:
      aws:
        # privateLink represents a VPC endpoint exposed by customers
        privateLink:
          # Endpoint VPC Endpoint Service name
          endpoint: com.amazonaws.vpce.<REGION>.<VPC_ENDPOINT_SERVICE_ID>
          # ports list of TCP ports to open for GKE in AWS Security Group, by default only 443
          ports:
            - 443
        # enabled only if customer uses Private DNS Domain, this can be done only after customers accepts VPC Endpoint
        privateDnsEnabled: false
  1. Generate additional resources in cdktf and apply
mi2 generate cdktf
mi2 instance cdktf deploy -auto-approve

If customer uses Private Domain for VPC Endpoint Service and accepted VPC Endpoint connection:

  1. enable Private DNS in VPC Endpoint
spec:
  infrastructure:
    privateCodeHost:
      aws:
        # enabled only if customer uses Private DNS Domain, this can be done only after customers accepts VPC Endpoint
        privateDnsEnabled: true
  1. Generate additional resources in cdktf and apply
mi2 generate cdktf
mi2 instance cdktf deploy -auto-approve

If customer uses Private Domain, addtional DNS proxy sidecar also needs to be used in Cloud Instance.

[Optional] Private code host domain

Every Cloud customer using private DNS domain for code hosts requires additional dns-proxy sidecar. DNS-proxy is rewriting from private customer domain to public, resolvable A record i.e. private.CUSTOMER.com -> public.CUSTOMER.com where:

  • private.CUSTOMER.com is customer code host address, only resolvable inside customer network
  • public.CUSTOMER.com - is resolvable to customer code host private IP Dns-proxy sidecar modifies pod /etc/resolv.conf file.
  1. Configure config.yaml
spec:
  infrastructure:
    privateDNS:
      routes:
        - source: private.CUSTOMER.com
          destination: public.CUSTOMER.com
          # resolverIP: IP_OF_CUSTOM_RESOLVER - optional
    gke:
      blue:
        additionalOauthScopes: # required for each node pool to pull dns-proxy private image from artifact registry
          - https://www.googleapis.com/auth/devstorage.read_only
  1. Generate and apply cdktf
mi2 generate cdktf
# modify GKE node pool and add artifact registry repository with permissions
mi2 instance tfc deploy -auto-approve
  1. Generate dns-proxy sidecars
# generate sidecars and copy dns-proxy from control plane project to customer project artifact registry repository
mi2 generate kustomize
cd kubernetes
kustomize build --load-restrictor LoadRestrictionsNone --enable-helm . | kubectl apply -f -

When a customer has private code hosts inside the AWS VPC and needs to expose it for Sourcegraph managed AWS VPC, customers can use AWS Documentation or follow these steps:

  1. Create Network Loadbalancer, VPC Endpoint Service and VPC Endpoint Service via terraform file:

Important: if https/ssl is used on NLB (recommended), customer has to create AWS Certificate and set certificate_arn and ssl_policy field in aws_lb_listener resources before apply.

resource "aws_lb" "nlb" {
  name               = "${var.customer_id}-nlb"
  internal           = true
  load_balancer_type = "network"
  subnets            = var.customer_vpc_public_subnets

  enable_deletion_protection = true

  tags = {
   // customer tags
  }
}

resource "aws_lb_listener" "tls" {
  load_balancer_arn = aws_lb.nlb.arn
  port              = "443"
  protocol          = "TLS"
  ssl_policy        = "ELBSecurityPolicy-TLS13-1-2-"
  certificate_arn   = "arn:aws:acm:<REGION>:<ACCOUNT:certificate/<CERTIFICATE_ID"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.customer_gitlab.arn
  }
}

resource "aws_vpc_endpoint_service" "customer_gitlab" {
  acceptance_required        = true
  network_load_balancer_arns = [aws_lb.nlb.arn]

  allowed_principals = [
    "arn:aws:iam::<SOURCEGRAPH_MANAGED_CUSTOMER_AWS_ACCOUNT_ID>:root"
  ]

  tags = {
   // customer tags
  }
}
  1. For private VPC Endpoint Service domain, customer has to follow verification steps to prove customer owns the domain.