How to use a k8s pod security policy to simulate admin restrictions

This describes how to simulate being a Sourcegraph admin with restrictions. ie an admin that can only schedule pods with security restrictions applied to the pods.

It is assumed you have a test cluster running in GCP or a similar k8s runtime and you are a cluster admin, ie you have super powers in the cluster:

kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user $(gcloud config get-value account)

First we need a pod security policy that matches as close as possible the restrictions at the customer site. For example (this is just an example, tweak as necessary):

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: nonroot-policy
spec:
  privileged: false
  allowPrivilegeEscalation: false
  # The rest fills in some required fields.
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: 'MustRunAs'
    ranges:
      # Forbid adding the root group.
      - min: 1
        max: 65535
  runAsUser:
    rule: 'MustRunAsNonRoot'
  fsGroup:
    rule: 'MustRunAs'
    ranges:
      # Forbid adding the root group.
      - min: 1
        max: 65535
  volumes:
    - '*'
  readOnlyRootFilesystem: true

Save this yaml into nonroot-policy.yaml and add the policy to the cluster:

kubectl apply -f nonroot-policy.yaml

From here on out we will do almost everything in a namespace. This is not strictly necessary (it works in the default namespace too) but it is cleaner and we’re migrating to namespaces anyways.

We create the namespace:

kubectl create namespace ns-sourcegraph

We now create a fake account, bind this policy to the account and impersonate it when we deploy and manage Sourcegraph. The impersonation will ensure that the policy is enforced for any pods that are scheduled/edited/managed by the fake account.

Let’s create the fake account. It will be a service account (so not a user). This enables us to later impersonate it.

kubectl create serviceaccount -n ns-sourcegraph fake-user

We create a role binding for it to declare that the fake user can edit the cluster in the namespace (schedule pods):

kubectl create rolebinding -n ns-sourcegraph fake-editor --clusterrole=edit --serviceaccount=ns-sourcegraph:fake-user

We create a role for the pod security policy:

kubectl create role -n ns-sourcegraph nonroot:unprivileged --verb=use --resource=podsecuritypolicy --resource-name=nonroot-policy

and then bind it to the fake user:

kubectl create rolebinding -n ns-sourcegraph fake-user:nonroot:unprivileged --role=nonroot:unprivileged --serviceaccount=ns-sourcegraph:fake-user

Our fake user is now under the spell of the pod security policy. Let’s check that:

kubectl --as=system:serviceaccount:ns-sourcegraph:fake-user -n ns-sourcegraph auth can-i use podsecuritypolicy/nonroot-policy

It should return yes.

From here on you can impersonate the fake user by adding --as=system:serviceaccount:ns-sourcegraph:fake-user -n ns-sourcegraph to your kubectl commands:

kubectl --as=system:serviceaccount:ns-sourcegraph:fake-user -n ns-sourcegraph get pods