Unable to deploy to Digital Ocean Kubernetes using GitLab CI

July 2, 2019 483 views
Kubernetes DigitalOcean

I’m moving my review apps from Google Kubernetes to Digital Ocean Kubernetes.

I have succesfully linked the Kubernetes clusted to GitLab. I have installed Helm Tiller, Ingress, Cert-Manager, and Prometheus using the GitLab integration. I have upgraded Tiller to the latest version manually, as did I on the previous Google cluster.

The cluster is RBAC-enabled,GitLab-managed, configured to use the default namespace, and tied to the * wildcard environment scope.

Now my deployments are failing. job output

$ helm upgrade "$CI_ENVIRONMENT_SLUG" config/charts/appsemble-docs --atomic --install --set "git.commit.sha=$CI_COMMIT_SHA" --set "image.tag=$CI_COMMIT_REF_NAME"
Error: pods is forbidden: User "system:serviceaccount:default:default-service-account" cannot list resource "pods" in API group "" in the namespace "gitlab-managed-apps"

Manual deployments on my own laptop work fine.

What could be causing this issue?

1 Answer
jkwiatkoski July 2, 2019
Accepted Answer

Hi there,

This looks like an RBAC permissions issue. The error you’re seeing is saying the service-account “default-service-account” in namespace “default” doesnt have access to pods in the ‘gitlab-managed-apps’ namespace.

You can fix this by either creating a role that allows this access then binding the sa to that role witha rolebinding:

See below for examples to achieve something similar:

This will create a role pod-reader that can get, list, and watch pods

kubectl create role pod-reader –verb=get,list,watch –resource=pods -n <namespace to read pods>

Then we will create the rolebinding to associate a serviceaccount to this role that can view pods.

kubectl create rolebinding sa-read-pods –role=pod-reader –user=system:serviceaccount:default:default-service-account -n rbac

Let me know if you have any further questions.

Regards,

John Kwiatkoski
Senior Developer Support Engineer

  • I deleted the cluster and created a new cluster. I changed the namespace in GitLab to appsemble instead, so the command for the role binding changed to:

    kubectl create rolebinding sa-read-pods --role=pod-reader --user=system:serviceaccount:appsemble:appsemble-service-account -n gitlab-managed-apps
    

    I also needed to be able to port forward, so I changed the role to this:

    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: pod-reader
      namespace: gitlab-managed-apps
    rules:
    - apiGroups:
      - ""
      resources:
      - pods
      - pods/portforward
      verbs:
      - get
      - list
      - watch
    - apiGroups:
      - ""
      resources:
      - pods/portforward
      verbs:
      - create
      - delete
      - deletecollection
      - patch
      - update
    

    I do wonder, any clue why this is necessary for the cluster in Digital Ocean, but not for the one in Google Cloud?

  • Thanks by the way! :D

Have another answer? Share your knowledge.