How to authorize a kubernetes service account without doctl?

So I am trying to launch to production via a popular CI tool, and I cannot yet understand why doctl is necessary. Can it be done without doctl? Here is my rationale and the issues I am facing.

My main motivation is that there some principle of giving the lowest amount of access as is necessary. A doctl API token has control over my whole DO account and I do not think it is wise to have that level of access on my CI. I would prefer to use some sort of Service Account as creating a service account and using the token to access the cluster should be doable and acceptable Kubernetes Authorization Docs.

Also, each time I download the config or when I create a new service account using the CLI a new DO API token is automatically generated. For the service accounts I do not get to see the secret token because I created the service account using the CLI. I figured that the token in the ClusterRoleBinding would work or be equivalent to the auto generated API token but it is not.

What is the right way to do this?

If it should be working here is what I am working with:

Using the handy download config options provided by DigitalOcean I have copied the format and inserted my service account name and base 64 decoded service account token (shown below). This kubeconfig file which I can encrypt and pass to my CI to use when making kubectl commands to my cluster. The problem is it hangs my CI asking for a UserName and Password. So…

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <CADATA>
    server: <SERVER>
  name: <CLUSTERNAME>
contexts:
- context:
    cluster: <CLUSTERNAME>
    user: <CLUSTERNAME>-admin
  name: <CLUSTERNAME>
current-context: <CLUSTERNAME>
kind: Config
preferences: {}
users:
- name: <SERVICEACCOUNT-USERNAME>
  user:
    token: <BASE64 decoded SERVICE ACCOUNT Token>

So I rechecked my kubeconfig file on my desktop that is using doctl and found a few extra lines to add to this file. But now I am getting “Error: Unable to initialize DigitalOcean API client: access token is required. (hint: run ‘doctl auth init’)”

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <CADATA>
    server: <SERVER>
  name: <CLUSTERNAME>
contexts:
- context:
    cluster: <CLUSTERNAME>
    user: <CLUSTERNAME>-admin
  name: <CLUSTERNAME>
current-context: <CLUSTERNAME>
kind: Config
preferences: {}
users:
- name: <CLUSTERNAME>-admin
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - kubernetes
      - cluster
      - kubeconfig
      - exec-credential
      - --version=v1beta1
      - --context=default
      - <SERVERID>
      command: /home/<ME>/doctl
      env: null
      provideClusterInfo: false
- name: <SERVICEACCOUNT-USERNAME>
  user:
    token: <BASE64 decoded SERVICE ACCOUNT Token>

So any suggestions? Thanks in advance for your help.