Docker registry with Spaces as storage

Hi everyone,

I’m trying to set up a private docker registry using DigitalOcean S3 compatible Spaces as a storage, but bumped into a small issue that I am not sure how to resolve. Hoping to get some thoughts on how one could resolve this, as I’m sure that I’m missing something.

I have a cluster set up on DigitalOcean, with a few managers and worker nodes. I won’t go into detail on that, as it doesn’t seem necessary. I am setting up my registry as a service, as I am using Swarm Mode. I’m also using Docker Flow Proxy, essentially to expose things in a neat way. So, here’s how I’m deploying the service.

docker service create --name registry \
--network df-proxy --label com.df.notify=true \
--label com.df.port=5000 --label com.df.servicePath=/registry \
--label com.df.httpsOnly=true --label com.df.sslVerifyNone=true \
--secret registry_crt --secret registry_key \
--constraint "node.role==manager" --env-file registry.env registry:2

Nothing out of the ordinary here, basically exposing the service as /registry, with SSL passthrough on Docker Flow Proxy, so that it’s handled by the registry, and passing out the SSL certificate along with it.

The contents of the registry.env is as follows:

REGISTRY_HTTP_ADDR=0.0.0.0:5000
REGISTRY_STORAGE=s3
REGISTRY_STORAGE_S3_ACCESSKEY=my-digitalocean-spaces-key
REGISTRY_STORAGE_S3_SECRETKEY=my-digitalocean-spaces-secret
REGISTRY_STORAGE_S3_BUCKET=bucket-name
REGISTRY_STORAGE_S3_REGION=us-west-1
REGISTRY_STORAGE_S3_REGIONENDPOINT=https://my-endpoint.digitaloceanspaces.com
REGISTRY_ENCRYPT=false
REGISTRY_HTTP_TLS_CERTIFICATE=/run/secrets/registry_crt
REGISTRY_HTTP_TLS_KEY=/run/secrets/registry_key
REGISTRY_LOG_LEVEL=info

After I set this up, I navigate into the public IP address with /registry/v2 and I get a good response, with an empty result

{ }

However, a few seconds later, the same request will start returning 503 (service unavailable) and I get a message to check /debug/health. The logs don’t say much, basically I get requests that start with 200 and then change to 503.

So I went on and activated the debug health port, and got this from the output.

{"storagedriver_s3":"s3aws: NoSuchKey: \n\tstatus code: 404, request id: xxxxxxxxxxx"}

It seems the registry isn’t liking something from the S3 configuration, but I can’t figure out what. The access key and secret key seem to be correct, so is the bucket name and the endpoint…

Can anypoint point towards a solution or what could the problem actually be?

Cheers