Hiding spaces behind cloudflare

November 23, 2017 2.5k views
Storage

I'm wondering if it is possible to use spaces behind the cloudflare proxy for cost and security reasons.
As far as I understood, this would involve the following:

  • Set up something to deny all requests to xxx.digitaloceanspaces.com from non-cloudflare IPs
  • Set up a custom subdomain on cloudflare that points to the space

Can spaces be used this way?

5 comments
  • Judging by the complete lack of responses, I'm assuming this isn't possible right now? Maybe it could be added in the future?

  • Hi there - I'm the product manager for Spaces. There is no way to do the 1st thing (limit Spaces access by IP) today. Some CDNs (not sure about cloudflare) will let you input an access/secret pair so you can limit reads that way. Is that what you're trying to accomplish or is it a hard requirement to block at the network level?

    Re: the 2nd thing, this works with any CDN and Spaces. However, the Cloudflare free plan doesn't allow host header re-writes, which what is required to make this work today. We are working on making it so that customers can do this with the cloudflare free, too. My hope is that we can release this in early Q1.

    Let me know if you have other feedback or can share more about your use case. Thanks so much for leaving the comment/question.

  • My use case is actually fairly simple, I'm looking to run the dynamic part of my site on a droplet, store all the static files in spaces, and hide the whole thing behind cloudflare: https://i.imgur.com/kR22efV.png

    Unfortunately I've not found anything on cloudflare about giving it a special access key. They always recommend you to deny all requests that do not come from these ip addresses, as early as possible: https://www.cloudflare.com/ips/ . Otherwise someone could simply bypass cloudflare by requesting the file from xxx.digitaloceanspaces.com directly and cause an unexpected high invoice using bot downloads.

    It looks like host header re-writes are limited to enterprise customers, not just paid plans, so this would not be useful for most people. But since you've already got custom domain support coming soon, the second part is solved.

3 Answers

We're working on it (custom domain support delivered via a new endpoint that will talk HTTP) -- still targeting early Q1.

  • Will using a custom domain completely hide the name of the space? That would mostly solve the problem of not being able to deny non-cloudflare traffic, simply name the space some random long string and noone will ever find it outside the custom subdomain that goes through cloudflare.

    • What do you mean by hide? @Zeblote

      • If I have a space called "hello", and make it accessible using a custom domain "static.stuff.com", will it be impossible for someone to figure out the space is called "hello" when all he has is an url "static.stuff.com/file.txt"?

I think Zeblote's comments describes very well the thing that he wants to do, I'm looking myself for the same thing and really haven't found any real answer.

Basically the thing is:

1.- Store web application on droplet, using a cdn like cloudflare point to the application by the domain 'stuff.com' and enjoy benefits from using cdn.

2.- Store all static files (.jpeg, .png, .css, .js, .mp4, etc.) on spaces. However by doing this anyone could access directly the files, consuming bandwith and resources of the spaces bucket, or simply perform malicious actions directly on the bucket itself.

3.- By hiding the bucket behind a cloudflare cdn I think he means (at least from my understanding), is that he wants to create a cname for 'stuff.com' alias 'static.stuff.com' wich points to the spaces bucket therefore the files stored at the bucket. So instead of accessing the files by 'stuff.nyc3.digitaloceanspaces.com/img-example.jpeg' we can access them simply like 'static.stuff.com/img-example.jpeg', wich is easy to remember and better brand representing than the first one, and enjoy benefits from using a cdn.

Nevertheless using a cdn brings a lot of benefits to the table, like security, file caching, etc...

In summary:

'stuff.com' in cloudflare points to web application ip in droplet.

'static.stuff.com' in cloudflare points to 'stuff.nyc3.digitaloceanspaces.com' in spaces.

Is there an actual way of doing this?

johngannon please give real answers.

  • They've launched a beta for the new "custom domain" feature since I asked this question, however I was not able to get this working with that either. Spaces will always respond to cloudflares requests with "404 NoSuchBucket" for no appearant reason.

    Followed by a very long support ticket (the messages have to be read bottom to top)
    https://i.imgur.com/QNxbmUV.png

    So, it looks like we can still not use spaces behind cloudflare. Why? Noone knows. It returns the file to a request from my browser and 404 to an identical request from cloudflare. Grey cloud has the same result as not using cloudflare at all so that's not really helpful either.

    I'm now thinking of using droplets with nginx to handle requests from cloudflare and forward the stuff from spaces, supposedly bandwidth between spaces and droplets in the same datacenter is free.

    • Yeah the forwarding stuff for me is a no go, I really (please please) expect my services to grow and eventually will have to load balance my servers, I can imagine the forwarding stuff with load balancing beeing a real headache.

      After hours and hours of trying I finally gave up with spaces, I really really wanted to go with spaces because I absolutely love DO droplets and wanted my application to be in the same platform. But this feature is really important for me and my brand and for future scalability purposes.

      I've set this up in google cloud storage and was a no brainer, you basically verify ownership of your domain, create your storage instance (name it to 'static.stuff.com' as has to match the url of your domain. The instance has to be called 'static.stuff.com' and the cname of your DNS has to be 'static' and your domain 'stuff.com' because they have to match), create cname 'static' (or whatever you want) of your domain 'stuff.com' (or whatever your domain is) in cloudflare to point to the google cloud storage URI ('c.storage.googleapis.com' by the time of this writing) and thats really it, now you can simple point your static assets to 'static.stuff.com/file.extension'.

      Until DO doesn't solve this I'm stuck with google cloud storage... :(

      I'm keeping my servers in DO droplets by the way, I absolutely love them!.

      • That's an option. But it doesn't look like a very good one:

        Running a few identical forwarder droplets costs you $5/TB. Miscalculating your scaling for a month, and paying 100% of the bandwidth as overages, costs you $20/TB.

        ...meanwhile google cloud storage eats a whopping $80 to $120/TB for no reason!

        And since this whole thing sits behind the cloudflare cache, you can probably get by with just a small number of droplets.

What you could do today is setup a reverse proxy using Nginx (or any other frontend server), rewriting the requests from something like assets.example.com to yourspace.region.digitaloceanspaces.com and put cloudflare in front of assets.example.com, so you should never pay for exceding transfer usage (as included one should be more than enough)

Have another answer? Share your knowledge.