Managing user uploads across a load balanced set up

September 4, 2017 1.1k views
Load Balancing Nginx Ubuntu
derekalmond
By:
derekalmond

What are the available strategies for managing file uploads in a load balanced set up?

at present we have the load balancer on www.example.com - then various heads w-s1.example.com, w-s2.example.com ...w-sX.example.com

if a user uploads a file it ends up on which ever head that session has connected to.

What's the ideal way to manage the file so it is accessible from sessions that connect to the other boxes, WITHOUT creating a single point of failure, a minimal delay (a minute or so) is acceptable, longer is not.

as i understand it the DO block storage devices can only be mounted to a single device at a time?

rysnc or similar methods of syncing all boxes might work - but it's difficult to automate fully so that deletions and mirroring all take place correctly.

Log In to Comment
2 Answers
mikebywater September 4, 2017

The new digital ocean "spaces" object storage seems a good fit for me. Its got a similar api to amazon s3 storage and you would configure the endpoint for all 3 of your webservers and whichever webapp the upload posted to will upload it into object storage in a central place.

Reply
jtittle MOD September 7, 2017

@derekalmond

When it comes to HA and Load Balancing, ideally, you'd have a central storage cluster that each of the application servers connects to. GlusterFS is a popular, but somewhat complex option.

You'd be setting up a client server and then N number of additional servers that would either replicate or distribute your data. The application servers would mount the client server, which is what would be the link to the other storage clients in the cluster.

GlusterFS does have fail over, though I've not yet been able to get it to work as expected (i.e if the client you mounted on each application server fails, it should bring up the next, but that was never the case when I was working with it).

That said, the other alternative would be bi-directional syncing using rsync or lsyncd, You'd have to set the syncing up on each server and make sure only the changed files sync.

Reply
Have another answer? Share your knowledge.

Related Questions