Question
Managing user uploads across a load balanced set up
- Posted September 4, 2017
- Load BalancingNginxUbuntu
What are the available strategies for managing file uploads in a load balanced set up?
at present we have the load balancer on www.example.com - then various heads w-s1.example.com, w-s2.example.com …w-sX.example.com
if a user uploads a file it ends up on which ever head that session has connected to.
What’s the ideal way to manage the file so it is accessible from sessions that connect to the other boxes, WITHOUT creating a single point of failure, a minimal delay (a minute or so) is acceptable, longer is not.
as i understand it the DO block storage devices can only be mounted to a single device at a time?
rysnc or similar methods of syncing all boxes might work - but it’s difficult to automate fully so that deletions and mirroring all take place correctly.
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
@derekalmond
When it comes to HA and Load Balancing, ideally, you’d have a central storage cluster that each of the application servers connects to. GlusterFS is a popular, but somewhat complex option.
You’d be setting up a client server and then N number of additional servers that would either replicate or distribute your data. The application servers would mount the client server, which is what would be the link to the other storage clients in the cluster.
GlusterFS does have fail over, though I’ve not yet been able to get it to work as expected (i.e if the client you mounted on each application server fails, it should bring up the next, but that was never the case when I was working with it).
That said, the other alternative would be bi-directional syncing using rsync or lsyncd, You’d have to set the syncing up on each server and make sure only the changed files sync.
The new digital ocean “spaces” object storage seems a good fit for me. Its got a similar api to amazon s3 storage and you would configure the endpoint for all 3 of your webservers and whichever webapp the upload posted to will upload it into object storage in a central place.