Question
Poor fsync performance on DO Block Storage
- Posted on May 24, 2020
- MySQLBlock StorageStorage
Asked by vasval
Hi there!
I have noticed very poor MySQL (InnoDB) performance on my droplet.
I am running MySQL (Percona Server) 5.6 with fine tuned config.
Database directory placed on BlockStorage.
I used fio utility to test BlockStorage disk performance and everything is ok, if I running fio as described here https://www.digitalocean.com/community/tutorials/how-to-benchmark-digitalocean-volumes . Burst performance approximately near 7500 iops, after 1 minute test it slow down to approx. 5000 iops.
As InnoDB using fsync for each commit operation to compliant data consistency I tested fio performance with fsync for each IO operation (—fsync=1). In test with fsync=1 I noticed only 50-100 iops!
Can anybody answer me why performance so poor in my case?
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
Want to learn more? Join the DigitalOcean Community!
Join our DigitalOcean community of over a million developers for free! Get help and share knowledge in Q&A, subscribe to topics of interest, and get courses and tools that will help you grow as a developer and scale your project or business.
I’ve found similar.
After some investigation (and reading up) I think the problem is that DO’s hypervisors do not use battery backed RAID controllers [would be nice if they verified this]. This means an fsync has to pysically flush to disk which is slow even on SSDs.
I see an average of about 20ms which matches your 50/sec. However, I am measuring from within the app which requires a localhost network round trip too… and I’m actually committing real data.
In a previous incarnation I used to manage database RAID arrays and one problem is that once the battery goes (which shows up as a tiny light on the controller and happens every year or two) the performance drops away.