Question

14.04 Munin Disk latency

Posted April 25, 2014 14.9k views
I am currently upgrading a server from Ubuntu 13.10 to 14.04. Building the 14.04 server from scratch and just moving data over to it. Problem I am seeing is munin keeps giving me warning of Disk latency per device on the 14.04 server currently not under any load. Disk latency per device Graphs shows on 14.04 cur min avg max VDA 11.25 366.56n 18.20 43.37 13.10 server under load cur min avg max VDA 222.42u 142.09u 1.44m 25.53m for Average latency for dev/vda 14.04 I am just going to show average so far Device IO Time 309.27u IO Wait Time 18.29 Read IO wait time 10.15 Write IO wait time 21.53 for Average latency for dev/vda 13.10 I am just going to show average so far Device IO Time 209.54u IO Wait Time 1.43m Read IO wait time 318.84u Write IO wait time 7.95m ioping currently shows about 260u on both servers. Is this just a munin bug or do I have a problem. If bug how do I fix it?
Add a comment
vps
By:
vps
Add a comment
2 comments

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
Submit an Answer
7 answers
edujoseg May 1, 2014
Apparently is a bug of Trusty kernel compilation:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1297522

Reply Report
  • draknor September 9, 2014

    Yep, kernel issue! I’m on Ubuntu 12.04 x64 and just started seeing the problem when I switched kernels from default to 3.13.0-32.

    Thanks @edujoseg - per that thread, looks like it’s fixed in 3.14, and maybe 3.13.0-35 (which is available for Ubuntu 14.04)? I’ll post a support ticket to DO to see about getting some updated kernels available.

    Reply Report
  • TracyWebTech September 10, 2014

    @draknor, I’ve checked and 3.13.0-35 still broken.

    The fix landed on the proposal package 3.13.0-36. It should be released as stable very soon (couple weeks).

    Reply Report
edujoseg April 27, 2014
I have exactly same problem !.
Reply Report
vps April 28, 2014
I was going to build another test server in different location to see if problem follows. My first test server was in New York 2. If yours was in different location let me know!

Thinking this is something with munin or 14.04 rather than DO but have no clue how to correct.

Reply Report
edujoseg April 28, 2014
My location is New York 2.
Yes, real performance of disk is fast, apparently the problem is focused in kernel diskstats info reported with /proc/diskstats. Comparing with another VPS of different provider running Xen shows:

202 0 xvda 25327 394 557386 14570 1284988 1778684 25167696 4052686 0 369456 4065963

while my DO VPS shows:

253 0 vda 260183 50 9542362 295950084 2318499 867088 55409896 388007768 0 1041416 13675936

Check seventh value corresponding to time spent reading (ms):

14570 versus 295950084

Kernel bug?

Edu
Reply Report
vps April 30, 2014
For the heck of it I created a droplet on NY 1. Problem follows.


Here is why I came up with 13.10 (normal load)
253 0 vda 23038975 2222175 838772522 6626356 2441149 3566029 107325192 24848616 0 3147096 31456468

14.04 not doing anything
253 0 vda 638140 88126 27884970 843291604 241292 396640 13099616 682631036 0 238300 2472584

So are we looking at a kernel bug?
Reply Report
asb May 1, 2014
@edujoseg: Thanks for finding that!
Reply Report
TracyWebTech September 22, 2014

Ubuntu just released the kernel 3.13.0-36. Now digitalocean needs to update their systems to allow us to change the kernel on panel (and maybe the default as well).

Reply Report

Looking for something else?

Ask a new question Search for more help