14.04 Munin Disk latency

April 25, 2014 6.8k views
I am currently upgrading a server from Ubuntu 13.10 to 14.04. Building the 14.04 server from scratch and just moving data over to it. Problem I am seeing is munin keeps giving me warning of Disk latency per device on the 14.04 server currently not under any load. Disk latency per device Graphs shows on 14.04 cur min avg max VDA 11.25 366.56n 18.20 43.37 13.10 server under load cur min avg max VDA 222.42u 142.09u 1.44m 25.53m for Average latency for dev/vda 14.04 I am just going to show average so far Device IO Time 309.27u IO Wait Time 18.29 Read IO wait time 10.15 Write IO wait time 21.53 for Average latency for dev/vda 13.10 I am just going to show average so far Device IO Time 209.54u IO Wait Time 1.43m Read IO wait time 318.84u Write IO wait time 7.95m ioping currently shows about 260u on both servers. Is this just a munin bug or do I have a problem. If bug how do I fix it?
7 Answers
Apparently is a bug of Trusty kernel compilation:

  • Yep, kernel issue! I'm on Ubuntu 12.04 x64 and just started seeing the problem when I switched kernels from default to 3.13.0-32.

    Thanks @edujoseg - per that thread, looks like it's fixed in 3.14, and maybe 3.13.0-35 (which is available for Ubuntu 14.04)? I'll post a support ticket to DO to see about getting some updated kernels available.

  • @draknor, I've checked and 3.13.0-35 still broken.

    The fix landed on the proposal package 3.13.0-36. It should be released as stable very soon (couple weeks).

I have exactly same problem !.
I was going to build another test server in different location to see if problem follows. My first test server was in New York 2. If yours was in different location let me know!

Thinking this is something with munin or 14.04 rather than DO but have no clue how to correct.

My location is New York 2.
Yes, real performance of disk is fast, apparently the problem is focused in kernel diskstats info reported with /proc/diskstats. Comparing with another VPS of different provider running Xen shows:

202 0 xvda 25327 394 557386 14570 1284988 1778684 25167696 4052686 0 369456 4065963

while my DO VPS shows:

253 0 vda 260183 50 9542362 295950084 2318499 867088 55409896 388007768 0 1041416 13675936

Check seventh value corresponding to time spent reading (ms):

14570 versus 295950084

Kernel bug?

For the heck of it I created a droplet on NY 1. Problem follows.

Here is why I came up with 13.10 (normal load)
253 0 vda 23038975 2222175 838772522 6626356 2441149 3566029 107325192 24848616 0 3147096 31456468

14.04 not doing anything
253 0 vda 638140 88126 27884970 843291604 241292 396640 13099616 682631036 0 238300 2472584

So are we looking at a kernel bug?
@edujoseg: Thanks for finding that!

Ubuntu just released the kernel 3.13.0-36. Now digitalocean needs to update their systems to allow us to change the kernel on panel (and maybe the default as well).

Have another answer? Share your knowledge.