Question

Does the monitoring agent poll for metrics, or does it look at differences in performance counters over time?

Posted July 1, 2020 158 views
Monitoring

I noticed recently that a batch job I’m running every 3h has far lower reported Disk I/O than is actually occurring (0.3 B/s rather than 30KB/s over a 20 minute window).

Most of the disk activity is concentrated in a small subset of that 20 minute window, so plausibly the monitoring agent is polling and recording the small amount of changes to file metadata and whatnot while completely missing the bulk of the data being written (as opposed to recording total changes since the last time the monitoring agent examined the disk).

Does that explanation make sense? Does the monitoring agent poll for current throughput, or is there something else going on?

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
1 answer

Hi there @hansvm,

I tried to reproduce this at my end but it seems to be working as expected.

I ran an I/O heavy operation while having iotop running and then I checked the graphs which represented the value accordingly:

Disk I/O DigitalOcean

However, if you believe that the data that you are seeing is way off the actual I/O, you could submit a GitHub issue here:

https://github.com/digitalocean/do-agent

Hope that this helps!
Regards,
Bobby

  • Hi Bobby,

    Thanks for taking the time to respond.

    Perhaps I’m just misunderstanding the units involved. During a heavy I/O workload, it looks like DO is reporting 25 bytes per second for you. What did iotop report?

    FWIW, DO’s graphs have been proportional to the expected disk I/O, just low by a factor of 10,000 or so.

    Thanks again!
    Hans

    • Hi there Hans,

      Hm yes indeed, good point. The value/number is correct but it was actually MB/s and not B/s.

      I will try to do some more tests to see if this is consistent. I will update you as soon as I have some more information.

      Regards,
      Bobby

    • Hi there @hansvm,

      This should now be all fixed. I would suggest running apt update && apt upgrade just to make sure that you are running the latest version.

      Let me know if you have any questions.

      Regards,
      Bobby

      • Hi Bobby,

        Everything looks great! Thank you for the fast fix :)

        Do you have time to give a brief summary of or link to whatever was going on? I didn’t notice anything telling in the latest do-agent releases.

        Best Wishes,
        Hans

        • Hi there @hansvm,

          Happy to hear that it looks good at your end as well!

          Actually, by the time I reported this, there was already a patch for that on the application level. As you mentioned the values were correct but were showing in B rather than MB.

          Hope that this helps!
          Regards,
          Bobby

Submit an Answer