Question

mysqld process 100% and "waiting TTFB" is almost 9 seconds

Posted January 21, 2020 171 views
MySQL

Hello community.

I’m coming across an issue with my website.

I started my project with a 1cpu and 2gb ram droplet, but after a while I start to see that there is 1 Mysqld process that run 100% and the website take almost 9 second to respond.

The website run wordpress and include 134809 posts at the moment.

This one is the Mysqltuner script result:

-------- Log file Recommendations ------------------------------------------------------------------
[OK] Log file /var/log/mysql/error.log exists
[--] Log file: /var/log/mysql/error.log(0B)
[OK] Log file /var/log/mysql/error.log is readable.
[!!] Log file /var/log/mysql/error.log is empty
[OK] Log file /var/log/mysql/error.log is smaller than 32 Mb
[OK] /var/log/mysql/error.log doesn't contain any warning.
[OK] /var/log/mysql/error.log doesn't contain any error.
[--] 0 start(s) detected in /var/log/mysql/error.log
[--] 0 shutdown(s) detected in /var/log/mysql/error.log

-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +Aria +CSV +InnoDB +MEMORY +MRG_MyISAM +MyISAM +PERFORMANCE_SCHEMA +SEQUENCE
[--] Data in InnoDB tables: 1.5G (Tables: 35)
[OK] Total fragmented tables: 0

-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.

-------- Security Recommendations ------------------------------------------------------------------
[OK] There are no anonymous accounts for any database users
[!!] User 'debian-sys-maint@localhost' has no password set.
[!!] There is no basic password file list!

-------- CVE Security Recommendations --------------------------------------------------------------
[--] Skipped due to --cvefile option undefined

-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 9h 57m 28s (21M q [613.584 qps], 6K conn, TX: 25G, RX: 2G)
[--] Reads / Writes: 99% / 1%
[--] Binary logging is disabled
[--] Physical Memory     : 1.9G
[--] Max MySQL memory    : 3.1G
[--] Other process memory: 0B
[--] Total buffers: 320.0M global + 18.8M per thread (151 max threads)
[--] P_S Max memory usage: 0B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 451.7M (22.66% of installed RAM)
[!!] Maximum possible memory usage: 3.1G (158.59% of installed RAM)
[!!] Overall possible memory usage with other process exceeded memory
[OK] Slow queries: 0% (354/21M)
[OK] Highest usage of available connections: 4% (7/151)
[OK] Aborted connections: 0.05%  (3/6452)
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[!!] Query cache may be disabled by default due to mutex contention.
[!!] Query cache efficiency: 1.0% (216K cached / 22M selects)
[!!] Query cache prunes per day: 47198665
[OK] Sorts requiring temporary tables: 0% (3K temp sorts / 7M sorts)
[!!] Joins performed without indexes: 171
[!!] Temporary tables created on disk: 99% (7M on disk / 7M total)
[OK] Thread cache hit rate: 99% (7 created / 6K connections)
[OK] Table cache hit rate: 80% (84 open / 105 opened)
[OK] table_definition_cache(400) is upper than number of tables(194)
[OK] Open file limit used: 0% (33/4K)
[OK] Table locks acquired immediately: 100% (24 immediate / 24 locks)

-------- Performance schema ------------------------------------------------------------------------
[--] Performance schema is disabled.
[--] Memory used by P_S: 0B
[--] Sys schema isn't installed.

-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is enabled.
[--] Thread Pool Size: 2 thread(s).
[--] Using default value is good enough for your version (10.3.18-MariaDB-1:10.3.18+maria~bionic)

-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 18.3% (3M used / 16M cache)
[OK] Key buffer size / total MyISAM indexes: 16.0M/124.0K

-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[!!] InnoDB buffer pool / data size: 128.0M/1.5G
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (75 %): 48.0M * 2/128.0M should be equal to 25%
[OK] InnoDB buffer pool instances: 1
[--] Number of InnoDB Buffer Pool Chunk : 1 for 1 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 99.39% (8765963307 hits/ 8819656689 total)
[!!] InnoDB Write Log efficiency: 12.43% (1950 hits/ 15694 total)
[OK] InnoDB log waits: 0.00% (0 waits / 13744 writes)

-------- AriaDB Metrics ----------------------------------------------------------------------------
[--] AriaDB is enabled.
[OK] Aria pagecache size / total Aria indexes: 128.0M/1B
[!!] Aria pagecache hit rate: 77.4% (31M cached / 7M reads)

-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.

-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.

-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.

-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: MIXED
[--] XA support enabled: ON
[--] Semi synchronous replication Master: OFF
[--] Semi synchronous replication Slave: OFF
[--] This is a standalone server

-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    Set up a Secure Password for debian-sys-maint@localhost user: SET PASSWORD FOR 'debian-sys-maint'@'SpecificDNSorIp' = PASSWORD('secure_password');
    MySQL was started within the last 24 hours - recommendations may be inaccurate
    Reduce your overall MySQL memory footprint for system stability
    Dedicate this server to your database for highest performance.
    Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
    We will suggest raising the 'join_buffer_size' until JOINs not using indexes are found.
             See https://dev.mysql.com/doc/internals/en/join-buffer-size.html
             (specially the conclusions at the bottom of the page).
    When making adjustments, make tmp_table_size/max_heap_table_size equal
    Reduce your SELECT DISTINCT queries which have no LIMIT clause
    Performance schema should be activated for better diagnostics
    Consider installing Sys schema from https://github.com/mysql/mysql-sys for MySQL
    Consider installing Sys schema from https://github.com/FromDual/mariadb-sys for MariaDB
    Before changing innodb_log_file_size and/or innodb_log_files_in_group read this: https://bit.ly/2TcGgtU
Variables to adjust:
  *** MySQL's maximum memory usage is dangerously high ***
  *** Add RAM before increasing MySQL buffer variables ***
    query_cache_size (=0)
    query_cache_type (=0)
    query_cache_limit (> 1M, or use smaller result sets)
    query_cache_size (> 16M)
    join_buffer_size (> 256.0K, or always use indexes with JOINs)
    tmp_table_size (> 16M)
    max_heap_table_size (> 16M)
    performance_schema = ON enable PFS
    innodb_buffer_pool_size (>= 1.5G) if possible.
    innodb_log_file_size should be (=16M) if possible, so InnoDB total log files size equals to 25% of buffer pool size.

Do you have any suggestions on what I can check or configure, in order to improve the performance?

Thanks

edited by MattIPv4

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
1 answer

Hi @marcomaneng,

It seems like your droplet is not able to cope with the traffic and the size of your website.

What I’ll recommend is to try and optimize both your server and your website.

Server side optimization

  • I’ll recommend adding SWAP if you having already done that.
  • Enable OpCache on your droplet so that you have somesorts of server side caching
  • Upgrade the droplet if the above doesn’t help
  • Start using Redis

Website Optimization

  • Install a caching plugin on your website
  • Install a Plugin to connect to your redis service on your droplet
  • Add Gzip compression

Those are the standard optimizations you’ll need to perform on both your website and droplet before you can remedy the issue experienced.

Additionally, the 9 TTB might be due to full Apache Workers, if you are using Apache that is. If you are, consider upping the maximum allowed connections and apache workers as soon as you update your droplet.

Regards,
KDSys

  • First of all thanks for the suggestions! Better than the ticketing system support :D. If the mod in this forum/community are not paid I would be surprised.

    I still have some delay on loading pages like this one: https://uquotes.club/quotes-sentences-and-memes/

    I just migrate to a 2cpu droplet with the same amount of ram (2gb).
    The droplet include already LiteSpeed Cache

    About the SWAP memory:
    SwapTotal: 4194300 kB

    About the OpCache:
    https://ibb.co/swFpz4d
    https://ibb.co/jG4Lyz5

    This one is the new mysqltuner Log after the migration:

    -------- Log file Recommendations ----------------------------------------------                                                                                                                                             --------------------
    [OK] Log file /var/log/mysql/error.log exists
    [--] Log file: /var/log/mysql/error.log(1K)
    [OK] Log file /var/log/mysql/error.log is readable.
    [OK] Log file /var/log/mysql/error.log is not empty
    [OK] Log file /var/log/mysql/error.log is smaller than 32 Mb
    [!!] /var/log/mysql/error.log contains 1 warning(s).
    [OK] /var/log/mysql/error.log doesn't contain any error.
    [--] 1 start(s) detected in /var/log/mysql/error.log
    [--] 1) 2020-01-21 14:46:15 0 [Note] /usr/sbin/mysqld: ready for connections.
    [--] 0 shutdown(s) detected in /var/log/mysql/error.log
    
    -------- Storage Engine Statistics ---------------------------------------------                                                                                                                                             --------------------
    [--] Status: +Aria +CSV +InnoDB +MEMORY +MRG_MyISAM +MyISAM +PERFORMANCE_SCHEMA                                                                                                                                              +SEQUENCE
    [--] Data in InnoDB tables: 787.3M (Tables: 32)
    [OK] Total fragmented tables: 0
    
    -------- Analysis Performance Metrics ------------------------------------------                                                                                                                                             --------------------
    [--] innodb_stats_on_metadata: OFF
    [OK] No stat updates during querying INFORMATION_SCHEMA.
    
    -------- Security Recommendations ----------------------------------------------                                                                                                                                             --------------------
    [OK] There are no anonymous accounts for any database users
    [!!] User 'debian-sys-maint@localhost' has no password set.
    [!!] There is no basic password file list!
    
    -------- CVE Security Recommendations ------------------------------------------                                                                                                                                             --------------------
    [--] Skipped due to --cvefile option undefined
    
    -------- Performance Metrics ---------------------------------------------------                                                                                                                                             --------------------
    [--] Up for: 11h 44m 27s (25M q [609.763 qps], 8K conn, TX: 29G, RX: 3G)
    [--] Reads / Writes: 99% / 1%
    [--] Binary logging is disabled
    [--] Physical Memory     : 1.9G
    [--] Max MySQL memory    : 3.1G
    [--] Other process memory: 0B
    [--] Total buffers: 320.0M global + 18.8M per thread (151 max threads)
    [--] P_S Max memory usage: 0B
    [--] Galera GCache Max memory usage: 0B
    [OK] Maximum reached memory usage: 545.8M (27.38% of installed RAM)
    [!!] Maximum possible memory usage: 3.1G (158.59% of installed RAM)
    [!!] Overall possible memory usage with other process exceeded memory
    [OK] Slow queries: 0% (62/25M)
    [OK] Highest usage of available connections: 7% (12/151)
    [OK] Aborted connections: 0.01%  (1/8162)
    [!!] name resolution is active : a reverse name resolution is made for each new                                                                                                                                              connection and can reduce performance
    [!!] Query cache may be disabled by default due to mutex contention.
    [!!] Query cache efficiency: 1.0% (255K cached / 25M selects)
    [!!] Query cache prunes per day: 47625169
    [OK] Sorts requiring temporary tables: 0% (2K temp sorts / 8M sorts)
    [!!] Joins performed without indexes: 170
    [!!] Temporary tables created on disk: 99% (8M on disk / 8M total)
    [OK] Thread cache hit rate: 99% (12 created / 8K connections)
    [OK] Table cache hit rate: 92% (70 open / 76 opened)
    [OK] table_definition_cache(400) is upper than number of tables(191)
    [OK] Open file limit used: 0% (25/4K)
    [OK] Table locks acquired immediately: 100% (18 immediate / 18 locks)
    
    -------- Performance schema ----------------------------------------------------                                                                                                                                             --------------------
    [--] Performance schema is disabled.
    [--] Memory used by P_S: 0B
    [--] Sys schema isn't installed.
    
    -------- ThreadPool Metrics ----------------------------------------------------                                                                                                                                             --------------------
    [--] ThreadPool stat is enabled.
    [--] Thread Pool Size: 2 thread(s).
    [--] Using default value is good enough for your version (10.3.18-MariaDB-1:10.3                                                                                                                                             .18+maria~bionic)
    
    -------- MyISAM Metrics --------------------------------------------------------                                                                                                                                             --------------------
    [!!] Key buffer used: 18.3% (3M used / 16M cache)
    [OK] Key buffer size / total MyISAM indexes: 16.0M/124.0K
    
    -------- InnoDB Metrics --------------------------------------------------------                                                                                                                                             --------------------
    [--] InnoDB is enabled.
    [--] InnoDB Thread Concurrency: 0
    [OK] InnoDB File per table is activated
    [!!] InnoDB buffer pool / data size: 128.0M/787.3M
    [!!] Ratio InnoDB log file size / InnoDB Buffer pool size (75 %): 48.0M * 2/128.                                                                                                                                             0M should be equal to 25%
    [OK] InnoDB buffer pool instances: 1
    [--] Number of InnoDB Buffer Pool Chunk : 1 for 1 Buffer Pool Instance(s)
    [OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb                                                                                                                                             _buffer_pool_instances
    [OK] InnoDB Read buffer efficiency: 99.07% (4439220154 hits/ 4481098340 total)
    [!!] InnoDB Write Log efficiency: 1.65% (212 hits/ 12883 total)
    [OK] InnoDB log waits: 0.00% (0 waits / 13095 writes)
    
    -------- AriaDB Metrics --------------------------------------------------------                                                                                                                                             --------------------
    [--] AriaDB is enabled.
    [OK] Aria pagecache size / total Aria indexes: 128.0M/1B
    [!!] Aria pagecache hit rate: 77.5% (37M cached / 8M reads)
    
    -------- TokuDB Metrics --------------------------------------------------------                                                                                                                                             --------------------
    [--] TokuDB is disabled.
    
    -------- XtraDB Metrics --------------------------------------------------------                                                                                                                                             --------------------
    [--] XtraDB is disabled.
    
    -------- Galera Metrics --------------------------------------------------------                                                                                                                                             --------------------
    [--] Galera is disabled.
    
    -------- Replication Metrics ---------------------------------------------------                                                                                                                                             --------------------
    [--] Galera Synchronous replication: NO
    [--] No replication slave(s) for this server.
    [--] Binlog format: MIXED
    [--] XA support enabled: ON
    [--] Semi synchronous replication Master: OFF
    [--] Semi synchronous replication Slave: OFF
    [--] This is a standalone server
    
    -------- Recommendations -------------------------------------------------------                                                                                                                                             --------------------
    General recommendations:
        Control warning line(s) into /var/log/mysql/error.log file
        Set up a Secure Password for debian-sys-maint@localhost user: SET PASSWORD F                                                                                                                                             OR 'debian-sys-maint'@'SpecificDNSorIp' = PASSWORD('secure_password');
        MySQL was started within the last 24 hours - recommendations may be inaccura                                                                                                                                             te
        Reduce your overall MySQL memory footprint for system stability
        Dedicate this server to your database for highest performance.
        Configure your accounts with ip or subnets only, then update your configurat                                                                                                                                             ion with skip-name-resolve=1
        We will suggest raising the 'join_buffer_size' until JOINs not using indexes                                                                                                                                              are found.
                 See https://dev.mysql.com/doc/internals/en/join-buffer-size.html
                 (specially the conclusions at the bottom of the page).
        When making adjustments, make tmp_table_size/max_heap_table_size equal
        Reduce your SELECT DISTINCT queries which have no LIMIT clause
        Performance schema should be activated for better diagnostics
        Consider installing Sys schema from https://github.com/mysql/mysql-sys for M                                                                                                                                             ySQL
        Consider installing Sys schema from https://github.com/FromDual/mariadb-sys                                                                                                                                              for MariaDB
        Before changing innodb_log_file_size and/or innodb_log_files_in_group read t                                                                                                                                             his: https://bit.ly/2TcGgtU
    Variables to adjust:
      *** MySQL's maximum memory usage is dangerously high ***
      *** Add RAM before increasing MySQL buffer variables ***
        query_cache_size (=0)
        query_cache_type (=0)
        query_cache_limit (> 1M, or use smaller result sets)
        query_cache_size (> 16M)
        join_buffer_size (> 256.0K, or always use indexes with JOINs)
        tmp_table_size (> 16M)
        max_heap_table_size (> 16M)
        performance_schema = ON enable PFS
        innodb_buffer_pool_size (>= 787.3M) if possible.
        innodb_log_file_size should be (=16M) if possible, so InnoDB total log files                                                                                                                                              size equals to 25% of buffer pool size.
    
    edited by MattIPv4
Submit an Answer