Question

How to restore Mysql Galera Cluster after upgrade Ubuntu 18

Posted May 17, 2021 98 views
MySQL

I followed this guide to start mysql cluster with 2 nodes: Galera Digital Ocean
Everything works fine until I upgrade APT and forget to ignore Galera&Mysql package.

root@prod-1:/home/bar# apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  linux-headers-4.15.0-132 linux-headers-4.15.0-132-generic linux-image-4.15.0-132-generic linux-modules-4.15.0-132-generic linux-modules-extra-4.15.0-132-generic
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
2 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up mysql-wsrep-server-5.7 (5.7.33-25.25-1ubuntu18.04) ...
Skipping profile in /etc/apparmor.d/disable: usr.sbin.mysqld
Job for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xe" for details.
dpkg: error processing package mysql-wsrep-server-5.7 (--configure):
 installed mysql-wsrep-server-5.7 package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of mysql-wsrep-5.7:
 mysql-wsrep-5.7 depends on mysql-wsrep-server-5.7 (= 5.7.33-25.25-1ubuntu18.04); however:
  Package mysql-wsrep-server-5.7 is not configured yet.

dpkg: error processing package mysql-wsrep-5.7 (--configure):
 dependency problems - leaving unconfigured
No apport report written because the error message indicates its a followup error from a previous failure.
                                                                                                          Errors were encountered while processing:
 mysql-wsrep-server-5.7
 mysql-wsrep-5.7
E: Sub-process /usr/bin/dpkg returned an error code (1)
root@prod-1:/home/bar#

It happens in 2 nodes and I cannot start my cluster anymore.
What I have to do to fix this issue.
Thanks

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
Submit an Answer
1 answer

Hi @barwalk,

I’ll suggest checking your MySQL logs or when you try to start your service check the journal:

journalctl -xe

Additionally, the logs should indicate where MySQL is failing and from there you can search how to fix the exact error behind the issue.

I’m uncertain whether it would be easier to fix the error or just create the cluster from zero again.

Regards,
KFSys

  • Mysql log shows:

    2021-05-17T03:03:47.822968Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
    2021-05-17T03:03:47.824624Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.33) starting as process 6643 ...
    2021-05-17T03:03:47.828016Z 0 [Note] WSREP: Read nil XID from storage engines, skipping position init
    2021-05-17T03:03:47.828030Z 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
    2021-05-17T03:03:47.828619Z 0 [Note] WSREP: wsrep_load(): Galera 3.33(rd992932) by Codership Oy <info@codership.com> loaded successfully.
    2021-05-17T03:03:47.828650Z 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
    2021-05-17T03:03:47.828925Z 0 [Note] WSREP: Found saved state: d3047cfc-84a4-11eb-a56d-1a459f7210eb:-1, safe_to_bootstrap: 0
    2021-05-17T03:03:47.829861Z 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 172.27.211.198; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S
    2021-05-17T03:03:47.842076Z 0 [Note] WSREP: GCache history reset: d3047cfc-84a4-11eb-a56d-1a459f7210eb:0 -> d3047cfc-84a4-11eb-a56d-1a459f7210eb:13837
    2021-05-17T03:03:47.850243Z 0 [Note] WSREP: Assign initial position for certification: 13837, protocol version: -1
    2021-05-17T03:03:47.850270Z 0 [Note] WSREP: wsrep_sst_grab()
    2021-05-17T03:03:47.850274Z 0 [Note] WSREP: Start replication
    2021-05-17T03:03:47.850282Z 0 [Note] WSREP: Setting initial position to d3047cfc-84a4-11eb-a56d-1a459f7210eb:13837
    2021-05-17T03:03:47.850336Z 0 [Note] WSREP: protonet asio version 0
    2021-05-17T03:03:47.850448Z 0 [Note] WSREP: Using CRC-32C for message checksums.
    2021-05-17T03:03:47.850468Z 0 [Note] WSREP: backend: asio
    2021-05-17T03:03:47.850540Z 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 
    2021-05-17T03:03:47.850609Z 0 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
    2021-05-17T03:03:47.850614Z 0 [Note] WSREP: restore pc from disk failed
    2021-05-17T03:03:47.850845Z 0 [Note] WSREP: GMCast version 0
    2021-05-17T03:03:47.850988Z 0 [Note] WSREP: (7fc2b71e, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
    2021-05-17T03:03:47.850995Z 0 [Note] WSREP: (7fc2b71e, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
    2021-05-17T03:03:47.851282Z 0 [Note] WSREP: EVS version 0
    2021-05-17T03:03:47.851353Z 0 [Note] WSREP: gcomm: connecting to group 'my_cluster', peer '172.27.211.198:,172.27.211.134:'
    2021-05-17T03:03:47.851847Z 0 [Note] WSREP: (7fc2b71e, 'tcp://0.0.0.0:4567') Found matching local endpoint for a connection, blacklisting address tcp://172.27.211.198:4567
    2021-05-17T03:03:50.853065Z 0 [Warning] WSREP: no nodes coming from prim view, prim not possible
    2021-05-17T03:03:50.853209Z 0 [Note] WSREP: view(view_id(NON_PRIM,7fc2b71e,1) memb {
        7fc2b71e,0
    } joined {
    } left {
    } partitioned {
    })
    2021-05-17T03:03:51.353362Z 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.50205S), skipping check
    

    I think there’s something wrong with the WSREP: Recovered position and I need to start the last shutdown server before any other.