Docker swarm persistent volume with rexray/dobs and mariadb fails

Hey!

The issue you’re facing with mounting a Docker-created volume using rexray/dobs specifically for the Bitnami/MariaDB image in a Docker Swarm environment may be related to the nature of how block storage volumes interact with Docker Swarm.

Since DigitalOcean Block Storage volumes are designed to be attached to a single Droplet at a time, attempting to mount the same volume on multiple nodes in a Swarm could lead to the error you’re experiencing.

Here’s a structured approach to diagnose and potentially resolve this issue:

1. Docker Swarm might attempt to deploy the MariaDB service on a different node where the volume isn’t mounted, causing the mount to fail. Verify if the MariaDB service is trying to start on multiple nodes and whether the volume is accessible from those nodes.

2. Ensure that the volume mysql-data is accessible from the node where the MariaDB container is trying to start. If the volume is attached to a different Droplet, it won’t be accessible from the MariaDB container’s host.

3. To mitigate the issue, you can use placement constraints to ensure that the MariaDB service always starts on the same node where the mysql-data volume is mounted. Modify your Docker stack file to include a constraint under the service definition:

   services:
     mariadb:
       image: bitnami/mariadb:latest
       volumes:
         - mysql-data:/bitnami/mariadb
       deploy:
         placement:
           constraints:
             - node.hostname == target-node-hostname
   

   Replace target-node-hostname with the hostname of the node where the volume is accessible.

4. Review the volume configuration in your Docker Compose or stack file. Ensure that the volume is correctly referenced and matches the name you’ve created (mysql-data).

5. Look at the REX-Ray logs for more detailed error messages that might give you additional insights into what’s going wrong.

6. If the application architecture allows, consider using a replicated database setup where each node in the Swarm has its own MariaDB instance and volume, thus avoiding shared volume issues. This setup would require database replication to keep the instances in sync.

What you can also try is to test the volume mount operation on the node outside the Swarm context to ensure that the volume can be mounted successfully. Also try deploying a simple service using the same volume to check if the issue is specific to the MariaDB image.

Best,

Bobby