Docker swarm persistent volume with rexray/dobs and mariadb fails

Question

I am unable to mount a docker created volume with rexray/dobs into my mariadb container. It only happens with the optnami/mariadb image. i created the docker volume with:

docker volume create --name=mysql-data --opt=size=1 --driver=rexray/dobs

i do not assign the volume to a droplet via do dashboard or run the init configuration. i added the rexray/dobs plugin with:

docker plugin install rexray/dobs
DOBS_REGION=fra1
DOBS_STATUSINITIALDELAY=1000ms
DOBS_STATUSMAXATTEMPTS=100
DOBS_STATUSTIMEOUT=5m
DOBS_TOKEN=xxxx

i followed: https://github.com/rexray/rexray/issues/1321 https://medium.com/@adrian.gheorghe.dev/docker-swarm-volume-data-persistence-on-digital-ocean-with-rexray-cd418f718131

When i docker stack deploy my services i get:

“VolumeDriver.Mount: docker-legacy: Mount: mysql-data: failed: Error while waiting for storage action to finish”

and the service fails and restarts. The strange thing is that it happens only with the mariadb images - i have other images where it works

Bobby · Answer

Hey! The issue you’re facing with mounting a Docker-created volume using rexray/dobs specifically for the Bitnami/MariaDB image in a Docker Swarm environment may be related to the nature of how block storage volumes interact with Docker Swarm. Since DigitalOcean Block Storage volumes are designed to be attached to a single Droplet at a time, attempting to mount the same volume on multiple nodes in a Swarm could lead to the error you’re experiencing. Here’s a structured approach to diagnose and potentially resolve this issue: Docker Swarm might attempt to deploy the MariaDB service on a different node where the volume isn’t mounted, causing the mount to fail. Verify if the MariaDB service is trying to start on multiple nodes and whether the volume is accessible from those nodes. Ensure that the volume mysql-data is accessible from the node where the MariaDB container is trying to start. If the volume is attached to a different Droplet, it won’t be accessible from the MariaDB container’s host. To mitigate the issue, you can use placement constraints to ensure that the MariaDB service always starts on the same node where the mysql-data volume is mounted. Modify your Docker stack file to include a constraint under the service definition: services: mariadb: image: bitnami/mariadb:latest volumes: - mysql-data:/bitnami/mariadb deploy: placement: constraints: - node.hostname == target-node-hostname Replace target-node-hostname with the hostname of the node where the volume is accessible. Review the volume configuration in your Docker Compose or stack file. Ensure that the volume is correctly referenced and matches the name you’ve created (mysql-data). Look at the REX-Ray logs for more detailed error messages that might give you additional insights into what’s going wrong. If the application architecture allows, consider using a replicated database setup where each node in the Swarm has its own MariaDB instance and volume, thus avoiding shared volume issues. This setup would require database replication to keep the instances in sync. What you can also try is to test the volume mount operation on the node outside the Swarm context to ensure that the volume can be mounted successfully. Also try deploying a simple service using the same volume to check if the issue is specific to the MariaDB image. Best, Bobby

Become a contributor for community

DigitalOcean Documentation

Resources for startups and AI-native businesses

The developer cloud

Start building today

Report this

Docker swarm persistent volume with rexray/dobs and mariadb fails

Report this

Docker swarm persistent volume with rexray/dobs and mariadb fails