Hello. I have 2 servers, running on Centos 7.7 and docker 19.03 CE. I tried to use sshfs volumes, but something goes wrong.

I successfully creates volume with this command:[lanadmin@mordred ~]$ docker volume create --driver vieux/sshfs -o sshcmd=lanadmin@10.220.127.247:/home/test -o password=mypass sshvolume

But when i tries to use this volume in container i have an error like this:

[lanadmin@mordred ~]$ docker run -it -v sshvolume:/etc busybox ls /etc
docker: Error response from daemon: failed to copy file info for /var/lib/docker/plugins/518a1874166cf493dd0700d9efec1c3e2d7649cb4a560d44a2af1a6aa16f0526/propagated-mount/5cd59d97ef5e801349f7c06a8f2c113d: failed to chown /var/lib/docker/plugins/518a1874166cf493dd0700d9efec1c3e2d7649cb4a560d44a2af1a6aa16f0526/propagated-mount/5cd59d97ef5e801349f7c06a8f2c113d: lchown /var/lib/docker/plugins/518a1874166cf493dd0700d9efec1c3e2d7649cb4a560d44a2af1a6aa16f0526/propagated-mount/5cd59d97ef5e801349f7c06a8f2c113d: permission denied. See 'docker run --help'.

Also if i tries to use empty directory everything work fine.

And the most funny thing is when i put some file in /home/test there is no error with coping files when container runs and i have this files in /home/test:

lanadmin@ganimed ~]$ ls -lah /home/test/ 
итого 8,0K 
drwxrwxrwx. 2 lanadmin lanadmin 4,0K сен 21 23:15 . 
drwxr-xr-x. 5 root root 4,0K сен 21 18:48 .. 
-rwxr-xr-x. 1 lanadmin lanadmin 0 сен 21 23:15 hostname 
-rwxr-xr-x. 1 lanadmin lanadmin 0 сен 21 23:15 hosts 
-rw-rw-r--. 1 lanadmin lanadmin 0 сен 21 23:14 notest.file 
-rwxr-xr-x. 1 lanadmin lanadmin 0 сен 21 23:15 resolv.conf

Can anyone help me with that?

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
1 answer

Hello,

It looks like a permissions issue, have you tried mounting a different folder besides /etc like /tmp for example?

I’ve tried recreating the setup at my end and it worked as expected with when using root:

[root@docker-test ~]# docker run -it -v sshvolume_test2:/etc busybox ls /etc
group        hosts        mtab         passwd       shadow
hostname     localtime    network      resolv.conf
[root@docker-question ~]# 

Regards,
Bobby

  • Yes. I’ve tried to do this things using root, and after your advice I’ve tried to mount /tmp directory. Nothing changed. Also I’ve tried to use nginx image with mounting html directory, and it doesn’t work.

    • Hello,

      What was the exact docker plugin install command that you use? Did you use the one with the ssh key?

      Can you also provide me with the output of:

      docker plugin inspect vieux/sshfs
      

      One last thing, have you tried enabling debug and see if you get a more descriptive error?

      Regards,
      Bobby

      • I use several methods, with ssh keys, without them, with Debug=1 and without it. Unfortunately debug=1 doesn’t help, because i can’t find any additional messages from stdout or with journalctl. Maybe I’m trying to find them in wrong place?
        To be sure about there aren’t my mistakes here, i repeate all steps again:

        Server 1.

        [lanadmin@mordred ~]$ mkdir /home/lanadmin/test
        [lanadmin@mordred ~]$ ls -lah /home/lanadmin/test/
        итого 8,0K
        drwxrwxrwt. 2 lanadmin lanadmin 4,0K сен 24 10:27 .
        drwx------. 8 lanadmin lanadmin 4,0K сен 24 10:27 ..
        [lanadmin@mordred ~]$ 
        
        

        Server 2.

        [lanadmin@redqueen ~]$ docker plugin install --grant-all-permissions vieux/sshfs DEBUG=1 sshkey.source=/home/lanadmin/.ssh/keys/                                                                            
        latest: Pulling from vieux/sshfs                                                                      
        52d435ada6a4: Download complete 
        Digest: sha256:1d3c3e42c12138da5ef7873b97f7f32cf99fb6edde75fa4f0bcf9ed277855811                       
        Status: Downloaded newer image for vieux/sshfs:latest                                                 
        Installed plugin vieux/sshfs       
        
        [lanadmin@redqueen ~]$ docker volume create -d vieux/sshfs -o sshcmd=lanadmin@10.220.127.246:/home/lanadmin/test -o IdentityFile=/root/.ssh/ sshvolume
        sshvolume            
        
        [lanadmin@redqueen ~]$ docker run -it -v sshvolume:/tmp busybox ls /tmp                               
        docker: Error response from daemon: failed to copy file info for /var/lib/docker/plugins/bad35f56f4cd5b3286e0457f27076bf1eb84387570a7750a065e2794fd5b8c55/propagated-mount/87a1a256c177fd471c935c699baf6673: failed to chown /var/lib/docker/plugins/bad35f56f4cd5b3286e0457f27076bf1eb84387570a7750a065e2794fd5b8c55/propagated-mount/87a1a256c177fd471c935c699baf6673: lchown /var/lib/docker/plugins/bad35f56f4cd5b3286e0457f27076bf1eb84387570a7750a065e2794fd5b8c55/propagated-mount/87a1a256c177fd471c935c699baf6673: permission denied.
        See 'docker run --help'.  
        
        [lanadmin@redqueen ~]$ systemctl status docker
        ● docker.service - Docker Application Container Engine                                                                                                                                                         
           Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)                                                                                                                   
           Active: active (running) since Вс 2019-09-22 11:04:19 MSK; 1 day 23h ago                                                                                                                                    
             Docs: https://docs.docker.com                                                                                                                                                                             
         Main PID: 1059 (dockerd)                                                                                                                                                                                      
            Tasks: 11                                                                                                                                                                                                  
           Memory: 162.1M                                                                                                                                                                                              
           CGroup: /system.slice/docker.service                                                                                                                                                                        
                   └─1059 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock                                                                                                                       
        
        сен 24 10:29:00 redqueen.mts dockerd[1059]: time="2019-09-24T10:29:00+03:00" level=error msg="2019/09/24 07:29:00 Entering go-plugins-helpers capabilitiesPath" plugin=bad35f56f4cd5b3286e04...e2794fd5b8c55   
        сен 24 10:29:00 redqueen.mts dockerd[1059]: time="2019-09-24T10:29:00+03:00" level=error msg="time=\"2019-09-24T07:29:00Z\" level=debug method=capabilities" plugin=bad35f56f4cd5b3286e0457f...e2794fd5b8c55   
        сен 24 10:29:00 redqueen.mts dockerd[1059]: time="2019-09-24T10:29:00+03:00" level=error msg="2019/09/24 07:29:00 Entering go-plugins-helpers getPath" plugin=bad35f56f4cd5b3286e0457f27076b...e2794fd5b8c55   
        сен 24 10:29:00 redqueen.mts dockerd[1059]: time="2019-09-24T10:29:00+03:00" level=error msg="time=\"2019-09-24T07:29:00Z\" level=debug msg=\"&volume.GetRequest{Name:\\\"sshvolume\\\"}\" m...e2794fd5b8c55   
        сен 24 10:29:00 redqueen.mts dockerd[1059]: time="2019-09-24T10:29:00+03:00" level=error msg="2019/09/24 07:29:00 Entering go-plugins-helpers capabilitiesPath" plugin=bad35f56f4cd5b3286e04...e2794fd5b8c55   
        сен 24 10:29:00 redqueen.mts dockerd[1059]: time="2019-09-24T10:29:00+03:00" level=error msg="time=\"2019-09-24T07:29:00Z\" level=debug method=capabilities" plugin=bad35f56f4cd5b3286e0457f...e2794fd5b8c55   
        сен 24 10:29:00 redqueen.mts dockerd[1059]: time="2019-09-24T10:29:00+03:00" level=error msg="2019/09/24 07:29:00 Entering go-plugins-helpers unmountPath" plugin=bad35f56f4cd5b3286e0457f27...e2794fd5b8c55   
        сен 24 10:29:00 redqueen.mts dockerd[1059]: time="2019-09-24T10:29:00+03:00" level=error msg="time=\"2019-09-24T07:29:00Z\" level=debug msg=\"&volume.UnmountRequest{Name:\\\"sshvolume\\\", ID:\\\"f6585...   
        сен 24 10:29:00 redqueen.mts dockerd[1059]: time="2019-09-24T10:29:00+03:00" level=error msg="time=\"2019-09-24T07:29:00Z\" level=debug msg=\"umount /mnt/volumes/87a1a256c177fd471c935c699b...e2794fd5b8c55   
        сен 24 10:29:01 redqueen.mts dockerd[1059]: time="2019-09-24T10:29:01.390208370+03:00" level=error msg="Handler for POST /v1.40/containers/create returned error: failed to copy file info f...f6673: failed   
        Hint: Some lines were ellipsized, use -l to show in full.                                                                                                                                                      
        [lanadmin@redqueen ~]$ journalctl -u docker -r -b
        
        

        And here is result for docker plugin inspect vieux/sshfs:

        
        [lanadmin@redqueen ~]$ docker plugin inspect vieux/sshfs:latest 
        [
            {
                "Config": {
                    "Args": {
                        "Description": "",
                        "Name": "",
                        "Settable": null,
                        "Value": null
                    },
                    "Description": "sshFS plugin for Docker",
                    "DockerVersion": "18.05.0-ce-rc1",
                    "Documentation": "https://docs.docker.com/engine/extend/plugins/",
                    "Entrypoint": [
                        "/docker-volume-sshfs"
                    ],
                    "Env": [
                        {
                            "Description": "",
                            "Name": "DEBUG",
                            "Settable": [
                                "value"
                            ],
                            "Value": "0"
                        }
                    ],
                    "Interface": {
                        "Socket": "sshfs.sock",
                        "Types": [
                            "docker.volumedriver/1.0"
                        ]
                    },
                    "IpcHost": false,
                    "Linux": {
                        "AllowAllDevices": false,
                        "Capabilities": [
                            "CAP_SYS_ADMIN"
                        ],
                        "Devices": [
                            {
                                "Description": "",
                                "Name": "",
                                "Path": "/dev/fuse",
                                "Settable": null
                            }
                        ]
                    },
                    "Mounts": [
                        {
                            "Description": "",
                            "Destination": "/mnt/state",
                            "Name": "state",
                            "Options": [
                                "rbind"
                            ],
                            "Settable": [
                                "source"
                            ],
                            "Source": "/var/lib/docker/plugins/",
                            "Type": "bind"
                        },
                        {
                            "Description": "",
                            "Destination": "/root/.ssh",
                            "Name": "sshkey",
                            "Options": [
                                "rbind"
                            ],
                            "Settable": [
                                "source"
                            ],
                            "Source": "/home/lanadmin/.ssh/keys/",
                            "Type": "bind"
                        }
                    ],
                    "Network": {
                        "Type": "host"
                    },
                    "PidHost": false,
                    "PropagatedMount": "/mnt/volumes",
                    "User": {},
                    "WorkDir": "",
                    "rootfs": {
                        "diff_ids": [
                            "sha256:ce2b7a99c5db05cfe263bcd3640f2c1ce7c6f4619339633d44e65a8168ec3587"
                        ],
                        "type": "layers"
                    }
                },
                "Enabled": true,
                "Id": "bad35f56f4cd5b3286e0457f27076bf1eb84387570a7750a065e2794fd5b8c55",
                "Name": "vieux/sshfs:latest",
                "PluginReference": "docker.io/vieux/sshfs:latest",
                "Settings": {
                    "Args": [],
                    "Devices": [
                        {
                            "Description": "",
                            "Name": "",
                            "Path": "/dev/fuse",
                            "Settable": null
                        }
                    ],
                    "Env": [
                        "DEBUG=1"
                    ],
                    "Mounts": [
                        {
                            "Description": "",
                            "Destination": "/mnt/state",
                            "Name": "state",
                            "Options": [
                                "rbind"
                            ],
                            "Settable": [
                                "source"
                            ],
                            "Source": "/var/lib/docker/plugins/",
                            "Type": "bind"
                        },
                        {
                            "Description": "",
                            "Destination": "/root/.ssh",
                            "Name": "sshkey",
                            "Options": [
                                "rbind"
                            ],
                            "Settable": [
                                "source"
                            ],
                            "Source": "/home/lanadmin/.ssh/keys/",
                            "Type": "bind"
                        }
                    ]
                }
            }
        ]
        [lanadmin@redqueen ~]$ 
        
        • Hi @kvp94best,

          This is quite interesting, the process looks correct. I have a couple of questions:

          • Do you have any other docker plugins that might be causing the issue due to some conflicts?

          • Do you have sshfs installed on the system?

          • By default your keys should be stored at /home/lanadmin/.ssh/ and not /home/lanadmin/.ssh/keys/, is this a custom directory that you keep your keys in?

          Regards,
          Bobby

            1. No. That was my first plugin I’ve installed.
            2. First times i tried, there wasn’t sshfs installed, after that issue, i have thought like you too, and installed sshfs on host.
            3. Yes. It’s a custom directory because i have several keys, and it’s more comfortable for me to keep them in keys directory

            P.S. Have you noticed that when you have a file in test directory, plugin works fine?

Submit an Answer