docker sshfs plugin issue

September 22, 2019 1.9k views
Docker

Hello. I have 2 servers, running on Centos 7.7 and docker 19.03 CE. I tried to use sshfs volumes, but something goes wrong.

I successfully creates volume with this command:[lanadmin@mordred ~]$ docker volume create --driver vieux/sshfs -o sshcmd=lanadmin@10.220.127.247:/home/test -o password=mypass sshvolume

But when i tries to use this volume in container i have an error like this:

[lanadmin@mordred ~]$ docker run -it -v sshvolume:/etc busybox ls /etc
docker: Error response from daemon: failed to copy file info for /var/lib/docker/plugins/518a1874166cf493dd0700d9efec1c3e2d7649cb4a560d44a2af1a6aa16f0526/propagated-mount/5cd59d97ef5e801349f7c06a8f2c113d: failed to chown /var/lib/docker/plugins/518a1874166cf493dd0700d9efec1c3e2d7649cb4a560d44a2af1a6aa16f0526/propagated-mount/5cd59d97ef5e801349f7c06a8f2c113d: lchown /var/lib/docker/plugins/518a1874166cf493dd0700d9efec1c3e2d7649cb4a560d44a2af1a6aa16f0526/propagated-mount/5cd59d97ef5e801349f7c06a8f2c113d: permission denied. See 'docker run --help'.

Also if i tries to use empty directory everything work fine.

And the most funny thing is when i put some file in /home/test there is no error with coping files when container runs and i have this files in /home/test:

lanadmin@ganimed ~]$ ls -lah /home/test/ 
итого 8,0K 
drwxrwxrwx. 2 lanadmin lanadmin 4,0K сен 21 23:15 . 
drwxr-xr-x. 5 root root 4,0K сен 21 18:48 .. 
-rwxr-xr-x. 1 lanadmin lanadmin 0 сен 21 23:15 hostname 
-rwxr-xr-x. 1 lanadmin lanadmin 0 сен 21 23:15 hosts 
-rw-rw-r--. 1 lanadmin lanadmin 0 сен 21 23:14 notest.file 
-rwxr-xr-x. 1 lanadmin lanadmin 0 сен 21 23:15 resolv.conf

Can anyone help me with that?

1 Answer

Hello,

It looks like a permissions issue, have you tried mounting a different folder besides /etc like /tmp for example?

I’ve tried recreating the setup at my end and it worked as expected with when using root:

[root@docker-test ~]# docker run -it -v sshvolume_test2:/etc busybox ls /etc
group        hosts        mtab         passwd       shadow
hostname     localtime    network      resolv.conf
[root@docker-question ~]# 

Regards,
Bobby

  • Yes. I’ve tried to do this things using root, and after your advice I’ve tried to mount /tmp directory. Nothing changed. Also I’ve tried to use nginx image with mounting html directory, and it doesn’t work.

    • Hello,

      What was the exact docker plugin install command that you use? Did you use the one with the ssh key?

      Can you also provide me with the output of:

      docker plugin inspect vieux/sshfs
      

      One last thing, have you tried enabling debug and see if you get a more descriptive error?

      Regards,
      Bobby

      • I use several methods, with ssh keys, without them, with Debug=1 and without it. Unfortunately debug=1 doesn’t help, because i can’t find any additional messages from stdout or with journalctl. Maybe I’m trying to find them in wrong place?
        To be sure about there aren’t my mistakes here, i repeate all steps again:

        Server 1.

        [lanadmin@mordred ~]$ mkdir /home/lanadmin/test
        [lanadmin@mordred ~]$ ls -lah /home/lanadmin/test/
        итого 8,0K
        drwxrwxrwt. 2 lanadmin lanadmin 4,0K сен 24 10:27 .
        drwx------. 8 lanadmin lanadmin 4,0K сен 24 10:27 ..
        [lanadmin@mordred ~]$ 
        
        

        Server 2.

        [lanadmin@redqueen ~]$ docker plugin install --grant-all-permissions vieux/sshfs DEBUG=1 sshkey.source=/home/lanadmin/.ssh/keys/                                                                            
        latest: Pulling from vieux/sshfs                                                                      
        52d435ada6a4: Download complete 
        Digest: sha256:1d3c3e42c12138da5ef7873b97f7f32cf99fb6edde75fa4f0bcf9ed277855811                       
        Status: Downloaded newer image for vieux/sshfs:latest                                                 
        Installed plugin vieux/sshfs       
        
        [lanadmin@redqueen ~]$ docker volume create -d vieux/sshfs -o sshcmd=lanadmin@10.220.127.246:/home/lanadmin/test -o IdentityFile=/root/.ssh/ sshvolume
        sshvolume            
        
        [lanadmin@redqueen ~]$ docker run -it -v sshvolume:/tmp busybox ls /tmp                               
        docker: Error response from daemon: failed to copy file info for /var/lib/docker/plugins/bad35f56f4cd5b3286e0457f27076bf1eb84387570a7750a065e2794fd5b8c55/propagated-mount/87a1a256c177fd471c935c699baf6673: failed to chown /var/lib/docker/plugins/bad35f56f4cd5b3286e0457f27076bf1eb84387570a7750a065e2794fd5b8c55/propagated-mount/87a1a256c177fd471c935c699baf6673: lchown /var/lib/docker/plugins/bad35f56f4cd5b3286e0457f27076bf1eb84387570a7750a065e2794fd5b8c55/propagated-mount/87a1a256c177fd471c935c699baf6673: permission denied.
        See 'docker run --help'.  
        
        [lanadmin@redqueen ~]$ systemctl status docker
        ● docker.service - Docker Application Container Engine                                                                                                                                                         
           Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)                                                                                                                   
           Active: active (running) since Вс 2019-09-22 11:04:19 MSK; 1 day 23h ago                                                                                                                                    
             Docs: https://docs.docker.com                                                                                                                                                                             
         Main PID: 1059 (dockerd)                                                                                                                                                                                      
            Tasks: 11                                                                                                                                                                                                  
           Memory: 162.1M                                                                                                                                                                                              
           CGroup: /system.slice/docker.service                                                                                                                                                                        
                   └─1059 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock                                                                                                                       
        
        сен 24 10:29:00 redqueen.mts dockerd[1059]: time="2019-09-24T10:29:00+03:00" level=error msg="2019/09/24 07:29:00 Entering go-plugins-helpers capabilitiesPath" plugin=bad35f56f4cd5b3286e04...e2794fd5b8c55   
        сен 24 10:29:00 redqueen.mts dockerd[1059]: time="2019-09-24T10:29:00+03:00" level=error msg="time=\"2019-09-24T07:29:00Z\" level=debug method=capabilities" plugin=bad35f56f4cd5b3286e0457f...e2794fd5b8c55   
        сен 24 10:29:00 redqueen.mts dockerd[1059]: time="2019-09-24T10:29:00+03:00" level=error msg="2019/09/24 07:29:00 Entering go-plugins-helpers getPath" plugin=bad35f56f4cd5b3286e0457f27076b...e2794fd5b8c55   
        сен 24 10:29:00 redqueen.mts dockerd[1059]: time="2019-09-24T10:29:00+03:00" level=error msg="time=\"2019-09-24T07:29:00Z\" level=debug msg=\"&volume.GetRequest{Name:\\\"sshvolume\\\"}\" m...e2794fd5b8c55   
        сен 24 10:29:00 redqueen.mts dockerd[1059]: time="2019-09-24T10:29:00+03:00" level=error msg="2019/09/24 07:29:00 Entering go-plugins-helpers capabilitiesPath" plugin=bad35f56f4cd5b3286e04...e2794fd5b8c55   
        сен 24 10:29:00 redqueen.mts dockerd[1059]: time="2019-09-24T10:29:00+03:00" level=error msg="time=\"2019-09-24T07:29:00Z\" level=debug method=capabilities" plugin=bad35f56f4cd5b3286e0457f...e2794fd5b8c55   
        сен 24 10:29:00 redqueen.mts dockerd[1059]: time="2019-09-24T10:29:00+03:00" level=error msg="2019/09/24 07:29:00 Entering go-plugins-helpers unmountPath" plugin=bad35f56f4cd5b3286e0457f27...e2794fd5b8c55   
        сен 24 10:29:00 redqueen.mts dockerd[1059]: time="2019-09-24T10:29:00+03:00" level=error msg="time=\"2019-09-24T07:29:00Z\" level=debug msg=\"&volume.UnmountRequest{Name:\\\"sshvolume\\\", ID:\\\"f6585...   
        сен 24 10:29:00 redqueen.mts dockerd[1059]: time="2019-09-24T10:29:00+03:00" level=error msg="time=\"2019-09-24T07:29:00Z\" level=debug msg=\"umount /mnt/volumes/87a1a256c177fd471c935c699b...e2794fd5b8c55   
        сен 24 10:29:01 redqueen.mts dockerd[1059]: time="2019-09-24T10:29:01.390208370+03:00" level=error msg="Handler for POST /v1.40/containers/create returned error: failed to copy file info f...f6673: failed   
        Hint: Some lines were ellipsized, use -l to show in full.                                                                                                                                                      
        [lanadmin@redqueen ~]$ journalctl -u docker -r -b
        
        

        And here is result for docker plugin inspect vieux/sshfs:

        
        [lanadmin@redqueen ~]$ docker plugin inspect vieux/sshfs:latest 
        [
            {
                "Config": {
                    "Args": {
                        "Description": "",
                        "Name": "",
                        "Settable": null,
                        "Value": null
                    },
                    "Description": "sshFS plugin for Docker",
                    "DockerVersion": "18.05.0-ce-rc1",
                    "Documentation": "https://docs.docker.com/engine/extend/plugins/",
                    "Entrypoint": [
                        "/docker-volume-sshfs"
                    ],
                    "Env": [
                        {
                            "Description": "",
                            "Name": "DEBUG",
                            "Settable": [
                                "value"
                            ],
                            "Value": "0"
                        }
                    ],
                    "Interface": {
                        "Socket": "sshfs.sock",
                        "Types": [
                            "docker.volumedriver/1.0"
                        ]
                    },
                    "IpcHost": false,
                    "Linux": {
                        "AllowAllDevices": false,
                        "Capabilities": [
                            "CAP_SYS_ADMIN"
                        ],
                        "Devices": [
                            {
                                "Description": "",
                                "Name": "",
                                "Path": "/dev/fuse",
                                "Settable": null
                            }
                        ]
                    },
                    "Mounts": [
                        {
                            "Description": "",
                            "Destination": "/mnt/state",
                            "Name": "state",
                            "Options": [
                                "rbind"
                            ],
                            "Settable": [
                                "source"
                            ],
                            "Source": "/var/lib/docker/plugins/",
                            "Type": "bind"
                        },
                        {
                            "Description": "",
                            "Destination": "/root/.ssh",
                            "Name": "sshkey",
                            "Options": [
                                "rbind"
                            ],
                            "Settable": [
                                "source"
                            ],
                            "Source": "/home/lanadmin/.ssh/keys/",
                            "Type": "bind"
                        }
                    ],
                    "Network": {
                        "Type": "host"
                    },
                    "PidHost": false,
                    "PropagatedMount": "/mnt/volumes",
                    "User": {},
                    "WorkDir": "",
                    "rootfs": {
                        "diff_ids": [
                            "sha256:ce2b7a99c5db05cfe263bcd3640f2c1ce7c6f4619339633d44e65a8168ec3587"
                        ],
                        "type": "layers"
                    }
                },
                "Enabled": true,
                "Id": "bad35f56f4cd5b3286e0457f27076bf1eb84387570a7750a065e2794fd5b8c55",
                "Name": "vieux/sshfs:latest",
                "PluginReference": "docker.io/vieux/sshfs:latest",
                "Settings": {
                    "Args": [],
                    "Devices": [
                        {
                            "Description": "",
                            "Name": "",
                            "Path": "/dev/fuse",
                            "Settable": null
                        }
                    ],
                    "Env": [
                        "DEBUG=1"
                    ],
                    "Mounts": [
                        {
                            "Description": "",
                            "Destination": "/mnt/state",
                            "Name": "state",
                            "Options": [
                                "rbind"
                            ],
                            "Settable": [
                                "source"
                            ],
                            "Source": "/var/lib/docker/plugins/",
                            "Type": "bind"
                        },
                        {
                            "Description": "",
                            "Destination": "/root/.ssh",
                            "Name": "sshkey",
                            "Options": [
                                "rbind"
                            ],
                            "Settable": [
                                "source"
                            ],
                            "Source": "/home/lanadmin/.ssh/keys/",
                            "Type": "bind"
                        }
                    ]
                }
            }
        ]
        [lanadmin@redqueen ~]$ 
        
        • Hi @kvp94best,

          This is quite interesting, the process looks correct. I have a couple of questions:

          • Do you have any other docker plugins that might be causing the issue due to some conflicts?

          • Do you have sshfs installed on the system?

          • By default your keys should be stored at /home/lanadmin/.ssh/ and not /home/lanadmin/.ssh/keys/, is this a custom directory that you keep your keys in?

          Regards,
          Bobby

            1. No. That was my first plugin I’ve installed.
            2. First times i tried, there wasn’t sshfs installed, after that issue, i have thought like you too, and installed sshfs on host.
            3. Yes. It’s a custom directory because i have several keys, and it’s more comfortable for me to keep them in keys directory

            P.S. Have you noticed that when you have a file in test directory, plugin works fine?

Have another answer? Share your knowledge.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!