Share

Question

I have been experimenting with the early access of Kubernetes. In GKE, for example, it’s easy to setup a standard VM with an NFS server configured, and then install the Helm chart for nfs-client-provisioner so you can have shared storage among pods (for things like redundant Django apps running in different pods). I tried this with DO and it did not work, even when I monkeyed with the firewall settings for the k8s nodes.

I am unaware of another inexpensive shared storage option, the standard PVCs offered by DO don’t permit ReadWriteMany. Does anyone have a better solution?

Answer 1

cro December 14, 2018

Accepted Answer

Digital Ocean support comes through again–they let me know that they only just started to deploy the NFS tools on worker nodes and if I deployed a cluster with k8s version 1.11.3-do.1 it should work. I tried again today and nfs-server-provisioner works now, as well as rook.io with CephFS.

Quick bonnie++ test, two different clusters, same worker node configuration:

CephFS via Rook:

Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
magical-ardinghe 4G   547  89 37557   4 24877   3   946  93 88786   5  3006  44
Latency               266ms    2444ms    3881ms   25122us     257ms     295ms
Version  1.97       ------Sequential Create------ --------Random Create--------
magical-ardinghelli -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16   727   1 +++++ +++   481   1   794   2  1881   2   456   1
Latency              1128ms   23441us     672ms    1445ms   10984us     173ms
1.97,1.97,magical-ardinghelli-3l9v,1,1544812020,4G,,547,89,37557,4,24877,3,946,93,88786,5,3006,44,16,,,,,727,1,+++++
,+++,481,1,794,2,1881,2,456,1,266ms,2444ms,3881ms,25122us,257ms,295ms,1128ms,23441us,672ms,1445ms,10984us,173ms

For NFS:

Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
nfs-busybox-dfdb 4G  1000  95 31489   2 135902  13  1287  93 229139  11  2707  33
Latency             19663us   36927ms     182ms     117ms   80444us   27791us
Version  1.97       ------Sequential Create------ --------Random Create--------
nfs-busybox-dfdb5c6 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  1157  14 23094  15  1565  20   249   3  4797  22  1559  19
Latency              9404us   29618us   10571us     662ms    4625us   17487us
1.97,1.97,nfs-busybox-dfdb5c685-vckmj,1,1544823234,4G,,1000,95,31489,2,135902,13,1287,93,229139,11,2707,33,16,,,,,1157,14,23094,15,1565,20,249,3,4797,22,1559,19,19663us,36927ms,182ms,117ms,80444us,27791us,9404us,29618us,10571us,662ms,4625us,17487us0

Reply

Answer 2

Trurl December 5, 2018

Looks like workers just don’t have nfs tools to mount NFS. Both nfs-server-provisioner over “do-block-storage” and external nfs server (already used on my old k8s cluster) failed to mount.

Reply

Answer 3

cro December 5, 2018

Good thinking to try nfs-server-provisioner. I suspected that the nfs tools packages were not installed on the nodes. I wish we could ssh into the nodes, could have answered that question much quicker.

Any other ideas for shared storage? I considered rook.io and fired up a cluster just to test that but couldn’t get it working. Also there are big scary warnings on the Ceph site saying that CephFS is not for production use.

Reply