DigitalOcean Kubernetes: new control plane is faster and free, enable HA for 99.95% uptime SLA

Related

Docker, Discourse and Nginx
Question
DigitalOcean for Non-Developers with Good Linux Experience
Question

Question

Why minio servers can't find each other and having troubles with disks

On my local machine everything works fine: docker-compose file

minio1:
    image: minio/minio:RELEASE.2020-12-18T03-27-42Z
    volumes:
      - data1-1:/data1
      - data1-2:/data2
#      - ./docker/nginx/fullchain.pem:/root/.minio/certs/public.crt
#      - ./docker/nginx/privkey.pem:/root/.minio/certs/private.key
    expose:
      - "9000"
    environment:
      MINIO_ACCESS_KEY: KAJKUQ_TLIABIWEMKKANT0EN
      MINIO_SECRET_KEY: pJG-t0rZV5Xo11pd8XAu7IxHQyy0wCMVacW6SiONnjUxr01w
    command: server http://minio{1...4}/data{1...2}
    healthcheck:
      test: [ "CMD", "curl", "-f", "--insecure", "https://localhost:9000/minio/health/live" ]
      interval: 30s
      timeout: 20s
      retries: 3

  minio2:
    image: minio/minio:RELEASE.2020-12-18T03-27-42Z
    volumes:
      - data2-1:/data1
      - data2-2:/data2
#      - ./docker/nginx/fullchain.pem:/root/.minio/certs/public.crt
#      - ./docker/nginx/privkey.pem:/root/.minio/certs/private.key
    expose:
      - "9000"
    environment:
      MINIO_ACCESS_KEY: KAJKUQ_TLIABIWEMKKANT0EN
      MINIO_SECRET_KEY: pJG-t0rZV5Xo11pd8XAu7IxHQyy0wCMVacW6SiONnjUxr01w
    command: server http://minio{1...4}/data{1...2}
    healthcheck:
      test: [ "CMD", "curl", "-f", "--insecure", "https://localhost:9000/minio/health/live" ]
      interval: 30s
      timeout: 20s
      retries: 3

  minio3:
    image: minio/minio:RELEASE.2020-12-18T03-27-42Z
    volumes:
      - data3-1:/data1
      - data3-2:/data2
#      - ./docker/nginx/fullchain.pem:/root/.minio/certs/public.crt
#      - ./docker/nginx/privkey.pem:/root/.minio/certs/private.key
    expose:
      - "9000"
    environment:
      MINIO_ACCESS_KEY: KAJKUQ_TLIABIWEMKKANT0EN
      MINIO_SECRET_KEY: pJG-t0rZV5Xo11pd8XAu7IxHQyy0wCMVacW6SiONnjUxr01w
    command: server http://minio{1...4}/data{1...2}
    healthcheck:
      test: [ "CMD", "curl", "-f", "--insecure", "https://localhost:9000/minio/health/live" ]
      interval: 30s
      timeout: 20s
      retries: 3
  minio4:
    image: minio/minio:RELEASE.2020-12-18T03-27-42Z
    volumes:
      - data4-1:/data1
      - data4-2:/data2
#      - ./docker/nginx/fullchain.pem:/root/.minio/certs/public.crt
#      - ./docker/nginx/privkey.pem:/root/.minio/certs/private.key
    expose:
      - "9000"
    environment:
      MINIO_ACCESS_KEY: KAJKUQ_TLIABIWEMKKANT0EN
      MINIO_SECRET_KEY: pJG-t0rZV5Xo11pd8XAu7IxHQyy0wCMVacW6SiONnjUxr01w
    command: server http://minio{1...4}/data{1...2}
    healthcheck:
      test: [ "CMD", "curl", "-f", "--insecure", "https://localhost:9000/minio/health/live" ]
      interval: 30s
      timeout: 20s
      retries: 3

  nginx:
    image: nginx:1.19.2-alpine
    volumes:
      - ./docker/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./docker/nginx/fullchain.pem:/etc/nginx/fullchain.pem
      - ./docker/nginx/privkey.pem:/etc/nginx/privkey.pem
    ports:
      - "9000:9000"
    depends_on:
      - minio1
      - minio2
      - minio3
      - minio4

volumes:
  data1-1:
  data1-2:
  data2-1:
  data2-2:
  data3-1:
  data3-2:
  data4-1:
  data4-2:
  db-data:

nginx.conf

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;

    upstream minio {
        server minio1:9000;
        server minio2:9000;
        server minio3:9000;
        server minio4:9000;
    }

    server {
        listen       9009 ssl;
        listen  [::]:9000 ssl;
        #listen       9000;
        #listen  [::]:9000;
        server_name  localhost;

         ssl_certificate fullchain.pem;
         ssl_certificate_key privkey.pem;
        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";

         # To allow special characters in headers
         ignore_invalid_headers off;
         # Allow any size file to be uploaded.
         # Set to a value such as 1000m; to restrict file size to a specific value
         client_max_body_size 0;
         # To disable buffering
         proxy_buffering off;

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;

            proxy_pass https://minio;
        }
    }
}

Error

minio2_1          | API: SYSTEM()
minio2_1          | Time: 08:55:47 UTC 02/15/2021
minio2_1          | Error: Marking http://minio4:9000/minio/storage/data2/v22 temporary offline; caused by Post "http://minio4:9000/minio/storage/data2/v22/readall?disk-id=&file-path=format.json&volume=.minio.sys": dial tcp 172.19.0.5:9000: connect: connection refused (*fmt.wrapError)
minio2_1          |        6: cmd/rest/client.go:122:rest.(*Client).Call()
minio2_1          |        5: cmd/storage-rest-client.go:132:cmd.(*storageRESTClient).call()
minio2_1          |        4: cmd/storage-rest-client.go:420:cmd.(*storageRESTClient).ReadAll()
minio2_1          |        3: cmd/format-erasure.go:415:cmd.loadFormatErasure()
minio2_1          |        2: cmd/format-erasure.go:324:cmd.loadFormatErasureAll.func1()
minio2_1          |        1: pkg/sync/errgroup/errgroup.go:55:errgroup.(*Group).Go.func1()
minio2_1          | 
minio2_1          | API: SYSTEM()
minio2_1          | Time: 08:55:47 UTC 02/15/2021
minio2_1          | Error: Marking http://minio4:9000/minio/storage/data1/v22 temporary offline; caused by Post "http://minio4:9000/minio/storage/data1/v22/readall?disk-id=&file-path=format.json&volume=.minio.sys": dial tcp 172.19.0.5:9000: connect: connection refused (*fmt.wrapError)
minio2_1          |        6: cmd/rest/client.go:122:rest.(*Client).Call()
minio2_1          |        5: cmd/storage-rest-client.go:132:cmd.(*storageRESTClient).call()
minio2_1          |        4: cmd/storage-rest-client.go:420:cmd.(*storageRESTClient).ReadAll()
minio2_1          |        3: cmd/format-erasure.go:415:cmd.loadFormatErasure()
minio2_1          |        2: cmd/format-erasure.go:324:cmd.loadFormatErasureAll.func1()
minio2_1          |        1: pkg/sync/errgroup/errgroup.go:55:errgroup.(*Group).Go.func1()
minio2_1          | Client http://minio4:9000/minio/storage/data1/v22 online
minio2_1          | Client http://minio4:9000/minio/storage/data2/v22 online

Then

API: SYSTEM()
minio4_1          | Time: 08:55:49 UTC 02/15/2021
minio4_1          | DeploymentID: 5d5c6db4-f7c9-417d-b548-a6d33565bdf9
minio4_1          | Error: Storage resources are insufficient for the read operation .minio.sys/buckets/.bloomcycle.bin (cmd.InsufficientReadQuorum)
minio4_1          |        1: cmd/data-crawler.go:95:cmd.runDataCrawler()
minio4_1          | Client http://minio3:9000/minio/storage/data1/v22 online
minio4_1          | Client http://minio3:9000/minio/lock/v4 online
minio4_1          | Client http://minio3:9000/minio/storage/data2/v22 online
minio2_1          | Client http://minio3:9000/minio/storage/data1/v22 online
minio2_1          | Client http://minio3:9000/minio/lock/v4 online
minio2_1          | Client http://minio3:9000/minio/storage/data2/v22 online
minio2_1          | All MinIO sub-systems initialized successfully
minio2_1          | 
minio2_1          | API: SYSTEM()
minio2_1          | Time: 08:55:52 UTC 02/15/2021
minio2_1          | DeploymentID: 5d5c6db4-f7c9-417d-b548-a6d33565bdf9
minio2_1          | Error: disk not found:  (*errors.errorString)
minio2_1          |        2: cmd/erasure.go:132:cmd.getDisksInfo.func1()
minio2_1          |        1: pkg/sync/errgroup/errgroup.go:55:errgroup.(*Group).Go.func1()
minio2_1          | 
minio2_1          | API: SYSTEM()
minio2_1          | Time: 08:55:52 UTC 02/15/2021
minio2_1          | DeploymentID: 5d5c6db4-f7c9-417d-b548-a6d33565bdf9
minio2_1          | Error: disk not found:  (*errors.errorString)
minio2_1          |        2: cmd/erasure.go:132:cmd.getDisksInfo.func1()
minio2_1          |        1: pkg/sync/errgroup/errgroup.go:55:errgroup.(*Group).Go.func1()
minio2_1          | Waiting for all MinIO IAM sub-system to be initialized.. lock acquired
minio2_1          | Use `mc admin info` to look for latest server/disk info
minio2_1          |  Status:         6 Online, 2 Offline. 
minio2_1          | Endpoint:  http://172.19.0.6:9000  http://127.0.0.1:9000

minio1_1          | Object API (Amazon S3 compatible):
minio1_1          |    Go:         https://docs.min.io/docs/golang-client-quickstart-guide
minio1_1          |    Java:       https://docs.min.io/docs/java-client-quickstart-guide
minio1_1          |    Python:     https://docs.min.io/docs/python-client-quickstart-guide
minio1_1          |    JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
minio1_1          |    .NET:       https://docs.min.io/docs/dotnet-client-quickstart-guide
minio1_1          | Waiting for all MinIO IAM sub-system to be initialized.. lock acquired
minio1_1          | fatal error: runtime: out of memory
minio1_1          | 
minio1_1          | runtime stack:
minio1_1          | runtime.throw(0x1f2209c, 0x16)
minio1_1          | 	runtime/panic.go:1116 +0x72
minio1_1          | runtime.sysMap(0xc008000000, 0x4000000, 0x3458a58)
minio1_1          | 	runtime/mem_linux.go:169 +0xc6
minio1_1          | runtime.(*mheap).sysAlloc(0x343c160, 0x800000, 0x42c117, 0x343c168)
minio1_1          | 	runtime/malloc.go:727 +0x1e5
minio1_1          | runtime.(*mheap).grow(0x343c160, 0x201, 0x0)
minio1_1          | 	runtime/mheap.go:1344 +0x85
minio1_1          | runtime.(*mheap).allocSpan(0x343c160, 0x201, 0x100, 0x3458a68, 0x2cf83d8)
minio1_1          | 	runtime/mheap.go:1160 +0x6b6
minio1_1          | runtime.(*mheap).alloc.func1()
minio1_1          | 	runtime/mheap.go:907 +0x65
minio1_1          | runtime.(*mheap).alloc(0x343c160, 0x201, 0xc0002b0101, 0xc0002b3380)
minio1_1          | 	runtime/mheap.go:901 +0x85
minio1_1          | runtime.largeAlloc(0x40000c, 0xc002090101, 0x1752c08)
minio1_1          | 	runtime/malloc.go:1177 +0x92
minio1_1          | runtime.mallocgc.func1()
minio1_1          | 	runtime/malloc.go:1071 +0x46
minio1_1          | runtime.systemstack(0x1)
minio1_1          | 	runtime/asm_amd64.s:370 +0x66
minio1_1          | runtime.mstart()
minio1_1          | 	runtime/proc.go:1116
Subscribe
Share
Submit an answer
You can type!ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
App Platform: Run Docker containers without managing serversProduct

With App Platform, you can:

  • Build, deploy, and scale apps and static sites by simply pointing to your GitHub repository
  • Let us manage the infrastructure, app runtimes, and other dependencies
  • Get started by building and deploying three static sites for free

Related

Docker, Discourse and Nginx
Question
DigitalOcean for Non-Developers with Good Linux Experience
Question

Try DigitalOcean for free

Click below to sign up and get $100 of credit to try our products over 60 days!

Sign up