I’m trying to follow the DigitalOcean CoreOS tutorial, however I can’t get past the “Verify Cluster” section of the tutorial. Specifically, when I type in fleetctl list-machines, I receive the following error:
$ fleetctl list-machines
Error retrieving list of active machines: googleapi: Error 503: fleet server unable to communicate with etcd
When I initially created the 3 droplets, I typed this user data verbatim into the web interface to create the droplets:
#cloud-config
coreos:
etcd2:
# generate a new token for each unique cluster from https://discovery.etcd.io/new:
discovery: https://discovery.etcd.io/d8ea4388ae9d4d41818b88f49c8ed80c
# multi-region deployments, multi-cloud deployments, and Droplets without
# private networking need to use $public_ipv4:
advertise-client-urls: http://$private_ipv4:2379,http://$private_ipv4:4001
initial-advertise-peer-urls: http://$private_ipv4:2380
# listen on the official ports 2379, 2380 and one legacy port 4001:
listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
listen-peer-urls: http://$private_ipv4:2380
fleet:
public-ip: $private_ipv4 # used for fleetctl ssh command
units:
- name: etcd2.service
command: start
- name: fleet.service
command: start
It seems to match closely to the tutorial’s configuration. I created 3 droplets on the nyc2 datacenter. The discovery url was generated with the size parameter set to 3.
I’m not sure what the issue is currently, but here are some of the statuses of the relevant services:
$ systemctl status fleet
● fleet.service - fleet daemon
Loaded: loaded (/usr/lib64/systemd/system/fleet.service; disabled; vendor preset: disabled)
Drop-In: /run/systemd/system/fleet.service.d
└─20-cloudinit.conf
Active: active (running) since Sun 2016-06-05 23:56:54 UTC; 1h 7min ago
Main PID: 1144 (fleetd)
Memory: 10.1M
CPU: 199ms
CGroup: /system.slice/fleet.service
└─1144 /usr/bin/fleetd
Jun 05 23:56:54 coreos-512mb-nyc2-01 systemd[1]: Started fleet daemon.
Jun 05 23:56:54 coreos-512mb-nyc2-01 fleetd[1144]: INFO fleetd.go:64: Starting fleetd version 0.11.7
Jun 05 23:56:54 coreos-512mb-nyc2-01 fleetd[1144]: INFO fleetd.go:168: No provided or default config file found - proceeding without
Jun 05 23:56:54 coreos-512mb-nyc2-01 fleetd[1144]: INFO server.go:157: Establishing etcd connectivity
$ systemctl status etcd2
● etcd2.service - etcd2
Loaded: loaded (/usr/lib64/systemd/system/etcd2.service; disabled; vendor preset: disabled)
Drop-In: /run/systemd/system/etcd2.service.d
└─20-cloudinit.conf
Active: active (running) since Sun 2016-06-05 23:56:54 UTC; 1h 8min ago
Main PID: 1135 (etcd2)
Memory: 24.2M
CPU: 18.091s
CGroup: /system.slice/etcd2.service
└─1135 /usr/bin/etcd2
Jun 06 01:05:18 coreos-512mb-nyc2-01 etcd2[1135]: 4692ba3abd59cdef is starting a new election at term 3020
Jun 06 01:05:18 coreos-512mb-nyc2-01 etcd2[1135]: 4692ba3abd59cdef became candidate at term 3021
Jun 06 01:05:18 coreos-512mb-nyc2-01 etcd2[1135]: 4692ba3abd59cdef received vote from 4692ba3abd59cdef at term 3021
Jun 06 01:05:18 coreos-512mb-nyc2-01 etcd2[1135]: 4692ba3abd59cdef [logterm: 1, index: 3] sent vote request to afc0c7d0eccba6c at term 3021
Jun 06 01:05:18 coreos-512mb-nyc2-01 etcd2[1135]: 4692ba3abd59cdef [logterm: 1, index: 3] sent vote request to d6907df338461404 at term 3021
Jun 06 01:05:19 coreos-512mb-nyc2-01 etcd2[1135]: 4692ba3abd59cdef is starting a new election at term 3021
Jun 06 01:05:19 coreos-512mb-nyc2-01 etcd2[1135]: 4692ba3abd59cdef became candidate at term 3022
Jun 06 01:05:19 coreos-512mb-nyc2-01 etcd2[1135]: 4692ba3abd59cdef received vote from 4692ba3abd59cdef at term 3022
Jun 06 01:05:19 coreos-512mb-nyc2-01 etcd2[1135]: 4692ba3abd59cdef [logterm: 1, index: 3] sent vote request to afc0c7d0eccba6c at term 3022
Jun 06 01:05:19 coreos-512mb-nyc2-01 etcd2[1135]: 4692ba3abd59cdef [logterm: 1, index: 3] sent vote request to d6907df338461404 at term 3022
If you couldn’t tell, I’m a bit of a newbie when it comes to CoreOS and it’s configuration. Can anyone give me some guidance as to why these fleet cannot connect to etcd2?
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Within a few hours of posting this, I received a notification from the DigitalOcean Team noting that the droplets I created were created with an invalid networking configuration. As a result, the droplets were not able to properly communicate with eachother.
In case anyone else is wondering if this effects you, here is the notification that I received with resolution paths that you can take:
[DigitalOcean] Action Required on Your Recently Created Droplet
Hi there,
We’ve identified an issue with your recently created Droplet(s). Droplets created between 18:18 UTC and 23:57 UTC on June 5, 2016 were created with an invalid networking configuration.
As a result, networking will not work properly on your Droplet and regretfully we have had to disable networking on your Droplet.
In order to use your Droplet, you will need to destroy and re-create it, for which there are a few options:
1) Snapshot and Re-create: If you want to retain the data on your Droplet, please take a snapshot and redeploy from the snapshot.
2) Re-create from your preferred image: If you do not need any data on the Droplet, please destroy and re-create from your preferred image.
Locating the Droplet(s) to destroy: You can identify which Droplet(s) to destroy by looking at the Droplet ID, it will be within the following range: 16777116 to 16792599
We apologize for the service interruption you experienced today. This is our mistake and we are committed to making this right for you. Following the full resolution of this issue and internal post-mortem to identify the root cause.
We will be issuing SLA credits, at which time you will receive an email notification once they’re applied to your account.
Should you have any questions about the required steps, or to identify which Droplet(s) is/are affected please reply to this email and we’ll be happy to assist.
Thank you, Team DigitalOcean
The status page has also noted this issues: Status Page
After recreating the droplets, everything seems to be working now.
Become a contributor for community
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
DigitalOcean Documentation
Full documentation for every DigitalOcean product.
Resources for startups and SMBs
The Wave has everything you need to know about building a business, from raising funding to marketing your product.
The developer cloud
Scale up as you grow — whether you're running one virtual machine or ten thousand.
Get started for free
Sign up and get $200 in credit for your first 60 days with DigitalOcean.*
*This promotional offer applies to new accounts only.