pauldixon
By:
pauldixon

How to update CoreOS cloud-config?

February 9, 2015 17.3k views

Once you've booted a new CoreOS host on Digital Ocean, how can you update the cloud-config?

10 comments
  • The cloud-config that is passed to a droplet on creation can not be updated. It is generally used to provide a script to be run on first boot.

    Could you tell us a bit more about your use case? I'm interested to know why you might want to update it.

    The DigitalOcean metadata service includes a field called "user-data", which can be used to specify a script that will be run as your server is brought online. The CloudInit program, which runs these scripts, can process a special script type called "cloud-config". In this guide, we'll explore how to create cloud-config files and the best ways to leverage their power.
  • i am interested in this too. i use the cloud-config to start the coreos joined to my fleet.

    I just created a coreos instance, how do i get it my could-config? I see that the root cloud-config (i assume) is in /usr/share/oem/cloud-config.yml. It fires up this:

    /usr/bin/coreos-cloudinit --oem=digitalocean

    It would be nice to change that line to do something like:

    /usr/bin/coreos-cloudinit --from-url="http://my.com/my-cloud-config?hostname"

    how do you customize the coreos image? I tried doing this:
    I created my own cloud-config.yml file. Then, ran the program:
    core@d2 ~ $ /usr/bin/coreos-cloudinit -from-file=cloud-config
    Checking availability of "local-file"
    Fetching user-data from datasource of type "local-file"
    Fetching meta-data from datasource of type "local-file"
    2015/02/18 20:52:25 Parsing user-data as cloud-config
    Processing cloud-config from user-data
    2015/02/18 20:52:25 Set hostname to d2
    2015/02/18 20:52:25 Authorized SSH keys for core user
    2015/02/18 20:52:25 Writing drop-in unit "20-cloudinit.conf" to filesystem
    2015/02/18 20:52:25 Writing file to "/run/systemd/system/etcd.service.d/20-cloudinit.conf"
    Failed to apply cloud-config: mkdir /run/systemd/system/etcd.service.d: permission denied

    Didn't work as core user.

    -g

  • I think is pretty common change the cloud-config since sometime the initial configuration ins't enough anymore. As @lgfausak said the coreos provides a command for that coreos-cloudinit. I also couldn't execute it with the core user. But with the root user that did work. Example: https://gist.github.com/maxcnunes/b93a118914caa9ea87ea

  • I haven't found a way to enable iptables except through cloud-config (as described here: http://www.jimmycuadra.com/posts/securing-coreos-with-iptables.)

    So the rules-save file gets rewritten at every boot. What to do if you want to change the rules? Your stuck.

    But perhaps there is a better way to configure and run iptables?

  • @asb This might well be CoreOS related, though, but consider the use case when you need to just add new meta data to the machine in a cluster. Is there a way to do it without destroying it? coreos-cloudinit works just once in my experience.

  • +1, I'd like to be able to properly update ssh keys and have them persist over reboots, seems like the way to do this is set them in the cloud-config, but then if there's a change to a ssh key I have to rebuild the whole server? Am I missing something?

  • I also had the need to change the data controlled by cloud-init and issues with the values being reverted after reboot. However, I do think we should rather treat CoreOS VPS as cattle, not pets and rebuilding a server should be trivial in that sense.

  • Any update to this? @asb I've had to rebuild my cluster 3 times so far because I've needed to make changes to cloud-config. Once because I started my cluster out has http, and only found out later that in order to use https you have to set it in the cloud-config, an then another time so I could set environment variables for etcdctl so I don't have to set them every time I login, or create a .bashrc / .profile every time I create a new user... As I understand it, AWS has the ability to modify the cloud-config file and this ability may be a deal breaker for us when it comes to choosing between DO and another platform...

  • Use cases for updating cloud-config without reprovisioning the host:

    • Update etcd2 and fleet client urls when SSL is added later
    • Add additional units
    • Put iptables-restore config into cloud-config and change it as cluster services evolve
    • Adjust SSH keys

    Workaround is to reprovision host with new metadata. This is tedious and risky, and has to be done fairly often, and writing python scripts to do it all automatically has taken quite a bit of effort.

4 Answers

You can find the cloud-config in :

  • /var/lib/coreos-install/user_data for baremetal
  • /var/lib/coreos-vagrant/vagrant-user-data for vagrant

update it, reboot, if no error, the new configuration should be done.
I don't know if it's the good way but it seems to work.

@bgrayburn you can update ssh keys without doing anything crazy: https://github.com/coreos/init/blob/master/bin/update-ssh-keys

I have a script that I use to push them out across my fleet:

$ cat fleetctl-inject-ssh.sh
#!/bin/bash -x

# Usage:
#   cat public.key | fleetctl-inject-ssh.sh keyname

name=$1
if [ -z $name ]; then
  echo "Provide a name for the injected SSH key"
  exit 1
fi

shift 1

pubkey=$(cat)

for machine in $(fleetctl -strict-host-key-checking=false $@ list-machines --no-legend --full | awk '{ print $1;}'); do
  fleetctl  -strict-host-key-checking=false $@ ssh $machine "echo '${pubkey}' | update-ssh-keys -a $name -n"
done

so to run you could do something like:

echo 'THE_CONTENTS_OF_PUBKEY' |  ./fleetctl-inject-ssh.sh <SOME_INTELLIGENT_NAME_FOR_KEY>

or

cat <PUBKEY> | ./fleetctl-inject-ssh.sh <SOME_INTELLIGENT_NAME_FOR_KEY>

I have found a solution for updating cloud-config.yml!

The file /usr/share/oem/cloud-config.yml is writable by root user, so you can change it.
I first tried to change it to read yml from file (--from-file), but it was not working.
I discovered in the source of coreos-cloudinit, that it gets the user-data from DO's metadata API, but also get network information as well, and coreos-cloudinit sets up the network. So if you just change the line in the cloud-config.yml, the network won't work.
I also discovered that coreos-cloudinit 1st read the userdata and then read the network info. The "--oem=digitalocean" param is just a shortcut for the "--from-digitalocean-metadata http://169.254.169.254/ --convert-netconf digitalocean" parameters.

So the solution is to change the API to localhost and simulate it this way:

until /bin/curl -s -i -o /tmp/v1 http://169.254.169.254/metadata/v1/; do sleep 1; done \
&& until /bin/curl -s -o /tmp/v1.json http://169.254.169.254/metadata/v1.json; do sleep 1; done \
&& ((cat /tmp/v1 | ncat -lp 80 \
&& echo -e "HTTP/1.1 200 OK\r\n\r" | cat - /home/core/cloud-config.yml | ncat -lp 80 \
&& echo -e "HTTP/1.1 200 OK\r\n\r" | cat - /tmp/v1.json | ncat -lp 80) &>/dev/null &) \
&& /usr/bin/coreos-cloudinit --from-digitalocean-metadata http://localhost/ --convert-netconf digitalocean \
&& rm /tmp/v1 && rm /tmp/v1.json
  • Download the needed answers from metadata service to /tmp. The "untils" are very important, because the metadata API is not accessible just a little bit later, so we need to wait for it.
  • Start netcats on localhost port 80 to simulate HTTP webserver, we will have 3 requests, so start 3 netcats in order
  • 1st http request is checking metadata api: http://169.254.169.254/metadata/v1/
  • 2nd request is the new cloud-config.yml from /home/core/cloud-config.yml
  • 3rd request is a JSON with the needed network data (it also contains the user-data, but not used, so we don't care): http://169.254.169.254/metadata/v1.json
  • So, netcats are waiting for requests one-by-one, we can start coreos-cloudinit with localhost as metadata URL, it will get our new cloud-config.yml
  • After successfull initialization we can delete the downloaded answers from /tmp

The full cloud-config is this:

#cloud-config

coreos:
    units:
      - name: 99-public-interface.network
        runtime: yes
        content: |
          [Match]
          Path=*:03.0

          [Network]
          IPv4LL=true
      - name: systemd-networkd.service
        command: restart
      - name: oem-cloudinit.service
        command: restart
        runtime: yes
        content: |
          [Unit]
          Description=Cloudinit from DigitalOcean metadata

          [Service]
          Type=oneshot
          ExecStart=/bin/sh -c 'until /bin/curl -s -i -o /tmp/v1 http://169.254.169.254/metadata/v1/; do sleep 1; done && until /bin/curl -s -o /tmp/v1.json http://169.254.169.254/metadata/v1.json; do sleep 1; done && ((cat /tmp/v1 | ncat -lp 80 && echo -e "HTTP/1.1 200 OK\r\n\r" | cat - /home/core/cloud-config.yml | ncat -lp 80 && echo -e "HTTP/1.1 200 OK\r\n\r" | cat - /tmp/v1.json | ncat -lp 80) &>/dev/null &) && /usr/bin/coreos-cloudinit --from-digitalocean-metadata http://localhost/ --convert-netconf digitalocean && rm /tmp/v1 && rm /tmp/v1.json'
    oem:
      id: digitalocean
      name: DigitalOcean
      version-id: 0.0.4
      home-url: https://www.digitalocean.com/
      bug-report-url: https://github.com/coreos/bugs/issues

It is a hacky way, but works very good. You only need to upload the new config into /home/core/cloud-config.yml and all your changes are alive after reboot.

  • My solution is working with newer CoreOS versions, though with a little modifications.
    The new oem-cloudinit.service is in /etc/systemd/system/oem-cloudinit.service file, which is writable. You just need to change to this:

    [Unit]
    Description=Cloudinit from DigitalOcean metadata
    
    [Service]
    Type=oneshot
    ExecStart=/bin/sh -c 'until /bin/curl -s -i -o /tmp/v1 http://169.254.169.254/metadata/v1/; do sleep 1; done && until /bin/curl -s -o /tmp/v1.json http://169.254.169.254/metadata/v1.json; do sleep 1; done && ((cat /tmp/v1 | ncat -lp 80 && echo -e "HTTP/1.1 200 OK\r\n\r" | cat - /home/core/cloud-config.yml | ncat -lp 80 && echo -e "HTTP/1.1 200 OK\r\n\r" | cat - /tmp/v1.json | ncat -lp 80) &>/dev/null &) && /usr/bin/coreos-cloudinit --from-digitalocean-metadata http://localhost/ && rm /tmp/v1 && rm /tmp/v1.json'
    
    [Install]
    WantedBy=multi-user.target
    

    It still use the /home/core/cloud-config.yml as the new user-data.

Has anyone able to update the user-data without using @Wallner solution?

  • I updated my solution for newer CoreOS versions. I'm afraid the user-data is still cannot be modified.

Have another answer? Share your knowledge.