This is a really interesting and I guess the correct answer here is that this depends. I would like to write down some of the advantages of CoreOs and RancherOs, because people usually choose to run their dokcer containers on those OS as they are optimized for running docker containers.
CoreOS is a barebones Linux distribution designed to make large multiple-machine deployments, using different software and dependencies, easier to scale and easier to manage. It is built for high availability and security. It is focused on large-scale deployments, mostly targeting enterprises. It comes bundled with a few really interesting tools developed by the CoreOS team, such as etcd, fleet, and flannel. These tools help you get started with a cluster of CoreOS very quickly. They’re also great starting points to get to understand the concepts behind service discovery, resource scheduling, and container networking.
etcd is a distributed key/value store which is designed to have no single point of failure due to its multiple node architecture.
If you care about high availability at all, deploying using CoreOS is one of the best ways to achieve that with a relatively low price point. Also lets you serve different versions of software on different machines and update machines without any downtime. Also their open source community is really active so if you run into an issue, probably it will be something that someone else have faced before and it will be easy for you the find the solution.
You can have a look at this article on Digital Ocean website even it’s listed as outdated, as you will receive a general infomration about the OS itself: https://www.digitalocean.com/community/tutorials/an-introduction-to-coreos-system-components
It’s lightweight Container Operating System. They run a system Docker as PID 1 and then launch a container that runs the user Docker for all the user containers. Basically everything in RancherOS is a Docker container.
Their team claim that it’s really simple to run containers at scale in development, test, and production. It comes with Improved Security, Automated OS Configuration by using cloud-init to parse the cloud-config file. It also have up-to-date versions of docker and linux but 24/7 support comes with the enterpirse edition only.
You can have a look at their startup guide: https://rancher.com/docs/os/v1.x/en/quick-start-guide/
You can always use Ubuntu LTS, CentOs and Fedora as well. It is just that the above mentioned are claimed to be optimized for running docker containers. I’ve run docker on both Ubuntu and CentOs and I’ve had no issues running it, but it really depends from your needs and your main goal as well.
You can always try one of those to see if you will find the OS optimized for your needs. The purpose of docker-compose is to function as docker cli but to issue multiple commands much more quickly. If you will need to deploy multiple machines with different software you can always try CoreOS for example as it should be a lot easier to manage it.
Hope this help and feel free to ask any questions as this is a really interesting topic.