Multiple High Availability between datacenters
I'm learning the ropes for high availability ops work. I have requirements that my HA setup also be capable of failing over to different data centers (or simply include more than one datacenter actively at once). Does this sound like a reasonable setup for what I'm trying to accomplish?
1) Create high availability setups in two different data centers. This likely includes 2 load balancers, web servers, and a database server in each physical location.
2) Use DNS round robin to distribute traffic to both data centers, using floating ips pointed at the primary load balancer of each data center. If one data center goes down entirely, remove that DNS record to route traffic to the other datacenter (albeit with a small delay due to TTLs).
Questions about the above:
In terms of database replication between data centers, I presume I'll need to use master master replication? In this scenario, are there suggested best practices for connecting to the databases for replication since I cannot utilize private networking features?
I understand that DNS round robin is essentially random. Is there a better way to use DNS to smartly send traffic to the correct datacenter based on geographical location of the user?