This tutorial is out of date and no longer maintained.
With DigitalOcean’s simple interface with API access, and full root level access, here is a great opportunity to create an automated setup that scales itself on-demand. You can have your websites scale up or down automatically to meet the traffic demands.
Both vertical and horizontal scalability is possible with this API, which can be accessed with just a bash script. It is best to create an admin droplet which has its public key on all other droplets. This would allow you to push webpage updates with rsync, as well as issue commands to other droplets, and query them for Nagios, NRPE, and SNMPD vitals. You would need to create your own level of logic to determine how and when you wish to scale things – for example if a server reaches 85% CPU usage, double the RAM (vertical scaling) or create a clone (horizontal scaling). The choice is a price-benefit analysis and is left to the reader.
Here is a bash script to get things started. Make sure to modify Client_Key and API_Key to your own variables from API Access Page - click “Generate a new API Key”. For our example we have the following:
Client Key: A2a9SfT4NeFBl6df5cu42 API Key: mnqWGdu4OfLBwwJPee6cpjkeY70qv9mKicqZYvtHJ
#!/bin/bash # Be Sure To Change This! Client_Key=A2a9SfT4NeFBl6df5cu42 API_Key=mnqWGdu4OfLBwwJPee6cpjkeY70qv9mKicqZYvtHJ droplets() { echo "Your current droplets:" All_Droplets=`curl -s "https://api.digitalocean.com/droplets/?client_id=$Client_Key&api_key=$API_Key"` echo $All_Droplets | sed -e 's/,/\n/g' | sed -e 's/{/\n/g' | sed -e 's/}/\n/g' | sed -e 's/"//g' } spinup() { Droplet_Name=$1 Size_ID=$2 Image_ID=$3 Region_ID=$4 echo "Spinning up a new droplet $Droplet_Name" curl -s "https://api.digitalocean.com/droplets/new?name=$Droplet_Name&size_id=$Size_ID&image_id=$Image_ID®ion_id=$Region_ID&client_id=$Client_Key&api_key=$API_Key" } resize() { Droplet_ID=$1 Size_ID=$2 echo "Resizing a droplet ID: $Droplet_ID to Size ID: $2" curl -s "https://api.digitalocean.com/droplets/$Droplet_ID/resize/?size_id=$Size_ID&client_id=$Client_Key&api_key=$API_Key" } sizes() { sizes=`curl -s "https://api.digitalocean.com/sizes/?client_id=$Client_Key&api_key=$API_Key"` echo $sizes | sed -e 's/,/\n/g' | sed -e 's/{//g' | sed -e 's/}//g' | sed -e 's/"//g' | sed -e 's/\[/\n/g' } snapshot() { Droplet_ID=$1 Snapshot_Name=$2 echo "Taking a snapshot of Droplet $1 with Name: $Snapshot_Name" curl -s "https://api.digitalocean.com/droplets/$Droplet_ID/snapshot/?name=$Snapshot_Name&client_id=$Client_Key&api_key=$API_Key" } go() { # Display all current droplets. Region_ID: 1 for US, 2 for Amsterdam. droplets # Show possible Size IDs, for RAM, also tied to amount of CPU cores and HDD allocated - refer to https://www.digitalocean.com/pricing echo "Possible droplets sizes by RAM:" sizes # Take a snapshot of an existing droplet # The syntax is: snapshot Droplet_ID Snapshot_Name # For example to take a snapshot of droplet with ID "72100": #snapshot 72100 domain.com # Vertical Scaling - Increase RAM, CPU, Disk # The syntax is: resize Droplet_ID New_Size_ID # For example to resize a 512MB droplet to a 1GB droplet with ID "72100": #resize 72100 63 # Horizontal Scaling - Clone a server from a snapshot # The syntax is: spinup Droplet_Name Size_ID Image_ID Region_ID # For example, to spinup a 512MB clone of domain.com webserver with image ID "12573" in New York datacenter (Region 1): #spinup domain.com 66 12574 1 } go
Suppose we have created a webserver droplet that has the Nginx/Apache/Memcached stack with Nagios, NRPE, SNMPD configured. Our admin droplet’s SSH key in /root/.ssh/authorized_keys and iptables rules in place, only allowing admin droplet to connect via SSH. The domain being hosted is domain.com.
A sample output of the script, with this webserver droplet being displayed:
id:72100 name:domain.com image_id:12574 size_id:66 region_id:1 backups_active:null ip_address:192.34.56.29 status:active
Note the Droplet_ID is 72100 and Image_ID (snapshot) is 12574 To take a snapshot of this droplet, uncomment the following line and run it:
snapshot 72100 domain.com
Taking a snapshot of Droplet 72100 with Name: domain.com
{"status":"OK","event_id":473473}
This creates a snapshot of our droplet and saves it as “domain.com” :
To scale this droplet vertically from 512MB to 1GB uncomment the following line and run:
resize 72100 63
Resizing a droplet ID: 72100 to Size ID: 63 {"status":"OK","event_id":473530}
To scale this droplet horizontally, we will spin up a new one by uncommenting this line and running Scaler.sh :
spinup domain.com 66 12574 1
Spinning up a new droplet domain.com
{"status":"OK","droplet":{"id":72124,"name":"domain.com","image_id":12574,"size_id":66,"event_id":473545}}
Latest DigitalOcean droplet sizes and IDs:
ID | Name |
66 | 512MB |
63 | 1GB |
62 | 2GB |
64 | 4GB |
65 | 8GB |
61 | 16GB |
60 | 32GB |
70 | 48GB |
70 | 48GB |
69 | 64GB |
68 | 96GB |
Recently DigitalOcean has upgraded to SSD drives, and the smallest droplet comes with 512MB of RAM and 1 CPU core. The next step up is 1GB of RAM and 1 CPU core. You can run Nginx on backend with PHP-FPM handling PHP requests, and Varnish cache in front of Nginx.
Benchmarks of Varnish cache with Nginx and PHP+FPM reveal 420 requests/second compared against 22.25 requests/second served by Nginx and PHP-FPM alone.
As far as droplets are concerned, there is also an interesting concept of a self-replicating VM. A host can be self-aware of being ‘overloaded’ and decide to clone itself and add the clone into DNS rotation. There are many ways of implementing this, and simplest way would be to use SNMPD and Nagios for polling data.
You can even setup an orchestration VM that has its public key on all VMs. A level of automation depends on your imagination and desire for complexity.
Having essential monitoring tools like Nagios, SNMPD, NRPE, and your SSH keys on the droplet that you are cloning is important. It will allow you to sync new content to this server down the line, further automating the process. You can setup an admin droplet to which you’ll upload webpages and from which all syncs will be done with a crontab. This admin droplet should have its key placed on all your droplets, and SSH port allowed only from that admin droplet’s IP address. Whether you choose to have an OpenVPN connection to this admin droplet is up to you.
Once you have spun up your new webserver from a snapshot, you’ll need to place the server into rotation. This can be accomplished by DNS round robin, Nginx reverse proxy, a dedicated load balancer setup, and so on. The choice would depend on your infrastructure needs and budget.
Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Sign up for Infrastructure as a Newsletter.
Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
“Benchmarks of Varnish cache with Nginx and PHP+FPM reveal 420 requests/second compared against 22.25 requests/second served by Nginx and PHP-FPM alone.”
This is true if you haven’t configured nginx well. How? Use nginx as a reverse proxy for itself (this cuts out the need for Varnish). Nginx will slow down if you have to hit php5-fpm and more so if you have to query a database. Varnish basically lets you skip interaction with php and the database, giving it the extra speed. But nginx can do that on its own. Options:
(1) Implement microcaching in a flat file system. (2) Implement microcaching integrated with memcache (faster). (3) Compile Google’s Pagespeed module with nginx and use its caching in a flat file system. (4) Compile Google’s Pagespeed module with nginx and integrate it with memcache (faster).
These caches can be configured to work globally or per user.
Also, for dynamic pages that cannot be cached: they can… to some extent. Varnish provides nice Edge Side Includes to cache parts of dynamic pages so that the entire page need not be rendered each time. Nginx can do this with its equivalent to ESI: Server-Side includes. Both ESI and SSI are a pain to setup because they are so idiosyncratic to the structure of your site, but, if you have a truly high-traffic site that needs to be dynamic and you’ve already leveraged everything possible out of caches/memcache, then it might be worth looking into.
url for scale should be corrected in sin up
d=$Image_ID®ion_id=$Region_ID&client_id=$Client_Key&api_key=$API_Key"
®ion_id=$Region_ID
Is there a possibility of scaling up SSD Disk storage without changing the Droplet processing specifications? My application database growth is not proportional to the Droplet processing requirements. The storage grows about 100 times before i need to increase the Droplet processing power. Your Droplets are wonderful but I am stack here because in your current droplets storage scales up with processing power
Cheers Kamal thanks for confirming the proper procedure for snapshots :)
ah good old pastebin sample of output http://pastebin.com/Us4anXMz :D
@GEORGE: Please paste code at a pastebin service such as <a href=“http://pastie.org”>pastie.org</a>.
The API docs are available here: <a href=“https://api.digitalocean.com/”>https://api.digitalocean.com/</a>
You have to power off the droplet first and /then/ take a snapshot.
looks like some of the code is broken posted here :o
Did the same for formatting image output too
i.e.
allimages() { curl -s “${APIBASEURL}/images/?client_id=${CLIENTID}&api_key=${APIKEY}” | awk -F ‘{"’ ‘{for(i=1;i<=NF;i++) print $i}’ | sed -e ‘s/}//g’ | sed -e ‘s/]//g’ | sed -e ‘s/,/ | /g’ | sed -e ‘s/"//g’ echo }
ends up outputting like
./doapi.sh imglist
status:OK | images:[ id:1601 | name:CentOS 5.8 x64 | slug:null | distribution:CentOS | public:true | id:1602 | name:CentOS 5.8 x32 | slug:null | distribution:CentOS | public:true | id:12573 | name:Debian 6.0 x64 | slug:null | distribution:Debian | public:true | id:12575 | name:Debian 6.0 x32 | slug:null | distribution:Debian | public:true | id:14097 | name:Ubuntu 10.04 x64 | slug:null | distribution:Ubuntu | public:true | id:14098 | name:Ubuntu 10.04 x32 | slug:null | distribution:Ubuntu | public:true | id:32387 | name:Fedora 17 x32 | slug:null | distribution:Fedora | public:true | id:32399 | name:Fedora 17 x32 Desktop | slug:null | distribution:Fedora | public:true | id:32419 | name:Fedora 17 x64 Desktop | slug:null | distribution:Fedora | public:true | id:32428 | name:Fedora 17 x64 | slug:null | distribution:Fedora | public:true | id:284203 | name:Ubuntu 12.04 x64 | slug:null | distribution:Ubuntu | public:true | id:284211 | name:Ubuntu 12.04 x32 | slug:null | distribution:Ubuntu | public:true | id:303619 | name:Debian 7.0 x32 | slug:null | distribution:Debian | public:true | id:308287 | name:Debian 7.0 x64 | slug:null | distribution:Debian | public:true | id:345791 | name:Ubuntu 13.04 x32 | slug:null | distribution:Ubuntu | public:true | id:350076 | name:Ubuntu 13.04 x64 | slug:null | distribution:Ubuntu | public:true | id:350424 | name:Arch Linux 2013.05 x64 | slug:null | distribution:Arch Linux | public:true | id:361740 | name:Arch Linux 2013.05 x32 | slug:null | distribution:Arch Linux | public:true | id:376568 | name:CentOS 6.4 x32 | slug:null | distribution:CentOS | public:true | id:433240 | name:Ubuntu 12.10 x32 | slug:null | distribution:Ubuntu | public:true | id:459444 | name:LAMP on Ubuntu 12.04 | slug:null | distribution:Ubuntu | public:true | id:464235 | name:Ruby on Rails on Ubuntu 12.10 (Nginx + Unicorn) | slug:null | distribution:Ubuntu | public:true | id:473123 | name:Ubuntu 12.10 x64 | slug:null | distribution:Ubuntu | public:true | id:473136 | name:Ubuntu 12.10 x64 Desktop | slug:null | distribution:Ubuntu | public:true | id:483575 | name:Redmine on Ubuntu 12.04 | slug:null | distribution:Ubuntu | public:true | id:562354 | name:CentOS 6.4 x64 | slug:null | distribution:CentOS | public:true | id:682275 | name:Wordpress on Ubuntu 12.10 | slug:null | distribution:Ubuntu | public:true | id:696598 | name:Fedora 19 x86-64 | slug:null | distribution:Fedora | public:true | id:697056 | name:Fedora 19 x86 | slug:null | distribution:Fedora | public:true | id:719193 | name:Docker on Ubuntu 13.04 (9/2) | slug:null | distribution:Ubuntu | public:true
for regions
listregions() { curl -s “${APIBASEURL}/regions/?client_id=${CLIENTID}&api_key=${APIKEY}” | awk -F ‘{"’ ‘{for(i=1;i<=NF;i++) print $i"\n"}’ | sed -e ‘s/}//g’ | sed -e ‘s/]//g’ | sed -e ‘s/,/\n/g’ | sed -e ‘s/"//g’ echo }
./doapi.sh regions
status:OK regions:[
id:2 name:Amsterdam 1 slug:ams1
id:3 name:San Francisco 1 slug:sfo1
id:4 name:New York 2 slug:nyc2
still working things out like proper snapshot procedure etc. If it’s indeed power off then snapshot run then i’ll have 2 functions to process when snapshot command is called
snapshot) poff snapshotdroplet ;;
so far
./doapi.sh /root/tools/doapi/doapi.sh reboot /root/tools/doapi/doapi.sh recycle /root/tools/doapi/doapi.sh poweron /root/tools/doapi/doapi.sh poweroff /root/tools/doapi/doapi.sh listdroplets /root/tools/doapi/doapi.sh dropletinfo /root/tools/doapi/doapi.sh spinup /root/tools/doapi/doapi.sh resize /root/tools/doapi/doapi.sh snapshot /root/tools/doapi/doapi.sh regions /root/tools/doapi/doapi.sh imglist /root/tools/doapi/doapi.sh imgid /root/tools/doapi/doapi.sh transferimgid /root/tools/doapi/doapi.sh listsizes /root/tools/doapi/doapi.sh listdomains /root/tools/doapi/doapi.sh showdomain /root/tools/doapi/doapi.sh listrecords
Thanks Kamal, maybe this is best place to ask as I tried DO support ticket and was asked and directed to read the api doc page heh.
Now I have question with regards to that bash script example above of creating a snapshot, is that valid ? As from the control panel on snapshot page it says to power off droplet from command line BEFORE creating snapshot ?
Does api snapshot command automatically power off the droplet or do I need to first run api command to shutdown or power off the droplet BEFORE running api cmd to create snapshot ?
also is it shutdown ?
https://api.digitalocean.com/droplets/[droplet_id]/power_off/?client_id=[your_client_id]&api_key=[your_api_key]
or power off ?
https://api.digitalocean.com/droplets/[droplet_id]/shutdown/?client_id=[your_client_id]&api_key=[your_api_key]
that i need to run before running snapshor command ?
For powering off at command line that is
shutdown -P now
or
shutdown -h now
?
Only then after powering off, then create snapshot
https://api.digitalocean.com/droplets/[droplet_id]/snapshot/?name=[snapshot_name]&client_id=[your_client_id]&api_key=[your_api_key]
Would be great to get some clarification for proper way to create a snapshot via command line and via api. If indeed need to power off, then the above bash script article needs updating or correcting for proper way to create snapshots :)
How can I Install a dinamic Load Balancer between my instances?
Great job George! Good luck :]
Make sure you put it on github so people can contribute as well!