The worst thing is that I cannot re-add a domain. As soon as a domain gets deleted the limit prohibits te re-creation.
Anyone else has encountered the same issue?
]]>DOCN rebooted by droplet and now I am getting a 502 Bad Gateway error (nginx/1.18.0 (Ubuntu)). Please could someone assist what I need to do. I tried running “pm2 restart app.js” in console but getting [PM2][ERROR] Process or Namespace app.js not found.
This is really frustrating as my service is down because of Digital Ocean issue - have raised a support ticket.
]]>If this keeps happening, check out status.digitalocean.com.
This is happening for the past few weeks, like almost 50% of the time, im in NYC.
]]>After resizing it. We could no longer fetch from the server, even though the application is 100% running, I’ve checked many times, via pm2. Logs are being printed out constantly.
I’ve read countless threads and tried countless solutions, none that worked so far. Port seems fine. Firewalls are disabled.
The 2 errors we are getting are net::ERR_CONNECTION_REFUSED and TypeError: Failed to fetch.
The application runs fine on localhost, just not on the droplet. Would appreciate any help. If there’s any further information needed please let me know.
Edit: Not sure how relevant, but I ran “service nginx status” and got the following:
● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Wed 2024-01-03 21:00:38 UTC; 13h ago Docs: man:nginx(8) Process: 903 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)
Jan 03 21:00:37 glorious-nexus-api systemd[1]: Starting A high performance web server and a reverse proxy server… Jan 03 21:00:37 glorious-nexus-api nginx[903]: nginx: [alert] could not open error log file: open() “/var/log/nginx/error.log” failed Jan 03 21:00:38 glorious-nexus-api nginx[903]: 2024/01/03 21:00:37 [emerg] 903#903: open() “/var/log/nginx/access.log” failed (2: No Jan 03 21:00:38 glorious-nexus-api nginx[903]: nginx: configuration file /etc/nginx/nginx.conf test failed Jan 03 21:00:38 glorious-nexus-api systemd[1]: nginx.service: Control process exited, code=exited status=1 Jan 03 21:00:38 glorious-nexus-api systemd[1]: nginx.service: Failed with result ‘exit-code’. Jan 03 21:00:38 glorious-nexus-api systemd[1]: Failed to start A high performance web server and a reverse proxy server.
]]>I couldn’t interpret what was given in the documentation, so please help code through this.
]]>My design was that I have a function to keep updating database , then webpage in flask app can show records in database. I am using APschedule to call function - “DataBaseUpdate”. However, it doesn’t run.
Someone told me create another non-web worker service in App platform. My questions are #1. update database, the worker service should connect internet, is non-web service will work? #2. Is the code base for this service independent than the flask-app? If so, can I use one dockerfile to deploy both this service and flask-app in future for convenience?
if __name__ == "__main__":
with app.app_context():
db.create_all() # <--- create db object.
start = datetime.date.today()+ datetime.timedelta(days=1) #set start and end time
end= start + datetime.timedelta(days=10)
scheduler = BackgroundScheduler()
scheduler.add_job(DataBaseUpdate, 'interval', seconds=30)
scheduler.start()
port = int(os.environ.get('PORT', 5000))
app.run(debug=True, host='0.0.0.0', port=port)
]]>So I’m wondering: 1) what images are the basis for DO’s supported droplet OS images? (so I can start with the closest thing to supported) and 2) how were they “built” or modified? (And what’s the best tool for building/modifying a cloud image offline?)
]]>Additionally it seems like there’s the issue of what software (especially closed source) will continue to support 32 bit systems.
So, uh, what to do? Is the 64 bit memory waste really that bad?
]]>I am not technical. We have a Digital Ocean server and need to set up a mail server. Would doing this be easier if we first install cPanel?
I feel we probably need cPanel; now anyway to make managing the server easier.
THank you.
]]>The loss of “built in” support, particularly given that with a custom image upload (which does boot and run) you can’t have IPv6, is serious trouble.
I have gone through the trouble to take my existing FreeBSD dropelets, clone them, strip out all the “extra” stuff and as it turns out I can then upgrade them with a modest amount of care and indeed the snapshot I generate with those can indeed be cloned into new droplets, so for myself I can handle this, albeit at the cost of paying to store the “stripped” base snapshots.
But as it turns out the changes are extraordinarily trivial for DO to support this once again. Specifically, the biggest is that the “digitcalocean” init is missing as a “REQUIRE” ldconfig because it uses jq, and jq is a package; if it runs too early it will fail because a shared library it needs is in /usr/local/lib, which isn’t in the search list until ldconfig runs. This means if rcorder spits it out the wrong way (pure luck on that one) you get no networking when it comes up because the init to set it fails. There is also a “do not screw with sshd’s config” flag that is a good idea because otherwise the update process can hose that file (FreeBSD’s development is aware of this one as it can bite people in other environments.) The other change you can (and Digital Ocean should) make is to permit a single-user boot from the console (which you can do) to drop to a root shell, which by default on their previous images was not enabled, so if you got hosed to the point where you can’t sign in and do a “sudo” you’re dead.
These are all trivial changes that took me under a half-hour to get nailed down. I now have both “old-style” UFS and ZFS filesystem snapshots I can clone from on 13.2, and all is well with them. That in turn means I can upgrade my existing stuff without fear in that if something goes wrong I’m not instantly blown out with no way to recover from it.
Please consider restoring FreeBSD as an OS option. I can handle this for my own personal use but this withdrawal of support has meant that for people I talk with on a professional basis who use FreeBSD (and that’s many) I’ve had to withdraw my recommendation they use Digital Ocean as a host for their projects, and that’s sad.
Thank you for your consideration.
]]>I have a website (www.domain.com) hosted on DO App platform. We also have a small merch shop that’s hosted on Wix (owners idea, not mine) We now want to have the wix page as a subdomain. How do I set this up? I want the URL as follows: some-subdomain.domain.com. I can only find tutorials on how to either fully set up the DNS settings over at Wix or to set up the full domain as the wix page.
]]>Can you please help me with this? I understand you probably have follow-up questions and I will do my best to provide an answer. On my previous computer I could just click the VPN, then connect to any server I had on forge via ssh forge@ip.
As far as I can tell I think I used an SSH key provided by Digital Ocean to gain access to my Laravel Forge servers via Outline VPN. and from there I could connect to the server I wanted.
Could somebody help me please figure out which key from Digital Ocean I should paste into Outline VPN?
]]>doctl app create --spec .do/app.yaml
but it gives me this error
Error: POST https://api.digitalocean.com/v2/apps: 400 (request “304ba4d8-b2c3-439d-8b7f-2d7f9b14a01a”) Account does not have access to the repo
I verified the doctl account I’m at via doctl auth list
and verified also that the personal access token attached to auth is working as I used it to connect to digitalocean API.
Unto the Digitalocean app platform, I can browse the repo and create the app as well. Any help?
]]>Just wanted to inform that Retroachievements changed its server. Its IP was 178.128.41.234 before, now it’s 165.232.42.21.
]]>I did some server maintenance just running the regular maintenance commands.
sudo apt-get update
sudo apt-get upgrade
sudo reboot
After the reboot Apache didn’t start. And all the websites are down. I wasn’t sure what the problem was so I restored a droplet backup from the previous day.
This didn’t seem to work right away. (even though the restore process appeared to be finished) An hour later, I saw my sites were loading again. So I assumed restoring the backup had rolled back the problems caused by the updates. This morning however all the sites are now down again and apache is not able to start.
I thought that restoring the backup would have rolled back any of the server maintenance updates?
]]>https://docs.digitalocean.com/products/networking/ddos/how-to/manage-plan/
But I’m not able to find that option, do I need to do something to enable?
]]>I only found the “Droplet Bandwidth” but nothing more for any of the above. Is it possible to do so with the current API ?
I’m trying to replicate the graphs that cloud.digitalocean provides when clicking on “network” - basically I want all that data from an API, without having to access the dashboard itself.
Is this possible ?
]]>Thank you.
I uploaded the relevant files to my Digital Ocean Droplet (AKA Ubuntu Instance) and installed NodeJS and NPM and ran ‘npm install’ and ‘npm start’ as I would on my local machine with no success.
]]>Operating system must be Unix-like. Just how Unix-like does my OS have to be? What in particular makes it necessary to have a Unix-like OS? What specific Unix features does Digital Ocean expect, and what does it use them for?
OS must support ext3 or ext4 file systems. Does this mean the OS needs to be on an ext fs, or just needs to read one? What exactly is ext required for? I’d prefer to have no filesystem at all if possible.
cloud-init is required. What exactly is cloud-init used for? The page linked says that droplets without it won’t function correctly, but it doesn’t provide details regarding why. I was reading through the metadata in the cloud-init drive on an Ubuntu droplet, and it seems to contain some network config info. As I understand it, custom images use DHCP to configure their network systems, so would any of the network config info even be relevant? There’s also some sort of droplet agent install script, but I’m not sure what this does exactly. What exactly do I need cloud-init for? If it is necessary for operation, is it feasible to replicate the necessary functionality without cloud-init?
I need my OS to do exactly two things: read/write network packets that can reach the internet and read/write from its disk. As long as I can achieve this, I shouldn’t need any further configuration.
This page also listed SSH as a requirement, but I don’t think that’s relevant to me, as I plan to access it over TLS.
Thanks for your time.
]]>The app is about 60MB.
]]>Assuming I’ll have a dev, qa and prod environments, will that translate to 19 * 3 = $57/month (give or take, given actual use), or are there any shared pricing calculations taking place?
]]>Then I went to the abuse page… same problem there. No way to let DO know about anything except abuse ORIGINATING from DO. Nothing to warn about abuse TARGETING DO.
Both the ticketing system and the abuse system need to have an “Other” topic category so that support issues not fitting into the limited subset available can be entered.
Even THIS community system’s tags/topics are too restricted and don’t allow entry of new tags. In this case, there are no tags for “Support” or “Abuse”. Please make it easier for us to submit tickets for topics you haven’t thought of.
]]>I’m using App Platform since 3 months (come from Heroku) and i have two strange behaviors :
<!DOCTYPE html>
<html>
<head>
<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">
<meta name=\"robots\" content=\"noindex\">
<style>body,html{height:100%;margin:0}body{display:flex;align-items:center;justify-content:center;flex-direction:column;-webkit-font-smoothing:antialiased;text-rendering:optimizeLegibility}p{text-align:center;font-family:-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen,Ubuntu,Cantarell,Fira Sans,Droid Sans,Helvetica Neue,sans-serif;color:#000;font-size:14px;margin-top:-50px}p.code{font-size:24px;font-weight:500;border-bottom:1px solid #e0e1e2;padding:0 20px 15px}p.text{margin:0}a,a:visited{color:#aaa}</style>
</head>
<body>
<p class=\"code\">
Error
</p>
<p class=\"text\">
We encountered an error when trying to load your application and your page could not be served. Check the logs for your application in the App Platform dashboard.
</p>
<div style=\"display:none;\">
<h1>
upstream_reset_before_response_started{connection_termination} (503 UC) </h1>
<p data-translate=\"connection_timed_out\">App Platform failed to forward this request to the application.</p>
</div>
</body>
</html>
It’s critical becauses it breaks somes requests. I have no logs for that too.
Thank you for your help.
]]>I’m using App Platform since 3 months (come from Heroku) and i have two strange behaviors :
<!DOCTYPE html>
<html>
<head>
<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">
<meta name=\"robots\" content=\"noindex\">
<style>body,html{height:100%;margin:0}body{display:flex;align-items:center;justify-content:center;flex-direction:column;-webkit-font-smoothing:antialiased;text-rendering:optimizeLegibility}p{text-align:center;font-family:-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen,Ubuntu,Cantarell,Fira Sans,Droid Sans,Helvetica Neue,sans-serif;color:#000;font-size:14px;margin-top:-50px}p.code{font-size:24px;font-weight:500;border-bottom:1px solid #e0e1e2;padding:0 20px 15px}p.text{margin:0}a,a:visited{color:#aaa}</style>
</head>
<body>
<p class=\"code\">
Error
</p>
<p class=\"text\">
We encountered an error when trying to load your application and your page could not be served. Check the logs for your application in the App Platform dashboard.
</p>
<div style=\"display:none;\">
<h1>
upstream_reset_before_response_started{connection_termination} (503 UC) </h1>
<p data-translate=\"connection_timed_out\">App Platform failed to forward this request to the application.</p>
</div>
</body>
</html>
It’s critical becauses it breaks somes requests. I have no logs for that too.
Thank you for your help.
]]>The web socket runs on the same port 3000
as the application, with:
import startSocketServer from "./server/sockets"
export default defineNuxtConfig({
...
hooks: {
listen: (nuxtServer) => startSocketServer(nuxtServer)
},
...
});
import { Server } from "socket.io";
export default (nuxtServer) => {
const io = new Server(nuxtServer)
io.on("connection", (socket) => {
console.log(`connection ${socket.id}`)
});
};
And on the client I have:
const socketDomain = (isLocal) ? 'localhost:3000' : 'my.domain.com'
const socket = io(socketDomain, {
transports: ['websocket'],
socket.on("connect", () => {
console.log('connect')
});
socket.on("connect_error", (err) => {
console.log(`connect_error due to ${err}`);
});
The application run perfectly locally with nuxi dev
or nuxi build && node .output/server/index.mjs
but when I deploy on DigitalOcean I get the following:
WebSocket connection to 'wss://my.domain.com/socket.io/?EIO=4&transport=websocket' failed:
doOpen @ entry.3f5d167e.js:21
open @ entry.3f5d167e.js:20
open @ entry.3f5d167e.js:21
$i @ entry.3f5d167e.js:21
A1.exports @ entry.3f5d167e.js:21
open @ entry.3f5d167e.js:21
(anonymous) @ entry.3f5d167e.js:21
invoice.22658294.js:1 connect_error due to Error: websocket error
I also tried with the web socket on a different port, but from my understanding it is not supported on DO.
I also tried with transports: ['polling', 'websocket']
but I got xhr poll error
.
What could be the issue?
]]>
server {
server_name domain www.domain;
location / {
proxy_pass http://localhost:3000;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/domain/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/domain/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
if ($host = www.domain) {
return 301 https://$host$request_uri;
} # managed by Certbot
if ($host = domain) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name domain www.domain;
return 404; # managed by Certbot
}
]]>Meaning that if you decide to go to (let it be) fraud.com, you will see the content of my web application (api requests are redirected from it to me as well).
As a result my content is indexed on its pages, that’s bad for my SEO.
I am using:
Also godaddies WHOIS service says that it is using DigitalOcean’s DNS.
Maybe support can handle it? How can I stop it?
]]>not install packages due to an OSError
Yesterday it deployed the sam app with the same requirements.txt just fine. Can anybody help to understand what the problem is?
023-03-07 23:34:50] │ -----> Requirements file has been changed, clearing cached dependencies
[2023-03-07 23:34:51] │ -----> Installing python-3.10.6
[2023-03-07 23:34:53] │ -----> Installing pip 22.2.2, setuptools 63.4.3 and wheel 0.37.1
[2023-03-07 23:34:59] │ -----> Installing SQLite3
[2023-03-07 23:35:06] │ -----> Installing requirements with pip
[2023-03-07 23:35:06] │ Processing /opt/concourse/worker/volumes/live/6ca6f098-d773-4461-5c91-a24a17435bda/volume/appnope_1606859448531/work
[2023-03-07 23:35:06] │ ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/opt/concourse/worker/volumes/live/6ca6f098-d773-4461-5c91-a24a17435bda/volume/appnope_1606859448531/work'
[2023-03-07 23:35:06] │
[2023-03-07 23:35:06] │ ERROR: failed to build: exit status 1
]]>There will be something wrong with the server settings, where…?
[jecnapss] [2023-02-25 20:48:32] Detected 1.0GiB of RAM
[jecnapss] [2023-02-25 20:48:32] PHP memory_limit is 128M Bytes
[jecnapss] [2023-02-25 20:48:33] Starting php-fpm with 8 workers...
[jecnapss] [2023-02-25 20:48:35] Starting httpd...
[jecnapss] [2023-02-25 20:48:36] Application ready for connections on port 8080.
[jecnapss] [2023-02-25 20:48:58] [25-Feb-2023 20:48:57 UTC] PHP Notice: Undefined index: login_name in /workspace/admin/topbar.php on line 23
[jecnapss] [2023-02-25 20:48:58] [25-Feb-2023 20:48:57 UTC] PHP Notice: Undefined index: login_id in /workspace/admin/topbar.php on line 36
[jecnapss] [2023-02-25 20:48:58] [25-Feb-2023 20:48:57 UTC] PHP Notice: Undefined index: system in /workspace/admin/navbar.php on line 7
[jecnapss] [2023-02-25 20:48:58] [25-Feb-2023 20:48:57 UTC] PHP Notice: Trying to access array offset on value of type null in /workspace/admin/navbar.php on line 7
[jecnapss] [2023-02-25 20:48:58] [25-Feb-2023 20:48:57 UTC] PHP Notice: Undefined index: login_type in /workspace/admin/navbar.php on line 19
[jecnapss] [2023-02-25 20:49:33] 10.244.31.108 - - [25/Feb/2023:20:49:33 +0000] "-" 408 - "-" "-
[jecnapss] [2023-02-25 20:49:35] [25-Feb-2023 20:49:35] WARNING: [pool www] child 180, script '/workspace/admin/index.php' (request: "GET /admin/index.php") execution timed out (37.820128 sec), terminating
[jecnapss] [2023-02-25 20:49:35] [25-Feb-2023 20:49:35] WARNING: [pool www] child 180 exited on signal 15 (SIGTERM) after 60.232039 seconds from start
[jecnapss] [2023-02-25 20:49:35] [Sat Feb 25 20:49:35.418051 2023] [proxy_fcgi:error] [pid 197:tid 140112654501632] [client 10.244.2.105:40858] AH01067: Failed to read FastCGI header
[jecnapss] [2023-02-25 20:49:35] [Sat Feb 25 20:49:35.418257 2023] [proxy_fcgi:error] [pid 197:tid 140112654501632] (104)Connection reset by peer: [client 10.244.2.105:40858] AH01075: Error dispatching request to :
[jecnapss] [2023-02-25 20:49:35] 10.244.2.105 - - [25/Feb/2023:20:48:57 +0000] "GET /admin/ HTTP/1.1" 503 299 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 Edg/110.0.1587.56
[jecnapss] [2023-02-25 20:49:43] 10.244.31.108 - - [25/Feb/2023:20:49:43 +0000] "-" 408 - "-" "-
[jecnapss] [2023-02-25 20:49:45] [25-Feb-2023 20:49:45] WARNING: [pool www] child 186, script '/workspace/login.php' (request: "GET /login.php") execution timed out (36.135580 sec), terminating
[jecnapss] [2023-02-25 20:49:45] [25-Feb-2023 20:49:45] WARNING: [pool www] child 186 exited on signal 15 (SIGTERM) after 70.134275 seconds from start
[jecnapss] [2023-02-25 20:49:45] [Sat Feb 25 20:49:45.422294 2023] [proxy_fcgi:error] [pid 195:tid 140112654501632] [client 10.244.2.105:47566] AH01067: Failed to read FastCGI header
[jecnapss] [2023-02-25 20:49:45] [Sat Feb 25 20:49:45.422843 2023] [proxy_fcgi:error] [pid 195:tid 140112654501632] (104)Connection reset by peer: [client 10.244.2.105:47566] AH01075: Error dispatching request to :
[jecnapss] [2023-02-25 20:49:45] 10.244.2.105 - - [25/Feb/2023:20:49:09 +0000] "GET /login.php HTTP/1.1" 503 299 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 Edg/110.0.1587.56
[jecnapss] [2023-02-25 20:49:53] 10.244.31.108 - - [25/Feb/2023:20:49:53 +0000] "-" 408 - "-" "-
[jecnapss] [2023-02-25 20:50:03] 10.244.31.108 - - [25/Feb/2023:20:50:03 +0000] "-" 408 - "-" "-
[jecnapss] [2023-02-25 20:50:13] 10.244.31.108 - - [25/Feb/2023:20:50:13 +0000] "-" 408 - "-" "-
[jecnapss] [2023-02-25 20:50:15] [25-Feb-2023 20:50:15] WARNING: [pool www] child 185, script '/workspace/login.php' (request: "GET /login.php") execution timed out (37.714901 sec), terminating
[jecnapss] [2023-02-25 20:50:15] [25-Feb-2023 20:50:15] WARNING: [pool www] child 183, script '/workspace/admin/index.php' (request: "GET /admin/index.php") execution timed out (37.715245 sec), terminating
[jecnapss] [2023-02-25 20:50:15] [25-Feb-2023 20:50:15] WARNING: [pool www] child 185 exited on signal 15 (SIGTERM) after 100.209739 seconds from start
[jecnapss] [2023-02-25 20:48:05] Detected 1.0GiB of RAM
[jecnapss] [2023-02-25 20:50:15] [25-Feb-2023 20:50:15] WARNING: [pool www] child 183 exited on signal 15 (SIGTERM) after 100.222834 seconds from start
[jecnapss] [2023-02-25 20:50:15] [25-Feb-2023 20:50:15 UTC] PHP Notice: Undefined index: login_name in /workspace/topbar.php on line 23
[jecnapss] [2023-02-25 20:50:15] [25-Feb-2023 20:50:15 UTC] PHP Notice: Undefined index: login_id in /workspace/topbar.php on line 35
[jecnapss] [2023-02-25 20:48:05] PHP memory_limit is 128M Bytes
[jecnapss] [2023-02-25 20:48:05] Starting php-fpm with 8 workers...
[jecnapss] [2023-02-25 20:48:07] Starting httpd...
[jecnapss] [2023-02-25 20:48:08] Application ready for connections on port 8080.
[jecnapss] [2023-02-25 20:49:01] [25-Feb-2023 20:49:01 UTC] PHP Notice: Undefined index: login_name in /workspace/admin/topbar.php on line 23
[jecnapss] [2023-02-25 20:49:01] [25-Feb-2023 20:49:01 UTC] PHP Notice: Undefined index: login_id in /workspace/admin/topbar.php on line 36
[jecnapss] [2023-02-25 20:49:01] [25-Feb-2023 20:49:01 UTC] PHP Notice: Undefined index: system in /workspace/admin/navbar.php on line 7
[jecnapss] [2023-02-25 20:49:01] [25-Feb-2023 20:49:01 UTC] PHP Notice: Trying to access array offset on value of type null in /workspace/admin/navbar.php on line 7
[jecnapss] [2023-02-25 20:49:01] [25-Feb-2023 20:49:01 UTC] PHP Notice: Undefined index: login_type in /workspace/admin/navbar.php on line 19
[jecnapss] [2023-02-25 20:49:03] 10.244.21.88 - - [25/Feb/2023:20:49:03 +0000] "-" 408 - "-" "-
[jecnapss] [2023-02-25 20:50:15] [25-Feb-2023 20:50:15 UTC] PHP Notice: Undefined index: login_id in /workspace/index.php on line 227
[jecnapss] [2023-02-25 20:50:15] [Sat Feb 25 20:50:15.423176 2023] [proxy_fcgi:error] [pid 279:tid 140112537061120] [client 10.244.2.105:43608] AH01067: Failed to read FastCGI header
[jecnapss] [2023-02-25 20:50:15] [Sat Feb 25 20:50:15.423560 2023] [proxy_fcgi:error] [pid 279:tid 140112537061120] (104)Connection reset by peer: [client 10.244.2.105:43608] AH01075: Error dispatching request to :
[jecnapss] [2023-02-25 20:50:15] [Sat Feb 25 20:50:15.423537 2023] [proxy_fcgi:error] [pid 279:tid 140112547546880] [client 10.244.2.105:43596] AH01067: Failed to read FastCGI header
[jecnapss] [2023-02-25 20:50:15] [Sat Feb 25 20:50:15.423802 2023] [proxy_fcgi:error] [pid 279:tid 140112547546880] (104)Connection reset by peer: [client 10.244.2.105:43596] AH01075: Error dispatching request to :
[jecnapss] [2023-02-25 20:50:15] 10.244.2.105 - - [25/Feb/2023:20:49:37 +0000] "GET /login.php HTTP/1.1" 503 299 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 Edg/110.0.1587.56
[jecnapss] [2023-02-25 20:50:15] 10.244.2.105 - - [25/Feb/2023:20:49:37 +0000] "GET /admin/ HTTP/1.1" 503 299 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 Edg/110.0.1587.56
[jecnapss] [2023-02-25 20:50:15] 10.244.2.105 - - [25/Feb/2023:20:49:47 +0000] "GET / HTTP/1.1" 302 11824 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 Edg/110.0.1587.56
]]>Suppose your website or application is more sluggish than usual. How do you begin to investigate the problem? There are so many causes for a slow application, but sometimes it’s because your server’s CPUs are maxed out. This guide will help you find out whether that is the case for you.
We’ll start by understanding two of the most commonly referenced metrics of resource usage on any Linux server: CPU utilization and load average. We’ll explore the difference between these two metrics and then see how to check their values using two command-line utilities, uptime
and top
. Finally, we’ll look at how to track these metrics over time with DigitalOcean Monitoring and set an alerting policy so you can get notified by email or Slack when the metrics rise above their usual values.
To follow along with this guide, you need:
stress
utility installed on your Droplet. Use apt
or dnf
to install stress
from your distribution’s repositories.The two command-line utilities featured in this guide, uptime
and top
, are available by default on most Linux distributions.
Before we look at how to check and track CPU utilization and load average, let’s consider them conceptually. What do they measure?
These two often-checked metrics are sometimes conflated because both have to do with the CPUs and both may be high when a server is apparently overburdened. But they do not measure the same thing, and they do not always directly correlate to one another. CPU utilization tells you how busy the CPUs are right now, and load average tells you how many tasks are waiting for or running on the CPUs, on average, over the recent past.
Note: Because CPU task queues can shrink and swell dramatically from one millisecond to the next, an instantaneous snapshot of CPU load is not very helpful. That is why the common command-line utilities report load average: an average of the CPU load taken over some time interval.
Imagine a server’s CPUs as cashiers in a grocery store and tasks as the customers. CPU load is a count of the total number of people in line, including those currently being served at the cash registers. CPU utilization is how intensely the cashiers are working.
Of course, these two metrics are related: one obvious reason the checkout lines could be chronically long is if the store has too slow (or too few) cashiers. But the lines may get backed up for other reasons as well. If a customer asks for a price check on an item, the cashier asks another employee to do that. The cashier may continue scanning other items while waiting for the price check, but if the check has not come back by the time the other items have been scanned—maybe the price checker stepped away for a moment—the cashier and the customer will be waiting idly while the line grows. This is like a task waiting on some slow or unavailable I/O device (i.e., a disk), and indeed, I/O-heavy workloads may lead to high load averages but low CPU utilization.
Now imagine that the price checker is gone for 30 minutes on lunch break. In the meantime, a few more customers need price checks. (Assume that an idle cashier cannot do price checks herself, just as a CPU cannot perform the functions of a disk.) The cashier sets all these customers aside and works through the remaining customers, emptying the checkout line. The cashier is now idle (~0% utilization) but there remain several customers standing around (non-zero load) who still need her time, even though she cannot complete their checkouts yet because everyone is waiting on the price checker to return. There is demand for the cashier, yet she is not busy, so if any new customer approaches the cash register, he will be served immediately.
You can probably imagine other grocery store problems that would cause wait times to grow through no fault of the cashiers, but considering those would lead us into a more thorough discussion of CPU utilization, load average, I/O wait, and CPU scheduling. Such a discussion is out of scope for this guide, and so for now, just remember:
So how does a server represent these metrics?
CPU utilization is given as a percentage of the total CPU capacity on the system. On a single-processor system, the total CPU capacity is always 100%. On a multiprocessor system, the capacity can be represented in two different ways: normalized, or non-normalized.
Normalized capacity means the combined capacity of all the processors is considered as 100%, regardless of the number of processors. If both CPUs in a two-processor system are at 50% utilization, the normalized total utilization is 50%.
Non-normalized capacity counts each processor as a unit with 100% capacity, so that a two-processor system has 200% capacity, a four-processor system has 400% capacity, and so on. If both CPUs in a two-processor system are at 50% utilization, the non-normalized total utilization is 100% (half of the 200% maximum), which could be confusing if you don’t understand that you’re looking at a non-normalized figure on a two-processor system.
Load average is given as a decimal number that, again, represents the average number of tasks waiting for—or currently being served by—all CPUs. On a four-processor system, there are four processor queues, and if, on average, the four CPUs are each serving one task with no tasks waiting in their queues, the load average will be 4.0. No problem. On the other hand, a load average of 4.0 on a single-processor system would be more worrisome. That would mean that on average, the lone processor is serving one task with three more waiting in its queue.
The values for load average are never normalized. No tool will report the four-processor-four-tasks case as having a load average of 1.0. If on average there are four total tasks running or waiting for the CPUs, the load average will always be 4.0, however many CPUs there are.
So, to make sense of the metrics you’re about to explore, you first need to know how many CPUs your server has.
You already know you have a two-core Droplet, but you can use the nproc
command with the --all
option to display the number of processors on any server. Without the --all
flag, nproc
will display the number of processing units available to the current process, which will be less than the total number of processors if any are in use.
- nproc --all
Output of nproc2
On most modern Linux distributions, you can also use the lscpu
command, which displays not only the number of processors but also the architecture, model name, speed, and much more:
- lscpu
Output of lscpuArchitecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 2
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Model name: Intel(R) Xeon(R) CPU E5-2650L v3 @ 1.80GHz
Stepping: 2
CPU MHz: 1797.917
BogoMIPS: 3595.83
Virtualization: VT-x
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 30720K
NUMA node0 CPU(s): 0,1
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl eagerfpu pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm vnmi ept fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
The optimal CPU utilization for a server depends on its role. Some servers are meant to make full use of the CPUs. Others are not. An application or batch job performing statistical analysis or cryptographic work may consistently tax the CPUs at or near full capacity, whereas a web server usually should not. If your web servers are maxing out their CPUs, do not unthinkingly upgrade to servers with more or faster CPUs. Rather, you should check that there is not malware or some other unruly process (legitimate or not) chewing up CPU cycles.
Although you never want to see sustained CPU utilization at 100% (normalized) on any server, it is appropriate for a computationally-intensive application to hover just under 100%. If it’s not, you may have more CPU power than you need, and perhaps a server downgrade (or using fewer servers) could save you money.
The optimal value for load average is anything at or below the number of CPUs. As long as a two-processor system’s load average remains below 2.0, you can be certain that there is rarely contention for CPU time. (Though the queues almost certainly swell for brief periods.)
uptime
The uptime
command is a quick way to check load average. It can be helpful because it is lightweight and runs quickly even when a system is otherwise responding slowly at the command line (i.e., due to high load).
The uptime
command shows:
- uptime
Output 14:08:15 up 22:54, 2 users, load average: 2.00, 1.37, 0.63
In this example, the command was run at 2:08pm on a server that had been up for almost 23 hours. Two users were logged in. The first load average figure, 2.00
, is the one-minute load average. Since this server has two CPUs, the load average of 2.00 means that during the minute before uptime
was run, on average, two processes were using the CPUs and no processes were waiting. The five-minute load average indicates that for that interval, there was an idle processor about 60% of the time. The 15-minute value indicates that there was even more available CPU time. The three figures together show an increase in CPU demand over the last fifteen minutes.
This utility provides a helpful glance at the load average, but to get more in-depth information, we’ll turn to top
.
top
Like uptime
, top is available on both Linux and Unix systems, but in addition to displaying the same load averages as uptime
, it provides both total-system and per-process figures for CPU utilization, memory usage, and more. Whereas uptime
runs and exits, top
is interactive. It presents a dashboard of system resource usage and stays in the foreground, refreshing every few seconds until you exit by pressing q
.
Before running top
, give your Droplet something to work on. Run the following command to place some strain on the CPUs:
- stress -c 2
Now open another terminal to your Droplet and run top
:
- top
This will bring up an interactive screen that shows a summary of resource utilization as well as a table of all system processes. Let’s look first at the summary information at the top.
Here are the first five lines in top
:
Outputtop - 22:05:28 up 1:02, 3 users, load average: 2.26, 2.27, 2.19
Tasks: 109 total, 3 running, 106 sleeping, 0 stopped, 0 zombie
%Cpu(s): 99.8 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.2 st
MiB Mem : 1975.7 total, 1134.7 free, 216.4 used, 624.7 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 1612.0 avail Mem
The first line is almost identical to the output of uptime
. As with uptime
, the one, five, and fifteen minute load averages are displayed. The only difference between this line and the output of uptime
is that the beginning of the line shows the command name, top
, and the time is updated each time top
refreshes the data.
The second line summarizes the states of all system tasks: the total number of processes, followed by how many of them are running, sleeping, stopped, or zombie.
The third line tells you about the CPU utilization. These figures are normalized and displayed as percentages (without the % symbol) so that all the values on this line should add up to 100% regardless of the number of CPUs. Since you started the stress
command with the -c
(CPU) option just before running top
, the first figure—us
, or user processes—should be near 100%.
The fourth and fifth lines tell you about memory and swap usage, respectively. This guide does not explore memory and swap usage, but it’s there for you when you need it.
This header block is followed by a table with information about each individual process, which we’ll look at in a moment.
In the example header block below, the one-minute load average exceeds the number of processors by .77, indicating a short queue with a little bit of wait time. The CPUs are 100% utilized and there’s plenty of free memory.
Outputtop - 14:36:05 up 23:22, 2 users, load average: 2.77, 2.27, 1.91
Tasks: 117 total, 3 running, 114 sleeping, 0 stopped, 0 zombie
%Cpu(s): 98.3 us, 0.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.2 si, 1.3 st
KiB Mem : 4046532 total, 3244112 free, 72372 used, 730048 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 3695452 avail Mem
. . .
Let’s look at each of the fields on the CPU line in more depth:
us, user: time running un-niced user processes
This category refers to user processes that were started with no explicit scheduling priority.
Linux systems use the nice
command to set a process’s scheduling priority, so “un-niced” means that nice
wasn’t used to change the default value. The user and nice values account for all the user processes. High CPU use in this category may indicate a runaway process, and you can use the output in the process table to identify any culprits.
sy, system: time running kernel processes
Most applications have both user and kernel components. When the Linux kernel is doing something like making system calls, checking permissions, or interacting with devices on behalf of an application, the kernel’s use of the CPU is displayed here. When a process is doing its own work, it will appear in either the user figure described above or, if its priority was explicitly set using the nice
command, in the nice figure that follows.
ni, nice: time running niced user processes Like user, nice refers to CPU time that does not involve the kernel. But with nice, the scheduling priority for these tasks was set explicitly. The niceness (priority) of a process is indicated in the fourth column in the process table under the header NI. Processes with a niceness value between 1 and 20 that consume a lot of CPU time are generally not a problem because tasks with normal or higher priority will be able to get processing power when they need it. But if tasks with elevated niceness, between -1 and -20, are taking a disproportionate amount of CPU time, they can easily affect the responsiveness of the system and warrant further investigation. Note that many processes that run with the highest scheduling priority, -19 or -20 depending on the system, are spawned by the kernel to do important tasks that affect system stability. If you’re not sure about the processes you see, it’s prudent to investigate them rather than killing them.
id, idle: time spent in the kernel idle handler This figure indicates the percentage of time that the CPU was both available and idle. A system is generally in good working order with respect to CPU when the user, nice, and idle figures combined are near 100%.
wa, IO-wait: time waiting for I/O completion The IO-wait figure shows how much time a processor has spent waiting for I/O operations to complete. Read/write operations for network resources like NFS and LDAP will count in IO-wait as well. As with idle time, spikes here are normal, but any kind of frequent or sustained time in this state suggests an inefficient task, a slow device or network, or a potential hard disk problem.
hi, hardware interrupts: time spent servicing hardware interrupts This is the time spent on physical interrupts sent to the CPU from peripheral devices like disks and hardware network interfaces. When the hardware interrupt value is high, you may have a peripheral device that has gone bad.
si, software interrupts: time spent servicing software interrupts Software interrupts are sent by processes rather than physical devices. Unlike hardware interrupts that occur at the CPU level, software interrupts occur at the kernel level. When the software interrupt value is high, investigate the specific processes that are using the CPU.
st, steal: time stolen from this vm by the hypervisor The steal value shows how long a virtual CPU spends waiting for a physical CPU while the hypervisor is servicing itself or a different virtual CPU. Essentially, the amount of CPU use in this field indicates how much processing power your VM is ready to use, but which is not available to your application because it is being used by the physical host or another virtual machine. Generally, seeing a steal value of up to 10% for brief periods of time is not a cause for concern. Larger amounts of steal for longer periods of time may indicate that the physical server has more demand for CPU than it can support.
Now that we’ve looked at the summary of CPU usage supplied in top
’s header, we’ll take a look at the process table that appears below it, paying attention to the CPU-specific columns.
Here are the first six lines of the process table in some example top
output:
Output
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9966 sammy 20 0 9528 96 0 R 99.7 0.0 0:40.53 stress
9967 sammy 20 0 9528 96 0 R 99.3 0.0 0:40.38 stress
7 root 20 0 0 0 0 S 0.3 0.0 0:01.86 rcu_sched
1431 root 10 -10 7772 4556 2448 S 0.3 0.1 0:22.45 iscsid
9968 root 20 0 42556 3604 3012 R 0.3 0.1 0:00.08 top
9977 root 20 0 96080 6556 5668 S 0.3 0.2 0:00.01 sshd
. . .
All processes on the server, in any state, are listed below the header. By default, the process table is sorted by the %CPU
column, so you will see the processes that are consuming the most CPU time at the top.
The %CPU
values are presented as a percent value, but unlike the CPU utilization figures in the header, those in the process table are not normalized, so on this two-core system, the total of all the values in the process table should add up to 200% when both processors are fully utilized.
Note: If you prefer to see normalized values here, press SHIFT+I
to switch the display from Irix mode to Solaris mode. Now the sum of the %CPU
values will not exceed 100%. When you switch to Solaris mode, you’ll get a brief message that Irix mode is off, and in this example, the values for your stress
processes will switch from nearly 100% each to around 50% each.
Output PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10081 sammy 20 0 9528 96 0 R 50.0 0.0 0:49.18 stress
10082 sammy 20 0 9528 96 0 R 50.0 0.0 0:49.08 stress
1439 root 20 0 223832 27012 14048 S 0.2 0.7 0:11.07 snapd
1 root 20 0 39832 5868 4020 S 0.0 0.1 0:07.31 systemd
You can interactively control top
—and the server’s processes—using different keystrokes, including:
SHIFT-M
to sort processes by memory usage.SHIFT-P
to sort processes by CPU usage.k
, followed by a process ID (PID) and a signal, to kill an unruly process. After entering a PID, try a signal of 15 (SIGTERM) first, which is the default. Do not use 9 (SIGKILL) carelessly.r
, followed by a process ID, to renice (reprioritize) a process.c
to toggle the command descriptions between a short description (the default) and the full command string.d
to change the refresh interval for the figures in top
.And many more. Read top
’s man page (man top
) for more details.
So far, you’ve examined two Linux commands that are commonly used to look into load average and CPU utilization. As useful as these tools are, they only provide a glimpse at the state of your server over a brief window of time. There’s only so long you can sit there and watch top
. To get a better idea of long-term patterns of server resource usage, you need some kind of monitoring solution. In the next section, you’ll briefly look into the monitoring tools available at no additional cost for DigitalOcean Droplets.
By default, all Droplets display Bandwidth, CPU, and Disk I/O graphs when you click the Droplet name in the Control Panel. You don’t have to install the DigitalOcean Agent to get these three graphs.
These graphs help you visualize each resource’s use for the last 1 hour, 6 hours, 24 hours, 7 days, and 14 days. The CPU graph provides CPU utilization information. The dark blue line indicates CPU use by user processes. The light blue indicates CPU use by system processes. The values on the graphs and their detail are normalized so that the total capacity is 100% regardless of the number of virtual cores.
The graphs let you see whether you’re experiencing an intermittent or sustained change in usage and are helpful in spotting changes in a server’s CPU usage pattern.
To supplement these default graphs, you can install the DigitalOcean Agent on a Droplet to collect and display additional data like load average, disk usage, and memory usage. The Agent also allows you to set Alert Policies for the system. How To Install and Use the DigitalOcean Agent for Monitoring can help you get set up.
Once the Agent is installed, you can set alert policies to notify you about resource usage. The thresholds you choose will depend on the server’s typical workload.
Note: when you install the DigitalOcean Monitoring Agent:
If you’re using a Droplet primarily for integrating and soak testing code, you might set an alert that’s just slightly above historical patterns to detect if new code pushed to the server has increased CPU usage. For the graph above where CPU never reaches 1%, a threshold of 1% CPU use for 5 minutes could provide early warning about code-based changes affecting CPU use.
In the DigitalOcean Control Panel, under the MANAGE menu on the left, click Monitoring. Under the Resource Alerts tab (which is highlighted by default), click Create Resource Alert. Fill out the form and click Create resource alert.
On most systems, there’s a good chance this threshold would be completely inappropriate, but if you can determine typical CPU utilization during normal workloads by examining CPU graphs going back a few weeks, you can create alerts with a threshold slightly above the normal utilization and learn early on when new code or new services impact the CPU utilization.
You might also want to set a threshold far above the average, one that you consider critical and which would warrant prompt investigation. For example, if a server that experienced sustained CPU use of 50% for five-minute intervals pretty regularly suddenly shot up to 90%, you might want to log in immediately to investigate the situation. Again, the thresholds are specific to the workload of the system.
In this article, we’ve explored uptime
and top
, two common Linux utilities that provide insight into CPU utilization and load average on Linux systems, as well as how to use DigitalOcean Monitoring to see the historical CPU utilization on a Droplet and alert you to changes and emergencies. To learn more about DigitalOcean Monitoring, read An Introduction to Metrics, Monitoring, and Alerting. You may also want to read our documentation.
For help in deciding what kind of Droplet you need for your workload, read Choosing the Right Droplet Plan.
]]>However, the very last step (use local computer to connect to the DigitalOcean-Droplet) on the web-browser keeps failing :
I am getting this : http://localhost:8891/?token=XXXXXXXXXXXXX But when I past it into my local Chrome web-browser, I just keep getting error message : LocalHost refused to connect.
HELP!
]]>What’s the best way to manage Moodle sites, hosted on DO server? We recently took over a client with two Moodle sites. We have never managed these type of sites before and moved the two sites from a shared hosting environment (with cPanel) to a new DO server. We’ve since worked with a freelance sysdev who claims that it would have been better to keep the sites on the old, shared server, as cPanel is better when it comes to things such as updating the Moodle app. Is he correct and what can we do now, seeing as we don’t really want to setup a new shared hosting server just for these two Moodle sites. The sysdev has told me that he spent up to 4 hours doing backups, updating the two Moodle sites (from 4.0.2 - 4.1) and also setting cron jobs, where she said it would only have taken 15 minutes with cPanel. Is he right about cPanel being better/easier for Moodle and what can we do (within DO) to make future updates / service easier and faster?
Thanks!
]]>
]]>I read all of the answers about this question before and I think this feature has not been implemented yet.
This feature is a must. We should be able to redirect our requests to another place. How can we make it ? I don’t want to use nginx etc. If I want then I would use droplet. This is the reason for me to use DO apps platform. If you don’t do this then what is the purpose of APPS platform ?
]]>I’ve encountered an error during a routine Power Cycle, and now I’m getting the message, “Powering on Droplet Issue.”
Full text reads: Sorry, we had a problem turning your Droplet on. Please try again.
No further details are provided. The first half of the power cycle (turning the droplet off) seems to have worked fine, which means I’m at DigitalOcean’s mercy here – I’m not sure what I can do, beyond continuously trying to Turn On the Droplet and expect a different result.
I’ve filed a Support ticket as well, but writing this here as well in case others have had similar issues in the past and can share how they resolved it! Any help is welcome here, thank you!
running 8 GB Memory / 160 GB Disk / Ubuntu
]]>Currently I am running into a problem where the collectstatic command will not collect all my static files.
settings.py - static files
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/4.1/howto/static-files/
STATICFILES_DIRS = [
os.path.join(BASE_DIR, 'static'),
]
AWS_ACCESS_KEY_ID = 'my_access_key'
AWS_SECRET_ACCESS_KEY = 'my_secret_key'
AWS_STORAGE_BUCKET_NAME = 'lrev'
AWS_S3_ENDPOINT_URL = 'https://fra1.digitaloceanspaces.com'
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
STATICFILES_STORAGE = 'custom_storages.StaticStorage'
DEFAULT_FILE_STORAGE = 'custom_storages.MediaStorage'
# Use AWS_S3_ENDPOINT_URL here if you haven't enabled the CDN and got a custom domain.
STATIC_URL = '{}/{}/'.format(AWS_S3_ENDPOINT_URL, 'static')
STATIC_ROOT = '/static/'
MEDIA_URL = '{}/{}/'.format(AWS_S3_ENDPOINT_URL, 'media')
MEDIA_ROOT = '/media/'
TEMP = "{}/{}".format(MEDIA_ROOT, 'temp')
My Static Files:
static
- accounts
- core
- images
- lrdc
- terms
- jquery.js
Digital Ocean Spaces:
static
- images
- images
- admin
- jquery.js
Any help will be greatly appreciated! Thank you :D
]]>I am currently getting this error:
File "G:\Projects\Python\Django\laterevision\src\manage.py", line 22, in <module>
main()
File "G:\Projects\Python\Django\laterevision\src\manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "G:\Projects\Python\Django\laterevision\venv\lib\site-packages\django\core\management\__init__.py", line 446, in execute_from_command_line
utility.execute()
File "G:\Projects\Python\Django\laterevision\venv\lib\site-packages\django\core\management\__init__.py", line 440, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "G:\Projects\Python\Django\laterevision\venv\lib\site-packages\django\core\management\base.py", line 402, in run_from_argv
self.execute(*args, **cmd_options)
File "G:\Projects\Python\Django\laterevision\venv\lib\site-packages\django\core\management\base.py", line 448, in execute
output = self.handle(*args, **options)
File "G:\Projects\Python\Django\laterevision\venv\lib\site-packages\django\contrib\staticfiles\management\commands\collectstatic.py", line 209, in handle
collected = self.collect()
File "G:\Projects\Python\Django\laterevision\venv\lib\site-packages\django\contrib\staticfiles\management\commands\collectstatic.py", line 135, in collect
handler(path, prefixed_path, storage)
File "G:\Projects\Python\Django\laterevision\venv\lib\site-packages\django\contrib\staticfiles\management\commands\collectstatic.py", line 378, in copy_file
self.storage.save(prefixed_path, source_file)
File "G:\Projects\Python\Django\laterevision\venv\lib\site-packages\django\core\files\storage.py", line 56, in save
name = self._save(name, content)
File "G:\Projects\Python\Django\laterevision\venv\lib\site-packages\storages\backends\s3boto3.py", line 457, in _save
obj.upload_fileobj(content, ExtraArgs=params, Config=self._transfer_config)
File "G:\Projects\Python\Django\laterevision\venv\lib\site-packages\boto3\s3\inject.py", line 725, in object_upload_fileobj
return self.meta.client.upload_fileobj(
File "G:\Projects\Python\Django\laterevision\venv\lib\site-packages\boto3\s3\inject.py", line 636, in upload_fileobj
return future.result()
File "G:\Projects\Python\Django\laterevision\venv\lib\site-packages\s3transfer\futures.py", line 103, in result
return self._coordinator.result()
File "G:\Projects\Python\Django\laterevision\venv\lib\site-packages\s3transfer\futures.py", line 266, in result
raise self._exception
File "G:\Projects\Python\Django\laterevision\venv\lib\site-packages\s3transfer\tasks.py", line 139, in __call__
return self._execute_main(kwargs)
File "G:\Projects\Python\Django\laterevision\venv\lib\site-packages\s3transfer\tasks.py", line 162, in _execute_main
return_value = self._main(**kwargs)
File "G:\Projects\Python\Django\laterevision\venv\lib\site-packages\s3transfer\upload.py", line 758, in _main
client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
File "G:\Projects\Python\Django\laterevision\venv\lib\site-packages\botocore\client.py", line 514, in _api_call
return self._make_api_call(operation_name, kwargs)
File "G:\Projects\Python\Django\laterevision\venv\lib\site-packages\botocore\client.py", line 938, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown
My settings for spaces look like this:
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/4.1/howto/static-files/
AWS_ACCESS_KEY_ID = 'XXX'
AWS_SECRET_ACCESS_KEY = 'XXXXXXX'
AWS_STORAGE_BUCKET_NAME = 'lrev'
AWS_S3_ENDPOINT_URL = 'https://fra1.digitaloceanspaces.com'
# I enabled the CDN, so you get a custom domain. Use the end point in the AWS_S3_CUSTOM_DOMAIN setting.
AWS_S3_CUSTOM_DOMAIN = 'lrev.fra1.digitaloceanspaces.com'
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
AWS_DEFAULT_ACL = 'x-amz-acl'
STATICFILES_STORAGE = 'custom_storages.StaticStorage'
DEFAULT_FILE_STORAGE = 'custom_storages.MediaStorage'
# Use AWS_S3_ENDPOINT_URL here if you haven't enabled the CDN and got a custom domain.
STATIC_URL = '{}/{}/'.format(AWS_S3_CUSTOM_DOMAIN, 'static')
STATIC_ROOT = 'static/'
MEDIA_URL = '{}/{}/'.format(AWS_S3_CUSTOM_DOMAIN, 'media')
MEDIA_ROOT = 'media/'
and custom_storages looks like this:
from storages.backends.s3boto3 import S3Boto3Storage
class StaticStorage(S3Boto3Storage):
bucket_name = 'lrev'
location = 'static'
class MediaStorage(S3Boto3Storage):
bucket_name = 'lrev'
location = 'media'
Any help would be appriciated!
]]>I’m looking for my SFTP credentials, but i don’t find them… I tried with :
Host: my public id User: root or my Digital Ocean email login Password: my Digital Ocean password
Thanks a lot! Vincent
]]>Thanks
]]>Whenever I try to upload file bigger than 60MB node server gets killed.
My Nginx Default file config.
proxy_pass http://localhost:8081;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_cache_bypass $http_upgrade;
proxy_max_temp_file_size 0;
proxy_connect_timeout 1800s;
proxy_send_timeout 1800s;
proxy_read_timeout 1800s;
proxy_buffer_size 128k;
proxy_buffers 128 256k;
proxy_busy_buffers_size 256k;
proxy_redirect off;
proxy_request_buffering off;
proxy_buffering off;
My nginx.conf file
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 1800;
types_hash_max_size 2048;
client_max_body_size 4096M;
fastcgi_read_timeout 1800s;
client_body_timeout 1800s;
client_header_timeout 1800s;
I am trying to solve this issue for hours and stuck.
Please help me out here!!
]]>This morning I have an issues with my droplet. It suddenly power off and I cannot access my website. I tried to power on my droplet, but it keeps loading. How long does it take to power on droplet?
]]>However I found out the issue ,** the issue is images from external sources(aws S3 bucket , cloudinary ) are getting loaded very slow**.
s3 image url format -> https://xxxx.s3.yyyy.amazonaws.com/zzzz.jpeg
xxxx: bucket name, yyyy:region zzzz:filename
I have already defined s3 domain in next.config.org as per nextjs docs.
But local images are getting loaded very fast.
I have gone through every possible docs of digital ocean but did not find a solution.
If anybody know how to resolve it Please let me know. .
]]>Regards and thanks
]]>Could anybody please advise how could I resolve this issue.
]]>Systemd Description=goapp
[Service]
Type=simple
Restart=always
RestartSec=5s
ExecStart=/home/.../goapp/main
[Install]
WantedBy=multi-user.target
I got this error
goapp.service - rediate
Loaded: loaded (/lib/systemd/system/goapp.service; disabled;
vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Thu
2022-09-29 08:14:10 UTC; 66ms ago
Process: 21628
ExecStart=/home/.../go/goapp/main (code=exited, status=2)
Main PID: 21628 (code=exited, status=2)
CPU: 9ms
]]>We manage the DNS in digitalocean and already set up our A records. In our domain hosting we already point our nameservers to our droplets’ IP Addresses but none seems to be working.
Am I missing something or what? Please help
domain is stalinks.com
]]>Authors who contribute to our growing collection of tutorials on open-source software deployment, configuration, and development will also have the opportunity to give back to a growing list of tech-focused nonprofits. Our goals are to enable authors to give back to the DigitalOcean and open-source communities, and to strengthen these communities by supporting the work of charities, nonprofits, and activists.
It all starts with a proposal. When you apply to write with us, you’ll submit a a proposal with the following details:
Once we receive your proposal, we’ll review your proposal as a team. If we’re interested in your proposal, we’ll either accept it as is or work with you to refine it further.
Once we agree on the topic and outline, we’ll send you a contract to sign which lets us publish your work. That’s also when we’ll ask for your contact and payment details.
Then you’ll begin writing in Markdown, turning your outline into a full article. When your article is done, we ask that you thoroughly test it by reading through it and following it as a reader would. We will provide testing credit for this. You won’t have to incur any cost to test your tutorials.
When your draft is complete and tested, you’ll submit it to us, and the draft will be entered into our editorial queue. All submitted drafts enter the queue in the order we receive them. We sometimes have higher than normal volumes of tutorials, so it may take a little time before an editor can work on your draft.
When an editor is available, they’ll contact you to let you know they’ve started working on your tutorial, and will then verify that your tutorial is technically correct by following it as written. Our editors all have technical backgrounds and will look for technical best practices as they test your article.
If everything works as written, your editor will then do a thorough review of the structure, tone, and style of your article. When they’re done, they will provide editorial feedback on your article and send you feedback.
When you receive the feedback, you’ll address it and send it back.
Your editor will then review it again, sending it back to you if there are further revisions necessary. Once there are no more revisions to make, another editor on the team reviews the tutorial for formatting and other issues.
Once your article is ready for publication, we’ll publish the tutorial, send you the link, and process your payment. Your editor will then encourage you to submit another tutorial idea.
Our team reviews each proposal we receive. We look for clear explanations, strong writing ability, and solid teaching skills. Our tutorials are aimed at all audiences, and we strongly prefer writing samples that follow our tutorial style. Unfortunately, because of the high volume of samples we receive, we cannot accept every author who applies and we cannot provide individual feedback on each sample we review.
Content on DigitalOcean must be original, first-run content that’s not published anywhere else. We pay authors for content and then we relicense it under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. However, you can submit your existing tutorial as your writing sample when you apply to write with us.
The typical payout for community authors in the Write for DOnations program is $300 for typical tutorial content. We then make an additional payout in the form of a contribution from DigitalOcean to the author’s choice of tech-focused nonprofits from our list of funds.
We will send payment for your article when we publish your final draft.
You can choose to receive your payout via PayPal or in DigitalOcean credit. You must have a PayPal account that can receive funds.
DigitalOcean will make donations using Bright Funds on your behalf, but you’ll tell us which fund or charity that will receive the donation.
Unfortunately we can only make payments via PayPal at this time, and only directly to PayPal addresses that can receive money as well as send it.
If you cannot receive payment via PayPal, we also offer the same amount in DigitalOcean credits.
In response to the current spread of COVID-19, we have put together the following fund focused on fighting the global pandemic and providing relief to those affected:
We’ve curated four tech-related funds, which are collections of nonprofits with similar missions:
Free and Open Source Fund, which includes the Open Source Initiative, Apache Software Foundation, FreeBSD Foundation, and Software in the Public Interest.
Diversity in Tech Fund, which includes Open Sourcing Mental Illness, Girls Who Code, Out In Tech, Techbridge Girls, Society of Women Engineers, Project Include, /dev/color, and Code 2040.
Open Internet/Free Speech Fund, which includes the Electronic Frontier Foundation, Mozilla Foundation, Wikimedia Foundation, Creative Commons, Code Labs for America, Owasp Foundation, Internet Archive, and Free Press.
Tech Education Fund, which includes IEEE, Code.org, Techsoup Global, NPower, Vets Who Code, and Computer History Museum.
We are continuously exploring additional charities and funds which may be a good match.
You can choose to donate to one of the funds we support, which splits the donation equally among the nonprofits in the fund. You can alternatively choose to donate the full amount to one specific nonprofit from within those funds.
At this time, it is not possible for authors to select a charity or nonprofit that is not on our list, but we hope to expand the program in the future to include this capability.
Because DigitalOcean will be processing the donation through Bright Funds, it won’t be possible for you to claim it as a charitable deduction on your tax return.
At this time, we ask all first-time authors to first write an original tutorial in order to gain a better understanding of our style and technical requirements.
Due to legal reasons, we can only work with authors 18 and older.
At this time, we are not accepting translations for publication. You may publish a translation of your own on your own web site, using the guidelines described in the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Tutorials on the DigitalOcean Community site are published under a Creative Commons license – specifically, an Attribution-NonCommercial-ShareAlike 4.0 license. This means that you can republish your content as long as you give credit (we prefer a canonical link), share it under the same license, and it is not for commercial purposes.
There are two ways to share thoughts or questions about the program. The first is to write to us directly at writefordonations@digitalocean.com. Though we may not be able to reply to each email, we will read all of them and use your feedback to guide our future work. If you have a question that you think may help others, please feel free to leave it in the comments section below.
]]>Now I can access by ipv4-address. but I want to setup a subdomain to this django droplet.
So I created a new record with subdomain eg. dev.domainname.com
but I can see only this screen
Welcome to nginx!
If you see this page, the nginx web server is successfully installed and working. Further configuration is required.
For online documentation and support please refer to nginx.org. Commercial support is available at nginx.com.
Thank you for using nginx.
How can I do now?
Thanks for your any advice.
Thanks
]]>When I did -
sudo do-release-upgrade
It was upgrading but it got stuck on this error -
Error during update
A problem occurred during the update. This is usually some sort of
network problem, please check your network connection and retry.
W:Updating from such a repository can't be done securely, and is
therefore disabled by default., W:See apt-secure(8) manpage for
repository creation and user configuration details., W:GPG error:
http://dl.google.com/linux/mod-pagespeed/deb stable Release: The
following signatures couldn't be verified because the public key is
not available: NO_PUBKEY 78BD65473CB3BD13, E:The repository
'http://dl.google.com/linux/mod-pagespeed/deb stable Release' is not
signed.
Restoring original system state
Aborting
Reading package lists... Done
Building dependency tree
Reading state information... Done
=== Command detached from window (Sun Sep 11 13:42:03 2022) ===
=== Command terminated with exit status 1 (Sun Sep 11 13:42:13 2022) ===
I’ve tried the following fix but it didn’t work - https://itsfoss.com/solve-gpg-error-signatures-verified-ubuntu/
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 78BD65473CB3BD13
What can I do to fix?
Please advise.
Thank you
]]>Whereas an internet speed test on my computer at home, shows i’m able to download at 12MB/s
Also if I try a speed test from the digitalocean server,
so, ookla commandline speed test
Retrieving speedtest.net configuration… Testing from Digital Ocean (…)… Retrieving speedtest.net server list… Selecting best server based on ping… Hosted by EGI Hosting (Santa Clara, CA) [4.38 km]: 2.584 ms Testing download speed… Download: 5274.92 Mbit/s Testing upload speed… Upload: 1931.09 Mbit/s
That’s slow, less than 1MB/s download and 0.2MB/s upload
So when i’m Downloading from the VPS it’s surprising I even manage to get 1MB/s 'cos i’d be limited by the VPS only being able to upload 0.2MB/s
My computer at home though is capable of 12MB/s
Are there digitalocean servers that would get better results from an internet speed test?
]]>Now i want sub domain, lets say “abc.example.com” to point to different droplet which is not under loadbalancer.
What i did: Under domain name “example.com” i added A record, with hostname “abc” and in will direct to field i added “ip address of the droplet which is not under loadbalancer”. Doing this my hostname “abc.example.com” is not resolved. tried doing ping, returned no ping and tried dig it does not have answer.
Is there any way i can do this?
]]>I was trying to do an apt install and got the following error repeated over ad over.
“It is held by process 1 Waiting for cache lock: Could not get lock /var/lib/dpkg/lock-frontend.”
I googled it, and was told that an update was probably taking place. I’ve waited and tried again several times. Do I just need to wait longer? Or does DigitalOcean for some reason not like tmux installations?
Thank you for your time norm
]]>but when use a CHR image router mikrotik work fine. but i want use my custon image mikrotik x86 because it is full licensed.
]]>This guide is an effort to summarize established best practices and strong recommendations for authors of DigitalOcean tutorials. It is intended to provide a foundation for consistency, technical correctness, and ease of use in DigitalOcean’s instructional material.
This guide is by nature both a work in progress and an opinionated document, based on the growing experience of in-house technical writers and editors, community managers, and engineers. Its recommendations are subject to change, and are written specifically for educational content with a broad range of readers and end-users in mind.
Many tutorials will rely on existing tutorials as prerequisites. Put all prerequisites for the article (including any nested prerequisites for the prerequisites) into the article, rather than having deeply nested prereq lists.
In rough, descending order of preference, use the following installation mechanisms:
curl | bash
patterns) require a judgement call on whether or not to use them.wget
or curl
install scripts piped to the shell, with an appropriate warning about inspecting scripts.In general, avoid unnecessary complication. For unpackaged software installed from source or binaries, you should generally accept the default installation prefix unless it’s very unusual or introduces conflicts.
An init script, conforming to official recommendations for the distribution, should be given for service-oriented software, if not provided by the package or other installation method.
On Linux systems, put self-contained binaries or directories in /opt
and standalone scripts in /usr/local/bin
.
Ubuntu and Debian systems should have unattended-upgrades
with at least security updates installed and configured. We recommend no auto-reboot or auto-update all, given context.
We generally recommend using the apt
command for Ubuntu 18.04 and newer. Change apt-get
and apt-cache
to use apt
. When updating apt commands, do not use the -y
flag; readers should be guided through any necessary inputs and prompts.
For CentOS and Rocky Linux tutorials, we now recommend using dnf
, which has superseded yum
and provides better performance.
If a tutorial relies on the latest updates, call update
and upgrade
during Step 1 setup or as needed. Call update
first so that your server pulls the latest versions of packages. When including upgrade
, which downloads and installs new versions for every package, please be aware that some users may choose to keep certain packages at a lower version.
Make sure to use native init system commands, even when legacy compatibility commands are available. For instance, use sudo systemctl start [service_name]
even though sudo service [service_name] start
will work.
Provide information about how to enable or disable the service from starting at boot. Indicate how to inspect outcome of service-related commands when not clearly indicated by the interface (journalctl -u
, systemctl status
, etc).
Prefer restarts over reloads for services as a rule of thumb. In most cases, it’s more important to ensure a known state than avoid a split-second service interruption, and restarts are also more useful in the case of a complete service failure.
Unless it’s part of a config management workflow, prefer user-data scripts, and prefer cloudinit scripts to bash scripts in user-data in most cases.
Explain where and how to access logs for installed services. Where relevant, explain systemctl
and journalctl
commands for checking service status and log output. Where possible, offer concise suggestions for diagnosing common failure cases.
Make sure to handle log rotation for any cases where it’s not handled by packages or other installation mechanisms.
For following plaintext log files, use tail -F
, not tail -f
, as the latter will not track a file across renames and might cause confusion if logs are rotated while a user is watching them.
Create sudo
users instead of using root directly. Reference the appropriate initial server setup guides which explain this task as a prerequisite.
On Debian-based distributions, add and remove users with adduser sammy
and deluser --remove-home sammy
respectively; on RHEL-based distributions, use adduser sammy
(set a password with passwd sammy
if necessary) and userdel -r sammy
.
Grant sudo
privileges with usermod -aG sudo sammy
on Ubuntu. CentOS is a little more complicated. Modern versions use usermod -aG wheel sammy
, but some versions require visudo
to uncomment wheel
group permissions first. Specifically, on CentOS 5, sudo
needs to be installed and the wheel group needs to be uncommented with visudo
; on CentOS 6, sudo
is already installed, but wheel needs to be uncommented; CentOS 7 has sudo
and the wheel group is already set up.
When using privilege escalated commands, make sure to test them as written. To pass environment variables through sudo
, use sudo -E command_to_run
(if trusted with entire environment) or sudo FOO=BAR command_to_run
. For instances that require a root shell, use sudo -i
. For instances that require redirection, use tee -a
to append to rather than replace the destination file: [sudo] command_to_run | sudo tee [-a] file_to_change
.
For interactive shells, assume Bash on GNU/Linux systems, mentioned explicitly when relevant. On FreeBSD, use tcsh, as it’s available out of the box and has useful features.
For text editors, we include the copy “use [preferred] or your favorite text editor”, and include the following beginner-friendly editors in commands for those copy and pasting. On Linux, we default to nano
; on FreeBSD, we default to ee
. vi(m) is permissible, but avoid it in introductory topics where it might present a stumbling block for beginners.
For file transfer, we generally recommend sftp
in most cases for its interactive and scp-alike uses, though it lacks push functionality, so scp
is acceptable as well. rsync
is useful for backups and large transfers (or many small files). Do not use FTP under any circumstances. We also make an effort to standardize on curl
over wget
because of its robustness. wget
’s advantage is mostly recursive download (i.e. a special use case which is not common for our kind of content).
On machines that ship with iproute2
utilities, we prefer them to the net-tools
suite, which is considered obsolete. In general, iproute2
utilities like ss
will have better support for multiple interfaces, IPv6, new kernel functionality, etc. So likewise, we should use ip route
over route
, ip addr show
over ifconfig
, etc. Sometimes the older utilities output is a bit cleaner by default, but the output itself is a bit less trustworthy since they don’t handle edge cases as well. When possible, we will control the more verbose output using available flags.
Within the context of systems administration tutorials, generally avoid lengthy custom scripts and long shell scripts.
Author-written scripts (and possibly other resources) should live in a per-article repository in the do-community GitHub account with a link back to the published tutorial. Follow good scripting practices in general. For example, put any variables the user will have to fill in at the top of the script, preferably in a well marked section. Provide in-line comments where needed to provide human-readable scripts. Ensure that your descriptions of the code are not exclusively provided in comments, but that you also provide prose descriptions with further explanation than appears in comments.
Prefer /bin/sh
to bash
and avoid Bash-specific features when portability or cross-platform reuse are a concern. Use the shell and coreutils/standard Unix tools for small tasks; avoid introducing new dependencies purely for glue-language tasks unless the benefits are substantial. Prefer #!/usr/bin/env interpreter
to #!/path/to/interpreter
.
In general, use cron
for scheduling recurring tasks, but systemd timers are also acceptable.
Use your_server_ip
and your_domain
when possible instead of documentation IP ranges or example.com
.
When downloading scripts or data, ensure that the user is in a writeable directory or paths are explicitly specified. For files which should be available for reference or reuse, use the user’s home directory, unless they belong in some standard well-defined path elsewhere on the filesystem (such as /opt
or /etc
). For throwaway files, use /tmp
.
When a tutorial relies on a specific directory, be sure to provide the cd
commands that route the reader to that directory before they run commands.
We recommend the Debian-style configuration directories for distributions that don’t structure it that way by default. Always test configuration changes (Apache uses sudo apachectl configtest
, and Nginx uses sudo nginx -t
).
/var/www/html
should be used as the document root for all web servers. Nginx’s /usr/share/nginx/html
default should be changed because that directory is owned by and can potentially be modified by package updates. This is no longer a problem in Ubuntu 16.04, but will remain relevant for previous releases.
For Apache or Nginx tutorials, use dedicated virtual host blocks instead of editing the default config files. This route can avoid common mistakes and maintain the default files as the fallback configuration as intended.
Encrypt and authenticate all connections between systems. Do not encourage (explicitly or implicitly) users to send credentials or transmit non-public data in the clear.
Specifically, passwords and key material must not be transmitted over unsecured connections. Database connections, logging, cluster management, and other services should ideally be encrypted at all times.
Web-based control panels must be served over HTTPS connections, and TLS/SSL should be used for services where it’s supported. All web servers should be HTTPS-enabled (or capable, at least). Use a certbot prerequisite to provide SSL certification. Public-facing services like plain HTTP are permissible, as users may still want or need to offer them, but should be strongly discouraged in the general case, especially for dynamic content. For articles that provide a plain HTTP connection, add a note or warning label to discourage plain HTTP and encourage HTTPS.
Avoid practices which constitute low- benefit security through obscurity or theatrics, like changing the default SSH port. Do configure a firewall. Our distro-specific recommendations are ufw
for Ubuntu, iptables
for Debian, and firewalld
for CentOS. However, iptables
is most consistent across platforms, and has many tools that hook into it.
Maintain the default SSH port as a norm. Changing the port should only be done in specific situations where that is a primary concern.
Disable password authentication and use key-only authentication for root or, alternatively, disable root login completely. Use strong SSH keys: at least 2048-bit RSA but recommended 4096; ECDSA is no longer recommended for technical reasons; and Ed25519 and elliptic curve algorithms are not widely supported enough.
Use passphrases for any interactive keys, but not for non-interactive processes. Set up or copy and change ownership on SSH keys from the root account to the user home directory. Install fail2ban where it’s practical.
Note that while SSH Agent Forwarding is necessary for normal workflows on platforms like CoreOS, it comes with some security concerns. Essentially, anyone with permissions on your host will be able to use the forwarding socket to connect to your local ssh-agent.
We strongly encourage the use of Let’s Encrypt for ease of use, and recommend TLS. Do use strong SSL security; look at https://cipherli.st/ (both modern and legacy recommendations).
For hosts without a domain name, we suggest a self-signed certificate of adequate strength. Again, look at https://cipherli.st/ plus the self-signed cert creation used in guides like this Self-Signed Certification on Nginx guide. Set up a strong Diffie-Hellman key when enabling encryption, as in Self-Signed SSL Certifications on Apache and Nginx.
We recommend VPNs as a solution for general encrypted communication between servers. VPNs become increasingly valuable when multiple services need to be protected between servers; instead of encrypting each service individually, all internal communication can be piped to the VPN. This approach is particularly useful if the services in question don’t support native encryption.
We generally recommend Tinc over OpenVPN for server-to-server communication. You can read this article on Ansible and Tinc for more details and implementation.
This is inherently an opinionated, living document, and will be updated often. Tech changes quickly and we at DigitalOcean do our best to keep up, but if you notice any errors or have feedback, we’ll be monitoring the comments below.
]]>ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=1/FAILURE)
systemd[1]: mysql.service: Control process exited, code=exited status=1
Is there any way to repair or reset mysql from this state?
]]> Forbidden
You don't have permission to access this resource.
Apache/2.4.41 (Ubuntu) Server at example.com Port 80
I check log
sudo tail -100 /var/log/apache2/error.log
This is result
[Wed Jul 20 07:18:22.838769 2022] [rewrite:error] [pid 234347] [client 49.228.236.32:37690] AH00670: Options FollowSymLinks and SymLinksIfOwnerMatch are both off, so the RewriteRule directive is also forbidden due to its similar ability to circumvent directory restrictions : /var/www/example/
I fix in apache2.conf and .htaccess
<VirtualHost *:80>
<Directory /usr/share>
AllowOverride All
Require all granted
</Directory>
<Directory /var/www>
Options +SymLinksIfOwnerMatch
AllowOverride All
Order deny,allow
Allow from all
</Directory>
#<Directory /srv>
# Options Indexes FollowSymLinks
# AllowOverride All
# Require all granted
#</Directory>
</VirtualHost>
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress
Bug can not fix issue
]]>I want to upload multiple files from frontend is there any way to do that?
I know I can bypass the backend by getting a pre-signed URL from the backend and then I can upload files directly from frontend which is okay.
But I need to upload multiple files do I need to get pre-signed URLs for all the files? Please let me know how can I upload multiple files into spaces.
Thank you in advance!
]]>I need your help to transfer this website.
I have already installed WordPress on the Digitalocean server, and my previous host provided me with a backup file but I have no idea how to upload this backup file to the new server.
Please help me move this website.
]]>and http://www.The example.com/ go to (Success!The example .com server block is working!)
please advise
i go through https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-20-04
]]>This is my database.YML
default: &default
adapter: postgresql
encoding: unicode
pool: <%= ENV.fetch(“RAILS_MAX_THREADS”) { 5 } %>
production:
<<: *default
secret_key_base: <%= ENV[“SECRET_KEY”] %>
username: <%= ENV[“DATABASE_USER”] %>
password: <%= ENV[“DATABASE_PASSWORD”] %>
database: <%= ENV[“DATABASE”] %>
host: <%= ENV[“DATABASE_HOST”] %>
these are the steps im following
- Cd Into rails app on VM
- rails s -d -b 0.0.0.0 RAILS_ENV=production
At this point the production server is “online”
then i want to get migrations by running this:
- rake db:migrate RAILS_ENV=production
and this is the issue:
Caused by:PG::ConnectionBad: FATAL: database “root” does not exist
Rails -v 6.0.3.4
Ruby -v 2.7.1
The app is basically a CRM with some extra bells and whistles, like file storage and management, a bit of IoT to use remote printers, etc.
I already have a good chunk of the frontend using the Quasar framework and am comfortable building the backend with node.
My plan is to develop an “Environment” where I can host the business static website, and at the same time host the business Postgres database and app server.
I think the easiest way to achieve this would be to just get a server instance and have it do it all, however, I am quite junior so I am good with front and back end development, but I know very little about apache/nginx and I know nothing (I am a bit scared to be honest) about security and hacking.
I started developing this app as a web app using Firebase, but I found the noSQL database a little limited in functionality. I am using firebase Auth though, so I dont know if DO has an auth system or I should stick to google only on that.
The question is if using DO AppPlatform would be able to provide me with the security management and all the resources that I need for this to work at an affordable price? / is there a way to have my own server instance but have the security managed by DO? / If I dont want to manage security should I stick to some other service like firebase?
Any info/suggestions would be really appreciated, just remember I am a bit of a newbie.
]]>I don’t understand why this would be? Is it because the component (web service) within my App is accessing the database and that hasn’t been whitelisted yet? If so, how would i whitelist it?
This just doesn’t really make any sense to me. Why would the IP whitelisting not work for just this web app but work fine for other droplets? I don’t see what I am doing wrong.
]]>Accidently, I created 2 tickets. One is 5 days old and another one is 2 days old.
How long does it take for Digital Ocean to respond to such reports?
I think 5 days is too much. The website hosted at Digital Ocean keeps copying my contents.
This makes me feeling bad every day like nightmares.
]]>Cloud hosting is a method of using online virtual servers that can be created, modified, and destroyed on demand. Cloud servers are allocated resources like CPU cores and memory by the physical they are hosted on, and can be configured with any operating system and accompanying software. Cloud hosting can be used for hosting websites, distributing web-based applications, or other services.
In this guide, we will go over some of the basic concepts involved in cloud hosting, including how virtualization works, the components in a virtual environment, and comparisons with other common hosting methods.
“The Cloud” is a common term that refers to internet-accessible servers that are available for public use, either through paid leasing or as part of a software or platform service. A cloud-based service can take many forms, including web hosting, file hosting and sharing, and software distribution. “The Cloud” can also refer to cloud computing, i.e., transparently spanning a task across multiple servers. Instead of running a complex process on a single powerful machine, cloud computing distributes the task across many smaller nodes.
Cloud hosting environments are broken down into two main parts: the virtual servers that apps and websites can be hosted on, and the physical hosts that manage the virtual servers. Virtualization makes cloud hosting possible: the relationship between host and virtual server provides flexibility and scaling that are not available through other hosting methods.
The most common form of cloud hosting today is the use of a virtual private server, or VPS. A VPS is a virtual server that acts like a real computer with its own operating system. While virtual servers share resources that are allocated to them by the host, they are totally isolated in practice, so operations on one VPS won’t affect the others.
Virtual servers are deployed and managed by the hypervisor of a physical host. Each virtual server has an operating system installed by the hypervisor, that is available to the user. For practical purposes, a virtual server is identical in use to a dedicated physical server, though the a virtual server needs to share physical hardware resources with other servers on the same host.
Resources are allocated to a virtual server by the physical server that it is hosted on. This host uses a software layer called a hypervisor to deploy, manage, and grant resources to the virtual servers that are under its control. The term “hypervisor” is also often used to refer to the physical hosts that hypervisors (and their virtual servers) are installed on.
The host is in charge of allocating memory, CPU cores, and a network connection to a virtual server when one is launched. An ongoing duty of the hypervisor is to schedule processes between the virtual CPU cores and the physical ones, since multiple virtual servers may be utilizing the same physical cores. Hypervisors differ from one another in the nuances of process scheduling and resource sharing.
There are a few common hypervisors available for cloud hosts today. These different virtualization methods have some key differences, but they all provide the tools that a host needs to deploy, maintain, move, and destroy virtual servers as needed.
KVM, short for “Kernel-Based Virtual Machine”, is a virtualization infrastructure that is built in to the Linux kernel. When activated, this kernel module turns the Linux machine into a hypervisor, allowing it to begin hosting virtual servers. This method contrasts with how other hypervisors usually work, as KVM does not need to create or emulate kernel components that are used for virtual hosting.
Xen is one of the most common hypervisors. Unlike KVM, Xen uses its own microkernel, which provides the tools needed to support virtual servers without modifying the host’s kernel. Xen supports two distinct methods of virtualization: paravirtualization, which skips the need to emulate hardware but requires special modifications made to the virtual servers’ operating system, and hardware-assisted virtualization (or HVM), which uses special hardware features to efficiently emulate a virtual server so that they can use unmodified operating systems. HVM became widespread on consumer CPUs around 2006, allowing most desktops and laptops to achieve similar performance when running virtual machines or microkernel-based containers (e.g. through Docker).
ESXi is an enterprise-level hypervisor offered by VMware. ESXi is unique in that it doesn’t require the host to have an underlying operating system. This is referred to as a “type 1” hypervisor and is extremely efficient due to the lack of a “middleman” between the hardware and the virtual servers. With type 1 hypervisors like ESXi, no operating system needs to be loaded on the host because the hypervisor itself acts as the operating system.
Hyper-V is one of the most popular methods of virtualizing Windows servers and is available as a system service in Windows Server. This makes Hyper-V a common choice for developers working within a Windows software environment. Hyper-V is included in modern versions of Windows and is also available as a stand-alone server without an existing installation of Windows Server. WSL2, the Windows Subsystem for Linux, is implemented via Hyper-V.
The features offered by virtualization lend themselves well to a cloud hosting environment. Virtual servers can be configured with a wide range of hardware resource allocations, and can often have resources added or removed as needs change over time. Some cloud hosts can move a virtual server from one hypervisor to another with little or no downtime, or duplicate the server for redundancy in case of a node failure.
Developers often prefer to work in a VPS due to the control that they have over the virtual environment. Most virtual servers running Linux offer access to the root (administrator) account or sudo
privileges by default, giving a developer the ability to install and modify whatever software they need.
This freedom of choice begins with the operating system. Most hypervisors are capable of hosting nearly any guest operating system, from open source software like Linux and BSD to proprietary systems like Windows. From there, developers can begin installing and configuring the building blocks needed for whatever they are working on. A cloud server’s configurations might include a web server, database, or an app that has been developed and is ready for distribution.
Cloud servers are very flexible in their ability to scale. Scaling methods fall into two broad categories: horizontal scaling and vertical scaling. Most hosting methods can scale one way or the other, but cloud hosting is unique in its ability to scale both horizontally and vertically. This is due to the virtual environment that a cloud server is built on: as its resources are an allocated portion of a larger physical pool, these resources can be adjusted or duplicated to other hypervisors.
Horizontal scaling, often referred to as “scaling out”, is the process of adding more nodes to a clustered system. This might involve adding more web servers to better manage traffic, adding new servers to a region to reduce latency, or adding more database workers to increase data transfer speed.
Vertical scaling, or “scaling up”, is when a single server is upgraded with additional resources. This might be an expansion of available memory, an allocation of more CPU cores, or some other upgrade that increases that server’s capacity. These upgrades usually pave the way for additional software instances, like database workers, to operate on that server. Before horizontal scaling became cost-effective, vertical scaling was the de facto way to respond to increasing demand.
With cloud hosting, developers can scale depending on their application’s needs — they can scale out by deploying additional VPS nodes, scale up by upgrading existing servers, or do both when server needs have dramatically increased.
By now, you should have an understanding of how cloud hosting works, including the relationship between hypervisors and the virtual servers that they are responsible for, as well as how cloud hosting compares to other common hosting methods. With this information in mind, you can choose the best hosting for your needs.
For a broader view of the overall cloud computing landscape, you can read A General Introduction to Cloud Computing.
]]>Having an automated deployment process is a requirement for a scalable and resilient application, and GitOps, or Git-based DevOps, has rapidly become a popular method of organizing CI/CD with a Git repository as a “single source of truth.” Tools like CircleCI integrate with your GitHub repository, allowing you to test and deploy your code automatically every time you make a change to your repository. When this kind of CI/CD is combined with the flexibility of Kubernetes infrastructure, you can build an application that scales easily with changing demand.
In this article you will use CircleCI to deploy a sample application to a DigitalOcean Kubernetes (DOKS) cluster. After reading this tutorial, you’ll be able to apply these same techniques to deploy other CI/CD tools that are buildable as Docker images.
To follow this tutorial, you’ll need to have:
A DigitalOcean account, which you can set up by following the Sign up for a DigitalOcean Account documentation.
Docker installed on your workstation, and knowledge of how to build, remove, and run Docker images. You can install Docker on Ubuntu 20.04 by following the tutorial on How To Install and Use Docker on Ubuntu 20.04.
Knowledge of how Kubernetes works and how to create deployments and services on it. It’s highly recommended to read the Introduction to Kubernetes article.
The kubectl
command line interface tool installed on the computer from which you will control your cluster.
An account on Docker Hub to be used to store your sample application image.
A GitHub account and knowledge of Git basics. You can follow the tutorial series Introduction to Git: Installation, Usage, and Branches and How To Create a Pull Request on GitHub to build this knowledge.
For this tutorial, you will use Kubernetes version 1.22.7
and kubectl
version 1.23.5
.
Note: You can skip this section if you already have a running DigitalOcean Kubernetes cluster.
In this first step, you will create the DigitalOcean Kubernetes (DOKS) cluster from which you will deploy your sample application. The kubectl
commands executed from your local machine will change or retrieve information directly from the Kubernetes cluster.
Go to the Kubernetes page on your DigitalOcean account.
Click Create a Kubernetes cluster, or click the green Create button at the top right of the page and select Kubernetes from the dropdown menu.
The next page is where you are going to specify the details of your cluster. On Select a Kubernetes version pick version 1.22.7-do.0. If this one is not available, choose the latest recommended version.
For Choose a datacenter region, choose the region closest to you. This tutorial will use San Francisco.
You then have the option to build your Node pool(s). On Kubernetes, a node is a worker machine, which contains the services necessary to run pods. On DigitalOcean, each node is a Droplet. Your node pool will consist of a single Basic node. Select the 1GB/1vCPU configuration and change to 1 Node on the number of nodes.
You can add extra tags if you want; this can be useful if you plan to use the DigitalOcean API or just to better organize your node pools.
On Choose a name, for this tutorial, use kubernetes-deployment-tutorial
. This will make it easier to follow throughout while reading the next sections. Finally, click the green Create Cluster button to create your cluster. Wait until the cluster creation has completed.
After cluster creation, there will be instructions to connect to your cluster. Follow the instructions on the Automated (recommended) tab or download the kubeconfig file under the Manual tab. This is the file you will be using to authenticate the kubectl
commands you are going to run against your cluster. Download it to your kubectl
machine.
The default way to use that file is to always pass the --kubeconfig
flag and the path to it on all commands you run with kubectl
. For example, if you downloaded the config file to Desktop
, you would run the kubectl get pods
command like this:
- kubectl --kubeconfig ~/Desktop/kubernetes-deployment-tutorial-kubeconfig.yaml get pods
This would yield the following output:
OutputNo resources found.
This means you accessed your cluster. The No resources found.
message is correct, since you don’t have any pods on your cluster.
If you are not maintaining any other Kubernetes clusters you can copy the kubeconfig file to a folder on your home directory called .kube
. Create that directory in case it does not exist:
- mkdir -p ~/.kube
Then copy the config file into the newly created .kube
directory and rename it config
:
- cp current_kubernetes-deployment-tutorial-kubeconfig.yaml_file_path ~/.kube/config
The config file should now have the path ~/.kube/config
. This is the file that kubectl
reads by default when running any command, so there is no need to pass --kubeconfig
anymore. Run the following:
- kubectl get pods
You will receive the following output:
OutputNo resources found in default namespace.
Now access the cluster with the following:
- kubectl get nodes
You will receive the list of nodes on your cluster. The output will be similar to this:
OutputNAME STATUS ROLES AGE VERSION
pool-upkissrv3-uzm8z Ready <none> 12m v1.22.7
In this tutorial you are going to use the default
namespace for all kubectl
commands and manifest files, which are files that define the workload and operating parameters of work in Kubernetes. Namespaces are like virtual clusters inside your single physical cluster. You can change to any other namespace you want; just make sure to always pass it using the --namespace
flag to kubectl
, and/or specifying it on the Kubernetes manifests metadata field. They are a great way to organize the deployments of your team and their running environments; read more about them in the official Kubernetes overview on Namespaces.
By finishing this step you are now able to run kubectl
against your cluster. In the next step, you will create the local Git repository you are going to use to house your sample application.
You are now going to structure your sample deployment in a local Git repository. You will also create some Kubernetes manifests that will be global to all deployments you are going to do on your cluster.
Note: This tutorial has been tested on Ubuntu 20.04, and the individual commands are styled to match this OS. However, most of the commands here can be applied to other Linux distributions with little to no change needed, and commands like kubectl
are platform-agnostic.
First, create a new Git repository locally that you will push to GitHub later on. Create an empty folder called do-sample-app
in your home directory and cd
into it:
- mkdir ~/do-sample-app
- cd ~/do-sample-app
Now create a new Git repository in this folder with the following command:
- git init .
Inside this repository, create an empty folder called kube
:
- mkdir ~/do-sample-app/kube/
This will be the location where you are going to store the Kubernetes resources manifests related to the sample application that you will deploy to your cluster.
Now, create another folder called kube-general
, but this time outside of the Git repository you just created. Make it inside your home directory:
- mkdir ~/kube-general/
This folder is outside of your Git repository because it will be used to store manifests that are not specific to a single deployment on your cluster, but common to multiple ones. This will allow you to reuse these general manifests for different deployments.
With your folders created and the Git repository of your sample application in place, it’s time to arrange the authentication and authorization of your DOKS cluster.
It’s generally not recommended to use the default admin user to authenticate from other Services into your Kubernetes cluster. If your keys on the external provider got compromised, your whole cluster would become compromised.
Instead you are going to use a single Service Account with a specific Role, which is all part of the RBAC Kubernetes authorization model.
This authorization model is based on Roles and Resources. You start by creating a Service Account, which is basically a user on your cluster, then you create a Role, in which you specify what resources it has access to on your cluster. Finally, you create a Role Binding, which is used to make the connection between the Role and the Service Account previously created, granting to the Service Account access to all resources the Role has access to.
The first Kubernetes resource you are going to create is the Service Account for your CI/CD user, which this tutorial will name cicd
.
Create the file cicd-service-account.yml
inside the ~/kube-general
folder, and open it with your favorite text editor:
- nano ~/kube-general/cicd-service-account.yml
Write the following content on it:
apiVersion: v1
kind: ServiceAccount
metadata:
name: cicd
namespace: default
This is a YAML file; all Kubernetes resources are represented using one. In this case you are saying this resource is from Kubernetes API version v1
(internally kubectl
creates resources by calling Kubernetes HTTP APIs), and it is a ServiceAccount
.
The metadata
field is used to add more information about this resource. In this case, you are giving this ServiceAccount
the name cicd
, and creating it on the default
namespace.
You can now create this Service Account on your cluster by running kubectl apply
, like the following:
- kubectl apply -f ~/kube-general/
You will receive output similar to the following:
Outputserviceaccount/cicd created
To make sure your Service Account is working, try to log in to your cluster using it. To do that you first need to obtain their respective access token and store it in an environment variable. Every Service Account has an access token which Kubernetes stores as a Secret.
You can retrieve this secret using the following command:
- TOKEN=$(kubectl get secret $(kubectl get secret | grep cicd-token | awk '{print $1}') -o jsonpath='{.data.token}' | base64 --decode)
Some explanation on what this command is doing:
$(kubectl get secret | grep cicd-token | awk '{print $1}')
This is used to retrieve the name of the secret related to our cicd
Service Account. kubectl get secret
returns the list of secrets on the default namespace, then you use grep
to search for the lines related to your cicd
Service Account. Then you return the name, since it is the first thing on the single line returned from the grep
.
kubectl get secret preceding-command -o jsonpath='{.data.token}' | base64 --decode
This will retrieve only the secret for your Service Account token. You then access the token field using jsonpath
, and pass the result to base64 --decode
. This is necessary because the token is stored as a Base64 string. The token itself is a JSON Web Token.
You can now try to retrieve your pods with the cicd
Service Account. Run the following command, replacing server-from-kubeconfig-file
with the server URL that can be found after server:
in ~/.kube/config
(the config file you downloaded for the cluster). This command will give a specific error that you will learn about later in this tutorial:
- kubectl --insecure-skip-tls-verify --kubeconfig="/dev/null" --server=server-from-kubeconfig-file --token=$TOKEN get pods
--insecure-skip-tls-verify
skips the step of verifying the certificate of the server, since you are just testing and do not need to verify this. --kubeconfig="/dev/null"
is to make sure kubectl
does not read your config file and credentials but instead uses the token provided.
The output should be similar to this:
OutputError from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:cicd" cannot list resource "pods" in API group "" in the namespace "default"
This is an error, but it shows us that the token worked. The error you received is about your Service Account not having the neccessary authorization to list the resource secrets
, but you were able to access the server itself. If your token had not worked, the error would have been the following one:
Outputerror: You must be logged in to the server (Unauthorized)
Now that the authentication was a success, the next step is to fix the authorization error for the Service Account. You will do this by creating a role with the necessary permissions and binding it to your Service Account.
Kubernetes has two ways to define roles: using a Role
or a ClusterRole
resource. The difference between the former and the latter is that the first one applies to a single namespace, while the other is valid for the whole cluster.
As you are using a single namespace on this tutorial, you will use a Role
.
Create the file ~/kube-general/cicd-role.yml
and open it with your favorite text editor:
- nano ~/kube-general/cicd-role.yml
The basic idea is to grant access to do everything related to most Kubernetes resources in the default
namespace. Your Role
would look like this:
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cicd
namespace: default
rules:
- apiGroups: ["", "apps", "batch", "extensions"]
resources: ["deployments", "services", "replicasets", "pods", "jobs", "cronjobs"]
verbs: ["*"]
This YAML has some similarities with the one you created previously, but here you are saying this resource is a Role
, and it’s from the Kubernetes API rbac.authorization.k8s.io/v1
. You are naming your role cicd
, and creating it on the same namespace you created your ServiceAccount
, the default
one.
Then you have the rules
field, which is a list of resources this role has access to. In Kubernetes resources are defined based on the API group they belong to, the resource kind itself, and what actions you can do on then, which is represented by a verb. Those verbs are similar to the HTTP ones.
In this case you are saying that your Role
is allowed to do everything, *
, on the following resources: deployments
, services
, replicasets
, pods
, jobs
, and cronjobs
. This also applies to those resources belonging to the following API groups: ""
(empty string), apps
, batch
, and extensions
. The empty string means the root API group. If you use apiVersion: v1
when creating a resource it means this resource is part of this API group.
A Role
by itself does nothing; you must also create a RoleBinding
, which binds a Role
to something, in this case, a ServiceAccount
.
Create the file ~/kube-general/cicd-role-binding.yml
and open it:
- nano ~/kube-general/cicd-role-binding.yml
Add the following lines to the file:
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cicd
namespace: default
subjects:
- kind: ServiceAccount
name: cicd
namespace: default
roleRef:
kind: Role
name: cicd
apiGroup: rbac.authorization.k8s.io
Your RoleBinding
has some specific fields that have not yet been covered in this tutorial. roleRef
is the Role
you want to bind to something; in this case it is the cicd
role you created earlier. subjects
is the list of resources you are binding your role to; in this case it’s a single ServiceAccount
called cicd
.
Note: If you had used a ClusterRole
, you would have to create a ClusterRoleBinding
instead of a RoleBinding
. The file would be almost the same. The only difference would be that it would have no namespace
field inside the metadata
.
With those files created you will be able to use kubectl apply
again. Create those new resources on your Kubernetes cluster by running the following command:
- kubectl apply -f ~/kube-general/
You will receive output similar to the following:
Outputrolebinding.rbac.authorization.k8s.io/cicd created
role.rbac.authorization.k8s.io/cicd created
serviceaccount/cicd unchanged
Now, try the command you ran previously:
- kubectl --insecure-skip-tls-verify --kubeconfig="/dev/null" --server=server-from-kubeconfig-file --token=$TOKEN get pods
Since you have no pods, this will yield the following output:
OutputNo resources found in default namespace.
In this step, you gave the Service Account you are going to use on CircleCI the necessary authorization to do meaningful actions on your cluster like listing, creating, and updating resources. Now it’s time to create your sample application.
Note: All commands and files created from now on will start from the folder ~/do-sample-app
you created earlier. This is becase you are now creating files specific to the sample application that you are going to deploy to your cluster.
The Kubernetes Deployment
you are going to create will use the Nginx image as a base, and your application will be a simple static HTML page. This is a great start because it allows you to test if your deployment works by serving HTML directly from Nginx. As you will see later on, you can redirect all traffic coming to a local address:port
to your deployment on your cluster to test if it’s working.
Inside the repository you set up earlier, create a new Dockerfile
file and open it with your text editor of choice:
- nano ~/do-sample-app/Dockerfile
Write the following on it:
FROM nginx:1.21
COPY index.html /usr/share/nginx/html/index.html
This will tell Docker to build the application container from an nginx
image.
Now create a new index.html
file and open it:
- nano ~/do-sample-app/index.html
Write the following HTML content:
<!DOCTYPE html>
<title>DigitalOcean</title>
<body>
Kubernetes Sample Application
</body>
This HTML will display a simple message that will let you know if your application is working.
You can test if the image is correct by building and then running it.
First, build the image with the following command, replacing dockerhub-username
with your own Docker Hub username. You must specify your username here so when you push it later on to Docker Hub it will work:
- docker build ~/do-sample-app/ -t dockerhub-username/do-kubernetes-sample-app
Now run the image. Use the following command, which starts your image and forwards any local traffic on port 8080
to the port 80
inside the image, the port Nginx listens to by default:
- docker run --rm -it -p 8080:80 dockerhub-username/do-kubernetes-sample-app
The command prompt will stop being interactive while the command is running. Instead you will see the Nginx access logs. If you open localhost:8080
on any browser it should show an HTML page with the content of ~/do-sample-app/index.html
. In case you don’t have a browser available, you can open a new terminal window and use the following curl
command to fetch the HTML from the webpage:
- curl localhost:8080
You will receive the following output:
Output<!DOCTYPE html>
<title>DigitalOcean</title>
<body>
Kubernetes Sample Application
</body>
Stop the container (CTRL
+ C
on the terminal where it’s running), and submit this image to your Docker Hub account. To do this, first log in to Docker Hub:
- docker login
Fill in the required information about your Docker Hub account, then push the image with the following command (don’t forget to replace the dockerhub-username
with your own):
- docker push dockerhub-username/do-kubernetes-sample-app
You have now pushed your sample application image to your Docker Hub account. In the next step, you will create a Deployment on your DOKS cluster from this image.
With your Docker image created and working, you will now create a manifest telling Kubernetes how to create a Deployment from it on your cluster.
Create the YAML deployment file ~/do-sample-app/kube/do-sample-deployment.yml
and open it with your text editor:
- nano ~/do-sample-app/kube/do-sample-deployment.yml
Write the following content on the file, making sure to replace dockerhub-username
with your Docker Hub username:
apiVersion: apps/v1
kind: Deployment
metadata:
name: do-kubernetes-sample-app
namespace: default
labels:
app: do-kubernetes-sample-app
spec:
replicas: 1
selector:
matchLabels:
app: do-kubernetes-sample-app
template:
metadata:
labels:
app: do-kubernetes-sample-app
spec:
containers:
- name: do-kubernetes-sample-app
image: dockerhub-username/do-kubernetes-sample-app:latest
ports:
- containerPort: 80
name: http
Kubernetes deployments are from the API group apps
, so the apiVersion
of your manifest is set to apps/v1
. On metadata
you added a new field you have not used previously, called metadata.labels
. This is useful to organize your deployments. The field spec
represents the behavior specification of your deployment. A deployment is responsible for managing one or more pods; in this case it’s going to have a single replica by the spec.replicas
field. That is, it’s going to create and manage a single pod.
To manage pods, your deployment must know which pods it’s responsible for. The spec.selector
field is the one that gives it that information. In this case the deployment will be responsible for all pods with tags app=do-kubernetes-sample-app
. The spec.template
field contains the details of the Pod
this deployment will create. Inside the template you also have a spec.template.metadata
field. The labels
inside this field must match the ones used on spec.selector
. spec.template.spec
is the specification of the pod itself. In this case it contains a single container, called do-kubernetes-sample-app
. The image of that container is the image you built previously and pushed to Docker Hub.
This YAML file also tells Kubernetes that this container exposes the port 80
, and gives this port the name http
.
To access the port exposed by your Deployment
, create a Service. Make a file named ~/do-sample-app/kube/do-sample-service.yml
and open it with your favorite editor:
- nano ~/do-sample-app/kube/do-sample-service.yml
Next, add the following lines to the file:
apiVersion: v1
kind: Service
metadata:
name: do-kubernetes-sample-app
namespace: default
labels:
app: do-kubernetes-sample-app
spec:
type: ClusterIP
ports:
- port: 80
targetPort: http
name: http
selector:
app: do-kubernetes-sample-app
This file gives your Service
the same labels used on your deployment. This is not required, but it helps to organize your applications on Kubernetes.
The service resource also has a spec
field. The spec.type
field is responsible for the behavior of the service. In this case it’s a ClusterIP
, which means the service is exposed on a cluster-internal IP, and is only reachable from within your cluster. This is the default spec.type
for services. spec.selector
is the label selector criteria that should be used when picking the pods to be exposed by this service. Since your pod has the tag app: do-kubernetes-sample-app
, you used it here. spec.ports
are the ports exposed by the pod’s containers that you want to expose from this service. Your pod has a single container which exposes port 80
, named http
, so you are using it here as targetPort
. The service exposes that port on port 80
too, with the same name, but you could have used a different port/name combination than the one from the container.
With your Service
and Deployment
manifest files created, you can now create those resources on your Kubernetes cluster using kubectl
:
- kubectl apply -f ~/do-sample-app/kube/
You will receive the following output:
Outputdeployment.apps/do-kubernetes-sample-app created
service/do-kubernetes-sample-app created
Test if this is working by forwarding one port on your machine to the port that the service is exposing inside your Kubernetes cluster. You can do that using kubectl port-forward
:
- kubectl port-forward $(kubectl get pod --selector="app=do-kubernetes-sample-app" --output jsonpath='{.items[0].metadata.name}') 8080:80
The subshell command $(kubectl get pod --selector="app=do-kubernetes-sample-app" --output jsonpath='{.items[0].metadata.name}')
retrieves the name of the pod matching the app
tag you used. Otherwise you could have retrieved it from the list of pods by using kubectl get pods
.
After you run port-forward
, the shell will stop being interactive, and will instead output the requests redirected to your cluster:
OutputForwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
Opening localhost:8080
on any browser should render the same page you saw when you ran the container locally, but it’s now coming from your Kubernetes cluster. As before, you can also use curl
in a new terminal window to check if it’s working:
- curl localhost:8080
You will receive the following output:
Output<!DOCTYPE html>
<title>DigitalOcean</title>
<body>
Kubernetes Sample Application
</body>
Next, it’s time to push all the files you created to your GitHub repository. To do this you must first create a repository on GitHub called digital-ocean-kubernetes-deploy
.
In order to keep this repository simple for demonstration purposes, do not initialize the new repository with a README
, license
, or .gitignore
file when asked on the GitHub UI. You can add these files later on.
With the repository created, point your local repository to the one on GitHub. To do this, press CTRL
+ C
to stop kubectl port-forward
and get the command line back, then run the following commands to add a new remote called origin
:
- cd ~/do-sample-app/
- git remote add origin https://github.com/your-github-account-username/digital-ocean-kubernetes-deploy.git
There should be no output from the preceding command.
Next, commit all the files you created up to now to the GitHub repository. First, add the files:
- git add --all
Next, commit the files to your repository, with a commit message in quotation marks:
- git commit -m "initial commit"
This will yield output similar to the following:
Output[master (root-commit) db321ad] initial commit
4 files changed, 47 insertions(+)
create mode 100644 Dockerfile
create mode 100644 index.html
create mode 100644 kube/do-sample-deployment.yml
create mode 100644 kube/do-sample-service.yml
Finally, push the files to GitHub:
- git push -u origin master
You will be prompted for your username and password. Once you have entered this, you will see output like this:
OutputCounting objects: 7, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 907 bytes | 0 bytes/s, done.
Total 7 (delta 0), reused 0 (delta 0)
To github.com:your-github-account-username/digital-ocean-kubernetes-deploy.git
* [new branch] master -> master
Branch master set up to track remote branch master from origin.
If you go to your GitHub repository page you will now see all the files there. With your project up on GitHub, you can now set up CircleCI as your CI/CD tool.
For this tutorial, you will use CircleCI to automate deployments of your application whenever the code is updated, so you will need to log in to CircleCI using your GitHub account and set up your repository.
First, go to their homepage https://circleci.com
, and press Sign Up.
You are using GitHub, so click the green Sign Up with GitHub button.
CircleCI will redirect to an authorization page on GitHub. CircleCI needs some permissions on your account to be able to start building your projects. This allows CircleCI to obtain your email, deploy keys and permission to create hooks on your repositories, and add SSH keys to your account. If you need more information on what CircleCI is going to do with your data, check their documentation about GitHub integration.
After authorizing CircleCI you will be redirected to the Projects page. Here you can set up your GitHub repository in CircleCI. Select Set Up Project in the entry for your digital-ocean-kubernetes-deploy
repo. Then select the Faster: Commit a starter CI pipeline to a new branch option. This will create a new circleci-project-setup branch for your project.
Next, specify some environment variables in the CircleCI settings. You can find the settings of the project by selecting the Project Settings button in the top right section of the page then selecting Environment Variables. Press Add Environment Variable to create new environment variables.
First, add two environment variables called DOCKERHUB_USERNAME
and DOCKERHUB_PASS
, which will be needed later on to push the image to Docker Hub. Set the values to your Docker Hub username and password, respectively.
Then add three more: KUBERNETES_TOKEN
, KUBERNETES_SERVER
, and KUBERNETES_CLUSTER_CERTIFICATE
.
The value of KUBERNETES_TOKEN
will be the value of the local environment variable you used earlier to authenticate on your Kubernetes cluster using your Service Account user. If you have closed the terminal, you can always run the following command to retrieve it again:
- kubectl get secret $(kubectl get secret | grep cicd-token | awk '{print $1}') -o jsonpath='{.data.token}' | base64 --decode
KUBERNETES_SERVER
will be the string you passed as the --server
flag to kubectl
when you logged in with your cicd
Service Account. You can find this after server:
in the ~/.kube/config
file, or in the file kubernetes-deployment-tutorial-kubeconfig.yaml
downloaded from the DigitalOcean dashboard when you made the initial setup of your Kubernetes cluster.
KUBERNETES_CLUSTER_CERTIFICATE
should also be available on your ~/.kube/config
file. It’s the certificate-authority-data
field on the clusters
item related to your cluster. It should be a long string; make sure to copy all of it.
Those environment variables must be defined here because most of them contain sensitive information, and it is not secure to place them directly on the CircleCI YAML config file.
With CircleCI listening for changes on your repository and the environment variables configured, it’s time to create the configuration file.
Make a directory called .circleci
inside your sample application repository:
- mkdir ~/do-sample-app/.circleci/
Inside this directory, create a file named config.yml
and open it with your favorite editor:
- nano ~/do-sample-app/.circleci/config.yml
Add the following content to the file, making sure to replace dockerhub-username
with your Docker Hub username:
version: 2.1
jobs:
build:
docker:
- image: circleci/buildpack-deps:bullseye
environment:
IMAGE_NAME: dockerhub-username/do-kubernetes-sample-app
working_directory: ~/app
steps:
- checkout
- setup_remote_docker
- run:
name: Build Docker image
command: |
docker build -t $IMAGE_NAME:latest .
- run:
name: Push Docker Image
command: |
echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin
docker push $IMAGE_NAME:latest
workflows:
version: 2
build-deploy-master:
jobs:
- build:
filters:
branches:
only: master
This sets up the workflow build-deploy-master
, which right now has a single job called build
. This job will run for every commit to the master
branch.
The build
job is using the image circleci/buildpack-deps:bullseye
to run its steps, which is an image from CircleCI based on the official buildpack-deps
Docker image, but with some extra tools installed, like the Docker binaries themselves.
The workflow has four steps:
checkout
retrieves the code from GitHub.setup_remote_docker
sets up a remote, isolated environment for each build. This is required before you use any docker
command inside a job step. This is necessary because as the steps are running inside a docker image, setup_remote_docker
allocates another machine to run the commands there.run
step builds the image, as you did previously in your local environment. For that you are using the environment variable you declared in environment:
, IMAGE_NAME
.run
step pushes the image to Docker Hub, using the environment variables you configured on the project settings to authenticate.There are other registries besides Docker Hub that you can use to store container images and collaborate on them with others. For instance, DigitalOcean has its own container registry to which you can push container images.
To push an image to the DigitalOcean Container Registry, you could replace the second run
step with one like the following:
. . .
- run: |
docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD registry.digitalocean.com/your_registry
docker push registry.digitalocean.com/your_registry/do-kubernetes-sample-app
. . .
This assumes you’ve securely set $DOCKER_USERNAME
and $DOCKER_PASSWORD
as environment variables previously in the file, as this will avoid exposing them in the CircleCI job output. Also, note that both of these values should be set to the same DigitalOcean API token.
To learn more about the DigitalOcean container registry, check out our product documentation.
Commit the new file to your repository and push the changes upstream:
- cd ~/do-sample-app/
- git add .circleci/
- git commit -m "add CircleCI config"
- git push
This will trigger a new build on CircleCI. The CircleCI workflow is going to correctly build and push your image to Docker Hub.
Now that you have created and tested your CircleCI workflow, you can set your DOKS cluster to retrieve the up-to-date image from Docker Hub and deploy it automatically when changes are made.
Now that your application image is being built and sent to Docker Hub every time you push changes to the master
branch on GitHub, it’s time to update your deployment on your Kubernetes cluster so that it retrieves the new image and uses it as a base for deployment.
To do that, first fix one issue with your deployment: it’s currently depending on an image with the latest
tag. This tag does not tell us which version of the image you are using. You cannot easily lock your deployment to that tag because it’s overwritten everytime you push a new image to Docker Hub, and by using it like that you lose the reproducibility of containerized applications.
You can read more about that on Vladislav Supalov’s article about why depending on the latest
tag is an anti-pattern.
To correct this, you first must make some changes to your Push Docker Image
build step in the ~/do-sample-app/.circleci/config.yml
file. Open up the file:
- nano ~/do-sample-app/.circleci/config.yml
Then add the highlighted lines to your Push Docker Image
step:
...
- run:
name: Push Docker Image
command: |
echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin
docker tag $IMAGE_NAME:latest $IMAGE_NAME:$CIRCLE_SHA1
docker push $IMAGE_NAME:latest
docker push $IMAGE_NAME:$CIRCLE_SHA1
...
CircleCI has some special environment variables set by default. One of them is CIRCLE_SHA1
, which contains the hash of the commit it’s building. The changes you made to ~/do-sample-app/.circleci/config.yml
will use this environment variable to tag your image with the commit it was built from, always tagging the most recent build with the latest tag. That way, you always have specific images available, without overwriting them when you push something new to your repository.
Save and exit the file.
Next, change your deployment manifest file to point to that file. This would be a small change if inside ~/do-sample-app/kube/do-sample-deployment.yml
you could set your image as dockerhub-username/do-kubernetes-sample-app:$COMMIT_SHA1
, but kubectl
doesn’t do variable substitution inside the manifests when you use kubectl apply
. To account for this, you can use envsubst
. envsubst
is a CLI tool that is part of the GNU gettext project. It allows you to pass some text to it, and if it finds any variable inside the text that has a matching environment variable, it replaces the variable with the respective value. The resulting text is then returned as output.
To use this, you will create a bash script that will be responsible for your deployment. Make a new folder called scripts
inside ~/do-sample-app/
:
- mkdir ~/do-sample-app/scripts/
Inside that folder create a new bash script called ci-deploy.sh
and open it with your favorite text editor:
- nano ~/do-sample-app/scripts/ci-deploy.sh
Inside it write the following bash script:
#! /bin/bash
# exit script when any command ran here returns with non-zero exit code
set -e
COMMIT_SHA1=$CIRCLE_SHA1
# Export it so it's available for envsubst
export COMMIT_SHA1=$COMMIT_SHA1
# Since the only way for envsubst to work on files is using input/output redirection,
# it's not possible to do in-place substitution, so you will save the output to another file
# and overwrite the original with that one.
envsubst <./kube/do-sample-deployment.yml >./kube/do-sample-deployment.yml.out
mv ./kube/do-sample-deployment.yml.out ./kube/do-sample-deployment.yml
echo "$KUBERNETES_CLUSTER_CERTIFICATE" | base64 --decode > cert.crt
./kubectl \
--kubeconfig=/dev/null \
--server=$KUBERNETES_SERVER \
--certificate-authority=cert.crt \
--token=$KUBERNETES_TOKEN \
apply -f ./kube/
Let’s go through this script, using the comments in the file. First, there is the following:
set -e
This line makes sure any failed command stops the execution of the bash script. That way if one command fails, the next ones are not executed.
COMMIT_SHA1=$CIRCLE_SHA1
export COMMIT_SHA1=$COMMIT_SHA1
These lines export the CircleCI $CIRCLE_SHA1
environment variable with a new name. If you had just declared the variable without exporting it using export
, it would not be visible for the envsubst
command.
envsubst <./kube/do-sample-deployment.yml >./kube/do-sample-deployment.yml.out
mv ./kube/do-sample-deployment.yml.out ./kube/do-sample-deployment.yml
envsubst
cannot do in-place substitution. That is, it cannot read the content of a file, replace the variables with their respective values, and write the output back to the same file. Therefore, you will redirect the output to another file and then overwrite the original file with the new one.
echo "$KUBERNETES_CLUSTER_CERTIFICATE" | base64 --decode > cert.crt
The environment variable $KUBERNETES_CLUSTER_CERTIFICATE
you created earlier on CircleCI’s project settings is in reality a Base64 encoded string. To use it with kubectl
you must decode its contents and save it to a file. In this case you are saving it to a file named cert.crt
inside the current working directory.
./kubectl \
--kubeconfig=/dev/null \
--server=$KUBERNETES_SERVER \
--certificate-authority=cert.crt \
--token=$KUBERNETES_TOKEN \
apply -f ./kube/
Finally, you are running kubectl
. The command has similar arguments to the one you ran when you were testing your Service Account. You are calling apply -f ./kube/
, since on CircleCI the current working directory is the root folder of your project. ./kube/
here is your ~/do-sample-app/kube
folder.
Save the file and make sure it’s executable:
- chmod +x ~/do-sample-app/scripts/ci-deploy.sh
Now, edit ~/do-sample-app/kube/do-sample-deployment.yml
:
- nano ~/do-sample-app/kube/do-sample-deployment.yml
Change the tag of the container image value to look like the following one:
...
containers:
- name: do-kubernetes-sample-app
image: dockerhub-username/do-kubernetes-sample-app:$COMMIT_SHA1
ports:
- containerPort: 80
name: http
Save and close the file. You must now add some new steps to your CI configuration file to update the deployment on Kubernetes.
Open ~/do-sample-app/.circleci/config.yml
in your favorite text editor:
- nano ~/do-sample-app/.circleci/config.yml
Write the following new job, right below the build
job you created previously:
...
command: |
echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin
docker tag $IMAGE_NAME:latest $IMAGE_NAME:$CIRCLE_SHA1
docker push $IMAGE_NAME:latest
docker push $IMAGE_NAME:$CIRCLE_SHA1
deploy:
docker:
- image: circleci/buildpack-deps:bullseye
working_directory: ~/app
steps:
- checkout
- run:
name: Install envsubst
command: |
sudo apt-get update && sudo apt-get -y install gettext-base
- run:
name: Install kubectl
command: |
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod u+x ./kubectl
- run:
name: Deploy Code
command: ./scripts/ci-deploy.sh
...
The first two steps of your new deploy
job are installing some dependencies, first envsubst
and then kubectl
. The Deploy Code
step is responsible for running our deploy script.
Now you will add this job to the build-deploy-master
workflow you created previously. Inside the build-deploy-master
workflow configuration, write the following new entry right after the existing entry for the build
job:
...
workflows:
version: 2
build-deploy-master:
jobs:
- build:
filters:
branches:
only: master
- deploy:
requires:
- build
filters:
branches:
only: master
This adds the deploy
job to the build-deploy-master
workflow. The deploy
job will only run for commits to master
, and only after the build
job is completed.
The contents of ~/do-sample-app/.circleci/config.yml
will now be like this:
version: 2.1
jobs:
build:
docker:
- image: circleci/buildpack-deps:bullseye
environment:
IMAGE_NAME: dockerhub-username/do-kubernetes-sample-app
working_directory: ~/app
steps:
- checkout
- setup_remote_docker
- run:
name: Build Docker image
command: |
docker build -t $IMAGE_NAME:latest .
- run:
name: Push Docker Image
command: |
echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin
docker tag $IMAGE_NAME:latest $IMAGE_NAME:$CIRCLE_SHA1
docker push $IMAGE_NAME:latest
docker push $IMAGE_NAME:$CIRCLE_SHA1
deploy:
docker:
- image: circleci/buildpack-deps:bullseye
working_directory: ~/app
steps:
- checkout
- run:
name: Install envsubst
command: |
sudo apt-get update && sudo apt-get -y install gettext-base
- run:
name: Install kubectl
command: |
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod u+x ./kubectl
- run:
name: Deploy Code
command: ./scripts/ci-deploy.sh
workflows:
version: 2
build-deploy-master:
jobs:
- build:
filters:
branches:
only: master
- deploy:
requires:
- build
filters:
branches:
only: master
You can now save and exit the file.
To make sure the changes are really going to be reflected on your Kubernetes deployment, edit your index.html
. Change the HTML to something else, like:
<!DOCTYPE html>
<title>DigitalOcean</title>
<body>
Automatic Deployment is Working!
</body>
Once you have saved the above change, commit all the modified files to the repository, and push the changes upstream:
- cd ~/do-sample-app/
- git add --all
- git commit -m "add deploy script and add new steps to circleci config"
- git push
You will see the new build running on CircleCI, and successfully deploying the changes to your Kubernetes cluster.
Wait for the build to finish, then run the same command you ran previously:
- kubectl port-forward $(kubectl get pod --selector="app=do-kubernetes-sample-app" --output jsonpath='{.items[0].metadata.name}') 8080:80
Make sure everything is working by opening your browser on the URL localhost:8080
or by making a curl
request to it. It should show the updated HTML:
- curl localhost:8080
You will receive the following output:
Output<!DOCTYPE html>
<title>DigitalOcean</title>
<body>
Automatic Deployment is Working!
</body>
This means that you have successfully set up automated deployment with CircleCI.
This was a basic tutorial on how to do deployments to DigitalOcean Kubernetes using CircleCI. From here, you can improve your pipeline in many ways. The first thing you can do is create a single build
job for multiple deployments, each one deploying to different Kubernetes clusters or different namespaces. This can be useful when you have different Git branches for development/staging/production environments, ensuring that the deployments are always separated.
You could also build your own image to be used on CircleCI, instead of using buildpack-deps
. This image could be based on it, but could already have kubectl
and envsubst
dependencies installed.
If you would like to learn more about CI/CD on Kubernetes, check out the tutorials for our CI/CD on Kubernetes Webinar Series, or for more information about apps on Kubernetes, see Modernizing Applications for Kubernetes.
]]>Modern development practices often distinguish between deploying and releasing software. Deployment is the step that involves getting the new code onto the servers. Releasing is the step where the new code begins to receive production traffic.
Blue-green deployment is a strategy for deploying and releasing software. It relies on maintaining two separate production-capable environments, nicknamed blue and green for ease of discussion. In this guide, we will discuss how to use blue-green deployments on DigitalOcean to improve the process of transitioning your users to a new version of your software.
In order to complete this guide, you will need two Ubuntu 20.04 servers deployed in an environment that allows you to move IP addresses between hosts. On DigitalOcean, Reserved IPs can provide this functionality. These servers will represent two parallel environments that are alternatively used for staging and production. You can call these servers whatever you’d like, but in this guide, we will be referring to them as “blue” and “green”.
On each of these servers, you should have a non-root user with sudo
configured for administrative functions. You can configure these users by following our Ubuntu 20.04 initial server setup guide.
The basic concept behind blue-green deployment, a technique made popular by this post from Martin Fowler, is that two sets of environments, each capable of serving your application in production, are maintained. These two environments should be nearly identical. By convention, these are referred to as the blue and the green environments.
Only one of these environments is active and receiving production traffic at any one time. In front of the web endpoints for these environments (either web servers or load balancers), a router or other traffic directing mechanism pushes all production traffic to the currently active environment.
When a new release is planned, it is deployed to the non-active environment. For blue-green deployments, the non-active environment functions as a final staging environment. It mirrors the production environment very closely and can be used for final testing before deciding to push changes live.
Once you have tested your deployment internally and have gained confidence in its robustness, you can quickly release the new version by adjusting the routing mechanism. Basically, you flip the switch at the traffic directing layer so that all production traffic begins to move to your new software version. The previously active environment becomes non-active and your previous staging environment becomes your new production environment.
At this point, your previous software version is non-active, but still accessible. If your newly active deployment suffers from any serious issues, you can revert to your previous version by modifying the routing mechanism again.
To demonstrate this general concept, we will set up two server environments. Each will have a web server installed. Keep in mind that in this example, the web server represents an entire application stack which could include a load balancer, multiple web servers, and distributed or replicated databases in the backend. We are using a web server in this guide because it represents the smallest environment that can demonstrate this release pattern.
We will start to develop an “app” on our local computer. In reality, this will only be an index.html
page that we can deploy to our servers. We will configure a git
post receive hook on each of our servers so that we can deploy by issuing a git push
. We will deploy the initial version of our application to both of our servers.
In this guide, we will be using a DigitalOcean Reserved IP address as our routing mechanism. Reserved IPs provide a mechanism for moving traffic from one server to another. We will create a Reserved IP and point it at our green server to set this as our initial production machine.
We will then modify our application and deploy it to our blue server. Production traffic will still be served from the unchanged green server at this point. We can then test the blue server to ensure that our deployment was successful and that there were no bugs. When we are ready, we can move the Reserved IP to the new version of the code by reassigning the Reserved IP address to the blue server.
We will start by creating our “application”. As stated above, this is actually just an index page that our web servers can display. It allows us to demonstrate different “versions” of the app without the overhead of actual development.
On your local system (or on another Droplet), install git
using your platform’s preferred method. If your local machine is running Ubuntu, you can install by typing:
- sudo apt update
- sudo apt install git
On Mac, Windows, or a different Linux environment, you should install git following its documentation if it is not already installed.
We need to set a few configuration settings in order to commit to a git
repository. We will set our name and email address by typing:
- git config --global user.name "Your Name"
- git config --global user.email "username@email.com"
With our configuration set, we can create a directory for our new application and move into it:
- mkdir ~/sample_app
- cd ~/sample_app
Initialize a git repository in our application directory by typing:
- git init
Now, create the index.html
file that represents our application, using nano
or your favorite text editor:
- nano index.html
Inside, we will just specify the version number of our application. This way, we can tell which version of our app is on each server:
App v1
Save and close the file when you are finished. If you are using nano
, press Ctrl+X
, then when prompted, Y
and the Enter.
To finish up, we can add the index.html
file to the git
staging area and then commit by typing:
- git add .
- git commit -m "initializing repository with version 1"
With our file committed, we will stop our application development on our local machine momentarily and focus on setting up our blue and green web servers.
Next, we’ll work on setting up our green and blue environments with functional web servers. We will use Apache in this guide. Log into your servers with your sudo
user to get started.
Note: The steps in this section should be completed on both the blue and green servers.
We can install Apache with apt
. Update the local package index and install the web server software by typing:
- sudo apt update
- sudo apt install apache2
This should install and start Apache on both of your web servers.
Next, we should create and configure a “deploy” user. This user will have access to Apache’s web root and will own the bare git
repository where we will push our app to.
Create a deploy
user by typing:
- sudo adduser --disabled-password deploy
This will create a new user with password authentication disabled.
We will give this new user ownership over Apache’s default web root. This is located at /var/www/html
. Change the ownership of this directory by typing:
- sudo chown -R deploy:deploy /var/www/html
This is all we need for our deployment which only relies on moving files into the web root.
Note: If you are deviating from this guide and your deployment steps would require root privileges, you’ll want to configure passwordless sudo
privileges for needed commands for use with the deploy
account.
This can be done by creating a new sudoers
file within the /etc/sudoers.d
directory:
- sudo visudo -f /etc/sudoers.d/90-deployment
Within this file, you can add the commands that you need to run during deployment. These can be specified like this:
deploy ALL=(ALL) NOPASSWD: first_deployment_command, second_deployment_command, ...
Save and close the file when you are finished. This should allow the deploy
user to correctly execute the required commands without a password.
Now that we have Apache installed and a user configured to execute the deployment, we can configure a bare git
repository to push our application to. We can then set up a post-receive
hook that will automatically deploy the newest version of our main branch when we push it to our servers.
Note: The steps in this section should be completed on both the blue and green servers.
Start off by installing git
on both of your servers:
- sudo apt install git
Next, we need to log in as our deploy
user. We can do that with sudo
by typing:
- sudo su - deploy
In our deploy
user’s home directory, create a directory for our sample application, just like we did on our local computer. Move into the directory after creation:
- mkdir ~/sample_app
- cd ~/sample_app
We will initialize a git
repo in this directory like we did on our local system. However, on our servers, we will include the --bare
option. This will create a git
repo without a working directory. Instead, the contents usually hidden in a .git
directory will be placed into the main folder:
- git init --bare
We will set up a post-receive
hook next. This is just a script that will deploy our changes after a git push
occurs. You can learn more about this deployment strategy by reading this guide. We should place this script in the hooks
directory of our repo. Create and open the file by typing:
- nano hooks/post-receive
Inside, paste in the following deployment script. This is basically the same script outlined in the article linked to above. We are using a GIT_DIR
variable to indicate our git
repo on the server, the WORK_TREE
variable to specify our Apache document root, and HOSTNAME
to grab our server’s hostname for progress messages. This script will deploy all changes in the main
branch to the web directory. No changes should be needed in the script below:
#!/bin/bash
GIT_DIR=/home/deploy/sample_app
WORK_TREE=/var/www/html
HOSTNAME=$(hostname)
while read oldrev newrev ref
do
if [[ $ref =~ .*/main$ ]];
then
echo "Main ref received. Deploying main branch to $HOSTNAME..."
git --work-tree=$WORK_TREE --git-dir=$GIT_DIR checkout -f
echo "Git hooks deploy complete."
else
echo "Ref $ref successfully received. Doing nothing: only the main branch may be deployed on this server."
fi
done
If you are deviating from this guide and need more complex deployment steps, add them to the then
clause in the script above. Make sure that any steps that require elevated privileges in this section use the sudo
command. Also, make sure that all commands that use sudo
here are added to the sudoers
file as specified at the bottom of the last section.
Save and close the file when you are finished.
Modify the permissions on the post-receive
hook so that git
can execute it at the appropriate time:
- chmod +x hooks/post-receive
Next, we will configure SSH keys so that git
can push changes to our web servers without prompting for a password.
On your local or development computer, check to see if you have an SSH key already configured by typing:
- cat ~/.ssh/id_rsa.pub
If you already have an SSH key pair available, you should see something that looks like this:
Outputssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDilFdzkgBcSKdh6tx5pLf+HH6Pv7z7jRZ7cSo6lQvecWOOgGl/wHCVZWx1ULvrF7VgJpgugLwxYsFh3E39sm1+7zeAlRxhFrbWvATwpAEwh5m0+48LTmvXCnJ8/om+GfmAwplmzGk/DNs5trVeagG62Css0rypdoNuLrVdCVKUXGXbO6KnpOsBqoM2HvZKtQ8j1gx+1UUnvK9LYes+ZzC2XZZeBh2dGABe7HNnd8+6e1f2ZjPEKAEV2fPJGAGaAQOnzSKJkUt/B9PdKFbCjnnG1sT0kQoxMRIAiqfR7wa7PUQCM5Orm5S92OTNcnRr8bWVjN18bWCyXkpxxWbIvVU/ user@devel
If the command executes correctly, copy the text that is displayed in its entirety. We will use this in the next section. You can safely skip there now.
If you do not have SSH keys on your local machine, you will probably see an error that looks like this:
Outputcat: /home/user/.ssh/id_rsa.pub: No such file or directory
If this is the case, you can create a new public and private key pair by typing:
- ssh-keygen
Press Enter through all of the prompts to accept the default values. When the keys are created, re-type the cat
command to display the new public key:
- cat ~/.ssh/id_rsa.pub
This should execute correctly this time. Copy the displayed lines to use in the next section.
Back on your green and blue servers, we will authorize our account on our local or development machine to connect to our deploy
users.
As your deploy
user, create an ~/.ssh
directory. Inside, open up a file called authorized_keys
:
- mkdir ~/.ssh
- nano ~/.ssh/authorized_keys
Within this file, paste the output that you copied from your local machine:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDilFdzkgBcSKdh6tx5pLf+HH6Pv7z7jRZ7cSo6lQvecWOOgGl/wHCVZWx1ULvrF7VgJpgugLwxYsFh3E39sm1+7zeAlRxhFrbWvATwpAEwh5m0+48LTmvXCnJ8/om+GfmAwplmzGk/DNs5trVeagG62Css0rypdoNuLrVdCVKUXGXbO6KnpOsBqoM2HvZKtQ8j1gx+1UUnvK9LYes+ZzC2XZZeBh2dGABe7HNnd8+6e1f2ZjPEKAEV2fPJGAGaAQOnzSKJkUt/B9PdKFbCjnnG1sT0kQoxMRIAiqfR7wa7PUQCM5Orm5S92OTNcnRr8bWVjN18bWCyXkpxxWbIvVU/ user@devel
Save and close the file when you are finished.
Next, lock down the permissions so that SSH can use the file you created:
- chmod 600 ~/.ssh/authorized_keys
- chmod 700 ~/.ssh
Now that we have SSH key access configured to our web servers and our application directory set up on each server, we can add our blue and green servers as remotes in our local git
app repository.
On your local machine, move back into your application directory:
- cd ~/sample_app
Add remote references so that git
can push changes to your green and blue web servers:
- git remote add blue deploy@blue_server_ip:sample_app
- git remote add green deploy@green_server_ip:sample_app
We should now be able to push our app to both of our servers. Let’s test it out by pushing version 1 of our application to both servers.
- git push blue main
- git push green main
You may have to accept each server’s key fingerprint on your first deploy. You should see output that looks something like this:
OutputThe authenticity of host '111.111.111.111 (111.111.111.111)' can't be established.
ECDSA key fingerprint is 30:a1:2c:8b:ec:98:a3:3c:7f:4a:db:46:2b:96:b5:06.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '111.111.111.111' (ECDSA) to the list of known hosts.
Counting objects: 3, done.
Writing objects: 100% (3/3), 246 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: Main ref received. Deploying main branch to blue...
remote: Git hooks deploy complete.
To deploy@111.111.111.111:sample_app
* [new branch] main -> main
As you can see, the lines that begin with “remote:” contain the echo
statements from the post-receive
hook on our server. Remember to push your app to both of your servers.
We can test that the initial deployment of our app was successful with curl:
- curl blue_server_ip
- curl green_server_ip
For both of these calls, the response should be the following:
OutputApp v1
This indicates that our deployment script works correctly.
Now that we have the initial version of our application deployed, we can create a Reserved IP address and point it to our “green” server initially.
In the DigitalOcean control panel, click on the “Networking” tab and then the “Reserved IPs” menu item. In the menu provided, select your green server and click on the “Assign Reserved IP” button:
After a few seconds, the IP should be assigned to your green server:
You can now use this IP address as the primary entry point into your production application deployment. If you wanted to set up a domain name for your web app, you would point the domain to this Reserved IP address.
Test that your application is accessible through the Reserved IP address by typing:
- curl reserved_IP_addr
You should see version 1 of your application:
OutputApp v1
The green server is currently supplying this response.
Now that our configuration is complete, we can demonstrate how blue-green deployment works in practice. Currently, our Reserved IP address is pointing to our green server. As stated previously, the Reserved IP address represents production traffic and would be the location where we would attach our application’s domain name.
On your local or development machine, we can make some changes to our application. Open the index file:
- cd ~/sample_app
- nano index.html
Let’s make a visible change to our application by incrementing the version number:
App v2
Save and close the file when you are finished.
Add the file to the git
staging area and commit your changes by typing:
- git add .
- git commit -m "Application version 2"
Next, we can push our new changes to our non-active environment. This will give us the opportunity to test our deployment without impacting our production server.
Since our Reserved IP address is currently pointing to our green environment, we will deploy to our blue server. Push the new changes to the blue environment by typing the following on your local development machine:
- git push blue main
If we visit our Reserved IP address, we should see that version 1 of our application is still being served:
- curl reserved_IP_addr
OutputApp v1
However, if we check out our blue server’s regular IP address, we can test out version 2 of our application:
- curl blue_server_IP
OutputApp v2
This is what we expect and what we want. We can now run our blue server environment through whatever internal testing that we need. All the while, the green server will continue to serve our production traffic.
Once you have tested the newest version of your application and are confident that it is performing as expected, we can switch the production traffic over to the blue server.
To do so, visit the DigitalOcean control panel. Click on the “Networking” tab and then select the “Reserved IPs” navigation item. In the “Reserved IPs” list, you should see your Reserved IP, which is currently pointing to the green server:
Before we switch over, in one of your terminal windows, start a while
loop so that we can make repeated requests through the Reserved IP address. This allow us to immediately see our production application version transition from v1 to v2:
- while true; do curl reserved_ip_addr; sleep 2; done
It should start outputting the results of the web requests:
OutputApp v1
App v1
App v1
App v1
Now, to make the switch and “release” your the new version of your software, click on the blue button on the right-hand side of the Reserved IP assignment to re-assign the IP address. Select your blue server:
In a few seconds, your Reserved IP will be reassigned to your blue server. In your terminal window, the change over should be evident:
OutputApp v1
App v1
App v2
App v2
Stop the while
loop by pressing “Ctrl+C”.
Your production traffic is now being routed to the new version of your application. Your previous production server, the green server, is now set up as both your rollback machine and your next staging area.
If, after moving traffic over to the new version of application, you discover a problem, this release strategy allows you to roll back to your previous version quickly and painlessly. To do so, just reverse the process and point your Reserved IP address back to the green server.
The scenario outlined above was scoped down in order to focus on the deployment and release strategy itself. We didn’t cover more complex, but common setups, like those involving databases.
There are a few different strategies that you can use to handle the persistent data between your two environments.
It is possible to maintain a separate database for each of your environments. However, this strategy would require that you replicate the data in the production database to the non-active database and stop transactions during the moments when you are initiating a switch. Basically, it would require a live database migration as well as a few moments of down time on each deployment. This could quickly become very time consuming and error prone.
A better alternative is usually to share a single database system between the green and blue environments. The application code will be switchable using the blue-green release strategy, while the database itself will be used by both environments.
The main concern with this approach is how to deploy and release updates that include non-backwards compatible database migrations. If we deploy a new release to staging that adds to or alters the database in a way that doesn’t work with the current production deployment, we will break our application.
To prevent this from happening, it is often best to deploy your migrations separate from your code base deployments and in stages where necessary. This modified process is sometimes called blue-turquoise-green deployment. Basically, it hinges on deploying an intermediate version of your application code that can handle both the old and new versions of your database.
The intermediate application code is almost completely the same as the older version, but with some additional logic that prepares it for the new data structures that will exist after the migration takes place. Often, this is accomplished by constructing the migrations so that they create completely new data structures instead of modifying existing ones. This way, you can keep the old data structure, let’s say a table, and create a new one that includes the breaking changes.
The intermediate turquoise deployment is deployed as the first step in the migration process. This deployment will at first read from and write to the old table, but it will check for the existence of the new structure. Next, the migration itself is run, creating the new version of the data structure alongside the old version. The turquoise deployment’s logic should be configured to recognize that the new structure is in place and it should start writing changes to both the old structure and the new structure. It will continue to read from the old structure for the time being.
At this point, all new activity will be recorded in both data structures. You can backfill the new structure with the data from the old structure, transforming it along the way to satisfy the conditions of the new structure. When this is complete, all of your records should exist in both locations. To continue the transition, the next application deployment might continue to write to both structures, but may read from the new structure. After everything is confirmed to be running smoothly, another deployment might cut off writes from the old structure and the old structure may be deleted.
This process can seem fairly involved at first, but it is usually not too much additional work in practice. The main work involves building a safety net that will use both the legacy and new structures temporarily. This gives you time to test your migrations in depth before committing to them, and allows you to roll back at any point to the previous working version of your data structure. For an example of how this data migration could take place, take a look at some of these slides from Mike Brittain at Etsy.
While there are many other strategies that can be employed to separate deployment from the actual release of your new code, blue-green deployment is a mechanism that is quick to implement. It provides a good staging environment that completely mirrors the production environment, while offering immediate rollback opportunities following a release if things didn’t go as expected.
Next, you may want to learn how to perform blue-green deployments using modern Kubernetes tooling such as ArgoCD.
]]>The Secure Shell Protocol (or SSH) is a cryptographic network protocol that allows users to securely access a remote computer over an unsecured network.
Though SSH supports password-based authentication, it is generally recommended that you use SSH keys instead. SSH keys are a more secure method of logging into an SSH server, because they are not vulnerable to common brute-force password hacking attacks.
Generating an SSH key pair creates two long strings of characters: a public and a private key. You can place the public key on any server, and then connect to the server using an SSH client that has access to the private key.
When the public and private keys match up, the SSH server grants access without the need for a password. You can increase the security of your key pair even more by protecting the private key with an optional (but highly encouraged) passphrase.
Note: If you are looking for information about setting up SSH keys in your DigitalOcean account, please refer to our DigitalOcean product documentation on SSH Keys
The first step is to create a key pair on the client machine. This will likely be your local computer. Type the following command into your local command line:
- ssh-keygen -t ed25519
OutputGenerating public/private ed25519 key pair.
You will see a confirmation that the key generation process has begun, and you will be prompted for some information, which we will discuss in the next step.
Note: if you are on an older system that does not support creating ed25519
key pairs, or the server you’re connecting to does not support them, you should create a strong rsa
keypair instead:
- ssh-keygen -t rsa -b 4096
This changes the -t
“type” flag to rsa
, and adds the -b 4096
“bits” flag to create a 4096 bit key.
The first prompt from the ssh-keygen
command will ask you where to save the keys:
OutputEnter file in which to save the key (/home/sammy/.ssh/id_ed25519):
You can press ENTER
here to save the files to the default location in the .ssh
directory of your home directory.
Alternately, you can choose another file name or location by typing it after the prompt and hitting ENTER
.
The second and final prompt from ssh-keygen
will ask you to enter a passphrase:
OutputEnter passphrase (empty for no passphrase):
It’s up to you whether you want to use a passphrase, but it is strongly encouraged: the security of a key pair, no matter the encryption scheme, still depends on the fact that it is not accessible to anyone else.
Should a private key with no passphrase fall into an unauthorized user’s possession, they will be able to log in to any server you’ve configured with the associated public key.
The main downside to having a passphrase — typing it in — can be mitigated by using an ssh-agent
service, which will temporarily store your unlocked key and make it accessible to the SSH client. Many of these agents are integrated with your operating system’s native keychain, making the unlocking process even more seamless.
To recap, the entire key generation process looks like this:
- ssh-keygen -t ed25519
OutputGenerating public/private ed25519 key pair.
Enter file in which to save the key (/home/sammy/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/sammy/.ssh/id_ed25519
Your public key has been saved in /home/sammy/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:EGx5HEXz7EqKigIxHHWKpCZItSj1Dy9Dqc5cYae+1zc sammy@hostname
The key's randomart image is:
+--[ED25519 256]--+
| o+o o.o.++ |
|=oo.+.+.o + |
|*+.oB.o. o |
|*. + B . . |
| o. = o S . . |
|.+ o o . o . |
|. + . ... . |
|. . o. . E |
| .. o. . . |
+----[SHA256]-----+
The public key is now located in /home/sammy/.ssh/id_ed25519.pub
. The private key is now located in /home/sammy/.ssh/id_ed25519
.
Once the key pair is generated, it’s time to place the public key on the server that you want to connect to.
You can copy the public key into the server’s authorized_keys
file with the ssh-copy-id
command. Make sure to replace the example username and address:
- ssh-copy-id sammy@your_server_address
Once the command completes, you will be able to log into the server via SSH without being prompted for a password. However, if you set a passphrase when creating your SSH key, you will be asked to enter the passphrase at that time. This is your local ssh
client asking you to decrypt the private key, it is not the remote server asking for a password.
Once you have copied your SSH keys onto the server, you may want to completely prohibit password logins by configuring the SSH server to disable password-based authentication.
Warning: before you disable password-based authentication, be certain you can successfully log onto the server with your SSH key, and that there are no other users on the server using passwords to log in.
In order to disable password-based SSH authentication, open up the SSH configuration file. It is typically found at the following location:
- sudo nano /etc/ssh/sshd_config
This command will open up the file within the nano
text editor. Find the line in the file that includes PasswordAuthentication
(or create the line if it doesn’t exist), make sure it is not commented out with a #
at the beginning of the line, and change it to no
:
PasswordAuthentication no
Save and close the file when you are finished. In nano
, use CTRL+O
to save, hit ENTER
to confirm the filename, then CTRL+X
to exit.
Reload the sshd
service to put these changes into effect:
- sudo systemctl reload sshd
Before exiting your current SSH session, make a test connection in another terminal to verify you can still connect.
In this tutorial we created an SSH key pair, copied our public key to a server, and (optionally) disabled password-based authentication completely.
For more information about SSH and the SSH service, including how to set up multifactor authentication, please read our related tutorials:
]]>Things do not always go according to plan: deployments can fail, existing resources may break unexpectedly, and you and your team could be forced to fix the issue as soon as possible. Understanding the methods to approach debugging your Terraform project is crucial when you need to make a swift response.
Similarly to developing with other programming languages and frameworks, setting log levels in Terraform to gain insight into its internal workflows with the necessary verbosity is a feature that can help you when troubleshooting. By logging the internal actions, you can uncover implicitly hidden errors, such as variables defaulting to an unsuitable data type. Also common with frameworks is the ability to import and use third-party modules (or libraries) to reduce code duplication between projects.
In this tutorial, you’ll verify that variables always have sensible values and you’ll specify exactly which versions of providers and modules you need to prevent conflicts. You’ll also enable various levels of debug mode verbosity, which can help you diagnose an underlying issue in Terraform itself.
terraform-troubleshooting
, instead of loadbalance
. During Step 2, do not include the pvt_key
variable and the SSH key resource.Note: This tutorial has specifically been tested with Terraform 1.0.2
.
Although the ability to make use of third-party modules and providers can minimize code duplication and effort when writing your Terraform code, it is also possible for developers of third-party modules to release new versions that can potentially bring breaking changes to your specific code. To prevent this, Terraform allows you to specify version boundaries to ensure that only the versions you want are installed and used. Specifying versions means that you will require the versions you’ve tested in your code, but it also leaves the possibility for a future update.
Version constraints are specified as strings and passed in to the version
parameter when you define module or provider requirements. As part of the Prerequisites, you’ve already requested the digitalocean
provider in provider.tf
. Run the following command to review what version requirements it specifies:
- cat provider.tf
You’ll find the provider code as follows:
terraform {
required_providers {
digitalocean = {
source = "digitalocean/digitalocean"
version = "~> 2.0"
}
}
}
...
In this case, you have requested the digitalocean
provider, and set the version to 2.x
, meaning that any version starting with 2
will be matched. You could also specify an explicit version, such as 2.10.1
.
Version constraints can be more complex than just specifying one version, as is the case above. They can contain one or more groups of conditions, separated by a comma (,
). The groups each define an acceptable range of versions and may include operators, such as:
>
, <
, >=
, <=
: for comparisons, such as >=1.0
, which would require the version to be equal to or greater than 1.0
.!=
: for excluding a specific version—!= 1.0
would deny version 1.0
from being used and requested.~>
: for matching the specified version up to the right-most version part, which is allowed to increment (~>1.5.10
will match 1.5.10
and 1.5.11
, but won’t match 1.5.9
).Here are two examples of version constraints with multiple groups:
>=1.0, <2.0
: allows all versions from the 1.0
series onward, up to 2.0
.>1.0, != 1.5
: allows versions greater than, but not equal to 1.0
, with the exception of 1.5
, which it also excludes.For a potential available version to be selected, it must pass every specified constraint and remain compatible with other modules and providers, as well as the version of Terraform that you’re using. If Terraform deems no combination acceptable, it won’t be able to perform any tasks because the dependencies remain unresolved. When Terraform identifies acceptable versions satisfying the constraints, it uses the latest one available.
In this section, you’ve learned about locking the range of module and resource versions that you can install in your project by specifying version constraints. This is useful when you want stability by using only tested and approved versions of third-party code. In the next section, you’ll configure Terraform to show more verbose logs, which are necessary for bug reports and further debugging in case of crashes.
There could be a bug or malformed input within your workflow, which may result in your resources not provisioning as intended. In such rare cases, it’s important to know how to access detailed logs describing what Terraform is doing. They may aid in pinpointing the cause of the error, tell you if it’s user-made, or prompt you to report the issue to Terraform developers if it’s an internal bug.
Terraform exposes the TF_LOG
environment variable for setting the level of logging verbosity. There are five levels:
TRACE
: the most elaborate verbosity, as it shows every step taken by Terraform and produces enormous outputs with internal logs.DEBUG
: describes what happens internally in a more concise way compared to TRACE
.ERROR
: shows errors that prevent Terraform from continuing.WARN
: logs warnings, which may indicate misconfiguration or mistakes, but are not critical to execution.INFO
: shows general, high-level messages about the execution process.To specify a desired log level, you’ll have to set the environment variable to the appropriate value:
- export TF_LOG=log_level
If TF_LOG
is defined, but the value is not one of the five listed verbosity levels, Terraform will default to TRACE
.
You’ll now define a Droplet resource and try deploying it with different log levels. You’ll store the Droplet definition in a file named droplets.tf
, so create and open it for editing:
- nano droplets.tf
Add the following lines:
resource "digitalocean_droplet" "test-droplet" {
image = "ubuntu-18-04-x64"
name = "test-droplet"
region = "fra1"
size = "s-1vcpu-1gb"
}
This Droplet will run Ubuntu 18.04 with one CPU core and 1GB RAM in the fra1
region; you’ll call it test-droplet
. That is all you need to define, so save and close the file.
Before deploying the Droplet, set the log level to DEBUG
by running:
- export TF_LOG=DEBUG
Then, plan the Droplet provisioning:
- terraform plan -var "do_token=${DO_PAT}"
The output will be very long, and you can inspect it more closely to find that each line starts with the level of verbosity (importance) in double brackets. You’ll see that most of the lines start with [DEBUG]
.
[WARN]
and [INFO]
are also present; that’s because TF_LOG
sets the lowest log level. This means that you’d have to set TF_LOG
to TRACE
to show TRACE
and all other log levels at the same time.
If an internal error occurred, Terraform will show the stack trace and exit, stopping execution. From there, you’ll be able to locate where in the source code the error occurred, and if it’s a bug, report it to Terraform developers. Otherwise, if it’s an error in your code, Terraform will point it out to you, so you can fix it in your project.
Here is how the log output would be when the DigitalOcean backend can’t verify your API token. It throws a user error because of incorrect input:
Output...
digitalocean_droplet.test-droplet: Creating...
2021/01/20 06:54:35 [ERROR] eval: *terraform.EvalApplyPost, err: Error creating droplet: POST https://api.digitalocean.com/v2/droplets: 401 Unable to authenticate you
2021/01/20 06:54:35 [ERROR] eval: *terraform.EvalSequence, err: Error creating droplet: POST https://api.digitalocean.com/v2/droplets: 401 Unable to authenticate you
Error: Error creating droplet: POST https://api.digitalocean.com/v2/droplets: 401 Unable to authenticate you
on droplets.tf line 1, in resource "digitalocean_droplet" "test-droplet":
1: resource "digitalocean_droplet" "test-droplet" {
...
To disable debug mode and reset the logging verbosity to its default level, clear the TF_LOG
environment variable by running:
- unset TF_LOG
You’ve now learned to enable more verbose logging modes. They are very useful for diagnosing crashes and unexpected Terraform behavior. In the next section, you’ll review verifying variables and preventing edge cases.
In this section, you’ll ensure that variables always have sensible and appropriate values according to their type and validation parameters.
In HCL (HashiCorp Configuration Language), when defining a variable you do not necessarily need to specify anything except its name. You would declare an example variable called test_ip
like this:
variable "test_ip" { }
You can then use this value through the code, passing its value in when you run Terraform.
While that will work, this definition has two shortcomings: first, you can not pass a value in at runtime; and second, it can be of any type (bool
, string
, and so on), which may not be suitable for its purpose. To remedy this, you should always specify its default value and type:
variable "test_ip" {
type = string
default = "8.8.8.8"
}
By setting a default
value, you ensure that the code referencing the variable remains operational in the event that a more specific value was not provided. When you specify a type, Terraform can validate the new value the variable should be set to, and show an error if it’s non-conforming to the type. An instance of this behavior would be trying to fit a string
into a number
.
You can provide a validation routine for variables that can give an error message if the validation fails. Examples of validation would be checking the length of the new value if it’s a string, or looking for at least one match with a RegEx expression in the case of structured data.
To add input validation to your variable, define a validation
block:
variable "test_ip" {
type = string
default = "8.8.8.8"
validation {
condition = can(regex("\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}", var.test_ip))
error_message = "The provided value is not a valid IP address."
}
}
Under validation
, you can specify two parameters within the curly braces:
condition
that accepts a bool
it will calculate, which will signify if the validation passes.error_message
that specifies the error message in case the validation does not pass.In this example, you compute the condition
by searching for a regex
match in the variable value. You pass that to the can
function. The can
function returns true
if the function that’s passed in as a parameter ran without errors, so it’s useful for checking when a function completed successfully or returned results.
The regex
function we’re using here accepts a Regular Expression (RegEx), applies it to a given string, and returns the matched substrings. The RegEx matches four pairs of three digit numbers, separated by dots between them. You can learn more about RegEx by visiting the Introduction to Regular Expressions tutorial.
You now know how to specify a default value for a variable, how to set its type, and how to enable input validation using RegEx expressions.
In this tutorial, you’ve troubleshooted Terraform by enabling debug mode and setting the log verbosity to appropriate levels. You’ve also learned about some of the advanced features of variables, such as declaring validation procedures and setting good defaults. Leaving out default values is a common pitfall that may cause strange issues further along in your project’s development.
For the stability of your project, locking third-party module and provider versions is recommended, since it leaves you with a stable system and an upgrade path when it becomes necessary.
Verification of input values for variables is not confined to matching with regex
. For more built-in functions that you can make use of, visit the official docs.
This tutorial is part of the How To Manage Infrastructure with Terraform series. The series covers a number of Terraform topics, from installing Terraform for the first time to managing complex projects.
]]>Terraform outputs are used to extract information about the infrastructure resources from the project state. Using other features of the Hashicorp Configuration Language (HCL), which Terraform uses, resource information can be queried and transformed into more complex data structures, such as lists and maps. Outputs are useful for providing information to external software, which can operate on the created infrastructure resources.
In this tutorial, you’ll learn about Terraform output syntax and its parameters by creating a simple infrastructure that deploys Droplets. You’ll also parse the outputs programmatically by converting them to JSON.
terraform-outputs
, instead of loadbalance
. During Step 2, do not include the pvt_key
variable and the SSH key resource.Note: This tutorial has specifically been tested with Terraform 1.0.2
.
In this section, you’ll declare a Droplet, deploy it to the cloud, and learn about outputs by defining one that will show the Droplet’s IP address.
From the terraform-outputs
directory you created as a prerequisite, create and open the droplets.tf
file for editing:
- nano droplets.tf
Add the following Droplet resource and output definition:
resource "digitalocean_droplet" "web" {
image = "ubuntu-20-04-x64"
name = "test-droplet"
region = "fra1"
size = "s-1vcpu-1gb"
}
output "droplet_ip_address" {
value = digitalocean_droplet.web.ipv4_address
}
You first declare a Droplet resource, called web
. Its actual name in the cloud will be test-droplet
, in the region fra1
, running Ubuntu 20.04.
Then, you declare an output called droplet_ip_address
. In Terraform, outputs are used to export and show internal and computed values and information about the resources. Here, you set the value
parameter, which accepts the data to output, to the IP address of the declared Droplet. At declare time, it’s unknown, but it will become available once the Droplet is deployed. Outputs are shown and accessible after each deployment.
Save and close the file, then deploy the project by running the following command:
- terraform apply -var "do_token=${DO_PAT}"
Enter yes
to apply when prompted. The end of the output will be similar to this:
Output...
digitalocean_droplet.web: Creating...
...
digitalocean_droplet.web: Creation complete after 32s [id=207631771]
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
Outputs:
droplet_ip_address = ip_address
The highlighted IP address belongs to your newly deployed Droplet. Applying the project deploys the resources to the cloud and shows the outputs at the end, when all resource attributes are available. Without the droplet_ip_address
output, Terraform would show no further information about the Droplet, except that it’s deployed.
Outputs can also be shown using the output
command:
- terraform output
The output will list all outputs
in the project:
Outputdroplet_ip_address = ip_address
You can also query a specific output by name by specifying it as an argument:
- terraform output output_name
For droplet_ip_address
, the output will consist of the IP address only:
Outputip_address
Except for specifying the mandatory value
, outputs have a few optional parameters:
description
: embeds short documentation detailing what the output shows.depends_on
: a meta parameter available at each resource that allows you to explicitly specify resources the output depends on that Terraform is not able to automatically deduce during planning.sensitive
: accepts a boolean value, which prevents the content of the output from being shown after deploying if set to true
.The sensitive
parameter is useful when the logs of the Terraform deployment will be publicly available, but the output contents should be kept hidden. You’ll now add it to your Droplet resource definition.
Open droplets.tf
for editing and add the highlighted line:
resource "digitalocean_droplet" "web" {
image = "ubuntu-20-04-x64"
name = "test-droplet"
region = "fra1"
size = "s-1vcpu-1gb"
}
output "droplet_ip_address" {
value = digitalocean_droplet.web.ipv4_address
sensitive = true
}
Save and close the file when you’re done. Deploy the project again by running:
- terraform apply -var "do_token=${DO_PAT}"
Enter yes
when prompted. You’ll see that the output is redacted:
Output...
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
droplet_ip_address = <sensitive>
Even if it’s marked as sensitive
, the output and its contents will still be available through other channels, such as viewing the Terraform state or querying the outputs directly.
In the next step, you’ll create a different Droplet and output structure, so destroy the currently deployed ones by running:
- terraform destroy -var "do_token=${DO_PAT}"
The output at the very end will be:
Output...
Destroy complete! Resources: 1 destroyed.
You’ve declared and deployed a Droplet and created an output that shows its IP address. You’ll now learn about using outputs to show more complex structures such as lists and maps.
In this section, you’ll deploy multiple Droplets from the same definition using the count
keyword, and output their IP addresses in various formats.
for
loopYou’ll need to modify the Droplet resource definition, so open it for editing:
- nano droplets.tf
Modify it to look like this:
resource "digitalocean_droplet" "web" {
count = 3
image = "ubuntu-20-04-x64"
name = "test-droplet-${count.index}"
region = "fra1"
size = "s-1vcpu-1gb"
}
You’ve specified that three Droplets should be created using the count
key and added the current index to the Droplet name, so that you’ll be able to later discern between them. Remove the existing output below. When you’re done, save and close the file.
Apply the code by running:
- terraform apply -var "do_token=${DO_PAT}"
Terraform will plan the creation of three numbered Droplets, called test-droplet-0
, test-droplet-1
, and test-droplet-2
. Enter yes
when prompted to finish the process. You’ll see the following output in the end:
Output...
Apply complete! Resources: 3 added, 0 changed, 0 destroyed.
This means that all three Droplets are successfully deployed and that all information about them is stored in the project state.
The easiest way to access their resource attributes is to use outputs, but creating one for each Droplet is not scalable. The solution is to use the for
loop to traverse through the list of Droplets and gather their attributes, or to alternatively use splat expressions (which you’ll learn about later in this step).
You’ll first define an output that will output the IP addresses of the three Droplets, paired with their names. Open droplets.tf
for editing:
- nano droplets.tf
Add the following lines:
resource "digitalocean_droplet" "web" {
count = 3
image = "ubuntu-20-04-x64"
name = "test-droplet-${count.index}"
region = "fra1"
size = "s-1vcpu-1gb"
}
output "droplet_ip_addresses" {
value = {
for droplet in digitalocean_droplet.web:
droplet.name => droplet.ipv4_address
}
}
The output value of droplet_ip_addresses
is constructed using a for
loop. Because it’s surrounded by braces, the resulting type will be a map. The loop traverses the list of Droplets, and for each instance, pairs its name with its IP address and appends it to the resulting map.
Save and close the file, then apply the project again:
- terraform apply -var "do_token=${DO_PAT}"
Enter yes
when prompted and you’ll receive the output contents at the end:
OutputApply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
droplet_ip_addresses = {
"test-droplet-0" = "ip_address"
"test-droplet-1" = "ip_address"
"test-droplet-2" = "ip_address"
}
The droplet_ip_addresses
output details the IP addresses of the three deployed droplets.
Using the Terraform output
command, you can get the contents of the output as JSON using its command argument:
- terraform output -json droplet_ip_addresses
The result will be similar to the following:
Output{"test-droplet-0":"ip_address","test-droplet-1":"ip_address","test-droplet-2":"ip_address"}
JSON parsing is widely used and supported in many programming languages. This way, you can programmatically parse the information about the deployed Droplet resources.
Splat expressions offer a compact way of iterating over all elements of a list, and collecting contents of an attribute from each of them, resulting in a list. A splat expression that would extract the IP addresses of the three deployed droplets would have the following syntax:
digitalocean_droplet.web[*].ipv4_address
The [*]
symbol traverses the list on its left and for each of the elements, takes the contents of its attribute specified on the right. If the reference on the left is not a list by itself, it will be converted to one where it will be the sole element.
You can open droplets.tf
for editing and modify the following lines to implement this:
resource "digitalocean_droplet" "web" {
count = 3
image = "ubuntu-20-04-x64"
name = "test-droplet-${count.index}"
region = "fra1"
size = "s-1vcpu-1gb"
}
output "droplet_ip_addresses" {
value = digitalocean_droplet.web[*].ipv4_address
}
After saving the file, apply the project by running the following command:
- terraform apply -var "do_token=${DO_PAT}"
You’ll receive output that is now a list, and which contains only the IP addresses of the Droplets:
Output...
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
droplet_ip_addresses = [
"ip_address",
"ip_address",
"ip_address",
]
To receive the output as JSON, run the following command:
- terraform output -json droplet_ip_addresses
The output will be a single array:
Output["ip_address","ip_address","ip_address"]
You’ve used outputs together with splat expressions and for
loops to export the IP addresses of the deployed Droplets. You’ve also received the output contents as JSON, and you’ll now use jq
—a tool for dynamically filtering JSON according to given expressions—to parse them.
jq
In this step, you’ll install and learn the basics of jq
, a tool for manipulating JSON documents. You’ll use it to parse the outputs of your Terraform project.
If you’re on Ubuntu, run the following command to install jq
:
- sudo snap install jq
On macOS, you can use Homebrew to install it:
- brew install jq
jq
applies the provided processing expression on given input, which can be piped in. The easiest task in jq
is to pretty print the input:
- terraform output -json droplet_ip_addresses | jq '.'
Passing in the identity operator (.
) means that the whole JSON document parsed from the input should be outputted without modifications:
Output[
"first_ip_address",
"second_ip_address",
"third_ip_address"
]
You can request just the second IP address using the array bracket notation, counting from zero:
- terraform output -json droplet_ip_addresses | jq '.[1]'
The output will be:
Output"second_ip_address"
To make the result of the processing an array, wrap the expression in brackets:
- terraform output -json droplet_ip_addresses | jq '[.[1]]'
You’ll get a pretty printed JSON array:
Output[
"second_ip_address"
]
You can retrieve parts of arrays instead of single elements by specifying a range of indexes inside the brackets:
- terraform output -json droplet_ip_addresses | jq '.[0:2]'
The output will be:
Output[
"first_ip_address",
"second_ip_address"
]
The range 0:2
returns the first two elements—the upper part of the range (2
) is not inclusive, so only elements at positions 0
and 1
are fetched.
You can now destroy the deployed resources by running:
- terraform destroy -var "do_token=${DO_PAT}"
In this step, you have installed jq
and used it to parse and manipulate the output of your Terraform project, which deploys three Droplets.
You have learned about Terraform outputs, using them to show details about the deployed resources and to export data structures for later external processing. You’ve also used outputs to show attributes of a single resource, as well as for showing constructed maps and lists containing resource attributes.
For more detailed information about the features of jq
, visit the official docs.
This tutorial is part of the How To Manage Infrastructure with Terraform series. The series covers a number of Terraform topics, from installing Terraform for the first time to managing complex projects.
]]>DigitalOcean Spaces is an object storage service that can be used to store large amounts of diverse, unstructured data. WordPress sites, which often include image and video assets, can be good candidates for object storage solutions. Using object storage for these types of static resources can optimize site performance by freeing up space and resources on your servers. For more information about object storage and WordPress check out our tutorial on How To Back Up a WordPress Site to Spaces.
In this tutorial, we will use a WordPress plugin that works directly with DigitalOcean Spaces to use it as the primary asset store. The DigitalOcean Spaces Sync plugin routes the data of our WordPress media library to Spaces and provides you with various configuration options based on your needs, streamlining the process of using object storage with your WordPress instance.
This tutorial assumes that you have a WordPress instance on a server as well as a DigitalOcean Space. If you do not have this set up, you can complete the following:
With these prerequisites in place, we’re ready to begin using this plugin.
Throughout this tutorial, we will be working with the wp-content/uploads
folder in our WordPress project, so it is important that this folder exists and has the correct permissions. You can create it with the mkdir
command using the -p
flag in order to create the folder if it doesn’t exist, and avoid throwing an error if it does:
- sudo mkdir -p /var/www/html/wp-content/uploads
You can now set permissions on the folder. First, set the ownership to your user (we will use sammy here, but be sure to use your non-root sudo
user), and group ownership to the www-data
group:
- sudo chown -R sammy:www-data /var/www/html/wp-content/uploads
Next, establish the permissions that will give the web server write access to this folder:
- sudo chmod -R g+w /var/www/html/wp-content/uploads
We will now be able to use our plugins to create a store in object storage for the assets in the wp-content/uploads
folder, and to engage with our assets from the WordPress interface.
The first step in using DigitalOcean Spaces Sync will be to install it in our WordPress folder. We can navigate to the plugin folder within our WordPress directory:
- cd /var/www/html/wp-content/plugins
From here, we can install DigitalOcean Spaces Sync using the wp
command:
- wp plugin install do-spaces-sync
To activate the plugin, we can run:
- wp plugin activate do-spaces-sync
From here, we can navigate to the plugins tab on the left-hand side of our WordPress administrative dashboard:
We should see DigitalOcean Spaces Sync in our list of activated plugins:
To manage the settings for DigitalOcean Spaces Sync, we can navigate to our Settings tab, and select DigitalOcean Spaces Sync from the menu:
DigitalOcean Spaces Sync will now give us options to configure our asset storage:
The Connection Settings field in the top half of the screen asks for our Spaces Access Key and Secret. It will then ask for our Container, which will be the name of our Space, and the Endpoint.
You can determine the endpoint of your Space based on its URL. For example, if the URL of your Space is https://example-name.nyc3.digitaloceanspaces.com
, then example-name
will be your bucket/container, and nyc3.digitaloceanspaces.com
will be your endpoint.
In the plugin’s interface, the Endpoint section will be pre-filled with the default https://ams3.digitaloceanspaces.com
. You should modify this endpoint if your Space lives in another region.
Next, you will be asked for File & Path Settings. In the field marked Full URL-path to files, you can input either a storage public domain, if your files will be stored only on your Space, or a full URL path, if you will store them on your Space and server.
For example, if your WordPress project is located in /var/www/html
, and you want to store files on both your server and Space, then you would enter:
http://your_server_ip/wp-content/uploads
in the Full URL-path to files field/var/www/html/wp-content/uploads
in the Local path fieldThe Storage prefix and Filemask settings are prefilled, and do not need to be modified unless you would like to specify certain types of files for your sync.
We will cover the specifics of storing files on your server and Space and on your Space alone in the following sections.
DigitalOcean Spaces Sync offers the option of saving files to your server while also syncing them to your Space. This utility can be helpful if you need to keep files on your server, but would also like backups stored elsewhere. We will go through the process of syncing a file to our Space while keeping it on our server. For the purposes of this example, we will assume that we have a file called sammy10x10.png
that we would like to store in our media library and on our Space.
First, navigate to the Settings tab on your WordPress administrative dashboard, and select DigitalOcean Spaces Sync from the menu of presented options.
Next, in the Connections Settings field, enter your Spaces Key and Secret, followed by your Container and Endpoint. Remember, if the URL of your Space is https://example-name.nyc3.digitaloceanspaces.com
, then example-name
will be your Container, and nyc3.digitaloceanspaces.com
will be your Endpoint. Test your connections by clicking the Check the Connection button at the bottom of the Connection Settings field:
Now we are ready to fill out the File & Path Settings.
In the Full URL-path to files field we can enter our full URL path, since we are saving our file on our server and on our Space. We will use our server’s IP here, but if you have a domain, you can swap out the IP address for your domain name. For more about registering domains with DigitalOcean, see our tutorial on How To Set Up a Host Name with DigitalOcean. In our case, the Full URL-path to files will be http://your_server_ip/wp-content/uploads
.
Next, we will fill out the Local path field with the local path to the uploads
directory: /var/www/html/wp-content/uploads
.
Because we are working with a single file, we do not need to modify the Storage prefix and Filemask sections. As your WordPress media library grows in size and variety, you can modify this setting to target individual file types using wildcards and extensions such as *.png
in the Filemask field.
Your final File & Path Settings will look like this:
Be sure to save your configuration changes by clicking the Save Changes button at the bottom of the screen.
Now we can add our file, sammy10x10.png
, to our WordPress media library. We will use the wp media import
command, which will import the file from our home directory to our WordPress media library. In this case, our home directory will belong to sammy, but in your case this will be your non-root sudo
user. As we move the file, we will use the --path
parameter to specify the location of our WordPress project:
- wp media import --path=/var/www/html/ /home/sammy/sammy10x10.png
Looking at our WordPress interface, we should now see our file in our Media Library. We can navigate there by following the Media Library tab on the left side of our WordPress administrative dashboard:
If we navigate to our Spaces page in the DigitalOcean control panel, we should also see the file in our Space.
Finally, we can navigate to our wp-content/uploads
folder, where WordPress will have created a sub-folder with the year and month. Within this folder we should see our sammy10x10.png
file.
The DigitalOcean Spaces Sync plugin has an additional option that will allow us to store files only on our Space, in case we would like to optimize space and resources on our server. We will work with another file, sammy-heart10x10.png
, and set our DigitalOcean Spaces Sync settings so that this file will be stored only on our Space.
First, let’s navigate back to the plugin’s main configuration page:
We can leave the Connection Settings information, but we will modify the File & Path Settings. First, in the Full URL-path to files, we will write the storage public domain. Again, we will use our server IP, but you can swap this out for a domain if you have one: http://uploads.your_server_ip
Next, we will navigate to Sync Settings, at the bottom of the page, and click the first box, which will allow us to “store files only in the cloud and delete after successful upload.” Your final File & Path Settings will look like this:
Be sure to save your changes by clicking the Save Changes button at the bottom of the screen.
Back on the command line, we will move sammy-heart10x10.png
from our user’s home directory to our Media Library using wp media import
:
- wp media import --path=/var/www/html/ /home/sammy/sammy-heart10x10.png
If we navigate back to our WordPress interface, we will not see sammy-heart10x10.png
or sammy10x10.png
in our Media Library. Next, if we return to the command line and navigate to our wp-content/uploads
directory, we should see that sammy-heart10x10.png
is missing from our timestamped sub-folder.
Finally, if we navigate to the Spaces page in the DigitalOcean control panel, we should see both files stored in our Space.
We have covered two different options you can use to store your WordPress media files to DigitalOcean Spaces using DigitalOcean Spaces Sync. This plugin offers additional options for customization, which you can learn more about by reading the developer’s article “Sync your WordPress media with DigitalOcean Spaces.”
If you would like more general information about working with Spaces, check out our introduction to DigitalOcean Spaces and our guide to best practices for performance on Spaces.
]]>Backing up important data is an essential part of managing any computer infrastructure. Although everyone’s needs are different when it comes to performing backups, maintaining backup data at an offsite location is a good practice.
The process of sending copies of data to an offsite location used to be a major logistical challenge. But with the advent of cloud-based storage services like Crashplan and Dropbox, as well as the development of object storage solutions like DigitalOcean Spaces, it has now become a much simpler task. Despite this, remembering to back up files and taking the time to upload them can still be a roadblock for some.
This is why people choose to use various tools to perform routine, automatic backups of their important data. In this tutorial, we will build a script around the s3cmd
command line tool which can be used to quickly upload data to DigitalOcean Spaces. We will then use crontab
to regularly invoke the backups script and upload files to our Space.
For this tutorial, you will need:
s3cmd
command line tool installed on your Droplet.Having some familiarity with shell scripting and the cron
job scheduler could also be helpful. For some guidance and additional context, consider reading “An Introduction to Shell Scripting” and “How To Schedule Routine Tasks With Cron and Anacron on a VPS.”
With our prerequisites in place, we’re ready to begin our backup automation process.
There are a number of tools which can be used to upload backup files to an object storage service automatically and at regular intervals. However, these can be difficult to configure and may not offer much in the way of flexibility. Using a shell script can be a much more elegant and straightforward approach to automating object storage backups.
For this tutorial, we will write a basic bash script which will create a backup of a file or directory using tar
. The script will then upload that backup to Spaces using the s3cmd
command line utility.
To get started, log into your Droplet and navigate to your home folder:
- cd ~
Once in the home folder, we’ll use nano
to create an empty file where we can write in our script:
- nano bkupscript.sh
We are now ready to begin writing our backups script in the text editor. As we build the script, we will explain each part of it in order, section by section.
At this point, bkupscript.sh
is only an empty text file. In order for our computer to invoke our executable file as commands, we need to begin our script with a hashbang. A hashbang is an interpreter directive which allows scripts or data files to be run as commands.
In our case, the hashbang will look like this:
#!/bin/bash
By including this at the top of the script, we are telling the shell to run the file’s commands in bash.
Next, we’ll tell our script the variables it will need to know in order to function correctly. We can add these directly below the hashbang at the top of the text file:
...
DATETIME=`date +%y%m%d-%H_%M_%S`
SRC=$1
DST=$2
GIVENNAME=$3
Let’s go through what we are assigning each of these variables to:
DATETIME
: This variable holds a timestamp to affix to the resulting filename so every file backed up to our Space has a unique name. This timestamp is created by invoking the date
command and formatting the output to show the last two digits of the year (%y
), the two digits of the month (%m
), the two digits of the day (%d
), the hour (%H
), the minutes (%M
) and the seconds (%S
).SRC
: This is the source path for the file or folder we want to backup. The $1
indicates that we are taking this value from the first parameter passed to the script.DST
: This variable represents the file’s destination. In our case this is the name of the Space to which we are uploading the backup. This name will come from the second parameter passed to the script as indicated by the $2
.GIVENNAME
: This variable hosts the user-chosen name for the destination file. The resulting filename will start with GIVENNAME
and will have the DATETIME
concatenated onto it. This name comes from the third parameter passed to the script ($3
).When writing a script, it’s helpful to add some tips or general advice which can assist users troubleshooting if their attempts to use it fail.
For our backups script, we will add a function called showhelp()
below our variables. This will print a series of messages to help users troubleshoot in case the script fails. When adding a function in bash, our syntax will look like this :
...
showhelp(){
}
This function will provide help messages by echoing a series of usage instructions on the screen. Each instruction should be presented as a string enclosed with double quotes. You’ll notice in our example below that some of the strings have \t
or \n
written at their beginning or end. These are escape characters which provide specific instructions for how the string should appear in the script’s output:
\t
indicates a tab space\n
indicates a line breakFeel free to add any usage details that would be helpful for you in between the curly brackets (just remember to precede any string with echo
). For demonstration purposes, we will add the following:
echo "\n\n############################################"
echo "# bkupscript.sh #"
echo "############################################"
echo "\nThis script will backup files/folders into a single compressed file and will store it in the current folder."
echo "In order to work, this script needs the following three parameters in the listed order: "
echo "\t- The full path for the folder or file you want to backup."
echo "\t- The name of the Space where you want to store the backup at (not the url, just the name)."
echo "\t- The name for the backup file (timestamp will be added to the beginning of the filename)\n"
echo "Example: sh bkupscript.sh ./testdir testSpace backupdata\n"<^>
The final showhelp
function should look something like this:
...
showhelp(
echo "\n\n############################################"
echo "# bkupscript.sh #"
echo "############################################"
echo "\nThis script will backup files/folders into a single compressed file and will store it in the current folder."
echo "In order to work, this script needs the following three parameters in the listed order: "
echo "\t- The full path for the folder or file you want to backup."
echo "\t- The name of the Space where you want to store the backup at (not the url, just the name)."
echo "\t- The name for the backup file (timestamp will be added to the beginning of the filename)\n"
echo "Example: sh bkupscript.sh ./testdir testSpace backupdata\n"
}
With our help text in place, we can move on to collecting the files we would like to backup in our Space.
Before our script can transfer anything to our Space, it first needs to gather the right files and consolidate them into a single package for us to upload. We can accomplish this by using the tar
utility and a conditional statement. Because we’re using tar
to create an archive file (sometimes known as a “zip” file), we will call this function tarandzip()
.
Let’s start by declaring the function and adding another echo
command to let users know that the script has begun gathering files:
...
tarandzip(){
echo "\n##### Gathering files #####\n"
}
Below the echo
command, we can add a tar
command which will do the work of collecting and compressing the files into a single output file.
tarandzip(){
echo "\n##### Gathering files #####\n"
tar -czvf $GIVENNAME-$DATETIME.tar.gz $SRC
}
You’ll notice that this tar
command is invoked with several options and variables:
c
: This flag tells tar
to compress the output file.z
: This instructs tar
to compress the file using gzip
.v
: This signifies the verbose
option, which instructs tar
to show more information in the output.f
: This flag instructs tar
to save the file with the filename indicated next.$GIVENNAME-$DATETIME.tar.gz
: The script calls these variables which we declared at the beginning in order to create the new file name. It does this by combining the $GIVENNAME
and $DATETIME
variables and adding the .tar.gz
extension to the end to form the new file name.$SRC
: This variable represents the source files or folders which we are instructing tar
to back up.This function should now be able to do what we want it to, but we can add a few more echo
calls to give the user some extra information about how the script is working. This can be done by adding a couple of conditional statements, like this:
if tar -czvf $GIVENNAME-$DATETIME.tar.gz $SRC; then
echo "\n##### Done gathering files #####\n"
return 0
else
echo "\n##### Failed to gather files #####\n"
return 1
fi
When the if
clause is invoked, it will execute the tar
command and wait for the result. If the result for the command is positive (meaning, it ran successfully), the lines between then
and else
will be executed. These are:
tar
process0
so that the portion of the code which invokes this function knows that everything worked fine.The else
portion of this function will only be executed if the tar
command finds an error while executing. In this case, the else
branch of the clause will:
tar
command failed1
, indicating that something went wrongFinally, we end the if/then/else
clause with a fi
, which in bash language means that the if
clause has ended.
The completed tarandzip()
function will look like this:
tarandzip(){
echo "\n##### Gathering files #####\n"
if tar -czvf $GIVENNAME-$DATETIME.tar.gz $SRC; then
echo "\n##### Done gathering files #####\n"
return 0
else
echo "\n##### Failed to gather files #####\n"
return 1
fi
}
With our tarandzip()
function in place, we are ready to set the script up to move our backups.
At this point, we can get our backup script to transfer a file to our Space by using the s3cmd
command. As with tarandzip
, we can also echo
a few strings and utilize an if/then/else
statement to keep users up to speed with how the script is working as it is running.
First we will declare our function. To keep things simple, name it movetoSpace()
:
...
movetoSpace(){
}
Now we can use s3cmd
and the variables we declared earlier to build the command which will push our backup files to our Space:
movetoSpace(){
s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST
}
Here’s what each part of this command means:
s3cmd
: This invokes s3cmd
, a command line tool used for managing object storage buckets.put
: This is a command used by s3cmd
for uploading data into a bucket.$GIVENNAME-$DATETIME.tar.gz
: This is the name of the backup that will be uploaded to our Space. It consists of the fourth and first variables we declared, followed by .tar.gz
, and is created by the tarandzip()
function from earlier.s3://$DST;
: This is the location where we want to upload the file. s3://
is a URI-like schema used specifically to describe object storage locations online, while $DST;
is the third variable we declared earlier.We now have a function which can upload our archived files to our Space. However, it doesn’t notify the user about its status. Let’s change this by echoing a string before the command to let the user know that it has been started, and after the function is complete, to let us know whether it was successful.
Let’s start by notifying the user that the process has begun:
movetoSpace(){
echo "\n##### MOVING TO SPACE #####\n"
s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST
}
Because the command will be either successful or unsuccessful (meaning, it will either upload the files to our Space or it won’t), we can let users know whether it worked by echoing one of two strings held in an if/then/else
statement, like this:
...
if s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST; then
echo "\n##### Done moving files to s3://"$DST" #####\n"
return 0
else
echo "\n##### Failed to move files to the Space #####\n"
return 1
fi
This conditional statement tells bash “If our s3cmd
command functions correctly, then let the user know that the script is done moving files to our Space. Otherwise, let the user know that the process failed.”
If the s3cmd
process completes successfully then the function prints a message to the screen (the first echo
string in the then
statement) indicating so, and returns a value of 0
, which informs the calling function that the operation has been completed. If the process fails, the then
clause prints the error message (the second echo
string), and returns a 1
so the rest of the script is aware that an error occurred.
Altogether, the movetoSpace()
function should look like this:
movetoSpace(){
echo "\n##### MOVING TO SPACE #####\n"
if s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST; then
echo "\n##### Done moving files to s3://"$DST" #####\n"
return 0
else
echo "\n##### Failed to move files to the Space #####\n"
return 1
fi
}
With the movetoSpace()
function written, we can move on to ensuring that the script is set up to call functions in the expected order through using conditional statements for flow control.
Though we have set up our script with functions, we have not provided an order for the script to complete those functions. At this point, we can introduce a calling function which will tell the rest of the script exactly how and when to run the other functions we’ve written.
Assuming everything has been configured correctly, when we run the script it should read the input command, assign the values from it to each variable, perform the tarandzip()
function, and follow that with the movetoSpace()
function. Should the script fail between any of those points, it should print the output of our showhelp()
function to help users with troubleshooting. We can order this and catch errors by adding a series of if/then/else
statements at the bottom of the file:
...
if [ ! -z "$GIVENNAME" ]; then
if tarandzip; then
movetoSpace
else
showhelp
fi
else
showhelp
fi
The first if
statement in the section above checks that the third variable passed is not empty. It does so in the following way:
[ ]
: The square brackets indicate that what lies between them is a test. In this case, the test is for a specific variable to not be empty.!
: In this case, this symbol means not
.-z
: This option indicates an empty string. So, combined with the !, we are asking for not an empty string.$GIVENNAME
: We are indicating here that the string we don’t want to be empty is the value assigned to the variable $GIVENNAME
. The reason we chose this approach is because this variable is assigned the value passed by the third parameter when calling the script from the command line. If we pass fewer than 3 parameters to the script, the code will not have a third parameter to assign the value to $GIVENNAME, so it will assign an empty string and this test will fail.Assuming that this first test is successful, it will then move on to the next if
statement and so on. If any of the if
statements return an error, the then
clause will call the showhelp
function and help text will be displayed in the output. Essentially, what this does is it glues together all the previous functions we’ve written out and gives bash the information it needs to perform them in the correct order.
Our script is now complete! You can verify that your script looks like the complete script we built in the section below.
The completed backups script that we created should look like this:
#!/bin/bash
DATETIME=`date +%y%m%d-%H_%M_%S`
SRC=$1
DST=$2
GIVENNAME=$3
showhelp(){
echo "\n\n############################################"
echo "# bkupscript.sh #"
echo "############################################"
echo "\nThis script will backup files/folders into a single compressed file and will store it in the current folder."
echo "In order to work, this script needs the following three parameters in the listed order: "
echo "\t- The full path for the folder or file you want to backup."
echo "\t- The name of the Space where you want to store the backup at (not the url, just the name)."
echo "\t- The name for the backup file (timestamp will be added to the beginning of the filename)\n"
echo "Example: sh bkupscript.sh ./testdir testSpace backupdata\n"
}
tarandzip(){
echo "\n##### Gathering files #####\n"
if tar -czvf $GIVENNAME-$DATETIME.tar.gz $SRC; then
echo "\n##### Done gathering files #####\n"
return 0
else
echo "\n##### Failed to gather files #####\n"
return 1
fi
}
movetoSpace(){
echo "\n##### MOVING TO SPACE #####\n"
if s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST; then
echo "\n##### Done moving files to s3://"$DST" #####\n"
return 0
else
echo "\n##### Failed to move files to the Space #####\n"
return 1
fi
}
if [ ! -z "$GIVENNAME" ]; then
if tarandzip; then
movetoSpace
else
showhelp
fi
else
showhelp
fi
Once you have verified your script, be sure to save and close the file (CTRL-x
, y
, then ENTER
) before exiting nano.
Now that we’re finished with building the script, we can move on to testing it out. Not only will this tell us whether we’ve written the script correctly, but it will also give us a chance to practice using the script.
When testing out a script like this, it’s usually a good idea to use dummy files. Even though we know it isn’t capable of destroying or removing data, it’s smart to play it safe by testing it with some unimportant files. We’ll first create a directory using the mkdir
command:
- mkdir backupthis
Next, we’ll create two empty files inside this directory using touch
:
- sudo touch backupthis/file1.txt
- sudo touch backupthis/file2.txt
We can now test the script by uploading the backupthis
directory and its contents to our Space. This is the format we will need to use in order to invoke the script:
- sh bkupscript.sh ./backupthis name_of_your_space testrun
Note Because the movetoSpace()
function automatically prepends s3://
to the destination variable (i.e., the name of your Space), this variable should just be the name of your Space and not its full URL. For example, if your Space’s URL is “https://example-space-name.nyc3.digitaloceanspaces.com”, you would write the test command like this:
- sh bkupscript.sh ./backupthis example-space-name testrun
The previous command will set the script in motion and will return output like this:
Output
##### Gathering files #####
./backupthis/
./backupthis/file1.txt
./backupthis/file2.txt
##### Done gathering files #####
##### MOVING TO SPACE #####
upload: 'testrun-210622-19_34_01.tar.gz' -> 's3://name_of_your_space /testrun-210622-19_34_01.tar.gz' [1 of 1]
162 of 162 100% in 8s 19.81 B/s done
##### Done moving files to s3://name_of_your_space #####
If you encounter any errors, please review your script to ensure that it matches our example. Also, make sure that your installation of s3cmd
has been properly configured and that both the access key and the secret key you’re using are correct.
After successfully testing the backups script, we can set up a cron
job which will use the script to perform regular backups to our Space. For the purpose of this tutorial, we’ll set it up to execute our backups script every minute.
First, we need to make the script executable:
- chmod +x bkupscript.sh
Now that the script can be executed as a command, we can edit the crontab
file to run the script every minute:
- crontab -e
The first time that you run crontab -e
, it will ask you to select an editor from a list:
no crontab for root - using an empty one
Select an editor. To change later, run 'select-editor'.
1. /bin/ed
2. /bin/nano <---- easiest
3. /usr/bin/vim.basic
4. /usr/bin/vim.tiny
Choose 1-4 [2]:
You can select the default nano, or another text editor of your choice.
Once in crontab
, we will add the following line at the bottom of the values that are already there:
* * * * * ~/bkupscript.sh ~/backupthis nameofyourspace cronupload
To save the changes, press CTRL-x
, then y
, then ENTER
.
After a minute or so, a new file will appear in your Space’s dashboard.
If you leave the cron
job running without making any changes, you will have a new file copied to your Space every minute. Once you’ve confirmed that cron
is running successfully, feel free to reconfigure crontab
to backup your files at your desired interval.
You now have a script that periodically compresses and ships your backups to a DigitalOcean Space!
In this tutorial, we have gone over how to create regularly scheduled offsite backups of your important files using a bash script, crontab
, and DigitalOcean Spaces. Although the script presented in this tutorial is only intended for demonstration purposes, it can be used as a foundation to build a production-ready version which could later be integrated with a CI/CD solution like Jenkins, Drone, or Travis CI.
If you want to learn more about CI/CD tools, you can do so by reading the following tutorials:
]]>Au cours de ce tutoriel, vous allez déployer une application de sondage Django conteneurisée dans un cluster Kubernetes.
Django est un framework web puissant qui peut vous aider à lancer votre application en Python. Il intègre plusieurs fonctionnalités pratiques comme un object-relational mapper, l’authentification des utilisateurs et une interface d’administration personnalisable pour votre application. Il comprend également un caching framework et encourage la conception de l’application propre grâce à son URL Dispatcher et son système de modèles.
Dans le tutoriel Comment créer une application Django et Gunicorn avec Docker, le tutoriel Django sur l’application de sondages a été modifié en prenant en considération la méthodologie du Twelve-Factor pour créer des applications web natives en nuage. Cette configuration conteneurisée a été cadrée et sécurisée par un certificat TLS Nginx reverse-proxy et Let’s Encrypt-signed dans le tutoriel Comment cadrer et sécuriser une application Django avec Docker, Nginx et Let’s Encrypt. Au cours de ce dernier tutoriel dans les séries Des conteneurs à Kubernetes avec Django, la nouvelle version de l’application de sondage Django sera déployée dans un cluster Kubernetes.
Kubernetes est un orchestrateur de conteneurs libre puissant qui automatise le déploiement, la mise à l’échelle et la gestion des applications conteneurisées. Les objets Kubernetes comme ConfigMaps et Secrets vous permettent de centraliser et de découpler la configuration de vos conteneurs, tandis que les contrôleurs comme les Deployments redémarrent automatiquement les conteneurs défaillants et permettent une mise à l’échelle rapide des répliques de conteneurs. L’activation du chiffrement TLS se fait à l’aide d’un objet Ingress et le contrôleur Ingress open-source ingress-nginx. L’add-on Kubernetes cert-manager renouvelle et publie des certificats à l’aide de l’autorité de certification gratuite Let’s Encrypt.
Pour suivre ce tutoriel, vous aurez besoin de :
kubectl
installé sur votre machine locale et configuré pour vous connecter à votre cluster. Vous pouvez en savoir plus sur l’installation de kubectl dans la documentation officielle
. Si vous utilisez un cluster DigitalOcean Kubernetes, veuillez consulter : Comment se connecter à un cluster DigitalOcean Kubernetes pour apprendre à vous connecter à votre cluster en utilisant kubectl
.your_domain.com
. Vous pouvez en obtenir un gratuitement sur Freenom ou utiliser le registre de domaine de votre choix.A
avec your_domain.com
pointant sur l’adresse IP publique du load balancer Ingress. Si vous utilisez DigitalOcean pour gérer les enregistrements DNS de votre domaine, consultez Comment gérer des enregistrements DNS pour apprendre à créer des enregistrements A
.Une fois ces composants configurés, vous êtes prêt à suivre ce guide.
Au cours de cette étape, nous allons cloner le code de l’application de GitHub et configurer des paramètres comme les identifiants de la base de données et les clés de stockage d’objets.
Vous pourrez trouver le code de l’application et le Dockerfile dans la branche polls-docker
du GitHub repository de l’application de sondage du tutorial Django. Ce référentiel contient le code pour l’exemple d’application de sondage utilisé dans la documentation de Django, qui vous apprend à développer une application de sondage à partir de zéro.
La branche polls-docker
contient une version dockerisée de l’application de sondage. Pour savoir de quelle manière l’application de sondage a été modifiée pour fonctionner efficacement dans un environnement conteneurisé, consultez Comment construire une application Django et Gunicorn avec Docker.
Tout d’abord, utilisez git
pour cloner la branche polls-docker
du GitHub repository de l’application de sondage Django sur votre machine locale :
- git clone --single-branch --branch polls-docker https://github.com/do-community/django-polls.git
Naviguez dans le répertoire django-polls
:
- cd django-polls
Ce répertoire contient le code Python de l’application Django, un Dockerfile
que Docker utilisera pour construire l’image du conteneur, ainsi qu’un fichier env
qui contient une liste de variables d’environnement à passer dans l’environnement d’exécution du conteneur. Inspectez le Dockerfile
:
- cat Dockerfile
OutputFROM python:3.7.4-alpine3.10
ADD django-polls/requirements.txt /app/requirements.txt
RUN set -ex \
&& apk add --no-cache --virtual .build-deps postgresql-dev build-base \
&& python -m venv /env \
&& /env/bin/pip install --upgrade pip \
&& /env/bin/pip install --no-cache-dir -r /app/requirements.txt \
&& runDeps="$(scanelf --needed --nobanner --recursive /env \
| awk '{ gsub(/,/, "\nso:", $2); print "so:" $2 }' \
| sort -u \
| xargs -r apk info --installed \
| sort -u)" \
&& apk add --virtual rundeps $runDeps \
&& apk del .build-deps
ADD django-polls /app
WORKDIR /app
ENV VIRTUAL_ENV /env
ENV PATH /env/bin:$PATH
EXPOSE 8000
CMD ["gunicorn", "--bind", ":8000", "--workers", "3", "mysite.wsgi"]
Ce Dockerfile utilise la Docker image officielle de Python 3.7.4 comme base et installe les exigences du paquet Python de Django et Gunicorn, telles que définies dans le fichier django-polls/requirements.txt
. Il supprime ensuite quelques fichiers de construction inutiles, copie le code de l’application dans l’image, et définit le PATH
d’exécution. Enfin, il déclare que le port 8000
sera utilisé pour accepter les connexions de conteneurs entrantes, et exécute gunicorn
avec 3 travailleurs, en écoutant sur le port 8000
.
Pour en savoir plus sur chacune des étapes de ce Dockerfile, consultez l’Étape 6 de Comment construire une application Django et Gunicorn avec Docker.
Maintenant, construisez l’image à l’aide de docker build
:
- docker build -t polls .
Nous nommons l’image polls
en utilisant le drapeau -t
et passons dans le répertoire courant comme contexte de construction, l’ensemble de fichiers à faire référence lors de la construction de l’image.
Après que Docker ait construit et étiqueté l’image, listez les images disponibles à l’aide de docker images
:
- docker images
Vous devriez voir l’image polls
listée :
OutputREPOSITORY TAG IMAGE ID CREATED SIZE
polls latest 80ec4f33aae1 2 weeks ago 197MB
python 3.7.4-alpine3.10 f309434dea3a 8 months ago 98.7MB
Avant de lancer le conteneur Django, nous devons configurer son environnement d’exécution à l’aide du fichier env
présent dans le répertoire actuel. Ce fichier sera transmis dans la commande docker run
utilisée pour exécuter le conteneur, et Docker injectera les variables d’environnement configurées dans l’environnement d’exécution du conteneur.
Ouvrez le fichier env
avec nano
ou votre éditeur préféré :
- nano env
DJANGO_SECRET_KEY=
DEBUG=True
DJANGO_ALLOWED_HOSTS=
DATABASE_ENGINE=postgresql_psycopg2
DATABASE_NAME=polls
DATABASE_USERNAME=
DATABASE_PASSWORD=
DATABASE_HOST=
DATABASE_PORT=
STATIC_ACCESS_KEY_ID=
STATIC_SECRET_KEY=
STATIC_BUCKET_NAME=
STATIC_ENDPOINT_URL=
DJANGO_LOGLEVEL=info
Remplissez les valeurs manquantes des clés suivantes :
DJANGO_SECRET_KEY
: définissez cette valeur à une valeur unique et imprévisible, comme indiqué dans les docs de Django. Une méthode de génération de cette clé est fournie dans Ajustement des paramètres du tutoriel sur les applications Django dimensionnables.DJANGO_ALLOWED_HOSTS
: : cette variable sécurise l’application et prévient les attaques d’en-tête d’hôte HTTP. Pour les besoins de test, définissez cette variable à *
, un joker qui correspondra à tous les hôtes. En production, vous devriez la définir sur your_domain.com
. Pour en savoir plus sur ce paramètre Django, consultez les paramètres de base dans les docs Django.DATABASE_USERNAME
: définissez ce paramètre sur l’utilisateur de la base de données PostgreSQL créé dans les étapes préalables.DATABASE_NAME
: définissez ce paramètres sur polls
ou le nom de la base de données PostgreSQL créée dans les étapes préalables.DATABASE_PASSWORD
: définissez ce paramètre sur le mot de passe de l’utilisateur PostgreSQL créé dans les étapes préalables.DATABASE_HOST
: définissez ce paramètre sur le nom d’hôte de votre base de données.DATABASE_PORT
: définissez ce paramètre sur le port de votre base de données.STATIC_ACCESS_KEY_ID
: définissez ce paramètre sur la clé d’accès de votre espace ou stockage d’objets.STATIC_SECRET_KEY
: définissez ce paramètre sur la clé d’accès de votre espace ou stockage d’objets Secret.STATIC_BUCKET_NAME
: définissez ce paramètre sur votre nom d’espace ou votre object storage bucket.STATIC_ENDPOINT_URL
: définissez ce paramètre sur les URL applicables de vos espaces ou du point final du stockage des abjets, par exemple https://space-name.nyc3.digitaloceanspaces.com
si votre espace se trouve dans la région nyc3
.Une fois que vous avez terminé vos modifications, enregistrez et fermez le fichier.
Au cours de la prochaine étape, nous allons exécuter un conteneur configuré localement et créer un schéma de base de données Nous allons également charger des actifs statiques, comme des feuilles de style ou des images, dans le stockage d’objets.
Une fois le conteneur créé et configuré, utilisez docker run
pour remplacer le paramètre CMD
dans le Dockerfile et créer le schéma de la base de données à l’aide des commandes manage.py
et manage.py migrate
:
- docker run --env-file env polls sh -c "python manage.py makemigrations && python manage.py migrate"
Nous lançons le container d’images polls:latest
, nous passons dans le fichier variable d’environnement que nous venons de modifier, et remplacons la commande Dockerfile par sh -c "python manage.py makemigrations python manage.py image"
, qui créera le schéma de base de données défini par le code de l’application.
Si vous exécutez cette opération pour la première fois, vous devriez voir apparaître ce qui suit :
OutputNo changes detected
Operations to perform:
Apply all migrations: admin, auth, contenttypes, polls, sessions
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying admin.0003_logentry_add_action_flag_choices... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying auth.0010_alter_group_name_max_length... OK
Applying auth.0011_update_proxy_permissions... OK
Applying polls.0001_initial... OK
Applying sessions.0001_initial... OK
Cela indique que le schéma de base de données a été créé avec succès.
Si vous exécutez migrate
une fois de plus, Django effectuera un no-op à moins que le schéma de base de données ait changé.
Ensuite, nous allons exécuter une autre instance du conteneur de l’application et utiliser un shell interactif à l’intérieur de celui-ci pour créer un utilisateur administratif pour le projet Django.
- docker run -i -t --env-file env polls sh
Vous obtiendrez une invite shell à l’intérieur du conteneur en cours d’exécution que vous pouvez utiliser pour créer l’utilisateur Django :
- python manage.py createsuperuser
Entrez un nom d’utilisateur, une adresse email et un mot de passe pour votre utilisateur, et après avoir créé l’utilisateur, appuyez sur CTRL+D
pour quitter le conteneur et le fermer.
Enfin, nous allons générer les fichiers statiques pour l’application et les télécharger sur l’espace DigitalOcean à l’aide de collectstatic
. Notez que cela peut prendre un peu de temps.
- docker run --env-file env polls sh -c "python manage.py collectstatic --noinput"
Une fois que ces fichiers sont générés et téléchargés, vous obtiendrez la sortie suivante.
Output121 static files copied.
Nous pouvons maintenant exécuter l’application :
- docker run --env-file env -p 80:8000 polls
Output[2019-10-17 21:23:36 +0000] [1] [INFO] Starting gunicorn 19.9.0
[2019-10-17 21:23:36 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
[2019-10-17 21:23:36 +0000] [1] [INFO] Using worker: sync
[2019-10-17 21:23:36 +0000] [7] [INFO] Booting worker with pid: 7
[2019-10-17 21:23:36 +0000] [8] [INFO] Booting worker with pid: 8
[2019-10-17 21:23:36 +0000] [9] [INFO] Booting worker with pid: 9
Ici, nous exécutons la commande par défaut définie dans le Dockerfile gunicorn --bind :8000 --workers 3 mysite.wsgi:application
et exposons le port de conteneur 8000
afin que le port 80
de votre machine locale soit mappé sur le port 8000
du conteneur polls
.
Vous devriez maintenant pouvoir naviguez jusqu’à l’application polls
à l’aide de votre navigateur web en tapant : http://localhost
. Comme il n’y a pas de route définie pour le chemin d’accès /
, vous obtiendrez probablement une erreur de recherche 404 Page Not Found
, qui est prévisible.
Naviguez sur http://localhost/polls
pour voir l’interface de l’application de sondage :
Pour voir l’interface administrative, allez à http://localhost/admin
. Vous devriez voir la fenêtre d’authentification de l’application de sondage :
Entrez le nom d’utilisateur administratif et le mot de passe que vous avez créé avec la commande createsuperuser
.
Après avoir été authentifié, vous pouvez accéder à l’interface administrative de l’application de sondage :
Notez que les actifs statiques pour les applications d’administration
et de sondage
sont livrées directement depuis le stockage d’objets. Pour confirmer ceci, consultez Testing Spaces Static File Delivery.
Lorsque vous avez terminé d’explorer, appuyez sur CTRL+C
dans la fenêtre de terminal en exécutant le conteneur de Docker pour terminer le conteneur.
Une fois que l’image Docker de l’application Django est testée, les actifs statiques chargés sur le stockage d’objets et le schéma de base de données configuré et fonctionnel avec votre application, vous êtes prêt à charger votre image de l’application Django sur un registre d’images comme Docker Hub.
Pour lancer votre application sur Kubernetes, votre image d’application doit être chargée sur un registre comme Docker Hub. Kubernetes ira extraire l’image de l’application de son référentiel, puis la déploiera sur votre cluster.
Vous pouvez utiliser un registre Docker privé, comme DigitalOcean Container Registry, actuellement gratuit en Early Access ou un registre Docker public comme Docker Hub. Docker Hub vous permet également de créer des référentiels Docker privés. Un référentiel public permet à quiconque de voir et d’extraire les images du conteneur, tandis qu’un référentiel privé vous permet de restreindre l’accès à vous et aux membres de votre équipe.
Au cours de ce tutoriel, nous allons pousser l’image Django sur le référentiel public Docker Hub créé dans les conditions préalablement citées. Vous pouvez également pousser votre image sur un référentiel privé, mais l’extraction d’images à partir d’un référentiel privé n’est pas traité dans le cadre de cet article. Pour en savoir plus sur l’authentification de Kubernetes avec Docker Hub et l’extraction d’images privées, veuillez consulter la section Extraire une image d’un référentiel privé dans les documents de Kubernetes.
Commencez par vous connecter à Docker Hub à partir de votre machine locale :
- docker login
OutputLogin with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username:
Pour vous connecter, utilisez votre nom d’utilisateur et votre mot de passe Docker Hub.
L’image Django a actuellement la balise polls:latest
. Pour la pousser sur votre référentiel Docker Hub, balisez à nouveau l’image en utilisant votre nom d’utilisateur Docker Hub et votre nom de référentiel :
- docker tag polls:latest your_dockerhub_username/your_dockerhub_repo_name:latest
Poussez l’image sur le référentiel :
- docker push sammy/sammy-django:latest
Pour ce tutoriel, nous utilisons le nom d’utilisateur Docker Hub sammy et le nom de référentiel sammy-django. Vous devez remplacer ces valeurs par votre propre nom d’utilisateur Docker Hub et votre nom de référentiel.
Vous verrez quelques résultats se mettre à jour alors que les couches des images sont poussées sur Docker Hub.
Maintenant que Kubernetes peut accéder à votre image sur Docker Hub, vous pouvez commencer à la déployer dans votre cluster.
Lorsque nous avons exécuté le conteneur Django localement, nous avons passé le fichier env
dans docker run
pour injecter des variables de configuration dans l’environnement d’exécution. Sur Kubernetes, les variables de configuration peuvent être injectées à l’aide de ConfigMaps et Secrets.
ConfigMaps permet de stocker des informations de configuration non confidentielles comme les paramètres de l’application. Secrets permet de stocker des informations sensibles comme les clés API et les identifiants de la base de données. Ils sont tous les deux injectés dans des conteneurs de manière similaire, mais Secrets dispose d’un contrôle d’accès et de fonctionnalités de sécurité supplémentaires comme encryption at rest. Secrets stocke également les données dans base64, tandis que ConfigMaps stocke les données en texte clair.
Pour commencer, créez un répertoire que vous nommerez yaml
dans lequel nous allons stocker nos manifestes Kubernetes. Naviguez dans le répertoire.
- mkdir yaml
- cd
Ouvrez un fichier appelé polls-configmap.yaml
dans nano
ou votre éditeur de texte préféré :
- nano polls-configmap.yaml
Collez–y le manifeste ConfigMap suivant :
apiVersion: v1
kind: ConfigMap
metadata:
name: polls-config
data:
DJANGO_ALLOWED_HOSTS: "*"
STATIC_ENDPOINT_URL: "https://your_space_name.space_region.digitaloceanspaces.com"
STATIC_BUCKET_NAME: "your_space_name"
DJANGO_LOGLEVEL: "info"
DEBUG: "True"
DATABASE_ENGINE: "postgresql_psycopg2"
Nous avons extrait la configuration non sensible du fichier env
modifié à l’étape 1 et l’avons collée dans un manifeste ConfigMap. L’objet ConfigMap se nomme polls-config
. Copiez-y les mêmes valeurs que celles que vous avez saisies dans le fichier env
à l’étape précédente.
Pour les besoins de test, laissez DJANGO_ALLOWED_HOSTS
avec *
pour désactiver le filtre configuré sur les en-têtes Host. Dans un environnement de production, vous devriez procéder à la même configuration sur le domaine de votre application.
Une fois que vous avez terminé de le modifier, enregistrez et fermez votre fichier.
Créez la ConfigMap dans votre cluster en utilisant kubectl apply
:
- kubectl apply -f polls-configmap.yaml
Outputconfigmap/polls-config created
Une fois la ConfigMap créée, nous allons créer le Secret que notre application utilisera à l’étape suivante.
Les valeurs de Secret doivent être base64-encoded, ce qui signifie que la création d’objets Secret dans votre cluster exige un peu plus d’implication que la création de ConfigMaps. Vous pouvez répéter le processus décrit à l’étape précédente, en codant manuellement les valeurs Secret en baseb64 et en les collant dans un fichier de manifeste. Vous pouvez également les créer à l’aide d’un fichier variable d’environnement, kubectl create
, et la balise --from-env-file
, ce que nous ferons au cours de cette étape.
Nous utiliserons à nouveau le fichier env
de l’étape 1, en supprimant les variables insérées dans la ConfigMap. Copiez le fichier env
appelé polls-secrets
dans le répertoire yaml
:
- cp ../env ./polls-secrets
Modifiez le fichier dans votre éditeur de texte préféré :
- nano polls-secrets
DJANGO_SECRET_KEY=
DEBUG=True
DJANGO_ALLOWED_HOSTS=
DATABASE_ENGINE=postgresql_psycopg2
DATABASE_NAME=polls
DATABASE_USERNAME=
DATABASE_PASSWORD=
DATABASE_HOST=
DATABASE_PORT=
STATIC_ACCESS_KEY_ID=
STATIC_SECRET_KEY=
STATIC_BUCKET_NAME=
STATIC_ENDPOINT_URL=
DJANGO_LOGLEVEL=info
Supprimez toutes les variables insérées dans le manifeste ConfigMap. Une fois que vous aurez terminé, vous devriez obtenir un résultat similaire à ce qui suit :
DJANGO_SECRET_KEY=your_secret_key
DATABASE_NAME=polls
DATABASE_USERNAME=your_django_db_user
DATABASE_PASSWORD=your_django_db_user_password
DATABASE_HOST=your_db_host
DATABASE_PORT=your_db_port
STATIC_ACCESS_KEY_ID=your_space_access_key
STATIC_SECRET_KEY=your_space_access_key_secret
Veillez à utiliser les mêmes valeurs que celles utilisées à l’étape 1. Une fois que vous avez terminé, enregistrez et fermez le fichier.
Créez le Secret dans votre cluster en utilisant kubectl create secret
:
- kubectl create secret generic polls-secret --from-env-file=poll-secrets
Outputsecret/polls-secret created
Ici, nous créons un objet Secret appelé polls-secret
dans lequel vous intégrerez le fichier secrets que nous venons de créer.
Vous pouvez inspecter le Secret en utilisant kubectl describe
:
- kubectl describe secret polls-secret
OutputName: polls-secret
Namespace: default
Labels: <none>
Annotations: <none>
Type: Opaque
Data
====
DATABASE_PASSWORD: 8 bytes
DATABASE_PORT: 5 bytes
DATABASE_USERNAME: 5 bytes
DJANGO_SECRET_KEY: 14 bytes
STATIC_ACCESS_KEY_ID: 20 bytes
STATIC_SECRET_KEY: 43 bytes
DATABASE_HOST: 47 bytes
DATABASE_NAME: 5 bytes
À ce stade, vous avez stocké la configuration de votre application dans votre cluster Kebernetes en utilisant des types d’objets Secret et ConfigMap. Nous sommes maintenant prêts à déployer l’application dans le cluster.
Au cours de cette étape, vous allez créer un Deployment pour votre application Django. Un Deployment est un contrôleur que vous pouvez utiliser pour gérer des applications apatrides dans votre cluster. Un contrôleur est une boucle de contrôle qui régule les charges de travail en les augmentant ou en les réduisant. Les contrôleurs redémarrent et suppriment les conteneurs défaillants.
Les Deployments contrôlent un ou plusieurs Pods, la plus petite unité déployable dans un cluster Kubernetes. Les Pods intègrent un ou plusieurs conteneurs. Pour en savoir plus sur les différents types de charges de travail que vous pouvez lancer, veuillez consulter Une introduction à Kubernetes.
Commencez par ouvrir le fichier appelé polls-deployment.yaml
dans votre éditeur de texte préféré :
- nano polls-deployment.yaml
Collez-y le manifeste de Deployment suivant :
apiVersion: apps/v1
kind: Deployment
metadata:
name: polls-app
labels:
app: polls
spec:
replicas: 2
selector:
matchLabels:
app: polls
template:
metadata:
labels:
app: polls
spec:
containers:
- image: your_dockerhub_username/app_repo_name:latest
name: polls
envFrom:
- secretRef:
name: polls-secret
- configMapRef:
name: polls-config
ports:
- containerPort: 8000
name: gunicorn
Renseignez le nom de l’image du conteneur applicable, qui doit faire référence à l’image Django Polls que vous avez poussée sur Docker Hub à l’étape 2.
Ici, nous allons définir un Deployment Kubernetes appelé polls-app
et l’étiqueter avec la paire de valeur app: pools
. Nous spécifions que nous souhaitons exécuter deux répliques du Pod défini sous le champ template
.
En utilisant envFrom
avec secretRef
et configMapRef
, nous spécifions que toutes les données du Secret polls-secret
et de la ConfigMap polls-config
doivent être injectées dans les conteneurs sous forme de variables d’environnement. Les clés ConfigMap et Secret deviennent les noms des variables d’environnement.
Enfin, nous allons exposer containerPort
8000
et le nommer gunicorn
.
Pour en savoir plus sur la configuration des Deployments de Kubernetes, veuillez consulter le document Deployments dans la documentation de Kubernetes.
Une fois que vous avez terminé de le modifier, enregistrez et fermez votre fichier.
Créez le Deployment dans votre cluster en utilisant kubectl apply -f
:
- kubectl apply -f polls-deployment.yaml
- deployment.apps/polls-app created
Vérifiez que le Deployment s’est correctement déployé en utilisant kubectl get
:
- kubectl get deploy polls-app
OutputNAME READY UP-TO-DATE AVAILABLE AGE
polls-app 2/2 2 2 6m38s
Si vous rencontrez une erreur ou que quelque chose qui ne fonctionne pas correctement, vous pouvez utiliser kubectl describe
pour inspecter le Deployment défaillant :
- kubectl describe deploy
Vous pouvez inspecter les deux Pods en utilisant kubectl get pod
:
- kubectl get pod
OutputNAME READY STATUS RESTARTS AGE
polls-app-847f8ccbf4-2stf7 1/1 Running 0 6m42s
polls-app-847f8ccbf4-tqpwm 1/1 Running 0 6m57s
Deux répliques de votre application Django sont maintenant opérationnelles dans le cluster. Pour accéder à l’application, vous devez créer un service Kubernetes, ce que nous ferons ensuite.
Au cours de cette étape, vous allez créer un Service pour votre application Django. Un Service Kubernetes est une abstraction qui vous permet d’exposer un ensemble de Pods en cours d’exécution en tant que service réseau. En utilisant un Service, vous pouvez créer un point final stable pour votre application qui ne change pas à mesure que les Pods périssent et sont recréés.
Il existe plusieurs types de Services, dont : les Services ClusterIP qui exposent le Service sur un IP interne de cluster, les NodePort Services qui exposent le Service sur chaque nœud au niveau d’un port statique appelé NodePort et les LoadBalancer Services qui intègrent un équilibreur de charge du trafic externe vers les Pods dans votre cluster (via NodePorts, ce qu’il crée automatiquement). Pour en savoir plus sur ces éléments, consultez le document Service dans la documentation de Kubernetes.
Pour notre configuration finale, nous allons utiliser un Service ClusterIP qui est exposé à l’aide d’un Ingress et du Controller Ingress configurés dans les conditions préalablement requises pour ce guide. Pour l’instant, pour vérifier que tout fonctionne correctement, nous allons créer un Service NodePort temporaire pour accéder à l’application Django.
Commencez par créer un fichier appelé polls-svc.yaml
en utilisant votre éditeur de texte préféré :
- nano polls-svc.yaml
Collez-y le manifeste de Service suivant :
apiVersion: v1
kind: Service
metadata:
name: polls
labels:
app: polls
spec:
type: NodePort
selector:
app: polls
ports:
- port: 8000
targetPort: 8000
Ici, nous créons un NodePort Service appelé polls
et lui donnons l’étiquette app: polls
. Nous sélectionnons ensuite les Pods de backend portant l’étiquette app: polls
et ciblons leurs ports 8000
.
Une fois que vous avez terminé de le modifier, enregistrez et fermez votre fichier.
Déploiement du Service avec kubectl apply
:
- kubectl apply -f polls-svc.yaml
Outputservice/polls created
Confirmez que votre Service a été créé en utilisant kubectl get svc
:
- kubectl get svc polls
OutputNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
polls NodePort 10.245.197.189 <none> 8000:32654/TCP 59s
Ce résultat affiche l’adresse IP interne du cluster de Service et NodePort (32654
). Pour nous connecter au service, nous avons besoin de l’adresse IP externe de nos nœuds de cluster :
- kubectl get node -o wide
OutputNAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
pool-7no0qd9e0-364fd Ready <none> 27h v1.18.8 10.118.0.5 203.0.113.1 Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9
pool-7no0qd9e0-364fi Ready <none> 27h v1.18.8 10.118.0.4 203.0.113.2 Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9
pool-7no0qd9e0-364fv Ready <none> 27h v1.18.8 10.118.0.3 203.0.113.3 Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9
Dans votre navigateur Web, consultez votre application de sondage en utilisant l’adresse IP externe de n’importe quel nœud et le NodePort. Compte tenu du résultat ci-dessus, l’URL de l’application devrait être : http://203.0.113.1:32654/polls
.
Vous devriez voir apparaître la même interface d’application de sondage que celle à laquelle vous avez accédé localement à l’étape 1 :
Vous pouvez répéter le même test en utilisant le chemin /admin
: http://203.0.113.1:32654/admin
. Vous devriez voir apparaître la même interface d’administration qu’auparavant :
À ce stade, vous avez déployé deux répliques du conteneur de l’application Django Polls en utilisant un Deployment. Vous avez également créé un terminal de réseau stable pour ces deux répliques et l’avez rendu accessible à l’extérieur à l’aide d’un Service NodePort.
La dernière étape de ce tutoriel consiste à sécuriser le trafic externe de votre application en utilisant HTTPS. Pour ce faire, nous allons utiliser le contrôleur Ingress ingress-nginx
installé dans les conditions préalables requises et créer un objet Ingress pour acheminer le trafic externe vers le Service Kubernetes polls
.
Les Ingresses de Kubernetes vous permettent d’acheminer le trafic de manière flexible depuis votre cluster Kubernetes vers Services à l’intérieur de votre cluster. Ceci se fait en utilisant des objets Ingress qui définissent des règles pour acheminer le trafic HTTP et HTTPS aux Services Kubernetes et les Ingress Controllers, qui implémentent les règles en équilibrant le trafic de charge et en l’acheminant vers les Services du terminal applicables.
Dans les conditions préalablement requises, vous avez installé le contrôleur Ingress ingress-nginx et l’add-on d’automatisation des certificats TLS cert-manager. Vous avez également défini des ClusterIssuers de simulation et de production pour votre domaine en utilisant l’autorité de certification Let’s Encrypt, et créé un Ingress pour tester l’émission de certificats et le cryptage TLS sur deux Services de backend factices. Avant de poursuivre avec cette étape, vous devez supprimer Ingress echo-ingress
créée dans le tutoriel préalable :
- kubectl delete ingress echo-ingress
Si vous le voulez, vous pouvez également supprimer les Services et Deployments factices en utilisant kubectl delete svc
et kubectl delete deploy
, mais cela n’est pas essentiel pour terminer ce tutoriel.
Vous devriez également avoir créé un dossier A
DNS avec your_domain.com
pointant sur l’adresse IP publique de l’équilibreur de charge Ingress. Si vous utilisez un équilibreur de charge DigitalOcean, vous pouvez trouver cette adresse IP dans la section Load Balancer du panneau de configuration. Si vous utilisez également DigitalOcean pour gérer les enregistrements DNS de votre domaine, consultez Comment gérer des enregistrements DNS pour apprendre à créer des enregistrements A
Si vous utilisez DigitalOcean Kubernetes, assurez-vous également de bien avoir implémenté le détour décrit à l’étape 5 de Comment configurer un Ingress Nginx avec Cert-Manager sur DigitalOcean Kubernetes.
Une fois que vous disposez d’un enregistrement A
pointant sur l’équilibreur de charge du contrôleur Ingress vous pouvez créer un Ingress pour your_domain.com
et le Service polls
.
Ouvrez un fichier appelé polls-ingress.yaml
en utilisant votre éditeur de texte préféré :
- nano polls-ingress.yaml
Collez-y le manifeste Ingress suivant :
[polls-ingress.yaml]
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: polls-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt-staging"
spec:
tls:
- hosts:
- your_domain.com
secretName: polls-tls
rules:
- host: your_domain.com
http:
paths:
- backend:
serviceName: polls
servicePort: 8000
Nous créons un objet Ingress appelé polls-ingress
et nous l’annotons pour instruire le plan de contrôle d’utiliser le contrôleur Ingress ingress-nginx et le ClusterIssuer de simulation. Nous activons également TLS pour your_domain.com
et stockons le certificat et la clé privée dans un secret appelé polls-tls
. Enfin, nous définissons une règle pour acheminer le trafic de l’hôte your_domain.com
vers le Service polls
sur le port 8000
.
Une fois que vous avez terminé de le modifier, enregistrez et fermez votre fichier.
Créez l’Ingress dans votre cluster en utilisant kubectl apply
:
- kubectl apply -f polls-ingress.yaml
Outputingress.networking.k8s.io/polls-ingress created
Vous pouvez utiliser kubectl describe
pour suivre l’état de l’Ingress que vous venez de créer :
- kubectl describe ingress polls-ingress
OutputName: polls-ingress
Namespace: default
Address: workaround.your_domain.com
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
polls-tls terminates your_domain.com
Rules:
Host Path Backends
---- ---- --------
your_domain.com
polls:8000 (10.244.0.207:8000,10.244.0.53:8000)
Annotations: cert-manager.io/cluster-issuer: letsencrypt-staging
kubernetes.io/ingress.class: nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 51s nginx-ingress-controller Ingress default/polls-ingress
Normal CreateCertificate 51s cert-manager Successfully created Certificate "polls-tls"
Normal UPDATE 25s nginx-ingress-controller Ingress default/polls-ingress
Vous pouvez également exécuter un describe
sur le certificat polls-tls
afin de confirmer à nouveau que sa création est probante :
- kubectl describe certificate polls-tls
Output. . .
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 3m33s cert-manager Issuing certificate as Secret does not exist
Normal Generated 3m32s cert-manager Stored new private key in temporary Secret resource "polls-tls-v9lv9"
Normal Requested 3m32s cert-manager Created new CertificateRequest resource "polls-tls-drx9c"
Normal Issuing 2m58s cert-manager The certificate has been successfully issued
Cela confirme que le certificat TLS a bien été émis et que le chiffrement HTTPS est maintenant actif pour your_domain.com
.
Étant donné que nous avons utilisé le ClusterIssuer de simulation, la plupart des navigateurs web ne feront pas confiance au faux certificat Let’s Encrypt qu’il a émis, de sorte que la navigation sur your_domain.com
vous renverra vers une page d’erreur.
Pour envoyer une requête de test, nous utiliserons wget
à partir de la ligne de commande :
- wget -O - http://your_domain.com/polls
Output. . .
ERROR: cannot verify your_domain.com's certificate, issued by ‘CN=Fake LE Intermediate X1’:
Unable to locally verify the issuer's authority.
To connect to your_domain.com insecurely, use `--no-check-certificate'.
Nous allons utiliser la balise --no-check-certificate
suggérée pour contourner la validation du certificat :
- wget --no-check-certificate -q -O - http://your_domain.com/polls
Output
<link rel="stylesheet" type="text/css" href="https://your_space.nyc3.digitaloceanspaces.com/django-polls/static/polls/style.css">
<p>No polls are available.</p>
Le résultat ainsi obtenu affiche le HTML de la page d’interface /polls
, confirmant également que la feuille de style est servie à partir du stockage d’objets.
Maintenant que vous avez réussi à tester la délivrance de certificats en utilisant le ClusterIssuer de simulation, vous pouvez modifier l’Ingress pour utiliser le ClusterIssuer de production.
Ouvrez polls-ingress.yaml
pour l’éditer à nouveau :
- nano polls-ingress.yaml
Modifiez l’annotation cluster-issuer
[polls-ingress.yaml]
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: polls-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
tls:
- hosts:
- your_domain.com
secretName: polls-tls
rules:
- host: your_domain.com
http:
paths:
- backend:
serviceName: polls
servicePort: 8000
Lorsque vous avez terminé, enregistrez et fermez le fichier. Mettez à jour l’Ingress en utilisant kubectl apply
:
- kubectl apply -f polls-ingress.yaml
Outputingress.networking.k8s.io/polls-ingress configured
Vous pouvez utiliser kubectl describe certificate polls-tls
et kubectl describe ingress polls-ingress
pour suivre l’état de délivrance du certificat :
- kubectl describe ingress polls-ingress
Output. . .
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 23m nginx-ingress-controller Ingress default/polls-ingress
Normal CreateCertificate 23m cert-manager Successfully created Certificate "polls-tls"
Normal UPDATE 76s (x2 over 22m) nginx-ingress-controller Ingress default/polls-ingress
Normal UpdateCertificate 76s cert-manager Successfully updated Certificate "polls-tls"
Le résultat ci-dessus confirme que le nouveau certificat de production a bien été émis et stocké avec succès dans le Secret polls-tls
.
Naviguez vers your_domain.com/polls
dans votre navigateur web pour confirmer que le cryptage HTTPS est activé et que tout fonctionne comme prévu. Vous devriez voir l’interface de l’application de sondage :
Vérifiez que le cryptage HTTPS est actif dans votre navigateur web. Si vous utilisez Google Chrome, tout fonctionne correctement si vous atteignez la page ci-dessus sans aucune erreur. En outre, vous devriez voir un cadenas dans la barre d’URL. Cliquez sur le cadenas pour inspecter les détails du certificat Let’s Encrypt.
Pour procéder à la tâche finale de nettoyage, vous pouvez facultativement commuter le type de Service polls
de NodePort à Type ClusterIP interne uniquement.
Modifiez polls-svc.yaml
en utilisant votre éditeur de texte :
- nano polls-svc.yaml
Changez le type
de NodePort
à ClusterIP
:
apiVersion: v1
kind: Service
metadata:
name: polls
labels:
app: polls
spec:
type: ClusterIP
selector:
app: polls
ports:
- port: 8000
targetPort: 8000
Une fois que vous avez terminé de le modifier, enregistrez et fermez votre fichier.
Déployez les changements en utilisant kubectl apply
:
- kubectl apply -f polls-svc.yaml --force
Outputservice/polls configured
Confirmez que votre Service a bien été modifié en utilisant kubectl get svc
:
- kubectl get svc polls
OutputNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
polls ClusterIP 10.245.203.186 <none> 8000/TCP 22s
Ce résultat montre que le type de Service est désormais configuré sur ClusterIP. La seule façon d’y accéder consiste à le faire via votre domaine et l’Ingress créé à cette étape.
Au cours de ce tutoriel, vous avez déployé une application Django évolutive et sécurisée HTTPS dans un cluster Kubernetes. Le contenu statique est directement extrait du stockage d’objets. Le nombre de Pods en cours d’exécution peut rapidement être augmenté ou diminué en utilisant le champ replicas
dans le manifeste de Deployment polls-app
.
Si vous utilisez un espace DigitalOcean, vous pouvez également activer la livraison d’actifs statiques via un réseau de distribution de contenu et créer un sous-domaine personnalisé pour votre espace. Veuillez consulter la section Activer CDN du document Comment configurer une application Django évolutive avec des bases de données et des espaces gérés par DigitalOcean pour en savoir plus.
Pour découvrir le reste de la série, veuillez consulter notre page sur la série Du conteneur aux Kubernetes avec Django.
]]>В этом учебном модуле мы развернем контейнеризованное приложение Django polls в кластере Kubernetes.
Django — это мощная веб-структура, позволяющая быстро развернуть ваше приложение Python с нуля. Она включает ряд удобных функций, в том числе реляционную карту объектов, аутентификацию пользователей и настраиваемый административный интерфейс для вашего приложения. Также она включает систему кэширования и помогает проектировать оптимизированные приложения с помощью диспетчера URL и системы шаблонов.
В учебном модуле Создание приложения Django и Gunicorn с помощью Docker мы изменили приложение Polls из учебного модуля Django согласно методологии Двенадцать факторов, предназначенной для построения масштабируемых веб-приложений, оптимизированных для работы в облаке. Для масштабирования и защиты контейнера использовался обратный прокси-сервер Nginx и подписанный Let’s Encrypt сертификат TLS (см. описание в учебном модуле Масштабирование и защита приложения Django с помощью Docker, Nginx и Let’s Encrypt). В этом последнем учебном модуле из серии От контейнеров к Kubernetes с Django мы покажем, как развернуть модернизированное приложение Django polls в кластере Kubernetes.
Kubernetes — мощная система оркестрации контейнеров с открытым исходным кодом, помогающая автоматизировать развертывание, масштабирование и управление контейнеризованными приложениями. Такие объекты Kubernetes как ConfigMaps и Secrets позволяют централизовать конфигурацию и отсоединить ее от контейнеров, а такие контроллеры как Deployments автоматически перезагружают неработающие контейнеры и позволяют быстро масштабировать реплики контейнеров. Шифрование TLS активируется с помощью объекта Ingress и контроллера ingress-nginx с открытым исходным кодом. Надстройка Kubernetes cert-manager проверяет и выпускает сертификаты, используя бесплатный центр сертификации Let’s Encrypt.
Для данного обучающего модуля вам потребуется следующее:
kubectl
, установленный на локальном компьютере и настроенный для подключения к вашему кластеру. Дополнительную информацию об установке kubectl
можно найти в официальной документации. Если вы используете кластер DigitalOcean Kubernetes, ознакомьтесь с руководством Подключение к кластеру DigitalOcean Kubernetes, чтобы узнать, как подключиться к кластеру с помощью kubectl
.your_domain.com
. Вы можете получить бесплатный домен на Freenom или зарегистрировать доменное имя по вашему выбору.A
для your_domain.com
, указывающая на публичный IP-адрес балансировщика нагрузки Ingress. Если вы используете DigitalOcean для управления записями DNS вашего домена, руководство Управление записями DNS поможет вам научиться создавать записи класса A
.Проверив наличие этих компонентов, вы можете начать прохождение этого обучающего модуля.
На этом шаге мы клонируем код приложения с GitHub и настроим такие параметры, как учетные данные БД и ключи хранения объектов.
Код приложения и файл Dockerfile можно найти в ветви polls-docker
репозитория Django Tutorial Polls App на GitHub. Этот репозиторий содержит код приложения Polls, используемого в документации по Django в качестве образца, на примере которого показывается, как создать приложение для опросов с нуля.
Ветвь polls-docker
содержит размещенную в контейнере Docker версию приложения Polls. Более подробную информацию об изменениях, внесенных в приложение Polls для эффективной работы в контейнеризированной среде, можно найти в руководстве Создание приложения Django и Gunicorn с помощью Docker.
Для начала используйте git
, чтобы клонировать ветвь polls-docker
репозитория Django Tutorial Polls App на GitHub на локальном компьютере:
- git clone --single-branch --branch polls-docker https://github.com/do-community/django-polls.git
Перейдите в каталог django-polls
:
- cd django-polls
В этом каталоге содержится код Python приложения Django, файл Dockerfile
, который Docker будет использовать для построения образа контейнера, а также файл env
, содержащий список переменных среды для передачи в рабочую среду контейнера. Проверьте файл Dockerfile
:
- cat Dockerfile
OutputFROM python:3.7.4-alpine3.10
ADD django-polls/requirements.txt /app/requirements.txt
RUN set -ex \
&& apk add --no-cache --virtual .build-deps postgresql-dev build-base \
&& python -m venv /env \
&& /env/bin/pip install --upgrade pip \
&& /env/bin/pip install --no-cache-dir -r /app/requirements.txt \
&& runDeps="$(scanelf --needed --nobanner --recursive /env \
| awk '{ gsub(/,/, "\nso:", $2); print "so:" $2 }' \
| sort -u \
| xargs -r apk info --installed \
| sort -u)" \
&& apk add --virtual rundeps $runDeps \
&& apk del .build-deps
ADD django-polls /app
WORKDIR /app
ENV VIRTUAL_ENV /env
ENV PATH /env/bin:$PATH
EXPOSE 8000
CMD ["gunicorn", "--bind", ":8000", "--workers", "3", "mysite.wsgi"]
В качестве базы в файле Dockerfile используется официальный образ Docker Python 3.7.4, который устанавливает требуемые пакеты Python для Django и Gunicorn в соответствии с файлом django-polls/requirements.txt
. Затем он удаляет некоторые ненужные файлы сборки, копирует код приложения в образ и устанавливает параметр выполнения PATH
. И в заключение заявляет, что порт 8000
будет использоваться для принятия входящих подключений контейнера, и запускает gunicorn
с тремя рабочими элементами, прослушивающими порт 8000
.
Дополнительную информацию о каждом этапе в Dockerfile см. в шаге 6 руководства Создание приложения Django и Gunicorn с помощью Docker.
Теперь создайте образ с помощью docker build
:
- docker build -t polls .
Назовем образ polls
, используя флаг -t
, и передадим в текущий каталог как контекст сборки набор файлов для справки при построении образа.
После того как Docker создаст и отметит образ, перечислите доступные образы, используя docker images
:
- docker images
Вы должны увидеть образ polls
:
OutputREPOSITORY TAG IMAGE ID CREATED SIZE
polls latest 80ec4f33aae1 2 weeks ago 197MB
python 3.7.4-alpine3.10 f309434dea3a 8 months ago 98.7MB
Перед запуском контейнера Django нам нужно настроить его рабочую среду с помощью файла env
, присутствующего в текущем каталоге. Этот файл передается в команду docker run
, которая используется для запуска контейнера, после чего Docker внедрит настроенные переменные среды в рабочую среду контейнера.
Откройте файл env
с помощью nano
или своего любимого редактора:
- nano env
DJANGO_SECRET_KEY=
DEBUG=True
DJANGO_ALLOWED_HOSTS=
DATABASE_ENGINE=postgresql_psycopg2
DATABASE_NAME=polls
DATABASE_USERNAME=
DATABASE_PASSWORD=
DATABASE_HOST=
DATABASE_PORT=
STATIC_ACCESS_KEY_ID=
STATIC_SECRET_KEY=
STATIC_BUCKET_NAME=
STATIC_ENDPOINT_URL=
DJANGO_LOGLEVEL=info
Заполните недостающие значения для следующих ключей:
DJANGO_SECRET_KEY
: задает уникальное, непрогнозируемое значение, как описано в документации Django. Один метод генерирования этого ключа представлен в разделе Изменение настроек приложения руководства Масштабируемое приложение Django.DJANGO_ALLOWED_HOSTS
: эта переменная защищает приложение и предотвращает атаки через заголовки хоста HTTP. С целью тестирования установите *
как подстановочный символ, подходящий для всех хостов. В производственной среде следует использовать значение your_domain.com
. Дополнительную информацию об этой настройке Django см. в разделе Core Settings (Базовые настройки) документации Django.DATABASE_USERNAME
: задает пользователя базы данных PostgreSQL, созданного на предварительных этапах.DATABASE_NAME
: задает polls
или имя базы данных, созданной на предварительных этапах.DATABASE_PASSWORD
: задает пароль пользователя базы данных PostgreSQL, созданного на предварительных этапах.DATABASE_HOST
: задает имя хоста базы данных.DATABASE_PORT
: укажите порт вашей базы данных.STATIC_ACCESS_KEY_ID
: укажите ключ доступа своего пространства или хранилища объектов.STATIC_SECRET_KEY
: укажите секретный ключ своего пространства или хранилища объектов.STATIC_BUCKET_NAME
: укажите имя своего пространства или корзину хранилища объектов.STATIC_ENDPOINT_URL
: укажите URL соответствующего пространства или конечного пункта хранилища объектов, например https://your_space_name.nyc3.digitaloceanspaces.com
, если ваше пространство находится в области nyc3
.После завершения редактирования сохраните и закройте файл.
На следующем шаге мы запустим настроенный контейнер в локальной среде и создадим схему базы данных. Также мы выгрузим в хранилище объектов статические ресурсы, такие как таблицы стилей и изображения.
После построения и настройки контейнера используйте docker run
, чтобы заменить заданную команду CMD
в файле Dockerfile и создайте схему базы данных, используя команды manage.py makemigrations
и manage.py migrate
:
- docker run --env-file env polls sh -c "python manage.py makemigrations && python manage.py migrate"
Мы запускаем образ контейнера polls:latest
, передаем в только что измененный файл переменной среды и переопределяем команду Dockerfile с помощью sh -c "python manage.py makemigrations && python manage.py migrate"
, которая создаст схему базы данных, определяемую кодом приложения.
Если запускаете эти команды впервые, вы должны увидеть следующее:
OutputNo changes detected
Operations to perform:
Apply all migrations: admin, auth, contenttypes, polls, sessions
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying admin.0003_logentry_add_action_flag_choices... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying auth.0010_alter_group_name_max_length... OK
Applying auth.0011_update_proxy_permissions... OK
Applying polls.0001_initial... OK
Applying sessions.0001_initial... OK
Это означает, что схема базы данных успешно создана.
При последующем запуске команды migrate
Django не будет выполнять никаких операций, если схема базы данных не изменилась.
Затем мы запустим еще один экземпляр контейнера приложения и используем внутри него интерактивную оболочку для создания административного пользователя проекта Django.
- docker run -i -t --env-file env polls sh
Вы получите строку оболочки внутри работающего контейнера, которую сможете использовать для создания пользователя Django:
- python manage.py createsuperuser
Введите имя пользователя, адрес электронной почты и пароль для пользователя, а после создания пользователя нажмите CTRL+D
для выхода из контейнера и его удаления.
В заключение мы создадим статические файлы приложения и загрузим их в пространство DigitalOcean с помощью collectstatic
. Обратите внимание, что для завершения процесса может потребоваться время.
- docker run --env-file env polls sh -c "python manage.py collectstatic --noinput"
После создания и загрузки файлов вы получите следующий вывод.
Output121 static files copied.
Теперь мы можем запустить приложение:
- docker run --env-file env -p 80:8000 polls
Output[2019-10-17 21:23:36 +0000] [1] [INFO] Starting gunicorn 19.9.0
[2019-10-17 21:23:36 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
[2019-10-17 21:23:36 +0000] [1] [INFO] Using worker: sync
[2019-10-17 21:23:36 +0000] [7] [INFO] Booting worker with pid: 7
[2019-10-17 21:23:36 +0000] [8] [INFO] Booting worker with pid: 8
[2019-10-17 21:23:36 +0000] [9] [INFO] Booting worker with pid: 9
Мы запустим определенную в Dockerfile команду по умолчанию gunicorn --bind :8000 --workers 3 mysite.wsgi:application
и откроем порт контейнера 8000
, чтобы установить соответствие порта 80
на локальном компьютере и порта 8000
контейнера polls
.
Теперь у вас должна появиться возможность открыть приложение polls
в браузере, для чего нужно ввести http://localhost
в адресную строку. Поскольку маршрут для пути /
не определен, скорее всего, вы получите ошибку 404 Страница не найдена
.
Введите в адресную строку http://localhost/polls
, чтобы увидеть интерфейс приложения Polls:
Чтобы открыть интерфейс администрирования, введите адрес http://localhost/admin
. Вы должны увидеть окно аутентификации администратора приложения «Опросы»:
Введите имя администратора и пароль, которые вы создали с помощью команды createsuperuser
.
После аутентификации вы сможете получить доступ к административному интерфейсу приложения «Опросы»:
Обратите внимание, что статические активы приложений admin
и polls
поступают напрямую из хранилища объекта. Чтобы убедиться в этом, ознакомьтесь с материалами Тестирование доставки статических файлов пространства.
После завершения изучения данных нажмите CTRL+C
в окне терминала, где запущен контейнер Docker, чтобы удалить контейнер.
Когда образ Docker приложения Django будет протестирован, статичные ресурсы будут выгружены в хранилище объектов, а схема базы данных будет настроена и готова к выгрузке образа приложения Django в реестр образов, например в Docker Hub.
Чтобы развернуть приложение в Kubernetes, необходимо будет выгрузить образ приложения в реестр, например, в Docker Hub. Kubernetes извлечет образ приложения из репозитория, а затем развернет его в кластере.
Вы можете использовать частный реестр Docker, например, реестр контейнеров DigitalOcean Container Registry, предварительная версия которого в настоящее время предоставляется бесплатно, или публичный реестр Docker, такой как Docker Hub. Docker Hub также позволяет создавать частные репозитории Docker. Публичный репозиторий позволяет любому пользователю видеть и извлекать образы контейнеров, а в частном репозитории доступ имеется только у вас и у членов вашей команды.
В этом учебном модуле мы разместим образ Django в публичном репозитории Docker Hub, созданном на этапе предварительных требований. Также вы можете перенести образ в частный репозиторий, однако извлечение образов из частного репозитория в этой статье не описывается. Чтобы получить более подробную информацию об аутентификации Kubernetes с Docker Hub и извлечении образов из частного репозитория, ознакомьтесь со статьей Pull an Image from a Private Registry (Извлечение образа из частного реестра) в документации по Kubernetes.
Для начала выполните вход в Docker Hub на локальном компьютере:
- docker login
OutputLogin with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username:
Введите имя пользователя и пароль Docker Hub для входа в систему.
Образ Django сейчас помечен тегом polls:latest
. Чтобы поместить его в ваш репозиторий Docker Hub, измените теги образа, указав свое имя пользователя Docker Hub и имя репозитория:
- docker tag polls:latest your_dockerhub_username/your_dockerhub_repo_name:latest
Разместите образ в репозитории:
- docker push sammy/sammy-django:latest
В этом учебном модуле мы используем имя пользователя Docker Hub sammy и имя репозитория sammy-django. Вам следует заменить эти значения своим именем пользователя Docker Hub и именем репозитория.
По мере размещения слоев образа в Docker Hub вы будете видеть на экране определенные сообщения.
Теперь ваш образ доступен Kubernetes в репозитории Docker Hub, и вы можете начать его развертывание в своем кластере.
При локальном запуске контейнера Django мы передали файл env
в docker run
для вставки переменных конфигурации в среду исполнения. Для вставки переменных конфигурации в Kubernetes используются элементы ConfigMap и Secret.
Элементы ConfigMap следует использовать для сохранения неконфиденциальной информации о конфигурации, такой как параметры приложения, а элементы Secret следует использовать для хранения важной информации, такой как ключи API и учетные данные для входа в базу данных. Они вставляются в контейнеры примерно одинаково, но у элементов Secret имеются дополнительные функции контроля доступа и безопасности, такие как шифрование в состоянии покоя. Элементы Secret хранят данные в формате base64, а элементы ConfigMap хранят данные в формате обычного текста.
Для начала необходимо создать каталог yaml
, где мы будем хранить наши манифесты Kubernetes. Перейдите в каталог.
- mkdir yaml
- cd
Откройте файл polls-configmap.yaml
в nano
или в другом предпочитаемом текстовом редакторе:
- nano polls-configmap.yaml
Вставьте в него следующий манифест ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: polls-config
data:
DJANGO_ALLOWED_HOSTS: "*"
STATIC_ENDPOINT_URL: "https://your_space_name.space_region.digitaloceanspaces.com"
STATIC_BUCKET_NAME: "your_space_name"
DJANGO_LOGLEVEL: "info"
DEBUG: "True"
DATABASE_ENGINE: "postgresql_psycopg2"
Мы извлекли неконфиденциальные данные конфигурации из файла env
, измененного на шаге 1, и передали их в манифест ConfigMap. Объект ConfigMap носит имя polls-config
. Скопируйте те же самые значения, введенные в файл env
на предыдущем шаге.
Для тестирования оставьте для DJANGO_ALLOWED_HOSTS
значение *
, чтобы отключить фильтрацию на основе заголовков хоста. В производственной среде здесь следует указать домен вашего приложения.
Когда вы завершите редактирование файла, сохраните и закройте его.
Создайте в своем кластере элемент ConfigMap, используя kubectl apply
:
- kubectl apply -f polls-configmap.yaml
Outputconfigmap/polls-config created
Мы создали элемент ConfigMap, а на следующем шаге мы создадим элемент Secret, который будет использоваться нашим приложением.
Значения Secret должны иметь кодировку base64, то есть создавать объекты Secret в кластере немного сложнее, чем объекты ConfigMaps. Вы можете повторить описанную на предыдущем шаге процедуру, выполнив кодирование значений Secret в формате base64 вручную и вставив их в файл манифеста. Также вы можете создавать их, используя файл переменных среды, kubectl create
и флаг --from-env-file
, что мы и сделаем на этом шаге.
Мы снова используем файл env
из шага 1, удалив переменные, вставленные в ConfigMap. Создайте копию файла env
с именем polls-secrets
в каталоге yaml
:
- cp ../env ./polls-secrets
Отредактируйте файл в предпочитаемом редакторе:
- nano polls-secrets
DJANGO_SECRET_KEY=
DEBUG=True
DJANGO_ALLOWED_HOSTS=
DATABASE_ENGINE=postgresql_psycopg2
DATABASE_NAME=polls
DATABASE_USERNAME=
DATABASE_PASSWORD=
DATABASE_HOST=
DATABASE_PORT=
STATIC_ACCESS_KEY_ID=
STATIC_SECRET_KEY=
STATIC_BUCKET_NAME=
STATIC_ENDPOINT_URL=
DJANGO_LOGLEVEL=info
Удалите все переменные, вставленные в манифест ConfigMap. Когда вы закончите, содержимое файла должно выглядеть следующим образом:
DJANGO_SECRET_KEY=your_secret_key
DATABASE_NAME=polls
DATABASE_USERNAME=your_django_db_user
DATABASE_PASSWORD=your_django_db_user_password
DATABASE_HOST=your_db_host
DATABASE_PORT=your_db_port
STATIC_ACCESS_KEY_ID=your_space_access_key
STATIC_SECRET_KEY=your_space_access_key_secret
Обязательно используйте те же значения, что и на шаге 1. Закончив, сохраните и закройте файл.
Создайте в кластере объект Secret, используя kubectl create secret
:
- kubectl create secret generic polls-secret --from-env-file=poll-secrets
Outputsecret/polls-secret created
Здесь мы создадим объект Secret с именем polls-secret
и передадим в него созданный нами файл secrets.
Вы можете проинспектировать объект Secret, используя kubectl describe
:
- kubectl describe secret polls-secret
OutputName: polls-secret
Namespace: default
Labels: <none>
Annotations: <none>
Type: Opaque
Data
====
DATABASE_PASSWORD: 8 bytes
DATABASE_PORT: 5 bytes
DATABASE_USERNAME: 5 bytes
DJANGO_SECRET_KEY: 14 bytes
STATIC_ACCESS_KEY_ID: 20 bytes
STATIC_SECRET_KEY: 43 bytes
DATABASE_HOST: 47 bytes
DATABASE_NAME: 5 bytes
Мы сохранили конфигурацию вашего приложения в кластере Kubernetes, используя типы объектов Secret и ConfigMap. Мы готовы развернуть приложение в кластере.
На этом шаге мы создадим Deployment для вашего приложения Django. Kubernetes Deployment — контроллер, который можно использовать для управления приложениями без состояния в вашем кластере. Контроллер — это контур управления, регулирующий рабочие задачи посредством вертикального масштабирования. Контроллеры также перезапускают и очищают неисправные контейнеры.
Контроллеры Deployment контролируют один или несколько подов. Под — это наименьшая развертываемая единица в кластере Kubernetes. Поды содержат один или несколько контейнеров. Чтобы узнать о различных типах рабочих задач, ознакомьтесь с учебным модулем Введение в Kubernetes.
Для начала откройте файл polls-deployment.yaml
в предпочитаемом текстовом редакторе:
- nano polls-deployment.yaml
Вставьте в него следующий манифест Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: polls-app
labels:
app: polls
spec:
replicas: 2
selector:
matchLabels:
app: polls
template:
metadata:
labels:
app: polls
spec:
containers:
- image: your_dockerhub_username/app_repo_name:latest
name: polls
envFrom:
- secretRef:
name: polls-secret
- configMapRef:
name: polls-config
ports:
- containerPort: 8000
name: gunicorn
Введите соответствующее имя образа контейнера, ссылаясь на образ Django Polls, который вы разместили в Docker Hub на шаге 2.
Здесь мы определяем контроллер Kubernetes Deployment с именем polls-app
и присваиваем ему пару ключ-значение app: polls
. Мы указываем, что хотим запустить две реплики пода, определенного под полем template
.
Используя envFrom
с secretRef
и configMapRef
, мы указываем, что все данные из объектов polls-secret
и polls-config
следует вставить в контейнеры как переменные среды. Ключи ConfigMap и Secret становятся именами переменных среды.
В заключение мы откроем порт containerPort
8000
и назовем его gunicorn
.
Чтобы узнать больше о настройке контроллеров Kubernetes Deployment, ознакомьтесь с разделом Deployments в документации по Kubernetes.
Когда вы завершите редактирование файла, сохраните и закройте его.
Создайте в кластере контроллер Deployment, используя kubectl apply -f
:
- kubectl apply -f polls-deployment.yaml
- deployment.apps/polls-app created
Убедитесь, что контроллер Deployment развернут правильно, используя kubectl get
:
- kubectl get deploy polls-app
OutputNAME READY UP-TO-DATE AVAILABLE AGE
polls-app 2/2 2 2 6m38s
Если вы столкнетесь с ошибкой или что-то не будет работать, вы можете использовать kubectl describe
для проверки неработающего контроллера Deployment:
- kubectl describe deploy
Чтобы проинспектировать два пода, используйте kubectl get pod
:
- kubectl get pod
OutputNAME READY STATUS RESTARTS AGE
polls-app-847f8ccbf4-2stf7 1/1 Running 0 6m42s
polls-app-847f8ccbf4-tqpwm 1/1 Running 0 6m57s
В кластеры запущены и работают две реплики вашего приложения Django. Чтобы получить доступ к приложению, вам нужно создать Kubernetes Service, и мы сделаем это на следующем шаге.
На этом шаге мы создаем Service для нашего приложения Django. Kubernetes Service — это абстракция, позволяющая предоставить доступ к набору работающих подов как к сетевой службе. Используя Service, вы можете создать для вашего приложения стабильный конечный пункт, который не будет изменяться по мере уничтожения и воссоздания подов.
Существует несколько типов служб Service, в том числе службы ClusterIP, открывающие доступ к Service через внутренний IP-адрес кластера, службы NodePort, открывающие доступ к Service на каждом узле статического порта NodePort, и службы LoadBalancer, предоставляющие облачный балансировщик нагрузки для управления внешним трафиком подов вашего кластера (через порты NodePort, которые он создает автоматически). Чтобы узнать больше об этом, ознакомьтесь с разделом Service в документации по Kubernetes.
На заключительном шаге настройки мы используем службу ClusterIP Service, доступ к которой открыт через Ingress и контроллер Ingress, настроенный на этапе подготовки к этому учебному модулю. Сейчас мы убедимся, что все элементы работают корректно. Для этого мы создадим временную службу NodePort Service для доступа к приложению Django.
Для начала создайте файл с именем polls-svc.yaml
в предпочитаемом редакторе:
- nano polls-svc.yaml
Вставьте в него следующий манифест Service:
apiVersion: v1
kind: Service
metadata:
name: polls
labels:
app: polls
spec:
type: NodePort
selector:
app: polls
ports:
- port: 8000
targetPort: 8000
Здесь мы создаем службу NodePort под названием polls
и присваиваем ей ярлык app: polls
. Мы выберем поды серверной части с ярлыком app: polls
и установим в качестве цели порты 8000
.
Когда вы завершите редактирование файла, сохраните и закройте его.
Разверните Service с помощью команды kubectl apply
:
- kubectl apply -f polls-svc.yaml
Outputservice/polls created
Убедитесь, что служба была создана с использованием kubectl get svc
:
- kubectl get svc polls
OutputNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
polls NodePort 10.245.197.189 <none> 8000:32654/TCP 59s
Этот вывод показывает внутренний IP-адрес кластера службы и номер порта NodePort (32654
). Чтобы подключиться к службе, нам потребуется внешний IP-адрес для нашего узла кластера:
- kubectl get node -o wide
OutputNAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
pool-7no0qd9e0-364fd Ready <none> 27h v1.18.8 10.118.0.5 203.0.113.1 Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9
pool-7no0qd9e0-364fi Ready <none> 27h v1.18.8 10.118.0.4 203.0.113.2 Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9
pool-7no0qd9e0-364fv Ready <none> 27h v1.18.8 10.118.0.3 203.0.113.3 Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9
Откройте в браузере приложение Polls, используя внешний IP-адрес и порт NodePort любого узла. С учетом приведенного выше вывода URL приложения будет выглядеть так: http://203.0.113.1:32654/polls
.
Вы должны увидеть тот же интерфейс приложения Polls, который вы открыли локально на шаге 1:
Вы можете повторить ту же проверку, используя маршрут /admin
: http://203.0.113.1:32654/admin
. Вы увидите тот же интерфейс администрирования, что и раньше:
Мы развернули две реплики контейнера приложения Django Polls, используя Deployment. Также мы создали стабильный конечный пункт сети для этих двух реплик и обеспечили внешний доступ к ним через службу NodePort.
Заключительный шаг этого учебного модуля заключается в том, чтобы защитить внешний трафик нашего приложения, используя HTTPS. Для этого мы используем контроллер ingress-nginx
, установленный на этапе предварительных требований, и создадим объект Ingress для перенаправления внешнего трафика в службу Kubernetes polls
.
Сущности Ingress в Kubernetes обеспечивают гибкую маршрутизацию внешнего трафика кластера Kubernetes среди служб внутри кластера. Это достигается с помощью ресурсов Ingress, которые определяют правила маршрутизации трафика HTTP и HTTPS для служб Kubernetes, и контроллеров Ingress, которые реализуют правила посредством балансировки нагрузки трафика и его перенаправления на соответствующие службы серверной части.
На этапе предварительных требований мы установили контроллер ingress-nginx и надстройку cert-manager для автоматизации сертификатов TLS. Также мы настроили испытательную и производственную среду ClusterIssuers для вашего домена, используя центр сертификации Let’s Encrypt, и создали объект Ingress для проверки выдачи сертификатов и шифрования TLS для двух фиктивных серверных служб. Прежде чем продолжить выполнение этого шага, удалите объект echo-ingress
, созданный в подготовительном учебном модуле:
- kubectl delete ingress echo-ingress
При желании вы также можете удалить фиктивные службы и развертывания, используя команды kubectl delete svc
и kubectl delete deploy
, но для прохождения этого учебного модуля это не обязательно.
Также вы должны были создать запись DNS типа A
для your_domain.com
, указывающую на публичный IP-адрес балансировщика нагрузки Ingress. Если вы используете балансировщик нагрузки DigitalOcean, вы можете найти этот IP-адрес в разделе Load Balancers панели управления. Если вы также используете DigitalOcean для управления записями DNS вашего домена, руководство Управление записями DNS поможет вам научиться создавать записи класса A
.
Если вы используете DigitalOcean Kubernetes, убедитесь, что вы внедрили обходное решение, описанное в шаге 5 учебного модуля Настройка Nginx Ingress с помощью Cert-Manager в DigitalOcean Kubernetes.
Когда у вас будет запись A
, указывающая на балансировщик нагрузки контроллера Ingress, вы можете создать Ingress для your_domain.com
и службы polls
.
Откройте файл с именем polls-ingress.yaml
в предпочитаемом текстовом редакторе:
- nano polls-ingress.yaml
Вставьте следующий манифест Ingress:
[polls-ingress.yaml]
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: polls-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt-staging"
spec:
tls:
- hosts:
- your_domain.com
secretName: polls-tls
rules:
- host: your_domain.com
http:
paths:
- backend:
serviceName: polls
servicePort: 8000
Мы создаем объект Ingress под именем polls-ingress
и добавляем к нему аннотацию, чтобы уровень управления использовал контроллер ingress-nginx и размещение ClusterIssuer. Также мы включаем TLS для домена your_domain.com
и сохраняем сертификат и закрытый ключ в объекте Secret с именем polls-tls
. В заключение мы определяем правило перенаправления трафика для хоста your_domain.com
в службу polls
через порт 8000
.
Когда вы завершите редактирование файла, сохраните и закройте его.
Создайте в кластере объект Ingress, используя kubectl apply
:
- kubectl apply -f polls-ingress.yaml
Outputingress.networking.k8s.io/polls-ingress created
Вы можете использовать kubectl describe
для отслеживания состояния только что созданного объекта Ingress:
- kubectl describe ingress polls-ingress
OutputName: polls-ingress
Namespace: default
Address: workaround.your_domain.com
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
polls-tls terminates your_domain.com
Rules:
Host Path Backends
---- ---- --------
your_domain.com
polls:8000 (10.244.0.207:8000,10.244.0.53:8000)
Annotations: cert-manager.io/cluster-issuer: letsencrypt-staging
kubernetes.io/ingress.class: nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 51s nginx-ingress-controller Ingress default/polls-ingress
Normal CreateCertificate 51s cert-manager Successfully created Certificate "polls-tls"
Normal UPDATE 25s nginx-ingress-controller Ingress default/polls-ingress
Также вы можете запустить команду describe
для сертификата polls-tls
, чтобы убедиться, что он был успешно создан:
- kubectl describe certificate polls-tls
Output. . .
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 3m33s cert-manager Issuing certificate as Secret does not exist
Normal Generated 3m32s cert-manager Stored new private key in temporary Secret resource "polls-tls-v9lv9"
Normal Requested 3m32s cert-manager Created new CertificateRequest resource "polls-tls-drx9c"
Normal Issuing 2m58s cert-manager The certificate has been successfully issued
Это подтверждает, что сертификат TLS успешно выдан, и шифрование HTTPS для домена your_domain.com
активно.
Поскольку мы использовали тестовую версию ClusterIssuer, большинство браузеров не будут доверять поддельному сертификату Let’s Encrypt, который она выдает, и поэтому при переходе по адресу your_domain.com
появится страница с сообщением об ошибке.
Чтобы отправить тестовый запрос, мы используем wget
из командной строки:
- wget -O - http://your_domain.com/polls
Output. . .
ERROR: cannot verify your_domain.com's certificate, issued by ‘CN=Fake LE Intermediate X1’:
Unable to locally verify the issuer's authority.
To connect to your_domain.com insecurely, use `--no-check-certificate'.
Мы используем предлагаемый флаг --no-check-certificate
, чтобы пропустить проверку сертификата:
- wget --no-check-certificate -q -O - http://your_domain.com/polls
Output
<link rel="stylesheet" type="text/css" href="https://your_space.nyc3.digitaloceanspaces.com/django-polls/static/polls/style.css">
<p>No polls are available.</p>
В этом выводе показан код HTML для страницы интерфейса /polls
, а также подтверждается выдача таблицы стилей из хранилища объектов.
Мы успешно проверили выдачу сертификата с помощью тестовой версии ClusterIssuer и теперь можем изменить Ingress для использования производственной версии ClusterIssuer.
Снова откройте файл polls-ingress.yaml
для редактирования:
- nano polls-ingress.yaml
Измените аннотацию cluster-issuer
:
[polls-ingress.yaml]
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: polls-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
tls:
- hosts:
- your_domain.com
secretName: polls-tls
rules:
- host: your_domain.com
http:
paths:
- backend:
serviceName: polls
servicePort: 8000
Внесите необходимые изменения, после чего сохраните и закройте файл. Обновите Ingress, используя kubectl apply
:
- kubectl apply -f polls-ingress.yaml
Outputingress.networking.k8s.io/polls-ingress configured
Вы можете использовать команды kubectl describe certificate polls-tls
и kubectl describe ingress polls-ingress
для отслеживания статуса выдачи сертификата:
- kubectl describe ingress polls-ingress
Output. . .
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 23m nginx-ingress-controller Ingress default/polls-ingress
Normal CreateCertificate 23m cert-manager Successfully created Certificate "polls-tls"
Normal UPDATE 76s (x2 over 22m) nginx-ingress-controller Ingress default/polls-ingress
Normal UpdateCertificate 76s cert-manager Successfully updated Certificate "polls-tls"
Приведенный выше вывод подтверждает, что новый производственный сертификат успешно выдан и сохранен в объекте Secret polls-tls
.
Откройте в браузере адрес your_domain.com/polls
, чтобы убедиться, что шифрование HTTPS включено, и все работает ожидаемым образом. Вы должны увидеть интерфейс приложения Polls:
Убедитесь, что в браузере включено шифрование HTTPS. Если вы используете Google Chrome, то если вышеуказанная страница откроется без ошибок, это будет означать, что все работает правильно. Кроме того, на панели URL должно быть изображение замка. Нажав на замок, вы сможете просмотреть детали сертификата Let’s Encrypt.
Для окончательной очистки вы можете переключить тип службы polls
с NodePort на внутренний тип ClusterIP.
Отредактируйте файл polls-svc.yaml
в текстовом редакторе:
- nano polls-svc.yaml
Измените type
с NodePort
на ClusterIP
:
apiVersion: v1
kind: Service
metadata:
name: polls
labels:
app: polls
spec:
type: ClusterIP
selector:
app: polls
ports:
- port: 8000
targetPort: 8000
Когда вы завершите редактирование файла, сохраните и закройте его.
Разверните изменения с помощью команды kubectl apply
:
- kubectl apply -f polls-svc.yaml --force
Outputservice/polls configured
Убедитесь, что служба Service была изменена, используя kubectl get svc
:
- kubectl get svc polls
OutputNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
polls ClusterIP 10.245.203.186 <none> 8000/TCP 22s
Этот вывод показывает, что тип Service — ClusterIP. Для доступа можно использовать только домен и Ingress, созданный на этом шаге.
В этом учебном модуле мы развернули масштабируемое приложение Django с защитой HTTPS в кластере Kubernetes. Статичный контент выдается напрямую из хранилища объектов, а количество работающих подов можно быстро увеличивать или уменьшать, используя поле replicas
в манифесте Deployment polls-app
.
Если вы используете пространство DigitalOcean Space, вы также можете включить доставку статичных ресурсов через сеть доставки контента и создать настраиваемый субдомен для своего пространства. Дополнительную информацию можно найти в разделе Включение CDN учебного модуля Настройка масштабируемого приложения Django с управляемыми базами данных и пространствами DigitalOcean.
С остальными частями серии можно ознакомиться на странице От контейнеров к Kubernetes с Django.
]]>Neste tutorial, você irá implantar um aplicativo de pesquisa do Django em contêiner em um cluster do Kubernetes.
O Django é um framework Web poderoso, que pode ajudar a acelerar a implantação do seu aplicativo Python. Ele inclui diversas funcionalidades convenientes, como um mapeador relacional do objeto, autenticação de usuário e uma interface administrativa personalizável para seu aplicativo. Ele também inclui um framework de cache e encoraja o design de aplicativos organizados através do seu URL Dispatcher e Sistema de modelos.
Em Como construir um aplicativo Django e Gunicorn com o Docker, o aplicativo Polls de tutorial do Django foi modificado de acordo com a metodologia do Twelve-Factor para a construção de aplicativos Web escaláveis e nativos na nuvem. Essa configuração em contêiner foi dimensionada e protegida com um proxy reverso do Nginx e autenticada pelo Let’s Encrypt com certificados TLS seguindo Como dimensionar e proteger um aplicativo Django com o Docker, Nginx e Let’s Encrypt. Neste tutorial final da série De contêineres ao Kubernetes com o Django, o aplicativo modernizado de pesquisa do Django será implantado em um cluster do Kubernetes.
O Kubernetes é um orquestrador de contêineres de código aberto poderoso, que automatiza a implantação, dimensionamento e gerenciamento de aplicativos em contêiner. Os objetos do Kubernetes como os ConfigMaps e Segredos permitem centralizar e desacoplar a configuração dos seus contêineres, enquanto os controladores como Deployments reiniciam automaticamente contêineres que falharam e habilitam o dimensionamento rápido de réplicas de contêineres. A criptografia TLS é habilitada com um objeto Ingress e o Controlador Ingress de código aberto ingress-nginx. O add-on cert-manager do Kubernetes renova e emite certificados usando a autoridade de certificação gratuita Let’s Encrypt.
Para seguir este tutorial, será necessário:
kubectl
de linha de comando instalada em sua máquina local e configurada para se conectar ao seu cluster. Você pode ler mais sobre como instalar o kubectl
na documentação oficial. Se você estiver usando um cluster do Kubernetes da DigitalOcean, consulte Como se conectar a um cluster do Kubernetes da DigitalOcean para aprender como se conectar ao seu cluster usando o kubectl
.your_domain.com
do início ao fim. Você pode obter um domínio gratuitamente através do Freenom, ou usar o registrador de domínios de sua escolha.A
com your_domain.com
apontando para o endereço IP público do Balanceador de carga do Ingress. Se estiver usando a DigitalOcean para gerenciar os registros de DNS do seu domínio, consulte Como gerenciar os registros de DNS para aprender como criar um registro A
.Assim que tiver esses componentes configurados, tudo estará pronto para seguir o guia.
Neste passo, vamos clonar o código do aplicativo do GitHub e definir algumas configurações como as credenciais de banco de dados e chaves de armazenamento de objetos.
O código do aplicativo e o Dockerfile podem ser encontrados na ramificação polls-docker
do repositório GitHub do aplicativo Polls de tutorial do Django. Esse repositório contém códigos para o aplicativo de amostra Polls da documentação do Django, que ensina como construir um aplicativo de pesquisa a partir do zero.
A ramificação polls-docker
contém uma versão em Docker do aplicativo Polls. Para aprender como o aplicativo Polls foi modificado para funcionar efetivamente em um ambiente em contêiner, consulte Como construir um aplicativo Django e Gunicorn com o Docker.
Comece usando o git
para clonar o branch polls-docker
do repositório GitHub do aplicativo Polls do tutorial do Django em sua máquina local:
- git clone --single-branch --branch polls-docker https://github.com/do-community/django-polls.git
Navegue até o diretório django-polls
:
- cd django-polls
Esse diretório contém o código Python do aplicativo Django, um Dockerfile
que o Docker usará para compilar a imagem do contêiner, bem como um arquivo env
que contém uma lista de variáveis de ambiente a serem passadas para o ambiente de execução do contêiner. Verifique o Dockerfile
:
- cat Dockerfile
OutputFROM python:3.7.4-alpine3.10
ADD django-polls/requirements.txt /app/requirements.txt
RUN set -ex \
&& apk add --no-cache --virtual .build-deps postgresql-dev build-base \
&& python -m venv /env \
&& /env/bin/pip install --upgrade pip \
&& /env/bin/pip install --no-cache-dir -r /app/requirements.txt \
&& runDeps="$(scanelf --needed --nobanner --recursive /env \
| awk '{ gsub(/,/, "\nso:", $2); print "so:" $2 }' \
| sort -u \
| xargs -r apk info --installed \
| sort -u)" \
&& apk add --virtual rundeps $runDeps \
&& apk del .build-deps
ADD django-polls /app
WORKDIR /app
ENV VIRTUAL_ENV /env
ENV PATH /env/bin:$PATH
EXPOSE 8000
CMD ["gunicorn", "--bind", ":8000", "--workers", "3", "mysite.wsgi"]
Esse Dockerfile usa a imagem Docker oficial do Python 3.7.4 como base e instala os requisitos de pacote Python do Django e do Gunicorn, conforme definido no arquivo django-polls/requirements.txt
. Em seguida, ele remove alguns arquivos de compilação desnecessários, copia o código do aplicativo na imagem e define o PATH
de execução. Por fim, ele declara que a porta 8000
será usada para aceitar conexões de contêiner recebidas e executa gunicorn
com 3 trabalhadores, escutando na porta 8000
.
Para aprender mais sobre cada um dos passos nesse Dockerfile, confira o Passo 6 de Como construir um aplicativo Django e Gunicorn com o Docker.
Agora, crie a imagem usando o docker build
:
- docker build -t polls .
Nós demos o nome de polls
para a imagem usando o sinalizador -t
e passamos o diretório atual como um contexto de compilação, que é o conjunto de arquivos de referência ao compilar a imagem.
Depois que o Docker compilar e marcar a imagem, liste as imagens disponíveis usando docker images
:
- docker images
Você deve ver a imagem polls
listada:
OutputREPOSITORY TAG IMAGE ID CREATED SIZE
polls latest 80ec4f33aae1 2 weeks ago 197MB
python 3.7.4-alpine3.10 f309434dea3a 8 months ago 98.7MB
Antes de executarmos o contêiner Django, precisamos configurar seu ambiente de execução usando o arquivo env
presente no diretório atual. Esse arquivo será passado para o comando docker run
usado para executar o contêiner, e o Docker irá injetar as variáveis de ambiente configuradas no ambiente de execução do contêiner.
Abra o arquivo env
com o nano
ou com o seu editor favorito:
- nano env
DJANGO_SECRET_KEY=
DEBUG=True
DJANGO_ALLOWED_HOSTS=
DATABASE_ENGINE=postgresql_psycopg2
DATABASE_NAME=polls
DATABASE_USERNAME=
DATABASE_PASSWORD=
DATABASE_HOST=
DATABASE_PORT=
STATIC_ACCESS_KEY_ID=
STATIC_SECRET_KEY=
STATIC_BUCKET_NAME=
STATIC_ENDPOINT_URL=
DJANGO_LOGLEVEL=info
Preencha os valores que estão faltando para as seguintes chaves:
DJANGO_SECRET_KEY
: defina isso como um valor único e imprevisível, conforme detalhado na documentação do Django. Um método para gerar essa chave é fornecido em Ajustando as configurações do aplicativo do tutorial sobre o Aplicativo Django escalável.DJANGO_ALLOWED_HOSTS
: essa variável protege o aplicativo e impede ataques de cabeçalho de host HTTP. Para fins de teste, defina isso como *
, um coringa que irá corresponder a todos os hosts. Na produção, isso deve ser definido como your_domain.com
. Para aprender mais sobre esse ajuste do Django, consulte as Core Settings da documentação do Django.DATABASE_USERNAME
: defina isso como o usuário do banco de dados PostgreSQL criado nos passos pré-requisitos.DATABASE_NAME
: defina isso como polls
ou o nome do banco de dados PostgreSQL criado nos passos pré-requisitos.DATABASE_PASSWORD
: defina isso como a senha do usuário do banco de dados PostgreSQL criada nos passos pré-requisitos.DATABASE_HOST
: defina isso como o nome do host do seu banco de dados.DATABASE_PORT
: defina isso como a porta do seu banco de dados.STATIC_ACCESS_KEY_ID
: defina isso como a chave de acesso do seu espaço ou armazenamento de objetos.STATIC_SECRET_KEY
: defina isso como o segredo da chave de acesso do seu espaço ou armazenamento de objetos.STATIC_BUCKET_NAME
: defina isso como seu nome de espaço ou bucket de armazenamento de objetos.STATIC_ENDPOINT_URL
: defina isso como o URL do ponto de extremidade do espaço apropriado ou armazenamento de objetos, como https://your_space_name.nyc3.digitaloceanspaces.com
se o espaço estiver localizado na região nyc3
.Assim que terminar a edição, salve e feche o arquivo.
No próximo passo, vamos executar o contêiner configurado localmente e criar o esquema de banco de dados. Além disso, vamos carregar ativos estáticos como folhas de estilos e imagens para o armazenamento de objetos.
Com o contêiner construído e configurado, use o docker run
para substituir o conjunto CMD
no Dockerfile e criar o esquema de banco de dados usando os comandos manage.py makemigrations
e manage.py migrate
:
- docker run --env-file env polls sh -c "python manage.py makemigrations && python manage.py migrate"
Executamos a imagem de contêiner polls:latest
, passamos o arquivo de variável de ambiente que acabamos de modificar e substituímos o comando do Dockerfile com sh -c "python manage.py makemigrations && python manage.py migrate"
, o que irá criar o esquema de banco de dados definido pelo código do aplicativo.
Se estiver executando isso pela primeira vez, você deve ver:
OutputNo changes detected
Operations to perform:
Apply all migrations: admin, auth, contenttypes, polls, sessions
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying admin.0003_logentry_add_action_flag_choices... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying auth.0010_alter_group_name_max_length... OK
Applying auth.0011_update_proxy_permissions... OK
Applying polls.0001_initial... OK
Applying sessions.0001_initial... OK
Isso indica que o esquema de banco de dados foi criado com sucesso.
Se estiver executando migrate
uma outra vez, o Django irá cancelar a operação a menos que o esquema de banco de dados tenha sido alterado.
Em seguida, vamos executar outra instância do contêiner de aplicativo e usar um shell interativo dentro dela para criar um usuário administrativo para o projeto Django.
- docker run -i -t --env-file env polls sh
Isso lhe fornecerá um prompt do shell dentro do contêiner em execução que você pode usar para criar o usuário do Django:
- python manage.py createsuperuser
Digite um nome de usuário, endereço de e-mail e senha para o seu usuário e, depois de criá-lo, pressione CTRL+D
para sair do contêiner e encerrá-lo.
Por fim, vamos gerar os arquivos estáticos para o aplicativo e fazer o upload deles para o espaço da DigitalOcean usando o collectstatic
. Observe que esse processo pode demorar um pouco de tempo para ser concluído.
- docker run --env-file env polls sh -c "python manage.py collectstatic --noinput"
Depois que esses arquivos forem gerados e enviados, você receberá a seguinte saída.
Output121 static files copied.
Agora, podemos executar o aplicativo:
- docker run --env-file env -p 80:8000 polls
Output[2019-10-17 21:23:36 +0000] [1] [INFO] Starting gunicorn 19.9.0
[2019-10-17 21:23:36 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
[2019-10-17 21:23:36 +0000] [1] [INFO] Using worker: sync
[2019-10-17 21:23:36 +0000] [7] [INFO] Booting worker with pid: 7
[2019-10-17 21:23:36 +0000] [8] [INFO] Booting worker with pid: 8
[2019-10-17 21:23:36 +0000] [9] [INFO] Booting worker with pid: 9
Aqui, executamos o comando padrão definido no Dockerfile, gunicorn --bind :8000 --workers 3 mysite.wsgi:application
e expomos a porta do contêiner 8000
para que a porta 80
na sua máquina local seja mapeada para a porta 8000
do contêiner polls
.
Agora, você deve ser capaz de navegar até o aplicativo polls
usando seu navegador Web digitando http://localhost
na barra de URL. Como não há nenhuma rota definida para o caminho /
, você provavelmente receberá um erro 404 Page Not Found
, o que é esperado.
Navegue até http://localhost/polls
para ver a interface do aplicativo Polls:
Para visualizar a interface administrativa, visite http://localhost/admin
. Você deve ver a janela de autenticação do administrador do aplicativo Polls:
Digite o nome e a senha do usuário administrativo que você criou com o comando createsuperuser
.
Depois de autenticar-se, você pode acessar a interface administrativa do aplicativo Polls:
Observe que os ativos estáticos para os aplicativos admin
e polls
estão sendo entregues diretamente do armazenamento de objetos. Para confirmar isso, consulte Testando a entrega de arquivos estáticos de espaços.
Quando terminar de explorar, aperte CTRL+C
na janela do terminal executando o contêiner Docker para encerrar o contêiner.
Com a imagem Docker do aplicativo Django testada, ativos estáticos carregados no armazenamento de objetos e o esquema de banco de dados configurado e pronto para ser usado com seu aplicativo, tudo está pronto para carregar sua imagem do aplicativo Django em um registro de imagem como o Docker Hub.
Para implementar seu aplicativo no Kubernetes, sua imagem de aplicativo deve ser carregada em um registro como o Docker Hub. O Kubernetes irá puxar a imagem do aplicativo do seu repositório e implantá-la em seu cluster.
É possível usar um registro privado do Docker, como o registro do contêiner da DigitalOcean, atualmente gratuito em acesso antecipado, ou um registro público do Docker como o Docker Hub. O Docker Hub também permite criar repositórios privados do Docker. Um repositório público permite que qualquer pessoa veja e puxe as imagens do contêiner, enquanto um repositório privado permite restringir o acesso a você e seus membros de equipe.
Neste tutorial, vamos enviar a imagem do Django ao repositório público do Docker Hub criado nos pré-requisitos. Também é possível enviar sua imagem a um repositório privado, mas puxar imagens de um repositório privado está além do escopo deste artigo. Para aprender mais sobre a autenticação do Kubernetes com o Docker Hub e puxar imagens privadas, consulte Puxar uma imagem de um registro privado dos documentos do Kubernetes.
Comece fazendo login no Docker Hub em sua máquina local:
- docker login
OutputLogin with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username:
Digite seu nome de usuário e senha do Docker Hub para fazer login.
A imagem do Django possui atualmente o sinalizador polls:latest
. Para enviá-la ao seu repositório do Docker Hub, sinalize novamente a imagem com seu nome de usuário e nome de repositório do Docker Hub:
- docker tag polls:latest your_dockerhub_username/your_dockerhub_repo_name:latest
Carregue a imagem no repositório:
- docker push sammy/sammy-django:latest
Neste tutorial, o nome de usuário do Docker Hub é sammy e o nome do repositório é sammy-django. Você deve substituir esses valores pelo seu nome de usuário e nome de repositório do Docker Hub.
Você verá um resultado que se atualiza conforme as camadas da imagem são enviadas ao Docker Hub.
Agora que sua imagem está disponível no Kubernetes no Docker Hub, você pode começar a implantá-la em seu cluster.
Quando executamos o contêiner do Django localmente, passamos o arquivo env
ao docker run
para injetar variáveis de configuração no ambiente de tempo de execução. No Kubernetes, as variáveis de configuração podem ser injetadas usando o ConfigMaps e Segredos.
Os ConfigMaps devem ser usados para armazenar informações de configuração não confidenciais, como as configurações do aplicativo, enquanto os Segredos devem ser usados para informações confidenciais como chaves de API e credenciais de banco de dados. Ambos são injetados em contêineres de maneira similar, mas os Segredos têm recursos de controle de acesso e segurança adicionais, como a criptografia em repouso. Os Segredos também armazenam dados em base64, enquanto os ConfigMaps armazenam dados em texto simples.
Para começar, crie um diretório chamado yaml
no qual vamos armazenar nossos manifestos do Kubernetes. Navegue até o diretório.
- mkdir yaml
- cd
Abra um arquivo chamado polls-configmap.yaml
no nano
ou seu editor de texto preferido:
- nano polls-configmap.yaml
Cole o seguinte manifesto do ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: polls-config
data:
DJANGO_ALLOWED_HOSTS: "*"
STATIC_ENDPOINT_URL: "https://your_space_name.space_region.digitaloceanspaces.com"
STATIC_BUCKET_NAME: "your_space_name"
DJANGO_LOGLEVEL: "info"
DEBUG: "True"
DATABASE_ENGINE: "postgresql_psycopg2"
As configurações não confidenciais foram extraídas do arquivo env
modificado no Passo 1 e coladas em um manifesto do ConfigMap. O objeto do ConfigMap chama-se polls-config
. Copie os mesmos valores inseridos no arquivo env
no passo anterior.
Por motivos de teste, deixe o DJANGO_ALLOWED_HOSTS
como *
para desativar a filtragem baseada em cabeçalhos de host. Em um ambiente de produção, isso deve ser definido como o domínio do seu aplicativo.
Quando terminar de editar o arquivo, salve e feche-o.
Crie o ConfigMap em seu cluster usando o kubectl apply
:
- kubectl apply -f polls-configmap.yaml
Outputconfigmap/polls-config created
Com o ConfigMap criado, vamos criar o Segredo usado pelo nosso aplicativo no próximo passo.
Os valores do Segredo devem ser codificados em base64, o que significa que criar objetos Segredo em seu cluster é ligeiramente mais complicado do que criar ConfigMaps. É possível repetir o processo do passo anterior, manualmente codificar os valores do Segredo em base64 e colá-los em um arquivo de manifesto. Também é possível criá-los usando um arquivo de variável de ambiente, kubectl create
e o sinalizador --from-env-file
, o que será feito neste passo.
Novamente, vamos usar o arquivo env
do Passo 1, removendo variáveis inseridas no ConfigMap. Faça uma cópia do arquivo env
chamado polls-secrets
no diretório yaml
:
- cp ../env ./polls-secrets
Edite o arquivo em seu editor de preferência:
- nano polls-secrets
DJANGO_SECRET_KEY=
DEBUG=True
DJANGO_ALLOWED_HOSTS=
DATABASE_ENGINE=postgresql_psycopg2
DATABASE_NAME=polls
DATABASE_USERNAME=
DATABASE_PASSWORD=
DATABASE_HOST=
DATABASE_PORT=
STATIC_ACCESS_KEY_ID=
STATIC_SECRET_KEY=
STATIC_BUCKET_NAME=
STATIC_ENDPOINT_URL=
DJANGO_LOGLEVEL=info
Exclua todas as variáveis inseridas no manifesto do ConfigMap. Quando terminar, ele deve ficar parecido com isto:
DJANGO_SECRET_KEY=your_secret_key
DATABASE_NAME=polls
DATABASE_USERNAME=your_django_db_user
DATABASE_PASSWORD=your_django_db_user_password
DATABASE_HOST=your_db_host
DATABASE_PORT=your_db_port
STATIC_ACCESS_KEY_ID=your_space_access_key
STATIC_SECRET_KEY=your_space_access_key_secret
Certifique-se de usar os mesmos valores usados no Passo 1. Quando terminar, salve e feche o arquivo.
Crie o Segredo em seu cluster usando o kubectl create secret
:
- kubectl create secret generic polls-secret --from-env-file=poll-secrets
Outputsecret/polls-secret created
Aqui, criamos um objeto Segredo chamado polls-secret
e o passamos no arquivo de Segredos que acabamos de criar.
Verifique o Segredo usando o kubectl describe
:
- kubectl describe secret polls-secret
OutputName: polls-secret
Namespace: default
Labels: <none>
Annotations: <none>
Type: Opaque
Data
====
DATABASE_PASSWORD: 8 bytes
DATABASE_PORT: 5 bytes
DATABASE_USERNAME: 5 bytes
DJANGO_SECRET_KEY: 14 bytes
STATIC_ACCESS_KEY_ID: 20 bytes
STATIC_SECRET_KEY: 43 bytes
DATABASE_HOST: 47 bytes
DATABASE_NAME: 5 bytes
Neste ponto, você armazenou a configuração do seu aplicativo em seu cluster do Kubernetes usando os tipos de objeto Segredo e ConfigMap. Agora, estamos prontos para implantar o aplicativo no cluster.
Neste passo, você criará um Deployment (implantação) para seu aplicativo do Django. Uma Implantação do Kubernetes é um controlador que pode ser usado para gerenciar aplicativos sem estado em seu cluster. Um controlador é um loop de controle que regula cargas de trabalho aumentando ou diminuindo-as. Os controladores também reiniciam e limpam contêineres com falhas.
As Implantações controlam um ou mais Pods, a menor unidade implantável em um cluster do Kubernetes. Os Pods incluem um ou mais contêineres. Para aprender mais sobre os diferentes tipos de cargas de trabalho que você pode inicializar, consulte Uma introdução ao Kubernetes.
Inicie abrindo um arquivo chamado polls-deployment.yaml
no seu editor de texto favorito:
- nano polls-deployment.yaml
Cole o manifesto de Implantação a seguir:
apiVersion: apps/v1
kind: Deployment
metadata:
name: polls-app
labels:
app: polls
spec:
replicas: 2
selector:
matchLabels:
app: polls
template:
metadata:
labels:
app: polls
spec:
containers:
- image: your_dockerhub_username/app_repo_name:latest
name: polls
envFrom:
- secretRef:
name: polls-secret
- configMapRef:
name: polls-config
ports:
- containerPort: 8000
name: gunicorn
Preencha o nome apropriado da imagem do contêiner, referenciando a imagem do Polls do Django que você enviou para o Docker Hub no Passo 2.
Aqui, definimos uma Implantação do Kubernetes chamada polls-app
e a rotulamos com o par de chave-valor app: polls
. Especificamos que queremos executar duas réplicas do Pod definido abaixo do campo template
.
Usando o envFrom
com o secretRef
e o configMapRef
, especificamos que todos os dados do Segredo polls-secret
e do ConfigMap polls-config
devem ser injetados nos contêineres como variáveis de ambiente. As chaves ConfigMap e Segredo tornam-se os nomes das variáveis de ambiente.
Por fim, expomos a containerPort
8000
e a nomeamos gunicorn
.
Para aprender mais sobre como configurar as Implantações do Kubernetes, consulte Deployments na documentação do Kubernetes.
Quando terminar de editar o arquivo, salve e feche-o.
Crie uma Implantação no seu cluster usando o kubectl apply -f
:
- kubectl apply -f polls-deployment.yaml
- deployment.apps/polls-app created
Verifique se a Implantação foi implantada corretamente usando o kubectl get
:
- kubectl get deploy polls-app
OutputNAME READY UP-TO-DATE AVAILABLE AGE
polls-app 2/2 2 2 6m38s
Se encontrar um erro ou algo não estiver funcionando, use o kubectl describe
para verificar o Deployment que falhou:
- kubectl describe deploy
Verifique os dois Pods usando o kubectl get pod
:
- kubectl get pod
OutputNAME READY STATUS RESTARTS AGE
polls-app-847f8ccbf4-2stf7 1/1 Running 0 6m42s
polls-app-847f8ccbf4-tqpwm 1/1 Running 0 6m57s
Agora, duas réplicas do seu aplicativo Django estão em funcionamento no cluster. Para acessar o aplicativo, é necessário criar um Service (serviço) do Kubernetes, que vamos fazer em seguida.
Neste passo, você irá criar um Serviço para seu aplicativo Django. Um Serviço do Kubernetes é uma abstração que permite expor um conjunto de Pods em execução como um serviço de rede. Ao usar um Serviço, é possível criar um ponto de extremidade estável para seu aplicativo que não muda à medida que os Pods são destruídos e recriados.
Existem vários tipos de Serviço, incluindo os Serviços de ClusterIP, que expõem o Serviço em um IP interno do cluster, os Serviços de NodePort, que expõem o Serviço em cada nó em uma porta estática chamada NodePort, além de os Serviços de LoadBalancer, que fornecem um balanceador de carga em nuvem para direcionar o tráfego externo aos Pods no seu cluster (através dos NodePorts, criados por ele automaticamente). Para aprender mais sobre isso, consulte Service nos documentos do Kubernetes.
Em nossa configuração final, vamos usar um Serviço de ClusterIP que é exposto usando um Ingress e um Controlador Ingress configurados nos pré-requisitos deste guia. Por enquanto, para testar se tudo está funcionando corretamente, vamos criar um Serviço de NodePort temporário para acessar o aplicativo Django.
Inicie criando um arquivo chamado polls-svc.yaml
usando seu editor favorito:
- nano polls-svc.yaml
Cole o manifesto de Serviço a seguir:
apiVersion: v1
kind: Service
metadata:
name: polls
labels:
app: polls
spec:
type: NodePort
selector:
app: polls
ports:
- port: 8000
targetPort: 8000
Aqui, criamos um Serviço de NodePort chamado polls
e damos a ele o rótulo app: polls
. Em seguida, selecionamos os Pods de backend com o rótulo app: polls
e miramos em suas portas 8000
.
Quando terminar de editar o arquivo, salve e feche-o.
Implemente o Serviço usando o kubectl apply
:
- kubectl apply -f polls-svc.yaml
Outputservice/polls created
Confirme se seu Serviço foi criado usando o kubectl get svc
:
- kubectl get svc polls
OutputNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
polls NodePort 10.245.197.189 <none> 8000:32654/TCP 59s
Esse resultado mostra o IP interno do cluster do Serviço e o NodePort (32654
). Para nos conectar ao serviço, precisamos dos endereços IP externos para nossos nós de cluster:
- kubectl get node -o wide
OutputNAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
pool-7no0qd9e0-364fd Ready <none> 27h v1.18.8 10.118.0.5 203.0.113.1 Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9
pool-7no0qd9e0-364fi Ready <none> 27h v1.18.8 10.118.0.4 203.0.113.2 Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9
pool-7no0qd9e0-364fv Ready <none> 27h v1.18.8 10.118.0.3 203.0.113.3 Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9
Em seu navegador Web, visite seu aplicativo Polls usando um endereço IP externo de qualquer nó e o NodePort. De acordo com o resultado acima, a URL do aplicativo seria: http://203.0.113.1:32654/polls
.
Você deve ver a mesma interface do aplicativo Polls que você acessou localmente no Passo 1:
É possível repetir o mesmo teste usando a rota /admin
: http://203.0.113.1:32654/admin
. Você deve ver a mesma interface de administrador que antes:
Neste estágio, você já implantou duas réplicas do contêiner do aplicativo Polls do Django usando uma Implantação. Você também criou um ponto de extremidade de rede estável para essas duas réplicas, e o tornou externamente acessível usando um Serviço de NodePort.
O passo final neste tutorial é proteger o tráfego externo para seu aplicativo usando HTTPS. Para fazer isso, vamos usar o Controlador Ingress ingress-nginx
instalado nos pré-requisitos e criar um objeto Ingress para rotear o tráfego externo para o Serviço polls
do Kubernetes.
Os Ingresses do Kubernetes permitem o roteamento do tráfego externo ao cluster do seu Kubernetes de maneira flexível para os Serviços dentro de seu cluster. Isso é alcançado usando os objetos do Ingress, que definem as regras para rotear o tráfego HTTP e HTTPS para os Serviços do Kubernetes e para os Controladores do Ingress, os quais implementam as regras fazendo o balanceamento da carga do tráfego e o seu roteamento para os Serviços de backend apropriados.
Nos pré-requisitos, você instalou o Controlador Ingress ingress-nginx e o add-on de automação de certificados TLS cert-manager. Você também definiu a preparação e a produção de ClusterIssuers para seu domínio usando a autoridade de certificação Let’s Encrypt e criou um Ingress para testar a emissão de certificados e a criptografia TLS em dois Serviços de backend fictícios. Antes de continuar com este passo, deve-se excluir o Ingress echo-ingress
criado no tutorial pré-requisito:
- kubectl delete ingress echo-ingress
Se desejar, também pode excluir os Serviços e Implantações fictícios usando o kubectl delete svc
e o kubectl delete deploy
, mas isso não é essencial para completar este tutorial.
Você também deve ter criado um registro de DNS A
com your_domain.com
apontando para o endereço IP público do balanceador de carga do Ingress. Se estiver usando um balanceador de carga da DigitalOcean, é possível encontrar esse endereço IP na seção Load Balancers do Painel de controle. Se estiver usando a DigitalOcean para gerenciar os registros de DNS do seu domínio, consulte Como gerenciar os registros de DNS para aprender como criar registros A
.
Se estiver usando o Kubernetes da DigitalOcean, também certifique-se de ter implementado a solução descrita no Passo 5 de Como configurar um Nginx Ingress com o Cert-Manager no Kubernetes da DigitalOcean.
Assim que tiver um registro A
apontando para o balanceador de carga do Ingress, crie um Ingress para your_domain.com
e o Serviço polls
.
Abra um arquivo chamado polls-ingress.yaml
no seu editor favorito:
- nano polls-ingress.yaml
Cole o manifesto de Ingress a seguir:
[polls-ingress.yaml]
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: polls-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt-staging"
spec:
tls:
- hosts:
- your_domain.com
secretName: polls-tls
rules:
- host: your_domain.com
http:
paths:
- backend:
serviceName: polls
servicePort: 8000
Criamos um objeto Ingress chamado polls-ingress
e o anotamos para instruir o plano de controle para usar o Controlador Ingress ingress-nginx e o ClusterIssuer de preparo. Também habilitamos o TLS para your_domain.com
e armazenamos o certificado e a chave privada em um segredo chamado polls-tls
. Por fim, definimos uma regra para rotear o tráfego para o host your_domain.com
para o Serviço polls
na porta 8000
.
Quando terminar de editar o arquivo, salve e feche-o.
Crie o Ingress no seu cluster usando o kubectl apply
:
- kubectl apply -f polls-ingress.yaml
Outputingress.networking.k8s.io/polls-ingress created
É possível usar o kubectl describe
para rastrear o estado do Ingress que acabou de ser criado:
- kubectl describe ingress polls-ingress
OutputName: polls-ingress
Namespace: default
Address: workaround.your_domain.com
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
polls-tls terminates your_domain.com
Rules:
Host Path Backends
---- ---- --------
your_domain.com
polls:8000 (10.244.0.207:8000,10.244.0.53:8000)
Annotations: cert-manager.io/cluster-issuer: letsencrypt-staging
kubernetes.io/ingress.class: nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 51s nginx-ingress-controller Ingress default/polls-ingress
Normal CreateCertificate 51s cert-manager Successfully created Certificate "polls-tls"
Normal UPDATE 25s nginx-ingress-controller Ingress default/polls-ingress
Também é possível executar um describe
no certificado polls-tls
para confirmar ainda mais se sua criação foi bem-sucedida:
- kubectl describe certificate polls-tls
Output. . .
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 3m33s cert-manager Issuing certificate as Secret does not exist
Normal Generated 3m32s cert-manager Stored new private key in temporary Secret resource "polls-tls-v9lv9"
Normal Requested 3m32s cert-manager Created new CertificateRequest resource "polls-tls-drx9c"
Normal Issuing 2m58s cert-manager The certificate has been successfully issued
Isso confirma que o certificado TLS foi emitido com sucesso e a criptografia do HTTPS agora está ativa para your_domain.com
.
Como usamos o ClusterIssuer de preparo, a maior parte dos navegadores Web não irá confiar no certificado falso do Let’s Encrypt que ele emitiu, de forma que navegar até your_domain.com
irá resultar em uma página de erro.
Para enviar um pedido de teste, vamos usar o wget
a partir da linha de comando:
- wget -O - http://your_domain.com/polls
Output. . .
ERROR: cannot verify your_domain.com's certificate, issued by ‘CN=Fake LE Intermediate X1’:
Unable to locally verify the issuer's authority.
To connect to your_domain.com insecurely, use `--no-check-certificate'.
Vamos usar o sinalizador sugerido --no-check-certificate
para ignorar a validação de certificados:
- wget --no-check-certificate -q -O - http://your_domain.com/polls
Output
<link rel="stylesheet" type="text/css" href="https://your_space.nyc3.digitaloceanspaces.com/django-polls/static/polls/style.css">
<p>No polls are available.</p>
Esse resultado mostra o HTML para a página de interface de /polls
, o que também confirma que a folha de estilos está sendo exibida a partir do armazenamento de objetos.
Agora que você testou com sucesso a emissão de certificados usando o ClusterIssuer de preparo, modifique o Ingress para usar o ClusterIssuer de produção.
Abra o polls-ingress.yaml
para editar mais uma vez:
- nano polls-ingress.yaml
Modifique a anotação do cluster-issuer
:
[polls-ingress.yaml]
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: polls-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
tls:
- hosts:
- your_domain.com
secretName: polls-tls
rules:
- host: your_domain.com
http:
paths:
- backend:
serviceName: polls
servicePort: 8000
Quando terminar, salve e feche o arquivo. Atualize o Ingress usando o kubectl apply
:
- kubectl apply -f polls-ingress.yaml
Outputingress.networking.k8s.io/polls-ingress configured
É possível usar o kubectl describe certificate polls-tls
e o kubectl describe ingress polls-ingress
para rastrear o status da emissão de certificados:
- kubectl describe ingress polls-ingress
Output. . .
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 23m nginx-ingress-controller Ingress default/polls-ingress
Normal CreateCertificate 23m cert-manager Successfully created Certificate "polls-tls"
Normal UPDATE 76s (x2 over 22m) nginx-ingress-controller Ingress default/polls-ingress
Normal UpdateCertificate 76s cert-manager Successfully updated Certificate "polls-tls"
O resultado acima confirma que o novo certificado de produção foi emitido com sucesso e armazenado no Secredo polls-tls
.
Navegue até your_domain.com/polls
no seu navegador Web para confirmar se a criptografia do HTTPS está habilitada e tudo está funcionando como esperado. Você deve ver a interface do aplicativo Polls:
Verifique se a criptografia do HTTPS está ativa no seu navegador Web. Se estiver usando o Google Chrome, chegar na página acima sem erros confirma que tudo está funcionando corretamente. Além disso, você deve ver um cadeado na barra de URL. Clicar no cadeado permitirá verificar os detalhes do certificado do Let’s Encrypt.
Como uma tarefa de limpeza final, você pode alterar opcionalmente o tipo de Serviço polls
do NodePort para o tipo exclusivamente interno ClusterIP.
Modifique o polls-svc.yaml
usando seu editor:
- nano polls-svc.yaml
Altere o type
de NodePort
para ClusterIP
:
apiVersion: v1
kind: Service
metadata:
name: polls
labels:
app: polls
spec:
type: ClusterIP
selector:
app: polls
ports:
- port: 8000
targetPort: 8000
Quando terminar de editar o arquivo, salve e feche-o.
Implemente as alterações usando o kubectl apply
:
- kubectl apply -f polls-svc.yaml --force
Outputservice/polls configured
Confirme se seu Serviço foi modificado usando o kubectl get svc
:
- kubectl get svc polls
OutputNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
polls ClusterIP 10.245.203.186 <none> 8000/TCP 22s
Esse resultado mostra que o tipo de Serviço é agora ClusterIP. A única maneira de acessá-lo é através do seu domínio e do Ingress criados neste passo.
Neste tutorial, você implantou um aplicativo Django dimensionável e seguro via HTTPS em um cluster do Kubernetes. O conteúdo estático é servido diretamente do armazenamento de objetos, e o número de Pods em execução pode ser aumentado ou reduzido rapidamente usando o campo replicas
no manifesto de Implantação polls-app
.
Se estiver usando um espaço da DigitalOcean, também é possível habilitar a entrega de ativos estáticos através de uma rede de entrega de conteúdo e criar um subdomínio personalizado para seu espaço. Por favor, consulte Habilitando o CDN de Como configurar um aplicativo Django dimensionável com os bancos de dados e espaços gerenciados da DigitalOcean para aprender mais.
Para revisar o resto da série, visite nossa página da série De contêineres ao Kubernetes com o Django.
]]>En este tutorial, implementará una aplicación de encuestas de Django en contenedor en un clúster de Kubernetes.
Django es un poderoso framework web que puede ayudarle a poner en marcha rápidamente su aplicación o sitio web con Python. Incluye varias características convenientes como un mapeo objeto-relacional, la autenticación de usuarios y una interfaz administrativa personalizable para su aplicación. También incluye un marco de almacenamiento en caché y fomenta el diseño limpio de aplicaciones a través de su Despachador URL y sistema de Plantillas
En Cómo crear una aplicación Django y Gunicorn con Docker, la aplicación de encuestas del Tutorial de Django fue modificada de acuerdo con la metodología de Twelve-Factor para crear aplicaciones web escalables y nativas en la nube. Esta configuración en contenedores se escaló y protegió con un proxy inverso Nginx y certificados TLS proporcionados por Let’s Encrypt en Cómo escalar y proteger una aplicación de Django con Docker, Nginx y Let’s Encrypt. En este tutorial final de la serie “De contenedores a Kubernetes con Django”, la aplicación de encuestas de Django modernizada se implementará en un clúster de Kubernetes.
Kubernetes es un potente orquestador de contenedores de código abierto que automatiza la implementación, el escalamiento y la administración de aplicaciones en contenedores. Los objetos de Kubernetes, como ConfigMaps y Secrets, permiten centralizar y desvincular la configuración de los contenedores, mientras que los controladores como las implementaciones reinician automáticamente los contenedores fallidos y habilitar el escalamiento rápido de las réplicas de contenedores. El cifrado TLS se activa con un objeto Ingress y el controlador de Ingress de código abierto llamado ingress-nginx. El complemento cert-manager de Kubernetes renueva y emite certificados usando la autoridad de certificación de Let’s Encrypt gratuita.
Para seguir este tutorial, necesitará lo siguiente:
kubectl
instalada en su equipo local y configurada para conectarse a su clúster. Puede leer más sobre la instalación de kubectl
en la documentación oficial. Si está usando un clúster de Kubernetes de DigitalOcean, consulte Cómo conectarse a un clúster Kubernetes de DigitalOcean para aprender a conectarse a su clúster usando kubectl
.your_domain.com
en todo momento. Puede obtener un ejemplar gratis en Freenom o utilizar el registrador de dominios que desee.A
con your_domain.com
orientado a la dirección IP pública del equilibrador de cargas de Ingress. Si usa DigitalOcean para administrar los registros DNS de su dominio, consulte Cómo administrar registros DNS para aprender a crear registros A
.Una vez que haya configurado estos componentes, estará listo para comenzar con esta guía.
En este paso, clonaremos el código de la aplicación de GitHub y configuraremos ajustes como las credenciales de la base de datos y las claves de almacenamiento de objetos.
El código de la aplicación y Dockerfile se encuentran en la rama polls-docker
del repositorio de GitHub de la aplicacipon Polls en el tutorial de Django. Este repositorio contiene el código para la aplicación Polls de muestra de la documentación de Django, que le enseña cómo crear una aplicación de encuestas desde cero.
La rama polls-docker
contiene una versión con Docker de esta aplicación Polls. Para obtener información sobre cómo se modificó la aplicación Polls para que funcionara de forma eficaz en un entorno de contenedores, consulte Cómo crear una aplicación de Django y Gunicorn con Docker.
Comience usando git
para clonar la rama polls-docker
del repositorio de GitHub de la aplicación Polls del tutorial de Django en su máquina local:
- git clone --single-branch --branch polls-docker https://github.com/do-community/django-polls.git
Diríjase al directorio django-polls
:
- cd django-polls
Este directorio contiene el código de Python de la aplicación de Django, un Dockerfile
que Docker utilizará para crear la imagen del contenedor, y un archivo env
que contiene una lista de las variables de entorno que se pasarán al entorno de ejecución del contenedor. Inspeccione el Dockerfile
:
- cat Dockerfile
OutputFROM python:3.7.4-alpine3.10
ADD django-polls/requirements.txt /app/requirements.txt
RUN set -ex \
&& apk add --no-cache --virtual .build-deps postgresql-dev build-base \
&& python -m venv /env \
&& /env/bin/pip install --upgrade pip \
&& /env/bin/pip install --no-cache-dir -r /app/requirements.txt \
&& runDeps="$(scanelf --needed --nobanner --recursive /env \
| awk '{ gsub(/,/, "\nso:", $2); print "so:" $2 }' \
| sort -u \
| xargs -r apk info --installed \
| sort -u)" \
&& apk add --virtual rundeps $runDeps \
&& apk del .build-deps
ADD django-polls /app
WORKDIR /app
ENV VIRTUAL_ENV /env
ENV PATH /env/bin:$PATH
EXPOSE 8000
CMD ["gunicorn", "--bind", ":8000", "--workers", "3", "mysite.wsgi"]
Este Dockerfile utiliza la imagen de Docker oficial de Python 3.7.4 como base e instala los requisitos del paquete de Python de Django y Gunicorn, tal como se define en el archivo django-polls/requirements.txt
. A continuación, elimina algunos archivos de compilación innecesarios, copia el código de la aplicación en la imagen y establece el PATH
de ejecución. Por último, declara que el puerto 8000
se utilizará para aceptar conexiones de contenedores entrantes y ejecuta gunicorn
con 3 trabajadores, escuchando en el puerto 8000
.
Para obtener más información sobre cada uno de los pasos de este Dockerfile, consulte el Paso 6 de Cómo crear una aplicación de Django y Gunicorn con Docker.
Ahora, compile la imagen con docker build
:
- docker build -t polls .
Nombramos la imagen polls
con el indicador -t
y pasamos el directorio actual como contexto de compilación, el conjunto de archivos a los que se debe hacer referencia al construir la imagen.
Una vez que Docker haya compilado y etiquetado la imagen, enumere las imágenes disponibles utilizando docker images
:
- docker images
Debería ver la imagen polls
enumerada:
OutputREPOSITORY TAG IMAGE ID CREATED SIZE
polls latest 80ec4f33aae1 2 weeks ago 197MB
python 3.7.4-alpine3.10 f309434dea3a 8 months ago 98.7MB
Antes de ejecutar el contenedor de Django, debemos configurar su entorno de ejecución utilizando el archivo env
presente en el directorio actual. Este archivo se pasará al comando docker run
que se utiliza para ejecutar el contenedor y Docker insertará las variables de entorno configuradas en el entorno en ejecución del contenedor.
Abra el archivo env
con nano
o su editor favorito:
- nano env
DJANGO_SECRET_KEY=
DEBUG=True
DJANGO_ALLOWED_HOSTS=
DATABASE_ENGINE=postgresql_psycopg2
DATABASE_NAME=polls
DATABASE_USERNAME=
DATABASE_PASSWORD=
DATABASE_HOST=
DATABASE_PORT=
STATIC_ACCESS_KEY_ID=
STATIC_SECRET_KEY=
STATIC_BUCKET_NAME=
STATIC_ENDPOINT_URL=
DJANGO_LOGLEVEL=info
Complete los valores que faltan para las siguientes claves:
DJANGO_SECRET_KEY
: establézcala en un valor único e impredecible, como se detalla en la documentación de Django. Se proporciona un método para generar esta clave en la sección Ajustar la configuración de la aplicación del tutorial Aplicaciones escalables de Django.DJANGO_ALLOWED_HOSTS
: esta variable asegura la aplicación y evita ataques a través del encabezado de host HTTP. Para propósitos de prueba, establézcala en *
, un comodín que coincidirá con todos los hosts. En producción, debe establecerlo en your_domain.com
. Para obtener más información sobre esta configuración de Django, consulte la sección Configuración principal en la documentación de Django.DATABASE_USERNAME
: establézcalo en el usuario de la base de datos de PostgreSQL creado en los pasos de requisitos previos.DATABASE_NAME
: establézcalo en polls
o el nombre de la base de datos de PostgreSQL creado en los pasos de requisitos previos.DATABASE_PASSWORD
: establézcala en la contraseña de la base de datos de PostgreSQL creada en los pasos de requisitos previos.DATABASE_HOST
: establézcalo en el nombre de host de su base de datos.DATABASE_PORT
: establézcalo en el puerto de su base de datos.STATIC_ACCESS_KEY_ID
: establézcala en la clave de acceso de Space o en el almacenamiento de objetos.STATIC_SECRET_KEY
: establézcala en Secret de la clave de acceso de su Space o en el almacenamiento de objetos.STATIC_BUCKET_NAME
: establézcalo en el nombre de su Space o en el depósito de almacenamiento de objetos.STATIC_ENDPOINT_URL
: establézcala en la URL de extremo de Spaces o del almacenamiento del objeto que corresponda, por ejemplo, https://your_space_name.nyc3.digitaloceanspaces.com
, si su Space está ubicado en la región nyc3
.Una vez que haya finalizado la edición, guarde y cierre el archivo.
En el siguiente paso, ejecutaremos el contenedor configurado a nivel local y crearemos el esquema de la base de datos. También cargaremos activos estáticos como hojas de estilos e imágenes al almacenamiento de objetos.
Con el contenedor creado y configurado, utilizaremos docker run
para anular el CMD
establecido en Dockerfile y crear el esquema de la base de datos utilizando los comandos manage.py makemigrations
y manage.py migrate
:
- docker run --env-file env polls sh -c "python manage.py makemigrations && python manage.py migrate"
Ejecutamos la imagen del contenedor polls:latest
, pasamos el archivo de variables de entorno que acabamos de modificar y anulamos el comando de Dockerfile con sh -c "python manage.py makemigrations && python manage.py migrate"
, lo que creará el esquema de la base de datos definido mediante el código de la aplicación.
Si lo está ejecutando por primera vez, debería ver lo siguiente:
OutputNo changes detected
Operations to perform:
Apply all migrations: admin, auth, contenttypes, polls, sessions
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying admin.0003_logentry_add_action_flag_choices... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying auth.0010_alter_group_name_max_length... OK
Applying auth.0011_update_proxy_permissions... OK
Applying polls.0001_initial... OK
Applying sessions.0001_initial... OK
Esto indica que el esquema de la base de datos se ha creado correctamente.
Si no está ejecutando migrate
por primera vez, Django realizará un no-op a menos que el esquema de la base de datos haya cambiado.
A continuación, ejecutaremos otra instancia del contenedor de la aplicación y utilizaremos una shell interactiva en su interior para crear un usuario administrativo para el proyecto de Django.
- docker run -i -t --env-file env polls sh
Esto le proporcionará una línea de comandos de shell dentro del contenedor en ejecución que puede usar para crear el usuario de Django:
- python manage.py createsuperuser
Ingrese un nombre de usuario, una dirección de correo electrónico y una contraseña para su usuario y, una vez que haya creado el usuario, presione CTRL+D
para salir del contenedor y cerrarlo.
Por último, generaremos los archivos estáticos de la aplicación y los subiremos al Space de DigitalOcean utilizando collectstatic
. Tenga en cuenta que esta operación puede tardar un poco en completarse.
- docker run --env-file env polls sh -c "python manage.py collectstatic --noinput"
Una vez que estos archivos se hayan generado y cargado, obtendrá el siguiente resultado.
Output121 static files copied.
Ahora, podemos ejecutar la aplicación:
- docker run --env-file env -p 80:8000 polls
Output[2019-10-17 21:23:36 +0000] [1] [INFO] Starting gunicorn 19.9.0
[2019-10-17 21:23:36 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
[2019-10-17 21:23:36 +0000] [1] [INFO] Using worker: sync
[2019-10-17 21:23:36 +0000] [7] [INFO] Booting worker with pid: 7
[2019-10-17 21:23:36 +0000] [8] [INFO] Booting worker with pid: 8
[2019-10-17 21:23:36 +0000] [9] [INFO] Booting worker with pid: 9
Aquí, ejecutamos el comando predeterminado definido en el Dockerfile, gunicorn --bind :8000 --workers 3 mysite.wsgi:application
y exponemos el puerto del contenedor 8000
para que el puerto 80
en su máquina local se asigne al puerto 8000
del contenedor polls
.
Ahora, debería poder navegar a la aplicación polls
desde su navegador web al http://localhost
en la barra de direcciones URL. Dado que no hay una ruta definida para la ruta /
, es probable que reciba un error de 404 Page Not Found
(Página no encontrada), lo que es de esperar.
Diríjase a http://localhost/polls
para ver la interfaz de la aplicación Polls:
Para ver la interfaz administrativa, visite http://localhost/admin
. Debería ver la ventana de autenticación de administración de la aplicación Polls:
Ingrese el nombre de usuario administrativo y la contraseña que creó con el comando createsuperuser
.
Después de la autenticación, podrá acceder a la interfaz administrativa de la aplicación Polls:
Tenga en cuenta que los que los recursos estáticos de las aplicaciones admin
y polls
se entregan directamente desde el almacenamiento de objetos. Para confirmar esto, consulte Prueba de la entrega de archivos estáticos de Spaces.
Cuando haya terminado de explorar, presione CTRL+C
en la ventana de terminal que está ejecutando el contenedor de Docker para cerrar el contenedor.
Con la imagen de Docker de la aplicación de Django probada, los activos estáticos que se cargan en el almacenamiento de objetos y el esquema de la base de datos configurado y listo para ser utilizado en su aplicación, estará listo para subir la imagen de su aplicación de Django a un registro de imágenes como Docker Hub.
Para implementar su aplicación en Kubernetes, la imagen de su aplicación debe cargarse en un registro como Docker Hub. Kubernetes extraerá la imagen de la aplicación de su repositorio y luego la implementará en su clúster.
Puede usar un registro de Docker privado, como DigitalOcean Container Registry, actualmente gratuito en Early Access o un registro de Docker público como Docker Hub. Docker Hub también le permite crear repositorios de Docker privados. Un repositorio público permite a cualquiera ver y extraer las imágenes del contenedor, mientras que un repositorio privado permite restringir el acceso a usted y a los miembros de su equipo.
En este tutorial, introduciremos la imagen de Django en el repositorio de Docker Hub público creado en los requisitos previos. También puede introducir su imagen en un repositorio privado, pero extraer imágenes de un repositorio privado no es el tema de este artículo. Para obtener más información sobre la autenticación de Kubernetes con Docker Hub y la extracción de imágenes privadas, consulte Extraer una imagen de un registro privado en la documentación de Kubernetes.
Comience iniciando sesión en Docker Hub en su máquina local:
- docker login
OutputLogin with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username:
Ingrese su nombre de usuario y contraseña de Docker Hub para iniciar sesión.
La imagen de Django actualmente tiene la etiqueta polls:latest
. Para introducirla en su repositorio de Docker Hub, vuelva a etiquetar la imagen con su nombre de usuario de Docker Hub y nombre de repositorio:
- docker tag polls:latest your_dockerhub_username/your_dockerhub_repo_name:latest
Introduzca la imagen en el repositorio:
- docker push sammy/sammy-django:latest
En este tutorial, el nombre de usuario de Docker Hub es sammy y el nombre de repositorio es sammy-django. Debe sustituir estos valores por su propio nombre de usuario y nombre de repositorio de Docker Hub.
Verá un resultado que se actualiza a medida que las capas de imágenes se insertan en Docker Hub.
Ahora que su imagen está disponible para Kubernetes en Docker Hub, puede comenzar a implementarla en su clúster.
Cuando ejecutamos el contenedor de Django a nivel local, pasamos el archivo env
a docker run
para insertar variables de configuración en el entorno de tiempo de ejecución. En Kubernetes, las variables de configuración pueden insertarse usando ConfigMaps y Secrets.
ConfigMaps debe usarse para almacenar información de configuración no confidencial, la configuración de la aplicación, mientras que Secrets debe usarse para información confidencial, como claves de API y credenciales de la base de datos. Ambos se insertan en contenedores de forma similar, pero Secrets tienen funciones de control de acceso y seguridad adicionales como encriptación en reposo. Secrets también almacenan datos en base64, mientras que ConfigMaps almacenan datos en texto simple.
Para comenzar, cree un directorio llamado yaml
en el que almacenaremos nuestros manifiestos de Kubernetes. Diríjase al directorio.
- mkdir yaml
- cd
Abra un archivo llamado polls-configmap.yaml
en nano
o su editor de texto preferido:
- nano polls-configmap.yaml
Pegue el siguiente manifiesto de ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: polls-config
data:
DJANGO_ALLOWED_HOSTS: "*"
STATIC_ENDPOINT_URL: "https://your_space_name.space_region.digitaloceanspaces.com"
STATIC_BUCKET_NAME: "your_space_name"
DJANGO_LOGLEVEL: "info"
DEBUG: "True"
DATABASE_ENGINE: "postgresql_psycopg2"
Extrajimos la configuración no sensible del archivo env
modificado en el Paso 1 y la pegamos en un manifiesto de ConfigMap. El objeto ConfigMap se llama polls-config
. Copie los mismos valores que ingresó en el archivo env
en el paso anterior.
Para propósitos de prueba, establezca DJANGO_ALLOWED_HOSTS
como *
para desactivar el filtrado del encabezado de host. En un entorno de producción, debe establecerlo en el dominio de su aplicación.
Cuando haya terminado de editar el archivo, guárdelo y ciérrelo.
Cree ConfigMap en su clúster usando kubectl apply
:
- kubectl apply -f polls-configmap.yaml
Outputconfigmap/polls-config created
Después de crear ConfigMap, crearemos el Secret que usó nuestra aplicación en el siguiente paso.
Los valores de Secret deben estar codificados en base64, lo que significa que crear objetos de Secret en su clúster es ligeramente más complicado que crear ConfigMaps. Puede repetir el proceso del paso anterior, codificando de forma manual los valores de Secret de base64 y pegándolos en un archivo de manifiesto. También puede crearlos usando un archivo de variable de entorno, kubectl create
y el indicador --from-env-file
, que realizaremos en este paso.
Una vez más, usaremos el archivo env
del Paso 1, eliminando las variables insertadas en ConfigMap. Cree una copia del archivo env
llamado polls-secrets
en el directorio yaml
:
- cp ../env ./polls-secrets
Edite el archivo en el editor de su preferencia:
- nano polls-secrets
DJANGO_SECRET_KEY=
DEBUG=True
DJANGO_ALLOWED_HOSTS=
DATABASE_ENGINE=postgresql_psycopg2
DATABASE_NAME=polls
DATABASE_USERNAME=
DATABASE_PASSWORD=
DATABASE_HOST=
DATABASE_PORT=
STATIC_ACCESS_KEY_ID=
STATIC_SECRET_KEY=
STATIC_BUCKET_NAME=
STATIC_ENDPOINT_URL=
DJANGO_LOGLEVEL=info
Elimine todas las variables insertadas en el manifiesto de ConfigMap. Cuando haya terminado, debe verse así:
DJANGO_SECRET_KEY=your_secret_key
DATABASE_NAME=polls
DATABASE_USERNAME=your_django_db_user
DATABASE_PASSWORD=your_django_db_user_password
DATABASE_HOST=your_db_host
DATABASE_PORT=your_db_port
STATIC_ACCESS_KEY_ID=your_space_access_key
STATIC_SECRET_KEY=your_space_access_key_secret
Asegúrese de usar los mismos valores que se utilizan en el Paso 1. Cuando haya terminado, guarde y cierre el archivo.
Cree el Secret en su clúster usando kubectl create secret
:
- kubectl create secret generic polls-secret --from-env-file=poll-secrets
Outputsecret/polls-secret created
Aquí creamos un objeto Secret llamado polls-secret
y pasamos el archivo de secrets que acabamos de crear.
Puede inspeccionar el Secret usando kubectl describe
:
- kubectl describe secret polls-secret
OutputName: polls-secret
Namespace: default
Labels: <none>
Annotations: <none>
Type: Opaque
Data
====
DATABASE_PASSWORD: 8 bytes
DATABASE_PORT: 5 bytes
DATABASE_USERNAME: 5 bytes
DJANGO_SECRET_KEY: 14 bytes
STATIC_ACCESS_KEY_ID: 20 bytes
STATIC_SECRET_KEY: 43 bytes
DATABASE_HOST: 47 bytes
DATABASE_NAME: 5 bytes
En este punto, almacenó la configuración de su aplicación en el clúster de Kubernetes usando los tipos de objeto Secret y ConfigMap. Ahora estamos listos para implementar la aplicación en el clúster.
En este paso, creará una implementación para su aplicación de Django. Una implementación de Kubernetes es un controlador que puede usarse para administrar aplicaciones sin estado en su clúster. Un controlador es un bucle de control que regula las cargas de trabajo aumentándolas o reduciéndolas. Los controladores también reinician y eliminan los contenedores fallidos.
Las implementaciones controlan uno o más Pods, la unidad implementable más pequeña de un clúster de Kubernetes. Los Pods están compuestos por uno o más contenedores. Para obtener más información sobre los diferentes tipos de cargas de trabajo que puede iniciar, consulte Introducción a Kubernetes.
Empiece por abrir un archivo llamado polls-deployment.yaml
en su editor favorito:
- nano polls-deployment.yaml
Pegue el siguiente manifiesto de implementación:
apiVersion: apps/v1
kind: Deployment
metadata:
name: polls-app
labels:
app: polls
spec:
replicas: 2
selector:
matchLabels:
app: polls
template:
metadata:
labels:
app: polls
spec:
containers:
- image: your_dockerhub_username/app_repo_name:latest
name: polls
envFrom:
- secretRef:
name: polls-secret
- configMapRef:
name: polls-config
ports:
- containerPort: 8000
name: gunicorn
Complete el nombre de la imagen del contenedor correspondiente, haciendo referencia a la imagen de Django Polls que agregó a Docker Hub en el Paso 2.
Aquí definimos una implementación de Kubernetes llamada polls-app
y la etiquetamos con el par clave-valor app: polls
. Especificamos que queremos ejecutar dos réplicas del Pod especificado debajo del campo template
.
Usando envFrom
con secretRef
y configMapRef
, especificamos que todos los datos del Secret polls-secret
y el ConfigMap polls-config
deben insertarse en los contenedores como variables de entorno. Las claves ConfigMap y Secret se convierten en los nombres de las variables de entorno.
Por último, exponemos el containerPort
8000
y lo nombramos gunicorn
.
Para obtener más información sobre la configuración de las implementaciones de Kubernetes, consulte Implementaciones en la documentación de Kubernetes.
Cuando haya terminado de editar el archivo, guárdelo y ciérrelo.
Cree la implementación en su clúster usando kubectl apply -f
:
- kubectl apply -f polls-deployment.yaml
- deployment.apps/polls-app created
Compruebe que la implementación se implementó correctamente usando kubectl get
:
- kubectl get deploy polls-app
OutputNAME READY UP-TO-DATE AVAILABLE AGE
polls-app 2/2 2 2 6m38s
Si encuentra un error o considera que algo no está funcionando, puede usar kubectl describe
para inspeccionar la implementación fallida:
- kubectl describe deploy
Puede inspeccionar los dos Pods usando kubectl get pod
:
- kubectl get pod
OutputNAME READY STATUS RESTARTS AGE
polls-app-847f8ccbf4-2stf7 1/1 Running 0 6m42s
polls-app-847f8ccbf4-tqpwm 1/1 Running 0 6m57s
Ahora hay dos réplicas de su aplicación de Django ejecutándose en el clúster. Para acceder a la aplicación, es necesario crear un Service de Kubernetes, lo cual es lo que haremos a continuación.
En este paso, creará un Service para su aplicación de Django. Un Service de Kubernetes es una abstracción que le permite exponer un conjunto de Pods en ejecución como servicio de red. Mediante un Service, puede crear un extremo estable para su aplicación que no cambia a medida que los Pods mueren y se vuelvan a crear.
Existen varios tipos de Service, incluidos ClusterIP Services, que exponen el servicio en un IP interno del clúster, NodePort Services que exponen el servicio en cada Nodo en un puerto estático llamado NodePort y LoadBalancer Services que proporcionan un equilibrador de carga en la nube para dirigir el tráfico externo a los Pods de su clúster (a través de NodePorts, que crea automáticamente). Para obtener más información sobre esos servicios, consulte el artículo Service en la documentación de Kubernetes.
En nuestra configuración final, usaremos un Service ClusterIP que se expone usando un Ingress y el controlador de Ingress configurado en los requisitos previos de esta guía. Por ahora, para comprobar que todo funciona correctamente, crearemos un servicio de NodePort temporal para acceder a la aplicación de Django.
Comience creando un archivo llamado polls-svc.yaml
usando el editor de su preferencia:
- nano polls-svc.yaml
Pegue el siguiente manifiesto de Service:
apiVersion: v1
kind: Service
metadata:
name: polls
labels:
app: polls
spec:
type: NodePort
selector:
app: polls
ports:
- port: 8000
targetPort: 8000
Aquí creamos un Service NodePort llamado polls
y le asignamos la etiqueta app: polls
. A continuación, seleccionamos los Pods backend con la etiqueta app: polls
y orientamos sus puertos 8000
.
Cuando haya terminado de editar el archivo, guárdelo y ciérrelo.
Implemente el Service usando kubectl apply
:
- kubectl apply -f polls-svc.yaml
Outputservice/polls created
Confirme que su Service se creó usando kubectl get svc
:
- kubectl get svc polls
OutputNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
polls NodePort 10.245.197.189 <none> 8000:32654/TCP 59s
Este resultado muestra la IP y NodePort internos del clúster de Service (32654
). Para conectarse al servicio, se necesitan las direcciones IP externas de los nodos del clúster:
- kubectl get node -o wide
OutputNAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
pool-7no0qd9e0-364fd Ready <none> 27h v1.18.8 10.118.0.5 203.0.113.1 Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9
pool-7no0qd9e0-364fi Ready <none> 27h v1.18.8 10.118.0.4 203.0.113.2 Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9
pool-7no0qd9e0-364fv Ready <none> 27h v1.18.8 10.118.0.3 203.0.113.3 Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9
En su navegador web, visite su aplicación Polls usando la dirección IP externa de cualquier nodo y el NodePort. Dado el resultado anterior, la URL de la aplicación sería la siguiente: http://203.0.113.1:32654/polls
.
Debería poder ver la misma interfaz de la aplicación Polls a la que accedió a nivel local en el Paso 1:
Puede repetir la misma prueba usando la ruta /admin:
http://203.0.113.1:32654/admin
. Debería ver la misma interfaz administrativa que antes:
En esta etapa, ya implementó dos réplicas del contenedor de la aplicación Polls de Django usando una implementación. También creó un extremo de red estable para estas dos réplicas e hizo que se pueda acceder a ellas externamente usando un servicio de NodePort
El paso final de este tutorial es proteger el tráfico externo a su aplicación usando HTTPS. Para ello, usaremos el controlador de Ingress-nginx
instalado en los requisitos previos y crearemos un objeto de Ingress para dirigir el tráfico externo al servicio de Kubernetes de polls
.
Los Ingress de Kubernetes le permiten dirigir de manera flexible el tráfico del exterior de su clúster de Kubernetes a servicios dentro de su clúster. Esto se realiza usando objetos de Ingress, que definen reglas para dirigir el tráfico HTTP y HTTPS a servicios de Kubernetes, y controladores de Ingress, que implementan las reglas equilibrando la carga de tráfico y dirigiéndola a los servicios de backend correspondientes.
En los requisitos previos, instaló el controlador de Ingress-nginx y el complemento para automatizar certificados TLS cert-manager. También ha configurado el ClusterIssuers de ensayo y producción de su dominio usando la autoridad de certificación Let’s Encrypt, y ha creado un Ingress para probar la emisión de certificados y el cifrado TLS en dos servicios de backend ficticios. Antes de continuar con este paso, debe eliminar el Ingress echo-ingress
creado en el tutorial de requisitos previos:
- kubectl delete ingress echo-ingress
Si desea, también puede eliminar los servicios y las implementaciones ficticios usando kubectl delete svc
y kubectl delete deploy
, pero no es necesario que lo haga para completar este tutorial.
También debería haber creado un registro DNS A
con your_domain.com
orientado a la dirección IP pública del equilibrador de cargas de Ingress. Si usa un equilibrador de carga de DigitalOcean, puede encontrar esta dirección IP en la sección Equilibradores de carga del panel de control. Si también usa DigitalOcean para administrar los registros DNS de su dominio, consulte Cómo administrar registros DNS para aprender a crear registros A
.
Si usa Kubernetes de DigitalOcean, también asegúrese de implementar la solución descrita en el Paso 5 de Cómo configurar un Ingress de Nginx con Cert-Manager en Kubernetes de DigitalOcean.
Una vez que tenga un registro A
que apunte al equilibrador de carga del controlador de Ingress, puede crear un Ingress para your_domain.com
y el servicio polls
.
Abra un archivo llamado polls-ingress.yaml
usando el editor de su preferencia:
- nano polls-ingress.yaml
Pegue el siguiente manifiesto de Ingress:
[polls-ingress.yaml]
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: polls-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt-staging"
spec:
tls:
- hosts:
- your_domain.com
secretName: polls-tls
rules:
- host: your_domain.com
http:
paths:
- backend:
serviceName: polls
servicePort: 8000
Creamos un objeto de Ingress llamado polls-ingress
y lo anotamos para indicarle al plano de control que utilice el controlador de Ingress-nginx y el ClusterIssuer de ensayo. También habilitamos TLS para your_domain.com
y almacenamos el certificado y la clave privada en un secret llamado polls-tls
. Por último, definimos una regla para dirigir el tráfico del host your_domain.com
al servicio polls
del puerto 8000
.
Cuando haya terminado de editar el archivo, guárdelo y ciérrelo.
Cree el Ingress en su clúster usando kubectl apply
:
- kubectl apply -f polls-ingress.yaml
Outputingress.networking.k8s.io/polls-ingress created
Puede usar kubectl describe
para rastrear el estado del Ingress que acaba de crear:
- kubectl describe ingress polls-ingress
OutputName: polls-ingress
Namespace: default
Address: workaround.your_domain.com
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
polls-tls terminates your_domain.com
Rules:
Host Path Backends
---- ---- --------
your_domain.com
polls:8000 (10.244.0.207:8000,10.244.0.53:8000)
Annotations: cert-manager.io/cluster-issuer: letsencrypt-staging
kubernetes.io/ingress.class: nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 51s nginx-ingress-controller Ingress default/polls-ingress
Normal CreateCertificate 51s cert-manager Successfully created Certificate "polls-tls"
Normal UPDATE 25s nginx-ingress-controller Ingress default/polls-ingress
También puede ejecutar un describe
en el certificado polls-tls
para confirmar una vez más que se creó de manera correcta:
- kubectl describe certificate polls-tls
Output. . .
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 3m33s cert-manager Issuing certificate as Secret does not exist
Normal Generated 3m32s cert-manager Stored new private key in temporary Secret resource "polls-tls-v9lv9"
Normal Requested 3m32s cert-manager Created new CertificateRequest resource "polls-tls-drx9c"
Normal Issuing 2m58s cert-manager The certificate has been successfully issued
Esto confirma que el certificado TLS ha sido emitido con éxito y que el cifrado HTTPS ahora está activo para your_domain.com
.
Dado que usamos el ClusterIssuer de ensayo, la mayoría de los navegadores de Internet no confiarán en el certificado Let’s Encrypt que emitió. Por lo tanto, dirigirse a your_domain.com
lo llevará a una página de error.
Para enviar una solicitud de prueba, usaremos wget
desde la línea de comandos:
- wget -O - http://your_domain.com/polls
Output. . .
ERROR: cannot verify your_domain.com's certificate, issued by ‘CN=Fake LE Intermediate X1’:
Unable to locally verify the issuer's authority.
To connect to your_domain.com insecurely, use `--no-check-certificate'.
Usaremos el indicador --no-check-certificate
sugerido para omitir la validación del certificado:
- wget --no-check-certificate -q -O - http://your_domain.com/polls
Output
<link rel="stylesheet" type="text/css" href="https://your_space.nyc3.digitaloceanspaces.com/django-polls/static/polls/style.css">
<p>No polls are available.</p>
Este resultado muestra el HTML de la página de la interfaz /polls
, lo que confirma también que la hoja de estilos se toma desde el almacenamiento de objetos.
Ahora que probó con éxito la emisión de certificados usando el ClusterIssuer de ensayo, puede modificar el Ingress para usar el ClusterIssuer de producción.
Abra polls-ingress.yaml
para editar una vez más:
- nano polls-ingress.yaml
Modifique la anotación cluster-issuer
:
[polls-ingress.yaml]
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: polls-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
tls:
- hosts:
- your_domain.com
secretName: polls-tls
rules:
- host: your_domain.com
http:
paths:
- backend:
serviceName: polls
servicePort: 8000
Cuando termine, guarde y cierre el archivo. Actualice el Ingress usando kubectl apply
:
- kubectl apply -f polls-ingress.yaml
Outputingress.networking.k8s.io/polls-ingress configured
Puede usar kubectl describe certificate polls-tls
y kubectl describe ingress polls-ingress
para rastrear el estado de emisión del certificado:
- kubectl describe ingress polls-ingress
Output. . .
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 23m nginx-ingress-controller Ingress default/polls-ingress
Normal CreateCertificate 23m cert-manager Successfully created Certificate "polls-tls"
Normal UPDATE 76s (x2 over 22m) nginx-ingress-controller Ingress default/polls-ingress
Normal UpdateCertificate 76s cert-manager Successfully updated Certificate "polls-tls"
El resultado anterior confirma que el nuevo certificado de producción se emitió y almacenó con éxito en el Secret polls-tls
.
Diríjase a your_domain.com/polls
en su navegador web para confirmar que el cifrado HTTPS está habilitado y que todo funciona según lo previsto. Debería ver la interfaz de la aplicación Polls:
Verifique que el cifrado HTTPS está activo en su navegador web. Si usa Google Chrome, llegar a la página anterior sin errores confirma que todo funciona correctamente. Además, debe ver un candado en la barra de direcciones URL. Al hacer clic en el candado, se le permitirá inspeccionar los detalles del certificado Let’s Encrypt.
Como tarea de limpieza final, puede cambiar opcionalmente el tipo de servicio polls
de NodePort al tipo ClusterIP interno solamente.
Modifique polls-svc.yaml
usando su editor:
- nano polls-svc.yaml
Cambie el type
de NodePort
a ClusterIP
:
apiVersion: v1
kind: Service
metadata:
name: polls
labels:
app: polls
spec:
type: ClusterIP
selector:
app: polls
ports:
- port: 8000
targetPort: 8000
Cuando haya terminado de editar el archivo, guárdelo y ciérrelo.
Implemente los cambios usando kubectl apply
:
- kubectl apply -f polls-svc.yaml --force
Outputservice/polls configured
Confirme que su Service se modificó usando kubectl get svc:
- kubectl get svc polls
OutputNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
polls ClusterIP 10.245.203.186 <none> 8000/TCP 22s
Este resultado muestra que el tipo de servicio es ahora ClusterIP. La única forma de acceder a él es a través de su dominio y del Ingress creado en este paso.
En este tutorial, implementó una aplicación de Django escalable y protegida con HTTPS en un clúster de Kubernetes. El contenido estático se toma directamente desde el almacenamiento de objetos y el número de Pods en ejecución puede aumentar o disminuir rápidamente usando el campo réplicas
en el manifiesto de implementación polls-app
.
Si usa un Space de DigitalOcean, también puede habilitar la entrega de activos estáticos a través de una red de entrega de contenido y crear un subdominio personalizado para su Space. Consulte Habilitar CDN en Cómo configurar una aplicación de Django escalable con bases de datos y Spaces administrados por DigitalOcean para obtener más información.
Para revisar el resto de la serie, visite nuestra página “De contenedores a Kubernetes con Django”.
]]>In diesem Tutorial stellen Sie eine containerisierte Django-Umfrageanwendung in einem Kubernetes-Cluster bereit.
Django ist ein leistungsfähiges Web-Framework, das Ihnen dabei helfen kann, Ihre Python-Anwendung schnell bereitzustellen. Es enthält mehrere praktische Funktionen wie einen objektrelationalen Mapper, eine Benutzerauthentifizierung und eine anpassbare Verwaltungsoberfläche für Ihre Anwendung. Es beinhaltet auch ein Caching-Framework und fördert ein sauberes App-Design durch seinen URL Dispatcher und das Vorlagensystem.
In Erstellen einer Django- und Gunicorn-Anwendung mit Docker wurde die Django Tutorial Umfrageanwendung gemäß der Zwölf-Faktor-Methodik zur Erstellung skalierbarer Cloud-nativer Web-Apps modifiziert. Diese containerisierte Einrichtung wurde mit einem Nginx Reverse-Proxy und Let’s Encrypt-signierten TLS-Zertifikaten in Skalieren und Sichern einer Django-Anwendung mit Docker, Nginx und Let’s Encrypt skaliert und gesichert. In diesem letzten Tutorial der Reihe Von Containern zu Kubernetes mit Django wird die modernisierte Django-Umfrageanwendung in einem Kubernetes-Cluster bereitgestellt.
Kubernetes ist ein leistungsstarker Open-Source-Container-Orchestrator, der die Bereitstellung, das Skalieren und die Verwaltung von containerisierten Anwendungen automatisiert. Mit Kubernetes-Objekten wie ConfigMaps und Secrets können Sie die Konfiguration zentralisieren und von Ihren Containern entkoppeln, während Controller wie Deployments fehlgeschlagene Container automatisch neu starten und eine schnelle Skalierung von Container-Replikaten ermöglichen. Die TLS-Verschlüsselung wird mit einem Ingress-Objekt und dem Open-Source Ingress-Controller ingress-nginx aktiviert. Das Kubernetes Add-on cert-manager erneuert und stellt Zertifikate mit der kostenlosen Zertifizierungsstelle Let’s Encrypt aus.
Um dieser Anleitung zu folgen, benötigen Sie:
kubectl
, das auf Ihrem lokalen Rechner installiert und für die Verbindung mit Ihrem Cluster konfiguriert ist. Weitere Informationen zur Installation von kubectl
finden Sie in der offiziellen Dokumentation. Wenn Sie einen DigitalOcean Kubernetes-Cluster verwenden, lesen Sie bitte Verbinden mit einem DigitalOcean Kubernetes-Cluster, um zu erfahren, wie Sie sich mit kubectl
mit Ihrem Cluster verbinden können.your_domain.com
verwendet. Einen Domänennamen können Sie kostenlos bei Freenom erhalten oder Sie nutzen eine Domänenregistrierungsstelle Ihrer Wahl.A
-DNS-Datensatz mit your_domain.com
, der auf die öffentliche IP-Adresse des Ingress Load Balancers verweist. Wenn Sie DigitalOcean zum Verwalten der DNS-Datensätze Ihrer Domäne verwenden, konsultieren Sie bitte Verwalten von DNS-Datensätzen, um zu erfahren, wie Sie A
-Datensätze erstellen.Sobald Sie diese Komponenten eingerichtet haben, können Sie mit diesem Leitfaden beginnen.
In diesem Schritt klonen wir den Anwendungscode von GitHub und konfigurieren Einstellungen wie Datenbankzugangsdaten und Objektspeicherschlüssel.
Der Anwendungscode und das Dockerfile befinden sich im Zweig polls-docker
der Django Tutorial Umfrageanwendung GitHub-Repository. Dieses Repository enthält Code für die Beispiel-Umfrageanwendung aus der Django-Dokumentation, in der Sie lernen, wie Sie eine Umfrageanwendung von Grund auf erstellen.
Der Zweig polls-docker
enthält eine dockerisierte Version der Umfrageanwendung. Um zu erfahren, wie die Umfrageanwendung modifiziert wurde, um effektiv in einer containerisierten Umgebung zu arbeiten, lesen Sie bitte Erstellen einer Django- und Gunicorn-Anwendung mit Docker.
Beginnen Sie mit der Verwendung von git
zum Klonen des Zweigs polls-docker
der Django Tutorial Umfrageanwendung GitHub-Repository auf Ihren lokalen Rechner:
- git clone --single-branch --branch polls-docker https://github.com/do-community/django-polls.git
Navigieren Sie in das Verzeichnis django-polls
:
- cd django-polls
Dieses Verzeichnis enthält den Python-Code der Django-Anwendung, ein Dockerfile
, das Docker zum Erstellen des Container-Images verwendet, sowie eine Datei env
, die eine Liste von Umgebungsvariablen enthält, die an die laufende Umgebung des Containers übergeben werden müssen. Prüfen Sie das Dockerfile
:
- cat Dockerfile
OutputFROM python:3.7.4-alpine3.10
ADD django-polls/requirements.txt /app/requirements.txt
RUN set -ex \
&& apk add --no-cache --virtual .build-deps postgresql-dev build-base \
&& python -m venv /env \
&& /env/bin/pip install --upgrade pip \
&& /env/bin/pip install --no-cache-dir -r /app/requirements.txt \
&& runDeps="$(scanelf --needed --nobanner --recursive /env \
| awk '{ gsub(/,/, "\nso:", $2); print "so:" $2 }' \
| sort -u \
| xargs -r apk info --installed \
| sort -u)" \
&& apk add --virtual rundeps $runDeps \
&& apk del .build-deps
ADD django-polls /app
WORKDIR /app
ENV VIRTUAL_ENV /env
ENV PATH /env/bin:$PATH
EXPOSE 8000
CMD ["gunicorn", "--bind", ":8000", "--workers", "3", "mysite.wsgi"]
Dieses Dockerfile verwendet das offizielle Python 3.7.4 Docker-Image als Basis und installiert die Python-Paketanforderungen von Django und Gunicorn, wie sie in der Datei django-polls/requirements.txt
definiert sind. Anschließend entfernt es einige unnötige Builddateien, kopiert den Anwendungscode in das Image und legt den Ausführungspfad PATH
fest. Schließlich gibt es an, dass Port 8000
verwendet wird, um eingehende Container-Verbindungen zu akzeptieren und gunicorn
mit 3 Workern ausgeführt wird, die Port 8000
abhören.
Um mehr über die einzelnen Schritte in diesem Dockerfile zu erfahren, lesen Sie bitte Schritt 6 von Erstellen einer Django- und Gunicorn-Anwendung mit Docker.
Erstellen Sie nun das Image mit docker build
:
- docker build -t polls .
Wir benennen das Image polls
mit dem Flag -t
und übergeben im aktuellen Verzeichnis als Build-Kontext den Satz von Daten, auf den beim Erstellen des Images verwiesen werden soll.
Nachdem Docker das Image erstellt und mit Tags versehen hat, listen wir die verfügbaren Images mit docker images
auf:
- docker images
Sie sollten die polls
-Images aufgelistet sehen:
OutputREPOSITORY TAG IMAGE ID CREATED SIZE
polls latest 80ec4f33aae1 2 weeks ago 197MB
python 3.7.4-alpine3.10 f309434dea3a 8 months ago 98.7MB
Bevor wir den Django-Container ausführen, müssen wir seine Betriebsumgebung mithilfe der im aktuellen Verzeichnis vorhandenen Datei env
konfigurieren. Diese Datei wird an den Befehl docker run
übergeben, der zum Ausführen des Containers verwendet wird, und Docker injiziert die konfigurierten Umgebungsvariablen in die Betriebsumgebung des Containers.
Öffnen Sie die Datei env
mit nano
oder Ihrem bevorzugten Editor:
- nano env
DJANGO_SECRET_KEY=
DEBUG=True
DJANGO_ALLOWED_HOSTS=
DATABASE_ENGINE=postgresql_psycopg2
DATABASE_NAME=polls
DATABASE_USERNAME=
DATABASE_PASSWORD=
DATABASE_HOST=
DATABASE_PORT=
STATIC_ACCESS_KEY_ID=
STATIC_SECRET_KEY=
STATIC_BUCKET_NAME=
STATIC_ENDPOINT_URL=
DJANGO_LOGLEVEL=info
Geben Sie die fehlenden Werte für die folgenden Schlüssel ein:
DJANGO_SECRET_KEY
: Setzen Sie diesen auf einen eindeutigen, nicht vorhersagbaren Wert, wie in den Django-Dokumentationen beschrieben. Eine Methode zur Generierung dieses Wertes wird in Anpassen der Anwendungseinstellungen in dem Tutorial Skalierbare Django-Anwendung angeboten.DJANGO_ALLOWED_HOSTS
: Diese Variable sichert die Anwendung und verhindert HTTP-Host-Header-Angriffe. Setzen Sie diese Variable für Testzwecke auf *
, einen Platzhalter, der auf alle Hosts zutrifft. In der Produktion sollten Sie diese Variable auf your_domain.com
setzen. Um mehr über diese Django-Einstellungen zu erfahren, konsultieren Sie die Core-Einstellungen der Django-Dokumentation.DATABASE_USERNAME
: Setzen Sie diesen auf den in den vorbereitenden Schritten erstellten PostgreSQL Datenbankbenutzer.DATABASE_NAME
: Setzen Sie diesen auf polls
oder den in den vorbereitenden Schritten erstellten Namen der PostgreSQL-Datenbank.DATABASE_PASSWORD
: Setzen Sie dieses auf das in den vorbereitenden Schritten erstellte Passwort für den PostgreSQL Benutzer.DATABASE_HOST
: Setzen Sie diesen Wert auf den Hostnamen Ihrer Datenbank.DATABASE_PORT
: Setzen Sie diesen Wert auf den Port Ihrer Datenbank.STATIC_ACCESS_KEY_ID
: Setzen Sie dies auf den Zugriffsschlüssel Ihres Space oder Objektspeichers.STATIC_SECRET_KEY
: Setzen Sie dies auf den Zugriffsschlüssel Ihres Space- oder Objektspeicher-Secret.STATIC_BUCKET_NAME
: Setzen Sie dies auf den Namen Ihres Space- oder Objektspeicher-Buckets.STATIC_ENDPOINT_URL
: Setzen Sie diese auf die entsprechende Endpunkt-URL des Space oder Objektspeichers, z. B. https://space-name.nyc3.digitaloceanspaces.com
, wenn sich Ihr Space in der Region nyc3
befindet.Wenn Sie die Bearbeitung abgeschlossen haben, speichern und schließen Sie die Datei.
Im nächsten Schritt führen wir den konfigurierten Container lokal aus und erstellen das Datenbankschema. Wir laden ebenfalls statische Assets wie Stylesheets und Images in den Objektspeicher hoch.
Nachdem der Container erstellt und konfiguriert ist, verwenden wir nun docker run
, um den CMD
-Satz in dem Dockerfile zu überschreiben und das Datenbankschema mit den Befehlen manage.py makemigrations
und manage.py migrate
zu erstellen:
- docker run --env-file env polls sh -c "python manage.py makemigrations && python manage.py migrate"
Wir führen das Container-Image polls:latest
aus, übergeben die von uns gerade modifizierte Umgebungsvariablendatei und überschreiben den Dockerfile-Befehl mit sh -c "python manage.py makemigrations && python manage.py migrate"
, wodurch das durch den Anwendungscode definierte Datenbankschema erstellt wird.
Wenn Sie dies zum ersten Mal ausführen, sollten Sie Folgendes sehen:
OutputNo changes detected
Operations to perform:
Apply all migrations: admin, auth, contenttypes, polls, sessions
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying admin.0003_logentry_add_action_flag_choices... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying auth.0010_alter_group_name_max_length... OK
Applying auth.0011_update_proxy_permissions... OK
Applying polls.0001_initial... OK
Applying sessions.0001_initial... OK
Dies zeigt an, dass das Datenbankschema erfolgreich erstellt wurde.
Wenn Sie migrate
zu einem späteren Zeitpunkt ausführen, führt Django eine Nulloperation durch, es sei denn, das Datenbankschema wurde geändert.
Als Nächstes führen wir eine weitere Instanz des Anwendungscontainers aus und verwenden darin eine interaktive Shell, um einen Administratorbenutzer für das Django-Projekt zu erstellen.
- docker run -i -t --env-file env polls sh
Dadurch erhalten Sie eine Shell-Eingabeaufforderung innerhalb des laufenden Containers, die Sie zum Erstellen des Django-Benutzers verwenden können:
- python manage.py createsuperuser
Geben Sie einen Benutzernamen, eine E-Mail-Adresse und ein Passwort für Ihren Benutzer ein. Drücken Sie nach dem Erstellen des Benutzers STRG+D
, um den Container zu verlassen und zu beenden.
Schließlich generieren wir die statischen Dateien für die Anwendung und laden sie mit collectstatic
in den DigitalOcean Space hoch. Beachten Sie, dass dies möglicherweise einige Zeit dauern kann.
- docker run --env-file env polls sh -c "python manage.py collectstatic --noinput"
Nachdem diese Dateien generiert und hochgeladen sind, erhalten Sie folgende Ausgabe.
Output121 static files copied.
Wir können die Anwendung nun ausführen:
- docker run --env-file env -p 80:8000 polls
Output[2019-10-17 21:23:36 +0000] [1] [INFO] Starting gunicorn 19.9.0
[2019-10-17 21:23:36 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
[2019-10-17 21:23:36 +0000] [1] [INFO] Using worker: sync
[2019-10-17 21:23:36 +0000] [7] [INFO] Booting worker with pid: 7
[2019-10-17 21:23:36 +0000] [8] [INFO] Booting worker with pid: 8
[2019-10-17 21:23:36 +0000] [9] [INFO] Booting worker with pid: 9
Hier führen wir den in dem Dockerfile definierten Standardbefehl gunicorn ---bind :8000 --workers 3 mysite.wsgi:application
aus und geben den Container-Port 8000
frei, sodass Port 80
auf Ihrem lokalen Rechner dem Port 8000
des Containers polls
zugeordnet wird.
Sie sollten nun über Ihren Webbrowser zu der Anwendung polls
navigieren können, indem Sie http://localhost
in die URL-Leiste eingeben. Da für den Pfad /
keine Route definiert ist, erhalten Sie wahrscheinlich einen 404 Page Not Found
-Fehler, der zu erwarten ist.
Navigieren Sie zu http://localhost/polls
, um die Benutzeroberfläche der Umfrageanwendung zu sehen:
Um die administrative Oberfläche anzuzeigen, besuchen Sie http://localhost/admin
. Sie sollten das Authentifizierungsfenster für den Administrator der Umfrageanwendung sehen:
Geben Sie den administrativen Benutzernamen und das Passwort ein, das Sie mit dem Befehl createsuperuser
erstellt haben.
Nach der Authentifizierung können Sie auf die administrative Oberfläche der Umfrageanwendung zugreifen:
Beachten Sie, dass statische Assets für die Anwendungen admin
und polls
direkt aus dem Objektspeicher bereitgestellt werden. Um dies zu bestätigen, konsultieren Sie Prüfen der statischen Dateizustellung von Spaces.
Wenn Sie die Erkundung abgeschlossen haben, drücken Sie STRG+
C im Terminalfenster, in dem der Docker-Container ausgeführt wird, um den Container zu beenden.
Nachdem das Docker-Image der Django-Anwendung getestet, die statischen Assets in den Objektspeicher hochgeladen und das Datenbankschema konfiguriert und für die Verwendung mit Ihrer Anwendung bereit ist/sind, können Sie Ihr Django-App-Image in eine Bildregistrierung wie Docker Hub hochladen.
Um Ihre Anwendung unter Kubernetes bereitzustellen, muss Ihr App-Image in eine Registrierung wie Docker Hub hochgeladen werden. Kubernetes zieht das App-Image aus seinem Repository und stellt es dann in Ihren Cluster bereit.
Sie können eine private Docker-Registrierung verwenden, wie die DigitalOcean Container Registry, die derzeit kostenlos in Early Access ist, oder eine öffentliche Docker-Registrierung wie Docker Hub. Mit Docker Hub können Sie auch private Docker-Repositorys erstellen. Ein öffentliches Repository erlaubt jedem, die Container-Images zu sehen und abzurufen, während Sie mit einem privaten Repository den Zugriff auf Sie und Ihre Teammitglieder beschränken können.
In diesem Tutorial verschieben wir das Django-Image in das öffentliche Docker Hub-Repository, das in den Voraussetzungen erstellt wurde. Sie können Ihr Image auch in ein privates Repository verschieben, aber das Abrufen von Images aus einem privaten Repository liegt außerhalb des Rahmens dieses Artikels. Um mehr über die Authentifizierung von Kubernetes mit Docker Hub und das Abrufen von privaten Images zu erfahren, lesen Sie bitte Abrufen eines Images aus einer privaten Registrierung aus den Kubernetes-Dokumenten.
Beginnen Sie mit der Anmeldung an Docker Hub auf Ihrem lokalen Rechner:
- docker login
OutputLogin with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username:
Geben Sie Ihren Docker Hub-Benutzernamen und Ihr Passwort ein, um sich anzumelden.
Das Django-Image hat derzeit die Markierung polls:latest
. Um es in Ihr Docker Hub-Repository zu verschieben, markieren Sie das Image erneut mit Ihrem Docker Hub-Benutzernamen und dem Repository-Namen:
- docker tag polls:latest your_dockerhub_username/your_dockerhub_repo_name:latest
Verschieben Sie das Image in das Repository:
- docker push sammy/sammy-django:latest
In diesem Tutorial ist der Docker Hub-Benutzername sammy und der Repository-Name ist sammy-django. Sie sollten diese Werte durch Ihren eigenen Docker Hub-Benutzernamen und Repository-Namen ersetzen.
Sie sehen eine Ausgabe, die sich aktualisiert, wenn Image-Schichten in Docker Hub verschoben werden.
Nachdem Ihr Image nun für Kubernetes unter Docker Hub verfügbar ist, können Sie es in Ihrem Cluster bereitstellen.
Als wir den Django-Container lokal ausgeführt haben, haben wir die Datei env
an docker run
übergeben, um Konfigurationsvariablen in die Laufzeitumgebung zu injizieren. Bei Kubernetes können Konfigurationsvariablen mit ConfigMaps und Secrets injiziert werden.
ConfigMaps sollte verwendet werden, um nicht vertrauliche Konfigurationsdaten wie App-Einstellungen zu speichern, und Secrets sollte für sensible Informationen wie API-Schlüssel und Datenbank-Zugangsdaten verwendet werden. Sie werden beide auf ähnliche Weise in Container injiziert, aber Secrets haben zusätzliche Zugriffskontrolle und Sicherheitsfunktionen wie Verschlüsselung im Ruhezustand. Secrets speichern außerdem Daten in base64, während ConfigMaps Daten im Klartext speichern.
Erstellen Sie zunächst ein Verzeichnis namens yaml
, in dem wir unsere Kubernetes-Manifeste speichern werden. Navigieren Sie in das Verzeichnis.
- mkdir yaml
- cd
Öffnen Sie eine Datei namens polls-configmap.yaml
in nano
oder Ihrem bevorzugten Texteditor:
- nano polls-configmap.yaml
Fügen Sie das folgende ConfigMap-Manifest ein:
apiVersion: v1
kind: ConfigMap
metadata:
name: polls-config
data:
DJANGO_ALLOWED_HOSTS: "*"
STATIC_ENDPOINT_URL: "https://your_space_name.space_region.digitaloceanspaces.com"
STATIC_BUCKET_NAME: "your_space_name"
DJANGO_LOGLEVEL: "info"
DEBUG: "True"
DATABASE_ENGINE: "postgresql_psycopg2"
Wir haben die nicht sensible Konfiguration aus der in Schritt 1
geänderten Datei env extrahiert und in ein ConfigMap-Manifest eingefügt. Das ConfigMap-Objekt wird polls-config
genannt. Kopieren Sie die gleichen Werte hinein, die Sie im vorherigen Schritt in die Datei env
eingegeben haben.
Lassen Sie für Testzwecke DJANGO_ALLOWED_HOSTS
auf *
stehen, um die Host-Header-basierte Filterung zu deaktivieren. In einer Produktionsumgebung sollten Sie dies auf die Domäne Ihrer Anwendung setzen.
Wenn Sie mit der Bearbeitung der Datei fertig sind, speichern und schließen Sie sie.
Erstellen Sie die ConfigMap in Ihrem Cluster mit kubectl apply
:
- kubectl apply -f polls-configmap.yaml
Outputconfigmap/polls-config created
Nachdem die ConfigMap erstellt wurde, erstellen wir im nächsten Schritt das von unserer Anwendung verwendete Secret.
Secret-Werte müssen base64-kodiert sein, d. h. das Erstellen von Secret-Objekten in Ihrem Cluster ist etwas aufwendiger als das Erstellen von ConfigMaps. Sie können den Vorgang aus dem vorherigen Schritt wiederholen, indem Sie Secret-Werte manuell base64-kodieren und in eine Manifestdatei einfügen. Sie können sie auch mit einer Umgebungsvariablendatei kubectl create
und dem Flag --from-env-file
erstellen, was wir in diesem Schritt tun werden.
Wir verwenden erneut die Datei env
von Schritt 1 und entfernen die in die ConfigMap eingefügten Variablen. Erstellen Sie eine Kopie der Datei env
mit dem Namen polls-secrets
im Verzeichnis yaml
:
- cp ../env ./polls-secrets
Bearbeiten Sie die Datei in Ihrem bevorzugten Editor:
- nano polls-secrets
DJANGO_SECRET_KEY=
DEBUG=True
DJANGO_ALLOWED_HOSTS=
DATABASE_ENGINE=postgresql_psycopg2
DATABASE_NAME=polls
DATABASE_USERNAME=
DATABASE_PASSWORD=
DATABASE_HOST=
DATABASE_PORT=
STATIC_ACCESS_KEY_ID=
STATIC_SECRET_KEY=
STATIC_BUCKET_NAME=
STATIC_ENDPOINT_URL=
DJANGO_LOGLEVEL=info
Löschen Sie alle in das ConfigMap-Manifest eingefügten Variablen. Wenn Sie fertig sind, sollte die Datei wie folgt aussehen:
DJANGO_SECRET_KEY=your_secret_key
DATABASE_NAME=polls
DATABASE_USERNAME=your_django_db_user
DATABASE_PASSWORD=your_django_db_user_password
DATABASE_HOST=your_db_host
DATABASE_PORT=your_db_port
STATIC_ACCESS_KEY_ID=your_space_access_key
STATIC_SECRET_KEY=your_space_access_key_secret
Stellen Sie sicher, dass Sie die gleichen Werte wie in Schritt 1 verwenden. Wenn Sie fertig sind, speichern und schließen Sie die Datei.
Erstellen Sie das Secret in Ihrem Cluster mit kubectl create secret
:
- kubectl create secret generic polls-secret --from-env-file=poll-secrets
Outputsecret/polls-secret created
Hier erstellen wir ein Secret-Objekt namens polls-secret
und übergeben die soeben erstellte Secrets-Datei.
Sie können das Secret mit kubectl describe
inspizieren:
- kubectl describe secret polls-secret
OutputName: polls-secret
Namespace: default
Labels: <none>
Annotations: <none>
Type: Opaque
Data
====
DATABASE_PASSWORD: 8 bytes
DATABASE_PORT: 5 bytes
DATABASE_USERNAME: 5 bytes
DJANGO_SECRET_KEY: 14 bytes
STATIC_ACCESS_KEY_ID: 20 bytes
STATIC_SECRET_KEY: 43 bytes
DATABASE_HOST: 47 bytes
DATABASE_NAME: 5 bytes
Zu diesem Zeitpunkt haben Sie die Konfiguration Ihrer Anwendung in Ihrem Kubernetes-Cluster mithilfe der Objekttypen Secret und ConfigMap gespeichert. Wir sind nun bereit, die Anwendung in dem Cluster bereitzustellen.
In diesem Schritt erstellen Sie ein Deployment für Ihre Django-Anwendung. Eine Kubernetes Deployment ist ein Controller, der zur Verwaltung von zustandslosen Anwendungen in Ihrem Cluster verwendet werden kann. Ein Controller ist eine Kontrollschleife, die Arbeitslasten reguliert, indem er sie herauf- oder herunterskaliert. Controller starten auch fehlgeschlagene Container neu und leeren sie aus.
Deployments steuern ein oder mehrere Pods, die kleinste einsetzbare Einheit in einem Kubernetes-Cluster. Pods umschließen einen oder mehrere Container. Um mehr über die verschiedenen Arten von Arbeistlasten zu erfahren, die Sie starten können, lesen Sie bitte Eine Einführung in Kubernetes.
Öffnen Sie zunächst eine Datei namens polls-deployment.yaml
in Ihrem bevorzugten Editor:
- nano polls-deployment.yaml
Fügen Sie das folgende Deployment-Manifest ein:
apiVersion: apps/v1
kind: Deployment
metadata:
name: polls-app
labels:
app: polls
spec:
replicas: 2
selector:
matchLabels:
app: polls
template:
metadata:
labels:
app: polls
spec:
containers:
- image: your_dockerhub_username/app_repo_name:latest
name: polls
envFrom:
- secretRef:
name: polls-secret
- configMapRef:
name: polls-config
ports:
- containerPort: 8000
name: gunicorn
Geben Sie den entsprechenden Container-Image-Namen ein und verweisen Sie dabei auf das Django Umfrage-Image, das Sie in Schritt 2 in Docker Hub verschoben haben.
Hier definieren wir eine Kubernetes Deployment namens polls-app
und kennzeichnen es mit dem Schlüsselwertpaar app: polls
. Wir geben an, dass wir zwei Replikate des Pods ausführen möchten, der unterhalb des Felds template
definiert ist.
Durch die Verwendung von envFrom
mit secretRef
und configMapRef
legen wir fest, dass alle Daten aus dem Secret polls-secret
und der ConfigMap polls-config
als Umgebungsvariablen in die Container injiziert werden sollen. Die Schlüssel ConfigMap und Secret werden zu den Namen der Umgebungsvariablen.
Abschließend geben wir containerPort
8000
frei und nennen ihn gunicorn
.
Um mehr über die Konfiguration von Kubernetes Deployments zu erfahren, konsultieren Sie bitte Deployments aus der Kubernetes-Dokumentation.
Wenn Sie mit der Bearbeitung der Datei fertig sind, speichern und schließen Sie sie.
Erstellen Sie das Deployment in Ihrem Cluster mit kubectl apply -f
:
- kubectl apply -f polls-deployment.yaml
- deployment.apps/polls-app created
Überprüfen Sie mit kubectl get
, ob das Deployment korrekt bereitgestellt wurde:
- kubectl get deploy polls-app
OutputNAME READY UP-TO-DATE AVAILABLE AGE
polls-app 2/2 2 2 6m38s
Wenn Sie auf einen Fehler stoßen, oder etwas nicht so ganz funktioniert, können Sie kubectl describe
verwenden, um das fehlgeschlagene Deployment zu inspizieren:
- kubectl describe deploy
Sie können die beiden Pods mit kubectl get pod
inspizieren:
- kubectl get pod
OutputNAME READY STATUS RESTARTS AGE
polls-app-847f8ccbf4-2stf7 1/1 Running 0 6m42s
polls-app-847f8ccbf4-tqpwm 1/1 Running 0 6m57s
Zwei Replikate Ihrer Django-Anwendung sind nun im Cluster in Betrieb. Um auf die Anwendung zuzugreifen, müssen Sie einen Kubernetes-Dienst erstellen, was wir als Nächstes tun.
In diesem Schritt erstellen Sie einen Dienst für Ihre Django-Anwendung. Ein Kubernetes-Dienst ist eine Abstraktion, die es Ihnen ermöglicht, einen Satz laufender Pods als Netzwerkdienst bereitzustellen. Mit einem Dienst können Sie einen stabilen Endpunkt für Ihre Anwendung erstellen, der sich nicht ändert, wenn Pods sterben und neu erstellt werden.
Es gibt mehrere Dienstarten, einschließlich ClusterIP-Dienste, die den Dienst auf einer clusterinternen IP-Adresse bereitstellen, NodePort-Dienste, die den Dienst auf jedem Knoten an einem statischen Port, dem Nodeport, bereitstellen, und LoadBalancer-Dienste, die einen Cloud-Load-Balancer bereitstellen, um externen Datenverkehr zu den Pods in Ihrem Cluster zu leiten (über NodePorts, die er automatisch erstellt). Um mehr über diese zu erfahren, lesen Sie bitte Service in den Kubernetes-Dokumenten.
In unserer endgültigen Einrichtung verwenden wir einen ClusterIP-Dienst, der über einen Ingress und den in den Voraussetzungen für diesen Leitfaden eingerichteten Ingress-Controller freigegeben wird. Um zu testen, ob alles korrekt funktioniert, erstellen wir zunächst einen temporären NodePort-Dienst, um auf die Django-Anwendung zuzugreifen.
Beginnen Sie mit dem Erstellen einer Datei namens polls-svc.yaml
mit Ihrem bevorzugten Editor:
- nano polls-svc.yaml
Fügen Sie das folgende Dienst-Manifest ein:
apiVersion: v1
kind: Service
metadata:
name: polls
labels:
app: polls
spec:
type: NodePort
selector:
app: polls
ports:
- port: 8000
targetPort: 8000
Hier erstellen wir einen NodePort-Dienst namens polls
und geben ihm die Kennzeichnung app: polls
. Dann wählen wir Backend-Pods mit der Kennzeichnung app: polls
, und zielen auf deren Ports 8000
.
Wenn Sie mit der Bearbeitung der Datei fertig sind, speichern und schließen Sie sie.
Stellen Sie den Dienst mit kubectl apply
bereit:
- kubectl apply -f polls-svc.yaml
Outputservice/polls created
Bestätigen Sie mit kubectl get svc
, dass Ihr Dienst erstellt wurde:
- kubectl get svc polls
OutputNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
polls NodePort 10.245.197.189 <none> 8000:32654/TCP 59s
Diese Ausgabe zeigt die clusterinterne IP des Dienstes und den NodePort (32654
). Um eine Verbindung mit dem Dienst herzustellen, benötigen wir die externen IP-Adressen für unsere Cluster-Knoten:
- kubectl get node -o wide
OutputNAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
pool-7no0qd9e0-364fd Ready <none> 27h v1.18.8 10.118.0.5 203.0.113.1 Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9
pool-7no0qd9e0-364fi Ready <none> 27h v1.18.8 10.118.0.4 203.0.113.2 Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9
pool-7no0qd9e0-364fv Ready <none> 27h v1.18.8 10.118.0.3 203.0.113.3 Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9
Rufen Sie in Ihrem Webbrowser Ihre Umfrageanwendung mit der externen IP-Adresse eines beliebigen Knotens und dem NodePort auf. Mit der vorstehenden Ausgabe würde die URL der Anwendung lauten: http://203.0.113.1:32654/polls
.
Sie sollten die gleiche Oberfläche der Umfrageanwendung sehen, auf die Sie in Schritt 1 lokal zugegriffen haben:
Sie können den gleichen Test mit der Route /admin
wiederholen: http://203.0.113.1:32654/admin
. Sie sollten die gleiche Admin-Oberfläche wie zuvor sehen:
Zu diesem Zeitpunkt haben Sie zwei Replikate des Django Umfrageanwendungs-Containers mithilfe eines Deployments bereitgestellt. Außerdem haben Sie einen stabilen Netzwerk-Endpunkt für diese beiden Replikate erstellt und ihn mit einem NodePort-Dienst von außen zugänglich gemacht.
Der letzte Schritt in diesem Tutorial ist die Sicherung des externen Datenverkehrs zu Ihrer Anwendung mit HTTPS. Dazu verwenden wir den in den Voraussetzungen installierten Ingress-Controller ingress-nginx
und erstellen ein Ingress-Objekt, um den externen Verkehr in den Kubernetes-Dienst polls
zu leiten.
Mit Ingresses von Kubernetes können Sie den Datenverkehr von außerhalb Ihres Kubernetes-Clusters flexibel an Dienste innerhalb Ihres Clusters leiten. Dies geschieht mit der Verwendung von Ingress-Objekten, die Regeln für das Routing von HTTP- und HTTPS-Verkehr zu Kubernetes-Diensten definieren, und Ingress-Controllern, die die Regeln umsetzen, indem sie den Verkehr durch Lastverteilung an die entsprechenden Backend-Dienste weiterleiten.
In den Voraussetzungen haben den Ingress-Controller ingress-nginx und das TLS-Zertifizierungsautomatisierungs-Add-on cert-manager installiert. Außerdem haben Sie die Staging- und Produktions-ClusterIssuers für Ihre Domäne unter Verwendung der Zertifizierungsstelle Let’s Encrypt eingerichtet und einen Ingress erstellt, um die Ausstellung von Zertifikaten und die TLS-Verschlüsselung für zwei Dummy-Backend-Dienste zu testen. Bevor Sie mit diesem Schritt fortfahren, sollten Sie den in dem Voraussetzungs-Tutorial
erstellten Ingress echo-ingress löschen:
- kubectl delete ingress echo-ingress
Wenn Sie möchten, können Sie auch die Dummy-Dienste und -Deployments mit kubectl delete svc
und kubectl delete delploy
löschen, aber dies ist für die Durchführung des Tutorials nicht unbedingt erforderlich.
Sie sollten auch einen DNS-A
-Datensatz mit your_domain.com
erstellt haben, der auf die öffentliche IP-Adresse des Ingress Load Balancers verweist. Wenn Sie einen DigitalOcean Load Balancer verwenden, finden Sie diese IP-Adresse im Abschnitt Load Balancers des Bedienfelds. Wenn Sie DigitalOcean auch für die Verwaltung der DNS-Datensätze Ihrer Domäne verwenden, konsultieren Sie bitte Verwalten von DNS-Datensätzen, um zu erfahren, wie Sie A
-Datensätze erstellen.
Wenn Sie DigitalOcean Kubernetes verwenden, stellen Sie außerdem sicher, dass Sie die in Schritt 5 von Einrichten eines Nginx-Ingress mit Cert-Manager unter DigitalOcean Kubernetes beschriebene Problemumgehung umgesetzt haben.
Sobald Sie einen A
-Datensatz haben, der auf den Ingress Controller Load Balancer verweist, können Sie einen Ingress für your_domain.com
und den Dienst polls
erstellen.
Öffnen Sie eine Datei namens polls-ingress.yaml
mit Ihrem bevorzugten Editor:
- nano polls-ingress.yaml
Fügen Sie das folgende Ingress-Manifest ein:
[polls-ingress.yaml]
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: polls-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt-staging"
spec:
tls:
- hosts:
- your_domain.com
secretName: polls-tls
rules:
- host: your_domain.com
http:
paths:
- backend:
serviceName: polls
servicePort: 8000
Wir erstellen ein Ingress-Objekt namens polls-ingress
und annotieren es, um die Steuerungsebene anzuweisen, den Ingress-Controller ingress-nginx und den ClusterIssuer „staging“ zu verwenden. Außerdem aktivieren wir TLS für your_domain.com
und speichern das Zertifikat und den privaten Schlüssel in einem Secret namens polls-tls
. Schließlich definieren wir eine Regel, um den Verkehr für den Host your_domain.com
an den Dienst polls
auf Port 8000
zu leiten.
Wenn Sie mit der Bearbeitung der Datei fertig sind, speichern und schließen Sie sie.
Erstellen Sie den Ingress in Ihrem Cluster mit kubectl apply
:
- kubectl apply -f polls-ingress.yaml
Outputingress.networking.k8s.io/polls-ingress created
Sie können kubectl describe
verwenden, um den Status des soeben erstellten Ingress zu verfolgen:
- kubectl describe ingress polls-ingress
OutputName: polls-ingress
Namespace: default
Address: workaround.your_domain.com
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
polls-tls terminates your_domain.com
Rules:
Host Path Backends
---- ---- --------
your_domain.com
polls:8000 (10.244.0.207:8000,10.244.0.53:8000)
Annotations: cert-manager.io/cluster-issuer: letsencrypt-staging
kubernetes.io/ingress.class: nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 51s nginx-ingress-controller Ingress default/polls-ingress
Normal CreateCertificate 51s cert-manager Successfully created Certificate "polls-tls"
Normal UPDATE 25s nginx-ingress-controller Ingress default/polls-ingress
Sie können auch describe
für das Zertifikat polls-tls
ausführen, um dessen erfolgreiche Erstellung weiter zu bestätigen:
- kubectl describe certificate polls-tls
Output. . .
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 3m33s cert-manager Issuing certificate as Secret does not exist
Normal Generated 3m32s cert-manager Stored new private key in temporary Secret resource "polls-tls-v9lv9"
Normal Requested 3m32s cert-manager Created new CertificateRequest resource "polls-tls-drx9c"
Normal Issuing 2m58s cert-manager The certificate has been successfully issued
Dies bestätigt, dass das TLS-Zertifikat erfolgreich ausgestellt wurde und die HTTPS-Verschlüsselung nun für your_domain.com
aktiv ist.
Da wir den Staging-ClusterIssuer verwendet haben, werden die meisten Webbrowser dem gefälschten Let’s Encrypt-Zertifikat nicht vertrauen, das es ausgestellt hat, sodass die Navigation zu your_domain.com
Sie zu einer Fehlerseite führt.
Um eine Testanfrage zu senden, verwenden wir wget
von der Befehlszeile aus:
- wget -O - http://your_domain.com/polls
Output. . .
ERROR: cannot verify your_domain.com's certificate, issued by ‘CN=Fake LE Intermediate X1’:
Unable to locally verify the issuer's authority.
To connect to your_domain.com insecurely, use `--no-check-certificate'.
Wir verwenden das vorgeschlagene Flag --no-check-certificate
, um die Zertifikatsprüfung zu umgehen:
- wget --no-check-certificate -q -O - http://your_domain.com/polls
Output
<link rel="stylesheet" type="text/css" href="https://your_space.nyc3.digitaloceanspaces.com/django-polls/static/polls/style.css">
<p>No polls are available.</p>
Diese Ausgabe zeigt das HTML für die Benutzeroberflächenseite /polls
und bestätigt auch, dass das Stylesheet aus dem Objektspeicher bedient wird.
Nachdem Sie nun die Zertifikatausstellung mit dem Staging-ClusterIssuer erfolgreich getestet haben, können Sie den Ingress ändern, um den ClusterIssuer, zu verwenden.
Öffnen Sie polls-ingress.yaml
zur erneuten Bearbeitung:
- nano polls-ingress.yaml
Ändern Sie die Annotation cluster-issuer
:
[polls-ingress.yaml]
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: polls-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
tls:
- hosts:
- your_domain.com
secretName: polls-tls
rules:
- host: your_domain.com
http:
paths:
- backend:
serviceName: polls
servicePort: 8000
Wenn Sie fertig sind, speichern und schließen Sie die Datei. Aktualisieren Sie den Ingress mit kubectl apply
:
- kubectl apply -f polls-ingress.yaml
Outputingress.networking.k8s.io/polls-ingress configured
Um den Status der Zertifikatausstellung zu verfolgen, können Sie kubectl describe certificate polls-tls
und kubectl describe ingress polls-ingress
verwenden:
- kubectl describe ingress polls-ingress
Output. . .
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 23m nginx-ingress-controller Ingress default/polls-ingress
Normal CreateCertificate 23m cert-manager Successfully created Certificate "polls-tls"
Normal UPDATE 76s (x2 over 22m) nginx-ingress-controller Ingress default/polls-ingress
Normal UpdateCertificate 76s cert-manager Successfully updated Certificate "polls-tls"
Die obige Ausgabe bestätigt, dass das neue Produktionszertifikat erfolgreich ausgestellt und im Secret polls-tls
gespeichert wurde.
Navigieren Sie in Ihrem Webbrowser zu your_domain.com/polls
, um zu bestätigen, dass die HTTPS-Verschlüsselung aktiviert ist und alles wie erwartet funktioniert. Sie sollten die Oberfläche der Umfrageanwendung sehen:
Überprüfen Sie, ob die HTTPS-Verschlüsselung in Ihrem Webbrowser aktiv ist. Wenn Sie Google Chrome verwenden, bestätigt die Ankunft auf der obigen Seite ohne Fehler, dass alles korrekt funktioniert. Außerdem sollten Sie ein Vorhängeschloss in der URL-Leiste sehen. Durch Klicken auf das Vorhängeschloss können Sie die Details des Let’s Encrypt-Zertifikats einsehen.
Als letzte Bereinigungsaufgabe können Sie optional den Typ des Dienstes polls
von NodePort auf den nut internen Typ ClusterIP umstellen.
Ändern Sie polls-svc.yaml
mit Ihrem Editor:
- nano polls-svc.yaml
Ändern Sie den type
von NodePort
auf ClusterIP
:
apiVersion: v1
kind: Service
metadata:
name: polls
labels:
app: polls
spec:
type: ClusterIP
selector:
app: polls
ports:
- port: 8000
targetPort: 8000
Wenn Sie mit der Bearbeitung der Datei fertig sind, speichern und schließen Sie sie.
Stellen Sie die Änderungen mit kubectl apply
bereit:
- kubectl apply -f polls-svc.yaml --force
Outputservice/polls configured
Bestätigen Sie mit kubectl get svc
, dass Ihr Dienst geändert wurde:
- kubectl get svc polls
OutputNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
polls ClusterIP 10.245.203.186 <none> 8000/TCP 22s
Diese Ausgabe zeigt, dass der Diensttyp nun ClusterIP ist. Der einzige Weg darauf zuzugreifen, ist über Ihre Domäne und den in diesem Schritt erstellten Ingress.
In diesem Tutorial haben Sie eine skalierbare HTTPS-gesicherte Django-Anwendung in einem Kubernetes-Cluster bereitgestellt. Statische Inhalte werden direkt aus dem Objektspeicher bedient, und die Anzahl der laufenden Pods kann über das Feld replicas
in dem Deployment-Manifest polls-app
schnell herauf- oder herunterskaliert werden.
Wenn Sie einen DigitalOcean Space verwenden, können Sie auch die Bereitstellung statischer Assets über ein Content Delivery-Netzwerk aktivieren und eine benutzerdefinierte Subdomäne für Ihren Space erstellen. Lesen Sie bitte Aktivieren von CDN aus Einrichten einer skalierbaren Django-Anwendung mit von DigitalOcean verwalteten Datenbanken und Spaces, um mehr zu erfahren.
Um den Rest der zu lesen, besuchen Sie bitte unsere Seite Von Containern zu Kubernetes mit Django.
]]>Velero is a convenient backup tool for Kubernetes clusters that compresses and backs up Kubernetes objects to object storage. It also takes snapshots of your cluster’s Persistent Volumes using your cloud provider’s block storage snapshot features, and can then restore your cluster’s objects and Persistent Volumes to a previous state.
The DigitalOcean Velero Plugin allows you to use DigitalOcean block storage to snapshot your Persistent Volumes, and Spaces to back up your Kubernetes objects. When running a Kubernetes cluster on DigitalOcean, this allows you to quickly back up your cluster’s state and restore it should disaster strike.
In this tutorial we’ll set up and configure the velero
command line tool on a local machine, and deploy the server component into our Kubernetes cluster. We’ll then deploy a sample Nginx app that uses a Persistent Volume for logging and then simulate a disaster recovery scenario.
If you’re looking for a managed Kubernetes hosting service, check out our simple, managed Kubernetes service built for growth.
Before you begin this tutorial, you should have the following available to you:
On your local computer:
kubectl
command-line tool, configured to connect to your cluster. You can read more about installing and configuring kubectl
in the official Kubernetes documentation.git
command-line utility. You can learn how to install git
in Getting Started with Git.In your DigitalOcean account:
1.7.5
or later) on DigitalOcean Droplets.Read/Write
permissions or snapshots will not work.Once you have all of this set up, you’re ready to begin with this guide.
The Velero backup tool consists of a client installed on your local computer and a server that runs in your Kubernetes cluster. To begin, we’ll install the local Velero client.
In your web browser, navigate to the Velero GitHub repo releases page, find the release corresponding to your OS and system architecture, and copy the link address. For the purposes of this guide, we’ll use an Ubuntu 18.04 server on an x86-64 (or AMD64) processor as our local machine, and the Velero v1.2.0
release.
Note: To follow this guide, you should download and install v1.2.0 of the Velero client.
Then, from the command line on your local computer, navigate to the temporary /tmp
directory and cd
into it:
- cd /tmp
Use wget
and the link you copied earlier to download the release tarball:
- wget https://link_copied_from_release_page
Once the download completes, extract the tarball using tar
(note the filename may differ depending on the release version and your OS):
- tar -xvzf velero-v1.2.0-linux-amd64.tar.gz
The /tmp
directory should now contain the extracted velero-v1.2.0-linux-amd64
directory as well as the tarball you just downloaded.
Verify that you can run the velero
client by executing the binary:
- ./velero-v1.2.0-linux-amd64/velero help
You should see the following help output:
OutputVelero is a tool for managing disaster recovery, specifically for Kubernetes
cluster resources. It provides a simple, configurable, and operationally robust
way to back up your application state and associated data.
If you're familiar with kubectl, Velero supports a similar model, allowing you to
execute commands such as 'velero get backup' and 'velero create schedule'. The same
operations can also be performed as 'velero backup get' and 'velero schedule create'.
Usage:
velero [command]
Available Commands:
backup Work with backups
backup-location Work with backup storage locations
bug Report a Velero bug
client Velero client related commands
completion Output shell completion code for the specified shell (bash or zsh)
create Create velero resources
delete Delete velero resources
describe Describe velero resources
get Get velero resources
help Help about any command
install Install Velero
plugin Work with plugins
restic Work with restic
restore Work with restores
schedule Work with schedules
snapshot-location Work with snapshot locations
version Print the velero version and associated image
. . .
At this point you should move the velero
executable out of the temporary /tmp
directory and add it to your PATH
. To add it to your PATH
on an Ubuntu system, simply copy it to /usr/local/bin
:
- sudo mv velero-v1.2.0-linux-amd64/velero /usr/local/bin/velero
You’re now ready to configure secrets for the Velero server and then deploy it to your Kubernetes cluster.
Before setting up the server component of Velero, you will need to prepare your DigitalOcean Spaces keys and API token. Again navigate to the temporary directory /tmp
using the cd
command:
- cd /tmp
Now we’ll download a copy of the Velero plugin for DigitalOcean. Visit the plugin’s Github releases page and copy the link to the file ending in .tar.gz
.
Use wget
and the link you copied earlier to download the release tarball:
- wget https://link_copied_from_release_page
Once the download completes, extract the tarball using tar
(again note that the filename may differ depending on the release version):
- tar -xvzf v1.0.0.tar.gz
The /tmp
directory should now contain the extracted velero-plugin-1.0.0
directory as well as the tarball you just downloaded.
Next we’ll cd
into the velero-plugin-1.0.0
directory:
- cd velero-plugin-1.0.0
Now we can save the access keys for our DigitalOcean Space and API token for use as a Kubernetes Secret. First, open up the examples/cloud-credentials
file using your favorite editor.
- nano examples/cloud-credentials
The file will look like this:
[default]
aws_access_key_id=<AWS_ACCESS_KEY_ID>
aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
Edit the <AWS_ACCESS_KEY_ID>
and <AWS_SECRET_ACCESS_KEY>
placeholders to use your DigitalOcean Spaces keys. Be sure to remove the <
and >
characters.
The next step is to edit the 01-velero-secret.patch.yaml
file so that it includes your DigitalOcean API token. Open the file in your favourite editor:
- nano examples/01-velero-secret.patch.yaml
It should look like this:
---
apiVersion: v1
kind: Secret
stringData:
digitalocean_token: <DIGITALOCEAN_API_TOKEN>
type: Opaque
Change the entire <DIGITALOCEAN_API_TOKEN>
placeholder to use your DigitalOcean personal API token. The line should look something like digitalocean_token: 18a0d730c0e0....
. Again, make sure to remove the <
and >
characters.
A Velero installation consists of a number of Kubernetes objects that all work together to create, schedule, and manage backups. The velero
executable that you just downloaded can generate and install these objects for you. The velero install
command will perform the preliminary set-up steps to get your cluster ready for backups. Specifically, it will:
Create a velero
Namespace.
Add the velero
Service Account.
Configure role-based access control (RBAC) rules to grant permissions to the velero
Service Account.
Install Custom Resource Definitions (CRDs) for the Velero-specific resources: Backup
, Schedule
, Restore
, Config
.
Register Velero Plugins to manage Block snapshots and Spaces storage.
We will run the velero install
command with some non-default configuration options. Specifically, you will to need edit each of the following settings in the actual invocation of the command to match your Spaces configuration:
--bucket velero-backups
: Change the velero-backups
value to match the name of your DigitalOcean Space. For example if you called your Space ‘backup-bucket’, the option would look like this: --bucket backup-bucket
--backup-location-config s3Url=https://nyc3.digitaloceanspaces.com,region=nyc3
: Change the URL and region to match your Space’s settings. Specifically, edit both nyc3
portions to match the region where your Space is hosted. For example, if your Space is hosted in the fra1
region, the line would look like this: --backup-location-config s3Url=https://fra1.digitaloceanspaces.com,region=fra1
. The identifiers for regions are: nyc3
, sfo2
, sgp1
, and fra1
.Once you are ready with the appropriate bucket and backup location settings, it is time to install Velero. Run the following command, substituting your values where required:
- velero install \
- --provider velero.io/aws \
- --bucket velero-backups \
- --plugins velero/velero-plugin-for-aws:v1.0.0,digitalocean/velero-plugin:v1.0.0 \
- --backup-location-config s3Url=https://nyc3.digitaloceanspaces.com,region=nyc3 \
- --use-volume-snapshots=false \
- --secret-file ./examples/cloud-credentials
You should see the following output:
OutputCustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: created
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource
CustomResourceDefinition/backupstoragelocations.velero.io: created
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource
CustomResourceDefinition/deletebackuprequests.velero.io: created
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource
CustomResourceDefinition/downloadrequests.velero.io: created
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource
CustomResourceDefinition/podvolumebackups.velero.io: created
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource
CustomResourceDefinition/podvolumerestores.velero.io: created
CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource
CustomResourceDefinition/resticrepositories.velero.io: created
CustomResourceDefinition/restores.velero.io: attempting to create resource
CustomResourceDefinition/restores.velero.io: created
CustomResourceDefinition/schedules.velero.io: attempting to create resource
CustomResourceDefinition/schedules.velero.io: created
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource
CustomResourceDefinition/serverstatusrequests.velero.io: created
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource
CustomResourceDefinition/volumesnapshotlocations.velero.io: created
Waiting for resources to be ready in cluster...
Namespace/velero: attempting to create resource
Namespace/velero: created
ClusterRoleBinding/velero: attempting to create resource
ClusterRoleBinding/velero: created
ServiceAccount/velero: attempting to create resource
ServiceAccount/velero: created
Secret/cloud-credentials: attempting to create resource
Secret/cloud-credentials: created
BackupStorageLocation/default: attempting to create resource
BackupStorageLocation/default: created
Deployment/velero: attempting to create resource
Deployment/velero: created
Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.
You can watch the deployment logs using the kubectl
command from the output. Once your deploy is ready, you can proceed to the next step, which is configuring the server. A successful deploy will look like this (with a different AGE column):
- kubectl get deployment/velero --namespace velero
OutputNAME READY UP-TO-DATE AVAILABLE AGE
velero 1/1 1 1 2m
At this point you have installed the server component of Velero into your Kubernetes cluster as a Deployment. You have also registered your Spaces keys with Velero using a Kubernetes Secret.
Note: You can specify the kubeconfig
that the velero
command line tool should use with the --kubeconfig
flag. If you don’t use this flag, velero
will check the KUBECONFIG
environment variable and then fall back to the kubectl
default (~/.kube/config
).
When we installed the Velero server, the option --use-volume-snapshots=false
was part of the command. Since we want to take snapshots of the underlying block storage devices in our Kubernetes cluster, we need to tell Velero to use the correct plugin for DigitalOcean block storage.
Run the following command to enable the plugin and register it as the default snapshot provider:
- velero snapshot-location create default --provider digitalocean.com/velero
You will see the following output:
OutputSnapshot volume location "default" configured successfully.
In the previous step we created block storage and object storage objects in the Velero server. We’ve registered the digitalocean/velero-plugin:v1.0.0
plugin with the server, and installed our Spaces secret keys into the cluster.
The final step is patching the cloud-credentials
Secret that we created earlier to use our DigitalOcean API token. Without this token the snapshot plugin will not be able to authenticate with the DigitalOcean API.
We could use the kubectl edit
command to modify the Velero Deployment object with a reference to the API token. However, editing complex YAML objects by hand can be tedious and error prone. Instead, we’ll use the kubectl patch
command since Kubernetes supports patching objects. Let’s take a quick look at the contents of the patch files that we’ll apply.
The first patch file is the examples/01-velero-secret.patch.yaml
file that you edited earlier. It is designed to add your API token to the secrets/cloud-credentials
Secret that already contains your Spaces keys. cat
the file:
- cat examples/01-velero-secret.patch.yaml
It should look like this (with your token in place of the <DIGITALOCEAN_API_TOKEN>
placeholder):
. . .
---
apiVersion: v1
kind: Secret
stringData:
digitalocean_token: <DIGITALOCEAN_API_TOKEN>
type: Opaque
Now let’s look at the patch file for the Deployment:
- cat examples/02-velero-deployment.patch.yaml
You should see the following YAML:
. . .
---
apiVersion: v1
kind: Deployment
spec:
template:
spec:
containers:
- args:
- server
command:
- /velero
env:
- name: DIGITALOCEAN_TOKEN
valueFrom:
secretKeyRef:
key: digitalocean_token
name: cloud-credentials
name: velero
This file indicates that we’re patching a Deployment’s Pod spec that is called velero
. Since this is a patch we do not need to specify an entire Kubernetes object spec or metadata. In this case the Velero Deployment is already configured using the cloud-credentials
secret because the velero install
command created it for us. So all that this patch needs to do is register the digitalocean_token
as an environment variable with the already deployed Velero Pod.
Let’s apply the first Secret patch using the kubectl patch
command:
- kubectl patch secret/cloud-credentials -p "$(cat examples/01-velero-secret.patch.yaml)" --namespace velero
You should see the following output:
Outputsecret/cloud-credentials patched
Finally we will patch the Deployment. Run the following command:
- kubectl patch deployment/velero -p "$(cat examples/02-velero-deployment.patch.yaml)" --namespace velero
You will see the following if the patch is successful:
Outputdeployment.apps/velero patched
Let’s verify the patched Deployment is working using kubectl get
on the velero
Namespace:
- kubectl get deployment/velero --namespace velero
You should see the following output:
OutputNAME READY UP-TO-DATE AVAILABLE AGE
velero 1/1 1 1 12s
At this point Velero is running and fully configured, and ready to back up and restore your Kubernetes cluster objects and Persistent Volumes to DigitalOcean Spaces and Block Storage.
In the next section, we’ll run a quick test to make sure that the backup and restore functionality works as expected.
Now that we’ve successfully installed and configured Velero, we can create a test Nginx Deployment, with a Persistent Volume and Service. Once the Deployment is running we will run through a backup and restore drill to ensure that Velero is configured and working properly.
Ensure you are still working in the /tmp/velero-plugin-1.0.0
directory. The examples
directory contains a sample Nginx manifest called nginx-example.yaml
.
Open this file using your editor of choice:
- nano examples/nginx-example.yaml
You should see the following text:
Output. . .
---
apiVersion: v1
kind: Namespace
metadata:
name: nginx-example
labels:
app: nginx
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: nginx-logs
namespace: nginx-example
labels:
app: nginx
spec:
storageClassName: do-block-storage
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deploy
namespace: nginx-example
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
volumes:
- name: nginx-logs
persistentVolumeClaim:
claimName: nginx-logs
containers:
- image: nginx:stable
name: nginx
ports:
- containerPort: 80
volumeMounts:
- mountPath: "/var/log/nginx"
name: nginx-logs
readOnly: false
---
apiVersion: v1
kind: Service
metadata:
labels:
app: nginx
name: nginx-svc
namespace: nginx-example
spec:
ports:
- port: 80
targetPort: 80
selector:
app: nginx
type: LoadBalancer
In this file, we observe specs for:
nginx-example
nginx:stable
container imagenginx-logs
), using the do-block-storage
StorageClassLoadBalancer
Service that exposes port 80
Create the objects using kubectl apply
:
- kubectl apply -f examples/nginx-example.yaml
You should see the following output:
Outputnamespace/nginx-example created
persistentvolumeclaim/nginx-logs created
deployment.apps/nginx-deploy created
service/nginx-svc created
Check that the Deployment succeeded:
- kubectl get deployments --namespace=nginx-example
You should see the following output:
OutputNAME READY UP-TO-DATE AVAILABLE AGE
nginx-deploy 1/1 1 1 1m23s
Once Available
reaches 1, fetch the Nginx load balancer’s external IP using kubectl get
:
- kubectl get services --namespace=nginx-example
You should see both the internal CLUSTER-IP
and EXTERNAL-IP
for the my-nginx
Service:
OutputNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-svc LoadBalancer 10.245.147.61 159.203.48.191 80:30232/TCP 3m1s
Note the EXTERNAL-IP
and navigate to it using your web browser.
You should see the following NGINX welcome page:
This indicates that your Nginx Deployment and Service are up and running.
Before we simulate our disaster scenario, let’s first check the Nginx access logs (stored on a Persistent Volume attached to the Nginx Pod):
Fetch the Pod’s name using kubectl get
:
- kubectl get pods --namespace nginx-example
OutputNAME READY STATUS RESTARTS AGE
nginx-deploy-694c85cdc8-vknsk 1/1 Running 0 4m14s
Now, exec
into the running Nginx container to get a shell inside of it:
- kubectl exec -it nginx-deploy-694c85cdc8-vknsk --namespace nginx-example -- /bin/bash
Once inside the Nginx container, cat
the Nginx access logs:
- cat /var/log/nginx/access.log
You should see some Nginx access entries:
Output10.244.0.119 - - [03/Jan/2020:04:43:04 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
10.244.0.119 - - [03/Jan/2020:04:43:04 +0000] "GET /favicon.ico HTTP/1.1" 404 153 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
Note these down (especially the timestamps), as we will use them to confirm the success of the restore procedure. Exit the pod:
- exit
We can now perform the backup procedure to copy all nginx
Kubernetes objects to Spaces and take a Snapshot of the Persistent Volume we created when deploying Nginx.
We’ll create a backup called nginx-backup
using the velero
command line client:
- velero backup create nginx-backup --selector app=nginx
The --selector app=nginx
instructs the Velero server to only back up Kubernetes objects with the app=nginx
Label Selector.
You should see the following output:
OutputBackup request "nginx-backup" submitted successfully.
Run `velero backup describe nginx-backup` or `velero backup logs nginx-backup` for more details.
Running velero backup describe nginx-backup --details
should provide the following output after a short delay:
OutputName: nginx-backup
Namespace: velero
Labels: velero.io/backup=nginx-backup
velero.io/pv=pvc-6b7f63d7-752b-4537-9bb0-003bed9129ca
velero.io/storage-location=default
Annotations: <none>
Phase: Completed
Namespaces:
Included: *
Excluded: <none>
Resources:
Included: *
Excluded: <none>
Cluster-scoped: auto
Label selector: app=nginx
Storage Location: default
Snapshot PVs: auto
TTL: 720h0m0s
Hooks: <none>
Backup Format Version: 1
Started: 2020-01-02 23:45:30 -0500 EST
Completed: 2020-01-02 23:45:34 -0500 EST
Expiration: 2020-02-01 23:45:30 -0500 EST
Resource List:
apps/v1/Deployment:
- nginx-example/nginx-deploy
apps/v1/ReplicaSet:
- nginx-example/nginx-deploy-694c85cdc8
v1/Endpoints:
- nginx-example/nginx-svc
v1/Namespace:
- nginx-example
v1/PersistentVolume:
- pvc-6b7f63d7-752b-4537-9bb0-003bed9129ca
v1/PersistentVolumeClaim:
- nginx-example/nginx-logs
v1/Pod:
- nginx-example/nginx-deploy-694c85cdc8-vknsk
v1/Service:
- nginx-example/nginx-svc
Persistent Volumes:
pvc-6b7f63d7-752b-4537-9bb0-003bed9129ca:
Snapshot ID: dfe866cc-2de3-11ea-9ec0-0a58ac14e075
Type: ext4
Availability Zone:
IOPS: <N/A>
This output indicates that nginx-backup
completed successfully. The list of resources shows each of the Kubernetes objects that was included in the backup. The final section shows the PersistentVolume was also backed up using a filesystem snapshot.
To confirm from within the DigitalOcean Cloud Control Panel, navigate to the Space containing your Kubernetes backup files.
You should see a new directory called nginx-backup
containing the Velero backup files.
Using the left-hand navigation bar, go to Images and then Snapshots. Within Snapshots, navigate to Volumes. You should see a Snapshot corresponding to the PVC listed in the above output.
We can now test the restore procedure.
Let’s first delete the nginx-example
Namespace. This will delete everything in the Namespace, including the Load Balancer and Persistent Volume:
- kubectl delete namespace nginx-example
Verify that you can no longer access Nginx at the Load Balancer endpoint, and that the nginx-example
Deployment is no longer running:
- kubectl get deployments --namespace=nginx-example
OutputNo resources found in nginx-example namespace.
We can now perform the restore procedure, once again using the velero
client:
- velero restore create --from-backup nginx-backup
Here we use create
to create a Velero Restore
object from the nginx-backup
object.
You should see the following output:
- OutputRestore request "nginx-backup-20200102235032" submitted successfully.
- Run `velero restore describe nginx-backup-20200102235032` or `velero restore logs nginx-backup-20200102235032` for more details.
Check the status of the restored Deployment:
- kubectl get deployments --namespace=nginx-example
OutputNAME READY UP-TO-DATE AVAILABLE AGE
nginx-deploy 1/1 1 1 58s
Check for the creation of a Persistent Volume:
- kubectl get pvc --namespace=nginx-example
OutputNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nginx-logs Bound pvc-6b7f63d7-752b-4537-9bb0-003bed9129ca 5Gi RWO do-block-storage 75s
The restore also created a LoadBalancer. Sometimes the Service will be re-created with a new IP address. You will need to find the EXTERNAL-IP
address again:
- kubectl get services --namespace nginx-example
OutputNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-svc LoadBalancer 10.245.15.83 159.203.48.191 80:31217/TCP 97s
Navigate to the Nginx Service’s external IP once again to confirm that Nginx is up and running.
Finally, check the logs on the restored Persistent Volume to confirm that the log history has been preserved post-restore.
To do this, once again fetch the Pod’s name using kubectl get
:
- kubectl get pods --namespace nginx-example
OutputNAME READY STATUS RESTARTS AGE
nginx-deploy-694c85cdc8-vknsk 1/1 Running 0 2m20s
Then exec
into it:
- kubectl exec -it nginx-deploy-694c85cdc8-vknsk --namespace nginx-example -- /bin/bash
Once inside the Nginx container, cat
the Nginx access logs:
- cat /var/log/nginx/access.log
Output10.244.0.119 - - [03/Jan/2020:04:43:04 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
10.244.0.119 - - [03/Jan/2020:04:43:04 +0000] "GET /favicon.ico HTTP/1.1" 404 153 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
You should see the same pre-backup access attempts (note the timestamps), confirming that the Persistent Volume restore was successful. Note that there may be additional attempts in the logs if you visited the Nginx landing page after you performed the restore.
At this point, we’ve successfully backed up our Kubernetes objects to DigitalOcean Spaces, and our Persistent Volumes using Block Storage Volume Snapshots. We simulated a disaster scenario, and restored service to the test Nginx application.
In this guide we installed and configured the Velero Kubernetes backup tool on a DigitalOcean-based Kubernetes cluster. We configured the tool to back up Kubernetes objects to DigitalOcean Spaces, and back up Persistent Volumes using Block Storage Volume Snapshots.
Velero can also be used to schedule regular backups of your Kubernetes cluster for disaster recovery. To do this, you can use the velero schedule
command. Velero can also be used to migrate resources from one cluster to another.
To learn more about DigitalOcean Spaces, consult the official Spaces documentation. To learn more about Block Storage Volumes, consult the Block Storage Volume documentation.
This tutorial builds on the README found in StackPointCloud’s ark-plugin-digitalocean
GitHub repo.
WordPress is one of the most widely used content management systems (CMS) in the world, making up over 33% of the web and providing technical and non-technical users alike a way to build and grow. As WordPress sites scale however, managing large sets of data and assets can sometimes surface latency errors that can affect end user load times.
In this tutorial, learn how to install Spaces, an S3-compatible object storage service that lets you store and serve large amounts of data, to optimize your WordPress site’s speed and performance.
In order to complete this tutorial, you’ll need a DigitalOcean account, and ideally a WordPress installation on Ubuntu 20.04. There are many different starting points for your WordPress installation, including:
Whichever you choose, this tutorial will start with the assumption that you have a DigitalOcean account and WordPress installation configured with an administrative user on Ubuntu 20.04.
From your DigitalOcean control panel, click the Spaces button, in the sidebar on the left:
If you do not have pre existing Spaces created, you’ll see a button prompting you to create a space. You can click that button and follow along. If you have an existing Spaces installation, you’ll see a list of your existing Spaces. In this case, you can use the Create
button on the top right of the screen to create a new Space and follow the rest of this tutorial.
Next, you’ll make a series of choices to customize your Space. Let’s review each choice for your WordPress installation.
Pick the datacenter region closest to your customers general location.
A content delivery network (CDN) is a geographically distributed group of servers optimized to deliver static content to end users. This static content can be almost any sort of data, but CDNs are most commonly used to deliver web pages and their related files, streaming video and audio, and large software packages.
Using a CDN with your WordPress installation can give site visitors the speed and efficiency they’ve come to expect with WordPress. Should you choose to enable the CDN, you’ll be presented with two choices:
Enter a subdomain for your Space. For help on configuring your domain or subdomain with DigitalOcean servers, visit our tutorial, “How to Point to DigitalOcean Nameservers From Common Domain Registrars”. Edge Cache TTL is set at 1 hour by default – feel free to change this to fit your use case.
You can use Static Site Hosting to serve your static html, images and js files. This can be helpful for WordPress installations to reduce content load time — if you decide to use it for your WordPress installation, you can access your Space via http protocol by adding a CNAME record to your DNS host. CDN custom domain cannot be used with static site hosting.
File listing allows for anyone who queries to list the content of your Space. Choose whether or now to enable this setting, keeping in mind that restricting file listing only restricts access to the metadata detailing file contents. The file itself may still be visible.
Add your unique subdomain name and select the Project that you’d like your space to be assigned to:
If everything looks correct in your Spaces configuration, click Create Space. You’ll be taken to your new space, once created:
There are several ways to add connectivity between your WordPress installation and your new Space.
If you intend to use DigitalOcean Spaces for your WordPress media library, follow our tutorial, “How to Store WordPress Assets on DigitalOcean Spaces”, to use the DigitalOcean Spaces Sync WordPress plugin to connect your installation with your new Space.
To extend the speed and performance of asset delivery using your new Space, our tutorial “How to Speed Up WordPress Asset Delivery Using DigitalOcean Spaces CDN” shares tips and suggestions on optimizing the Spaces CDN to deliver media assets efficiently.
Using Spaces with your WordPress installation enables you to provide a secure, fast, user-focused experience for site visitors. To dive deeper into using your new Space to benefit the speed and performance of your WordPress installation, visit the following tutorials:
To learn more about how to optimize your WordPress installation, visit our complete list of resources for Wordpress.
]]>The majority of Let’s Encrypt certificates are issued using HTTP validation, which allows for the installation of certificates on a single server. However, HTTP validation is not always suitable for issuing certificates for use on load-balanced websites, nor can you use this validation to issue wildcard certificates.
DNS validation allows for certificate issuance requests to be verified using DNS records, rather than by serving content over HTTP. This means that certificates can be issued simultaneously for a cluster of web servers running behind a load balancer, or for a system that isn’t directly accessible over the internet.
In this tutorial, you will use the certbot-dns-digitalocean hook for Certbot to issue a Let’s Encrypt certificate using DNS validation via the DigitalOcean API.
You can use the certbot-dns-digitalocean tool to integrate Certbot with DigitalOcean’s DNS management API, allowing the certificate validation records to be automatically configured on-the-fly when you request a certificate.
Another key benefit of certbot-dns-digitalocean is that you can use it to issue certificates for individual servers that may be running behind a load balancer, or are otherwise not directly accessible over HTTP. In these cases, you can’t use traditional HTTP certificate validation, unless you set the validation files on each and every server, which can be inconvenient. The certbot-dns-digitalocean tool is also useful if you want to issue a certificate for a server that isn’t accessible over the internet, for example an internal system or staging environment.
certbot-dns-digitalocean also fully supports wildcard certificates, which can only be issued using DNS validation.
To complete this tutorial, you will need:
An Ubuntu 20.04 server set up by following the Initial Server Setup with Ubuntu 20.04, including a sudo non-root user.
A domain name managed via your DigitalOcean account—that is, for managing DNS records. In this particular example, we will use your_domain
and subdomain.your_domain
, as well as *.your_domain
for a wildcard certificate, however you can adjust this for other domains or subdomains if required.
A DigitalOcean API key (Personal Access Token) with read and write permissions. To create one, visit How to Create a Personal Access Token.
Once you have these ready, log in to your server as your non-root user to begin.
In this step, you will install Certbot, which is a program to issue and manage Let’s Encrypt certificates.
Certbot is available within the official Ubuntu Apt repositories, so you can install it using the default system package manager:
- sudo apt update
- sudo apt install certbot
Once the installation has completed, you can check with the following command:
- certbot --version
This will output something similar to the following:
Outputcertbot 0.40.0
In this step you installed Certbot. Next, you will download and install the acme-dns-certbot hook.
Now that you’ve installed the base Certbot program, you can download and install certbot-dns-digitalocean, which will allow Certbot to operate in DNS validation mode using the DigitalOcean DNS management API.
Like Certbot itself, which you installed in Step 1, the certbot-dns-digitalocean utility is available within Ubuntu’s default repositories. However, the Certbot repository contains a more reliably updated version, so it is always recommended to use this where possible.
Continue by installing the package for certbot-dns-digitalocean:
- sudo apt install python3-certbot-dns-digitalocean
Once the installation has completed, you need to set up a configuration file containing the DigitalOcean API key/Personal Access Token that you generated as part of the prerequisites.
Begin by creating the creds.ini
file in a private location:
- touch ~/certbot-creds.ini
Next, restrict the permissions on the file in order to make sure no other user on your server can read it:
- chmod go-rwx ~/certbot-creds.ini
Finally, open the file using your text editor and add your DigitalOcean access token:
- nano ~/certbot-creds.ini
The content of the file will be as follows:
dns_digitalocean_token = your_digitalocean_access_token
Once done, save and close the file.
Warning: Your DigitalOcean access token grants access to your DigitalOcean account, so you must protect it as you would a password. Do not share it with anyone or check it into a public code repository.
In this step, you downloaded and installed the certbot-dns-digitalocean utility and created a configuration file containing your API credentials.
In this step, you’ll issue a certificate using Certbot and the DigitalOcean API.
To issue your first certificate, run Certbot using the following arguments, making sure to specify the correct path to your credentials file as well as your domains:
- sudo certbot certonly --dns-digitalocean --dns-digitalocean-credentials ~/certbot-creds.ini -d your_domain -d subdomain.your_domain
Note: If you see an unsafe permissions on credentials configuration file
warning, this indicates that the file permissions have not been correctly restricted, thus allowing other users on your server to access your token. Please double-check with the chmod
command in Step 2.
Certbot will take a few seconds to request the certificate; you will then receive a message confirming that it has issued your certificate:
Output...
Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/your_domain/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/your_domain/privkey.pem
...
In the event that the certificate issuance fails, this may be because there wasn’t sufficient time for the DNS changes to propagate. You can optionally increase the DNS propagation delay to give more time for the verification DNS records to propagate and be picked up by Let’s Encrypt. The delay is 10 seconds by default, but you can increase this using the --dns-digitalocean-propagation-seconds
argument:
- sudo certbot certonly --dns-digitalocean --dns-digitalocean-credentials ~/certbot-creds.ini --dns-digitalocean-propagation-seconds 30 -d your_domain -d subdomain.your_domain
Finally, you can also use certbot-dns-digitalocean to issue wildcard certificates for your domain:
- sudo certbot certonly --dns-digitalocean --dns-digitalocean-credentials ~/certbot-creds.ini -d \*.your_domain
Note: In some cases, requesting multiple certificates for the same hostnames in a short time period can cause issuance to begin failing. This is due to rate limits and the DNS time-to-live (TTL) value, which can sometimes cause delays in new DNS changes being propagated.
To mitigate this, you may wish to wait out the duration of the TTL, or consider adjusting the --dns-digitalocean-propagation-seconds
option that was detailed earlier in this step.
In this step, you used Certbot with certbot-dns-digitalocean for the first time and issued your initial certificates.
In this final step, you will renew certificates using Certbot with certbot-dns-digitalocean.
Once your certificates are nearing expiry, Certbot is able to automatically renew them for you:
- sudo certbot renew
The renewal process can run start-to-finish without user interaction. It will also remember the configuration options that you specified during the initial setup.
By default, Certbot will run this as an automatic scheduled system task, meaning that no further maintenance is needed for your certificates. You can check that the scheduled task has been correctly installed by printing out the status of the associated system service, which is certbot.timer
:
- sudo systemctl status certbot.timer
This will output something similar to the following, which shows the loaded task scheduled to run twice per day:
Output● certbot.timer - Run certbot twice daily
Loaded: loaded (/lib/systemd/system/certbot.timer; enabled; vendor preset: enabled)
Active: active (waiting) since Sun 2020-11-22 18:18:40 UTC; 2 weeks 6 days ago
Trigger: Sun 2020-12-13 7:17:57 UTC; 11h left
Nov 22 18:18:40 droplet1 systemd[1]: Started Run certbot twice daily.
However, to test that this is working without having to wait until nearer the expiry date of your certificate(s), you can trigger a ‘dry run’. This will simulate the renewal process without making any actual changes to your configuration.
You can trigger a dry run using the standard renew
command, but with the --dry-run
argument:
- sudo certbot renew --dry-run
This will output something similar to the following, which will provide assurance that the renewal process is functioning correctly:
Output...
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator dns-digitalocean, Installer None
Renewing an existing certificate
Performing the following challenges:
dns-01 challenge for your_domain
dns-01 challenge for subdomain.your_domain
Waiting 10 seconds for DNS changes to propagate
Waiting for verification...
Cleaning up challenges
...
In this final step, you tested the automatic renewal process within Certbot.
In this tutorial, you set up Certbot with certbot-dns-digitalocean to issue certificates using DNS validation with the DigitalOcean DNS management API.
If you’re interested in learning more about certbot-dns-digitalocean, you may wish to review the official documentation for the utility:
Alternatively, if you aren’t using DigitalOcean to manage your DNS records, you may wish to check out How to Acquire a Let’s Encrypt Certificate using DNS Validation with acme-dns-certbot on Ubuntu 18.04, which is a provider-agnostic alternative to certbot-dns-digitalocean.
Finally, if you would like some further technical reading, you could dig into the details of ACME DNS validation by reviewing the relevant section of the official RFC document, which outlines how the process works:
]]>If you’ve built a static website in a local environment, the next step is to decide how to publish it to the web. One way to publish your site is to deploy it as an application through DigitalOcean App Platform, which offers free hosting for three static sites. Deploying applications often requires setting up underlying server infrastructure. App Platform automates this work, allowing you to deploy your static website to the cloud from a GitHub repository.
This tutorial will guide you through all the steps of deploying a static website to the cloud using App Platform, GitHub (a software development platform), and GitHub’s Desktop Application. The instructions here should work for any static website you’ve built in a local environment, including websites created with our tutorial series How To Build a Website With HTML. We will also walk you through how to use our sample HTML website for this tutorial if you don’t have a website ready to deploy, or would just like to test out App Platform. By the end of this tutorial, you should have a published website and an understanding of how to deploy websites to the cloud from a GitHub repository with App Platform.
Note: If you already have a GitHub account and a GitHub repository for your website project, you can skip to Step 6 for instructions on getting started with App Platform.
Deploy your frontend applications from GitHub using DigitalOcean App Platform. Let DigitalOcean focus on scaling your app.
If you don’t already have a GitHub account, you’ll need to register for one so that you can create a GitHub repository for your project. GitHub is a software development platform that allows developers to host, share, and collaborate on coding projects. To create a free account, sign up on GitHub’s homepage.
Once you have confirmed your account, you are ready to proceed to the next step. Remember your login credentials, as you’ll need them in Step 3.
Many developers use the command-line interface (CLI) tool Git to interact with GitHub, but you can also use the GitHub Desktop app if you are not familiar with using your computer’s terminal. (If you’d like to learn more about using the CLI tool Git, you can visit our guide How To Contribute To Open Source: Getting Started With Git. This tutorial will proceed with instructions for using the GitHub Desktop app.
Download the GitHub Desktop app by following the instructions on the GitHub Desktop homepage. Next, open the downloaded application file and complete the installation process as instructed.
After the installation is complete, you are ready to proceed to the next step.
In this step, we’ll use the GitHub Desktop app to create a local repository on your machine for your website project.
First, open the GitHub Desktop app. Click the “Sign in to GitHub.com” blue button :
Follow the prompts to connect the GitHub Desktop app with your GitHub account. Once the Desktop app is connected with your account, a window should appear with options for getting started. Click on the “Create a New Repository on Your Hard Drive” button (third large button from the top):
Next, you will be prompted to fill out the details of your new repository:
In this window, enter the following information:
You can leave the automatically-generated Local Path as it is. This is where GitHub Desktop will store your project on your local machine.
If you’d like to add a file to store your site’s documentation, you can check the option to initialize the repository with a README. In general, it is good practice to create a README for your repositories, but you can also leave this option unchecked for the purpose of the tutorial.
The Git Ignore option allows you to select a template for ignoring certain files. The “License” option allows you to choose an open source license for your work. To learn more about different open source license options, you can visit the Open Source Initiative’s list of Licenses and Standards. If you don’t know what to select for these options, you can keep “none” selected for both options for the purpose of this tutorial.
Click “Create Repository.” The Desktop app should now show the details of your newly-created repository. We’ll go over what these different panels display in the next step.
Once your repository is created, you should be ready to proceed to the next step.
In this step, we’ll copy the files of your website project and place them in the newly-created GitHub repository folder.
Note: If you want to use our sample website to explore App Platform, download the zip file from the GitHub repository by clicking the green “Code” button in the upper right and selecting the option to “Download ZIP”:
Once the ZIP file has finished downloading, unzip the file to access the folder that contains the website files. This folder will serve as your website project’s working folder in the steps below.
First, on your desktop, open your website project’s working folder, or the folder that is currently storing all of your website project’s files and folders. In this example, the working folder is called “html-site”.
Next, find and open the newly created repository folder that you named in Step 3. In this example, the repository folder is called “my-static-site”.
Copy the files from your working folder to your repository folder. To copy the files you can select all of your website files and simultaneously click Right Click
(on Windows) or CTRL + Left Click
(on Mac) and select “Copy X items”. Then, to paste copies of your files into the repository folder, click into the repository folder, click Right Click
(on Windows) or CTRL + Left Click
(on Mac), and select “Paste X items”:
After pasting the files into your repository folder, the GitHub Desktop app should display the files in the “Changes” panel on the left side of the app window:
If you are using a macOS operating system, don’t be alarmed if you see the addition of a .DS_STORE
file in the “Changes” panel. This is an automatically-generated file that stores information about the folder and should not affect your project.
Once your folders are in your local repository folder, you are ready to save your changes to the repository. On GitHub, saved changes are called commits. Each time you commit changes, you must make a comment describing your changes.
To commit your changes, add a comment in the field that says “Summary (required)” and any additional info you’d like to include in the field “Description” in the bottom left corner of the Desktop app:
Then click the blue button “Commit to master” located below the text fields, This action will save your changes to the “main” branch of your project. Note that GitHub previously used the word “master” instead of “main” for the primary branch of users’ repositories. Please see their information on renaming these conventions, and refer to GitHub on their timeline for rolling these changes out. On GitHub, the main or master branch is the definitive branch of the project, which can be copied to work on different versions of the same repository simultaneously. To learn more about branches, you can visit our tutorial How To Use Git Branches or GitHub’s documentation.
Once you have committed your changes to the main branch, your files in the left hand panel will disappear as this panel only displays files that contain uncommitted changes. You should receive a message in the bottom left corner noting that your commit was successful.
In the last step, you committed your changes to the repository on your local machine. This repository, however, has not yet been pushed to your GitHub account. In this step, we will push this commit to your repository on GitHub, which will add your website files to your GitHub repository.
To publish your local repository to your GitHub repository, click on the blue “Publish repository” button:
Once you click the button, a modal will appear asking you to fill out the name and description of your repository. Fill out your details. You may keep the repository private if you wish.
After filling out your details, click the blue “Publish repository” button. Once your files finish uploading, your repository should be available on your account on GitHub. To check, visit the relevant URL, which will have the following format:
https://github.com/your_github_account_name/your_repository_name
Be sure to replace the highlighted text with your account name and repository name. You should receive a webpage that shows your repository’s files:
Now that your website files are hosted on GitHub, we can use them with App Platform. First, however, we’ll need to create a DigitalOcean account.
To create a DigitalOcean account, visit the sign up page and choose among the following options:
If you choose to use an email address and password, you will need to verify your email address using the email automatically sent to you.
Note that you will need to enter a payment method to verify your identity and keep spammers out. You will not be charged. You may see a temporary pre-authorization charge to verify the card, which will be reversed within a week.
Once you have verified your account, you should be able to access App Platform. For complete documentation about signing up for a DigitalOcean account, please visit our guide Sign up for a DigitalOcean Account
You are now ready to proceed to the next step.
In this step we’ll deploy our static website with App Platform.
First, visit the DigitalOcean App Platform portal and click on the blue “Launch Your App” button:
On the next page, you will be prompted to select your GitHub repository. Since you have not yet connected your App Platform account to your GitHub account, you’ll need to click on the “Link Your GitHub Account” button:
You will then be prompted to sign into your GitHub account (if you aren’t already signed in) and select the account that you want to connect to App Platform. Once selected, you will be directed to a page where you can select which repositories to permit App Platform to access. Click the “Only select repositories” button and select the repository that you pushed to your GitHub account in Step 5:
When you are done, click the “Save” button at the bottom of the webpage. You will now be directed back to App Platform, where you should now be able to select your repository in the dropdown menu:
After selecting your repository, click “Next.” You will then be prompted to choose the name, branch, and options for Autodeploy. If the Autodeploy box is checked, any future changes you make to your repository files will be immediately pushed to your live site. Make your selections and click “Next”:
Next, you will be taken to a page where you can configure your App. This page should automatically detect your component type as a “Static Site":
You should not need to make any changes on this page. Scroll down and click the blue button “Next” at the bottom of the page. You will be directed to a new window where you can select the “Starter” plan if you’d like to deploy this site as one of your free three static sites:
Select your desired plan and click the “Launch Your Starter App” button. You will be directed to your app’s admin page. When your app is finished deploying, you will see the “Deployed Successfully!” message:
.
You will also see a link under your app’s name at the top of the page. Click on the link to make sure your site is working properly. You should be directed to a new web page with your published website. If your site is not appearing, go back and check for errors.
Your static site should now be published to the web through App Platform. Anyone with the app link will be able to access your site. If you’d like to add a custom domain to your site, please visit our How To Manage Custom Domains guide in App Platform product documentation.
In this tutorial, you have learned how to deploy a static site using App Platform, with a GitHub account, and the GitHub Desktop app. If you wish to make changes to your website, edit your files on your local machine and commit and push changes to your GitHub repository as instructed in Steps 4 and 5. Once your changes are pushed to your GitHub repository, they should automatically update on your site if you kept the “Automatically deploy on push” option selected in Step 7.
For further information about App Platform, please visit the official App Platform product documentation. Remember, you can host up to three free static sites. If you wish to delete your app, please follow the instructions in the section Destroy an App in the product documentation.
]]>In this tutorial you’ll deploy a containerized Django polls application into a Kubernetes cluster.
Django is a powerful web framework that can help you get your Python application off the ground quickly. It includes several convenient features like an object-relational mapper, user authentication, and a customizable administrative interface for your application. It also includes a caching framework and encourages clean app design through its URL Dispatcher and Template system.
In How to Build a Django and Gunicorn Application with Docker, the Django Tutorial Polls application was modified according to the Twelve-Factor methodology for building scalable, cloud-native web apps. This containerized setup was scaled and secured with an Nginx reverse-proxy and Let’s Encrypt-signed TLS certificates in How To Scale and Secure a Django Application with Docker, Nginx, and Let’s Encrypt. In this final tutorial in the From Containers to Kubernetes with Django series, the modernized Django polls application will be deployed into a Kubernetes cluster.
Kubernetes is a powerful open-source container orchestrator that automates the deployment, scaling and management of containerized applications. Kubernetes objects like ConfigMaps and Secrets allow you to centralize and decouple configuration from your containers, while controllers like Deployments automatically restart failed containers and enable quick scaling of container replicas. TLS encryption is enabled with an Ingress object and the ingress-nginx open-source Ingress Controller. The cert-manager Kubernetes add-on renews and issues certificates using the free Let’s Encrypt certificate authority.
If you’re looking for a managed Kubernetes hosting service, check out our simple, managed Kubernetes service built for growth.
To follow this tutorial, you will need:
kubectl
command-line tool installed on your local machine and configured to connect to your cluster. You can read more about installing kubectl
in the official documentation. If you are using a DigitalOcean Kubernetes cluster, please refer to How to Connect to a DigitalOcean Kubernetes Cluster to learn how to connect to your cluster using kubectl
.your_domain.com
throughout. You can get one for free at Freenom, or use the domain registrar of your choice.A
DNS record with your_domain.com
pointing to the Ingress Load Balancer’s public IP address. If you are using DigitalOcean to manage your domain’s DNS records, consult How to Manage DNS Records to learn how to create A
recordsOnce you have these components set up, you’re ready to begin with this guide.
In this step we’ll clone the application code from GitHub and configure settings like database credentials and object storage keys.
The application code and Dockerfile can be found in the polls-docker
branch of the Django Tutorial Polls App GitHub repository. This repo contains code for the Django documentation’s sample Polls application, which teaches you how to build a polling application from scratch.
The polls-docker
branch contains a Dockerized version of this Polls app. To learn how the Polls app was modified to work effectively in a containerized environment, please see How to Build a Django and Gunicorn Application with Docker.
Begin by using git
to clone the polls-docker
branch of the Django Tutorial Polls App GitHub repository to your local machine:
- git clone --single-branch --branch polls-docker https://github.com/do-community/django-polls.git
Navigate into the django-polls
directory:
- cd django-polls
This directory contains the Django application Python code, a Dockerfile
that Docker will use to build the container image, as well as an env
file that contains a list of environment variables to be passed into the container’s running environment. Inspect the Dockerfile
:
- cat Dockerfile
OutputFROM python:3.7.4-alpine3.10
ADD django-polls/requirements.txt /app/requirements.txt
RUN set -ex \
&& apk add --no-cache --virtual .build-deps postgresql-dev build-base \
&& python -m venv /env \
&& /env/bin/pip install --upgrade pip \
&& /env/bin/pip install --no-cache-dir -r /app/requirements.txt \
&& runDeps="$(scanelf --needed --nobanner --recursive /env \
| awk '{ gsub(/,/, "\nso:", $2); print "so:" $2 }' \
| sort -u \
| xargs -r apk info --installed \
| sort -u)" \
&& apk add --virtual rundeps $runDeps \
&& apk del .build-deps
ADD django-polls /app
WORKDIR /app
ENV VIRTUAL_ENV /env
ENV PATH /env/bin:$PATH
EXPOSE 8000
CMD ["gunicorn", "--bind", ":8000", "--workers", "3", "mysite.wsgi"]
This Dockerfile uses the official Python 3.7.4 Docker image as a base, and installs Django and Gunicorn’s Python package requirements, as defined in the django-polls/requirements.txt
file. It then removes some unnecessary build files, copies the application code into the image, and sets the execution PATH
. Finally, it declares that port 8000
will be used to accept incoming container connections, and runs gunicorn
with 3 workers, listening on port 8000
.
To learn more about each of the steps in this Dockerfile, please see Step 6 of How to Build a Django and Gunicorn Application with Docker.
Now, build the image using docker build
:
- docker build -t polls .
We name the image polls
using the -t
flag and pass in the current directory as a build context, the set of files to reference when constructing the image.
After Docker builds and tags the image, list available images using docker images
:
- docker images
You should see the polls
image listed:
OutputREPOSITORY TAG IMAGE ID CREATED SIZE
polls latest 80ec4f33aae1 2 weeks ago 197MB
python 3.7.4-alpine3.10 f309434dea3a 8 months ago 98.7MB
Before we run the Django container, we need to configure its running environment using the env
file present in the current directory. This file will be passed into the docker run
command used to run the container, and Docker will inject the configured environment variables into the container’s running environment.
Open the env
file with nano
or your favorite editor:
- nano env
DJANGO_SECRET_KEY=
DEBUG=True
DJANGO_ALLOWED_HOSTS=
DATABASE_ENGINE=postgresql_psycopg2
DATABASE_NAME=polls
DATABASE_USERNAME=
DATABASE_PASSWORD=
DATABASE_HOST=
DATABASE_PORT=
STATIC_ACCESS_KEY_ID=
STATIC_SECRET_KEY=
STATIC_BUCKET_NAME=
STATIC_ENDPOINT_URL=
DJANGO_LOGLEVEL=info
Fill in missing values for the following keys:
DJANGO_SECRET_KEY
: Set this to a unique, unpredictable value, as detailed in the Django docs. One method of generating this key is provided in Adjusting the App Settings of the Scalable Django App tutorial.DJANGO_ALLOWED_HOSTS
: This variable secures the app and prevents HTTP Host header attacks. For testing purposes, set this to *
, a wildcard that will match all hosts. In production you should set this to your_domain.com
. To learn more about this Django setting, consult Core Settings from the Django docs.DATABASE_USERNAME
: Set this to the PostgreSQL database user created in the prerequisite steps.DATABASE_NAME
: Set this to polls
or the name of the PostgreSQL database created in the prerequisite steps.DATABASE_PASSWORD
: Set this to the PostgreSQL user password created in the prerequisite steps.DATABASE_HOST
: Set this to your database’s hostname.DATABASE_PORT
: Set this to your database’s port.STATIC_ACCESS_KEY_ID
: Set this to your Space or object storage’s access key.STATIC_SECRET_KEY
: Set this to your Space or object storage’s access key Secret.STATIC_BUCKET_NAME
: Set this to your Space name or object storage bucket.STATIC_ENDPOINT_URL
: Set this to the appropriate Spaces or object storage endpoint URL, for example https://your_space_name.nyc3.digitaloceanspaces.com
if your Space is located in the nyc3
region.Once you’ve finished editing, save and close the file.
In the next step we’ll run the configured container locally and create the database schema. We’ll also upload static assets like stylesheets and images to object storage.
With the container built and configured, use docker run
to override the CMD
set in the Dockerfile and create the database schema using the manage.py makemigrations
and manage.py migrate
commands:
- docker run --env-file env polls sh -c "python manage.py makemigrations && python manage.py migrate"
We run the polls:latest
container image, pass in the environment variable file we just modified, and override the Dockerfile command with sh -c "python manage.py makemigrations && python manage.py migrate"
, which will create the database schema defined by the app code.
If you’re running this for the first time you should see:
OutputNo changes detected
Operations to perform:
Apply all migrations: admin, auth, contenttypes, polls, sessions
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying admin.0003_logentry_add_action_flag_choices... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying auth.0010_alter_group_name_max_length... OK
Applying auth.0011_update_proxy_permissions... OK
Applying polls.0001_initial... OK
Applying sessions.0001_initial... OK
This indicates that the database schema has successfully been created.
If you’re running migrate
a subsequent time, Django will perform a no-op unless the database schema has changed.
Next, we’ll run another instance of the app container and use an interactive shell inside of it to create an administrative user for the Django project.
- docker run -i -t --env-file env polls sh
This will provide you with a shell prompt inside of the running container which you can use to create the Django user:
- python manage.py createsuperuser
Enter a username, email address, and password for your user, and after creating the user, hit CTRL+D
to quit the container and kill it.
Finally, we’ll generate the static files for the app and upload them to the DigitalOcean Space using collectstatic
. Note that this may take a bit of time to complete.
- docker run --env-file env polls sh -c "python manage.py collectstatic --noinput"
After these files are generated and uploaded, you’ll receive the following output.
Output121 static files copied.
We can now run the app:
- docker run --env-file env -p 80:8000 polls
Output[2019-10-17 21:23:36 +0000] [1] [INFO] Starting gunicorn 19.9.0
[2019-10-17 21:23:36 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
[2019-10-17 21:23:36 +0000] [1] [INFO] Using worker: sync
[2019-10-17 21:23:36 +0000] [7] [INFO] Booting worker with pid: 7
[2019-10-17 21:23:36 +0000] [8] [INFO] Booting worker with pid: 8
[2019-10-17 21:23:36 +0000] [9] [INFO] Booting worker with pid: 9
Here, we run the default command defined in the Dockerfile, gunicorn --bind :8000 --workers 3 mysite.wsgi:application
, and expose container port 8000
so that port 80
on your local machine gets mapped to port 8000
of the polls
container.
You should now be able to navigate to the polls
app using your web browser by typing http://localhost
in the URL bar. Since there is no route defined for the /
path, you’ll likely receive a 404 Page Not Found
error, which is expected.
Navigate to http://localhost/polls
to see the Polls app interface:
To view the administrative interface, visit http://localhost/admin
. You should see the Polls app admin authentication window:
Enter the administrative username and password you created with the createsuperuser
command.
After authenticating, you can access the Polls app’s administrative interface:
Note that static assets for the admin
and polls
apps are being delivered directly from object storage. To confirm this, consult Testing Spaces Static File Delivery.
When you are finished exploring, hit CTRL+C
in the terminal window running the Docker container to kill the container.
With the Django app Docker image tested, static assets uploaded to object storage, and database schema configured and ready for use with your app, you’re ready to upload your Django app image to an image registry like Docker Hub.
To roll your app out on Kubernetes, your app image must be uploaded to a registry like Docker Hub. Kubernetes will pull the app image from its repository and then deploy it to your cluster.
You can use a private Docker registry, like DigitalOcean Container Registry, currently free in Early Access, or a public Docker registry like Docker Hub. Docker Hub also allows you to create private Docker repositories. A public repository allows anyone to see and pull the container images, while a private repository allows you to restrict access to you and your team members.
In this tutorial we’ll push the Django image to the public Docker Hub repository created in the prerequisites. You can also push your image to a private repository, but pulling images from a private repository is beyond the scope of this article. To learn more about authenticating Kubernetes with Docker Hub and pulling private images, please see Pull an Image from a Private Registry from the Kubernetes docs.
Begin by logging in to Docker Hub on your local machine:
- docker login
OutputLogin with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username:
Enter your Docker Hub username and password to login.
The Django image currently has the polls:latest
tag. To push it to your Docker Hub repo, re-tag the image with your Docker Hub username and repo name:
- docker tag polls:latest your_dockerhub_username/your_dockerhub_repo_name:latest
Push the image to the repo:
- docker push sammy/sammy-django:latest
In this tutorial the Docker Hub username is sammy and the repo name is sammy-django. You should replace these values with your own Docker Hub username and repo name.
You’ll see some output that updates as image layers are pushed to Docker Hub.
Now that your image is available to Kubernetes on Docker Hub, you can begin rolling it out in your cluster.
When we ran the Django container locally, we passed the env
file into docker run
to inject configuration variables into the runtime environment. On Kubernetes, configuration variables can be injected using ConfigMaps and Secrets.
ConfigMaps should be used to store non-confidential configuration information like app settings, and Secrets should be used for sensitive information like API keys and database credentials. They are both injected into containers in a similar fashion, but Secrets have additional access control and security features like encryption at rest. Secrets also store data in base64, while ConfigMaps store data in plain text.
To begin, create a directory called yaml
in which we’ll store our Kubernetes manifests. Navigate into the directory.
- mkdir yaml
- cd
Open a file called polls-configmap.yaml
in nano
or your preferred text editor:
- nano polls-configmap.yaml
Paste in the following ConfigMap manifest:
apiVersion: v1
kind: ConfigMap
metadata:
name: polls-config
data:
DJANGO_ALLOWED_HOSTS: "*"
STATIC_ENDPOINT_URL: "https://your_space_name.space_region.digitaloceanspaces.com"
STATIC_BUCKET_NAME: "your_space_name"
DJANGO_LOGLEVEL: "info"
DEBUG: "True"
DATABASE_ENGINE: "postgresql_psycopg2"
We’ve extracted the non-sensitive configuration from the env
file modified in Step 1 and pasted it into a ConfigMap manifest. The ConfigMap object is called polls-config
. Copy in the same values entered into the env
file in the previous step.
For testing purposes leave DJANGO_ALLOWED_HOSTS
as *
to disable Host header-based filtering. In a production environment you should set this to your app’s domain.
When you’re done editing the file, save and close it.
Create the ConfigMap in your cluster using kubectl apply
:
- kubectl apply -f polls-configmap.yaml
Outputconfigmap/polls-config created
With the ConfigMap created, we’ll create the Secret used by our app in the next step.
Secret values must be base64-encoded, which means creating Secret objects in your cluster is slightly more involved than creating ConfigMaps. You can repeat the process from the previous step, manually base64-encoding Secret values and pasting them into a manifest file. You can also create them using an environment variable file, kubectl create
, and the --from-env-file
flag, which we’ll do in this step.
We’ll once again use the env
file from Step 1, removing variables inserted into the ConfigMap. Make a copy of the env
file called polls-secrets
in the yaml
directory:
- cp ../env ./polls-secrets
Edit the file in your preferred editor:
- nano polls-secrets
DJANGO_SECRET_KEY=
DEBUG=True
DJANGO_ALLOWED_HOSTS=
DATABASE_ENGINE=postgresql_psycopg2
DATABASE_NAME=polls
DATABASE_USERNAME=
DATABASE_PASSWORD=
DATABASE_HOST=
DATABASE_PORT=
STATIC_ACCESS_KEY_ID=
STATIC_SECRET_KEY=
STATIC_BUCKET_NAME=
STATIC_ENDPOINT_URL=
DJANGO_LOGLEVEL=info
Delete all the variables inserted into the ConfigMap manifest. When you’re done, it should look like this:
DJANGO_SECRET_KEY=your_secret_key
DATABASE_NAME=polls
DATABASE_USERNAME=your_django_db_user
DATABASE_PASSWORD=your_django_db_user_password
DATABASE_HOST=your_db_host
DATABASE_PORT=your_db_port
STATIC_ACCESS_KEY_ID=your_space_access_key
STATIC_SECRET_KEY=your_space_access_key_secret
Be sure to use the same values used in Step 1. When you’re done, save and close the file.
Create the Secret in your cluster using kubectl create secret
:
- kubectl create secret generic polls-secret --from-env-file=poll-secrets
Outputsecret/polls-secret created
Here we create a Secret object called polls-secret
and pass in the secrets file we just created.
You can inspect the Secret using kubectl describe
:
- kubectl describe secret polls-secret
OutputName: polls-secret
Namespace: default
Labels: <none>
Annotations: <none>
Type: Opaque
Data
====
DATABASE_PASSWORD: 8 bytes
DATABASE_PORT: 5 bytes
DATABASE_USERNAME: 5 bytes
DJANGO_SECRET_KEY: 14 bytes
STATIC_ACCESS_KEY_ID: 20 bytes
STATIC_SECRET_KEY: 43 bytes
DATABASE_HOST: 47 bytes
DATABASE_NAME: 5 bytes
At this point you’ve stored your app’s configuration in your Kubernetes cluster using the Secret and ConfigMap object types. We’re now ready to deploy the app into the cluster.
In this step you’ll create a Deployment for your Django app. A Kubernetes Deployment is a controller that can be used to manage stateless applications in your cluster. A controller is a control loop that regulates workloads by scaling them up or down. Controllers also restart and clear out failed containers.
Deployments control one or more Pods, the smallest deployable unit in a Kubernetes cluster. Pods enclose one or more containers. To learn more about the different types of workloads you can launch, please review An Introduction to Kubernetes.
Begin by opening a file called polls-deployment.yaml
in your favorite editor:
- nano polls-deployment.yaml
Paste in the following Deployment manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
name: polls-app
labels:
app: polls
spec:
replicas: 2
selector:
matchLabels:
app: polls
template:
metadata:
labels:
app: polls
spec:
containers:
- image: your_dockerhub_username/app_repo_name:latest
name: polls
envFrom:
- secretRef:
name: polls-secret
- configMapRef:
name: polls-config
ports:
- containerPort: 8000
name: gunicorn
Fill in the appropriate container image name, referencing the Django Polls image you pushed to Docker Hub in Step 2.
Here we define a Kubernetes Deployment called polls-app
and label it with the key-value pair app: polls
. We specify that we’d like to run two replicas of the Pod defined below the template
field.
Using envFrom
with secretRef
and configMapRef
, we specify that all the data from the polls-secret
Secret and polls-config
ConfigMap should be injected into the containers as environment variables. The ConfigMap and Secret keys become the environment variable names.
Finally, we expose containerPort
8000
and name it gunicorn
.
To learn more about configuring Kubernetes Deployments, please consult Deployments from the Kubernetes documentation.
When you’re done editing the file, save and close it.
Create the Deployment in your cluster using kubectl apply -f
:
- kubectl apply -f polls-deployment.yaml
- deployment.apps/polls-app created
Check that the Deployment rolled out correctly using kubectl get
:
- kubectl get deploy polls-app
OutputNAME READY UP-TO-DATE AVAILABLE AGE
polls-app 2/2 2 2 6m38s
If you encounter an error or something isn’t quite working, you can use kubectl describe
to inspect the failed Deployment:
- kubectl describe deploy
You can inspect the two Pods using kubectl get pod
:
- kubectl get pod
OutputNAME READY STATUS RESTARTS AGE
polls-app-847f8ccbf4-2stf7 1/1 Running 0 6m42s
polls-app-847f8ccbf4-tqpwm 1/1 Running 0 6m57s
Two replicas of your Django app are now up and running in the cluster. To access the app, you need to create a Kubernetes Service, which we’ll do next.
In this step, you’ll create a Service for your Django app. A Kubernetes Service is an abstraction that allows you to expose a set of running Pods as a network service. Using a Service you can create a stable endpoint for your app that does not change as Pods die and are recreated.
There are multiple Service types, including ClusterIP Services, which expose the Service on a cluster-internal IP, NodePort Services, which expose the Service on each Node at a static port called the NodePort, and LoadBalancer Services, which provision a cloud load balancer to direct external traffic to the Pods in your cluster (via NodePorts, which it creates automatically). To learn more about these, please see Service from the Kubernetes docs.
In our final setup we’ll use a ClusterIP Service that is exposed using an Ingress and the Ingress Controller set up in the prerequisites for this guide. For now, to test that everything is functioning correctly, we’ll create a temporary NodePort Service to access the Django app.
Begin by creating a file called polls-svc.yaml
using your favorite editor:
- nano polls-svc.yaml
Paste in the following Service manifest:
apiVersion: v1
kind: Service
metadata:
name: polls
labels:
app: polls
spec:
type: NodePort
selector:
app: polls
ports:
- port: 8000
targetPort: 8000
Here we create a NodePort Service called polls
and give it the app: polls
label. We then select backend Pods with the app: polls
label and target their 8000
ports.
When you’re done editing the file, save and close it.
Roll out the Service using kubectl apply
:
- kubectl apply -f polls-svc.yaml
Outputservice/polls created
Confirm that your Service was created using kubectl get svc
:
- kubectl get svc polls
OutputNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
polls NodePort 10.245.197.189 <none> 8000:32654/TCP 59s
This output shows the Service’s cluster-internal IP and NodePort (32654
). To connect to the service, we need the external IP addresses for our cluster nodes:
- kubectl get node -o wide
OutputNAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
pool-7no0qd9e0-364fd Ready <none> 27h v1.18.8 10.118.0.5 203.0.113.1 Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9
pool-7no0qd9e0-364fi Ready <none> 27h v1.18.8 10.118.0.4 203.0.113.2 Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9
pool-7no0qd9e0-364fv Ready <none> 27h v1.18.8 10.118.0.3 203.0.113.3 Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9
In your web browser, visit your Polls app using any Node’s external IP address and the NodePort. Given the output above, the app’s URL would be: http://203.0.113.1:32654/polls
.
You should see the same Polls app interface that you accessed locally in Step 1:
You can repeat the same test using the /admin
route: http://203.0.113.1:32654/admin
. You should see the same Admin interface as before:
At this stage, you’ve rolled out two replicas of the Django Polls app container using a Deployment. You’ve also created a stable network endpoint for these two replicas, and made it externally accessible using a NodePort Service.
The final step in this tutorial is to secure external traffic to your app using HTTPS. To do this we’ll use the ingress-nginx
Ingress Controller installed in the prerequisites, and create an Ingress object to route external traffic to the polls
Kubernetes Service.
Kubernetes Ingresses allow you to flexibly route traffic from outside your Kubernetes cluster to Services inside of your cluster. This is accomplished using Ingress objects, which define rules for routing HTTP and HTTPS traffic to Kubernetes Services, and Ingress Controllers, which implement the rules by load balancing traffic and routing it to the appropriate backend Services.
In the prerequisites you installed the ingress-nginx Ingress Controller and cert-manager TLS certificate automation add-on. You also set up staging and production ClusterIssuers for your domain using the Let’s Encrypt certificate authority, and created an Ingress to test certificate issuance and TLS encryption to two dummy backend Services. Before continuing with this step, you should delete the echo-ingress
Ingress created in the prerequisite tutorial:
- kubectl delete ingress echo-ingress
If you’d like you can also delete the dummy Services and Deployments using kubectl delete svc
and kubectl delete deploy
, but this is not essential to complete this tutorial.
You should also have created a DNS A
record with your_domain.com
pointing to the Ingress Load Balancer’s public IP address. If you’re using a DigitalOcean Load Balancer, you can find this IP address in the Load Balancers section of the Control Panel. If you are also using DigitalOcean to manage your domain’s DNS records, consult How to Manage DNS Records to learn how to create A
records.
If you’re using DigitalOcean Kubernetes, also ensure that you’ve implemented the workaround described in Step 5 of How to Set Up an Nginx Ingress with Cert-Manager on DigitalOcean Kubernetes.
Once you have an A
record pointing to the Ingress Controller Load Balancer, you can create an Ingress for your_domain.com
and the polls
Service.
Open a file called polls-ingress.yaml
using your favorite editor:
- nano polls-ingress.yaml
Paste in the following Ingress manifest:
[polls-ingress.yaml]
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: polls-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt-staging"
spec:
tls:
- hosts:
- your_domain.com
secretName: polls-tls
rules:
- host: your_domain.com
http:
paths:
- backend:
serviceName: polls
servicePort: 8000
We create an Ingress object called polls-ingress
and annotate it to instruct the control plane to use the ingress-nginx Ingress Controller and staging ClusterIssuer. We also enable TLS for your_domain.com
and store the certificate and private key in a secret called polls-tls
. Finally, we define a rule to route traffic for the your_domain.com
host to the polls
Service on port 8000
.
When you’re done editing the file, save and close it.
Create the Ingress in your cluster using kubectl apply
:
- kubectl apply -f polls-ingress.yaml
Outputingress.networking.k8s.io/polls-ingress created
You can use kubectl describe
to track the state of the Ingress you just created:
- kubectl describe ingress polls-ingress
OutputName: polls-ingress
Namespace: default
Address: workaround.your_domain.com
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
polls-tls terminates your_domain.com
Rules:
Host Path Backends
---- ---- --------
your_domain.com
polls:8000 (10.244.0.207:8000,10.244.0.53:8000)
Annotations: cert-manager.io/cluster-issuer: letsencrypt-staging
kubernetes.io/ingress.class: nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 51s nginx-ingress-controller Ingress default/polls-ingress
Normal CreateCertificate 51s cert-manager Successfully created Certificate "polls-tls"
Normal UPDATE 25s nginx-ingress-controller Ingress default/polls-ingress
You can also run a describe
on the polls-tls
Certificate to further confirm its successful creation:
- kubectl describe certificate polls-tls
Output. . .
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 3m33s cert-manager Issuing certificate as Secret does not exist
Normal Generated 3m32s cert-manager Stored new private key in temporary Secret resource "polls-tls-v9lv9"
Normal Requested 3m32s cert-manager Created new CertificateRequest resource "polls-tls-drx9c"
Normal Issuing 2m58s cert-manager The certificate has been successfully issued
This confirms that the TLS certificate was successfully issued and HTTPS encryption is now active for your_domain.com
.
Given that we used the staging ClusterIssuer, most web browsers won’t trust the fake Let’s Encrypt certificate that it issued, so navigating to your_domain.com
will bring you to an error page.
To send a test request, we’ll use wget
from the command-line:
- wget -O - http://your_domain.com/polls
Output. . .
ERROR: cannot verify your_domain.com's certificate, issued by ‘CN=Fake LE Intermediate X1’:
Unable to locally verify the issuer's authority.
To connect to your_domain.com insecurely, use `--no-check-certificate'.
We’ll use the suggested --no-check-certificate
flag to bypass certificate validation:
- wget --no-check-certificate -q -O - http://your_domain.com/polls
Output
<link rel="stylesheet" type="text/css" href="https://your_space.nyc3.digitaloceanspaces.com/django-polls/static/polls/style.css">
<p>No polls are available.</p>
This output shows the HTML for the /polls
interface page, also confirming that the stylesheet is being served from object storage.
Now that you’ve successfully tested certificate issuance using the staging ClusterIssuer, you can modify the Ingress to use the production ClusterIssuer.
Open polls-ingress.yaml
for editing once again:
- nano polls-ingress.yaml
Modify the cluster-issuer
annotation:
[polls-ingress.yaml]
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: polls-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
tls:
- hosts:
- your_domain.com
secretName: polls-tls
rules:
- host: your_domain.com
http:
paths:
- backend:
serviceName: polls
servicePort: 8000
When you’re done, save and close the file. Update the Ingress using kubectl apply
:
- kubectl apply -f polls-ingress.yaml
Outputingress.networking.k8s.io/polls-ingress configured
You can use kubectl describe certificate polls-tls
and kubectl describe ingress polls-ingress
to track the certificate issuance status:
- kubectl describe ingress polls-ingress
Output. . .
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 23m nginx-ingress-controller Ingress default/polls-ingress
Normal CreateCertificate 23m cert-manager Successfully created Certificate "polls-tls"
Normal UPDATE 76s (x2 over 22m) nginx-ingress-controller Ingress default/polls-ingress
Normal UpdateCertificate 76s cert-manager Successfully updated Certificate "polls-tls"
The above output confirms that the new production certificate was successfully issued and stored in the polls-tls
Secret.
Navigate to your_domain.com/polls
in your web browser to confirm that HTTPS encryption is enabled and everything is working as expected. You should see the Polls app interface:
Verify that HTTPS encryption is active in your web browser. If you’re using Google Chrome, arriving at the above page without any errors confirms that everything is working correctly. In addition, you should see a padlock in the URL bar. Clicking on the padlock will allow you to inspect the Let’s Encrypt certificate details.
As a final cleanup task, you can optionally switch the polls
Service type from NodePort to the internal-only ClusterIP type.
Modify polls-svc.yaml
using your editor:
- nano polls-svc.yaml
Change the type
from NodePort
to ClusterIP
:
apiVersion: v1
kind: Service
metadata:
name: polls
labels:
app: polls
spec:
type: ClusterIP
selector:
app: polls
ports:
- port: 8000
targetPort: 8000
When you’re done editing the file, save and close it.
Roll out the changes using kubectl apply
:
- kubectl apply -f polls-svc.yaml --force
Outputservice/polls configured
Confirm that your Service was modified using kubectl get svc
:
- kubectl get svc polls
OutputNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
polls ClusterIP 10.245.203.186 <none> 8000/TCP 22s
This output shows that the Service type is now ClusterIP. The only way to access it is via your domain and the Ingress created in this step.
In this tutorial you deployed a scalable, HTTPS-secured Django app into a Kubernetes cluster. Static content is served directly from object storage, and the number of running Pods can be quickly scaled up or down using the replicas
field in the polls-app
Deployment manifest.
If you’re using a DigitalOcean Space, you can also enable delivery of static assets via a content delivery network and create a custom subdomain for your Space. Please consult Enabling CDN from How to Set Up a Scalable Django App with DigitalOcean Managed Databases and Spaces to learn more.
To review the rest of the series, please visit our From Containers to Kubernetes with Django series page.
]]>Baixe o e-book completo!
Como programar em Go eBook no formato EPUB
Como programar em Go eBook em formato PDF
Este livro foi desenvolvido para apresentar você à escrita de programas com a linguagem de programação Go. Você aprenderá como escrever ferramentas e aplicações úteis que possam ser executadas em servidores remotos, ou sistemas locais Windows, macOS e Linux para desenvolvimento.
Este livro baseia-se na série de tutoriais Como programar em Go encontrada na Comunidade da DigitalOcean. Os tópicos que ele aborda incluem como:
Instale e configure um ambiente de desenvolvimento Go local em sistemas Windows, macOS e Linux
Desenvolva seus programas com lógica condicional, incluindo as declarações switch para controlar o fluxo do programa
Defina suas próprias estruturas de dados e crie interfaces para elas para um código reutilizável
Escreva funções personalizadas para tratamento de erros
Criando e instalando seus programas Go para que eles possam ser executados em sistemas operacionais e arquiteturas de CPU diferentes.
Utilizando flags para passar argumentos para seus programas, para substituir as opções padrão
Cada capítulo pode ser lido independentemente ou usado como referência, ou você pode seguir os capítulos do início ao fim. Sinta-se livre para saltar para o capítulo ou capítulos que melhor se adapte ao seu propósito enquanto você está aprendendo Go com este livro.
Baixe o e-book em formato EPUB ou PDF, seguindo os links abaixo.
Baixe o e-book completo!
Depois que você terminar este livro, se quiser saber mais sobre como construir ferramentas e aplicações com o Go, visite a seção Go da Comunidade da DigitalOcean.
]]>Télécharger le eBook complet
eBook Comment coder dans Go au format EPUB
eBook Comment coder dans Go au format PDF
Cet eBook a été conçu pour vous introduire à l’écriture de programmes avec le langage de programmation Go. Vous apprendez à écrire des outils et des applications utiles exécutables sur des serveurs distants ou des systèmes locaux Windows, macOS et Linux dans le cadre du développement.
Cet eBook s’inspire de la série de tutoriels Comment coder dans Go, que vous trouverez dans DigitalOcean Community. Les sujets traités sont les suivants :
Installer et configurer un environnement de développement local Go sur les systèmes Windows, macOS et Linux
Concevoir vos programmes avec une logique conditionnelle, et notamment les instructions de commutation permettant de contrôler le flux du programme
Définir vos propres structures de données et créer des interfaces pour le code réutilisable
Écrire des fonctions personnalisées de gestion des erreurs
Développer et installer vos programmes Go afin qu’ils puissent fonctionner sur différents systèmes d’exploitation et sur différentes architectures de processeur
Utiliser des balises pour transférer des arguments à vos programmes, réécrire les options par défaut
Chaque chapitre peut être lu indépendamment des autres ou être utilisé à titre de référence. Vous pouvez également suivre l’intégralité des chapitres du début à la fin. N’hésitez pas à passer au(x) chapitre(s) le(s) plus adapté(s) à vos objectifs au cours de votre apprentissage de Go avec cet eBook.
Vous pouvez télécharger le eBook au format EPUB ou PDF en suivant les liens ci-dessous.
Télécharger le eBook complet
Si, une fois que vous aurez atteint la fin de cet eBook, vous souhaitez en savoir plus sur la manière de construire des outils et des applications avec Go, consultez la section Go de DigitalOcean Community.
]]>Das komplette eBook herunterladen!
Codieren in Go eBook im EPUB-Format
Codieren in Go eBook im PDF-Format <$>
Dieses Buch soll Sie mit dem Schreiben von Programmen mit der Programmiersprache Go vertraut machen. Sie werden erfahren, wie Sie nützliche Tools und Anwendungen schreiben können, die sich auf Remoteservern oder in lokalen Windows-, macOS- und Linux-Systemen für die Entwicklung ausführen lassen.
Dieses Buch basiert auf der Tutorial-Reihe Codieren in Go, die Sie in der DigitalOcean Community finden können. Darin werden folgende Themen behandelt:
Installieren und Einrichten einer lokalen Go-Entwicklungsumgebung in Windows-, macOS- und Linux-Systemen
Gestalten Ihrer Programme mit bedingter Logik, einschließlich Switch-Anweisungen zur Steuerung des Programmflusses
Definieren eigener Datenstrukturen und Erstellen von Schnittstellen zu ihnen für wiederverwendbaren Code
Schreiben benutzerdefinierter Fehlerbehandlungsfunktionen
Erstellen und Installieren Ihrer Go-Programme, damit diese in verschiedenen Betriebssystemen und CPU-Architekturen ausgeführt werden können
Verwenden von Flags zur Übergabe von Argumenten an Ihre Programme, um Standardoptionen zu überschreiben
Jedes Kapitel kann für sich allein gelesen oder als Referenz verwendet werden; alternativ können Sie die Kapitel von Anfang bis Ende lesen. Wechseln Sie nach Belieben zwischen Kapiteln, die für Ihre aktuelle Aufgabe relevant sind, während Sie Go mit diesem Buch erlernen.
Sie können das eBook im EPUB- oder PDF-Format unter den nachstehenden Links herunterladen.
<$>[note] [label Das komplette eBook herunterladen!] Codieren in Go eBook im EPUB-Format
Codieren in Go eBook im PDF-Format
Nachdem Sie das Buch gelesen haben, können Sie mehr über die Erstellung von Tools und Anwendungen mit Go erfahren. Besuchen Sie dazu den Bereich Go der DigitalOcean Community.
]]>Загрузите полную электронную книгу!
Электронная книга «Программирование на Go» в формате EPUB
Электронная книга «Программирование на Go» в формате PDF
Эта книга расскажет вам о написании программ на языке программирования Go. Вы узнаете, как писать полезные инструменты и приложения, которые можно запускать на удаленных серверах или локальных компьютерах Windows, macOS и Linux для разработки.
Эта книга основана на серии обучающих модулей «Программирование на Go» сообщества DigitalOcean. В ней освещены следующие темы:
Установка и настройка локальной среды разработки Go на системах под управлением Windows, macOS и Linux
Проектирование программ с условной логикой, в том числе операторами switch для управления ходом выполнения программы
Определение собственных структур данных и создание интерфейсов для их многоразового использования в коде
Написание собственных функций обработки ошибок
Сборка и установка программ Go для их запуска на разных операционных системах и процессорных архитектурах
Использование флагов для передачи аргументов в программы для переопределения параметров по умолчанию
Каждую главу можно читать отдельно или использовать как справочник, также можно читать всю книгу от начала до конца. Изучая Go с помощью этой книги, вы можете переходить к любым интересующим вас в данный момент главам.
Вы можете загрузить электронную книгу в формате EPUB или PDF по одной из следующих ссылок.
Загрузите полную электронную книгу!
Если после прочтения этой книги вы захотите узнать больше о создании инструментов и приложений на Go, посетите раздел Go сообщества DigitalOcean.
]]>Descargue todo el EBook
Libro electrónico Cómo crear código en Go en formato EPUB
Libro electrónico Cómo crear código en Go en formato PDF
Este libro está diseñado como una introducción a la escritura de programas con el lenguaje de programación Go. Aprenderá cómo escribir herramientas y aplicaciones útiles para el desarrollo que puedan ejecutarse en servidores remotos o en sistemas locales de Windows, macOS y Linux.
Este libro toma como base la serie de tutoriales publicados por la comunidad de DigitalOcean sobre Cómo crear código en Go. Los temas que abarca incluyen:
Instalar y establecer un entorno local de desarrollo de Go en sistemas Windows, macOS y Linux
Diseñar los programas con lógica condicional, incluidas las instrucciones de conmutación para controlar el flujo del programa
Definir estructuras de datos propias y crear interfaces a dichas estructuras para el código reutilizable
Escribir funciones personalizadas para la gestión de errores
Compilar e instalar los programas de Go para que puedan ejecutarse en diferentes sistemas operativos y arquitecturas de CPU
Utilizar indicadores para pasar argumentos a los programas, a fin de anular las opciones predeterminadas
Cada capítulo puede leerse de manera independiente o utilizarse como referencia, o bien puede leerlos en orden. A medida que aprenda a utilizar Go con este libro, no dude en ir directamente al capítulo o a los capítulos que mejor se adapten a su propósito.
Puede descargar el libro electrónico en formato EPUB o PDF siguiendo los enlaces que se muestran a continuación.
Descargue todo el EBook
Una vez que haya terminado de leer el libro, si quiere obtener más información sobre cómo crear herramientas y aplicaciones con Go, visite la sección sobre Go de la comunidad de DigitalOcean.
]]>Virtual Private Cloud (VPC) is now active on DigitalOcean and we would like to give you a tour to understand how it works and how you can organize your networking on DigitalOcean.
This Tech Talk covers a brief overview of all the network features you can use and show a few examples of how to use them.
Rafael Rosa is a Senior Product Manager at DigitalOcean.
]]>Download the Complete eBook!
How To Code in Go eBook in EPUB format
How To Code in Go eBook in PDF format
This book is designed to introduce you to writing programs with the Go programming language. You’ll learn how to write useful tools and applications that can run on remote servers, or local Windows, macOS, and Linux systems for development.
This book is based on the How To Code in Go tutorial series found on DigitalOcean Community. The topics that it covers include how to:
Install and set up a local Go development environment on Windows, macOS, and Linux systems
Design your programs with conditional logic, including switch statements to control program flow
Define your own data structures and create interfaces to them for reusable code
Write custom error handling functions
Building and installing your Go programs so that they can run on different operating systems and different CPU architectures
Using flags to pass arguments to your programs, to override default options
Each chapter can be read on its own or used as a reference, or you can follow the chapters from beginning to end. Feel free to jump to the chapter or chapters that best suits your purpose as you are learning Go with this book.
You can download the eBook in either the EPUB or PDF format by following the links below.
Download the Complete eBook!
After you’re finished this book, if you’d like to learn more about how to build tools and applications with Go, visit the DigitalOcean Community’s Go section.
]]>Laravel es un marco PHP de código abierto que ofrece un conjunto de herramientas y recursos para crear aplicaciones PHP modernas. Con un ecosistema completo que aprovecha sus funciones integradas, la popularidad de Laravel ha aumentado rápidamente en los últimos años, y muchos desarrolladores lo han adoptado como su marco preferido para un proceso de desarrollo agilizado.
En esta guía, instalará y configurará una nueva aplicación de Laravel en un servidor de Ubuntu 20.04, usando Composer para descargar y administrar las dependencias del marco y Nginx para presentar la aplicación. Cuando termine, tendrá una aplicación de demostración funcional de Laravel que extrae contenido de una base de datos MySQL 8.
Para completar esta guía, primero deberá realizar las siguientes tareas en su servidor Ubuntu 20.04:
sudo
y habilite ufw
. Para configurar esto, puede seguir nuestro tutorial Configuración inicial para servidores con Ubuntu 20.04.Antes de que pueda instalar Laravel, deberá instalar algunos módulos PHP que el marco necesita. Usaremos apt
para instalar los módulos PHP php-mbstring
, php-xml
y php-bcmath
. Estas extensiones PHP ofrecen un soporte adicional para tratar con la codificación de caracteres, el XML y las matemáticas de precisión.
Si esta es la primera vez que utliza apt
en esta sesión, primero debería ejecutar el comando update
para actualizar el caché del administrador de paquetes:
- sudo apt update
Ahora puede instalar los paquetes necesarios con:
- sudo apt install php-mbstring php-xml php-bcmath
Su sistema está ahora listo para ejecutar la instalación de Laravel a través de Composer, pero antes de hacerlo, necesitará una base de datos para su aplicación.
Para demostrar la instalación y uso básicos de Laravel, crearemos una aplicación travel list para mostrar una lista de lugares a los que un usuario le gustaría viajar, y una lista de lugares que ya ha visitado. Esto puede almacenarse en una tabla places con un campo para las ubicaciones al que llamaremos name y otro campo para marcarlas como visited (visitadas) o not visited (no visitadas, al que llamaremos visited. Además, incluiremos un campo id para identificar cada entrada de forma única.
Para conectar con la base de datos desde la aplicación Laravel, crearemos un usuario MySQL dedicado, y concederemos a este usuario privilegios completos sobre la base de datos travellist
.
En el momento de escribir este artículo, la biblioteca PHP de MySQL nativa mysqlnd
no admite caching_sha2_authentication
, el método de autenticación predeterminado para MySQL 8. Necesitaremos configurar el usuario de nuestra base de datos con el método de autenticación mysql_native_password
para poder conectar con la base MySQL desde PHP.
Para comenzar, inicie sesión en la consola de MySQL como usuario root de la base de datos con:
- sudo mysql
Para crear una base de datos nueva, ejecute el siguiente comando desde su consola de MySQL:
- CREATE DATABASE travellist;
Ahora puede crear un nuevo usuario y concederle privilegios completos sobre la base de datos personalizada que acaba de crear. En este ejemplo, estamos creando un usuario llamado travellist_user con la contraseña password
, aunque debería cambiarla a una contraseña segura:
- CREATE USER 'travellist_user'@'%' IDENTIFIED WITH mysql_native_password BY 'password';
Ahora, debemos darle permiso a este usuario a la base de datos travellist
:
- GRANT ALL ON travellist.* TO 'travellist_user'@'%';
Esto proporcionará al usuario travellist_user privilegios completos sobre la base de datos travellist
, al mismo tiempo que se impide que este usuario cree o modifique otras bases de datos en su servidor.
Después de esto, cierre el shell de MySQL:
- exit
Puede probar si el usuario nuevo tiene los permisos adecuados al volver a iniciar sesión en la consola de MySQL, esta vez, con las credenciales de usuario personalizadas:
- mysql -u travellist_user -p
Observe el indicador -p
en este comando, que le solicitará la contraseña que utilizó cuando creó el usuario travellist_user. Tras iniciar sesión en la consola MySQL, confirme que tiene acceso a la base de datos travellist
:
- SHOW DATABASES;
Con esto se generará el siguiente resultado:
Output+--------------------+
| Database |
+--------------------+
| information_schema |
| travellist |
+--------------------+
2 rows in set (0.01 sec)
A continuación, cree una tabla llamada places
en la base de datos travellist
. Desde la consola de MySQL, ejecute la siguiente instrucción:
- CREATE TABLE travellist.places (
- id INT AUTO_INCREMENT,
- name VARCHAR(255),
- visited BOOLEAN,
- PRIMARY KEY(id)
- );
Ahora, complete la tabla places
con algunos datos de ejemplo:
- INSERT INTO travellist.places (name, visited)
- VALUES ("Tokyo", false),
- ("Budapest", true),
- ("Nairobi", false),
- ("Berlin", true),
- ("Lisbon", true),
- ("Denver", false),
- ("Moscow", false),
- ("Olso", false),
- ("Rio", true),
- ("Cincinnati", false),
- ("Helsinki", false);
Para confirmar que los datos se guardaron correctamente en su tabla, ejecute lo siguiente:
- SELECT * FROM travellist.places;
Verá un resultado similar a este:
Output+----+-----------+---------+
| id | name | visited |
+----+-----------+---------+
| 1 | Tokyo | 0 |
| 2 | Budapest | 1 |
| 3 | Nairobi | 0 |
| 4 | Berlin | 1 |
| 5 | Lisbon | 1 |
| 6 | Denver | 0 |
| 7 | Moscow | 0 |
| 8 | Oslo | 0 |
| 9 | Rio | 1 |
| 10 | Cincinnati| 0 |
| 11 | Helsinki | 0 |
+----+-----------+---------+
11 rows in set (0.00 sec)
Después de confirmar que haya datos válidos en su tabla de prueba, puede cerrar la consola de MySQL:
- exit
Ahora está listo para crear la aplicación y configurarla para que se conecte con la nueva base de datos.
Ahora creará una nueva aplicación Laravel usando el comando composer create-project
. Este comando de Composer normalmente se usa para arrancar nuevas aplicaciones en base a marcos y sistemas de gestión de contenido existentes.
A lo largo de esta guía, usaremos travellist
como aplicación de ejemplo, pero puede cambiar esto a otra cosa. La aplicación travellist
mostrará una lista de ubicaciones extraídas de un servidor MySQL local, que pretende demostrar la configuración básica de Laravel y confirmar que puede conectar con la base de datos.
Primero, vaya al directorio de inicio de su usuario:
- cd ~
El siguiente comando creará un nuevo directorio travellist
que contiene una aplicación Laravel barebones basada en los ajustes predeterminados:
- composer create-project --prefer-dist laravel/laravel travellist
Verá un resultado similar a este:
OutputInstalling laravel/laravel (v5.8.17)
- Installing laravel/laravel (v5.8.17): Downloading (100%)
Created project in travellist
> @php -r "file_exists('.env') || copy('.env.example', '.env');"
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 80 installs, 0 updates, 0 removals
- Installing symfony/polyfill-ctype (v1.11.0): Downloading (100%)
- Installing phpoption/phpoption (1.5.0): Downloading (100%)
- Installing vlucas/phpdotenv (v3.4.0): Downloading (100%)
- Installing symfony/css-selector (v4.3.2): Downloading (100%)
...
Cuando finalice la instalación, acceda al directorio de la aplicación y ejecute el comando artisan
de Laravel para verificar que todos los componentes se instalaron correctamente.
- cd travellist
- php artisan
Verá un resultado similar a este:
OutputLaravel Framework 7.11.0
Usage:
command [options] [arguments]
Options:
-h, --help Display this help message
-q, --quiet Do not output any message
-V, --version Display this application version
--ansi Force ANSI output
--no-ansi Disable ANSI output
-n, --no-interaction Do not ask any interactive question
--env[=ENV] The environment the command should run under
-v|vv|vvv, --verbose Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug
...
Este resultado confirma que los archivos de la aplicación están listos y que las herramientas de la línea de comando de Laravel funcionan como se espera. Sin embargo, aún debemos configurar la aplicación para configurar la base de datos y algunos detalles más.
Los archivos de configuración de Laravel se encuentran en un directorio llamado config
, dentro del directorio root de la aplicación. Además, cuando instala Laravel con Composer, crea un archivo de entorno. Este archivo contiene ajustes que son específicos para el entorno actual en el que se ejecuta la aplicación, y tendrá precedencia sobre los valores establecidos en los archivos de configuración regulares ubicados en el directorio config
. Para cada instalación en un nuevo entorno se requiere un archivo de entorno personalizado a fin de definir elementos como las configuraciones de conexión de bases de datos, las opciones de depuración y las URL de aplicación, entre otros elementos que pueden variar dependiendo del entorno en el que se ejecute la aplicación.
Advertencia: En el archivo de configuración del entorno se encuentra información confidencial sobre su servidor, incluidas las credenciales de bases de datos y las claves de seguridad. Por ese motivo, nunca debe compartir públicamente este archivo.
Ahora editaremos el archivo .env
para personalizar las opciones de configuración para el entorno actual de la aplicación.
Abra el archivo .env
usando el editor de línea de comandos que prefiera. Aquí usaremos nano
:
- nano .env
Aunque existen muchas variables de configuración en este archivo, no es necesario configurarlas todas ahora. La siguiente lista contiene una descripción general de las variables que requieren atención inmediata:
APP_NAME
: nombre de la aplicación, usado para notificaciones y mensajes.APP_ENV
: entorno actual de la aplicación.APP_KEY
: se utiliza para generar salts y hashes; esta clave única se crea automáticamente cuando se instala Laravel a través de Composer, de forma que no es necesario cambiarla.APP_DEBUG
: decida si desea mostrar o no la información de depuración en el lado del cliente.APP_URL
: URL base para la aplicación, usada para generar enlaces de la aplicación.DB_DATABASE
: nombre de la base de datos.DB_USERNAME
: nombre de usuario para conectar con la base de datos.DB_PASSWORD
: contraseña para conectar con la base de datos.Por defecto, estos valores se configuran para un entorno de desarrollo local que utiliza Homestead, un cuadro de Vagrant pre-empaquetado proporcionada por Laravel. Cambiaremos estos valores para que reflejen los ajustes actuales del entorno de nuestra aplicación de ejemplo.
En el caso de instalar Laravel en un entorno de desarrollo o de prueba, puede dejar la opción APP_DEBUG
habilitada, ya que esto le proporcionará información de depuración importante mientras realiza pruebas a la aplicación desde un navegador. La variable APP_ENV
debería establecerse a desarrollo
o prueba
en este caso.
En el caso de que esté instalando Laravel en un entorno de producción, debería deshabilitar la opción APP_DEBUG
, porque muestra al usuario final información sensible sobre su aplicación. La opción APP_ENV
en este caso debería configurarse a producción
.
El siguiente archivo .env
configura nuestra aplicación de ejemplo para desarrollo:
Nota: La variable APP_KEY
contiene una clave única que se generó automáticamente cuando instaló Laravel a través de Composer. No es necesario que cambie este valor. Si desea generar una nueva clave segura, puede usar el comando php artisan key:generate
.
APP_NAME=TravelList
APP_ENV=development
APP_KEY=APPLICATION_UNIQUE_KEY_DONT_COPY
APP_DEBUG=true
APP_URL=http://domain_or_IP
LOG_CHANNEL=stack
DB_CONNECTION=mysql
DB_HOST=127.0.0.1
DB_PORT=3306
DB_DATABASE=travellist
DB_USERNAME=travellist_user
DB_PASSWORD=password
...
Ajuste las variables según corresponda. Cuando termine de editar, guarde y cierre el archivo para conservar sus cambios. Si utiliza nano
, puede hacer esto escribiendo CTRL+X
y, luego, Y
y ENTER
para confirmar.
Su aplicación Laravel ahora está configurada, pero aún debemos configurar el servidor web para poder acceder a ella desde un navegador. En el siguiente paso, configuraremos Nginx para presentar su aplicación Laravel.
Hemos instalado Laravel en una carpeta local del directorio de inicio de su usuario remoto, y, aunque esto funciona bien para los entornos de desarrollo local, no es una práctica recomendada para los servidores web que están abiertos al Internet público. Moveremos la carpeta de la aplicación a /var/www
, que es la ubicación habitual para las aplicaciones web que se ejecutan en Nginx.
Primero, utilice el comando mv
para mover la carpeta de la aplicación con todo su contenido a /var/www/travellist
:
- sudo mv ~/travellist /var/www/travellist
Ahora, deberá proporcionar al usuario del servidor web acceso de escritura para las carpetas storage
y cache
, donde Laravel guarda los archivos generados por la aplicación.
- sudo chown -R www-data.www-data /var/www/travellist/storage
- sudo chown -R www-data.www-data /var/www/travellist/bootstrap/cache
Los archivos de la aplicación ahora están en orden, pero aún necesitamos configurar Nginx para presentar el contenido. Para hacer esto, crearemos un nuevo archivo de configuración del host virtual en /etc/nginx/sites-available
:
- sudo nano /etc/nginx/sites-available/travellist
El siguiente archivo de configuración contiene los ajustes recomendados para las aplicaciones de Laravel en Nginx:
server {
listen 80;
server_name server_domain_or_IP;
root /var/www/travellist/public;
add_header X-Frame-Options "SAMEORIGIN";
add_header X-XSS-Protection "1; mode=block";
add_header X-Content-Type-Options "nosniff";
index index.html index.htm index.php;
charset utf-8;
location / {
try_files $uri $uri/ /index.php?$query_string;
}
location = /favicon.ico { access_log off; log_not_found off; }
location = /robots.txt { access_log off; log_not_found off; }
error_page 404 /index.php;
location ~ \.php$ {
fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
include fastcgi_params;
}
location ~ /\.(?!well-known).* {
deny all;
}
}
Copie este contenido a su archivo /etc/nginx/sites-available/travellist
y, si es necesario, ajuste los valores resaltados para que se adapten a su propia configuración. Guarde y cierre el archivo cuando finalice la edición.
Para activar el nuevo archivo de configuración del host virtual, cree un enlace simbólico a travellist
en sites-enabled
:
- sudo ln -s /etc/nginx/sites-available/travellist /etc/nginx/sites-enabled/
Nota: Si tiene otro archivo de host virtual que fue configurado previamente para el mismo server_name
usado en el host virtual travellist
, es posible que deba desactivar la configuración antigua eliminando el enlace simbólico correspondiente dentro de /etc/nginx/sites-enabled/
.
Para confirmar que la configuración no contiene errores de sintaxis, puede usar:
- sudo nginx -t
Debería ver el siguiente resultado:
- Outputnginx: the configuration file /etc/nginx/nginx.conf syntax is ok
- nginx: configuration file /etc/nginx/nginx.conf test is successful
Para aplicar los cambios, vuelva a cargar Nginx con:
- sudo systemctl reload nginx
Ahora vaya a su navegador y acceda a la aplicación usando el nombre de dominio del servidor o la dirección IP, como se define en la directiva server_name
en su archivo de configuración:
http://server_domain_or_IP
Verá una página como la siguiente:
Eso confirma que su servidor Nginx está configurado correctamente para presentar Laravel. Desde este punto, puede comenzar a crear su aplicación sobre el esqueleto proporcionado por la instalación predeterminada.
En el siguiente paso, modificaremos la ruta principal de la aplicación para que consulte datos usando la fachada DB
de Laravel.
Asumiendo que ha seguido todos los pasos de esta guía, debería tener una aplicación Laravel funcionando y una tabla de base de datos llamada places
que contiene algunos datos de ejemplo.
Ahora editaremos la ruta principal de la aplicación para que consulte la base de datos y devuelva contenido a la vista de la aplicación.
Abra el archivo de la ruta principal, routes/web.php
:
- nano routes/web.php
Este archivo viene por defecto con el siguiente contenido:
<?php
/*
|--------------------------------------------------------------------------
| Web Routes
|--------------------------------------------------------------------------
|
| Here is where you can register web routes for your application. These
| routes are loaded by the RouteServiceProvider within a group which
| contains the "web" middleware group. Now create something great!
|
*/
Route::get('/', function () {
return view('welcome');
});
Las rutas se definen con este archivo usando el método estático Route::get
, que recibe una ruta y una función de invocación como argumentos.
El siguiente código sustituye la función de invocación de la ruta principal. Realiza dos consultas a la base de datos usando el indicador visited
para filtrar los resultados. Devuelve los resultados a una vista llamada travellist
, que vamos a crear a continuación. Copie este contenido a su archivo routes/web.php
, sustituyendo el código que ya está ahí:
<?php
use Illuminate\Support\Facades\DB;
Route::get('/', function () {
$visited = DB::select('select * from places where visited = ?', [1]);
$togo = DB::select('select * from places where visited = ?', [0]);
return view('travellist', ['visited' => $visited, 'togo' => $togo ] );
});
Guarde y cierre el archivo cuando finalice la edición. Ahora crearemos la vista que mostrará los resultados de la base de datos al usuario. Cree un nuevo archivo de vista dentro de resources/views
:
- nano resources/views/travellist.blade.php
La siguiente plantilla crea dos listas de lugares basadas en las variables visited
y togo
. Copie este contenido a su nuevo archivo de vista:
<html>
<head>
<title>Travel List</title>
</head>
<body>
<h1>My Travel Bucket List</h1>
<h2>Places I'd Like to Visit</h2>
<ul>
@foreach ($togo as $newplace)
<li>{{ $newplace->name }}</li>
@endforeach
</ul>
<h2>Places I've Already Been To</h2>
<ul>
@foreach ($visited as $place)
<li>{{ $place->name }}</li>
@endforeach
</ul>
</body>
</html>
Guarde y cierre el archivo cuando termine. Ahora vaya a su navegador y vuelva a cargar la aplicación. Verá una página como la siguiente:
Ahora tiene una aplicación de Laravel funcional que extrae contenido de una base de datos MySQL.
En este tutorial, ha configurado una nueva aplicación de Laravel sobre una pila LEMP (Linux, Nginx, MySQL y PHP) que se ejecuta en un servidor Ubuntu 20.04. También ha personalizado su ruta predeterminada para consultar el contenido de la base de datos y mostrar los resultados en una vista personalizada.
A partir de aquí, puede crear nuevas rutas y vistas para cualquier página adicional que necesite su aplicación. Consulte la documentación oficial de Laravel para obtener más información sobre las rutas, las vistas y la compatibilidad con bases de datos. Si está implementando para producción, debería consultar también la sección de optimización para ver diferentes formas en las cuales puede mejorar el rendimiento de su aplicación.
]]>Usually, Linux-based servers don’t come with a graphical user interface (GUI) preinstalled. Whenever you want to run GUI applications on your instance, the typical solution is to employ Virtual Network Computing (VNC). Unfortunately, VNC is notoriously sluggish, insecure by default, and requires a lot of manual configuration to get up and running.
By contrast, X2Go provides a working “cloud desktop,” complete with all the advantages of an always-online, remotely-accessible, and easily-scalable computing system with a very fast network. It is also more responsive and more secure than many VNC solutions. In this tutorial, you’ll use X2Go to create a Debian 10 XFCE desktop environment that you can access remotely. This “cloud desktop” will include the exact same utilities that you would obtain had you installed Debian 10 and the XFCE environment on your personal computer.
The setup described in this tutorial is useful when:
Before starting this tutorial, you’ll need:
A Debian 10 x64 instance with 2GB of RAM or more. 2GB is minimal and it’s recommended to use a server with 4GB or more if you have memory-hungry applications you plan to run. You can use a DigitalOcean Droplet if you like.
A user with sudo
privileges and an SSH key. Follow this guide to get started: Initial Server Setup with Debian 10. Make sure you complete Step 4 and configure your firewall to restrict all connections except for OpenSSH.
With your server up and your firewall configured, you are now ready to install the graphical environment for the X2Go server.
In this tutorial, you are installing XFCE as the desktop environment. XFCE doesn’t use fancy graphical effects like compositing, which makes it more compatible with X2Go and allows it to optimize screen updates. For reference, LXDE
and MATE
(with compositing disabled) also work fine but you’ll have to change the command in this tutorial where you install the desktop environment. For example, instead of sudo apt-get install xfce4
you would type sudo apt-get install lxde
.
There are two ways to easily install XFCE, but you only need to choose one—either the Minimal Desktop Environment or the Full Desktop Environment.
Full Desktop Environment: Recommended. If you don’t want to handpick every component you need and would rather have a default set of packages, like a word processor, web browser, email client, and other accessories pre-installed, then you can choose task-xfce-desktop
.
Install and configure the Full Desktop Environment. This is similar to what you would get if you installed Debian from a bootable DVD/USB memory stick to your local PC:
- sudo apt-get install task-xfce-desktop
Minimal Desktop Environment: Alternately, if you want to install a small, core set of packages and then build on top of them by manually adding whatever you need afterward, you can use the xfce4
metapackage.
A metapackage doesn’t contain software of its own, it just depends on other packages to be installed, allowing for an entire collection of packages to be installed at once without having to type each package name individually at the command line.
Install xfce4
and all of the additional dependencies needed to support it:
- sudo apt-get install xfce4
Now that your graphical environment is installed, you will establish a way to view it remotely.
X2Go comes with two main components: the server, which starts and manages the graphical session on the remote machine, and the client, which you install on your local computer to view and control the remote desktop or application.
In previous versions of Debian, x2goserver
wasn’t included in the default repositories, so you’d have to follow steps like these to get the software package. We’re leaving the link here, just for reference, in case the package gets dropped in future versions of Debian. Fortunately, Debian 10, codenamed Buster, includes the package you need in its default repositories, so the installation is a bit easier.
To install X2Go on your server, type the following command:
- sudo apt-get install x2goserver x2goserver-xsession
At this point, no further setup is required on your server. However, keep in mind that since SSH password authentication is disabled, for increased security, you’ll need to have your SSH private key available on any machine that you want to log in from. This will be the case if you followed the recommendation of setting up SSH keys in the Initial Server Setup with Debian 10 tutorial listed in the prerequisites. If you didn’t, you can still log in with a password, but make sure you choose a strong one.
Remember that if you run out of RAM, some applications might be abruptly terminated by the Linux kernel, and then you could lose work in progress. If you are using a DigitalOcean Droplet and you notice that your programs require more RAM, you can temporarily power off your Droplet and upgrade (resize) to one with more memory.
You are now done setting up the server and you can type exit
or close the terminal window. The rest of the steps will focus on configuring the client on your local machine.
X2Go is ready to use out of the box. If you’re using Windows or Mac on your local machine, you can download the X2Go client software here. If you’re using Debian or Ubuntu you can install the X2Go client with this command on your local machine:
- sudo apt-get install x2goclient
After downloading the software you are ready to install it. Open the installer and select your preferred language. Now agree to the license and let the wizard guide you through the remaining steps. Normally, there shouldn’t be any reason to change the pre-filled, default values in these steps.
X2Go works well out of the box but it is also highly customizable. If you’d like additional information, visit X2Go’s official documentation.
Now that the desktop client is installed, you can configure its settings and connect to the X2Go server to use your remote XFCE desktop.
When you first open the X2Go client, the following window will appear. If it doesn’t, click Session in the top-left menu and then select New session ….
In the Session name field, enter something to help differentiate between servers. This can be particularly useful if you plan on connecting to multiple machines.
Enter your server’s IP address or hostname in the Host field under Server.
Enter the username you used for your SSH connection in the Login field.
And, since you installed XFCE in Step Two, choose XFCE
as your Session type.
Finally, because you are connecting to the server with SSH keys, click the folder icon next to Use RSA/DSA key for ssh connection and browse to your private key. If you didn’t opt to use the more secure SSH keys, just leave this empty and the X2Go client will ask for a password each time you log in.
The rest of the default settings will suffice for now, but as you get more familiar with the software, you can fine tune the client based on your individual preferences.
After pressing the OK button, you can start your graphical session by clicking the white box that includes your session’s name on the top-right side of the box.
If you are running OS X on your local machine, you may be prompted to install xquartz, which is required to run X11. If so, follow the instructions to install it now.
In a few seconds, your remote desktop will be displayed and you can start interacting with it. During your first login, XFCE will ask if you want to Use default config or One empty panel. The first option will create a rectangular panel docked at the bottom of the screen, containing a few useful application shortcuts (e.g. a file manager, a terminal emulator, a browser, etc.). This option will also add a top panel to the desktop that includes utilities like an application launcher, a clock, a shutdown menu, and more.
Unless you’re already familiar with XFCE, opting for an empty panel can be more complicated because you’ll be starting from scratch. There will be no taskbar, no clock, and no pre-configured start menu; it will be up to you to add everything to an empty panel on your own.
Additionally, on Windows and Linux-based operating systems, there are a few useful keyboard shortcuts you can use for a better experience:
CTRL+ALT+F
will toggle full-screen mode on and off. Working in full-screen mode can feel more like a local desktop experience. Plus, other keyboard shortcuts will be grabbed by the remote OS instead of the local one.
CTRL+ALT+M
will minimize the remote view, even if you are in full-screen mode
CTRL+ALT+T
will disconnect from the session but leave the GUI running on the server. It’s just a quick way of disconnecting without logging off or closing applications on the server. The same will happen if you click the window’s close button.
Lastly, there are two ways you can end the remote session and close all of the graphical programs running in it. You can log off remotely from XFCE’s start menu, or you can click the button marked with a circle and a small line (like a power/standby icon) in the bottom-right corner of the main portion of the X2Go screen.
The first method is cleaner but may leave programs like session managing software running. The second method will close everything but may do so forcefully if a process can’t cleanly exit. In either case, be sure to save your work before proceeding.
And lastly, although it’s not required, let’s go into XFCE’s control panel, navigate to power settings and disable all standby features. If you don’t use the desktop for a while, XFCE will try to trigger a standby. Although the server normally won’t do anything when it receives this request, it’s better to avoid any unforeseen behaviors altogether.
By going to the Applications menu, then Settings, you can open Power Manager.
In this application, navigate to the System tab and drag the When inactive for slider all the way to the left. You should now see the text Never under this slider, which means XFCE won’t try to put the system in standby after it’s inactive for a while.
You have now successfully accessed and configured your remote desktop.
In this tutorial you used X2Go to create a robust and remote GUI-environment for the Debian operating system. Here are a few additional ideas about how to use this desktop. You could centralize your development work by creating a git repository, installing an IDE/code editor like NetBeans or Eclipse, or configuring a web server for testing web applications. You could also enhance your remote desktop with a good backup scheme to make sure that your work environment and important data are safely preserved in case something ever goes wrong. With DigitalOcean, you can also snapshot your Droplets when you’re happy with a certain setup. This way, you can test risky changes and always come back to a known good state in case you break something.
If you’d like to learn more, visit X2Go’s official documentation website.
]]>Baixe o e-book completo!
Este livro destaca habilidades práticas de administrador de sistema e arquiteturas comuns. Além disso, também ele aborda as melhores práticas aplicáveis aos sistemas de automatização e execução, em qualquer escala, de um notebook ou servidor para 1.000 ou mais. Ele foi criado para orientar você dentro da disciplina, e, com sorte, encorajar você a aprender mais sobre a administração de sistemas.
Este livro baseia-se no programa Como fazer servidores funcionarem: um guia prático para a administração do sistema Linux, encontrado na Comunidade da DigitalOcean. O livro foi estruturado em torno de alguns tópicos centrais:
Tópicos introdutórios
Pilhas de tecnologia LAMP e LEMP
Protegendo seus servidores
Automação com o Ansible
Controle de versão e integração contínua
Sinta-se à vontade para escolher tópicos neste livro que lhe interessam e explore-os usando esses capítulos como guias. Trabalhar com este livro irá expor você a uma grande variedade de tecnologias, termos técnicos e abordagens conceituais para gerenciar servidores Linux. Aprenda de cada capítulo ou seção no seu próprio ritmo e na ordem que quiser.
Baixe o e-book em formato EPUB ou PDF, seguindo os links abaixo.
Baixe o e-book completo!
Para obter recursos adicionais sobre a administração de sistemas para ajudar você a começar, e para participar da comunidade DigitalOcean de outros desenvolvedores e administradores, confira nossa biblioteca crescente de tutoriais, perguntas e projetos com o sinalizador Getting Started (Começando).
]]>Das komplette eBook herunterladen!
Server zum Laufen bringen: Ein praktischer Leitfaden für Systemadministrator eBook im EPUB-Format
Server zum Laufen bringen: Ein praktischer Leitfaden für Systemadministrator eBook im PDF-Format
Dieses Buch behandelt praktische Fähigkeiten von Systemadministratoren, gängige Architekturen, denen Sie begegnen werden, und bewährte Verfahren, die für die Automatisierung und den Betrieb von Systemen jeder Größenordnung gelten, von einem Laptop oder Server bis hin zu 1.000 oder mehr. Es soll Ihnen helfen, sich innerhalb des Fachgebiets zu orientieren, und ermutigt Sie hoffentlich dazu, mehr über Systemadministration zu lernen.
Dieses Buch basiert auf dem Lehrplan Server zum Laufen bringen: Ein praktischer Leitfaden für die Linux-Systemadministration in der DigitalOcean Community. Es ist nach einigen Hauptthemen gegliedert:
Einführende Themen
LAMP- und LEMP-Technologie-Stapel
Sicherung Ihrer Server
Automatisierung mit Ansible
Versionskontrolle und kontinuierliche Integration
Suchen Sie sich in diesem Buch Themen aus, die Sie interessieren, und erkunden Sie sie anhand dieser Kapitel als Leitfaden. Beim Durcharbeiten dieses Buches werden Sie mit einer Vielzahl von Technologien, technischen Begriffen und konzeptionellen Ansätzen zur Verwaltung von Linux-Servern konfrontiert. Sie können jedes Kapitel oder jeden Abschnitt in Ihrem eigenen Tempo und in der von Ihnen gewählten Reihenfolge durcharbeiten.
Sie können das eBook im EPUB- oder PDF-Format unter den nachstehenden Links herunterladen.
Das komplette eBook herunterladen!
Server zum Laufen bringen: Ein praktischer Leitfaden für Systemadministrator eBook im EPUB-Format
Server zum Laufen bringen: Ein praktischer Leitfaden für Systemadministrator eBook im PDF-Format
Wenn Sie zusätzliche Sysadmin-Ressourcen benötigen, die Ihnen den Einstieg erleichtern, und wenn Sie an der DigitalOcean-Community anderer Entwickler und Administratoren teilnehmen möchten, sehen Sie sich unsere wachsende Bibliothek mit Tutorials, Fragen und Projekten mit dem Tag Erste Schritte an.
]]>Descargue todo el EBook
En este libro, se destacan las habilidades prácticas de los administradores de sistemas, las arquitecturas comunes que encontrará y las mejores prácticas que se aplican a la automatización y la ejecución de sistemas a cualquier escala, desde una computadora portátil o un servidor hasta 1000 o más. Está diseñado para ayudar a orientarlo dentro de la disciplina, y esperamos que lo anime a aprender más sobre la administración de sistemas.
Este libro se basa en el plan de estudios Cómo hacer funcionar servidores: guía práctica para la administración de sistemas Linux, disponible en la comunidad de DigitalOcean. Su estructura se basa en algunos temas centrales:
Temas introductorios
Pilas de tecnología LAMP y LEMP
Proteger sus servidores
Automatización con Ansible
Control de versiones e integración continua
Puede elegir los temas de este libro que le interesen y explorarlos usando estos capítulos como guía. Trabajar con este libro lo expondrá a una amplia variedad de tecnologías, términos técnicos y enfoques conceptuales para administrar servidores de Linux. Puede trabajar con cada capítulo o sección a su propio ritmo y en cualquier orden que desee.
Puede descargar el libro electrónico en formato EPUB o PDF siguiendo los enlaces que se muestran a continuación.
[label Descargue todo el EBook] Libro electrónico Cómo hacer funcionar los servidores: Guía práctica para la administración de sistemas en formato EPUB
Para obtener recursos adicionales de administración de sistemas que lo ayuden a comenzar, y para participar en la comunidad de DigitalOcean de otros desarrolladores y administradores, consulte nuestra creciente biblioteca de tutoriales, preguntas y proyectos con la etiqueta “Introducción”.
]]>Télécharger le eBook complet
Ce livre met en lumière les compétences pratiques des administrateurs système, les architectures courantes que vous rencontrerez et les meilleures pratiques qui s’appliquent à l’automatisation et à l’exécution de systèmes à toute échelle, d’un ordinateur portable ou d’un serveur à 1 000 ou plus. Il est destiné à vous aider à vous orienter dans la discipline et, nous l’espérons, vous encourager à en apprendre davantage sur l’administration du système.
Ce livre est basé sur le programme d’études Faire fonctionner les serveurs : Un guide pratique pour l’administration du système Linux que l’on trouve dans la communauté DigitalOcean. Il se structure autour de quelques thèmes centraux :
Sujets d’introduction
Les piles technologiques LAMP et LEMP
Sécuriser vos serveurs
Automatisation avec Ansible
Contrôle de la version et intégration continue
N’hésitez pas à choisir les sujets qui vous intéressent dans ce livre et à les explorer en utilisant ces chapitres comme des guides. En parcourant ce livre, vous découvrirez une grande variété de technologies, de termes techniques et d’approches conceptuelles de la gestion des serveurs Linux. Vous pouvez parcourir chaque chapitre ou section à votre propre rythme et dans l’ordre que vous souhaitez.
Vous pouvez télécharger le eBook au format EPUB ou PDF en suivant les liens ci-dessous.
Télécharger le eBook complet
Pour des ressources sysadmin supplémentaires qui vous aideront à démarrer, et pour participer à la communauté DigitalOcean d’autres développeurs et administrateurs, consultez notre bibliothèque sans cesse croissante de tutoriels, de questions et de projets avec le tag Getting Started.
]]>[label Загрузите полную электронную книгу!] Электронная книга «Как заставить серверы работать: практическое руководство по системному администрированию» в формате EPUB
В книге рассказывается о практических навыках системного администрирования, распространенных архитектурах и передовых практиках, применимых для автоматизации и эксплуатации систем любого масштаба, от одного ноутбука или сервера до 1000 и более серверов. Книга поможет сориентироваться в этой области и возможно вызовет желание узнать больше о системном администрировании.
Книга основана на учебном курсе «Как заставить серверы работать: практическое руководство по администрированию систем Linux», который можно найти в сообществе DigitalOcean. Она включает несколько основных тем:
Вводные темы
Технологические стеки LAMP и LEMP
Обеспечение безопасности серверов
Автоматизация с помощью Ansible
Контроль версий и непрерывная интеграция
Выбирайте в книге интересные вам темы и изучайте их, используя эти главы как руководство. Эта книга познакомит вас с разнообразными технологиями, техническими терминами и концептуальными подходами к управлению серверами Linux. Вы можете изучать главы или разделы в любом выбранном порядке в удобном для вас темпе.
Вы можете загрузить электронную книгу в формате EPUB или PDF по одной из следующих ссылок.
[label Загрузите полную электронную книгу!] Электронная книга «Как заставить серверы работать: практическое руководство по системному администрированию» в формате EPUB
Дополнительные ресурсы для системных администраторов, которые помогут вам начать работать в этой области и участвовать в сообществе DigitalOcean вместе с другими разработчиками и администраторами, можно найти в нашей постоянно растущей библиотеке обучающих руководств, ответов на вопросы и проектов с тегом Getting Started (Начало работы).
]]>Das komplette eBook herunterladen!
Kubernetes for Full-Stack Developers eBook im EPUB-Format
Kubernetes for Full-Stack Developers eBook im PDF-Format
Dieses Buch soll Newcomern und erfahrenen Benutzern dabei helfen, sich über Kubernetes zu informieren. Die Kapitel im Buch stellen Ihnen zuerst die wichtigsten Kubernetes-Konzepte vor und bauen dann schrittweise darauf auf, bis schließlich das Ausführen einer Anwendung auf einem Produktionscluster leicht, wiederholbar und automatisch wird. Daran anschließend werden fortgeschrittenere Themen vorgestellt, wie das Management eines Kubernetes-Clusters.
Es gibt zahlreiche Tools, Netzwerkkonfigurationen und Prozesse, die dazu beitragen können, Kubernetes verständlicher zu machen. Dieses Buch erörtert nacheinander die einzelnen Themen, damit jeder, der es liest, in der Lage ist, selbst ein Kubernetes-Cluster zu erstellen, zu verwalten und zu überwachen.
Dieses Buch basiert auf dem Leitfaden Kubernetes for Full-Stack Developers, der in der DigitalOcean Community zu finden ist. Es ist in einige Hauptthemen untergliedert:
Erlernen der Kubernetes-Grundkonzepte
Modernisieren von Anwendungen, sodass sie mit Containern arbeiten
Containerisieren von Awendungen
Der Einsatz von Anwendungen auf Kubernetes
Das Management des Cluster-Betriebs
Sie müssen die Themen nicht unbedingt in einer bestimmten Reihenfolge bearbeiten. Wenn ein Abschnitt für Sie interessanter oder relevanter ist, können Sie Teile überspringen und später wieder darauf zurückkommen. Und wenn Sie die Konzepte und Tools eines Abschnitts bereits beherrschen, können Sie den Abschnitt überspringen und sich anderen Themen zuwenden.
Sie können das eBook im EPUB- oder PDF-Format unter den nachstehenden Links herunterladen.
[label Das komplette eBook herunterladen!] Kubernetes for Full-Stack Developers eBook im EPUB-Format
Für weitere Kubernetes-Ressourcen und zur Teilnahme in der DigitalOcean Community anderer Developer, die Kubernetes verwenden, können Sie sich unsere immer größer werdende Bibliothek mit Tutorials, Fragen und Projekten mit der Markierung Kubernetes ansehen.
]]>Télécharger le eBook complet
eBook Kubernetes pour Développeurs Full-Stack au format EPUB
eBook Kubernetes pour Développeurs Full-Stack au format PDF
Ce livre a été conçu pour aider les nouveaux venus et les utilisateurs expérimentés à se familiariser avec Kubernetes. Ses chapitres ont pour but de présenter les concepts fondamentaux de Kubernetes et de les développer de telle sorte que l’exécution d’une application sur un cluster de production soit un processus familier, reproductible et automatisé. À partir de là, des sujets plus complexes sont introduits, comme par exemple la gestion d’un cluster de Kubernetes lui-même.
Il existe de nombreux outils, configurations de réseau et processus qui peuvent contribuer à rendre Kubernetes plus accessible. Ce livre examinera chaque sujet à tour de rôle afin que quiconque puisse construire, gérer et surveiller un cluster Kubernetes par lui-même.
Ce livre s’appuie sur le programme Kubernetes for Full-Stack Developers qui se trouve sur DigitalOcean Community. Il se structure autour de quelques thèmes centraux :
Apprendre les concepts fondamentaux de Kubernetes
Moderniser les apps pour travailler avec des conteneurs
Mettre en conteneur les apps
Déployer des apps vers Kubernetes
Gérer des opérations de cluster
Vous n’êtes pas obligés de suivre les sujets dans un ordre particulier. Si vous trouvez qu’une section est plus intéressante ou plus pertinente qu’une autre, explorez-la et revenez aux autres plus tard si vous préférez. De même, si vous êtes déjà familier avec les concepts et les outils d’une section donnée, n’hésitez pas à sauter celle-ci et à vous concentrer sur d’autres sujets.
Vous pouvez télécharger le eBook au format EPUB ou PDF en suivant les liens ci-dessous.
Télécharger le eBook complet
Kubernetes pour Développeurs Full-Stack eBook au format EPUB
Pour obtenir des ressources Kubernetes supplémentaires et pour accéder à la communauté DigitalOcean des autres développeurs utilisant Kubernetes, consultez notre bibliothèque de tutoriels, de questions et de projets en constante expansion, grâce au tag Kubernetes.
]]>Download the Complete eBook!
Making Servers Work: A Practical Guide to System Administration eBook in EPUB format
Making Servers Work: A Practical Guide to System Administration eBook in PDF format
This book highlights practical sysadmin skills, common architectures that you’ll encounter, and best practices that apply to automating and running systems at any scale, from one laptop or server to 1,000 or more. It is intended to help orient you within the discipline, and hopefully encourages you to learn more about system administration.
This book is based on the Making Servers Work: A Practical Guide to Linux System Administration curriculum found on DigitalOcean Community. It is structured around a few central topics:
Introductory Topics
LAMP and LEMP Technology Stacks
Securing your Servers
Automation with Ansible
Version Control and Continuous Integration
Feel free to pick topics in this book that interest you and explore them using these chapters as guides. Working through this book will expose you to a wide variety of technologies, technical terms, and conceptual approaches to managing Linux servers. You can work through each chapter or section at your own pace, and in any order that you choose.
You can download the eBook in either the EPUB or PDF format by following the links below.
Download the Complete eBook!
Making Servers Work: A Practical Guide to System Administration eBook in EPUB format
Making Servers Work: A Practical Guide to System Administration eBook in PDF format
For additional sysadmin resources to help you get started, and to participate in the DigitalOcean community of other developers and administrators, check out our growing library of tutorials, questions, and projects with the Getting Started tag.
]]>Baixe o e-book completo!!
e-Book do Kubernetes para desenvolvedores de pilha completa, em formato EPUB
e-Book do Kubernetes para desenvolvedores de pilha completa, em formato PDF
Este livro foi desenvolvido para ajudar usuários novatos e experientes a aprender sobre o Kubernetes. Seus capítulos foram elaborados para apresentar os principais conceitos do Kubernetes e para expandi-los a um nível em que a execução de um aplicativo em um cluster de produção torne-se um processo familiar, que pode ser repetido e automatizado. A partir daí, serão introduzidos tópicos mais avançados, por exemplo, como gerenciar o próprio cluster do Kubernetes.
Existem várias ferramentas, configurações de rede e processos que podem ajudar a tornar o Kubernetes mais acessível. Esse livro examinará um tópico de cada vez, de modo que todos que seguirem as instruções serão capazes de compilar, gerenciar e monitorar um cluster do Kubernetes sozinhos.
Esse livro baseia-se no Currículo Kubernetes para desenvolvedores de pilha completa, encontrado na Comunidade do DigitalOcean. O livro foi estruturado em torno de alguns tópicos centrais:
Aprendendo os conceitos principais do Kubernetes
Modernizando os aplicativos para funcionarem com contêineres
Conteinerizando os aplicativos
Implantando aplicativos para o Kubernetes
Gerenciando operações de cluster
Você não precisa seguir os tópicos em nenhuma ordem específica. Se uma seção for mais interessante ou relevante para você, explore-a e volte para as outras mais tarde, caso preferir. De igual modo, se já estiver familiarizado com os conceitos e ferramentas de uma determinada seção, sinta-se à vontade para ignorar a mesma e se concentrar em outros tópicos.
Você pode baixar o e-book em formato EPUB ou PDF, seguindo os links abaixo.
Baixe o e-book completo!!
e-Book do Kubernetes para desenvolvedores de pilha completa, em formato EPUB
e-Book do Kubernetes para desenvolvedores de pilha completa, em formato PDF
Para obter recursos adicionais do Kubernetes e participar da comunidade do DigitalOcean de outros desenvolvedores usando o Kubernetes, confira nossa crescente biblioteca de tutoriais, dúvidas e projetos com o rótulo de Kubernetes.
]]>Descargue el libro electrónico completo
Libro electrónico Kubernetes for Full-Stack Developers en formato PDF
Este libro está diseñado para que tanto usuarios principiantes como experimentados aprendan sobre Kubernetes. Los capítulos están elaborados para introducir conceptos básicos sobre Kubernetes e interiorizarlos hasta un nivel en el cual ejecutar una aplicación en un clúster de producción sea un proceso familiar, repetible y automatizado. Luego, se introducen temas más avanzados; por ejemplo, como administrar un clúster Kubernetes.
Existe una gran cantidad de herramientas, configuraciones de red y procesos que pueden facilitar el acercamiento a Kubernetes. En este libro, se examinará un tema a la vez para que quien lo siga pueda crear, administrar y controlar un clúster Kubernetes por su cuenta.
Este libro se basa en el plan de estudios de Kubernetes para desarrolladores “full-stack” disponible en la comunidad de DigitalOcean. Su estructura se basa en algunos temas centrales:
Aprender conceptos básicos de Kubernetes
Modernizar aplicaciones para trabajar con contenedores
Disponer aplicaciones en contenedores
Implementar aplicaciones en Kubernetes
Administrar operaciones de clústeres
No es necesario seguir los temas en ningún orden en particular. Si una sección le resulta más interesante o pertinente, puede explorarla y regresar a las demás más adelante, si así lo prefiere. Asimismo, si ya está familiarizado con los conceptos y las herramientas de una sección, puede omitirla y concentrarse en otros temas.
Puede descargar el libro electrónico en formato EPUB o PDF siguiendo los enlaces que se muestran a continuación.
Descargue todo el EBook
Libro electrónico Kubernetes for “Full-Stack” Developers en formato EPUB
Libro electrónico Kubernetes for “Full-Stack” Developers en **formato PDF **
Para acceder a recursos adicionales de Kubernetes y participar en la comunidad de DigitalOcean de otros desarrolladores que usan Kubernetes, consulte nuestra biblioteca de tutoriales, preguntas y proyectos con la etiqueta Kubernetes.
]]>Загрузите полную электронную книгу!
Электронная книга Kubernetes для разработчиков широкого профиля в формате EPUB
Электронная книга Kubernetes для разработчиков широкого профиля в формате PDF
Эта книга поможет начинающим и опытным пользователям узнать много интересного о Kubernetes. В главах книги представлены основные концепции Kubernetes и их развитие до уровня, когда запуск приложения на рабочем кластере становится знакомым, воспроизводимым и автоматизированным процессом. Затем в книге рассказывается о более сложных вопросах, в частности об управлении кластером Kubernetes.
Для упрощения работы с Kubernetes существует множество инструментов, сетевых конфигураций и процессов. В этой книге все темы рассматриваются по очереди, так что с помощью их указаний можно самостоятельно собрать кластер Kubernetes, управлять им и контролировать его работу.
Эта книга основана на учебном плане Kubernetes для разработчиков широкого профиля, который можно найти в сообществе DigitalOcean. Она включает несколько основных тем:
Изучение основных концепций Kubernetes
Модернизация приложений для работы с контейнерами
Контейнеризация приложений
Развертывание приложений в Kubernetes
Управление кластерными операциями
Необязательно проходить темы в определенном порядке. Если какой-то раздел интересует вас больше, чем другие, вы можете вначале ознакомиться с ним, а затем вернуться к остальным. Если вы знакомы с концепциями и инструментами, описанными в определенном разделе, вы можете пропустить его и сосредоточиться на других темах.
Вы можете загрузить электронную книгу в формате EPUB или PDF по одной из следующих ссылок.
Загрузите полную электронную книгу!
Электронная книга Kubernetes для разработчиков широкого профиля в формате EPUB
Электронная книга Kubernetes для разработчиков широкого профиля в формате PDF
Чтобы найти дополнительные ресурсы по Kubernetes и участвовать в сообществе DigitalOcean вместе с другими разработчиками, использующими Kubernetes, вы можете ознакомиться с нашей растущей библиотекой обучающих руководств, вопросов и проектов с тегом Kubernetes.
]]>O painel de controle web da DigitalOcean fornece uma interface de gerenciamento de Droplets usando apontar e clicar . No entanto, você pode preferir uma ferramenta de linha de comando se tiver muitos Droplets para gerenciar, precisar administrar Droplets a partir do terminal sem uma área de trabalho gráfica disponível ou tiver tarefas que se beneficiariam de uma interface com script.
doctl
é o cliente oficial de linha de comandos da DigitalOcean. Ele utiliza a API da DigitalOcean para fornecer acesso à maioria dos recursos da conta e dos Droplets.
Para seguir este tutorial, você precisará de:
doctl
instalado seguindo as instruções de instalação e configuração do projeto.Este tutorial pretende ser uma referência para a maioria das operações do doctl
. Como os comandos doctl
são intimamente ligados à API, também pode ser útil ler a documentação da API e Como Usar a API da DigitalOcean v2.
doctl
No doctl
, os recursos individuais são executados, fornecendo ao utilitário um comando, um ou mais subcomandos e, às vezes, uma ou mais opções que especificam valores particulares. Os comandos são agrupados em três categorias principais:
account
para informações relacionadas à contaauth
para autenticação com a DigitalOceancompute
para o gerenciamento de infraestruturaPara ver uma visão geral de todos os comandos, você pode chamar o doctl
por si só. Para ver todos os comandos disponíveis em uma das três categorias principais, você pode usar doctl categoria
, como doctl compute
. Para um guia de uso de um comando específico, digite o comando com a flag --help
, como em doctl compute droplet --help
.
Em ambientes de script, ou ao trabalhar na linha de comando com ferramentas de processamento de dados, geralmente é útil obter uma saída legível por máquina a partir de um comando.
Por padrão, o doctl
formata sua saída em colunas de texto legível por humanos, mas pode produzir uma saída JSON detalhada usando a opção --output json
.
- doctl compute droplet get droplet_id --output json
Sample Output{
"id": droplet_id,
"name": "nome_do_droplet",
"memory": 1024,
"vcpus": 1,
"disk": 30,
"region": {
"slug": "nyc3",
"name": "New York 3",
"sizes": [
...
Além de ser um formato legível com bibliotecas padrão na maioria das linguagens de programação, a saída JSON pode permitir uma inspeção mais detalhada dos Droplets e de outros recursos.
Geralmente, é útil obter apenas um conjunto de campos da saída. Para fazer isso, você pode usar a flag --format
, que pega a lista de campos pelo seu nome. Por exemplo, se você deseja obter apenas o ID, nome e endereço IP dos seus Droplets, você pode usar o seguinte comando:
- doctl compute droplet list --format "ID,Name,PublicIPv4"
Sample outputID Name Public IPv4
50513569 doctl-1 67.205.152.65
50513570 test 67.205.148.128
50513571 node-1 67.205.131.88
O comando doctl compute droplet get
suporta template de saída, o que lhe permite personalizar o formato da saída. Para usar esse recurso, especifique o template formatado para Go via a flag --template
Por exemplo, se você deseja obter o nome de um Droplet no formato droplet_name: nome_do_droplet
, use o seguinte comando get
:
- doctl compute droplet get 12345678 --template "nome_do_droplet: {{ .Name}}"
Outputnome_do_droplet: ubuntu-1gb-nyc3-01
Para obter uma lista de recursos, como Droplets, você pode usar o comando list
sem parâmetros.
- doctl compute droplet list
Sample output for list commandID Name Public IPv4 Private IPv4 Public IPv6 Memory VCPUs Disk Region Image Status Tags
50513569 test-1 67.205.152.65 512 1 20 nyc1 Ubuntu 16.04.2 x64 active
50513571 test-2 67.205.131.88 512 1 20 nyc1 Ubuntu 16.04.2 x64 active
50513570 site 67.205.148.128 512 1 20 nyc1 Ubuntu 16.04.2 x64 active
O comando list
suporta um glob como um parâmetro opcional. Um glob representa um padrão com caracteres curinga que podem ser usados para filtrar recursos específicos por nome. Por exemplo, para obter uma lista de Droplets cujos nomes começam com test
, você pode usar o seguinte comando:
doctl compute droplet list 'test*'
Sample output for list command with 'doctl-' as globID Name Public IPv4 Private IPv4 Public IPv6 Memory VCPUs Disk Region Image Status Tags
50513569 test-1 67.205.152.65 512 1 20 nyc1 Ubuntu 16.04.2 x64 active
50513571 test-2 67.205.131.88 512 1 20 nyc1 Ubuntu 16.04.2 x64 active
Criar um recurso requer comandos mais longos com parâmetros mais detalhados. Para criar um Droplet, você precisará especificar a imagem que deseja usar, a região do datacenter e o tipo de Droplet que deseja usando o slug associado. Veja New Size Slugs for Droplet Plan Changes
para encontrar o slug que você quer usar. Como alternativa, execute o comando doctl compute size list
.
Por exemplo, o comando a seguir cria um Droplet Debian 8 de 64 bits chamado test com 1 GB de memória, uma CPU, uma chave SSH e backups ativados.
- doctl compute droplet create test --size s-1vcpu-1gb --image debian-8-x64 --region nyc1 --ssh-keys 4d:23:e6:e4:8c:17:d2:cf:89:47:36:b5:c7:33:40:4e --enable-backups
Você verá esta saída:
Sample Droplet creation outputID Name Public IPv4 Memory VCPUs Disk Region Image Status
11450164 test 1024 1 30 nyc1 Debian 8.3 x64 new
A exclusão de um recurso requer um ID de recurso como argumento ou um nome de recurso no caso de não existir um ID para determinado recurso (por exemplo, tags). Para confirmar suas intenções, você precisa confirmar todas as ações de exclusão, respondendo à pergunta de confirmação com y
ou yes
.
doctl compute droplet delete 123456
OutputWarning: Are you sure you want to delete droplet(s) (y/N) ?
Não fornecer uma resposta ou fornecer uma resposta diferente de y
ou yes
cancelará a ação sem excluir o recurso. Você pode fazer com que o doctl
assuma uma resposta afirmativa sem fornecê-la explicitamente, usando a flag -f
(--force
):
doctl compute droplet delete -f 123456
O comando de criação de Droplets requer uma série de identificadores, como nyc1
para a região NYC1, debian-8-x64
para a imagem Debian e uma impressão digital ou fingerprint da chave SSH como 4d:23:e6:e4:8c:17:d2:cf:89:47:36:b5:c7:33:40:4e
.
Vários recursos, como Droplets e imagens, são identificados por um valor (geralmente numérico) exclusivo no banco de dados da DigitalOcean.
Você pode obter os identificadores exclusivos necessários para a maioria dos comandos da API:
Comando | Notas |
---|---|
doctl compute droplet list |
Seus Droplets. Alguns comandos também levam o nome; a maioria exige o valor numérico da coluna ID. |
doctl compute ssh-key list |
As chaves SSH associadas à sua conta. Para a criação de Droplets, você pode especificar o ID numérico ou fingerprint. |
doctl compute region list |
Regiões disponíveis. Use a string na coluna Slug. |
doctl compute image list |
Imagens disponíveis, incluindo snapshots, backups e imagens base de distribuição. Use a string na coluna Slug para a criação de novos Droplets. |
doctl compute size list |
Tamanhos de Droplets disponíveis. Use a string na coluna Slug. |
doctl compute tag list |
Tags disponíveis. Use a string na coluna Name. |
O comando doctl compute droplet
lhe permite criar, excluir e inspecionar Droplets. Novamente, a maioria dos comandos para trabalhar com Droplets individuais exige o ID exclusivo do Droplet, e estes podem ser encontrados na saída do comando doctl droplet list
.
Subcomando doctl compute droplet |
Notas |
---|---|
actions droplet_id |
Exibir um histórico de ações executadas para um Droplet. |
backups droplet_id |
Listar backups para um Droplet. |
create name --size s-1vcpu-1gb --image image_slug --region nyc1 |
Criar um Droplet. Tamanho, imagem e região são todos obrigatórios. |
delete droplet_id_ou_nome |
Excluir um Droplet por ID ou nome. |
get droplet_id |
Obter detalhes de um Droplet específico. |
kernels droplet_id |
Listar kernels para um Droplet. |
list |
Listar seus Droplets atuais. |
neighbors droplet_id |
Listar seus Droplets em execução no mesmo hardware físico que um Droplet específico. |
snapshots droplet_id |
Listar snapshots para um Droplet. |
tag droplet_id/nome_do_droplet |
Colocar uma Tag em um Droplet. |
untag droplet_id/nome_do_droplet |
Retirar uma tag de um Droplet. |
O comando doctl compute droplet-action
lhe permite disparar várias ações para um Droplet, incluindo ações de gerenciamento de energia e alternância de recursos como backups e redes privadas.
Subcomando doctl compute droplet-action |
Notas |
---|---|
get droplet_id --action-id action_id |
Obter detalhes sobre a ação em um Droplet. |
disable-backups droplet_id |
Desativar backups para um Droplet. |
reboot droplet_id |
Reiniciar um Droplet. |
power-cycle droplet_id |
Desligar e ligar novamente um Droplet. |
shutdown droplet_id |
Desligar um Droplet. |
power-off droplet_id |
Desligar um Droplet. O Droplet deve estar ligado. Geralmente, é melhor fazer isso na linha de comando do próprio Droplet para evitar perda de dados. |
power-on droplet_id |
Ligue um Droplet. O Droplet deve estar desligado. |
power-reset droplet_id |
Resetar um Droplet. |
enable-ipv6 droplet_id |
Ativar ipv6 para um Droplet. |
enable-private-networking droplet_id |
Ativar rede privada para um Droplet. |
upgrade droplet_id |
Fazer upgrade de um Droplet. |
restore droplet_id --image-id image_id |
Restaurar um Droplet para uma imagem de backup específica. O image_id deve ser um backup do Droplet. |
resize droplet_id --size 2gb |
Redimensionar um Droplet. O Droplet deve estar desligado. Por padrão, o disco não é redimensionado, o que permite que o Droplet seja rebaixado. Você pode redimensionar o disco usando a flag --resize-disk . |
rebuild droplet_id --image-id image_id |
Reconstruir um Droplet a partir de uma imagem específica. |
rename droplet_id --droplet-name novo_nome |
Renomear um Droplet para novo_nome . |
change-kernel droplet_id --kernel-id kernel_id |
Alterar o kernel de um Droplet para kernel_id . |
snapshot droplet_id --snapshot-name nome_do_snapshot |
Tirar um instantâneo ou snapshot de um Droplet, nomeando-o como nome_do_snapshot . |
Para se conectar a um Droplet individual com SSH, geralmente é necessário saber seu endereço IP ou nome de domínio totalmente qualificado. Você pode usar o doctl
para se conectar a um Droplet pelo seu nome, ID numérico ou IP privado:
- doctl compute ssh nome_do_droplet
- doctl compute ssh droplet_id
- doctl compute ssh --ssh-private-ip ip_privado_do_droplet
Além disso, você pode fornecer um comando para executar assim que a conexão SSH for estabelecida usando a flag --ssh-command
. Isso executará o comando, cuja saída será impressa no seu terminal local, e então a sessão SSH será fechada.
- doctl compute ssh --ssh-command comando
Nota: No momento, o encaminhamento de comandos SSH não está disponível no Windows.
O nome do usuário SSH padrão é root (core para o CoreOS) e a porta padrão é a 22
. Você pode usar flags para definir valores não padrão e ativar outros recursos:
Flag | Descrição |
---|---|
--ssh-user string |
Nome de usuário a ser usado para a sessão SSH. |
--ssh-port int |
A porta para a sessão SSH. |
--ssh-key-path string |
Caminho para a chave SSH. |
--ssh-agent-forwarding |
Ativar encaminhamento de agente. |
Você também pode alterar os valores de configuração padrão em um arquivo de configuração. O arquivo README do projeto tem mais detalhes sobre como fazer isso.
Você pode gerenciar as chaves públicas SSH associadas à sua conta usando o comando doctl compute ssh-key
. A maioria dos comandos que fazem referência às chaves SSH aceitam o ID numérico da chave ou sua fingerprint.
Subcomando doctl compute ssh-key |
Notas |
---|---|
list |
Listar chaves SSH associadas à sua conta. |
get ssh_key_id_ou_fingerprint |
Obter informações sobre uma chave específica, por ID numérico ou fingerprint da chave. |
create novo_nome_da_chave --public-key "chave_pública" |
Associar uma chave pública à sua conta, especificando seu conteúdo. |
import novo_nome_da_chave --public-key-file ~/.ssh/id_rsa.pub |
Associar uma chave pública à sua conta especificando um arquivo de origem. |
delete ssh_key_id_ou_fingerprint |
Excluir uma chave da sua conta por ID numérico ou fingerprint. |
update ssh_key_id_ou_fingerprint --key-name novo_nome_da_chave |
Alterar o nome de uma chave por ID numérico ou fingerprint. |
Um IP flutuante é um endereço IP estático acessível ao público que pode ser atribuído a um de seus Droplets. Para uma descrição detalhada do recurso, você pode ler Como usar IPs flutuantes na DigitalOcean. Você pode manipular IPs flutuantes com doctl compute floating-ip
.
Subcomando doctl compute floating-ip |
Notas |
---|---|
list |
Listar todos os endereços IP flutuantes. |
get endereço_ip_flutuante |
Obter detalhes de um endereço IP flutuante. |
create --region nyc1 |
Crie um IP flutuante na região nyc1 . |
delete endereço_ip_flutuante |
Excluir um endereço IP flutuante. |
O comando doctl compute floating-ip-action
é usado para atribuir ou cancelar a atribuição de um IP flutuante de um Droplet.
Subcomando doctl compute floating-ip-action |
Notas |
---|---|
assign ip_flutuante droplet_id |
Atribuir um IP flutuante ao Droplet pelo seu ID numérico. |
unassign ip_flutuante |
Cancelar a atribuição de um IP flutuante. |
get ip_flutuante action_id |
Obter detalhes sobre uma ação de IP flutuante pelo seu ID numérico. |
O comando doctl compute domain
é usado para gerenciar domínios. Veja nossa Série de Introdução ao gerenciamento de DNS para uma visão geral mais ampla do assunto.
Subcomando doctl compute domain |
Notas |
---|---|
list |
Listar domínios. |
create nome_de_domínio --ip-address endereço_ip_do_droplet |
Criar um domínio com registros padrão para endereço_ip_do_droplet . |
get nome_de_domínio |
Obter um registro de domínio. |
delete nome_de_domínio |
Excluir um domínio. |
O comando doctl compute domain records
pode ser usado para criar, excluir, atualizar ou obter informações sobre os registros DNS do domínio.
Subcomando doctl compute domain records |
Notas |
---|---|
list nome_de_domínio |
Listar registros para determinado domínio. |
create nome_de_domínio --record-type tipo_de_registro |
Criar um registro para o domínio. |
delete nome_de_domínio record_id |
Excluir registro pelo ID numérico. |
update nome_de_domínio --record-id record_id |
Atualizar registro pelo ID numérico. |
O comando doctl compute volume
pode ser usado para criar, excluir ou obter informações sobre os volumes de armazenamento em bloco da DigitalOcean. Para obter mais informações sobre esse recurso, leia nosso guia Como usar o armazenamento em bloco na DigitalOcean.
Subcomando doctl compute volume |
Notas |
---|---|
list |
Listar volumes. |
create nome_do_volume --region região_do_volume --size tamanho_do_volume |
Criar um volume. O nome, região e tamanho são obrigatórios. |
get volume_ID |
Obter volume pelo ID numérico. |
delete volume_ID |
Excluir volume. |
snapshot volume_ID |
Fazer um Snapshot de um volume. |
O comando doctl compute volume-action
lhe permite disparar ações para um volume, incluindo anexar volumes e desanexar volumes de Droplets.
Subcomando doctl compute volume-action |
Notas |
---|---|
attach volume_id droplet_id |
Anexar um volume a um Droplet. |
detach volume_id droplet_id |
Desanexar um volume de um Droplet. |
resize volume_id --region região_do_volume --size novo_tamanho |
Redimensionar um volume. |
O comando doctl compute load-balancer
pode ser usado para criar, excluir ou obter informações sobre os balanceadores de carga ou Load Balancers da DigitalOcean. Para obter mais informações sobre esse recurso, leia nossa Introdução aos balanceadores de carga da DigitalOcean.
Subcomando doctl compute load-balancer |
Notas |
---|---|
list |
Listar load balancers. |
create --name nome_do_lb --region região_do_lb --tag-name nome_da_tag --forwarding-rules regra_de_encaminhamento |
Criar um Load Balancer. O nome, a região, uma tag ou lista de IDs de Droplets e pelo menos uma regra de encaminhamento são obrigatórios. |
update --name nome_do_lb --region região_do_lb --tag-name nome_da_tag --forwarding-rules regra_de_encaminhamento |
Atualizar um Load Balancer. O nome, a região, uma tag ou lista de IDs de Droplets e pelo menos uma regra de encaminhamento são obrigatórios. |
get lb_ID |
Obter um load balancer pelo ID. |
delete lb_ID |
Excluir um load balancer pelo ID. |
add-droplets lb_ID --droplet-ids droplet_ID |
Adicionar Droplets a um load balancer. |
remove-droplets lb_ID --droplet-ids droplet_ID |
Remover Droplets de um load balancer. |
add-forwarding-rules lb_ID --forwarding-rules regra_de_encaminhamento |
Adicionar regras de encaminhamento a um load balancer. |
remove-forwarding-rules lb_ID --forwarding-rules regra_de_encaminhamento |
Remover regras de encaminhamento de um load balancer. |
Quando usado como argumento para o doctl
, as regras de encaminhamento devem ser expressas como: entry_protocol:protocolo,entry_port:porta,target_protocol:protocolo,target_port:porta
.
O subcomando doctl compute certificate
lhe permite fazer upload e gerenciar certificados SSL, chaves privadas e cadeias de certificados.
Subcomando doctl compute certificate |
Notas |
---|---|
list |
Listar todos os certificados. |
get certificate_id |
Obter um certificado pelo ID. |
create --name nome_do_certificado --leaf-certificate-path caminho_do_cetrificado_em_folha |
Criar um certificado. O nome e o caminho do certificado em folha são obrigatórios. |
delete certificate_id |
Excluir um certificado pelo ID. |
O comando doctl compute snapshot
pode ser usado para listar, excluir ou obter informações sobre snapshots de Droplet e de Volume.
Subcomando doctl compute snapshot |
Notas |
---|---|
list |
Listar todos os Snapshots. |
get snapshot_ID |
Obter um Snapshot pelo ID. |
delete snapshot_ID |
Excluir um Snapshot. |
Para criar um novo Snapshot, você precisa usar o comando apropriado na árvore de comandos do recurso relevante. Por exemplo:
doctl compute droplet-action snapshot droplet_ID
cria um Snapshot de um Droplet.doctl compute volume snapshot volume_ID
cria um Snapshot de um Volume.O comando doctl compute image
lhe permite gerenciar todas as imagens, incluindo imagens de distribuição, imagens de aplicações e imagens criadas pelo usuário, como backups e snapshots. Recomendamos o uso do comando snapshot
para gerenciar snapshots, pois ele fornece mais detalhes, possui funcionalidade de exclusão e suporta snapshots de Armazenamento em Bloco.
Subcomando doctl compute image |
Notas |
---|---|
list --public |
Listar todas imagens. |
list-distribution --public |
Listar todas imagens de distribuição disponíveis |
list-application --public |
Listar todas as imagens de Aplicações de Um Clique. |
list-user |
Listar todas as imagens criadas pelo usuário. |
get image_id |
Obter uma imagem pelo ID. |
update image_id --name nome_da_imagem |
Atualizar o nome da imagem. O nome é obrigatório. |
delete image_id |
Excluir uma imagem pelo ID. |
O comando doctl compute image-action
lhe permite transferir imagens e obter detalhes sobre as ações invocadas nas imagens.
Subcomando doctl compute image-action |
Notas |
---|---|
get image_id --action-id action_id |
Obter uma ação para a imagem pelo seu ID. O ID da ação é obrigatório. |
transfer image_id --region região |
Transferir uma imagem para outra região. O ID da imagem e a região são obrigatórios. |
O comando doctl compute firewall
lhe permite criar e gerenciar Firewalls, incluindo a criação e a manutenção de regras. Para obter mais informações sobre como administrar Firewalls usando o doctl
, consulte o tutorial How To Secure Web Server Infrastructure With DigitalOcean Cloud Firewalls Using Doctl.
Comando doctl compute firewall |
Notas |
---|---|
list |
Listar todos os Firewalls. |
list-by-droplet droplet_id |
Listar todos os Firewalls pelo ID numérico do Droplet. |
create --name nome_do_firewall --inbound-rules regras_de_entrada --outbound-rules regras_de_saída |
Criar um Firewall. O nome e pelo menos uma regra de entrada ou saída são obrigatórios. |
update firewall_id --name nome_do_firewall --inbound-rules regras_de_entrada --outbound-rules regras_de_saída |
Atualizar um Firewall. O ID numérico, o nome e pelo menos uma regra de entrada ou saída são obrigatórios. |
get firewall_id |
Obter um Firewall pelo seu ID numérico. |
delete firewall_id |
Excluir um firewall pelo ID numérico. |
add-droplets firewall_id --droplet-ids droplet_IDs |
Adicionar Droplets pelo seu ID numérico ao Firewall. |
remove-droplets firewall_id --droplet-ids droplet_IDs |
Remover Droplets do Firewall pelos seus IDs numéricos. |
add-tags firewall_id --tag-names tags |
Adicionar Tags ao Firewall. |
remove-tags firewall_id --tag-names tags |
Remover Tags do Firewall. |
add-rules firewall_id --inbound-rules regras_de_entrada --outbound-rules regras_de_saída |
Adicionar regras de entrada ou saída ao Firewall. |
remove-rules firewall_id --inbound-rules regras_de_entrada --outbound-rules regras_de_saída |
Remover as regras de entrada ou saída do Firewall. |
Quando usado como argumento para o doctl
, as regras de entrada ou saída devem ser expressas como: protocol:protocolo,ports:portas,droplet_id:droplet-id
.
As Tags são usadas para aplicar rótulos personalizados aos recursos, permitindo que você os filtre facilmente. Você pode aprender mais sobre Tags no tutorial How To Tag DigitalOcean Droplets.
Subcomando doctl compute tag |
Notas |
---|---|
create nome_da_tag |
Criar uma Tag. |
get nome_da_tag |
Obter uma Tag pelo nome. |
list |
Listar todas as Tags. |
delete nome_da_tag |
Excluir uma Tag pelo nome. |
O sistema da DigitalOcean registra um histórico das ações executadas em seus Droplets, IPs flutuantes e outros recursos. Você pode acessar esses dados com o comando doctl compute action
:
- doctl compute action list
Você pode ver ações para um Droplet específico da seguinte maneira:
- doctl compute droplet actions droplet_id
Você pode descobrir detalhes básicos sobre sua conta, como seu endereço de e-mail configurado e o limite de Droplet:
- doctl account get
Como as solicitações de API são limitadas por uma taxa de acessos, pode ser útil ver quantas solicitações você fez recentemente e quando o limite deve ser redefinido:
- doctl account ratelimit
O utilitário doctl
é uma ferramenta útil para gerenciar Droplets e outros recursos na linha de comando. Isso pode reduzir bastante a quantidade de interação manual com interfaces baseadas em web, necessárias para tarefas diárias de desenvolvimento e de administração.
Além de aprender sobre a API subjacente, você pode querer explorar bibliotecas que envolvem a API para linguagens de programação populares, e ferramentas como Ansible para automatizar tarefas no nível do sistema.
]]>Download the Complete eBook!
Kubernetes for Full-Stack Developers eBook in EPUB format
Kubernetes for Full-Stack Developers eBook in PDF format
This book is designed to help newcomers and experienced users alike learn about Kubernetes. Its chapters are designed to introduce core Kubernetes concepts and to build on them to a level where running an application on a production cluster is a familiar, repeatable, and automated process. From there, more advanced topics are introduced, like how to manage a Kubernetes cluster itself.
There are numerous tools, networking configurations, and processes that can help make Kubernetes more approachable. This book will examine each topic in turn so that anyone who follows along will be able to build, manage, and monitor a Kubernetes cluster on their own.
This book is structured around a few central topics:
Learning Kubernetes core concepts
Modernizing applications to work with containers
Containerizing applications
Deploying applications to Kubernetes
Managing cluster operations
You should not feel obliged to follow the topics in any particular order. If one section is more interesting or relevant to you, explore it and come back to the others later if you prefer. Likewise, if you are already familiar with the concepts and tools in a given section, feel free to skip that one and focus on other topics.
You can download the eBook in either the EPUB or PDF format by following the links below.
Download the Complete eBook!
For additional Kubernetes resources and to participate in the DigitalOcean community of other developers using Kubernetes, check out our growing library of tutorials, questions, and projects with the Kubernetes tag.
]]>This comic compares imperative and declarative Kubernetes management. If you would like to learn more about Kubernetes, please check our our Kubernetes Course for Full-Stack Developers.
]]>O DigitalOcean Spaces é um serviço de armazenamento de objetos projetado para facilitar e reduzir o custo de armazenar e servir grandes quantidades de dados.
Neste guia, abordaremos como migrar dados entre regiões do Spaces, usando o Rclone para transferir dados entre dois Spaces. Demonstraremos como instalar o Rclone, as definições de configuração necessárias para acessar várias regiões e os comandos que você pode usar para sincronizar seus arquivos entre regiões e verificar sua integridade.
Antes de começarmos a instalar e configurar o Rclone para copiar nossos objetos entre os Spaces, precisaremos de algumas informações sobre nossa conta no DigitalOcean Spaces. Vamos precisar de uma chave da API do Spaces e precisamos conhecer as regiões e os nomes dos nossos Spaces de origem e destino.
Para criar uma chave de API do DigitalOcean Spaces, siga a seção “Creating an Access Key” de nossa documentação em How To Create a DigitalOcean Space API Key.
Salve o ID da chave de acesso e a chave secreta. Nós os usaremos posteriormente para configurar o Rclone para acessar nossa conta.
Em seguida, precisamos encontrar o endpoint para cada Space. Você pode visualizar o endpoint do Space no Painel de controle da DigitalOcean, selecionando o Space e visualizando a guia Settings:
O endpoint sempre estará na região em que você criou o Space, seguido por .digitaloceanspaces.com
. Anote o endpoint para ambos os seus Spaces. Usaremos essas informações ao criar nossa configuração rclone
.
Agora você está pronto para instalar o Rclone. Você pode fazer isso na sua máquina local ou – se a largura de banda estiver limitada – você poderá instalar o Rclone em um Droplet localizado na região do Spaces de origem ou de destino.
Visite a seção de Downloads do site do projeto para encontrar binários do utilitário compilado para diferentes plataformas. Para começar, baixe o binário compactado que corresponde ao sistema operacional do seu computador.
Depois de baixar o arquivo zip do Rclone para o seu computador, siga a seção abaixo que corresponda à sua plataforma.
Antes de podermos extrair o arquivo, precisamos garantir que o utilitário unzip
esteja disponível.
Se você estiver executando o Ubuntu ou o Debian, você pode atualizar o índice local de pacotes e instalar o unzip
digitando:
- sudo apt update
- sudo apt install unzip
Se você estiver usando CentOS ou Fedora, você pode instalar o unzip
digitando:
- sudo yum install unzip
Com o unzip
instalado, navegue até o diretório onde você baixou o arquivo zip do rclone
:
- cd ~/Downloads
Em seguida, descompacte o arquivo e vá para o diretório recém-criado:
- unzip rclone*
- cd rclone-v*
A partir daqui, podemos copiar o binário no diretório /usr/local/bin
para que ele esteja disponível em todo o sistema:
- sudo cp rclone /usr/local/bin
Em seguida, adicionamos a página de manual ao nosso sistema, para que possamos obter facilmente ajuda na sintaxe de comando e nas opções disponíveis. Verifique se o diretório local de manual está disponível e copie o arquivo rclone.1
:
- sudo mkdir -p /usr/local/share/man/man1
- sudo cp rclone.1 /usr/local/share/man/man1
Atualize o banco de dados man
para adicionar a nova página de manual ao sistema:
- sudo mandb
Por fim, podemos criar o diretório .config do Rclone e abrir um arquivo de configuração:
- mkdir -p ~/.config/rclone
- nano ~/.config/rclone/rclone.conf
Isso abrirá seu editor de texto com um novo arquivo em branco. Avance para a seção Configurando o Rclone para prosseguir.
Se você estiver executando o macOS, comece navegando no terminal até o diretório em que você baixou o arquivo zip rclone
:
- cd ~/Downloads
Em seguida, descompacte o arquivo e vá para o diretório recém-criado:
- unzip -a rclone*
- cd rclone-v*
Depois, certifique-se de que o diretório /usr/local/bin
está disponível e então copie o binário rclone
para ele:
- sudo mkdir -p /usr/local/bin
- sudo cp rclone /usr/local/bin
Por fim, podemos criar o diretório .config e abrir um arquivo de configuração:
- mkdir -p ~/.config/rclone
- nano ~/.config/rclone/rclone.conf
Isso abrirá seu editor de texto com um novo arquivo em branco. Avance para a seção Configurando o Rclone para prosseguir.
Se você estiver executando o Windows, comece navegando até o diretório Downloads no Gerenciador de Arquivos do Windows. Selecione o arquivo zip rclone
e clique com o botão direito do mouse. No menu de contexto que aparece, clique em Extrair tudo…:
Siga as instruções para extrair os arquivos do arquivo zip.
O utilitário rclone.exe
deve ser executado a partir da linha de comando. Abra uma nova janela Prompt de Comando (o programa cmd.exe
) clicando no botão Windows no canto inferior esquerdo, digitando cmd e selecionando Prompt de Comando.
Dentro do prompt, navegue até o caminho do rclone
que você extraiu digitando:
- cd "%HOMEPATH%\Downloads\rclone*\rclone*"
Liste o conteúdo do diretório para verificar se você está no local correto:
- dir
Output10/23/2017 01:02 PM <DIR> .
10/23/2017 01:02 PM <DIR> ..
10/23/2017 01:02 PM 17 git-log.txt
10/23/2017 01:02 PM 296,086 rclone.1
10/23/2017 01:02 PM 16,840,192 rclone.exe
10/23/2017 01:02 PM 315,539 README.html
10/23/2017 01:02 PM 261,497 README.txt
5 File(s) 17,713,331 bytes
2 Dir(s) 183,296,266,240 bytes free
Você precisará estar neste diretório sempre que quiser usar o comando rclone.exe
.
Nota: No macOS e no Linux, executamos a ferramenta digitando rclone
, mas no Windows, o comando é chamado rclone.exe
. No restante deste guia, forneceremos comandos como rclone
, portanto, certifique-se de substituir o rclone.exe
sempre que executar no Windows.
Em seguida, podemos criar o diretório .config e abrir um arquivo de configuração para definir nossas credenciais do S3 e do Spaces:
- mkdir "%HOMEPATH%\.config\rclone"
- notepad "%HOMEPATH%\.config\rclone\rclone.conf"
Isso abrirá seu editor de texto com um novo arquivo em branco. Continue em frente para aprender como definir suas regiões do Spaces no arquivo de configuração.
Vamos configurar nossas duas regiões do DigitalOcean Spaces como Rclone “remotas” no arquivo de configuração do Rclone. Cole a seção a seguir no arquivo de configuração para definir a primeira região:
[spaces-sfo2]
type = s3
env_auth = false
access_key_id = access_key_do_seu_spaces
secret_access_key = secret_key_do_seu_spaces
endpoint = sfo2.digitaloceanspaces.com
acl = private
Aqui, definimos um novo rclone
“remoto” chamado spaces-sfo2
. Altere o nome da região para corresponder à região do Spaces que você está configurando.
Definimos o type
como s3
para que o rclone
saiba a maneira apropriada de interagir e gerenciar o recurso de armazenamento remoto. Definiremos as credenciais de acesso do Spaces neste arquivo de configuração, de forma que possamos definir env_auth
como false
.
A seguir, definimos as variáveis access_key_id
e secret_access_key
como access key e secret key do nosso Spaces, respectivamente. Certifique-se de alterar os valores para as credenciais associadas à sua conta.
Definimos o endpoint
como o endpoint do Spaces que vimos anteriormente.
Finalmente, definimos o acl
como private
para proteger nossos ativos até que desejemos compartilhá-los.
Em seguida, faça uma duplicata do bloco de configuração que você acabou de criar, então atualize o nome e a região do endpoint para refletir sua segunda região:
. . .
[spaces-nyc3]
type = s3
env_auth = false
access_key_id = access_key_do_seu_spaces
secret_access_key = secret_key_do_seu_spaces
endpoint = nyc3.digitaloceanspaces.com
acl = private
O restante da configuração deve permanecer o mesmo que para a primeira região. Salve e feche o arquivo quando terminar.
No macOS e no Linux, certifique-se de bloquear as permissões do arquivo de configuração, pois nossas credenciais estão dentro dele:
- chmod 600 ~/.config/rclone/rclone.conf
No Windows, as permissões são negadas a usuários não administrativos, a menos que sejam concedidas explicitamente, portanto, não precisamos ajustar o acesso manualmente.
Em seguida, usaremos o rclone
para explorar nossos Spaces e sincronizar dados entre eles.
Agora que nossa configuração está completa, estamos prontos para transferir nossos arquivos.
Comece verificando os remotos configurados no rclone
:
- rclone listremotes
Outputspaces-nyc3:
spaces-sfo2:
As duas regiões que definimos são exibidas.
Podemos ver os Spaces disponíveis, pedindo ao rclone
para listar os “diretórios” associados aos remotos (certifique-se de adicionar dois pontos ao final do nome do remoto):
- rclone lsd spaces-sfo2:
Output -1 2019-09-23 13:07:54 -1 source-space
A saída acima indica que um Space, chamado source-space
foi encontrado na região sfo2
.
Você pode repetir o procedimento para visualizar a outra região:
- rclone lsd spaces-nyc3:
Output -1 2019-09-23 13:08:28 -1 destination-space
Para visualizar o conteúdo de um Space, você pode usar o comando tree
. Passe o nome remoto, seguido por dois pontos e o nome do “diretório” que você deseja listar (o nome do Space):
- rclone tree spaces-sfo2:source-space
Output/
├── Photos
│ ├── 2019.01.24-23.10.27.png
│ ├── 2019.01.24-23.11.39.png
│ ├── 2019.01.24-23.18.00.png
│ ├── 2019.01.24-23.18.18.png
│ ├── 2019.01.24-23.18.30.png
│ ├── 2019.01.24-23.19.32.png
│ ├── 2019.01.24-23.23.06.png
│ ├── 2019.01.24-23.23.53.png
│ ├── 2019.01.24-23.25.14.png
│ ├── 2019.01.24-23.26.22.png
│ ├── 2019.01.25-12.43.35.png
│ ├── 2019.03.13-14.35.34.png
│ └── 2019.03.13-14.40.52.png
└── Photos.zip
1 directories, 14 files
Quando estiver pronto, você poderá copiar os arquivos entre os Spaces, digitando:
- rclone sync spaces-sfo2:source-space spaces-nyc3:destination-space
Assumindo que tudo correu bem, o rclone
começará a copiar objetos entre os dois Spaces.
Nota: Se você não criou anteriormente o Space de destino na região especificada, o rclone
tentará criar um para você com o nome dado. Isso vai falhar se o nome fornecido já estiver sendo usado por outra conta ou se o nome não atender aos requisitos de nomenclatura para o DigitalOcean Spaces (somente letras minúsculas, números e traços).
Quando a transferência estiver concluída, você pode verificar se os objetos foram transferidos visualizando-os com o subcomando tree
:
- rclone tree spaces-nyc3:destination-space
Output/
├── Photos
│ ├── 2019.01.24-23.10.27.png
│ ├── 2019.01.24-23.11.39.png
│ ├── 2019.01.24-23.18.00.png
│ ├── 2019.01.24-23.18.18.png
│ ├── 2019.01.24-23.18.30.png
│ ├── 2019.01.24-23.19.32.png
│ ├── 2019.01.24-23.23.06.png
│ ├── 2019.01.24-23.23.53.png
│ ├── 2019.01.24-23.25.14.png
│ ├── 2019.01.24-23.26.22.png
│ ├── 2019.01.25-12.43.35.png
│ ├── 2019.03.13-14.35.34.png
│ └── 2019.03.13-14.40.52.png
└── Photos.zip
1 directories, 14 files
Para uma verificação mais robusta, use o subcomando check
para comparar os objetos nas duas regiões:
- rclone check spaces-sfo2:source-space spaces-nyc3:destination-space
Output2019/09/23 14:29:11 NOTICE: S3 bucket destination-space: 0 differences found
2019/09/23 14:29:11 NOTICE: S3 bucket destination-space: 14 matching files
Isso irá comparar os valores de hash de cada objeto nos dois remotos. Você pode receber uma mensagem indicando que alguns hashes não podem ser comparados. Nesse caso, você pode executar novamente o comando com a flag --size-only
(que compara com base apenas no tamanho do arquivo) ou a flag --download
(que baixa cada objeto de ambos os remotos para comparar localmente) para verificar a integridade da transferência.
Neste guia, abordamos como transferir objetos entre duas regiões do DigitalOcean Spaces. Reunimos credenciais de API e informações de endpoint do serviço Spaces, instalamos e configuramos o utilitário rclone
em nosso computador local e, em seguida, copiamos todos os objetos de um Space de origem para um Space de destino.
O cliente rclone
pode ser usado para muitas outras tarefas de gerenciamento de armazenamento de objetos, incluindo upload ou download de arquivos, montagem de Spaces no sistema de arquivos local e criação ou exclusão de Spaces adicionais. Confira a página de man
para aprender mais sobre as funcionalidades que a ferramenta fornece.
O Kubernetes ingress facilita a exposição de web services para a internet. Quando se trata de serviços privados, no entanto, você provavelmente desejará limitar quem pode acessá-los. O oauth2_proxy pode servir como uma barreira entre a Internet pública e os serviços privados. O oauth2_proxy é um servidor e proxy reverso que fornece autenticação usando diferentes provedores, como o GitHub, e valida os usuários pelo seu endereço de email ou outras propriedades.
Neste tutorial, você usará o oauth2_proxy com o GitHub para proteger seus serviços. Quando terminar, você terá um sistema de autorização semelhante ao do diagrama a seguir:
Para concluir este tutorial, você precisará de:
Após seguir o tutorial indicado na seção Pré-requisitos, você terá dois web services em execução no cluster: echo1
e echo2
. Você também terá um ingress que mapeia echo1.seu_domínio
e echo2.seu_domínio
aos seus serviços correspondentes.
Neste tutorial, usaremos as seguintes convenções:
.int.seu_domínio
, como service.int.seu_domínio
. O agrupamento de serviços privados em um subdomínio é ideal porque o cookie de autenticação será compartilhado por todos os subdomínios *.int.seu_domínio
auth.int.seu_domínio
.Nota: Certifique-se de substituir seu_domínio
pelo seu próprio nome de domínio onde quer que ele apareça neste tutorial.
Para começar, atualize a definição atual do ingress para mover os serviços echo1
e echo2
em .int.seu_domínio
. Abra echo_ingress.yaml
em seu editor de textos para poder alterar os domínios:
- nano echo_ingress.yaml
Renomeie todas as instâncias de echo1.seu_domínio
para echo1.int.seu_domínio
, e substitua todas as instâncias de echo2.seu_domínio
com echo2.int.seu_domínio
:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: echo-ingress
annotations:
kubernetes.io/ingress.class: nginx
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
spec:
tls:
- hosts:
- echo1.int.seu_domínio
- echo2.int.seu_domínio
secretName: letsencrypt-prod
rules:
- host: echo1.int.seu_domínio
http:
paths:
- backend:
serviceName: echo1
servicePort: 80
- host: echo2.int.seu_domínio
http:
paths:
- backend:
serviceName: echo2
servicePort: 80
Salve o arquivo e aplique as alterações:
- kubectl apply -f echo_ingress.yaml
Isso atualizará os certificados TLS para seus serviços echo1
e echo2
também.
Agora atualize sua configuração de DNS para refletir as alterações que você fez. Primeiro, procure o endereço IP de seu ingress Nginx executando o seguinte comando para imprimir seus detalhes:
- kubectl get svc --namespace=ingress-nginx
Você verá o endereço IP abaixo de EXTERNAL-IP
na saída:
OutputNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx LoadBalancer 10.245.247.67 203.0.113.0 80:32486/TCP,443:32096/TCP 20h
Copie o endereço IP externo para a sua área de transferência. Navegue até o seu serviço de gerenciamento de DNS e localize os registros A para echo1-2.seu_domínio
para apontar para esse endereço IP externo. Se você estiver usando a DigitalOcean para gerenciar seus registros DNS, veja How to Manage DNS Records para instruções.
Delete os registros para echo1
e echo2
. Adicione um novo registro A
para o hostname *.int.seu_domínio
e o aponte para o endereço IP externo do ingress.
Agora, qualquer solicitação para qualquer subdomínio em *.int.seu_domínio
será roteada para o ingress Nginx, para que você possa usar esses subdomínios no seu cluster.
Em seguida, você irá configurar o GitHub como seu provedor de login.
O oauth2_proxy suporta vários provedores de login. Neste tutorial, você usará o provedor GitHub. Para começar, crie uma nova aplicação GitHub OAuth.
Na guia OAuth Apps da página de configurações do desenvolvedor da sua conta, clique no botão New OAuth App.
Os campos Application name e Homepage URL podem ser qualquer coisa que você quiser. No campo Authorization callback URL, digite https://auth.int.seu_domínio/oauth2/callback
.
Depois de registrar a aplicação, você receberá um Client ID e um Secret. Tome nota dos dois, pois você precisará deles no próximo passo.
Agora que você criou uma aplicação GitHub OAuth, você pode instalar e configurar o oauth2_proxy.
Você usará o Helm para instalar o oauth2_proxy no cluster. Primeiro, você criará um secret do Kubernetes para armazenar o Client ID e o Secret da aplicação GitHub, bem como um secret de criptografia para os cookies do navegador definidos pelo oauth2_proxy.
Execute o seguinte comando para gerar um secret de cookie seguro:
- python -c 'import os,base64; print base64.b64encode(os.urandom(16))'
Copie o resultado para sua área de transferência.
Em seguida, crie o secret do Kubernetes, substituindo os valores destacados pelo seu secret de cookie, seu ID de cliente GitHub e sua chave secreta do GitHub:
- kubectl -n default create secret generic oauth2-proxy-creds \
- --from-literal=cookie-secret=SEU_COOKIE_SECRET \
- --from-literal=client-id=SEU_GITHUB_CLIENT_ID \
- --from-literal=client-secret=SEU_GITHUB_SECRET
Você verá a seguinte saída:
Outputsecret/oauth2-proxy-creds created
Em seguida, crie um novo arquivo chamado oauth2-proxy-config.yaml
que conterá a configuração para o oauth2_proxy
:
- nano oauth2-proxy-config.yaml
Os valores que você definir neste arquivo substituirão os valores padrão do Helm. Adicione o seguinte código ao arquivo:
config:
existingSecret: oauth2-proxy-creds
extraArgs:
whitelist-domain: .int.seu_domínio
cookie-domain: .int.seu_domínio
provider: github
authenticatedEmailsFile:
enabled: true
restricted_access: |-
permitido@user1.com
permitido@user2.com
ingress:
enabled: true
path: /
hosts:
- auth.int.seu_domínio
annotations:
kubernetes.io/ingress.class: nginx
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
tls:
- secretName: oauth2-proxy-https-cert
hosts:
- auth.int.seu_domínio
Este código faz o seguinte:
auth.int.seu_domínio
com um certificado TLS da Let’s Encrypt.Agora que você tem o secret e o arquivo de configuração prontos, você pode instalar o oauth2_proxy
. Execute o seguinte comando:
- helm repo update \
- && helm upgrade oauth2-proxy --install stable/oauth2-proxy \
- --reuse-values \
- --values oauth2-proxy-config.yaml
Pode levar alguns minutos para que o certificado Let’s Encrypt seja emitido e instalado.
Para testar se o deploy foi bem-sucedido, navegue até https://auth.int.seu_domínio
. Você verá uma página que solicita que você efetue login no GitHub.
Com oauth2_proxy configurado e em execução, tudo o que falta é exigir autenticação nos seus serviços.
Para proteger um serviço, configure seu Nginx ingress para forçar a autenticação via oauth2_proxy. O Nginx e o nginx-ingress suportam essa configuração nativamente, portanto, você só precisa adicionar algumas anotações à definição do ingress.
Vamos proteger os serviços echo1
e echo2
que você configurou no tutorial de pré-requisito. Abra echo_ingress.yaml
em seu editor:
- nano echo_ingress.yaml
Adicione essas duas anotações adicionais ao arquivo para exigir autenticação:
annotations:
kubernetes.io/ingress.class: nginx
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/auth-url: "https://auth.int.seu_domínio/oauth2/auth"
nginx.ingress.kubernetes.io/auth-signin: "https://auth.int.seu_domínio/oauth2/start?rd=https%3A%2F%2F$host$request_uri"
Salve o arquivo e aplique as alterações:
- kubectl apply -f echo_ingress.yaml
Agora, quando você navega até https://echo1.int.seu_domínio
, você será solicitado a fazer login usando o GitHub para acessá-lo. Após fazer o login com uma conta válida, você será redirecionado de volta ao serviço echo1
. O mesmo vale para echo2
.
Neste tutorial, você configurou o oauth2_proxy no seu cluster Kubernetes e protegeu um serviço privado atrás de um login do GitHub. Para qualquer outro serviço que você precise proteger, basta seguir as instruções descritas no Passo 4.
O oauth2_proxy suporta muitos provedores diferentes, além do GitHub. Para saber mais sobre diferentes provedores, consulte a documentação oficial.
Além disso, existem muitos parâmetros de configuração que você pode precisar ajustar, embora os padrões adotados sejam adequados à maioria das necessidades. Para uma lista de parâmetros, consulte a documentação do Helm charts e a documentação do oauth2_proxy.
]]>DigitalOcean’s web-based control panel provides a point-and-click interface for managing Droplets. However, you may prefer a command-line tool if you have many Droplets to manage, need to administer Droplets from the terminal without a graphical desktop available, or have tasks which would benefit from a scriptable interface.
doctl
is the official DigitalOcean command-line client. It uses the DigitalOcean API to provide access to most account and Droplet features.
To follow this tutorial, you will need:
doctl
installed by following the project’s installation and configuration instructions.This tutorial is intended as a reference for most of doctl
’s operations. Because doctl
commands closely parallel the API, it may also be helpful to read the API documentation and How To Use the DigitalOcean API v2.
doctl
UsageIn doctl
, individual features are invoked by giving the utility a command, one or more sub-commands, and sometimes one or more options specifying particular values. Commands are grouped under three main categories:
account
for account-related informationauth
for authenticating with DigitalOceancompute
for managing infrastructureTo see an overview of all commands, you can invoke doctl
by itself. To see all available commands under one of the three main categories, you can use doctl category
, like doctl compute
. For a usage guide on a specific command, enter the command with the --help
flag, as in doctl compute droplet --help
.
In scripting environments, or when working on the command line with data-processing tools, it’s often helpful to get machine-readable output from a command.
By default, doctl
formats its output in columns of human-readable text, but can produce detailed JSON output using the --output json
option.
- doctl compute droplet get droplet_id --output json
Sample Output{
"id": droplet_id,
"name": "droplet_name",
"memory": 1024,
"vcpus": 1,
"disk": 30,
"region": {
"slug": "nyc3",
"name": "New York 3",
"sizes": [
...
In addition to being a format readable with standard libraries in most programming languages, the JSON output may allow more fine-grained inspection of Droplets and other resources.
It’s often useful to obtain only a set of fields from output. To do this, you can use the --format
flag followed by a list of your desired fields. For example, if you want to obtain only the ID, name, and IP address of your Droplets, you can use the following command:
- doctl compute droplet list --format "ID,Name,PublicIPv4"
Sample outputID Name Public IPv4
50513569 doctl-1 67.205.152.65
50513570 test 67.205.148.128
50513571 node-1 67.205.131.88
The doctl compute droplet get
command supports output templating, which lets you customize the format of the output. To use this feature, specify the Go-formatted template via the --template
flag.
For example, if you want to get a Droplet’s name in the format droplet_name: droplet_name
, you would use the following get
command:
- doctl compute droplet get 12345678 --template "droplet_name: {{ .Name}}
Outputdroplet_name: ubuntu-1gb-nyc3-01
To get a list of resources, like Droplets, you can use the list
command with no parameters.
- doctl compute droplet list
Sample output for list commandID Name Public IPv4 Private IPv4 Public IPv6 Memory VCPUs Disk Region Image Status Tags
50513569 test-1 67.205.152.65 512 1 20 nyc1 Ubuntu 16.04.2 x64 active
50513571 test-2 67.205.131.88 512 1 20 nyc1 Ubuntu 16.04.2 x64 active
50513570 site 67.205.148.128 512 1 20 nyc1 Ubuntu 16.04.2 x64 active
The list
command supports a glob as an optional parameter. A glob represents pattern with wildcard characters which can be used to filter specific resources by name. For example, to get a list of Droplets whose names start with test
, you can use the following command:
doctl compute droplet list 'test*'
Sample output for list command with 'doctl-' as globID Name Public IPv4 Private IPv4 Public IPv6 Memory VCPUs Disk Region Image Status Tags
50513569 test-1 67.205.152.65 512 1 20 nyc1 Ubuntu 16.04.2 x64 active
50513571 test-2 67.205.131.88 512 1 20 nyc1 Ubuntu 16.04.2 x64 active
Creating a resource requires longer commands with more detailed parameters. To create a Droplet, you’ll need to specify the image you want to use, the datacenter region, and the kind of Droplet you want by using its associated slug. See New Size Slugs for Droplet Plan Changes
to find the slug you want to use. Alternatively, run the doctl compute size list
command.
For example, the following command creates a 64-bit Debian 8 Droplet named test with 1GB of memory, one CPU, an SSH key, and backups enabled.
- doctl compute droplet create test --size s-1vcpu-1gb --image debian-8-x64 --region nyc1 --ssh-keys 4d:23:e6:e4:8c:17:d2:cf:89:47:36:b5:c7:33:40:4e --enable-backups
You’ll see this output:
Sample Droplet creation outputID Name Public IPv4 Memory VCPUs Disk Region Image Status
11450164 test 1024 1 30 nyc1 Debian 8.3 x64 new
Deleting a resource requires a resource ID as an argument, or a resource name in the event that an ID doesn’t exist for given resource (e.g. tags). To confirm your intentions, you need to confirm all delete actions by answering the confirmation question with y
or yes
.
doctl compute droplet delete 123456
OutputWarning: Are you sure you want to delete droplet(s) (y/N) ?
Not providing an answer or providing an answer different from y
or yes
will cancel the action without deleting the resource. You can make doctl
assume an affirmative answer without explicitly providing it, using --f
(--force
) flag:
doctl compute droplet delete -f 123456
The Droplet creation command requires a series of identifiers, like nyc1
for the NYC1 region, debian-8-x64
for the Debian image, and an SSH key fingerprint like 4d:23:e6:e4:8c:17:d2:cf:89:47:36:b5:c7:33:40:4e
.
A number of resources, such as Droplets and images, are identified by a value (often numeric) unique within DigitalOcean’s database.
You can get the required unique identifiers for most commands from the API:
Command | Notes |
---|---|
doctl compute droplet list |
Your Droplets. Some commands also take the name; most require the numeric value from the ID column. |
doctl compute ssh-key list |
The SSH keys associated with your account. For Droplet creation, you can specify either the numeric ID or fingerprint. |
doctl compute region list |
Available regions. Use the string in the Slug column. |
doctl compute image list |
Available images, including snapshots, backups, and base distribution images. Use the string in the Slug column for creating new Droplets. |
doctl compute size list |
Available Droplet sizes. Use the string in the Slug column. |
doctl compute tag list |
Available Tags. Use the string in the Name column. |
The doctl compute droplet
command lets you create, delete, and inspect Droplets. Again, most commands for working with individual Droplets require the Droplet’s unique ID, and these can be found in the output from doctl droplet list
.
doctl compute droplet subcommand |
Notes |
---|---|
actions droplet_id |
Display a history of actions taken for a Droplet. |
backups droplet_id |
List backups for a Droplet. |
create name --size s-1vcpu-1gb --image image_slug --region nyc1 |
Create a Droplet. Size, image and region are all mandatory. |
delete droplet_id_or_name |
Delete a Droplet by id or name. |
get droplet_id |
Get details for a particular Droplet. |
kernels droplet_id |
List kernels for a Droplet. |
list |
List your current Droplets. |
neighbors droplet_id |
List your Droplets running on the same physical hardware as a specific Droplet. |
snapshots droplet_id |
List snapshots for a Droplet. |
tag droplet_id/droplet_name |
Tag a Droplet. |
untag droplet_id/droplet_name |
Untag a Droplet. |
The doctl compute droplet-action
command lets you trigger various actions for a Droplet, including power management actions and toggling features like backups and private networking.
doctl compute droplet-action subcommand |
Notes |
---|---|
get droplet_id --action-id action_id |
Get details about action on a Droplet. |
disable-backups droplet_id |
Disable backups for a Droplet. |
reboot droplet_id |
Reboot a Droplet. |
power-cycle droplet_id |
Turn a Droplet off and back on again. |
shutdown droplet_id |
Shut down a Droplet. |
power-off droplet_id |
Power off a Droplet. The Droplet must be powered on. It’s usually best to do this from the command line of the Droplet itself in order to prevent data loss. |
power-on droplet_id |
Power on a Droplet. The Droplet must be powered off. |
power-reset droplet_id |
Power reset Droplet. |
enable-ipv6 droplet_id |
Enable ipv6 for a Droplet. |
enable-private-networking droplet_id |
Enable private networking for a Droplet. |
upgrade droplet_id |
Upgrade a Droplet. |
restore droplet_id --image-id image_id |
Restore a Droplet to a specific backup image. The image_id must be a backup of the Droplet. |
resize droplet_id --size 2gb |
Resize a Droplet. The Droplet must be powered off. By default, disk is not resized, which allows Droplet to be downgraded. You can resize disk using the --resize-disk flag. |
rebuild droplet_id --image-id image_id |
Rebuild a Droplet from a specific image. |
rename droplet_id --droplet-name new_name |
Rename a Droplet to new_name . |
change-kernel droplet_id --kernel-id kernel_id |
Change a Droplet’s kernel to kernel_id . |
snapshot droplet_id --snapshot-name snapshot_name |
Take a snapshot of a Droplet, naming it snapshot_name . |
In order to connect to an individual Droplet with SSH, it’s usually necessary to know either its IP address or fully-qualified domain name. You can instead use doctl
to connect to a Droplet by its name, numeric ID or Private IP:
- doctl compute ssh droplet_name
- doctl compute ssh droplet_id
- doctl compute ssh --ssh-private-ip droplet_private_ip
Also, you can provide a command to execute once the SSH connection is established using the --ssh-command
flag. This will run the command, the output of which will be printed on your local terminal, and then the SSH session will close.
- doctl compute ssh --ssh-command command
Note: SSH command forwarding is currently not available on Windows.
The default SSH user name is root (core for CoreOS) and the default port is 22
. You can use flags to set non-default values and enable other features:
Flag | Description |
---|---|
--ssh-user string |
User name to use for the SSH session. |
--ssh-port int |
The port for the SSH session. |
--ssh-key-path string |
Path to SSH key. |
--ssh-agent-forwarding |
Enable agent forwarding. |
You can also change the default configuration values in a configuration file. The project’s README file has more detail on how to do this.
You can manage the SSH public keys associated with your account using the doctl compute ssh-key
command. Most commands which reference SSH keys accept either the numeric ID for the key or its fingerprint.
doctl compute ssh-key subcommand |
Notes |
---|---|
list |
List SSH keys associated with your account. |
get ssh_key_id_or_fingerprint |
Get info on a specific key, by numeric ID or key’s fingerprint. |
create new_key_name --public-key "public_key" |
Associate a public key with your account by specifying its contents. |
import new_key_name --public-key-file ~/.ssh/id_rsa.pub |
Associate a public key with your account by specifying a source file. |
delete ssh_key_id_or_fingerprint |
Delete a key from your account by numeric ID or fingerprint. |
update ssh_key_id_or_fingerprint --key-name new_key_name |
Change a key’s name by numeric ID or fingerprint. |
A Floating IP is a publicly-accessible static IP address that can be assigned to one of your Droplets. For a detailed description of the feature, you can read How To Use Floating IPs on DigitalOcean. You can manipulate floating IPs with doctl compute floating-ip
.
doctl compute floating-ip subcommand |
Notes |
---|---|
list |
List all Floating IP addresses. |
get floating_ip_address |
Get the details for a Floating IP address. |
create --region nyc1 |
Create a Floating IP in nyc1 region. |
delete floating_ip_address |
Delete a floating IP address. |
The doctl compute floating-ip-action
command is used to assign or unassign a Floating IP from a Droplet.
doctl compute floating-ip-action subcommand |
Notes |
---|---|
assign floating_ip droplet_id |
Assign a Floating IP to the Droplet by its numeric ID. |
unassign floating_ip |
Unassign a Floating IP. |
get floating_ip action_id |
Get details about a Floating IP action by its numeric ID. |
The doctl compute domain
command is used to manage domains. See our Introduction to Managing DNS series for a broad overview of the subject.
doctl compute domain subcommand |
Notes |
---|---|
list |
List domains. |
create domain_name --ip-address droplet_ip_address |
Create a domain with default records for droplet_ip_address . |
get domain_name |
Get a domain record. |
delete domain_name |
Delete a domain. |
The doctl compute domain records
command can be used to create, delete, update or get information about domain’s DNS records.
doctl compute domain records subcommand |
Notes |
---|---|
list domain_name |
List records for given domain. |
create domain_name --record-type record_type |
Create an record for domain. |
delete domain_name record_id |
Delete record by numeric ID. |
update domain_name --record-id record_id |
Update record by numeric ID. |
The doctl compute volume
command can be used to create, delete, or get information about DigitalOcean’s Block Storage volumes. For more information about this feature, read our guide on How To Use Block Storage on DigitalOcean.
doctl compute volume subcommand |
Notes |
---|---|
list |
List volumes. |
create volume_name --region volume_region --size volume_size |
Create a volume. The name, region, and size are mandatory. |
get volume_ID |
Get volume by numeric ID. |
delete volume_ID |
Delete volume. |
snapshot volume_ID |
Snapshot volume. |
The doctl compute volume-action
command lets you trigger actions for a volume, including attaching volumes to and detaching volumes from Droplets.
doctl compute volume-action subcommand |
Notes |
---|---|
attach volume_id droplet_id |
Attach a volume to a Droplet. |
detach volume_id droplet_id |
Detach a volume from a Droplet. |
resize volume_id --region volume_region --size new_size |
Resize a volume. |
The doctl compute load-balancer
command can be used to create, delete, or get information about DigitalOcean’s Load Balancers. For more information about this feature, read our Introduction to DigitalOcean Load Balancers.
doctl compute load-balancer subcommand |
Notes |
---|---|
list |
List load balancers. |
create --name lb_name --region lb_region --tag-name tag_name --forwarding-rules forwarding_rule |
Create a Load Balancer. The name, region, a tag or list of Droplet IDs, and at least one forwarding rule are mandatory. |
update --name lb_name --region lb_region --tag-name tag_name --forwarding-rules forwarding_rule |
Create a Load Balancer. The name, region, a tag or list of Droplet IDs, and at least one forwarding rule are mandatory. |
get lb_ID |
Get a load balancer. |
delete lb_ID |
Delete a load balancer. |
add-droplets lb_ID --droplet-ids droplet_ID |
Add Droplets to a load balancer. |
remove-droplets lb_ID --droplet-ids droplet_ID |
Remove Droplets from a load balancer. |
add-forwarding-rules lb_ID --forwarding-rules forwarding_rule |
Add forwarding rules to a load balancer. |
remove-forwarding-rules lb_ID --forwarding-rules forwarding_rule |
Remove forwarding rules from a load balancer. |
When used as an argument to doctl
, forwarding rules should be expressed like: entry_protocol:protocol,entry_port:port,target_protocol:protocol,target_port:port
.
The doctl compute certificate
subcommand allows you to upload and manage SSL certificates, private keys, and certificate chains.
doctl compute certificate subcommand |
Notes |
---|---|
list |
List all Certificates. |
get certificate_id |
Get a Certificate by ID. |
create --name certificate_name --leaf-certificate-path leaf_certificate_path |
Create a Certificate. Name and Leaf Certificate Path are mandatory. |
delete certificate_id |
Delete a Certificate by ID. |
The doctl compute snapshot
command can be used to list, delete, or get information about Droplet and Volume Snapshots.
doctl compute snapshot subcommand |
Notes |
---|---|
list |
List all Snapshots. |
get snapshot_ID |
Get a Snapshot. |
delete snapshot_ID |
Delete a Snapshot. |
To create a new Snapshot, you need to use the appropriate command under the relevant resource command tree. For example:
doctl compute droplet-action snapshot droplet_ID
creates a Snapshot from a Droplet.doctl compute volume snapshot volume_ID
creates a Snapshot from a Volume.The doctl compute image
command allows you to manage all images, including distribution images, application images, and user-created images such as backups and snapshots. We recommend using the snapshot
command for managing snapshots because it provides more detail, has delete functionality, and supports Block Storage snapshots.
doctl compute image subcommand |
Notes |
---|---|
list --public |
List all images. |
list-distribution --public |
List all available distribution images. |
list-application --public |
List all available One-Click Applications. |
list-user |
List all user-created images. |
get image_id |
Get an Image by ID. |
update image_id --name image_name |
Update Image’s name. Name is mandatory. |
delete image_id |
Delete an Image by ID. |
The doctl compute image-action
command allows you to transfer images and get details about actions invoked on images.
doctl compute image-action subcommand |
Notes |
---|---|
get image_id --action-id action_id |
Get an Action for Image by its ID. Action ID is mandatory. |
transfer image_id --region region |
Transfer an Image to the another region. Image ID and region are mandatory. |
The doctl compute firewall
command lets you create and manage Firewalls, including creating and maintaining rules. For more about information about administering Firewalls using doctl
, check out the How To Secure Web Server Infrastructure With DigitalOcean Cloud Firewalls Using Doctl tutorial.
doctl compute firewall command |
Notes |
---|---|
list |
List all Firewalls. |
list-by-droplet droplet_id |
List all Firewalls by Droplet’s numeric ID. |
create --name firewall_name --inbound-rules inbound_rules --outbound-rules outbound_rules |
Create a Firewall. The name and at least an inbound or outbound rule are mandatory. |
update firewall_id --name firewall_name --inbound-rules inbound_rules --outbound-rules outbound_rules |
Update a Firewall. The numeric ID, name and at least an inbound or outbound rule are mandatory. |
get firewall_id |
Get a Firewall by its numeric ID. |
delete firewall_id |
Delete a Firewall by numeric ID. |
add-droplets firewall_id --droplet-ids droplet_IDs |
Add Droplets by their numeric ID to the Firewall. |
remove-droplets firewall_id --droplet-ids droplet_IDs |
Remove Droplets from the Firewall by their numeric IDs. |
add-tags firewall_id --tag-names tags |
Add Tags to the Firewall. |
remove-tags firewall_id --tag-names tags |
Remove Tags from the Firewall. |
add-rules firewall_id --inbound-rules inbound_rules --outbound-rules outbound_rules |
Add inbound or outbound rules to the Firewall. |
remove-rules firewall_id --inbound-rules inbound_rules --outbound-rules outbound_rules |
Remove inbound or outbound rules to the Firewall. |
When used as an argument to doctl
, inbound or outbound rules should be expressed like: protocol:protocol,ports:ports,droplet_id:droplet-id
.
Tags are used to apply custom labels to resources, allowing you to easily filter them. You can learn more about Tags in the How To Tag DigitalOcean Droplets tutorial.
doctl compute tag subcommand |
Notes |
---|---|
create tag_name |
Create a Tag. |
get tag_name |
Get a Tag by name. |
list |
List all Tags. |
delete tag_name |
Delete a Tag by name. |
The DigitalOcean system logs a history of the actions taken on your Droplets, Floating IPs, and other resources. You can access this data with the doctl compute action
command:
- doctl compute action list
You can see actions for a specific Droplet like so:
- doctl compute droplet actions droplet_id
You can discover basic details about your account, such as your configured e-mail address and Droplet limit:
- doctl account get
Because API requests are rate-limited, it may be helpful to see how many requests you’ve made recently, and when the limit is due to reset:
- doctl account ratelimit
The doctl
utility is a helpful tool for managing Droplets and other resources at the command line. It can greatly reduce the amount of manual interaction with web-based interfaces needed for daily development and administrative tasks.
In addition to learning about the underlying API, you may want to explore libraries which wrap the API for popular programming languages, and tools such as Ansible for automating system-level tasks.
]]>Kubernetes ingresses make it easy to expose web services to the internet. When it comes to private services, however, you will likely want to limit who can access them. oauth2_proxy can serve as a barrier between the public internet and private services. oauth2_proxy is a reverse proxy and server that provides authentication using different providers, such as GitHub, and validates users by their email address or other properties.
In this tutorial you’ll use oauth2_proxy with GitHub to protect your services. When you’re done, you will have an authorization system that looks like the one in the following diagram:
To complete this tutorial, you’ll need:
After following the tutorial linked in the Prerequisites section, you will have two web services running on your cluster: echo1
and echo2
. You will also have one ingress that maps echo1.your_domain
and echo2.your_domain
to their corresponding services.
In this tutorial, we will use the following conventions:
.int.your_domain
subdomain, like service.int.your_domain
. Grouping private services under one subdomain is ideal because the authentication cookie will be shared across all *.int.your_domain
subdomains.auth.int.your_domain
.Note: Be sure to replace your_domain
with your own domain name wherever it appears in this tutorial.
To start, update the existing ingress definition to move the echo1
and echo2
services under .int.your_domain
. Open echo_ingress.yaml
in your text editor so you can change the domains:
- nano echo_ingress.yaml
Rename all instances of echo1.your_domain
to echo1.int.your_domain
, and replace all instances of echo2.your_domain
with echo2.int.your_domain
:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: echo-ingress
annotations:
kubernetes.io/ingress.class: nginx
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
spec:
tls:
- hosts:
- echo1.int.your_domain
- echo2.int.your_domain
secretName: letsencrypt-prod
rules:
- host: echo1.int.your_domain
http:
paths:
- backend:
serviceName: echo1
servicePort: 80
- host: echo2.int.your_domain
http:
paths:
- backend:
serviceName: echo2
servicePort: 80
Save the file and apply the changes:
- kubectl apply -f echo_ingress.yaml
This will update the TLS certificates for your echo1
and echo2
services as well.
Now update your DNS configuration to reflect the changes you made. First, look up the IP address of your Nginx ingress by running the following command to print its details:
- kubectl get svc --namespace=ingress-nginx
You will see the IP address under EXTERNAL-IP
in the output:
OutputNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx LoadBalancer 10.245.247.67 203.0.113.0 80:32486/TCP,443:32096/TCP 20h
Copy the external IP address to your clipboard. Browse to your DNS management service and locate the A records for echo1-2.your_domain
to point to that external IP address. If you are using DigitalOcean to manage your DNS records, see How to Manage DNS Records for instructions.
Delete the records for echo1
and echo2
. Add a new A
record for the hostname *.int.your_domain
and point it to the External IP address of the ingress.
Now any request to any subdomain under *.int.your_domain
will be routed to the Nginx ingress, so you can use these subdomains within your cluster.
Next you’ll configure GitHub as your login provider.
oauth2_proxy supports various login providers. In this tutorial, you will use the GitHub provider. To get started, create a new GitHub OAuth App.
In the OAuth Apps tab of the Developer settings page of your account, click the New OAuth App button.
The Application name and Homepage URL fields can be anything you want. In the Authorization callback URL field, enter https://auth.int.your_domain/oauth2/callback
.
After registering the application, you will receive a Client ID and Secret. Note the two as you will need them in the next step.
Now that you’ve created a GitHub OAuth application, you can install and configure oauth2_proxy.
You’ll use Helm to install oauth2_proxy onto the cluster. First, you’ll create a Kubernetes secret to hold the GitHub application’s Client ID and Secret, as well as an encryption secret for browser cookies set by oauth2_proxy.
Run the following command to generate a secure cookie secret:
- python -c 'import os,base64; print base64.b64encode(os.urandom(16))'
Copy the result to your clipboard
Then, create the Kubernetes secret, substituting the highlighted values for your cookie secret, your GitHub client ID, and your GitHub secret key:
- kubectl -n default create secret generic oauth2-proxy-creds \
- --from-literal=cookie-secret=YOUR_COOKIE_SECRET \
- --from-literal=client-id=YOUR_GITHUB_CLIENT_ID \
- --from-literal=client-secret=YOUR_GITHUB_SECRET
You’ll see the following output:
Outputsecret/oauth2-proxy-creds created
Next, create a new file named oauth2-proxy-config.yaml
which will contain the configuration for oauth2_proxy
:
- nano oauth2-proxy-config.yaml
The values you’ll set in this file will override the Helm chart’s defaults. Add the following code to the file:
config:
existingSecret: oauth2-proxy-creds
extraArgs:
whitelist-domain: .int.your_domain
cookie-domain: .int.your_domain
provider: github
authenticatedEmailsFile:
enabled: true
restricted_access: |-
allowed@user1.com
allowed@user2.com
ingress:
enabled: true
path: /
hosts:
- auth.int.your_domain
annotations:
kubernetes.io/ingress.class: nginx
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
tls:
- secretName: oauth2-proxy-https-cert
hosts:
- auth.int.your_domain
This code does the following:
auth.int.your_domain
with a TLS certificate from Let’s Encrypt.Now that you have the secret and configuration file ready, you can install oauth2_proxy
. Run the following command:
- helm repo update \
- && helm upgrade oauth2-proxy --install stable/oauth2-proxy \
- --reuse-values \
- --values oauth2-proxy-config.yaml
It might take a few minutes for the Let’s Encrypt certificate to be issued and installed.
To test that the deployment was successful, browse to https://auth.int.your_domain
. You’ll see a page that prompts you to log in with GitHub.
With oauth2_proxy set up and running, all that is left is to require authentication on your services.
In order to protect a service, configure its Nginx ingress to enforce authentication via oauth2_proxy. Nginx and nginx-ingress support this configuration natively, so you only need to add a couple of annotations to the ingress definition.
Let’s protect the echo1
and echo2
services that you set up in the prerequisite tutorial. Open echo_ingress.yaml
in your editor:
- nano echo_ingress.yaml
Add these two additional annotations to the file to require authentication:
annotations:
kubernetes.io/ingress.class: nginx
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/auth-url: "https://auth.int.your_domain/oauth2/auth"
nginx.ingress.kubernetes.io/auth-signin: "https://auth.int.your_domain/oauth2/start?rd=https%3A%2F%2F$host$request_uri"
Save the file and apply the changes:
- kubectl apply -f echo_ingress.yaml
Now when you browse to https://echo1.int.your_domain
, you will be asked to log in using GitHub in order to access it. After logging in with a valid account, you will be redirected back to the echo1
service. The same is true for echo2
.
In this tutorial, you set up oauth2_proxy on your Kubernetes cluster and protected a private service behind a GitHub login. For any other services you need to protect, simply follow the instructions outlined in Step 4.
oauth2_proxy supports many different providers other than GitHub. To learn more about different providers, see the official documentation.
Additionally, there are many configuration parameters that you might need to adjust, although the defaults will suit most needs. For a list of parameters, see the Helm chart’s documentation and oauth2_proxy’s documentation.
]]>Download the Complete eBook!
Machine Learning Projects: Python eBook in EPUB format
Machine Learning Projects: Python eBook in PDF format
Machine Learning Projects: Python eBook in Mobi format
As machine learning is increasingly leveraged to find patterns, conduct analysis, and make decisions — sometimes without final input from humans who may be impacted by these findings — it is crucial to invest in bringing more stakeholders into the fold. This book of Python projects in machine learning tries to do just that: to equip the developers of today and tomorrow with tools they can use to better understand, evaluate, and shape machine learning to help ensure that it is serving us all.
This book will set you up with a Python programming environment if you don’t have one already, then provide you with a conceptual understanding of machine learning in the chapter “An Introduction to Machine Learning.” What follows next are three Python machine learning projects. They will help you create a machine learning classifier, build a neural network to recognize handwritten digits, and give you a background in deep reinforcement learning through building a bot for Atari.
These chapters originally appeared as articles on DigitalOcean Community, written by members of the international software developer community. If you are interested in contributing to this knowledge base, consider proposing a tutorial to the Write for DOnations program. DigitalOcean offers payment to authors and provides a matching donation to tech-focused nonprofits.
If you are learning Python or are looking for reference material, you can download our free Python eBook, How To Code in Python 3
For other programming languages and DevOps engineering articles, check out our knowledge base of over 2,100 tutorials.
You can download the eBook in either the EPUB, PDF, or Mobi format by following the links below.
]]>Despite being a commercial failure after its first publication, Herman Melville’s allegorical adventure novel Moby-Dick; or, The Whale is today one of the most popular and influential novels in the American canon. Artists as diverse as William Faulkner, Ralph Ellison, and Bob Dylan have acknowledged the novel’s impact on their work, and one can spot references to it in films, television, music, and, of course, open-source projects.
In this article, we will analyze several nautically-themed open-source projects and how they pay tribute to Moby-Dick.
Warning: While it isn’t necessary that you read Moby-Dick prior to reading this article, this article does contain a few spoilers. If you haven’t read the novel but would like to, you may want to hold off from reading this article until you’ve finished it.
To follow along with this tutorial, you’ll need:
Docker is an open-source program that performs operating system-level virtualization, also known as containerization. The influence of Moby-Dick is obvious with this project: Docker’s logo and mascot is a whale affectionately known as Moby Dock. However, there are some substantial differences between Moby Dick and Moby Dock.
First, Moby Dock’s species isn’t immediately obvious. It’s clear from the beginning of the novel that Moby Dick is a sperm whale, and while it’s possible that Moby Dock is a sperm whale as well, there are several clues that suggest otherwise:
Another important difference between these Mobys is that Moby Dock is helpfully carrying a few stacks of containers; Moby Dick would never be so accommodating. In fact, one can easily imagine Moby Dick going out of his way to knock over such a neatly organized pile of shipping containers. Perhaps Moby Dock is meant to be seen as a warmer, friendlier cousin of Moby Dick. After all, it’s probably bad marketing to associate one’s product with a ferocious leviathan bent on destroying everything in its path.
OpenFaaS is an open-source project that aims to make serverless functions simple through the use of Docker containers, allowing users to run complex infrastructures with far greater flexibility and without the fear of vendor lock-in.
The OpenFaaS logo focuses entirely on a whale’s tail, which is significant because Melville dedicates an entire chapter to describing the tails of sperm whales. In it, Ishmael reveals his deep appreciation of whales’ tails:
Such is the subtle elasticity of [the tail], that whether wielded in sport, or in earnest, or in anger, whatever be the mood it be in, its flexions are invariably marked by exceeding grace. Therein no fairy’s arm can transcend it.
The OpenFaaS whale is shown to be peaking its flukes, presumably as it is about to dive. In the same chapter, Ishmael opines that “excepting the sublime breach…this peaking of the whale’s flukes is perhaps the grandest sight to be seen in all animated nature.” Perhaps the OpenFaaS team chose a whale’s tail as their logo to convey the grace and power that OpenFaaS brings to managing functions. It could even be that the whale is “diving in” to the realm of functions as a service.
Because OpenFaaS is closely related to Docker, it’s obvious why the project’s logo also features a whale. However, are these supposed to be the same whale? Let us not forget that Moby Dick was believed to be “ubiquitous”, with sailors swearing up and down that they had encountered him “in opposite latitudes at one and the same instant of time.” This may be a clue that Moby Dock and the OpenFaaS whale are indeed one and the same.
Perhaps in choosing this logo the OpenFaaS team was trying to signal their hope that the framework would become ubiquitous in future software projects. Interestingly, while an omnipresent whale may strike fear in the hearts of whalers, software is generally seen as safer and more secure if it’s widely used. The OpenFaaS team should be thankful that coders are generally less superstitious than whalers.
Kubernetes is an open-source container orchestration system that helps to automate the deployment, scaling, and management of applications. The name “Kubernetes” comes from the Greek word “κυβερνήτης,” which translates to English as “captain” or “helmsman.” Appropriately, its logo consists of a ship’s wheel, or helm, conveying the control and steadiness required to manage complex container orchestration with ease.
Curiously, the Pequod doesn’t have a wheel; instead, it has a tiller made out of a whale’s jawbone. This is seen by some readers as underscoring the shared histories of Captain Ahab and the ship, as Ahab lost his leg to the great white whale and replaced it with a whalebone prosthesis.
Though a helm or tiller can convey steadiness and control, as the Kubernetes logo designers intended, Moby-Dick shows us the deeper questions that the project maintainers might have brushed aside. Who is at the helm when it comes to Kubernetes? Even more, who is at the helm in our everyday lives? Do we drive software, or does software drive us? Of all these things the helm is the symbol.
MySQL is the world’s most widely deployed open-source database management system (DBMS). MySQL’s logo features the outline of a dolphin, affectionately known as Sakila.
While dolphins aren’t prominently featured in the plot of Moby-Dick, Melville discusses them at length in one of the books famous pseudoscientific asides. In Chapter 32, “Cetology,” Ishmael refers to dolphins as “Huzza Porpoises,” so called because sailors see them as an omen of good luck:
Their appearance is generally hailed with delight by the mariner…. If you yourself can withstand three cheers at beholding these vivacious fish, then heaven help ye; the spirit of godly gamesomeness is not in ye.
Mayhaps the MySQL developers chose a dolphin to represent their DBMS to impart this same sense of hopeful joy to those who use it. By associating the database with a dolphin, they hope users will see it as being similarly fast, agile, and fun-loving. After all, who doesn’t have fun running correlated subqueries?
MariaDB is a community-supported fork of MySQL, as indicated by its similarly nautical logo. Both the MariaDB and MySQL logos include the respective RDBMS’s name and feature an aquatic animal: in MariaDB’s case, this animal is a pinniped.
Interestingly, there’s some confusion about what kind of animal is depicted in the MariaDB logo. According to the project’s trademarks page, the animal in the logo is a sea lion. However, some members of the MariaDB community see it as a seal. MariaDB’s official sources are fairly consistent in referring to their mascot as a sea lion, though not always. Certainly, the mascot’s shape does seem to more closely resemble that of a sea lion, but it’s also missing the telltale ears which would distinguish it as such.
The idea that human perception is inherently biased and unreliable runs as a theme throughout the novel. Perhaps by keeping the pinniped’s species vague, the MariaDB team is making a Melvillian comment on how truth isn’t always obvious and, in some cases, can never be known for certain. Is it a seal or a sea lion? Is Moby Dick real or imagined? Is Vim or Emacs the superior text editor? Riddles like these abound throughout the world we live in, which, like a magician’s glass, to each and every man in turn but mirrors back his own mysterious self. Great pains, small gains for those who ask the world to solve them.
Of course, it’s also possible that the logo is simply meant to represent a sea lion. Perhaps when the MariaDB team asked the designer to draw ears, they responded “I would prefer not to.”
Clearly, Melville’s influence extends far beyond the realm of literature, and well into the world of open-source technology. As this article has highlighted, these five projects (and likely many more) pay homage to his great whaling tale through subtle references in their names and logos, as well as how they challenge our perceptions of truth and human nature.
We hope that by reading this article, you’ll go on to create your own Melville-inspired, nautically-themed, open-source project. Here are a few ideas to help you get started:
Note: Some readers may be wondering why this article hasn’t yet mentioned DigitalOcean’s own Sammy the Shark. The simple reason is that Sammy has little in common with the sharks depicted in Moby-Dick. Throughout the novel, sharks are depicted as ravenous beasts dominated by instinct. Melville’s sharks eat anything and everything in their path, and are violent, dangerous creatures who pose a serious risk to the crew of the Pequod (though not as great a risk as whales, apparently).
Clearly, Melville never encountered a shark like Sammy. After all, Sammy is a vegetarian, and a very friendly one at that!
O recurso Custom Images ou Imagens Personalizadas da DigitalOcean lhe permite trazer seu disco virtual personalizado de Linux e Unix-like de um ambiente local ou de outra plataforma de nuvem para a DigitalOcean e utilizá-lo para iniciar Droplets na DigitalOcean.
Como descrito na documentação do Custom Images, os seguintes tipos de imagens são suportados nativamente pela ferramenta de upload do Custom Images:
Embora imagens com formato ISO não sejam oficialmente suportadas, você pode aprender como criar e carregar uma imagem compatível usando o VirtualBox seguindo o tutorial How to Create a DigitalOcean Droplet from an Ubuntu ISO Format Image.
Se você ainda não tem uma imagem compatível para carregar na DigitalOcean, você pode criar e comprimir uma imagem de disco do seu sistema Unix-like ou Linux, desde que ela tenha o software e os drivers de pré-requisitos instalados.
Vamos começar assegurando que sua imagem atende ao requisitos do Custom Images. Para fazer isso, vamos configurar o sistema e instalar alguns pré-requisitos de software. Depois, vamos criar a imagem utilizando o utilitário de linha de comando dd
e comprimí-la usando o gzip
. Na sequência, vamos fazer o upload desse arquivo de imagem compactado para o DigitalOcean Spaces, de onde podemos importá-lo como uma Imagem Personalizada. Finalmente, vamos inicializar um droplet usando a imagem enviada
Se possível, você deve usar uma das imagens fornecidas pela DigitalOcean como base, ou uma imagem de nuvem oficial fornecida pela distribuição como o Ubuntu Cloud. Então você pode instalar softwares e aplicaçoes em cima dessa imagem de base para fazer uma nova imagem usando ferramentas como o Packer e o VirtualBox. Muitos provedores de nuvem e ambientes de virtualização também fornecem ferramentas para exportar discos virtuais para um dos formatos compatíveis listados acima, assim, se possível, você deve usá-las para simplificar o processo de importação. Nos casos em que você precisa criar manualmente uma imagem de disco do seu sistema você pode seguir as instruções nesse guia. Observe que essas instruções só foram testadas com um sistema Ubuntu 18.04 e as etapas podem variar dependendo do sistema operacional e da configuração do seu servidor.
Antes de começar com este tutorial, você deve ter o seguinte disponível para você:
Um sistema Linux ou Unix-like que atenda a todos os requisitos listados na documentação de produto do Custom Images. Por exemplo, seu disco de boot deve ter:
Um tamanho máximo de 100GB
Um tabela de partição MBR ou GPT com um gerenciador de boot grub
Drivers do VirtIO instalados
Um usuário não-root com privilégios administrativos disponível para você no sistema que você está fazendo imagem. Para criar um novo usuário e conceder a ele privilégios administrativos no Ubuntu 18.04, siga nosso tutorial de Configuração Inicial de Servidor com Ubuntu 18.04. Para aprender como fazer isto no Debian 9, consulte Configuração Inicial de Servidor com Debian 9.
Um dispositivo de armazenamento adicional usado para armazenar a imagem de disco criada neste guia, preferivelmente tão grande quanto o disco que está sendo copiado. Isso pode ser um volume de armazenamento em blocos anexado, um drive externo USB, um espaço em disco adicional, etc.
Um Space na DigitalOcean e o utilitário de transferência de arquivos s3cmd
configurado para uso com o seu Space. Para aprender como criar um Space, consulte o Guia Rápido do Spaces. Para aprender como configurar o s3cmd
para uso com o seu Space, consulte o Guia de Configuração do s3cmd 2.x.
Para começar, vamos instalar o pacote de inicialização do cloud-Init. O cloud-init é um conjunto de scripts que executam no boot para configurar certas propriedades da instância de nuvem como a localidade padrão, hostname, chaves SSH e dispositivos de rede.
Os passos para a instalação do cloud-init vão variar dependendo do sistema operacional que você instalou. Em geral, o pacote cloud-init
deve estar disponível no gerenciador de pacotes do seu SO, assim se você não estiver utilizando uma distribuição baseada no Debian, você deve substituir o apt
nos seguintes passos pelo seu comando do gerenciador de pacotes específico da distribuição.
cloud-init
Neste guia, vamos utilizar um servidor Ubuntu 18.04 e então usaremos o apt
para baixar e instalar o pacote cloud-init
. Observe que o cloud-init
pode já estar instalado em seu sistema (algumas distribuições Linux instalam o cloud-init
por padrão). Para verificar, efetue o login em seu servidor e execute o seguinte comando:
- cloud-init
Se você vir a seguinte saída, o cloud-init
já foi instalado no seu servidor e você pode continuar configurando-o para uso com a DigitalOcean:
Outputusage: /usr/bin/cloud-init [-h] [--version] [--file FILES] [--debug] [--force]
{init,modules,single,query,dhclient-hook,features,analyze,devel,collect-logs,clean,status}
...
/usr/bin/cloud-init: error: the following arguments are required: subcommand
Se, em vez disso, você vir o seguinte, você precisa instalar o cloud-init
:
Outputcloud-init: command not found
Para instalar o cloud-init
, atualize o índice de pacotes e em seguida instale o pacote usando o apt
:
- sudo apt update
- sudo apt install cloud-init
Agora que instalamos o cloud-init
, vamos configurá-lo para uso com a DigitalOcean, assegurando que ele utilize o datasource ConfigDrive
. O datasource do cloud-init determina como o cloud-init
procurará e atualizará a configuração e os metadados da instância. Os Droplets da DigitalOcean usam o datasource ConfigDrive
, por isso, vamos verificar se ele vem em primeiro lugar na lista de datasources que o cloud-init
pesquisa sempre que o Droplet inicializa.
cloud-init
Por padrão, no Ubuntu 18.04, o cloud-init
configura a si mesmo para utilizar o datasource NoCloud
primeiro. Isso irá causar problemas ao executar a imagem na DigitalOcean, por isso precisamos reconfigurar o cloud-init
para utilizar o datasource ConfigDdrive
e garantir que o cloud-init
execute novamente quando a imagem é lançada na DigitalOcean.
A partir da linha de comando, navegue até o diretório /etc/cloud/cloud.cfg.d
:
- cd /etc/cloud/cloud.cfg.d
Use o comando ls
para listar os arquivos de configuração do cloud-init
presentes dentro do diretório:
- ls
Output05_logging.cfg 50-curtin-networking.cfg 90_dpkg.cfg curtin-preserve-sources.cfg README
Dependendo da sua instalação, alguns desses arquivos podem não estar presentes. Se presente, exclua o arquivo 50-curtin-networking.cfg
, que configura as interfaces de rede para seu servidor Ubuntu. Quando a imagem é lançada na DigitalOcean, o cloud-init
irá executar e reconfigurar estas interfaces automaticamente, portanto esse arquivo não é necessário. Se esse arquivo não for excluído, o Droplet da DigitalOcean criado a partir dessa imagem Ubuntu terá suas interfaces configuradas incorretamente e não serão acessíveis pela internet:
- sudo rm 50-curtin-networking.cfg
Em seguida, vamos executar dpkg-reconfigure cloud-init
para remover o datasource NoCloud
, garantindo que o cloud-init procure e localize o datasource ConfigDrive
usado na DigitalOcean:
- sudo dpkg-reconfigure cloud-init
Você deve ver o seguinte menu gráfico:
O datasource NoCloud
está inicialmente destacado. Pressione ESPAÇO
para desmarcá-lo e, em seguida, pressione ENTER
.
Finalmente, navegue até /etc/netplan
:
- cd /etc/netplan
Remova o arquivo 50-cloud-init.yaml
, que foi gerado a partir do arquivo de rede cloud-init
que removemos anteriormente:
- sudo rm 50-cloud-init.yaml
A etapa final é garantir que limpemos a configuração da execução inicial do cloud-init
para que ela seja executada novamente quando a imagem for lançada na DigitalOcean.
Para fazer isso, execute cloud-init clean
:
- sudo cloud-init clean
Neste ponto você instalou e configurou o cloud-init
para uso com a DigitalOcean. Agora você pode seguir para ativar o acesso SSH ao seu droplet.
Depois que você instalou e configurou o cloud-init
, o próximo passo é assegurar que você tenha um usuário e senha de administrador não-root disponível para você em sua máquina, conforme descrito nos pré-requisitos. Este passo é essencial para diagnosticar quaisquer erros que possam surgir após o upload da sua imagem e o lançamento do seu droplet. Se uma configuração de rede preexistente ou uma configuração incorreta do cloud-init
tornar o seu Droplet inacessível na rede, você pode utilizar esse usuário em combinação ao Console do Droplet da DigitalOcean para acessar seu sistema e diagnosticar quaisquer problemas que possam ter surgido.
Depois que você tiver configurado seu usuário administrativo não-root, a etapa final é garantir que você tenha um servidor SSH instalado e executando. O SSH geralmente vem pré-instalado em muitas distribuições populares do Linux. O procedimento para verificar se um processo está executando irá variar dependendo do sistema operacional do seu servidor. Se você não tiver certeza de como fazer isso, consulte a documentação do seu sistema operacional sobre o gerenciamento de serviços. No Ubuntu, você pode verificar que o SSH está funcionando utilizando este comando:
- sudo service ssh status
Você deve ver a seguinte saída:
Output● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2018-10-22 19:59:38 UTC; 8 days 1h ago
Docs: man:sshd(8)
man:sshd_config(5)
Process: 1092 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Main PID: 1115 (sshd)
Tasks: 1 (limit: 4915)
Memory: 9.7M
CGroup: /system.slice/ssh.service
└─1115 /usr/sbin/sshd -D
Se o SSH não estiver em execução, você pode instalá-lo usando o apt
(nas distribuições baseadas em Debian):
- sudo apt install openssh-server
Por padrão, o servidor SSH vai iniciar no boot a menos que esteja configurado de outra forma. Isso é desejável ao executar o sistema na nuvem, já que a DigitalOcean pode copiar automaticamente sua chave pública e conceder acesso SSH imediato ao seu Droplet após a criação.
Depois que você criou um usuário administrativo não-root, ativou o SSH, e instalou o cloud-init, você está pronto para continuar criando uma imagem do seu disco de boot.
Neste passo, vamos criar uma imagem de disco de formato RAW usando o utilitário de linha de comando dd
, e compactá-lo usando o gzip
. Vamos então carregar a imagem para o Spaces da DigitalOcean usando o s3cmd
.
Para começar, efetue login em seu servidor, e inspecione o arranjo de dispositivos de bloco para o seu sistema usando lsblk
:
- lsblk
Você deverá ver algo como o seguinte:
OutputNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 12.7M 1 loop /snap/amazon-ssm-agent/495
loop1 7:1 0 87.9M 1 loop /snap/core/5328
vda 252:0 0 25G 0 disk
└─vda1 252:1 0 25G 0 part /
vdb 252:16 0 420K 1 disk
Nesse caso, observamos que nosso disco principal de boot é o /dev/vda
, um disco de 25GB, e a partição primária, montada no /
, é a /dev/vda1
. Na maioria dos casos o disco contendo a partição montada no /
será o disco de origem para a imagem. Vamos usar o dd
para criar uma imagem do /dev/vda
.
Neste ponto, você deve decidir onde você quer armazenar a imagem de disco. Uma opção é anexar outro dispositivo de armazenamento em bloco, de preferência tão grande quanto o disco que você está fazendo a imagem. Em seguida, você pode salvar a imagem neste disco temporário anexado e enviá-la para o Spaces da DigitalOcean.
Se você tem acesso físico ao servidor, você pode adicionar um drive adicional à máquina ou anexar outro dispositivo de armazenamento, como um disco USB externo.
Outra opção, que iremos demonstrar nesse guia, é copiar a imagem por SSH para uma máquina local, a partir da qual você pode enviá-la para o Spaces.
Independentemente do método escolhido, verifique se o dispositivo de armazenamento no qual você salvou a imagem compactada tem espaço livre suficiente. Se o disco que você está fazendo imagem está quase vazio, você pode esperar que o arquivo de imagem compactado seja significativamente menor que o disco original.
Atenção: Antes de rodar o seguinte comando dd
, certifique-se de que todos os aplicativos críticos tenham sido parados e seu sistema esteja o mais folgado possível. Copiar um disco sendo usado ativamente pode resultar em alguns arquivos corrompidos, portanto, certifique-se de interromper qualquer operação que use muitos dados e encerre o máximo possível de aplicativos em execução.
A sintaxe para o comando dd
que vamos executar é a seguinte:
- dd if=/dev/vda bs=4M conv=sparse | pv -s 25G | gzip > /mnt/tmp_disk/ubuntu.gz
Neste caso, estamos selecionando /dev/vda
como o disco de entrada para se fazer imagem, e definindo o tamanho dos blocos de entrada/saída para 4MB (sendo que o padrão é 512 bytes). Isso geralmente acelera um pouco as coisas. Além disso, estamos usando a flag conv=sparse
para minimizar o tamanho do arquivo de saída pulando o espaço vazio. Para aprender mais sobre parâmetros do dd
, consulte a sua manpage.
Em seguida, fazemos um pipe da saída para o utilitário de visualização de pipe pv
para que possamos acompanhar o progresso da transferência visualmente (esse pipe é opcional e requer a instalação do pv
usando o gerenciador de pacotes). Se você sabe o tamanho do disco inicial (nesse caso é 25GB), você pode adicionar -s 25G
ao pipe do pv
para ter uma estimativa de quando a transferência será concluída.
Fazemos então um pipe de tudo isso para o gzip
e salvamos em um arquivo chamado ubuntu.gz
no volume de armazenamento de bloco temporário que anexamos ao servidor. Substitua /mnt/tmp_disk
com o caminho para o dispositivo de armazenamento externo que você anexou ao seu servidor.
Em vez de provisionar armazenamento adicional para sua máquina remota, você também pode executar a cópia via SSH se tiver espaço em disco suficiente disponível na sua máquina local. Observe que, dependendo da largura de banda disponível para você, isso pode ser lento e você pode incorrer em custos adicionais para a transferência de dados pela rede.
Para copiar e compactar o disco via SSH, execute o seguinte comando em sua máquina local:
- ssh usuário_remoto@ip_do_seu_servidor "sudo dd if=/dev/vda bs=4M conv=sparse | gzip -1 -" | dd of=ubuntu.gz
Neste caso, estamos fazendo SSH para o nosso servidor remoto executando o comando dd
lá, e fazendo um pipe da saída para o gzip
. Em seguida, transferimos a saída do gzip
pela rede e a salvamos localmente como ubuntu.gz
. Certifique-se de que você tenha o utilitário dd
disponível em sua máquina local antes de executar esse comando:
- which dd
Output/bin/dd
Crie o arquivo de imagem compactado usando qualquer um dos métodos acima. Isso pode levar várias horas, dependendo do tamanho do disco que você está criando e do método que você está usando para criar a imagem.
Depois de criar o arquivo de imagem compactado, você pode passar a enviá-lo para seus Spaces da DigitalOcean usando o s3cmd
.
Conforme descrito nos pré-requisitos, você deve ter o s3cmd
instalado e configurado para uso com seu Space da DigitalOcean na máquina que contém sua imagem compactada.
Localize o arquivo de imagem compactado e faça o upload dele para seu Space usando o s3cmd
:
Nota: Você deve substituir your_space_name
pelo nome do seu Space e não a sua URL. Por exemplo, se a URL do seu Space é https://example-space-name.nyc3.digitaloceanspaces.com
, então o nome do seu Space é example-space-name
.
- s3cmd put /caminho_da_imagem/ubuntu.gz s3://your_space_name
Quando o upload estiver concluído, navegue até seu Space usando o Painel de Controle da DigitalOcean, e localize a imagem na lista de arquivos. Tornaremos a imagem publicamente acessível temporariamente para que o Custom Images possa acessá-la e salvar uma cópia.
À direita da lista de imagens, clique no menu suspenso More e, em seguida, clique em Manage Permissions:
Em seguida, clique no botão de opção ao lado de Public e clique em Update para tornar a imagem publicamente acessível.
Atenção: Sua imagem estará temporariamente acessível publicamente para qualquer pessoa com o caminho do seu Space durante este processo. Se você gostaria de evitar tornar sua imagem temporariamente pública, você pode criar sua Imagem Personalizada usando a API da DigitalOcean. Certifique-se de definir sua imagem como Private usando o procedimento acima depois que sua imagem for transferida com sucesso para o Custom Images.
Busque a URL do Spaces para sua imagem passando o mouse sobre o nome da imagem no Painel de controle, e clique em Copy URL na janela que aparece.
Agora, navegue para Images na barra de navegação à esquerda, e depois para Custom Images.
A partir daqui, envie sua imagem usando esta URL, conforme detalhado na Documentação de Produto do Custom Images.
Você pode então criar um Droplet a partir desta imagem. Observe que você precisa adicionar uma chave SSH ao Droplet na criação. Para aprender como fazer isso, consulte How to Add SSH Keys to Droplets.
Uma vez que o seu Droplet inicializa, se você puder fazer SSH nele, você lançou com sucesso a sua Imagem Personalizada como um Droplet da DigitalOcean.
Se você tentar fazer SSH no seu Droplet e não conseguir conectar, certifique-se de que sua imagem atenda aos requisitos listados e tenha o cloud-init
e o SSH instalados e configurados corretamente. Se você ainda não conseguir acessar o Droplet, você pode tentar utilizar o Console do Droplet da DigitalOcean e o usuário não-root que você criou anteriormente para explorar o sistema e fazer o debug das configurações de sua rede, do cloud-init
e do SSH. Outra maneira de fazer o debug de sua imagem é usar uma ferramenta de virtualização como o Virtualbox para inicializar sua imagem de disco dentro de uma máquina virtual, e fazer o debug da configuração do seu sistema a partir da VM.
Neste guia, você aprendeu como criar uma imagem de disco de um sistema Ubuntu 18.04 usando o utilitário de linha de comando dd
e fazer o upload dela para a DigitalOcean como uma Custom Image ou Imagem Personalizada a partir da qual você pode lançar Droplets.
As etapas neste guia podem variar dependendo do seu sistema operacional, do hardware existente e da configuração do kernel, mas, em geral, as imagens criadas a partir de distribuições populares do Linux devem funcionar usando esse método. Certifique-se de seguir cuidadosamente as etapas de instalação e configuração do cloud-init
e de garantir que o sistema atenda a todos os requisitos listados na seção pré-requisitos acima.
Para aprender mais sobre Custom Images, consulte a documentação de produto do Custom Images.
Por Hanif Jetha
]]>DigitalOcean’s Custom Images feature allows you to bring your custom Linux and Unix-like virtual disk images from an on-premises environment or another cloud platform to DigitalOcean and use them to start DigitalOcean Droplets.
As described in the Custom Images documentation, the following image types are supported natively by the Custom Images upload tool:
Although ISO format images aren’t officially supported, you can learn how to create and upload a compatible image using VirtualBox by following How to Create a DigitalOcean Droplet from an Ubuntu ISO Format Image.
If you don’t already have a compatible image to upload to DigitalOcean, you can create and compress a disk image of your Unix-like or Linux system, provided it has the prerequisite software and drivers installed.
We’ll begin by ensuring that our image meets the Custom Images requirements. To do this, we’ll configure the system and install some software prerequisites. Then, we’ll create the image using the dd
command-line utility and compress it using gzip
. Following that, we’ll upload this compressed image file to DigitalOcean Spaces, from which we can import it as a Custom Image. Finally, we’ll boot up a Droplet using the uploaded image.
If possible, you should use one of the DigitalOcean-provided images as a base, or an official distribution-provided cloud image like Ubuntu Cloud. You can then install software and applications on top of this base image to bake a new image, using tools like Packer and VirtualBox. Many cloud providers and virtualization environments also provide tools to export virtual disks to one of the compatible formats listed above, so, if possible, you should use these to simplify the import process. In the cases where you need to manually create a disk image of your system, you can follow the instructions in this guide. Note that these instructions have only been tested with an Ubuntu 18.04 system, and steps may vary depending on your server’s OS and configuration.
Before you begin with this tutorial, you should have the following available to you:
A Linux or Unix-like system that meets all of the requirements listed in the Custom Images product documentation. For example, your boot disk must have:
grub
bootloaderA non-root user with administrative privileges available to you on the system you’re imaging. To create a new user and grant it administrative privileges on Ubuntu 18.04, follow our Initial Server Setup with Ubuntu 18.04. To learn how to do this on Debian 9, consult Initial Server Setup with Debian 9.
An additional storage device used to store the disk image created in this guide, preferably as large as the disk being copied. This can be an attached block storage volume, an external USB drive, an additional physical disk, etc.
A DigitalOcean Space and the s3cmd
file transfer utility configured for use with your Space. To learn how to create a Space, consult the Spaces Quickstart. To learn how set up s3cmd
for use with your Space, consult the s3cmd 2.x Setup Guide.
To begin, we will install the cloud-Init initialization package. Cloud-init is a set of scripts that runs at boot to configure certain cloud instance properties like default locale, hostname, SSH keys and network devices.
Steps for installing cloud-init will vary depending on the operating system you have installed. In general, the cloud-init
package should be available in your OS’s package manager, so if you’re not using a Debian-based distribution, you should substitute apt
in the following steps with your distribution-specific package manager command.
cloud-init
In this guide, we’ll use an Ubuntu 18.04 server and so will use apt
to download and install the cloud-init
package. Note that cloud-init
may already be installed on your system (some Linux distributions install cloud-init
by default). To check, log in to your server and run the following command:
- cloud-init
If you see the following output, cloud-init
has already been installed on your server and you can continue on to configuring it for use with DigitalOcean:
Outputusage: /usr/bin/cloud-init [-h] [--version] [--file FILES] [--debug] [--force]
{init,modules,single,query,dhclient-hook,features,analyze,devel,collect-logs,clean,status}
...
/usr/bin/cloud-init: error: the following arguments are required: subcommand
If instead you see the following, you need to install cloud-init
:
Outputcloud-init: command not found
To install cloud-init
, update your package index and then install the package using apt
:
- sudo apt update
- sudo apt install cloud-init
Now that we’ve installed cloud-init
, we’ll configure it for use with DigitalOcean, ensuring that it uses the ConfigDrive
datasource. Cloud-init datasources dictate how cloud-init
will search for and update instance configuration and metadata. DigitalOcean Droplets use the ConfigDrive
datasource, so we will check that it comes first in the list of datasources that cloud-init
searches whenever the Droplet boots.
cloud-init
By default, on Ubuntu 18.04, cloud-init
configures itself to use the NoCloud
datasource first. This will cause problems when running the image on DigitalOcean, so we need to reconfigure cloud-init
to use the ConfigDrive
datasource and ensure that cloud-init
reruns when the image is launched on DigitalOcean.
From the command line, navigate to the /etc/cloud/cloud.cfg.d
directory:
- cd /etc/cloud/cloud.cfg.d
Use the ls
command to list the cloud-init
config files present in the directory:
- ls
Output05_logging.cfg 50-curtin-networking.cfg 90_dpkg.cfg curtin-preserve-sources.cfg README
Depending on your installation, some of these files may not be present. If present, delete the 50-curtin-networking.cfg
file, which configures networking interfaces for your Ubuntu server. When the image is launched on DigitalOcean, cloud-init
will run and reconfigure these interfaces automatically, so this file is not necessary. If this file is not deleted, the DigitalOcean Droplet created from this Ubuntu image will have its interfaces misconfigured and won’t be accessible from the internet:
- sudo rm 50-curtin-networking.cfg
Next, we’ll run dpkg-reconfigure cloud-init
to remove the NoCloud
datasource, ensuring that cloud-init
searches for and finds the ConfigDrive
datasource used on DigitalOcean:
- sudo dpkg-reconfigure cloud-init
You should see the following graphical menu:
The NoCloud
datasource is initially highlighted. Press SPACE
to unselect it, then hit ENTER
.
Finally, navigate to /etc/netplan
:
- cd /etc/netplan
Remove the 50-cloud-init.yaml
file, which was generated from the cloud-init
networking file we removed previously:
- sudo rm 50-cloud-init.yaml
The final step is ensuring that we clean up configuration from the initial cloud-init
run so that it reruns when the image is launched on DigitalOcean.
To do this, run cloud-init clean
:
- sudo cloud-init clean
At this point you’ve installed and configured cloud-init
for use with DigitalOcean. You can now move on to enabling SSH access to your droplet.
Once you’ve installed and configured cloud-init
, the next step is to ensure that you have a non-root admin user and password available to you on your machine, as outlined in the prerequisites. This step is essential to diagnose any errors that may arise after uploading your image and launching your Droplet. If a preexisting network configuration or bad cloud-init
configuration renders your Droplet inaccesible over the network, you can use this user in combination with the DigitalOcean Droplet Console to access your system and diagnose any problems that may have surfaced.
Once you’ve set up your non-root administrative user, the final step is to ensure that you have an SSH server installed and running. SSH often comes preinstalled on many popular Linux distributions. The process for checking whether a service is running will vary depending on your server’s operating system… If you aren’t sure of how to do this, consult your OS’s documentation on managing services. On Ubuntu, you can verify that SSH is up and running using the following command:
- sudo service ssh status
You should see the following output:
Output● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2018-10-22 19:59:38 UTC; 8 days 1h ago
Docs: man:sshd(8)
man:sshd_config(5)
Process: 1092 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Main PID: 1115 (sshd)
Tasks: 1 (limit: 4915)
Memory: 9.7M
CGroup: /system.slice/ssh.service
└─1115 /usr/sbin/sshd -D
If SSH isn’t up and running, you can install it using apt
(on Debian-based distributions):
- sudo apt install openssh-server
By default, the SSH server will start on boot unless configured otherwise. This is desirable when running the system in the cloud, as DigitalOcean can automatically copy in your public key and grant you immediate SSH access to your Droplet after creation.
Once you’ve created a non-root administrative user, enabled SSH, and installed cloud-init, you’re ready to move on to creating an image of your boot disk.
In this step, we’ll create a RAW format disk image using the dd
command-line utility, and compress it using gzip
. We’ll then upload the image to DigitalOcean Spaces using s3cmd
.
To begin, log in to your server, and inspect the block device arrangement for your system using lsblk
:
- lsblk
You should see something like the following:
OutputNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 12.7M 1 loop /snap/amazon-ssm-agent/495
loop1 7:1 0 87.9M 1 loop /snap/core/5328
vda 252:0 0 25G 0 disk
└─vda1 252:1 0 25G 0 part /
vdb 252:16 0 420K 1 disk
In this case, we notice that our main boot disk is /dev/vda
, a 25GB disk, and the primary partition, mounted at /
, is /dev/vda1
. In most cases the disk containing the partition mounted at /
will be the source disk to image. We are going to use dd
to create an image of /dev/vda
.
At this point, you should decide where you want to store the disk image. One option is to attach another block storage device, preferably as large as the disk you are going to image. You can then save the image to this attached temporary disk and upload it to DigitalOcean Spaces.
If you have physical access to the server, you can add an additional drive to the machine or attach another storage device, like an external USB disk.
Another option, which we’ll demonstrate in this guide, is copying the image over SSH to a local machine, from which you can upload it to Spaces.
No matter which method you choose to follow, ensure that the storage device to which you save the compressed image has enough free space. If the disk you’re imaging is mostly empty, you can expect the compressed image file to be significantly smaller than the original disk.
Warning: Before running the following dd
command, ensure that any critical applications have been stopped and your system is as quiet as possible. Copying an actively-used disk may result in some corrupted files, so be sure to halt any data-intensive operations and shut down as many running applications as possible.
The syntax for the dd
command we’re going to execute looks as follows:
- dd if=/dev/vda bs=4M conv=sparse | pv -s 25G | gzip > /mnt/tmp_disk/ubuntu.gz
In this case, we are selecting /dev/vda
as the input disk to image, and setting the input/output block sizes to 4MB (from the default 512 bytes). This generally speeds things up a little bit. In addition, we are using the conv=sparse
flag to minimize the output file size by skipping over empty space. To learn more about dd
’s parameters, consult the dd
manpage.
We then pipe the output to the pv
pipe viewer utility so we can visually track the progress of the transfer (this pipe is optional, and requires installing pv
using your package manager). If you know the size of the initial disk (in this case it’s 25G), you can add the -s 25G
to the pv
pipe to get an ETA for when the transfer will complete.
We then pipe it all to gzip
, and save it in a file called ubuntu.gz
on the temporary block storage volume we’ve attached to the server. Replace /mnt/tmp_disk
with the path to the external storage device you’ve attached to your server.
Instead of provisioning additional storage for your remote machine, you can also execute the copy over SSH if you have enough disk space available on your local machine. Note that depending on the bandwidth available to you, this can be slow and you may incur additional costs for data transfer over the network.
To copy and compress the disk over SSH, execute the following command on your local machine:
- ssh remote_user@your_server_ip "sudo dd if=/dev/vda bs=4M conv=sparse | gzip -1 -" | dd of=ubuntu.gz
In this case, we are SSHing into our remote server, executing the dd
command there, and piping the output to gzip
. We then transfer the gzip
output over the network and save it as ubuntu.gz
locally. Ensure you have the dd
utility available on your local machine before running this command:
- which dd
Output/bin/dd
Create the compressed image file using either of the above methods. This may take several hours, depending on the size of the disk you’re imaging and the method you’re using to create the image.
Once you’ve created the compressed image file, you can move on to uploading it to your DigitalOcean Spaces using s3cmd
.
As described in the prerequisites, you should have s3cmd
installed and configured for use with your DigitalOcean Space on the machine containing your compressed image.
Locate the compressed image file, and upload it to your Space using s3cmd
:
Note: You should replace your_space_name
with your Space’s name and not its URL. For example, if your Space’s URL is https://example-space-name.nyc3.digitaloceanspaces.com
, then your Space’s name is example-space-name
.
- s3cmd put /path_to_image/ubuntu.gz s3://your_space_name
Once the upload completes, navigate to your Space using the DigitalOcean Control Panel, and locate the image in the list of files. We will temporarily make the image publicly accessible so that Custom Images can access it and save a copy.
At the right-hand side of the image listing, click the More drop down menu, then click into Manage Permissions:
Then, click the radio button next to Public and hit Update to make the image publicly accessible.
Warning: Your image will temporarily be publicly accessible to anyone with its Spaces path during this process. If you’d like to avoid making your image temporarily public, you can create your Custom Image using the DigitalOcean API. Be sure to set your image to Private using the above procedure after your image has successfully been transferred to Custom Images.
Fetch the Spaces URL for your image by hovering over the image name in the Control Panel, and hit Copy URL in the window that pops up.
Now, navigate to Images in the left hand navigation bar, and then Custom Images.
From here, upload your image using this URL as detailed in the Custom Images Product Documentation.
You can then create a Droplet from this image. Note that you need to add an SSH key to the Droplet on creation. To learn how to do this, consult How to Add SSH Keys to Droplets.
Once your Droplet boots up, if you can SSH into it, you’ve successfully launched your Custom Image as a DigitalOcean Droplet.
If you attempt to SSH into your Droplet and are unable to connect, ensure that your image meets the listed requirements and has both cloud-init
and SSH installed and properly configured. If you still can’t access the Droplet, you can attempt to use the DigitalOcean Droplet Console and the non-root user you created earlier to explore the system and debug your networking, cloud-init
and SSH configurations. Another way of debugging your image is to use a virtualization tool like Virtualbox to boot up your disk image inside of a virtual machine, and debug your system’s configuration from within the VM.
In this guide, you’ve learned how to create a disk image of an Ubuntu 18.04 system using the dd
command line utility and upload it to DigitalOcean as a Custom Image from which you can launch Droplets.
The steps in this guide may vary depending on your operating system, existing hardware, and kernel configuration but, in general, images created from popular Linux distributions should work using this method. Be sure to carefully follow the steps for installing and configuring cloud-init
, and ensure that your system meets all the requirements listed in the prerequisites section above.
To learn more about Custom Images, consult the Custom Images product documentation.
]]>DigitalOcean’s Custom Images feature allows you to bring your virtual disk images from an on-premise environment or another cloud platform to DigitalOcean and use them to start DigitalOcean Droplets.
As described in the Custom Images documentation, the following image types are supported natively by the Custom Images upload tool:
ISO is another popular image format which you may want to use with Custom Images. ISO images are frequently provided by Linux distributions as a convenient method for installing Linux. Unfortunately, ISO images aren’t currently supported by the upload tool, although support is planned for the end of 2018.
In this tutorial, we’ll demonstrate how to use the free and open-source VirtualBox virtualization tool to create a DigitalOcean-compatible VDI image (VirtualBox Disk Image) from an Ubuntu 18.04 ISO. The steps in this guide can be adapted to work with your preferred distribution’s ISO images.
Before you begin, you’ll need the following available to you:
A local machine or remote server (with GUI access) onto which you’ll install and use VirtualBox. In this tutorial we’ll use a Mac OS X local machine, but you can use any system supported by VirtualBox. To learn more about supported systems, consult the VirtualBox Manual. The GUI menu options should be similar across operating systems, but may not be identical.
An ISO-format Ubuntu 18.04 Server OS image. The ubuntu-18.04.1-live-server-amd64.iso
image meets the two requirements listed in the Custom Images Image Requirements:
cloud-init
0.7.7, cloudbase-init
, coreos-cloudinit
, iginition
, or bsd-cloudinit
installed (Ubuntu 18.04 Server comes with cloud-init
installed)If you’re adapting these steps for another distribution’s ISO and your image does not have cloud-init
installed and configured, you must install and configure it manually after installing the OS.
Once you have these prerequisites available to you, you’re ready to begin with this guide.
The tool we’ll use to convert the ISO-format image in this guide is VirtualBox, a free and open-source virtualizer for x86 hardware. By default, VirtualBox uses a GUI, which we’ll use to create the VDI image in this guide.
To begin, download and install VirtualBox from the downloads page. Follow the appropriate link in the VirtualBox 5.2.20 platform packages section depending on your host operating system. In this guide, we’ll be using an OSX system, so we’ll download and install VirtualBox using the provided DMG.
Once you’ve installed VirtualBox, open the application.
You should see the following welcome screen:
Click on New to begin creating your Ubuntu virtual machine.
The following window should pop up, allowing you to name your virtual machine (VM) and select its OS:
In this tutorial, we’ll name our VM Ubuntu 18.04
, but feel free to give the VM a more descriptive name.
For Type, select Linux, and for Version, select Ubuntu (64-bit). Then, hit Continue.
The following screen should appear, allowing you to specify how much memory to allocate to your virtual machine:
Unless you have a more complex use case, 1024 MB should be enough memory for your virtual machine. If you need to adjust memory size, enter the amount of memory to be allocated to the VM, then hit Continue.
You should see the following screen:
This window allows you to create a virtual hard disk for your VM. This virtual hard disk is the image that you’ll upload to DigitalOcean in a later step. The Ubuntu operating system will be installed from the ISO you downloaded to this virtual hard disk. Make sure Create a virtual hard disk now is selected, and hit Create.
The following Hard disk file type window should appear, allowing you to select the format you’d like to use for your image:
All three types are supported by DigitalOcean Custom Images, so unless you have a strong preference, select VDI (VirtualBox Disk Image). Hit Continue.
You should then see the following window:
This window allows you to choose between a Dynamically allocated or Fixed size hard disk file. We’ll use the default Dynamically allocated option and allow the file to grow as we install the Ubuntu OS and packages. Hit Continue.
The next window allows you to name your hard disk file (as well as choose the path to which it will be saved), and specify its maximum size:
Be sure to give yourself enough disk space to install the operating system as well as additional packages you may need. The default 10 GB should be fine for most purposes, but if you anticipate installing a large number of packages or storing a lot of data in the image, you should bump this up to your anticipated disk usage.
Once you’ve selected the size of the virtual hard disk, hit Create.
At this point, you’ll be returned to the initial welcome screen, where you’ll see the virtual machine you just created:
We can now begin installing Ubuntu onto the virtual machine.
In this step we’ll install and configure the Ubuntu operating system onto our virtual machine.
To begin, from the VirtualBox welcome screen, select your virtual machine, and hit the Start button in the toolbar.
You should see the following virtual machine window, prompting you to select the ISO file from which you’ll boot the system:
Select the Ubuntu 18.04 Server ISO you downloaded, and hit Start.
In the VM, the Ubuntu installer will begin booting from the ISO, and you should be brought to the following menu:
Choose your preferred language using the arrow keys, and hit ENTER
to continue.
You should then see the following Keyboard configuration screen:
Choose your preferred keyboard configuration, select Done, and hit ENTER
.
Next, you’ll be brought to the following installer selection screen:
Select Install Ubuntu, and hit ENTER
.
The following Network connections screen should appear:
This screen allows you to configure the network interfaces for your Ubuntu server. Since we’re performing the installation on a virtual machine, we’ll just use the default option as the configured interface will be overwritten when we launch the image on the DigitalOcean platform.
Select Done and hit ENTER
.
You’ll then be brought to the following Configure proxy screen:
If you require a proxy, enter it here. Then, select Done, and hit ENTER
.
The next screen will allow you to choose an Ubuntu archive mirror:
Unless you require a specific mirror, the default should be fine here. Select Done and hit ENTER
.
Next, you’ll be prompted to partition your virtual disk:
Unless you’d like to set up Logical Volume Manager (LVM) or manually partition the virtual disk, select Use An Entire Disk to use the entire attached virtual disk, and hit ENTER
.
The following screen allows you to select the virtual disk that will be partitioned:
As described in the prompt text, the installer will create a partition for the bootloader, and use the remaining virtual disk space to create an ext4
partition to which the Ubuntu OS will be installed.
Select the attached virtual disk and hit ENTER
.
The following screen displays a summary of the filesystem installer options before partitioning:
The ext4
partition will be mounted to /
, and a second partition (1 MB) will be created for the GRUB bootloader. Once you’ve gone over and confirmed the partitioning scheme for your virtual disk, select Done and hit ENTER
.
In the confirmation screen that appears, select Continue and hit ENTER
.
The next screen will allow you to configure the system hostname, as well as an Ubuntu user:
Note that as you fill out this screen, the installer will continue copying files to the virtual disk in the background.
In this tutorial, we’ll create a user named sammy and call our server ubuntu. The server name will likely be overwritten when this image is run on the DigitalOcean platform, so feel free to give it a temporary name here.
You can upload your SSH keys to DigitalOcean and automatically embed them into created Droplets, so for now we won’t Import SSH identity. To learn how to upload your SSH keys to DigitalOcean, consult the Droplet Product Documentation.
Once you’ve filled in all the required fields, the prompt should look something like this:
Select Done and hit ENTER
.
The next screen will prompt you to select popular snaps for your Ubuntu server. Snaps are prepackaged bundles of software that contain an application, its dependencies, and configuration. To learn more about snaps, consult the Snap Documentation.
In this guide we won’t install any snaps and will manually install packages in a later step. If you’d like to install a snap, select or deselect it using SPACE
and scroll down to Done. Then, hit ENTER
.
Regardless of your selection in the snap screen, you’ll then be brought to an installation progress and summary screen:
Once the installation completes, select Reboot Now and hit ENTER
.
The installer will shut down and prompt you to remove the installation medium (in this case this is the ISO image we selected earlier). In most cases, the ISO will be detached automatically upon reboot, so you can simply hit ENTER
.
To double check, in the VirtualBox GUI menu, navigate to Devices, and then Optical Drives. If the Remove disk from virtual drive option is available to you, click on it to detach the ISO from the virtual machine. Then, back in the virtual machine window, hit ENTER
.
The system will reboot in the virtual machine, this time from the virtual disk to which we installed Ubuntu.
Since cloud-init
is installed by default on Ubuntu 18.04 Server, the first time Ubuntu boots, cloud-init
will run and configure itself. In the virtual machine window, you should see some cloud-init
log items and have a prompt available to you. Hit ENTER
.
You can then log in to your Ubuntu server using the user you created in the installer.
Enter your username and hit ENTER
, then enter your password and hit ENTER
.
You should now have access to a command prompt, indicating that you’ve successfully completed the Ubuntu 18.04 installation, and are now logged in as the user you created previously.
In the next step of this guide, we’ll reconfigure cloud-init
and set it up to run when the Ubuntu image is launched as a Droplet on the DigitalOcean platform.
cloud-init
Now that we’ve installed Ubuntu 18.04 to a virtual disk and have the system up and running, we need to reconfigure cloud-init
to use the appropriate datasource for the DigitalOcean platform. A cloud-init
datasource is a source of config data for cloud-init
that typically consists of userdata (like shell scripts) or server metadata, like hostname, instance-id, etc. To learn more about cloud-init
datasources, consult the official cloud-init
docs.
By default, on Ubuntu 18.04, cloud-init
configures itself to use the DataSourceNoCloud
datasource. This will cause problems when running the image on DigitalOcean, so we need to reconfigure cloud-init
to use the ConfigDrive
datasource and ensure that cloud-init
reruns when the image is launched on DigitalOcean.
To begin, ensure that you’ve started your Ubuntu 18.04 virtual machine and have logged in as the user you created earlier.
From the command line, navigate to the /etc/cloud/cloud.cfg.d
directory:
- cd /etc/cloud/cloud.cfg.d
Use the ls
command to list the cloud-init
config files present in the directory:
- ls
Output05_logging.cfg 50-curtin-networking.cfg 90_dpkg.cfg curtin-preserve-sources.cfg README
First, delete the 50-curtin-networking.cfg
file, which configures networking interfaces for your Ubuntu server. When the image is launched on DigitalOcean, cloud-init
will run and reconfigure these interfaces automatically. If this file is not deleted, the DigitalOcean Droplet created from this Ubuntu image will have its interfaces misconfigured and won’t be accessible from the internet.
- sudo rm 50-curtin-networking.cfg
Next, we’ll run dpkg-reconfigure cloud-init
to remove the NoCloud
datasource, ensuring that cloud-init
searches for and finds the ConfigDrive
datasource used on DigitalOcean:
- sudo dpkg-reconfigure cloud-init
You should see the following graphical menu:
The NoCloud
datasource is initially highlighted. Press SPACE
to unselect it, then hit ENTER
.
Finally, navigate to /etc/netplan
:
- cd /etc/netplan
Remove the 50-cloud-init.yaml
file (this was generated from the cloud-init
networking file we removed earlier):
- sudo rm 50-cloud-init.yaml
The final step is ensuring that we clean up configuration from the initial cloud-init
run so that it reruns when the image is launched on DigitalOcean.
To do this, run cloud-init clean
:
- sudo cloud-init clean
At this point, your image is ready to be launched on the DigitalOcean platform. You can install additional packages and software into your image. Once you’re done, shutdown your virtual machine:
- sudo shutdown -h now
We can now move on to uploading and launching this custom image on the DigitalOcean platform.
Now that we’ve created an Ubuntu 18.04 VDI image and configured it for use on DigitalOcean, we can upload it using the Custom Images upload tool.
On macOS, the Ubuntu virtual disk image we created and configured will be located by default at ~/VirtualBox VMs/your_VM_name/your_virtual_disk_name.vdi
. This path may vary slightly depending on the OS you’re using with VirtualBox.
Before we upload the image, we’ll compress it to speed up the file transfer to DigitalOcean.
On your host OS (not inside the virtual machine), navigate to the directory containing your VDI image file:
- cd ~/VirtualBox\ VMs/Ubuntu\ 18.04/
Now, use gzip
to compress the file:
- gzip < Ubuntu\ 18.04.vdi > Ubuntu\ 18.04.gz
In this command we pipe the source Ubuntu 18.04.vdi
file into gzip
, specifying as output the Ubuntu 18.04.gz
compressed file.
Once gzip
finishes compressing your file, upload the .gz
file to DigitalOcean, following instructions in the Custom Images Quickstart.
You should now be able to create and use Droplets from your custom Ubuntu 18.04 Server image.
In this tutorial, we learned how to create a custom VDI image from a vanilla Ubuntu 18.04 ISO using the VirtualBox virtualization tool. We adjusted cloud-init
so it can properly configure Droplet networking on DigitalOcean, and finally compressed and uploaded the image using the Custom Images upload tool.
You can adjust the steps in this tutorial to work with your preferred Linux distribution’s ISO images. Ensure that you have an SSH server installed and configured to start on boot, and that cloud-init
has been installed and properly configured to use the ConfigDrive
datasource. Finally, ensure that any stale networking configuration files have been purged.
You may also wish to use a tool like Packer to automate the creation of your machine images.
To learn more about DigitalOcean Custom Images, consult the Custom Images product docs and launch blog post.
]]>DigitalOcean Spaces é um serviço de armazenamento de objetos que torna mais fácil e econômico armazenar e fornecer grandes quantidades de dados. Spaces individuais podem ser criados e colocados em uso rapidamete, sem necessidade de configuração adicional.
Neste tutorial, iremos utilizar o Painel de Controle da DigitalOcean para criar um novo Space. Em seguida, iremos recuperar uma Chave de API ou API Key e um secret que podem ser utilizados para conceder acesso ao Space para quaisquer clientes ou bibliotecas compatíveis com S3.
Para completar este tutorial, você vai precisar de uma conta na DigitalOcean. Se você já não tiver uma, você pode registrar uma nova na página de inscrição.
Faça o login no Painel de Controle da DigitalOcean para começar.
Para criar um novo Space, utilize o botão Create no canto superior direito do Painel de Controle. Clique no botão, em seguida escolha Spaces na lista suspensa:
Se você nunca criou um Space antes, você também pode criar um diretamente da página do Spaces. Para fazer isso, clique Spaces na navegação principal do Painel de Controle, e então clique em Create a space. Qualquer uma das opções o levarão à tela Create a Space:
Primeiro, escolha um nome para o seu Space. Esse nome deve ser único entre todos os Spaces (ou seja, nenhum outro usuário do Spaces pode ter o mesmo nome em qualquer região), deve ter de 3 a 63 caracteres, e pode conter apenas letras minúsculas, números e traços.
Em seguida, escolha a região do datacenter onde você gostaria que seu Space estivesse. No momento em que esta captura de tela foi feita, nyc3
e ams3
eram as escolhas possíveis. Outras se tornarão dsponíveis ao longo do tempo.
Finalmente, escolha se deseja que os usuários não autenticados possam listar todos os arquivos em seu Space. Isso não afeta o acesso a arquivos individuais (que é definido em uma base por aquivo), mas apenas a capacidade de obter uma lista de todos os arquivos. A escolha padrão de Private é segura, a menos que você tenha alguns scripts ou clientes que precisem buscar listagens de arquivos sem uma chave de acesso ou access key.
Quando seu nome e as opções estiverem todos definidos, desça e clique no botão Create a Space. Seu Space será criado, e você será levado para a interface do navegador de arquivos:
Se este é o seu primeiro Space, você terá um arquivo welcome.html, do contrário, o Space estará vazio.
Tome nota da URL do seu Space. Está disponível logo abaixo do nome do Space na visualização do navegador de arquivos. Nesse caso de exemplo, a URL completa é https://example-name.nyc3.digitaloceanspaces.com. O nome do Space aqui (geralmente chamado de nome do bucket) é example-name. A URL do servidor (ou endereço) é a parte restante, consistindo do nome do datacenter seguido por .digitaloceanspaces.com: https://nyc3.digitaloceanspaces.com.
Existem algumas maneiras diferentes pelas quais os clientes e bibliotecas solicitarão essas informações. Alguns vão querer no mesmo formato dado no Painel de Controle. Alguns exigem que o nome do bucket siga a URL do servidor, como em https://nyc3.digitaloceanspaces.com/example-name. Outros ainda pedirão para você inserir o endereço do servidor e o nome do bucket ou Space separadamente. Consulte a documentação do seu cliente ou biblioteca para mais orientações nesse item.
A seguir, criaremos a chave que precisamos para acessar nossos Spaces a partir de clientes de terceiros.
Para acessar seus arquivos de fora do Painel de Controle da DigitalOcean, precisamos gerar uma chave de acesso ou access key e um secret. Estes são um par de tokens aleatórios que servem como nome de usuário e senha para conceder acesso ao seu Space.
Primeiro, clique no link da API na navegação principal do Painel de Controle. A página resultante lista seus tokens de API da DigitalOcean e as chaves de acesso do Spaces. Role para baixo até a parte do Spaces:
Se este é o seu primeiro Space, você não pode ter nenhuma chave listada. Clique no botão Generate New Key. A caixa de diálogo New Spaces key será exibida:
Digite um nome para a chave. Você pode criar quantas chaves quiser, portanto, lembre-se de que a única maneira de revogar o acesso a uma chave é excluí-la. Desse modo, você pode querer particionar as chaves por pessoa, por equipe ou pelo software cliente no qual você as estiver utilizando.
Neste caso, estamos criando uma chave chamada example-token. Clique no botão Generate Key para completar o processo. Você retornará à tela da API listando todas as suas chaves. Observe que a nova chave tem dois tokens longos exibidos:
O primeiro é a sua access key. Isso não é secreto e continuará visível no Painel de Controle. A segunda string é o seu secret ou secret key. Isso só será exibido uma vez. Registre-a em um local seguro para uso posterior. Na próxima vez que você visitar a página da API, esse valor será eliminado e não há como recuperá-lo.
Diferentes clientes compatíveis com S3 podem ter nomes sutilmente diferentes para access key e secret. A terminologia usada é normalmente próxima o suficiente para deixar claro qual token deve ir para onde. Caso contrário, consulte a documentação do seu cliente ou biblioteca para obter mais informações.
Neste tutorial criamos um novo Space na DigitalOcean e uma nova access key e secret. Agora sabemos nossa URL de servidor, nome do bucket (ou nome do Space), access key, e secret. Com essas informações você pode conectar praticamente qualquer cliente ou biblioteca compatível com S3 ao seu novo Space na DigitalOcean!
Por Brian Boucheron
]]>Implementing a CDN, or Content Delivery Network, to deliver your WordPress site’s static assets can greatly decrease your servers’ bandwidth usage as well as speed up page load times for geographically dispersed users. WordPress static assets include images, CSS stylesheets, and JavaScript files. Leveraging a system of edge servers distributed worldwide, a CDN caches copies of your site’s static assets across its network to reduce the distance between end users and this bandwidth-intensive content.
In a previous Solutions guide, How to Store WordPress Assets on DigitalOcean Spaces, we covered offloading a WordPress site’s Media Library (where images and other site content gets stored) to DigitalOcean Spaces, a highly redundant object storage service. We did this using the DigitalOcean Spaces Sync plugin, which automatically syncs WordPress uploads to your Space, allowing you to delete these files from your server and free up disk space.
In this Solutions guide, we’ll extend this procedure by enabling the Spaces CDN and rewriting Media Library asset URLs. This forces users’ browsers to download static assets directly from the CDN, a geographically distributed set of cache servers optimized for delivering static content. We’ll go over how to enable the CDN for Spaces, how to rewrite links to serve your WordPress assets from the CDN, and finally how to test that your website’s assets are being correctly delivered by the CDN.
We’ll demonstrate how to implement Media Library offload and link rewriting using the free and open-source Spaces Sync plugin. We’ll also cover how to do this using two popular paid WordPress plugins: WP Offload Media and Media Library Folders Pro. You should choose the plugin that suits your production needs best.
Before you begin this tutorial, you should have a running WordPress installation on top of a LAMP or LEMP stack. You should also have WP-CLI installed on your WordPress server, which you can learn to set up by following these instructions.
To offload your Media Library, you’ll need a DigitalOcean Space and an access key pair:
s3cmd
tool, consult s3cmd 2.x Setup, also on the DigitalOcean product documentation site.There are a few WordPress plugins that you can use to offload your WordPress assets:
Using a custom domain with Spaces CDN is highly recommended. This will drastically improve Search Engine Optimization (SEO) for your site by keeping your offloaded asset URLs similar to your Wordpress site’s URLs. To use a custom domain with Spaces CDN, you need to ensure that you first add your domain to your DigitalOcean account:
For testing purposes, be sure to have a modern web browser such as Google Chrome or Firefox installed on your client (e.g. laptop) computer.
Once you have a running WordPress installation and have created a DigitalOcean Space, you’re ready to enable the CDN for your Space and begin with this guide.
We’ll begin this guide by enabling the CDN for your DigitalOcean Space. This will not affect the availability of existing objects. With the CDN enabled, objects in your Space will be “pushed out” to edge caches across the content delivery network, and a new CDN endpoint URL will be made available to you. To learn more about how CDNs work, consult Using a CDN to Speed Up Static Content Delivery.
First, enable the CDN for your Space by following How to Enable the Spaces CDN.
If you’d like to use a custom domain with Spaces CDN (recommended), create the subdomain CNAME record and appropriate SSL certificates by following How to Customize the Spaces CDN Endpoint with a Subdomain. Note down the subdomain you’ll be using with Spaces CDN, as we’ll need to use this when configuring the WordPress asset offload plugin.
Navigate back to your Space and reload the page. You should see a new Endpoints link under your Space name:
These endpoints contain your Space name. We’re using wordpress-offload
in this tutorial.
Notice the addition of the new Edge endpoint. This endpoint routes requests for Spaces objects through the CDN, serving them from the edge cache as much as possible. Note down this Edge endpoint, which you’ll use to configure your WordPress plugin in future steps. If you created a subdomain for Spaces CDN, this subdomain is an alias for the Edge endpoint.
Now that you have enabled the CDN for your Space, you’re ready to begin configuring your asset offload and link rewriting plugin.
If you’re using DigitalOcean Spaces Sync and continuing from How to Store WordPress Assets on DigitalOcean Spaces, begin reading from the following section. If you’re not using Spaces Sync, skip to either the WP Offload Media section or the [Media Library Folders Pro section] (https://www.digitalocean.com/community/tutorials/how-to-speed-up-wordpress-asset-delivery-using-digitalocean-spaces-cdn#media-library-folders-pro-and-cdn-enabler-plugins), depending on the plugin you choose to use.
If you’d like to use the free and open-source DigitalOcean Spaces Sync and CDN Enabler plugins to serve your files from the CDN’s edge caches, follow the steps outlined in this section.
We’ll begin by ensuring that our WordPress installation and Spaces Sync plugin are configured correctly and are serving assets from DigitalOcean Spaces.
Continuing from How To Store WordPress Assets on DigitalOcean Spaces, your Media Library should be offloaded to your DigitalOcean Space and your Spaces Sync plugin settings should look as follows:
If you haven’t completed the How To Store WordPress Assets on DigitalOcean Spaces tutorial, you can still follow this guide by installing the Spaces Sync plugin using the built-in plugin installer. If you encounter any errors, please consult the steps in this prerequisite guide.
We are going to make some minor changes to ensure that our configuration allows us to offload WordPress themes and other directories, beyond the wp-content/uploads
Media Library folder.
First, we’re going to modify the Full URL-path to files field so that the Media Library files are served from our Space’s CDN and not locally from the server. This setting essentially rewrites links to Media Library assets, changing them from file links hosted locally on your WordPress server, to file links hosted on the DigitalOcean Spaces CDN.
Recall the Edge endpoint you noted down in the Enabling Spaces CDN step. If you are using a custom subdomain with Spaces CDN, you’ll use that subdomain instead of the Edge endpoint.
In this tutorial, the Space’s name is wordpress-offload
and the Space’s CDN endpoint is:
https://wordpress-offload.nyc3.cdn.digitaloceanspaces.com
Now, in the Spaces Sync plugin settings page, replace the URL in the Full URL-path to files field with your Spaces CDN endpoint, followed by /wp-content/uploads
.
In this tutorial, using the above Spaces CDN endpoint, the full URL would be:
https://wordpress-offload.nyc3.cdn.digitaloceanspaces.com/wp-content/uploads
If you’re using a custom subdomain, say https://assets.example.com
, the full URL would look as follows:
https://assets.example.com/wp-content/uploads
Next, for the Local path field, enter the full path to the wp-content/uploads
directory on your WordPress server. In this tutorial, the path to the WordPress installation on the server is /var/www/html/
, so the full path to uploads
would be /var/www/html/wp-content/uploads
.
Note: If you’re continuing from How To Store WordPress Assets on DigitalOcean Spaces, this guide will slightly modify the path to files in your Space to enable you to optionally offload themes and other wp-content
assets. You should clear out your Space before doing this (be sure to save a copy of the files), or alternatively you can transfer existing files into the correct wp-content/uploads
Space directory using s3cmd.
In the Storage prefix field, we’re going to enter /wp-content/uploads
, which will ensure that we build the correct wp-content
directory hierarchy so that we can offload other WordPress directories to this Space.
Filemask can remain wildcarded with *
, unless you’d like to exclude certain files.
It’s not necessary to check the Store files only in the cloud and delete… option; only check this box if you’d like to delete the Media Library assets from your server after they’ve been successfully uploaded to your DigitalOcean Space.
Your final settings should look something like this:
Be sure to replace the above values with the values corresponding to your WordPress installation and Spaces configuration.
Finally, hit Save Changes.
You should see a Settings saved box appear at the top of your screen, confirming that the Spaces Sync plugin settings have successfully been updated.
Future WordPress Media Library uploads should now be synced to your DigitalOcean Space, and served using the Spaces Content Delivery Network.
In this step, we did not offload the WordPress theme or other wp-content
assets. To learn how to transfer these assets to Spaces and serve them using the Spaces CDN, skip to Offloading Additional Assets.
To verify and test that your Media Library uploads are being delivered from the Spaces CDN, skip to Test CDN Caching.
The DeliciousBrains WordPress Offload Media plugin allows you to quickly and automatically upload your Media Library assets to DigitalOcean Spaces and rewrite links to these assets so that you can deliver them directly from Spaces or via the Spaces CDN. In addition, the Assets Pull addon allows you to quickly offload additional WordPress assets like JS, CSS, and font files in combination with a pull CDN. Setting up this addon is beyond the scope of this guide but to learn more you can consult the DeliciousBrains documentation.
We’ll begin by installing and configuring the WP Offload Media plugin for a sample WordPress site.
To begin, you must purchase a copy of the plugin on the DeliciousBrains plugin site. Choose the appropriate version depending on the number of assets in your Media Library, and support and feature requirements for your site.
After going through checkout, you’ll be brought to a post-purchase site with a download link for the plugin and a license key. The download link and license key will also be sent to you at the email address you provided when purchasing the plugin.
Download the plugin and navigate to your WordPress site’s admin interface (https://your_site_url/wp-admin
). Log in if necessary. From here, hover over Plugins and click on Add New.
Click Upload Plugin and the top of the page, Choose File, and then select the zip archive you just downloaded.
Click Install Now, and then Activate Plugin. You’ll be brought to WordPress’s plugin admin interface.
From here, navigate to the WP Offload Media plugin’s settings page by clicking Settings under the plugin name.
You’ll be brought to the following screen:
Click the radio button next to DigitalOcean Spaces. You’ll now be prompted to either configure your Spaces Access Key in the wp-config.php
file (recommended), or directly in the web interface (the latter will store your Spaces credentials in the WordPress database).
We’ll configure our Spaces Access Key in wp-config.php
.
Log in to your WordPress server via the command line, and navigate to your WordPress root directory (in this tutorial, this is /var/www/html
). From here, open up wp-config.php
in your favorite editor:
- sudo nano wp-config.php
Scroll down to the line that says /* That's all, stop editing! Happy blogging. */
, and before it insert the following lines containing your Spaces Access Key pair (to learn how to generate an access key pair, consult the Spaces product docs):
. . .
define( 'AS3CF_SETTINGS', serialize( array(
'provider' => 'do',
'access-key-id' => 'your_access_key_here',
'secret-access-key' => 'your_secret_key_here',
) ) );
/* That's all, stop editing! Happy blogging. */
. . .
Once you’re done editing, save and close the file. The changes will take effect immediately.
Back in the WordPress Offload Media plugin admin interface, select the radio button next to Define access keys in wp-config.php and hit Save Changes.
You should be brought to the following interface:
On this configuration page, select the appropriate region for your Space using the Region dropdown and enter your Space name next to Bucket (in this tutorial, our Space is called wordpress-offload
).
Then, hit Save Bucket.
You’ll be brought to the main WP Offload Media configuration page. At the top you should see the following warning box:
Click on enter your license key, and on the subsequent page enter the license key found in your email receipt or on the checkout page and hit Activate License.
If you entered your license key correctly, you should see License activated successfully.
Now, navigate back to main WP Offload Media configuration page by clicking on Media Library at the top of the window.
At this point, WP Offload Media has successfully been configured for use with your DigitalOcean Space. You can now begin offloading assets and delivering them using the Spaces CDN.
Now that you’ve linked WP Offload Media with your DigitalOcean Space, you can begin offloading assets and configuring URL rewriting to deliver media from the Spaces CDN.
You should see the following configuration options on the main WP Offload Media configuration page:
These defaults should work fine for most use cases. If your Media Library exists at a nonstandard path within your WordPress directory, enter the path in the text box under the Path option.
If you’d like to change asset URLs so that they are served directly from Spaces and not your WordPress server, ensure the toggle is set to On next to Rewrite Media URLs.
To deliver Media Library assets using the Spaces CDN, ensure you’ve enabled the CDN for your Space (see Enable Spaces CDN to learn how) and have noted down the URL for the Edge endpoint. Hit the toggle next to Custom Domain (CNAME), and In the text box that appears, enter the CDN Edge endpoint URL, without the https://
prefix.
In this guide the Spaces CDN endpoint is:
https://wordpress-offload.nyc3.cdn.digitaloceanspaces.com
So here we enter:
wordpress-offload.nyc3.cdn.digitaloceanspaces.com
If you’re using a custom subdomain with Spaces CDN, enter that subdomain here:
your_subdomain.example.com
To improve security, we’ll force HTTPS for requests to Media Library assets (now served using the CDN) by setting the toggle to On.
You can optionally clear out files that have been offloaded to Spaces from your WordPress server to free up disk space. To do this, hit On next to Remove Files From Server.
Once you’ve finished configuring WP Offload Media, hit Save Changes at the bottom of the page to save your settings.
The URL Preview box should display a URL containing your Spaces CDN endpoint. It should look something like the following:
https://wordpress‑offload.nyc3.cdn.digitaloceanspaces.com/wp‑content/uploads/2018/09/21211354/photo.jpg
If you’re using a custom subdomain with Spaces CDN, the URL preview should contain this subdomain.
This URL indicates that WP Offload Media has been successfully configured to deliver Media Library assets using the Spaces CDN. If the path doesn’t contain cdn
, ensure that you correctly entered the Edge endpoint URL and not the Origin URL (this does not apply when using a custom subdomain).
At this point, WP Offload Media has been set up to deliver your Media Library using Spaces CDN. Any future uploads to your Media Library will be automatically copied over to your DigitalOcean Space and served using the CDN.
You can now bulk offload existing assets in your Media Library using the built-in upload tool.
We’ll use the plugin’s built-in “Upload Tool” to offload existing files in our WordPress Media Library.
On the right-hand side of the main WP Offload Media configuration page, you should see the following box:
Click Offload Now to upload your Media Library files to your DigitalOcean Space.
If the upload procedure gets interrupted, the box will change to display the following:
Hit Offload Remaining Now to transfer the remaining files to your DigitalOcean Space.
Once you’ve offloaded the remaining items from your Media Library, you should see the following new boxes:
At this point you’ve offloaded your Media Library to your Space and are delivering the files to users using the Spaces CDN.
At any point in time, you can download the files back to your WordPress server from your Space by hitting Download Files.
You can also clear out your DigitalOcean Space by hitting Remove Files. Before doing this, ensure that you’ve first downloaded the files back to your WordPress server from Spaces.
In this step, we learned how to offload our WordPress Media Library to DigitalOcean Spaces and rewrite links to these Library assets using the WP Offload Media plugin.
To offload additional WordPress assets like themes and JavaScript files, you can use the Asset Pull addon or consult the Offload Additional Assets section of this guide.
To verify and test that your Media Library uploads are being delivered from the Spaces CDN, skip to Testing CDN Caching.
The MaxGalleria Media Library Folders Pro plugin is a convenient WordPress plugin that allows you to better organize your WordPress Media Library assets. In addition, the free Spaces addon allows you to bulk offload your Media Library assets to DigitalOcean Spaces, and rewrite URLs to those assets to serve them directly from object storage. You can then enable the Spaces CDN and use the Spaces CDN endpoint to serve your library assets from the distributed delivery network. To accomplish this last step, you can use the CDN Enabler plugin to rewrite CDN endpoint URLs for your Media Library assets.
We’ll begin by installing and configuring the Media Library Folders Pro (MLFP) plugin, as well as the MLFP Spaces addon. We’ll then install and configure the CDN Enabler plugin to deliver Media Library assets using the Spaces CDN.
After purchasing the MLFP plugin, you should have received an email containing your MaxGalleria account credentials as well as a plugin download link. Click on the plugin download link to download the MLFP plugin zip archive to your local computer.
Once you’ve downloaded the archive, log in to your WordPress site’s administration interface (https://your_site_url/wp-admin
), and navigate to Plugins and then Add New in the left-hand sidebar.
From the Add Plugins page, click Upload Plugin and then select the zip archive you just downloaded.
Click Install Now to complete the plugin installation, and from the Installing Plugin screen, click Activate Plugin to activate MLFP.
You should then see a Media Library Folders Pro menu item appear in the left-hand sidebar. Click it to go to the Media Library Folders Pro interface. Covering the plugin’s various features is beyond the scope of this guide, but to learn more, you can consult the MaxGalleria site and forums.
We’ll now activate the plugin. Click into Settings under the MLFP menu item, and enter your license key next to the License Key text box. You can find your MLFP license key in the email sent to you when you purchased the plugin. Hit Save Changes and then Activate License. Next, hit Update Settings.
Your MLFP plugin is now active, and you can use it to organize existing or new Media Library assets for your WordPress site.
We’ll now install and configure the Spaces addon plugin so that you can offload and serve these assets from DigitalOcean Spaces.
To install the Spaces Addon, log in to your MaxGalleria account. You can find your account credentials in an email sent to you when you purchased the MLFP plugin.
Navigate to the Addons page in the top menu bar and scroll down to Media Sources. From here, click into the Media Library Folders Pro S3 and Spaces option.
From this page, scroll down to the Pricing section and select the option that suits the size of your WordPress Media Library (for Media Libraries with 3000 images or less, the addon is free).
After completing the addon “purchase,” you can navigate back to your account page (by clicking the Account link in the top menu bar), from which the addon plugin will now be available.
Click on the Media Library Folders Pro S3 image and the plugin download should begin.
Once the download completes, navigate back to your WordPress administration interface, and install the downloaded plugin using the same method as above, by clicking Upload Plugin. Once again, hit Activate Plugin to activate the plugin.
You will likely receive a warning about configuring access keys in your wp-config.php
file. We’ll configure these now.
Log in to your WordPress server using the console or SSH, and navigate to your WordPress root directory (in this tutorial, this is /var/www/html
). From here, open up wp-config.php
in your favorite editor:
- sudo nano wp-config.php
Scroll down to the line that says /* That's all, stop editing! Happy blogging. */
, and before it insert the following lines containing your Spaces Access Key pair and a plugin configuration option (to learn how to generate an access key pair, consult the Spaces product docs):
. . .
define('MF_AWS_ACCESS_KEY_ID', 'your_access_key_here');
define( 'MF_AWS_SECRET_ACCESS_KEY', 'your_secret_key_here');
define('MF_CLOUD_TYPE', 'do')
/* That's all, stop editing! Happy blogging. */
. . .
Once you’re done editing, save and close the file.
Now, navigate to your DigitalOcean Space from the Cloud Control Panel, and create a folder called wp-content
by clicking on New Folder.
From here, navigate back to the WordPress administration interface, and click into Media Library Folders Pro and then S3 & Spaces Settings in the sidebar.
The warning banner about configuring access keys should now have disappeared. If it’s still present, you should double check your wp-config.php
file for any typos or syntax errors.
In the License Key text box, enter the license key that was emailed to you after purchasing the Spaces addon. Note that this license key is different from the MLFP license key. Hit Save Changes and then Activate License.
Once activated, you should see the following configuration pane:
From here, click Select Image Bucket & Region to select your DigitalOcean Space. Then select the correct region for your Space and hit Save Bucket Selection.
You’ve now successfully connected the Spaces offload plugin to your DigitalOcean Space. You can begin offloading your WordPress Media Library assets.
The Use files on the cloud server checkbox allows you to specify where Media Library assets will be served from. If you check the box, assets will be served from DigitalOcean Spaces, and URLs to images and other Media Library objects will be correspondingly rewritten. If you plan on using the Spaces CDN to serve your Media Library assets, do not check this box, as the plugin will use the Spaces Origin endpoint and not the CDN Edge endpoint. We will configure CDN link rewriting in a future step.
Click the Remove files from local server box to delete local Media Library assets once they’ve been successfully uploaded to DigitalOcean Spaces.
The Remove individual downloaded files from the cloud server checkbox should be used when bulk downloading files from Spaces to your WordPress server. If checked, these files will be deleted from Spaces after successfully downloading to your WordPress server. We can ignore this option for now.
Since we’re configuring the plugin for use with the Spaces CDN, leave the Use files on the cloud server box unchecked, and hit Copy Media Library to the cloud server to sync your site’s WordPress Media Library to your DigitalOcean Space.
You should see a progress box appear, and then Upload complete. indicating the Media Library sync has concluded successfully.
Navigate to your DigitalOcean Space to confirm that your Media Library files have been copied to your Space. They should be available in the uploads
subdirectory of the wp-content
directory you created earlier in this step.
Once your files are available in your Space, you’re ready to move on to configuring the Spaces CDN.
To use the Spaces CDN to serve your now offloaded files, first ensure that you’ve enabled the CDN for your Space.
Once the CDN has been enabled for your Space, you can now install and configure the CDN Enabler WordPress plugin to rewrite links to your Media Library assets. The plugin will rewrite links to these assets so that they are served from the Spaces CDN endpoint.
To install CDN Enabler, you can either use the Plugins menu from the WordPress administration interface, or install the plugin directly from the command line. We’ll demonstrate the latter procedure here.
First, log in to your WordPress server. Then, navigate to your plugins directory:
- cd /var/www/html/wp-content/plugins
Be sure to replace the above path with the path to your WordPress installation.
From the command line, use the wp-cli
interface to install the plugin:
- wp plugin install cdn-enabler
Now, activate the plugin:
- wp plugin activate cdn-enabler
You can also install and activate the CDN Enabler plugin using the built-in plugin installer.
Back in the WordPress Admin Area, under Settings, you should see a new link to CDN Enabler settings. Click into CDN Enabler.
You should see the following settings screen:
Modify the displayed fields as follows:
https://wordpress-offload.nyc3.cdn.digitaloceanspaces.com
. If you’re using a custom subdomain with Spaces CDN, enter that subdomain here. For example, https://assets.example.com
.wp-content/uploads
. We’ll learn how to serve other wp-content
directories in the Offload Additional Assets section..php
Then, hit Save Changes to save these settings and enable them for your WordPress site.
At this point you’ve successfully offloaded your WordPress site’s Media Library to DigitalOcean Spaces and are serving them to end users using the CDN.
In this step, we did not offload the WordPress theme or other wp-content
assets. To learn how to transfer these assets to Spaces and serve them using the Spaces CDN, skip to Offload Additional Assets.
To verify and test that your Media Library uploads are being delivered from the Spaces CDN, skip to Testing CDN Caching.
In previous sections of this guide, we’ve learned how to offload our site’s WordPress Media Library to Spaces and serve these files using the Spaces CDN. In this section, we’ll cover offloading and serving additional WordPress assets like themes, JavaScript files, and fonts.
Most of these static assets live inside of the wp-content
directory (which contains wp-themes
). To offload and rewrite URLs for this directory, we’ll use CDN Enabler, an open-source plugin developed by KeyCDN.
If you’re using the WP Offload Media plugin, you can use the Asset Pull addon to serve these files using a pull CDN. Installing and configuring this addon is beyond the scope of this guide. To learn more, consult the DeliciousBrains product page.
First, we’ll install CDN Enabler. We’ll then copy our WordPress themes over to Spaces, and finally configure CDN Enabler to deliver these using the Spaces CDN.
If you’ve already installed CDN Enabler in a previous step, skip to Step 2.
To install CDN Enabler, log in to your WordPress server. Then, navigate to your plugins directory:
- cd /var/www/html/wp-content/plugins
Be sure to replace the above path with the path to your WordPress installation.
From the command line, use the wp-cli
interface to install the plugin:
- wp plugin install cdn-enabler
Now, activate the plugin:
- wp plugin activate cdn-enabler
You can also install and activate the CDN Enabler plugin using the built-in plugin installer.
Back in the WordPress Admin Area, under Settings, you should see a new link to CDN Enabler settings. Click into CDN Enabler.
You should see the following settings screen:
At this point you’ve successfully installed CDN Enabler. We’ll now upload our WordPress themes to Spaces.
In this tutorial, to demonstrate a basic plugin configuration, we’re only going to serve wp-content/themes
, the WordPress directory containing WordPress themes’ PHP, JavaScript, HTML, and image files. You can optionally extend this process to other WordPress directories, like wp-includes
, and even the entire wp-content
directory.
The theme used by the WordPress installation in this tutorial is twentyseventeen
, the default theme for a fresh WordPress installation at the time of writing. You can repeat these steps for any other theme or WordPress content.
First, we’ll upload our theme to our DigitalOcean Space using s3cmd
. If you haven’t yet configured s3cmd
, consult the DigitalOcean Spaces Product Documentation.
Navigate to your WordPress installation’s wp-content
directory:
- cd /var/www/html/wp-content
From here, upload the themes
directory to your DigitalOcean Space using s3cmd
. Note that at this point you can choose to upload only a single theme, but for simplicity and to offload as much content as possible from our server, we will upload all the themes in the themes
directory to our Space.
We’ll use find
to build a list of non-PHP (therefore cacheable) files, which we’ll then pipe to s3cmd
to upload to Spaces. We’ll exclude CSS stylesheets as well in this first command as we need to set the text/css
MIME type when uploading them.
- find themes/ -type f -not \( -name '*.php' -or -name '*.css' \) | xargs -I{} s3cmd put --acl-public {} s3://wordpress-offload/wp-content/{}
Be sure to replace wordpress-offload
in the above command with your Space name.
Here, we instruct find
to search for files within the themes/
directory, and ignore .php
and .css
files. We then use xargs -I{}
to iterate over this list, executing s3cmd put
for each file, and set the file’s permissions in Spaces to public
using --acl-public
.
Next, we’ll do the same for CSS stylesheets, adding the --mime-type="text/css"
flag to set the text/css
MIME type for the stylesheets on Spaces. This will ensure that Spaces serves your theme’s CSS files using the correct Content-Type: text/css
HTTP header:
- find themes/ -type f -name '*.css' | xargs -I{} s3cmd put --acl-public --mime-type="text/css" {} s3://wordpress-offload/wp-content/{}
Again, be sure to replace wordpress-offload
in the above command with your Space name.
Now that we’ve uploaded our theme, let’s verify that it can be found at the correct path in our Space. Navigate to your Space using the DigitalOcean Cloud Control Panel.
Enter the wp-content
directory, followed by the themes
directory. You should see your theme’s directory here. If you don’t, verify your s3cmd
configuration and re-upload your theme to your Space.
Now that our theme lives in our Space, and we’ve set the correct metadata, we can begin serving its files using CDN Enabler and the DigitalOcean Spaces CDN.
Navigate back to the WordPress Admin Area and click into Settings and then CDN Enabler.
Here, modify the displayed fields as follows:
https://wordpress-offload.nyc3.cdn.digitaloceanspaces.com
. If you’re using a custom subdomain with Spaces CDN, enter that subdomain here. For example, https://assets.example.com
.wp-content/themes
. If you are, this should be wp-content/uploads,wp-content/themes
.php
Your final settings should look something like this:
Hit Save Changes to save these settings and enable them for your WordPress site.
At this point you’ve successfully offloaded your WordPress site’s theme assets to DigitalOcean Spaces and are serving them to end users using the CDN. We can confirm this using Chrome’s DevTools, following the procedure described below.
Using the CDN Enabler plugin, you can repeat this process for other WordPress directories, like wp-includes
, and even the entire wp-content
directory.
In this section, we’ll demonstrate how to determine where your WordPress assets are being served from (e.g. your host server or the CDN) using Google Chrome’s DevTools.
To begin, we’ll first upload a sample image to our Media Library, and verify that it’s being served from the DigitalOcean Spaces CDN servers. You can upload an image using the WordPress Admin web interface, or using the wp-cli
command-line tool. In this guide, we’ll use wp-cli
to upload the sample image.
Log in to your WordPress server using the command line, and navigate to the home directory for the non-root user you’ve configured. In this tutorial, we’ll use the user sammy.
- cd
From here, use curl
to download the DigitalOcean logo to your Droplet (if you already have an image you’d like to test with, skip this step):
- curl https://assets.digitalocean.com/logos/DO_Logo_horizontal_blue.png > do_logo.png
Now, use wp-cli
to import the image to your Media Library:
- wp media import --path=/var/www/html/ /home/sammy/do_logo.png
Be sure to replace /var/www/html
with the correct path to the directory containing your WordPress files.
You may see some warnings, but the output should end in the following:
OutputImported file '/home/sammy/do_logo.png' as attachment ID 10.
Success: Imported 1 of 1 items.
Which indicates that our test image has successfully been copied to the WordPress Media Library, and also uploaded to our DigitalOcean Space, using your preferred offload plugin.
Navigate to your DigitalOcean Space to confirm:
This indicates that your offload plugin is functioning as expected and automatically syncing WordPress uploads to your DigitalOcean Space. Note that the exact path to your Media Library uploads in the Space will depend on the plugin you’re using to offload your WordPress files.
Next, we will verify that this file is being served using the Spaces CDN, and not from the server running WordPress.
From the WordPress admin area (https://your_domain/wp-admin
), navigate to Pages in the left-hand side navigation menu.
We will create a sample page containing our uploaded image to determine where it’s being served from. You can also run this test by adding the image to an existing page on your WordPress site.
From the Pages screen, click into Sample Page, or any existing page. You can alternatively create a new page.
In the page editor, click on Add Media, and select the DigitalOcean logo (or other image you used to test this procedure).
An Attachment Details pane should appear on the right-hand side of your screen. From this pane, add the image to the page by clicking on Insert into page.
Now, back in the page editor, click on either Publish (if you created a new sample page) or Update (if you added the image to an existing page) in the Publish box on the right-hand side of your screen.
Now that the page has successfully been updated to contain the image, navigate to it by clicking on the Permalink under the page title. You’ll be brought to this page in your web browser.
For the purposes of this tutorial, the following steps will assume that you’re using Google Chrome, but you can use most modern web browsers to run a similar test.
From the rendered page preview in your browser, right click on the image and click on Inspect:
A DevTools window should pop up, highlighting the img
asset in the page’s HTML:
You should see the CDN endpoint for your DigitalOcean Space in this URL (in this tutorial, our Spaces CDN endpoint is https://wordpress-offload.nyc3.cdn.digitaloceanspaces.com
), indicating that the image asset is being served from the DigitalOcean Spaces CDN edge cache. If you’re using a custom subdomain with Spaces CDN, the asset URL should be using this custom subdomain.
This confirms that your Media Library uploads are being synced to your DigitalOcean Space and served using the Spaces CDN.
From the DevTools window, we’ll run one final test. Click on Network in the toolbar at the top of the window.
Once in the blank Network window, follow the displayed instructions to reload the page.
The page assets should populate in the window. Locate your test image in the list of page assets:
Once you’ve located your test image, click into it to open an additional information pane. Within this pane, click on Headers to show the response headers for this asset:
You should see the Cache-Control
HTTP header, which is a CDN response header. This confirms that this image was served from the Spaces CDN.
If you offloaded your wp-themes
(or other) directory as described in Offload Additional Assets, you should perform the following brief check to verify that your theme’s assets are being served from the Spaces CDN.
Navigate to your WordPress site in Google Chrome, and right-click anywhere in the page. In the menu that appears, click on Inspect.
You’ll once again be brought to the Chrome DevTools interface.
From here, click into Sources.
In the left-hand pane, you should see a list of your WordPress site’s assets. Scroll down to your CDN endpoint (or custom subdomain), and expand the list by clicking the small arrow next to the endpoint name:
Observe that your WordPress theme’s header image, JavaScript, and CSS stylesheet are now being served from the Spaces CDN.
In this tutorial, we’ve shown how to offload static content from your WordPress server to DigitalOcean Spaces, and serve this content using the Spaces CDN. In most cases, this should reduce bandwidth on your host infrastructure and speed up page loads for end users, especially those located further away geographically from your WordPress server.
We demonstrated how to offload and serve both Media Library and themes
assets using the Spaces CDN, but these steps can be extended to further unload the entire wp-content
directory, as well as wp-includes
.
Implementing a CDN to deliver static assets is just one way to optimize your WordPress installation. Other plugins like W3 Total Cache can further speed up page loads and improve the SEO of your site. A helpful tool to measure your page load speed and improve it is Google’s PageSpeed Insights. Another helpful tool that provides a waterfall breakdown of request and response times as well as suggested optimizations is Pingdom.
To learn more about Content Delivery Networks and how they work, consult Using a CDN to Speed Up Static Content Delivery.
]]>Download the Kubernetes White Paper
Running Cloud Native Applications on DigitalOcean Kubernetes (PDF)
The Running Cloud Native Applications on DigitalOcean Kubernetes White Paper brings readers through a variety of cloud native topics, introducing them to how they may leverage Kubernetes in order to manage and scale their applications.
This white paper provides further insight into:
Throughout the White Paper, a photo-sharing app called “Snappy” is used as a running example to demonstrate the value of implementing Cloud Native best practices.
In today’s fast-moving software landscape, advances in operations technologies have fostered the dramatic reduction of application release cycles. Traditionally, software releases follow a time-based schedule, but it has become increasingly common to see applications and services continuously delivered and deployed to users throughout the day. This truncating of the traditional software release cycle has its roots both in technological developments — such as the explosive growth of cloud platforms, containers, and microservices-oriented architectures — as well as cultural developments — with tech-savvy and mobile-enabled users increasingly expecting new features, fast bug fixes, and a responsive and continuously developing product.
This symbiotic relationship between end users and developers has become increasingly linked. Shifting organizational structures and application architectures allow developers to quickly incorporate feedback and react to user demands. This accelerated development cadence often accompanies the packaging of applications into containers, and the use of systems that automate their deployment and orchestration, like Docker Swarm, Marathon, and Kubernetes. These open-source platforms, now stable enough for large-scale production deployments, allow service owners to launch and scale applications themselves, effortlessly managing hundreds of running containers.
Kubernetes, initially open-sourced by Google in 2014, has today grown to become one of the highest velocity projects on GitHub, with over 11,300 contributing developers and 75,000 commits. The growth of its thriving open-source community mirrors its popularity in the private sector, with over 50% of Fortune 100 companies relying on Kubernetes every day to rapidly deploy new features and bug fixes to users.
DigitalOcean Kubernetes enables development teams both small and large to quickly take advantage of this market-leading container orchestration platform without the lead time required to provision, install, and operate a cluster. With its simplicity and developer-friendly interfaces, DigitalOcean Kubernetes empowers developers to launch their containerized applications into a managed, production-ready cluster without having to maintain and configure the underlying infrastructure. Seamlessly integrating with the rest of the DigitalOcean suite — including Load Balancers, Firewalls, Object Storage Spaces, and Block Storage Volumes — and with built-in support for public and private image registries like Docker Hub and Quay.io, developers can now run and scale container-based workloads with ease on the DigitalOcean platform.
With full programmatic control of their cluster using the exposed Kubernetes REST API, developers can benefit from the rich ecosystem of open-source tools while still reaping the convenience of managed infrastructure. Teams can flexibly deploy and scale their Cloud Native applications. A Certified Kubernetes conformant platform, DigitalOcean Kubernetes helps developers launch their application containers and bring their Kubernetes workloads into the DigitalOcean cloud with minimal configuration and operations overhead.
To learn more about scaling and managing Cloud Native applications, microservices, containers, and Kubernetes, download your free copy of Running Cloud Native Applications on DigitalOcean Kubernetes*!*
Download the Kubernetes White Paper
Running Cloud Native Applications on DigitalOcean Kubernetes (PDF)
GitLab is an open-source tool used by software teams to manage their complete development and delivery lifecycle. GitLab provides a broad set of functionality: issue tracking, git repositories, continuous integration, container registry, deployment, and monitoring. These features are all built from the ground up as a single application. You can host GitLab on your own servers or use GitLab.com, a cloud service where open-source projects get all the top-tier features for free.
GitLab’s continuous integration / continuous delivery (CI/CD) functionality is an effective way to build the habit of testing all code before it’s deployed. GitLab CI/CD is also highly scalable thanks to an additional tool, GitLab Runner, which automates scaling your build queue in order to avoid long wait times for development teams trying to release code.
In this guide, we will demonstrate how to configure a highly scalable GitLab infrastructure that manages its own costs, and automatically responds to load by increasing and decreasing available server capacity.
We’re going to build a scalable CI/CD process on DigitalOcean that automatically responds to demand by creating new servers on the platform and destroys them when the queue is empty.
These reusable servers are spawned by the GitLab Runner process and are automatically deleted when no jobs are running, reducing costs and administration overhead for your team.
As we’ll explain in this tutorial, you are in control of how many machines are created at any given time, as well as the length of time they’re retained before being destroyed.
We’ll be using three separate servers to build this project, so let’s go over terminology first:
GitLab: Your hosted GitLab instance or self-hosted instance where your code repositories are stored.
GitLab Bastion: The bastion server or Droplet is the core of what we’ll be configuring. It is the control instance that is used to interact with the DigitalOcean API to create Droplets and destroy them when necessary. No jobs are executed on this server.
GitLab Runners: Your runners are transient servers or Droplets that are created on the fly by the bastion server when needed to execute a CI/CD job in your build queue. These servers are disposable, and are where your code is executed or tested before your build is marked as passing or failing.
By leveraging each of the GitLab components, the CI/CD process will enable you to scale responsively based on demands. With these goals in mind, we are ready to begin setting up our continuous deployment with GitLab and DigitalOcean.
This tutorial will assume you have already configured GitLab on your own server or through the hosted service, and that you have an existing DigitalOcean account.
To set this up on an Ubuntu 16.04 Droplet, you can use the DigitalOcean one-click image, or follow our guide: “How To Install and Configure GitLab on Ubuntu 16.04.”
For the purposes of this tutorial, we assume you have private networking enabled on this Droplet, which you can achieve by following our guide on “How To Enable DigitalOcean Private Networking on Existing Droplets,” but it is not compulsory.
Throughout this tutorial, we’ll be using non-root users with admin privileges on our Droplets.
To begin, we will create a new example project in your existing GitLab instance containing a sample Node.js application.
Login to your GitLab instance and click the plus icon, then select New project from the dropdown menu.
On the new project screen, select the Import project tag, then click Repo by URL to import our example project directly from GitHub.
Paste the below clone URL into the Git repository URL:
https://github.com/do-community/hello_hapi.git
This repository is a basic JavaScript application for the purposes of demonstration, which we won’t be running in production. To complete the import, click the New Project button.
Your new project will now be in GitLab and we can get started setting up our CI pipeline.
Our GitLab Code Runner requires specific configuration as we’re planning to programmatically create Droplets to handle CI load as it grows and shrinks.
We will create two types of machines in this tutorial: a bastion instance, which controls and spawns new machines, and our runner instances, which are temporary servers spawned by the bastion Droplet to build code when required. The bastion instance uses Docker to create your runners.
Here are the DigitalOcean products we’ll use, and what each component is used for:
Flexible Droplets — We will create memory-optimized Droplets for our GitLab Runners as it’s a memory-intensive process which will run using Docker for containerization. You can shrink or grow this Droplet in the future as needed, however we recommend the flexible Droplet option as a starting point to understand how your pipeline will perform under load.
DigitalOcean Spaces (Object Storage) — We will use DigitalOcean Spaces to persist cached build components across your runners as they’re created and destroyed. This reduces the time required to set up a new runner when the CI pipeline is busy, and allows new runners to pick up where others left off immediately.
Private Networking — We will create a private network for your bastion Droplet and GitLab runners to ensure secure code compilation and to reduce firewall configuration required.
To start, we’ll create the bastion Droplet. Create a new Droplet, then under choose an image, select the One-click apps tab. From there, select Docker 17.12.0-ce on 16.04 (note that this version is current at the time of writing), then choose the smallest Droplet size available, as our bastion Droplet will manage the creation of other Droplets rather than actually perform tests.
It is recommended that you create your server in a data center that includes DigitalOcean Spaces in order to use the object storage caching features mentioned earlier.
Select both the Private networking and Monitoring options, then click Create Droplet.
We also need to set up our storage space which will be used for caching. Follow the steps in “How To Create a DigitalOcean Space and API Key” to create a new Space in the same or nearest data center as your hosted GitLab instance, along with an API Key.
Note this key down, as we’ll need it later in the tutorial.
Now it’s time to get our CI started!
With the fresh Droplet ready, we can now configure GitLab Runner. We’ll be installing scripts from GitLab and GitHub repositories.
As a best practice, be sure to inspect scripts to confirm what you will be installing prior to running the full commands below.
Connect to the Droplet using SSH, move into the /tmp
directory, then add the official GitLab Runner repository to Ubuntu’s package manager:
- cd /tmp
- curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
Once added, install the GitLab Runner application:
- sudo apt-get install gitlab-runner
We also need to install Docker Machine, which is an additional Docker tool that assists with automating the deployment of containers on cloud providers:
- curl -L https://github.com/docker/machine/releases/download/v0.14.0/docker-machine-`uname -s`-`uname -m` >/tmp/docker-machine && \
- sudo install /tmp/docker-machine /usr/local/bin/docker-machine
With these installations complete, we can move on to connecting our GitLab Runner to our GitLab install.
To link GitLab Runner to your existing GitLab install, we need to link the two instances together by obtaining a token that authenticates your runner to your code repositories.
Login to your existing GitLab instance as the admin user, then click the wrench icon to enter the admin settings area.
On the left of your screen, hover over Overview and select Runners from the list that appears.
On the Runners page under the How to setup a shared Runner for a new project section, copy the token shown in Step 3, and make a note of it along with the publicly accessible URL of your GitLab instance from Step 2. If you are using HTTPS for Gitlab, make sure it is not a self-signed certificate, or GitLab Runner will fail to start.
Back in your SSH connection with your bastion Droplet, run the following command:
- sudo gitlab-runner register
This will initiate the linking process, and you will be asked a series of questions.
On the next step, enter the GitLab instance URL from the previous step:
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com)
https://example.digitalocean.com
Enter the token you obtained from your GitLab instance:
Please enter the gitlab-ci token for this runner
sample-gitlab-ci-token
Enter a description that will help you recognize it in the GitLab web interface. We recommend naming this instance something unique, like runner-bastion
for clarity.
Please enter the gitlab-ci description for this runner
[yourhostname] runner-bastion
If relevant, you may enter the tags for code you will build with your runner. However, we recommend this is left blank at this stage. This can easily be changed from the GitLab interface later.
Please enter the gitlab-ci tags for this runner (comma separated):
code-tag
Choose whether or not your runner should be able to run untagged jobs. This setting allows you to choose whether your runner should build repositories with no tags at all, or require specific tags. Select true in this case, so your runner can execute all repositories.
Whether to run untagged jobs [true/false]: true
Choose if this runner should be shared among your projects, or locked to the current one, which blocks it from building any code other than those specified. Select false for now, as this can be changed later in GitLab’s interface:
Whether to lock Runner to current project [true/false]: false
Choose the executor which will build your machines. Because we’ll be creating new Droplets using Docker, we’ll choose docker+machine
here, but you can read more about the advantages of each approach in this compatibility chart:
Please enter the executor: ssh, docker+machine, docker-ssh+machine, kubernetes, docker, parallels, virtualbox, docker-ssh, shell:
docker+machine
You’ll be asked which image to use for projects that don’t explicitly define one. We’ll choose a basic, secure default:
Please enter the Docker image (e.g. ruby:2.1):
alpine:latest
Now you’re done configuring the core bastion runner! At this point it should appear within the GitLab Runner page of your GitLab admin settings, which we accessed to obtain the token.
If you encounter any issues with these steps, the GitLab Runner documentation includes options for troubleshooting.
To speed up Droplet creation when the build queue is busy, we’ll leverage Docker’s caching tools on the Bastion Droplet to store the images for your commonly used containers on DigitalOcean Spaces.
To do so, upgrade Docker Machine on your SSH shell using the following command:
- curl -L https://github.com/docker/machine/releases/download/v0.14.0/docker-machine-`uname -s`-`uname -m` >/tmp/docker-machine && sudo install /tmp/docker-machine /usr/local/bin/docker-machine
With Docker Machine upgraded, we can move on to setting up our access tokens for GitLab Runner to use.
Now we need to create the credentials that GitLab Runner will use to create new Droplets using your DigitalOcean account.
Visit your DigitalOcean dashboard and click API. On the next screen, look for Personal access tokens and click Generate New Token.
Give the new token a name you will recognize such as GitLab Runner Access
and ensure that both the read and write scopes are enabled, as we need the Droplet to create new machines without human intervention.
Copy the token somewhere safe as we’ll use it in the next step. You can’t retrieve this token again without regenerating it, so be sure it’s stored securely.
To bring all of these components together, we need to finish configuring our bastion Droplet to communicate with your DigitalOcean account.
In your SSH connection to your bastion Droplet, use your favorite text editor, such as nano, to open the GitLab Runner configuration file for editing:
- nano /etc/gitlab-runner/config.toml
This configuration file is responsible for the rules your CI setup uses to scale up and down on demand. To configure the bastion to autoscale on demand, you need to add the following lines:
concurrent = 50 # All registered Runners can run up to 50 concurrent builds
[[runners]]
url = "https://example.digitalocean.com"
token = "existinggitlabtoken" # Note this is different from the registration token used by `gitlab-runner register`
name = "example-runner"
executor = "docker+machine" # This Runner is using the 'docker+machine' executor
limit = 10 # This Runner can execute up to 10 builds (created machines)
[runners.docker]
image = "alpine:latest" # Our secure image
[runners.machine]
IdleCount = 1 # The amount of idle machines we require for CI if build queue is empty
IdleTime = 600 # Each machine can be idle for up to 600 seconds, then destroyed
MachineName = "gitlab-runner-autoscale-%s" # Each machine will have a unique name ('%s' is required and generates a random number)
MachineDriver = "digitalocean" # Docker Machine is using the 'digitalocean' driver
MachineOptions = [
"digitalocean-image=coreos-stable", # The DigitalOcean system image to use by default
"digitalocean-ssh-user=core", # The default SSH user
"digitalocean-access-token=DO_ACCESS_TOKEN", # Access token from Step 7
"digitalocean-region=nyc3", # The data center to spawn runners in
"digitalocean-size=1gb", # The size (and price category) of your spawned runners
"digitalocean-private-networking" # Enable private networking on runners
]
[runners.cache]
Type = "s3" # The Runner is using a distributed cache with the S3-compatible Spaces service
ServerAddress = "nyc3.spaces.digitaloceanspaces.com"
AccessKey = "YOUR_SPACES_KEY"
SecretKey = "YOUR_SPACES_SECRET"
BucketName = "your_bucket_name"
Insecure = true # We do not have a SSL certificate, as we are only running locally
Once you’ve added the new lines, customize the access token, region and Droplet size based on your setup. For the purposes of this tutorial, we’ve used the smallest Droplet size of 1GB and created our Droplets in NYC3. Be sure to use the information that is relevant in your case.
You also need to customize the cache component, and enter your Space’s server address from the infrastructure configuration step, access key, secret key and the name of the Space that you created.
When completed, restart GitLab Runner to make sure the configuration is being used:
- gitlab-runner restart
If you would like to learn about more all available options, including off-peak hours, you can read GitLab’s advanced documentation.
At this point, our GitLab Runner bastion Droplet is configured and is able to create DigitalOcean Droplets on demand, as the CI queue fills up. We’ll need to test it to be sure it works by heading to your GitLab instance and the project we imported in Step 1.
To trigger a build, edit the readme.md
file by clicking on it, then clicking edit, and add any relevant testing text to the file, then click Commit changes.
Now a build will be automatically triggered, which can be found under the project’s CI/CD option in the left navigation.
On this page you should see a pipeline entry with the status of running. In your DigitalOcean account, you’ll see a number of Droplets automatically created by GitLab Runner in order to build this change.
Congratulations! Your CI pipeline is cloud scalable and now manages its own resource usage. After the specified idle time, the machines should be automatically destroyed, but we recommend verifying this manually to ensure you aren’t unexpectedly billed.
In some cases, GitLab may report that the runner is unreachable and as a result perform no actions, including deploying new runners. You can troubleshoot this by stopping GitLab Runner, then starting it again in debug mode:
- gitlab-runner stop
- gitlab-runner --debug start
The output should throw an error, which will be helpful in determining which configuration is causing the issue.
If your configuration creates too many machines, and you wish to remove them all at the same time, you can run this command to destroy them all:
- docker-machine rm $(docker-machine ls -q)
For more troubleshooting steps and additional configuration options, you can refer to GitLab’s documentation.
You’ve successfully set up an automated CI/CD pipeline using GitLab Runner and Docker. From here, you could configure higher levels of caching with Docker Registry to optimize performance or explore the use of tagging code builds to specific GitLab code runners.
For more on GitLab Runner, see the detailed documentation, or to learn more, you can read GitLab’s series of blog posts on how to make the most of your continuous integration pipeline.
This post also appears on the GitLab Blog.
]]>FileZilla Pro is a file transfer solution that works with FTP, SFTP, FTPS, and WebDAV protocols. In 2001, the original FileZilla project brought an open-source, cross-platform file access and transfer application to users. Today, FileZilla Pro offers support for a growing number of network and cloud protocols.
DigitalOcean Spaces is an object storage solution that allows users to store and serve large amounts of data. Because its API is interoperable with the AWS S3 API, you can use FileZilla Pro to transfer files to and access files from your DigitalOcean Space.
In this tutorial, we’ll walk you through configuring FileZilla Pro to connect to a DigitalOcean Spaces repository.
In order to complete this tutorial, you should have access to the following:
With these prerequisites in place, we can begin setting up FileZilla Pro to work with DigitalOcean Spaces.
To connect to your DigitalOcean Spaces repository with FileZilla Pro, you will need to configure the FileZilla providers list with DigitalOcean endpoints. Open FileZilla Pro and follow these steps:
Name | Description | Endpoints |
---|---|---|
nyc3 | New York 3 | nyc3.digitaloceanspaces.com |
sgp1 | Singapore 1 | sgp1.digitaloceanspaces.com |
ams3 | Amsterdam 3 | ams3.digitaloceanspaces.com |
You can add other regions later. The completed Regions list will look like this:
.digitaloceanspaces.com
into this box. Be sure to include the leading dot before the text.{region}.digitaloceanspaces.com
. Your Settings page should now look like this:From here, we’ll connect our Spaces buckets to FileZilla.
Now we can add our Spaces bucket information — including our repository URL and Access and Secret Keys — to FileZilla in order to connect to each of our Spaces.
DigitalOcean
.sammys-bucket.nyc3.digitaloceanspaces.com
, with sammys-bucket
being your Space Bucket Name, and nyc3.digitaloceanspaces.com
being your endpoint.With these information fields complete, your Site Manager page should now look like this:
When you have finished and ensured that everything appears to be correctly filled in, press the OK button at the bottom of the screen to complete the setup.
With the setup complete, you can now connect to DigitalOcean Spaces with FileZilla Pro.
You have now successfully integrated DigitalOcean Spaces with FileZilla Pro and can now transfer files to your Spaces bucket.
With FileZilla Pro, you can transfer files seamlessly between your local machine and remote servers, managing all of your transfers without needing to worry about how many files are in your source directory. Optimized for speed, FileZilla Pro allows you to adjust the pace of your transfers to best suit your needs, providing you with considerable flexibility.
To learn more about how to use FileZilla, you can read the FileZilla Client Tutorial.
]]>Regular database backups are a crucial step in guarding against unintended data loss events. Designing an effective backup and recovery strategy often involves trading off performance impact, implementation costs, and data storage costs with recovery speed, data integrity, and backup coverage. The optimal solution will depend on your recovery point and time objectives and database scale and architecture.
In this guide we’ll demonstrate how to perform a live (or “hot”) physical backup of a running MySQL database using LVM snapshots. We’ll then compress and store the data in a DigitalOcean Space.
The procedure presented in this tutorial is well suited for large MySQL databases, databases using a mixture of storage engines (such as InnoDB, TokuDB, and MyISAM), and database servers with multiple block storage volumes attached, managed using LVM.
We’ll begin by ensuring that our Ubuntu 16.04 server can take and mount an LVM snapshot. Next, we’ll take an LVM snapshot of the logical volume containing MySQL’s data directory. We’ll then mount this snapshot volume (frozen logical volume), and compress and ship the MySQL data directory to DigitalOcean Spaces for storage. To conclude, we’ll briefly run through a sample recovery scenario.
To use this guide, you’ll need to have the following prerequisites available to you:
An Ubuntu 16.04 Droplet with a non-root user that has sudo privileges, as detailed in Initial Server Setup with Ubuntu 16.04
A running MySQL 5.7+ installation, as detailed in How To Install MySQL on Ubuntu 16.04
An LVM logical volume used to store your MySQL database’s data directory:
/var/lib/mysql
location, consult How To Move a MySQL Data Directory to a New Location on Ubuntu 16.04A DigitalOcean Space and set of API credentials, as detailed in How To Create a DigitalOcean Space and API Key.
The s3cmd
command-line file transfer client (2.X) installed as detailed in Step 1 of How To Use Logrotate and S3cmd to Archive Logs to Object Storage on Ubuntu 16.04
s3cmd
configured to access your Space, as detailed in How To Configure s3cmd 2.x To Manage DigitalOcean Spaces
Once you have all of this set up, you’re ready to begin with this guide.
To begin, we’ll locate our MySQL data directory and note those details about our LVM configuration.
datadir
To find the path to your MySQL data directory, run the following command:
- mysqladmin -u root -p variables | grep datadir
Enter your MySQL root
password when prompted. You should see output similar to the following:
Output| datadir | /data/mysql/
For the MySQL installation used in this guide, the data directory is /data/mysql
.
We now need to confirm that /data/mysql
lives on an LVM logical volume. To confirm this, we’ll run lsblk
:
- lsblk
You should see output similar to the following:
OutputNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 600G 0 disk
└─vg1-mysql_data 252:0 0 475G 0 lvm /data
vda 253:0 0 160G 0 disk
├─vda1 253:1 0 159.9G 0 part /
├─vda14 253:14 0 4M 0 part
└─vda15 253:15 0 106M 0 part /boot/efi
From this we observe that /data
is in fact the mount point for an LVM logical volume called mysql_data
. It is a member of the volume group vg1
.
We now need to ensure that we have enough free space available in our volume group vg1
to take an LVM snapshot.
It’s important to note that output from the commands described in this section will vary depending on your server’s hardware and LVM configuration. Let’s quickly investigate the hardware and LVM configuration for the Ubuntu 16.04 server used in this guide.
To begin, let’s find out how many physical volumes we have using pvscan
:
- sudo pvscan
You should see output similar to the following:
Output PV /dev/sda VG vg1 lvm2 [500.00 GiB / 25.00 GiB free]
Total: 1 [500.00 GiB] / in use: 1 [500.00 GiB] / in no VG: 0 [0 ]
We observe that we have one 500GB physical volume (/dev/sda
) which is in one volume group (vg1
). 475GB of this physical volume has been allocated off to logical volumes, while 25GB remains free for use by the volume group.
We can confirm this by taking a deeper look into the vg1
volume group using the vgdisplay
command:
- sudo vgdisplay
You should see output that resembles the following:
Output--- Volume group ---
VG Name vg1
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 2
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size 500.00 GiB
PE Size 4.00 MiB
Total PE 127999
Alloc PE / Size 121600 / 475.00 GiB
Free PE / Size 6399 / 25.00 GiB
VG UUID KEsoDE-zON7-NdyO-ioxb-6FSl-CB4m-S3QCRj
From the Alloc PE / Size and Free PE / Size lines, we observe that we have 475GB allocated, and 25GB free in the vg1
volume group. The Cur PV line shows us that we have 1 physical volume in this volume group. The Cur LV line indicates that we’ve used the pool of space in this volume group to create 1 logical volume.
Let’s now take a look at this logical volume using lvdisplay
:
- sudo lvdisplay
You should see output similar to the following:
Output --- Logical volume ---
LV Path /dev/vg1/mysql_data
LV Name mysql_data
VG Name vg1
LV UUID T98x9c-zvC1-f0Rw-4ipn-Cxo2-duwk-KUwQQc
LV Write Access read/write
LV Creation host, time LVM, 2018-04-18 20:11:48 +0000
LV Status available
# open 1
LV Size 475.00 GiB
Current LE 121600
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 252:0
From LV Size we see that we have one 475GB logical volume, mysql_data
, found at /dev/vg1/mysql_data
(recall that vg1
was the name of mysql_data
’s volume group).
To summarize, on the Ubuntu 16.04 server used for this tutorial, we have one 500GB physical volume (/dev/sda
) used to back one volume group (vg1
), from which we’ve created a single 475GB logical volume (mysql_data
). This leaves 25GB of free space in the volume group which can be used to create further logical volumes (and snapshots).
Your hardware and LVM configuration will likely be different; you may have several block storage devices attached, pooling into single or multiple volume groups. The procedure for taking a snapshot of a given logical volume will nevertheless be the same.
Using the series of commands presented in this section, you should now have a general sense of your LVM and hardware configuration.
In the next step, we’ll prepare your database server for LVM snapshotting.
To ensure that we can safely take an LVM snapshot, we need to provision enough disk space to cover any writes or changes that may occur during the backup and file transfer to Spaces. Depending on the size of your database, this backup may take several hours to complete, so it’s best to err on the side of caution here. If your snapshot volume runs out of space while performing the backup, the snapshot volume will become invalid and you will no longer have a consistent backup.
In the previous step we observed that the volume group (vg1
) containing our principal logical volume (mysql_data
) only had 25GB free. Although it may be the case that 25GB of changes won’t be written to disk in the time it takes to back up our database, we’d ideally like a margin of safety of at least 100GB. In a production setting, it would be best practice to measure the average amount of data being written to disk during the scheduled backup window and scale the snapshot volume size accordingly.
To add an additional 75GB of space to the vg1
volume group, we can either attach a block storage device, or increase the size of the volume currently attached to the Droplet. In this tutorial, we’ll expand the already attached block storage volume; to learn more about attaching an additonal block storage volume, you can consult An Introduction to DigitalOcean Block Storage.
Note: Some regions do not yet support Block Storage and you may not be able to attach a Block Storage volume to your Droplet. A reasonable workaround in this case would be to Snapshot your Droplet and create a new Droplet using this Snapshot image, to which you can then add Block Storage.
Let’s expand the block storage volume attached to this Droplet.
Navigate to DigitalOcean’s web control panel and from the Dashboard, navigate to your Droplet.
In the sidebar, click into Volumes:
From this pane, you should see any block storage volumes attached to your Droplet. For the Ubuntu Droplet used in this guide, we have one attached block storage volume:
Click on More and then Resize volume.
From here, you can select one of several predefined volume sizes, or choose your own volume size. Let’s increase the 500GB volume by 100GB to 600GB:
Press Continue. Your attached block storage volume has now been increased by 100GB.
To propagate this device change to LVM, we need to run pvresize
.
Log in to your server, and run pvscan
again to scan for physical volumes:
- sudo pvscan
You should see the same output as before for our /dev/sda
physical volume:
Output PV /dev/sda VG vg1 lvm2 [500.00 GiB / 25.00 GiB free]
Total: 1 [500.00 GiB] / in use: 1 [500.00 GiB] / in no VG: 0 [0 ]
Now, run pvresize
on the volume to fill out the extra space that we just added:
- sudo pvresize /dev/sda
You should see the following output:
OutputPhysical volume "/dev/sda" changed
1 physical volume(s) resized / 0 physical volume(s) not resized
Let’s confirm that our physical volume is now 100GB larger by running another pvscan
:
- sudo pvscan
We observe that the /dev/sda
physical volume is now 600GB:
Output PV /dev/sda VG vg1 lvm2 [600.00 GiB / 125.00 GiB free]
Total: 1 [600.00 GiB] / in use: 1 [600.00 GiB] / in no VG: 0 [0 ]
Let’s now confirm that our volume group’s free space has also increased by 100GB:
- sudo vgdisplay
You should then see the following output:
Output --- Volume group ---
VG Name vg1
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size 600.00 GiB
PE Size 4.00 MiB
Total PE 153599
Alloc PE / Size 121600 / 475.00 GiB
Free PE / Size 31999 / 125.00 GiB
VG UUID KEsoDE-zON7-NdyO-ioxb-6FSl-CB4m-S3QCRj
This indicates that we now have 125GB of free space available from which to create our snapshot volume.
For the purposes of this tutorial, 125GB will be plenty for absorbing writes and changes during the backup and upload procedure, but in a production setting the snapshot volume size should be scaled proportionally to anticipated disk usage during the backup window.
Now that we have enough space available in our volume group to cover any writes or changes that may occur during the snapshot and backup, we can move on to creating the snapshot volume.
Warning: While the LVM snapshot is active, there will be some performance degradation when writing to disk. You should test this procedure first using a non-production database with simulated load to verify that this method will work in your production deployment.
We’ll now create a snapshot of the mysql_data
logical volume using lvcreate
. Before we do this, we need to freeze writes to the database using FLUSH TABLES WITH READ LOCK
so that we can guarantee data consistency. The tables only need to be read locked until we run lvcreate
, at which point they can be unlocked. If you script this series of commands, the total lock time should be very small depending on currently executing write queries.
Let’s begin by flushing tables. From the terminal on your database server, use mysql
to log into your MySQL database:
- mysql -u root -p
From the MySQL shell, run the FLUSH TABLES
command to read-lock your database.
Warning: After running the following command, all open tables will be closed and all tables for all databases will be locked with a global read lock. If running this on a production database, it’s best practice to execute this command on a replica or as part of a script to minimize the amount of time the database will be locked.
- FLUSH TABLES WITH READ LOCK;
You should see the following output:
OutputQuery OK, 0 rows affected (0.00 sec)
Which indicates that your database has been read-locked. Don’t exit the MySQL prompt, we’ll need to keep it open.
We’ll now create and mount an LVM snapshot of the logical volume used to house our MySQL data.
###Create and Mount Snapshot Volume
Keeping this MySQL client connection open, log in to your database server from a new terminal window.
Warning: If you close this connection, the lock will be discarded and writes will resume, rendering the snapshot inconsistent.
We can now take a snapshot of the mysql_data
logical volume. We’ll allocate 100GB of buffer space to absorb writes and other changes as we perform the physical backup. To create the LVM snapshot, run the following lvcreate
command:
sudo lvcreate -L 100G -s -n mysql_data_snap /dev/vg1/mysql_data
The -L
flag specifies the size of the logical volume, in this case 100GB. -s
indicates that the logical volume will be a snapshot, in this case of the /dev/vg1/mysql_data
logical volume. We’ve chosen to name this snapshot volume mysql_data_snap
.
You should see the following result:
OutputLogical volume "mysql_data_snap" created.
This indicates that we now have a copy of the mysql_data
logical volume from which we can perform a backup.
Now that we’ve essentially “frozen” our MySQL data files at a point in time, we can unlock our database tables and resume writes. From your open MySQL connection, run the following command:
- UNLOCK TABLES;
You should see the following result:
OutputQuery OK, 0 rows affected (0.00 sec)
The tables have been unlocked and you can now safely close this connection.
At this point, your database is still live and accepting incoming connections and writes, but we have a consistent snapshot of the data at the point in time when we ran FLUSH TABLES WITH READ LOCK
(or to be completely accurate, the point in time when the last write query after the FLUSH
completed).
The final step consists of mounting this snapshot so that we can access these frozen data files.
First, we’ll create a mount point called /backup_src
:
- sudo mkdir /backup_src
Now, we’ll mount the snapshot volume to /backup_src
:
- sudo mount /dev/vg1/mysql_data_snap /backup_src
We can now access the frozen data files. Let’s take a look:
- cd /backup_src
- ls
You should see your MySQL data directory:
Outputlost+found mysql
Now that we have access to a consistent snapshot of our data, we can back it up to a DigitalOcean Space.
To upload this backup to our DigitalOcean Space, we’ll use the s3cmd
tool which we installed and configured in the prerequisite steps.
We’ll first test our s3cmd
configuration and attempt to access our backups Space (in this tutorial our space is named mysql-backup-demo
):
- s3cmd info s3://mysql-backup-demo/
You should see the following output:
Outputs3://mysql-backup-demo/ (bucket):
Location: nyc3
Payer: BucketOwner
Expiration Rule: none
Policy: none
CORS: none
ACL: 3587522: FULL_CONTROL
This output indicates that the connection was successful and s3cmd
can transfer objects to the Space.
We’ll now compress and upload our MySQL data directory to the mysql-backup-demo
space:
- sudo tar -czvf - /backup_src/mysql | s3cmd put - s3://mysql-backup-demo/mysql_backup_180423.tar.gz
Here, we use tar
to compress and archive the MySQL data directory, and pipe the output to s3cmd
, which we use to transfer the compressed archive to Spaces. We’ve named the compressed archive mysql_backup_180423.tar.gz
.
Since we used tar
in verbose mode, you’ll see the list of files being compressed (to hide this output, omit the -v
flag in the above command).
The output will conclude with the following file transfer information:
Output...
upload: '<stdin>' -> 's3://mysql-backup-demo/mysql_backup_180423.tar.gz' [part 1, 1417kB]
1451996 of 1451996 100% in 0s 1993.41 kB/s done
Once the transfer completes, we’ll verify that the file was successfully transferred to our Space by listing the Space contents:
- s3cmd ls s3://mysql-backup-demo/
You should see the backup archive file:
Output2018-04-23 20:39 297 s3://mysql-backup-demo/mysql_backup_180423.tar.gz
At this point, we’ve successfully completed a physical MySQL backup to DigitalOcean Spaces.
We’ll now unmount and drop the snapshot volume, restoring the used space to our volume group vg1
.
Now that our data has been backed up, we no longer have any use for the snapshot volume we created earlier in this tutorial and can safely drop it.
To unmount the volume, run the following command:
- sudo umount /backup_src
Replace /backup_src
with your snapshot volume’s mount point.
We can now drop the snapshot volume. To do so, run the following command:
- sudo lvremove vg1/mysql_data_snap
Here, vg1
corresponds to your volume group name, and mysql_data_snap
to your snapshot volume name.
You’ll be prompted to confirm removal, to which you should respond Y.
You should see the following output:
Output Logical volume "mysql_data_snap" successfully removed
The snapshot volume has successfully been dropped. You’ve now completed a full MySQL physical backup and uploaded it to your DigitalOcean Space.
We’ll conclude this tutorial by quickly running through a recovery scenario.
To restore our MySQL database from the physical backup we previously uploaded to Spaces, we’ll transfer the backup to our database server, and then use the extracted files as our restored MySQL data directory.
Let’s first transfer the backup from our Space back to the user’s home directory on the database server:
- s3cmd get s3://mysql-backup-demo/mysql_backup_180423.tar.gz ~/mysql_backup_180423.tar.gz
You should see some file transfer output:
Outputdownload: 's3://mysql-backup-demo/mysql_backup_180423.tar.gz' -> '~/mysql_backup_180423.tar.gz' [1 of 1]
1451889 of 1451889 100% in 0s 38.49 MB/s done
We’ll now stop the running database server and purge the existing data directory, as we’d like to test a clean restore from the physical backup files.
First, stop the MySQL server:
- sudo service mysql stop
Now, delete the contents of your MySQL data directory:
- sudo rm -rf /data/*
Recall that in this tutorial, the non-default MySQL data directory path is /data
.
Now, extract the physical backup archive to your MySQL data directory:
- sudo tar -xzvf ~/mysql_backup_180423.tar.gz -C /data
Now that the data files have been restored, we can restart the MySQL database and allow it to recover:
- sudo service mysql start
Finally, we can log in to our database server to verify that the restore completed successfully:
- mysql -u root -p
After entering your password, you should see the MySQL client prompt:
OutputWelcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.7.21-0ubuntu0.16.04.1 (Ubuntu)
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
From here you can scan some tables to verify that your data is intact.
In this tutorial, we’ve demonstrated how to leverage LVM’s snapshot feature to freeze the filesystem and perform a full physical backup and restore of a running MySQL instance. If you use LVM to manage one or more storage volumes that contain your MySQL data, this feature provides a convenient method for backing up your production database.
In a production setting, this procedure should ideally be scripted and scheduled with proper logging, monitoring, and alerting. In addition, a FLUSH TABLES WITH READ LOCK
(no matter how brief) should not be run on the master server but on a minimally loaded replica. Note that with slight modifications, you can also adapt the above procedure to quickly spin up replicas from a master physical backup.
If your MySQL instance exclusively uses InnoDB as its storage engine, you can also use Percona XtraBackup to perform physical backups of your database in a similar fashion. To learn more, consult our tutorial on How To Back Up MySQL Databases to Object Storage with Percona on Ubuntu 16.04.
A reasonable alternative to uploading physical backup files to Spaces would be to use LVM snapshots in combination with Droplet Snapshots. To learn more about Droplet Snapshots, consult DigitalOcean Backups and Snapshots Explained.
To learn more about DigitalOcean Spaces, the object store used in this guide, consult An Introduction To DigitalOcean Spaces.
]]>Benchmarking allows you to estimate infrastructure performance so that you can determine whether or not a particular setup can serve your workload needs. This is an important component of maintaining high server performance and scaling to meet growing computing needs. With benchmark testing, you can monitor server resources, optimize for performance, manage utilization, and predict issues that may occur.
In this tutorial, we’ll go over best practices to benchmark your DigitalOcean Block Storage Volumes by simulating workloads that mimic your application.
You’ll want to determine the specifications of the workload you are testing in order to configure an appropriate Droplet and Block Storage Volume setup. If you are benchmarking DigitalOcean Volumes against an alternate product, make sure you select plans with similar configuration so you get a more proximate comparison.
For guidance on setting up a Droplet, refer to our tutorial “How To Create Your First DigitalOcean Droplet.” The Block Storage Volume can be created simultaneously with the Droplet, and in some cases you can create it later from the Volumes tab under Droplets in the Control Panel. To learn more about creating and managing volumes, you can read “An Introduction to DigitalOcean Block Storage.”
Format the volume with the filesystem that meets your performance requirements. The most popular default filesystem is Ext4, which is more performant than previous generations Ext3 and Ext2. The XFS filesystem specializes in performance and large data files. You can read more about filesystems in the “How To Partition and Format DigitalOcean Block Storage Volumes in Linux.” If you have no specific file system or configuration preference, the process of creating and mounting block volumes for you is automated in the Control Panel.
With your setup in place, we can move on to discussing benchmarking tools and configurations you can use to get the most out of your performance testing.
We’ll be discussing the performance measuring tool fio for testing performance, as it is extremely flexible and supported by most distributions. Alternate benchmarking tools you may want to research and use include Bonnie++, btest, and Filebench.
To install fio on an Ubuntu server, you should first update your package list and then install it, by using the following commands:
- sudo apt update
- sudo apt install fio
Each benchmark tool comes with a variety of parameters that you can tune in order to get the optimal performance for your test.
One parameter worth tuning is the queue depth, which is the parallelism at which the volume exhibits the best performance. Usually a queue depth of 1
indicates that the workload cannot start another transaction until the prior transaction has been completed. Only use low queue depth numbers for your tests if you want to simulate a highly parallelized application. Otherwise, keep raising the queue depth until you get the performance you desire for your application.
With the benchmarking tool fio, some typical configuration options include:
Option | Recommendations |
---|---|
iodepth |
The queue depth that fio will issue to the file. In order to hit best input/output (I/O) rates, a number greater than iodepth=64 is recommended. |
bs |
Block size in bytes for I/O to use. File systems use 4K for metadata but tend to store files in much larger block sizes. Databases typically issue 8-16K sized I/O. For peak bandwidth tests, we recommend a block size of bs=64k or greater. |
runtime |
The time in seconds to run the benchmark. We recommend a runtime greater than 60 seconds, usually in the range of runtime=120s to runtime=300s . |
ioengine |
The ioengine option defines how the job issues I/O to the file. We recommend ioengine=libaio which refers to Linux native asynchronous I/O. |
direct |
Takes a Boolean value: 0 uses the filesystem cache returning a value closest to the application behavior which may result in higher than typical benchmark results; and 1 skips any file system caching behavior, returning the closest performance to what the block volume can do. We recommend direct=1 . |
sync |
Use synchronous I/O for buffered writes. This option takes a Boolean value: 0 means not to force unit access, allowing for writeback cache behavior to operate like it does on normal disk drives with fio behaving more like a filesystem; 1 forces unit access, meaning I/O will not complete until the disk has guaranteed physical placement. We recommend sync=0 . |
size |
Size of test file, taking an integer. We typically recommend at least 20 gigabytes. Note that DigitalOcean’s performance does not vary with volume size. |
With these configurations in mind, we can now go over some example benchmark tests that you may want to run.
Here are some example benchmarks you can run. In the next section of this tutorial, we’ll go over how to inspect the output you receive in greater depth.
In the following commands, we’re pointing to a fio.test
file on a volume based in the NYC3 data center, be sure to update it to point to the specific filesystem you would like to use.
This test performs a random write of 1MB onto the block volumes.
- fio --filename=/mnt/volume-nyc3-04/fio.test \
- --direct=1 \
- --rw=randwrite \
- --ioengine=libaio \
- --bs=1024K \
- --iodepth=32 \
- --name=bw-test \
- --runtime=120s \
For a standard Droplet, we expect output of 200MB/sec
. If you’re working with a High CPU Droplet, you should expect output of 300MB/sec
.
This will measure how quickly multiple small files can be read from a device.
- fio --filename=/mnt/volume-nyc3-04/fio.test \
- --direct=1 \
- --rw=randread \
- --ioengine=libaio \
- --bs=4K \
- --iodepth=128 \
- --name=rand-r \
- --runtime=120s \
For a standard Droplet we expect output of 5000
I/O operations per second (IOPS). For a High CPU Droplet, we expect an output greater than 6000
IOPS.
This will measure how quickly multiple small files can be written.
- fio --filename=/mnt/volume-nyc3-04/fio.test \
- --direct=1 \
- --rw=randwrite \
- --ioengine=libaio \
- --bs=4K \
- --iodepth=128 \
- --name=rand-w \
- --runtime=120s \
An output of 5000
IOPS is what we expect for a standard Droplet, while an output that is greater than 6000
IOPS is what we expect for a High CPU Droplet.
We’ll determine the time required to find and access the proper data blocks on the disk with a read latency test.
- fio --filename=/mnt/volume-nyc3-04/fio.test \
- --direct=1 \
- --rw=randread \
- --ioengine=libaio \
- --bs=4K \
- --iodepth=1 \
- --name=lat-read \
- --runtime=120s \
For this test, we expect an output that returns less than 5ms
.
This test measures the delay from when a disk write request is created until it’s completed.
- fio --filename=/mnt/volume-nyc3-04/fio.test \
- --direct=1 \
- --rw=randwrite \
- --ioengine=libaio \
- --bs=4K \
- --iodepth=1 \
- --name=lat-write \
- --runtime=120s \
Here, we also expect an output of less than 5ms
for this test.
Once you run your test, you’ll be inspecting the resulting output to examine how many read and write operations were serviced by DigitalOcean Volumes. You’ll want to pay attention to how long it took for each test to complete.
Below is sample output from a write bandwidth test.
- fio --filename=/mnt/volume-nyc3-04/test.fio --direct=1 --rw=randwrite --ioengine=libaio --bs=1024k --iodepth=32 --name=bw-test --runtime=120s
Outputbw-test: (groupid=0, jobs=1): err= 0: pid=2584: Fri Apr 20 17:14:19 2018
write: io=22937MB, bw=195468KB/s, iops=190, runt=120160msec
slat (usec): min=54, max=622, avg=135.46, stdev=23.21
clat (msec): min=7, max=779, avg=167.48, stdev=26.28
lat (msec): min=7, max=779, avg=167.62, stdev=26.28
clat percentiles (msec):
| 1.00th=[ 101], 5.00th=[ 155], 10.00th=[ 159], 20.00th=[ 163],
| 30.00th=[ 165], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 167],
| 70.00th=[ 169], 80.00th=[ 169], 90.00th=[ 172], 95.00th=[ 178],
| 99.00th=[ 306], 99.50th=[ 363], 99.90th=[ 420], 99.95th=[ 474],
| 99.99th=[ 545]
bw (KB /s): min=137730, max=254485, per=100.00%, avg=195681.88, stdev=9590.24
lat (msec) : 10=0.01%, 20=0.03%, 50=0.37%, 100=0.58%, 250=97.55%
lat (msec) : 500=1.44%, 750=0.03%, 1000=0.01%
cpu : usr=1.76%, sys=1.83%, ctx=22777, majf=0, minf=11
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=22937/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: io=22937MB, aggrb=195468KB/s, minb=195468KB/s, maxb=195468KB/s, mint=120160msec, maxt=120160msec
The highlighted line in the output above shows our average bandwidth as bw=195468KB/s
, as well as our I/O operations per second (IOPS) as iops=190
. In this particular scenario, the IOPS is low because we were performing 1MB writes at a peak rate of 200MB per second (190iops * 1M =~ 190MB/sec
).
When performing a read latency test, you will receive the metrics on lines that look like the following:
Outputlat-read: (groupid=0, jobs=1): err= 0: pid=2628: Fri Apr 20 17:32:51 2018
read : io=855740KB, bw=7131.2KB/s, iops=1782, runt=120001msec
slat (usec): min=8, max=434, avg=16.77, stdev= 5.92
clat (usec): min=2, max=450994, avg=539.15, stdev=2188.85
lat (usec): min=53, max=451010, avg=556.61, stdev=2188.91
In the example above, we can see that I/O latency is 556 usec or microseconds (or .5 ms or milliseconds). This indicates the time it takes to perform a single 4K I/O to the block volume.
Latency is impacted by several factors, including the performance of the storage system, the size of the I/O, the queue depth, and any throttle limits that are triggered.
When your benchmark tests are complete, you can delete the Droplet and the volume.
Note: DigitalOcean Block Storage Volumes have additional performance enhancing features (like bursting) that may result in higher than typical benchmarks. The effect of this can be mitigated by running a profile longer then a few seconds.
DigitalOcean Block Storage Volumes are SAN-based SSD storage. Because of this, the performance of a given volume scales with the block size and queue depth. That is, the storage performs best when it is given a lot of work to do all at the same time.
Below is a graph showing an example of performance on parallelism. Click the image to expand it.
DigitalOcean performance is limited by max IOPS and max bandwidth, whichever is reached first. On DigitalOcean, performance does not change with volume size. However, because IOPS and bandwidth are capped at certain rates, there is a crossover effect. That is, increasing queue depth on small-sized I/O will not reach peak bandwidth of 200MB/sec as it will hit the IOPS limit instead. For large blocks, peak IOPS of 5000 will not be reached as it will hit the bandwidth limit of 200MB/sec instead.
As a rule of thumb, using a block size of 32K or greater will result in less than 5000 IOPS because it will hit the bandwidth limit of 200MB/sec, while using a block size or 16K or less will result in less than the bandwidth limit of 200MB/sec because it will hit 5000 IOPS.
As a practical example, let’s compare IOPS and bandwidth.
The first graph shows typical bandwidth given a block size and queue depth of an application. We’ll consider two different block sizes of 4K and 256K.
When looking at the 4K values we see that it hits 5000 IOPS at a queue depth of 4, but we only ever see it hit a bandwidth of 20MB/sec. That is because 5000 IOPS * 4K = 20MB/sec
. This means that a 4K workload will never exceed 20MB/sec since it is I/O capped.
When looking at the 256K workload, we see that it reaches 200MB/sec at a queue depth of 2, but it will never see an IOPS rate higher than 800 IOPS because 200MB/sec / 256K = 800 IOPS
.
DigitalOcean Block Storage Volumes are tuned for typical filesystem workloads of 16K-64K. At these block sizes, we see a good tradeoff between IOPS and bandwidth.
Block Size | IOPS | Bandwidth |
---|---|---|
4K | ~5000 IOPS | ~20MB/sec |
16K | ~5000 IOPS | ~80MB/sec |
32K | ~5000 IOPS | ~160MB/sec |
64K | ~3200 IOPS | ~200MB/sec |
256K | ~800 IOPS | ~200MB/sec |
Once you receive the output from benchmarking a simulated I/O workload, you’ll be able to analyze the best setup based on the needs of your app.
In this guide, we covered how to carry out benchmarking your DigitalOcean Droplets and Block Storage Volumes in order to simulate anticipated workflows. Incorporating benchmark testing as part of your workflow can ensure that you maintain appropriate scale for your applications and predict issues before they arise.
To continue learning more about how to work with DigitalOcean Block Storage, you can read “How to Create an Encrypted File System on a DigitalOcean Block Storage Volume” and “How To Work with DigitalOcean Block Storage Using Doctl.”
]]>Regular database backups are a crucial step in guarding against unintended data loss events. In general, there are two broad categories of backups: filesystem-level (“physical”) backups and logical backups.
Filesystem-level backups involve snapshotting the underlying data files at a point in time, and allowing the database to cleanly recover using the state captured in the snapshotted files. They are instrumental in backing up large databases quickly, especially when used in tandem with filesystem snapshots, such as LVM snapshots, or block storage volume snapshots, such as DigitalOcean Block Storage Snapshots.
Logical backups involve using a tool (e.g. mongodump
or pg_dump
) to export data from the database into backup files, which are then restored using a corresponding restore tool (e.g. mongorestore
or pg_restore
). They offer granular control over what data to back up and restore and backups are often portable across database versions and installations. As logical backup tools read all data being backed up through memory, they can be slow and cause non-trivial additional load for particularly large databases.
Designing an effective backup and recovery strategy often involves trading off performance impact, implementation costs, and data storage costs with recovery speed, data integrity, and backup coverage. The optimal solution will depend on your recovery point and time objectives and database scale and architecture.
In this guide, we’ll demonstrate how to back up a MongoDB database using mongodump
, a built-in logical backup tool. We’ll then show how to compress and upload the resulting serialized data backup files to DigitalOcean Spaces, a highly redundant object store. We’ll also show how to regularly schedule the backup and upload operation using Bash and cron
, and finally conclude with a sample data recovery scenario.
By the end of this tutorial, you’ll have implemented the framework for an extensible automated backup strategy that will allow you to quickly recover should your application suffer from data loss. For smaller to medium-sized databases, logical backups using mongodump
give you fine-grained control over what data to back up and recover. Storage of these compressed backup archives in DigitalOcean Spaces ensures that they are readily available in a durable object store, so that your application data is protected and quickly recoverable should a data loss event occur.
Note: There may be some performance impact when using the mongodump
tool, especially on highly loaded databases. You should test this procedure first using a non-production database with simulated load to verify that this method will work in your production deployment.
Before you get started with this guide, make sure you have the following prerequisites available to you:
s3cmd
command-line file transfer client (2.X) installed as detailed in Step 1 of How To Use Logrotate and S3cmd to Archive Logs to Object Storage on Ubuntu 16.04s3cmd
configured to access your Space, as detailed in How To Configure s3cmd 2.x To Manage DigitalOcean SpacesOnce you’ve logged in to your Droplet, have MongoDB up and running, and have created your Space, you’re ready to get started.
If you’re starting from a clean MongoDB installation and haven’t stored any data yet, you should first insert some sample data into a dummy restaurants
collection for test purposes. If you already have some collections and documents stored in your database, feel free to skip this step and continue on to Step 2.
First, connect to the running database using the MongoDB shell:
mongo
You’ll see the following Mongo shell prompt:
MongoDB shell version: 3.2.19
connecting to: test
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
http://docs.mongodb.org/
Questions? Try the support group
http://groups.google.com/group/mongodb-user
Server has startup warnings:
2018-04-11T20:30:57.320+0000 I CONTROL [initandlisten]
2018-04-11T20:30:57.320+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2018-04-11T20:30:57.320+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2018-04-11T20:30:57.320+0000 I CONTROL [initandlisten]
>
By default, the shell connects to the test
database.
Let’s list the collections present in the test
database:
- show collections
Since we haven’t inserted anything into the database yet, there are no collections, and we’re brought back to the prompt with no output.
Let’s insert a document into a dummy restaurants
collection, which will automatically be created (as it doesn’t yet exist):
- db.restaurants.insert({'name': 'Pizzeria Sammy'})
You’ll see the following output:
OutputWriteResult({ "nInserted" : 1 })
This indicates that the insert operation was successful.
Let’s list collections once again:
show collections
We now see our newly created restaurants
collection:
Outputrestaurants
To exit the MongoDB shell, press CTRL
+ D
.
Now that we’ve stored some sample data in the database, we’re ready to back it up.
mongodump
to Back Up MongoDB DataWe’ll now use the built-in mongodump
utility to back up (or “dump”) an entire MongoDB database to a compressed archive file.
First, let’s create a temporary directory called backup
to store the archive created by mongodump
:
- mkdir backup
Now, let’s back up the test
database in this MongoDB instance to a compressed archive file called test_dump.gz
. If your instance contains other databases, you can substitute another database name for test
after the --db
flag. You also may omit the --db
flag to back up all databases in your MongoDB instance.
Note: The following command should be run from the terminal and not the Mongo shell.
- mongodump --db test --archive=./backup/test_dump.gz --gzip
Here, we use the --archive
flag to specify that we’d like to save all the data to a single archive file (whose location is specified by the archive
parameter) , and the --gzip
flag to specify that we’d like to compress this file. In addition, you optionally may use the --collection
or --query
flags to select a given collection or query to archive. To learn more about these flags, consult the mongodump
documentation.
After running the dump command, you will see the following output:
Output2018-04-13T16:29:32.191+0000 writing test.restaurants to archive './backup/test_dump.gz'
2018-04-13T16:29:32.192+0000 done dumping test.restaurants (1 document)
This indicates that our test data has successfully been dumped.
In the next step, we’ll upload this backup archive to object storage.
To upload this archive to our DigitalOcean Space, we’ll need to use the s3cmd
tool, which we installed and configured in the Prerequisites.
We’ll first test our s3cmd
configuration and attempt to access our backups Space. In this tutorial, we’ll use mongo-backup-demo
as our Space name, but you should fill in the actual name of your Space:
- s3cmd info s3://mongo-backup-demo/
You’ll see the following output:
Outputs3://mongo-backup-demo/ (bucket):
Location: nyc3
Payer: BucketOwner
Expiration Rule: none
Policy: none
CORS: none
ACL: 3587522: FULL_CONTROL
Which indicates the connection was successful and s3cmd
can transfer objects to the Space.
Let’s transfer the archive we created in Step 2 to our Space using the put
command:
- s3cmd put ./backup/test_dump.gz s3://mongo-backup-demo/
You’ll see some file transfer output:
Outputupload: './backup/test_dump.gz' -> 's3://mongo-backup-demo/test_dump.gz' [1 of 1]
297 of 297 100% in 0s 25.28 kB/s done
Once the transfer completes, we’ll verify that the file was successfully transferred to our Space by listing the Space contents:
- s3cmd ls s3://mongo-backup-demo/
You should see the backup archive file:
Output2018-04-13 20:39 297 s3://mongo-backup-demo/test_dump.gz
At this point you’ve successfully backed up the test
MongoDB database and transferred the backup archive to your DigitalOcean Space.
In the next section we’ll cover how to script the above procedure using Bash so that we can schedule it using cron
.
Now that we’ve backed up our MongoDB database to a compressed archive file and transferred this file to our Space, we can combine these manual steps into a single Bash script.
We’ll first write a script combining the mongodump
and s3cmd put
commands, and add a few extra bells and whistles, like some logging (using echo
s).
Open a blank file in your preferred text editor (here we’ll use nano
):
- nano backup_mongo.sh
Paste in the following code snippets, being sure to update the relevant values to refer to your own Space, database, and file names. We’ll call the file backup_mongo.sh
, but you may name this file however you’d like. You can also find the full script at the end of this section.
Let’s go through this script piece by piece:
#!/bin/bash
set -e
...
Here, #!/bin/bash
tells the shell to interpret the script as Bash code. set -e
tells the interpreter to exit immediately if any of the script commands fail.
...
SPACE_NAME=mongo-backup-demo
BACKUP_NAME=$(date +%y%m%d_%H%M%S).gz
DB=test
...
In this section, we’re setting three variables that we’ll use later on:
SPACE_NAME
: The name of the DigitalOcean space to which we’re uploading our backup fileBACKUP_NAME
: The backup archive’s name. Here, we set it to a basic date-time string.DB
: Specifies which MongoDB database the script will back up. If you’re backing up the entire MongoDB instance (all databases), this variable won’t be used....
date
echo "Backing up MongoDB database to DigitalOcean Space: $SPACE_NAME"
echo "Dumping MongoDB $DB database to compressed archive"
mongodump --db $DB --archive=$HOME/backup/tmp_dump.gz --gzip
echo "Copying compressed archive to DigitalOcean Space: $SPACE_NAME"
s3cmd put $HOME/backup/tmp_dump.gz s3://$SPACE_NAME/$BACKUP_NAME
...
We then print the date and time (for logging purposes), and begin the backup by running the mongodump
command we tested above. We once again save the backup archive to ~/backup/
.
We next use s3cmd
to copy this archive to the location specified by those two SPACE_NAME
and BACKUP_NAME
variables. For example, if our Space name is mongo-backup-demo
and the current date and time is 2018/04/12 12:42:21
, the backup will be named 180412_124221.gz
and it’ll be saved to the mongo-backup-demo
Space.
...
echo "Cleaning up compressed archive"
rm $HOME/backup/tmp_dump.gz
echo 'Backup complete!'
Here we remove the backup archive from the ~/backup
directory as we’ve successfully copied it to our Space, with final output indicating that the backup is complete.
After combining all these code snippets, the full script should look like this:
#!/bin/bash
set -e
SPACE_NAME=mongo-backup-demo
BACKUP_NAME=$(date +%y%m%d_%H%M%S).gz
DB=test
date
echo "Backing up MongoDB database to DigitalOcean Space: $SPACE_NAME"
echo "Dumping MongoDB $DB database to compressed archive"
mongodump --db $DB --archive=$HOME/backup/tmp_dump.gz --gzip
echo "Copying compressed archive to DigitalOcean Space: $SPACE_NAME"
s3cmd put $HOME/backup/tmp_dump.gz s3://$SPACE_NAME/$BACKUP_NAME
echo "Cleaning up compressed archive"
rm $HOME/backup/tmp_dump.gz
echo 'Backup complete!'
Be sure to save this file when you’re done.
Next, we’ll test this script to validate that all the subcommands work.
Let’s quickly run the backup_mongo.sh
script.
First, make the script executable:
- chmod +x backup_mongo.sh
Now, run the script:
- ./backup_mongo.sh
You will see the following output:
OutputMon Apr 16 22:20:26 UTC 2018
Backing up MongoDB database to DigitalOcean Space: mongo-backup-demo
Dumping MongoDB test database to compressed archive
2018-04-16T22:20:26.664+0000 writing test.restaurants to archive '/home/sammy/backup/tmp_dump.gz'
2018-04-16T22:20:26.671+0000 done dumping test.restaurants (1 document)
Copying compressed archive to DigitalOcean Space: mongo-backup-demo
upload: '/home/sammy/backup/tmp_dump.gz' -> 's3://mongo-backup-demo/180416_222026.gz' [1 of 1]
297 of 297 100% in 0s 3.47 kB/s done
Cleaning up compressed archive
Backup complete!
We’ve successfully created a backup shell script and can now move on to scheduling it using cron
.
To schedule a nightly run of the backup script, we’ll use cron
, a job scheduling utility built-in to Unix-like operating systems.
First, we’ll create a directory to store the logs for our backup script. Next, we’ll add the backup script to the crontab (cron
’s configuration file) so cron
schedules it to run nightly. Because cron
supports any regular frequency, you can optionally schedule weekly or monthly backups.
Let’s create a directory to store our backup script’s log files. These logs will allow us to periodically check up on the backup script to ensure that all is well, and debug should some command fail.
Create a mongo_backup
subdirectory in /var/log
(by convention used for logging):
- sudo mkdir /var/log/mongo_backup
Now, make that directory writeable to our Unix user. In this case, our user’s name is sammy, but you should use the relevant non-root username with sudo privileges for your server.
- sudo chown sammy:sammy /var/log/mongo_backup
Our Unix user sammy can now write to /var/log/mongo_backup
. Since the cronjob will run as sammy, it can now write its log files to this directory.
Let’s create the scheduled cronjob.
To create the cronjob, we’ll edit the file containing the list of scheduled jobs, called the “crontab.” Note that there are multiple crontabs, one per user, and a system-wide crontab at /etc/crontab
. In this tutorial, we’ll run the backup script as our user sammy; depending on your use case, you may elect to run it from the system-wide crontab.
Open up the crontab for editing:
- crontab -e
You’ll see the following menu allowing you to choose your preferred text editor:
Outputno crontab for sammy - using an empty one
Select an editor. To change later, run 'select-editor'.
1. /bin/ed
2. /bin/nano <---- easiest
3. /usr/bin/vim.basic
4. /usr/bin/vim.tiny
Choose 1-4 [2]: no crontab for sammy - using an empty one
Select your preferred editor; to choose nano
enter 2
. Now, append the following line to the file, following the commented-out section:
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h dom mon dow command
0 2 * * * /home/sammy/backup_mongo.sh >>/var/log/mongo_backup/mongo_backup.log 2>&1
Be sure to include a trailing newline at the end of the crontab. Save and close the file.
You’ll see the following output:
Outputno crontab for sammy - using an empty one
crontab: installing new crontab
The backup script will now run at 2:00 AM every morning. Both stdout
and stderr
(the output and error streams) will be piped and appended to a log file called mongo_backup.log
in the log directory we created earlier.
You may change 0 2 * * *
(execute nightly at 2:00 AM in cron syntax) to your desired backup frequency and time. To learn more about cron and its syntax, consult our tutorial on How To Use Cron To Automate Tasks On A VPS.
We’ll conclude this tutorial with a quick recovery exercise to ensure that our backups are functional.
Any backup strategy should contain a recovery procedure that is routinely tested. Here, we’ll quickly test a restore from the compressed backup file we uploaded to DigitalOcean spaces.
First, we’ll download test_dump.gz
from our Space to the home directory in our MongoDB Droplet:
- s3cmd get s3://mongo-backup-demo/test_dump.gz
You will see the following output:
Outputdownload: 's3://mongo-backup-demo/test_dump.gz' -> './test_dump.gz' [1 of 1]
297 of 297 100% in 0s 1305.79 B/s done
If you began this tutorial with a fresh MongoDB instance, you’ll recall that it only contained the test
database, which in turn was the only database we backed up.
For demonstration purposes, we’ll now drop this test database so that we can perform a clean restore. If we don’t perform this first step, the restore procedure will encounter the original documents, which it’ll skip. In your particular use case restoring only new documents may be acceptable, but for the purposes of this tutorial we’d like to explicitly test a full restore into an empty database.
Connect to your MongoDB instance using the mongo
shell:
- mongo
Now, use
the test
database, and drop it from the MongoDB instance:
- use test
- db.dropDatabase()
You’ll see the following output confirming the test
drop:
Output{ "dropped" : "test", "ok" : 1 }
Now, exit the mongo
shell and execute the mongorestore
command:
- mongorestore --gzip --archive=test_dump.gz --db test
Here, we specify that the source backup file is compressed and in “archive file” form (recall that we used the --archive
and --gzip
flags when calling mongodump
), and that we’d like to restore to the test
database.
You will see the following output:
Output2018-04-16T23:10:07.317+0000 creating intents for archive
2018-04-16T23:10:07.453+0000 reading metadata for test.restaurants from archive 'test_dump.gz'
2018-04-16T23:10:07.497+0000 restoring test.restaurants from archive 'test_dump.gz'
2018-04-16T23:10:07.541+0000 restoring indexes for collection test.restaurants from metadata
2018-04-16T23:10:07.541+0000 finished restoring test.restaurants (1 document)
2018-04-16T23:10:07.541+0000 done
This indicates that the test
restore succeeded.
To conclude, let’s confirm that our initial restaurants
data has successfully been restored.
Open up the MongoDB shell and query the restaurants
collection:
- db.restaurants.find()
You should see the object we saved in the first step of this tutorial:
Output{ "_id" : ObjectId("5ace7614dbdf8137afe60025"), "name" : "Pizzeria Sammy" }
You’ve now successfully implemented and tested this MongoDB backup strategy.
In this tutorial, we’ve learned how to implement and test a strategy for nightly logical MongoDB backups.
This guide can be extended or modified in many ways. Here are some quick suggestions:
As the mongodump
procedure involves quickly reading through all the dumped data, this backup method is most suitable for small- to medium-sized databases, particularly for partial backups such as a specific collection or result set. Filesystem-level backups are recommended for larger deployments. To learn more about filesystem-level MongoDB backups, consult this tutorial on How To Back Up MongoDB Using Droplet Snapshots. To learn more about various methods of backing up a MongoDB database, you can consult the MongoDB manual.
The solution presented in this tutorial leverages mongodump
for granular control over backup data coverage and DigitalOcean Spaces for cost-effective and durable long-term data storage. To learn more about the mongodump
backup utility, consult its reference page in the MongoDB manual. To learn more about DigitalOcean Spaces, you can read An Introduction To DigitalOcean Spaces.
Duplicity is a command-line utility written in Python that produces encrypted tar volumes for storage on a local or remote repository. It uses the GNU Privacy Guard (GPG) to encrypt and sign its archives and the rsync algorithm to create incremental, space-efficient backups. Backups can be transmitted to a variety of repositories, including local file storage, SFTP or FTP servers, and S3-compatible object stores.
In this tutorial, we will install Duplicity and go over how to back up project data to DigitalOcean Spaces, an S3-compatible object storage service. We will create a Spaces repository for this purpose, and cover how to manually back up data to it. Finally, we will automate this process by creating a script that will set up incremental and weekly full backup schedules.
For this tutorial, you will need:
One Ubuntu 16.04 server, set up following our Initial Server Setup with Ubuntu 16.04 tutorial. After following this tutorial, you should have a non-root sudo user.
A DigitalOcean Space and API key, created by following How To Create a DigitalOcean Space and API Key. Be sure to note the following credentials for your Space:
Once you have your Space set up and this information in hand, you can move on to installing Duplicity.
To get an up-to-date version of Duplicity, we can install it from the Duplicity releases Personal Package Archive (PPA):
- sudo apt-add-repository ppa:duplicity-team/ppa
We will also install the python-boto
package to have access to Boto, a Python package that provides interfaces to Amazon Web Services. This will help us take advantage of Spaces’ interoperability with the AWS S3 API. We will install python-boto
from the official Ubuntu repositories, since this version is compatible with the version of Python that ships with our Ubuntu server image. If you would prefer to use Boto3, you can install it from source, although feature compatibility with Python 3.3+ is still under development.
In addition to python-boto
, we will also install Haveged, a tool that will help us generate the entropy necessary to create our GPG keys. In order to create these keys, GPG draws on the level of entropy or unpredictability in our system. Installing haveged
will help us speed up the key creation process.
Before installing these packages, update the local repository index:
- sudo apt-get update
Then install duplicity
, python-boto
, and haveged
by typing:
- sudo apt-get install duplicity haveged python-boto
Press y
when prompted to confirm installation. We now have Duplicity installed on our system and are ready to create our project folders and configuration files.
To demonstrate how the backup process works, we will create a directory for our backups in our non-root user’s home directory, along with some sample data. We will call our directory sammy_backups
:
- mkdir ~/sammy_backups
Next, create a sample project file called historical_sharks.txt
:
- echo "The ancient Megalodon shark reached lengths of up to 59 feet, and is widely regarded as one of history's most fearsome predators." >> ~/sammy_backups/historical_sharks.txt
With our backup directory and test data in place, we are ready to generate a GPG key for our non-root user.
Next, we will generate a GPG key pair for our user. To ensure the secure transmission of information, GPG uses public key encryption. What this means in our context is that data will be encrypted to our public key and sent to our repository. For more about GPG keys and encryption see our tutorial on How To Use GPG to Sign and Encrypt Messages.
Our keyrings will be stored on our user account in a directory called ~/.gnupg
, which will be created when we generate the keys. When we use the duplicity
command, we will specify a public key identifier that points to our key pair. Using this identifier enables data encryption and the signature that verifies our ownership of the private key. The encrypted data will be transmitted to our repository, where it will be difficult to infer much more than file size and upload time from the files themselves. This protects our data, which our user can restore in full at any time with the private key.
GPG should be installed on our server by default. To test this, type:
- gpg --version
Once you have verified that GPG is installed, you can generate a key pair as follows:
- gpg --gen-key
You will be asked a series of questions to configure your keys:
ENTER
will confirm the default size of 2048 bits.your-GPG-key-passphrase
.After you have created these settings, gpg
will generate the keys based on the level of entropy in the system. Since we installed haveged
, our keys should be generated either very quickly or right away. You will see output that includes the following:
Output...
gpg: /home/sammy/.gnupg/trustdb.gpg: trustdb created
gpg: key your-GPG-public-key-id marked as ultimately trusted
public and secret key created and signed.
...
Take note of your-GPG-public-key-id
, as we will be using it in the next section to configure our local environment variables.
We will now set environment variables so we do not need to enter any confidential information on the command line while running the duplicity
command. These variables will be available to our user during our current session, and we will store them in a hidden directory so that they are available for later use. The variables that duplicity
will need, which we will define as environment variables, include our Spaces Access Key and Secret, and our GPG public key ID and passphrase.
To begin, let’s create a hidden directory in our user’s home directory that will store the configuration file:
- mkdir ~/.duplicity
Next, let’s create a file called .env_variables.conf
to define our variables, which we will do using export
statements. These statements will make the variables available to programs for later use. Open the file by typing:
- nano ~/.duplicity/.env_variables.conf
Within the file, set your Spaces Access Key and Secret, as well as your GPG public key ID and passphrase:
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export GPG_KEY="your-GPG-public-key-id"
export PASSPHRASE="your-GPG-key-passphrase"
Save and close the file when you are finished.
We can now set permissions on the file to ensure that only our current non-root user has read and write access:
- chmod 0600 ~/.duplicity/.env_variables.conf
Make these variables available for use in the current Bash session by typing:
- source ~/.duplicity/.env_variables.conf
Next, we will run duplicity
to create a manual, full backup of our ~/sammy_backups
directory. Running duplicity
without the full
action will create an initial full backup, followed by incremental backups. We will create a full backup in our first use of the command, but should you wish to create another full manual backup of this directory, you would need to specify the full
action.
Other options that we will define in our command include:
--verbosity
: This will specify the level of information we would like in our output. We will specify info
, which will provide more detail than the default notice
setting.--encrypt-sign-key
: This will tell duplicity
to encrypt to the public key in the pair we identified with your-GPG-public-key-id
in the GPG_KEY
variable. It will also tell duplicity
to use the same identifier to enable the signing function.--log-file
: This option will specify a location for the log files that will also be available to other programs. This will give us a straightforward place to look in case we need to troubleshoot. We will specify the log file location as /home/sammy/.duplicity/info.log
.Finally, we will specify the directory we are backing up and our repository endpoint. We will back up the ~/sammy_backups
directory in our user’s home directory. Our repository will be our Space, which we will define using the following information: s3://spaces_endpoint/bucket_name/
. You can determine your endpoint and bucket name as follows: if the URL of your Space is https://sammys-bucket.nyc3.digitaloceanspaces.com
, then sammys-bucket
is your bucket name, and nyc3.digitaloceanspaces.com
is your endpoint.
Our duplicity
command will ultimately look like this:
- duplicity --verbosity info --encrypt-sign-key=$GPG_KEY --log-file /home/sammy/.duplicity/info.log /home/sammy/sammy_backups \
- s3://nyc3.digitaloceanspaces.com/sammys-bucket/
After running this command, we will see output like the following:
Output...
--------------[ Backup Statistics ]--------------
StartTime 1522417021.39 (Fri Mar 30 13:37:01 2018)
EndTime 1522417021.40 (Fri Mar 30 13:37:01 2018)
ElapsedTime 0.01 (0.01 seconds)
SourceFiles 2
SourceFileSize 4226 (4.13 KB)
NewFiles 2
NewFileSize 4226 (4.13 KB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 2
RawDeltaSize 130 (130 bytes)
TotalDestinationSizeChange 955 (955 bytes)
Errors 0
-------------------------------------------------
To check that the files uploaded to your Space as intended, you can navigate to your Spaces page in the DigitalOcean control panel to check that they are there.
To test that we can restore our data, we will now remove our sample file and restore it from our repository. To restore files with Duplicity, we can use the --file-to-restore
option. It is also necessary to reverse the order of items in our duplicity
command: our repository URL will now act as the origin, and our backup directory will be the destination for our restored file.
Remove the file by typing:
- rm ~/sammy_backups/historical_sharks.txt
Check to make sure that the file was removed:
- cat ~/sammy_backups/historical_sharks.txt
You should see the following output:
Outputcat: /home/sammy/sammy_backups/historical_sharks.txt: No such file or directory
Next, let’s restore this file from our Space. The --file-to-restore
option allows us to specify the path of the file we would like to restore. This path should be relative to the directory that we have backed up; in our case, our relative path will be historical_sharks.txt
. We will also reverse the order of our Space URL and backup directory to indicate that we are restoring the file from our repository:
- duplicity --verbosity info --encrypt-sign-key=$GPG_KEY --log-file /home/sammy/.duplicity/info.log --file-to-restore historical_sharks.txt \
- s3://nyc3.digitaloceanspaces.com/sammys-bucket /home/sammy/sammy_backups/historical_sharks.txt
You will see output like the following:
Output...
Processing local manifest /home/sammy/.cache/duplicity/d9911d387bb9ee345a171141106ab714/duplicity-full.20180402T170008Z.manifest (195)
Found 1 volumes in manifest
Deleting /tmp/duplicity-e66MEL-tempdir/mktemp-_A24DP-6
Processed volume 1 of 1
Running cat
again will output the contents of the restored historical_sharks.txt
file:
- cat ~/sammy_backups/historical_sharks.txt
OutputThe ancient Megalodon shark reached lengths of up to 59 feet, and is widely regarded as one of history's most fearsome predators.
Now that we have created a manual backup of the ~/sammy_backups
directory and restored data from our repository, we are ready to move on to automating the backup process.
Automating the backup process can help ensure that the data in our ~/sammy_backups
directory remains recoverable and up-to-date. We can use the cron
job scheduler to create a backup schedule that will include a full backup each week and incremental backups otherwise. To learn more about using cron
to schedule tasks, check out our tutorial on How To Schedule Routine Tasks With Cron and Anacron on a VPS.
First, let’s create a backup script in our ~/.duplicity
directory:
- nano ~/.duplicity/.backup.sh
Within this file, we will first specify that this script will be run by the Bash shell:
#!/bin/bash
Next, we will create a HOME
variable to use with our source
and duplicity
commands. Be sure to replace the highlighted username, backup directory, and bucket name with your information:
...
HOME="/home/sammy"
source "$HOME/.duplicity/.env_variables.conf"
duplicity \
--verbosity info \
--encrypt-sign-key="$GPG_KEY" \
--full-if-older-than 7D \
--log-file "$HOME/.duplicity/info.log" \
/home/sammy/sammy_backups \
s3://nyc3.digitaloceanspaces.com/sammys-bucket/
The source
and duplicity
commands do the same work here that they did when we created our manual backup: source
loads our environment variables into the current context, while duplicity
creates encrypted tar volumes to send to our repository. Our options all remain the same, except for the addition of the --full-if-older-than
option. Set at 7D
, this option specifies that a full backup will happen each week, once the last full backup is older than seven days.
The final elements in our script will be unset
commands that will remove our environment variables as a security measure:
...
unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY
unset GPG_KEY
unset PASSPHRASE
The complete script will look like this:
#!/bin/bash
HOME="/home/sammy"
source "$HOME/.duplicity/.env_variables.conf"
duplicity \
--verbosity info \
--encrypt-sign-key="$GPG_KEY" \
--full-if-older-than 7D \
--log-file "$HOME/.duplicity/info.log" \
/home/sammy/sammy_backups \
s3://nyc3.digitaloceanspaces.com/sammys-bucket/
unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY
unset GPG_KEY
unset PASSPHRASE
When you are satisfied with the script, you can save and close the file. We’ll also set permissions to ensure that only our current non-sudo user will have the ability to read, write, and execute the file:
- chmod 0700 ~/.duplicity/.backup.sh
Finally, we can automate our backup schedule by editing our user’s crontab
file. Open this file for editing by typing:
- crontab -e
Because this is our first time editing this file, we will be asked to choose an editor:
no crontab for root - using an empty one
Select an editor. To change later, run 'select-editor'.
1. /bin/ed
2. /bin/nano <---- easiest
3. /usr/bin/vim.basic
4. /usr/bin/vim.tiny
Choose 1-4 [2]:
...
You can select 2
for nano, or enter the number corresponding to the editor of your choice.
At the bottom of the file, we will add a line to specify how often our script should run. To test its functionality, we can set our time interval to two minutes as follows:
...
*/2 * * * * /home/sammy/.duplicity/.backup.sh
Save and close the file. After two minutes, you can navigate to your Spaces page in the DigitalOcean control panel, where you should see incremental backup files. You can now modify the crontab
file to specify the time interval you would like to use for your incremental backups.
In this tutorial, we have covered how to back up the contents of a specific directory to a Spaces repository. Using a configuration file to store our repository information, we created a manual backup of our data, which we tested by restoring a sample file, and an automated backup schedule.
For more information on Duplicity, you can check out the project website as well as the duplicity
man page. This documentation covers Duplicity’s many features, and offers guidance on creating full system backups.
This tutorial will cover setting up a Hadoop cluster on DigitalOcean. The Hadoop software library is an Apache framework that lets you process large data sets in a distributed way across server clusters through leveraging basic programming models. The scalability provided by Hadoop allows you to scale up from single servers to thousands of machines. It also provides failure detection at the application layer, so it can detect and handle failures as a high-availability service.
There are 4 important modules that we will be working with in this tutorial:
In this tutorial, we will be setting up and running a Hadoop cluster on four DigitalOcean Droplets.
This tutorial will require the following:
Four Ubuntu 16.04 Droplets with non-root sudo users set up. If you do not have this set up, follow along with steps 1-4 of the Initial Server Setup with Ubuntu 16.04. This tutorial will assume that you are using an SSH key from a local machine. Per Hadoop’s language, we’ll refer to these Droplets by the following names:
hadoop-master
hadoop-worker-01
hadoop-worker-02
hadoop-worker-03
Additionally, you may want to use DigitalOcean Snapshots after the initial server set up and the completion of Steps 1 and 2 (below) of your first Droplet.
With these prerequisites in place, you will be ready to begin setting up a Hadoop cluster.
We’re going to be installing Java and Hadoop on each of our four Droplets. If you don’t want to repeat each step on each Droplet, you can use DigitalOcean Snapshots at the end of Step 2 in order to replicate your initial installation and configuration.
First, we’ll update Ubuntu with the latest software patches available:
- sudo apt-get update && sudo apt-get -y dist-upgrade
Next, let’s install the headless version of Java for Ubuntu on each Droplet. “Headless” refers to the software that is capable of running on a device without a graphical user interface.
- sudo apt-get -y install openjdk-8-jdk-headless
To install Hadoop on each Droplet, let’s make the directory where Hadoop will be installed. We can call it my-hadoop-install
and then move into that directory.
- mkdir my-hadoop-install && cd my-hadoop-install
Once we’ve created the directory, let’s install the most recent binary from the Hadoop releases list. At the time of this tutorial, the most recent is Hadoop 3.0.1
.
Note: Keep in mind that these downloads are distributed via mirror sites, and it is recommended that it be checked first for tampering using either GPG or SHA-256.
When you are satisfied with the download you have selected, you can use the wget
command with the binary link you have chosen, such as:
- wget http://mirror.cc.columbia.edu/pub/software/apache/hadoop/common/hadoop-3.0.1/hadoop-3.0.1.tar.gz
Once your download is complete, unzip the file’s contents using tar
, a file archiving tool for Ubuntu:
- tar xvzf hadoop-3.0.1.tar.gz
We’re now ready to start our initial configuration.
For each Droplet node, we’ll need to set up JAVA_HOME
. Open the following file with nano or another text editor of your choice so that we can update it:
- nano ~/my-hadoop-install/hadoop-3.0.1/etc/hadoop/hadoop-env.sh
Update the following section, where JAVA_HOME
is located:
...
###
# Generic settings for HADOOP
###
# Technically, the only required environment variable is JAVA_HOME.
# All others are optional. However, the defaults are probably not
# preferred. Many sites configure these options outside of Hadoop,
# such as in /etc/profile.d
# The java implementation to use. By default, this environment
# variable is REQUIRED on ALL platforms except OS X!
# export JAVA_HOME=
# Location of Hadoop. By default, Hadoop will attempt to determine
# this location based upon its execution path.
# export HADOOP_HOME=
...
To look like this:
...
###
# Generic settings for HADOOP
###
# Technically, the only required environment variable is JAVA_HOME.
# All others are optional. However, the defaults are probably not
# preferred. Many sites configure these options outside of Hadoop,
# such as in /etc/profile.d
# The java implementation to use. By default, this environment
# variable is REQUIRED on ALL platforms except OS X!
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
# Location of Hadoop. By default, Hadoop will attempt to determine
# this location based upon its execution path.
# export HADOOP_HOME=
...
We’ll also need to add some environment variables to run Hadoop and its modules. They should be added to the bottom of the file so it looks like the following, where sammy
would be your sudo non-root user’s username.
Note: If you are using a different username across your cluster Droplets, you will need to edit this file in order to reflect the correct username for each specific Droplet.
...
#
# To prevent accidents, shell commands be (superficially) locked
# to only allow certain users to execute certain subcommands.
# It uses the format of (command)_(subcommand)_USER.
#
# For example, to limit who can execute the namenode command,
export HDFS_NAMENODE_USER="sammy"
export HDFS_DATANODE_USER="sammy"
export HDFS_SECONDARYNAMENODE_USER="sammy"
export YARN_RESOURCEMANAGER_USER="sammy"
export YARN_NODEMANAGER_USER="sammy"
At this point, you can save and exit the file. Next, run the following command to apply our exports:
- source ~/my-hadoop-install/hadoop-3.0.1/etc/hadoop/hadoop-env.sh
With the hadoop-env.sh
script updated and sourced, we need to create a data directory for the Hadoop Distributed File System (HDFS) to store all relevant HDFS
files.
- sudo mkdir -p /usr/local/hadoop/hdfs/data
Set the permissions for this file with your respective user. Remember, if you have different usernames on each Droplet, be sure to allow your respective sudo user to have these permissions:
- sudo chown -R sammy:sammy /usr/local/hadoop/hdfs/data
If you would like to use a DigitalOcean Snapshot to replicate these commands across your Droplet nodes, you can create your Snapshot now and create new Droplets from this image. For guidance on this, you can read An Introduction to DigitalOcean Snapshots.
When you have completed the steps above across all four Ubuntu Droplets, you can move on to completing this configuration across nodes.
At this point, we need to update the core_site.xml
file for all 4 of your Droplet nodes. Within each individual Droplet, open the following file:
- nano ~/my-hadoop-install/hadoop-3.0.1/etc/hadoop/core-site.xml
You should see the following lines:
...
<configuration>
</configuration>
Change the file to look like the following XML so that we include each Droplet’s respective IP inside of the property value, where we have server-ip
written. If you are using a firewall, you’ll need to open port 9000.
...
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://server-ip:9000</value>
</property>
</configuration>
Repeat the above writing in the relevant Droplet IP for all four of your servers.
Now all of the general Hadoop settings should be updated for each server node, and we can continue onto connecting our nodes via SSH keys.
In order for Hadoop to work properly, we need to set up passwordless SSH between the master node and the worker nodes (the language of master
and worker
is Hadoop’s language to refer to primary
and secondary
servers).
For this tutorial, the master node will be hadoop-master
and the worker nodes will be collectively referred to as hadoop-worker
, but you’ll have three of them in total (referred to as -01
, -02
, and -03
). We first need to create a public-private key-pair on the master node, which will be the node with the IP address belonging to hadoop-master
.
While on the hadoop-master
Droplet, run the following command. You’ll press enter
to use the default for the key location, then press enter
twice to use an empty passphrase:
- ssh-keygen
For each of the worker nodes, we need to take the master node’s public key and copy it into each of the worker nodes’ authorized_keys
file.
Get the public key from the master node by running cat
on the id_rsa.pub
file located in your .ssh
folder, to print to console:
- cat ~/.ssh/id_rsa.pub
Now log into each worker node Droplet, and open the authorized_keys
file:
- nano ~/.ssh/authorized_keys
You’ll copy the master node’s public key — which is the output you generated from the cat ~/.ssh/id_rsa.pub
command on the master node — into each Droplet’s respective ~/.ssh/authorized_keys
file. Be sure to save each file before closing.
When you are finished updating the 3 worker nodes, also copy the master node’s public key into its own authorized_keys
file by issuing the same command:
- nano ~/.ssh/authorized_keys
On hadoop-master
, you should set up the ssh
configuration to include each of the hostnames of the related nodes. Open the configuration file for editing, using nano:
- nano ~/.ssh/config
You should modify the file to look like the following, with relevant IPs and usernames added.
Host hadoop-master-server-ip
HostName hadoop-example-node-server-ip
User sammy
IdentityFile ~/.ssh/id_rsa
Host hadoop-worker-01-server-ip
HostName hadoop-worker-01-server-ip
User sammy
IdentityFile ~/.ssh/id_rsa
Host hadoop-worker-02-server-ip
HostName hadoop-worker-02-server-ip
User sammy
IdentityFile ~/.ssh/id_rsa
Host hadoop-worker-03-server-ip
HostName hadoop-worker-03-server-ip
User sammy
IdentityFile ~/.ssh/id_rsa
Save and close the file.
From the hadoop-master
, SSH into each node:
- ssh sammy@hadoop-worker-01-server-ip
Since it’s your first time logging into each node with the current system set up, it will ask you the following:
Outputare you sure you want to continue connecting (yes/no)?
Reply to the prompt with yes
. This will be the only time it needs to be done, but it is required for each worker node for the initial SSH connection. Finally, log out of each worker node to return to hadoop-master
:
- logout
Be sure to repeat these steps for the remaining two worker nodes.
Now that we have successfully set up passwordless SSH for each worker node, we can now continue to configure the master node.
For our Hadoop cluster, we need to configure the HDFS properties on the master node Droplet.
While on the master node, edit the following file:
- nano ~/my-hadoop-install/hadoop-3.0.1/etc/hadoop/hdfs-site.xml
Edit the configuration
section to look like the XML below:
...
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///usr/local/hadoop/hdfs/data</value>
</property>
</configuration>
Save and close the file.
We’ll next configure the MapReduce
properties on the master node. Open mapred.site.xml
with nano or another text editor:
- nano ~/my-hadoop-install/hadoop-3.0.1/etc/hadoop/mapred-site.xml
Then update the file so that it looks like this, with your current server’s IP address reflected below:
...
<configuration>
<property>
<name>mapreduce.jobtracker.address</name>
<value>hadoop-master-server-ip:54311</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
Save and close the file. If you are using a firewall, be sure to open port 54311.
Next, set up YARN on the master node. Again, we are updating the configuration section of another XML file, so let’s open the file:
- nano ~/my-hadoop-install/hadoop-3.0.1/etc/hadoop/yarn-site.xml
Now update the file, being sure to input your current server’s IP address:
...
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop-master-server-ip</value>
</property>
</configuration>
Finally, let’s configure Hadoop’s point of reference for what the master and worker nodes should be. First, open the masters
file:
- nano ~/my-hadoop-install/hadoop-3.0.1/etc/hadoop/masters
Into this file, you’ll add your current server’s IP address:
hadoop-master-server-ip
Now, open and edit the workers
file:
- nano ~/my-hadoop-install/hadoop-3.0.1/etc/hadoop/workers
Here, you’ll add the IP addresses of each of your worker nodes, underneath where it says localhost
.
localhost
hadoop-worker-01-server-ip
hadoop-worker-02-server-ip
hadoop-worker-03-server-ip
After finishing the configuration of the MapReduce
and YARN
properties, we can now finish configuring the worker nodes.
We’ll now configure the worker nodes so that they each have the correct reference to the data directory for HDFS.
On each worker node, edit this XML file:
- nano ~/my-hadoop-install/hadoop-3.0.1/etc/hadoop/hdfs-site.xml
Replace the configuration section with the following:
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///usr/local/hadoop/hdfs/data</value>
</property>
</configuration>
Save and close the file. Be sure to replicate this step on all three of your worker nodes.
At this point, our worker node Droplets are pointing to the data directory for HDFS, which will allow us to run our Hadoop cluster.
We have reached a point where we can start our Hadoop cluster. Before we start it up, we need to format the HDFS on the master node. While on the master node Droplet, change directories to where Hadoop is installed:
- cd ~/my-hadoop-install/hadoop-3.0.1/
Then run the following command to format HDFS:
- sudo ./bin/hdfs namenode -format
A successful formatting of the namenode will result in a lot of output, consisting of mostly INFO
statements. At the bottom you will see the following, confirming that you’ve successfully formatted the storage directory.
Output...
2018-01-28 17:58:08,323 INFO common.Storage: Storage directory /usr/local/hadoop/hdfs/data has been successfully formatted.
2018-01-28 17:58:08,346 INFO namenode.FSImageFormatProtobuf: Saving image file /usr/local/hadoop/hdfs/data/current/fsimage.ckpt_0000000000000000000 using no compression
2018-01-28 17:58:08,490 INFO namenode.FSImageFormatProtobuf: Image file /usr/local/hadoop/hdfs/data/current/fsimage.ckpt_0000000000000000000 of size 389 bytes saved in 0 seconds.
2018-01-28 17:58:08,505 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2018-01-28 17:58:08,519 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop-example-node/127.0.1.1
************************************************************/
Now, start the Hadoop cluster by running the following scripts (be sure to check scripts before running by using the less
command):
- sudo ./sbin/start-dfs.sh
You’ll then see output that contains the following:
OutputStarting namenodes on [hadoop-master-server-ip]
Starting datanodes
Starting secondary namenodes [hadoop-master]
Then run YARN, using the following script:
- ./sbin/start-yarn.sh
The following output will appear:
OutputStarting resourcemanager
Starting nodemanagers
Once you run those commands, you should have daemons running on the master node and one on each of the worker nodes.
We can check the daemons by running the jps
command to check for Java processes:
- jps
After running the jps
command, you will see that the NodeManager
, SecondaryNameNode
, Jps
, NameNode
, ResourceManager
, and DataNode
are running. Something similar to the following output will appear:
Output9810 NodeManager
9252 SecondaryNameNode
10164 Jps
8920 NameNode
9674 ResourceManager
9051 DataNode
This verifies that we’ve successfully created a cluster and verifies that the Hadoop daemons are running.
In a web browser of your choice, you can get an overview of the health of your cluster by navigating to:
http://hadoop-master-server-ip:9870
If you have a firewall, be sure to open port 9870. You’ll see something that looks similar to the following:
From here, you can navigate to the Datanodes
item in the menu bar to see the node activity.
In this tutorial, we went over how to set up and configure a Hadoop multi-node cluster using DigitalOcean Ubuntu 16.04 Droplets. You can also now monitor and check the health of your cluster using Hadoop’s DFS Health web interface.
To get an idea of possible projects you can work on to utilize your newly configured cluster, check out Apache’s long list of projects powered by Hadoop.
]]>Ricotta cheese is a soft, creamy, slightly-sweet Italian cheese with a very fine curd. It is quite versatile and can be found in both savory dishes such as lasagne, and sweet desserts such as cannoli.
Though Ricotta cheese is traditionally compiled from the whey left over from traditional cheesemaking, hobbyist fromagineers can create an adequate simulation using whole milk and a variety of household acids.
In this tutorial we will coagulate milk proteins by manipulating the temperature and acidity of cow’s milk, then compile and containerize the resulting curd.
Before getting started, you will need to prepare your development environment. Gather the following prerequisite ingredients, which should be available in most well-stocked food repositories:
You will also need the following hardware:
Prepare your straining apparatus before proceeding. First, line the strainer with cheesecloth. If your cheesecloth is loosely woven (as is often the case with grocery store cheesecloth), you should use multiple layers to ensure the filter does not let any errant curds slip through.
Set the lined strainer in a bowl that’s large enough to hold all of our liquid.
Note: You may use the excess whey to make bread and soft drinks, or to feed pigs and other useful omnivores.
Next, we will combine our ingredients and apply heat.
In your pot, combine milk, cream, and salt.
Put the pot on your heat source and add energy until the mixture boils, stirring periodically to prevent scorching.
Add the vinegar or lemon juice to the boiling mixture, then lower heat to a simmer, continuing to stir occasionally. The milk and cream will start to curdle quite quickly, and in two or three minutes the process will be complete. You should now have a pot full of small white curds floating in a yellowy-green whey. You may now proceed to the next step, where we will filter the curds from the whey.
Empty the pot into the lined strainer. The white curds will be caught in our filter, while the liquid whey will end up in the bowl beneath. Let the curds drain for at least an hour.
When your Ricotta is done draining, it is ready to eat! Try it with a little olive oil, salt, and pepper, or some sweet or savory jams. You may store the cheese by transferring it to a sealed container, and refrigerating for up to two days.
In this tutorial we built and containerized Ricotta cheese. You should now be ready to deploy this versatile product into appetizer, main dish, or dessert environments.
]]>Access Blueprints Repositories
MySQL Group Replication with ProxySQL
DigitalOcean Blueprints provide you with fast and flexible infrastructure to support you as you scale. You can leverage and incorporate Blueprints as part of your development workflow to spend more time crafting code and less time setting up your infrastructure.
DigitalOcean Blueprints offer automated multi-server infrastructure setups. The goal of each Blueprint is to give developers a way to streamline the infrastructure setup process so they can spend more time bringing ideas and projects to life.
Blueprints can be the foundation of a project or a component in a multi-server environment. As a starting point for further work, Blueprints leave configuration and content creation within developers’ hands while giving them a tool for getting started quickly.
Each Blueprint uses Terraform and Ansible to create an infrastructure setup with DigitalOcean products that addresses a different use case:
Node.js Web Application: This Blueprint can be used to set up a two-node infrastructure with Nginx, Node.js, and MongoDB. The web and application layers are deployed on one server, while the database is located on the other. Data from the database is stored on a block storage device, and Cloud Firewalls are configured in front of each server to regulate traffic.
MySQL Group Replication with ProxySQL: This Blueprint provides a replicated database group setup using MySQL group replication and ProxySQL. The cloned setup creates a three-node replication database layer to handle project data. It also creates a ProxySQL server that is configured to manage queries and changes to the project’s backend.
Status Page Application: This Blueprint creates a status page using Cachet, an open-source status page application, and a two-node infrastructure. One of the two servers runs MySQL, while the other runs the Cachet application with Nginx and PHP-FRM. The two servers communicate over DigitalOcean’s private network, and customizable Cloud Firewalls are in place to further restrict access. Nginx is also configured with SSL/TLS certificates using Let’s Encrypt.
Each of these Blueprints can lay the groundwork for various use cases and provide a pattern that can be modified based on your needs.
Each Blueprint will be ready to clone and use after a few prerequisites are in place. You will need:
With these prerequisites in place, you will be able to take the following steps to get each Blueprint up and running:
Clone the repository.
Configure definitions and credentials for the Docker image and local repository.
Create your infrastructure.
From here, you will be able to customize your infrastructure and adapt it to your needs and use cases.
A good first step in putting the Blueprints to use will be to read each project’s README.md
in full. There, you will find detailed instructions for installation, as well as discussions of how to test, customize, and deprovision your infrastructure.
Access Blueprints Repositories
A high-availability web application setup offers advantages to developers who are looking to eliminate single points of failure and minimize downtime. Within this general framework, however, there are a number of possible variations. Developers will make choices based on the specific needs of their application and their performance goals.
This high-availability application setup was designed as a hypothetical solution to potentially offer:
In this article, we will go over the specific features of this setup and discuss its components at a more general level. At the end of each section, we’ll link out to additional resources on the topic to support you as you consider methodologies and best practices.
A typical multi-tier setup separates the presentation layer from our application logic. Separating application functions into layers makes the processes of troubleshooting and scaling easier in the long term.
As we select servers and resources, we can consider the following factors:
Our monitoring tools will help us scale our application and build out resources at this and other levels. An additional step we can take for cost-saving and security measures is to assign our application’s resources, including our front-end servers, to a shared private network. Data can then be transferred between servers without incurring additional bandwidth costs or leaving a single datacenter.
To ensure that our application’s resources remain highly available and performant, we can create load balancers to manage our front-end workload. These load balancers will redirect incoming traffic, using regular health checks and failover mechanisms to manage server failure or malfunction. They will also balance traffic more generally, making sure individual servers don’t become overloaded.
To optimize their configuration, we can consider the following factors:
These factors will enable us to select the optimal algorithm for our configuration. There is an additional security component to the load balancers’ work as well: we can configure them to listen on specific ports and to redirect traffic between ports. It is also possible to use them to decrypt messages for our back-end servers.
Creating our application’s backend involves another set of resource calculations. Again, the nature of our application’s work will determine the size and resources of our servers. Factors to consider include the type and volume of processing work our servers will do at this level. This is where distinctions between data types and processing tasks will come into play. If, for example, we are working with image assets and consumer data, we can consider load and latency requirements as they apply to each.
Monitoring will also be important at this level to address issues like:
We can place the resources at this level within our shared private network to account for potential bandwidth charges.
Similarly to how our load balancers handle external requests, HAProxy manages the flow of communication between our front-end and application layers. In its function as a load balancer, HAProxy can be configured to listen on and redirect traffic from particular ports. This can add another layer of security to our application’s internal operations. When we need to scale, we can configure HAProxy to add and remove nodes automatically.
For a certain segment of our application data we will use a SQL database. This is for data that needs to be current, accurate, and consistent. Things like sales transactions, login/logoff information, and password changes, which are uniform in structure and need to be secure, make a reasonable case for the use of a SQL database.
Again, we will want to consider our metrics: How many transactional or secure requests are we processing? If our load is high, we may want to consider using tools like ProxySQL to balance incoming requests. We can take an additional step to improve performance and ensure high availability if we set up replication between our SQL databases. This will also prove useful if we need to scale our data processing.
With data that is less uniform or schematic, we can use a NoSQL database. For pictures, videos, or blog posts, for example, a NoSQL database offers the ability to store item metadata in a non-schematic way. When using this type of solution, our data will be highly available, and its consistency will be eventual. As we think about performance, we want to consider the type and volume of requests we anticipate to these databases.
Factors that can optimize performance, depending on request load and type, include: using a load balancing solution to manage traffic between databases, distributing data across databases and storage solutions, and adding or destroying databases (rather than replicating them).
Our setup separates database storage functionality from our application’s other operations. The goal is to enhance the security of our data and our application’s overall performance. As another part of this isolation process, we can create a backup solution for our SQL database files. Block storage solutions such as DigitalOcean’s Block Storage volumes can do this job well, thanks to their low latency I/O, and schematic file system structure. They also offer options for scaling, since they can be easily destroyed, resized, or multiplied.
Monitoring our application’s performance will inform the decisions we make as we scale and refine our setup. To do this work, we can use a centralized logging solution such as an Elastic/ELK stack. Our stack includes components that gather and visualize logs: Logstash, which processes logs; Elasticsearch, which stores them; and Kibana, which allows them to be searched and visually organized. If we situate this stack behind a Reserved IP, we will be able to access it remotely with a static IP. Additionally, if we include our stack in our shared private network, we will have another security advantage: our reporting agents will not need to transfer information to the stack over the internet.
When storing our application’s static assets, we want to ensure their availability while maintaining a high performance. Object storage solutions like DigitalOcean Spaces can meet this need. Specifically, if we decide to store large objects in our databases, they may experience performance issues with the influx of data, making our backups very large. In this scenario, we could move our data to object storage. By storing a URL in our database, we can point to our resources from the database without impacting its storage capacity. This is an optimal solution for data that we anticipate will remain static, and offers additional options for scaling.
Once our high-availability setup is in place, we can point our application’s domain name to our load balancers using DNS. With a round robin algorithm, we can balance query responses between our application’s distributed resources. This will maximize the availability of these resources, while also distributing workloads across resource clusters. Additionally, we can use geographic routing to match requests to proximate resources.
Our recovery strategy will include tools and functions to back up and restore our data in the case of administrative or other failures. For each of our Droplets, we can leverage and automate DigitalOcean Snapshots to copy and store images of Droplets on DigitalOcean servers. Additionally, we can use dedicated tools and services such as Percona, Restic, or Bacula, along with storage devices like DigitalOcean Backups and Spaces to copy our data. As we evaluate these tools and create our strategy, we will think about the data at each layer of our application, and how often it needs to be backed up in order for us to have a reasonable point from which to restore our application’s functionality.
In this article, we have discussed a potential setup for a highly-available web application that depends on infrastructure components like Droplets, Load Balancers, Spaces, and Block Storage to deliver a high level of operational performance. This setup could support a processing solution for images and other media, with a focus on storage and retrieval, as well as purchasing, scorekeeping, or blogging capabilities that could be integrated with ecommerce solutions.
Ultimately, there are many directions developers can take to meet particular needs and use cases while maintaining high availability, and each application setup will reflect these differences in the specificity of its architecture.
]]>Regular database backups are a crucial step in guarding against unintended data loss events. In general, there are two broad categories of backups: filesystem-level (“physical”) backups and logical backups. Filesystem-level backups involve snapshotting the underlying data files at a point in time, and allowing the database to cleanly restore itself using the state captured in the snapshotted files. Logical backups involve using a tool (e.g. mongodump
or pg_dump
) to export data from the database into backup files, which are then restored using a corresponding restore tool (e.g. mongorestore
or psql <
).
In this guide, we’ll demonstrate how to perform a filesystem-level backup of a running MongoDB installation using Droplet Snapshots. In addition, we’ll cover how to perform a restore from the snapshot image.
Note: As detailed in the DigitalOcean backups guide, there is some performance impact when using Droplet snapshots, especially on highly loaded databases. You should test this procedure first using a non-production database with simulated load to verify that this method will work in your production deployment.
Before you get started with this guide, make sure you’ve completed the following prerequisite steps:
This guide will assume you have MongoDB 3.2+ installed, using the default WiredTiger storage engine with journaling enabled. In addition, to use this guide, it’s important that the dbpath
directory (the directory containing the data files, by default /var/lib/mongodb
) is mapped to a single volume. If you haven’t attached additional block storage volumes to your Droplet, you can follow this guide.
Once you’re logged in to your Droplet and have MongoDB up and running, you’re ready to get started.
We’re first going to check that journaling has been enabled.
Journaling is a MongoDB feature that provides durability in the event of a database failure by writing operations to journal files. To learn more about MongoDB journaling, consult the MongoDB Manual.
If you followed the above guide, journaling will be enabled by default. To confirm that this is the case, we can inspect the MongoDB configuration file.
Open up /etc/mongod.conf
using your favorite text editor, such as nano for instance:
- nano /etc/mongod.conf
You should see the following block:
- # Where and how to store data.
- storage:
- dbPath: /var/lib/mongodb
- journal:
- enabled: true
- # engine:
- # mmapv1:
- # wiredTiger:
This indicates that journaling has been enabled. If you’re on MongoDB 3.2+, the default storage engine is WiredTiger (MMAPv1 was MongoDB’s original storage engine).
We’ll now insert some dummy data to test the backup and restore procedure.
If you have started with a clean server and don’t have any data yet, we can insert some sample data into a dummy restaurants collection for demonstration purposes. If you already have some collections and documents stored in your database, feel free to skip this step.
First, connect to the running database using the MongoDB shell:
mongo
You should see the following Mongo shell prompt:
MongoDB shell version: 3.2.19
connecting to: test
Server has startup warnings:
2018-02-16T02:40:13.071+0000 I CONTROL [initandlisten]
2018-02-16T02:40:13.071+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2018-02-16T02:40:13.071+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2018-02-16T02:40:13.071+0000 I CONTROL [initandlisten]
2018-02-16T02:40:13.071+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2018-02-16T02:40:13.071+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2018-02-16T02:40:13.071+0000 I CONTROL [initandlisten]
>
The default database the shell uses is the test database.
Let’s list the collections present in the test database:
- show collections
Since we haven’t inserted anything into the database yet, there are no collections, and we’re brought back to the prompt with no output.
Let’s insert a document into a dummy restaurants collection, which we’ll create at the same time:
- db.restaurants.insert({'name': 'Sammy's Pizzeria'})
You should see the following output:
- WriteResult({ "nInserted" : 1 })
This indicates that the insert operation was successful. Since the restaurants collection didn’t previously exist, it was created at the same time.
Let’s list collections once again:
show collections
We now see our newly created restaurants collection:
restaurants
Now that we’ve stored some sample data in the database, we’re ready to back it up.
##Step 3 — Snapshot the MongoDB Droplet
To perform the backup, we’re going to take advantage of DigitalOcean Droplet Snapshots. Droplet Snapshots allow us to create an image of the Droplet at the point in time the snapshot was initiated. This image can then be restored to a new Droplet, where further recovery operations can take place.
Given that we’re using MongoDB 3.2+ (with WiredTiger and journaling enabled), we don’t need to suspend writes to the filesystem while the snapshot occurs. Once we restore the image and start up the database, MongoDB will restore itself from a checkpoint, and then replay operations from the journal files until it reaches the point in time that the snapshot occurred. If you’re interested in exploring journaling further, consult the MongoDB Manual),
To begin the snapshot process, log in to your DigitalOcean account, navigate to your MongoDB Droplet, and click the Snapshots link in the sidebar.
You should see the following prompt:
Note: Although it’s recommended to power down your Droplet before taking a snapshot, in production deployments this may not always be possible. MongoDB’s journaling feature enables consistent and valid snapshots, even while the database and Droplet are running.
Give your snapshot a descriptive name and click the Take Live Snapshot button to begin the snapshot process.
You should see the following snapshot progress indicator:
Once the snapshot completes, you’ll be able to create a new Droplet from the image, or restore the running Droplet to the state captured in your snapshot image.
We’re now ready to perform a restore and validation of the backup procedure.
We’ll now create a new Droplet that will be restored from the image we just created. The data available in our MongoDB database will be the same data available at the time the snapshot was taken.
Navigate back to Snapshots using the sidebar, and locate your completed Droplet snapshot.
Click into More and select Create Droplet.
You’ll be taken to the Create Droplet menu, where you’ll be able to spin up a new Droplet from your snapshot.
Choose the image corresponding to the snapshot you took earlier. In this case, we’ll use the mongo-backup-test image.
Finish configuring your restore Droplet and click Create. Once your restore Droplet is up and running, log into it.
If you configured MongoDB to start upon Droplet boot, it should now be running. You can check this using systemctl
:
- sudo systemctl status mongod
You should see the following output:
Output● mongod.service - High-performance, schema-free document-oriented database
Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2018-02-14 21:14:40 UTC; 4min 53s ago
Docs: https://docs.mongodb.org/manual
Main PID: 1302 (mongod)
Tasks: 19
Memory: 87.2M
CPU: 1.802s
CGroup: /system.slice/mongod.service
└─1302 /usr/bin/mongod --quiet --config /etc/mongod.conf
Indicating that all is well and MongoDB started up correctly.
If MongoDB is not running, we’ll first need to delete the lock file, and then start the service:
- rm /var/lib/mongodb/mongod.lock
- sudo systemctl start mongod
Verify that MongoDB started up correctly using systemctl status
.
Once MongoDB is up and running, it will begin to clean itself up and restore its state to the point in time when the snapshot occurred. This can take several minutes and the mongo
shell may not be available until this completes.
Once the server becomes available, we can log in using the mongo
command:
- mongo
You’ll now be given the mongo shell prompt:
OutputMongoDB shell version: 3.2.19
connecting to: test
Server has startup warnings:
2018-02-14T21:14:41.923+0000 I CONTROL [initandlisten]
2018-02-14T21:14:41.923+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2018-02-14T21:14:41.923+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2018-02-14T21:14:41.923+0000 I CONTROL [initandlisten]
2018-02-14T21:14:41.923+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2018-02-14T21:14:41.923+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2018-02-14T21:14:41.923+0000 I CONTROL [initandlisten]
>
If you’ve made it to this point, congratulations! You’ve successfully performed a backup and restore of your MongoDB database.
As an additional precautionary measure, we can check the integrity of our collections.
Before production use of this backup data, it’s useful to check restored collections for invalid BSON objects.
Note: The validate
command can be slow on very large collections. In addition, all reads and writes will be blocked on the collection until the validate
command returns.
In this example, we have a collection called restaurants on which we want to run the validate
command.
From the mongo shell, run the validate command:
- db.restaurants.validate({full:true})
You should see similar output to the following:
- {
- "ns" : "test.restaurants",
- "nrecords" : 1,
- "nIndexes" : 1,
- "keysPerIndex" : {
- "test.restaurants.$_id_" : 1
- },
- "indexDetails" : {
- "test.restaurants.$_id_" : {
- "valid" : true
- }
- },
- "valid" : true,
- "errors" : [ ],
- "ok" : 1
- }
If you see valid: true
, all aspects of your collection are valid and you can safely use data from this collection in production.
In this tutorial we’ve learned how to complete a physical filesystem-level backup of a running MongoDB database server.
To learn more about various methods of backing up a MongoDB database, consult the MongoDB manual.
This particular backup technique was made possible due to DigitalOcean’s convenient Droplet Snapshots feature. To learn more about Droplet Snapshots, consult the Snapshot docs.
In addition, you can schedule these snapshots to occur automatically using the Backups feature. To learn more about Droplet Backups, consult the Backups Introduction.
]]>Duplicacy is a cross-platform backup tool that offers a number of functionalities — including incremental backups, concurrent backup, and client-side encryption — which aim to streamline the process of backing up data to the cloud. The CLI (command-line interface) Linux version is free for personal use but requires a paid license for commercial users. Additionally, Duplicacy is available for MacOS and Windows with a graphical interface, and this version requires both personal and commercial users to pay for a license.
Built on the idea of lock-free deduplication, Duplicacy was designed to safely manage backups to a wide range of cloud storage services. When a Duplicacy client creates and stores a new chunk, other clients using the same storage bucket can see that the chunk already exists and therefore won’t upload it again. This allows separate clients to share and back up identical data without involving any additional effort to track backups.
This tutorial provides a high-level overview of how to install the CLI version of Duplicacy and use it to manage a typical data backup process with a DigitalOcean Space. We’ll also demonstrate how to back up a shared repository from multiple Droplets to the same Space, as well as how to back up snapshots to multiple Spaces for greater data security.
##Prerequisites
To follow along with this tutorial, you will need:
With these prerequisites in place, you are ready to install Duplicacy.
##Installing Duplicacy
The latest Duplicacy CLI version can be downloaded (with no license required for personal users) from the Duplicacy GitHub repository using wget
.
Run the following commands on both server-01 and server-02 to download Duplicacy onto each of them (substituting the download link for that of the latest release):
- sudo wget -O /opt/duplicacy https://github.com/gilbertchen/duplicacy/releases/download/v2.0.10/duplicacy_linux_x64_2.0.10
Next, create a symbolic link between the download location and a new directory within /usr/local/bin
:
- sudo ln -s /opt/duplicacy /usr/local/bin/duplicacy
Finally, make duplicacy
executable:
- sudo chmod 0755 /opt/duplicacy
Duplicacy should now be installed on each of your Droplets and you are now ready to configure it to use your Space.
##Initializing Your Repository and Configuring Duplicacy
Duplicacy backs up data from the directory level, so before you can begin uploading files to your Space it must be associated with a specific directory or repository on both of your Droplets. To do this, you will need to create a project repository and initialize it using Duplicacy’s init
command.
The init
command accepts the following syntax:
duplicacy init repository_id s3://region@endpoint/space_name
repository_id
: This is the label used by Duplicacy to distinguish between different repositories. If you plan to back up the same repository from multiple sources (as we will in the next step of this tutorial), the repository ID should be the same on both Droplets.region
: The region
is the name of the region in which your Droplet is located.endpoint
: An endpoint is a static location used by server-side web APIs to specify where certain resources are found. For DigitalOcean Spaces, the endpoint will be the region followed by .digitaloceanspaces.com
, as in nyc3.digitaloceanspaces.com
. Your Space’s endpoint name can also be copied directly from the Spaces Control Panel under the “Settings” tab.space_name
: This is the name of your Space which was specified during its creation. Be aware that this is not your Space’s URL. If your Space’s URL is https://example_space.nyc3.digitaloceanspaces.com
, its name would just be example_space
.If you’d like your backups to go to a specific folder within your Space, add the name of the folder after the name of your space when you run the init
command. When doing so, remember to follow the folder’s name with a trailing slash:
- duplicacy init repository_id s3://region@endpoint/space_name/folder_name/
Once you have those details on hand, you are ready to create a repository directory on each of your Droplets using the mkdir
command. After that, navigate into your new repositories with cd
:
- mkdir project-repository
- cd project-repository/
With project-repository/
as your working directory, run the following init
command on server-01. Be sure to replace the highlighted values with your own details:
- duplicacy init project_01 s3://nyc3@nyc3.digitaloceanspaces.com/example_space
It is worth noting that you can choose to enable encryption with Duplicacy by using the -e
option with the init
command, as follows:
- duplicacy init -e project_01 s3://nyc3@nyc3.digitaloceanspaces.com/example_space
When encryption is enabled, Duplicacy will prompt you to enter your encryption password each time you use it to interact with your Space.
Once the init
command runs, Duplicacy will prompt you for your Access and Secret keys, which can be copied over from the Control Panel’s API page.
OutputEnter S3 Access Key ID:ExampleAccessKeyGBBI
Enter S3 Secret Access Key:ExampleSecretKeyEC1wnP2YiHobVcSBaZvLoFXXlnA
And it will output the following:
Output/home/sammy/project-repository will be backed up to s3://nyc3@nyc3.digitaloceanspaces.com/example_space with id project_01
When you run the init
command, it creates a directory within your repository called .duplicacy/
which holds a file named preferences
. This file contains information about your Space as well as any encryption or storage options you’ve specified. If at a later point you decide to make changes to your Duplicacy configuration, you can either edit the preferences
file directly or delete it. The file will be recreated the next time you run the init
command in that repository.
Now repeat the init
command on your second Droplet:
- duplicacy init project_01 s3://nyc3@nyc3.digitaloceanspaces.com/example_space
After adding your Access keys again, you will see a slightly different output than on your first Droplet:
OutputThe storage 's3://nyc3@nyc3.digitaloceanspaces.com/example_space' has already been initialized
Compression level: 100
Average chunk size: 4194304
Maximum chunk size: 16777216
Minimum chunk size: 1048576
Chunk seed: 6475706c6963616379
/home/sammy/project-repository will be backed up to s3://nyc3@nyc3.digitaloceanspaces.com/example_space with id project_01
Both your servers’ repositories are now initialized, but there’s one more step you may want to take to configure Duplicacy. As it stands, Duplicacy will prompt you for your Access and Secret keys every time you back up your data, which would become tedious rather quickly. To avoid this, you can use Duplicacy’s set
command to write your Space’s credentials to Duplicacy’s preferences
file. Run the following commands on each of your servers to have Duplicacy save your Access and Secret keys, respectively:
- duplicacy set -key s3_id -value ExampleAccessKeyGBBI
- duplicacy set -key s3_secret -value ExampleSecretKeyEC1wnP2YiHobVcSBaZvLoFXXlnA
You are now ready to use Duplicacy to back up each of your Droplets’ repositories to one of your Spaces!
##Backing Up One Repository from Multiple Sources
Distributed teams can benefit from unobtrusive cloud backup solutions that prevent file conflicts and data loss. By taking a snapshot of an entire repository and uploading it to a Space with a single command, Duplicacy streamlines backups while avoiding file conflicts across multiple machines.
To test out Duplicacy’s backup functionality, use touch
to populate the project-repository
on each of your Droplets with a couple dummy files:
- touch /project-repository/file-1.txt
- touch /project-repository/file-2.txt
Next, on server-01, use Duplicacy’s backup
command to create a snapshot of your repository and upload it to your Space. Because you’ve initiated your repository with only one storage location, you won’t need to specify any other options to back up your files:
- duplicacy backup
The resulting output should look something like this:
OutputNo previous backup found
Indexing /home/mark/project-repository
Listing all chunks
Packed file-1.txt (0)
Packed file-2.txt (0)
Backup for /home/sammy/project-repository at revision 1 completed
Now try backing up your repository from server-02:
- duplicacy backup
OutputLast backup at revision 1 found
Indexing /home/sammy/project-repository
Backup for /home/sammy/project-repository at revision 2 completed
You’ll notice that because the repositories on server-01 and server-02 were identical, Duplicacy didn’t pack any files like it did when you ran the backup
command on your first Droplet. To see what will happen when you back up a slightly different snapshot, open up one of the dummy files on server-02 and add some text to it:
- nano file-1.txt
The quick brown fox jumped over the lazy dogs.
Save and close the file by entering CTRL - X
, Y
, then ENTER
, and then run the backup
command once again:
- duplicacy backup
OutputStorage set to s3://nyc3@nyc3.digitaloceanspaces.com/example_space
Last backup at revision 2 found
Indexing /home/sammy/project-repository
Packed file-1.txt (45)
Backup for /home/sammy/project-repository at revision 3 completed
Because there were new changes to one of the files in your repository, Duplicacy packed that file and uploaded it as part of revision 3.
You can use the restore
command to revert your repository back to a previous revision by using the -r
option and specifying the revision number. Note that it will not overwrite existing files unless the -overwrite
option is specified, like this:
- duplicacy restore -overwrite -r 2
After running the restore
command, you can confirm that Duplicacy did indeed rewrite file-1.txt
by checking whether it has any contents:
- cat file-1.txt
If this command doesn’t produce any output, then file-1.txt
is back to being an empty file and you have successfully rolled back your repository to the previous revision.
##Backing up to Multiple Storage Locations
Storing backups at mulitple offsite locations has been a common data security practice for many years. However, the process of backing up files to multiple destinations can prove tedious and cause a drop in productivity. There are a number of third-party backup tools, though, that can provide a quick solution to back up data to multiple locations in the cloud.
To demonstrate this functionality in Duplicacy, add your second Space to the repository on server-01. You will not be able to do this by running the init
command again because that repository has already been initiated by Duplicacy and associated with your first Space. For these scenarios, you will need to use the add
command which connects an already-initialized repository to another storage bucket.
Duplicacy’s add
command uses the following syntax:
- duplicacy add storage_id repository_id s3://region@endpoint/example_space_02
This looks mostly similar to the init
command used earlier, with the main difference being that it requires you to specify an ID for the new storage location. When you ran the init
command above, Duplicacy assigned the default
ID to your first storage bucket, since that is the default location where it will send backups. The storage name you provide for your second Space can be whatever you’d like, but it may be helpful for it to be something descriptive so you remember which Space it represents.
With that information in mind, add your second Space to the repository:
- duplicacy add space_02 project_01 s3://nyc3@nyc3.digitaloceanspaces.com/example_space_02
You are now all set to back up your repository to your second Space. It’s recommended that you do this by first backing up your repository to your default storage location, and then using Duplicacy’s copy
command to copy an identical backup over to your second storage location:
- duplicacy backup
- duplicacy copy -from default -to space_02
This will copy over each chunk and snapshot from your first Space over to your second. It’s important to note that the copy
command is non-destructive, and it will not write over any existing files.
##Conclusion
When combined with DigitalOcean Spaces, Duplicacy allows users to manage cloud backups with flexibility. If you need to back up the same repository from multiple computers or you need to back up one repository to multiple places in the cloud, Duplicacy could become an integral part of your backups solution.
If you’re interested in learning more about how to use Duplicacy, you can check out the project wiki on GitHub. Alternatively, if you’d like to learn more about backup strategies in general, see our guide on How To Choose an Effective Backup Strategy for your VPS or our comparison between Object Storage vs. Block Storage Services.
]]>Download the Complete eBook!
How To Code in Python eBook in EPUB format
How To Code in Python eBook in PDF format
DigitalOcean’s How To Code in Python 3 tutorial series is available for free as an open educational eBook in both EPUB and PDF formats. Having these tutorials together in an eBook format provides you with a resource that you can use on your favorite e-reader without maintaining a constant internet connection. This way, you can use the book to learn key concepts while offline, or as a reference guide while you are coding on your computer.
Because it is available in an eBook format, How To Code in Python can be used as an Open Educational Resource and therefore as an alternative to a textbook in the classroom. Additionally, the eBook can be made available for the wider public through libraries.
This eBook can be used in a variety of ways, so this guide will break down how the general reader may want to approach the book, how teachers and students can use the book as part of their classes, and how public and university librarians can increase availability of this eBook as an educational resource. Finally, for any reader who works through the book and wants guidance on what to do next, additional resources are discussed at the bottom of this page.
This book is designed to be used in a way that makes sense for you. While it is arranged to ramp up an emerging developer, do not be constrained by the order: feel free to move throughout the book to suit your needs. Once you are familiar with the concepts, you can continue to use the book as a source of reference.
If you use the book in the order it is laid out, you’ll begin your exploration in Python by understanding the key differences between Python 3 and the previous versions of the language. From there, you’ll set up a programming environment for your relevant local or server-based system, and begin by learning general Python code structure, syntax, and data types. Along the way, you’ll gain a solid grounding in computational logic within Python, a set of skills relevant even when moving to other programming languages in the future. While the beginning of the book focuses on scripting in Python, object-oriented coding concepts are gradually introduced to help make your code more modular, flexible, and complex without repetition. By the end of the book, you’ll learn how to debug your Python code and finally how to port Python code across versions.
If you’re a student, you can let your teacher, professor, or Computer Science department know about the availability of this free eBook on Python programming. Your school or university may have an open educational resource repository where they can make the eBook available to students or teachers. You can also share this eBook with clubs or groups you belong to that may be interested in learning more about Python programming. In addition to Computer Science clubs and programs, those involved in Data Science, Statistics, and the Digital Humanities may find this free resource useful.
If you’re a teacher offering classes or workshops on Python programming, you can use this open educational eBook for free with your students. You can follow the order of the chapters in the eBook for your own curriculum, or you can pick and choose based on what you are planning to accomplish within your class. You can also supplement the eBook with the growing number of DigitalOcean project-based tutorials that are available for free online and can support students as they put their programming knowledge into practice by working through solutions to real-world scenarios.
If you’re a librarian, you can consider adding How To Code in Python to your library’s catalogue. Having the eBook available in a greater number of libraries will increase public access to a free resource that will support people as they learn to code. While not everyone is interested in a career in computer programming, everyone who learns some coding principles will be able to contribute to a larger discourse around software development and technology.
This free eBook is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License and is available to be used as an Open Educational Resource.
The following information will be useful to librarians seeking to add the eBook to their library’s collection:
If the DigitalOcean Community team can provide additional guidance, feel free to reach out in the comments section below.
When you are done with the book, you can take a look at project-based tutorials to put your knowledge into play while creating projects that can help you solve problems. While you are working on these projects, you can continue to refer to the chapters in this book as reference material.
Anyone who learns how to code can contribute to an open-source project. Open-source software is software that is freely available to use, redistribute, and modify.
Contributing to open-source projects helps improve software by ensuring that it is representative of the broad base of technology end-users. When users contribute to open-source projects through code, technical documentation, or repository maintenance, their diverse perspectives provide added value to the project and the larger developer community.
You can download the eBook in either the EPUB or PDF format by following the links below.
Download the Complete eBook!
For additional Python resources and to participate in discussion with others, check out our growing library of tutorials, questions, and projects with the Python tag.
If you are interested in learning about software development in general or other programming languages such as Ruby and JavaScript, check out the Development tag.
You can learn more about how and why the eBook was made by reading this blog post.
]]>New Droplet plans are available immediately through the cloud control panel and API.
All Droplet resources and rates are updated our pricing page.
No. In order to decrease impact on existing applications and workflows, these plans are being introduced as completely new plans. This means there is no impact to your current Droplet and these changes will only impact newly created Droplets. To take advantage of the new plans, you must resize your Droplet.
In order to take advantage of the new resources and plans, you must resize your Droplet. This means there is no impact to your current Droplet and these changes will only impact newly created Droplets or Droplets that you resize.
You can resize from first generation Droplets to the new Standard Droplet plans assuming there are no local disk space constraints.
For more information, see: How To Resize Your Droplets on DigitalOcean
No. You can continue using your existing API slugs that map to the same Droplet plans as they always have. To get access to the new Droplet plans, you’ll need to update the API slug. We will be supporting the First Generation Droplet slugs until July 1st, 2018.
Optimized Droplets’ vCPUs provide dedicated resources from best-in-class Intel Processors. These Droplets run on entirely different, compute optimized servers meant for different types of workloads that are CPU intensive such as CI/CD, batch processing and data analysis. These Droplets are meant to give you a consistent and reliable measure of performance at all times.
Standard Droplet vCPUs are shared resources that are able to burst up as resources are available. If you are less sensitive to maximum CPU performance at all times, these plans are good to use and priced a bit cheaper relative to Optimized.
For more information, see: Choosing the Right Droplet for Your Application
If you have an active Optimized Droplet - you can run either a power-cycle or combination of power-off/power-on events either through the Droplet’s page on the cloud control panel or via the API. When your Droplet has completed either of these events, you’ll see the updated SSD values.
All newly created Optimized Droplets will come with this local SSD by default upon creation.
DigitalOcean is always looking for ways to optimize efficiency and costs and will make pricing changes more frequently into the future. Our blog will share any pricing related changes as they happen.
As a result of heavily upgrading the resources with the Standard Plans, we’ve made the decision to remove High Memory Droplets in their current form. We recommend customers using High Memory Droplets to begin taking advantage of the larger Standard Droplet plans with ample amounts of RAM and SSD.
The API will support High Memory Droplets created until July 1st, 2018, but we recommend transitioning over to the new Standard Droplet Plans before then. (If you have an active High Memory Droplet, it will simply continue to be charged at the same rate for the duration that it remains active.)
In the future, there is a possibility we may re-release the High Memory Droplets but do not have details on that at this time.
]]>Note: This is a comic adaptation of an introductory load balancing article. If you’d like to learn more, we recommend taking a look at the original piece: What is Load Balancing?
]]>Block Storage allows you to manage additional storage for your DigitalOcean Droplets in a way similar to using hard drives. Adding block storage to our Droplets can be done with a few clicks from DigitalOcean’s streamlined GUI or graphical user interface. However, this is not a practical way of operating in larger and more complex environments, so DigitalOcean offers an API to work at scale. We can interact directly with the API through doctl
, DigitalOcean’s official command-line tool.
In this tutorial, we’ll learn how to use doctl
to create, list, attach, detach, and delete block storage volumes from our Droplets.
Before starting this tutorial, you should familiarize yourself with doctl
and DigitalOcean’s block storage. The following articles will be helpful:
You should make sure you have the latest version of doctl
(at the time of this writing it’s 1.7.1) installed and authenticated before continuing. Check your doctl
version by running doctl version
. You will also need to have an SSH key added to your DigitalOcean account.
Last but not least, to follow this tutorial you will need a Droplet created on one of the regions that allow the use of block storage (at the time of this writing, the following regions offer block storage: BLR1, FRA1, LON1, NYC1, NYC3, SFO2, SGP1, and TOR1).
Note: While the regions mentioned above support block storage, this feature is not currently available for all Droplets in the noted regions. To ensure that the Droplet you are working on supports block storage, you will have to attach a volume to it at the time of creation.
In order to create a volume with doctl
, you need to provide the following parameters to the command:
You can also provide a description with the desc flag but that’s entirely optional. The full command will look like this:
- doctl compute volume create firstvolume --region nyc3 --size 100GiB
You should see an output similar to this:
OutputID Name Size Region Droplet IDs
______your_volume_ID1_______ firstvolume 100 GiB nyc3
At this point, you know the doctl
command and what information is required to create a new volume. Next, you will learn how to print a complete list of existing volumes.
doctl
provides us with the ability to show existing volumes in a formatted list. There are a few reasons why you would want to list your volumes. The two most common ones being to display the ID of each volume to be used in later commands, and to display which Droplets have volumes assigned to them.
To list all current block storage volumes, you can run the following command.
- doctl compute volume list
This is the output of the list
command run in our example.:
OutputID Name Size Region Droplet IDs
______your_volume_ID1______ firstvolume 100 GiB nyc3
______your_volume_ID1______ secondvolume 4096 GiB nyc3
______your_volume_ID1_______ thirdvolume 100 GiB nyc3 [ID]
In this section, you have learned the doctl
command to see a list of volumes you have created. In the next section, we’ll go over how to attach a volume to a Droplet.
Sometimes your Droplet may need extra space to handle assets such as application data and configuration files. Adding a volume is a great way to add this space without disrupting service.
To attach volumes you will need two pieces of information:
In the previous section, we’ve seen how to obtain the volume ID by using the doctl compute volume list
command.
We can get our Droplet ID by running the following command to display information about our account’s Droplets:
- doctl compute droplet list
Once we have both the volume and Droplet IDs, we can proceed with the following command to attach a volume to a Droplet:
- doctl compute volume-action attach your_volume_ID your_droplet_ID
This will produce an output similar to this:
OutputID Status Type Started At Completed At Resource ID Resource Type Region
346253669 in-progress attach_volume 2017-12-28 19:53:28 +0000 UTC <nil> 0 backend nyc3
Earlier in this tutorial, it was recommended that you attach a volume to the Droplet at creation time, in order to ensure that the Droplet is using infrastructure that supports block storage. If you created a Droplet without attaching a volume at that time, you might see the following error when trying to attach a volume to it:
OutputError: POST https://api.digitalocean.com/v2/volumes/your_volume_ID/actions: 422 Droplet can't attach volumes due to a region restriction
If you encounter this error, you will not be able to attach the volume to the specified Droplet and will need to try again.
Once you have a volume successfully attached to a Droplet that accepts the volume, you can go on to the next section to learn how you can detach the volume in the case that you no longer need the extra space.
There may be times in which you may need to attach a volume only temporarily to a Droplet, like when you are debugging an issue that requires a large amount of logs, or creating a backup of certain time-bound data. In these cases, we’ll need to be able to detach a volume once we are done using it.
Detaching a volume is similar to attaching a volume and uses the same pieces of information. The command and the output vary slightly.
- doctl compute volume-action detach your_volume_ID your_droplet_ID
OutputID Status Type Started At Completed At Resource ID Resource Type Region
346254931 in-progress detach_volume 2017-12-28 19:57:51 +0000 UTC <nil> 0 backend nyc3
At this point, you know how you can detach a volume with doctl
. In the next section, you will learn how to delete a volume that you no longer need.
When you no longer need a certain block storage volume, you can detach it and then remove it from your account through deletion. Once you have detached a volume, to delete it you will need its ID.
- doctl compute volume delete your_volume_id
Running this command will prompt for a confirmation:
OutputWarning: Are you sure you want to delete volume (y/N) ?
If you are satisfied that you would like to delete the volume, press y
to confirm.
Once the volume is deleted you will be returned to the command prompt. You can verify that the volume has been deleted by using the list
command.
If you need information about a specific volume, you can request it by invoking the following command
- doctl compute volume get your_volume_id
You will find the output of this command familiar because it is automatically run when creating a volume.
OutputID Name Size Region Droplet IDs
______your_volume_ID1_______ firstvolume 100 GiB nyc3
In this section you have learned how you can delete a volume that is no longer needed.
You now have all the information you need to successfully use doctl
to work with DigitalOcean block storage volumes.
In this tutorial, we’ve learned how to use doctl
to add, attach, detach, list, and delete volumes from our Droplets.
Now that you know how to do this, you may want to explore creating scripts and adding these scripts to your favorite automation tool, such as Jenkins or Drone.
]]>Relying on a source code repository for versioning is a best practice that can get us back up and running when a code change causes our application to crash or to behave erratically. However, in case of a catastrophic event like a full branch getting accidentally deleted or losing access to a repository, we should leverage additional disaster recovery strategies.
Backing up our code repository into an object storage infrastructure provides us with an off-site copy of our data that we can recover when needed. Spaces is DigitalOcean’s object storage solution that offers a destination for users to store backups of digital assets, documents, and code.
Compatible with the S3 API, Spaces allows us to use S3 tools like S3cmd to interface with it. S3cmd is a client tool that we can use for uploading, retrieving, and managing data from object storage through the command line or through scripting.
In this tutorial we will demonstrate how to back up a remote Git repository into a DigitalOcean Space using S3cmd. To achieve this goal, we will install and configure Git, install S3cmd, and create scripts to back up the Git repository into our Space.
In order to work with Spaces, you’ll need a DigitalOcean account. If you don’t already have one, you can register on the signup page.
From there, you’ll need to set up your DigitalOcean Space and create an API key, which you can achieve by following our tutorial How To Create a DigitalOcean Space and API Key.
Once created, you’ll need to keep the following details about your Space handy:
Additionally, you should have an Ubuntu 16.04 server set up with a sudo non-root user. You can get guidance for setting this up by following this Ubuntu 16.04 initial server setup tutorial.
Once you have your Spaces information and server set up, proceed to the next section to install Git.
In this tutorial, we’ll be working with a remote Git repository that we’ll clone to our server. Ubuntu has Git installed and ready to use in its default repositories, but this version may be older than the most recent available release.
We can use the apt
package management tools to update the local package index and to download and install the most recent available version of Git.
- sudo apt-get update
- sudo apt-get install git
For a more flexible way to install Git and to ensure that you have the latest release, you can consider installing Git from Source.
We’ll be backing up from a Git repository’s URL, so we will not need to configure Git in this tutorial. For guidance on configuring Git, read this section on How To Set Up Git.
Now we’ll move on to cloning our remote Git repository.
In order to clone our Git repository, we’ll create a script to perform the task. Creating a script allows us to use variables and helps ensure that we do not make errors on the command line.
To write our executable script, we’ll create a new shell script file called cloneremote.sh
with the text editor nano.
- nano cloneremote.sh
Within this blank file, let’s write the following script.
#!/bin/bash
remoterepo=your_remote_repository_url
localclonedir=repos
clonefilename=demoprojectlocal.git
git clone --mirror $remoterepo $localclonedir/$clonefilename
Let’s walk through each element of this script.
The first line — #!/bin/bash
— indicates that the script will be run by the Bash shell. From there, we define the variables that will be used in the command, which will run once we execute the script. These variables define the following pieces of configuration:
remoterepo
is being assigned the remote Git repository URL that we will be backing up fromlocalclonedir
refers to the server directory or folder that we will be cloning the remote repository into, in this case we have called it repos
clonefilename
refers to the filename we will provide to the local cloned repository, in this case we have called it demoprojectlocal.git
Each of these variables are then called directly in the command at the end of the script.
The last line of the script uses the Git command line client beginning with the git
command. From there, we are requesting to clone a repository with clone
, and executing it as a mirror version of the repository with the --mirror
tag. This means that the cloned repository will be exactly the same as the original one. The three variables that we defined above are called with $
.
When you are satisfied that the script you have written is accurate, you can exit nano by typing the CTRL
+ x
keys, and when prompted to save the file press y
.
At this point we can run the shell script with the following command.
- sh cloneremote.sh
Once you run the command, you’ll receive output similar to the following.
OutputCloning into bare repository './repos/demoprojectlocal.git'...
remote: Counting objects: 3, done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (3/3), done.
Checking connectivity... done.
At this point, if you list the items in your current directory, you should see your backup directory there, and if you move into that directory you’ll see the sub-folder with the filename that you provided in the script. That subdirectory is the clone of the Git repository.
With our remote Git repository cloned, we can now move on to installing S3cmd, which we can use to back up the repository into object storage.
The S3cmd tool allows us to connect to the Spaces environment from the command line. We’ll download the latest version of S3cmd from its public GitHub repository and follow the recommended guidelines for installing it.
Before installing S3cmd, we need to install Python’s Setuptools, as it will help with our installation (S3cmd is written in Python).
- sudo apt-get install python-setuptools
Press y
to continue.
With this downloaded, we can now download the S3cmd tar.gz
file with curl
.
- cd /tmp
- curl -LO https://github.com/s3tools/s3cmd/releases/download/v2.0.1/s3cmd-2.0.1.tar.gz
Note that we are downloading the file into our tmp
directory. This a common practice when downloading files onto our server.
You can check to see if there is a newer version of S3cmd available by visiting the Releases page of the tool’s GitHub repository. If you find a newer version, you can copy the tar.gz
URL and substitute it into the curl
command above.
When the download has completed, unzip and unpack the file using the tar utility:
- cd ~
- tar xf /tmp/s3cmd-*.tar.gz
In the commands above, we changed back to our home directory then executed the tar
command. We used two flags with the command, the x
indicates that we want to extract from a tar file and, and the f
indicates that the immediately adjacent string will be the full path name of the file that we want to expand from. In the file path of the tar file, we also indicate that it is in the tmp
directory.
Once the file is extracted, change into the resulting directory and install the software using sudo:
- cd s3cmd-*
- sudo python setup.py install
For the above command to run, we need to use sudo
. The python
command is a call to the Python interpreter to install the setup.py
Python script.
Test the install by asking S3cmd for its version information:
- s3cmd --version
Outputs3cmd version 2.0.1
If you see similar output, S3cmd has been successfully installed. Next, we’ll configure S3cmd to connect to our object storage service.
S3cmd has an interactive configuration process that can create the configuration file we need to connect to our object storage server. During the configuration process, you will be asked for your Access Key and Secret Key, so have them readily available.
Let’s start the configuration process by typing the following command:
- s3cmd --configure
We are prompted to enter our keys, so let’s paste them in and then accept US
for the Default Region. It is worth noting that being able to modify the Default Region is relevant for the AWS infrastructure that the S3cmd tool was originally created to work with. Because DigitalOcean requires fewer pieces of information for configuration, this is not relevant so we accept the default.
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key []: EXAMPLE7UQOTHDTF3GK4
Secret Key []: b8e1ec97b97bff326955375c5example
Default Region [US]:
Next, we’ll enter the DigitalOcean endpoint, nyc3.digitaloceanspaces.com
.
Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: nyc3.digitaloceanspaces.com
Because Spaces supports DNS-based buckets, at the next prompt we’ll supply the bucket value in the required format:
%(bucket)s.nyc3.digitaloceanspaces.com
Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars c
an be used if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket []: %(bucket)s.nyc3.digitaloceanspaces.com
At this point, we’re asked to supply an encryption password. We’ll enter a password so it will be available in the event we want to use encryption.
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: secure_password
Path to GPG program [/usr/bin/gpg]:
We’re next prompted to connect via HTTPS, but DigitalOcean Spaces does not support unencrypted transfer, so we’ll press ENTER
to accept the default, Yes
.
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]:
Since we aren’t using an HTTP Proxy server, we’ll leave the next prompt blank and press ENTER
.
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:
After the prompt for the HTTP Proxy server name, the configuration script presents a summary of the values it will use, followed by the opportunity to test them. When the test completes successfully, enter Y
to save the settings.
Once you save the configuration, you’ll receive confirmation of its location.
When you have completed all the installation steps, you can double-check that your setup is correct by running the following command.
- s3cmd ls
This command should output a list of Spaces that you have available under the credentials you provided.
Output2017-12-15 02:52 s3://demospace
This confirms that we have successfully connected to our DigitalOcean Spaces. We can now move on to backing up our Git repository into object storage.
With all of our tools installed and configured, we are now going to create a script that will zip the local repository and push it into our DigitalOcean Space.
From our home directory, let’s call our script movetospaces.sh
and open it in nano.
- cd ~
- nano movetospaces.sh
We’ll write our script as follows.
#!/bin/sh
tar -zcvf archivedemoproject.tar.gz /repos/demoprojectlocal.git
./s3cmd-2.0.1/s3cmd put archivedemoproject.tar.gz s3://demospace
Earlier in this tutorial, we’ve used tar
to unzip s3cmd
, we are now using tar
to zip the Git repository before sending it to Spaces. In the tar
command, we specify four flags:
z
compresses using the gzip methodc
creates a new file instead of using an existing onev
indicates that we are being verbose about the files being included in the compressed filef
names the resulting file with the name defined in the next stringAfter the flags, we are providing a file name for the compressed file, in this case archivedemoproject.tar.gz
. We are also providing the name of the directory that we want to zip /repos/demoprojectlocal.git
.
The script then executes s3cmd put
to send archivedemoproject.tar.gz
to our destination Space s3://demospace
.
Among the commands you may commonly use with S3cmd, the put
command sends files to Spaces. Other commands that may be useful include the get
command to download files from the Space, and the delete
command to delete files. You can obtain a list of all commands accepted by S3cmd by executing s3cmd
with no options.
To copy your backup into your Space, we’ll execute the script.
- sh movetospaces.sh
You will see the following output:
Outputdemoprojectlocal.git/
...
demoprojectlocal.git/packed-refs
upload: 'archivedemoproject.tar.gz' -> 's3://demobucket/archivedemoproject.tar.gz' [1 of 1]
6866 of 6866 100% in 0s 89.77 kB/s done
You can check that the process worked correctly by running the following command:
- s3cmd ls s3://demospace
You’ll see the following output, indicating that the file is in your Space.
Output2017-12-18 20:31 6866 s3://demospace/archivedemoproject.tar.gz
We now have successfully backed up our Git repository into our DigitalOcean Space.
To ensure that code can be quickly recovered if needed, it is important to maintain backups. In this tutorial, we covered how to back up a remote Git repository into a DigitalOcean Space through using Git, the S3cmd client, and shell scripts. This is just one method of dozens of possible scenarios in which you can use Spaces to help with your disaster recovery and data consistency strategies.
You can learn more about what we can store in object storage by reading the following tutorials:
]]>Python Decouple is a Python library aimed at making it easier for developers to separate their configuration settings from code. Originally designed for Django, it is now a generic Python tool for storing parameters and defining constant values separate from your code.
In this tutorial we will go over how to install Python Decouple and how to use it in a basic Django application utilizing DigitalOcean’s object storage solution, Spaces.
In order to be adequately prepared for this tutorial, you will need the following:
sudo
privileges set up on an Ubuntu or Debian Linux server. If you haven’t set this up already, follow the initial server setup for Ubuntu 16.04 or Debian tutorial.With an initial server setup and a DigitalOcean Space and API key, you’re ready to get started.
Before we begin, if you haven’t done so already, now is a good time to update and upgrade your server.
- sudo apt-get update && sudo apt-get -y upgrade
Your server should have shipped with Python 3. Run the following command to verify that it is installed:
- sudo apt-get install python3
Next, let’s install pip the package manager for Python.
- sudo apt-get install -y python3-pip
Finally, we will need to install the virtualenv module so that we can set up our programming environment:
- sudo pip3 install virtualenv
For further guidance and information on the setup and utilization of programming environments, check out this tutorial on setting up a virtual environment.
We are now ready to move into our Python programming environment.
We must next create the Django app and install the required dependencies in order to utilize DigitalOcean Spaces and Python Decouple.
While in the server’s home directory, we have to create the directory that will contain our Django application. Run the following command to create a directory called django-apps
, or another name of your choice. Then navigate to the directory.
- mkdir django-apps
- cd django-apps
While inside the django-apps
directory, create your virtual environment. Let’s call it env
.
- virtualenv env
Now, activate the virtual environment with the following command:
- . env/bin/activate
You’ll know it’s activated once the prefix is changed to (env)
, which will look similar to the following depending on what directory you are in:
-
Within the environment, install the Django package using pip. Installing Django allows us to create and run Django applications. To learn more about Django, read our tutorial series on Django Development.
- pip install django
Now let’s create a Django project called mysite
using the following command:
- django-admin startproject mysite
Next, we need to install Boto 3, an AWS SDK for Python, which will allow us to integrate object storage solutions, like DigitalOcean Spaces, with our Django application.
At the time of writing, Boto 3 has explicit compatibility with S3. Because of Space’s interoperability with S3, Spaces is also compatible with Boto 3. For more details on the comparisons between Amazon S3 and DigitalOcean Spaces object storage read the Spaces docs.
Run the following command to install Boto 3:
- pip install boto3
We also need to install django-storages, a collection of custom storage backends for Django and boto3.
- pip install django-storages
Finally, let’s install Python Decouple**.
- pip install python-decouple
You have setup your dependencies within the environment of your Django app and are now ready to set up static and template directories.
With our environment set up with all dependencies, you can now switch to the mysite/mysite
directory,
- cd ~/django-apps/mysite/mysite
Within the mysite/mysite
directory, run the following commands to create the static and template directories.
- mkdir static && mkdir templates
We’ll next create the subdirectories for images and CSS to live within the static
directory.
- mkdir static/img && mkdir static/css
Once you’ve made the directories, we’ll download a test file that we’ll eventually add to our object storage. Switch to the img
directory since we’ll be downloading an image.
- cd ~/django-apps/mysite/mysite/static/img
Within this directory, we’ll download the DigitalOcean logo image using Wget’s wget
command. This is a commonly used GNU program, preinstalled on Ubuntu distros, to retrieve content from web servers.
- wget https://assets.digitalocean.com/logos/DO_Logo_icon_blue.png
Once you hit ENTER
, you’ll see output similar to the following:
OutputResolving www.digitalocean.com (www.digitalocean.com)... 104.16.24.4, 104.16.25.4
Connecting to www.digitalocean.com (www.digitalocean.com)|104.16.24.4|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1283 (1.3K) [image/png]
Saving to: ‘DO_Logo_icon_blue.png’
DO_Logo_icon_blue-6edd7377 100%[=====================================>] 1.25K --.-KB/s in 0s
2017-11-05 12:26:24 (9.60 MB/s) - ‘DO_Logo_icon_blue.png’ saved [1283/1283]
At this point, if you run the command ls
, you’ll notice that an image named DO_Logo_icon_blue.png
now exists in the static/img/
directory.
With these directories set up and the image we’ll be storing downloaded to the server, we can move on to editing the files associated with our Django app.
We’ll start by editing the style sheet. You should move into the css
directory so that we can add a basic style sheet for our web app.
- cd ~/django-apps/mysite/mysite/static/css
Use nano, or another text editor of your choice, to edit the document.
- nano app.css
Once the file opens, add the following CSS:
body {
margin: 0;
background-color: #f1f1f1;
font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
}
.container {
width: 80%;
border: 1px solid #ddd;
background-color: #fff;
padding: 20px;
margin: 40px auto;
}
form {
margin-bottom: 20px;
padding: 10px;
border: 1px solid #ff9900;
width: 350px;
}
table {
border-collapse: collapse;
width: 100%;
}
table td,
table th {
border: 1px solid #eceeef;
padding: 5px 8px;
text-align: left;
}
table thead {
border-bottom: 2px solid #eceeef;
}
Once you are finished, you can save and close the file.
From here, navigate to the templates
directory.
- cd ~/django-apps/mysite/mysite/templates
We need to open a file called home.html
and add HTML into it for how our basic web app will be displayed. Using nano, open the file so it’s ready for editing:
- nano home.html
Within the document, add the following:
{% load static %}
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Spaces + Django Tutorial</title>
<link rel="stylesheet" type="text/css" href="{% static 'css/app.css' %}">
</head>
<body>
<center>
<header>
<h1>Spaces + Django Tutorial</h1>
</header>
<main>
<img src="{% static 'img/DO_Logo_icon_blue.png' %}">
<h2>Congratulations, you’re using Spaces!</h2>
</main>
</center>
</body>
</html>
Save and close the file. The last file we will update is the urls.py
file so that it points to your newly created home.html
file. We need to move into the following directory:
- cd ~/django-apps/mysite/mysite
Use nano to edit the urls.py file.
- nano urls.py
You can delete everything in the file and then add the following:
from django.conf.urls import url
from django.views.generic import TemplateView
urlpatterns = [
url(r'^$', TemplateView.as_view(template_name='home.html'), name='home'),
]
With these files set up, we can next work on configuring our settings in order to integrate with object storage.
Within your virtual environment, navigate to the location of the settings.py
file. This is where we will create the settings.ini
file to store your credentials separately.
- cd ~/django-apps/mysite/mysite
Create the settings.ini
file using touch
, a Linux command that creates new, empty files in the directory from which it is called.
- touch settings.ini
The settings file being created will have an .ini
file extension. This file will be looked at by Python Decouple for settings data, and it is also where your settings file will refer to for the API key. You can also use .env
as an alternative extension to .ini
.
Now, open the settings.ini
file using your favorite text editor, such as nano.
- nano settings.ini
In this document, we’ll have a section header, [settings]
, required by Python Decouple, and will add our Spaces credentials by assigning them to variables. Your full file should look like the following:
[settings]
SPACES_ACCESS_KEY=your-spaces-access-key
SPACES_SECRET_ACCESS_KEY=your-spaces-secret-access-key
In order to access these credentials, we’ll need to refer to the settings.ini
file from the settings.py
file.
In the next step, we’ll go through configuring the settings.py
file completely.
Now it’s time to update your settings file with your Spaces credentials so that we can take advantage of the page we’ve setup to display the image.
Ensure that you’re in the correct location to access your settings file.
- cd ~/django-apps/mysite/mysite
Open the file for editing with nano or another text editor:
- nano settings.py
At the top of the file, we’ll need to add an import
statement in order to use the config module from Decouple.
...
import os
from decouple import config
...
Move down in the file to the allowed hosts and add your server IP.
...
ALLOWED_HOSTS = ['your-server-ip']
...
Then add storages
to the installed apps section of the settings file and remove django.contrib.admin
since we won’t be using that in this tutorial. It should look like the following.
...
# Application definition
INSTALLED_APPS = [
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'storages'
]
...
Replace and add the highlighted text to the TEMPLATES
section of the settings file, so that the project knows where to locate your home.html
file.
...
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [os.path.join(BASE_DIR, 'mysite/templates')],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
],
},
},
]
...
Finally, let’s update your settings at the bottom of the file. We’ll be adding the following below the # Static files
section. The first two lines add the reference to the settings.ini
file, so that it can retrieve the configuration parameters.
Below that, be sure to add your own bucket name. At the time of writing, NYC3 is the only region where Spaces currently are, so that is being passed as the endpoint URL.
For a terminal location, add the directory into which you would like to import your files. You can add a directory through your Spaces interface in-browser.
...
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.11/howto/static-files/
AWS_ACCESS_KEY_ID = config('SPACES_ACCESS_KEY')
AWS_SECRET_ACCESS_KEY = config('SPACES_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = 'your-storage-bucket-name'
AWS_S3_ENDPOINT_URL = 'https://nyc3.digitaloceanspaces.com'
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
AWS_LOCATION = 'your-spaces-files-folder'
STATICFILES_DIRS = [
os.path.join(BASE_DIR, 'mysite/static'),
]
STATIC_URL = 'https://%s/%s/' % (AWS_S3_ENDPOINT_URL, AWS_LOCATION)
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
Now we’ve abstracted our Spaces credentials away from the Python code and our settings file is ready to integrate our Django app with object storage.
Let’s run the Django web application to verify that everything has been setup correctly.
Now we’ll run collectstatic
and you’ll notice files being transferred, including the image that we’ve saved in our static directory. It will get transferred to the Spaces location that we’ve identified in the settings file.
To accomplish this, let’s navigate to ~/django-apps/mysite/
:
- cd ~/django-apps/mysite
Within the directory, run the following command:
- python manage.py collectstatic
You’ll see the following output and should respond yes
when prompted.
OutputYou have requested to collect static files at the destination
location as specified in your settings.
This will overwrite existing files!
Are you sure you want to do this?
Type 'yes' to continue, or 'no' to cancel:
Then you’ll see some more output telling you the file has been copied to Spaces.
OutputCopying '/root/django-apps/mysite/mysite/static/css/app.css'
1 static file copied, 1 unmodified.
At this point, if you return to your bucket from your DigitalOcean Cloud account, you’ll see the css
and img
directories added to the folder you pointed them to, with app.css
in the css
directory, and the DO-Logo_icon_blue-.png
image in the img
directory.
If you have a UFW firewall set up, let’s first allow inbound traffic to pass through port 8000 by issuing the following command:
- sudo ufw allow 8000
With your virtual environment still activated, let’s navigate to the location of the manage.py
file and run the application, using the following commands:
- cd ~/django-apps/mysite
- python manage.py runserver <your-server-ip>:8000
In a web browser, navigate to the http://your-server-ip
:8000 to see the result of the Django application you’ve created. You will see the following output in your browser:
When you are done with testing your app, you can press CTRL
+ C
to stop the runserver
command. This will return you to the your programming environment.
When you are ready to leave your Python environment, you can run the deactivate
command:
- deactivate
Deactivating your programming environment will put you back to the terminal command prompt.
In this tutorial you have successfully created a Django application that serves files from DigitalOcean Spaces while abstracting away the Spaces credentials from the Python code. In the process you’ve learned about static files, how to manage static files, how to serve them from a cloud service and how to decouple your configuration parameters from your python settings file.
You can continue learning about web development with Python and Django by reading our tutorial series on Django Development.
]]>DigitalOcean Spaces is an object storage solution, ideal for unstructured data such as audio, video, images or large amounts of text. To learn more about Spaces and object storage, you can read through An Introduction to DigitalOcean Spaces.
In this tutorial, we will be covering how to setup your Django application to work with Spaces.
In order to begin this tutorial, you should have a few things set up:
sudo
privileges set up on a Debian or Ubuntu Linux server. If you haven’t set this up already, follow the initial server setup for Ubuntu 16.04 or Debian tutorial.With an initial server set up and a DigitalOcean Space and API key, you’re ready to begin.
If you haven’t already, first update and upgrade your server.
- sudo apt-get update && sudo apt-get -y upgrade
Your server should ship with Python 3, but you can run the following command to ensure that it is installed:
- sudo apt-get install python3
Next, install pip to manage software packages for Python:
- sudo apt-get install -y python3-pip
Finally, we can install the virtualenv module so we can use it to set up a programming environment:
sudo pip3 install virtualenv
For additional guidance and information about programmig environments, you can read about setting up a virtual environment.
We’ll now move on to creating the Django app that will be utilizing our DigitalOcean Space.
While in the server’s home directory, run the following command to create a directory (in this case, we’ll name it django-apps
) to hold the project and navigate to the directory:
- mkdir django-apps
- cd django-apps
Within this directory, create a virtual environment with the following command. We’ll call it env
, but you can call it whatever you would like.
- virtualenv env
You can now activate the environment and will receive feedback that you’re in the environment by the change in your command line’s prefix.
. env/bin/activate
You will receive feedback that you’re in the environment by the change in your command line’s prefix. It will look something like this, but will change depending on what directory you are in:
-
Within the environment, install the Django package with pip so that we can create and run a Django app. To learn more about Django, read our tutorial series on Django Development.
- pip install django
Then create the project with the following command, in this case we’ll call it mysite
.
- django-admin startproject mysite
Next we’ll install Boto 3, which is an AWS SDK for Python that will allow our application to interact with things like S3, EC2 and DigitalOcean Spaces. Because DigitalOcean Spaces is interoperable with Amazon S3, Spaces can interact with tools such as Boto 3 with ease. For more details on the comparison between S3 and Spaces please review the Spaces docs.
- sudo pip install boto3
Another library that is crucial for our project is django-storages, which is a collection of custom storage backends for Django. We’ll also install this with pip.
- sudo pip install django-storages
You have setup your dependencies within the environment of your Django app and are now ready to set up static and template directories.
With our environment set up with all dependencies, you can now switch to the mysite/mysite
directory,
- cd ~/django-apps/mysite/mysite
Within the mysite/mysite
directory, run the following commands to create the static and template directories.
- mkdir static && mkdir templates
We’ll next create the subdirectories for images and CSS to live within the static
directory.
- mkdir static/img && mkdir static/css
Once you’ve made the directories, we’ll download a test file that we’ll eventually add to our object storage. Switch to the img
directory since we’ll be downloading an image.
- cd ~/django-apps/mysite/mysite/static/img
Within this directory, we’ll download the DigitalOcean logo image using Wget’s wget
command. This is a commonly used GNU program, preinstalled on Ubuntu distros, to retrieve content from web servers.
- wget https://assets.digitalocean.com/logos/DO_Logo_icon_blue.png
You’ll see the output similar to the following:
OutputResolving www.digitalocean.com (www.digitalocean.com)... 104.16.24.4, 104.16.25.4
Connecting to www.digitalocean.com (www.digitalocean.com)|104.16.24.4|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1283 (1.3K) [image/png]
Saving to: ‘DO_Logo_icon_blue.png’
DO_Logo_icon_blue-6edd7377 100%[=====================================>] 1.25K --.-KB/s in 0s
2017-11-05 12:26:24 (9.60 MB/s) - ‘DO_Logo_icon_blue.png’ saved [1283/1283]
At this point, if you run the command ls
, you’ll notice that an image named DO_Logo_icon_blue.png
now exists in the static/img/
directory.
With these directories set up and the image will be storing downloaded to the server, we can move on to editing the files associated with our Django app.
We’ll start by editing the style sheet. You should move into the css
directory so that we can add a basic style sheet for our web app.
- cd ~/django-apps/mysite/mysite/static/css
Use nano, or another text editor of your choice, to edit the document.
- nano app.css
Once the file opens, add the following CSS:
body {
margin: 0;
background-color: #f1f1f1;
font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
}
.container {
width: 80%;
border: 1px solid #ddd;
background-color: #fff;
padding: 20px;
margin: 40px auto;
}
form {
margin-bottom: 20px;
padding: 10px;
border: 1px solid #ff9900;
width: 350px;
}
table {
border-collapse: collapse;
width: 100%;
}
table td,
table th {
border: 1px solid #eceeef;
padding: 5px 8px;
text-align: left;
}
table thead {
border-bottom: 2px solid #eceeef;
}
Once you are finished, you can save and close the file. From here, navigate to the templates
directory.
- cd ~/django-apps/mysite/mysite/templates
We need to open a file called home.html
and add HTML into it for how our basic web app will be displayed. Using nano, open the file so it’s ready for editing:
- nano home.html
Within the document, add the following:
{% load static %}
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Spaces + Django Tutorial</title>
<link rel="stylesheet" type="text/css" href="{% static 'css/app.css' %}">
</head>
<body>
<center>
<header>
<h1>Spaces + Django Tutorial</h1>
</header>
<main>
<img src="{% static 'img/DO_Logo_icon_blue.png' %}">
<h2>Congratulations, you’re using Spaces!</h2>
</main>
</center>
</body>
</html>
Save and close the file. The last file we will update is the urls.py
file so that it points to your newly created home.html
file. We need to move into the following directory:
- cd ~/django-apps/mysite/mysite
Use nano to edit the urls.py file.
- nano urls.py
You can delete everything in the file and then add the following:
from django.conf.urls import url
from django.views.generic import TemplateView
urlpatterns = [
url(r'^$', TemplateView.as_view(template_name='home.html'), name='home'),
]
With these files set up, we can move on to editing our settings.py
file in order to integrate it with object storage.
Now it’s time to update your settings file with your Spaces credentials, so that we can take advantage of the page we’ve setup to display the image.
Keep in mind that in this example we will be hardcoding our credentials for brevity, but this is not secure enough for a production setup. It is recommended that you use a package like Python Decouple something like to mask your Spaces credentials. This package will separate the settings parameters from your source code, which is necessary for a production-grade Django application.
We’ll start by navigating to the location of your settings file.
- cd ~/django-apps/mysite/mysite
Open the file for editing, using nano:
- nano settings.py
Add your server ip as an allowed host.
...
ALLOWED_HOSTS = ['your-server-ip']
...
Then add storages
to the installed apps section of the settings file and remove django.contrib.admin
since we won’t be using that in this tutorial. It should look like the following.
...
# Application definition
INSTALLED_APPS = [
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'storages'
]
...
Replace and add the highlighted text to the TEMPLATES
section of the settings file, so that the project knows where to locate your home.html file.
...
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [os.path.join(BASE_DIR, 'mysite/templates')],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
],
},
},
]
...
Finally, let’s update your settings at the bottom of the file. We’ll be adding the following below the # Static files
section. Be sure to add your own access keys, bucket name, and the directory you would like your files to live. You can add a directory through your Spaces interface in-browser. At the time of writing, NYC3 is the only region where Spaces currently are, so that is being passed as the endpoint URL.
...
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.11/howto/static-files/
AWS_ACCESS_KEY_ID = 'your-spaces-access-key'
AWS_SECRET_ACCESS_KEY = 'your-spaces-secret-access-key'
AWS_STORAGE_BUCKET_NAME = 'your-storage-bucket-name'
AWS_S3_ENDPOINT_URL = 'https://nyc3.digitaloceanspaces.com'
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
AWS_LOCATION = 'your-spaces-files-folder'
STATICFILES_DIRS = [
os.path.join(BASE_DIR, 'mysite/static'),
]
STATIC_URL = 'https://%s/%s/' % (AWS_S3_ENDPOINT_URL, AWS_LOCATION)
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
Now our settings file is ready to integrate our Django app with object storage.
Now we’ll run collectstatic
and you’ll notice files being transferred, including the image that we’ve saved in our static directory. It will get transferred to the Spaces location that we’ve identified in the settings file.
To accomplish this, let’s navigate to ~/django-apps/mysite/
:
- cd ~/django-apps/mysite
Within the directory, run the following command:
- python manage.py collectstatic
You’ll see the following output and should respond yes when prompted.
OutputYou have requested to collect static files at the destination
location as specified in your settings.
This will overwrite existing files!
Are you sure you want to do this?
Type 'yes' to continue, or 'no' to cancel:
Then you’ll see some more output telling you the file has been copied to Spaces.
OutputCopying '/root/django-apps/mysite/mysite/static/css/app.css'
1 static file copied, 1 unmodified.
At this point, if you return to your bucket from your DigitalOcean Cloud account, you’ll see the css
and img
directories added to the folder you pointed them to, with app.css
in the css
directory, and the DO-Logo_icon_blue-.png
image in the img
directory.
With everything set up and our files in our object storage, we can now test our application by navigating to the page in which our static file is being served.
First, let’s ensure that our firewall will allow traffic to pass through port 8000 by issuing the following command:
- sudo ufw allow 8000
Now, we can run our server by referring to our server’s IP address and using port 8000.
- python manage.py runserver your-server-ip:8000
In a web browser, navigate to the http://your-server-ip:8000
to see the result of the Django application you’ve created. You will see the following output in your browser:
When you are done with testing your app, you can press CTRL
+ C
to stop the runserver
command. This will return you to the your programming environment.
When you are ready to leave your Python environment, you can run the deactivate
command:
- deactivate
Deactivating your programming environment will put you back to the terminal command prompt.
In this tutorial you have successfully created a Django application that serves files from DigitalOcean Spaces. In the process you’ve learned about static files, how to manage static files and how to serve them from a cloud service.
You can continue learning about web development with Python and Django by reading our tutorial series on Django Development.
]]>In this tutorial we will demonstrate how to use the DigitalOcean API to horizontally scale your server setup using DOProxy, a Ruby script that, once configured, provides a command line interface to scale your HTTP application server tier up or down.
DOProxy was written specifically for this tutorial to provide a simple way for creating and deleting application server Droplets by using the DigitalOcean API as well as managing their membership to an HAProxy load balancer. This basic scaling model allows users to access your application through the HAProxy server which will, in turn, forward them to the back-end application servers in a load balanced fashion.
DOProxy performs three primary functions:
Note: The primary purpose of this tutorial is to teach the minimum required concepts that are necessary to programmatically scale your DigitalOcean server architecture through the API. You should not run DOProxy in a production environment since it was not designed with resiliency in mind and it performs only very basic error checking. With that being said, getting familiar with this script is a great way to get you started with learning about horizontal scaling through the DigitalOcean API.
This tutorial uses the following technologies that you may want to read about before proceeding:
Because DOProxy is written in Ruby, knowledge of Ruby can be beneficial. To gain more of a familiarity with Ruby, you can read our series on How To Code in Ruby. If you’re less familiar with Ruby, we provide some pseudocode to explain the gist of the DOProxy code. In order to simplify our calls to the API, we are using DropletKit which is the official DigitalOcean Ruby wrapper.
Before we get into the details of how DOProxy works we will install and use it on a server.
Let’s install DOProxy on an Ubuntu 16.04 Droplet now.
First, create an Ubuntu 16.04 Droplet in the NYC3 region, the region DOProxy uses by default. If you wish to use another region, you will need to configure the region
variable in the doproxy.yml
file after installing DOProxy. This Droplet will run the HAProxy load balancer and the DOProxy scaling script, so choose a size that you think will be adequate for your desired scale potential. Because this tutorial is a basic demonstration of scaling with no real traffic expected, the 512MB size is probably adequate.
For the length of this document, we will refer to this Droplet as the DOProxy server.
Next, log into the server and follow the Installation and Configuration (including doproxy config and Userdata) sections in the DOProxy GitHub repository README to install DOProxy on this server. Be sure to replace the YOUR_DO_API_TOKEN
and YOUR_SSH_KEY_FINGERPRINT
values in the DOproxy configuration file or the script will not work.
Now that you have DOProxy and HAProxy installed on your server, let’s try and scale the environment.
Log into your DOProxy server as root and get to the directory where you cloned DOProxy.
Run DOProxy without any arguments:
- ruby doproxy.rb
This should print out the available commands:
OutputCommands:
doproxy.rb print # Print backend Droplets in inventory file
doproxy.rb create # Create a new backend Droplet and reload
doproxy.rb delete <LINE_NUMBER> # Delete a Droplet and reload
doproxy.rb reload # Generate HAProxy config and reload HAProxy
doproxy.rb generate # Generate HAProxy config based on inventory
At this point, DOProxy hasn’t yet created any Droplets. Let’s create some to get our HTTP service online, and scale up.
Run the create
command to create the first Droplet that is managed by DOProxy:
- ruby doproxy.rb create
This will take some time before returning to the prompt (because the script creates a new Droplet via the API and waits for it to boot up). We’ll talk about how the API call is made when we go through the pseudocode.
Once the script is done, you should see a success message that contains the Droplet ID:
OutputSuccess: 4202645 created and added to backend.
It is suggested that you wait a couple minutes after the prompt has returned before proceeding with the next steps since it is possible that the userdata script may not have yet run and hence, the HAProxy may not have started passing traffic.
Once you are ready to continue, visit your DOProxy server’s public IP address in a web browser. You should see a page that lists your new Droplet’s hostname, id, and public IP address.
We’ll use DOProxy to create two more Droplets, for a total of three. Feel free to create more if you want:
- ruby doproxy.rb create
- ruby doproxy.rb create
Now visit your DOProxy server’s public IP address in a web browser again. If you refresh the page, you will notice that the information on the page will change while cycling through the Droplets that you’ve created. This is because they are all being load balanced by HAProxy which added each Droplet to its configuration when created with DOProxy.
If you happen to look in the DigitalOcean Control Panel, you will notice that these new Droplets will be listed there (along with the rest of your Droplets):
Let’s take a closer look at the Droplets that were created by looking at DOProxy’s inventory.
DOProxy provides a print
command that will print out all of the Droplets that are part of its inventory:
- ruby doproxy.rb print
You should see output that looks something like this:
Output0) auto-nginx-0 (pvt ip: 192.0.2.175, status: active, id: 4202645)
1) auto-nginx-1 (pvt ip: 192.0.2.176, status: active, id: 4205587)
2) auto-nginx-2 (pvt ip: 192.0.2.172, status: active, id: 4205675)
In the example output, we see information about the three Droplets that we created, including their hostnames, status, and Droplet IDs. The hostnames and IDs should match what you saw in your web browser when you accessed the HAProxy load balancer (via DOProxy’s public IP address).
As you may have noticed, DOProxy only printed information about Droplets that it created. This is because it maintains an inventory of the Droplets it creates.
Check out the contents of the inventory
file now:
- cat inventory
You should see the ID of each Droplet, one per line. Each time a Droplet is created, its ID is stored in this inventory file.
As you may have guessed, DOProxy’s print
command iterates through the Droplet IDs in the inventory file and performs an API call to retrieve information about each of them.
It should be noted that storing your server inventory in a single file is not the best solution — it can easily be corrupted or deleted — but it demonstrates a simple implementation that works. A distributed key value store, such as etcd, would be a better solution. You would also want to save more than just the Droplet ID in the inventory (so you don’t have to make API calls every time you want to look at certain Droplet information).
DOProxy also has a delete
command that lets you delete Droplets in your inventory. The delete
command requires that you provide the line number of the Droplet to delete (as displayed by the print
command).
Before running this command you will probably want to print your inventory:
- ruby doproxy.rb print
So, for example, if you want to delete the third Droplet, you would supply 2
as the line number:
- ruby doprorxy.rb delete 2
After a moment, you’ll see the confirmation message:
OutputSuccess: 4205675 deleted and removed from backend.
The delete
command deletes the Droplet via the API, removes it from the HAProxy configuration and deletes it from the inventory. Feel free to verify that the Droplet was deleted by using the DOProxy print command or by checking the DigitalOcean control panel. You will also notice that it is no longer part of the load balancer.
The last piece of DOProxy that we haven’t discussed yet is how HAProxy is configured.
When you run the create
or delete
DOProxy command, the information for each Droplet in the inventory is retrieved and some of the information is used to modify an HAProxy configuration file. In particular, the Droplet ID and private IP address are used to add each Droplet as a backend server.
Look at the last few lines of the generated haproxy.cfg
file like this:
- tail haproxy.cfg
You should see something like this:
frontend www-http
bind 203.0.113.43:80
reqadd X-Forwarded-Proto:\ http
default_backend www-backend
backend www-backend
server www-4202645 192.0.2.175:80 check # id:4202645, hostname:auto-nginx-0
server www-4205587 192.0.2.176:80 check # id:4205587, hostname:auto-nginx-1
The frontend
section should contain the public IP address of your DOProxy server, and the backend
section should contain lines that refer to each of the Droplets that were created.
Note: At this point, you may want to delete the rest of the Droplets that were created with DOProxy (ruby doproxy.rb delete 0
until all of the servers are gone).
Now that you’ve seen DOProxy’s scaling in action, let’s take a closer look at the code.
In this section, we will look at the pertinent files and lines of code that make DOProxy work. Seeing how DOProxy was implemented should give you some ideas of how you can use the API to manage and automate your own server infrastructure.
Since you cloned the repository to your server, you can look at the files there or you can look at the files at the DOProxy repository (https://github.com/scanevari/doproxy).
Important files:
doproxy.rb
: DOProxy Ruby script. Provides the command line interface and logic behind DOProxydoproxy.yml
: DOProxy configuration file. Contains the API token and specifies Droplet create optionshaproxy.cfg.erb
: HAProxy configuration template. Used to generate load balancer configuration with proper backend server informationinventory
: Droplet inventory file. Stores IDs of created Dropletsuser-data.yml
: Userdata file. A cloud-config file that will run on a new Droplet when it is createdLet’s dive into the configuration files first.
These are the important lines in doproxy.yml
:
token: YOUR_DO_API_TOKEN
ssh_key_ids:
- YOUR_SSH_KEY_FINGERPRINT
...
droplet_options:
hostname_prefix: auto-nginx
region: nyc3
size: 1gb
image: ubuntu-16-04-x64
The token
property is the one that must hold your read and write API token.
The other lines specify the options that will be used when DOProxy creates a new Droplet. For example, installing of the specified SSH key (by ID or fingerprint) and prefixing the hostnames with “auto-nginx”.
More information about valid Droplet options can be found in the DigitalOcean API documentation.
This is the file that will be executed by cloud-init when each new Droplet is created. This means that you can supply a cloud-config file or a script to install your application software on each new Droplet.
The sample userdata file contains a simple bash script that installs Nginx on an Ubuntu server and replaces its default configuration file with the Droplet’s hostname, ID, and public IP address:
#!/bin/bash
apt-get -y update
apt-get -y install nginx
export DROPLET_ID=$(curl http://169.254.169.254/metadata/v1/id)
export HOSTNAME=$(curl -s http://169.254.169.254/metadata/v1/hostname)
export PUBLIC_IPV4=$(curl -s http://169.254.169.254/metadata/v1/interfaces/public/0/ipv4/address)
echo Droplet: $HOSTNAME, ID: $DROPLET_ID, IP Address: $PUBLIC_IPV4 > /var/www/html/index.html
Those curl
commands are retrieving the information about the Droplet (hostname, ID, and IP address) using the DigitalOcean Metadata service.
In a production implementation, this file would contain the commands to, for instance, install and configure your application. You can also use this to automate the integration of your Droplets into your overall infrastructure by doing things like automatically installing SSH keys and connecting to your configuration management or monitoring tools.
To read more about userdata, cloud-config, and metadata, check out these links:
The HAProxy configuration template contains most of the load balancer configuration, with some Ruby code that will be replaced with backend Droplet information.
We’ll just look at the Ruby section that generates the backend configuration:
backend www-backend
<% @Droplets.each_with_index do |droplet, index| %>
server www-<%= droplet.id %> <%= droplet.private_ip %>:80 check # id:<%= droplet.id %>, hostname:<%= droplet.name -%>
<% end %>
This code iterates through each of the Droplets in the inventory and adds a new HAProxy backend entry for each one of them (based on the private IP address).
For example, a line like this will be produced for each Droplet:
server www-4202645 192.0.2.175:80 check # id:4202645, hostname:auto-nginx-0
Whenever a Droplet is created or deleted, DOProxy generates a new HAProxy configuration file containing the changes.
This Ruby script consists mainly of a DOProxy class that contains the methods that perform the Droplet creation and deletion, inventory management and HAProxy configuration generation.
If you understand Ruby, check out the file on GitHub: https://github.com/scanevari/doproxy/blob/master/doproxy.rb.
If you don’t understand Ruby, here is some simplified pseudocode
that explains each method. It may be useful to compare this against the actual Ruby code to help you understand what is happening.
def initialize
Executed every time DOProxy runs with any valid arguments:
doproxy.yml
configuration file and get API token and Droplet options.def get\_inventory
Retrieves information for each Droplet in the inventory file. It must be executed before any of the other methods are executed.
def print\_inventory
This method prints Droplet information for each of the Droplet IDs in the inventory file. It is invoked with the doproxy.rb print
command.
def create\_server
When invoked through the doproxy.rb create
command, this method creates a new Droplet and adds it to the inventory file. Then, it calls reload_haproxy
to re-generate the HAProxy configuration file and reload the load balancer.
reload_haproxy
to regenerate the HAProxy configuration file and reload the load balancerdef delete\_server(line\_number)
When the doproxy.rb delete
command is used, this method deletes the specified Droplet and deletes its ID from the inventory file. It then calls reload_haproxy
to re-generate the HAProxy configuration file and reload the load balancer.
reload_haproxy
to re-generate the HAProxy configuration file and reload the load balancerdef generate\_haproxy\_cfg
This is a supporting method that creates new HAProxy configuration files based on the Droplets in the inventory.
haproxy.cfg.erb
)haproxy.cfg
file to diskdef reload\_haproxy
This is another supporting method that copies the HAProxy configuration file into the proper location and reloads HAProxy. This relies on generate_haproxy_cfg
.
haproxy.cfg
to the location where HAProxy will look for it on reloadThat’s all of the important code that makes DOProxy work. The last thing we will discuss is DropletKit, the API wrapper that we used in DOProxy.
DOProxy uses the DropletKit gem which is the official DigitalOcean API v2 Ruby wrapper that facilitates calls made to the DigitalOcean API. DropletKit allows us to easily write Ruby programs that do things like:
This tutorial focused on these particular API endpoints, but keep in mind that there are many other endpoints that can help facilitate programmatic management of your DigitalOcean server infrastructure.
Now that you’ve seen how a simple script can help scale a server environment by leveraging the DigitalOcean API, cloud-config, and metadata, you can apply all these concepts to scale your own server setup. Although DOProxy isn’t intended for production use, it should give you a good set of ideas for implementing your own scaling solution.
Remember that the scaling setup described here with DOProxy is informational, but it could be greatly improved by using it in conjunction with our monitoring system. This would allow you to automatically scale your application server tier up and down depending on certain conditions such as server resource utilization.
]]>Object storage is a popular and scalable method of storing and serving static assets such as audio, images, text, PDFs, and other types of unstructured data. Cloud providers offer object storage in addition to traditional local or block storage, which is used to store dynamic application files and databases. Read Object Storage vs. Block Storage to learn about the use cases and differences between the two.
Spaces is a simple object storage service offered by DigitalOcean. In addition to being able to login and upload, manage, and delete stored files through a control panel, you can also access your DigitalOcean Space through the command line and the Spaces API.
In this tutorial, we will create a Node.js application that allows a user to upload a file to their DigitalOcean Space by submitting a form on the front-end of a website.
To follow along with this tutorial, you will need:
You should now have a DigitalOcean account, a Space with access key, and Node.js and npm installed on your computer.
DigitalOcean Spaces is compatible with the Amazon Simple Storage Service (S3) API, and we will be using the AWS SDK for JavaScript in Node.js to connect to the Space we created.
The first step is to create a credentials file, to place the access key and secret access key you obtained when you created your DigitalOcean Space. The file will be located at ~/.aws/credentials
on Mac and Linux, or C:\Users\USERNAME\.aws\credentials
on Windows. If you have previously saved AWS credentials, you can read about keeping multiple sets of credentials for further guidance.
Open your command prompt, make sure you’re in your Users directory, have access to an administrative sudo
user, and create the .aws
directory with the credentials
file inside.
- sudo mkdir .aws && touch .aws/credentials
Open the file, and paste the following code inside, replacing your_access_key
and your_secret_key
with your respective keys.
[default]
aws_access_key_id=your_access_key
aws_secret_access_key=your_secret_key
Now your access to Spaces via the AWS SDK will be authenticated, and we can move on to creating the application.
To begin, create a directory in which you would like to place your Node.js application and navigate to the directory. For this demonstration, we will create our project in spaces-node-app
in the sites
directory.
- mkdir sites/spaces-node-app && cd sites/spaces-node-app
Create a new package.json
file for your project. Paste the code below into the file.
{
"name": "spaces-node-app",
"version": "1.0.0",
"main": "server.js",
"scripts": {
"start": "node server.js"
},
"license": "MIT"
}
This is a basic package.json
file listing the name, version number, and license of our application. The scripts
field will allow us to run a Node.js server by typing npm start
instead of node server.js
.
We will install all of our dependencies with the npm install
command, followed by the names of the four dependencies in our project.
- npm install aws-sdk express multer multer-s3
After running this command, the package.json
file should be updated. These dependencies will aid us in connecting to the DigitalOcean Spaces API, creating a web server, and handling file uploads.
aws-sdk
— AWS SDK for JavaScript will allow us to access S3 through a JavaScript API.express
— Express is a web framework that will allow us to quickly and efficiently set up a server.multer
— Multer is middleware that will handle file uploads.multer-s3
— Multer S3 extends file uploads to S3 object storage, and in our case, DigitalOcean Spaces.Now that we have our project location and dependencies set up, we can set up the server and front-end views.
Note: npm install
saves dependencies to the package.json
file by default in current versions of Node. If you are running an older version of Node, you will have to add the --save
flag to your npm install
command to ensure that package.json
gets updated.
First, let’s create files for the public views of our application. This is what the user will see on the front end. Create a public directory in your project, with index.html
, success.html
, and error.html
. All three of these files will have the below HTML skeleton, with different contents in the body
. Write the following code into each file.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>DigitalOcean Spaces Tutorial</title>
<link rel="stylesheet" href="./style.css">
</head>
<body>
<!-- contents will go here -->
</body>
</html>
Write an error message in the body
of error.html
.
...
<h1>Something went wrong!</h1>
<p>File was not uploaded successfully.</p>
...
Write a success message in the body
of success.html
.
...
<h1>Success!</h1>
<p>File uploaded successfully.</p>
...
In index.html
, we will create an HTML form
with multipart/form-data
. It will consist of a simple file upload input
and a submit button.
...
<h1>DigitalOcean Spaces Tutorial</h1>
<p>Please select a file and submit the form to upload an asset to your DigitalOcean Space.</p>
<form method="post" enctype="multipart/form-data" action="/upload">
<label for="file">Upload a file</label>
<input type="file" name="upload">
<input type="submit" class="button">
</form>
...
Finally, let’s create style.css
and add just enough CSS to make the application easy to read.
html {
font-family: sans-serif;
line-height: 1.5;
color: #333;
}
body {
margin: 0 auto;
max-width: 500px;
}
label,
input {
display: block;
margin: 5px 0;
}
With these three files, we have an upload form which makes up the main page of our small application, and we have success and error pages for the user.
We’ve created all the files for the front end of our application, but we currently don’t have a server set up or any way to view them. We will set up a Node server with the Express web framework.
In the root directory of the project, create a server.js
file. At the top, load in our four dependencies with require()
. We will route our application through the app
instance of express
.
// Load dependencies
const aws = require('aws-sdk');
const express = require('express');
const multer = require('multer');
const multerS3 = require('multer-s3');
const app = express();
Our front end is located in the public
directory, so set that configuration below the dependencies.
...
// Views in public directory
app.use(express.static('public'));
We will route index.html
, success.html
, and error.html
relative to the root of the server.
...
// Main, error and success views
app.get('/', function (request, response) {
response.sendFile(__dirname + '/public/index.html');
});
app.get("/success", function (request, response) {
response.sendFile(__dirname + '/public/success.html');
});
app.get("/error", function (request, response) {
response.sendFile(__dirname + '/public/error.html');
});
Finally, we will tell the server which port to listen on. In this example, 3001
is used, but you can set it to any available port.
...
app.listen(3001, function () {
console.log('Server listening on port 3001.');
});
Save server.js
and start the server. You can do this by running node server.js
, or with npm start
, the shortcut we set in package.json
.
- npm start
Output> node server.js
Server listening on port 3001.
Navigate to http://localhost:3001
, and you will see the upload form, since we set index.html
to be the root of the server.
You can also navigate to http://localhost:3001/success
and http://localhost:3001/error
to ensure those pages are routing properly.
Now that we have our server environment up and running properly, the last step is to integrate the form with Multer and Multer S3 to make a file upload to Spaces.
You can use new aws.S3()
to connect to the Amazon S3 client. For use with DigitalOcean Spaces, we’ll need to set a new endpoint to ensure it uploads to the correct location. At the time of writing, nyc3
is the only region available for Spaces.
In server.js
, scroll back up to the top and paste the following code below the constant declarations.
...
const app = express();
// Set S3 endpoint to DigitalOcean Spaces
const spacesEndpoint = new aws.Endpoint('nyc3.digitaloceanspaces.com');
const s3 = new aws.S3({
endpoint: spacesEndpoint
});
Using the example from the multer-s3 documentation, we will create an upload
function, setting the bucket
property to your unique Space name. Setting acl
to public-read
will ensure our file is accessible to the public; leaving this blank will default to private, making the files inaccessible from the web.
...
// Change bucket property to your Space name
const upload = multer({
storage: multerS3({
s3: s3,
bucket: 'your-space-here',
acl: 'public-read',
key: function (request, file, cb) {
console.log(file);
cb(null, file.originalname);
}
})
}).array('upload', 1);
The upload
function is complete, and our last step is to connect the upload form with code to send the file through and route the user accordingly. Scroll to the bottom of server.js
, and paste this code right above the app.listen()
method at the end of the file.
...
app.post('/upload', function (request, response, next) {
upload(request, response, function (error) {
if (error) {
console.log(error);
return response.redirect("/error");
}
console.log('File uploaded successfully.');
response.redirect("/success");
});
});
When the user clicks submit, a POST request goes through to /upload
. Node is listening for this POST, and calls the upload()
function. If an error is found, the conditional statement will redirect the user to the /error
page. If it went through successfully, the user will be redirected to the /success
page, and the file will be uploaded to your Space.
Here is the entire code for server.js
.
// Load dependencies
const aws = require('aws-sdk');
const express = require('express');
const multer = require('multer');
const multerS3 = require('multer-s3');
const app = express();
// Set S3 endpoint to DigitalOcean Spaces
const spacesEndpoint = new aws.Endpoint('nyc3.digitaloceanspaces.com');
const s3 = new aws.S3({
endpoint: spacesEndpoint
});
// Change bucket property to your Space name
const upload = multer({
storage: multerS3({
s3: s3,
bucket: 'your-space-here',
acl: 'public-read',
key: function (request, file, cb) {
console.log(file);
cb(null, file.originalname);
}
})
}).array('upload', 1);
// Views in public directory
app.use(express.static('public'));
// Main, error and success views
app.get('/', function (request, response) {
response.sendFile(__dirname + '/public/index.html');
});
app.get("/success", function (request, response) {
response.sendFile(__dirname + '/public/success.html');
});
app.get("/error", function (request, response) {
response.sendFile(__dirname + '/public/error.html');
});
app.post('/upload', function (request, response, next) {
upload(request, response, function (error) {
if (error) {
console.log(error);
return response.redirect("/error");
}
console.log('File uploaded successfully.');
response.redirect("/success");
});
});
app.listen(3001, function () {
console.log('Server listening on port 3001.');
});
Stop the Node server by typing CONTROL
+ C
in the command prompt, and restart it to ensure the new changes are applied.
- npm start
Navigate to the root of the project, select a file, and submit the form. If everything was set up properly, you will be redirected to the success page, and a public file will be available on your DigitalOcean Space.
Assuming the file you uploaded was test.txt
, the URL of the file will be https://your-space-here.nyc3.digitaloceanspaces.com/test.txt
.
Common reasons for an unsuccessful transaction would be wrong credentials, credentials file in the wrong location, or an incorrect bucket name.
Congratulations, you’ve set up a Node.js and Express application to upload static assets to object storage!
You can play around with the code of this DigitalOcean Spaces Node App by remixing the project here.
Additional precautions such as authentication must be taken to put this type of application into production, but this is a good starting point to make your web app functional with DigitalOcean Spaces. For more information about object storage, read An Introduction to DigitalOcean Spaces.
]]>Heartbeat is an open-source program that provides cluster infrastructure capabilities — cluster membership and messaging — to client servers. These capabilities are a critical component in a high availability (HA) server infrastructure. In this tutorial, we will demonstrate how to create a 2-node HA server setup by simply using Heartbeat and a DigitalOcean Reserved IP.
Heartbeat is typically used in conjunction with a cluster resource manager (CRM), such as Pacemaker, to achieve a complete HA setup. If you are looking to create a more robust HA setup, look into using Corosync and Pacemaker or Keepalived.
When completed, the HA setup will consist of two Ubuntu 16.04 servers in an active/passive configuration. This will be accomplished by pointing to a Reserved IP, which is how your users will access your services or website, to point to the primary — or active — server unless a failure is detected. In the event that the Heartbeat service detects that the primary server is unavailable, the secondary server will automatically run a script to reassign the Reserved IP to itself via the DigitalOcean API. Thus, subsequent network traffic to the Reserved IP will be directed to your secondary server, which will act as the active server until the primary server becomes available again (at which point, the primary server will reassign the Reserved IP to itself).
Note: This tutorial is intended for demonstration purposes and only covers some of the aspects of setting up a reliable HA solution. The main takeaways of this document are the details on how to install active/passive nodes at the gateway level and to tie them up to a Reserved IP. To keep the tutorial simpler, instead of configuring reverse-proxy load balancers on each server, we will configure them to respond with their respective hostname and public IP address.
To achieve this goal, we will follow these steps:
With this goal in mind, we can begin working on setting up our HA setup.
In order to automate the reserved IP reassignment, we must use the DigitalOcean API. This means that you need to generate a Personal Access Token (PAT), which is an API token that can be used to authenticate to your DigitalOcean account, with read and write access. You can achieve this by following the How To Generate a Personal Access Token section of the API tutorial. Your PAT will be used in a script that will be added to both servers in your cluster. It is important that you keep it somewhere safe for reference, as it allows full access to your DigitalOcean account.
In addition to the API, this tutorial utilizes the following DigitalOcean features:
Please read the linked tutorials if you want to learn more about them.
The first step is to create two Ubuntu Droplets in the same datacenter, which will act as the primary and secondary servers described above. In our example setup, we will name them “primary” and “secondary” for easy reference. We will install Nginx on both Droplets and replace their index pages with information that uniquely identifies them. This will allow us a simple way to demonstrate that the HA setup is working. For a production setup, your servers should run the web server or load balancer of your choice.
Create two Ubuntu 16.04 Droplets, primary and secondary, with this bash script as the user data:
#!/bin/bash
apt-get -y update
apt-get -y install nginx
export HOSTNAME=$(curl -s http://169.254.169.254/metadata/v1/hostname)
export PUBLIC_IPV4=$(curl -s http://169.254.169.254/metadata/v1/interfaces/public/0/ipv4/address)
echo Droplet: $HOSTNAME, IP Address: $PUBLIC_IPV4 > /var/www/html/index.html
This will install Nginx and replace the contents of index.html
with the Droplet’s hostname and IP address (by referencing the Metadata service). Accessing either Droplet via its public IP address will show a basic webpage with the Droplet hostname and IP address, which will be useful for testing which Droplet the Reserved IP is pointing to at any given moment.
In the DigitalOcean Control Panel, click Networking, in the top menu, then Reserved IPs in the sub menu.
Assign a Reserved IP to your primary Droplet, then click the Assign Reserved IP button.
After the Reserved IP has been assigned, check that you can reach the Droplet that it was assigned to by visiting it in a web browser.
http://your_reserved_ip
You should see the index page of your primary Droplet.
If you want to be able to access your HA setup via a domain name, go ahead and create an A record in your DNS that points your domain to your Reserved IP address. If your domain is using DigitalOcean’s nameservers, follow step three of the How To Set Up a Host Name with DigitalOcean tutorial. Once that propagates, you may access your active server via the domain name.
The example domain name we’ll use is example.com
. If you don’t have a domain name right now, you should use the Reserved IP address instead.
The next step is to install Heartbeat on both servers. The simplest way to install Heartbeat is to use apt-get:
sudo apt-get update
sudo apt-get install heartbeat
Heartbeat is now installed but it needs to be configured before it will do anything.
In order to get our desired cluster up and running, we must create and set up these Heartbeat configuration files identically in both servers’ /etc/ha.d
directories:
ha.cf
— Global configuration of the Heartbeat cluster, including its member nodesauthkeys
— Contains a security key that provides nodes a way to authenticate to the clusterharesources
— Specifies the services that are managed by the cluster and the node that is the preferred owner of the services. Note that this file is not used in a setup that uses a CRM like PacemakerWe will also need to provide a script that will perform the Reserved IP reassignment in the event that the primary Droplet’s availability changes.
Before configuring ha.cf
, we should look up the names of each node. Heartbeat requires that each node name matches their respective uname -n
output.
On both servers, run this command to look up the appropriate node names:
- uname -n
Note the output of the command. The example node names are “primary” and “secondary”, which matches what we named the Droplets.
To determine which nodes are available, we will also need to look up the network interface and IP address that each node will use to communicate with the rest of the cluster. You may use any network interface, as long as each node can reach the other nodes in the cluster. We’ll use the public interface of our Droplets, which happens to be eth0
.
On both servers, use this command to look up the IP address of the eth0
interface (or look it up in the DigitalOcean Control Panel):
- ip addr show eth0
ip addr show eth0 output:2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 04:01:76:a5:45:01 brd ff:ff:ff:ff:ff:ff
inet 198.51.100.5/24 brd 198.51.100.255 scope global eth0
valid_lft forever preferred_lft forever
inet 10.17.0.28/16 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::601:76ff:fea5:4501/64 scope link
valid_lft forever preferred_lft forever
Note the IP address of the network interface (highlighted in the example). Be sure to get the IP addresses of both servers.
On both servers, open /etc/ha.d/ha.cf
in your favorite editor. We’ll use nano:
- sudo nano /etc/ha.d/ha.cf
The file should be new and empty. We need to add the network interfaces and names of each node in our cluster.
Copy and paste this configuration into the file, then replace the respective node names and IP addresses with the values that we looked up earlier. In this example, primary’s IP address is 198.51.100.5
and secondary’s IP address is 198.51.100.6
:
node primary
ucast eth0 198.51.100.5
node secondary
ucast eth0 198.51.100.6
Save and exit the file. Next, we’ll set up the cluster’s authorization key.
The authorization key is used to allow cluster members to join a cluster. We can simply generate a random key for this purpose.
On the primary node, run these commands to generate a suitable authorization key in an environment variable named AUTH_KEY
:
if [ -z "${AUTH_KEY}" ]; then
export AUTH_KEY="$(command dd if='/dev/urandom' bs=512 count=1 2>'/dev/null' \
| command openssl sha1 \
| command cut --delimiter=' ' --fields=2)"
fi
Then write the /etc/ha.d/authkeys
file with these commands:
sudo bash -c "{
echo auth1
echo 1 sha1 $AUTH_KEY
} > /etc/ha.d/authkeys"
Check the contents of the authkeys
file like this:
- sudo cat /etc/ha.d/authkeys
It should look something like this (with a different authorization key):
auth1
1 sha1 d1e6557e2fcb30ff8d4d3ae65b50345fa46a2faa
Ensure that the file is only readable by the root user:
- sudo chmod 600 /etc/ha.d/authkeys
Now copy the /etc/ha.d/authkeys
file from your primary node to your secondary node. You can do this manually, or with scp
.
On the secondary server, be sure to set the permissions of the authkeys
file:
- sudo chmod 600 /etc/ha.d/authkeys
At this point, both servers should have an identical /etc/ha.d/authkeys
file.
The haresources
file specifies preferred hosts paired with services that the cluster manages. The preferred host is the node that should run the associated service(s) if the node is available. If the preferred host is not available, i.e. it is not reachable by the cluster, one of the other nodes will take over. In other words, the secondary server will take over if the primary server goes down.
On both servers, open the haresources
file in your favorite editor. We’ll use nano:
- sudo nano /etc/ha.d/haresources
Now add this line to the file, substituting in your primary node’s name if it is different:
primary floatip
Save and exit the file. This configures the primary server as the preferred host for the floatip
service, which is currently undefined. Let’s set up the floatip
service next.
Our Heartbeat cluster is configured to maintain the floatip
service, which a node can use to assign the Reserved IP to itself, but we still need to create the service. Before we set up the service itself, however, let’s create a script that will assign the Reserved IP, via the DigitalOcean API, to the node that runs it. Then we will create the floatip
service which will run the Reserved IP reassignment script.
For our example, we’ll download a basic Python script that assigns a Reserved IP to a given Droplet ID, using the DigitalOcean API.
On both servers, download the assign-ip
Python script:
- sudo curl -L -o /usr/local/bin/assign-ip http://do.co/assign-ip
On both servers, make it executable:
- sudo chmod +x /usr/local/bin/assign-ip
Since our script is making a request to an API, we’ll need the Python Requests library installed:
sudo apt-get install python-requests
Use of the assign-ip
script requires the following details:
DO_TOKEN
, your read/write DigitalOcean PATFeel free to review the contents of the script before continuing.
Now we’re ready to create the floatip
service.
floatip
ServiceTo create the floatip
service, all we need to do is create an init script that invokes the assign-ip
script that we created earlier, and responds to start
and stop
subcommands. This init script will be responsible for looking up the Droplet ID of the server, via the Droplet Metadata service. Also, it will require the Reserved IP that will be reassigned, and the DigitalOcean API token (the Personal Access Token mentioned in the prerequisites section).
On both servers, add open /etc/init.d/floatip
in an editor:
- sudo nano /etc/init.d/floatip
Then copy and paste in this init script, replacing the highlighted parts with your DigitalOcean API key and the Reserved IP that should be reassigned:
- #!/bin/bash
-
- param=$1
-
- export DO_TOKEN='your_DO_API_token'
- IP='your_reserved_IP_address'
- ID=$(curl -s http://169.254.169.254/metadata/v1/id)
-
- if [ "start" == "$param" ] ; then
- python /usr/local/bin/assign-ip $IP $ID
- exit 0
- elif [ "stop" == "$param" ] ; then
- exit 0;
- elif [ "status" == "$param" ] ; then
- exit 0;
- else
- echo "no such command $param"
- exit 1;
- fi
Save and exit the file.
Make the script executable:
- sudo chmod u+x /etc/init.d/floatip
When this floatip
service is started, it will simply call the assign-ip
Python script and assign the specified Reserved IP to the Droplet that executed the script. This is the script that will be called by the secondary server, if the primary server fails, to reassign the Reserved IP to itself, . Likewise, the same script will be used by the primary server, to reclaim the Reserved IP, once it rejoins the cluster.
Now that Heartbeat is configured and all the scripts it relies on are set up, we’re ready to start the Heartbeat cluster!
On both servers, run this command to start Heartbeat:
- sudo systemctl start heartbeat
Our HA setup is now complete! Before moving on, let’s test that it works as intended.
It’s important to test that a high availability setup works, so let’s do that now.
Currently, the Reserved IP is assigned to the primary node. Accessing the Reserved IP now, via the IP address or by the domain name that is pointing to it, will simply show the index page of the primary server. If you used the example user data script, it will look something like this:
Reserved IP is pointing to primary serverDroplet: primary, IP Address: 198.51.100.5
This indicates that the Reserved IP is, in fact, assigned to the primary Droplet.
Now, let’s open a local terminal and use curl
to access the Reserved IP on a 1-second loop. Use this command to do so, but be sure to replace the URL with your domain or Reserved IP address:
- while true; do curl http://example.com; sleep 1; done
Currently, this will output the same Droplet name and IP address of the primary server. If we cause the primary server to fail, by powering it off or stopping the Heartbeat service, we will see if the Reserved IP gets reassigned to the secondary server.
Let’s reboot the primary server now. Do so via the DigitalOcean Control Panel or by running this command on the primary server:
- sudo reboot
After a few moments, the primary server should become unavailable. Pay attention to the output of the curl
loop that is running in the terminal. You should notice output that looks like this:
curl loop output:Droplet: primary, IP Address: 198.51.100.5
...
curl: (7) Failed to connect to example.com port 80: Connection refused
Droplet: secondary, IP Address: 198.51.100.6
Droplet: secondary, IP Address: 198.51.100.6
...
That is, the Reserved IP address should be reassigned to point to the IP address of the secondary server. That means that your HA setup is working, as a successful automatic failover has occurred.
You may or may not see the Connection refused
error, which can occur if you try and access the Reserved IP between the primary server failure and the Reserved IP reassignment completion.
Now, you may power on your primary Droplet, via the DigitalOcean Control Panel. Because Heartbeat is configured with the primary Droplet as the preferred host to run the Reserved IP reassignment script, the Reserved IP will automatically point back to the primary server as soon as it becomes available again.
Congratulations! You now have a basic HA server setup using Heartbeat and a DigitalOcean Reserved IP.
If you are looking to create a more robust HA setup, look into using Corosync and Pacemaker or Keepalived.
In this example, we’ve installed Nginx as a basic load balancer but if you wanted to improve your Heartbeat setup utilizing a reverse-proxy load balancer, you could do so by either configuring Nginx as one, or using HAProxy.
Please keep in mind that with either alternative you choose to use, you will want to bind your load balancer/reverse-proxy to the anchor IP address so that your users can only access your servers via the Reserved IP address (and not via the public IP address of each server).
]]>DigitalOcean Spaces is an object storage service designed to make it easy and cost effective to store and serve large amounts of data. If you have previously relied on other object storage services, migrating data to Spaces may be one of your first tasks.
In this guide, we will cover how to migrate data to DigitalOcean Spaces from Amazon’s S3 block storage service using the rclone
utility. We will demonstrate how to install rclone
, the configuration settings to use to access both storage services, and the commands that you can use to synchronize your files and verify their integrity within Spaces.
Before we begin installing and configuring rclone
to copy our objects to Spaces, we will need some information about our Amazon S3 and DigitalOcean Spaces accounts. We will need a set of API keys for both services that the tool can use and we will need to know the region and location constraint values for our buckets.
To create a DigitalOcean Spaces API key, follow the “Creating an Access Key” section of our How To Create a DigitalOcean Space and API Key tutorial.
Save the access key ID and the secret key so that we can configure rclone
to access our account.
Next, we need to find the appropriate API endpoint. If you’ve already created a DigitalOcean Space you wish to transfer your objects to, you can view the Space’s endpoint within the DigitalOcean Control Panel by selecting the Space and viewing the Settings tab:
If you have not created a Space yet, rclone
can automatically create the space you select as part of the copying process. The endpoint in that case would be the Spaces region you wish to use followed by .digitaloceanspaces.com
. You can find the available regions for Spaces in the DigitalOcean Control Panel by viewing the selection options on the Spaces creation page. At the time of this writing only the “nyc3” region is available (with endpoint of nyc3.digitaloceanspaces.com
).
If you do not already have an Amazon API key with permission to manage S3 assets, you will need to generate those now. In your AWS Management Console, click on your account name and select My Security Credentials from the drop down menu:
Next, select Users in the left-hand menu and then click the Add user button:
Type in a User name and select Programmatic access in the Access type section. Click the Next: Permissions button to continue:
On the page that follows, select the Attach existing policies directly option at the top and then type s3read in the Policy type filter. Check the AmazonS3ReadOnlyAccess policy box and then click the Next: Review button to continue:
Review the user details on the next page and then click the Create user button when ready:
On the final page, you will see the credentials for your new user. Click the Show link under the Secret access key column to view the credentials:
Copy the Access key ID and the Secret access key somewhere secure so that you can configure rclone
to use those credentials. You can also click the Download .csv button to save the credentials to your computer.
Now, we need to find the region and location constraint values for our S3 bucket.
Click Services in the top menu and type S3 in the search bar that appears. Select the S3 service to go to the S3 management console.
We need to look for the region name of the bucket we wish to transfer. The region will be displayed next to the bucket name:
We need to find the region string and the matching location restraint associated with our bucket’s region. Look for your bucket’s region name in this S3 region chart from Amazon to find the appropriate region and location constraint strings. In our example, our region name is “US East (N. Virginia)”, so we would use us-east-1
as the region string and our location constraint would be blank.
Now that we have the appropriate information from our Amazon account, we can install and configure rclone
using this information.
You’re now ready to install rclone
on your local computer.
Visit the Downloads section of the project’s website to find binaries of the utility compiled for different platforms. Download the zipped binary that matches your computer’s operating system to your Downloads directory to get started.
Once you have the rclone
zip file downloaded to your computer, follow the section below that matches your platform.
Before we can extract the archive, we will need to ensure that the unzip
utility is available.
If you are running Ubuntu or Debian, you can update the local package index and install unzip
by typing:
- sudo apt-get update
- sudo apt-get install unzip
If you are running CentOS or Fedora, you can install unzip
by typing:
- sudo yum install unzip
With unzip
installed, navigate to the directory where you downloaded the rclone
zip file:
- cd ~/Downloads
Next, unzip the archive and move into the new directory:
- unzip rclone*
- cd rclone-v*
From here, we can copy the binary to the /usr/local/bin
directory so that it is available system-wide:
- sudo cp rclone /usr/local/bin
Next, we can add the manual page to the system so that we can easily get help on the command syntax and available options. Make sure that the local manual directory we require is available and then copy the rclone.1
file:
- sudo mkdir -p /usr/local/share/man/man1
- sudo cp rclone.1 /usr/local/share/man/man1
Update the man
database to add the new manual page to the system:
- sudo mandb
Finally, we can create the configuration directory and open up a configuration file to define our S3 and Spaces credentials:
- mkdir -p ~/.config/rclone
- nano ~/.config/rclone/rclone.conf
This will open up your text editor with a new blank file. Skip ahead to the section on defining your object storage accounts to continue.
If you are running macOS, begin by navigating in the terminal to the directory where you downloaded the rclone
zip file:
- cd ~/Downloads
Next, unzip the file and move into the new directory level:
- unzip -a rclone*
- cd rclone-v*
Next, make sure the /usr/local/bin
directory is available and then move the rclone
binary inside:
- sudo mkdir -p /usr/local/bin
- sudo cp rclone /usr/local/bin
Finally, we can create the configuration directory and open up a configuration file to define our S3 and Spaces credentials:
- mkdir -p ~/.config/rclone
- nano ~/.config/rclone/rclone.conf
This will open up your text editor with a new blank file. Skip ahead to the section on defining your object storage accounts to continue.
If you are running Windows, begin by navigating to the Downloads directory in the Windows File Explorer. Select the rclone
zip file and right-click. In the context menu that appears, click Extract All…:
Follow the prompts to extract the files from the zip archive.
The rclone.exe
utility must be run from the command line. Open a new Command Prompt (the cmd.exe
program) window by clicking the Windows button in the lower-left corner, typing cmd, and selecting Command Prompt.
Inside, navigate to the rclone
path you extracted by typing:
- cd "%HOMEPATH%\Downloads\rclone*\rclone*"
List the directory contents to verify that you are in the correct location:
- dir
Output10/23/2017 01:02 PM <DIR> .
10/23/2017 01:02 PM <DIR> ..
10/23/2017 01:02 PM 17 git-log.txt
10/23/2017 01:02 PM 296,086 rclone.1
10/23/2017 01:02 PM 16,840,192 rclone.exe
10/23/2017 01:02 PM 315,539 README.html
10/23/2017 01:02 PM 261,497 README.txt
5 File(s) 17,713,331 bytes
2 Dir(s) 183,296,266,240 bytes free
You will need to be in this directory whenever you want to use the rclone.exe
command.
Note: On macOS and Linux, we run the tool by typing rclone
, but on Windows, the command is called rclone.exe
. Throughout the rest of this guide, we will be providing commands as rclone
, so be sure to substitute rclone.exe
each time when running on Windows.
Next, we can create the configuration directory and open up a configuration file to define our S3 and Spaces credentials:
- mkdir "%HOMEPATH%\.config\rclone"
- notepad "%HOMEPATH%\.config\rclone\rclone.conf"
This will open up your text editor with a new blank file. Continue ahead to learn how to define your object storage accounts in the configuration file.
We can define our Amazon S3 and DigitalOcean Spaces configuration in the new file so that rclone
can manage content between our two accounts.
Let’s start by defining our S3 account. Paste the following section in the configuration file:
[s3]
type = s3
env_auth = false
access_key_id = aws_access_key
secret_access_key = aws_secret_key
region = aws_region
location_constraint = aws_location_constraint
acl = private
Here, we define a new rclone
“remote” called s3
. We set the type
to s3
so that rclone
knows the appropriate way to interact with and manage the remote storage resource. We will define the S3 credentials in the configuration file itself, so we set env_auth
to false
.
Next, we set the access_key_id
and secret_access_key
variables to our S3 access key and secret key, respectively. Be sure to change the values to the S3 credentials associated with your account.
We set the region and location constraint according to the properties of our S3 bucket that we found in the Amazon region chart. Finally, we set the access control policy to “private” so that assets are not public by default.
Now, we can define a similar section for our DigitalOcean Spaces configuration. Paste the following section in the configuration file:
. . .
[spaces]
type = s3
env_auth = false
access_key_id = spaces_access_key
secret_access_key = spaces_secret_key
endpoint = nyc3.digitaloceanspaces.com
acl = private
In this section, we are defining a new remote called “spaces”. Again, we are setting type
to s3
since Spaces offers an S3-compatible API. We turn off env_auth
so that we can define the Spaces credentials within the configuration file.
Next, we set the access_key_id
and secret_access_key
variables to the values generated for our DigitalOcean account. We set the endpoint
to the appropriate Spaces endpoint we determined earlier. Finally, we set the acl
to private
again to protect our assets until we want to share them.
Save and close the file when you are finished.
On macOS and Linux, be sure to lock down the permissions of the configuration file since our credentials are inside:
- chmod 600 ~/.config/rclone/rclone.conf
On Windows, permissions are denied to non-administrative users unless explicitly granted, so we shouldn’t need to adjust access manually.
Now that our configuration is complete, we are ready to transfer our files.
Begin by checking the rclone
configured remotes:
- rclone listremotes
Outputs3:
spaces:
Both of the sections we defined are displayed.
We can view the available S3 buckets by asking rclone
to list the “directories” associated with the s3
remote (make sure to add the colon to the end of the remote name):
- rclone lsd s3:
Output -1 2017-10-20 15:32:28 -1 source-of-files
The above output indicates that one bucket, called source-of-files
was found in our S3 account.
If you have already created a DigitalOcean Space, you can repeat the procedure to view your Spaces:
- rclone lsd spaces:
Output -1 2017-10-25 19:00:35 -1 existing-space
To view the contents of an S3 bucket or DigitalOcean Space, you can use the tree
command. Pass in the remote name, followed by a colon and the name of the “directory” you wish to list (the bucket or Space name):
- rclone tree s3:source-of-files
Output/
├── README.txt
├── demo_dir
│ ├── demo1
│ └── demo2
└── media
├── Social Rebrand Presentation 032815.ppt
├── TechnicLauncher.jar
├── nda_template.docx
├── textfile.txt
└── the_mother_of_all_demos.mp4
2 directories, 8 files
When you are ready, you can copy the files from your S3 bucket to a DigitalOcean Space by typing:
- rclone sync s3:source-of-files spaces:dest-of-files
If you hadn’t previously created the Space you selected, rclone
will attempt to create one for you with the given name. This will fail if the name provided is already being used by another account or if the name doesn’t meet the naming requirements for DigitalOcean Spaces (lowercase letters, numbers, and dashes only).
Assuming everything went well, rclone
will begin copying objects from S3 to Spaces.
When the transfer is complete, you can visually check that the objects have transferred by viewing them with the tree
subcommand:
- rclone tree spaces:dest-of-files
Output/
├── README.txt
├── demo_dir
│ ├── demo1
│ └── demo2
└── media
├── Social Rebrand Presentation 032815.ppt
├── TechnicLauncher.jar
├── nda_template.docx
├── textfile.txt
└── the_mother_of_all_demos.mp4
2 directories, 8 files
For more robust verification, use the check
subcommand to compare the objects in both remotes:
- rclone check s3:source-of-files spaces:dest-of-files
Output2017/10/25 19:51:36 NOTICE: S3 bucket dest-of-files: 0 differences found
2017/10/25 19:51:36 NOTICE: S3 bucket dest-of-files: 2 hashes could not be checked
This will compare the hash values of each object in both remotes. You may receive a message indicating that some hashes could not be compared. In that case, you can rerun the command with the --size-only
flag (which just compares based on file size) or the --download
flag (which downloads each object from both remotes to compare locally) to verify the transfer integrity.
In this guide, we’ve covered how to transfer objects from Amazon S3 to DigitalOcean Spaces. We created API credentials for both services, installed and configured the rclone
utility on our local computer, and then copied all objects from an S3 bucket to a DigitalOcean Space.
The rclone
client can be used for many other object storage management tasks including uploading or downloading files, mounting buckets on the local filesystem, and creating or deleting additional buckets. Check out the man
page to learn more about the functionality the tool provides.
Databases often store some of the most valuable information in your infrastructure. Because of this, it is important to have reliable backups to guard against data loss in the event of an accident or hardware failure.
The Percona XtraBackup backup tools provide a method of performing “hot” backups of MySQL data while the system is running. They do this by copying the data files at the filesystem level and then performing a crash recovery to achieve consistency within the dataset.
In a previous guide, we installed Percona’s backup utilities and created a series of scripts to perform rotating local backups. This works well for backing up data to a different drive or network mounted volume to handle problems with your database machine. However, in most cases, data should be backed up off-site where it can be easily maintained and restored. In this guide, we will extend our previous backup system to upload our compressed, encrypted backup files to an object storage service. We will be using DigitalOcean Spaces as an example in this guide, but the basic procedures are likely applicable for other S3-compatible object storage solutions as well.
Before you start this guide, you will need a MySQL database server configured with the local Percona backup solution outlined in our previous guide. The full set of guides you need to follow are:
sudo
privileges and configure a basic firewall.In addition to the above tutorials, you will also need to generate an access key and secret key to interact with your object storage account using the API. If you are using DigitalOcean Spaces, you can find out how to generate these credentials by following our How to Create a DigitalOcean Space and API Key guide. You will need to save both the API access key and API secret value.
When you are finished with the previous guides, log back into your server as your sudo
user to get started.
We will be using some Python and Bash scripts to create our backups and upload them to remote object storage for safekeeping. We will need the boto3
Python library to interact with the object storage API. We can download this with pip
, Python’s package manager.
Refresh our local package index and then install the Python 3 version of pip
from Ubuntu’s default repositories using apt-get
by typing:
- sudo apt-get update
- sudo apt-get install python3-pip
Because Ubuntu maintains its own package life cycle, the version of pip
in Ubuntu’s repositories is not kept in sync with recent releases. However, we can update to a newer version of pip
using the tool itself. We will use sudo
to install globally and include the -H
flag to set the $HOME
variable to a value pip
expects:
- sudo -H pip3 install --upgrade pip
Afterwards, we can install boto3
along with the pytz
module, which we will use to compare times accurately using the offset-aware format that the object storage API returns:
- sudo -H pip3 install boto3 pytz
We should now have all of the Python modules we need to interact with the object storage API.
Our backup and download scripts will need to interact with the object storage API in order to upload files and download older backup artifacts when we need to restore. They will need to use the access keys we generated in the prerequisite section. Rather than keeping these values in the scripts themselves, we will place them in a dedicated file that can be read by our scripts. This way, we can share our scripts without fear of exposing our credentials and we can lock down the credentials more heavily than the script itself.
In the last guide, we created the /backups/mysql
directory to store our backups and our encryption key. We will place the configuration file here alongside our other assets. Create a file called object_storage_config.sh
:
- sudo nano /backups/mysql/object_storage_config.sh
Inside, paste the following contents, changing the access key and secret key to the values you obtained from your object storage account and the bucket name to a unique value. Set the endpoint URL and region name to the values provided by your object storage service (we will use the values associated with DigitalOcean’s NYC3 region for Spaces here):
#!/bin/bash
export MYACCESSKEY="my_access_key"
export MYSECRETKEY="my_secret_key"
export MYBUCKETNAME="your_unique_bucket_name"
export MYENDPOINTURL="https://nyc3.digitaloceanspaces.com"
export MYREGIONNAME="nyc3"
These lines define two environment variables called MYACCESSKEY
and MYSECRETKEY
to hold our access and secret keys respectively. The MYBUCKETNAME
variable defines the object storage bucket we want to use to store our backup files. Bucket names must be universally unique, so you must choose a name that no other user has selected. Our script will check the bucket value to see if it is already claimed by another user and automatically create it if it is available. We export
the variables we define so that any processes we call from within our scripts will have access to these values.
The MYENDPOINTURL
and MYREGIONNAME
variables contain the API endpoint and the specific region identifier offered by your object storage provider. For DigitalOcean spaces, the endpoint will be https://region_name.digitaloceanspaces.com
. You can find the available regions for Spaces in the DigitalOcean Control Panel (at the time of this writing, only “nyc3” is available).
Save and close the file when you are finished.
Anyone who can access our API keys has complete access to our object storage account, so it is important to restrict access to the configuration file to the backup
user. We can give the backup
user and group ownership of the file and then revoke all other access by typing:
- sudo chown backup:backup /backups/mysql/object_storage_config.sh
- sudo chmod 600 /backups/mysql/object_storage_config.sh
Our object_storage_config.sh
file should now only be accessible to the backup
user.
Now that we have an object storage configuration file, we can go ahead and begin creating our scripts. We will be creating the following scripts:
object_storage.py
: This script is responsible for interacting with the object storage API to create buckets, upload files, download content, and prune older backups. Our other scripts will call this script anytime they need to interact with the remote object storage account.remote-backup-mysql.sh
: This script backs up the MySQL databases by encrypting and compressing the files into a single artifact and then uploading it to the remote object store. It creates a full backup at the beginning of each day and then an incremental backup every hour afterwards. It automatically prunes all files from the remote bucket that are older than 30 days.download-day.sh
: This script allows us to download all of the backups associated with a given day. Because our backup script creates a full backup each morning and then incremental backups throughout the day, this script can download all of the assets necessary to restore to any hourly checkpoint.Along with the new scripts above, we will leverage the extract-mysql.sh
and prepare-mysql.sh
scripts from the previous guide to help restore our files. You can view the scripts in the repository for this tutorial on GitHub at any time. If you do not want to copy and paste the contents below, you can download the new files directly from GitHub by typing:
- cd /tmp
- curl -LO https://raw.githubusercontent.com/do-community/ubuntu-1604-mysql-backup/master/object_storage.py
- curl -LO https://raw.githubusercontent.com/do-community/ubuntu-1604-mysql-backup/master/remote-backup-mysql.sh
- curl -LO https://raw.githubusercontent.com/do-community/ubuntu-1604-mysql-backup/master/download-day.sh
Be sure to inspect the scripts after downloading to make sure they were retrieved successfully and that you approve of the actions they will perform. If you are satisfied, mark the scripts as executable and then move them into the /usr/local/bin
directory by typing:
- chmod +x /tmp/{remote-backup-mysql.sh,download-day.sh,object_storage.py}
- sudo mv /tmp/{remote-backup-mysql.sh,download-day.sh,object_storage.py} /usr/local/bin
Next, we will set up each of these scripts and discuss them in more detail.
If you didn’t download the object_storage.py
script from GitHub, create a new file in the /usr/local/bin
directory called object_storage.py
:
- sudo nano /usr/local/bin/object_storage.py
Copy and paste the script contents into the file:
#!/usr/bin/env python3
import argparse
import os
import sys
from datetime import datetime, timedelta
import boto3
import pytz
from botocore.client import ClientError, Config
from dateutil.parser import parse
# "backup_bucket" must be a universally unique name, so choose something
# specific to your setup.
# The bucket will be created in your account if it does not already exist
backup_bucket = os.environ['MYBUCKETNAME']
access_key = os.environ['MYACCESSKEY']
secret_key = os.environ['MYSECRETKEY']
endpoint_url = os.environ['MYENDPOINTURL']
region_name = os.environ['MYREGIONNAME']
class Space():
def __init__(self, bucket):
self.session = boto3.session.Session()
self.client = self.session.client('s3',
region_name=region_name,
endpoint_url=endpoint_url,
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
config=Config(signature_version='s3')
)
self.bucket = bucket
self.paginator = self.client.get_paginator('list_objects')
def create_bucket(self):
try:
self.client.head_bucket(Bucket=self.bucket)
except ClientError as e:
if e.response['Error']['Code'] == '404':
self.client.create_bucket(Bucket=self.bucket)
elif e.response['Error']['Code'] == '403':
print("The bucket name \"{}\" is already being used by "
"someone. Please try using a different bucket "
"name.".format(self.bucket))
sys.exit(1)
else:
print("Unexpected error: {}".format(e))
sys.exit(1)
def upload_files(self, files):
for filename in files:
self.client.upload_file(Filename=filename, Bucket=self.bucket,
Key=os.path.basename(filename))
print("Uploaded {} to \"{}\"".format(filename, self.bucket))
def remove_file(self, filename):
self.client.delete_object(Bucket=self.bucket,
Key=os.path.basename(filename))
def prune_backups(self, days_to_keep):
oldest_day = datetime.now(pytz.utc) - timedelta(days=int(days_to_keep))
try:
# Create an iterator to page through results
page_iterator = self.paginator.paginate(Bucket=self.bucket)
# Collect objects older than the specified date
objects_to_prune = [filename['Key'] for page in page_iterator
for filename in page['Contents']
if filename['LastModified'] < oldest_day]
except KeyError:
# If the bucket is empty
sys.exit()
for object in objects_to_prune:
print("Removing \"{}\" from {}".format(object, self.bucket))
self.remove_file(object)
def download_file(self, filename):
self.client.download_file(Bucket=self.bucket,
Key=filename, Filename=filename)
def get_day(self, day_to_get):
try:
# Attempt to parse the date format the user provided
input_date = parse(day_to_get)
except ValueError:
print("Cannot parse the provided date: {}".format(day_to_get))
sys.exit(1)
day_string = input_date.strftime("-%m-%d-%Y_")
print_date = input_date.strftime("%A, %b. %d %Y")
print("Looking for objects from {}".format(print_date))
try:
# create an iterator to page through results
page_iterator = self.paginator.paginate(Bucket=self.bucket)
objects_to_grab = [filename['Key'] for page in page_iterator
for filename in page['Contents']
if day_string in filename['Key']]
except KeyError:
print("No objects currently in bucket")
sys.exit()
if objects_to_grab:
for object in objects_to_grab:
print("Downloading \"{}\" from {}".format(object, self.bucket))
self.download_file(object)
else:
print("No objects found from: {}".format(print_date))
sys.exit()
def is_valid_file(filename):
if os.path.isfile(filename):
return filename
else:
raise argparse.ArgumentTypeError("File \"{}\" does not exist."
.format(filename))
def parse_arguments():
parser = argparse.ArgumentParser(
description='''Client to perform backup-related tasks with
object storage.''')
subparsers = parser.add_subparsers()
# parse arguments for the "upload" command
parser_upload = subparsers.add_parser('upload')
parser_upload.add_argument('files', type=is_valid_file, nargs='+')
parser_upload.set_defaults(func=upload)
# parse arguments for the "prune" command
parser_prune = subparsers.add_parser('prune')
parser_prune.add_argument('--days-to-keep', default=30)
parser_prune.set_defaults(func=prune)
# parse arguments for the "download" command
parser_download = subparsers.add_parser('download')
parser_download.add_argument('filename')
parser_download.set_defaults(func=download)
# parse arguments for the "get_day" command
parser_get_day = subparsers.add_parser('get_day')
parser_get_day.add_argument('day')
parser_get_day.set_defaults(func=get_day)
return parser.parse_args()
def upload(space, args):
space.upload_files(args.files)
def prune(space, args):
space.prune_backups(args.days_to_keep)
def download(space, args):
space.download_file(args.filename)
def get_day(space, args):
space.get_day(args.day)
def main():
args = parse_arguments()
space = Space(bucket=backup_bucket)
space.create_bucket()
args.func(space, args)
if __name__ == '__main__':
main()
This script is responsible for managing the backups within your object storage account. It can upload files, remove files, prune old backups, and download files from object storage. Rather than interacting with the object storage API directly, our other scripts will use the functionality defined here to interact with remote resources. The commands it defines are:
upload
: Uploads to object storage each of the files that are passed in as arguments. Multiple files may be specified.download
: Downloads a single file from remote object storage, which is passed in as an argument.prune
: Removes every file older than a certain age from the object storage location. By default this removes files older than 30 days. You can adjust this by specifying the --days-to-keep
option when calling prune
.get_day
: Pass in the day to download as an argument using a standard date format (using quotations if the date has whitespace in it) and the tool will attempt to parse it and download all of the files from that date.The script attempts to read the object storage credentials and bucket name from environment variables, so we will need to make sure those are populated from the object_storage_config.sh
file before calling the object_storage.py
script.
When you are finished, save and close the file.
Next, if you haven’t already done so, make the script executable by typing:
- sudo chmod +x /usr/local/bin/object_storage.py
Now that the object_storage.py
script is available to interact with the API, we can create the Bash scripts that use it to back up and download files.
Next, we will create the remote-backup-mysql.sh
script. This will perform many of the same functions as the original backup-mysql.sh
local backup script, with a more basic organization structure (since maintaining backups on the local filesystem is not necessary) and some additional steps to upload to object storage.
If you did not download the script from the repository, create and open a file called remote-backup-mysql.sh
in the /usr/local/bin
directory:
- sudo nano /usr/local/bin/remote-backup-mysql.sh
Inside, paste the following script:
#!/bin/bash
export LC_ALL=C
days_to_keep=30
backup_owner="backup"
parent_dir="/backups/mysql"
defaults_file="/etc/mysql/backup.cnf"
working_dir="${parent_dir}/working"
log_file="${working_dir}/backup-progress.log"
encryption_key_file="${parent_dir}/encryption_key"
storage_configuration_file="${parent_dir}/object_storage_config.sh"
now="$(date)"
now_string="$(date -d"${now}" +%m-%d-%Y_%H-%M-%S)"
processors="$(nproc --all)"
# Use this to echo to standard error
error () {
printf "%s: %s\n" "$(basename "${BASH_SOURCE}")" "${1}" >&2
exit 1
}
trap 'error "An unexpected error occurred."' ERR
sanity_check () {
# Check user running the script
if [ "$(id --user --name)" != "$backup_owner" ]; then
error "Script can only be run as the \"$backup_owner\" user"
fi
# Check whether the encryption key file is available
if [ ! -r "${encryption_key_file}" ]; then
error "Cannot read encryption key at ${encryption_key_file}"
fi
# Check whether the object storage configuration file is available
if [ ! -r "${storage_configuration_file}" ]; then
error "Cannot read object storage configuration from ${storage_configuration_file}"
fi
# Check whether the object storage configuration is set in the file
source "${storage_configuration_file}"
if [ -z "${MYACCESSKEY}" ] || [ -z "${MYSECRETKEY}" ] || [ -z "${MYBUCKETNAME}" ]; then
error "Object storage configuration are not set properly in ${storage_configuration_file}"
fi
}
set_backup_type () {
backup_type="full"
# Grab date of the last backup if available
if [ -r "${working_dir}/xtrabackup_info" ]; then
last_backup_date="$(date -d"$(grep start_time "${working_dir}/xtrabackup_info" | cut -d' ' -f3)" +%s)"
else
last_backup_date=0
fi
# Grab today's date, in the same format
todays_date="$(date -d "$(date -d "${now}" "+%D")" +%s)"
# Compare the two dates
(( $last_backup_date == $todays_date ))
same_day="${?}"
# The first backup each new day will be a full backup
# If today's date is the same as the last backup, take an incremental backup instead
if [ "$same_day" -eq "0" ]; then
backup_type="incremental"
fi
}
set_options () {
# List the xtrabackup arguments
xtrabackup_args=(
"--defaults-file=${defaults_file}"
"--backup"
"--extra-lsndir=${working_dir}"
"--compress"
"--stream=xbstream"
"--encrypt=AES256"
"--encrypt-key-file=${encryption_key_file}"
"--parallel=${processors}"
"--compress-threads=${processors}"
"--encrypt-threads=${processors}"
"--slave-info"
)
set_backup_type
# Add option to read LSN (log sequence number) if taking an incremental backup
if [ "$backup_type" == "incremental" ]; then
lsn=$(awk '/to_lsn/ {print $3;}' "${working_dir}/xtrabackup_checkpoints")
xtrabackup_args+=( "--incremental-lsn=${lsn}" )
fi
}
rotate_old () {
# Remove previous backup artifacts
find "${working_dir}" -name "*.xbstream" -type f -delete
# Remove any backups from object storage older than 30 days
/usr/local/bin/object_storage.py prune --days-to-keep "${days_to_keep}"
}
take_backup () {
find "${working_dir}" -type f -name "*.incomplete" -delete
xtrabackup "${xtrabackup_args[@]}" --target-dir="${working_dir}" > "${working_dir}/${backup_type}-${now_string}.xbstream.incomplete" 2> "${log_file}"
mv "${working_dir}/${backup_type}-${now_string}.xbstream.incomplete" "${working_dir}/${backup_type}-${now_string}.xbstream"
}
upload_backup () {
/usr/local/bin/object_storage.py upload "${working_dir}/${backup_type}-${now_string}.xbstream"
}
main () {
mkdir -p "${working_dir}"
sanity_check && set_options && rotate_old && take_backup && upload_backup
# Check success and print message
if tail -1 "${log_file}" | grep -q "completed OK"; then
printf "Backup successful!\n"
printf "Backup created at %s/%s-%s.xbstream\n" "${working_dir}" "${backup_type}" "${now_string}"
else
error "Backup failure! If available, check ${log_file} for more information"
fi
}
main
This script handles the actual MySQL backup procedure, controls the backup schedule, and automatically removes older backups from remote storage. You can choose how many days of backups you’d like to keep on-hand by adjusting the days_to_keep
variable.
The local backup-mysql.sh
script we used in the last article maintained separate directories for each day’s backups. Since we are storing backups remotely, we will only store the latest backup locally in order to minimize the disk space devoted to backups. Previous backups can be downloaded from object storage as needed for restoration.
As with the previous script, after checking that a few basic requirements are satisfied and configuring the type of backup that should be taken, we encrypt and compress each backup into a single file archive. The previous backup file is removed from the local filesystem and any remote backups that are older than the value defined in days_to_keep
are removed.
Save and close the file when you are finished. Afterwards, ensure that the script is executable by typing:
- sudo chmod +x /usr/local/bin/remote-backup-mysql.sh
This script can be used as a replacement for the backup-mysql.sh
script on this system to switch from making local backups to remote backups.
Finally, download or create the download-day.sh
script within the /usr/local/bin
directory. This script can be used to download all of the backups associated with a particular day.
Create the script file in your text editor if you did not download it earlier:
- sudo nano /usr/local/bin/download-day.sh
Inside, paste the following contents:
#!/bin/bash
export LC_ALL=C
backup_owner="backup"
storage_configuration_file="/backups/mysql/object_storage_config.sh"
day_to_download="${1}"
# Use this to echo to standard error
error () {
printf "%s: %s\n" "$(basename "${BASH_SOURCE}")" "${1}" >&2
exit 1
}
trap 'error "An unexpected error occurred."' ERR
sanity_check () {
# Check user running the script
if [ "$(id --user --name)" != "$backup_owner" ]; then
error "Script can only be run as the \"$backup_owner\" user"
fi
# Check whether the object storage configuration file is available
if [ ! -r "${storage_configuration_file}" ]; then
error "Cannot read object storage configuration from ${storage_configuration_file}"
fi
# Check whether the object storage configuration is set in the file
source "${storage_configuration_file}"
if [ -z "${MYACCESSKEY}" ] || [ -z "${MYSECRETKEY}" ] || [ -z "${MYBUCKETNAME}" ]; then
error "Object storage configuration are not set properly in ${storage_configuration_file}"
fi
}
main () {
sanity_check
/usr/local/bin/object_storage.py get_day "${day_to_download}"
}
main
This script can be called to download all of the archives from a specific day. Since each day starts with a full backup and accumulates incremental backups throughout the rest of the day, this will download all of the relevant files necessary to restore to any hourly snapshot.
The script takes a single argument which is a date or day. It uses the Python’s dateutil.parser.parse
function to read and interpret a date string provided as an argument. The function is fairly flexible and can interpret dates in a variety of formats, including relative strings like “Friday”, for example. To avoid ambiguity however, it is best to use more well-defined dates. Be sure to wrap dates in quotations if the format you wish to use contains whitespace.
When you are ready to continue, save and close the file. Make the script executable by typing:
- sudo chmod +x /usr/local/bin/download-day.sh
We now have the ability to download the backup files from object storage for a specific date when we want to restore.
Now that we have our scripts in place, we should test to make sure they function as expected.
Begin by calling the remote-mysql-backup.sh
script with the backup
user. Since this is the first time we are running this command, it should create a full backup of our MySQL database.
- sudo -u backup remote-backup-mysql.sh
Note: If you receive an error indicating that the bucket name you selected is already in use, you will have to select a different name. Change the value of MYBUCKETNAME
in the /backups/mysql/object_storage_config.sh
file and delete the local backup directory (sudo rm -rf /backups/mysql/working
) so that the script can attempt a full backup with the new bucket name. When you are ready, rerun the command above to try again.
If everything goes well, you will see output similar to the following:
OutputUploaded /backups/mysql/working/full-10-17-2017_19-09-30.xbstream to "your_bucket_name"
Backup successful!
Backup created at /backups/mysql/working/full-10-17-2017_19-09-30.xbstream
This indicates that a full backup has been created within the /backups/mysql/working
directory. It has also been uploaded to remote object storage using the bucket defined in the object_storage_config.sh
file.
If we look within the /backups/mysql/working
directory, we can see files similar to those produced by the backup-mysql.sh
script from the last guide:
- ls /backups/mysql/working
Outputbackup-progress.log full-10-17-2017_19-09-30.xbstream xtrabackup_checkpoints xtrabackup_info
The backup-progress.log
file contains the output from the xtrabackup
command, while xtrabackup_checkpoints
and xtrabackup_info
contain information about options used, the type and scope of the backup, and other metadata.
Let’s make a small change to our equipment
table in order to create additional data not found in our first backup. We can enter a new row in the table by typing:
- mysql -u root -p -e 'INSERT INTO playground.equipment (type, quant, color) VALUES ("sandbox", 4, "brown");'
Enter your database’s administrative password to add the new record.
Now, we can take an additional backup. When we call the script again, an incremental backup should be created as long as it is still the same day as the previous backup (according to the server’s clock):
- sudo -u backup remote-backup-mysql.sh
OutputUploaded /backups/mysql/working/incremental-10-17-2017_19-19-20.xbstream to "your_bucket_name"
Backup successful!
Backup created at /backups/mysql/working/incremental-10-17-2017_19-19-20.xbstream
The above output indicates that the backup was created within the same directory locally, and was again uploaded to object storage. If we check the /backups/mysql/working
directory, we will find that the new backup is present and that the previous backup has been removed:
- ls /backups/mysql/working
Outputbackup-progress.log incremental-10-17-2017_19-19-20.xbstream xtrabackup_checkpoints xtrabackup_info
Since our files are uploaded remotely, deleting the local copy helps reduce the amount of disk space used.
Since our backups are stored remotely, we will need to pull down the remote files if we need to restore our files. To do this, we can use the download-day.sh
script.
Begin by creating and then moving into a directory that the backup
user can safely write to:
- sudo -u backup mkdir /tmp/backup_archives
- cd /tmp/backup_archives
Next, call the download-day.sh
script as the backup
user. Pass in the day of the archives you’d like to download. The date format is fairly flexible, but it is best to try to be unambiguous:
- sudo -u backup download-day.sh "Oct. 17"
If there are archives that match the date you provided, they will be downloaded to the current directory:
OutputLooking for objects from Tuesday, Oct. 17 2017
Downloading "full-10-17-2017_19-09-30.xbstream" from your_bucket_name
Downloading "incremental-10-17-2017_19-19-20.xbstream" from your_bucket_name
Verify that the files have been downloaded to the local filesystem:
- ls
Outputfull-10-17-2017_19-09-30.xbstream incremental-10-17-2017_19-19-20.xbstream
The compressed, encrypted archives are now back on the server again.
Once the files are collected, we can process them the same way we processed local backups.
First, pass the .xbstream
files to the extract-mysql.sh
script using the backup
user:
- sudo -u backup extract-mysql.sh *.xbstream
This will decrypt and decompress the archives into a directory called restore
. Enter that directory and prepare the files with the prepare-mysql.sh
script:
- cd restore
- sudo -u backup prepare-mysql.sh
OutputBackup looks to be fully prepared. Please check the "prepare-progress.log" file
to verify before continuing.
If everything looks correct, you can apply the restored files.
First, stop MySQL and move or remove the contents of the MySQL data directory:
sudo systemctl stop mysql
sudo mv /var/lib/mysql/ /tmp/
Then, recreate the data directory and copy the backup files:
sudo mkdir /var/lib/mysql
sudo xtrabackup --copy-back --target-dir=/tmp/backup_archives/restore/full-10-17-2017_19-09-30
Afterward the files are copied, adjust the permissions and restart the service:
sudo chown -R mysql:mysql /var/lib/mysql
sudo find /var/lib/mysql -type d -exec chmod 750 {} \;
sudo systemctl start mysql
The full backup in the /tmp/backup_archives/restore
directory should now be prepared. We can follow the instructions in the output to restore the MySQL data on our system.
Before we restore the backup data, we need to move the current data out of the way.
Start by shutting down MySQL to avoid corrupting the database or crashing the service when we replace its data files.
- sudo systemctl stop mysql
Next, we can move the current data directory to the /tmp
directory. This way, we can easily move it back if the restore has problems. Since we moved the files to /tmp/mysql
in the last article, we can move the files to /tmp/mysql-remote
this time:
- sudo mv /var/lib/mysql/ /tmp/mysql-remote
Next, recreate an empty /var/lib/mysql
directory:
- sudo mkdir /var/lib/mysql
Now, we can type the xtrabackup
restore command that the prepare-mysql.sh
command provided to copy the backup files into the /var/lib/mysql
directory:
- sudo xtrabackup --copy-back --target-dir=/tmp/backup_archives/restore/full-10-17-2017_19-09-30
Once the process completes, modify the directory permissions and ownership to ensure that the MySQL process has access:
- sudo chown -R mysql:mysql /var/lib/mysql
- sudo find /var/lib/mysql -type d -exec chmod 750 {} \;
When this finishes, start MySQL again and check that our data has been properly restored:
- sudo systemctl start mysql
- mysql -u root -p -e 'SELECT * FROM playground.equipment;'
Output+----+---------+-------+--------+
| id | type | quant | color |
+----+---------+-------+--------+
| 1 | slide | 2 | blue |
| 2 | swing | 10 | yellow |
| 3 | sandbox | 4 | brown |
+----+---------+-------+--------+
The data is available, which indicates that it has been successfully restored.
After restoring your data, it is important to go back and delete the restore directory. Future incremental backups cannot be applied to the full backup once it has been prepared, so we should remove it. Furthermore, the backup directories should not be left unencrypted on disk for security reasons:
- cd ~
- sudo rm -rf /tmp/backup_archives/restore
The next time we need clean copies of the backup directories, we can extract them again from the backup archive files.
We created a cron
job to automatically backup up our database locally in the last guide. We will set up a new cron
job to take remote backups and then disable the local backup job. We can easily switch between local and remote backups as necessary by enabling or disabling the cron
scripts.
To start, create a file called remote-backup-mysql
in the /etc/cron.hourly
directory:
- sudo nano /etc/cron.hourly/remote-backup-mysql
Inside, we will call our remote-backup-mysql.sh
script with the backup
user through the systemd-cat
command, which allows us to log the output to journald
:
#!/bin/bash
sudo -u backup systemd-cat --identifier=remote-backup-mysql /usr/local/bin/remote-backup-mysql.sh
Save and close the file when you are finished.
We will enable our new cron
job and disable the old one by manipulating the executable
permission bit on both files:
- sudo chmod -x /etc/cron.hourly/backup-mysql
- sudo chmod +x /etc/cron.hourly/remote-backup-mysql
Test the new remote backup job by executing the script manually:
- sudo /etc/cron.hourly/remote-backup-mysql
Once the prompt returns, we can check the log entries with journalctl
:
- sudo journalctl -t remote-backup-mysql
[seconary_label Output]
-- Logs begin at Tue 2017-10-17 14:28:01 UTC, end at Tue 2017-10-17 20:11:03 UTC. --
Oct 17 20:07:17 myserver remote-backup-mysql[31422]: Uploaded /backups/mysql/working/incremental-10-17-2017_22-16-09.xbstream to "your_bucket_name"
Oct 17 20:07:17 myserver remote-backup-mysql[31422]: Backup successful!
Oct 17 20:07:17 myserver remote-backup-mysql[31422]: Backup created at /backups/mysql/working/incremental-10-17-2017_20-07-13.xbstream
Check back in a few hours to make sure that additional backups are being taken on schedule.
One final consideration that you will have to handle is how to back up the encryption key (found at /backups/mysql/encryption_key
).
The encryption key is required to restore any of the files backed up using this process, but storing the encryption key in the same location as the database files eliminates the protection provided by encryption. Because of this, it is important to keep a copy of the encryption key in a separate location so that you can still use the backup archives if your database server fails or needs to be rebuilt.
While a complete backup solution for non-database files is outside the scope of this article, you can copy the key to your local computer for safekeeping. To do so, view the contents of the file by typing:
- sudo less /backups/mysql/encryption_key
Open a text file on your local computer and paste the value inside. If you ever need to restore backups onto a different server, copy the contents of the file to /backups/mysql/encryption_key
on the new machine, set up the system outlined in this guide, and then restore using the provided scripts.
In this guide, we’ve covered how take hourly backups of a MySQL database and upload them automatically to a remote object storage space. The system will take a full backup every morning and then hourly incremental backups afterwards to provide the ability to restore to any hourly checkpoint. Each time the backup script runs, it checks for backups in object storage that are older than 30 days and removes them.
]]>DigitalOcean Cloud Firewalls provide a powerful firewall service at the network level, protecting your resources from unauthorized traffic.
Although you can configure Cloud Firewalls through the DigitalOcean Control Panel, when you have many Droplets to manage, need to script a process, or prefer working from the terminal, a command-line interface can be a better choice.
In this tutorial we’ll learn how to use doctl
—the official DigitalOcean Command-Line Client—to create and manage Cloud Firewalls for a web server.
For this tutorial, you will need:
doctl
version 1.7.0 installed and authenticated by following the official installation instructions in the doctl
GitHub repository. (Use the doctl version
command to verify which version of doctl
you’re running.)
An SSH key added to your DigitalOcean account by following the How To Use SSH Keys with DigitalOcean Droplets tutorial.
We’ll be creating a one-click LAMP (Linux, Apache, MySQL, PHP) stack image running Ubuntu 16.04, in the nyc1 region, and we’ll put this on a 512MB Droplet. Before beginning this tutorial, though, we recommend that you familiarize yourself with doctl
and Cloud Firewalls by reading How To Use Doctl, the Official DigitalOcean Command-Line Client and An Introduction To DigitalOcean Cloud Firewalls.
First, we’ll choose a region for our Droplet. We’ll be using nyc1 in this tutorial, but you can see all of the regions and their slugs with the following command:
- doctl compute region list
OutputSlug Name Available
nyc1 New York 1 true
sfo1 San Francisco 1 true
ams2 Amsterdam 2 true
sgp1 Singapore 1 true
lon1 London 1 true
nyc3 New York 3 true
ams3 Amsterdam 3 true
fra1 Frankfurt 1 true
tor1 Toronto 1 true
sfo2 San Francisco 2 true
blr1 Bangalore 1 true
Since we don’t want to send passwords over the network and we want to reduce the possibility of a brute-force attack, we’ll secure our web server with SSH key authentication.
To create a Droplet that includes an SSH key, doctl
requires the SSH key fingerprint, which you can obtain with the command:
- doctl compute ssh-key list
OutputID Name FingerPrint
9763174 sammy_rsa your_ssh_key_fingerprint
Copy the fingerprint of the SSH key you want to use with your Droplet.
Now, let’s bring everything together in a single command that will create a 512MB Droplet named web-1 in the nyc1 region, using a one-click LAMP stack image running Ubuntu 16.04 with our SSH key.
- doctl compute droplet create web-1 \
- --region nyc1 \
- --image lamp-16-04 \
- --ssh-keys your_ssh_key_fingerprint \
- --size 512mb
The output gives us an overview of the Droplet we just created, including the Droplet’s ID, name, IPv4 address, Memory, and more:
OutputID Name Public IPv4 Private IPv4 Public IPv6 Memory VCPUs Disk Region Image Status Tags
52059458 web-1 512 1 20 nyc1 Ubuntu LAMP on 16.04 new
Note: You will need to wait a few minutes for the provisioning process to complete. Once provisioned, the Droplet will have an IPv4 address and a status of active
instead of new
.
Use the following command to check your Droplet’s status, and, if it’s fully provisioned, make note of the ID as we’ll need it when assigning the firewall to the Droplet in Step 2. Do not move past this step until your Droplet’s status reads active
.
- doctl compute droplet list web-1
OutputID Name Public IPv4 Private IPv4 Public IPv6 Memory VCPUs Disk Region Image Status Tags
52059458 web-1 203.0.113.1 512 1 20 nyc1 Ubuntu LAMP on 16.04 active
Next, use doctl
to log into the Droplet via SSH, enabling your LAMP installation and getting additional instructions about how to prepare your server for production use. If you get a connection refused
error message, your Droplet is not yet ready. Wait a few minutes and then re-run the list
command to verify that your Droplet’s status is set to active
before continuing.
- doctl compute ssh web-1
Output...
-------------------------------------------------------------------------------
Thank you for using DigitalOcean's LAMP Application.
LAMP has now been enabled. You can access your LAMP instance at:
Your web root is located at /var/www/html and can be seen from
http://203.0.113.1
...
After you configure the Droplet for your needs, exit the SSH session.
- [environment]
- exit
Finally, point your web browser to the Droplet’s IP address to make sure that the LAMP stack is working correctly. You should see the default DigitalOcean one-click LAMP stack landing page with the message: “Please log into your droplet via SSH to configure your LAMP installation.” If you don’t, re-trace the preceeding steps to ensure that you’ve enabled LAMP and that you’ve correctly copied your Droplet’s IP address into your browser.
Because we’ve already completed the LAMP configuration required for this tutorial, we’re ready to move on to protecting the Droplet from unauthorized traffic.
To begin, we’ll use the Droplet ID that we got from the doctl compute droplet list
command in Step 1 to create a Cloud Firewall named web-firewall
that allows inbound SSH connections on port 22
and all outbound TCP, UDP and ICMP connections. This will let us administer the server from the command line while still giving many fundamental services the ability to operate normally.
The protocol
field is required and must be set either to tcp
, udp
, or icmp
, and you must include a ports
value for all protocols except icmp
which, by its specification, doesn’t require one.
The address
field specifies which IP addresses are allowed to access a given port. If you want to allow traffic from all IPv4 addresses, use 0:0:0:0/0
, and if you want to allow traffic from all IPv6 addresses, use ::0/0
.
Lastly, each Firewall that you create must have at least one rule, either under the --inbound-rules
or --outbound-rules
flag, and all values must be entered as comma-separated key:value
lists. Use a quoted string of space-separated values for multiple rules.
Now, use the create
command create the firewall:
- doctl compute firewall create --name web-firewall \
- --droplet-ids your_droplet_id \
- --inbound-rules "protocol:tcp,ports:22,address:0.0.0.0/0,address:::/0" \
- --outbound-rules "protocol:icmp,address:0.0.0.0/0,address:::/0 protocol:tcp,ports:all,address:0.0.0.0/0,address:::/0 protocol:udp,ports:all,address:0.0.0.0/0,address:::/0"
The output contains a basic overview of the new Cloud Firewall. Make note of the Cloud Firewall’s ID, as you’ll use it in Step 3 to add additional rules to the Firewall.
OutputID Name Status Created At Inbound Rules Outbound Rules Droplet IDs Tags Pending Changes
c7b39b43-4fcc-4594-88f2-160a64aaddd4 web-firewall waiting 2017-06-17T21:20:38Z protocol:tcp,ports:22,address:0.0.0.0/0,address:::/0 protocol:icmp,ports:0,address:0.0.0.0/0,address:::/0 protocol:tcp,ports:0,address:0.0.0.0/0,address:::/0 protocol:udp,ports:0,address:0.0.0.0/0,address:::/0 your_droplet_id droplet_id:your_droplet_id,removing:false,status:waiting
If you ever need to specify a port range, use the following format:
--inbound-rules "protocol:tcp,ports:8000-8080,address:0.0.0.0/0,address:::/0"
You can also use the droplet_id
flag instead of the address
flag. This can be particularly useful in setups that involve multiple Droplets communicating with each other.
--inbound-rules "protocol:tcp,ports:8000-8080,droplet_id:your_droplet_id"
And, you can combine multiple address
or droplet_id
fields into a single rule, like:
--inbound-rules "protocol:tcp,ports:8000-8080,droplet_id:your_first_droplet_id,droplet_id:your_second_droplet_id"
At this point, confirm that the Cloud Firewall is working correctly by pointing your web browser to the Droplet’s IP address. You should see a message indicating that the site is no longer reachable. If you don’t, double-check the output from the previous create
command to make sure you didn’t miss any error messages.
Lastly, even though our inbound rule should already allow for SSH, we’ll verify it using doctl
.
- doctl compute ssh web-1
If you’re unable to connect to the Droplet, the How To Troubleshoot SSH tutorial series will help you diagnose the problem.
Once you’ve successfully connected to the Droplet, exit the SSH session:
- [environment]
- exit
As we’ve now verified that the Cloud Firewall is working correctly, we’ll add an additional rule to allow for incoming traffic to the web server.
Using the Firewall ID that we got from the doctl compute firewall create
command in Step 2, we are now going to add a rule to allow inbound TCP traffic for Apache on port 80
.
We’ll use the add-rules
command, which requires a Firewall ID and at least one rule. Rules are specified using --outbound-rules
and --inbound-rules
flags, just like in Step 2.
- doctl compute firewall add-rules c7b39b43-4fcc-4594-88f2-160a64aaddd4 \
- --inbound-rules "protocol:tcp,ports:80,address:0.0.0.0/0,address:::/0"
If you need HTTPS, allow inbound TCP traffic on port 443
.
- doctl compute firewall add-rules c7b39b43-4fcc-4594-88f2-160a64aaddd4 \
- --inbound-rules "protocol:tcp,ports:443,address:0.0.0.0/0,address:::/0"
If successful, this command will produce no output. If you receive an error message, follow the on-screen instructions to diagnose the problem.
Now, re-point your web browser to your Droplet’s IP address. This time you should see the default DigitalOcean one-click LAMP stack landing page again. If you don’t, double-check that you’ve correctly copied your IP address into your web browser and then re-trace the preceeding steps.
If you have additional web servers that you’d like to protect, continue on to Step 4. Otherwise, skip ahead to Step 5 where we’ll manage Cloud Firewalls with tags.
If you have multiple Droplets, you can apply the same Cloud Firewall to each of them.
Use the add-droplets
command to add additional Droplets to a Cloud Firewall. This command requires a Cloud Firewall ID as an argument, and it uses the droplet-ids
flag to determine which Droplets to apply the Firewall to.
If you don’t know the Cloud Firewall’s ID, use the list
command:
- doctl compute firewall list
OutputID Name Status Created At Inbound Rules Outbound Rules Droplet IDs Tags Pending Changes
c7b39b43-4fcc-4594-88f2-160a64aaddd4 web-firewall succeeded 2017-06-17T21:20:38Z protocol:tcp,ports:22,address:0.0.0.0/0,address:::/0 protocol:tcp,ports:80,address:0.0.0.0/0,address:::/0 protocol:icmp,ports:0,address:0.0.0.0/0,address:::/0 protocol:tcp,ports:0,address:0.0.0.0/0,address:::/0 protocol:udp,ports:0,address:0.0.0.0/0,address:::/0 52059458
You can also use the list
command to get Droplets’ IDs:
- doctl compute droplet list
OutputID Name Public IPv4 Private IPv4 Public IPv6 Memory VCPUs Disk Region Image Status Tags
51146959 test-1 203.0.113.1 512 1 20 nyc1 Ubuntu LAMP on 16.04 active
52059458 web-1 203.0.113.2 512 1 20 nyc1 Ubuntu LAMP on 16.04 active
Using the following doctl
command, we’ll add the test-1
Droplet to the web-servers
Firewall, which has an ID of c7b39b43-4fcc-4594-88f2-160a64aaddd4
:
- doctl compute firewall add-droplets c7b39b43-4fcc-4594-88f2-160a64aaddd4 \
- --droplet-ids 51146959
If you don’t receive any output, the command was successful. If you receive an error message, follow the on-screen instructions to diagnose the problem.
And, if you want to add multiple Droplets at once, separate them out using commas. Note that there are no spaces between two IDs:
--droplet-ids 51146959,52059458
Now, let’s use Tags for easier Cloud Firewall management.
At this point, we’ve added individual Droplets to the Cloud Firewall, but Cloud Firewalls also support Tags for easier management of multiple resources. To better understand how Tags work, see How To Tag DigitalOcean Droplets.
In this step, we’ll tag Droplets, add Tags to the Cloud Firewall, and then remove the individual Droplet IDs from the Firewall keeping the Droplets secure by way of Tags.
Before we can add a Tag to a Droplet using doctl
, we need to first create the Tag with the tag create
command:
- doctl compute tag create web-servers
OutputName Droplet Count
web-servers 0
Once the Tag is created, apply it to the Droplet using the droplet tag
command. This command takes the Droplet ID as an argument, and it gets the Tag name from the --tag-name
flag.
- doctl compute droplet tag 52059458 \
- --tag-name "web-servers"
If you want to secure multiple Droplets with one Cloud Firewall, repeat the previous command for each Droplet.
Next, add the Tag to the Cloud Firewall with the add-tags
command, which takes the Firewall ID as an argument and gets the list of Tag names to use from the --tag-names
flag:
- doctl compute firewall add-tags c7b39b43-4fcc-4594-88f2-160a64aaddd4 \
- --tag-names web-servers
If you don’t receive any output, the command was successful. If you receive an error message, follow the on-screen instructions to diagnose the problem.
And, if you need to add multiple Tags, provide them as a comma-separated list:
--tag-names web-servers,backend-servers
Finally, we can remove the Droplet’s ID from the Firewall, because the Droplet is part of the web-servers
Tag, and that entire Tag is now protected.
- doctl compute firewall remove-droplets c7b39b43-4fcc-4594-88f2-160a64aaddd4 \
- --droplet-ids 52059458
Repeat the previous step for each Droplet you want to secure by Tag only.
Warning: Removing non-tagged Droplets from the Cloud Firewall leaves the Droplets unprotected from unauthorized traffic.
You now have a fully configured Cloud Firewall which will protect your web server from unauthorized traffic. If you also want to delete a rule from the Firewall, continue on to Step 6.
If you want to remove a rule from a Cloud Firewall, use the remove-rules
command.
The remove-rules
command takes a Firewall ID as its argument, and rules are specified using the --outbound-rules
and --inbound-rules
flags. Note that the specified rule must be exactly the same as the rule that was used during creation.
- doctl compute firewall remove-rules c7b39b43-4fcc-4594-88f2-160a64aaddd4 \
- --inbound-rules protocol:tcp,ports:80,address:0.0.0.0/0,address:::/0
If you don’t receive any output, the command was successful. If you receive an error message, follow the on-screen instructions to diagnose the problem.
In this tutorial, we used doctl
to create DigitalOcean Cloud Firewalls, add rules to those Firewalls, add additional Droplets to the Firewalls, manage Firewalls with Tags, and remove rules from Firewalls.
To learn other ways to use Cloud Firewalls, see How To Organize DigitalOcean Cloud Firewalls.
And, to learn about troubleshooting Cloud Firewalls, visit How To Troubleshoot DigitalOcean Firewalls.
]]>DigitalOcean Load Balancers allow you to split incoming traffic between multiple backend servers. Often this is used to distribute HTTP requests among a group of application servers to increase overall capacity. This is a common way to scale your application.
Load Balancers also offer other use cases. For example, they can increase the reliability of your site, or improve your deployment and testing processes. In this tutorial, we will review five Load Balancer use cases.
Before we begin, you should familiarize yourself with the basics of DigitalOcean’s Load Balancers by reading our tutorial An Introduction to DigitalOcean Load Balancers.
As mentioned above, scaling traffic is the most common use case for a Load Balancer. Often times scaling is discussed in vertical and horizontal terms. Vertical scaling is basically moving your application to a more powerful server to meet increasing performance demands. Horizontal scaling is distributing your traffic among multiple servers to share the load. Load Balancers facilitate horizontal scaling.
DigitalOcean Load Balancers allow you to distribute load via two different algorithms: round robin and least connections. Round robin will send requests to each available backend server in turn, whereas least connections will send requests to the server with the fewest connections. Round robin is by far the most frequently used scheme for load balancing, but if you have an application that keeps connections open for a long time, least connections may do a better job of preventing any one server from becoming overloaded.
A side benefit of horizontal scaling with load balancers is the chance to increase your service’s reliability. We’ll talk about that next.
Related Tutorials:
High availability is a term that describes efforts to decrease downtime and increase system reliability. This is often addressed by improving performance and eliminating single points of failure.
A Load Balancer can increase availability by performing repeated health checks on your backend servers and automatically removing failed servers from the pool.
Health checks can be customized in the Settings area of the Load Balancer control panel:
By default, the Load Balancer will fetch a web page every ten seconds to make sure the server is responding properly. If this fails three times in a row, the server will be removed until the problem is resolved.
Related Tutorials:
Blue/green deployments refer to a technique where you deploy your new software on production infrastructure, test it thoroughly, then switch traffic over to it only after verifying that everything is working as you expect. If the deploy ends up failing in new and unexpected ways, you can easily recover by switching the Load Balancer back to the old version.
DigitalOcean Load Balancers make blue/green deployments simple through their use of the Droplet tagging feature. Load Balancers can send traffic to a group of servers based on their tag, so you can have one set of Droplets tagged blue and the other green. When it’s time to cut over, switch the tag in the Load Balancer control panel or through the API:
After you save your changes, traffic will quickly switch over to the new set of Droplets.
Related Tutorials:
Canary deployments are a way of testing a new version of your application on a subset of users before updating your entire pool of application servers. With DigitalOcean Load Balancers you could do this by, for instance, adding just one canary server to your Load Balancer’s pool. If you don’t see any increase in errors or other undesirable results through your logging and monitoring infrastructure, you can then proceed to deploy updates to the rest of the pool.
You’ll want to turn on sticky sessions for this use case, so that your users aren’t bounced between different versions of your application when making new connections through the Load Balancer:
Sticky sessions will use a cookie to ensure that future connections from a particular browser will continue to be routed to the same server. You can access this feature in the Advanced settings area of the Load Balancer’s control panel.
A/B deployments are functionally similar to canary deployments, but the purpose is different. A/B deployments test a new feature on a portion of your users in order to gather information that will inform your marketing and development efforts. You’ll need to do this in conjunction with your existing monitoring and logging infrastructure to get back meaningful results.
On the server side, you’ll add one or more B servers to your existing pool of A servers. If you need to launch multiple B servers to gather enough data, you could organize this with tags as we did for blue/green deployments.
Although Load Balancers are most often considered when scale is needed, we’ve shown that there are many other cases where it’s useful to have the ability to distribute or shuffle traffic among various backend servers. Whether it’s for high availability or leveraging various deployment techniques, Load Balancers are a flexible and powerful tool in your production infrastructure.
For more in-depth and specialized information on DigitalOcean Load Balancers, take a look at the following tutorials:
]]>SSH is the primary way to connect to remote Linux servers. Securing this channel is essential to maintaining a secure infrastructure. The most common way to authenticate to a remote server over SSH is to use public/private key pairs. Add the public key to the authorized keys list on the remote server and you’re ready to go.
The more difficult question is how to store your private key securely.
Typically, developers store their private keys in the ~/.ssh
directory. However, you can read your private key with a simple cat ~/.ssh/id_rsa
command. Any application on your machine can potentially read your SSH private key, even if it’s encrypted with a passphrase.
A common solution to this security risk is to add a second factor (i.e. enabling multi-factor authentication, or MFA). The downside to this is twofold: setup cost and usability. For every server you create, you have to configure the OpenSSH server to use the OATH-TOTP PAM module and load the shared secret on to it. This is a time-consuming process and there are a lot of places to make mistakes. Additionally, every time you SSH into a your server, you have to open an app on your phone, read a six digit code, and type it into your terminal. This can significantly slow down your workflow.
To avoid the drawbacks of configuring MFA, developers often use USB Hardware Security Modules (HSMs), like NitroKey or YubiKey, to generate and store SSH public-private key pairs. These are small USB devices that hold your SSH key pair. Every time you want to SSH into a server, you plug the USB device into your computer and press a button on the device.
But HSMs are expensive; SSH compatible devices cost as much as $50 USD. It’s yet another device to carry around and every time you SSH into a server, you have to plug a USB device into your computer and press a physical button on it. USB HSMs also typically do not have any display screen, so you do not know which login you’re actually approving and there’s no way to view an audit log of what you’ve authenticated to.
Kryptonite is a new solution to protecting your SSH private key. It’s free, easy to set up, user friendly, and has additional built-in security protections. It requires no server-side changes and lets you approve login requests via push notifications to your phone (without opening an app). Known hosts are always with you on your phone no matter which machine or server you’re SSHing to or from.
In this guide, you will generate an SSH key pair with Kryptonite on your phone, pair your phone with your local computer, and use Kryptonite to SSH into a DigitalOcean Droplet.
To follow this guide, you will need:
The first step is to download the Kryptonite app by going to get.krypt.co on your iOS or Android phone.
Once the app is installed, open it and tap Generate Key Pair to create your Kryptonite SSH key pair. Enter an email if you want to identify your public key with it (or skip this step).
Next, you’ll need to install Kryptonite’s command-line utility.
kr
The next step continues on your local computer. You’ll need to install the kr
command line utility, which enables SSH to authenticate with a key stored in Kryptonite. You can install kr
with your preferred package manager (like npm
or brew
) or simply use curl
, as we’ll do here.
For security reasons, if you want to inspect the installation script before installing, you can run curl https://krypt.co/kr > install_kr
and take a look. You can read more about how it works and alternative ways to install in the kr documentation.
When you’re ready, install kr
.
- curl https://krypt.co/kr | sh
You will be asked to enable push notifications. This is necessary for Kryptonite to send login approval requests via push notifications.
Now that you have the application, a key pair, and kr
, the next step is to pair your computer with Kryptonite.
After kr
is successfully installed, run:
- kr pair
A QR code will appear in the terminal. If your terminal window is small you may have to make it bigger so that the whole QR code is visible or make the font-size smaller.
In the Kryptonite app, tap Allow Camera Access on the bottom of the screen. Once the camera appears, scan the QR code in the terminal. After a few seconds, the Kryptonite app will show a successful pairing and the terminal will print out your Kryptonite SSH public key.
Let’s test that this key pair works.
To check that everything works, try SSHing into the public me.krypt.co
server:
- ssh me.krypt.co
You will notice a request appear on the Kryptonite app asking you to approve an SSH authentication with three options:
me.krypt.co
.Tap Allow Once. You will see a successful SSH login to me.krypt.co
, which will quickly exits from the pseudo-shell and show the shield logo.
If you lock your device and try to SSH into me.krypt.co
again, it will send a push notification to your device with the intended command, asking for your approval from the lock screen.
Now that Kryptonite is paired with your computer, you can quickly add your public key to all the servers and tools you use over SSH.
To add your public key to DigitalOcean, run the following command:
- kr digitalocean
You’ll see output with instructions specific to DigitalOcean, like this:
OutputPublic key copied to clipboard.
Press ENTER to open your web browser to DigitalOcean.
Then click “Add SSH Key” and paste your public key.
This is what you’ll need to do next:
ENTER
from your terminal to automatically navigate to your DigitalOcean settings page, logging in if necessary.You can find detailed instructions on adding your SSH key in Step 3 of this SSH on DigitalOcean tutorial.
Uploading your key to DigitalOcean makes it easy to add it to a new Droplet. Just select the box for your Kryptonite key when you create the server. Next, let’s add this key to an existing Droplet.
The kr
command line tool can be used to add your Kryptonite public key to an already running Droplet that you have access to with a local SSH key or a password.
Run the following command to add your Kryptonite public key to the Droplet’s authorized users file, making sure to substitute in your username and the IP address of your Droplet.
- kr add user@your_server_ip
Once you’ve done this, test that it works by trying to SSH in.
- ssh user@your_server_ip
You’ll get a Kryptonite SSH login request on your phone.
Now that you have set up Kryptonite and successfully added your Kryptonite public key to your DigitalOcean account, you can now SSH into any of your Droplets from any paired computer.
Your private key is securely stored on your phone and never leaves your device. When you allow a request, the private key is used to cryptographically sign an SSH login nonce locally on your device. This signature is then sent back to your computer to complete the SSH authentication.
For more information about how Kryptonite works, take a look at Kryptonite’s system architecture blog post and the Kryptonite and kr
source code.
DigitalOcean Cloud Firewalls provide a powerful firewall service at the network level, leaving your servers free to do their job of serving your applications and storing your data. In this tutorial, we will adapt a two-server Wordpress and MySQL setup to use Cloud Firewalls, and demonstrate some of the advantages this service can provide. If you’d like more background on this firewall service before beginning, please read our Introduction To DigitalOcean Cloud Firewalls tutorial.
Before starting this tutorial, you’ll need to have created the infrastructure outlined in How To Set Up a Remote Database to Optimize Site Performance with MySQL on Ubuntu 16.04. This will leave you with two servers, an Nginx web server with PHP and WordPress installed, and a standalone MySQL server. Throughout this tutorial we will call these servers frontend-01 and database-01 respectively.
Right now, both of our servers have firewalls set up using the ufw
utility. ufw
is an easy-to-use wrapper around Linux’s iptables firewall engine. Log in to both servers now and let’s check the status of our firewalls:
First, on the web server, frontend-01:
- sudo ufw status verbose
OutputStatus: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip
To Action From
-- ------ ----
22/tcp (OpenSSH) ALLOW IN Anywhere
80,443/tcp (Nginx Full) ALLOW IN Anywhere
22/tcp (OpenSSH (v6)) ALLOW IN Anywhere (v6)
80,443/tcp (Nginx Full (v6)) ALLOW IN Anywhere (v6)
In the output, after Default:
we are shown that the firewall is, by default, denying all incoming connections and allowing all outgoing connections. Additionally we have four rules that allow incoming IPv4 and IPv6 TCP connections (ALLOW IN
) to ports 22 (SSH), 80 (HTTP), and 443 (HTTPS).
Let’s do the same thing on the database server, database-01:
- sudo ufw status verbose
OutputStatus: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip
To Action From
-- ------ ----
22/tcp (OpenSSH) ALLOW IN Anywhere
3306 ALLOW IN Anywhere
22/tcp (OpenSSH (v6)) ALLOW IN Anywhere (v6)
3306 (v6) ALLOW IN Anywhere (v6)
This output is similar, except we’ve swapped the two Nginx ports for port 3306, which is the standard MySQL port. Now that we know our current setup, let’s plan our replacement.
Although we could just make two Cloud Firewalls, one tailored for each specific server, and apply one to frontend-01 and the other to database-01, we’re going take a more flexible approach to how we organize our rules.
First, we want to leave ourselves prepared for a future where we may need to add a third type of service to this system (perhaps a cache server). So we’re going to split up our firewall rules based on roles, not by physical server. We can apply multiple Cloud Firewalls to each Droplet, so it’s not a problem to make these firewalls fine-grained and modular.
Note: If you would like a more in-depth exploration of best-practices regarding structuring your Cloud Firewalls, please read How To Organize DigitalOcean Cloud Firewalls.
If we break things down a bit, we notice that both of our servers actually have multiple functions. There’s the primary function of either serving web pages or database information, and there’s also a management function provided by the SSH service. It would make good sense for us to create a management firewall, a frontend firewall, and a database firewall.
To handle the future scenario where we scale our web or database services to multiple hosts, we’ll use DigitalOcean’s tagging feature to organize our Droplets by role. Tags are simple labels we can apply to Droplets to categorize them and address whole groups of servers at once. The Cloud Firewall service can apply firewall rules to all Droplets in a tag, making it easy to provision new Droplets with the correct firewall rules already in place.
An additional bonus – and something that would be difficult do in a dynamic way using ufw
– is that Cloud Firewalls can restrict inbound access based on tags. So for instance, our database servers only need to be accessible from our frontend servers. The current ufw
setup has the database open to anybody on the network. We’ll lock that down to only our Droplets tagged with frontend.
Let’s summarize the three firewalls we need to set up, in plain language:
We’re not going to restrict outbound traffic at all in this tutorial. It’s not a bad idea, but it does take some care to make sure you don’t break auto-update mechanisms and other critical features of the underlying operating system.
Now that we have a plan for our new firewalls, let’s get started.
First, we’ll tag our Droplets by role, in preparation for our firewall rules. Navigate to the DigitalOcean Control Panel. The default view is a list of your Droplets. Click on the More button to the right of your frontend-01 Droplet, and select Add tags:
A text box will pop up where you can enter tags for this Droplet. Enter frontend and click the Add Tags button:
Do the same for your database server, giving it a database tag. The tags will show up in your Droplet list:
When creating future Droplets, you could apply these tags during the initial provisioning process. The Droplets will then automatically inherit the corresponding firewall rules.
We’ll set up those rules in the next step.
We’re going to set up our Cloud Firewalls now. We’ll do the frontend firewall first, followed by database, then management. This order should result in no service disruptions for your website visitors, but we will temporarily lose the ability to make new SSH connections. This will not affect already established connections.
The Firewalls service is available under the Networking section on the DigitalOcean Control Panel. Once there, click the Firewalls tab, then click the Create Firewall button to get started.
On the Create Firewall page, we need to fill out a Name, configure our Inbound Rules, and select which Droplets to apply the firewall to. We will leave the Outbound Rules section as is.
We’re creating the frontend firewall first, so put frontend-fw in the Name field.
Note: We’ll add -fw to the end of our firewall names to disambiguate them. Though the Control Panel interface uses icons to differentiate between resource types, it could get confusing if you’re using the command line or API and have multiple frontend items, for instance.
Next, we need to delete the default SSH rule from the Inbound Rules section. We’ll break this rule out into the management firewall for flexibility. Use the Delete link on the right-hand side of the page to delete the SSH rule now.
Then, click on the New rule dropdown and select HTTP. This will autofill the correct protocol (TCP) and port (80), and by default allow traffic from all IPv4 and IPv6 addresses. This is what we want.
If you have HTTPS enabled, repeat the above process to create a second rule, selecting HTTPS this time. Your Inbound Rules section will end up like this:
Finally, in the Apply to Droplets field, start typing frontend then select the frontend tag when it is auto-suggested.
Click the Create Firewall button. The new firewall will be created and applied to any Droplet with the frontend tag. You will be returned to an updated firewall summary page showing your new firewall:
Now we’ll create the database firewall.
On the Firewalls page, click Create Firewall again. The process will be mostly the same as for our frontend firewall.
Type database-fw into the Name field.
In Inbound Rules, delete the default SSH rule. Then, create a new rule using the dropdown, selecting MySQL. A default MySQL rule will be created allowing access to port 3306 from all IPs. Delete All IPv4 and All IPv6 from the Sources field. We want only our frontend servers to be able to access the database. Start typing frontend into the Sources box, and select the frontend tag when it is auto-suggested. Now any Droplet with that tag applied will be allowed access to the database server. All other IPs are blocked.
Leave the Outbound Rules as is. Under Apply to Droplets, apply this firewall to the database tag, then click Create Firewall. Once again, you’ll be returned to the firewall summary page:
Note that both firewalls show that they are applied to one Droplet each. If you load your website, it should still load fine. Now let’s re-enabled management via SSH.
Click Create Firewall one last time. Add management-fw to the Name field.
The default SSH rule is all we need for this firewall. This will allow any IP to connect to port 22.
Alternately, you could change the Sources field of the SSH rule to a specific IP that you’ll be connecting from. For instance, if your office has a static IP, and you want to restrict SSH access to only connections from the office, put that IP in Sources, replacing All IPv4 and All IPv6. If your IP ever changes in the future, you’ll just have to update this one rule to restore management access, another advantage of planning ahead and making our rules modular.
Under Apply to Droplets, add both the frontend and database tags, then click Create Firewall. Let’s take a look at our final firewall summary:
At this point, our Cloud Firewall should be fully functional, but we also still have the host-based ufw
firewalls active. Let’s disable those, then test our connections.
We need to disable the ufw
firewall on both hosts. First, on frontend-01:
- sudo ufw disable
OutputFirewall stopped and disabled on system startup
Then on database-01:
- sudo ufw disable
OutputFirewall stopped and disabled on system startup
This stops the current firewall, flushes out all of the rules, and prevents the rules from being re-enabled on startup.
At this point, all of our connectivity should be restored. Try creating a new SSH session to one of your servers. Then load your website to verify that the web server is still connecting to the database and returning webpages to the browser.
Being able to connect to all of our services doesn’t actually prove that a firewall is functioning though. Let’s do a little more testing to verify that our firewalls are actually in place.
To test our firewalls, we’re going to log in to a third server, and use a utility called nmap
to scan our web and database servers. nmap
is a port scanner that will scan our hosts and tell us which ports are open, closed, or filtered.
Log in to another Ubuntu 16.04 server that’s in the same region as your frontend-01 and database-01 servers. Then install nmap
:
- sudo apt-get update
- sudo apt-get install nmap
Then, use nmap
to scan the web server’s public IP:
- nmap -Pn frontend-01_public_ip
OutputStarting Nmap 7.01 ( https://nmap.org ) at 2017-06-05 17:08 UTC
Nmap scan report for 203.0.113.11
Host is up (0.0022s latency).
Not shown: 997 filtered ports
PORT STATE SERVICE
22/tcp open ssh
80/tcp open http
443/tcp open https
Nmap done: 1 IP address (1 host up) scanned in 4.54 seconds
Note the output about filtered ports
. If the firewall was not functioning, these would show as closed ports
. Filtered means that nmap
can’t even connect to determine if the port is open or closed.
Note also that we see our SSH, HTTP, and HTTPS ports are open, as expected.
Next, we’ll scan the database server. Be sure to use the Droplet’s private IP if you’ve set it up that way, as that’s what the MySQL database will be listening on:
- nmap -Pn database-01_private_ip
OutputStarting Nmap 7.01 ( https://nmap.org ) at 2017-06-05 17:21 UTC
Nmap scan report for 198.51.100.20
Host is up (0.0024s latency).
Not shown: 999 filtered ports
PORT STATE SERVICE
22/tcp open ssh
Nmap done: 1 IP address (1 host up) scanned in 8.17 seconds
We see that most ports are filtered, as before. However, we’re only seeing the SSH port as open, with no MySQL port available. Recall that we restricted database access to only those servers tagged with frontend. Switch back over to the DigitalOcean Control Panel and add the frontend tag to the server you’re using nmap
from. Then rerun the command:
- nmap -Pn database-01_private_ip
OutputStarting Nmap 7.01 ( https://nmap.org ) at 2017-06-05 17:22 UTC
Nmap scan report for 198.51.100.20
Host is up (0.0033s latency).
Not shown: 998 filtered ports
PORT STATE SERVICE
22/tcp open ssh
3306/tcp open mysql
Nmap done: 1 IP address (1 host up) scanned in 4.46 seconds
The MySQL port now shows as open. We’ve verified that both of our servers are now protected by our Cloud Firewall rules. You can now restore this test server’s original firewall settings by returning to the Control Panel and removing the Droplet’s frontend tag.
In this tutorial we’ve replaced a ufw
firewall setup with a flexible and powerful network-based Cloud Firewall configuration. For more information on using Cloud Firewalls via doctl
or the DigitalOcean API, please see the following articles:
Packer, by Hashicorp, is a command-line tool for quickly creating identical machine images for multiple platforms and environments. With Packer, you use a configuration file, called a template, to create a machine image containing a preconfigured operating system and software. You can then use this image to create new machines. You can even use a single template to orchestrate the simultaneous creation of your production, staging, and development environments.
In this tutorial, you’ll use Packer to configure an Nginx web server on Ubuntu 16.04. You’ll then use Packer to create a snapshot of this Droplet and make it immediately available in your DigitalOcean dashboard so you can use it to create new Droplets.
Before you can get up and running with Packer you will need a few things.
After logging in to your server, you will download the Packer binary package, install Packer for the current user, and check that your installation was successful.
The easiest way to get Packer running on your system is to download the latest binary package from the official Hashicorp releases website. At the time of writing, the latest version is 0.12.2.
Use the curl
utility to download the binary package from the Hashicorp website.
- curl -O https://releases.hashicorp.com/packer/0.12.2/packer_0.12.2_linux_amd64.zip
Once downloaded, install the unzip
utility and use it to unzip the package contents into the /usr/local/bin
directory to make Packer available to all users:
- sudo apt install -y unzip
- sudo unzip -d /usr/local/bin packer_0.12.2_linux_amd64.zip
Verify that the installation was successful by checking that packer
is available on the command line:
- packer
A successful installation will output the following:
Outputusage: packer [--version] [--help] <command> [<args>]
Available commands are:
build build image(s) from template
fix fixes templates from old versions of packer
inspect see components of a template
push push a template and supporting files to a Packer build service
validate check that a template is valid
version Prints the Packer version
Packer is now installed and working on your machine. In the next step you will set up a project directory and configure the template to produce a basic Ubuntu snapshot.
We want Packer to create a Droplet, install some software and configuration files, and then turn that Droplet into an image we can use to create new machines. Packer uses a configuration file called a template that contains all the details that tell Packer how to build an image. We write this configuration using JSON, a common format for configuration files.
In Packer-speak, a builder is a JSON object that contains the blueprint for the image you want Packer to create. Using the digitalocean
builder, you are going to instruct Packer to create a 512 MB Ubuntu 16.04 Droplet that will be launched in the NYC1 region.
Create and change to a new directory which will hold the template and configuration files we’ll create in this tutorial:
- mkdir ~/packerProject
- cd ~/packerProject
Now that you have a project directory, open a new file called template.json
, in your text editor:
- nano ~/packerProject/template.json
Each builder needs to go into the builders
section of template.json
. Add this section now and include the digitalocean
builder by placing this code into the file:
{
"builders": [
{
"type": "digitalocean"
}]
}
The type
key defines which builder Packer uses to create your image. The digitalocean
builder creates DigitalOcean Droplets from which Packer creates snapshots.
Packer now knows that you want to create an image for DigitalOcean, but it still needs a few more key-value pairs to be able to complete the build.
Finish configuring your Droplet by adding these keys and values to produce a snapshot from a 512 MB Ubuntu 16.04 Droplet that is launched in the NYC1 region. Modify your file so it looks like this:
{
"builders": [
{
"type": "digitalocean",
"ssh_username": "root",
"api_token": "YOUR_DIGITALOCEAN_API_TOKEN",
"image": "ubuntu-16-04-x64",
"region": "nyc1",
"size": "512mb"
}]
}
Packer connects to Droplets using the ssh_username
value. This value needs to be set to “root” in order for Packer to work properly.
Save template.json
and exit your text editor.
The preceding block of code contains the minimum amount of configuration needed to create a DigitalOcean Droplet but there are additional configuration options available, as shown in the following table:
Key | Value | Required | Description |
---|---|---|---|
api_token |
String | Yes | The API token to use to access your account. It can also be specified via environment variable DIGITALOCEAN_API_TOKEN , if set. |
image |
String | Yes | The name (or slug) of the base image to use. This is the image that will be used to launch a new Droplet and provision it. See https://developers.digitalocean.com/documentation/v2/#list-all-images for details on how to get a list of the the accepted image names/slugs. |
region |
String | Yes | The name (or slug) of the region to launch the Droplet in. Consequently, this is the region where the snapshot will be available. See https://developers.digitalocean.com/documentation/v2/#list-all-regions for the accepted region names/slugs. |
size |
String | Yes | The name (or slug) of the Droplet size to use. See https://developers.digitalocean.com/documentation/v2/#list-all-sizes for the accepted size names/slugs. |
api_url |
String | No | The URL of a non-standard API endpoint. Set this if you are using a DigitalOcean API-compatible service. |
droplet_name |
String | No | The name assigned to the Droplet. DigitalOcean sets the hostname of the machine to this value. |
private_networking |
Boolean | No | Set to true to enable private networking for the Droplet being created. This defaults to false , or not enabled. |
snapshot_name |
String | No | The name of the resulting snapshot that will appear in your account. This must be unique. |
state_timeout |
String | No | The time to wait, as a duration string, for a Droplet to enter a desired state (such as “active”) before timing out. The default state timeout is “6m”. |
user_data |
String | No | User data to launch with the Droplet. See An Introduction to Droplet Metadata for more information. |
You now have a valid template, but your API token is hard-coded in your template. This is a bad practice and a potential security risk. In the next step you will create a variable for this token and move it out of template.json
.
Packer lets you create and store the values of variables in a separate file. This file can then be passed to Packer via the command line when you are ready to build your image.
Storing variables in a separate file is an ideal way to keep sensitive information or environment-specific data out of your template. This is crucial if you intend to share it with team members or store it in a public facing repository such as GitHub.
Even if you will only be saving a local copy, it’s a Packer best practice to store variables outside of a template.
Create and open a new JSON file in the packerProject
directory to store this information:
- nano ~/packerProject/variables.json
Now, add a my_token
variable and set its value to your DigitalOcean API token:
{
"my_token": "YOUR_DIGITALOCEAN_API_TOKEN"
}
Save variables.json
and exit your editor.
Now let’s configure our template to use variables. Before you use the my_token
variable, or any other variable, you first need to tell Packer the variable exists by defining it in a variables
section at the beginning of the template.json
file.
Open template.json
in your editor:
- nano template.json
Add a new variables
section above the builders
section you previously defined. Within this new section, declare the my_token
variable and set its default value to an empty string:
{
"variables": {
"my_token":""
},
"builders": [
...
}
Variables defined in the variables
section are available globally.
Next, replace your API token in the builders
section with a call to my_token
:
{
...
"builders": [
{
"type": "digitalocean",
"api_token": "{{ user `my_token` }}",
...
}]
}
As you can see, calls to user variables must use a specific format: "{{ user `variable_name` }}
. The quotes and the backticks are required, as are the double curly braces.
Save the file and exit the editor.
You now have a working template that produces a basic snapshot and a separate variables file to store your API key. Before you validate and build your image, let’s add a provisioners
section to our template which will configure Packer to install and set up the Nginx web server on the machine before creating the image.
The provisioners
section is where Packer installs and configures software on the running Droplet before turning it into a machine image. Like builders, there are different types of provisioners you can use to configure a Droplet.
In order to configure Nginx, you are going to use Packer’s file
provisioner to upload configuration files to the server, and then use the shell
provisioner to execute an installation script that uses those files. The file
provisioner lets you move files and directories to and from a running machine before it is turned into an image. With the shell
provisioner, you can remotely execute shell scripts on that machine.
Provisioners execute in the same order in which they appear within the template. This means putting the file
provisioner first since your shell scripts need the uploaded files.
Add a provisioners
section immediately following the builders
section in template.json
and set the two provisioners you will use:
{
...
"builders": [
{
...
}],
"provisioners": [
{
"type": "file"
},
{
"type": "shell"
}]
}
The file
provisioner requires a source
, which points to a local file path, and a destination
, which points to an existing file path on the running machine. Packer can only move files to destinations that already exist. For this reason, we generally upload files to the /tmp
directory.
Configure the file
provisioner by adding the highlighted lines to template.json
:
{
...
"provisioners": [
{
"type": "file",
"source": "configs/",
"destination": "/tmp"
},
...
}
We’ll create the configs
folder on our local machine in the next step. Before we do, let’s finish editing the configuration file by setting up the shell
provisioner.
The shell
provisioner takes a scripts
key which contains array of scripts that should be passed to the running machine. Each script is uploaded and executed in the order specified in your template.
Now, configure the shell
provisioner by providing the full path to your script:
{
...
"provisioners": [
{
"type": "file",
"source": "configs/",
"destination": "/tmp"
},
{
"type": "shell",
"scripts": [
"scripts/configureNginx.sh"
]
}]
}
Scripts must be listed individually, which allows you to control the execution order of the scripts.
The provisioners
section of your template is complete. If you’re using nano, hold down the ctrl
key and press x
to exit the file, and press y
when prompted to save your changes.
Now let’s create the shell scripts and configuration files that Packer will use to create your image.
We want our image to ship with a fully-configured Nginx installation, with the proper configuration files and a default web page. In this section, you’ll create these files from some predefined configuration based on the tutorial How To Set Up Nginx Server Blocks (Virtual Hosts) on Ubuntu 16.04, as Nginx configuration is beyond the scope of this tutorial.
We’ll provision the server with Nginx by creating and uploading three separate configuration files that are handled by a single installation script.
First, create a new directory within the project folder to store the configuration files.
- mkdir ~/packerProject/configs
Change to /configs
to create your Nginx configuration files:
- cd ~/packerProject/configs
First, you need a default web page to serve from your new domain. Create the file index.html.new
:
- nano index.html.new
In this new file, insert the following:
HELLO FROM YOUR TEST PAGE
Next, you need an Nginx configuration file that defines the server block for your domain that, in turn, defines the listening port and the location of your web pages for the domain. Create a file called newDomain.conf
:
- nano newDomain.conf
Place the following configuration in this file:
server {
listen 80;
listen [::]:80;
server_name example.com;
location / {
root /var/www/html/newDomain;
index index.html index.htm;
}
}
In this example, we’re using example.com
as a placeholder value. When you create a new machine from your image, you’ll have to log in to the new machine and change this file to reflect the actual domain or IP address that points to the machine.
Finally, you want Nginx to load your domain’s configuration from a new directory, /etc/nginx/vhost.d/
. This means editing the main Nginx configuration file.
Create nginx.conf.new
:
- nano nginx.conf.new
We’ll use a default Nginx config file, but we’ll modify it to include our specific site configuration and ensure that Nginx runs as the www-data
user. Put the following contents into this file:
user www-data;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/vhost.d/*.conf;
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
root /usr/share/nginx/html;
include /etc/nginx/default.d/*.conf;
location / {
}
error_page 404 /404.html;
location = /40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
}
Save and exit the file.
With the configuration files in place, let’s define the scripts that Packer will use to install the software we need. Create a new folder to store your scripts:
- mkdir ~/packerProject/scripts
Now change to this new directory and create the installation script, configureNginx.sh
, which installs, configures, enables, and starts the Nginx web server:
- cd ~/packerProject/scripts
- nano configureNginx.sh
Paste the following into the file, which installs, configures, and starts Nginx, using the configuration files you just created:
#!/bin/bash
# Script to install Nginx and enable on boot.
# Update your system:
apt-get update -y
apt-get upgrade -y
# Install Nginx:
apt-get install -y nginx
#Start Nginx service and enable to start on boot:
systemctl enable nginx
systemctl start nginx
# Create new 'vhost' directory for domain configuration:
mkdir /etc/nginx/vhost.d
# Create a new directory to serve new content.
mkdir -p /var/www/html/newDomain
# Create a copy of original configuration files and import configuration:
cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.original
cp /tmp/nginx.conf.new /etc/nginx/nginx.conf
# Copy over the server block configuration:
cp /tmp/newDomain.conf /etc/nginx/vhost.d/newDomain.conf
# Copy over the html test page:
cp /tmp/index.html.new /var/www/html/newDomain/index.html
# Restart Nginx:
systemctl restart nginx
Your template is finished and you’re now ready to validate and build your snapshot.
It’s time to test your template using Packer’s validate
subcommand. Once your template validates successfully you will build your Droplet and create the snapshot.
Change to the root of your project:
- cd ~/packerProject
The validate
subcommand will check your template for valid syntax and configuration options:
- packer validate -var-file=variables.json template.json
The -var-file
flag reads variables.json
and sets the value for my_token
within template.json
.
You’ll see the following output:
OutputTemplate validated successfully.
If there is something wrong with template.json
you will get an error message. This message will vary depending on the error but most can be fixed by double checking syntax and correcting any typos.
The build
subcommand runs the build that you defined in the builders
section of your template. In other words, it tells Packer to build your Droplet and then create a snapshot of that Droplet in your DigitalOcean dashboard.
Call packer build
to build the Droplet and create the snapshot:
- packer build -var-file=variables.json template.json
Note that the -var-file
flag operates in the exact same manner for both the build
and validate
subcommands.
The output of a successful build will look similar to the following:
Outputdigitalocean output will be in this color.
==> digitalocean: Creating temporary ssh key for Droplet...
==> digitalocean: Creating Droplet...
==> digitalocean: Waiting for Droplet to become active...
==> digitalocean: Waiting for SSH to become available...
==> digitalocean: Connected to SSH!
==> digitalocean: Gracefully shutting down Droplet...
==> digitalocean: Creating snapshot: packer-1488487459
==> digitalocean: Waiting for snapshot to complete...
==> digitalocean: Destroying Droplet...
==> digitalocean: Deleting temporary ssh key...
Build 'digitalocean' finished.
==> Builds finished. The artifacts of successful builds are:
--> digitalocean: A snapshot was created: 'packer-1488487459' (ID: 18252043) in region 'nyc1'
After a successful build you will find a new snapshot in your DigitalOcean snapshots repository. You can find the name of the snapshot in the output. In this example, it’s packer-1488487459
.
From here, visit your DigitalOcean dashboard, select Images, and the new Snapshot will appear in your list:
You can now use this new snapshot to create new Droplets. Select More and choose Create Droplet. Then complete the form to create your new machine.
Once the machine is online, determine its IP address from your dashboard and log in to your new machine:
- ssh root@your_new_server_ip_address
Then edit the Nginx server configuration file:
- nano /etc/nginx/vhost.d/newDomain.conf
And replace example.com
with either the IP address of the machine or the domain name you’ll use:
server {
listen 80;
listen [::]:80;
server_name your_new_server_ip_address;
location / {
root /var/www/html/newDomain;
index index.html index.htm;
}
}
Alternatively, you can use the sed
command to replace the value in the file, like this:
- sudo sed -i 's/^.*server_name example.com/server_name your_new_server_ip_address/' /etc/nginx/vhost.d/newDomain.conf
You can learn more about sed
in this tutorial.
Then restart the Nginx server to apply the changes:
- sudo systemctl restart nginx
Occasionally, you may run into an issue that isn’t adequately explained by the error message. In these scenarios, you can extract more detail about your build by enabling debug mode, inspecting the Packer logs, or both.
Debug mode provides builder-specific debugging information for each step in a remote build. Enabling debug mode for a DigitalOcean build will also produce a temporary private key in your project folder which you can use to connect to and inspect a running Droplet before it is turned into a snapshot.
You can enter debug mode by passing the -debug
flag to packer build
on the command line:
- packer build -debug --var-file=variables.json template.json
If you are unable to diagnose the issue in debug mode you can try enabling the Packer logs. These logs are primarily used to debug local builders, but they may provide helpful information on remote builds as well.
To enable the Packer logs, set the PACKER_LOG
environmental variable to any value except “0” or an empty string:
- PACKER_LOG=1 packer build --var-file=variables.json template.json
Logs will print to the console unless you also set the PACKER_LOG_PATH
environmental variable.
If you are still having problems, you may want to try reaching out to someone in the Packer community.
Now that you are comfortable with the basics of Packer, you may be interested in building on this foundation.
Try adding a second builder to your template to create a local testing environment alongside your DigitalOcean snapshot. The virtualbox-iso
builder, for example, produces images for VirtualBox, a free, open-source virtualization product used by both enterprises and hobbyists. You can define a post-processor
to the VirtualBox image and create Vagrant environments that mirror your DigitalOcean snapshots. This will allow you to test website changes locally before pushing them to a live Droplet. You can learn more in the Vagrant post-processor documentation.
Or you may want to connect your web server to a database. Add a second digitalocean
builder and use the only
key in your provisioners
section to apply different provisioning to each build.
If you’re more comfortable using a configuration management tools, Packer comes with out of the box support for Ansible, Puppet, Chef, and more. Try using one of these provisioners to further configure your Droplet to match your use case. If you’ve never tried configuration management before, take a look at How To Create Ansible Playbooks to Automate System Configuration on Ubuntu.
]]>Packer, by Hashicorp, is a command-line tool for quickly creating identical machine images for multiple platforms and environments. With Packer, you use a configuration file, called a template, to create a machine image containing a preconfigured operating system and software. You can then use this image to create new machines. You can even use a single template to orchestrate the simultaneous creation of your production, staging, and development environments.
In this tutorial, you’ll use Packer to configure an Nginx web server on CentOS 7. You’ll then use Packer to create a snapshot of this Droplet and make it immediately available in your DigitalOcean dashboard so you can use it to create new Droplets.
Before you can get up and running with Packer you will need a few things.
After logging in to your server, you will download the Packer binary package, install Packer for the current user, and check that your installation was successful.
The easiest way to get Packer running on your system is to download the latest binary package from the official Hashicorp releases website. At the time of writing, the latest version is 0.12.2.
Use the curl
utility to download the binary package from the Hashicorp website.
- curl -O https://releases.hashicorp.com/packer/0.12.2/packer_0.12.2_linux_amd64.zip
Once downloaded, install the unzip
utility and use it to unzip the package contents into the /usr/local
directory, the recommended location to make Packer available to all users.
- sudo yum install -y unzip
- sudo unzip -d /usr/local packer_0.12.2_linux_amd64.zip
CentOS already includes a program called packer
, and while you could simply type out the full path each time you run a command, a more efficient method to work around this issue is to create a symlink that maps packer.io
to /usr/local/packer
. Make the symlink in the /usr/local/bin
folder with the following command:
- sudo ln -s /usr/local/packer /usr/local/bin/packer.io
Verify that the installation was successful by checking that packer.io
is available on the command line:
- packer.io
A successful installation will output the following:
Outputusage: packer [--version] [--help] <command> [<args>]
Available commands are:
build build image(s) from template
fix fixes templates from old versions of packer
inspect see components of a template
push push a template and supporting files to a Packer build service
validate check that a template is valid
version Prints the Packer version
Packer is now installed and working on your machine. In the next step you will set up a project directory and configure the template to produce a basic CentOS snapshot.
We want Packer to create a Droplet, install some software and configuration files, and then turn that Droplet into an image we can use to create new machines. Packer uses a configuration file called a template that contains all the details that tell Packer how to build an image. We write this configuration using JSON, a common format for configuration files.
In Packer-speak, a builder is a JSON object that contains the blueprint for the image you want Packer to create. Using the digitalocean
builder, you are going to instruct Packer to create a 512 MB CentOS 7.3 Droplet that will be launched in the NYC1 region.
Create and change to a new directory which will hold the template and configuration files we’ll create in this tutorial:
- mkdir ~/packerProject
- cd ~/packerProject
Now that you have a project directory, open a new file called template.json
, in your text editor:
- vi ~/packerProject/template.json
Each builder needs to go into the builders
section of template.json
. Add this section now and include the digitalocean
builder by placing this code into the file:
{
"builders": [
{
"type": "digitalocean"
}]
}
The type
key defines which builder Packer uses to create your image. The digitalocean
builder creates DigitalOcean Droplets from which Packer creates snapshots.
Packer now knows that you want to create an image for DigitalOcean, but it still needs a few more key-value pairs to be able to complete the build.
Finish configuring your Droplet by adding these keys and values to produce a snapshot from a 512 MB CentOS 7 Droplet that is launched in the NYC1 region. Modify your file so it looks like this:
{
"builders": [
{
"type": "digitalocean",
"ssh_username": "root",
"api_token": "YOUR_DIGITALOCEAN_API_TOKEN",
"image": "centos-7-x64",
"region": "nyc1",
"size": "512mb"
}]
}
Packer connects to Droplets using the ssh_username
value. This value needs to be set to “root” in order for Packer to work properly.
Save template.json
and exit your text editor.
The preceding block of code contains the minimum amount of configuration needed to create a DigitalOcean Droplet but there are additional configuration options available, as shown in the following table:
Key | Value | Required | Description |
---|---|---|---|
api_token |
String | Yes | The API token to use to access your account. It can also be specified via environment variable DIGITALOCEAN_API_TOKEN , if set. |
image |
String | Yes | The name (or slug) of the base image to use. This is the image that will be used to launch a new Droplet and provision it. See https://developers.digitalocean.com/documentation/v2/#list-all-images for details on how to get a list of the the accepted image names/slugs. |
region |
String | Yes | The name (or slug) of the region to launch the Droplet in. Consequently, this is the region where the snapshot will be available. See https://developers.digitalocean.com/documentation/v2/#list-all-regions for the accepted region names/slugs. |
size |
String | Yes | The name (or slug) of the Droplet size to use. See https://developers.digitalocean.com/documentation/v2/#list-all-sizes for the accepted size names/slugs. |
api_url |
String | No | The URL of a non-standard API endpoint. Set this if you are using a DigitalOcean API-compatible service. |
droplet_name |
String | No | The name assigned to the Droplet. DigitalOcean sets the hostname of the machine to this value. |
private_networking |
Boolean | No | Set to true to enable private networking for the Droplet being created. This defaults to false , or not enabled. |
snapshot_name |
String | No | The name of the resulting snapshot that will appear in your account. This must be unique. |
state_timeout |
String | No | The time to wait, as a duration string, for a Droplet to enter a desired state (such as “active”) before timing out. The default state timeout is “6m”. |
user_data |
String | No | User data to launch with the Droplet. See An Introduction to Droplet Metadata for more information. |
You now have a valid template, but your API token is hard-coded in your template. This is a bad practice and a potential security risk. In the next step you will create a variable for this token and move it out of template.json
.
Packer lets you create and store the values of variables in a separate file. This file can then be passed to Packer via the command line when you are ready to build your image.
Storing variables in a separate file is an ideal way to keep sensitive information or environment-specific data out of your template. This is crucial if you intend to share it with team members or store it in a public facing repository such as GitHub.
Even if you will only be saving a local copy, it’s a Packer best practice to store variables outside of a template.
Create and open a new JSON file in the packerProject
directory to store this information:
- vi ~/packerProject/variables.json
Now, add a my_token
variable and set its value to your DigitalOcean API token:
{
"my_token": "YOUR_DIGITALOCEAN_API_TOKEN"
}
Save variables.json
and exit your editor.
Now let’s configure our template to use variables. Before you use the my_token
variable, or any other variable, you first need to tell Packer the variable exists by defining it in a variables
section at the beginning of the template.json
file.
Open template.json
in your editor:
- vi template.json
Add a new variables
section above the builders
section you previously defined. Within this new section, declare the my_token
variable and set its default value to an empty string:
{
"variables": {
"my_token":""
},
"builders": [
...
}
Variables defined in the variables
section are available globally.
Next, replace your API token in the builders
section with a call to my_token
:
{
...
"builders": [
{
"type": "digitalocean",
"api_token": "{{ user `my_token` }}",
...
}]
}
As you can see, calls to user variables must use a specific format: "{{ user `variable_name` }}
. The quotes and the backticks are required, as are the double curly braces.
Save the file and exit the editor.
You now have a working template that produces a basic snapshot and a separate variables file to store your API key. Before you validate and build your image, let’s add a provisioners
section to our template which will configure Packer to install and set up the Nginx web server on the machine before creating the image.
The provisioners
section is where Packer installs and configures software on the running Droplet before turning it into a machine image. Like builders, there are different types of provisioners you can use to configure a Droplet.
In order to configure Nginx, you are going to use Packer’s file
provisioner to upload configuration files to the server, and then use the shell
provisioner to execute an installation script that uses those files. The file
provisioner lets you move files and directories to and from a running machine before it is turned into an image. With the shell
provisioner, you can remotely execute shell scripts on that machine.
Provisioners execute in the same order in which they appear within the template. This means putting the file
provisioner first since your shell scripts need the uploaded files.
Add a provisioners
section immediately following the builders
section in template.json
and set the two provisioners you will use:
{
...
"builders": [
{
...
}],
"provisioners": [
{
"type": "file"
},
{
"type": "shell"
}]
}
The file
provisioner requires a source
, which points to a local file path, and a destination
, which points to an existing file path on the running machine. Packer can only move files to destinations that already exist. For this reason, we generally upload files to the /tmp
directory.
Configure the file
provisioner by adding the highlighted lines to template.json
:
{
...
"provisioners": [
{
"type": "file",
"source": "configs/",
"destination": "/tmp"
},
...
}
We’ll create the configs
folder on our local machine in the next step. Before we do, let’s finish editing the configuration file by setting up the shell
provisioner.
The shell
provisioner takes a scripts
key which contains array of scripts that should be passed to the running machine. Each script is uploaded and executed in the order specified in your template.
Now, configure the shell
provisioner by providing the full path to your script:
{
...
"provisioners": [
{
"type": "file",
"source": "configs/",
"destination": "/tmp"
},
{
"type": "shell",
"scripts": [
"scripts/configureNginx.sh"
]
}]
}
Scripts must be listed individually, which allows you to control the execution order of the scripts.
The provisioners
section of your template is complete. Save the file and exit Vim.
Now let’s create the shell scripts and configuration files that Packer will use to create your image.
We want our image to ship with a fully-configured Nginx installation, with the proper configuration files and a default web page. In this section, you’ll create these files from some predefined configuration based on the tutorial How To Set Up Nginx Server Blocks (Virtual Hosts) on Ubuntu 16.04, as Nginx configuration is beyond the scope of this tutorial.
We’ll provision the server with Nginx by creating and uploading three separate configuration files that are handled by a single installation script.
First, create a new directory within the project folder to store the configuration files.
- mkdir ~/packerProject/configs
Change to /configs
to create your Nginx configuration files:
- cd ~/packerProject/configs
First, you need a default web page to serve from your new domain. Create the file index.html.new
:
- vi index.html.new
In this new file, insert the following:
HELLO FROM YOUR TEST PAGE
Next, you need an Nginx configuration file that defines the server block for your domain that defines the listening port and the location of your web pages for the domain. Create a file called newDomain.conf
:
- vi newDomain.conf
Place the following configuration in this file:
server {
listen 80;
listen [::]:80;
server_name example.com;
location / {
root /var/www/html/newDomain;
index index.html index.htm;
}
}
In this example, we’re using example.com
as a placeholder value. When you create a new machine from your image, you’ll have to log in to the new machine and change this file to reflect the actual domain or IP address that points to the machine.
Finally, you want Nginx to load your domain’s configuration from a new directory, /etc/nginx/vhost.d/
. This means editing the main Nginx configuration file.
Create nginx.conf.new
:
- vi nginx.conf.new
We’ll use a default Nginx config file, but we’ll modify it to include our specific site configuration. Put the following contents into this file:
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/vhost.d/*.conf;
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
root /usr/share/nginx/html;
include /etc/nginx/default.d/*.conf;
location / {
}
error_page 404 /404.html;
location = /40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
}
Save and exit the file.
With the configuration files in place, let’s define the scripts that Packer will use to install the software we need. Create a new folder to store your scripts:
- mkdir ~/packerProject/scripts
Now change to this new directory and create the installation script, configureNginx.sh
, which installs, configures, enables, and starts the Nginx web server:
- cd ~/packerProject/scripts
- vi configureNginx.sh
Paste the following into the file, which installs, configures, and starts Nginx, using the configuration files you just created:
#!/bin/bash
# Script to install Nginx and enable on boot.
# Update your system:
yum update -y
# Install EPEL Repository, update EPEL, and install Nginx:
yum install -y epel-release
yum update -y
yum install -y nginx
#Start Nginx service and enable to start on boot:
systemctl enable nginx
systemctl start nginx
# Create new 'vhost' directory for domain configuration:
mkdir /etc/nginx/vhost.d
# Create a new directory to serve new content.
mkdir -p /var/www/html/newDomain
# Create a copy of original configuration files and import configuration:
cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.original
cp /tmp/nginx.conf.new /etc/nginx/nginx.conf
# Copy over the server block configuration:
cp /tmp/newDomain.conf /etc/nginx/vhost.d/newDomain.conf
# Copy over the html test page:
cp /tmp/index.html.new /var/www/html/newDomain/index.html
# Restart Nginx:
systemctl restart nginx
Your template is finished and you’re now ready to validate and build your snapshot.
It’s time to test your template using Packer’s validate
subcommand. Once your template validates successfully you will build your Droplet and create the snapshot.
Change to the root of your project:
- cd ~/packerProject
The validate
subcommand will check your template for valid syntax and configuration options:
- packer.io validate -var-file=variables.json template.json
The -var-file
flag reads variables.json
and sets the value for my_token
within template.json
.
You’ll see the following output:
OutputTemplate validated successfully.
If there is something wrong with template.json
you will get an error message. This message will vary depending on the error but most can be fixed by double checking syntax and correcting any typos.
The build
subcommand runs the build that you defined in the builders
section of your template. In other words, it tells Packer to build your Droplet and then create a snapshot of that Droplet in your DigitalOcean dashboard.
Call packer.io build
to build the Droplet and create the snapshot:
- packer.io build -var-file=variables.json template.json
Note that the -var-file
flag operates in the exact same manner for both the build
and validate
subcommands.
The output of a successful build will look similar to the following:
Outputdigitalocean output will be in this color.
==> digitalocean: Creating temporary ssh key for Droplet...
==> digitalocean: Creating Droplet...
==> digitalocean: Waiting for Droplet to become active...
==> digitalocean: Waiting for SSH to become available...
==> digitalocean: Connected to SSH!
==> digitalocean: Gracefully shutting down Droplet...
==> digitalocean: Creating snapshot: packer-1467580504
==> digitalocean: Waiting for snapshot to complete...
==> digitalocean: Destroying Droplet...
==> digitalocean: Deleting temporary ssh key...
Build 'digitalocean' finished.
==> Builds finished. The artifacts of successful builds are:
--> digitalocean: A snapshot was created: 'packer-1487878703' (ID: 18252043) in region 'nyc1'
After a successful build you will find a new snapshot in your DigitalOcean snapshots repository. You can find the name of the snapshot in the output. In this example, it’s packer-1487878703
.
From here, visit your DigitalOcean dashboard, select Images, and the new Snapshot will appear in your list:
You can now use this new snapshot to create new Droplets. Select More and choose Create Droplet. Then complete the form to create your new machine.
Once the machine is online, determine its IP address from your dashboard and log in to your new machine:
- ssh root@your_new_server_ip_address
Then edit the Nginx server configuration file:
- vi /etc/nginx/vhost.d/newDomain.conf
And replace example.com
with either the IP address of the machine or the domain name you’ll use:
server {
listen 80;
listen [::]:80;
server_name your_new_server_ip_address;
location / {
root /var/www/html/newDomain;
index index.html index.htm;
}
}
Alternatively, you can use the sed
command to replace the value in the file, like this:
- sudo sed -i 's/^.*server_name example.com/server_name your_new_server_ip_address/' /etc/nginx/vhost.d/newDomain.conf
You can learn more about sed
in this tutorial.
Occasionally, you may run into an issue that isn’t adequately explained by the error message. In these scenarios, you can extract more detail about your build by enabling debug mode, inspecting the Packer logs, or both.
Debug mode provides builder-specific debugging information for each step in a remote build. Enabling debug mode for a DigitalOcean build will also produce a temporary private key in your project folder which you can use to connect to and inspect a running Droplet before it is turned into a snapshot.
You can enter debug mode by passing the -debug
flag to packer.io build
on the command line:
- packer.io build -debug --var-file=variables.json template.json
If you are unable to diagnose the issue in debug mode you can try enabling the Packer logs. These logs are primarily used to debug local builders, but they may provide helpful information on remote builds as well.
To enable the Packer logs, set the PACKER_LOG
environmental variable to any value except “0” or an empty string:
- PACKER_LOG=1 packer build --var-file=variables.json template.json
Logs will print to the console unless you also set the PACKER_LOG_PATH
environmental variable.
If you are still having problems, you may want to try reaching out to someone in the Packer community.
Now that you are comfortable with the basics of Packer, you may be interested in building on this foundation.
Try adding a second builder to your template to create a local testing environment alongside your DigitalOcean snapshot. The virtualbox-iso
builder, for example, produces images for VirtualBox, a free, open-source virtualization product used by both enterprises and hobbyists. You can define a post-processor
to the VirtualBox image and create Vagrant environments that mirror your DigitalOcean snapshots. This will allow you to test website changes locally before pushing them to a live Droplet. You can learn more in the Vagrant post-processor documentation.
Or you may want to connect your web server to a database. Add a second digitalocean
builder and use the only
key in your provisioners
section to apply different provisioning to each build.
If you’re more comfortable using a configuration management tools, Packer comes with out of the box support for Ansible, Puppet, Chef, and more. Try using one of these provisioners to further configure your Droplet to match your use case. If you’ve never tried configuration management before, take a look at How To Create Ansible Playbooks to Automate System Configuration on Ubuntu.
]]>Внимание: Более ранняя версия этой статьи включала упоминание Ubuntu 14.04. Хотя обновление с версии 14.04 может успешно завершиться, обновления с одной версии LTS до следующей версии LTS по умолчанию отключены до момента выхода первого обновления новой версии LTS (point release). Мы рекомендуем не обновляться до выхода версии 16.04.1. На серверах Digital Ocean системы с Ubuntu 14.04 после обновления будут использовать старое ядро, которое в течение первого времени нельзя будет обновить.
Следующий LTS релиз Ubuntu версии 16.04 (Xenial Xerus) будет доступен с 21 апреля 2016 года.
Хотя эта версия ещё не доступна для обновления на момент написания данной статьи, в настоящее время возможно обновить систему с 15.10 до девелоперской версии 16.04. Это может быть полезно для тестирования процесса обновления и новых возможностей 16.04 до момента официального выхода новой версии.
Эта статья описывает процесс обновления для систем с Ubuntu 15.10 (включая, но не ограничиваясь дроплетами Digital Ocean).
Внимание: Как и в случае любого другого обновления операционной системы этот процесс подвержен риску потери данных и конфигураций программного обеспечения. Рекомендуем сделать бэкапы перед началом процесса обновления и тщательно тестировать свои сервера после обновления.
Инструкции в этой статье предполагают, что у вы используете Ubuntu 15.10. Перед началом убедитесь, что у вас есть не-рутовый пользователь с привилегиями sudo
.
Несмотря на то, что многие системы могут обновиться без каких-либо проблем, чаще всего гораздо безопаснее и более предсказуемо обновляться на новую версию операционной системы путём установки с нуля, настройки конфигурации с одновременным её тестированием, перенося пользовательские данные только после завершения установки.
Не следует обновлять production систему без предварительного тестирования всего установленного программного обеспечения и сервисов с новой версией в staging среде. Помните, что библиотеки, языки программирования и системные сервисы могли значительно измениться в обновлении. В Ubuntu 16.04 важными изменениями по сравнению с предыдущей LTS версией стали переход к использованию системы инициализации systemd вместо Upstart, особый акцент на поддержке Python 3, а также использование PHP 7 вместо PHP 5.
Перед обновлением рекомендуем ознакомиться со списком изменений в Xenial Xerus.
Перед любым обновлением системы вам стоит убедиться, что вы не потеряете пользовательские данные в случае, если что-то пойдёт не так. Лучшим способом убедиться в этом является полный бэкап файловой системы. В случае, если это невозможно, убедитесь, что у вас есть копии пользовательских домашних директорий, всех необходимых конфигурационных файлов, а также данных, используемых сервисами, например, реляционные базы данных.
В случае дроплета Digital Ocean самым простым способом создания бэкапа является выключения сервера и создание снимка системы (snapshot). Выключение сервера при этом гарантирует, что файловая система будет сохранена в целостном состоянии. Для ознакомления с этим процессом рекомендуем прочитать статью Как использовать снимки системы Digital Ocean для автоматизации бэкапов ваших дроплетов. После того, как вы убедитесь, что обновление прошло успешно, вы можете удалить снимок системы, чтобы сэкономить деньги на его хранении.
Для поиска более универсального способа создания бэкапов, который будет работать на любой системе с Ubuntu, рекомендуем ознакомиться со статьёй Как выбрать эффективную стратегию создания бэкапов для вашего виртуального сервера.
Перед началом обновления системы лучше всего установить самые последние версии всех пакетов для текущей версии операционной системы.
Для этого сначала обновим список пакетов:
- sudo apt-get update
Далее обновим установленные пакеты до последних доступных версий:
- sudo apt-get upgrade
Вам будет представлен список обновлений пакетов. Отвечайте y и нажимайте Enter для продолжения.
Этот процесс может занять некоторое время. После его завершения используйте команду dist-upgrade
, которая выполнит обновления, затрагивающие зависимости пакетов, добавляя или удаляя пакеты при необходимости. В результате могут обновиться пакеты, которые не обновились в результате apt-get upgrade
:
- sudo apt-get dist-upgrade
Отвечайте y для продолжения и дождитесь завершения установки обновлений пакетов.
Теперь, когда у вас есть полностью обновлённая версия Ubuntu 15.10, вы можете использовать do-release-upgrade
для обновления до 16.04.
Сначала убедимся, что у вас установлен пакет update-manager-core
:
- sudo apt-get install update-manager-core
Традиционно дистрибутивы на базе Debian используют для обновления файл /etc/apt/sources.list
, указывающий на список репозиториев пакетов для apt, а также используют команду apt-get dist-upgrade
для выполнения самого обновления. Ubuntu основана на Debian, поэтому данный процесс, скорее всего, сработает. Тем не менее, мы будем использовать утилиту do-release-upgrade
, предоставляемую проектом Ubuntu, которая проверяет наличие нового релиза, обновляет sources.list
, а также выполняет ряд других действий. Это официально рекомендованный способ обновления серверов, который должен выполняться с помощью удалённого соединения.
Начнём с команды do-release-upgrade
без каких-либо опций:
- sudo do-release-upgrade
Если Ubuntu 16.04 ещё не доступна, вы увидите следующий вывод:
ВыводChecking for a new Ubuntu release
No new release found
Для обновления до версии 16.04 до момента её официального релиза, укажите ключ -d
для использования девелоперского релиза:
- sudo do-release-upgrade -d
Если вы зашли на свой сервер через SSH (как, например, в случае использования дроплета Digital Ocean), вас спросят, хотите ли ли вы продолжить установку.
В случае дроплета обновляться через SSH безопасно. Вы всегда можете использовать консоль в панели управления Digital Ocean для подключения к серверу не используя SSH.
При использовании виртуальный серверов или серверов других провайдеров вам необходимо иметь в виду, что потеря SSH соединения несёт риски, особенно, если у вас нет другого способа удалённо зайти на вашу машину. В случае других ваших систем, помните, что безопаснее всего выполнять обновления операционной системы только если у вас есть физический доступ к машине.
Ответьте y и нажмите Enter для продолжения:
Reading cache
Checking package manager
Continue running under SSH?
This session appears to be running under ssh. It is not recommended
to perform a upgrade over ssh currently because in case of failure it
is harder to recover.
If you continue, an additional ssh daemon will be started at port
'1022'.
Do you want to continue?
Continue [yN] y
Далее вас проинформируют, что do-release-upgrade
запустит новый процесс sshd
на порту 1022:
Starting additional sshd
To make recovery in case of failure easier, an additional sshd will
be started on port '1022'. If anything goes wrong with the running
ssh you can still connect to the additional one.
If you run a firewall, you may need to temporarily open this port. As
this is potentially dangerous it's not done automatically. You can
open the port with e.g.:
'iptables -I INPUT -p tcp --dport 1022 -j ACCEPT'
To continue please press [ENTER]
Нажмите Enter. Далее вы можете увидеть предупреждение о том, что зеркало для обновления не найдено. В системах Digital Ocean вы можете игнорировать это сообщение и продолжать обновление, поскольку локальное зеркало для 16.04 на самом деле является доступным. Введите y:
Updating repository information
No valid mirror found
While scanning your repository information no mirror entry for the
upgrade was found. This can happen if you run an internal mirror or
if the mirror information is out of date.
Do you want to rewrite your 'sources.list' file anyway? If you choose
'Yes' here it will update all 'trusty' to 'xenial' entries.
If you select 'No' the upgrade will cancel.
Continue [yN] y
После того, как новый список пакетов будет загружен, вас спросят, хотите ли вы начать обновление. Введите y для продолжения:
Do you want to start the upgrade?
6 installed packages are no longer supported by Canonical. You can
still get support from the community.
9 packages are going to be removed. 104 new packages are going to be
installed. 399 packages are going to be upgraded.
You have to download a total of 232 M. This download will take about
46 seconds with your connection.
Installing the upgrade can take several hours. Once the download has
finished, the process cannot be canceled.
Continue [yN] Details [d]y
Далее новые пакеты будут загружены, распакованы и установлены. Даже если ваша система имеет быстрое соединение, это займёт какое-то время.
В процессе установки вам могут задавать различные вопросы. Например, вас могут спросить, хотите ли вы автоматически перезапускать сервисы, когда это необходимо:
В этом случае вы можете ответить “Yes”. В других случаях вас могут спросить, хотите ли вы заменить изменённый вами конфигурационный файл на файл по умолчанию распространяемый с устанавливаемым пакетом. Ответ на этот вопрос чаще всего требует знания специфики работы данного конкретного пакета, и находится за пределами тем, описываемых в данной статье.
После установки пакетов вас спросят, хотите ли вы удалить ненужные пакеты. На новой системе без каких-либо изменений конфигурации вы можете ответить y. На системе, которую вы значительно модифицировали, вы можете ответить d и изучить предлагаемый для удаления список пакетов на случай, если вы захотите переустановить некоторые из них позже.
Remove obsolete packages?
53 packages are going to be removed.
Continue [yN] Details [d]y
Наконец, если всё прошло успешно, вам сообщат, что обновление завершено и необходимо перезапустить машину. Введите y для продолжения:
System upgrade is complete.
Restart required
To finish the upgrade, a restart is required.
If you select 'y' the system will be restarted.
Continue [yN] y
При использовании SSH вы, скорее всего, увидите нечто похожее:
=== Command detached from window (Thu Apr 7 13:13:33 2016) ===
=== Command terminated normally (Thu Apr 7 13:13:43 2016) ===
Возможно, вам придётся нажать любую клавишу для выхода в локальную консоль, поскольку ваша SSH сессия была прервана на стороне сервера. Подождите, пока система перезагрузится и войдите в неё снова. В процессе логина вы должны увидеть приветствие, подтверждающее установку Xenial Xerus:
Welcome to Ubuntu Xenial Xerus (development branch) (GNU/Linux 4.4.0-17-generic x86_64)
Теперь у вас должен быть работающий сервер с Ubuntu 16.04. Далее вам, скорее всего, будет необходимо изучить требуемые изменения конфигурации сервисов и установленных приложений для работы с новой версии операционной системы. В следующие несколько недель мы начнём публиковать статьи от Digital Ocean на разные темы, касающиеся Ubuntu 16.04.
]]>Atenção: Uma versão anterior desse guia incluía uma menção aos sistemas Ubuntu 14.04. Embora uma atualização a partir do 14.04 possa completar com sucesso, atualizações entre versões LTS não estão habilitadas por padrão até o primeiro ponto de versão, e é recomendado aguardar até o ponto de versão 16.04.1 para atualizar. Nos sistemas da DigitalOcean, um sistema atualizado 14.04 será deixado com um kernel mais antigo que pode não ser atualizável por algum tempo.
A próxima versão Long Term Support do sistema operacional Ubuntu, versão 16.04 (Xenial Xerus), deve ser lançada em 21 de abril de 2016.
Embora ainda não tenha sido lançado no momento da redação deste artigo, já é possível atualizar um sistema 15.10 para a versão de desenvolvimento 16.04. Isso pode ser útil para testar tanto o processo de atualização quanto as próprias características do 16.04 antes da data oficial do lançamento.
Este guia irá explicar o processo para sistemas incluindo (mas não limitado a) Droplets DigitalOcean rodando Ubuntu 15.10.
Atenção: Como acontece com quase toda atualização entre as principais versões de um sistema operacional, este processo carrega um risco inerente de falha, perda de dados ou quebra de configuração de software. Backups completos e testes extensos são fortemente aconselhados.
Este guia assume que você tem um sistema rodando Ubuntu 15.10, configurado com um usuário não-root com privilégios sudo
para tarefas administrativas.
Embora muitos sistemas possam ser atualizados sem incidentes, muitas vezes é mais seguro e mais previsível migrar para uma nova versão principal instalando a distribuição a partir do zero, configurando os serviços com testes cuidadosos ao longo do caminho, e migrando aplicações ou dados de usuários como um passo separado.
Você nunca deve atualizar um sistema de produção sem primeiro testar todo o seu software e os serviços implantados contra a atualização em um ambiente de teste. Tenha em mente que aquelas bibliotecas, linguagens e serviços de sistema podem ter mudado substancialmente. No Ubuntu 16.04, mudanças importantes desde a liberação do LTS anterior incluem a transição para o sistema init systemd no lugar do Upstart, uma ênfase no suporte ao Python 3, e PHP 7 no lugar do PHP 5.
Antes da atualização, considere a leitura das Notas de Versão Xenial Xerus.
Antes de tentar uma atualização principal em qualquer sistema, certifique-se de que você não perderá dados se a atualização der errado. A melhor forma de se conseguir isso é realizando um backup do seu sistema de arquivos inteiro. Se isso falhar, assegure-se de que você tem cópias dos diretórios home dos usuários, quaisquer arquivos de configuração, e dados armazenados por serviços como bancos de dados relacionais.
Em um Droplet DigitalOcean, a abordagem mais fácil é desligar o sistema e tirar um instantâneo (desligando garante que o sistema de arquivos estará mais consistente). Veja How To Use DigitalOcean Snapshots to Automatically Backup your Droplets para mais detalhes do processo de se tirar instantâneos. Quando você tiver verificado que a atualização foi realizada com sucesso, você pode deletar o instantâneo para que você não seja mais cobrado por ele.
Para métodos de backup que irão funcionar na maioria dos sistemas Ubuntu, veja How To Choose an Effective Backup Strategy for your VPS.
Antes de iniciar a atualização de versão, é mais seguro instalar as versões mais recentes de todos os pacotes para a versão atual. Comece atualizando a lista de pacotes:
sudo apt-get update
Depois, atualize os pacotes instalados para a suas últimas versões disponíveis:
- sudo apt-get upgrade
Será exibida uma lista de atualizações e será solicitado que você continue. Responda y para sim e pressione Enter.
Esse processo pode levar algum tempo. Quando terminar, utilize o comando dist-upgrade
, que realizará atualizações envolvendo alterações de dependências, adição ou remoção de novos pacotes quando necessário. Isso irá lidar com um conjunto de atualizações que podem ter sido retidas pelo apt-get upgrade
:
- sudo apt-get dist-upgrade
Novamente, responda y quando solicitado a continuar, e aguarde para que as atualizações terminem.
Agora que você tem uma instalação atualizada do Ubuntu 15.10, você pode utilizar do-release-upgrade
para atualizar para a versão 16.04.
Primeiro, certifique-se de que você tem o pacote update-manager-core
instalado:
- sudo apt-get install update-manager-core
Tradicionalmente, as versões do Debian são atualizáveis através da alteração do arquivo do Apt /etc/apt/sources.list
que especifica os repositórios de pacotes, e utilizando apt-get dist-upgrade
para realizar a atualização propriamente dita. O Ubuntu ainda é uma distribuição derivada do Debian, assim esse processo ainda funcionaria. Em vez disso, contudo, iremos utilizar do-release-upgrade
, uma ferramenta fornecida pelo projeto Ubuntu, que lida com a checagem de uma nova versão, atualizando sources.list
, e uma série de outras tarefas. Esse é o caminho oficial recomendado para atualizações para servidores, que deve ser realizada através de uma conexão remota.
Comece executando do-release-upgrade
sem opções:
- sudo do-release-upgrade
Se o Ubuntu 16.04 não tiver sido lançado ainda, você deve ver o seguinte:
Checking for a new Ubuntu release
No new release found
Para realizar a atualização para 16.04 antes do seu lançamento oficial, especifique a opção -d
de forma a utilizar a versão de desenvolvimento:
- sudo do-release-upgrade -d
Se você está conectado em seu sistema via SSH, como é provável com um Droplet da DigitalOcean, você será perguntado se deseja continuar.
Em um Droplet, é seguro atualizar via SSH. Embora o do-upgrade-release
não nos tenha informado disso, você pode utilizar o console disponível através do Painel de Controle da DigitalOcean para conectar em seu Droplet sem executar SSH.
Para máquinas virtuais ou servidores gerenciados hospedados por outros provedores, você deve ter em mente que perder a conexão SSH é um risco, particularmente se você não tiver outros meios de conectar-se remotamente ao console do sistema. Para outros sistemas sob o seu controle, lembre-se que é mais seguro realizar atualizações principais de sistema operacional quando você tem acesso físico à máquina.
No prompt, digite y e pressione Enter para continuar:
Reading cache
Checking package manager
Continue running under SSH?
This session appears to be running under ssh. It is not recommended
to perform a upgrade over ssh currently because in case of failure it
is harder to recover.
If you continue, an additional ssh daemon will be started at port
'1022'.
Do you want to continue?
Continue [yN] y
Em seguida, você será informado de que o do-release-upgrade
está iniciando uma nova instância do sshd
na porta 1022:
Starting additional sshd
To make recovery in case of failure easier, an additional sshd will
be started on port '1022'. If anything goes wrong with the running
ssh you can still connect to the additional one.
If you run a firewall, you may need to temporarily open this port. As
this is potentially dangerous it's not done automatically. You can
open the port with e.g.:
'iptables -I INPUT -p tcp --dport 1022 -j ACCEPT'
To continue please press [ENTER]
Pressione Enter. Em seguida, você pode ser avisado que uma entrada de espelho não foi encontrada. Nos sistemas da DigitalOcean, é seguro ignorar esse aviso e prosseguir com a atualização, visto que um espelho local para a 16.04 está de fato disponível. Digite y:
Updating repository information
No valid mirror found
While scanning your repository information no mirror entry for the
upgrade was found. This can happen if you run an internal mirror or
if the mirror information is out of date.
Do you want to rewrite your 'sources.list' file anyway? If you choose
'Yes' here it will update all 'trusty' to 'xenial' entries.
If you select 'No' the upgrade will cancel.
Continue [yN] y
Uma vez que as listas de novos pacotes tenham sido baixadas e a alterações calculadas, você será perguntado se quer iniciar a atualização. Novamente, digite y para continuar:
Do you want to start the upgrade?
6 installed packages are no longer supported by Canonical. You can
still get support from the community.
9 packages are going to be removed. 104 new packages are going to be
installed. 399 packages are going to be upgraded.
You have to download a total of 232 M. This download will take about
46 seconds with your connection.
Installing the upgrade can take several hours. Once the download has
finished, the process cannot be canceled.
Continue [yN] Details [d]y
Os novos pacotes serão agora baixados, depois descompactados e instalados. Mesmo que seu sistema esteja em uma conexão rápida, isso irá levar algum tempo.
Durante a instalação você pode se deparar com diálogos interativos para várias questões. Por exemplo, você pode ser perguntado se quer reiniciar automaticamente os serviços quando necessário:
Nesse caso, é seguro responder “Yes”. Em outros casos, você pode ser perguntado se deseja substituir um arquivo de configuração que você tenha modificado com a versão padrão do pacote que está sendo instalado. Isso geralmente é uma questão que provavelmente irá exigir um conhecimento sobre o software específico, o que está fora do escopo desse tutorial.
Uma vez que os novos pacotes tenham terminado de instalar, você será perguntado se está pronto para remover os pacotes obsoletos. Em um sistema regular sem configuração personalizada, deve ser seguro digitar y aqui. Em um sistema que você tenha modificado muito, você pode querer digitar d e inspecionar a lista de pacotes a serem removidos, no caso de incluir qualquer coisa que você precise reinstalar mais tarde.
Remove obsolete packages?
53 packages are going to be removed.
Continue [yN] Details [d]y
Finalmente, assumindo que tudo correu bem, você será informado de que a atualização está completa e uma reinicialização é necessária. Digite y para continuar:
System upgrade is complete.
Restart required
To finish the upgrade, a restart is required.
If you select 'y' the system will be restarted.
Continue [yN] y
Em uma sessão SSH, você provavelmente verá algo assim:
=== Command detached from window (Thu Apr 7 13:13:33 2016) ===
=== Command terminated normally (Thu Apr 7 13:13:43 2016) ===
Você pode precisar pressionar uma tecla aqui para sair para seu prompt local, uma vez que a sua sessão SSH terá terminado no lado do servidor. Aguarde um momento para seu sistema reinicializar, e reconecte. No login, você deve ser saudado por uma mensagem confirmando que você está agora no Xenial Xerus:
Welcome to Ubuntu Xenial Xerus (development branch) (GNU/Linux 4.4.0-17-generic x86_64)
Agora você deve ter uma instalação de trabalho do Ubuntu 16.04. A partir daqui, é provável que você precise investigar as alterações de configuração necessárias para os serviços e aplicativos implantados. Nas próximas semanas, começaremos a postar guias da DigitalOcean específicos do Ubuntu 16.04 em uma ampla gama de tópicos.
]]>Recently, a large portion of the core development team at ownCloud left to start a new project called Nextcloud. While ownCloud will still continue to be developed, you might want to see what the new project has to offer. Nextcloud and ownCloud share a common code base which means migrating your existing ownCloud installation to Nextcloud should be a painless task.
In this tutorial, you’ll migrate an existing ownCloud installation to Nextcloud. The process involves swapping out the core application files with those from Nextcloud, and letting Nextcloud’s built-in updater do the heavy lifting. While the process is simple, there are a number of things that need to be done in the correct order to make sure everything goes smoothly.
Note: You can only update ownCloud and Nextcloud installations one major version number at a time. If you currently use ownCloud 9, you must migrate to Nextcloud 10 first, and then upgrade to Nextcloud 11. This tutorial covers this process.
In order to migrate you ownCloud installation to Nextcloud, you will need:
sudo
. You can configure this by following the How to Create a Sudo User on Ubuntu tutorial.Even if you’re working with a freshly configured install, it is a good idea to do a quick backup. You’re about to start moving and deleting things, so safety first!
Log into your server running ownCloud if you’re not already connected:
- ssh sammy@your_server_ip
It’s important to make sure nothing changes while you perform the backup and migration, so the easiest way to ensure that is to shut down the web server so users can’t access ownCloud. Execute this command:
- sudo service apache2 stop
Now that the web server has stopped, navigate to the directory where your server stores ownCloud. If you are using the One-Click installation for ownCloud on Ubuntu 14.04, your installation is located within the /var/www/
directory. Run the following commands to switch to this directory and verify that it contains owncloud/
:
- cd /var/www
- ls
You’ll see the owncloud
folder:
Outputhtml owncloud
Next, create the backup archive using the tar
command to compress a gzip file and display verbose output to the screen. The new archive will be called owncloud.tar.gz
and will contain the entire owncloud/
directory. Execute the following command:
- sudo tar czfv owncloud.tar.gz owncloud/
Now move the archive to your home directory for safe-keeping:
- sudo mv owncloud.tar.gz ~/
Note: Your ownCloud files are backed up, but if you are using MySQL or any other database instead of the internal data storage option, you should also make a backup of the database. For MySQL, create a backup by running this command:
- mysqldump -u username -p dbname > ~/owncloud_backup.sql
You can find the values for username
, password
, and dbname
in the configuration file located at /var/www/owncloud/config/config.php
.
You can find more information about backing up and restoring MySQL databases here.
Before installing Nextcloud, there is one more step specific to Ubuntu 14.04 servers.
If you are migrating from the One-Click installation on Ubuntu 14.04 you will need to upgrade PHP to be able to use any version of Nextcloud that is newer than 10.0.2. The standard Ubuntu 14.04 repositories only include PHP 5.5, but PHP 5.6 is required starting with NextCloud 11. Luckily, Ubuntu supports 3rd party repositories known as PPAs. If you have not installed PPAs before, execute this command to install a package called python-software-properties
:
- sudo apt-get install python-software-properties
Next, add the PPA that contains updated versions of PHP:
- sudo add-apt-repository ppa:ondrej/php
Then tell the package manager to update its list of known packages, which includes those in the PPA:
- sudo apt-get update
Now you can install PHP7 and all of the modules that are required by Nextcloud:
- sudo apt-get install php7.0 php7.0-sqlite php7.0-mysql php7.0-pgsql php7.0-zip php7.0-gd php7.0-mb php7.0-curl php7.0-xml php7.0-apc
Finally, switch the PHP module that your web server uses. For Apache, the commands to do this are:
- a2dismod php5
- a2enmod php7.0
Note: If you are using your server for anything other than ownCloud, you should make sure that your web server doesn’t need PHP5.5 before disabling that module.
Now let’s get Nextcloud installed.
At the Nextcloud release site you will find a list of every Nextcloud release in a number of different formats. Find the most recent .tar.gz
file for the release that is the same as, or one major version after, your current ownCloud version. For example, if you are migrating from the ownCloud 9 One-Click installation you would be looking for the file nextcloud-10.0.2.tar.bz2
.
When you find the file, don’t download it onto your personal computer. Instead, right click the file name and copy the link address so you can download the file to your server.
You’re going to download two files. The first will be the Nextcloud package that you found on the web site. The other file will be a verification file called an “md5 checksum”. The md5 file will have the exact same path as the package, but with the extra extension .md5
added to the end. Execute the follow commands to move to your home directory, then download the two files.
- cd ~
- wget https://download.nextcloud.com/server/releases/nextcloud-10.0.2.tar.bz2
- wget https://download.nextcloud.com/server/releases/nextcloud-10.0.2.tar.bz2.md5
Run the md5sum
command to generate its checksum to verify the integrity of the package file:
- md5sum nextcloud-10.0.2.tar.bz2
You’ll see something similar to this output:
Outputdc30ee58858d4f6f2373472264f7d147 nextcloud-10.0.2.tar.bz2
Then display the contents of the .md5
file that you downloaded:
- cat nextcloud-10.0.2.tar.bz2.md5
The output of this command should be identical to the output of the previous command:
Outputdc30ee58858d4f6f2373472264f7d147 nextcloud-10.0.2.tar.bz2
If the outputs are different, download Nextcloud again.
To unpack the file, use the tar
command again, but this time, extract the file with verbose output. Execute this command to extract the archive:
- tar xfv nextcloud-10.0.2.tar.bz2
Finally, copy the newly extracted nextcloud
folder to the /var/www
folder:
- sudo mv nextcloud /var/www/nextcloud
Now you can start migrating your files from ownCloud to Nextcloud.
Your existing ownCloud installation has two directories you’ll want to preserve: data/
and config/
. You’ll move these from their original locations into your nextcoud
directory, but first, you’ll want to remove the default versions that came with Nextclout.
First, execute the command to delete the default directories from your nextcloud
directory, if they exist:
- sudo rm -rf /var/www/nextcloud/data /var/www/nextcloud/config
Then move the old directories over from the owncloud
directory:
- sudo mv /var/www/owncloud/data /var/www/nextcloud/data
- sudo mv /var/www/owncloud/config /var/www/nextcloud/config
One consquence of moving files with the sudo
command is the files will all be owned by the root user. Nextcloud, however, is always run by the www-data user. This means you need to change the ownership of the /var/www/nextcloud
folder and its contents before you go any further. To do this run the chown
command with the -R
argument to recursivly change all of the file ownerships to the www-data user:
- sudo chown -R www-data:www-data /var/www/nextcloud/
Now that the files are in place, we need to tell the web server how to access them.
With all of the files in place, you can initiate the internal upgrade process. Nextcloud and ownCloud provide a tool to manage and upgrade installations called occ
. Navigate to the /var/www/nextcloud/
directory:
- cd /var/www/nextcloud
Before you can use occ
, you’ll have to update the /var/www/nextcloud/config/config.php
file to reflect the new location of the data directory. Specifically, the line 'datadirectory' => '/var/www/owncloud/data',
needs to be changed to 'datadirectory' => '/var/www/nextcloud/data',
. Use sed
to easily make ths change:
- sudo sed -i "s/owncloud\/data/nextcloud\/data/g" config/config.php
NOTE: Normally, sed
streams output to the screen, but the -i
flag tells it to modify the file in place. For information about how to use regular expressions, see An Introduction To Regular Expressions. And for more on sed
, look at The Basics of Using the Sed Stream Editor to Manipulate Text in Linux.
Now use occ
to put Nextcloud into maintenance mode. This locks down the files so no changes can be made externally while you upgrade the application. Run the following command to turn on maintenance mode:
- sudo -u www-data php occ maintenance:mode --on
Note that this uses sudo
to run commands as the www-data user.
You’ll see this output so you can confirm that maintenance mode is turned on:
[secondary_output]
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Maintenance mode enabled
Next, use occ
to initiate the internal upgrade process:
- sudo -u www-data php occ upgrade
This command displays a lot of output as it migrates all of the ownCloud data to Nextcloud, but in the end you’ll see the following messages:
Output...
Starting code integrity check...
Finished code integrity check
Update successful
Maintenance mode is kept active
Reset log level
If there were problems with the upgrade, the output will give you some feedback about what went wrong and how to solve the problem. Assuming the upgrade went smoothly its time to turn off maintenance mode.
- sudo -u www-data php occ maintenance:mode --off
Your ownCloud installation has now been migrated to Nextcloud, but it may still be out-of-date. If you migrated ownCloud 9 you will have only migrated to Nextcloud 10, but there is still a newer version so let’s upgrade.
To upgrade Nextcloud to a new major version, you use the same procedure you used in Steps 3 through 5 of this tutorial. First, move your currently installed Nextcloud folder out of the way with this command:
- sudo mv /var/www/nextcloud /var/www/nextcloud.old
Then find the .tar.gz
file from the Nextcloud release site, download it, and check its MD5 checksum just as you did in Step 3.
- wget https://download.nextcloud.com/server/releases/nextcloud-11.0.0.tar.bz2
- wget https://download.nextcloud.com/server/releases/nextcloud-11.0.0.tar.bz2.md5
- md5sum nextcloud-11.0.0.tar.bz2
- cat nextcloud-11.0.0.tar.bz2.md5
Once you’ve downloaded and verified the archive. unpack it and move it to the Nextcloud location on the web server:
- tar xfv nextcloud-11.0.0.tar.bz2
- mv nextcloud /var/www/nextcloud
Next, move the configuration and data files from the old installation to the new one as you did in Step 4:
- rm -rf /var/www/nextcloud/config /var/www/nextcloud/data
- mv /var/www/nextcloud.old/config /var/www/nextcloud
- mv /var/www/nextcloud.old/data /var/www/nextcloud
- sudo chown -R www-data:www-data /var/www/nextcloud/
Finally, use occ
to perform the upgrade:
- sudo -u www-data php occ maintenance:mode --on
- sudo -u www-data php occ upgrade
- sudo -u www-data php occ maintenance:mode --off
Repeat these steps for each major version of Nextcloud you need to upgrade through.
Now that everything is up-to-date, we can configure the web server to send traffic to Nextcloud.
The Apache web server directs to different directories through the use of virtual hosts, or vhosts. The folder /etc/apache2/sites-available/
contains a description of each vhost that is configured for the server. These vhosts are enabled by linking their associated files to the /etc/apache2/sites-enabled/
folder. The file /etc/apache2/sites-available/000-owncloud.conf
configures the server to read the /var/www/owcloud
and that configuration is enabled by the link located at /etc/apache2/sites-enabled/000-owncloud.conf
.
To convert the server to use the Nextcloud installation, create a copy of the ownCloud vhost configuration, edit it to point at Nextcloud, disable the ownCloud vhost, and enable the Nextcloud vhost.
Fist copy the ownCloud configuration file:
- sudo cp /etc/apache2/sites-available/000-owncloud.conf /etc/apache2/sites-available/000-nextcloud.conf
Next, replace all instances of owncloud
in the configuration file with nextcloud
. You can do this by opening /etc/apache2/sites-available/000-nextcloud.conf
with a text editor and making the changes yourself, or by using regular expressions and the sed
command.
Run the following command to convert the contents of the vhost configuration file with sed
:
- sudo sed -i "s/owncloud/nextcloud/g" /etc/apache2/sites-available/000-nextcloud.conf
Next, disable the ownCloud vhost by deleting the link /etc/apache2/sites-enabled/000-owncloud.conf
. Ubuntu provides the a2dissite
command to disable sites. Execute this command:
- sudo a2dissite 000-owncloud.conf
Finally, enable the Nextcloud vhost by creating a symbolic link to the Nextcloud configuration file. Use the a2ensite
command to create the link:
- sudo a2ensite 000-nextcloud.conf
Note: If you access ownCloud through HTTPS, you will also need to repeat these steps with the /etc/apache2/sites-available/owncloud-ssl.conf
vhost.
Now that the web server knows where to find Nextcloud, we can start it back up with this command:
- sudo service apache2 start
At this point everything should be up and running with your new Nextcloud installation. Open up a web browser and navigate to the location of your old ownCloud server and you’ll see the Nextcloud login screen. All of your old user names and passwords will work just as they did before the migration. Log in as the admin user, as you may need to re-enable some of your apps, including the Calendar and Contacts apps.
In this tutorial you backed up your previous ownCloud installation, migrated to Nextcloud, and disabled ownCloud. You can now log into Nextcloud using the web interface just as you did with ownCloud.
Now that your server has been migrated to Nextcloud, it is time to update any sync clients you are using. Just like ownCloud, Nextcloud provides a number of syncing clients for your desktop and mobile devices.
If you decide to switch back to ownCloud you can restore the data/
and config/
folders from the backup you created in Step 1, as well as any external database you backed up. Do not try to copy the data/
and config/
folders from /var/www/nextcloud
back to ownCloud. Once the backups have been restored, all you have to do is disable the Nextcloud vhost and enable the ownCloud one, using the same procedure in Step 4.
Advertencia: Una versión anterior de esta guía incluía mención de los sistemas Ubuntu 14.04. Mientras que una actualización de 14.04 puede completarse con éxito, las actualizaciones entre versiones LTS no están habilitadas por defecto hasta la primera liberación, y se recomienda esperar hasta la versión 16.04.1 para actualizar. En los sistemas de DigitalOcean, un sistema actualizado de Ubuntu 14.04 quedará con un kernel antiguo que puede no ser actualizable durante algún tiempo.
El próximo lanzamiento de Soporte a Largo Plazo (LTS) del sistema operativo Ubuntu, versión 16.04 (Xenial Xerus), se estrenará el 21 de abril de 2016.
Aunque todavía no se ha publicado en el momento de escribir este documento, ya es posible actualizar un sistema 15.10 a la versión de desarrollo de 16.04. Esto puede ser útil para probar tanto el proceso de actualización como las características de 16.04 antes de la fecha oficial de lanzamiento.
Esta guía explicará el proceso de los sistemas que incluyen (pero no limitado a) Droplets de DigitalOcean con Ubuntu 15.10.
Advertencia: Al igual que con casi cualquier actualización entre versiones principales de un sistema operativo, este proceso conlleva un riesgo inherente de falla, pérdida de datos o configuración rota de software. Se recomienda encarecidamente realizar copias de seguridad completas y realizar extensas pruebas.
Esta guía asume que usted tiene un sistema que ejecuta Ubuntu 15.10, configurado con una cuenta de usuario independiente que no sea root con privilegios de sudo
para tareas administrativas.
Aunque muchos sistemas pueden actualizarse sin incidentes, a menudo es más seguro y más predecible migrar a una nueva versión principal instalando la distribución desde cero, configurando los servicios con pruebas cuidadas a lo largo del proceso y migrando los datos de aplicaciones o usuarios como datos separados paso.
Nunca debe actualizarse un sistema de producción sin probar primero todo el software y los servicios desplegados frente a la actualización en un entorno de almacenamiento intermedio. Tenga en cuenta que las bibliotecas, lenguajes y servicios del sistema pueden haber cambiado sustancialmente. En Ubuntu 16.04, hay cambios importantes desde la versión anterior de LTS que incluyen una transición al sistema systemd init en lugar de upstart, un énfasis en soporte de Python 3 y PHP 7 en lugar de PHP 5.
Antes de actualizar, le recomendamos que lea las notas de la versión Xenial Xerus.
Antes de intentar realizar una actualización importante en cualquier sistema, asegúrese de no perder datos si la actualización falla. La mejor manera de lograr esto es hacer una copia de seguridad de todo tu sistema de archivos. De lo contrario, asegúrese de tener copias de los directorios personales de usuario, de cualquier archivo de configuración personalizado y de datos almacenados por servicios como bases de datos relacionales.
En un Droplet de DigitalOcean, el enfoque más fácil es apagar el sistema y tomar una copia instantánea (la desconexión garantiza que el sistema de archivos será más coherente). Consulte ¿Cómo utilizar las instantáneas de DigitalOcean para hacer una copia de seguridad automática de droplets? para obtener más detalles sobre el proceso de copias instantáneas. Cuando haya verificado que la actualización se realizó correctamente, puede eliminar la copia instantánea para que ya no se cargue.
Para los métodos de copia de seguridad que funcionarán en la mayoría de los sistemas de Ubuntu, consulte ¿Cómo elegir una estrategia de copia de seguridad eficaz para su VPS?.
Antes de comenzar la actualización de la versión, es más seguro instalar las versiones más recientes de todos los paquetes para la versión actual. Comience actualizando la lista de paquetes:
- sudo apt-get update
A continuación, actualizar los paquetes instalados a sus últimas versiones disponibles:
- sudo apt-get upgrade
Se mostrará una lista de actualizaciones, y preguntará si deseas continuar. Responder y para sí y pulse Enter.
Este proceso puede tomar algún tiempo. Una vez que termine, utilice el comando dist-upgrade
, que llevará a cabo actualizaciones que impliquen cambiar dependencias, agregar o quitar nuevos paquetes según sea necesario. Esto manejará un conjunto de actualizaciones que pueden haber sido retenidas por apt-get upgrade
:
- sudo apt-get dist-upgrade
Una vez más, presione y cuando se pida para continuar, y espera que las actualizaciones se terminen.
Ahora que tiene una instalación a la fecha de Ubuntu 15.10, puede utilizar do-release-upgrade
para actualizar a la versión 16.04.
En primer lugar, asegúrese de que tiene el paquete update-manager-core
instalado:
- sudo apt-get install update-manager-core
Tradicionalmente, las versiones de Debian se pueden actualizar mediante el cambio del archivo de Apt /etc/apt/sources.list
, que especifica los repositorios de paquetes, y utilizar apt-get dist-upgrade
para realizar la actualización en sí. Ubuntu es todavía una distribución derivada de Debian, por lo que este proceso sería probable que todavía trabajar. En lugar de ello, sin embargo, vamos a utilizar do-release-upgrade, una herramienta proporcionada por el proyecto Ubuntu, que se encarga de comprobar para una nueva versión, las actualizaciones de sources.list, y una serie de otras tareas. Esta es la ruta de actualización oficialmente recomendada para actualizaciones del servidor que deben realizarse a través de una conexión remota.
Comience a correr do-release-upgrade
sin opciones:
- sudo do-release-upgrade
Si Ubuntu 16.04 no se ha liberado aún, debería ver lo siguiente:
Checking for a new Ubuntu release
No new release found
Con el fin de actualizar a 16.04 antes de su lanzamiento oficial, especifique la opción -d
con el fin de utilizar la liberación de desarrollo:
- sudo do-release-upgrade -d
Si está conectado a su sistema a través de SSH, como es probable con un Droplet de DigitalOcean, se le preguntará si desea continuar.
En un Droplet, es seguro actualizar a través de SSH. A pesar de que do-upgrade-release
no nos ha informado de ello, puede utilizar la consola disponible en el panel de control de DigitalOcean para conectarse a su Droplet sin ejecutar SSH.
Para máquinas virtuales o servidores administrados alojados por otros proveedores, debe tener en cuenta que la pérdida de conectividad SSH es un riesgo, sobre todo si no dispone de otro medio de conexión remota a la consola del sistema. Para otros sistemas bajo su control, recuerde que es más seguro realizar las actualizaciones principales del sistema operativo sólo cuando se tiene acceso físico directo a la máquina.
En el indicador, escriba Y y pulse Enter para continuar:
Reading cache
Checking package manager
Continue running under SSH?
This session appears to be running under ssh. It is not recommended
to perform a upgrade over ssh currently because in case of failure it
is harder to recover.
If you continue, an additional ssh daemon will be started at port
'1022'.
Do you want to continue?
Continue [yN] y
A continuación, se te informará de que do-release-upgrade
está comenzando una nueva instancia de sshd
en el puerto 1022:
Starting additional sshd
To make recovery in case of failure easier, an additional sshd will
be started on port '1022'. If anything goes wrong with the running
ssh you can still connect to the additional one.
If you run a firewall, you may need to temporarily open this port. As
this is potentially dangerous it's not done automatically. You can
open the port with e.g.:
'iptables -I INPUT -p tcp --dport 1022 -j ACCEPT'
To continue please press [ENTER]
Presiona Enter. A continuación, se le puede advertir que no se encontró una entrada de espejo. En los sistemas de DigitalOcean, es seguro hacer caso omiso de esta advertencia y continuar con la actualización, ya que un espejo local para 16.04 está de hecho disponible. Introduzca y:
Updating repository information
No valid mirror found
While scanning your repository information no mirror entry for the
upgrade was found. This can happen if you run an internal mirror or
if the mirror information is out of date.
Do you want to rewrite your 'sources.list' file anyway? If you choose
'Yes' here it will update all 'trusty' to 'xenial' entries.
If you select 'No' the upgrade will cancel.
Continue [yN] y
Una vez descargadas las nuevas listas de paquetes y calculados los cambios, se le preguntará si deseas iniciar la actualización. Otra vez, ingrese y para continuar:
Do you want to start the upgrade?
6 installed packages are no longer supported by Canonical. You can
still get support from the community.
9 packages are going to be removed. 104 new packages are going to be
installed. 399 packages are going to be upgraded.
You have to download a total of 232 M. This download will take about
46 seconds with your connection.
Installing the upgrade can take several hours. Once the download has
finished, the process cannot be canceled.
Continue [yN] Details [d] y
Ahora se recuperarán los nuevos paquetes, luego se desempaquetarán e instalarán. Incluso si su sistema está en una conexión rápida, esto tomará un tiempo.
Durante la instalación, se le pueden presentar diálogos interactivos para varias preguntas. Por ejemplo, le puede preguntar si desea reiniciar automáticamente los servicios cuando sea necesario:
En este caso, es seguro que responder “Yes”. En otros casos, es posible que se le preguntará si desea reemplazar un archivo de configuración que ha sido modificado con la versión predeterminada del paquete que se está instalando. Esto es a menudo una cuestión de criterio, y es probable que requiera conocimientos sobre software específico que está fuera del alcance de este tutorial.
Una vez que los nuevos paquetes se han terminado de instalar, se le preguntará si está listo para eliminar los paquetes obsoletos. En un sistema por defecto sin necesidad de configuración personalizada, debe ser seguro pulsar y. En un sistema que se ha modificado en gran medida, es posible que desee pulsar d e inspeccionar la lista de paquetes que serán eliminados, en caso de que sea necesario reinstalar en el futuro.
Remove obsolete packages?
53 packages are going to be removed.
Continue [yN] Details [d] y
Por último, suponiendo que todo ha ido bien, se le informará de que la actualización se ha completado y se requiere un reinicio. Introduzca y* para continuar:
System upgrade is complete.
Restart required
To finish the upgrade, a restart is required.
If you select 'y' the system will be restarted.
Continue [yN] y
En una sesión SSH, es probable que vea algo como lo siguiente:
=== Command detached from window (Thu Apr 7 13:13:33 2016) ===
=== Command terminated normally (Thu Apr 7 13:13:43 2016) ===
Es posible que tenga que pulsar una tecla para salir a su local prompt, ya que la sesión de SSH se habrá terminado en el extremo del servidor. Espere un momento para que el sistema se reinicie, y vuelva a conectar. Al iniciar sesión, debe recibir un mensaje confirmando que está en Xenial Xerus:
- Welcome to Ubuntu Xenial Xerus (development branch) (GNU/Linux 4.4.0-17-generic x86_64)
Ahora debería tener una instalación de trabajo de Ubuntu 16.04. A partir de aquí, es probable que necesite investigar los cambios de configuración necesarios en los servicios y las aplicaciones implementadas. En las próximas semanas, vamos a comenzar la publicación de guías DigitalOcean específicas para Ubuntu 16.04 en una amplia gama de temas.
]]>Advertencia: Una versión anterior de esta guía incluía mención de los sistemas Ubuntu 14.04. Mientras que una actualización de 14.04 puede completarse con éxito, las actualizaciones entre versiones LTS no están habilitadas por defecto hasta la primera liberación, y se recomienda esperar hasta la versión 16.04.1 para actualizar. En los sistemas de DigitalOcean, un sistema actualizado de Ubuntu 14.04 quedará con un kernel antiguo que puede no ser actualizable durante algún tiempo.
El próximo lanzamiento de Soporte a Largo Plazo (LTS) del sistema operativo Ubuntu, versión 16.04 (Xenial Xerus), se estrenará el 21 de abril de 2016.
Aunque todavía no se ha publicado en el momento de escribir este documento, ya es posible actualizar un sistema 15.10 a la versión de desarrollo de 16.04. Esto puede ser útil para probar tanto el proceso de actualización como las características de 16.04 antes de la fecha oficial de lanzamiento.
Esta guía explicará el proceso de los sistemas que incluyen (pero no limitado a) Droplets de DigitalOcean con Ubuntu 15.10.
Advertencia: Al igual que con casi cualquier actualización entre versiones principales de un sistema operativo, este proceso conlleva un riesgo inherente de falla, pérdida de datos o configuración rota de software. Se recomienda encarecidamente realizar copias de seguridad completas y realizar extensas pruebas.
Esta guía asume que usted tiene un sistema que ejecuta Ubuntu 15.10, configurado con una cuenta de usuario independiente que no sea root con privilegios de sudo
para tareas administrativas.
Aunque muchos sistemas pueden actualizarse sin incidentes, a menudo es más seguro y más predecible migrar a una nueva versión principal instalando la distribución desde cero, configurando los servicios con pruebas cuidadas a lo largo del proceso y migrando los datos de aplicaciones o usuarios como datos separados paso.
Nunca debe actualizarse un sistema de producción sin probar primero todo el software y los servicios desplegados frente a la actualización en un entorno de almacenamiento intermedio. Tenga en cuenta que las bibliotecas, lenguajes y servicios del sistema pueden haber cambiado sustancialmente. En Ubuntu 16.04, hay cambios importantes desde la versión anterior de LTS que incluyen una transición al sistema systemd init en lugar de upstart, un énfasis en soporte de Python 3 y PHP 7 en lugar de PHP 5.
Antes de actualizar, le recomendamos que lea las notas de la versión Xenial Xerus.
Antes de intentar realizar una actualización importante en cualquier sistema, asegúrese de no perder datos si la actualización falla. La mejor manera de lograr esto es hacer una copia de seguridad de todo tu sistema de archivos. De lo contrario, asegúrese de tener copias de los directorios personales de usuario, de cualquier archivo de configuración personalizado y de datos almacenados por servicios como bases de datos relacionales.
En un Droplet de DigitalOcean, el enfoque más fácil es apagar el sistema y tomar una copia instantánea (la desconexión garantiza que el sistema de archivos será más coherente). Consulte ¿Cómo utilizar las instantáneas de DigitalOcean para hacer una copia de seguridad automática de droplets? para obtener más detalles sobre el proceso de copias instantáneas. Cuando haya verificado que la actualización se realizó correctamente, puede eliminar la copia instantánea para que ya no se cargue.
Para los métodos de copia de seguridad que funcionarán en la mayoría de los sistemas de Ubuntu, consulte ¿Cómo elegir una estrategia de copia de seguridad eficaz para su VPS?.
Antes de comenzar la actualización de la versión, es más seguro instalar las versiones más recientes de todos los paquetes para la versión actual. Comience actualizando la lista de paquetes:
sudo apt-get update
A continuación, actualizar los paquetes instalados a sus últimas versiones disponibles:
sudo apt-get upgrade
Se mostrará una lista de actualizaciones, y preguntará si deseas continuar. Responder y para sí y pulse Enter.
Este proceso puede tomar algún tiempo. Una vez que termine, utilice el comando dist-upgrade
, que llevará a cabo actualizaciones que impliquen cambiar dependencias, agregar o quitar nuevos paquetes según sea necesario. Esto manejará un conjunto de actualizaciones que pueden haber sido retenidas por apt-get upgrade
:
sudo apt-get dist-upgrade
Una vez más, presione y cuando se pida para continuar, y espera que las actualizaciones se terminen.
Ahora que tiene una instalación a la fecha de Ubuntu 15.10, puede utilizar do-release-upgrade
para actualizar a la versión 16.04.
En primer lugar, asegúrese de que tiene el paquete update-manager-core
instalado:
sudo apt-get install update-manager-core
Tradicionalmente, las versiones de Debian se pueden actualizar mediante el cambio del archivo de Apt /etc/apt/sources.list
, que especifica los repositorios de paquetes, y utilizar apt-get dist-upgrade
para realizar la actualización en sí. Ubuntu es todavía una distribución derivada de Debian, por lo que este proceso sería probable que todavía trabajar. En lugar de ello, sin embargo, vamos a utilizar do-release-upgrade, una herramienta proporcionada por el proyecto Ubuntu, que se encarga de comprobar para una nueva versión, las actualizaciones de sources.list, y una serie de otras tareas. Esta es la ruta de actualización oficialmente recomendada para actualizaciones del servidor que deben realizarse a través de una conexión remota.
Comience a correr do-release-upgrade
sin opciones:
sudo do-release-upgrade
Si Ubuntu 16.04 no se ha liberado aún, debería ver lo siguiente:
Output
Checking for a new Ubuntu release
No new release found
Con el fin de actualizar a 16.04 antes de su lanzamiento oficial, especifique la opción -d
con el fin de utilizar la liberación de desarrollo:
sudo do-release-upgrade -d
Si está conectado a su sistema a través de SSH, como es probable con un Droplet de DigitalOcean, se le preguntará si desea continuar.
En un Droplet, es seguro actualizar a través de SSH. A pesar de que do-upgrade-release
no nos ha informado de ello, puede utilizar la consola disponible en el panel de control de DigitalOcean para conectarse a su Droplet sin ejecutar SSH.
Para máquinas virtuales o servidores administrados alojados por otros proveedores, debe tener en cuenta que la pérdida de conectividad SSH es un riesgo, sobre todo si no dispone de otro medio de conexión remota a la consola del sistema. Para otros sistemas bajo su control, recuerde que es más seguro realizar las actualizaciones principales del sistema operativo sólo cuando se tiene acceso físico directo a la máquina.
En el indicador, escriba Y y pulse Enter para continuar:
Reading cache
Checking package manager
Continue running under SSH?
This session appears to be running under ssh. It is not recommended
to perform a upgrade over ssh currently because in case of failure it
is harder to recover.
If you continue, an additional ssh daemon will be started at port
'1022'.
Do you want to continue?
Continue [yN] y
A continuación, se te informará de que do-release-upgrade
está comenzando una nueva instancia de sshd
en el puerto 1022:
Starting additional sshd
To make recovery in case of failure easier, an additional sshd will
be started on port '1022'. If anything goes wrong with the running
ssh you can still connect to the additional one.
If you run a firewall, you may need to temporarily open this port. As
this is potentially dangerous it's not done automatically. You can
open the port with e.g.:
'iptables -I INPUT -p tcp --dport 1022 -j ACCEPT'
To continue please press [ENTER]
Presiona Enter. A continuación, se le puede advertir que no se encontró una entrada de espejo. En los sistemas de DigitalOcean, es seguro hacer caso omiso de esta advertencia y continuar con la actualización, ya que un espejo local para 16.04 está de hecho disponible. Introduzca y:
Updating repository information
No valid mirror found
While scanning your repository information no mirror entry for the
upgrade was found. This can happen if you run an internal mirror or
if the mirror information is out of date.
Do you want to rewrite your 'sources.list' file anyway? If you choose
'Yes' here it will update all 'trusty' to 'xenial' entries.
If you select 'No' the upgrade will cancel.
Continue [yN] y
Una vez descargadas las nuevas listas de paquetes y calculados los cambios, se le preguntará si deseas iniciar la actualización. Otra vez, ingrese y para continuar:
Do you want to start the upgrade?
6 installed packages are no longer supported by Canonical. You can
still get support from the community.
9 packages are going to be removed. 104 new packages are going to be
installed. 399 packages are going to be upgraded.
You have to download a total of 232 M. This download will take about
46 seconds with your connection.
Installing the upgrade can take several hours. Once the download has
finished, the process cannot be canceled.
Continue [yN] Details [d]y
Ahora se recuperarán los nuevos paquetes, luego se desempaquetarán e instalarán. Incluso si su sistema está en una conexión rápida, esto tomará un tiempo.
Durante la instalación, se le pueden presentar diálogos interactivos para varias preguntas. Por ejemplo, le puede preguntar si desea reiniciar automáticamente los servicios cuando sea necesario:
En este caso, es seguro que responder “Sí”. En otros casos, es posible que se le preguntará si desea reemplazar un archivo de configuración que ha sido modificado con la versión predeterminada del paquete que se está instalando. Esto es a menudo una cuestión de criterio, y es probable que requiera conocimientos sobre software específico que está fuera del alcance de este tutorial.
Una vez que los nuevos paquetes se han terminado de instalar, se le preguntará si está listo para eliminar los paquetes obsoletos. En un sistema por defecto sin necesidad de configuración personalizada, debe ser seguro pulsar y. En un sistema que se ha modificado en gran medida, es posible que desee pulsar d e inspeccionar la lista de paquetes que serán eliminados, en caso de que sea necesario reinstalar en el futuro.
Remove obsolete packages?
53 packages are going to be removed.
Continue [yN] Details [d]y
Por último, suponiendo que todo ha ido bien, se le informará de que la actualización se ha completado y se requiere un reinicio. Introduzca y* para continuar:
System upgrade is complete.
Restart required
To finish the upgrade, a restart is required.
If you select 'y' the system will be restarted.
Continue [yN] y
En una sesión SSH, es probable que vea algo como lo siguiente:
=== Command detached from window (Thu Apr 7 13:13:33 2016) ===
=== Command terminated normally (Thu Apr 7 13:13:43 2016) ===
Es posible que tenga que pulsar una tecla para salir a su local prompt, ya que la sesión de SSH se habrá terminado en el extremo del servidor. Espere un momento para que el sistema se reinicie, y vuelva a conectar. Al iniciar sesión, debe recibir un mensaje confirmando que está en Xenial Xerus:
Welcome to Ubuntu Xenial Xerus (development branch) (GNU/Linux 4.4.0-17-generic x86_64)
Ahora debería tener una instalación de trabajo de Ubuntu 16.04. A partir de aquí, es probable que necesite investigar los cambios de configuración necesarios en los servicios y las aplicaciones implementadas. En las próximas semanas, vamos a comenzar la publicación de guías DigitalOcean específicas para Ubuntu 16.04 en una amplia gama de temas.
]]>Warning: As with almost any upgrade between major releases of an operating system, this process carries an inherent risk of failure, data loss, or broken software configuration. Comprehensive backups and extensive testing are strongly advised.
To avoid these problems, when possible, we recommend migrating to a fresh Ubuntu 16.04 server rather than upgrading in-place. You may still need to review differences in software configuration when upgrading, but the core system will likely have greater stability. You can follow our series on how to migrate to a new Linux server to learn how to migrate between servers.
The Ubuntu operating system’s next Long Term Support release, version 16.04 (Xenial Xerus), is due to be released on April 21, 2016.
Although it hasn’t yet been released at the time of this writing, it’s already possible to upgrade a 15.10 system to the development version of 16.04. This may be useful for testing both the upgrade process and the features of 16.04 itself in advance of the official release date.
This guide will explain the process for systems including (but not limited to) DigitalOcean Droplets running Ubuntu 15.10.
This guide assumes that you have a system running Ubuntu 15.10, configured with a non-root user with sudo
privileges for administrative tasks.
Although many systems can be upgraded in place without incident, it is often safer and more predictable to migrate to a major new release by installing the distribution from scratch, configuring services with careful testing along the way, and migrating application or user data as a separate step.
You should never upgrade a production system without first testing all of your deployed software and services against the upgrade in a staging environment. Keep in mind that libraries, languages, and system services may have changed substantially. In Ubuntu 16.04, important changes since the preceding LTS release include a transition to the systemd init system in place of Upstart, an emphasis on Python 3 support, and PHP 7 in place of PHP 5.
Before upgrading, consider reading the Xenial Xerus Release Notes.
Before attempting a major upgrade on any system, you should make sure you won’t lose data if the upgrade goes awry. The best way to accomplish this is to make a backup of your entire filesystem. Failing that, ensure that you have copies of user home directories, any custom configuration files, and data stored by services such as relational databases.
On a DigitalOcean Droplet, the easiest approach is to power down the system and take a snapshot (powering down ensures that the filesystem will be more consistent). See How To Use DigitalOcean Snapshots to Automatically Backup your Droplets for more details on the snapshot process. When you have verified that the update was successful, you can delete the snapshot so that you will no longer be charged for it.
For backup methods which will work on most Ubuntu systems, see How To Choose an Effective Backup Strategy for your VPS.
Before beginning the release upgrade, it’s safest to install the latest versions of all packages for the current release. Begin by updating the package list:
- sudo apt-get update
Next, upgrade installed packages to their latest available versions:
- sudo apt-get upgrade
You will be shown a list of upgrades, and prompted to continue. Answer y for yes and press Enter.
This process may take some time. Once it finishes, use the dist-upgrade
command, which will perform upgrades involving changing dependencies, adding or removing new packages as necessary. This will handle a set of upgrades which may have been held back by apt-get upgrade
:
- sudo apt-get dist-upgrade
Again, answer y when prompted to continue, and wait for upgrades to finish.
Now that you have an up-to-date installation of Ubuntu 15.10, you can use do-release-upgrade
to upgrade to the 16.04 release.
First, make sure you have the update-manager-core
package installed:
- sudo apt-get install update-manager-core
Traditionally, Debian releases have been upgradeable by changing Apt’s /etc/apt/sources.list
, which specifies package repositories, and using apt-get dist-upgrade
to perform the upgrade itself. Ubuntu is still a Debian-derived distribution, so this process would likely still work. Instead, however, we’ll use do-release-upgrade
, a tool provided by the Ubuntu project, which handles checking for a new release, updating sources.list
, and a range of other tasks. This is the officially recommended upgrade path for server upgrades which must be performed over a remote connection.
Start by running do-release-upgrade
with no options:
- sudo do-release-upgrade
If Ubuntu 16.04 has not been released yet, you should see the following:
Checking for a new Ubuntu release
No new release found
In order to upgrade to 16.04 before its official release, specify the -d
option in order to use the development release:
- sudo do-release-upgrade -d
If you’re connected to your system over SSH, as is likely with a DigitalOcean Droplet, you’ll be asked whether you wish to continue.
On a Droplet, it’s safe to upgrade over SSH. Although do-upgrade-release
has not informed us of this, you can use the console available from the DigitalOcean Control Panel to connect to your Droplet without running SSH.
For virtual machines or managed servers hosted by other providers, you should keep in mind that losing SSH connectivity is a risk, particularly if you don’t have another means of remotely connecting to the system’s console. For other systems under your control, remember that it’s safest to perform major operating system upgrades only when you have direct physical access to the machine.
At the prompt, type y and press Enter to continue:
Reading cache
Checking package manager
Continue running under SSH?
This session appears to be running under ssh. It is not recommended
to perform a upgrade over ssh currently because in case of failure it
is harder to recover.
If you continue, an additional ssh daemon will be started at port
'1022'.
Do you want to continue?
Continue [yN] y
Next, you’ll be informed that do-release-upgrade
is starting a new instance of sshd
on port 1022:
Starting additional sshd
To make recovery in case of failure easier, an additional sshd will
be started on port '1022'. If anything goes wrong with the running
ssh you can still connect to the additional one.
If you run a firewall, you may need to temporarily open this port. As
this is potentially dangerous it's not done automatically. You can
open the port with e.g.:
'iptables -I INPUT -p tcp --dport 1022 -j ACCEPT'
To continue please press [ENTER]
Press Enter. Next, you may be warned that a mirror entry was not found. On DigitalOcean systems, it is safe to ignore this warning and proceed with the upgrade, since a local mirror for 16.04 is in fact available. Enter y:
Updating repository information
No valid mirror found
While scanning your repository information no mirror entry for the
upgrade was found. This can happen if you run an internal mirror or
if the mirror information is out of date.
Do you want to rewrite your 'sources.list' file anyway? If you choose
'Yes' here it will update all 'trusty' to 'xenial' entries.
If you select 'No' the upgrade will cancel.
Continue [yN] y
Once new package lists have been downloaded and changes calculated, you’ll be asked if you want to start the upgrade. Again, enter y to continue:
Do you want to start the upgrade?
6 installed packages are no longer supported by Canonical. You can
still get support from the community.
9 packages are going to be removed. 104 new packages are going to be
installed. 399 packages are going to be upgraded.
You have to download a total of 232 M. This download will take about
46 seconds with your connection.
Installing the upgrade can take several hours. Once the download has
finished, the process cannot be canceled.
Continue [yN] Details [d]y
New packages will now be retrieved, then unpacked and installed. Even if your system is on a fast connection, this will take a while.
During the installation, you may be presented with interactive dialogs for various questions. For example, you may be asked if you want to automatically restart services when required:
In this case, it is safe to answer “Yes”. In other cases, you may be asked if you wish to replace a configuration file that you have modified with the default version from the package that is being installed. This is often a judgment call, and is likely to require knowledge about specific software that is outside the scope of this tutorial.
Once new packages have finished installing, you’ll be asked whether you’re ready to remove obsolete packages. On a stock system with no custom configuration, it should be safe to enter y here. On a system you have modified heavily, you may wish to enter d and inspect the list of packages to be removed, in case it includes anything you’ll need to reinstall later.
Remove obsolete packages?
53 packages are going to be removed.
Continue [yN] Details [d]y
Finally, assuming all has gone well, you’ll be informed that the upgrade is complete and a restart is required. Enter y to continue:
System upgrade is complete.
Restart required
To finish the upgrade, a restart is required.
If you select 'y' the system will be restarted.
Continue [yN] y
On an SSH session, you’ll likely see something like the following:
=== Command detached from window (Thu Apr 7 13:13:33 2016) ===
=== Command terminated normally (Thu Apr 7 13:13:43 2016) ===
You may need to press a key here to exit to your local prompt, since your SSH session will have terminated on the server end. Wait a moment for your system to reboot, and reconnect. On login, you should be greeted by a message confirming that you’re now on Xenial Xerus:
Welcome to Ubuntu Xenial Xerus (development branch) (GNU/Linux 4.4.0-17-generic x86_64)
You should now have a working Ubuntu 16.04 installation. From here, you likely need to investigate necessary configuration changes to services and deployed applications. In the coming weeks, we’ll begin posting DigitalOcean guides specific to Ubuntu 16.04 on a wide range of topics.
]]>DigitalOcean’s Private Networking feature gives your Droplets a network interface which is only accessible to other Droplets on the same team or account and located in the same datacenter.
The droplan
utility can help secure private network interfaces on a Droplet by adding iptables firewall rules that only allow traffic from your other Droplets in the same datacenter. By installing and running the utility on each Droplet, you can ensure that your systems will only accept local traffic from one another.
This guide will cover installing droplan
on an individual Droplet, scheduling a cron
job to run it on a regular basis, and ensuring that firewall rules persist when the Droplet is rebooted or loses power.
This guide assumes that you have two or more Linux Droplets in the same region, each configured with a non-root user with sudo
privileges for administrative tasks. Specifically, it provides instructions for recent Debian, Ubuntu, and CentOS releases. On CentOS systems, it will disable firewalld
, so you should be aware that it may override any existing firewall configuration.
The droplan
utility In order to ask the API for a list of your Droplets, the droplan
command needs access to a personal access token with read scope. You can retrieve a token by accessing the DigitalOcean Control Panel, clicking on API in the top menu, and clicking the Generate New Token button. Enter a descriptive name for the new token, in the Token Name field, such as “droplan readonly”, and uncheck the Write (Optional) box:
Click Generate Token, and copy the resulting token to your local machine:
Note: Make sure you keep a copy of the token, or you’ll have to generate a new one. It can’t be retrieved from the Control Panel after the first time it’s shown.
For more details on this process, and the basics of API usage, see How To Use the DigitalOcean API v2.
If you are on Debian or a Debian-derived distribution such as Ubuntu, install the unzip
package using apt-get
:
- sudo apt-get install unzip iptables-persistent
We’ll need iptables-persistent
in a moment when we configure persistent firewall rules. You’ll likely be asked by the installer whether you want to save current firewall rules at the time of installation. It shouldn’t do any harm if you say yes.
If you are using CentOS 7, install the unzip
and iptables-services
package using yum
:
- sudo yum install unzip iptables-services
We’ll need iptables-services
in a moment when we configure persistent firewall rules.
Visit the releases page on the droplan
GitHub project, and find a URL for the latest release which supports your architecture. Copy the URL, log in to one of your Droplets, and retrieve the file with wget
or curl
:
- wget https://github.com/tam7t/droplan/releases/download/v1.0.0/droplan_1.0.0_linux_amd64.zip
Now, use the unzip
command to extract the droplan
binary from the release archive:
- unzip droplan_1.0.0_linux_amd64.zip
Create a directory in /opt
for droplan
, and move the binary there:
- sudo mkdir /opt/droplan
- sudo mv ./droplan /opt/droplan/
The /opt
directory is a standard location for software installed from sources other than a distribution’s official package repositories.
With the droplan
binary in place, you can use it to create rules. Run the command under sudo
, setting the DO_KEY
environment variable to your token:
- sudo DO_KEY=personal_access_token /opt/droplan/droplan
Now, check your iptables rules:
- sudo iptables -L
Assuming that you have two other Droplets in the same region, you should see something like the following:
OutputChain INPUT (policy ACCEPT)
target prot opt source destination
droplan-peers all -- anywhere anywhere
DROP all -- anywhere anywhere
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain droplan-peers (1 references)
target prot opt source destination
ACCEPT all -- droplet_ip1 anywhere
ACCEPT all -- droplet_ip2 anywhere
To confirm that these rules are applied only to eth1, you can add the -v
option for more verbose output, which will include interfaces:
- sudo iptables -L -v
For now, all of your other Droplets in the same region can connect to the current system, while traffic from systems you don’t control is blocked. If the system reboots, however, the iptables rules will disappear. It’s also likely that you will create new Droplets (or delete the existing ones) at some point in the future. In order to deal with these problems, we’ll make sure that rules persist on restart, and schedule droplan
to run on a regular basis and make any necessary changes to the firewall.
Firewall rules are kept in /etc/iptables/rules.v4
(and /etc/iptables/rules.v6
for ipv6 rules). You can generate a new version of this file using the iptables-save
command:
- sudo iptables-save | sudo tee /etc/iptables/rules.v4
By default, CentOS 7 uses the firewalld service in place of iptables. Since we already installed the iptables-services
package above, we can use systemctl
to stop this service and mask it, ensuring that it won’t be restarted:
- sudo systemctl stop firewalld
- sudo systemctl mask firewalld
Now enable the iptables
service:
- systemctl enable iptables
With the iptables
service in place, save the current firewall rules:
- sudo service iptables save
You may wish to reboot the system, reconnect, and check that the rules have persisted. First, reboot:
- sudo reboot
Now, reconnect to your Droplet (this will take a few seconds), and check the rules:
- sudo iptables -L
As a final step, we’ll make sure that droplan
runs periodically so that it catches changes in your collection of Droplets.
Begin by creating a new script called /opt/droplan/refresh.sh
, using nano
(or your editor of choice):
- sudo nano /opt/droplan/refresh.sh
Paste the following, uncommenting the appropriate line for your distribution by deleting the leading #
:
#!/usr/bin/env bash
/opt/droplan/droplan
# Uncomment for Centos:
# service iptables save
# Uncomment for Debian or Ubuntu:
# iptables-save > /etc/iptables/rules.v4
Exit and save the file, then mark it executable:
- sudo chmod +x /opt/droplan/refresh.sh
Next, create a new file at /etc/cron.d/droplan
:
- sudo nano /etc/cron.d/droplan
Add the following line to the file in order to run the script as root every 5 minutes:
*/5 * * * * root PATH=/sbin:/usr/bin:/bin DO_KEY=personal_access_token /opt/droplan/refresh.sh > /var/log/droplan.log 2>&1
This will run the refresh.sh
script once every 5 minutes, as indicated by */5
in the first field, and log its most recent output to /var/log/droplan.log
.
Exit and save the file. You can now use the watch
command, which displays the output of another command every few seconds, to make sure that the script runs successfully:
- sudo watch cat /var/log/droplan.log
Once the script runs, you should see output something like the following:
Sample CentOS OutputEvery 2.0s: cat droplan.log Fri Mar 25 01:14:45 2016
2016/03/25 01:14:02 Added 2 peers to droplan-peers
iptables: Saving firewall rules to /etc/sysconfig/iptables: [ OK ]
On Debian-derived systems, systemctl iptables save
won’t display any output.
Press Ctrl-C to exit watch
.
Note: Since the API is rate-limited, you may need to tune the frequency of updates if you have many Droplets. You can read more about cron
or the API itself.
Now that you’ve configured the firewall on a single Droplet, you’ll want to repeat this process with the rest of your infrastructure. For more than a handful of Droplets, it would probably be easiest to automate this process. If you’re using Hashicorp’s Terraform for provisioning systems, you can find example templates on the Droplan GitHub project. For a broad overview of automating tasks like this one, see An Introduction to Configuration Management.
For more detail on firewalls, see What is a Firewall and How Does It Work?
]]>The DigitalOcean API provides access to most of the features found in the DigitalOcean Control Panel, and offers a straightforward way to manipulate Droplets and other resources from the command line or your own code.
Droplet tagging is a new feature. The feature allows you to group and locate Droplets by applying tags, as well as initiate actions across all the Droplets with a specific tag.
This guide uses the curl
utility and Bash for all examples. It assumes that you are familiar with the use of the DigitalOcean API, and have already generated a personal access token. For details on this process, and the basics of API usage, see How To Use the DigitalOcean API v2.
Once you have a token, begin by setting the $TOKEN
variable in your shell. The export
command ensures that child processes can read the variable:
- export TOKEN=your_personal_access_token
We’ll use $TOKEN
for the rest of the examples in this document, always inside a double-quoted string so that its value, rather than the literal string $TOKEN
, will be interpolated. If you receive errors, first make certain that this value is correctly set in your current shell.
Tags can either be created before they are applied to resources or created and applied during resource creation.
Use curl
to send a POST to the API endpoint, including headers for the Content-Type
, your personal access token, and some JSON data to specify the tag name. Substitute your desired tag name for tag_name
:
curl -X POST \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $TOKEN" \
-d '{"name":"tag_name"}' \
"https://api.digitalocean.com/v2/tags"
Output{"tag":{"name":"tag_name","resources":{"droplets":{"count":0,"last_tagged":null}}}}
Note: This request, like any other request that makes a change to your account, requires that your token has “write” scope assigned to it.
Resources can also include a tags
attribute on creation. This is an array of tag names that will be applied upon creation. Tags that do not already exist will be created to satisfy the request. The only supported resource at the time of this writing is a Droplet, but eventually others will be available.
To create a Droplet and apply tags upon creation, you could type:
curl -X POST \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $TOKEN" \
-d '{"name":"example.com","region":"nyc3","size":"512mb","image":"ubuntu-14-04-x64","tags":["tag_name","another_tag"]}' \
"https://api.digitalocean.com/v2/droplets"
This will create a new Droplet with the tags tag_name
and another_tag
applied. If the example in the previous section was executed, then this command would apply the existing tag_name
tag and create and apply the another_tag
tag to the Droplet.
You can list all of your current tags with a GET request to /v2/tags
:
curl -X GET \
-H "Authorization: Bearer $TOKEN" \
"https://api.digitalocean.com/v2/tags"
Output{"tags":[{"name":"tag_name","resources":{"droplets":{"count":0,"last_tagged":null}}}],"links":{},"meta":{"total":1}}
To view an individual tag, use a GET request to /v2/tags/tag_name
:
curl -X GET \
-H "Authorization: Bearer $TOKEN" \
"https://api.digitalocean.com/v2/tags/tag_name"
Output{"tag":{"name":"tag_name","resources":{"droplets":{"count":0,"last_tagged":null}}}}
The example output above is brief. Notice that the resources.droplets.last_tagged
property is null
. Once you associate a tag with one or more Droplets, this property will contain detailed information about the last-tagged Droplet.
Tags can also be applied to existing resources. The only supported resource at the time of this writing is a Droplet, but eventually others will be available.
Droplets are associated to tags using their id
attribute. You can retrieve a JSON object containing a droplets
array listing all of your Droplets with a GET request to /v2/droplets
:
curl -X GET \
-H "Authorization: Bearer $TOKEN" \
"https://api.digitalocean.com/v2/droplets"
Once you know a Droplet’s id
, you can associate it with a tag by POSTing to /v2/tags/tag_name/resources
, including JSON data which sets resource_id
to the Droplet id
and resource_type
to the string droplet
:
curl -X POST \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $TOKEN" \
-d '{"resources":[{"resource_id":"droplet_id","resource_type":"droplet"}]}' \
"https://api.digitalocean.com/v2/tags/tag_name/resources"
Try a GET request for the tag again, and the resources.droplets.last_tagged
property should contain detailed information on the Droplet you just tagged:
curl -X GET \
-H "Authorization: Bearer $TOKEN" \
"https://api.digitalocean.com/v2/tags/tag_name"
To remove a tag from a specific Droplet, you can issue a DELETE request to /v2/tags/tag_name/resources/
, with the same data you used to tag the Droplet in the first place:
curl -X DELETE \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $TOKEN" \
-d '{"resources":[{"resource_id":"droplet_id","resource_type":"droplet"}]}' \
"https://api.digitalocean.com/v2/tags/tag_name/resources"
This will remove the tag from the resource.
To find all Droplets associated with a particular tag, issue a GET request to /v2/droplets?tag_name=tag_name
:
curl -X GET \
-H "Authorization: Bearer $TOKEN" \
"https://api.digitalocean.com/v2/droplets?tag_name=tag_name"
This will filter your Droplets by the requested tag.
You can perform a number of actions on all the Droplets associated with a specific tag:
Data | Notes |
---|---|
{"type":"power_cycle"} |
Turn Droplets off and back on again. |
{"type":"power_on"} |
Power Droplets on. Must be off. |
{"type":"power_off"} |
Power Droplets off. Must be on. |
{"type":"shutdown"} |
Shutdown Droplets, similar to powering down from the command line. |
{"type":"enable_private_networking"} |
Enable private networking. |
{"type":"enable_ipv6"} |
Enable IPv6 addresses for Droplets. |
{"type":"enable_backups"} |
Enable backups for Droplets. |
{"type":"disable_backups"} |
Disable backups. |
{"type":"snapshot, "name": "snapshot_name"} |
Take snapshots of Droplets. Droplets must be powered off first, and a name is mandatory. |
In order to perform an action, send a POST to /v2/droplets/actions?tag_name=tag_name
with JSON data specifying a type
and any additional values required by the action:
curl -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{"type":"action_type"}' \
"https://api.digitalocean.com/v2/droplets/actions?tag_name=tag_name"
You can retrieve a history of recent actions, including their completion status, with a GET request to /v2/actions
:
curl -X GET \
-H "Authorization: Bearer $TOKEN" \
"https://api.digitalocean.com/v2/actions"
This is a useful way to determine whether actions have completed or are still in-progress.
Suppose that you have a collection of Droplets associated with a tag named fileserver
, and you want to snapshot all of them.
First issue a shutdown
action:
curl -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{"type":"shutdown"}' \
"https://api.digitalocean.com/v2/droplets/actions?tag_name=fileserver"
Wait for the shutdown to finish on all Droplets, and issue a snapshot
action, including a name
value for the snapshot:
curl -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{"type":"snapshot", "name":"snapshot_name"}' \
"https://api.digitalocean.com/v2/droplets/actions?tag_name=fileserver"
Bear in mind that snapshots may take an hour or more to complete, depending on the size of the Droplet. Once the snapshots have finished, you can bring the Droplets back online with a power_on
action:
curl -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{"type":"power_on"}' \
"https://api.digitalocean.com/v2/droplets/actions?tag_name=fileserver"
This will start the Droplet again.
You can delete a tag itself, and remove its association to all resources, with a DELETE request to /v2/tags/tag_name
:
curl -X DELETE \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $TOKEN" \
"https://api.digitalocean.com/v2/tags/tag_name"
The tag will be completely removed.
Tagging is a simple abstraction, but in combination with basic scripting tools, it can offer a powerful mechanism for inventorying and managing your systems.
From here, you may wish to dig deeper into the detailed DigitalOcean API documentation, or investigate libraries which wrap the API for popular programming languages.
]]>Corosync is an open source cluster engine used to implement high availability within applications. Commonly referred to as a messaging layer, Corosync provides a cluster membership and closed communication model for creating replicated state machines, on top of which cluster resource managers like Pacemaker can run. Corosync can be seen as the underlying system that connects the cluster nodes together, while Pacemaker monitors the cluster and takes action in the event of a failure.
This tutorial will demonstrate how to use Corosync and Pacemaker to create a high availability (HA) infrastructure on DigitalOcean with CentOS 7 servers and Reserved IPs. To facilitate the process of setting up and managing the cluster nodes, we are going to use PCS, a command line interface that interacts with both Corosync and Pacemaker. ##Prerequisites
In order to follow this guide, you will need:
When creating these Droplets, use descriptive hostnames to uniquely identify them. For this tutorial, we will refer to these Droplets as primary and secondary.
When you are ready to move on, make sure you are logged into both of your servers with your sudo
user.
##Step 1 — Set Up Nginx
To speed things up, we are going to use a simple shell script that installs Nginx and sets up a basic web page containing information about that specific server. This way we can easily identify which server is currently active in our Reserved IP setup. The script uses DigitalOcean’s Metadata service to fetch the Droplet’s IP address and hostname.
In order to execute the script, run the following commands on both servers:
- sudo curl -L -o install.sh http://do.co/nginx-centos
- sudo chmod +x install.sh
- sudo ./install.sh
After the script is finished running, accessing either Droplet via its public IP address from a browser should give you a basic web page showing the Droplet’s hostname and IP address.
In order to reduce this tutorial’s complexity, we will be using simple web servers as cluster nodes. In a production environment, the nodes would typically be configured to act as redundant load balancers. For more information about load balancers, check out our Introduction to HAProxy and Load Balancing Concepts guide.
##Step 2 — Create and Assign Reserved IP The first step is to create a Reserved IP and assign it to the primary server. In the DigitalOcean Control Panel, click Networking in the top menu, then Reserved IPs in the side menu.
You should see a page like this:
Select your primary server and click on the “Assign Reserved IP” button. After the Reserved IP has been assigned, check that you can reach the primary Droplet by accessing the Reserved IP address from your browser:
http://your_reserved_ip
You should see the index page of your primary Droplet. ##Step 3 — Create IP Reassignment Script In this step, we’ll demonstrate how the DigitalOcean API can be used to reassign a Reserved IP to another Droplet. Later on, we will configure Pacemaker to execute this script when the cluster detects a failure in one of the nodes.
For our example, we are going to use a basic Python script that takes a Reserved IP address and a Droplet ID as arguments in order to assign the Reserved IP to the given Droplet. The Droplet’s ID can be fetched from within the Droplet itself using the Metadata service.
Let’s start by downloading the assign-ip
script and making it executable. Feel free to review the contents of the script before downloading it.
The following two commands should be executed on both servers (primary and secondary):
- sudo curl -L -o /usr/local/bin/assign-ip http://do.co/assign-ip
- sudo chmod +x /usr/local/bin/assign-ip
The assign-ip
script requires the following information in order to be executed:
###Testing the IP Reassignment Script
To monitor the IP reassignment taking place, we can use a curl
command to access the Reserved IP address in a loop, with an interval of 1 second between each request.
Open a new local terminal and run the following command, making sure to replace reserved_IP_address with your actual Reserved IP address:
- while true; do curl reserved_IP_address; sleep 1; done
This command will keep running in the active terminal until interrupted with a CTRL+C
. It simply fetches the web page hosted by the server that your Reserved IP is currently assigned to. The output should look like this:
OutputDroplet: primary, IP Address: primary_IP_address
Droplet: primary, IP Address: primary_IP_address
Droplet: primary, IP Address: primary_IP_address
...
Now, let’s run the assign-ip
script to reassign the Reserved IP to the secondary droplet. We will use DigitalOcean’s Metadata service to fetch the current Droplet ID and use it as an argument to the script. Fetching the Droplet’s ID from the Metadata service can be done with:
- curl -s http://169.254.169.254/metadata/v1/id
Where 169.254.169.254
is a static IP address used by the Metadata service, and therefore should not be modified. This information is only available from within the Droplet itself.
Before we can execute the script, we need to set the DO_TOKEN environment variable containing the DigitalOcean API token. Run the following command from the secondary server, and don’t forget to replace your_api_token with your read/write Personal Access Token to the DigitalOcean API:
- export DO_TOKEN=your_api_token
Still on the secondary server, run the assign-ip
script replacing reserved_IP_address with your Reserved IP address:
- assign-ip reserved_IP_address `curl -s http://169.254.169.254/metadata/v1/id`
OutputMoving IP address: in-progress
By monitoring the output produced by the curl
command on your local terminal, you will notice that the Reserved IP will change its assigned IP address and start pointing to the secondary Droplet after a few seconds:
OutputDroplet: primary, IP Address: primary_IP_address
Droplet: primary, IP Address: primary_IP_address
Droplet: secondary, IP Address: secondary_IP_address
You can also access the Reserved IP address from your browser. You should get a page showing the secondary Droplet information. This means that the reassignment script worked as expected.
To reassign the Reserved IP back to the primary server, repeat the 2-step process but this time from the primary Droplet:
- export DO_TOKEN=your_api_token
- assign-ip reserved_IP_address `curl -s http://169.254.169.254/metadata/v1/id`
After a few seconds, the Reserved IP should be pointing to your primary Droplet again. ##Step 4 — Install Corosync, Pacemaker and PCS The next step is to get Corosync, Pacemaker and PCS installed on your Droplets. Because Corosync is a dependency to Pacemaker, it’s usually a better idea to simply install Pacemaker and let the system decide which Corosync version should be installed.
Install the software packages on both servers:
- sudo yum install pacemaker pcs
The PCS utility creates a new system user during installation, named hacluster, with a disabled password. We need to define a password for this user on both servers. This will enable PCS to perform tasks such as synchronizing the Corosync configuration on multiple nodes, as well as starting and stopping the cluster.
On both servers, run:
- passwd hacluster
You should use the same password on both servers. We are going to use this password to configure the cluster in the next step.
The user hacluster has no interactive shell or home directory associated with its account, which means it’s not possible to log into the server using its credentials.
##Step 5 — Set Up the Cluster Now that we have Corosync, Pacemaker and PCS installed on both servers, we can set up the cluster. ###Enabling and Starting PCS To enable and start the PCS daemon, run the following on both servers:
- sudo systemctl enable pcsd.service
- sudo systemctl start pcsd.service
###Obtaining the Private Network IP Address for Each Node For improved network performance and security, the nodes should be connected using the private network. The easiest way to obtain the Droplet’s private network IP address is via the Metadata service. On each server, run the following command:
- curl http://169.254.169.254/metadata/v1/interfaces/private/0/ipv4/address && echo
This command will simply output the private network IP address of the Droplet you’re logged in. You can also find this information on your Droplet’s page at the DigitalOcean Control Panel (under the Settings tab).
Collect the private network IP address from both Droplets for the next steps. ###Authenticating the Cluster Nodes Authenticate the cluster nodes using the username hacluster and the same password you defined on step 3. You’ll need to provide the private network IP address for each node. From the primary server, run:
- sudo pcs cluster auth primary_private_IP_address secondary_private_IP_address
You should get output like this:
OutputUsername: hacluster
Password:
primary_private_IP_address: Authorized
secondary_private_IP_address: Authorized
###Generating the Corosync Configuration Still on the primary server, generate the Corosync configuration file with the following command:
- sudo pcs cluster setup --name webcluster \
- primary_private_IP_address secondary_private_IP_address
The output should look similar to this:
OutputShutting down pacemaker/corosync services...
Redirecting to /bin/systemctl stop pacemaker.service
Redirecting to /bin/systemctl stop corosync.service
Killing any remaining services...
Removing all cluster configuration files...
primary_private_IP_address: Succeeded
secondary_private_IP_address: Succeeded
Synchronizing pcsd certificates on nodes primary_private_IP_address, secondary_private_IP_address...
primary_private_IP_address: Success
secondary_private_IP_address: Success
Restaring pcsd on the nodes in order to reload the certificates...
primary_private_IP_address: Success
secondary_private_IP_address: Success
This will generate a new configuration file located at /etc/corosync/corosync.conf
based on the parameters provided to the pcs cluster setup
command. We used webcluster as the cluster name in this example, but you can use the name of your choice.
###Starting the Cluster
To start the cluster you just set up, run the following command from the primary server:
- sudo pcs cluster start --all
Outputprimary_private_IP_address: Starting Cluster...
secondary_private_IP_address: Starting Cluster...
You can now confirm that both nodes joined the cluster by running the following command on any of the servers:
- sudo pcs status corosync
OutputMembership information
----------------------
Nodeid Votes Name
2 1 secondary_private_IP_address
1 1 primary_private_IP_address (local)
To get more information about the current status of the cluster, you can run:
- sudo pcs cluster status
The output should be similar to this:
OutputCluster Status:
Last updated: Fri Dec 11 11:59:09 2015 Last change: Fri Dec 11 11:59:00 2015 by hacluster via crmd on secondary
Stack: corosync
Current DC: secondary (version 1.1.13-a14efad) - partition with quorum
2 nodes and 0 resources configured
Online: [ primary secondary ]
PCSD Status:
primary (primary_private_IP_address): Online
secondary (secondary_private_IP_address): Online
Now you can enable the corosync
and pacemaker
services to make sure they will start when the system boots. Run the following on both servers:
- sudo systemctl enable corosync.service
- sudo systemctl enable pacemaker.service
###Disabling STONITH STONITH (Shoot The Other Node In The Head) is a fencing technique intended to prevent data corruption caused by faulty nodes in a cluster that are unresponsive but still accessing application data. Because its configuration depends on a number of factors that are out of scope for this guide, we are going to disable STONITH in our cluster setup.
To disable STONITH, run the following command on one of the Droplets, either primary or secondary:
- sudo pcs property set stonith-enabled=false
##Step 6 — Create Reserved IP Reassignment Resource Agent The only thing left to do is to configure the resource agent that will execute the IP reassignment script when a failure is detected in one of the cluster nodes. The resource agent is responsible for creating an interface between the cluster and the resource itself. In our case, the resource is the assign-ip script. The cluster relies on the resource agent to execute the right procedures when given a start, stop or monitor command. There are different types of resource agents, but the most common one is the OCF (Open Cluster Framework) standard.
We will create a new OCF resource agent to manage the assign-ip service on both servers.
First, create the directory that will contain the resource agent. The directory name will be used by Pacemaker as an identifier for this custom agent. Run the following on both servers:
- sudo mkdir /usr/lib/ocf/resource.d/digitalocean
Next, download the FloatIP resource agent script and place it in the newly created directory, on both servers:
- sudo curl -L -o /usr/lib/ocf/resource.d/digitalocean/floatip http://do.co/ocf-floatip
Now make the script executable with the following command on both servers:
- sudo chmod +x /usr/lib/ocf/resource.d/digitalocean/floatip
We still need to register the resource agent within the cluster, using the PCS utility. The following command should be executed from one of the nodes (don’t forget to replace your_api_token with your DigitalOcean API token and reserved_IP_address with your actual Reserved IP address):
- sudo pcs resource create FloatIP ocf:digitalocean:floatip \
- params do_token=your_api_token \
- reserved_ip=reserved_IP_address
The resource should now be registered and active in the cluster. You can check the registered resources from any of the nodes with the pcs status
command:
- sudo pcs status
Output...
2 nodes and 1 resource configured
Online: [ primary secondary ]
Full list of resources:
FloatIP (ocf::digitalocean:floatip): Started primary
...
##Step 7 — Test Failover Your cluster should now be ready to handle a node failure. A simple way to test failover is to restart the server that is currently active in your Reserved IP setup. If you’ve followed all steps in this tutorial, this should be the primary server.
Again, let’s monitor the IP reassignment by using a curl
command in a loop. From a local terminal, run:
- while true; do curl reserved_IP_address; sleep 1; done
From the primary server, run a reboot command:
- sudo reboot
After a few moments, the primary server should become unavailable. This will cause the secondary server to take over as the active node. You should see output similar to this in your local terminal running curl
:
Output...
Droplet: primary, IP Address: primary_IP_address
Droplet: primary, IP Address: primary_IP_address
curl: (7) Failed connect to reserved_IP_address; Connection refused
Droplet: secondary, IP Address: secondary_IP_address
Droplet: secondary, IP Address: secondary_IP_address
…
The “Connection refused” error happens when the request is made right before or at the same time when the IP reassignment is taking place. It may or may not show up in the output.
If you want to point the Reserved IP back to the primary node while also testing failover on the secondary node, just repeat the process but this time from the secondary Droplet:
- sudo reboot
##Conclusion
In this guide, we saw how Reserved IPs can be used together with Corosync, Pacemaker and PCS to create a highly available web server environment on CentOS 7 servers. We used a rather simple infrastructure to demonstrate the usage of Reserved IPs, but this setup can be scaled to implement high availability at any level of your application stack.
]]>Heartbeat is an open source program that provides cluster infrastructure capabilities—cluster membership and messaging—to client servers, which is a critical component in a high availability (HA) server infrastructure. Heartbeat is typically used in conjunction with a cluster resource manager (CRM), such as Pacemaker, to achieve a complete HA setup. However, in this tutorial, we will demonstrate how to create a 2-node HA server setup by simply using Heartbeat and a DigitalOcean Reserved IP.
If you are looking to create a more robust HA setup, look into using Corosync and Pacemaker or Keepalived.
When completed, the HA setup will consist of two Ubuntu 14.04 servers in an active/passive configuration. This will be accomplished by pointing a Reserved IP, which is how your users will access your services or website, to point to the primary, or active, server unless a failure is detected. In the event that the Heartbeat service detects that the primary server is unavailable, the secondary server will automatically run a script to reassign the Reserved IP to itself via the DigitalOcean API. Thus, subsequent network traffic to the Reserved IP will be directed to your secondary server, which will act as the active server until the primary server becomes available again (at which point, the primary server will reassign the Reserved IP to itself).
Note: This tutorial only covers setting up active/passive high availability at the gateway level. That is, it includes the Reserved IP, and the load balancer servers—Primary and Secondary. Furthermore, for demonstration purposes, instead of configuring reverse-proxy load balancers on each server, we will simply configure them to respond with their respective hostname and public IP address.
To achieve this goal, we will follow these steps:
In order to automate the Reserved IP reassignment, we must use the DigitalOcean API. This means that you need to generate a Personal Access Token (PAT), which is an API token that can be used to authenticate to your DigitalOcean account, with read and write access by following the How To Generate a Personal Access Token section of the API tutorial. Your PAT will be used in a script that will be added to both servers in your cluster, so be sure to keep it somewhere safe—as it allows full access to your DigitalOcean account—for reference.
In addition to the API, this tutorial utilizes the following DigitalOcean features:
Please read the linked tutorials if you want to learn more about them.
The first step is to create two Ubuntu Droplets in the same datacenter, which will act as the primary and secondary servers described above. In our example setup, we will name them “primary” and “secondary” for easy reference. We will install Nginx on both Droplets and replace their index pages with information that uniquely identifies them. This will allow us a simple way to demonstrate that the HA setup is working. For a real setup, your servers should run the web server or load balancer of your choice.
Create two Ubuntu 14.04 Droplets, primary and secondary, with this bash script as the user data:
#!/bin/bash
apt-get -y update
apt-get -y install nginx
export HOSTNAME=$(curl -s http://169.254.169.254/metadata/v1/hostname)
export PUBLIC_IPV4=$(curl -s http://169.254.169.254/metadata/v1/interfaces/public/0/ipv4/address)
echo Droplet: $HOSTNAME, IP Address: $PUBLIC_IPV4 > /usr/share/nginx/html/index.html
This will install Nginx and replace the contents of index.html
with the droplet’s hostname and IP address (by referencing the Metadata service). Accessing either Droplet via its public IP address will show a basic webpage with the Droplet hostname and IP address, which will be useful for testing which Droplet the Reserved IP is pointing to at any given moment.
In the DigitalOcean Control Panel, click Networking, in the top menu, then Reserved IPs in the side menu.
Assign a Reserved IP to your primary Droplet, then click the Assign Reserved IP button.
After the Reserved IP has been assigned, check that you can reach the Droplet that it was assigned to by visiting it in a web browser.
http://your_reserved_ip
You should see the index page of your primary Droplet.
If you want to be able to access your HA setup via a domain name, go ahead and create an A record in your DNS that points your domain to your Reserved IP address. If your domain is using DigitalOcean’s nameservers, follow step three of the How To Set Up a Host Name with DigitalOcean tutorial. Once that propagates, you may access your active server via the domain name.
The example domain name we’ll use is example.com
. If you don’t have a domain name right now, you should use the Reserved IP address instead.
The next step is to install Heartbeat on both servers. The simplest way to install Heartbeat is to use apt-get:
sudo apt-get update
sudo apt-get install heartbeat
Heartbeat is now installed but it needs to be configured before it will do anything.
In order to get our desired cluster up and running, we must set up these Heartbeat configuration files in /etc/ha.d
, identically on both servers:
We will also need to provide a script that will perform the Reserved IP reassignment in the event that the primary Droplet’s availability changes.
Before configuring ha.cf
, we should look up the names of each node. Heartbeat requires that each node name matches their respective uname -n
output.
On both servers, run this command to look up the appropriate node names:
- uname -n
Note the output of the command. The example node names are “primary” and “secondary”, which matches what we named the Droplets.
We will also need to look up the network interface and IP address that each node will use to communicate with the rest of the cluster, to determine which nodes are available. You may use any network interface, as long as each node can reach the other nodes in the cluster. We’ll use the public interface of our Droplets, which happens to be eth0
.
On both servers, use this command to look up the IP address of the eth0
interface (or look it up in the DigitalOcean Control Panel):
- ip addr show eth0
ip addr show eth0 output:2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 04:01:76:a5:45:01 brd ff:ff:ff:ff:ff:ff
inet 104.236.6.11/18 brd 104.236.63.255 scope global eth0
valid_lft forever preferred_lft forever
inet 10.17.0.28/16 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::601:76ff:fea5:4501/64 scope link
valid_lft forever preferred_lft forever
Note the IP address of the network interface (highlighted in the example). Be sure to get the IP addresses of both servers.
On both servers, open /etc/ha.d/ha.cf
in your favorite editor. We’ll use vi
:
- sudo vi /etc/ha.d/ha.cf
The file should be new and empty. We need to add the network interfaces and names of each node in our cluster.
Copy and paste this configuration into the file, then replace the respective node names and IP addresses with the values that we looked up earlier. In this example, primary’s IP address is 104.236.6.11
and secondary’s IP address is 104.236.6.22
:
node primary
ucast eth0 104.236.6.11
node secondary
ucast eth0 104.236.6.22
Save and exit. Next, we’ll set up the cluster’s authorization key.
The authorization key is used to allow cluster members to join a cluster. We can simply generate a random key for this purpose.
On the primary node, run these commands to generate a suitable authorization key in an environment variable named AUTH_KEY
:
if [ -z "${AUTH_KEY}" ]; then
export AUTH_KEY="$(command dd if='/dev/urandom' bs=512 count=1 2>'/dev/null' \
| command openssl sha1 \
| command cut --delimiter=' ' --fields=2)"
fi
Then write the /etc/ha.d/authkeys
file with these commands:
sudo bash -c "{
echo auth1
echo 1 sha1 $AUTH_KEY
} > /etc/ha.d/authkeys"
Check the contents of the authkeys
file like this:
- sudo cat /etc/ha.d/authkeys
It should like something like this (with a different authorization key):
/etc/ha.d/authkeys example:auth1
1 sha1 d1e6557e2fcb30ff8d4d3ae65b50345fa46a2faa
Ensure that the file is only readable by root:
- sudo chmod 600 /etc/ha.d/authkeys
Now copy the /etc/ha.d/authkeys
file from your primary node to your secondary node. You can do this manually, or with scp
.
On the secondary server, be sure to set the permissions of the authkeys
file:
- sudo chmod 600 /etc/ha.d/authkeys
Both servers should have an identical /etc/ha.d/authkeys
file.
The haresources
file specifies preferred hosts paired with services that the cluster manages. The preferred host is the node that should run the associated service(s) if the node is available. If the preferred host is not available, i.e. it is not reachable by the cluster, one of the other nodes will take over. In other words, the secondary server will take over if the primary server goes down.
On both servers, open the haresources
file in your favorite editor. We’ll use vi
:
- sudo vi /etc/ha.d/haresources
Now add this line to the file, substituting in your primary node’s name:
- primary floatip
Save and exit. This configures the primary server as the preferred host for the floatip
service, which is currently undefined. Let’s set up the floatip
service next.
Our Heartbeat cluster is configured to maintain the floatip
service, which a node can use to assign the Reserved IP to itself, but we still need to create the service. Before we set up the service itself, however, let’s create a script that will assign the Reserved IP, via the DigitalOcean API, to the node that runs it. Then we will create the floatip
service which will run the Reserved IP reassignment script.
For our example, we’ll download a basic Python script that assigns a Reserved IP to a given Droplet ID, using the DigitalOcean API.
On both servers, download the assign-ip
Python script:
- sudo curl -L -o /usr/local/bin/assign-ip http://do.co/assign-ip
On both servers, make it executable:
- sudo chmod +x /usr/local/bin/assign-ip
Use of the assign-ip
script requires the following details:
DO_TOKEN
, your read/write DigitalOcean PATFeel free to review the contents of the script before continuing.
Now we’re ready to create the floatip
service.
To create the floatip
service, all we need to do is create an init script that invokes the assign-ip
script that we created earlier, and responds to start
and stop
subcommands. This init script will be responsible for looking up the Droplet ID of the server, via the Droplet Metadata service. Also, it will require the Reserved IP that will be reassigned, and the DigitalOcean API token (the Personal Access Token mentioned in the prerequisites section).
On both servers, add open /etc/init.d/floatip
in an editor:
- sudo vi /etc/init.d/floatip
Then copy and paste in this init script, replacing the highlighted parts with your DigitalOcean API key and the Reserved IP that should be reassigned:
- #!/bin/bash
-
- param=$1
-
- export DO_TOKEN='b7d03a6947b217efb6f3ec3bd3504582'
- IP='45.55.96.8'
- ID=$(curl -s http://169.254.169.254/metadata/v1/id)
-
- if [ "start" == "$param" ] ; then
- python /usr/local/bin/assign-ip $IP $ID
- exit 0
- elif [ "stop" == "$param" ] ; then
- exit 0;
- elif [ "status" == "$param" ] ; then
- exit 0;
- else
- echo "no such command $param"
- exit 1;
- fi
Save and exit.
Make the script executable:
- sudo chmod u+x /etc/init.d/floatip
When this floatip
service is started, it will simply call the assign-ip
Python script and assign the specified Reserved IP to the Droplet that executed the script. This is the script that will be called by the secondary server, to reassign the Reserved IP to itself, if the primary server fails. Likewise, the same script will be used by the primary server, to reclaim the Reserved IP, once it rejoins the cluster.
Now that Heartbeat is configured, and all of the scripts it relies on are set up, we’re ready to start the Heartbeat cluster!
On both servers, run this command to start Heartbeat:
- sudo service heartbeat start
You should see output like this:
Heartbeat output:Starting High-Availability services: Done.
Our HA setup is now complete! Before moving on, let’s test that it works as intended.
It’s important to test that a high availability setup works, so let’s do that now.
Currently, the Reserved IP is assigned to the primary node. Accessing the Reserved IP now, via the IP address or by the domain name that is pointing to it, will simply show the index page of the primary server. If you used the example user data script, it will look something like this:
Reserved IP is pointing to primary serverDroplet: primary, IP Address: 104.236.6.11
This indicates that the Reserved IP is, in fact, assigned to the primary Droplet.
Now, let’s open a terminal and use curl
to access the Reserved IP on a 1 second loop. Use this command to do so, but be sure to replace the URL with your domain or Reserved IP address:
- while true; do curl http://example.com; sleep 1; done
Currently, this will output the same Droplet name and IP address of the primary server. If we cause the primary server to fail, by powering it off or stopping the Heartbeat service, we will see if the Reserved IP gets reassigned to the secondary server.
Let’s power off the primary server now. Do so via the DigitalOcean Control Panel or by running this command on the primary server:
- sudo poweroff
After a few moments, the primary server should become unavailable. Pay attention to the output of the curl
loop that is running in the terminal. You should notice output that looks like this:
curl loop output:Droplet: primary, IP Address: 104.236.6.11
...
curl: (7) Failed to connect to example.com port 80: Connection refused
Droplet: secondary, IP Address: 104.236.6.22
Droplet: secondary, IP Address: 104.236.6.22
...
That is, the Reserved IP address should be reassigned to point to the IP address of the secondary server. That means that your HA setup is working, as a successful automatic failover has occurred.
You may or may not see the Connection refused
error, which can occur if you try and access the Reserved IP between the primary server failure and the Reserved IP reassignment completion.
Now, you may power on your primary Droplet, via the DigitalOcean Control Panel. Because Heartbeat is configured with the primary Droplet as the preferred host to run the Reserved IP reassignment script, the Reserved IP will automatically point back to the primary server as soon as it becomes available again.
Congratulations! You now have a basic HA server setup using Heartbeat and a DigitalOcean Reserved IP.
If you are looking to create a more robust HA setup, look into using Corosync and Pacemaker or Keepalived.
If you want to extend your Heartbeat setup, the next step is to replace the example Nginx setup with a reverse-proxy load balancer. You can use Nginx or HAProxy for this purpose. Keep in mind that you will want to bind your load balancer to the anchor IP address, so that your users can only access your servers via the Reserved IP address (and not via the public IP address of each server).
]]>In this guide, we will create and deploy a scalable WordPress instance consisting of a MySQL database server, a GlusterFS distributed filesystem, Nginx web servers, and an Nginx load balancer. By using user-data and Droplet meta-data, we will automate the deployment of our site. Finally, we will provide a Ruby script which will automate this entire process and ease the creation of scalable Wordpress sites. Through this tutorial, you will learn about the power and flexibility of user-data and Droplet meta-data in deploying services on DigitalOcean.
The deployment we create in this tutorial will consist of a single MySQL database server, multiple GlusterFS servers in a cluster, multiple Nginx web servers, and a single Nginx load balancer.
Before we begin we should know:
We can add additional nodes or scale up the nodes we created if we need to later. Once we have decided on these details, we can begin deploying our site.
We will begin by deploying our MySQL server. To do this, we will create a default Ubuntu 14.04 x64 Droplet using the following user-data.
#!/bin/bash
export DEBIAN_FRONTEND=noninteractive;
export PUBLIC_IP=$(curl -s http://169.254.169.254/metadata/v1/interfaces/public/0/ipv4/address)
export PRIVATE_IP=$(curl -s http://169.254.169.254/metadata/v1/interfaces/private/0/ipv4/address)
apt-get update;
apt-get -y install mysql-server;
mysqladmin -u root create wordpress;
mysqladmin -u root password "mysql_password";
sed -i.bak "s/127.0.0.1/$PRIVATE_IP/g" /etc/mysql/my.cnf;
service mysql restart;
mysql -uroot -pmysql_password -e "CREATE USER 'wordpress'@'%' IDENTIFIED BY 'mysql_password'";
mysql -uroot -pmysql_password -e "GRANT ALL PRIVILEGES ON wordpress.* TO 'wordpress'@'%'";
This user-data script will perform the following functions on our new Droplet:
First, we export a variable which tells apt-get
that we are running in non-interactive mode to prevent it from prompting for any input when it installs packages.
export DEBIAN_FRONTEND=noninteractive;
Next, we use Droplet meta-data to get the Droplet’s public and private IP addresses and assign them to variables:
export PUBLIC_IP=$(curl -s http://169.254.169.254/metadata/v1/interfaces/public/0/ipv4/address)
export PRIVATE_IP=$(curl -s http://169.254.169.254/metadata/v1/interfaces/private/0/ipv4/address)
Note: Droplet Meta-Data is not available in NYC1, NYC2, and AMS1 at this time.
We then use apt
to install the MySQL server.
apt-get update;
apt-get -y install mysql-server;
Now we need to create a new database called wordpress.
mysqladmin -u root create wordpress;
Then we set a password for our MySQL root user.
mysqladmin -u root password "mysql_password";
Because our MySQL server will be accepting queries from our web servers, we need to have it listen on the private IP address rather than only on localhost
. To do this, we will use sed
to update the MySQL configuration file by doing a find and replace and then restart the service.
sed -i.bak "s/127.0.0.1/$PRIVATE_IP/g" /etc/mysql/my.cnf;
service mysql restart;
Finally, we will create a new MySQL user called wordpress and give it permission to access the wordpress database.
mysql -uroot -pmysql_password -e "CREATE USER 'wordpress'@'%' IDENTIFIED BY 'mysql_password'";
mysql -uroot -pmysql_password -e "GRANT ALL PRIVILEGES ON wordpress.* TO 'wordpress'@'%'";
By deploying our new Droplet with this user-data script, we will have a configured MySQL server listening on its private IP address and with our configured database and user without ever logging in via SSH or the console.
Before deploying our GlusterFS cluster, we need to decide how many nodes we will deploy. There are two variables that will go into this decision. First, we need to decide how much space we need and then we need to decide on a replica setting to use. The replica setting tells GlusterFS how many copies of any file to store. For example, a replica setting of 2 will mean that every file is duplicated on at least 2 servers. This will cut our available storage in half since we are keeping two copies of each file but will provide improved redundancy. The number of GlusterFS nodes we create must be a multiple of our replica setting. For a cluster with a replica setting of 2, we will need to create our nodes in a multiple of 2 (so 2, 4, 6, or 8 nodes would be acceptable).
For this example, we will deploy a 4 node GlusterFS cluster using a replica setting of 2.
For our first 3 nodes, we will use the following user-data script:
#!/bin/bash
export DEBIAN_FRONTEND=noninteractive;
apt-get update;
apt-get install -y python-software-properties;
add-apt-repository -y ppa:gluster/glusterfs-3.5;
apt-get update;
apt-get install -y glusterfs-server;
Again, we first set the DEBIAN_FRONTEND
variable so apt
knows that we are running in non-interactive mode:
export DEBIAN_FRONTEND=noninteractive;
We then update our apt
database and install python-software-properties
, which is needed to add the PPA for GlusterFS.
apt-get update;
apt-get install -y python-software-properties;
Next we will add the GlusterFS PPA so we can grab our deb packages.
add-apt-repository -y ppa:gluster/glusterfs-3.5;
Then we will update our apt
database again and install glusterfs-server.
apt-get install -y glusterfs-server;
For our first three nodes, this is all we need to do. Make a note of the private IP addresses assigned to each of these new Droplets as we will need them when creating our final GlusterFS node and creating our volume.
For our final node, we will use the following user-data script:
#!/bin/bash
export DEBIAN_FRONTEND=noninteractive;
export PRIVATE_IP=$(curl -s http://169.254.169.254/metadata/v1/interfaces/private/0/ipv4/address)
apt-get update;
apt-get install -y python-software-properties;
add-apt-repository -y ppa:gluster/glusterfs-3.5;
apt-get update;
apt-get install -y glusterfs-server;
sleep 30;
gluster peer probe node1_private_ip;
gluster peer probe node2_private_ip;
gluster peer probe node3_private_ip;
gluster volume create file_store replica 2 transport tcp node1_private_ip:/gluster node2_private_ip:/gluster node3_private_ip:/gluster $PRIVATE_IP:/gluster force;
gluster volume start file_store;
Note: If you do not want to enable replication you should not include the "replica" setting in your "volume create" command.
The first section of this user-data script is pretty similar to the one we used on the other GlusterFS nodes, though we are assigning our new Droplet’s private IP to the $PRIVATE_IP variable. Once glusterfs-server
is installed, though, we do some additional work.
First, our script will wait 30 seconds for the new glusterfs-server to start up and be available.
sleep 30
Then we probe the three GlusterFS Droplets we created earlier in order to add all four to a cluster.
gluster peer probe node1_private_ip;
gluster peer probe node2_private_ip;
gluster peer probe node3_private_ip;
Next we will create our GlusterFS volume named “file_store” with a replica setting of 2 and including all four of our nodes. Since we wont know the IP address of our newest node yet we will use the $PRIVATE_IP variable for it.
gluster volume create file_store replica 2 transport tcp node1_private_ip:/gluster node2_private_ip:/gluster node3_private_ip:/gluster $PRIVATE_IP:/gluster force;
Finally, we will start the new volume to make it accessible from our clients:
gluster volume start file_store;
We now have a distributed filesystem where we can keep our WordPress files that will be accessible to all our web server nodes.
Now that we have a database server and a distributed filesystem all set, we can deploy our web servers. We will use the following user-data script to deploy our first Nginx web server node and configure our WordPress installation inside our GlusterFS volume.
#!/bin/bash
export DEBIAN_FRONTEND=noninteractive;
export PRIVATE_IP=$(curl -s http://169.254.169.254/metadata/v1/interfaces/private/0/ipv4/address)
apt-get update;
apt-get -y install nginx glusterfs-client php5-fpm php5-mysql;
sed -i s/\;cgi\.fix_pathinfo\=1/cgi\.fix_pathinfo\=0/g /etc/php5/fpm/php.ini;
mkdir /gluster;
mount -t glusterfs gluter_node_private_ip:/file_store /gluster;
echo "gluster_node_private_ip:/file_store /gluster glusterfs defaults,_netdev 0 0" >> /etc/fstab;
mkdir /gluster/www;
wget https://raw.githubusercontent.com/ryanpq/do-wpc/master/default -O /etc/nginx/sites-enabled/default;
service nginx restart;
# Get Wordpress Files
wget https://wordpress.org/latest.tar.gz -O /root/wp.tar.gz;
tar -zxf /root/wp.tar.gz -C /root/;
cp -Rf /root/wordpress/* /gluster/www/.;
cp /gluster/www/wp-config-sample.php /gluster/www/wp-config.php;
sed -i "s/'DB_NAME', 'database_name_here'/'DB_NAME', 'wordpress'/g" /gluster/www/wp-config.php;
sed -i "s/'DB_USER', 'username_here'/'DB_USER', 'wordpress'/g" /gluster/www/wp-config.php;
sed -i "s/'DB_PASSWORD', 'password_here'/'DB_PASSWORD', 'mysql_password'/g" /gluster/www/wp-config.php;
sed -i "s/'DB_HOST', 'localhost'/'DB_HOST', 'mysql_private_ip'/g" /gluster/www/wp-config.php;
chown -Rf www-data:www-data /gluster/www;
This script is a bit more complicated than our previous ones, so let’s break it down step by step.
First, we will again set the DEBIAN_FRONTEND
variable as we have in our previous scripts and populate our $PRIVATE_IP
variable.
export DEBIAN_FRONTEND=noninteractive;
export PRIVATE_IP=$(curl -s http://169.254.169.254/metadata/v1/interfaces/private/0/ipv4/address)
Next, we will update our apt
database and install Nginx, the glusterfs client, and the php libraries we will need.
apt-get update;
apt-get -y install nginx glusterfs-client php5-fpm php5-mysql;
Then we will use sed
’s find and replace functionality to update our php.ini
file and set the cgi.fixpathinfo variable to 0.
sed -i s/\;cgi\.fix_pathinfo\=1/cgi\.fix_pathinfo\=0/g /etc/php5/fpm/php.ini;
Now we’ll create a folder called /gluster
in the root of our disk image and mount our GlusterFS volume there. Then we will create an fstab entry so our GlusterFS volume is automatically mounted when the Droplet boots.
mkdir /gluster;
mount -t glusterfs gluter_node_private_ip:/file_store /gluster;
echo "gluster_node_private_ip:/file_store /gluster glusterfs defaults,_netdev 0 0" >> /etc/fstab;
Then we will create a folder called www
in our GlusterFS volume. This folder will act as our web root.
mkdir /gluster/www;
Next we will pull a new Nginx configuration file from a remote server. This file will set our web root to /gluster/www
and ensure Nginx is configured to use PHP. You can view this configuration file here. Once we have replaced our Nginx configuration file we will restart the service for this change to take effect.
wget https://raw.githubusercontent.com/ryanpq/do-wpc/master/default -O /etc/nginx/sites-enabled/default;
service nginx restart;
Now we will grab a copy of the latest version of WordPress, extract it and copy its contents to our new web root.
wget https://wordpress.org/latest.tar.gz -O /root/wp.tar.gz;
tar -zxf /root/wp.tar.gz -C /root/;
cp -Rf /root/wordpress/* /gluster/www/.;
Next, we will copy the sample WordPress configuration file to wp-config.php
.
cp /gluster/www/wp-config-sample.php /gluster/www/wp-config.php;
And update its variables to match our new environment, again using sed
’s find and replace function.
sed -i "s/'DB_NAME', 'database_name_here'/'DB_NAME', 'wordpress'/g" /gluster/www/wp-config.php;
sed -i "s/'DB_USER', 'username_here'/'DB_USER', 'wordpress'/g" /gluster/www/wp-config.php;
sed -i "s/'DB_PASSWORD', 'password_here'/'DB_PASSWORD', 'mysql_password'/g" /gluster/www/wp-config.php;
sed -i "s/'DB_HOST', 'localhost'/'DB_HOST', 'mysql_private_ip'/g" /gluster/www/wp-config.php;
And finally, we will make sure that the files in our web root are owned by the user www-data which our Nginx process will be running as.
chown -Rf www-data:www-data /gluster/www;
We now have our first web server node all set up and ready to receive requests.
Since each of our web server nodes shares the same GlusterFS volume for storage, there are fewer steps for each additional node we create. For additional nodes we will use the following user-data script:
#!/bin/bash
export DEBIAN_FRONTEND=noninteractive;
export PRIVATE_IP=$(curl -s http://169.254.169.254/metadata/v1/interfaces/private/0/ipv4/address)
apt-get update;
apt-get -y install nginx glusterfs-client php5-fpm php5-mysql;
sed -i s/\;cgi\.fix_pathinfo\=1/cgi\.fix_pathinfo\=0/g /etc/php5/fpm/php.ini;
mkdir /gluster;
mount -t glusterfs gluster_node_private_ip:/file_store /gluster;
echo "gluster_node_private_ip:/file_store /gluster glusterfs defaults,_netdev 0 0" >> /etc/fstab;
mkdir /gluster/www;
wget https://raw.githubusercontent.com/ryanpq/do-wpc/master/default -O /etc/nginx/sites-enabled/default;
service nginx restart;
For our additional web nodes we will still be installing the same packages, mounting our GlusterFS volume and replacing our Nginx configuration file but we will not need to do any setup of our WordPress instance since we did this when creating our first node.
The final step in this deployment is to create our load balancer. We will be using another Nginx server for this purpose. To set up this node we will use the following user-data script:
#!/bin/bash
export DEBIAN_FRONTEND=noninteractive;
apt-get update;
apt-get -y install nginx;
lbconf="
server {
listen 80 default_server;
listen [::]:80 default_server ipv6only=on;
root /usr/share/nginx/html;
index index.php index.html index.htm;
location / {
proxy_pass http://backend;
include proxy_params;
}
}
upstream backend {
ip_hash;
server web_node_1_private_ip
server web_node_2_private_ip
}
"
echo $lbconf > /etc/nginx/sites-enabled/default;
service nginx restart;
For our load balancer’s user-data script, we will be building our Nginx configuration directly in the script. We start off much as we did with our other Droplets by ensuring that apt
knows we are running in non-interactive mode.
export DEBIAN_FRONTEND=noninteractive;
Then we will install Nginx:
apt-get update;
apt-get -y install nginx;
Next we will create our new Nginx configuration in a variable called lbconf
. Adding an entry for each of our web servers in the upstream backend section.
lbconf="
server {
listen 80 default_server;
listen [::]:80 default_server ipv6only=on;
root /usr/share/nginx/html;
index index.php index.html index.htm;
location / {
proxy_pass http://backend;
include proxy_params;
}
}
upstream backend {
ip_hash;
server web_node_1_private_ip
server web_node_2_private_ip
}
"
We will then write the lbconf
variable to our Nginx configuration file, replacing its current content.
echo $lbconf > /etc/nginx/sites-enabled/default;
And finally, we’ll restart Nginx for this configuration to take effect.
service nginx restart;
Now, before we access our new WordPress site via the browser, we should set up a DNS entry for it. We will do this via the control panel.
In the DigitalOcean control panel, click on DNS. In the Add Domain form, enter your domain name and select your load balancer Droplet from the drop-down menu, then click Create Domain.
In order to use the www subdomain for your site you will need to create another record in this new domain.
Click Add Record and choose the CNAME record type. In the name field enter www and in the hostname field enter @. This will direct requests for the www subdomain to the same location as your main domain (your load balancer Droplet).
Now that we have launched all our Droplets and configured our domain, we can access our new WordPress site by visiting our newly configured domain in a web browser.
We will be prompted here to create a user account and give our new site a name. Once we have done this our deployment is complete and we can begin using the new site.
Now that we can create our WordPress deployment without ever needing to ssh into a Droplet, we can take things a step further and automate this process using the DigitalOcean API.
A sample Ruby script has been created based on this tutorial which will prompt the user to provide the relevant details and then automatically deploy a new scablable WordPress instance. You can find this script on GitHub.
We now have a scalable WordPress deployment but there are additional steps we can take to ensure our new site is secure and stable.
]]>In this tutorial we will create a master snapshot image with our software and configuration and then use the DigitalOcean API automate deployment of droplets using this image. The examples in this tutorial will be using the official DigitalOcean API client for Ruby DropletKit.
In this tutorial we will create a master image based on the LAMP One-Click Image, set up our default configuration and then use it to create a snapshot image. Snapshots cost $0.05 per gigabyte per month, based on the amount of utilized space within the filesystem. We will be able to deploy multiple instances of our customized LAMP stack using the DigitalOcean API.
We will start by creating a new droplet named lamp-master
from the control panel selecting the LAMP image in the Applications tab. This image will provide us with a pre-built Ubuntu 14.04 server with Apache, MySQL and PHP.
When creating the droplet we will use to generate our master snapshot image it is important to select the smallest droplet plan we can. Once we create our snapshot it can only be used to create droplets on the same plan or a larger one. For example, if we create our master snapshot using a 1GB droplet we could then use it to launch droplets on the 1GB, 2GB or other larger plans but we would not be able to launch a droplet with 512MB RAM from this snapshot.
Once our new droplet has been created, use an SSH client to connect to it.
Now that we are connected to our new droplet we can configure any settings or install any packages that we want to have on all droplets deployed from our master image. In this case we will install two extra php modules; curl and Imagemagick.
sudo apt-get update
sudo apt-get install php5-curl php5-imagick
Now that we have added the additional software we want we can power off our droplet and create our snapshot. While it is possible to take a snapshot of a running system, powering down ensures that the filesystem is in a consistent state:
sudo poweroff
While we could create our snapshot from the control panel, for the purposes of this tutorial we will use the API from this point forward to work with our DigitalOcean account. These examples can be run with interactive Ruby (irb
) or added to a script and run with the ruby
command. The first step will be to include the DropletKit client.
require 'droplet_kit'
token='[your api token]'
client = DropletKit::Client.new(access_token: token)
In order to create a snapshot from the API we will need to get the id for our master droplet. We can do this by making a call to the droplets endpoint of the API.
droplets = client.droplets.all
droplets.each do |droplet|
if droplet.name == "lamp-master"
puts droplet.id
end
end
This snippet of code will make a call to the droplets endpoint of the API and loop through the droplets in our account looking for one with the name lamp-master
. When it finds it, the script will then display the ID number for this droplet.
Now that we have our droplet ID number we can tell the API to create a snapshot of this droplet by passing the droplet ID to the snapshot action of the droplet endpoint. In addition to the droplet ID we will also pass a snapshot name which will be used for our new image. In this case we have decided to name our snapshot lamp-image
.
client.droplet_actions.snapshot(droplet_id: '1234567', name: 'lamp-image')
The snapshot request we made will return an event ID number which can be used to track the status of the snapshot process. This tutorial will provide more information on using event IDs.
We have now created a master snapshot image we can use to deploy droplets with our configuration. As we did with our droplet, we will now need to query the API to get the image ID for our new snapshot.
images = client.images.all(public:false)
images.each do |image|
if image.name == "lamp-image"
puts image.id
end
end
As with our droplet identification example above this code will loop through the snapshot and backup images on our account and display the ID for the image named lamp-image
.
Now that we have our image’s ID number we can start deploying droplets. The following code will create a new 2GB droplet using our master snapshot in the New York 3 region.
Note that our snapshot image needs to be present in the region we specify for our droplet creation. You can transfer an image to additional regions via the control panel or through the API’s image endpoint.
droplet = DropletKit::Droplet.new(name: 'my-lamp-server', region: 'nyc3', size: '2gb', image: '1234567')
client.droplets.create(droplet)
We can now deploy new droplets with our custom configuration using the API but we may want to further customize our new droplets individually. We can perform additional customization by sending user-data to our droplets when we create them.
For this example we will pre-load a custom index.html file on our new droplet including it’s name.
sitename = "example.org"
userdata = "
#cloud-config
runcmd:
- echo '<html><head><title>Welcome to #{sitename} </title></head><body><h1>This is #{sitename}</h1></body></html>' > /var/www/html/index.html
"
droplet = DropletKit::Droplet.new(name: sitename, region: 'nyc3', size: '2gb', image: '1234567', user_data: userdata)
client.droplets.create(droplet)
In this example we are simply using the echo
command inside our new droplet to drop some HTML into an index.html file in the web root. By using other commands you could choose to configure new virtualhosts directly on your droplet, pull down additional configuration details from a remote server or do just about anything you could do via an ssh connection. You can learn more about user-data here.
Now that we can deploy droplets based on our snapshot image via the API and customize their contents lets take it a step further and create an interactive script to launch new droplets based on our image. The following script assumes that we have already created our snapshot image and have it’s ID available.
require 'droplet_kit'
token='[Your API Token]'
client = DropletKit::Client.new(access_token: token)
region = 'nyc3'
image_id = '1234567'
droplet_size = '2gb'
puts "Enter a name for your new droplet:"
sitename = gets.chomp
userdata = "
#cloud-config
runcmd:
- echo '<html><head><title>Welcome to #{sitename} </title></head><body><h1>This is #{sitename}</h1></body></html>' > /var/www/html/index.html
"
sitename.gsub!(/\s/,'-')
droplet = DropletKit::Droplet.new(name: sitename, region: region, size: droplet_size, image: image_id, user_data: userdata)
client.droplets.create(droplet)
This script first includes the DropletKit client and initializes a new client connection using the API token you supply.
require 'droplet_kit'
token='[Your API Token]'
client = DropletKit::Client.new(access_token: token)
We then specify a few options for our droplet including the region, droplet size and the ID for our master snapshot image.
region = 'nyc3'
image_id = '1234567'
droplet_size = '2gb'
Then we prompt the user to provide a name for the new droplet and include this information in the user-data our script will provide to the creation process.
puts "Enter a name for your new droplet:"
sitename = gets.chomp
userdata = "
#cloud-config
runcmd:
- echo '<html><head><title>Welcome to #{sitename} </title></head><body><h1>This is #{sitename}</h1></body></html>' > /var/www/html/index.html
"
Once we have included our site name in our index.html page we need to sanitize it to make sure it can be used as a droplet name. Since droplet names cannot have spaces in them we will replace any spaces with dashes.
sitename.gsub!(/\s/,'-')
Then we bring all these variables together and submit our request to create the new droplet.
droplet = DropletKit::Droplet.new(name: sitename, region: region, size: droplet_size, image: image_id, user_data: userdata)
client.droplets.create(droplet)
Using the API we can create custom droplets on demand and include our own settings or files at creation. You may choose to expand on these basics by adding additional functionality to this script. Possible improvements include.
Using the DNS endpoint of the API to auto-configure DNS records for your new droplets when they are launched.
Prompting the user for additional inputs such as region or droplet size.
Using user-data runcmd calls to download web content to your new droplets or to populate MySQL databases.
In version two of the DigitalOcean API, each event that occurs creates an “Action” object. These serve both as records of events that have occurred in the past and as a way to check the progress of an on-going event. From creating a new Droplet to transferring an image to a new region, an Action object will provide you with useful information about the event.
This article will explain Action objects and show how they can be used in practice via DropletKit, the official Ruby gem for the DigitalOcean API.
This article assumes a basic understanding of the DigitalOcean API. To learn more about the API including how to obtain an access token which will be needed to complete this tutorial, see the following resources:
When initiating an event with the API, an Action object will be returned in the response. This object will contain information about the event including its status, the timestamps for when it was started and completed, and the associated resource type and ID. For instance if we where to take a snapshot of the Droplet with the ID 3164450:
curl -X POST -H 'Content-Type: application/json' \
-H 'Authorization: Bearer '$TOKEN'' \
-d '{"type":"snapshot","name":"Nifty New Snapshot"}' \
"https://api.digitalocean.com/v2/droplets/3164450/actions"
we would receive this in response:
{
"action": {
"id": 36805022,
"status": "in-progress",
"type": "snapshot",
"started_at": "2014-11-14T16:34:39Z",
"completed_at": null,
"resource_id": 3164450,
"resource_type": "droplet",
"region": "nyc3"
}
}
Note that the resource_type
is droplet
and the resource_id
is the ID of the Droplet. The status
is in-progress
. This will change to completed
once the event is finished. In order to check on the status of an Action, you can query the API for that Action directly.
curl -X GET -H 'Content-Type: application/json' \
-H 'Authorization: Bearer '$TOKEN'' \
"https://api.digitalocean.com/v2/actions/36805022"
This will return the requested action object:
{
"action": {
"id": 36805022,
"status": "completed",
"type": "snapshot",
"started_at": "2014-11-14T16:34:39Z",
"completed_at": "2014-11-14T16:38:52Z",
"resource_id": 3164450,
"resource_type": "droplet",
"region": "nyc3"
}
}
Notice how now that the status
is completed
, there is a timestamp for completed_at
as well as started_at
.
You can also access a complete history of all Actions taken on your account at the /actions
endpoint.
curl -X GET -H 'Content-Type: application/json' \
-H 'Authorization: Bearer '$TOKEN'' \
"https://api.digitalocean.com/v2/actions"
While listing all Action objects may be interesting in order to audit your history, in practice you will mostly use this endpoint in order check on the status of a process. We’ll be using droplet_kit
, the official Ruby gem for the DigitalOcean API, for these examples. It can be installed with:
gem install droplet_kit
To get started, enter the Ruby shell by running the command irb
Then import the droplet_kit
gem and set up your client using your API token:
irb(main):> require 'droplet_kit'
=> true
irb(main):> client = DropletKit::Client.new(access_token: DO_TOKEN)
Some actions are dependent on others being taken first. For instance, attempting to take a snapshot of a Droplet which is still powered on will lead to an error. A Droplet must be powered off in order to take a snapshot.
irb(main):> client.droplet_actions.snapshot(droplet_id: 4143310, name: 'Snapshot Name')
=> "{\"id\":\"unprocessable_entity\",\"message\":\"Droplet is currently on. Please power it off to run this event.\"}"
Attempting to take a snapshot immediately after initiating a shutdown action will also lead to that same error as you must ensure that the shutdown Action has completed before the snapshot can be taken. Actions can not be queued.
irb(main):> client.droplet_actions.shutdown(droplet_id: 4143310)
=> <DropletKit::Action {:@id=>43918785, :@status=>"in-progress", :@type=>"shutdown", :@started_at=>"2015-02-16T21:22:35Z", :@completed_at=>nil, :@resource_id=>4143310, :@resource_type=>"droplet", :@region=>"nyc3"}>
irb(main):> client.droplet_actions.snapshot(droplet_id: 4143310, name: 'Snapshot Name')
=> "{\"id\":\"unprocessable_entity\",\"message\":\"Droplet is currently on. Please power it off to run this event.\"}"
Like the curl
examples above, droplet_kit
also returns the Action object in response to a successfully initiated event. It can be accessed as a normal Ruby object. Saving the response into a variable will allow you to access its attributes directly:
irb(main):> snapshot = client.droplet_actions.snapshot(droplet_id: 4143310, name: 'Snapshot Name')
=> "{\"id\":\"unprocessable_entity\",\"message\":\"Droplet is currently on. Please power it off to run this event.\"}"
irb(main):> shutdown = client.droplet_actions.shutdown(droplet_id: 4143310)
=> <DropletKit::Action {:@id=>43919195, :@status=>"in-progress", :@type=>"shutdown", :@started_at=>"2015-02-16T21:32:03Z", :@completed_at=>nil, :@resource_id=>4143310, :@resource_type=>"droplet", :@region=>"nyc3"}>
irb(main):> shutdown.status
=> "in-progress"
irb(main):> shutdown.id
=> 43919195
You can then check the status of the actions:
irb(main):> action = client.actions.find(id: shutdown.id)
=> <DropletKit::Action {:@id=>43919195, :@status=>"completed", :@type=>"shutdown", :@started_at=>"2015-02-16T21:32:03Z", :@completed_at=>"2015-02-16T21:32:07Z", :@resource_id=>4143310, :@resource_type=>"droplet", :@region=>"nyc3"}>
irb(main):> action.status
=> "completed"
We can use an until
loop in Ruby to check on the progress of an Action until it has completed:
res = client.droplet_actions.shutdown(droplet_id: id)
until res.status == "completed"
res = client.actions.find(id: res.id)
sleep(2)
end
This Ruby script bellow is an example of how to check on the status of an action in practice. It powers a droplet off and uses the while
loop from above to make sure that the action has completed before moving on. Once the shutdown action has completed, it will then take a snapshot of the droplet.
#!/usr/bin/env ruby
require 'droplet_kit'
require 'json'
token = ENV['DO_TOKEN']
client = DropletKit::Client.new(access_token: token)
droplet_id = ARGV[0]
snapshot_name = ARGV[1] || Time.now.strftime("%b. %d, %Y - %H:%M:%S %Z")
def power_off(client, id)
res = client.droplet_actions.shutdown(droplet_id: id)
until res.status == "completed"
res = client.actions.find(id: res.id)
sleep(2)
end
puts " * Action status: #{res.status}"
rescue NoMethodError
puts JSON.parse(res)['message']
end
def take_snapshot(client, id, name)
res = client.droplet_actions.snapshot(droplet_id: id, name: name)
puts " * Action status: #{res.status}"
rescue NameError
puts JSON.parse(res)['message']
end
unless droplet_id.nil?
puts "Powering off droplet..."
power_off(client, droplet_id)
sleep(2)
puts "Taking snapshot..."
take_snapshot(client, droplet_id, snapshot_name)
else
puts "Power off and snapshot a droplet. Requires a droplet ID and optionally a snapshot name."
puts "Usage: #{$0} droplet_id ['snapshot name']"
end
If you save this script as a file named snapshot.rb
(or download it from this GitHub Gist), you can run it from the command line like so:
DO_TOKEN=YOUR_DO_API_TOKEN ruby snapshot.rb 12345 "My Snapshot"
Note that in order to use the script, you must export your API token as a environmental variable with the name DO_TOKEN
. The script takes two arguments, the ID of the droplet and optionally a name of the snapshot. If you do not provide a name, it will is the date and time.
Action items are an important part of the DigtialOcean API. Using them to check the status of actions is an important best practice to implement when using the API. Now that you understand how to use them, you are ready to move on to more complex use-cases of the API like:
Check out the DigitalOcean developer’s portal for more topics.
]]>In this tutorial, we will demonstrate how to use DigitalOcean API to horizontally scale your server setup. To do this, we will use DOProxy, a relatively simple Ruby script that, once configured, provides a command line interface to scale your HTTP application server tier up or down.
DOProxy, which was written specifically for this tutorial, provides a simple way for creating and deleting application server droplets, using the DigitalOcean API, and automatically managing them behind an HAProxy load balancer. This basic scaling model allows your users to access your application through the HAProxy server, which will forward them to the load-balanced, backend application servers.
DOProxy performs three primary functions:
Note: The primary purpose of this tutorial is to teach the minimally required concepts necessary to programmatically scale your DigitalOcean server architecture through the API. You should not run DOProxy, in its current form, in a production environment; it was not designed with resiliency in mind, and it performs just enough error checking to get by. With that said, if you are curious about learning about horizontal scaling through the API, it’s a great way to get started.
This tutorial touches on a variety of technologies that you might want to read up on before proceeding, including:
Because DOProxy is written in Ruby, knowledge of Ruby is a plus but not necessary; we will provide pseudocode to explain the gist of the DOProxy code. Also, we will use the official DigitalOcean Ruby wrapper, DropletKit, which enables us to easily make API calls in our Ruby code.
Before we get into the details of DOProxy works, we will install and use it on a server. Let’s install DOProxy on an Ubuntu 14.04 droplet now.
First, create an Ubuntu 14.04 droplet in the NYC3 region (you may use any region that supports private networking and userdata if you configure the region
variable in the doproxy.yml
file after installing DOProxy). This droplet will run the HAProxy load balancer, and the DOProxy scaling script, so choose a size that you think will be adequate for your desired scale potential. Because this tutorial is a basic demonstration of scaling, which won’t receive any real traffic, the 1GB size is probably adequate.
We will refer to this droplet as the DOProxy server.
Next, log in and follow the Installation and Configuration (including doproxy config and Userdata) sections in the DOProxy GitHub repository to install DOProxy on this server. Use the sample doproxy.yml
and user-data.yml
files by copying them, as noted in the directions. Be sure to replace the token
and ssh_key_ids
values in the DOproxy configuration file, or the script will not work.
Now that you have DOProxy and HAProxy installed on your server, let’s try and scale our environment.
Log in to your DOProxy server as root, and change to the directory where you cloned DOProxy, if you haven’t done so already.
Now run DOProxy without any arguments:
ruby doproxy.rb
This should print out the available commands like so:
Commands:
doproxy.rb print # Print backend droplets in inventory file
doproxy.rb create # Create a new backend droplet and reload
doproxy.rb delete <LINE_NUMBER> # Delete a droplet and reload
doproxy.rb reload # Generate HAProxy config and reload HAProxy
doproxy.rb generate # Generate HAProxy config based on inventory
Currently, DOProxy hasn’t created any droplets. Let’s create some to get our HTTP service online, and scale up.
Run the create command to create the first droplet that is managed by DOProxy:
ruby doproxy.rb create
This will take some time before returning to the prompt (because the script creates a new droplet via the API and waits for it to boot up). We’ll talk about how the API call is made, later, when we go through the DOProxy code.
When it is complete, you should see a success message that contains the droplet ID, like so:
Success: 4202645 created and added to backend.
If you visit your DOProxy server’s public IP address in a web browser. You should see a page that lists your new droplet’s hostname, id, and public IP address.
We’ll use DOProxy to create two more droplets, for a total of three. Feel free to create more if you want:
ruby doproxy.rb create
ruby doproxy.rb create
Now visit your DOProxy server’s public IP address in a web browser again. If you refresh the page, you will notice that the information on the page will change—it will cycle through the droplets that you created. This is because they are all being load balanced by HAProxy—each droplet is added to the load balancer configuration when it is created.
If you happen to look in the DigitalOcean Control Panel, you will notice that these new droplets will be listed there (along with the rest of your droplets):
Let’s take a closer look at the droplets that were created by looking at DOProxy’s inventory.
DOProxy provides a print command, that will print out all of the droplets that are part of its inventory:
ruby doproxy.rb print
You should see output that looks something like this:
0) auto-nginx-0 (pvt ip: 10.132.224.168, status: active, id: 4202645)
1) auto-nginx-1 (pvt ip: 10.132.228.224, status: active, id: 4205587)
2) auto-nginx-2 (pvt ip: 10.132.252.42, status: active, id: 4205675)
In the example output, we see information about the three droplets that we created, such as their hostnames, status, and droplet IDs. The hostnames and IDs should match what you saw when you accessed the HAProxy load balancer (via DOProxy’s public IP address).
As you may have noticed, DOProxy only printed information about droplets that it created. This is because it maintains an inventory of the droplets it creates.
Check out the contents of the inventory
file now:
cat inventory
You should see the ID of each droplet, one per line. Each time a droplet is created, its ID is stored in this inventory file.
As you may have guessed, DOProxy’s print
command iterates through the droplet IDs in the inventory file and performs an API call to retrieve droplet information about each one.
It should be noted that storing your server inventory in a single file is not the best solution—it can easily be corrupted or deleted—but it demonstrates a simple implementation that works. A distributed key value store, such as etcd, would be a better solution. You would also want to save more than just the droplet ID in the inventory (so you don’t have to make API calls every time you want to look at certain droplet information).
DOProxy also has a delete command that lets you delete droplets in your inventory. The delete command requires that you provide the line number of the droplet to delete (as displayed by the print
command).
Before running this command you will probably want to print your inventory:
ruby doproxy.rb print
So, for example, if you want to delete the third droplet, you would supply 2
as the line number:
ruby doprorxy.rb delete 2
After a moment, you’ll see the confirmation message:
Success: 4205675 deleted and removed from backend.
The delete command deletes the droplet via the API, removes it from the HAProxy configuration, and deletes it from the inventory. Feel free to verify that the droplet was deleted by using the DOProxy print command or by checking the DigitalOcean control panel. You will also notice that it is no longer part of the load balancer.
The last piece of DOProxy that we haven’t discussed is how HAProxy is configured.
When you run the create
or delete
DOProxy command, the information of each droplet in the inventory is retrieved, and some of the information is used to create an HAProxy configuration file. In particular, the droplet ID and private IP address is used to add each droplet as a backend server.
Look at the last few lines of the generated haproxy.cfg
file like this:
tail haproxy.cfg
You should see something like this:
frontend www-http
bind 104.236.236.43:80
reqadd X-Forwarded-Proto:\ http
default_backend www-backend
backend www-backend
server www-4202645 10.132.224.168:80 check # id:4202645, hostname:auto-nginx-0
server www-4205587 10.132.228.224:80 check # id:4205587, hostname:auto-nginx-1
The frontend
section should contain the public IP address of your DOProxy server, and the backend
section should contain lines that refer to each of the droplets that were created.
Note: At this point, you may want to delete the rest of the droplets that were created with DOProxy (ruby doproxy.rb delete 0
until all of the servers are gone).
Now that you’ve seen DOProxy’s scaling in action, let’s take a closer look at the code.
In this section, we will look at the pertinent files and lines of code that make DOProxy work. Seeing how DOProxy was implemented should give you some ideas of how you can use the API to manage and automate your own server infrastructure.
Since you cloned the repository to your server, you can look at the files there, or you can look at the files at the DOProxy repository (https://github.com/thisismitch/doproxy).
Important files:
Let’s dive into the configuration files first.
The important lines in the DOProxy configuration file, doproxy.yml
, are the following:
token: 878a490235d53e34b44369b8e78
ssh_key_ids: # DigitalOcean ID for your SSH Key
- 163420
...
droplet_options:
hostname_prefix: auto-nginx
region: nyc3
size: 1gb
image: ubuntu-14-04-x64
The token
is where you can configure your read and write API token.
The other lines specify the options that will be used when DOProxy creates a new droplet. For example, it will install the specified SSH key (by ID or fingerprint), and it will prefix the hostnames with “auto-nginx”.
More information about valid droplet options, check out the DigitalOcean API documentation.
The userdata file, user-data.yml
, is a file that will be executed by cloud-init on each new droplet, when it is created. This means that you can supply a cloud-config file or a script to install your application software on each new droplet.
The sample userdata file contains a simple bash script that installs Nginx on an Ubuntu server, and replaces its default configuration file with the droplet hostname, ID, and public IP address:
#!/bin/bash
apt-get -y update
apt-get -y install nginx
export DROPLET_ID=$(curl http://169.254.169.254/metadata/v1/id)
export HOSTNAME=$(curl -s http://169.254.169.254/metadata/v1/hostname)
export PUBLIC_IPV4=$(curl -s http://169.254.169.254/metadata/v1/interfaces/public/0/ipv4/address)
echo Droplet: $HOSTNAME, ID: $DROPLET_ID, IP Address: $PUBLIC_IPV4 > /usr/share/nginx/html/index.html
The droplet information (hostname, ID, and IP address) are retrieved through the DigitalOcean Metadata service—that’s what those curl
commands are doing.
Obviously, you would want to do something more useful than this, like install and configure your application. You can use this to automate the integration of your droplets into your overall infrastructure, by doing things like automatically installing SSH keys and connecting to your configuration management or monitoring tools.
To read more about userdata, cloud-config, and metadata, check out these links:
The HAProxy configuration template, haproxy.cfg.erb
, contains most of the load balancer configuration, with some Ruby code that will be replaced with backend droplet information.
We’ll just look at the Ruby section that generates the backend configuration:
backend www-backend
<% @droplets.each_with_index do |droplet, index| %>
server www-<%= droplet.id %> <%= droplet.private_ip %>:80 check # id:<%= droplet.id %>, hostname:<%= droplet.name -%>
<% end %>
This code iterates through each of the droplets in the inventory, and adds a new HAProxy backend for each one (based on the private IP address).
For example, a line like this will be produced for each droplet:
server www-4202645 10.132.224.168:80 check # id:4202645, hostname:auto-nginx-0
Whenever a droplet is created or deleted, DOProxy generates a new HAProxy configuration file—the haproxy.cfg
file that you looked at earlier.
The DOProxy Ruby script, doproxy.rb
, consists mainly of a DOProxy class that contains the methods that perform the droplet creation and deletion, inventory management, and HAProxy configuration generation.
If you understand Ruby, check out the file on GitHub: https://github.com/thisismitch/doproxy/blob/master/doproxy.rb.
If you don’t understand Ruby, here is some simplified pseudocode that explains each method. It may be useful to reference this against the actual Ruby code, to help you understand what is happening.
Executed every time DOProxy runs, unless no arguments are specified.
doproxy.yml
configuration file (get API token, and droplet options). 2ified.Retrieves information for each droplet in the inventory file. It must be executed before any of the following methods are executed.
When the “doproxy.rb print” command is used, prints droplet information to the screen. It relies on get_inventory
.
get_inventory
)When the “doproxy.rb create” command is used, creates a new droplet and adds it to the inventory file, then calls reload_haproxy
to generate HAProxy configuration and reload the load balancer.
reload_haproxy
to generate HAProxy configuration and reload the load balancerWhen the “doproxy.rb delete” command is used, deletes the specified droplet and deletes its ID from the inventory file, then calls reload_haproxy
to generate HAProxy configuration and reload the load balancer.
reload_haproxy
to generate HAProxy configuration and reload the load balancerThis is a supporting method that creates new HAProxy configuration files based on the droplets in the inventory.
haproxy.cfg.erb
haproxy.cfg
file to diskThis is a supporting method that copies the HAProxy configuration file into the proper location, and reloads HAProxy. This relies on generate_haproxy_cfg
.
haproxy.cfg
to the location where HAProxy will read on reloadThat’s all of the important code that makes DOProxy work. The last thing we will discuss is DropletKit, the API wrapper that we used in DOProxy.
DOProxy uses the DropletKit gem, the official DigitalOcean API v2 Ruby wrapper, to make DigitalOcean API calls. DropletKit allows us to easily write Ruby programs that do things like:
This tutorial focused on these particular API endpoints, but keep in mind that there are many other endpoints that can help facilitate programmatic management your DigitalOcean server infrastructure.
Now that you’ve seen how a simple script can help scale a server environment, by leveraging the DigitalOcean API, cloud-config, and metadata, hopefully you can apply these concepts to scale your own server setup. Although DOProxy isn’t production ready, it should give you some ideas for implementing your own scaling solution.
Remember that the scaling setup described here, with DOProxy, is great but it could be greatly improved by using it in conjunction with a monitoring system. This would allow you to automatically scale your application server tier up and down, depending on certain conditions, such as server resource utilization.
Have any questions or comments? Feel free to post them below!
]]>