uptime
just gives load average as load average: 0.0, 0.0, 0.0
whereas in both my local dev (WSL) and docker container has the proper load avg which I’m later using in my node app with func: os.loadavg().
I assume there are permission errors but isn’t there any way to get system load info i.e. cpu, mem, etc. in a web app env or will I need to recreate as a droplet? In the tutorial they also run uptime
with no issues.
Thanks
]]>When working with big JSON files, it can be hard to find and manipulate the information you need. You could copy and paste all relevant snippets to calculate totals manually, but this is a time-consuming process and could be prone to human error. Another option is to use general-purpose tools for finding and manipulating information. All modern Linux systems come installed with three established text processing utilities: sed
, awk
, and grep
. While these commands are helpful when working with loosely structured data, other options exist for machine-readable data formats like JSON.
jq
, a command-line JSON processing tool, is a good solution for dealing with machine-readable data formats and is especially useful in shell scripts. Using jq
can aid you when you need to manipulate data. For example, if you run a curl
call to a JSON API, jq
can extract specific information from the server’s response. You could also incorporate jq
into your data ingestion process as a data engineer. If you manage a Kubernetes cluster, you could use the JSON output of kubectl
as an input source for jq
to extract the number of available replicas for a specific deployment.
In this article, you will use jq
to transform a sample JSON file about ocean animals. You’ll apply data transformations using filters and merge pieces of transformed data into a new data structure. By the end of the tutorial, you will be able to use a jq
script to answer questions about the data you have manipulated.
To complete this tutorial, you will need the following:
jq
, a JSON parsing and transformation tool. It is available from the repositories for all major Linux distributions. If you are using Ubuntu, run sudo apt install jq
to install it.jq
CommandIn this step, you will set up your sample input file and test the setup by running a jq
command to generate an output of the sample file’s data. jq
can take input from either a file or a pipe. You will use the former.
You’ll begin by generating the sample file. Create and open a new file named seaCreatures.json
using your preferred editor (this tutorial uses nano
):
- nano seaCreatures.json
Copy the following contents into the file:
[
{ "name": "Sammy", "type": "shark", "clams": 5 },
{ "name": "Bubbles", "type": "orca", "clams": 3 },
{ "name": "Splish", "type": "dolphin", "clams": 2 },
{ "name": "Splash", "type": "dolphin", "clams": 2 }
]
You’ll work with this data for the rest of the tutorial. By the end of the tutorial, you will have written a one-line jq
command that answers the following questions about this data:
Save and close the file.
In addition to an input file, you will need a filter that describes the exact transformation you’d like to do. The .
(period) filter, also known as the identity operator, passes the JSON input unchanged as output.
You can use the identity operator to test whether your setup works. If you see any parse errors, check that seaCreatures.json
contains valid JSON.
Apply the identity operator to the JSON file with the following command:
- jq '.' seaCreatures.json
When using jq
with files, you always pass a filter followed by the input file. Since filters may contain spacing and other characters that hold a special meaning to your shell, it is a good practice to wrap your filter in single quotation marks. Doing so tells your shell that the filter is a command parameter. Rest assured that running jq
will not modify your original file.
You’ll receive the following output:
Output[
{
"name": "Sammy",
"type": "shark",
"clams": 5
},
{
"name": "Bubbles",
"type": "orca",
"clams": 3
},
{
"name": "Splish",
"type": "dolphin",
"clams": 2
},
{
"name": "Splash",
"type": "dolphin",
"clams": 2
}
]
By default, jq
will pretty print its output. It will automatically apply indentation, add new lines after every value, and color its output when possible. Coloring may improve readability, which can help many developers as they examine JSON data produced by other tools. For example, when sending a curl
request to a JSON API, you may want to pipe the JSON response into jq '.'
to pretty print it.
You now have jq
up and running. With your input file set up, you’ll manipulate the data using a few different filters in order to compute the values of all three attributes: creatures
, totalClams
, and totalDolphinClams
. In the next step, you’ll find the information from the creatures
value.
creatures
ValueIn this step, you will generate a list of all sea creatures, using the creatures
value to find their names. At the end of this step, you will have generated the following list of names:
Output[
"Sammy",
"Bubbles",
"Splish",
"Splash"
],
Generating this list requires extracting the names of the creatures and then merging them into an array.
You’ll have to refine your filter to get the names of all creatures and discard everything else. Since you’re working on an array, you’ll need to tell jq
you want to operate on the values of that array instead of the array itself. The array value iterator, written as .[]
, serves this purpose.
Run jq
with the modified filter:
- jq '.[]' seaCreatures.json
Every array value is now output separately:
Output{
"name": "Sammy",
"type": "shark",
"clams": 5
}
{
"name": "Bubbles",
"type": "orca",
"clams": 3
}
{
"name": "Splish",
"type": "dolphin",
"clams": 2
}
{
"name": "Splash",
"type": "dolphin",
"clams": 2
}
Instead of outputting every array item in full, you’ll want to output the value of the name
attribute and discard the rest. The pipe operator |
will allow you to apply a filter to each output. If you have used find | xargs
on the command line to apply a command to every search result, this pattern will feel familiar.
A JSON object’s name
property can be accessed by writing .name
. Combine the pipe with the filter and run this command on seaCreatures.json
:
- jq '.[] | .name' seaCreatures.json
You’ll notice that the other attributes have disappeared from the output:
Output"Sammy"
"Bubbles"
"Splish"
"Splash"
By default, jq
outputs valid JSON, so strings will appear in double quotation marks (""
). If you need the string without double quotes, add the -r
flag to enable raw output:
- jq -r '.[] | .name' seaCreatures.json
The quotation marks have disappeared:
OutputSammy
Bubbles
Splish
Splash
You now know how to extract specific information from the JSON input. You’ll use this technique to find other specific information in the next step and then to generate the creatures
value in the final step.
totalClams
Value with map
and add
In this step, you’ll find the total information for how many clams the creatures own. You can calculate the answer by aggregating a few pieces of data. Once you’re familiar with jq
, this will be faster than manual calculations and less prone to human error. The expected value at the end of this step is 12
.
In Step 2, you extracted specific bits of information from a list of items. You can reuse this technique to extract the values of the clams
attribute. Adjust the filter for this new attribute and run the command:
- jq '.[] | .clams' seaCreatures.json
The individual values of the clams
attribute will be output:
Output5
3
2
2
To find the sum of individual values, you will need the add
filter. The add
filter works on arrays. However, you are currently outputting array values, so you must wrap them in an array first.
Surround your existing filter with []
as follows:
- jq '[.[] | .clams]' seaCreatures.json
The values will appear in a list:
Output[
5,
3,
2,
2
]
Before applying the add
filter, you can improve the readability of your command with the map
function, which also makes it easier to maintain. Iterating over an array, applying a filter to each of those items, and then wrapping the results in an array can be achieved with one map
invocation. Given an array of items, map
will apply its argument as a filter to each item. For example, if you apply the filter map(.name)
to [{"name": "Sammy"}, {"name": "Bubbles"}]
, the resulting JSON object will be ["Sammy", "Bubbles"]
.
Rewrite the filter to generate an array to use a map
function instead, then run it:
- jq 'map(.clams)' seaCreatures.json
You will receive the same output as before:
Output[
5,
3,
2,
2
]
Since you have an array now, you can pipe it into the add
filter:
- jq 'map(.clams) | add' seaCreatures.json
You’ll receive a sum of the array:
Output12
With this filter, you have calculated the total number of clams, which you’ll use to generate the totalClams
value later. You’ve written filters for two out of three questions. You have one more filter to create, after which you can generate the final output.
totalDolphinClams
Value with the add
FilterNow that you know how many clams the creatures own, you can identify how many of those clams the dolphins have. You can generate the answer by adding only the values of array elements that satisfy a specific condition. The expected value at the end of this step is 4
, which is the total number of clams the dolphins have. In the final step, the resulting value will be used by the totalDolphinClams
attribute.
Instead of adding all clams
values as you did in Step 3, you’ll count only clams held by creatures with the "dolphin"
type. You’ll use the select
function to select a specific condition: select(condition)
. Any input for which the condition evaluates to true
is passed on. All other input is discarded. If, for example, your JSON input is "dolphin"
and your filter is select(. == "dolphin")
, the output would be "dolphin"
. For the input "Sammy"
, the same filter would output nothing.
To apply select
to every value in an array, you can pair it with map
. In doing so, array values that don’t satisfy the condition will be discarded.
In your case, you only want to retain array values whose type
value equals "dolphin"
. The resulting filter is:
- jq 'map(select(.type == "dolphin"))' seaCreatures.json
Your filter will not match Sammy the shark and Bubbles the orca, but it will match the two dolphins:
Output[
{
"name": "Splish",
"type": "dolphin",
"clams": 2
},
{
"name": "Splash",
"type": "dolphin",
"clams": 2
}
]
This output contains the number of clams per creature, as well as some information that isn’t relevant. To retain only the clams
value, you can append the name of the field to the end of map
’s parameter:
- jq 'map(select(.type == "dolphin").clams)' seaCreatures.json
The map
function receives an array as input and will apply map
’s filter (passed as an argument) to each array element. As a result, select
gets called four times, once per creature. The select
function will produce output for the two dolphins (as they match the condition) and omit the rest.
Your output will be an array containing only the clams
values of the two matching creatures:
Output[
2,
2
]
Pipe the array values into add
:
- jq 'map(select(.type == "dolphin").clams) | add' seaCreatures.json
Your output will return the sum of the clams
values from creatures of the "dolphin"
type:
Output4
You’ve successfully combined map
and select
to access an array, select array items matching a condition, transform them, and sum the result of that transformation. You can use this strategy to calculate totalDolphinClams
in the final output, which you will do in the next step.
In the previous steps, you wrote filters to extract and manipulate the sample data. Now, you can combine these filters to generate an output that answers your questions about the data:
To find the names of the sea creatures in list form, you used the map
function: map(.name)
. To find how many clams the creatures own in total, you piped all clams
values into the add
filter: map(.clams) | add
. To find how many of those clams are owned by dolphins, you used the select
function with the .type == "dolphin"
condition: map(select(.type == "dolphin").clams) | add
.
You’ll combine these filters into one jq
command that does all of the work. You will create a new JSON object that merges the three filters in order to create a new data structure that displays the information you desire.
As a reminder, your starting JSON file matches the following:
[
{ "name": "Sammy", "type": "shark", "clams": 5 },
{ "name": "Bubbles", "type": "orca", "clams": 3 },
{ "name": "Splish", "type": "dolphin", "clams": 2 },
{ "name": "Splash", "type": "dolphin", "clams": 2 }
]
Your transformed JSON output will generate the following:
Final Output{
"creatures": [
"Sammy",
"Bubbles",
"Splish",
"Splash"
],
"totalClams": 12,
"totalDolphinClams": 4
}
Here is a demonstration of the syntax for the full jq
command with empty input values:
- jq '{ creatures: [], totalClams: 0, totalDolphinClams: 0 }' seaCreatures.json
With this filter, you create a JSON object containing three attributes:
Output{
"creatures": [],
"totalClams": 0,
"totalDolphinClams": 0
}
That’s starting to look like the final output, but the input values are not correct because they have not been pulled from your seaCreatures.json
file.
Replace the hard-coded attribute values with the filters you created in each prior step:
- jq '{ creatures: map(.name), totalClams: map(.clams) | add, totalDolphinClams: map(select(.type == "dolphin").clams) | add }' seaCreatures.json
The above filter tells jq
to create a JSON object containing:
creatures
attribute containing a list of every creature’s name
value.totalClams
attribute containing a sum of every creature’s clams
value.totalDolphinClams
attribute containing a sum of every creature’s clams
value for which type
equals "dolphin"
.Run the command, and the output of this filter should be:
Output{
"creatures": [
"Sammy",
"Bubbles",
"Splish",
"Splash"
],
"totalClams": 12,
"totalDolphinClams": 4
}
You now have a single JSON object providing relevant data for all three questions. Should the dataset change, the jq
filter you wrote will allow you to re-apply the transformations at any time.
When working with JSON input, jq
can help you perform a wide variety of data transformations that would be difficult with text manipulation tools like sed
. In this tutorial, you filtered data with the select
function, transformed array elements with map
, summed arrays of numbers with the add
filter, and learned how to merge transformations into a new data structure.
To learn about jq
advanced features, dive into the jq
reference documentation. If you often work with non-JSON command output, you can explore our guides on sed
, awk
or grep
for information on text processing techniques that will work on any format.
#! /bin/bash
echo "hello"
mkdir "/karan/washere"
cron job i created, i want to run this cron job to run every min and want the log
#testing if cron job workes or not
1 * * * * /user/local/bin/bash /root/test.sh &> /root/crontest.log
I am signed in the droplet as root user
I also have given the permission for the script, using
sudo chmod u+x test.sh
I tried to log the syslog using
sudo grep CRON /var/log/syslog
but didn’t show there also,
let me know if you need any more info or other context,
]]>I’m looking for something that will help me visualize it a bit better in their console. They are new to all this technical stuff and I was searching for something to help them out.
]]>Apt is a command line frontend for the dpkg packaging system and is the preferred way of managing software from the command line for many distributions. It is the main package management system in Debian and Debian-based Linux distributions like Ubuntu.
While a tool called “dpkg” forms the underlying packaging layer, apt
and apt-cache
provide user-friendly interfaces and implement dependency handling. This allows users to efficiently manage large amounts of software easily.
In this guide, we will discuss the basic usage of apt
and apt-cache
and how they can manage your software. We will be practicing on an Ubuntu 22.04 cloud server, but the same steps and techniques should apply on any other Ubuntu or Debian-based distribution.
Apt operates on a database of known, available software. It performs installations, package searches, and many other operations by referencing this database.
Because of this, before beginning any packaging operations with apt
, we need to ensure that our local copy of the database is up-to-date.
Update the local database with apt update
. Apt requires administrative privileges for most operations:
- sudo apt update
You will see a list of servers we are retrieving information from. After this, your database should be up-to-date.
You can upgrade the packages on your system by using apt upgrade
. You will be prompted to confirm the upgrades, and restart any updated system services:
- sudo apt upgrade
If you know the name of a package you need to install, you can install it by using apt install
:
- sudo apt install package1 package2 …
You can see that it is possible to install multiple packages at one time, which is useful for acquiring all of the necessary software for a project in one step.
Apt installs not only the requested software, but also any software needed to install or run it.
You can install a program called sl
by typing:
- sudo apt install sl
After that, you’ll be able to run sl
on the command line.
To remove a package from your system, run apt remove
:
- sudo apt remove package_name
This command removes the package, but retains any configuration files in case you install the package again later. This way, your settings will remain intact, even though the program is not installed.
If you need to clean out the configuration files as well as the program, use apt purge
:
- sudo apt purge package_name
This uninstalls the package and removes any configuration files associated with the package.
To remove any packages that were installed automatically to support another program, that are no longer needed, type the following command:
- sudo apt autoremove
You can also specify a package name after the autoremove
command to uninstall a package and its dependencies.
There are a number of additional options that can be specified by the use of flags. We will go over some common ones.
To do a “dry run” of a procedure in order to get an idea of what an action will do, you can pass the -s
flag for “simulate”:
- sudo apt install -s htop
OutputReading package lists... Done
Building dependency tree... Done
Reading state information... Done
Suggested packages:
lm-sensors
The following NEW packages will be installed:
htop
0 upgraded, 1 newly installed, 0 to remove and 1 not upgraded.
Inst htop (3.0.5-7build2 Ubuntu:22.04/jammy [amd64])
Conf htop (3.0.5-7build2 Ubuntu:22.04/jammy [amd64])
In place of actual actions, you can see an Inst
and Conf
section specifying where the package would be installed and configured if the “-s” was removed.
If you do not want to be prompted to confirm your choices, you can also pass the -y
flag to automatically assume “yes” to questions.
- sudo apt remove -y htop
If you would like to download a package, but not install it, you can issue the following command:
- sudo apt install -d packagename
The files will be retained in /var/cache/apt/archives
.
If you would like to suppress output, you can pass the -qq
flag to the command:
- sudo apt remove -qq packagename
The apt packaging tool is actually a suite of related, complimentary tools that are used to manage your system software.
While apt
is used to upgrade, install, and remove packages, apt-cache
is used to query the package database for package information.
You can use apt-cache search
to search for a package that suits your needs. Note that apt-cache doesn’t usually require administrative privileges:
- apt-cache search what_you_are_looking_for
For instance, to find htop
, an improved version of the top
system monitor, you can use:
- apt-cache search htop
Outputhtop - interactive processes viewer
aha - ANSI color to HTML converter
bashtop - Resource monitor that shows usage and stats
bpytop - Resource monitor that shows usage and stats
btop - Modern and colorful command line resource monitor that shows usage and stats
libauthen-oath-perl - Perl module for OATH One Time Passwords
pftools - build and search protein and DNA generalized profiles
You can search for more generic terms also. In this example, we’ll look for mp3 conversion software:
- apt-cache search mp3 convert
Outputabcde - A Better CD Encoder
cue2toc - converts CUE files to cdrdao's TOC format
dir2ogg - audio file converter into ogg-vorbis format
easytag - GTK+ editor for audio file tags
ebook2cw - convert ebooks to Morse MP3s/OGGs
ebook2cwgui - GUI for ebook2cw
ffcvt - ffmpeg convert wrapper tool
. . .
To view information about a package, including an extended description, use the following syntax:
- apt-cache show package_name
This will also provide the size of the download and the dependencies needed for the package.
To see if a package is installed and to check which repository it belongs to, you can use apt-cache policy
:
- apt-cache policy package_name
You should now know enough about apt-get and apt-cache to manage most of the software on your server.
While it is sometimes necessary to go beyond these tools and the software available in the repositories, most software operations can be managed by these tools.
Next, you can read about Ubuntu and Debian package management in detail.
]]>sudo reboot
but the API is still down. Is there any way to restart the server and resolve this issue?
This is the banner: SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.6
]]>I then followed this guide to add a GUI - https://www.digitalocean.com/community/questions/how-to-install-graphical-interface
All the steps worked, until the step to establish a secure connection (Step 3 — Connecting to the VNC Desktop Securely).
I ran the code that was provided (changing the user parameter accordingly) -
> ssh -L 59000:localhost:5901 -C -N -l sammy your_server_ip
But I got the following error - Permission denied (publickey).
I do have SSH keys as the authentication method on root, and my non root user can login successfully. So I assume the key was copied successfully to that user.
I was required to set a password for the non root user, and I am not sure if that could be the cause of the error.
How can I fix this error and establish the SSH tunnel?
]]>apt-key
is a utility used to manage the keys that APT uses to authenticate packages. It’s closely related to the add-apt-repository
utility, which adds external repositories using keyservers to an APT installation’s list of trusted sources. However, keys added using apt-key
and add-apt-repository
are trusted globally by apt
. These keys are not limited to authorizing the single repository they were intended for. Any key added in this manner can be used to authorize the addition of any other external repository, presenting an important security concern.
Starting with Ubuntu 20.10, the use of apt-key
yields a warning that the tool will be deprecated in the near future; likewise, add-apt-repository
will also soon be deprecated. While these deprecation warnings do not strictly prevent the usage of apt-key
and add-apt-repository
with Ubuntu 22.04, it is not advisable to ignore them.
The current best practice is to use gpg
in place of apt-key
and add-apt-repository
, and in future versions of Ubuntu it will be the only option. apt-key
and add-apt-repository
themselves have always acted as wrappers, calling gpg
in the background. Using gpg
directly cuts out the intermediary. For this reason, the gpg
method is backwards compatible with older versions of Ubuntu and can be used as a drop-in replacement for apt-key
.
This tutorial will outline two procedures that use alternatives to apt-key
and add-apt-repository
, respectively. First will be adding an external repository using a public key with gpg
instead of using apt-key
. Second, as an addendum, this tutorial will cover adding an external repository using a keyserver with gpg
as an alternative to using add-apt-repository
.
To complete this tutorial, you will need an Ubuntu 22.04 server. Be sure to set this up according to our initial server setup guide for Ubuntu 22.04, with a non-root user with sudo
privileges and a firewall enabled.
PGP, or Pretty Good Privacy, is a proprietary encryption program used for signing, encrypting, and decrypting files and directories. PGP files are public key files, which are used in this process to authenticate repositories as valid sources within apt
. GPG, or GNU Privacy Guard, is an open-source alternative to PGP. GPG files are usually keyrings, which are files that hold multiple keys. Both of these file types are commonly used to sign and encrypt files.
gpg
is GPG’s command line tool that can be used to authorize external repositories for use with apt
. However, gpg
only accepts GPG files. In order to use this command line tool with PGP files, you must convert them.
Elasticsearch presents a common scenario for key conversion, and will be used as the example for this section. You’ll download a key formatted for PGP and convert it into an apt
compatible format with a .gpg
file extension. You’ll do this by running the gpg
command with the --dearmor
flag. Next, you’ll add the repository link to the list of package sources, while attaching a direct reference to your converted key. Finally, you will verify this process by installing the Elasticsearch package.
Projects that require adding repositories with key verification will always provide you with a public key and a repository URI representing its exact location. For our Elasticsearch example, the documentation gives these components on their installation page.
Here are the components given for Elasticsearch:
https://artifacts.elastic.co/GPG-KEY-elasticsearch
https://artifacts.elastic.co/packages/7.x/apt stable main
Next, you have to determine whether you are given a PGP or GPG file to work with. You can inspect at the key file by opening the URL with curl
:
- curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch
This will output the contents of the key file, which starts with the following:
Output-----BEGIN PGP PUBLIC KEY BLOCK-----
. . .
Despite having GPG
in the URL, the first line indicates that this is actually a PGP key file. Take note of this, because apt
only accepts the GPG format. Originally, apt-key
detected PGP files and converted it into GPG automatically by calling gpg
in the background. Step 2 will cover both manual conversion from PGP to GPG, and also what to do when conversion is not needed.
apt
Compatible File TypeWith the gpg
method, you must always download the key before adding to the list of package sources. Previously with apt-key
, this ordering was not always enforced. Now, you are required to reference the path to the downloaded key file in your sources list. If you have not downloaded the key, you obviously cannot reference an existing path.
With Elasticsearch you are working with a PGP file, so you will convert it to a GPG file format after download. The following example uses curl
to download the key, with the download being piped into a gpg
command. gpg
is called with the --dearmor
flag to convert the PGP key into a GPG file format, with -o
used to indicate the file output.
On Ubuntu, the /usr/share/keyrings
directory is the recommended location for your converted GPG files, as it is the default location where Ubuntu stores its keyrings. The file is named elastic-7.x.gpg
in this example, but any name works:
- curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elastic-7.x.gpg
This converts the PGP file into the correct GPG format, making it ready to be added to the list of sources for apt
.
Note: If the downloaded file was already in a GPG format, you could instead download the file straight to /usr/share/keyrings
without converting it using a command like the following example:
- curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo tee /usr/share/keyrings/elastic-7.x.gpg
In this case, the curl
command’s output would be piped into tee
to save the file in the correct location.
With the key downloaded and in the correct GPG file format, you can add the repository to the apt
packages source while explicitly linking it to the key you obtained. There are three methods to achieve this, all of which are related to how apt
finds sources. apt
pulls sources from a central sources.list
file, .list
files in the sources.list.d
directory, and .source
files in the sources.list.d
directory. Though there is no functional difference between the three options, it is recommended to consider the three options and choose the method that best fits your needs.
sources.list
DirectlyThe first method involves inserting a line representing the source directly into /etc/apt/sources.list
, the primary file containing apt
sources. There are multiple sources in this file, including the default sources that come with Ubuntu. It is perfectly acceptable to edit this file directly, though Option 2 and Option 3 will present a more modular solution that can be easier to edit and maintain.
Open /etc/apt/sources.list
with nano
or your preferred text editor:
- sudo nano /etc/apt/sources.list
Then add the external repository to the bottom of the file:
. . .
deb [arch=amd64,arm64 signed-by=/usr/share/keyrings/elastic-7.x.gpg] https://artifacts.elastic.co/packages/7.x/apt stable main
This line contains the following information about the source:
deb
: This specifies that the source uses a regular Debian architecture.arch=amd64,arm64
specifies the architectures the APT data will be downloaded to. Here it is amd64
and arm64
.signed-by=/usr/share/keyrings/elastic-7.x.gpg
: This specifies the key used to authorize this source, and here it points towards your .gpg
file stored in /usr/share/keyrings
. This portion of the line must be included, while it previously wasn’t required in the apt-key
method. This addition is the most critical change in porting away from apt-key
, since it ties the key to a singular repository it is allowed to authorize and fixes the original security flaw in apt-key
.https://artifacts.elastic.co/packages/7.x/apt stable main
: This is the URI representing the exact location the data within the repository can be found./etc/apt/sources.list.d/elastic-7.x.list
: This is the location and name of the new file to be created./dev/null
: This is used when the output of a command is not necessary. Pointing tee
to this location omits the output.Save and exit by hitting CTRL+O
then CTRL+X
.
.list
File in sources.list.d
With this option, you will instead create a new file in the sources.list.d
directory. apt
parses both this directory and sources.list
for repository additions. This method allows you to physically isolate repository additions within separate files. If you ever need to later remove this addition or make edits, you can delete this file instead of editing the central sources.list
file. Keeping your additions separate makes it easier to maintain, and editing sources.list
can be more error prone in a way that affects other repositories in the file.
To do this, pipe an echo
command into a tee
command to create this new file and insert the appropriate line. The file is named elastic-7.x.list
in the following example, but any name works as long as it is a unique filename in the directory:
- echo "deb [arch=amd64,arm64 signed-by=/usr/share/keyrings/elastic-7.x.gpg] https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-7.x.list > /dev/null
This command is identical to manually creating the file and inserting the appropriate line of text.
.sources
File in sources.list.d
The third method writes to a .sources
file instead of a .list
file. This method is relatively new, and uses the deb822
multiline format that is less ambiguous compared to the deb . . .
declaration, though is functionally identical. Create a new file:
- sudo nano /etc/apt/sources.list.d/elastic-7.x.sources
Then add the external repository using the deb822
format:
Types: deb
Architectures: amd64 arm64
Signed-By: /usr/share/keyrings/elastic-7.x.gpg
URIs: https://artifacts.elastic.co/packages/7.x/apt
Suites: stable
Components: main
Save and exit after you’ve inserted the text.
This is analogous to the one-line format, and doing a line-by-line comparison shows that the information in both is identical, just organized differently. One thing to note is that this format doesn’t use commas when there are multiple arguments (such as with amd64,arm64
), and instead uses spaces.
Next you will verify this process by doing a test installation.
You must call apt update
in order to prompt apt
to look through the main sources.list
file, and all the .list
and .sources
files in sources.list.d
. Calling apt install
without an update first will cause a failed install, or an installation of an out-of-date default package from apt
.
Update your repositories:
- sudo apt update
Then install your package:
- sudo apt install elasticsearch
Nothing changes in this step compared to the apt-key
method. Once this command finishes you will have completed the installation.
This section will briefly go over using gpg
with a keyserver instead of a public key to add an external repository. The process is nearly identical to the public key method, with the difference being how gpg
is called.
add-apt-repository
is the keyserver based counterpart to apt-key
, and both are up for deprecation. This scenario uses different components. Instead of a key and repository, you are given a keyserver URL and key ID. In this case, you can download from the keyserver directly into the appropriate .gpg
format without having to convert anything. Because add-apt-repository
will soon be deprecated, you will instead use gpg
to download to a file while overriding the default gpg
behavior of importing to an existing keyring.
Using the open-source programming language R as an example, here are the given components, which can also be found in the installation instructions on the official project site:
keyserver.ubuntu.com
E298A3A825C0D65DFD57CBB651716619E084DAB9
https://cloud.r-project.org/bin/linux/ubuntu jammy-cran40/
First, download from the keyserver directly using gpg
. Be aware that depending on download traffic, this download command may take a while to complete:
- sudo gpg --homedir /tmp --no-default-keyring --keyring /usr/share/keyrings/R.gpg --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9
This command includes the following flags, which are different from using gpg
with a public key:
--no-default-keyring
combined with --keyring
allows outputting to a new file instead of importing into an existing keyring, which is the default behavior of gpg
in this scenario.--keyserver
combined with --recv-keys
provides the specific key and location you’re downloading from.--homedir
is used to overwrite the gpg
default location for creating temporary files. gpg
needs to create these files to complete the command, otherwise gpg
will attempt to write to /root
which causes a permission error. Instead, this command places the temporary files in the appropriate /tmp
directory.Next, add the repository to a .list
file. This is done in the exact same manner as adding an external repository using a public key by piping an echo
command into a tee
command:
- echo "deb [arch=amd64 signed-by=/usr/share/keyrings/R.gpg] https://cloud.r-project.org/bin/linux/ubuntu jammy-cran40/" | sudo tee /etc/apt/sources.list.d/R.list > /dev/null
Next, update your list of repositories:
- sudo apt update
Then you can install the package:
- sudo apt install r-base
Using gpg
to add external repositories is similar between public keys and keyservers, with the difference being how you call gpg
.
Adding an external repository using a public key or a keyserver can be done through gpg
, without using apt-key
or add-apt-repository
as an intermediary. Use this method to ensure your process does not become obsolete in future Ubuntu versions, as apt-key
and add-apt-repository
are deprecated and will be removed in a future version. Adding external repositories using gpg
ensures that a key will only be used to authorize a single repository as you intend.
Authentication: PermitRootLogin yes PasswordAuthentication yes
ChallengeResponseAuthentication yes
UsePAM no
X11Forwarding yes PrintMotd no
]]>I logged to de DO dashboard and there was an update for the console, which I installed. Don’t know if that change is related to the problem.
Any ideas on how to troubleshoot this? :(
]]>After reading An Introduction to the Linux Terminal and A Linux Command Line Primer, you should have a good grasp on the fundamentals of working in a modern command line environment. However, many users who are new to the command line may still find it fairly restrictive. This tutorial is designed for these users to provide more background on command line interfaces, as well as advice around customization and cross-platform portability. The goal is to feel just as comfortable in a terminal environment as you are with using a computer in any other way.
Because getting started with a terminal on Windows can be less intuitive than other platforms, the first section of this tutorial will cover Windows terminal environments. If you are using macOS or Linux, you can skip to the following section.
On Windows, there are many choices for a Terminal equivalent. Historically, Windows did not use Unix-style command line shells, such as bash
, which have been ubiquitous on macOS and Linux since the early 2000s. It also lacked special features for highlighting text, opening multiple tabs, and so on. This is because it did not use common command-line terminal interfaces, sometimes called terminal emulators because they emulate the interfaces of older non-graphical computers.
Instead, Windows offered two of its own native command line interfaces: the Windows Command Prompt, and from Windows 7 onward, the Microsoft PowerShell. The Command Prompt, also called cmd.exe
, uses legacy MS-DOS syntax with relatively few additions. PowerShell provides somewhat more modern syntax relative to cmd.exe
(where “modern” in this context means “closer to a modern macOS or Linux shell”), as well as signal handling functions specific to certain Windows software, making it useful for Windows administrators.
Neither of these Windows shells include many fundamental features of modern Unix-style shells, and they are generally not well suited to most cloud development. Because of this, users who needed to work on cloud servers from Windows would usually install software like PuTTY (a tty
is the historical name of a Unix terminal), mobaXterm, or ConEmu. Each of these applications would usually include their own terminal interface, as well as a built-in SSH client for connecting to remote servers. These two features are not usually thought of as analogous on other platforms, but on Windows, it was usually a safe assumption that if you needed a Unix-style terminal, you were going to be working on a remote server, so they were often packaged together. For this reason, using a dedicated SSH GUI like PuTTY is still a popular way of working with cloud servers from Windows.
On other platforms, ssh
is just a command-line program that you can run from a terminal, and it is part of a core group of Linux utilities. It is also possible to install these core Linux utilities on Windows, along with a port of the standard bash
shell. Originally, this functionality was provided by the Cygwin project, which includes ports of many other Linux system tools. Installing Git on Windows also provides its own bash
shell from the MSYS2 project, which shares some functionality and upstream code with Cygwin. This would allow you to, for example, ssh
from the command line on Windows – as well as using common Linux utilities like grep
or cat
– without needing another program to provide this functionality.
These ports of Linux utilities to Windows are typically robust and well-maintained. However, they have a couple of significant downsides which have made them less popular than using similar functionality on macOS or Linux. One is that they generally do not include full-fledged package managers for installing other command line software as needed, which is a core assumption of most Linux environments (also provided by tools like Homebrew on macOS). In recent years, Windows has gained its own ecosystem of command line package managers, such as Chocolatey and Scoop, that can be used with Git’s bash
shell or other environments. Like Homebrew on macOS, these are especially useful on a local machine because they can be used to install desktop software in addition to command line tools. However, because most cloud software is still not designed to run natively on Windows, Chocolatey and Scoop’s package repositories are often less complete than their macOS or Linux equivalents. Using them often requires you to translate install documentation written for more common platforms into a working Windows equivalent, making them unintuitive for beginner users.
The other downside is that PuTTy, Git Bash, and other all-in-one Windows terminal environments usually have very barebones UI features, lacking support for syntax highlighting or tabs. Because the Windows Command Prompt has not visibly improved for some time, Windows users may be pushed toward more awkward workflows than users of a modern terminal like iTerm2 on macOS. To approximate an environment like this, you could instead combine Git’s bash
shell, Chocolatey’s package manager, and ConEmu’s terminal customization. This produces a very usable result, but requires a unique configuration that can be considered brittle or less reproducible than other platforms.
Note: It is also possible to open multiple tabs within a single shell without needing to rely on any additional features of your terminal emulator by using a Terminal Multiplexer such as tmux. However, this usually has a higher learning curve than making use of application-level features.
Nowadays, you can use the new Windows Terminal along with WSL2, also called the Windows Subsystem for Linux. These two projects substantially solve many outstanding issues. The Windows Terminal provides similar functionality to ConEmu, such as tabs, highlighting, and modern text rendering, and it can be used with any installed shell. It is also installed by default on Windows 11, significantly lowering the barrier to access and the need for third-party tools to do this work. WSL2 accomplishes a related goal: by allowing users to install a Linux compatibility environment that runs within Windows and is directly supported by Microsoft, they (mostly) no longer need to consider the entire set of other Windows command line tools.
WSL2 is a full Linux environment that runs within Windows. Because it is not a port of Linux tools to Windows, it comes with significant advantages. When working within a WSL2 environment, you can use apt
or Homebrew or any other native Linux tools to install and run software exactly as you would on a cloud server. This also has some downsides. Unlike with a macOS or Linux terminal, you can’t run native desktop software from WSL2, only Linux software that is installed into your Linux environment. In many cases this will be sufficient, especially if you are mostly using your terminal to deploy software to remote servers or make small configuration changes. However, you may still want to configure Windows Terminal to launch multiple different shell environments depending on your needs: for example, one using Git Bash and Chocolatey, which works with your native Windows software, and one using WSL2, which provides a full package manager and allows you to follow Linux documentation as-is. Many of these tools now have mutual support for one another built in, making this relatively straightforward, and very powerful.
Although bash
is the most common shell on modern Linux distributions, and its syntax is considered the most widely compatible with most environments, it is not the only one. Bourne shell syntax, or /bin/sh
, is a subset of bash
, and is sometimes still used in minimal environments such as containers. There is also the Z shell, or zsh
, which is becoming more popular due to its more flexible MIT license and its configurability. As of 2020, it is the default Terminal shell on macOS, and it is widely available in other environments.
Note: You can change your default shell in any modern command line environment by using the chsh command. This allows you to switch between bash
, zsh
, or others.
Zsh provided earlier and more widespread support for theming, text highlighting, and rendering of non-text characters (also called glyphs, essentially an earlier form of Emoji) than the default bash
shell in many environments. Because of this, there is a larger ecosystem of terminal customization tools for Zsh, such as Oh My Zsh. More importantly, Zsh tools and documentation usually prioritize installing fonts with supports for complex glyphs, such as the Powerline fonts, which is helpful for solving text rendering problems in other environments.
One downside that potential Zsh users quickly realize is that zsh
and bash
do not have strictly identical syntax, and bash
shell scripts are the most common by far. While most everyday shell conventions for navigating directories are the same, some differences arise. This includes testing equivalency with comparison operators, complex filesystem search strings, and so on.
As a general rule, if you are writing a shell script with syntax more complex than bash
accommodates, you should consider a different scripting language. Shell syntax is powerful, but it can also be unwieldy. The Go language, for example, has become popular for writing command line tools that use more advanced flow control than is appropriate for shell scripts. With some exceptions, even dedicated users of zsh
do not usually write and maintain standalone zsh
scripts.
With that in mind, you should not worry about any compatibility issues arising from using zsh
as your default, interactive shell. Virtually all environments where you can run zsh
will also have the bash
interpreter installed to run any bash
scripts as needed. When you are writing a standalone shell script, or running a shell script that you downloaded from elsewhere, the first line will normally contain #!/bin/bash
. This is known as a shebang, or an interpreter directive, which tells the computer which program, or interpreter, to use to run the script by default. This is especially important for shell scripts which cannot otherwise be distinguished by their file extension: both bash
scripts and other shell scripts all end in .sh
. Because bash
is always installed at /bin/bash
in compatible environments (including Git Bash on Windows), providing a complete path here is a widespread and safe convention.
An example of advanced shell syntax that is supported in zsh
by default, but has to be enabled manually in bash
, is the globstar, or **
pattern. Globbing is another name for performing fuzzy-match searching, i.e., searching for files using wildcard *
characters. The **
globstar allows you to use wildcard substitution not only within a single filename, but for entire directories as well. For example, if you were searching for a file named config.txt
somewhere in your home directory, but you didn’t know where to look, you might need to use a command like find
, which is designed to recursively search through nested directories. Even if you already know find
syntax, this would add an additional step to your process where you might otherwise have used a wildcard.
By using a globstar, you could instead provide a path like ~/**/config.txt
to any other command. The shell would automatically expand the search for you, without needing to invoke find
directly. Shells are good at providing this kind of functionality — what would normally require an entire other dependency can be incorporated into a single additional character substitution. To enable globstar use in bash,
you can run shopt -s globstar
.
- shopt -s globstar
You can also experiment with glob search strings using DigitalOcean’s Glob Tool.
As mentioned, the Go language has become very powerful for building modern command line tools. Another relatively new programming language that is especially relevant to command line environments is Rust. Rust is a low-level language that is generally considered similar to C++ and other C-likes, but with much more modern syntax and less of C’s accumulated baggage. Rust is popular for many use cases, including WebAssembly, but it is particularly useful for rewriting C code.
Nearly all of the core utilities that are thought of as essential to command line environments, such as ssh
, curl
, and cat
, are written in the C language, in many cases going back multiple decades. This is why many of them have so many dozens of options that have been added over years of active maintenance. In many ways, Linux has been built up around these specific tools, and they are unlikely ever to be officially replaced. However, there have been recent efforts to develop improved versions of each of them using Rust.
Note: “Core utilities,” or coreutils, is generally taken to be the actual name for this collection of programs. Many of these are often assumed to be actual terminal commands rather than programs – for example, cd
is terminal syntax, whereas cat
is a small, replaceable, program. On macOS, you can even use the Homebrew package manager to replace the built-in macOS coreutils with the more common GNU/Linux coreutils.
For example, bat provides more syntax highlighting and similar features to cat
, dust is similar to using du
to check disk usage but with more sophisticated graphical output, and ripgrep, or rg
, is a much faster implementation of grep
.
The relative advantages of these new Rust tools should not be taken as a judgment on C or Rust themselves, or on their relative performance. Rather, it is a consequence of maintaining the same codebases for a long time. Many core Linux utilities are not prioritizing getting faster, and in most cases, maintaining these utilities is about ensuring that they do not break compatibility with any legacy functionality. Most of these Rust utilities have been added to different platforms’ package managers, but they are still treated as new and optional.
The most significant downside to these new Rust tools is that many people may actually forget to use them, since their habits of using cat
or grep
will quickly become ingrained. To work around this, or to further customize your shell environment in general, you can use a profile file.
Both bash
and zsh
support profile files. A profile file is a file in your home directory called .bash_profile
or .bashrc
for bash, or .zshrc
for zsh. Often, they are created automatically when you first launch your shell, but because they start with a .
character, they are treated by the system as hidden, and remain invisible unless edited directly. Profile files can contain a number of settings that help to initialize your shell environment, such as aliases from one command to another.
With this method, you can create an alias in your ~/.bash_profile
to make the grep
command always run rg
instead of your system grep
. As mentioned, these Rust tools are not usually installed by default, so you would first need to install rg
. If you are using Homebrew, you can install it from the provided ripgrep package.
- brew install ripgrep
Next, open ~/.bash_profile
using nano
or your favorite text editor. This file may or may not already exist if you already have some profile settings configured.
- nano ~/.bash_profile
Then, add a line to the end of the file that includes the alias
command:
…
alias grep='rg'
Save and close the file, then close and open a new bash
terminal. From now on, when you run grep
, you’ll get rg
instead:
- grep --version
Outputripgrep 12.1.1
-SIMD -AVX (compiled)
+SIMD +AVX (runtime)
You should be aware that if you copy and paste a grep
command that you find elsewhere, and it uses functionality that isn’t present in rg
, it may not work correctly. For this reason, you should use aliases sparingly. However, they are useful for creating shortcuts that do not outright replace other commands. For example, the Python programming environment comes with a built-in webserver that can serve any small static sites by running python -m http.server port_number
. If you use this feature often, you may want to create an alias along the lines of alias pyserver="python -m http.server 8000"
.
Note: Aliases in ~/.bash_profile
will not override any commands in standalone bash scripts, because your ~/.bash_profile
is only loaded into interactive bash shells by default. You can manually load your ~/.bash_profile
by using the command source ~/.bash_profile
, but it is usually better for compatibility reasons to run standalone scripts in a stock environment.
Each terminal session is configured with a number of environment variables by default. Many of these are set automatically by the system, but you can provide additional overrides in ~/bash_profile
, or interactively within a single shell session. To view your current environment variables, run env
:
- env
OutputSHELL=/usr/local/bin/bash
ITERM_PROFILE=bash
COLORTERM=truecolor
XPC_FLAGS=0x0
TERM_PROGRAM_VERSION=3.4.15
…
Many of these contain information about your shell itself, which will not be particularly useful. One exception to this is the PATH
variable, which contains a list of all the directories that are automatically checked to run command-line programs. You can pipe with grep
to output only the line containing the PATH
variable from the env
command:
- env | grep PATH
OutputPATH=/usr/local/opt/mysql-client/bin:/Users/sammy/.gem/ruby/3.0.0/bin:/usr/local/opt/ruby/bin:/usr/local/lib/ruby/gems/3.0.0/bin:/Users/sammy/.cargo/bin:/Users/sammy/.rbenv/shims:/Users/sammy/.pyenv/shims:/Users/sammy/.pyenv/bin:/Users/sammy/Library/Python/3.9/bin:/usr/local/opt/gnu-sed/libexec/gnubin:/usr/local/sbin:/usr/local/opt/libpq/bin:/usr/local/opt/coreutils/libexec/gnubin:/usr/local/opt/findutils/libexec/gnubin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/opt/X11/bin:/Library/Apple/usr/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands
Note the directories in this list are separated by a :
character. In general, your PATH variable will be longer on Windows or macOS than on Linux. Linux tries to enforce installing command-line programs only into directories like bin
or /usr/local/bin
, whereas Windows and Mac software is often installed to other directories that need to be added to your PATH for those commands to be available on the command line. Installing software via a package manager usually takes care of this. While you can manually move programs into /usr/local/bin
or another directory on your path, be aware that package managers expect to be able to manipulate the contents of these directories. Any programs that you install manually could be overwritten, or cause an error when attempting to be overwritten.
You can check the installed location of a command line program by using the which
command:
- which python
Output/Users/sammy/.pyenv/shims/python
This is useful for verifying which version of a program will run by default when, for example, python
is run from the command line. If a program is present in multiple directories on your PATH, the directory that is listed first in your PATH variable will take precedence. Python is a notorious example. Because macOS and Linux both include a version of Python with the system that is sometimes updated either too frequently or too infrequently, tools like pyenv are designed to register a separate installation of Python as early on your path as possible. This ensures that you are always working directly with and installing additional libraries for the pyenv
-provided Python.
Any time you do not get the expected result from a command, ask yourself these questions:
The variable inheritance behavior of your PATH and your shell environment is usually straightforward — you may just have multiple programs making other, conflicting assumptions. Knowing how to check and configure your environment will go a long way toward helping you be comfortable on the command line.
In this tutorial, you reviewed many nuances of terminal environments, including configuration on Windows, differences between bash
and other shells, modern command-line utilities, and environment variables including paths and aliases. Many of these are overlooked by new developers, and each of them can make working either locally or in the cloud much more pleasant and effective.
Next, you may want to learn to work with DigitalOcean’s command line client, doctl.
]]>I recently regenerated my public/private keys for SSH access to all of my droplets. I successfully updated the public key on all accounts except for one. I used the cat ~/.ssh/id_rsa.pub
command to display the public key in Terminal and copied it to the Settings > Security > SSH Keys section on my client’s account. After doing that, I tried to login using SSH via the Terminal, but Permission denied (publickey) is returned. I have reviewed many related issues on Stack Overflow and other tech help websites, but nothing has helped resolve my issue.
Can someone help me out with resolving this issue?
]]>Most modern Unix-like operating systems offer a centralized mechanism for finding and installing software. Software is usually distributed in the form of packages, kept in repositories. Working with packages is known as package management. Packages provide the core components of an operating system, along with shared libraries, applications, services, and documentation.
A package management system does much more than one-time installation of software. It also provides tools for upgrading already-installed packages. Package repositories help to ensure that code has been vetted for use on your system, and that the installed versions of software have been approved by developers and package maintainers.
When configuring servers or development environments, it’s often necessary to look beyond official repositories. Packages in the stable release of a distribution may be out of date, especially where new or rapidly-changing software is concerned. Nevertheless, package management is a vital skill for system administrators and developers, and the wealth of packaged software for major distributions is a tremendous resource.
This guide is intended as a quick reference for the fundamentals of finding, installing, and upgrading packages on a variety of distributions, and should help you translate that knowledge between systems.
Most package systems are built around collections of package files. A package file is usually an archive which contains compiled applications and other resources used by the software, along with installation scripts. Packages also contain valuable metadata, including their dependencies, a list of other packages required to install and run them.
While their functionality and benefits are broadly similar, packaging formats and tools vary by platform:
.deb
packages installed by apt
and dpkg
.rpm
packages installed by yum
.txz
packages installed by pkg
In Debian and systems based on it, like Ubuntu, Linux Mint, and Raspbian, the package format is the .deb
file. apt
, the Advanced Packaging Tool, provides commands used for most common operations: Searching repositories, installing collections of packages and their dependencies, and managing upgrades. apt
commands operate as a front-end to the lower-level dpkg
utility, which handles the installation of individual .deb
files on the local system, and is sometimes invoked directly.
Recent releases of most Debian-derived distributions include a single apt
command, which offers a concise and unified interface to common operations that have traditionally been handled by the more-specific apt-get
and apt-cache
.
Rocky Linux, Fedora, and other members of the Red Hat family use RPM files. These used to use a package manager called yum
. In recent versions of Fedora and its derivatives, yum
has been supplanted by dnf
, a modernized fork which retains most of yum
’s interface.
FreeBSD’s binary package system is administered with the pkg
command. FreeBSD also offers the Ports Collection, a local directory structure and tools which allow the user to fetch, compile, and install packages directly from source using Makefiles. It’s usually much more convenient to use pkg
, but occasionally a pre-compiled package is unavailable, or you may need to change compile-time options.
Most systems keep a local database of the packages available from remote repositories. It’s best to update this database before installing or upgrading packages. As a partial exception to this pattern, dnf
will check for updates before performing some operations, but you can ask at any time whether updates are available.
sudo apt update
dnf check-update
sudo pkg update
sudo portsnap fetch update
Making sure that all of the installed software on a machine stays up to date would be an enormous undertaking without a package system. You would have to track upstream changes and security alerts for hundreds of different packages. While a package manager doesn’t solve every problem you’ll encounter when upgrading software, it does enable you to maintain most system components with a few commands.
On FreeBSD, upgrading installed ports can introduce breaking changes or require manual configuration steps. It’s best to read /usr/ports/UPDATING
before upgrading with portmaster
.
sudo apt upgrade
sudo dnf upgrade
sudo pkg upgrade
Most distributions offer a graphical or menu-driven front end to package collections. These can be a good way to browse by category and discover new software. Often, however, the quickest and most effective way to locate a package is to search with command-line tools.
apt search search_string
dnf search search_string
pkg search search_string
Note: On Rocky, Fedora, or RHEL, you can search package titles and descriptions together by using dnf search all
. On FreeBSD, you can search descriptions by using pkg search -D
When deciding what to install, it’s often helpful to read detailed descriptions of packages. Along with human-readable text, these often include metadata like version numbers and a list of the package’s dependencies.
apt show package
dnf info package
pkg info package
cd /usr/ports/category/port && cat pkg-descr
Once you know the name of a package, you can usually install it and its dependencies with a single command. In general, you can supply multiple packages to install at once by listing them all.
sudo apt install package
sudo dnf install package
sudo pkg install package
Sometimes, even though software isn’t officially packaged for a given operating system, a developer or vendor will offer package files for download. You can usually retrieve these with your web browser, or via curl
on the command line. Once a package is on the target system, it can often be installed with a single command.
On Debian-derived systems, dpkg
handles individual package files. If a package has unmet dependencies, gdebi
can often be used to retrieve them from official repositories.
On On Rocky Linux, Fedora, or RHEL, dnf
is used to install individual files, and will also handle needed dependencies.
sudo dpkg -i package.deb
sudo dnf install package.rpm
sudo pkg add package.txz
Since a package manager knows what files are provided by a given package, it can usually remove them cleanly from a system if the software is no longer needed.
sudo apt remove package
sudo dnf erase package
sudo pkg delete package
In addition to web-based documentation, keep in mind that Unix manual pages (usually referred to as man pages) are available for most commands from the shell. To read a page, use man
:
- man page
In man
, you can navigate with the arrow keys. Press / to search for text within the page, and q to quit.
man apt
man dnf
man pkg
man ports
This guide provides an overview of operations that can be cross-referenced between systems, but only scratches the surface of a complex topic. For greater detail on a given system, you can consult the following resources:
dnf
, and an official manual for dnf
itself.pkg
.Hope that this is helpful! If you have any questions post the below!
]]>Every computer system benefits from proper administration and monitoring. Keeping an eye on how your system is running will help you discover issues and resolve them quickly.
There are plenty of command line utilities created for this purpose. This guide will introduce you to some of the most helpful applications to have in your toolbox.
To follow along with this guide, you will need access to a computer running a Linux-based operating system. This can either be a virtual private server which you’ve connected to with SSH or your local machine. Note that this tutorial was validated using a Linux server running Ubuntu 20.04, but the examples given should work on a computer running any version of any Linux distribution.
If you plan to use a remote server to follow this guide, we encourage you to first complete our Initial Server Setup guide. Doing so will set you up with a secure server environment — including a non-root user with sudo
privileges and a firewall configured with UFW — which you can use to build your Linux skills.
You can see all of the processes running on your server by using the top
command:
- top
Outputtop - 15:14:40 up 46 min, 1 user, load average: 0.00, 0.01, 0.05
Tasks: 56 total, 1 running, 55 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1019600k total, 316576k used, 703024k free, 7652k buffers
Swap: 0k total, 0k used, 0k free, 258976k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 24188 2120 1300 S 0.0 0.2 0:00.56 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.07 ksoftirqd/0
6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
7 root RT 0 0 0 0 S 0.0 0.0 0:00.03 watchdog/0
8 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 cpuset
9 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper
10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
The first several lines of output provide system statistics, such as CPU/memory load and the total number of running tasks.
You can see that there is 1 running process, and 55 processes that are considered to be sleeping because they are not actively using CPU cycles.
The remainder of the displayed output shows the running processes and their usage statistics. By default, top
automatically sorts these by CPU usage, so you can see the busiest processes first. top
will continue running in your shell until you stop it using the standard key combination of Ctrl+C
to exit a running process. This sends a kill
signal, instructing the process to stop gracefully if it is able to.
An improved version of top
, called htop
, is available in most package repositories. On Ubuntu 20.04, you can install it with apt
:
- sudo apt install htop
After that, the htop
command will be available:
- htop
Output Mem[||||||||||| 49/995MB] Load average: 0.00 0.03 0.05
CPU[ 0.0%] Tasks: 21, 3 thr; 1 running
Swp[ 0/0MB] Uptime: 00:58:11
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
1259 root 20 0 25660 1880 1368 R 0.0 0.2 0:00.06 htop
1 root 20 0 24188 2120 1300 S 0.0 0.2 0:00.56 /sbin/init
311 root 20 0 17224 636 440 S 0.0 0.1 0:00.07 upstart-udev-brid
314 root 20 0 21592 1280 760 S 0.0 0.1 0:00.06 /sbin/udevd --dae
389 messagebu 20 0 23808 688 444 S 0.0 0.1 0:00.01 dbus-daemon --sys
407 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.02 rsyslogd -c5
408 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.00 rsyslogd -c5
409 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.00 rsyslogd -c5
406 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.04 rsyslogd -c5
553 root 20 0 15180 400 204 S 0.0 0.0 0:00.01 upstart-socket-br
htop
provides better visualization of multiple CPU threads, better awareness of color support in modern terminals, and more sorting options, among other features. Unlike top
, It is not always installed by default, but can be considered a drop-in-replacement. You can exit htop
by pressing Ctrl+C
as with top
.
Here are some keyboard shortcuts that will help you use htop more effectively:
There are many other options that you can access through help or setup. These should be your first stops in exploring htop’s functionality. In the next step, you’ll learn how to monitor your network bandwidth.
If your network connection seems overutilized and you are unsure which application is the culprit, a program called nethogs
is a good choice for finding out.
On Ubuntu, you can install nethogs with the following command:
- sudo apt install nethogs
After that, the nethogs
command will be available:
- nethogs
OutputNetHogs version 0.8.0
PID USER PROGRAM DEV SENT RECEIVED
3379 root /usr/sbin/sshd eth0 0.485 0.182 KB/sec
820 root sshd: root@pts/0 eth0 0.427 0.052 KB/sec
? root unknown TCP 0.000 0.000 KB/sec
TOTAL 0.912 0.233 KB/sec
nethogs
associates each application with its network traffic.
There are only a few commands that you can use to control nethogs
:
iptraf-ng
is another way to monitor network traffic. It provides a number of different interactive monitoring interfaces.
Note: IPTraf requires a screen size of at least 80 columns by 24 lines.
On Ubuntu, you can install iptraf-ng
with the following command:
- sudo apt install iptraf-ng
iptraf-ng
needs to be run with root privileges, so you should precede it with sudo
:
- sudo iptraf-ng
You’ll be presented with a menu that uses a popular command line interface framework called ncurses
.
With this menu, you can select which interface you would like to access.
For example, to get an overview of all network traffic, you can select the first menu and then “All interfaces”. It will give you a screen that looks like this:
Here, you can see what IP addresses you are communicating on all of your network interfaces.
If you would like to have those IP addresses resolved into domains, you can enable reverse DNS lookup by exiting the traffic screen, selecting Configure
and then toggling on Reverse DNS lookups
.
You can also enable TCP/UDP service names
to see the names of the services being run instead of the port numbers.
With both of these options enabled, the display may look like this:
The netstat
command is another versatile tool for gathering network information.
netstat
is installed by default on most modern systems, but you can install it yourself by downloading it from your server’s default package repositories. On most Linux systems, including Ubuntu, the package containing netstat
is net-tools
:
- sudo apt install net-tools
By default, the netstat
command on its own prints a list of open sockets:
- netstat
OutputActive Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 192.241.187.204:ssh ip223.hichina.com:50324 ESTABLISHED
tcp 0 0 192.241.187.204:ssh rrcs-72-43-115-18:50615 ESTABLISHED
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags Type State I-Node Path
unix 5 [ ] DGRAM 6559 /dev/log
unix 3 [ ] STREAM CONNECTED 9386
unix 3 [ ] STREAM CONNECTED 9385
. . .
If you add an -a
option, it will list all ports, listening and non-listening:
- netstat -a
OutputActive Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:ssh *:* LISTEN
tcp 0 0 192.241.187.204:ssh rrcs-72-43-115-18:50615 ESTABLISHED
tcp6 0 0 [::]:ssh [::]:* LISTEN
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 6195 @/com/ubuntu/upstart
unix 2 [ ACC ] STREAM LISTENING 7762 /var/run/acpid.socket
unix 2 [ ACC ] STREAM LISTENING 6503 /var/run/dbus/system_bus_socket
. . .
If you’d like to filter to see only TCP or UDP connections, use the -t
or -u
flags respectively:
- netstat -at
OutputActive Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:ssh *:* LISTEN
tcp 0 0 192.241.187.204:ssh rrcs-72-43-115-18:50615 ESTABLISHED
tcp6 0 0 [::]:ssh [::]:* LISTEN
See statistics by passing the “-s” flag:
- netstat -s
OutputIp:
13500 total packets received
0 forwarded
0 incoming packets discarded
13500 incoming packets delivered
3078 requests sent out
16 dropped because of missing route
Icmp:
41 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
echo requests: 1
echo replies: 40
. . .
If you would like to continuously update the output, you can use the -c
flag. There are many other options available to netstat which you learn by reviewing its manual page.
In the next step, you’ll learn some useful ways of monitoring your disk usage.
For a quick overview of how much disk space is left on your attached drives, you can use the df
program.
Without any options, its output looks like this:
- df
OutputFilesystem 1K-blocks Used Available Use% Mounted on
/dev/vda 31383196 1228936 28581396 5% /
udev 505152 4 505148 1% /dev
tmpfs 203920 204 203716 1% /run
none 5120 0 5120 0% /run/lock
none 509800 0 509800 0% /run/shm
This outputs disk usage in bytes, which may be a bit hard to read.
To fix this problem, you can specify output in a human-readable format:
- df -h
OutputFilesystem Size Used Avail Use% Mounted on
/dev/vda 30G 1.2G 28G 5% /
udev 494M 4.0K 494M 1% /dev
tmpfs 200M 204K 199M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 498M 0 498M 0% /run/shm
If you want to see the total disk space available on all filesystems, you can pass the --total
option. This will add a row at the bottom with summary information:
- df -h --total
OutputFilesystem Size Used Avail Use% Mounted on
/dev/vda 30G 1.2G 28G 5% /
udev 494M 4.0K 494M 1% /dev
tmpfs 200M 204K 199M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 498M 0 498M 0% /run/shm
total 32G 1.2G 29G 4%
df
can provide a useful overview. Another command, du
gives a breakdown by directory.
du
will analyze usage for the current directory and any subdirectories. The default output of du
running in a nearly-empty home directory looks like this:
- du
Output4 ./.cache
8 ./.ssh
28 .
Once again, you can specify human-readable output by passing it -h
:
- du -h
Output4.0K ./.cache
8.0K ./.ssh
28K .
To see file sizes as well as directories, type the following:
- du -a
Output0 ./.cache/motd.legal-displayed
4 ./.cache
4 ./.ssh/authorized_keys
8 ./.ssh
4 ./.profile
4 ./.bashrc
4 ./.bash_history
28 .
For a total at the bottom, you can add the -c
option:
- du -c
Output4 ./.cache
8 ./.ssh
28 .
28 total
If you are only interested in the total and not the specifics, you can issue:
- du -s
Output28 .
There is also an ncurses
interface for du
, appropriately called ncdu
, that you can install:
- sudo apt install ncdu
This will graphically represent your disk usage:
- ncdu
Output--- /root ----------------------------------------------------------------------
8.0KiB [##########] /.ssh
4.0KiB [##### ] /.cache
4.0KiB [##### ] .bashrc
4.0KiB [##### ] .profile
4.0KiB [##### ] .bash_history
You can step through the filesystem by using the up and down arrows and pressing Enter on any directory entry.
In the last section, you’ll learn how to monitor your memory usage.
You can check the current memory usage on your system by using the free
command.
When used without options, the output looks like this:
- free
Output total used free shared buff/cache available
Mem: 1004896 390988 123484 3124 490424 313744
Swap: 0 0 0
To display in a more readable format, you can pass the -m
option to display the output in megabytes:
- free -m
Output total used free shared buff/cache available
Mem: 981 382 120 3 478 306
Swap: 0 0 0
The Mem
line includes the memory used for buffering and caching, which is freed up as soon as needed for other purposes. Swap
is memory that has been written to a swapfile
on disk in order to conserve active memory.
Finally, the vmstat
command can output various information about your system, including memory, swap, disk io, and cpu information.
You can use the command to get another view into memory usage:
- vmstat
Outputprocs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 0 99340 123712 248296 0 0 0 1 9 3 0 0 100 0
You can see this in megabytes by specifying units with the -S
flag:
- vmstat -S M
Outputprocs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 0 96 120 242 0 0 0 1 9 3 0 0 100 0
To get some general statistics about memory usage, type:
- vmstat -s -S M
Output 495 M total memory
398 M used memory
252 M active memory
119 M inactive memory
96 M free memory
120 M buffer memory
242 M swap cache
0 M total swap
0 M used swap
0 M free swap
. . .
To get information about individual system processes’ cache usage, type:
- vmstat -m -S M
OutputCache Num Total Size Pages
ext4_groupinfo_4k 195 195 104 39
UDPLITEv6 0 0 768 10
UDPv6 10 10 768 10
tw_sock_TCPv6 0 0 256 16
TCPv6 11 11 1408 11
kcopyd_job 0 0 2344 13
dm_uevent 0 0 2464 13
bsg_cmd 0 0 288 14
. . .
This will give you details about what kind of information is stored in the cache.
Using these tools, you should begin to be able to monitor your server from the command line. There are many other monitoring utilities that are used for different purposes, but these are a good starting point.
Next, you may want to learn about Linux process management using ps, kill, and nice.
]]>A Linux server, like any modern computer, runs multiple applications. These are referred to and managed as individual processes.
While Linux will handle the low-level, behind-the-scenes management in a process’s life-cycle – i.e., startup, shutdown, memory allocation, and so on – you will need a way of interacting with the operating system to manage them from a higher level.
In this guide, you will learn some fundamental aspects of process management. Linux provides a number of standard, built-in tools for this purpose.
You will explore these ideas in a Ubuntu 20.04 environment, but any modern Linux distribution will operate in a similar way.
You can see all of the processes running on your server by using the top
command:
- top
Outputtop - 15:14:40 up 46 min, 1 user, load average: 0.00, 0.01, 0.05
Tasks: 56 total, 1 running, 55 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1019600k total, 316576k used, 703024k free, 7652k buffers
Swap: 0k total, 0k used, 0k free, 258976k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 24188 2120 1300 S 0.0 0.2 0:00.56 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.07 ksoftirqd/0
6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
7 root RT 0 0 0 0 S 0.0 0.0 0:00.03 watchdog/0
8 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 cpuset
9 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper
10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
The first several lines of output provide system statistics, such as CPU/memory load and the total number of running tasks.
You can see that there is 1 running process, and 55 processes that are considered to be sleeping because they are not actively using CPU cycles.
The remainder of the displayed output shows the running processes and their usage statistics. By default, top
automatically sorts these by CPU usage, so you can see the busiest processes first. top
will continue running in your shell until you stop it using the standard key combination of Ctrl+C
to exit a running process. This sends a kill
signal, instructing the process to stop gracefully if it is able to.
An improved version of top
, called htop
, is available in most package repositories. On Ubuntu 20.04, you can install it with apt
:
- sudo apt install htop
After that, the htop
command will be available:
- htop
Output Mem[||||||||||| 49/995MB] Load average: 0.00 0.03 0.05
CPU[ 0.0%] Tasks: 21, 3 thr; 1 running
Swp[ 0/0MB] Uptime: 00:58:11
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
1259 root 20 0 25660 1880 1368 R 0.0 0.2 0:00.06 htop
1 root 20 0 24188 2120 1300 S 0.0 0.2 0:00.56 /sbin/init
311 root 20 0 17224 636 440 S 0.0 0.1 0:00.07 upstart-udev-brid
314 root 20 0 21592 1280 760 S 0.0 0.1 0:00.06 /sbin/udevd --dae
389 messagebu 20 0 23808 688 444 S 0.0 0.1 0:00.01 dbus-daemon --sys
407 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.02 rsyslogd -c5
408 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.00 rsyslogd -c5
409 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.00 rsyslogd -c5
406 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.04 rsyslogd -c5
553 root 20 0 15180 400 204 S 0.0 0.0 0:00.01 upstart-socket-br
htop
provides better visualization of multiple CPU threads, better awareness of color support in modern terminals, and more sorting options, among other features. Unlike top
, It is not always installed by default, but can be considered a drop-in-replacement. You can exit htop
by pressing Ctrl+C
as with top
. You can also learn more about how to use top and htop.
In the next section, you’ll learn about how to use tools to query specific processes.
top
and htop
provide a dashboard interface to view running processes similar to a graphical task manager. A dashboard interface can provide an overview, but usually does not return directly actionable output. For this, Linux provides another standard command called ps
to query running processes.
Running ps
without any arguments provides very little information:
- ps
Output PID TTY TIME CMD
1017 pts/0 00:00:00 bash
1262 pts/0 00:00:00 ps
This output shows all of the processes associated with the current user and terminal session. This makes sense if you are only running the bash
shell and this ps
command within this terminal currently.
To get a more complete picture of the processes on this system, you can run ps aux
:
- ps aux
OutputUSER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.2 24188 2120 ? Ss 14:28 0:00 /sbin/init
root 2 0.0 0.0 0 0 ? S 14:28 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S 14:28 0:00 [ksoftirqd/0]
root 6 0.0 0.0 0 0 ? S 14:28 0:00 [migration/0]
root 7 0.0 0.0 0 0 ? S 14:28 0:00 [watchdog/0]
root 8 0.0 0.0 0 0 ? S< 14:28 0:00 [cpuset]
root 9 0.0 0.0 0 0 ? S< 14:28 0:00 [khelper]
…
These options tell ps
to show processes owned by all users (regardless of their terminal association) in a more human-readable format.
By making use of pipes, you can search within the output of ps aux
using grep
, in order to return the name of a specific process. This is useful if you believe it has crashed, or if you need to stop it for some reason.
- ps aux | grep bash
Outputsammy 41664 0.7 0.0 34162880 2528 s000 S 1:35pm 0:00.04 -bash
sammy 41748 0.0 0.0 34122844 828 s000 S+ 1:35pm 0:00.00 grep bash
This returns both the grep
process you just ran, and the bash
shell that’s currently running. It also return their total memory and CPU usage, how long they’ve been running, and in the highlighted output above, their process ID. In Linux and Unix-like systems, each process is assigned a process ID, or PID. This is how the operating system identifies and keeps track of processes.
A quick way of getting the PID of a process is with the pgrep
command:
- pgrep bash
Output1017
The first process spawned at boot, called init, is given the PID of “1”.
- pgrep init
Output1
This process is then responsible for spawning every other process on the system. The later processes are given larger PID numbers.
A process’s parent is the process that was responsible for spawning it. Parent processes have a PPID, which you can see in the column headers in many process management applications, including top
, htop
and ps
.
Any communication between the user and the operating system about processes involves translating between process names and PIDs at some point during the operation. This is why these utilities will always include the PID in their output. In the next section, you’ll learn how to use PIDs to send stop, resume, or other signals to running processes.
All processes in Linux respond to signals. Signals are an operating system-level way of telling programs to terminate or modify their behavior.
The most common way of passing signals to a program is with the kill
command. As you might expect, the default functionality of this utility is to attempt to kill a process:
- kill PID_of_target_process
This sends the TERM signal to the process. The TERM signal tells the process to please terminate. This allows the program to perform clean-up operations and exit smoothly.
If the program is misbehaving and does not exit when given the TERM signal, you can escalate the signal by passing the KILL
signal:
- kill -KILL PID_of_target_process
This is a special signal that is not sent to the program.
Instead, it is given to the operating system kernel, which shuts down the process. This is used to bypass programs that ignore the signals sent to them.
Each signal has an associated number that can be passed instead of the name. For instance, You can pass “-15” instead of “-TERM”, and “-9” instead of “-KILL”.
Signals are not only used to shut down programs. They can also be used to perform other actions.
For instance, many processes that are designed to run constantly in the background (sometimes called “daemons”) will automatically restart when they are given the HUP
, or hang-up signal. The Apache webserver typically operates this way.
- sudo kill -HUP pid_of_apache
The above command will cause Apache to reload its configuration file and resume serving content.
Note: Many background processes like this are managed through system services which provide an additional surface for interacting with them, and it is usually preferable to restart the service itself rather than sending a HUP
signal directly to one running process. If you review the configuration files of various services, you may find that the various service restart
hooks are designed to do exactly that – send signals to specific processes – while also providing logs and other reporting.
You can list all of the signals that are possible to send with kill
with the -l
flag:
- kill -l
Output1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP
6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1
11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM
Although the conventional way of sending signals is through the use of PIDs, there are also methods of doing this with regular process names.
The pkill
command works in almost exactly the same way as kill
, but it operates on a process name instead:
- pkill -9 ping
The above command is the equivalent of:
- kill -9 `pgrep ping`
If you would like to send a signal to every instance of a certain process, you can use the killall
command:
- killall firefox
The above command will send the TERM signal to every instance of firefox
running on the computer.
Often, you will want to adjust which processes are given priority in a server environment.
Some processes might be considered mission critical for your situation, while others may be executed whenever there are leftover resources.
Linux controls priority through a value called niceness.
High priority tasks are considered less nice, because they don’t share resources as well. Low priority processes, on the other hand, are nice because they insist on only taking minimal resources.
When you ran top
at the beginning of the article, there was a column marked “NI”. This is the nice value of the process:
- top
[secondary_label Output]
Tasks: 56 total, 1 running, 55 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1019600k total, 324496k used, 695104k free, 8512k buffers
Swap: 0k total, 0k used, 0k free, 264812k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1635 root 20 0 17300 1200 920 R 0.3 0.1 0:00.01 top
1 root 20 0 24188 2120 1300 S 0.0 0.2 0:00.56 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.11 ksoftirqd/0
Nice values can range between -19/-20 (highest priority) and 19/20 (lowest priority) depending on the system.
To run a program with a certain nice value, you can use the nice
command:
- nice -n 15 command_to_execute
This only works when beginning a new program.
To alter the nice value of a program that is already executing, you use a tool called renice
:
- renice 0 PID_to_prioritize
Process management is a fundamental part of Linux that is useful in almost every context. Even if you aren’t performing any hands-on system administration, being able to chase down stuck processes and handle them carefully is very helpful.
Next, you may want to learn how to use netstat
and du
to monitor other server resources.
The Go programming language comes with a rich toolchain that makes obtaining packages and building executables incredibly easy. One of Go’s most powerful features is the ability to cross-build executables for any Go-supported foreign platform. This makes testing and package distribution much easier, because you don’t need to have access to a specific platform in order to distribute your package for it.
In this tutorial, you’ll use Go’s tools to obtain a package from version control and automatically install its executable. Then you’ll manually build and install the executable so you can be familiar with the process. Then you’ll build an executable for a different architecture, and automate the build process to create executables for multiple platforms. When you’re done, you’ll know how to build executables for Windows and macOS, as well as other platforms you want to support.
To follow this tutorial, you will need:
Before we can create executables from a Go package, we have to obtain its source code. The go get
tool can fetch packages from version control systems like GitHub. Under the hood, go get
clones packages into subdirectories of the $GOPATH/src/
directory. Then, if applicable, it installs the package by building its executable and placing it in the $GOPATH/bin
directory. If you configured Go as described in the prerequisite tutorials, the $GOPATH/bin
directory is included in your $PATH
environmental variable, which ensures that you can use installed packages from anywhere on your system.
The syntax for the go get
command is go get package-import-path
. The package-import-path
is a string that unique identifies a package. It’s often the location of the package in a remote repository like Github, or a directory within the $GOPATH/src/
directory on your machine.
It’s common to use go get
with the -u
flag, which instructs go get
to obtain the package and its dependencies, or update those dependencies if they’re already present on the machine.
In this tutorial, we will install Caddy, a web server written in Go. Per Caddy’s instructionsx, we’ll use github.com/mholt/caddy/caddy
for the package import path. Use go get
to fetch and install Caddy:
- go get -u github.com/mholt/caddy/caddy
The command will take some time to complete, but you won’t see any progress while it fetches the package and installs it. No output actually indicates that the command executed successfully.
When the command completes, you’ll find Caddy’s source code available at $GOPATH/src/github.com/mholt/caddy
. In addition, since Caddy has an executable, it was automatically created and stored in the $GOPATH/bin
directory. Verify this by using which
to print the location of the executable:
- which caddy
You’ll see the following output:
Output/home/sammy/work/bin/caddy
Note: The go get
command installs packages from the default branch of a Git repository, which in many cases is the master
, or in-development branch. Make sure to review the instructions, usually located in the repository’s README
file, before using go get
.
You can use Git commands like git checkout
to select a different branch on sources obtained using the go get
command. Review the tutorial How to Use Git Branches to learn more about how to switch branches.
Let’s look at the process of installing Go packages in more detail, starting with creating executables from packages we’ve already obtained.
The go get
command downloaded the source and installed Caddy’s executable for us in a single command. But you may want to rebuild the executable yourself, or build an executable from your own code. The go build
command builds executables.
Although we already installed Caddy, let’s build it again manually so we can get comfortable with the process. Execute go build
and specify the package import path:
- go build github.com/mholt/caddy/caddy
As before, no output indicates successful operation. The executable will be generated in your current directory, with the same name as the directory containing the package. In this case, the executable will be named caddy
.
If you’re located in the package directory, you can omit the path to the package and simply run go build
.
To specify a different name or location for the executable, use the -o
flag. Let’s build an executable called caddy-server
and place it in a build
directory within the current working directory:
- go build -o build/caddy-server github.com/mholt/caddy/caddy
This command creates the executable, and also creates the ./build
directory if it doesn’t exist.
Now let’s look at installing executables.
Building an executable creates the executable in the current directory or the directory of your choice. Installing an executable is the process of creating an executable and storing it in $GOPATH/bin
. The go install
command works just like go build
, but go install
takes care of placing the output file in the right place for you.
To install an executable, use go install
, followed by the package import path. Once again, use Caddy to try this out:
- go install github.com/mholt/caddy/caddy
Just like with go build
, you’ll see no output if the command was successful. And like before, the executable is created with the same name as the directory containing the package. But this time, the executable is stored in $GOPATH/bin
. If $GOPATH/bin
is part of your $PATH
environmental variable, the executable will be available from anywhere on your operating system. You can verify its location using which
command:
- which caddy
You’ll see the following output:
Output of which/home/sammy/work/bin/caddy
Now that you understand how go get
, go build
, and go install
work, and how they’re related, let’s explore one of the most popular Go features: creating executables for other target platforms.
The go build
command lets you build an executable file for any Go-supported target platform, on your platform. This means you can test, release and distribute your application without building those executables on the target platforms you wish to use.
Cross-compiling works by setting required environment variables that specify the target operating system and architecture. We use the variable GOOS
for the target operating system, and GOARCH
for the target architecture. To build an executable, the command would take this form:
- env GOOS=target-OS GOARCH=target-architecture go build package-import-path
The env
command runs a program in a modified environment. This lets you use environment variables for the current command execution only. The variables are unset or reset after the command executes.
The following table shows the possible combinations of GOOS
and GOARCH
you can use:
GOOS - Target Operating System |
GOARCH - Target Platform |
---|---|
android |
arm |
darwin |
386 |
darwin |
amd64 |
darwin |
arm |
darwin |
arm64 |
dragonfly |
amd64 |
freebsd |
386 |
freebsd |
amd64 |
freebsd |
arm |
linux |
386 |
linux |
amd64 |
linux |
arm |
linux |
arm64 |
linux |
ppc64 |
linux |
ppc64le |
linux |
mips |
linux |
mipsle |
linux |
mips64 |
linux |
mips64le |
netbsd |
386 |
netbsd |
amd64 |
netbsd |
arm |
openbsd |
386 |
openbsd |
amd64 |
openbsd |
arm |
plan9 |
386 |
plan9 |
amd64 |
solaris |
amd64 |
windows |
386 |
windows |
amd64 |
Warning: Cross-compiling executables for Android requires the Android NDK, and some additional setup which is beyond the scope of this tutorial.
Using the values in the table, we can build Caddy for Windows 64-bit like this:
- env GOOS=windows GOARCH=amd64 go build github.com/mholt/caddy/caddy
Once again, no output indicates that the operation was successful. The executable will be created in the current directory, using the package name as its name. However, since we built this executable for Windows, the name ends with the suffix .exe
.
You should have caddy.exe
file in your current directory, which you can verify with the ls
command.
- ls caddy.exe
You’ll see the caddy.exe
file listed in the output:
Outputcaddy.exe
Note: You can use the -o
flag to rename the executable or place it in a different location. However, when building an executable for Windows and providing a different name, be sure to explicitly specify the .exe
suffix when setting the executable’s name.
Let’s look at scripting this process to make it easier to release software for multiple target environments.
The process of creating executables for many platforms can be a little tedious, but we can create a script to make things easier.
The script will take the package import path as an argument, iterate through a predefined list of operating system and platform pairs, and generate an executable for each pair, placing the output in the current directory. Each executable will be named with the package name, followed by the target platform and architecture, in the form package-OS-architecture
. This will be a universal script that you can use on any project.
Switch to your home directory and create a new file called go-executable-build.bash
in your text editor:
- cd ~
- nano go-executable-build.bash
We will start our script with a shebang line. This line defines which interpreter will parse this script when it’s run as an executable. Add the following line to specify that bash
should execute this script:
#!/usr/bin/env bash
We want to take the package import path as a command line argument. To do this, we will use $n
variable, where n
is a non-negative number. The variable $0
contains the name of the script you executed, while $1
and greater will contain user-provided arguments. Add this line to the script, which will take the first argument from the command line and store it in a variable called package
:
...
package=$1
Next, make sure that the user provided this value. If the value isn’t provided, exit the script with a message explaining how to use the script:
...
if [[ -z "$package" ]]; then
echo "usage: $0 <package-name>"
exit 1
fi
This if
statement checks the value of the $package
variable. If it’s not set, we use echo
to print the correct usage, and then terminate the script using exit
. exit
takes a return value as an argument, which should be 0
for successful executions and any non-zero value for unsuccessful executions. We use 1
here since the script wasn’t successful.
Note: If you want to make this script work with a predefined package, change the package
variable to point to that import path:
...
package="github.com/user/hello"
Next, we want to extract the package name from the path. The package import path is delimited by /
characters, with the package name located at the end of the path. First, we will split the package import path into an array, using the /
as the delimiter:
package_split=(${package//\// })
The package name should be the last element of this new $package_split
array. In Bash, you can use a negative array index to access an array from the end instead of the beginning. Add this line to grab the package name from the array and store it in a variable called package_name
:
...
package_name=${package_split[-1]}
Now, you’ll need to decide what platforms and architectures you want build executables for. In this tutorial, we’ll build executables for Windows 64-bit, Windows 32-bit, and 64-bit macOS. We will put these targets in an array with the format OS/Platform
, so we can split each pair into GOOS
and GOARCH
variables using the same method we used to extract the package name from the path. Add the platforms to the script:
...
platforms=("windows/amd64" "windows/386" "darwin/amd64")
Next, we’ll iterate through the array of platforms, split each platform entry into values for the GOOS
and GOARCH
environment variables, and use those to build the executable. We can do that with the following for
loop:
...
for platform in "${platforms[@]}"
do
...
done
The platform
variable will contain an entry from the platforms
array in each iteration. We need to split platform
to two variables - GOOS
and GOARCH
. Add the following lines to the for
loop:
for platform in "${platforms[@]}"
do
platform_split=(${platform//\// })
GOOS=${platform_split[0]}
GOARCH=${platform_split[1]}
done
Next, we will generate the executable’s name by combining the package name with the OS and architecture. When we’re building for Windows, we also need to add the .exe
suffix to the filename. Add this code to the for
loop:
for platform in "${platforms[@]}"
do
platform_split=(${platform//\// })
GOOS=${platform_split[0]}
GOARCH=${platform_split[1]}
output_name=$package_name'-'$GOOS'-'$GOARCH
if [ $GOOS = "windows" ]; then
output_name+='.exe'
fi
done
With the variables set, we use go build
to create the executable. Add this line to the body of the for
loop, right above the done
keyword:
...
if [ $GOOS = "windows" ]; then
output_name+='.exe'
fi
env GOOS=$GOOS GOARCH=$GOARCH go build -o $output_name $package
done
Finally, we should check to see if there were errors building the executable. For example, we might run into an error if we try to build a package we don’t have sources for. We can check the go build
command’s return code for a non-zero value. The variable $?
contains the return code from a previous command’s execution. If go build
returns anything other than 0
, there was a problem, and we’ll want to exit the script. Add this code to the for
loop, after the go build
command and above the done
keyword.
...
env GOOS=$GOOS GOARCH=$GOARCH go build -o $output_name $package
if [ $? -ne 0 ]; then
echo 'An error has occurred! Aborting the script execution...'
exit 1
fi
With that, we now have a script that will build multiple executables from our Go package. Here’s the completed script:
#!/usr/bin/env bash
package=$1
if [[ -z "$package" ]]; then
echo "usage: $0 <package-name>"
exit 1
fi
package_split=(${package//\// })
package_name=${package_split[-1]}
platforms=("windows/amd64" "windows/386" "darwin/amd64")
for platform in "${platforms[@]}"
do
platform_split=(${platform//\// })
GOOS=${platform_split[0]}
GOARCH=${platform_split[1]}
output_name=$package_name'-'$GOOS'-'$GOARCH
if [ $GOOS = "windows" ]; then
output_name+='.exe'
fi
env GOOS=$GOOS GOARCH=$GOARCH go build -o $output_name $package
if [ $? -ne 0 ]; then
echo 'An error has occurred! Aborting the script execution...'
exit 1
fi
done
Verify that your file matches the preceding code. Then save the file and exit the editor.
Before we can use the script, we have to make it executable with the chmod
command:
- chmod +x go-executable-build.bash
Finally, test the script by building executables for Caddy:
- ./go-executable-build.bash github.com/mholt/caddy/caddy
If everything goes well, you should have executables in your current directory. No output indicates successful script execution. You can verify are executables created using ls
command:
- ls caddy*
You should see all three versions:
Example ls outputcaddy-darwin-amd64 caddy-windows-386.exe caddy-windows-amd64.exe
To change the target platforms, just change the platforms
variable in your script.
In this tutorial, you’ve learned how to use Go’s tooling to obtain packages from version control systems, as well as build and cross-compile executables for different platforms.
You also created a script that you can use to cross-compile a single package for many platforms.
To make sure your application works correctly, you can take a look at testing and continuous integration like Travis-CI and AppVeyor for testing on Windows.
If you are interested in Caddy and how to use it, take a look at How to Host a Website with Caddy on Ubuntu 16.04.
]]>Adding and removing users on a Linux system is one of the most important system administration tasks to familiarize yourself with. When you create a new system, you are often only given access to the root account by default.
While running as the root user gives you complete control over a system and its users, it is also dangerous and possibly destructive. For common system administration tasks, it’s a better idea to add an unprivileged user and carry out those tasks without root privileges. You can also create additional unprivileged accounts for any other users you may have on your system. Each user on a system should have their own separate account.
For tasks that require administrator privileges, there is a tool installed on Ubuntu systems called sudo
. Briefly, sudo
allows you to run a command as another user, including users with administrative privileges. In this guide, you’ll learn how to create user accounts, assign sudo
privileges, and delete users.
Deploy your frontend applications from GitHub using DigitalOcean App Platform. Let DigitalOcean focus on scaling your app.
To complete this tutorial, you will need access to a server running Ubuntu 20.04. Ensure that you have root access to the server and firewall enabled. To set this up, follow our Initial Server Setup Guide for Ubuntu 20.04.
If you are signed in as the root user, you can create a new user at any time by running the following:
- adduser newuser
If you are signed in as a non-root user who has been given sudo
privileges, you can add a new user with the following command:
- sudo adduser newuser
Either way, you will be required to respond to a series of questions:
ENTER
if you don’t wish to utilize these fields.Y
to continue.Your new user is now ready for use and can be logged into with the password that you entered.
If you need your new user to have administrative privileges, continue on to the next section.
If your new user should have the ability to execute commands with root (administrative) privileges, you will need to give the new user access to sudo
. Let’s examine two approaches to this task: first, adding the user to a pre-defined sudo user group, and second, specifying privileges on a per-user basis in sudo
’s configuration.
By default, sudo
on Ubuntu 20.04 systems is configured to extend full privileges to any user in the sudo group.
You can view what groups your new user is in with the groups
command:
- groups newuser
Outputnewuser : newuser
By default, a new user is only in their own group because adduser
creates this in addition to the user profile. A user and its own group share the same name. In order to add the user to a new group, you can use the usermod
command:
- usermod -aG sudo newuser
The -aG
option tells usermod
to add the user to the listed groups.
Please note that the usermod
command itself requires sudo
privileges. This means that you can only add users to the sudo
group if you’re signed in as the root user or as another user that has already been added as a member of the sudo
group. In the latter case, you will have to precede this command with sudo
, as in this example:
- sudo usermod -aG sudo newuser
/etc/sudoers
As an alternative to putting your user in the sudo group, you can use the visudo
command, which opens a configuration file called /etc/sudoers
in the system’s default editor, and explicitly specify privileges on a per-user basis.
Using visudo
is the only recommended way to make changes to /etc/sudoers
because it locks the file against multiple simultaneous edits and performs a validation check on its contents before overwriting the file. This helps to prevent a situation where you misconfigure sudo
and cannot fix the problem because you have lost sudo
privileges.
If you are currently signed in as root, run the following:
- visudo
If you are signed in as a non-root user with sudo
privileges, run the same command with the sudo
prefix:
- sudo visudo
Traditionally, visudo
opened /etc/sudoers
in the vi
editor, which can be confusing for inexperienced users. By default on new Ubuntu installations, visudo
will use the nano
text editor, which provides a more convenient and accessible text editing experience. Use the arrow keys to move the cursor, and search for the line that reads like the following:
root ALL=(ALL:ALL) ALL
Below this line, add the following highlighted line. Be sure to change newuser
to the name of the user profile that you would like to grant sudo
privileges:
root ALL=(ALL:ALL) ALL
newuser ALL=(ALL:ALL) ALL
Add a new line like this for each user that should be given full sudo
privileges. When you’re finished, save and close the file by pressing CTRL + X
, followed by Y
, and then ENTER
to confirm.
Now your new user is able to execute commands with administrative privileges.
When signed in as the new user, you can execute commands as your regular user by typing commands as normal:
- some_command
You can execute the same command with administrative privileges by typing sudo
ahead of the command:
- sudo some_command
When doing this, you will be prompted to enter the password of the regular user account you are signed in as.
In the event that you no longer need a user, it’s best to delete the old account.
You can delete the user itself, without deleting any of their files, by running the following command as root:
- deluser newuser
If you are signed in as another non-root user with sudo
privileges, you would use the following:
- sudo deluser newuser
If, instead, you want to delete the user’s home directory when the user is deleted, you can issue the following command as root:
- deluser --remove-home newuser
If you’re running this as a non-root user with sudo
privileges, you would run the same command with the sudo
prefix:
- sudo deluser --remove-home newuser
If you previously configured sudo
privileges for the user you deleted, you may want to remove the relevant line again:
- visudo
Or use the following command if you are a non-root user with sudo
privileges:
- sudo visudo
root ALL=(ALL:ALL) ALL
newuser ALL=(ALL:ALL) ALL # DELETE THIS LINE
This will prevent a new user created with the same name from being accidentally given sudo
privileges.
You should now have a fairly good handle on how to add and remove users from your Ubuntu 20.04 system. Effective user management will allow you to separate users and give them only the access that they are required to do their job.
For more information about how to configure sudo
, check out our guide on how to edit the sudoers file.
It works but I’d love to make the restart happen automatically whenever I save the file(s). I know this can be done for HTML templates with an “auto-reload” setting in the .py file, but restarting the whole service seems like it needs to happen at a higher level, like a bash script or something. Any suggestions? Thanks.
]]>When I did the following command: sudo apt install python3-certbot-nginx
As you can see, the prompt was [Y/n], which made me think I need to type an uppercase Y to say yes. But when I did, the operation aborted. Very confusing. When I tried it again and used a lowercase y, it proceeded.
I will share this as a ticket, but I also want to share it in the community in case others have the same problem.
]]>However, instead of copying from my local windows machine, I want to set up the same process on the Digital Ocean droplet to connect to my windows machine, and copy files from a certain directory. That is, using some secure protocol I want to know the code that will allow me to do the following:
I was hoping this was an easy task
Thanks in advance
]]>root@ip-172-31-21-204:~# nginx -t
nginx: [emerg] unknown directive "<!DOCTYPE" in /etc/nginx/sites-enabled/site1:6
nginx: configuration file /etc/nginx/nginx.conf test failed
journalctl -xe
Feb 27 20:43:13 ip-172-31-21-204 dhclient[556]: XMT: Solicit on ens5, interval 109320ms.
Feb 27 20:45:03 ip-172-31-21-204 dhclient[556]: XMT: Solicit on ens5, interval 121150ms.
Feb 27 20:46:19 ip-172-31-21-204 sshd[2345]: Invalid user user from 92.255.85.237 port 61852
Feb 27 20:46:19 ip-172-31-21-204 sshd[2345]: Received disconnect from 92.255.85.237 port 61852:11: Bye Bye [preauth]
Feb 27 20:46:19 ip-172-31-21-204 sshd[2345]: Disconnected from invalid user user 92.255.85.237 port 61852 [preauth]
Feb 27 20:47:04 ip-172-31-21-204 dhclient[556]: XMT: Solicit on ens5, interval 119330ms.
Feb 27 20:49:03 ip-172-31-21-204 dhclient[556]: XMT: Solicit on ens5, interval 124670ms.
Feb 27 20:51:08 ip-172-31-21-204 dhclient[556]: XMT: Solicit on ens5, interval 126880ms.
Feb 27 20:51:55 ip-172-31-21-204 dhclient[480]: DHCPREQUEST for 172.31.21.204 on ens5 to 172.31.16.1 port 67
Feb 27 20:51:55 ip-172-31-21-204 dhclient[480]: DHCPACK of 172.31.21.204 from 172.31.16.1
Feb 27 20:51:55 ip-172-31-21-204 dhclient[480]: bound to 172.31.21.204 -- renewal in 1635 seconds.
Feb 27 20:53:15 ip-172-31-21-204 dhclient[556]: XMT: Solicit on ens5, interval 110250ms.
Feb 27 20:55:05 ip-172-31-21-204 dhclient[556]: XMT: Solicit on ens5, interval 116830ms.
Feb 27 20:57:02 ip-172-31-21-204 dhclient[556]: XMT: Solicit on ens5, interval 111480ms.
Feb 27 20:57:19 ip-172-31-21-204 kernel: [UFW BLOCK] IN=ens5 OUT= MAC=02:36:35:9d:52:f8:02:31:52:19:17:48:08:00 SRC=162.142.125.176 DST=172.31.21.204 LEN=44
root@ip-172-31-21-204:~# systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2022-02-27 20:32:05 UTC; 18min ago
Docs: man:nginx(8)
Process: 2263 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)
Feb 27 20:32:05 ip-172-31-21-204 systemd[1]: Starting A high performance web server and a reverse proxy server...
Feb 27 20:32:05 ip-172-31-21-204 nginx[2263]: nginx: [emerg] unknown directive "<!DOCTYPE" in /etc/nginx/sites-enabled/site1:6
Feb 27 20:32:05 ip-172-31-21-204 nginx[2263]: nginx: configuration file /etc/nginx/nginx.conf test failed
Feb 27 20:32:05 ip-172-31-21-204 systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Feb 27 20:32:05 ip-172-31-21-204 systemd[1]: nginx.service: Failed with result 'exit-code'.
Feb 27 20:32:05 ip-172-31-21-204 systemd[1]: Failed to start A high performance web server and a reverse proxy server.
root@ip-172-31-21-204:~# cat /etc/nginx/sites-enabled/site1
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
]]>Is it possible to change the root password with cloud-init?
]]>Adding and removing users on a Linux system is one of the most important system administration tasks to familiarize yourself with. When you create a new system, you are often only given access to the root account by default.
While running as the root user gives you complete control over a system and its users, it is also dangerous and possibly destructive. For common system administration tasks, it’s a better idea to add an unprivileged user and carry out those tasks without root privileges. You can also create additional unprivileged accounts for any other users you may have on your system. Each user on a system should have their own separate account.
For tasks that require administrator privileges, there is a tool installed on Ubuntu systems called sudo
. Briefly, sudo
allows you to run a command as another user, including users with administrative privileges. In this guide, you’ll learn how to create user accounts, assign sudo
privileges, and delete users.
To complete this tutorial, you will need access to a server running Ubuntu 18.04. Ensure that you have root access to the server and firewall enabled. To set this up, follow our Initial Server Setup Guide for Ubuntu 18.04.
If you are signed in as the root user, you can create a new user at any time by running the following:
- adduser newuser
If you are signed in as a non-root user who has been given sudo
privileges, you can add a new user with the following command:
- sudo adduser newuser
Either way, you will be required to respond to a series of questions:
ENTER
if you don’t wish to utilize these fields.Y
to continue.Your new user is now ready for use and can be logged into with the password that you entered.
If you need your new user to have administrative privileges, continue on to the next section.
If your new user should have the ability to execute commands with root (administrative) privileges, you will need to give the new user access to sudo
. Let’s examine two approaches to this task: first, adding the user to a pre-defined sudo user group, and second, specifying privileges on a per-user basis in sudo
’s configuration.
By default, sudo
on Ubuntu 18.04 systems is configured to extend full privileges to any user in the sudo group.
You can view what groups your new user is in with the groups
command:
- groups newuser
Outputnewuser : newuser
By default, a new user is only in their own group because adduser
creates this in addition to the user profile. A user and its own group share the same name. In order to add the user to a new group, you can use the usermod
command:
- usermod -aG sudo newuser
The -aG
option tells usermod
to add the user to the listed groups.
Please note that the usermod
command itself requires sudo
privileges. This means that you can only add users to the sudo
group if you’re signed in as the root user or as another user that has already been added as a member of the sudo
group. In the latter case, you will have to precede this command with sudo
, as in this example:
- sudo usermod -aG sudo newuser
/etc/sudoers
As an alternative to putting your user in the sudo group, you can use the visudo
command, which opens a configuration file called /etc/sudoers
in the system’s default editor, and explicitly specify privileges on a per-user basis.
Using visudo
is the only recommended way to make changes to /etc/sudoers
because it locks the file against multiple simultaneous edits and performs a validation check on its contents before overwriting the file. This helps to prevent a situation where you misconfigure sudo
and cannot fix the problem because you have lost sudo
privileges.
If you are currently signed in as root, run the following:
- visudo
If you are signed in as a non-root user with sudo
privileges, run the same command with the sudo
prefix:
- sudo visudo
Traditionally, visudo
opened /etc/sudoers
in the vi
editor, which can be confusing for inexperienced users. By default on new Ubuntu installations, visudo
will use the nano
text editor, which provides a more convenient and accessible text editing experience. Use the arrow keys to move the cursor, and search for the line that reads like the following:
root ALL=(ALL:ALL) ALL
Below this line, add the following highlighted line. Be sure to change newuser
to the name of the user profile that you would like to grant sudo
privileges:
root ALL=(ALL:ALL) ALL
newuser ALL=(ALL:ALL) ALL
Add a new line like this for each user that should be given full sudo
privileges. When you’re finished, save and close the file by pressing CTRL + X
, followed by Y
, and then ENTER
to confirm.
Now your new user is able to execute commands with administrative privileges.
When signed in as the new user, you can execute commands as your regular user by typing commands as normal:
- some_command
You can execute the same command with administrative privileges by typing sudo
ahead of the command:
- sudo some_command
When doing this, you will be prompted to enter the password of the regular user account you are signed in as.
In the event that you no longer need a user, it’s best to delete the old account.
You can delete the user itself, without deleting any of their files, by running the following command as root:
- deluser newuser
If you are signed in as another non-root user with sudo
privileges, you would use the following:
- sudo deluser newuser
If, instead, you want to delete the user’s home directory when the user is deleted, you can issue the following command as root:
- deluser --remove-home newuser
If you’re running this as a non-root user with sudo
privileges, you would run the same command with the sudo
prefix:
- sudo deluser --remove-home newuser
If you previously configured sudo
privileges for the user you deleted, you may want to remove the relevant line again:
- visudo
Or use the following command if you are a non-root user with sudo
privileges:
- sudo visudo
root ALL=(ALL:ALL) ALL
newuser ALL=(ALL:ALL) ALL # DELETE THIS LINE
This will prevent a new user created with the same name from being accidentally given sudo
privileges.
You should now have a fairly good handle on how to add and remove users from your Ubuntu 18.04 system. Effective user management will allow you to separate users and give them only the access that they are required to do their job.
For more information about how to configure sudo
, check out our guide on how to edit the sudoers file.
john ALL=ALL,!/usr/bin/apt-get #john won’t be able to run apt-get
My question comes when john can use cp, mv or ln to copy and change the name of the binary, for example:
john home$ sudo ln -s /usr/bin/apt-get myaptget
This renders my sudoers file configuration useless.
Any suggestions, plugin/app to restrict and differentiate sudoers/permissions?
Imagine I have 2 administrators on my system, but I’d like that one of them to have more control than the other…
Thank you very much for your answers, best regards!
]]>old.html new.html
oldpage newpage
error.php 404
So I should get something like this:
rewrite ^/old.html$ https://domain.com/new.html permanent;
rewrite ^/oldpage$ https://domain.com/newpage permanent;
rewrite ^/error.php$ https://domain.com/404 permanent;
For now, my script looks like this:
if [ "$#" -eq 0 ]; then
echo "Usage: ./rewrite.sh <domain> <file>"
exit
else
DOMAIN=${1}
FILE=${2}
fi
sed -e 's/$/ permanent;/' -i $FILE #end of each line
It does nothing but add “permanent ;” to the end of each line. How do I add “rewrite ^/” to the beginning of each line? I tried this way, but it doesn’t work :
REW=“rewrite ^/” sed -i -e “s/^/‘${REW}’/” $FILE
I also have no idea how to add “$ https://domain.com/” in the beginning of the new address.
]]>For example i try command ip-copy-id -i id_rsa.pub user@ip-address it will want me my password and than permission denied (public key, password)
anyone have any suggestions the others solution i tried was nano command, keygen etc…
]]>One problem users run into when first learning how to work with Linux is how to find the files they are looking for.
This guide will cover how to use the aptly named find
command. This will help you search for files on your system using a variety of filters and parameters. It will also briefly cover the locate
command, which can be used to search for files in a different way.
To follow along with this guide, you will need access to a computer running a Linux-based operating system. This can either be a virtual private server which you’ve connected to with SSH or your local machine. Note that this tutorial was validated using a Linux server running Ubuntu 20.04, but the examples given should work on a computer running any version of any Linux distribution.
If you plan to use a remote server to follow this guide, we encourage you to first complete our Initial Server Setup guide. Doing so will set you up with a secure server environment — including a non-root user with sudo
privileges and a firewall configured with UFW — which you can use to build your Linux skills.
Note: To illustrate how the find
and locate
commands work, the example commands in this guide search for files stored under /
, or the root directory. Because of this, if you’re logged into the terminal as a non-root user, some of the example commands may include Permission denied
in their output.
This is to be expected, since you’re searching for files within directories that regular users typically don’t have access to. However, these example commands should still work and be useful for understanding how these programs work.
The most obvious way of searching for files is by their name.
To find a file by name with the find
command, you would use the following syntax:
- find -name "query"
This will be case sensitive, meaning a search for query
is different from a search for Query
.
To find a file by name but ignore the case of the query, use the -iname
option:
- find -iname "query"
If you want to find all files that don’t adhere to a specific pattern, you can invert the search with -not
:
- find -not -name "query_to_avoid"
Alternatively, you can invert the search using an exclamation point (!
), like this:
- find \! -name "query_to_avoid"
Note that if you use !
, you must escape the character with a backslash (\
) so that the shell does not try to interpret it before find
can act.
You can specify the type of files you want to find with the -type
parameter. It works like this:
- find -type type_descriptor query
Here are some of the descriptors you can use to specify the type of file:
f
: regular filed
: directoryl
: symbolic linkc
: character devicesb
: block devicesFor instance, if you wanted to find all of the character devices on your system, you could issue this command:
- find /dev -type c
This command specifically only searches for devices within the /dev
directory, the directory where device files are typically mounted in Linux systems:
Output/dev/vcsa5
/dev/vcsu5
/dev/vcs5
/dev/vcsa4
/dev/vcsu4
/dev/vcs4
/dev/vcsa3
/dev/vcsu3
/dev/vcs3
/dev/vcsa2
/dev/vcsu2
/dev/vcs2
. . .
You can search for all files that end in .conf
with a command like the following. This example searches for matching files within the /usr
directory:
- find /usr -type f -name "*.conf"
Output/usr/src/linux-headers-5.4.0-88-generic/include/config/auto.conf
/usr/src/linux-headers-5.4.0-88-generic/include/config/tristate.conf
/usr/src/linux-headers-5.4.0-90-generic/include/config/auto.conf
/usr/src/linux-headers-5.4.0-90-generic/include/config/tristate.conf
/usr/share/adduser/adduser.conf
/usr/share/ufw/ufw.conf
/usr/share/popularity-contest/default.conf
/usr/share/byobu/keybindings/tmux-screen-keys.conf
/usr/share/libc-bin/nsswitch.conf
/usr/share/rsyslog/50-default.conf
. . .
Note: The previous example combines two find
query expressions; namely, -type f
and -name "*.conf"
. For any file to be returned, it must satisfy both of these expressions.
You can combine expressions like this by separating them with the -and
option, but as this example shows the -and
is implied any time you include two expressions. You can also return results that satisfy either expression by separating them with the -or
option:
- find -name query_1 -or -name query_2
This example will find any files whose names match either query_1
or query_2
.
find
gives you a variety of ways to filter results by size and time.
You can filter files by their size using the -size
parameter. To do this, you must add a special suffix to the end of a numerical size value to indicate whether you’re counting the size in terms of bytes, megabytes, gigabytes, or another size. Here are some commonly used size suffixes:
c
: bytesk
: kilobytesM
: megabytesG
: gigabytesb
: 512-byte blocksTo illustrate, the following command will find every file in the /usr
directory that is exactly 50 bytes:
- find /usr -size 50c
To find files that are less than 50 bytes, you can use this syntax instead:
- find /usr -size -50c
To find files in the /usr
directory that are more than 700 Megabytes, you could use this command:
- find /usr -size +700M
For every file on the system, Linux stores time data about access times, modification times, and change times.
Access Time: The last time a file was read or written to.
Modification Time: The last time the contents of the file were modified.
Change Time: The last time the file’s inode metadata was changed.
You can base your find
searches on these parameters using the -atime
, -mtime
, and -ctime
options, respectively. For any of these options, you must pass a value indicating how many days in the past you’d like to search. Similar to the size options outlined previously, you can prepend these options with the plus or minus symbols to specify “greater than” or “less than”.
For example, to find files in the /usr
directory that were modified within the last day, run the following command:
- find /usr -mtime 1
If you want files that were accessed less than a day ago, you could run this command:
- find /usr -atime -1
To find files that last had their meta information changed more than 3 days ago, you might execute the following:
- find /usr -ctime +3
These options also have companion parameters you can use to specify minutes instead of days:
- find /usr -mmin -1
This will give the files that have been modified in the last minute.
find
can also do comparisons against a reference file and return those that are newer:
- find / -newer reference_file
This syntax will return every file on the system that was created or changed more recently than the reference file.
You can also search for files by the user or group that owns the file using the -user
and -group
parameters, respectively. To find every file in the /var
directory that is owned by the syslog user run this command:
- find /var -user syslog
Similarly, you can specify files in the /etc
directory owned by the shadow group by typing:
- find /etc -group shadow
You can also search for files with specific permissions.
If you want to match an exact set of permissions, you use can this syntax specifying the permissions using octal notation:
- find / -perm 644
This will match files with exactly the permissions specified.
If you want to specify anything with at least those permissions, you can precede the permissions notation with a minus sign:
- find / -perm -644
This will match any files that have additional permissions. A file with permissions of 744
would be matched in this instance.
In this section, you will create an example directory structure that you’ll then use to explore filtering files by their depth within the structure.
If you’re following along with the examples in this tutorial, it would be prudent to create these files and directories within the /tmp/
directory. /tmp/
is a temporary directory, meaning that any files and directories within it will be deleted the next time the server boots up. This will be useful for the purposes of this guide, since you can create as many directories, files, and links as you’d like without having to worry about them clogging up your system later on.
After running the commands in this section, your /tmp/
directory will contain three levels of directories, with ten directories at the first level. Each directory (including the temporary directory) will contain ten files and ten subdirectories.
Create the example directory structure within the /tmp/
directory with the following command:
- mkdir -p /tmp/test/level1dir{1..10}/level2dir{1..10}/level3dir{1..10}
Following that, populate these directories with some sample files using the touch
command:
- touch /tmp/test/{file{1..10},level1dir{1..10}/{file{1..10},level2dir{1..10}/{file{1..10},level3dir{1..10}/file{1..10}}}}
With these files and directories in place, go ahead and navigate into the test/
directory you just created:
- cd /tmp/test
To get a baseline understanding of how find
will retrieve files from this structure, begin with a regular name search that matches any files named file1
:
- find -name file1
Output./level1dir7/level2dir8/level3dir9/file1
./level1dir7/level2dir8/level3dir3/file1
./level1dir7/level2dir8/level3dir4/file1
./level1dir7/level2dir8/level3dir1/file1
./level1dir7/level2dir8/level3dir8/file1
./level1dir7/level2dir8/level3dir7/file1
./level1dir7/level2dir8/level3dir2/file1
./level1dir7/level2dir8/level3dir6/file1
./level1dir7/level2dir8/level3dir5/file1
./level1dir7/level2dir8/file1
. . .
This will return a lot of results. If you pipe the output into a counter, you’ll find that there are 1111
total results:
- find -name file1 | wc -l
Output1111
This is probably too many results to be useful to you in most circumstances. To narrow it down, you can specify the maximum depth of the search under the top-level search directory:
- find -maxdepth num -name query
To find file1
only in the level1
directories and above, you can specify a max depth of 2 (1 for the top-level directory, and 1 for the level1
directories):
- find -maxdepth 2 -name file1
Output./level1dir7/file1
./level1dir1/file1
./level1dir3/file1
./level1dir8/file1
./level1dir6/file1
./file1
./level1dir2/file1
./level1dir9/file1
./level1dir4/file1
./level1dir5/file1
./level1dir10/file1
That is a much more manageable list.
You can also specify a minimum directory if you know that all of the files exist past a certain point under the current directory:
- find -mindepth num -name query
You can use this to find only the files at the end of the directory branches:
- find -mindepth 4 -name file1
Output./level1dir7/level2dir8/level3dir9/file1
./level1dir7/level2dir8/level3dir3/file1
./level1dir7/level2dir8/level3dir4/file1
./level1dir7/level2dir8/level3dir1/file1
./level1dir7/level2dir8/level3dir8/file1
./level1dir7/level2dir8/level3dir7/file1
./level1dir7/level2dir8/level3dir2/file1
. . .
Again, because of the branching directory structure, this will return a large number of results (1000).
You can combine the min and max depth parameters to focus in on a narrow range:
- find -mindepth 2 -maxdepth 3 -name file1
Output./level1dir7/level2dir8/file1
./level1dir7/level2dir5/file1
./level1dir7/level2dir7/file1
./level1dir7/level2dir2/file1
./level1dir7/level2dir10/file1
./level1dir7/level2dir6/file1
./level1dir7/level2dir3/file1
./level1dir7/level2dir4/file1
./level1dir7/file1
. . .
Combining these options like this narrows down the results significantly, with only 110 lines returned instead of the previous 1000.
find
ResultsYou can execute an arbitrary helper command on everything that find
matches by using the -exec
parameter using the following syntax:
- find find_parameters -exec command_and_options {} \;
The {}
is used as a placeholder for the files that find
matches. The \;
lets find
know where the command ends.
For instance, assuming you’re still in the test/
directory you created within the /tmp/
directory in the previous step, you could find the files in the previous section that had 644
permissions and modify them to have 664
permissions:
- find . -type f -perm 644 -exec chmod 664 {} \;
You could also change the directory permissions in a similar way:
- find . -type d -perm 755 -exec chmod 700 {} \;
This example finds every directory with permissions set to 755
and then modifies the permissions to 700
.
locate
An alternative to using find
is the locate
command. This command is often quicker and can search the entire file system with ease.
You can install the command on Debian or Ubuntu with apt
by updating your package lists and then installing the mlocate
package:
- sudo apt update
- sudo apt install mlocate
On Rocky Linux, CentOS, and other RedHat derived distributions, you can instead use the dnf
command to install mlocate
:
- sudo dnf install mlocate
The reason locate
is faster than find
is because it relies on a database that lists all the files on the filesystem. This database is usually updated once a day with a cron script, but you can update it manually with the updatedb
command. Run this command now with sudo
privileges:
- sudo updatedb
Remember, the locate
database must always be up-to-date if you want to find new files. If you add new files before the cron script is executed or before you run the updatedb
command, they will not appear in your query results.
locate
allows you to filter results in a number of ways. The most fundamental way you can use it to find files is to use this syntax:
- locate query
This will match any files and directories that contain the string query
anywhere in their file path. To return only files whose names contain the query itself, instead of every file that has the query in the directories leading up to it, you can include the -b
flag to search only for files whose “basename” matches the query:
- locate -b query
To have locate
only return results that still exist in the filesystem (meaning files that were not removed between the last updatedb
call and the current locate
call), use the -e
flag:
- locate -e query
You can retrieve statistics about the information that locate
has cataloged using the -S
option:
- locate -S
OutputDatabase /var/lib/mlocate/mlocate.db:
21015 directories
136787 files
7727763 bytes in file names
3264413 bytes used to store database
This can be useful for getting a high-level understanding of how many files and directories exist on your system.
Both the find
and locate
commands are useful tools for finding files on your system. Both are powerful commands that can be strengthened by combining them with other utilities through pipelines, but it’s up to you to decide which tool is appropriate for your given situation.
From here, we encourage you to continue experimenting with find
and locate
. You can read their respective man
pages to learn about other options not covered in this guide, and you can analyze and manipulate search results by piping them into other commands like wc
, sort
, and grep
.
While working in a server environment, you’ll spend a lot of your time on the command line. Most likely, you’ll be using the bash shell, which is the default of most distributions.
During a terminal session, you’ll likely be repeating some commands often, and typing variations on those commands even more frequently. While typing each command repeatedly can be good practice in the beginning, at some point, it crosses the line into being disruptive and an annoyance.
Luckily, the bash shell has some fairly well-developed history functions. Learning how to effectively use and manipulate your bash history will allow you to spend less time typing and more time getting actual work done. Many developers are familiar with the DRY philosophy of Don’t Repeat Yourself. Effective use of bash’s history allows you to operate closer to this principle and will speed up your workflow.
To follow along with this guide, you will need access to a computer running a Linux-based operating system. This can either be a virtual private server which you’ve connected to with SSH or your local machine. Note that this tutorial was validated using a Linux server running Ubuntu 20.04, but the examples given should work on a computer running any version of any Linux distribution.
If you plan to use a remote server to follow this guide, we encourage you to first complete our Initial Server Setup guide. Doing so will set you up with a secure server environment — including a non-root user with sudo
privileges and a firewall configured with UFW — which you can use to build your Linux skills.
Before you begin actually using your command history, it can be helpful for you to adjust some bash settings to make it more useful. These steps are not necessary, but they can make it easier to find and execute commands that you’ve run previously.
Bash allows you to adjust the number of commands that it stores in history. It actually has two separate options for this: the HISTFILESIZE
parameter configures how many commands are kept in the history file, while the HISTSIZE
controls the number stored in memory for the current session.
This means you can set a reasonable cap for the size of history in memory for the current session, and have an even larger history saved to disk that you can examine at a later time. By default, bash sets very conservative values for these options, but you can expand these to take advantage of a larger history. Some distributions already increase the default bash history settings with slightly more generous values.
Open your ~/.bashrc
file with your preferred text editor to change these settings. Here, we’ll use nano
:
- nano ~/.bashrc
Search for both the HISTSIZE
and HISTFILESIZE
parameters. If they are set, feel free to modify the values. If these parameters aren’t in your file, add them now. For the purposes of this guide, saving 10000 lines to disk and loading the last 5000 lines into memory will work fine. This is a conservative estimate for most systems, but you can adjust these numbers down if you notice a performance impact:
. . .
# for setting history length see HISTSIZE and HISTFILESIZE in bash(1)
HISTSIZE=5000
HISTFILESIZE=10000
. . .
By default, bash writes its history at the end of each session, overwriting the existing file with an updated version. This means that if you are logged in with multiple bash sessions, only the last one to exit will have its history saved.
You can work around this by setting the histappend
setting, which will append instead of overwriting the history. This may be set already, but if it is not, you can enable this by adding this line:
. . .
shopt -s histappend
. . .
If you want to have bash immediately add commands to your history instead of waiting for the end of each session (to enable commands in one terminal to be instantly be available in another), you can also set or append the history -a
command to the PROMPT_COMMAND
parameter, which contains commands that are executed before each new command prompt.
To configure this correctly, you’ll need to customize bash’s PROMPT_COMMAND
to change the way commands are recorded in the history file and in the current shell session’s memory:
history -a
.history -c
.history -r
command.Putting all these commands together in order in the PROMPT_COMMAND
shell variable will result in the the following, which you can paste into your .bashrc
file:
. . .
export PROMPT_COMMAND="history -a; history -c; history -r; $PROMPT_COMMAND"
. . .
When you are finished, save the file and exit. If you edited your .bashrc
file with nano
, do so by pressing CTRL + X
, Y
, and then ENTER
.
To implement your changes, either log out and back in again, or source
the file by running:
- source ~/.bashrc
With that, you’ve adjusted how your shell handles your command history. You can now get some practice finding your previous commands with the history
command.
The way to review your bash history is to use the history
command. This will print out our recent commands, one command per line. This should output, at most, the number of lines you selected for the HISTSIZE
variable. It will probably be fewer at this point:
- history
Output . . .
43 man bash
44 man fc
45 man bash
46 fc -l -10
47 history
48 ls -a
49 vim .bash_history
50 history
51 man history
52 history 10
53 history
Each command history
returns is associated with a number for easy reference. This guide will go over how this can be useful later on.
You can truncate the output by specifying a number after the command. For instance, if you want to only return the last 5 commands that were executed, you can type:
- history 5
Output 50 history
51 man history
52 history 10
53 history
54 history 5
To find all of the history
commands that contain a specific string, you can pipe the results into a grep
command that searches each line for a given string. For example, you can search for the lines that have cd
by typing:
- history | grep cd
Output 33 cd Pictures/
37 cd ..
39 cd Desktop/
61 cd /usr/bin/
68 cd
83 cd /etc/
86 cd resolvconf/
90 cd resolv.conf.d/
There are many situations where being able to retrieve a list of commands you’ve previously ran can be helpful. If you want to run one of those commands again, your instinct may be to copy one of the commands from your output and paste it into your prompt. This works, but bash comes with a number of shortcuts that allow you to retrieve and then automatically execute commands from your history.
Printing your command history can be useful, but it doesn’t really help you access those commands other than as reference.
You can recall and immediately execute any of the commands returned by a history
operation by its number preceded by an exclamation point (!
). Assuming your history
results aligned with those from the previous section, you could check out the man page for the history
command quickly by typing:
- !51
This will immediately recall and execute the command associated with the history number 51.
You can also execute commands relative to your current position by using the !-n
syntax, where “n” is replaced by the number of previous commands you want to recall.
As an example, say you ran the following two commands:
- ls /usr/share/common-licenses
- echo hello
If you wanted to recall and execute the command you ran before your most recent one (the ls
command), you could type !-2
:
- !-2
To re-execute the last command you ran, you could run !-1
. However, bash provides a shortcut consisting of two exclamation points which will substitute the most recent command and execute it:
- !!
Many people use this when they type a command but forgot that they needed sudo
privileges for it to execute. Typing sudo !!
will re-execute the command with sudo
in front of it:
- touch /etc/hello
Outputtouch: cannot touch `/etc/hello': Permission denied
- sudo !!
Outputsudo touch /etc/hello
[sudo] password for sammy:
This demonstrates another property of this syntax: these shortcuts are pure substitutions, and can be incorporated within other commands at will.
There are a few ways that you can scroll through your bash history, putting each successive command on the command line to edit.
The most common way of doing this is to press the up arrow key at the command prompt. Each additional press of the up arrow key will take you further back in your command line history.
If you need to go the other direction, the down arrow key traverses the history in the opposite direction, finally bringing you back to your current empty prompt.
If moving your hand all the way over to the arrow keys seems like a big hassle, you can move backwards in your command history using the CTRL + P
combination and use the CTRL + N
combination to move forward through your history again.
If you want to jump back to the current command prompt, you can do so by pressing META + >
. In most cases, the “meta” key is the ALT
key, so META + >
will mean pressing ALT + SHIFT + .
. This is useful if you find yourself far back in your history and want to get back to your empty prompt.
You can also go to the first line of your command history by doing the opposite maneuver and typing META + <
. This typically means pressing ALT + SHIFT + ,
.
To summarize, these are some keys and key combinations you can use to scroll through the history and jump to either end:
UP arrow key
: Scroll backwards in historyCTRL + P
: Scroll backwards in historyDOWN arrow key
: Scroll forwards in historyCTRL + N
: Scroll forwards in historyALT + SHIFT + .
: Jump to the end of the history (most recent)ALT+ SHIFT + ,
: Jump to the beginning of the history (most distant)Although piping the history
command through grep
can be a useful way to narrow down the results, it isn’t ideal in many situations.
Bash includes search functionality for its history. The typical way of using this is through searching backwards in history (most recent results returned first) using the CTRL + R
key combination.
For instance, you can type CTRL + R
, and begin typing part of the previous command. You only have to type out part of the command. If it matches an unwanted command instead, you can press CTRL + R
again for the next result.
If you accidentally pass the command you wanted, you can move in the opposite direction by typing CTRL + S
. This also can be useful if you’ve moved to a different point in your history using the keys in the last section and wish to search forward.
Be aware that, in many terminals, the CTRL + S
combination is mapped to suspend the terminal session. This will intercept any attempts to pass CTRL + S
to bash, and will “freeze” your terminal. To unfreeze, type CTRL + Q
to unsuspend the session.
This suspend and resume feature is not needed in most modern terminals, and you can turn it off without any problem by running the following command:
- stty -ixon
stty
is a utility that allows you to change your terminal’s settings from the command line. You could add this stty -ixon
command to the end of your ~/.bashrc
file to make this change permanent as well.
If you try searching with CTRL + S
again now, it should work as expected to allow you to search forwards.
A common scenario to find yourself in is to type in part of a command, only to then realize that you have executed it previously and can search the history for it.
The correct way of searching using what is already on your command line is to move your cursor to the beginning of the line with CTRL + A
, call the reverse history with CTRL + R
, paste the current line into the search with CTRL + Y
, and then using the CTRL + R
again to search in reverse.
For instance, suppose you want to update your package cache on an Ubuntu system. You’ve already typed this out recently, but you didn’t think about that until after you’ve typed the sudo
in the prompt again:
- sudo
At this point, you realize that this is an operation you’ve definitely done in the past day or so. You can press CTRL + A
to move your cursor to the beginning of the line. Then, press CTRL + R
to call your reverse incremental history search. This has a side effect of copying all of the content on the command line that was after our cursor position and putting it into your clipboard.
Next, press CTRL + Y
to paste the command segments that you just copied from the command line into the search. Lastly, press CTRL + R
to move backwards in your history, searching for commands containing the content you’ve just pasted.
Using shortcuts like this may seem tedious at first, but it can be quite useful when you get used to it. It is extremely helpful when you find yourself in a position where you’ve typed out half of a complex command and know you’re going to need the history to finish the rest.
Rather than thinking of these as separate key combinations, it may help you to think of them as a single compound operation. You can just hold the CTRL
key down and then press A
, R
, Y
, and then the R
key down in succession.
This guide has already touched on some of the most fundamental history expansion techniques that bash provides. Some of the ones we’ve covered so far are:
!!
: Expand to the last command!n
: Expand the command with history number “n”.!-n
: Expand to the command that was “n” number of commands before the current command in history.The above three examples are instances of event designators. These generally are ways of recalling previous history commands using certain criteria. They are the selection portion of your available operations.
For example, you can execute the last ssh
command that you ran by typing something like:
- !ssh
This searches for lines in your command history that begin with ssh
. If you want to search for a string that isn’t at the beginning of the command, you can surround it with ?
characters. For instance, to repeat a previous apt-cache search
command, you could likely run the following command to find and execute it:
- !?search?
Another event designator you can try involves substituting a string within your last command for another. To do this, enter a caret symbol (^
) followed by the string you want to replace, then immediately follow that with another caret, the replacement string, and a final caret at the end. Don’t include any spaces unless they’re part of the string you want to replace or part of the string you want to use as the replacement:
- ^original^replacement^
This will recall the previous command (just like !!
), search for an instance of original
within the command string, and replace it with replacement
. It will then execute the command using the replacement string.
This is useful for dealing with things like misspellings. For instance, say you mistakenly run this command when trying to read the contents of the /etc/hosts
file:
- cat /etc/hosst
Outputcat: /etc/hosst: No such file or directory
Rather then rewriting the entire command, you could run the following instead:
- ^hosst^hosts^
This will fix the error in the previous command and execute it successfully.
After event designators, you can add a colon (:
) followed by a word designator to select a portion of the matched command.
It does this by dividing the command into “words”, which are defined as any chunk separated by whitespace. This allows you some interesting opportunities to interact with your command parameters.
The word numbering starts at the initial command as “0”, the first argument as “1”, and continues on from there.
For instance, you could list the contents of a directory and then decide you want to navigate into that same directory. You could do so by running the following operations back to back:
- ls /usr/share/common-licenses
- cd !!:1
In cases like this where you are operating on the last command, you can shorten this by removing the second !
as well as the colon:
- cd !1
This will operate in the same way.
You can refer to the first argument with a caret (^
) and the final argument with a dollar sign ($
) if that makes sense for your purposes. These are more helpful when using ranges instead of specific numbers. For instance, you have three ways you can get all of the arguments from a previous command into a new command:
- !!:1*
- !!:1-$
- !!:*
The lone *
expands to all portions of the command being recalled other than the initial command. Similarly, you can use a number followed by *
to mean that everything after the specified word should be included.
Another thing you can do to augment the behavior of the history line you’re recalling is to modify the behavior of the recall to manipulate the text itself. To do this, you can add modifiers after a colon (:
) character at the end of an expansion.
For instance, you can chop off the path leading up to a file by using the h
modifier (it stands for “head”), which removes the path up until the final slash (/
) character. Be aware that this won’t work the way you want it to if you are using this to truncate a directory path and the path ends with a trailing slash.
A common use case for this is if you are modifying a file and realize you’d like to change to the file’s directory to do operations on related files.
For instance, say you run this command to print the contents of an open-source software license to your output:
- cat /usr/share/common-licenses/Apache-2.0
After being satisfied that the license suits your needs, you may want to change into the directory where it’s held. You can do this by calling the cd
command on the argument chain and chopping off the filename at the end:
- cd !!:$:h
If you run pwd
, which prints your current working directory, you’ll find that you’ve navigated to the directory included in the previous command:
- pwd
Output/usr/share/common-licenses
Once you’re there, you may want to open that license file again to double check, this time in a pager like less
.
To do this, you could perform the reverse of the previous manipulation by chopping off the path and using only the filename with the t
modifier, which stands for “tail”. You can search for your last cat
operation and use the t
flag to pass only the file name:
- less !cat:$:t
You could just as easily keep the full absolute path name and this command would work correctly in this instance. However, there may be other times when this isn’t true. For instance, you could be looking at a file nested within a few subdirectories below your current working directory using a relative path and then change to the subdirectory using the “h” modifier. In this case, you wouldn’t be able to rely on the relative path name to reach the file any more.
Another extremely helpful modifier is the r
modifier which strips the trailing extension. This could be useful if you are using tar
to extract a file and want to change into the resulting directory afterwards. Assuming the directory produced is the same name as the file, you could do something like:
- tar xzvf long-project-name.tgz
- cd !!:$:r
If your tarball uses the tar.gz
extension instead of tgz
, you can just pass the modifier twice:
- tar xzvf long-project-name.tgz
- cd !!:$:r:r
A similar modifier, e
, removes everything besides the trailing extension.
If you do not want to execute the command that you are recalling and only want to find it, you can use the p
modifier to have bash echo the command instead of executing it.
This is useful if you are unsure of if you’re selecting the correct piece. This not only prints it, but also puts it into your history for further editing if you’d like to modify it.
For instance, imagine you ran a find
command on your home directory and then realized that you wanted to run it from the root (/
) directory. You could check that you’re making the correct substitutions like this (assuming the original command is associated with the number 119 in your history):
- find ~ -name "file1"
- !119:0:p / !119:2*:p
Outputfind / -name "file1"
If the returned command is correct, you can execute it with the CTRL + P
key combination.
You can also make substitutions in your command by using the s/original/new/
syntax.
For instance, you could have accomplished that by typing:
- !119:s/~/\//
This will substitute the first instance of the search pattern (~
).
You can substitute every match by also passing the g
flag with the s
. For instance, if you want to create files named file1
, file2
, and file3
, and then want to create directories called dir1
, dir2
, dir3
, you could do this:
- touch file1 file2 file3
- mkdir !!:*:gs/file/dir/
Of course, it may be more intuitive to just run mkdir dir1 dir2 dir3
in this case. However, as you become comfortable using modifiers and the other bash history expansion tools, you can greatly expand your capabilities and productivity on the command line.
By reading this guide, you should now have a good idea of how you can leverage the history operations available to you. Some of these will probably be more useful than others, but it is good to know that bash has these capabilities in case you find yourself in a position where it would be helpful to dig them up.
If nothing else, the history
command alone, the reverse search, and the basic history expansions can do a lot to help you speed up your workflow.
Symbolic links allow you to link files and directories to other files and directories. They go by many names including symlinks, shell links, soft links, shortcuts, and aliases. From the user’s perspective, symbolic links are very similar to normal files and directories. However, when you interact with them, they will actually interact with the target at the other end. Think of them like wormholes for your file system.
This guide provides an overview of what symbolic links are and how to to create them from a Linux command line using the ln
command.
To follow along with this guide, you will need access to a computer running a Linux-based operating system. This can either be a virtual private server which you’ve connected to with SSH or your local machine. Note that this tutorial was validated using a Linux server running Ubuntu 20.04, but the examples given should work on a computer running any version of any Linux distribution.
If you plan to use a remote server to follow this guide, we encourage you to first complete our Initial Server Setup guide. Doing so will set you up with a secure server environment — including a non-root user with sudo
privileges and a firewall configured with UFW — which you can use to build your Linux skills.
The system call necessary to create symbolic links tends to be readily available on Unix-like and POSIX-compliant operating systems. The command we’ll be using to create the links is the ln
command.
You’re welcome to use existing files on your system to practice making symbolic links, but this section provides a few commands that will set up a practice environment you can use to follow along with this guide’s examples.
Begin by creating a couple directories within the /tmp/
directory. /tmp/
is a temporary directory, meaning that any files and directories within it will be deleted the next time the server boots up. This will be useful for the purposes of this guide, since you can create as many directories, files, and links as you’d like without having to worry about them clogging up your system later on.
The following mkdir
command creates three directories at once. It creates a directory named symlinks/
within the /tmp/
directory, and two directories (one named one/
and another named two/
) within symlinks/
:
- mkdir -p /tmp/symlinks/{one,two}
Navigate into the new symlinks/
directory:
- cd /tmp/symlinks
From there, create a couple sample files, one for both of the subdirectories within symlinks/
. The following command creates a file named one.txt
within the one/
subdirectory whose only contents are a single line reading one
:
- echo "one" > ./one/one.txt
Similarly, this command creates a file named two.txt
within the two/
subdirectory whose only contents are a single line reading two
:
- echo "two" > ./two/two.txt
If you were to run tree
at this point to display the contents of the entire /tmp/symlinks
directory and any nested subdirectories, its output would look like this:
- tree
Output.
├── one
│ └── one.txt
└── two
└── two.txt
2 directories, 2 files
Note: If tree
isn’t installed on your machine by default, you can install it using your system’s package manager. On Ubuntu, for example, you can install it with apt
:
- sudo apt install tree
With these sample documents in place, you’re ready to practice making symbolic links.
By default, the ln
command will make hard links instead of symbolic, or soft, links.
Say you have a text file. If you make a symbolic link to that file, the link is only a pointer to the original file. If you delete the original file, the link will be broken as it no longer has anything to point to.
A hard link is instead a mirror copy of an original file with the exact same contents. Like symbolic links, if you edit the contents of the original file those changes will be reflected in the hard link. If you delete the original file, though, the hard link will still work, and you can view and edit it as you would a normal copy of the original file.
Hard links serve their purpose in the world, but they should be avoided entirely in some cases. For instance, you should avoid using hard links when linking inside of a git
repository as they can cause confusion.
To ensure that you’re creating symbolic links, you can pass the -s
or --symbolic
option to the ln
command.
Note: Because symbolic links are typically used more frequently than hard links, some may find it beneficial to alias ln
to ln -s
:
- alias ln="ln -s"
This may save only a few keystrokes, but if you find yourself making a lot of symbolic links this could add up significantly.
As mentioned previously, symbolic linking is essentially like creating a file that contains the target’s filename and path. Because a symbolic link is just a reference to the original file, any changes that are made to the original will be immediately available in the symbolic link.
One potential use for symbolic links is to create local directories in a user’s home directory pointing to files being synchronized to an external application, like Dropbox. Another might be to create a symbolic link that points to the latest build of a project that resides in a dynamically-named directory.
Using the example files and directories from the first section, go ahead and try creating a symbolic link named three
that points to the one
directory you created previously:
- ln -s one three
Now you should have 3 directories, one of which is pointing back to another. To get a more detailed overview of the current directory structure, you can use the ls
command to print the contents of the current working directory:
- ls
Outputone three two
There are now three directories within the symlinks/
directory. Depending on your system, it may signify that three
is in fact a symbolic link. This is sometimes done by rendering the name of the link in a different color, or appending it with an @
symbol.
For even greater detail, you can pass the -l
argument to ls
to determine where the symbolic link is actually pointing:
- ls -l
Outputtotal 8
drwxrwxr-x 2 sammy sammy 4096 Oct 30 19:51 one
lrwxrwxrwx 1 sammy sammy 3 Oct 30 19:55 three -> one
drwxrwxr-x 2 sammy sammy 4096 Oct 30 19:51 two
Notice that the three
link is pointing to the one
directory as expected. Also, it begins with an l
, which indicates it’s a link. The other two begin with d
, meaning that they are regular directories.
Symbolic links can also contain symbolic links. As an example, link the one.txt
file from three
to the two
directory:
- ln -s three/one.txt two/one.txt
You should now have a file named one.txt
inside of the two
directory. You can check with the following ls
command:
- ls -l two/
Outputtotal 4
lrwxrwxrwx 1 sammy sammy 13 Oct 30 19:58 one.txt -> three/one.txt
-rw-rw-r-- 1 sammy sammy 4 Oct 30 19:51 two.txt
Depending on your terminal configuration, the link (highlighted in this example output) may be rendered in red text, indicating a broken link. Although the link was created, the way this example specified the path was relative. The link is broken because the two
directory doesn’t contain a three
directory with the one.txt
file in it.
Fortunately, you can remedy this situation by telling ln
to create the symbolic link relative to the link location using the -r
or --relative
argument.
Even with the -r
flag, though, you won’t be able to fix the broken symbolic link. The reason for this is the symbolic link already exists, and you won’t be able to overwrite it without including the -f
or --force
argument as well:
- ln -srf three/one.txt two/one.txt
With that, you now have two/one.txt
which was linked to three/one.txt
which is a link to one/one.txt
.
Nesting symbolic links like this can quickly get confusing, but many applications are equipped to make such linking structures more understandable. For instance, if you were to run the tree
command, the link target being shown is actually that of the original file location and not the link itself:
- tree
Output.
├── one
│ └── one.txt
├── three -> one
└── two
├── one.txt -> ../one/one.txt
└── two.txt
3 directories, 3 files
Now that things are linked up nicely, you can begin exploring how symbolic links work with files by altering the contents of these sample files.
To get a sense of what your files contain, run the following cat
command to print the contents of the one.txt
file in each of the three directories you’ve created in this guide:
- cat {one,two,three}/one.txt
Outputone
one
one
Next, update the contents of the original one.txt
file from the one/
directory:
- echo "1. One" > one/one.txt
Then check the contents of each file again:
- cat {one,two,three}/one.txt
Output1. One
1. One
1. One
As this output indicates, any changes you make to the original file will be reflected in any of its symbolic links.
Now try out the reverse. Run the following command to change the contents of one of the symbolic links. This example changes the contents of the one.txt
file within the three/
directory:
- echo "One and done" > three/one.txt
Then check the contents of each file once again:
- cat {one,two,three}/one.txt
OutputOne and done
One and done
One and done
Because the symbolic link you changed is just a pointer to another file, any change you make to the link will be immediately reflected in the original file as well as any of its other symbolic links.
Symbolic links can be incredibly useful, but they do have certain limitations. Keep in mind that if you were to move or delete the original file or directory, all of your existing symbolic links pointed to it will become broken. There’s no automatic updating in that scenario. As long as you’re careful, though, you can find many uses for symbolic links as you continue working with the command line.
]]>poweroff
I haven’t been able to make this work. The poweroff
command works from the CLI, if I am logged in as a user, even if the user is not a sudoer.
But if noone is logged onto the CLI, and the task finishes without a user being logged into the shell, the poweroff
command appears to have no effect.
How do I power off a droplet programmatically from a bash script?
]]>$1
. I use the following method for this:
cat > /etc/nginx/sites-available/website <<EOL
...
rewrite ^/api/?(.*)$ /web/api.php?url=$1 last;
...
EOL
This makes the output file look like this:
...
rewrite ^/api/?(.*)$ /web/api.php?url= last;
...
How do I make cat not interpret $1
as a parameter?
**<VirtualHost *:80> ServerName mydomain.com Redirect permanent / https://www.mydomain.com/ </VirtualHost> <VirtualHost *:443> ServerName mydomain.com Redirect permanent / https://www.mydomain.com/ DocumentRoot /var/www/html SSLEngine on SSLCertificateFile /etc/ssl/mydomain_com.crt SSLCertificateKeyFile /etc/ssl/mydomain_com.key SSLCertificateChainFile /etc/ssl/mydomain_com.ca-bundle </VirtualHost>
So, how can I stop the Redirect Loop? I also have /etc/apache2/sites-enabled/000-default.conf:30
**<VirtualHost *:443> ServerName mydomain.com DocumentRoot /var/www/html
SSLEngine on SSLCertificateFile /etc/ssl/mydomain_com.crt SSLCertificateKeyFile /etc/ssl/mydomain_com.key SSLCertificateChainFile /etc/ssl/mydomain_com.ca-bundle </VirtualHost>**
Does this look correct? Please take a look. And reply. With how I can stop the redirect. Loop. Thanks
]]>scp -r root@X.X.X.X:/home/userme ./
With this, it does download files, but I got a really old sql that I have since deleted. When I try to do specific newer files, it says: ‘No such file or directory’
I have not succeeded in the direction of doing it from the droplet terminal down to my local. I get permission denied when I tried that.
I had a lot of success of doing images (wp-content) from local up and down to the droplet. Looks like I had success with an sql file as well (this was two weeks ago). But today it is giving me old files.
]]>The site is Joomla-based and I am using Akeeba Backup to handle the backups. The recommendations for the command-line CRON jobs are as follows:
Use the following command in your host’s CRON interface: /path/to/php /var/www/domain.com/cli/akeeba-backup.php
Remember to substitute /path/to/php with the real path to your host’s PHP CLI (Command Line Interface) executable. Do remember that you must use the PHP CLI executable; the PHP CGI (Common Gateway Interface) executable will not work with our CRON scripts. If unsure what this means, please consult your host. They are the only people who can provide this information.
domain.com used as an example
I have CRON installed and running on Ubuntu 20.04.
The following code is in crontab -e
0 3 * * * /usr/bin/php7.4 /var/www/domain.com/cli/akeeba-backup.php --profile=1 --description="Full automated backup"
The site is running PHP 7.4 with FPM installed.
When I run which php
the following is returned: /usr/bin/php
When I run ls -l /usr/bin/php
the following is returned: lrwxrwxrwx 1 root root 21 Aug 31 18:09 /usr/bin/php -> /etc/alternatives/php
When I run ls -l /etc/alternatives/php
the following is returned: lrwxrwxrwx 1 root root 15 Aug 31 18:53 /etc/alternatives/php -> /usr/bin/php8.0
As this shows version 8.0 instead of 7.4, which the site is using, does this mean the CRON job should be using version 8 instead.
Sample log output from /var/log/syslog
Oct 1 11:39:01 domain CRON[38538]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Any help is greatly appreciated.
]]>I (by mistake) wanted to install libdwg… but dnf proposed “try --allowerasing”
I think it erased nscd and bunch of gcc stuff…
now, my droplet is not accessible, ssh don’t work.
When I try the recovery console, I cannot type :(
it says "Probing EDD (edd=off to disable)… ok
what can I do next ?
when I contact the support, they always point me “go to the recovery console” or “log to ssh”, and (in a loop) they keep repeating the same.
]]>The grep
command is one of the most useful commands in a Linux terminal environment. The name grep
stands for “global regular expression print”. This means that you can use grep
to check whether the input it receives matches a specified pattern. This seemingly trivial program is extremely powerful; its ability to sort input based on complex rules makes it a popular link in many command chains.
In this tutorial, you will explore the grep
command’s options, and then you’ll dive into using regular expressions to do more advanced searching.
To follow along with this guide, you will need access to a computer running a Linux-based operating system. This can either be a virtual private server which you’ve connected to with SSH or your local machine. Note that this tutorial was validated using a Linux server running Ubuntu 20.04, but the examples given should work on a computer running any version of any Linux distribution.
If you plan to use a remote server to follow this guide, we encourage you to first complete our Initial Server Setup guide. Doing so will set you up with a secure server environment — including a non-root user with sudo
privileges and a firewall configured with UFW — which you can use to build your Linux skills.
In this tutorial, you’ll use grep
to search the GNU General Public License version 3 for various words and phrases.
If you’re on an Ubuntu system, you can find the file in the /usr/share/common-licenses
folder. Copy it to your home directory:
- cp /usr/share/common-licenses/GPL-3 .
If you’re on another system, use the curl
command to download a copy:
- curl -o GPL-3 https://www.gnu.org/licenses/gpl-3.0.txt
You’ll also use the BSD license file in this tutorial. On Linux, you can copy that to your home directory with the following command:
- cp /usr/share/common-licenses/BSD .
If you’re on another system, create the file with the following command:
- cat << 'EOF' > BSD
- Copyright (c) The Regents of the University of California.
- All rights reserved.
-
- Redistribution and use in source and binary forms, with or without
- modification, are permitted provided that the following conditions
- are met:
- 1. Redistributions of source code must retain the above copyright
- notice, this list of conditions and the following disclaimer.
- 2. Redistributions in binary form must reproduce the above copyright
- notice, this list of conditions and the following disclaimer in the
- documentation and/or other materials provided with the distribution.
- 3. Neither the name of the University nor the names of its contributors
- may be used to endorse or promote products derived from this software
- without specific prior written permission.
-
- THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
- ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
- FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- SUCH DAMAGE.
- EOF
Now that you have the files, you can start working with grep
.
In the most basic form, you use grep
to match literal patterns within a text file. This means that if you pass grep
a word to search for, it will print out every line in the file containing that word.
Execute the following command to use grep
to search for every line that contains the word GNU
:
- grep "GNU" GPL-3
The first argument, GNU
, is the pattern you’re searching for, while the second argument, GPL-3
, is the input file you wish to search.
The resulting output will be every line containing the pattern text:
Output GNU GENERAL PUBLIC LICENSE
The GNU General Public License is a free, copyleft license for
the GNU General Public License is intended to guarantee your freedom to
GNU General Public License for most of our software; it applies also to
Developers that use the GNU GPL protect your rights with two steps:
"This License" refers to version 3 of the GNU General Public License.
13. Use with the GNU Affero General Public License.
under version 3 of the GNU Affero General Public License into a single
...
...
On some systems, the pattern you searched for will be highlighted in the output.
By default, grep
will search for the exact specified pattern within the input file and return the lines it finds. You can make this behavior more useful though by adding some optional flags to grep
.
If you want grep
to ignore the “case” of your search parameter and search for both upper- and lower-case variations, you can specify the -i
or --ignore-case
option.
Search for each instance of the word license
(with upper, lower, or mixed cases) in the same file as before with the following command:
- grep -i "license" GPL-3
The results contain: LICENSE
, license
, and License
:
Output GNU GENERAL PUBLIC LICENSE
of this license document, but changing it is not allowed.
The GNU General Public License is a free, copyleft license for
The licenses for most software and other practical works are designed
the GNU General Public License is intended to guarantee your freedom to
GNU General Public License for most of our software; it applies also to
price. Our General Public Licenses are designed to make sure that you
(1) assert copyright on the software, and (2) offer you this License
"This License" refers to version 3 of the GNU General Public License.
"The Program" refers to any copyrightable work licensed under this
...
...
If there was an instance with LiCeNsE
, that would have been returned as well.
If you want to find all lines that do not contain a specified pattern, you can use the -v
or --invert-match
option.
Search for every line that does not contain the word the
in the BSD license with the following command:
- grep -v "the" BSD
You’ll receive this output:
OutputAll rights reserved.
Redistribution and use in source and binary forms, with or without
are met:
may be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
...
...
Since you did not specify the “ignore case” option, the last two items were returned as not having the word the
.
It is often useful to know the line number that the matches occur on. You can do this by using the -n
or --line-number
option. Re-run the previous example with this flag added:
- grep -vn "the" BSD
This will return the following text:
Output2:All rights reserved.
3:
4:Redistribution and use in source and binary forms, with or without
6:are met:
13: may be used to endorse or promote products derived from this software
14: without specific prior written permission.
15:
16:THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
17:ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
...
...
Now you can reference the line number if you want to make changes to every line that does not contain the
. This is especially handy when working with source code.
In the introduction, you learned that grep
stands for “global regular expression print”. A “regular expression” is a text string that describes a particular search pattern.
Different applications and programming languages implement regular expressions slightly differently. In this tutorial you will only be exploring a small subset of the way that grep
describes its patterns.
In the previous examples in this tutorial, when you searched for the words GNU
and the
, you were actually searching for basic regular expressions which matched the exact string of characters GNU
and the
. Patterns that exactly specify the characters to be matched are called “literals” because they match the pattern literally, character-for-character.
It is helpful to think of these as matching a string of characters rather than matching a word. This will become a more important distinction as you learn more complex patterns.
All alphabetical and numerical characters (as well as certain other characters) are matched literally unless modified by other expression mechanisms.
Anchors are special characters that specify where in the line a match must occur to be valid.
For instance, using anchors, you can specify that you only want to know about the lines that match GNU
at the very beginning of the line. To do this, you could use the ^
anchor before the literal string.
Run the following command to search the GPL-3
file and find lines where GNU
occurs at the very beginning of a line:
- grep "^GNU" GPL-3
This command will return the following two lines:
OutputGNU General Public License for most of our software; it applies also to
GNU General Public License, you may choose any version ever published
Similarly, you use the $
anchor at the end of a pattern to indicate that the match will only be valid if it occurs at the very end of a line.
This command will match every line ending with the word and
in the GPL-3
file:
- grep "and$" GPL-3
You’ll receive this output:
Outputthat there is no warranty for this free software. For both users' and
The precise terms and conditions for copying, distribution and
License. Each licensee is addressed as "you". "Licensees" and
receive it, in any medium, provided that you conspicuously and
alternative is allowed only occasionally and noncommercially, and
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
provisionally, unless and until the copyright holder explicitly and
receives a license from the original licensors, to run, modify and
make, use, sell, offer for sale, import and otherwise run, modify and
The period character (.) is used in regular expressions to mean that any single character can exist at the specified location.
For example, to match anything in the GPL-3
file that has two characters and then the string cept
, you would use the following pattern:
- grep "..cept" GPL-3
This command returns the following output:
Outputuse, which is precisely where it is most unacceptable. Therefore, we
infringement under applicable copyright law, except executing it on a
tells the user that there is no warranty for the work (except to the
License by making exceptions from one or more of its conditions.
form of a separately written license, or stated as exceptions;
You may not propagate or modify a covered work except as expressly
9. Acceptance Not Required for Having Copies.
...
...
This output has instances of both accept
and except
and variations of the two words. The pattern would also have matched z2cept
if that was found as well.
By placing a group of characters within brackets (\[
and \]
), you can specify that the character at that position can be any one character found within the bracket group.
For example, to find the lines that contain too
or two
, you would specify those variations succinctly by using the following pattern:
- grep "t[wo]o" GPL-3
The output shows that both variations exist in the file:
Outputyour programs, too.
freedoms that you received. You must make sure that they, too, receive
Developers that use the GNU GPL protect your rights with two steps:
a computer network, with no transfer of a copy, is not conveying.
System Libraries, or general-purpose tools or generally available free
Corresponding Source from a network server at no charge.
...
...
Bracket notation gives you some interesting options. You can have the pattern match anything except the characters within a bracket by beginning the list of characters within the brackets with a ^
character.
This example is like the pattern .ode
, but will not match the pattern code
:
- grep "[^c]ode" GPL-3
Here’s the output you’ll receive:
Output 1. Source Code.
model, to give anyone who possesses the object code either (1) a
the only significant mode of use of the product.
notice like this when it starts in an interactive mode:
Notice that in the second line returned, there is, in fact, the word code
. This is not a failure of the regular expression or grep. Rather, this line was returned because earlier in the line, the pattern mode
, found within the word model
, was found. The line was returned because there was an instance that matched the pattern.
Another helpful feature of brackets is that you can specify a range of characters instead of individually typing every available character.
This means that if you want to find every line that begins with a capital letter, you can use the following pattern:
- grep "^[A-Z]" GPL-3
Here’s the output this expression returns:
OutputGNU General Public License for most of our software; it applies also to
States should not allow patents to restrict development and use of
License. Each licensee is addressed as "you". "Licensees" and
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
System Libraries, or general-purpose tools or generally available free
Source.
User Product is transferred to the recipient in perpetuity or for a
...
...
Due to some legacy sorting issues, it is often more accurate to use POSIX character classes instead of character ranges like you just used.
To discuss every POSIX character class would be beyond the scope of this guide, but an example that would accomplish the same procedure as the previous example uses the \[:upper:\]
character class within a bracket selector:
- grep "^[[:upper:]]" GPL-3
The output will be the same as before.
Finally, one of the most commonly used meta-characters is the asterisk, or *
, which means “repeat the previous character or expression zero or more times”.
To find each line in the GPL-3
file that contains an opening and closing parenthesis, with only letters and single spaces in between, use the following expression:
- grep "([A-Za-z ]*)" GPL-3
You’ll get the following output:
Output Copyright (C) 2007 Free Software Foundation, Inc.
distribution (with or without modification), making available to the
than the work as a whole, that (a) is included in the normal form of
Component, and (b) serves only to enable use of the work with that
(if any) on which the executable work runs, or a compiler used to
(including a physical distribution medium), accompanied by the
(including a physical distribution medium), accompanied by a
place (gratis or for a charge), and offer equivalent access to the
...
...
So far you’ve used periods, asterisks, and other characters in your expressions, but sometimes you need to search for those characters specifically.
There are times where you’ll need to search for a literal period or a literal opening bracket, especially when working with source code or configuration files. Because these characters have special meaning in regular expressions, you need to “escape” these characters to tell grep
that you do not wish to use their special meaning in this case.
You escape characters by using the backslash character (\
) in front of the character that would normally have a special meaning.
For instance, to find any line that begins with a capital letter and ends with a period, use the following expression which escapes the ending period so that it represents a literal period instead of the usual “any character” meaning:
- grep "^[A-Z].*\.$" GPL-3
This is the output you’ll see:
OutputSource.
License by making exceptions from one or more of its conditions.
License would be to refrain entirely from conveying the Program.
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
SUCH DAMAGES.
Also add information on how to contact you by electronic and paper mail.
Now let’s look at other regular expression options.
The grep
command supports a more extensive regular expression language by using the -E
flag or by calling the egrep
command instead of grep
.
These options open up the capabilities of “extended regular expressions”. Extended regular expressions include all of the basic meta-characters, along with additional meta-characters to express more complex matches.
One of the most useful abilities that extended regular expressions open up is the ability to group expressions together to manipulate or reference as one unit.
To group expressions together, wrap them in parentheses. If you would like to use parentheses without using extended regular expressions, you can escape them with the backslash to enable this functionality. This means that the following three expressions are functionally equivalent:
- grep "\(grouping\)" file.txt
- grep -E "(grouping)" file.txt
- egrep "(grouping)" file.txt
Similar to how bracket expressions can specify different possible choices for single character matches, alternation allows you to specify alternative matches for strings or expression sets.
To indicate alternation, use the pipe character |
. These are often used within parenthetical grouping to specify that one of two or more possibilities should be considered a match.
The following will find either GPL
or General Public License
in the text:
- grep -E "(GPL|General Public License)" GPL-3
The output looks like this:
Output The GNU General Public License is a free, copyleft license for
the GNU General Public License is intended to guarantee your freedom to
GNU General Public License for most of our software; it applies also to
price. Our General Public Licenses are designed to make sure that you
Developers that use the GNU GPL protect your rights with two steps:
For the developers' and authors' protection, the GPL clearly explains
authors' sake, the GPL requires that modified versions be marked as
have designed this version of the GPL to prohibit the practice for those
...
...
Alternation can select between more than two choices by adding additional choices within the selection group separated by additional pipe (|
) characters.
Like the *
meta-character that matched the previous character or character set zero or more times, there are other meta-characters available in extended regular expressions that specify the number of occurrences.
To match a character zero or one times, you can use the ?
character. This makes character or character sets that came before optional, in essence.
The following matches copyright
and right
by putting copy
in an optional group:
- grep -E "(copy)?right" GPL-3
You’ll receive this output:
Output Copyright (C) 2007 Free Software Foundation, Inc.
To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights. Therefore, you have
know their rights.
Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
"Copyright" also means copyright-like laws that apply to other kinds of
...
The +
character matches an expression one or more times. This is almost like the *
meta-character, but with the +
character, the expression must match at least once.
The following expression matches the string free
plus one or more characters that are not white space characters:
- grep -E "free[^[:space:]]+" GPL-3
You’ll see this output:
Output The GNU General Public License is a free, copyleft license for
to take away your freedom to share and change the works. By contrast,
the GNU General Public License is intended to guarantee your freedom to
When we speak of free software, we are referring to freedom, not
have the freedom to distribute copies of free software (and charge for
you modify it: responsibilities to respect the freedom of others.
freedomss that you received. You must make sure that they, too, receive
protecting users' freedom to change the software. The systematic
of the GPL, as needed to protect the freedom of users.
patents cannot be used to render the program non-free.
To specify the number of times that a match is repeated, use the brace characters ({
and }
). These characters let you specify an exact number, a range, or an upper or lower bounds to the amount of times an expression can match.
Use the following expression to find all of the lines in the GPL-3
file that contain triple-vowels:
- grep -E "[AEIOUaeiou]{3}" GPL-3
Each line returned has a word with three vowels:
Outputchanged, so that their problems will not be attributed erroneously to
authors of previous versions.
receive it, in any medium, provided that you conspicuously and
give under the previous paragraph, plus a right to possession of the
covered work so as to satisfy simultaneously your obligations under this
To match any words that have between 16 and 20 characters, use the following expression:
- grep -E "[[:alpha:]]{16,20}" GPL-3
Here’s this command’s output:
Output certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.
c) Prohibiting misrepresentation of the origin of that material, or
Only lines containing words within that length are displayed.
grep
is useful in finding patterns within files or within the file system hierarchy, so it’s worth spending time getting comfortable with its options and syntax.
Regular expressions are even more versatile, and can be used with many popular programs. For instance, many text editors implement regular expressions for searching and replacing text.
Furthermore, most modern programming languages use regular expressions to perform procedures on specific pieces of data. Once you understand regular expressions, you’ll be able to transfer that knowledge to many common computer-related tasks, from performing advanced searches in your text editor to validating user input.
]]>In a previous tutorial, we discussed how the ps
, kill
, and nice
commands can be used to control processes on your system. This guide highlights how bash
, the Linux system, and your terminal come together to offer process and job control.
This article will focus on managing foreground and background processes and will demonstrate how to leverage your shell’s job control functions to gain more flexibility in how you run commands.
To follow along with this guide, you will need access to a computer running the bash
shell interface. bash
is the default shell on many Linux-based operating systems, and it is available on many Unix-like operating systems, including macOS. Note that this tutorial was validated using a Linux virtual private server running Ubuntu 20.04.
If you plan to use a remote server to follow this guide, we encourage you to first complete our Initial Server Setup guide. Doing so will set you up with a secure server environment — including a non-root user with sudo
privileges and a firewall configured with UFW — which you can use to build your Linux skills.
Most processes that you start on a Linux machine will run in the foreground. The command will begin execution, blocking use of the shell for the duration of the process. The process may allow user interaction or may just run through a procedure and then exit. Any output will be displayed in the terminal window by default. We’ll discuss the basic way to manage foreground processes in the following subsections.
By default, processes are started in the foreground. This means that until the program exits or changes state, you will not be able to interact with the shell.
Some foreground commands exit very quickly and return you to a shell prompt almost immediately. For instance, the following command will print Hello World
to the terminal and then return you to your command prompt:
- echo "Hello World"
OutputHello World
Other foreground commands take longer to execute, blocking shell access for their duration. This might be because the command is performing a more extensive operation or because it is configured to run until it is explicitly stopped or until it receives other user input.
A command that runs indefinitely is the top
utility. After starting, it will continue to run and update its display until the user terminates the process:
- top
You can quit top
by pressing q
, but some other processes don’t have a dedicated quit function. To stop those, you’ll have to use another method.
Suppose you start a simple bash
loop on the command line. As an example, the following command will start a loop that prints Hello World
every ten seconds. This loop will continue forever, until explicitly terminated:
- while true; do echo "Hello World"; sleep 10; done
Unlike top
, loops like this have no “quit” key. You will have to stop the process by sending it a signal. In Linux, the kernel can send signals to running processes as a request that they exit or change states. Linux terminals are usually configured to send the “SIGINT” signal (short for “signal interrupt”) to current foreground process when the user presses the CTRL + C
key combination. The SIGINT signal tells the program that the user has requested termination using the keyboard.
To stop the loop you’ve started, hold the CTRL
key and press the C
key:
CTRL + C
The loop will exit, returning control to the shell.
The SIGINT signal sent by the CTRL + C
combination is one of many signals that can be sent to programs. Most signals do not have keyboard combinations associated with them and must instead be sent using the kill
command, which will be covered later on in this guide.
As mentioned previously, foreground process will block access to the shell for the duration of their execution. What if you start a process in the foreground, but then realize that you need access to the terminal?
Another signal that you can send is the “SIGTSTP” signal. SIGTSTP is short for “signal terminal stop”, and is usually represented as signal number 20. When you press CTRL + Z
, your terminal registers a “suspend” command, which then sends the SIGTSTP signal to the foreground process. Essentially, this will pause the execution of the command and return control to the terminal.
To illustrate, use ping
to connect to google.com
every 5 seconds. The following command precedes the ping
command with command
, which will allow you to bypass any shell aliases that artificially set a maximum count on the command:
- command ping -i 5 google.com
Instead of terminating the command with CTRL + C
, press CTRL + Z
instead. Doing so will return output like this:
Output[1]+ Stopped ping -i 5 google.com
The ping
command has been temporarily stopped, giving you access to a shell prompt again. You can use the ps
process tool to show this:
- ps T
Output PID TTY STAT TIME COMMAND
26904 pts/3 Ss 0:00 /bin/bash
29633 pts/3 T 0:00 ping -i 5 google.com
29643 pts/3 R+ 0:00 ps t
This output indicates that the ping
process is still listed, but that the “STAT” column has a “T” in it. Per the ps
man page, this means that a job that has been “stopped by [a] job control signal”.
This guide will outline how to change process states in greater depth, but for now you can resume execution of the command in the foreground again by typing:
- fg
Once the process has resumed, terminate it with CTRL + C
:
The main alternative to running a process in the foreground is to allow it to execute in the background. A background process is associated with the specific terminal that started it, but does not block access to the shell. Instead, it executes in the background, leaving the user able to interact with the system while the command runs.
Because of the way that a foreground process interacts with its terminal, there can be only a single foreground process for every terminal window. Because background processes return control to the shell immediately without waiting for the process to complete, many background processes can run at the same time.
You can start a background process by appending an ampersand character (&
) to the end of your commands. This tells the shell not to wait for the process to complete, but instead to begin execution and to immediately return the user to a prompt. The output of the command will still display in the terminal (unless redirected), but you can type additional commands as the background process continues.
For instance, you can start the same ping
process from the previous section in the background by typing:
- command ping -i 5 google.com &
The bash
job control system will return output like this:
Output[1] 4287
You’ll then receive the normal output from the ping
command:
OutputPING google.com (74.125.226.71) 56(84) bytes of data.
64 bytes from lga15s44-in-f7.1e100.net (74.125.226.71): icmp_seq=1 ttl=55 time=12.3 ms
64 bytes from lga15s44-in-f7.1e100.net (74.125.226.71): icmp_seq=2 ttl=55 time=11.1 ms
64 bytes from lga15s44-in-f7.1e100.net (74.125.226.71): icmp_seq=3 ttl=55 time=9.98 ms
However, you can also type commands at the same time. The background process’s output will be mixed among the input and output of your foreground processes, but it will not interfere with the execution of the foreground processes.
To list all stopped or backgrounded processes, you can use the jobs
command:
- jobs
If you still have the previous ping
command running in the background, the jobs
command’s output will be similar to this:
Output[1]+ Running command ping -i 5 google.com &
This indicates that you currently have a single background process running. The [1]
represents the command’s job spec or job number. You can reference this with other job and process control commands, like kill
, fg
, and bg
by preceding the job number with a percentage sign. In this case, you’d reference this job as %1
.
You can stop the current background process in a few ways. The most straightforward way is to use the kill
command with the associated job number. For instance, you can kill your running background process by typing:
- kill %1
Depending on how your terminal is configured, either immediately or the next time you hit ENTER
, the job termination status will appear in your output:
Output[1]+ Terminated command ping -i 5 google.com
If you check the jobs
command again, there won’t be any current jobs.
Now that you know how to start and stop processes in the background, you can learn about changing their state.
This guide already outlined one way to change a process’s state: stopping or suspending a process with CTRL + Z
. When processes are in this stopped state, you can move a foreground process to the background or vice versa.
If you forget to end a command with &
when you start it, you can still move the process to the background.
The first step is to stop the process with CTRL + Z
again. Once the process is stopped, you can use the bg
command to start it again in the background:
- bg
You will receive the job status line again, this time with the ampersand appended:
Output[1]+ ping -i 5 google.com &
By default, the bg
command operates on the most recently-stopped process. If you’ve stopped multiple processes in a row without starting them again, you can reference a specific process by its job number to move the correct process to the background.
Note that not all commands can be backgrounded. Some processes will automatically terminate if they detect that they have been started with their standard input and output directly connected to an active terminal.
You can also move background processes to the foreground by typing fg
:
- fg
This operates on your most recently backgrounded process (indicated by the +
in the jobs
command’s output). It immediately suspends the process and puts it into the foreground. To specify a different job, use its job number:
- fg %2
Once a job is in the foreground, you can kill it with CTRL + C
, let it complete, or suspend and move it to the background again.
Whether a process is in the background or in the foreground, it is rather tightly tied with the terminal instance that started it. When a terminal closes, it typically sends a SIGHUP signal to all of the processes (foreground, background, or stopped) that are tied to the terminal. This signals for the processes to terminate because their controlling terminal will shortly be unavailable.
There may be times, though, when you want to close a terminal but keep the background processes running. There are a number of ways of accomplishing this. One of the more flexible ways is to use a terminal multiplexer like screen
or tmux
. Another solution is to use a utility that provides the detach functionality of screen
and tmux
, like dtach
.
However, this isn’t always an option. Sometimes these programs aren’t available or you’ve already started the process you need to continue running. Sometimes these could even be overkill for what you need to accomplish.
nohup
If you know when starting the process that you will want to close the terminal before the process completes, you can start it using the nohup
command. This makes the started process immune to the SIGHUP signal. It will continue running when the terminal closes and will be reassigned as a child of the init system:
- nohup ping -i 5 google.com &
This will return a line like the following, indicating that the output of the command will be written to a file called nohup.out
:
Outputnohup: ignoring input and appending output to ‘nohup.out’
This file will be placed in your current working directory if writeable, but otherwise it will be placed in your home directory. This is to ensure that output is not lost if the terminal window is closed.
If you close the terminal window and open another one, the process will still be running. You will not find it in the output of the jobs
command because each terminal instance maintains its own independent job queue. Closing the terminal will cause the ping
job to be destroyed even though the ping
process is still running.
To kill the ping
process, you’ll have to find its process ID (or “PID”). You can do that with the pgrep
command (there is also a pkill
command, but this two-part method ensures that you are only killing the intended process). Use pgrep
and the -a
flag to search for the executable:
- pgrep -a ping
Output7360 ping -i 5 google.com
You can then kill the process by referencing the returned PID, which is the number in the first column:
- kill 7360
You may wish to remove the nohup.out
file if you don’t need it anymore.
disown
The nohup
command is helpful, but only if you know you will need it at the time you start the process. The bash
job control system provides other methods of achieving similar results with the built-in disown
command.
The disown
command, in its default configuration, removes a job from the jobs queue of a terminal. This means that it can no longer be managed using the job control mechanisms discussed previously in this guide, like fg
, bg
, CTRL + Z
, CTRL + C
. Instead, the job will immediately be removed from the list in the jobs
output and no longer associated with the terminal.
The command is called by specifying a job number. For instance, to immediately disown job 2, you could type:
- disown %2
This leaves the process in a state not unlike that of a nohup
process after the controlling terminal has been closed. The exception is that any output will be lost when the controlling terminal closes if it is not being redirected to a file.
Usually, you don’t want to remove the process completely from job control if you aren’t immediately closing your terminal window. You can pass the -h
flag to the disown
process instead in order to mark the process to ignore SIGHUP signals, but to otherwise continue on as a regular job:
- disown -h %1
In this state, you could use normal job control mechanisms to continue controlling the process until closing the terminal. Upon closing the terminal, you will, once again, be stuck with a process with nowhere to output if you didn’t redirect to a file when starting it.
To work around that, you can try to redirect the output of your process after it is already running. This is outside the scope of this guide, but this post provides an explanation of how you could do that.
huponexit
Shell Optionbash
has another way of avoiding the SIGHUP problem for child processes. The huponexit
shell option controls whether bash
will send its child processes the SIGHUP signal when it exits.
Note: The huponexit
option only affects the SIGHUP behavior when a shell session termination is initiated from within the shell itself. Some examples of when this applies are when the exit
command or CTRL + D
is pressed within the session.
When a shell session is ended through the terminal program itself (through closing the window, etc.), the command huponexit
will have no affect. Instead of bash
deciding on whether to send the SIGHUP signal, the terminal itself will send the SIGHUP signal to bash
, which will then correctly propagate the signal to its child processes.
Despite the aforementioned caveats, the huponexit
option is perhaps one of the easiest to manage. You can determine whether this feature is on or off by typing:
- shopt huponexit
To turn it on, type:
- shopt -s huponexit
Now, if you exit your session by typing exit
, your processes will all continue to run:
- exit
This has the same caveats about program output as the last option, so make sure you have redirected your processes’ output prior to closing your terminal if this is important.
Learning job control and how to manage foreground and background processes will give you greater flexibility when running programs on the command line. Instead of having to open up many terminal windows or SSH sessions, you can often get by with stopping processes early or moving them to the background as needed.
]]>./aa ./bb/data ./cc/files/docs
I need to remove all lines from it that contain subdirectories. So in the example above, only the first line should remain. Do you know how to do this in Linux console?
Regards
]]>Each site will have the following permission requirements:
* Allow the configuration.php (Joomla) to be updated using the administrator panel.
* Allow all directories and files within /var/www/domain.com to be editable by a custom user created by me. This will happen via SFTP for one single user.
* Allow the Joomla and WordPress update systems (core and plugins) to work from the respective admin panels. This should happen without the FTP Layer enabled (Joomla) or the equivalent on WordPress.
* Allow for image/media uploads (images for Joomla and wp-content/uploads for WordPress) to be allowed from the Joomla and WordPress admin panels.
* When new files or directories are created, they should inherit the user/group ownership and permissions that were previously set. Files or directories will be created from within Joomla/WordPress or via SFTP.
I have been testing out various user/group ownership settings and file permissions from Ask Ubuntu post, but nothing has fully worked so far. Either making config updates via the admin panels work and SFTP does not or vice-versa.
Any suggestions or tips are greatly appreciated.
]]>Ubuntu 20.04 one-click WordPress installed
How to clean up the memory, clean the disk space, clean the server logs files and clean all unnecessary data that auto-generate?
These kinds of data taking a lot of space.
]]>Wget is a networking command-line tool that lets you download files and interact with REST APIs. It supports the HTTP
,HTTPS
, FTP
, and FTPS
internet protocols. Wget can deal with unstable and slow network connections. In the event of a download failure, Wget keeps trying until the entire file has been retrieved. Wget also lets you resume a file download that was interrupted without starting from scratch.
You can also use Wget to interact with REST APIs without having to install any additional external programs. You can make GET
, POST
, PUT
, and DELETE
HTTP
requests with single and multiple headers right in the terminal.
In this tutorial, you will use Wget to download files, interact with REST API endpoints, and create and manage a Droplet in your DigitalOcean account.
You can use your local system or a remote server to open a terminal and run the commands there.
Deploy your frontend applications from GitHub using DigitalOcean App Platform. Let DigitalOcean focus on scaling your app.
To complete this tutorial, you will need:
Wget installed. Most Linux distributions have Wget installed by default. To check, type wget
in your terminal and press ENTER
. If it is not installed, it will display: command not found
. You can install it by running the following command: sudo apt-get install wget
.
A DigitalOcean account. If you do not have one, sign up for a new account.
A DigitalOcean Personal Access Token, which you can create via the DigitalOcean control panel. Instructions to do that can be found here: How to Generate a Personal Access Token.
In this section, you will use Wget to customize your download experience. For example, you will learn to download a single file and multiple files, handle file downloads in unstable network conditions, and, in the case of a download interruption, resume a download.
First, create a directory to save the files that you will download throughout this tutorial:
- mkdir -p DigitalOcean-Wget-Tutorial/Downloads
With the command above, you have created a directory named DigitalOcean-Wget-Tutorial
, and inside of it, you created a subdirectory named Downloads
. This directory and its subdirectory will be where you will store the files you download.
Navigate to the DigitalOcean-Wget-Tutorial
directory:
- cd DigitalOcean-Wget-Tutorial
You have successfully created the directory where you will store the files you download.
Downloading a file
In order to download a file using Wget, type wget
followed by the URL of the file that you wish to download. Wget will download the file in the given URL and save it in the current directory.
Let’s download a minified version of jQuery using the following command:
- wget https://code.jquery.com/jquery-3.6.0.min.js
Don’t worry if you don’t know what jQuery is – you could have downloaded any file available on the internet. All you need to know is that you successfully used Wget to download a file from the internet.
The output will look similar to this:
Output--2021-07-21 16:25:11-- https://code.jquery.com/jquery-3.6.0.min.js
Resolving code.jquery.com (code.jquery.com)... 69.16.175.10, 69.16.175.42, 2001:4de0:ac18::1:a:1a, ...
Connecting to code.jquery.com (code.jquery.com)|69.16.175.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 89501 (87K) [application/javascript]
Saving to: ‘jquery-3.6.0.min.js’
jquery-3.6.0.min.js 100%[===================>] 87.40K 114KB/s in 0.8s
2021-07-21 16:25:13 (114 KB/s) - ‘jquery-3.6.0.min.js’ saved [89501/89501]
According to the output above, you have successfully downloaded and saved a file named jquery-3.6.0.min.js
to your current directory.
You can check the contents of the current directory using the following command:
- ls
The output will look similar to this:
OutputDownloads jquery-3.6.0.min.js
Specifying the filename for the downloaded file
When downloading a file, Wget defaults to storing it using the name that the file has on the server. You can change that by using the -O
option to specify a new name.
Download the jQuery file you downloaded previously, but this time save it under a different name:
- wget -O jquery.min.js https://code.jquery.com/jquery-3.6.0.min.js
With the command above, you set the jQuery file to be saved as jquery.min.js
instead of jquery-3.6.0.min.js
The output will look similar to this:
Output--2021-07-21 16:27:01-- https://code.jquery.com/jquery-3.6.0.min.js
Resolving code.jquery.com (code.jquery.com)... 69.16.175.10, 69.16.175.42, 2001:4de0:ac18::1:a:2b, ...
Connecting to code.jquery.com (code.jquery.com)|69.16.175.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 89501 (87K) [application/javascript]
Saving to: ‘jquery.min.js’
jquery.min.js 100%[==================================>] 87.40K 194KB/s in 0.4s
2021-07-21 16:27:03 (194 KB/s) - ‘jquery.min.js’ saved [89501/89501]
According to the output above, you have successfully downloaded the jQuery file and saved it as jquery.min.js
.
You can use the ls
command to list the contents of your current directory, and you will see the jquery.min.js
file there:
- ls
The output will look similar to this:
OutputDownloads jquery-3.6.0.min.js jquery.min.js
So far, you have used wget
to download files to the current directory. Next, you will download to a specific directory.
Downloading a file to a specific directory
When downloading a file, Wget stores it in the current directory by default. You can change that by using the -P
option to specify the name of the directory where you want to save the file.
Download the jQuery file you downloaded previously, but this time save it in the Downloads
subdirectory.
- wget -P Downloads/ https://code.jquery.com/jquery-3.6.0.min.js
The output will look similar to this:
Output--2021-07-21 16:28:50-- https://code.jquery.com/jquery-3.6.0.min.js
Resolving code.jquery.com (code.jquery.com)... 69.16.175.42, 69.16.175.10, 2001:4de0:ac18::1:a:2b, ...
Connecting to code.jquery.com (code.jquery.com)|69.16.175.42|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 89501 (87K) [application/javascript]
Saving to: ‘Downloads/jquery-3.6.0.min.js’
jquery-3.6.0.min.js 100%[==================================>] 87.40K 43.6KB/s in 2.0s
2021-07-21 16:28:53 (43.6 KB/s) - ‘Downloads/jquery-3.6.0.min.js’ saved [89501/89501]
Notice the last line where it says that the jquery-3.6.0.min.js
file was saved in the Downloads
directory.
If you use the ls Downloads
command to list the contents of the Downloads
directory, you will see the jQuery file there:
Run the ls
command:
- ls Downloads
The output will look similar to this:
Outputjquery-3.6.0.min.js
Turning Wget’s output off
By default, Wget outputs a lot of information to the terminal when you download a file. You can use the -q
option to turn off all output.
Download the jQuery file, but this time without showing any output:
- wget -q https://code.jquery.com/jquery-3.6.0.min.js
You won’t see any output, but if you use the ls
command to list the contents of the current directory you will find a file named jquery-3.6.0.min.js.1
:
- ls
The output will look similar to this:
OutputDownloads jquery-3.6.0.min.js jquery-3.6.0.min.js.1 jquery.min.js
Before saving a file, Wget checks whether the file exists in the desired directory. If it does, Wget adds a number to the end of the file. If you ran the command above one more time, Wget would create a file named jquery-3.6.0.min.js.2
. This number increases every time you download a file to a directory that already has a file with the same name.
You have successfully turned off Wget’s output, but now you can’t monitor the download progress. Let’s look at how to show the download progress bar.
Showing the download progress bar
Wget lets you show the download progress bar but hide any other output by using the -q
option alongside the --show-progress
option.
Download the jQuery file, but this time only show the download progress bar:
- wget -q --show-progress https://code.jquery.com/jquery-3.6.0.min.js
The output will look similar to this:
Outputjquery-3.6.0.min.js.2 100%[================================================>] 87.40K 207KB/s in 0.4s
Use the ls
command to check the contents of the current directory and you will find the file you have just downloaded with the name jquery-3.6.0.min.js.2
From this point forward you will be using the -q
and --show-progress
options in most of the subsequent Wget commands.
So far you have only downloaded a single file. Next, you will download multiple files.
Downloading multiple files
In order to download multiples files using Wget, you need to create a .txt
file and insert the URLs of the files you wish to download. After inserting the URLs inside the file, use the wget
command with the -i
option followed by the name of the .txt
file containing the URLs.
Create a file named images.txt
:
- nano images.txt
In images.txt
, add the following URLs:
https://cdn.pixabay.com/photo/2016/12/13/05/15/puppy-1903313__340.jpg
https://cdn.pixabay.com/photo/2016/01/05/17/51/maltese-1123016__340.jpg
https://cdn.pixabay.com/photo/2020/06/30/22/34/dog-5357794__340.jpg
The URLs link to three random images of dogs found on Pixabay. After you have added the URLs, save and close the file.
Now you will use the -i
option alongside the -P
,-q
and --show-progress
options that you learned earlier to download all three images to the Downloads
directory:
- wget -i images.txt -P Downloads/ -q --show-progress
The output will look similar to this:
Outputpuppy-1903313__340.jp 100%[=========================>] 26.44K 93.0KB/s in 0.3s
maltese-1123016__340. 100%[=========================>] 50.81K --.-KB/s in 0.06s
dog-5357794__340.jpg 100%[=========================>] 30.59K --.-KB/s in 0.07s
If you use the ls Downloads
command to list the contents of the Downloads
directory, you will find the names of the three images you have just downloaded:
- ls Downloads
The output will look similar to this:
Outputdog-5357794__340.jpg jquery-3.6.0.min.js maltese-1123016__340.jpg puppy-1903313__340.jpg
Limiting download speed
So far, you have download files with the maximum available download speed. However, you might want to limit the download speed to preserve resources for other tasks. You can limit the download speed by using the --limit-rate
option followed by the maximum speed allowed in kiloBits per second
and the letter k
.
Download the first image in the images.txt
file with a speed of 15 kB/S
to the Downloads
directory:
- wget --limit-rate 15k -P Downloads/ -q --show-progress https://cdn.pixabay.com/photo/2016/12/13/05/15/puppy-1903313__340.jpg
The output will look similar to this:
Outputpuppy-1903313__340.jpg.1 100%[====================================================>] 26.44K 16.1KB/s in 1.6s
If you use the ls Downloads
command to check the contents of the Downloads
directory, you will see the file you have just downloaded with the name puppy-1903313__340.jpg.1
.
When downloading a file that already exists, Wget creates a new file instead of overwriting the existing file. Next, you will overwrite a downloaded file.
Overwriting a downloaded file
You can overwrite a file you have downloaded by using the -O
option alongside the name of the file. In the code below, you will first download the second image listed in the images.txt
file to the current directory and then you will overwrite it.
First, download the second image to the current directory and set the name to image2.jpg
:
- wget -O image2.jpg -q --show-progress https://cdn.pixabay.com/photo/2016/12/13/05/15/puppy-1903313__340.jpg
The output will look similar to this::
Outputimage2.jpg 100%[====================================================>] 26.44K --.-KB/s in 0.04s
If you use the ls
command to check the contents of the current directory, you will see the file you have just downloaded with the name image2.jpg
.
If you wish to overwrite this image2.jpg
file, you can run the same command you ran earlier :
- wget -O image2.jpg -q --show-progress https://cdn.pixabay.com/photo/2016/12/13/05/15/puppy-1903313__340.jpg
You can run the command above as many times as you like and Wget will download the file and overwrite the existing one. If you run the command above without the -O
option, Wget will create a new file each time you run it.
Resuming a download
Thus far, you have successfully downloaded multiple files without interruption. However, if the download was interrupted, you can resume it by using the -c
option.
Run the following command to download a random image of a dog found on Pixabay. Note that in the command, you have set the maximum speed to 1 KB/S
. Before the image finishes downloading, press Ctrl+C
to cancel the download:
- wget --limit-rate 1k -q --show-progress https://cdn.pixabay.com/photo/2018/03/07/19/51/grass-3206938__340.jpg
To resume the download, pass the -c
option. Note that this will only work if you run this command in the same directory as the incomplete file:
- wget -c --limit-rate 1k -q --show-progress https://cdn.pixabay.com/photo/2018/03/07/19/51/grass-3206938__340.jpg
Up until now, you have only downloaded files in the foreground. Next, you will download files in the background.
Downloading in the background
You can download files in the background by using the -b
option.
Run the command below to download a random image of a dog from Pixabay in the background:
- wget -b https://cdn.pixabay.com/photo/2018/03/07/19/51/grass-3206938__340.jpg
When you download files in the background, Wget creates a file named wget-log
in the current directory and redirects all output to this file. If you wish to watch the status of the download, you can use the following command:
- tail -f wget-log
The output will look similar to this:
OutputResolving cdn.pixabay.com (cdn.pixabay.com)... 104.18.20.183, 104.18.21.183, 2606:4700::6812:14b7, ...
Connecting to cdn.pixabay.com (cdn.pixabay.com)|104.18.20.183|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 33520 (33K) [image/jpeg]
Saving to: ‘grass-3206938__340.jpg’
0K .......... .......... .......... .. 100% 338K=0.1s
2021-07-20 23:49:52 (338 KB/s) - ‘grass-3206938__340.jpg’ saved [33520/33520]
Setting a timeout
Until this point, we have assumed that the server that you are trying to download files from is working properly. However, let’s assume that the server is not working properly. You can use Wget to first limit the amount of time that you wait for the server to respond and then limit the number of times that Wget tries to reach the server.
If you wish to download a file but you are unsure if the server is working properly, you can set a timeout by using the -T
option followed by the time in seconds.
In the following command, you are setting the timeout to 5
seconds:
- wget -T 5 -q --show-progress https://cdn.pixabay.com/photo/2016/12/13/05/15/puppy-1903313__340.jpg
Setting maximum number of tries
You can also set how many times Wget attempts to download a file after being interrupted by passing the --tries
option followed by the number of tries.
By running the command below, you are limiting the number of tries to 3
:
- wget --tries=3 -q --show-progress https://cdn.pixabay.com/photo/2018/03/07/19/51/grass-3206938__340.jpg
If you would like to try indefinitely you can pass inf
alongside the --tries
option:
- wget --tries=inf -q --show-progress https://cdn.pixabay.com/photo/2018/03/07/19/51/grass-3206938__340.jpg
In this section, you used Wget to download a single file and multiple files, resume downloads, and handle network issues. In the next section, you will learn to interact with REST API endpoints.
In this section, you will use Wget to interact with REST APIs without having to install an external program. You will learn the syntax to send the most commonly used HTTP
methods: GET
, POST
, PUT
, and DELETE
.
We are going to use JSONPlaceholder as the mock REST API. JSONPlaceholder is a free online REST API that you can use for fake data. (The requests you send to it won’t affect any databases and the data won’t be saved.)
Sending GET requests
Wget lets you send GET
requests by running a command that looks like the following:
- wget -O- [ URL ]
In the command above, the -
after the -O
option means standard output, so Wget will send the output of the URL to the terminal instead of sending it to a file as you did in the previous section. GET
is the default HTTP
method that Wget uses.
Run the following command in the terminal window:
- wget -O- https://jsonplaceholder.typicode.com/posts?_limit=2
In the command above, you used wget
to send a GET
request to JSON Placeholder in order to retrieve two posts from the REST API
.
The output will look similar to this:
Output--2021-07-21 16:52:51-- https://jsonplaceholder.typicode.com/posts?_limit=2
Resolving jsonplaceholder.typicode.com (jsonplaceholder.typicode.com)... 104.21.10.8, 172.67.189.217, 2606:4700:3032::6815:a08, ...
Connecting to jsonplaceholder.typicode.com (jsonplaceholder.typicode.com)|104.21.10.8|:443... connected.
HTTP request sent, awaiting response... 200 OK'
Length: 600 [application/json]
Saving to: ‘STDOUT’
- 0%[ ] 0 --.-KB/s [
{
"userId": 1,
"id": 1,
"title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
"body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
},
{
"userId": 1,
"id": 2,
"title": "qui est esse",
"body": "est rerum tempore vitae\nsequi sint nihil reprehenderit dolor beatae ea dolores neque\nfugiat blanditiis voluptate porro vel nihil molestiae ut reiciendis\nqui aperiam non debitis possimus qui neque nisi nulla"
}
- 100%[==================================>] 600 --.-KB/s in 0s
2021-07-21 16:52:53 (4.12 MB/s) - written to stdout [600/600]
Notice the line where it says HTTP request sent, awaiting response... 200 OK
, which means that you have successfully sent a GET
request to JSONPlaceholder.
If that is too much output you can use the -q
option that you learned in the previous section to restrict the output to the results of the GET
request:
- wget -O- -q https://jsonplaceholder.typicode.com/posts?_limit=2
The output will look similar to this:
Output[
{
"userId": 1,
"id": 1,
"title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
"body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
},
{
"userId": 1,
"id": 2,
"title": "qui est esse",
"body": "est rerum tempore vitae\nsequi sint nihil reprehenderit dolor beatae ea dolores neque\nfugiat blanditiis voluptate porro vel nihil molestiae ut reiciendis\nqui aperiam non debitis possimus qui neque nisi nulla"
}
]
Sending POST requests
Wget lets you send POST
requests by running a command that looks like the following:
- wget --method==[post] -O- --body-data=[ body in json format ] --header=[ String ] [ URL ]
Run the following command:
- wget --method=post -O- -q --body-data='{"title": "Wget POST","body": "Wget POST example body","userId":1}' --header=Content-Type:application/json https://jsonplaceholder.typicode.com/posts
In the command above, you used wget
to send a POST
request to JSON Placeholder to create a new post. You set the method
to post
, the Header
to Content-Type:application/json
and sent the following request body
to it :{"title": "Wget POST","body": "Wget POST example body","userId":1}
.
The output will look similar to this:
Output{
"title": "Wget POST",
"body": "Wget POST example body",
"userId": 1,
"id": 101
}
Sending PUT requests
Wget lets you send PUT
requests by running a command that looks like the following:
- wget --method==[put] -O- --body-data=[ body in json format ] --header=[ String ] [ URL ]
Run the following command:
- wget --method=put -O- -q --body-data='{"title": "Wget PUT", "body": "Wget PUT example body", "userId": 1, "id":1}' --header=Content-Type:application/json https://jsonplaceholder.typicode.com/posts/1
In the command above you used wget
to send a PUT
request to JSON Placeholder to edit the first post in this REST API. You set the method
to put
, the Header
to Content-Type:application/json
and sent the following request body
to it :{"title": "Wget PUT", "body": "Wget PUT example body", "userId": 1, "id":1}
.
The output will look similar to this:
Output{
"body": "Wget PUT example body",
"title": "Wget PUT",
"userId": 1,
"id": 1
}
Sending DELETE requests
Wget lets you send DELETE
requests by running a command that looks like the following:
- wget --method==[delete] -O- [ URL ]
Run the following command:
- wget --method=delete -O- -q --header=Content-Type:application/json https://jsonplaceholder.typicode.com/posts/1
In the command above you used wget
to send a DELETE
request to JSON Placeholder to delete the first post in this REST API. You set the method
to delete
, and set the post you want to delete to 1
in the URL.
The output will look similar to this:
Output{}
In this section, you learned how to use Wget to send GET
, POST
, PUT
and DELETE
requests with only one header field. In the next section, you will learn how to send multiple header fields in order to create and manage a Droplet in your DigitalOcean account.
In this section, you will apply what you learned in the previous section and use Wget to create and manage a Droplet in your DigitalOcean account. But before you do that, you will learn how to send multiple headers
fields in a HTTP method.
The syntax for a command to send multiple headers looks like this:
- wget --header=[ first header ] --header=[ second header] --header=[ N header] [ URL ]
You can have as many headers
fields as you like by repeating the --header
option as many times as you need.
To create a Droplet or interact with any other resource in the DigitalOcean API, you will need to send two request headers:
Content-Type: application/json
Authorization: Bearer your_personal_access_token
You already saw the first header in the previous section. The second header is what lets you authenticate your account. It has the String named Bearer
followed by your DigitalOcean account Personal Access Token.
Run the following command, replacing your_personal_access_token
with your DigitalOcean Personal Access Token:
- wget --method=post -O- -q --header="Content-Type: application/json" --header="Authorization: Bearer your_personal_access_token" --body-data='{"name":"Wget-example","region":"nyc1","size":"s-1vcpu-1gb","image":"ubuntu-20-04-x64","tags": ["Wget-tutorial"]}' https://api.digitalocean.com/v2/droplets
With the command above, you have created an ubuntu-20-04-x64
Droplet in the nyc1
region named Wget-example
with 1vcpu
and 1gb
of memory, and you have set the tag to Wget-tutorial
. For more information about the attributes in the body-data
field, see the DigitalOcean API documentation.
The output will look similar to this:
Output{"droplet":{"id":237171073,"name":"Wget-example","memory":1024,"vcpus":1,"disk":25,"locked":false,"status":"new","kernel":null,"created_at":"2021-03-16T12:38:59Z","features":[],"backup_ids":[],"next_backup_window":null,"snapshot_ids":[],"image":{"id":72067660,"name":"20.04 (LTS) x64","distribution":"Ubuntu","slug":"ubuntu-20-04-x64","public":true,"regions":["nyc3","nyc1","sfo1","nyc2","ams2","sgp1","lon1","ams3","fra1","tor1","sfo2","blr1","sfo3"],"created_at":"2020-10-20T16:34:30Z","min_disk_size":15,"type":"base","size_gigabytes":0.52,"description":"Ubuntu 20.04 x86","tags":[],"status":"available"},"volume_ids":[],"size":{"slug":"s-1vcpu-1gb","memory":1024,"vcpus":1,"disk":25,"transfer":1.0,"price_monthly":5.0,"price_hourly":0.00744,"regions":["ams2","ams3","blr1","fra1","lon1","nyc1","nyc2","nyc3","sfo1","sfo3","sgp1","tor1"],"available":true,"description":"Basic"},"size_slug":"s-1vcpu-1gb","networks":{"v4":[],"v6":[]},"region":{"name":"New York 1","slug":"nyc1","features":["backups","ipv6","metadata","install_agent","storage","image_transfer"],"available":true,"sizes":["s-1vcpu-1gb","s-1vcpu-1gb-intel","s-1vcpu-2gb","s-1vcpu-2gb-intel","s-2vcpu-2gb","s-2vcpu-2gb-intel","s-2vcpu-4gb","s-2vcpu-4gb-intel","s-4vcpu-8gb","c-2","c2-2vcpu-4gb","s-4vcpu-8gb-intel","g-2vcpu-8gb","gd-2vcpu-8gb","s-8vcpu-16gb","m-2vcpu-16gb","c-4","c2-4vcpu-8gb","s-8vcpu-16gb-intel","m3-2vcpu-16gb","g-4vcpu-16gb","so-2vcpu-16gb","m6-2vcpu-16gb","gd-4vcpu-16gb","so1_5-2vcpu-16gb","m-4vcpu-32gb","c-8","c2-8vcpu-16gb","m3-4vcpu-32gb","g-8vcpu-32gb","so-4vcpu-32gb","m6-4vcpu-32gb","gd-8vcpu-32gb","so1_5-4vcpu-32gb","m-8vcpu-64gb","c-16","c2-16vcpu-32gb","m3-8vcpu-64gb","g-16vcpu-64gb","so-8vcpu-64gb","m6-8vcpu-64gb","gd-16vcpu-64gb","so1_5-8vcpu-64gb","m-16vcpu-128gb","c-32","c2-32vcpu-64gb","m3-16vcpu-128gb","m-24vcpu-192gb","g-32vcpu-128gb","so-16vcpu-128gb","m6-16vcpu-128gb","gd-32vcpu-128gb","m3-24vcpu-192gb","g-40vcpu-160gb","so1_5-16vcpu-128gb","m-32vcpu-256gb","gd-40vcpu-160gb","so-24vcpu-192gb","m6-24vcpu-192gb","m3-32vcpu-256gb","so1_5-24vcpu-192gb"]},"tags":["Wget-tutorial"]},"links":{"actions":[{"id":1164336542,"rel":"create","href":"https://api.digitalocean.com/v2/actions/1164336542"}]}}
If you see an output similar to the one above that means that you have successfully created a Droplet.
Now let’s get a list of all the Droplets in your account that have the tag Wget-tutorial
. Run the following command, replacing your_personal_access_token
with your DigitalOcean Personal Access Token:
- wget -O- -q --header="Content-Type: application/json" --header="Authorization: Bearer your_personal_access_token" https://api.digitalocean.com/v2/droplets?tag_name=Wget-tutorial
You should see the name of the Droplet you have just created in the output:
Output{"droplets":[{"id":237171073,"name":"Wget-example","memory":1024,"vcpus":1,"disk":25,"locked":false,"status":"active","kernel":null,"created_at":"2021-03-16T12:38:59Z","features":["private_networking"],"backup_ids":[],"next_backup_window":null,"snapshot_ids":[],"image":{"id":72067660,"name":"20.04 (LTS) x64","distribution":"Ubuntu","slug":"ubuntu-20-04-x64","public":true,"regions":["nyc3","nyc1","sfo1","nyc2","ams2","sgp1","lon1","ams3","fra1","tor1","sfo2","blr1","sfo3"],"created_at":"2020-10-20T16:34:30Z","min_disk_size":15,"type":"base","size_gigabytes":0.52,"description":"Ubuntu 20.04 x86","tags":[],"status":"available"},"volume_ids":[],"size":{"slug":"s-1vcpu-1gb","memory":1024,"vcpus":1,"disk":25,"transfer":1.0,"price_monthly":5.0,"price_hourly":0.00744,"regions":["ams2","ams3","blr1","fra1","lon1","nyc1","nyc2","nyc3","sfo1","sfo3","sgp1","tor1"],"available":true,"description":"Basic"},"size_slug":"s-1vcpu-1gb","networks":{"v4":[{"ip_address":"10.116.0.2","netmask":"255.255.240.0","gateway":"","type":"private"},{"ip_address":"204.48.20.197","netmask":"255.255.240.0","gateway":"204.48.16.1","type":"public"}],"v6":[]},"region":{"name":"New York 1","slug":"nyc1","features":["backups","ipv6","metadata","install_agent","storage","image_transfer"],"available":true,"sizes":["s-1vcpu-1gb","s-1vcpu-1gb-intel","s-1vcpu-2gb","s-1vcpu-2gb-intel","s-2vcpu-2gb","s-2vcpu-2gb-intel","s-2vcpu-4gb","s-2vcpu-4gb-intel","s-4vcpu-8gb","c-2","c2-2vcpu-4gb","s-4vcpu-8gb-intel","g-2vcpu-8gb","gd-2vcpu-8gb","s-8vcpu-16gb","m-2vcpu-16gb","c-4","c2-4vcpu-8gb","s-8vcpu-16gb-intel","m3-2vcpu-16gb","g-4vcpu-16gb","so-2vcpu-16gb","m6-2vcpu-16gb","gd-4vcpu-16gb","so1_5-2vcpu-16gb","m-4vcpu-32gb","c-8","c2-8vcpu-16gb","m3-4vcpu-32gb","g-8vcpu-32gb","so-4vcpu-32gb","m6-4vcpu-32gb","gd-8vcpu-32gb","so1_5-4vcpu-32gb","m-8vcpu-64gb","c-16","c2-16vcpu-32gb","m3-8vcpu-64gb","g-16vcpu-64gb","so-8vcpu-64gb","m6-8vcpu-64gb","gd-16vcpu-64gb","so1_5-8vcpu-64gb","m-16vcpu-128gb","c-32","c2-32vcpu-64gb","m3-16vcpu-128gb","m-24vcpu-192gb","g-32vcpu-128gb","so-16vcpu-128gb","m6-16vcpu-128gb","gd-32vcpu-128gb","m3-24vcpu-192gb","g-40vcpu-160gb","so1_5-16vcpu-128gb","m-32vcpu-256gb","gd-40vcpu-160gb","so-24vcpu-192gb","m6-24vcpu-192gb","m3-32vcpu-256gb","so1_5-24vcpu-192gb"]},"tags":["Wget-tutorial"],"vpc_uuid":"5ee0a168-39d1-4c60-a89c-0b47390f3f7e"}],"links":{},"meta":{"total":1}}
Now let’s take the id
of the Droplet you have created and use it to delete the Droplet. Run the following command, replacing your_personal_access_token
with your DigitalOcean Personal Access Token and your_droplet_id
with your Droplet id
:
- wget --method=delete -O- --header="Content-Type: application/json" --header="Authorization: Bearer your_personal_access_token" https://api.digitalocean.com/v2/droplets/your_droplet_id
In the command above, you added your Droplet id
to the URL to delete it. If you are seeing a 204 No Content
in the output, that means that you succeeded in deleting the Droplet.
In this section, you used Wget to send multiple headers. Then, you created and managed a Droplet in your DigitalOcean account.
In this tutorial, you used Wget to download files in stable and unstable network conditions and interact with REST API endpoints. You then used this knowledge to create and manage a Droplet in your DigitalOcean account. If you would like to learn more about Wget, visit this tool’s manual page. For more Linux command-line tutorials visit DigitalOcean community tutorials.
]]>I’ve been searching online for the past 2 days and I’ve tried a lot of things but it still doesn’t work.
What else do you thing I can do about this issue please?
nodejs - server.js
import express from "express";
import cors from "cors";
import server from "http";
import { Server } from "socket.io";
import pkg from "dotenv";
pkg.config();
const app = express();
const serve = server.createServer(app);
const io = new Server(serve, {
cors: {
origin: process.env.CLIENT_SERVER,
methods: ["GET", "POST"],
},
});
const port = process.env.PORT || 5000;
app.get("/", (req, res) => {
res.send("SOCKET SERVER!");
});
app.use(cors());
app.use(express.json());
app.use(express.urlencoded({ extended: true }));
io.on("connection", (socket) => {
console.log("new socket connected");
socket.on("join-room", (roomId, userId, name, mic, cam, screen = false) => {
console.log(`new peer joined : ${name} `, userId);
socket.join(roomId);
socket.broadcast
.to(roomId)
.emit("user-connected", userId, name, mic, cam, screen);
socket.on("disconnect", () => {
console.log("disconnected", userId);
socket.broadcast.to(roomId).emit("user-disconnected", userId);
});
});
socket.on("toggle media", (userId, roomId, type) => {
console.log("toggle media for ", userId);
socket.broadcast.to(roomId).emit("user toggled media", userId, type);
});
});
// Server listen initilized
serve
.listen(port, () => {
console.log(`Listening on the port ${port}`);
})
.on("error", (e) => {
console.error(e);
});
nginx config - /etc/nginx/sites-available/socket.<mydomain>.com
server {
if ($host = socket.<mydomain>.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
}
server{
listen 443 ssl;
access_log /var/log/nginx/socket.<mydomain>.com.log;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass http://127.0.0.1:5000;
}
}
I also created a symbolic link at /etc/nginx/sites-enabled/socket.<mydomain>.com
/etc/nginx/nginx.conf
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
# multi_accept on;
}
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
When I run curl http://127.0.0.1:5000 and http://<myserverip>:5000 it shows the right output based on the nodejs app
After all of these, https://socket.<mydomain>.com still shows me the nginx welcome page on browser and with curl.
Please what do you think the problem is?
]]>I was working with a remote API and I exported my API key so that I could have it available as an environment variable for my current shell session, as follows:
- export API_KEY=MY_API_KEY_HERE
However my workflow requires that at some point this env variable should be deleted, but I don’t want to log out and log in back again each time.
What is a good way to delete an environment variable?
Thanks!
]]>Note: The interactive terminal in this tutorial is currently disabled as we work on improving our interactive learning experiences. You can still use this tutorial to learn about the command line and practice Linux commands, but you will need to use the terminal on your computer or a virtual machine.
Today, many of us are familiar with computers (desktops and laptops), smartphones, and tablets which have graphical user interfaces (also referred to as GUIs), allowing us to navigate apps, the web, and our files (like documents and photos) through a visual experience. The Windows, macOS, and Linux operating systems each present varieties of a desktop environment (with images of folders and files, for example), and dropdown menus, all of which provide access to computer programs, applications, and our own media.
Although GUIs can be an intuitive way to use a computer for many users, they often do not provide us with the greatest power over our machines, and they may prevent us from having full administrative access on our computers, including installing, modifying, or deleting software or files. Additionally, as GUIs are largely visual, they are often not as accessible as they could be for all users.
One way of navigating both your own personal computer and remote cloud servers without a GUI is through a text-based terminal or command-line interface (CLI).
Terminal interfaces exist on almost every computer operating system, and terminal emulators are also available as apps for tablets and smartphones. Terminals provide users with greater overall access to their machines through increased administrator access, greater ability to customize environments, and opportunities to automate processes. They also provide users with the ability to access remote computers, such as cloud servers.
This tutorial will provide users who are new to terminal environments with the basics of using a command-line interface through an embedded web terminal in your browser, which you can launch below. If you already have some familiarity with terminals, you may prefer to go through our Introduction to the Linux Terminal tutorial instead. Once you complete this tutorial, you should have an understanding of how to use a terminal on a Linux (or macOS) computer or server.
When you first get access to a new computer or smartphone, you likely want to turn it on and get a feel for how to use it by checking which apps are available, and to learn where things are so that you can customize the device to suit your needs. You can become familiar with a computer through a terminal in a similar way.
The interactive terminal you launched in this browser window, by clicking the Launch an Interactive Terminal!
button above, displays a white rectangle on the bottom of your browser window:
If you have not launched the terminal, please do so now using the button at the beginning of this tutorial.
In your interactive browser terminal, there should be a dollar sign, $
and a blinking cursor. This is where you will begin to type commands to tell the terminal what to do.
The terminal you have launched is an Ubuntu 20.04 terminal. Ubuntu is a popular distribution of Linux, which was originally based on the Unix operating system. The macOS operating system is also based on Unix. If you are reading this tutorial on a Linux or macOS machine, you should have a terminal on your operating system that works similarly to the embedded terminal we’ll be using in this guide.
In many of these Unix (or *nix-based) operating systems, the symbols at the end of the prompt may be a $
symbol or a #
symbol, which mean the following:
$
or dollar sign — you are logged in as a regular user#
or hashtag/pound symbol — you are logged in as a user with elevated privilegesThe user that is noted in the #
environment is also known as a root user, which is considered to be a super user, or administrator, by default.
For our purposes within the browser terminal below, you are logged in as a regular user, but you also have administrator privileges via the sudo
command. As this is a temporary terminal, you do not need to worry about what you type into the terminal, as we will destroy everything once we are done. Similarly, with a cloud server, it is possible to destroy a server and start fresh if something goes awry.
Please note that it is best to exercise more care when working on a local computer’s terminal as there may be changes you can make as an administrator on the terminal that can make permanent changes on the computer you are using.
At this point, with your terminal launched in the browser, you can begin to type into it using your local computer. Your text will appear at the blinking cursor. We’ll learn about what we can type here in the next sections.
We’ll begin working with the terminal by typing a command. A command is an instruction that is given by a user, communicating what it is that the user wants the computer to do. You will be typing your commands into the terminal and then pressing ENTER
or RETURN
when you are ready for the computer to execute on a given command.
Let’s type the following command followed by ENTER
. You can also copy the command, or ask it to run in a launched interactive terminal by clicking on the relevant links in the code block below when you hover over it with a mouse.
- pwd
Once you run this command, you’ll receive the following output:
Output/home/sammy
The pwd
command stands for “present working directory,” and it lets you know where you are within the current filesystem.
In this example, you are in the directory (or folder) called /home/sammy
, which stands for the user called sammy
. If you are logged in as root
, a user with elevated privileges, then the directory would be called /root
. On a personal computer, this directory may be called the name of the user who owns the computer. Sammy Shark’s computer may have /sammy
or /sammy-shark
or /home/sammy
as their primary user directory.
Right now, this directory is empty. Let’s create a directory to store the files we’ll be creating as we go through this tutorial, which we can call files
, for example.
To do this, we’ll use the mkdir
command, which stands for “make directory.” After we type the command, we’ll need to write the name of the folder, which will pass the value to the command so that the command can execute on creating this directory. This value (the name of the folder) is known as an argument, which is an input being given to the command. If you are familiar with natural language grammar, you can think of the argument as an object that is being acted upon by the verb of the command.
In order to create a new directory called files
we’ll write the following, with mkdir
being the command and files
being the argument:
- mkdir files
After you run this command, you won’t receive any output other than a new line with a blinking cursor. With this fresh line on your terminal, you are ready for your next command.
As we have not received any concrete feedback about our new directory yet, we’ll use a command to learn more about what is in our present working directory. You can confirm that the new directory is indeed there by listing out the files in the directory, with the ls
command (signifying “list”):
- ls
You’ll receive output that confirms the files
directory is there:
Outputfiles
This gives us general information about what is in our present working directory. If we want to have more details, we can run the ls
command with what is called a flag. In Linux commands, a flag is written with a hyphen -
and letters, passing additional options (and more arguments) to the command. In our example, we’ll add the -l
flag, which — when paired with ls
— denotes that we would like to use the option to use a long listing format with our command.
Let’s type this command and flag, like so:
- ls -l
Upon pressing ENTER
, we’ll receive the following output in our terminal:
Outputtotal 4
drwxr-xr-x 2 sammy sammy 4096 Nov 13 18:06 files
Here, there are two lines of output. The first line refers to computer memory blocks being allocated to this directory, the second line mostly refers to user permissions on the file.
To get a somewhat more human readable output, we can also pass the -h
or --human-readable
flag, which will print memory sizes in a human readable format, as below. Generally, one hyphen -
refers to single-letter options, and two hyphens --
refer to options that are written out in words. Note that some options can use both formats. We can build multiple options into a command by chaining flags together, as in -lh
.
For example, the two commands below deliver the same results even though they are written differently:
- ls -lh
- ls -l --human-readable
Both of these commands will return the following output, similar to the output above but with greater context of the memory blocks:
Outputtotal 4.0K
drwxr-xr-x 2 sammy sammy 4.0K Nov 13 18:06 files
The first line of output lets us know that 4K of computer memory is dedicated to the folder. The second line of output has many more details, which we’ll go over in more detail. A general high-level reference of all the information that we’ll cover is indicated in the table below.
File type | Permissions | Link count | Owner | Group | File size | Last modified date | File name |
---|---|---|---|---|---|---|---|
d | rwxr-xr-x | 2 | sammy | sammy | 4.0K | Nov 13 18:06 | files |
You’ll note that the name of our directory, files
, is at the end of the second line of output. This name indicates which specific item in the /home/sammy
user directory is being described by the line of output. If we had another file in the directory, we would have another line of output with details on that file.
At the front of the line, there is a list of characters and dashes. Let’s break down the meaning of each of the characters:
Character | Description |
---|---|
d | directory (or folder) — a type of file that can hold other files, useful for organizing a file system; if this were - instead, this would refer to a non-directory file |
r | read — permission to open and read a file, or list the contents of a directory |
w | write — permission to modify the content of a file; and to add, remove, rename files in a directory |
x | execute — permission to run a file that is a program, or to enter and access files within a directory |
In the first drwx
characters of the string, the first letter d
means that the item files
is a directory. If this were a file other than a directory, this string of characters would begin with a hyphen instead, as in -rwx
, where the first hyphen signifies a non-directory file. The following three letters, rwx
, represent the permissions for the owner of the directory files
, and mean that the directory files
can be read, written, and executed by the owner of the file. If any of these characters were replaced with hyphens, that would mean that the owner does not have the type of permission represented by that character. We’ll discuss how to identify the owner of a file in just a moment.
The next three characters in the output are r-x
, which represent the group permissions for the files
directory. In this instance, the group has read and execute permissions, but not write permissions, as the w
is replaced with a -
. We’ll discuss how to identify the group in just a moment.
The final three characters of the first string, r-x
represents the permissions for any other groups that have access to the machine. In this case, these user groups can also read and execute, but not write.
The number 2
in the output refers to the number of links to this file. In Linux, links provide a method to create shortcuts to help users navigate the filesystem. When you created this file, Linux did some background work to create an absolute link to the file, and a self-referential link to the file to allow for users to navigate along a relative path. We’ll discuss absolute and relative paths in the next section.
After the number 2
, the word sammy
is displayed twice. This part of the output gives information about the owner and group associated with the files
directory. The first instance of sammy
in this line refers to the owner of the directory, whose permissions we saw earlier are rwx
. The sammy
user is the owner as we created the files
directory as the sammy
user and are the current owner of the file. Though the sammy
user is the only user in our current environment, Unix-like operating systems often have more than one user and so it is useful to know which user has ownership of a file.
The second instance of sammy
refers to the group that has access to the files
directory, whose permissions we saw earlier are r-x
. In this case, the group name is the same as the owner username sammy
. In real-world environments, there may be other groups on the operating system that have access to the directory, such as staff
or a username like admin
.
The rest of the details on this output line are the 4.0K
for the memory allocation of the directory on the machine, and the date that the directory was last modified (so far, we have just created it).
With this greater understanding of file systems and permissions, we can move onto navigating the file system on our Linux terminal.
So far, we have learned how to determine where we are in a filesystem, how to make a new directory, how to list out files, and how to determine permissions.
Let’s next learn how to move around the file system. We have made a new directory, but we are still in the main /home/sammy
user directory. In order to move into the /home/sammy/files
directory that we have created, we’ll use the cd
command and pass the name of the directory we want to move into as the argument. The command cd
stands for “change directory,” and we’ll construct it like so:
- cd files
Again, you won’t receive output other than a new line with a blinking cursor, but we can check that we are in the /home/sammy/files
directory with the pwd
command we used earlier:
- pwd
You’ll get the following output, confirming where you are:
Output/home/sammy/files
This validates that you are in the /home/sammy/files
directory of the /home/sammy
user directory. Does this syntax look familiar to you? It may remind you of a website’s URL with its forward slashes, and, indeed, websites are structured on servers within directories, too.
Let’s move to the primary directory of the server. Regardless of where we are in a filesystem, we can always use the command cd /
to move to the primary directory:
- cd /
To confirm that we have moved and learn what is in this directory, let’s run our list command:
- ls
We’ll receive the following output:
Outputbin boot dev etc home lib lib32 lib64 libx32 media mnt opt proc root run s sbin srv sys tmp usr var
There are a lot of files in there! The /
directory is the main directory of a Linux server, referred to as the “root” directory. Note that the root directory is different from the default “root” user. You can think of the /
directory as the major artery of a Linux machine, as it contains all the folders necessary to run the computer. For example, the sys
directory holds the Linux kernel and system information virtual filesystem. If you would like to learn more about each of these directories, you can visit the Linux Foundation documentation.
You’ll also notice that there is a directory we have been in already, the /home
user folder. From the /
directory, we can change directories back into /home
then back into files
, or we can move directly back into that folder by typing the absolute path there with cd
:
- cd /home/sammy/files
Now, if you run pwd
you’ll receive /home/sammy/files
as your output.
A file path is the representation of where a file or directory is located on your computer or server. You can call a path to a file or directory in either a relative or absolute way. A relative path would be when we move to a location relative to our current working directory, like we did when we were already in /home/sammy/
and then moved into files/
. An absolute path is when we call the direct line to a location, as we did above with /home/sammy/files
, showing that we started in the /
directory, called the /home/sammy/
user directory and then the nested files/
directory.
Additionally, Linux leverages dot notation to help users navigate via relative paths. A single .
stands for the directory you are currently in, and a double ..
stands for the parent directory. So, from where we currently are (/home/sammy/files
), we can use two dots to return to the parent /home/sammy
user directory, like so:
- cd ..
If you run pwd
, you’ll receive /home/sammy
as your output, and if you run ls
, you’ll receive files
as your output.
Another important symbol to be familiar with is ~
which stands for the home directory of your machine. Here, our home directory is called /home/sammy
for the sammy user, but on a local machine it may be your own name as in sammy-shark/
.
You can type the following from anywhere on your machine and return to this home directory:
- cd ~
At this point, feel free to navigate around your file system with the commands you have learned so far. In the next section, we’ll begin working with text files.
Now that we have a foundation in the Linux file system and how to get around it, let’s start creating new files and learn about how to manipulate text on the command line.
Let’s first be sure that we’re in the files/
directory of the /home/sammy
user folder, which we can do by either verifying with pwd
, or by changing directories on the absolute path:
- cd /home/sammy/files
Now, we’ll create a new text file. We’ll be making a .txt
file, which is a standard file that can be read across operating systems. Unlike .doc
files, a .txt
file is composed of unformatted text. Unformatted text, including the text in.txt
files, can readily be used on the command line, and therefore can be used when working with textual data programmatically (as in, to automate text analysis, to pull information from text, and more).
We’ll begin by using the touch
command, which can create a new file or modify an existing file. To use it, you can use the command touch
and pass the name of the text file you want to create as the argument, as demonstrated below.
- touch ocean.txt
Once you press ENTER
, you’ll receive a new line of the command prompt, and you can list the current contents of files/
to ensure it was created.
- ls
Outputocean.txt
So far we have created an ocean.txt
file which contains no text at the time of creation.
If we want to create a text file that is initialized with text, we can use the echo
command, which is used to display strings of text in Linux.
We can use echo
directly on the command line to have the interface repeat after us. The traditional first program, "Hello, World!"
, can be written with echo
like so:
- echo Hello, World!
OutputHello, World!
Named for Echo of Ovid’s Metamorphosis, the echo
command returns back what we request. In this case, it echoed, “Hello, World!” On its own, however, the echo
command does not allow us to store the value of our text into a text file. In order to do that, we will need to type the following:
- echo "Sammy the Shark" > sammy.txt
The above command uses echo
, then the text we would like to add to our file in quotes, then the redirection operator >
, and finally the name of our new text file, sammy.txt
.
We can check that our new file exists, again with ls
.
- ls
Outputocean.txt sammy.txt
We now have two text files in our /home/sammy/files
user folder. Next, we can confirm that the file sammy.txt
does have the text we asked the terminal to echo into it. We can do that with the cat
command. Short for concatenate, the cat
command is very useful for working with files. Among its functions is showing the contents of a file.
- cat sammy.txt
Once we run the command, we’ll receive the following output:
OutputSammy the Shark
If we were to run cat
on the empty file ocean.txt
, we would receive nothing in return as there is no text in that file. We can add text to this existing file with echo
as well. Let’s add a quote from Zora Neale Hurston to the file.
- echo "Some people could look at a mud puddle and see an ocean with ships." > ocean.txt
Now, if we run cat
on the file, we’ll receive output of the text we just entered.
- cat ocean.txt
OutputSome people could look at a mud puddle and see an ocean with ships.
So far, we have created text files and have added text to these files, but we have not yet modified these files. If we would like to do that, we can use a command-line text editor. Several popular choices exist, including Vim and Emacs. For our purposes, we’ll use nano, which is a less complex CLI text editor program that we can use to begin our exploration.
The nano text editor can be summoned with the nano
command. If we want to edit our existing sammy.txt
file, we can do so by passing the file name as an argument.
- nano sammy.txt
The file will open up on your terminal:
Sammy the Shark
With your keyboard’s arrow keys, move your cursor to the end of the line and begin typing a few lines from the perspective of Sammy.
Note: On the command line, you can’t use your mouse or other pointer to navigate, both through the file system and within files. You’ll need to use your keyboard, and your arrow keys in particular, to move around textual files.
When you’re done with your file, it may read something like this:
Sammy the Shark
Hello, I am Sammy.
I am studying computer science.
Nice to meet you!
With your file now containing the text you would like, we can now save and close the file. You may notice that there is some guidance at the bottom of your terminal window:
^G Get Help ^O WriteOut ^R Read File ^Y Prev Page ^K Cut Text ^C Cur Pos
^X Exit ^J Justify ^W Where Is ^V Next Page ^U UnCut Text ^T To Spell
Because we are currently done with working on this file, we would like to Exit
the file. Here, the ^
symbol refers to the Control
or CTRL
key on your keyboard, and the output above tells us that we need to combine that key with X
(use this lower case, without pressing the SHIFT
key) in order to leave the file. Let’s press those two keys together:
CTRL x
The above is often written inline as CTRL + X
or Ctrl+x
in technical documentation.
At this point, you’ll receive the following prompt:
OutputSave modified buffer?
Y Yes
N No ^C Cancel
In order to save it, we’ll press the letter y
for yes:
y
You’ll receive feedback like the following.
OutputFile Name to Write: sammy.txt
There are additional options, including cancelling with CTRL + C
, but if you are comfortable with closing the file, you can press ENTER
at this point to save the file and exit it.
Let’s say that we want to make a few files of students at DigitalOcean University. Let’s create a new directory in files/
called students
:
- mkdir students
Next, let’s move sammy.txt
into the new students/
directory. The mv
command, which stands for move, will allow us to change the location of a file. The command is constructed by taking the file we want to move as the first argument, and the new location as the second argument. Both of the following executions will produce the same result.
- mv sammy.txt students
- mv sammy.txt students/sammy.txt
This latter option would be useful if we would like to change the name of the file, as in mv sammy.txt students/sammy-the-shark.txt
.
Now, if we run the ls
command, we’ll see that only ocean.txt
and the students/
directory are in our current directory (files/
). Let’s move into the students/
folder.
- cd students
In order to have a template for the other students, we can copy the sammy.txt
file to create more files. To do this, we can use the cp
command, which stands for copy. This command works similarly to the mv
command, taking the original file as the first argument, and the new file as the second argument. We’ll make a file for Alex the Leafy Seadragon:
- cp sammy.txt alex.txt
Now, we can open alex.txt
and inspect it.
- nano alex.txt
So far, alex.txt
looks identical to sammy.txt
. By replacing some of the words, we can modify this file to read like the following. Note that you can use CTRL + K
to remove an entire line.
Alex the Leafy Seadragon
Hello, I am Alex.
I am studying oceanography.
Nice to meet you!
You can save and close the file by pressing CTRL + X
then y
then ENTER
.
If you would like to get more practice with text files, consider creating files for Jamie the Mantis Shrimp, Jesse the Octopus, Drew the Squid, or Taylor the Yellowfin Tuna.
Once you feel comfortable with creating, editing, copying, and moving text files, we can move onto the next section.
Many versions of the command line, including the interactive terminal embedded in this tutorial, allow you to autocomplete and to reuse commands as you go. This supports you moving more quickly as it saves you typing time.
Try typing cat
along with the first few letters of one of the text files you have been working on — for example, cat sa
. Before you finish typing the whole file name of sammy.txt
, press the TAB
key instead. This should autocomplete the full file name, so that your terminal prompt displays the following:
- cat sammy.txt
Now, if you press ENTER
, the terminal should return the contents of the file to the command line.
Another shortcut is to press the UP
arrow key, which will let you cycle through the most recent commands you have run. On a new line with a blinking cursor, press the UP
arrow key a few times to have quick access to your previous commands.
If you need to replicate all the commands you have done in your terminal, you can also summon the entire history of this session with the aptly named history
command:
- history
Depending on how much you have practiced, you should receive 30 or more lines of commands, starting with the following output:
Output 1 pwd
2 mkdir files
3 ls
4 ls -l
...
Familiarizing yourself with these shortcuts will support you as you become more proficient with the command line interface.
One of the most exciting aspects of working on a command line interface connected to the internet is that you have access to all of the resources on the web, and can act on them in an automated way. With the terminal, you can also directly access cloud servers that you have credentials for, manage and orchestrate cloud infrastructure, build your own web apps, and more. For now, as we have already learned how to work with text files on the terminal, we’ll go over how to pull down a text file from the web so that the machine we are using has that text file available to us.
Let’s move back into the files/
directory:
- cd /home/sammy/files
From here, we’ll use the curl
command to transfer data from the web to our personal interactive terminal on the browser. The command curl
stands for client URL (web address).
We have uploaded a short passage from Jules Verne’s Twenty Thousand Leagues Under the Seas on a cloud server. We’ll pass the URL of that file to the curl
command, as demonstrated below.
- curl https://assets.digitalocean.com/articles/command-line-intro/verne_twenty-thousand-leagues.txt
Once we press ENTER
, we’ll receive the text of the passage as output to our terminal (excerpted below)
Output"You like the sea, Captain?"
"Yes; I love it! The sea is everything. It covers seven tenths of the terrestrial globe.
...
"Captain Nemo," said I to my host, who had just thrown himself on one of the divans, "this
is a library which would do honor to more than one of the continental palaces, and I am
absolutely astounded when I consider that it can follow you to the bottom of the seas."
While it’s interesting to have the text display on our terminal window, we do not have the file available to us, we have only transferred the data but have not stored it. (You can verify that the file is not there by running ls
).
In order to save the text to a file, we’ll need to run curl
with the -O
flag, which enables us to output the text to a file, taking the same name of the remote file for our local copy.
- curl -O https://assets.digitalocean.com/articles/command-line-intro/verne_twenty-thousand-leagues.txt
You’ll receive feedback from the terminal that your file has downloaded.
Output % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2671 100 2671 0 0 68487 0 --:--:-- --:--:-- --:--:-- 68487
If you would like to use a specific and alternate name of the file, you could do so with the -o
flag and pass the name of the new file as an argument (in this case, jules.txt
).
- curl -o jules.txt https://assets.digitalocean.com/articles/command-line-intro/verne_twenty-thousand-leagues.txt
We can now work on this file exactly as we would any other text file. Try using cat
, or editing it with nano
.
In the next section, we’ll clean up some of the files and exit our terminal.
As with any other computer, we sometimes need to remove files and folders that are no longer relevant, and exit the program we are using.
Let’s say that the students we know from DigitalOcean University have graduated and we need to clean up their files and the relevant folder. Ensure you are in the students/
directory:
- cd /home/sammy/files/students
If you run ls
, your folder may have a few files, like so:
Outputalex.txt drew.txt jamie.txt jesse.txt sammy.txt taylor.txt
We can remove individual files with the rm
command, which stands for remove. We’ll need to pass the file we want to remove as the argument.
Warning: Note that once you remove a file, it cannot be undone. Be sure that you want to remove the file before pressing ENTER
.
- rm sammy.txt
Now, if we run ls
, we’ll notice that sammy.txt
is no longer in the folder:
Outputalex.txt drew.txt jamie.txt jesse.txt taylor.txt
While we now know we can remove individual files with rm
, it is not very time efficient if we want to remove the entire students/
directory and all of its contents.
The command that is used to remove directories is called rmdir
, which stands for remove directory. Let’s move to the parent folder of files
so that we can work with the students/
directory from there (we would not be able to delete a folder we are presently in).
- cd ..
From the /home/sammy/
user directory, we can run rmdir
on students
.
- rmdir students
However, this does not work, as we receive the following feedback:
Outputrmdir: failed to remove 'students': Directory not empty
The command did not work as rmdir
only works on empty directories and the students
directory still has files in it. (Here, you can create a new, empty folder, and try rmdir
on it. Empty folders can be removed with rmdir
.)
To remove the directory with files still inside, we’ll need to try a different option. In computer science, recursion is commonly used to iteratively self-reference; so we can call both a primary item and all its dependencies. Using the rm
command, we can recursively remove the primary students
directory and all of its content dependencies. We’ll use the -r
flag, which stands for recursive, and pass the folder students
as the argument.
- rm -r students
At this point, if we run ls
, we’ll notice that students/
is no longer in our present directory, and none of the files it held are available either, as they have all been deleted.
When you are done with a terminal session, and especially when you are working on a remote server, you can exit the terminal with the exit
command. Once you feel comfortable with what you have achieved in this session (as you won’t be able to restore it), you can type the following, followed by ENTER
to leave the terminal.
- exit
On our interactive terminal, we’ll receive the following output, confirming that our session has ended.
OutputSession ended
With this session complete, you can refresh this page and then launch a new terminal to try out alternate commands, or create a new file system to explore.
Congratulations! You now know your way around the terminal interface, and are well on your way to doing more with computers and servers.
To continue your learning, you can take a guided pathway on setting up and managing remote servers with our Introduction to Cloud Computing curriculum.
]]>In this tutorial, you’ll use the curl
command to download a text file from a web server. You’ll view its contents, save it locally, and tell curl
to follow redirects if files have moved.
Downloading files off of the Internet can be dangerous, so be sure you are downloading from reputable sources. In this tutorial you’ll download files from DigitalOcean, and you won’t be executing any files you download.
Out of the box, without any command-line arguments, the curl
command will fetch a file and display its contents to the standard output.
Let’s give it a try by downloading the robots.txt
file from Digitalocean.com:
- curl https://www.digitalocean.com/robots.txt
You’ll see the file’s contents displayed on the screen:
OutputUser-agent: *
Disallow:
sitemap: https://www.digitalocean.com/sitemap.xml
sitemap: https://www.digitalocean.com/community/main_sitemap.xml.gz
sitemap: https://www.digitalocean.com/community/questions_sitemap.xml.gz
sitemap: https://www.digitalocean.com/community/users_sitemap.xml.gz
Give curl
a URL and it will fetch the resource and display its contents.
Fetching a file and display its contents is all well and good, but what if you want to actually save the file to your system?
To save the remote file to your local system, with the same filename as the server you’re downloading from, add the --remote-name
argument, or use the -O
option:
- curl -O https://www.digitalocean.com/robots.txt
Your file will download:
Output % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 286 0 286 0 0 5296 0 --:--:-- --:--:-- --:--:-- 5296
Instead of displaying the contents of the file, curl
displays a text-based progress meter and saves the file to the same name as the remote file’s name. You can check on things with the cat
command:
- cat robots.txt
The file contains the same contents you saw previously:
OutputUser-agent: *
Disallow:
sitemap: https://www.digitalocean.com/sitemap.xml
sitemap: https://www.digitalocean.com/community/main_sitemap.xml.gz
sitemap: https://www.digitalocean.com/community/questions_sitemap.xml.gz
sitemap: https://www.digitalocean.com/community/users_sitemap.xml.gz
Now let’s look at specifying a filename for the downloaded file.
You may already have a local file with the same name as the file on the remote server.
To avoid overwriting your local file of the same name, use the -o
or --output
argument, followed by the name of the local file you’d like to save the contents to.
Execute the following command to download the remote robots.txt
file to the locally named do-bots.txt
file:
- curl -o do-bots.txt https://www.digitalocean.com/robots.txt
Once again you’ll see the progress bar:
Output % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 286 0 286 0 0 6975 0 --:--:-- --:--:-- --:--:-- 7150
Now use the cat
command to display the contents of do-bots.txt
to verify it’s the file you downloaded:
- cat do-bots.txt
The contents are the same:
OutputUser-agent: *
Disallow:
sitemap: https://www.digitalocean.com/sitemap.xml
sitemap: https://www.digitalocean.com/community/main_sitemap.xml.gz
sitemap: https://www.digitalocean.com/community/questions_sitemap.xml.gz
sitemap: https://www.digitalocean.com/community/users_sitemap.xml.gz
By default, curl
doesn’t follow redirects, so when files move, you might not get what you expect. Let’s look at how to fix that.
Thus far all of the examples have included fully qualified URLs that include the https://
protocol. If you happened to try to fetch the robots.txt
file and only specified www.digitalocean.com
, you would not see any output, because DigitalOcean redirects requests from http://
to https://
:
You can verify this by using the -I
flag, which displays the request headers rather than the contents of the file:
- curl -I www.digitalocean.com/robots.txt
The output shows that the URL was redirected. The first line of the output tells you that it was moved, and the Location
line tells you where:
OutputHTTP/1.1 301 Moved Permanently
Cache-Control: max-age=3600
Cf-Ray: 65dd51678fd93ff7-YYZ
Cf-Request-Id: 0a9e3134b500003ff72b9d0000000001
Connection: keep-alive
Date: Fri, 11 Jun 2021 19:41:37 GMT
Expires: Fri, 11 Jun 2021 20:41:37 GMT
Location: https://www.digitalocean.com/robots.txt
Server: cloudflare
. . .
You could use curl
to make another request manually, or you can use the --location
or -L
argument which tells curl
to redo the request to the new location whenever it encounters a redirect. Give it a try:
- curl -L www.digitalocean.com/robots.txt
This time you see the output, as curl
followed the redirect:
OutputUser-agent: *
Disallow:
sitemap: https://www.digitalocean.com/sitemap.xml
sitemap: https://www.digitalocean.com/community/main_sitemap.xml.gz
sitemap: https://www.digitalocean.com/community/questions_sitemap.xml.gz
sitemap: https://www.digitalocean.com/community/users_sitemap.xml.gz
You can combine the -L
argument with some of the aforementioned arguments to download the file to your local system:
- curl -L -o do-bots.txt www.digitalocean.com/robots.txt
Warning: Many resources online will ask you to use curl
to download scripts and execute them. Before you run any scripts you have downloaded, it’s good practice to check their contents before making them executable and running them. Use the less
command to review the code to ensure it’s something you want to run.
curl
lets you quickly download files from a remote system. curl
supports many different protocols and can also make more complex web requests, including interacting with remote APIs to send and receive data.
You can learn more by viewing the manual page for curl
by running man curl
.
Starting and creating my docker container with these commands:
do i need to do something like -p 25:25? or -p 465:465?
What could be the problem? I already asked the support to open up the SMTP Ports for me, they did it. But it still won’t work.
]]>It would be very helpful ig you could help me out in a step by step manner on how to do it. Thanks!
]]>This is the second mini tutorial about Associative Arrays in Bash. You can check the previous one here:
In this mini-tutorial, we’ll be talking about how to access elements in an associative array. We’ll be accessing them individually with for loops.
]]>So recently I was working a project where I had to implement some new features. The project was primarily written in BASH. While I love BASH it’s for simpler tasks, or at least this is what I thought.
What is an Associative array:
In computer science, an associative array, map, symbol table, or dictionary is an abstract data type composed of a collection of (key, value) pairs, such that each possible key appears at most once in the collection.
I’ve used associative arrays in different projects when I had to work with PHP, Python, and other languages but never with BASH, I didn’t even know it was possible.
In this mini-tutorial, I’ll go over exactly that, associative arrays in BASH, how to control, populate it, how to define and use them.
]]>My server is running low on disk space and I think it is it is because of some Docker images.
How can I find the directory where the Docker images are stored at on the host/server itself?
Thanks!
]]>However when created a SAS Data set using SAS Programming Runtime Environment(SPRE) in Viya while logged in as that specific user, the permission being reflected are 0022.
Any one help me understand what could have happened to override the initial set umask 0002 to 0022 ?
Thanks in advance.
]]>I am writing a script that would take some user input but I want to make sure that the string provided by the user is consistent.
Is there a way to transform a string in bash to all lower case or all upper case?
Thanks!
]]>I am just getting started with Kubernetes and the kubectl
command specifically.
As there are so many flags and arguments, is there a way to enable autocompletion just like in git
?
So for example when I type kubectl get no[TAB]
it would autocomplete nodes
.
Thank you!
]]>I know some people have had trouble getting this configuration to work because UFW was blocking routing requests on the GW server, but the results are the same for me even if I disable UFW on both servers. I know the GW and BE are connected in the network because I can ping the BE via its private IP from the GW. Any help with why this isn’t working would be greatly appreciated. I also don’t know how to troubleshoot where in the rerouting process the problem is occurring, so tips in that vein would be useful as well.
]]>Auf einem Linux-Server werden wie auf jedem anderen Computer, mit dem Sie möglicherweise vertraut sind, Anwendungen ausgeführt. Auf dem Computer werden diese als „Prozesse“ bezeichnet.
Während Linux die Verwaltung auf niedriger Ebene hinter den Kulissen im Lebenszyklus eines Prozesses übernimmt, benötigen Sie eine Möglichkeit zur Interaktion mit dem Betriebssystem, um es von einer höheren Ebene aus zu verwalten.
In diesem Leitfaden werden wir einige einfache Aspekte der Prozessverwaltung erörtern. Linux bietet eine reichliche Sammlung von Tools für diesen Zweck.
Wir werden diese Ideen auf einem Ubuntu 12.04 VPS untersuchen, aber jede moderne Linux-Distribution funktioniert auf ähnliche Weise.
Der einfachste Weg, um herauszufinden, welche Prozesse auf Ihrem Server ausgeführt werden, besteht darin, den Befehl top
auszuführen:
top***
top - 15:14:40 bis 46 min, 1 Benutzer, Lastdurchschnitt: 0,00, 0,01, 0,05 Aufgaben: 56 insgesamt, 1 laufend, 55 inaktiv, 0 gestoppt, 0 Zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 1019600k gesamt, 316576k gebraucht, 703024k frei, 7652k Puffer Swap: 0k insgesamt, 0k verwendet, 0k frei, 258976k zwischengespeichert PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1 root 20 0 24188 2120 1300 S 0.0 0.2 0:00.56 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:00.07 ksoftirqd/0 6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 7 root RT 0 0 0 0 S 0.0 0.0 0:00.03 watchdog/0 8 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 cpuset 9 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper 10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
Der oberste Informationsblock enthält Systemstatistiken wie die Systemlast und die Gesamtzahl der Aufgaben.
Sie können leicht erkennen, dass 1 Prozess ausgeführt wird und 55 Prozesse inaktiv sind (auch bekannt als inaktiv/ohne CPU-Ressourcen).
Der untere Teil enthält die laufenden Prozesse und ihre Nutzungsstatistiken.
Eine verbesserte Version von top
namens htop
ist in den Repositorys verfügbar. Installieren Sie sie mit diesem Befehl:
sudo apt-get install htop
Wenn wir den Befehl htop
ausführen, sehen wir, dass es eine benutzerfreundlichere Anzeige gibt:
htop***
Mem[||||||||||| 49/995MB] Durchschnittslast: 0.00 0.03 0.05 CPU[ 0.0%] Aufgaben: 21, 3 thr; 1 laufend Swp[ 0/0MB] Betriebszeit: 00:58:11 PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 1259 root 20 0 25660 1880 1368 R 0.0 0.2 0:00.06 htop 1 root 20 0 24188 2120 1300 S 0.0 0.2 0:00.56 /sbin/init 311 root 20 0 17224 636 440 S 0.0 0.1 0:00.07 upstart-udev-brid 314 root 20 0 21592 1280 760 S 0.0 0.1 0:00.06 /sbin/udevd --dae 389 messagebu 20 0 23808 688 444 S 0.0 0.1 0:00.01 dbus-daemon --sys 407 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.02 rsyslogd -c5 408 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.00 rsyslogd -c5 409 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.00 rsyslogd -c5 406 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.04 rsyslogd -c5 553 root 20 0 15180 400 204 S 0.0 0.0 0:00.01 upstart-socket-br
Sie können hier mehr über die Verwendung von top und htop erfahren.
Sowohl top
als auch htop
bieten eine schöne Benutzeroberfläche, um laufende Prozesse zu sehen, die einem grafischen Aufgabenmanager ähneln.
Diese Tools sind jedoch nicht immer flexibel genug, um alle Szenarien angemessen zu behandeln. Ein leistungsfähiger Befehl namens ps
ist oft die Antwort auf diese Probleme.
Wenn er ohne Argumente aufgerufen wird, kann die Ausgabe etwas fehlerhafter sein:
ps***
PID TTY TIME CMD 1017 pts/0 00:00:00 bash 1262 pts/0 00:00:00 ps
Diese Ausgabe zeigt alle mit dem aktuellen Benutzer und der Terminalsitzung verknüpften Prozesse an. Dies ist sinnvoll, da wir derzeit nur bash
und ps
mit diesem Terminal ausführen.
Um ein vollständigeres Bild der Prozesse auf diesem System zu erhalten, können wir Folgendes ausführen:
ps aux***
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.2 24188 2120 ? Ss 14:28 0:00 /sbin/initroot 2 0.0 0.0 0 0 ? S 14:28 0:00 [kthreadd] root 3 0.0 0.0 0 0 ? S 14:28 0:00 [ksoftirqd/0] root 6 0.0 0.0 0 0 ? S 14:28 0:00 [migration/0] root 7 0.0 0.0 0 0 ? S 14:28 0:00 [watchdog/0] root 8 0.0 0.0 0 0 ? S< 14:28 0:00 [cpuset] root 9 0.0 0.0 0 0 ? S< 14:28 0:00 [khelper] . . .
Diese Optionen weisen ps
an, Prozesse, die allen Benutzern gehören (unabhängig von ihrer Terminalzuordnung), in einem benutzerfreundlichen Format anzuzeigen.
Um eine Baumansicht zu sehen, in der hierarchische Beziehungen illustriert werden, können wir den Befehl mit diesen Optionen ausführen:
ps axjf***
PPID PID PGID SID TTY TPGID STAT UID TIME COMMAND 0 2 0 0 ? -1 S 0 0:00 [kthreadd] 2 3 0 0 ? -1 S 0 0:00 \_ [ksoftirqd/0] 2 6 0 0 ? -1 S 0 0:00 \_ [migration/0] 2 7 0 0 ? -1 S 0 0:00 \_ [watchdog/0] 2 8 0 0 ? -1 S< 0 0:00 \_ [cpuset] 2 9 0 0 ? -1 S< 0 0:00 \_ [khelper] 2 10 0 0 ? -1 S 0 0:00 \_ [kdevtmpfs] 2 11 0 0 ? -1 S< 0 0:00 \_ [netns] . . .
Wie Sie sehen können, wird der Prozess kthreadd
als übergeordnetes Element des Prozesses ksoftirqd/0
und der anderen Prozesse angezeigt.
In Linux- und Unix-ähnlichen Systemen wird jedem Prozess einer Prozess-ID oder PID zugewiesen. So identifiziert und verfolgt das Betriebssystem Prozesse.
Eine schnelle Möglichkeit zum Abrufen der PID eines Prozesses ist mit dem Befehl pgrep
:
pgrep bash***
1017
Dadurch wird die Prozess-ID einfach abfragt und zurückgegeben.
Der erste beim Booten erzeugte Prozess namens init erhält die PID „1“.
pgrep init***
1
Dieser Prozess ist dann dafür verantwortlich, jeden anderen Prozess auf dem System zu erzeugen. Die späteren Prozesse erhalten größere PID-Nummern.
Das übergeordnete Element eines Prozesses ist der Prozess, der für das Ablegen verantwortlich war. Übergeordnete Prozesse verfügen über eine PPID, die Sie in den Spaltenüberschriften vieler Prozessverwaltungsanwendungen sehen können, einschließlich top
, htop
und ps
.
Jede Kommunikation zwischen dem Benutzer und dem Betriebssystem über Prozesse umfasst die Übersetzung zwischen Prozessnamen und PIDs zu einem bestimmten Zeitpunkt während des Vorgangs. Aus diesem Grund teilen Dienstprogramme Ihnen die PID mit.
Das Erstellen eines untergeordneten Prozesses erfolgt in zwei Schritten: fork(), das einen neuen Adressraum erstellt und die Ressourcen des übergeordneten Elements per Copy-on-Write kopiert, um dem untergeordneten Prozess zur Verfügung zu stehen; und exec(), das eine ausführbare Datei in den Adressraum lädt und ausführt.
Für den Fall, dass ein untergeordneter Prozess vor seinem übergeordneten Prozess beendet wird, wird der untergeordnete Prozess zu einem Zombie, bis der übergeordnete Prozess Informationen darüber gesammelt oder dem Kernel angezeigt hat, dass er diese Informationen nicht benötigt. Die Ressourcen aus dem untergeordneten Prozess werden dann freigegeben. Wenn der übergeordnete Prozess jedoch vor dem untergeordneten Prozess beendet wird, wird der untergeordnete Prozess von init übernommen, obwohl er auch einem anderen Prozess neu zugewiesen werden kann.
Alle Prozesse in Linux reagieren auf Signale. Signale sind eine Methode auf Betriebssystemebene, mit der Programme angewiesen werden, ihr Verhalten zu beenden oder zu ändern.
Die häufigste Art, Signale an ein Programm weiterzuleiten, ist mit dem Befehl kill
.
Wie Sie möglicherweise erwarten, besteht die Standardfunktion dieses Dienstprogramms darin, zu versuchen, einen Prozess zu beenden:
<pre>kill <span class=“highlight”>PID_of_target_process</span></pre>
Dadurch wird das TERM-Signal an den Prozess gesendet. Das TERM-Signal weist den Prozess an, zu beenden. Dadurch kann das Programm Reinigungsvorgänge durchführen und reibungslos beenden.
Wenn sich das Programm schlecht verhält und bei Erhalt des TERM-Signals nicht beendet wird, können wir das Signal durch Weiterleiten des KILL
-Signals eskalieren:
<pre>kill -KILL <span class=“highlight”>PID_of_target_process</span></pre>
Dies ist ein spezielles Signal, das nicht an das Programm gesendet wird.
Stattdessen wird es dem Betriebssystem-Kernel übergeben, der den Prozess herunterschaltet. Dies wird verwendet, um Programme zu umgehen, die die an sie gesendeten Signale ignorieren.
Jedem Signal ist eine Nummer zugeordnet, die anstelle des Namens übergeben werden kann. Beispielsweise können Sie „-15“ anstelle von „-TERM“ und „-9“ anstelle von „-KILL“ übergeben.
Signale werden nicht nur zum Herunterfahren von Programmen verwendet. Sie können auch verwendet werden, um andere Aktionen auszuführen.
Beispielsweise werden viele Daemons neu gestartet, wenn sie das HUP
- oder Auflegesignal erhalten. Apache ist ein Programm, das so funktioniert.
<pre>sudo kill -HUP <span class=“highlight”>pid_of_apache</span></pre>
Der obige Befehl führt dazu, dass Apache seine Konfigurationsdatei neu lädt und Inhalte wiederbelebt.
Sie können alle Signale auflisten, die mit kill gesendet werden können, indem Sie Folgendes eingeben:
kill -l***
1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP 6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1 11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM . . .
Obwohl die konventionelle Art des Sendens von Signalen durch die Verwendung von PIDs ist, gibt es auch Methoden, dies mit regulären Prozessnamen zu tun.
Der Befehl pkill
funktioniert fast genau so wie kill
, operiert jedoch stattdessen auf einem Prozessnamen:
pkill -9 ping
Der obige Befehl ist das Äquivalent von:
kill -9 `pgrep ping`
Wenn Sie ein Signal an jede Instanz eines bestimmten Prozesses senden möchten, können Sie den Befehl killall
verwenden:
killall firefox
Der obige Befehl sendet das TERM-Signal an jede Instanz von Firefox, das auf dem Computer ausgeführt wird.
Oft möchten Sie anpassen, welchen Prozessen in einer Serverumgebung Priorität eingeräumt wird.
Einige Prozesse können als geschäftskritisch für Ihre Situation angesehen werden, während andere ausgeführt werden können, wenn Ressourcen übrig bleiben.
Linux kontrolliert die Priorität durch einen Wert namens niceness.
Hohe Prioritätsaufgaben werden als weniger nett angesehen, da sie auch keine Ressourcen teilen. Prozesse mit niedriger Priorität sind dagegen nett, weil sie darauf bestehen, nur minimale Ressourcen zu verbrauchen.
Als wir am Anfang des Artikels top
ausgeführt haben, gab es eine Spalte mit der Bezeichnung „NI“. Dies ist der nette Wert des Prozesses:
top***
Aufgaben: 56 insgesamt, 1 laufend, 55 inaktiv, 0 gestoppt, 0 Zombie Cpu(s): 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 1019600k insgesamt, 324496k verwendet, 695104k frei, 8512k Puffer Swap: 0k insgesamt, 0k verwendet, 0k frei, 264812k zwischengespeichert PID-BENUTZER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1635 root 20 0 17300 1200 920 R 0.3 0.1 0:00.01 top 1 root 20 0 24188 2120 1300 S 0,0 0,2 0:00,56 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:00.11 ksoftirqd/0
Nette Werte können je nach System zwischen „-19/-20“ (höchste Priorität) und „19/20“ (niedrigste Priorität) liegen.
Um ein Programm mit einem bestimmten netten Wert auszuführen, können wir den Befehl nice
verwenden:
<pre>nice -n 15 <span class=“highlight”>command_to_execute</span></pre>
Dies funktioniert nur, wenn ein neues Programm gestartet wird.
Um den netten Wert eines Programms zu ändern, das bereits ausgeführt wird, verwenden wir ein Tool namens renice
:
<pre>renice 0 <span class=“highlight”>PID_to_prioritize</span></pre>
Hinweis: Während nice zwangsläufig mit einem Befehlsnamen funktioniert, ruft renice die Prozess-PID auf
Die Prozessverwaltung ist ein Thema, das für neue Benutzer manchmal schwer zu verstehen ist, da sich die verwendeten Tools von ihren grafischen Gegenstücken unterscheiden.
Die Ideen sind jedoch vertraut und intuitiv und werden mit ein wenig Übung zur Gewohnheit. Da Prozesse an allem beteiligt sind, was Sie mit einem Computersystem tun, ist es eine wesentliche Fähigkeit, zu lernen, wie man sie effektiv steuert.
<div class=“author”>Von Justin Ellingwood</div>
]]>Can anyone suggest how I can find the CPU hog(s)? These results suggest that some quick to run user space program are running and terminating before a normal ‘top’ can display them.
=============
top - 15:51:49 up 110 days, 23:23, 1 user, load average: 1.30, 1.36, 1.33 Tasks: 126 total, 2 running, 85 sleeping, 0 stopped, 0 zombie %Cpu(s): 83.7 us, 16.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 1008828 total, 137280 free, 291796 used, 579752 buff/cache KiB Swap: 0 total, 0 free, 0 used. 410896 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 18643 root 20 0 13732 2036 1292 S 0.7 0.2 531:27.70 /bin/bash /home/noisegraphs/bin/noise_graphs_daemon.sh -a 7 root 20 0 0 0 0 S 0.3 0.0 102:40.03 [ksoftirqd/0] 8 root 20 0 0 0 0 I 0.3 0.0 167:17.85 [rcu_sched] 21067 root 20 0 0 0 0 I 0.3 0.0 0:00.16 [kworker/u2:0] 1 root 20 0 225692 5684 2968 S 0.0 0.6 24:04.54 /lib/systemd/systemd --system --deserialize 35 2 root 20 0 0 0 0 S 0.0 0.0 0:00.61 [kthreadd] 4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 [kworker/0:0H]
]]>Сервер Linux, как и любой другой компьютер, использует приложения. Компьютер рассматривает эти приложения как процессы.
Хотя Linux автоматически выполняет все скрытые низкоуровневые задачи жизненного цикла процесса, нам необходим способ взаимодействия с операционной системой для управления на более высоком уровне.
В этом учебном модуле мы расскажем о некоторых простых аспектах управления процессами. Linux предоставляет широкий выбор инструментов для этой цели.
В качестве примера мы используем Ubuntu 12.04 VPS, но любые современные дистрибутивы Linux будут работать аналогичным образом.
Чтобы посмотреть, какие процессы запущены на вашем сервере, нужно запустить команду top
:
top***
top - 15:14:40 up 46 min, 1 user, load average: 0.00, 0.01, 0.05 Tasks: 56 total, 1 running, 55 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 1019600k total, 316576k used, 703024k free, 7652k buffers Swap: 0k total, 0k used, 0k free, 258976k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1 root 20 0 24188 2120 1300 S 0.0 0.2 0:00.56 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:00.07 ksoftirqd/0 6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 7 root RT 0 0 0 0 S 0.0 0.0 0:00.03 watchdog/0 8 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 cpuset 9 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper 10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
В верхней части подборки приведена статистика по системе, в том числе сведения о нагрузке и общем количестве задач.
Вы можете легко увидеть, что в системе запущен 1 процесс, а 55 процессов находятся в режиме сна (т. е. не активны/не используют ресурсы ЦП).
В нижней части отображаются запущенные процессы и статистика их использования.
В репозиториях доступна улучшенная версия top
, которая называется htop
. Установите ее с помощью следующей команды:
sudo apt-get install htop
Если мы запустим команду htop
, мы увидим отображение информации в более удобном формате:
htop***
Mem[||||||||||| 49/995MB] Load average: 0.00 0.03 0.05 CPU[ 0.0%] Tasks: 21, 3 thr; 1 running Swp[ 0/0MB] Uptime: 00:58:11 PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 1259 root 20 0 25660 1880 1368 R 0.0 0.2 0:00.06 htop 1 root 20 0 24188 2120 1300 S 0.0 0.2 0:00.56 /sbin/init 311 root 20 0 17224 636 440 S 0.0 0.1 0:00.07 upstart-udev-brid 314 root 20 0 21592 1280 760 S 0.0 0.1 0:00.06 /sbin/udevd --dae 389 messagebu 20 0 23808 688 444 S 0.0 0.1 0:00.01 dbus-daemon --sys 407 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.02 rsyslogd -c5 408 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.00 rsyslogd -c5 409 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.00 rsyslogd -c5 406 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.04 rsyslogd -c5 553 root 20 0 15180 400 204 S 0.0 0.0 0:00.01 upstart-socket-br
Вы можете узнать больше об использовании top и htop здесь.
И top
, и htop
предоставляют удобный интерфейс для просмотра работающих процессов, похожий на графический диспетчер задач.
Однако эти инструменты не всегда достаточно гибкие, чтобы охватывать все сценарии. Решить эту проблему может помочь мощная команда ps
.
При вызове без аргументов вывод может быть довольно сжатым:
ps***
PID TTY TIME CMD 1017 pts/0 00:00:00 bash 1262 pts/0 00:00:00 ps
Вывод показывает все процессы, связанные с текущим пользователем и текущим сеансом терминала. Это имеет смысл, потому что мы запускаем на этом терминале только bash
и ps
.
Чтобы получить более полное представление о процессах в данной системе, мы можем использовать следующую команду:
ps aux***
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.2 24188 2120 ? Ss 14:28 0:00 /sbin/init root 2 0.0 0.0 0 0 ? S 14:28 0:00 [kthreadd] root 3 0.0 0.0 0 0 ? S 14:28 0:00 [ksoftirqd/0] root 6 0.0 0.0 0 0 ? S 14:28 0:00 [migration/0] root 7 0.0 0.0 0 0 ? S 14:28 0:00 [watchdog/0] root 8 0.0 0.0 0 0 ? S< 14:28 0:00 [cpuset] root 9 0.0 0.0 0 0 ? S< 14:28 0:00 [khelper] . . .
Эти опции предписывают ps
показать процессы, принадлежащие всем пользователям (вне зависимости от привязки терминала) в удобном формате.
Чтобы посмотреть представление дерева с иллюстрацией иерархических отношений, данную команду можно запустить с этими опциями:
ps axjf***
PPID PID PGID SID TTY TPGID STAT UID TIME COMMAND 0 2 0 0 ? -1 S 0 0:00 [kthreadd] 2 3 0 0 ? -1 S 0 0:00 \_ [ksoftirqd/0] 2 6 0 0 ? -1 S 0 0:00 \_ [migration/0] 2 7 0 0 ? -1 S 0 0:00 \_ [watchdog/0] 2 8 0 0 ? -1 S< 0 0:00 \_ [cpuset] 2 9 0 0 ? -1 S< 0 0:00 \_ [khelper] 2 10 0 0 ? -1 S 0 0:00 \_ [kdevtmpfs] 2 11 0 0 ? -1 S< 0 0:00 \_ [netns] . . .
Как видите, процесс kthreadd
отображается как родитель процесса ksoftirqd/0
и других процессов.
В системах Linux и Unix каждому процессу назначается идентификатор процесса или PID. Операционная система использует их для идентификации и отслеживания процессов.
Чтобы быстро узнать PID процесса, вы можете использовать команду pgrep
:
pgrep bash***
1017
Эта команда просто запросит идентификатор процесса и выведет его.
Процессу init, который создается первым при загрузке, присваивается PID “1”.
pgrep init***
1
Этот процесс отвечает за создание всех остальных процессов в системе. Последующим процессам присваиваются большие номера PID.
Родитель процесса — это процесс, который отвечает за его создание. Родительские процессы имеют идентификатор PPID, который можно увидеть в заголовках столбцов многих приложений для управления процессами, включая top
, htop
и ps
.
Любое взаимодействие между пользователем и операционной системой, связанное с процессами, включает взаимное преобразование имен процессов и PID. Именно поэтому утилиты сообщают вам PID.
Создание дочернего процесса осуществляется в два этапа: fork() создает новое адресное пространство и копирует в него ресурсы, принадлежащие родительскому процессу, с помощью copy-on-write; а exec() загружает исполняемый блок в адресное пространство и выполняет его.
Если дочерний процесс завершается раньше родительского, он остается бесхозным, пока родитель не получит информацию о нем или не сообщит ядру, что эта информация не требуется. В этом случае ресурсы дочернего процесса освободятся. Если родительский процесс завершается раньше дочернего, дочерний процесс привязывается к процессу init, хотя его можно переназначить другому процессу.
Все процессы Linux реагируют на сигналы. Операционная система использует сигналы, чтобы отправить программам команду остановиться или изменить поведение.
Наиболее распространенный способ передачи сигналов в программу — использовать команду kill
.
Как вы можете догадаться, по умолчанию эта утилита пытается уничтожить процесс:
<pre>kill <span class=“highlight”>PID_of_target_process</span></pre>
Она отправляет процессору сигнал TERM. Сигнал TERM просит процесс остановиться. Это позволяет программе выполнить операции по очистке и нормально завершить работу.
Если программа работает неправильно и не завершает работу после получения сигнала TERM, мы можем отправить сигнал более высокого уровня — KILL
:
<pre>kill -KILL <span class=“highlight”>PID_of_target_process</span></pre>
Это специальный сигнал, который не отправляется программе.
Вместо этого он передается в ядро операционной системы, которое отключает процесс. Он используется, чтобы обходить программы, игнорирующие отправляемые им сигналы.
Каждому сигналу присвоено число, которое можно передать вместо имени. Например, вы можете передать “-15” вместо “-TERM” и “-9” вместо “-KILL”.
Сигналы используются не только для отключения программ. Их также можно использовать для выполнения других действий.
Например, многие демоны перезапускаются при получении сигнала HUP
или прекращения работы. Например, так работает Apache.
<pre>sudo kill -HUP <span class=“highlight”>pid_of_apache</span></pre>
Получив вышеуказанную команду, Apache перезагрузит файл конфигурации и возобновит вывод контента.
Вы можете вывести список сигналов, которые можно отправлять с помощью kill, используя следующую команду:
kill -l***
1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP 6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1 11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM . . .
Хотя обычно при отправке сигналов используются PID, существуют способы использовать для этой же цели обычные имена процессов.
Команда pkill
работает практически точно так же как и kill
, но использует имя процесса:
pkill -9 ping
Вышеуказанная команда эквивалентна команде:
kill -9 `pgrep ping`
Если вы хотите отправить сигнал каждому экземпляру определенного процесса, вы можете использовать команду killall
:
killall firefox
Приведенная выше команда отправит сигнал TERM всем экземплярам firefox, запущенным на этом компьютере.
Часто бывает необходимо изменить приоритет процессов в серверной среде.
Некоторые процессоры могут быть важными, а другие могут выполняться на излишках ресурсов.
Linux контролирует приоритеты с помощью значения вежливости.
Приоритетные задачи считаются менее вежливыми, потому что они вообще не делятся ресурсами. Процессы с низким приоритетом считаются более вежливыми, потому что они используют минимум ресурсов.
Когда мы запускали команду top
в начале этого учебного модуля, мы видели столбец “NI”. В этом столбце отображается значение вежливости процесса:
top***
Tasks: 56 total, 1 running, 55 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 1019600k total, 324496k used, 695104k free, 8512k buffers Swap: 0k total, 0k used, 0k free, 264812k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1635 root 20 0 17300 1200 920 R 0.3 0.1 0:00.01 top 1 root 20 0 24188 2120 1300 S 0.0 0.2 0:00.56 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:00.11 ksoftirqd/0
В зависимости от системы, значения вежливости могут различаться от “-19/-20” (наибольший приоритет) до “19/20” (наименьший приоритет).
Чтобы запустить программу с определенным значением вежливости, мы можем использовать команду nice
:
<pre>nice -n 15 <span class=“highlight”>command_to_execute</span></pre>
Это работает только в начале новой программы.
Чтобы изменить значение вежливости уже выполняемой программы, мы используем инструмент renice
:
<pre>renice 0 <span class=“highlight”>PID_to_prioritize</span></pre>
Примечание. Хотя nice по необходимости использует имя команды, renice вызывает PID процесса.
Управление процессами — это тема, которая иногда бывает сложной для новых пользователей, потому что используемые для этой цели инструменты отличаются от аналогичных инструментов с графическим интерфейсом.
Однако все эти идеи знакомы, интуитивно понятны и станут привычными после небольшой практики. Поскольку процессы используются в компьютерных системах повсеместно, умение эффективно управлять ими — критически важный навык.
<div class=“author”>Джастин Эллингвуд</div>
]]>Um servidor Linux, assim como qualquer outro computador que você conheça, executa aplicativos. Para o computador, eles são considerados “processos”.
Embora o Linux lide nos bastidores com o gerenciamento de baixo nível no ciclo de vida de um processo, você precisará de uma maneira de interagir com o sistema operacional para gerenciá-lo de um nível superior.
Neste guia, vamos discutir alguns aspectos simples do gerenciamento de processos. O Linux oferece uma coleção abundante de ferramentas para esse propósito.
Vamos explorar essas ideias em um VPS Ubuntu 12.04, mas qualquer distribuição moderna do Linux funcionará de maneira similar.
A maneira mais fácil de descobrir quais processos estão sendo executados no seu servidor é executando o comando top
:
top***
top - 15:14:40 up 46 min, 1 user, load average: 0.00, 0.01, 0.05 Tasks: 56 total, 1 running, 55 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 1019600k total, 316576k used, 703024k free, 7652k buffers Swap: 0k total, 0k used, 0k free, 258976k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1 root 20 0 24188 2120 1300 S 0.0 0.2 0:00.56 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:00.07 ksoftirqd/0 6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 7 root RT 0 0 0 0 S 0.0 0.0 0:00.03 watchdog/0 8 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 cpuset 9 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper 10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
A porção superior de informações mostra estatísticas do sistema, como a carga do sistema e o número total de tarefas.
É possível ver facilmente que há 1 processo em execução e 55 processos estão suspensos (ou seja, ociosos/sem utilizar recursos da CPU).
A parte inferior mostra os processos em execução e suas estatísticas de uso.
Uma versão melhorada de top
, chamada htop
, está disponível nos repositórios. Instale-o com este comando:
sudo apt-get install htop
Se executarmos o comando htop
, veremos que as informação são exibidas de uma maneira mais inteligível:
htop***
Mem[||||||||||| 49/995MB] Load average: 0.00 0.03 0.05 CPU[ 0.0%] Tasks: 21, 3 thr; 1 running Swp[ 0/0MB] Uptime: 00:58:11 PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 1259 root 20 0 25660 1880 1368 R 0.0 0.2 0:00.06 htop 1 root 20 0 24188 2120 1300 S 0.0 0.2 0:00.56 /sbin/init 311 root 20 0 17224 636 440 S 0.0 0.1 0:00.07 upstart-udev-brid 314 root 20 0 21592 1280 760 S 0.0 0.1 0:00.06 /sbin/udevd --dae 389 messagebu 20 0 23808 688 444 S 0.0 0.1 0:00.01 dbus-daemon --sys 407 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.02 rsyslogd -c5 408 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.00 rsyslogd -c5 409 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.00 rsyslogd -c5 406 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.04 rsyslogd -c5 553 root 20 0 15180 400 204 S 0.0 0.0 0:00.01 upstart-socket-br
Aprenda mais sobre como usar o top e htop aqui.
Tanto o top
quanto o htop
fornecem uma interface agradável para visualizar processos em execução, de maneira semelhante a um gerenciador de tarefas gráfico.
No entanto, essas ferramentas nem sempre são flexíveis o suficiente para abranger adequadamente todos os cenários. Um comando poderoso chamado ps
é geralmente a resposta para esses problemas.
Quando chamado sem argumentos, o resultado pode ser um pouco confuso:
ps***
PID TTY TIME CMD 1017 pts/0 00:00:00 bash 1262 pts/0 00:00:00 ps
Esse resultado mostra todos os processos associados ao usuário e sessão de terminal atuais. Isso faz sentido porque estamos executando apenas o bash
e ps
com esse terminal atualmente.
Para conseguirmos uma visão mais completa dos processos neste sistema, podemos executar o seguinte:
ps aux***
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.2 24188 2120 ? Ss 14:28 0:00 /sbin/init root 2 0.0 0.0 0 0 ? S 14:28 0:00 [kthreadd] root 3 0.0 0.0 0 0 ? S 14:28 0:00 [ksoftirqd/0] root 6 0.0 0.0 0 0 ? S 14:28 0:00 [migration/0] root 7 0.0 0.0 0 0 ? S 14:28 0:00 [watchdog/0] root 8 0.0 0.0 0 0 ? S< 14:28 0:00 [cpuset] root 9 0.0 0.0 0 0 ? S< 14:28 0:00 [khelper] . . .
Essas opções dizem ao ps
para mostrar processos de propriedade de todos os usuários (independentemente da sua associação de terminais) em um formato facilmente inteligível.
Para ver um modo de exibição em árvore, onde relações hierárquicas são mostradas, executamos o comando com essas opções:
ps axjf***
PPID PID PGID SID TTY TPGID STAT UID TIME COMMAND 0 2 0 0 ? -1 S 0 0:00 [kthreadd] 2 3 0 0 ? -1 S 0 0:00 \_ [ksoftirqd/0] 2 6 0 0 ? -1 S 0 0:00 \_ [migration/0] 2 7 0 0 ? -1 S 0 0:00 \_ [watchdog/0] 2 8 0 0 ? -1 S< 0 0:00 \_ [cpuset] 2 9 0 0 ? -1 S< 0 0:00 \_ [khelper] 2 10 0 0 ? -1 S 0 0:00 \_ [kdevtmpfs] 2 11 0 0 ? -1 S< 0 0:00 \_ [netns] . . .
Como se vê, o processo kthreadd
é mostrado como um pai do processo ksoftirqd/0
e de outros.
No Linux e em sistemas do tipo Unix, cada processo é atribuído a um ID de processo, ou PID. É assim que o sistema operacional identifica e mantém o controle dos processos.
Uma maneira rápida de obter o PID de um processo é com o comando pgrep
:
pgrep bash***
1017
Isso irá simplesmente consultar o ID do processo e retorná-lo.
Ao primeiro processo gerado na inicialização do sistema, chamado init, é atribuído o PID de “1".
pgrep init***
1
Esse processo é então responsável por gerar todos os processos no sistema. Os processos posteriores recebem números PID maiores.
O pai de um processo é o processo que foi responsável por gerá-lo. Os processos pais possuem um PPID, que pode ser visto no cabeçalho da coluna correspondente em muitos aplicativos de gerenciamento de processos, incluindo o top
, htop
e ps
.
Qualquer comunicação entre o usuário e o sistema operacional em relação aos processos envolve a tradução entre os nomes de processos e PIDs em algum ponto durante a operação. É por isso que os utilitários informam o PID.
A criação de um processo filho acontece em dois passos: fork(), que cria espaços de endereço e copia os recursos de propriedade do pai via copia em gravação para que fique disponível ao processo filho; e exec(), que carrega um executável no espaço de endereço e o executa.
Caso um processo filho morra antes do seu pai, o filho torna-se um zumbi até que o pai tenha coletado informações sobre ele ou indicado ao kernel que ele não precisa dessas informações. Os recursos do processo filho serão então libertados. No entanto, se o processo pai morrer antes do filho, o filho será adotado pelo init, embora também possa ser reatribuído a outro processo.
Todos os processos no Linux respondem a sinais. Os sinais são uma maneira ao nível de SO de dizer aos programas para terminarem ou modificarem seu comportamento.
A maneira mais comum de passar sinais para um programa é com o comando kill
.
Como se espera, a funcionalidade padrão desse utilitário é tentar encerrar um processo:
<pre>kill <span class=“highlight”>PID_of_target_process</span></pre>
Isso envia o sinal TERM ao processo. O sinal TERM pede ao processo para encerrar. Isso permite que o programa execute operações de limpeza e seja finalizado sem problemas.
Se o programa estiver se comportando incorretamente e não se encerrar quando um sinal TERM for passado, podemos escalar o sinal passando o sinal KILL
:
<pre>kill -KILL <span class=“highlight”>PID_of_target_process</span></pre>
Esse é um sinal especial que não é enviado ao programa.
Ao invés disso, ele é dado ao kernel do sistema operacional, que encerra o processo. Isso é usado para passar por cima de programas que ignoram os sinais que lhe são enviados.
Cada sinal possui um número associado que pode ser passado ao invés de seu nome. Por exemplo, é possível passar “-15" ao invés de ”-TERM” e ”-9" ao invés de ”-KILL”.
Os sinais não são usados apenas para encerrar programas. Eles também podem ser usados para realizar outras ações.
Por exemplo, muitos daemons são reiniciados quando recebem o HUP
, ou sinal de desligamento. O Apache é um programa que funciona dessa forma.
<pre>sudo kill -HUP <span class=“highlight”>pid_of_apache</span></pre>
O comando acima fará com que o Apache recarregue seu arquivo de configuração e retome o serviço de conteúdo.
Liste todos os sinais que podem ser enviados com o kill digitando:
kill -l***
1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP 6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1 11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM . . .
Embora a maneira convencional de enviar sinais seja através do uso de PIDs, também existem métodos para fazer isso com os nomes regulares dos processos.
O comando pkill
funciona de maneira bastante similar ao kill
, com a diferença de operar com um nome de processo:
pkill -9 ping
O comando acima é equivalente a:
kill -9 `pgrep ping`
Se quiser enviar um sinal para todas as instâncias de um determinado processo, utilize o comando killall
:
killall firefox
O comando acima enviará o sinal TERM para todas as instâncias do firefox em execução no computador.
Geralmente, é desejável ajustar quais processos recebem prioridade em um ambiente de servidor.
Alguns processos podem ser considerados críticos para sua situação, enquanto outros podem ser executados sempre que houver recursos sobrando no sistema.
O Linux controla a prioridade através de um valor chamado niceness (gentileza).
As tarefas de alta prioridade são consideradas menos nice (gentis), porque não compartilham recursos tão bem. Por outro lado, os processos de baixa prioridade são nice, porque insistem em utilizar apenas a menor quantidade de recursos.
Quando executamos o top
no início do artigo, havia uma coluna marcada “NI”. Esse é o valor de gentileza do processo:
top***
Tasks: 56 total, 1 running, 55 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 1019600k total, 324496k used, 695104k free, 8512k buffers Swap: 0k total, 0k used, 0k free, 264812k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1635 root 20 0 17300 1200 920 R 0.3 0.1 0:00.01 top 1 root 20 0 24188 2120 1300 S 0.0 0.2 0:00.56 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:00.11 ksoftirqd/0
Os valores de gentileza podem variar entre “-19/-20” (prioridade mais alta) e ”19/20” (prioridade mais baixa), dependendo do sistema.
Para executar um programa com um certo valor de gentileza, podemos usar o comando nice
:
<pre>nice -n 15 <span class=“highlight”>command_to_execute</span></pre>
Isso funciona apenas ao iniciar um novo programa.
Para alterar o valor de gentileza de um programa que já está sendo executado, usamos uma ferramenta chamada renice
:
<pre>renice 0 <span class=“highlight”>PID_to_prioritize</span></pre>
Nota: embora o comando nice opere necessariamente com um nome de comando, o renice opera chamando o PID do processo
O gerenciamento de processos é um tópico que pode ser de difícil compreensão para novos usuários, pois as ferramentas usadas são diferentes de seus equivalentes gráficos.
No entanto, as ideias são habituais e intuitivas e, com um pouco de prática, se tornarão naturais. Como os processos estão envolvidos em tudo o que é feito em um sistema de computador, aprender como controlá-los de maneira eficaz é uma habilidade essencial.
<div class=“author”>Por Justin Ellingwood</div>
]]>grep
est l’une des commandes les plus pratiques dans un environnement de terminaux Linux. L’accronyme grep
signifie « global regular expression print » (rechercher globalement les correspondances avec l’expression régulière). Cela signifie que vous pouvez utiliser grep
pour voir si l’entrée qu’il reçoit correspond à un modèle spécifié. Apparemment trivial, ce programme est extrêmement puissant. Sa capacité à trier les entrées en fonction de règles complexes fait qu’il est couramment utilisé dans de nombreuses chaînes de commande.
Au cours de ce tutoriel, vous allez explorer les options de la commande grep
. Ensuite, vous allez approfondir vos connaissances dans l’utilisation des expressions régulières qui vous permettront d’effectuer des recherches plus avancées.
[interactive_terminal bash]
Au cours de ce tutoriel, vous allez utiliser grep
pour rechercher plusieurs mots et plusieurs phrases dans GNU General Public License version 3.
Si vous utilisez un système Ubuntu, vous pouvez trouver le fichier dans le dossier /usr/share/common-licenses
. Copiez-le dans votre répertoire home :
- cp /usr/share/common-licenses/GPL-3 .
Si vous êtes sur un autre système, utilisez la commande curl
pour en télécharger une copie :
- curl -o GPL-3 https://www.gnu.org/licenses/gpl-3.0.txt
Vous allez également utiliser le fichier de licence BSD de ce tutoriel. Sous Linux, vous pouvez utiliser la commande suivante pour le copier dans votre répertoire home :
- cp /usr/share/common-licenses/BSD .
Si vous êtes sur un autre système, créez le fichier en utilisant la commande suivante :
- cat << 'EOF' > BSD
- Copyright (c) The Regents of the University of California.
- All rights reserved.
-
- Redistribution and use in source and binary forms, with or without
- modification, are permitted provided that the following conditions
- are met:
- 1. Redistributions of source code must retain the above copyright
- notice, this list of conditions and the following disclaimer.
- 2. Redistributions in binary form must reproduce the above copyright
- notice, this list of conditions and the following disclaimer in the
- documentation and/or other materials provided with the distribution.
- 3. Neither the name of the University nor the names of its contributors
- may be used to endorse or promote products derived from this software
- without specific prior written permission.
-
- THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
- ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
- FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- SUCH DAMAGE.
- EOF
Maintenant que vous avez les fichiers, vous pouvez commencer à travailler avec grep
.
Dans sa forme la plus basique, vous pouvez utiliser grep
pour trouver les correspondances avec des modèles littéraux dans un fichier texte. Cela signifie que si vous passez une commande grep
pour rechercher un mot, le système imprimera chaque ligne du fichier qui contient le mot en question.
Exécutez la commande suivante pour utiliser grep
et trouver ainsi toutes les lignes qui contiennent le mot GNU
:
- grep "GNU" GPL-3
Le premier argument, GNU
, est le modèle que vous recherchez alors que le second, GPL-3
, est le fichier saisi dans lequel vous souhaitez faire votre recherche.
La sortie affichera chacune des lignes qui contient le texte modèle :
Output GNU GENERAL PUBLIC LICENSE
The GNU General Public License is a free, copyleft license for
the GNU General Public License is intended to guarantee your freedom to
GNU General Public License for most of our software; it applies also to
Developers that use the GNU GPL protect your rights with two steps:
"This License" refers to version 3 of the GNU General Public License.
13. Use with the GNU Affero General Public License.
under version 3 of the GNU Affero General Public License into a single
...
...
Sur certains systèmes, le modèle que vous recherchez sera mis en surbrillance dans la sortie.
Par défaut, grep
recherchera le modèle exact spécifié dans le fichier d’entrée et renverra les lignes qu’il trouvera. Vous pouvez rendre ce comportement plus pratique en ajoutant quelques balises facultatives à grep
.
Si vous souhaitez que grep
ignore la « case » de votre paramètre de recherche et que vous recherchez des variations à la fois en majuscule et en minuscule, vous pouvez spécifier l’option -i
ou --ignore-case
.
Recherchez chaque instance du mot license
(en majuscule, minuscule ou les deux) dans le même fichier qu’auparavant avec la commande suivante :
- grep -i "license" GPL-3
Les résultats incluent : LICENSE
, license
et License
:
Output GNU GENERAL PUBLIC LICENSE
of this license document, but changing it is not allowed.
The GNU General Public License is a free, copyleft license for
The licenses for most software and other practical works are designed
the GNU General Public License is intended to guarantee your freedom to
GNU General Public License for most of our software; it applies also to
price. Our General Public Licenses are designed to make sure that you
(1) assert copyright on the software, and (2) offer you this License
"This License" refers to version 3 of the GNU General Public License.
"The Program" refers to any copyrightable work licensed under this
...
...
S’il y avait une instance avec LiCeNsE
, elle aurait également été renvoyée.
Si vous souhaitez trouver toutes les lignes qui ne contiennent pas le modèle spécifié, vous pouvez utiliser l’option -v
ou --invert-match
.
Recherchez chaque ligne qui ne contient pas le mot the
dans la licence BSD en exécutant la commande suivante :
- grep -v "the" BSD
Vous verrez la sortie suivante :
OutputAll rights reserved.
Redistribution and use in source and binary forms, with or without
are met:
may be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
...
...
Étant donné que vous n’avez pas précisé l’option « ignorer case », les deux derniers éléments ont été renvoyés comme ne comportant pas le mot the
.
Il vous sera souvent utile de connaître le numéro de la ligne sur laquelle les correspondances sont trouvées. Pour se faire, vous pouvez utiliser l’option -n
ou --line-number
. Ré-exécutez l’exemple précédent en ajoutant la balise précédente :
- grep -vn "the" BSD
Vous verrez le texte suivant :
Output2:All rights reserved.
3:
4:Redistribution and use in source and binary forms, with or without
6:are met:
13: may be used to endorse or promote products derived from this software
14: without specific prior written permission.
15:
16:THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
17:ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
...
...
Vous pouvez maintenant référencer le numéro des lignes dans le cas où vous souhaiteriez modifier chacune des lignes qui ne contient pas the
. Cela est particulièrement pratique lorsque vous travaillez avec un code source.
Au cours de l’introduction, vous avez appris que grep
signifiait « rechercher globalement les correspondances avec l’expression rationnelle ». Une « expression régulière » est une chaîne de texte qui décrit un modèle de recherche spécifique.
La manière dont les diverses applications et langages de programmation implémentent les expressions régulières est légèrement différente. Au cours de ce tutoriel, vous allez uniquement aborder un petit sous-ensemble de la façon dont grep
décrit ses modèles.
Dans les exemples précédents de ce tutoriel, lorsque vous avez recherché les mots GNU
et the
, en réalité, vous avez recherché les expressions régulières de base qui correspondaient à la chaîne exacte de caractères GNU
et the
. Les modèles qui spécifient exactement les caractères à mettre en correspondance se nomment « literals » car ils correspondent littéralement au modèle, character-for-character.
Il est plus facile de les considérer comme une correspondance à une chaîne de caractères plutôt qu’à un mot. La distinction deviendra plus importante à mesure que les modèles que vous aborderez seront plus complexes.
La correspondance se fera sur tous les caractères alphabétiques et numériques (ainsi que certains autres caractères) littéralement à moins qu’un autre mécanisme d’expression ne vienne la modifier.
Les ancres sont des caractères spéciaux qui spécifient à quel endroit de la ligne une correspondance est considérée comme valable.
Par exemple, en utilisant des ancres, vous pouvez spécifier que vous souhaitez uniquement obtenir les lignes qui comportent GNU
tout au début de la ligne. Pour cela, vous pouvez utiliser la balise ^
avant la chaîne littérale.
Exécutez la commande suivante pour rechercher le fichier GPL-3
et trouver les lignes où GNU
se trouve au tout début d’une ligne :
- grep "^GNU" GPL-3
Vous verrez apparaître les deux lignes suivantes :
OutputGNU General Public License for most of our software; it applies also to
GNU General Public License, you may choose any version ever published
De la même façon, vous devez utiliser l’ancre $
à la fin d’un modèle pour indiquer que la correspondance ne sera valable que si elle se trouve à la fin d’une ligne.
Cette commande permettra de faire la correspondance avec chaque ligne se terminant par le mot and
dans le fichier GPL-3
:
- grep "and$" GPL-3
Vous verrez la sortie suivante :
Outputthat there is no warranty for this free software. For both users' and
The precise terms and conditions for copying, distribution and
License. Each licensee is addressed as "you". "Licensees" and
receive it, in any medium, provided that you conspicuously and
alternative is allowed only occasionally and noncommercially, and
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
provisionally, unless and until the copyright holder explicitly and
receives a license from the original licensors, to run, modify and
make, use, sell, offer for sale, import and otherwise run, modify and
Dans les expressions régulières, le caractère du point (.) signifie que tout caractère individuel peut exister à l’endroit spécifié.
Par exemple, pour trouver toute correspondance dans le fichier GPL-3
qui contient deux caractères et puis la chaine cept
, vous devez utiliser le modèle suivant :
- grep "..cept" GPL-3
Vous verrez la sortie suivante :
Outputuse, which is precisely where it is most unacceptable. Therefore, we
infringement under applicable copyright law, except executing it on a
tells the user that there is no warranty for the work (except to the
License by making exceptions from one or more of its conditions.
form of a separately written license, or stated as exceptions;
You may not propagate or modify a covered work except as expressly
9. Acceptance Not Required for Having Copies.
...
...
Comme vous pouvez le voir, la sortie affiche des instances à la fois de accept
et except
ainsi que des variations des deux mots. Le modèle devrait également trouver les correspondances avec z2cept
s’il y en avait.
En plaçant un groupe de caractères entre parenthèses (\[
et \]
), vous pouvez spécifier que le caractère qui se trouve à cette position peut être l’un des caractères du groupe entre parenthèses.
Par exemple, pour trouver les lignes qui en contiennent too
ou two
, vous devez spécifier ces variations succinctement en utilisant le modèle suivant :
- grep "t[wo]o" GPL-3
La sortie montre que les deux variations existent dans le fichier :
Outputyour programs, too.
freedoms that you received. You must make sure that they, too, receive
Developers that use the GNU GPL protect your rights with two steps:
a computer network, with no transfer of a copy, is not conveying.
System Libraries, or general-purpose tools or generally available free
Corresponding Source from a network server at no charge.
...
...
La notation des parenthèses vous offre quelques options intéressantes. Vous pouvez utiliser un modèle qui trouve toutes les correspondances sauf les caractères entre parenthèses en commençant la liste de caractères entre parenthèses par un ^
Dans cet exemple, nous trouverons le modèle .ode
mais ne fera pas correspondre le modèle code
:
- grep "[^c]ode" GPL-3
Voici la sortie qui s’affichera :
Output 1. Source Code.
model, to give anyone who possesses the object code either (1) a
the only significant mode of use of the product.
notice like this when it starts in an interactive mode:
Notez que dans la seconde ligne renvoyée, il y a, en fait, le mot code
. Il ne s’agit pas d’un échec de l’expression régulière ou de grep. Au contraire, cette ligne a été renvoyée car plus tôt dans la ligne, le système a trouvé le modèle mode
qui se trouve dans le mot model
. La ligne a été renvoyée car il y avait une instance qui correspond au modèle.
Une autre des fonctionnalités utiles des parenthèses est que vous pouvez spécifier une gamme de caractères au lieu de saisir chaque caractère disponible individuellement.
Ainsi, si vous le souhaitez, vous pouvez trouver toutes les lignes qui commencent par une lettre majuscule avec le modèle suivant :
- grep "^[A-Z]" GPL-3
Voici la sortie que vous verrez :
OutputGNU General Public License for most of our software; it applies also to
States should not allow patents to restrict development and use of
License. Each licensee is addressed as "you". "Licensees" and
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
System Libraries, or general-purpose tools or generally available free
Source.
User Product is transferred to the recipient in perpetuity or for a
...
...
À cause de certains des problèmes de tri hérités, il est souvent plus juste d’utiliser les catégories de caractères POSIX à la place des gammes de caractères que vous venez d’utiliser.
Il existe plusieurs caractères qui sont en-dehors du champ de ce guide, mais voici un exemple qui accomplira la même procédure que celle de l’exemple précédent en utilisant la catégorie de caractères \[:upper:\]
dans un sélectionneur de parenthèses :
- grep "^[[:upper:]]" GPL-3
La sortie sera la même qu’auparavant.
Enfin, l’un des méta-caractères les plus couramment utilisés est l’astérisque ou *
, qui signifie « répétez le caractère précédent ou l’expression zero ou more times ».
Pour trouver chaque ligne dans le fichier GPL-3
qui contient des parenthèses d’ouverture et de fermeture, avec seulement des lettres et des espaces uniques entre les deux, utilisez l’expression suivante :
- grep "([A-Za-z ]*)" GPL-3
Vous verrez la sortie suivante :
Output Copyright (C) 2007 Free Software Foundation, Inc.
distribution (with or without modification), making available to the
than the work as a whole, that (a) is included in the normal form of
Component, and (b) serves only to enable use of the work with that
(if any) on which the executable work runs, or a compiler used to
(including a physical distribution medium), accompanied by the
(including a physical distribution medium), accompanied by a
place (gratis or for a charge), and offer equivalent access to the
...
...
Jusqu’à présent, vous avez utilisé les points, les astérisques et d’autres caractères dans vos expressions. Cependant, vous aurez parfois besoin de rechercher ces caractères spécifiques.
Parfois, vous aurez besoin de trouver un point ou une parenthèse d’ouverture en tant que tel, tout particulièrement lorsque vous travaillerez avec un code source ou des fichiers de configuration. Étant donné que ces caractères ont une signification spéciale dans les expressions régulières, vous devez « échapper » ces caractères pour indiquer à grep
que vous ne souhaitez pas utiliser leur signification spéciale dans ce cas.
Vous pouvez éviter des caractères en utilisant un caractère de barre oblique inverse (\
) devant le caractère qui devrait normalement avoir une signification spéciale.
Par exemple, pour trouver toutes les lignes qui commencent par une lettre majuscule et se termine par un point, utilisez l’expression suivante qui évite le point final afin qu’elle représente un point littéral et non pas le sens habituel « any character » :
- grep "^[A-Z].*\.$" GPL-3
Voici la sortie qui s’affichera :
OutputSource.
License by making exceptions from one or more of its conditions.
License would be to refrain entirely from conveying the Program.
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
SUCH DAMAGES.
Also add information on how to contact you by electronic and paper mail.
Prenons maintenant en considération d’autres options d’expressions régulières.
La commande grep
prend en charge un langage d’expression plus vaste en utilisant la balise -E
ou en appelant la commande egrep
au lieu de grep
.
Ces options vous donne la possibilité d’utiliser des « expressions régulières étendues ». Les expressions régulières étendues incluent tous les méta-caractères de base ainsi que les méta-caractères supplémentaires qui permettent d’exprimer des caractères plus complexes.
L’une des capacités les plus utiles qu’offrent les expressions régulières étendues est la possibilité de regrouper des expressions ensemble et de les manipuler ou d’y faire référence en tant qu’une seule unité.
Groupez les expressions ensemble à l’aide de parenthèses. Si vous souhaitez utiliser des parenthèses sans utiliser des expressions régulières étendues, vous pouvez les éviter avec la barre oblique inverse pour activer cette fonctionnalité.
Les trois expressions suivantes se valent en termes de fonctionnalité :
- grep "\(grouping\)" file.txt
- grep -E "(grouping)" file.txt
- egrep "(grouping)" file.txt
Tout comme les expressions entre parenthèses peuvent spécifier différents choix possibles de correspondance avec des caractères uniques, l’alternance vous permet de spécifier des correspondances alternatives pour les chaînes de caractères ou ensembles d’expression.
Pour indiquer une alternance, utilisez le caractère de barre droite |
. Ils sont souvent utilisés dans des regroupements entre parenthèses pour spécifier qu’une sur deux ou plusieurs des possibilités devraient être considérées comme une correspondance.
La commande suivante trouvera soit GPL
ou General Public Licence
dans le texte :
- grep -E "(GPL|General Public License)" GPL-3
Le résultat ressemblera à ce qui suit :
Output The GNU General Public License is a free, copyleft license for
the GNU General Public License is intended to guarantee your freedom to
GNU General Public License for most of our software; it applies also to
price. Our General Public Licenses are designed to make sure that you
Developers that use the GNU GPL protect your rights with two steps:
For the developers' and authors' protection, the GPL clearly explains
authors' sake, the GPL requires that modified versions be marked as
have designed this version of the GPL to prohibit the practice for those
...
...
Avec l’alternance, vous pouvez faire votre sélection entre plus de deux options en ajoutant des choix supplémentaires dans le groupe de sélection séparé par des caractères supplémentaires de barre oblique (|).
Comme le méta-caractère *
qui correspond au caractère précédent ou au caractère défini sur zero ou more times, d’autres méta-caractères disponibles dans des expressions régulières étendues permettent de spécifier le nombre d’événements.
Pour trouver une correspondance de caractère zero ou one times, vous pouvez utiliser le ?
. Cela rend le caractère ou les ensembles de caractères précédents optionnels, par essence.
La commande suivante cherche les correspondances de copyright
et right
en plaçant copy
dans un groupe optionnel :
- grep -E "(copy)?right" GPL-3
Vous verrez la sortie suivante :
Output Copyright (C) 2007 Free Software Foundation, Inc.
To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights. Therefore, you have
know their rights.
Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
"Copyright" also means copyright-like laws that apply to other kinds of
...
Le caractère +
correspond à l’apparition une ou plusieurs fois d’une expression. Il est presque similaire au méta-caractère *
, mais avec le caractère +
, l’expression doit correspondre au moins une fois.
L’expression suivante correspond à la chaîne free
plus un ou plusieurs caractères qui ne sont pas des caractères d’espacement :
- grep -E "free[^[:space:]]+" GPL-3
Vous verrez la sortie suivante :
Output The GNU General Public License is a free, copyleft license for
to take away your freedom to share and change the works. By contrast,
the GNU General Public License is intended to guarantee your freedom to
When we speak of free software, we are referring to freedom, not
have the freedom to distribute copies of free software (and charge for
you modify it: responsibilities to respect the freedom of others.
freedomss that you received. You must make sure that they, too, receive
protecting users' freedom to change the software. The systematic
of the GPL, as needed to protect the freedom of users.
patents cannot be used to render the program non-free.
Pour spécifier le nombre de fois qu’une correspondance est répétée, utilisez les caractères d’accolade ({
et }
). Ces caractères vous permettent de spécifier avec exactitude un numéro, une plage ou une limite supérieure ou inférieure pour le nombre de fois qu’une correspondance à une expression est trouvée.
Utilisez l’expression suivante pour trouver toutes les lignes dans le fichier GPL-3
qui contiennent trois voyelles :
- grep -E "[AEIOUaeiou]{3}" GPL-3
Chaque ligne renvoyée contient un mot avec trois voyelles :
Outputchanged, so that their problems will not be attributed erroneously to
authors of previous versions.
receive it, in any medium, provided that you conspicuously and
give under the previous paragraph, plus a right to possession of the
covered work so as to satisfy simultaneously your obligations under this
Pour trouver la correspondance avec tous les mots qui ont entre 16 et 20 caractères, utilisez l’expression suivante :
- grep -E "[[:alpha:]]{16,20}" GPL-3
Vous verrez la sortie suivante :
Output certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.
c) Prohibiting misrepresentation of the origin of that material, or
Seules les lignes contenant des mots de cette longueur s’afficheront.
grep
est une fonctionnalité pratique pour trouver des modèles dans des fichiers ou la hiérarchie du système de fichiers. Il est donc conseiller de passer un peu de temps pour se familiariser avec ses options et sa syntaxe.
Les expressions régulières sont encore plus versatiles et peuvent être utilisées avec plusieurs programmes populaires. Par exemple, de nombreux éditeurs de texte implémentent des expressions régulières pour rechercher et remplacer du texte.
En outre, la plupart des langages de programmation modernes utilisent des expressions régulières pour exécuter des procédures sur des données spécifiques. Une fois que vous aurez compris les expressions régulières, vous pourrez transférer ces connaissances sur plusieurs tâches informatiques courantes, de la recherche avancée dans votre éditeur de texte à la validation des entrées de l’utilisateur.
]]>Un serveur Linux, tout comme tous les autres ordinateurs que vous connaissez, exécute des applications. L’ordinateur considère cela comme des « processus ».
Bien que Linux se chargera de la gestion de bas niveau et en coulisses du cycle de vie d’un processus, vous avez besoin de pouvoir interagir avec le système d’exploitation et à le gérer ainsi à un niveau supérieur.
Dans ce guide, nous allons voir certains des aspects simples de la gestion de processus. Linux vous propose un vaste ensemble d’outils.
Nous allons voir ces idées sur un Ubuntu 12.04 VPS. Cependant, toute distribution Linux moderne fonctionnera de manière similaire.
Afin de déterminer quels sont les processus en cours d’exécution sur votre serveur, la façon la plus simple consiste à exécuter la commande top
:
top***
top - 15:14:40 up 46 min, 1 user, load average: 0.00, 0.01, 0.05 Tasks: 56 total, 1 running, 55 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 1019600k total, 316576k used, 703024k free, 7652k buffers Swap: 0k total, 0k used, 0k free, 258976k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1 root 20 0 24188 2120 1300 0.0 0.2 0:00.56 init 2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:00.07 ksoftirqd/0 6 root RT 0 0 0 S 0.0 0.0 0:00.00 migration/0 7 root RT 0 0 0 0 S 0.0 0.0 0:00.03 watchdog/0 8 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 cpuset 9 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper 10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
Le premier morceau d’information vous donne les statistiques sur les systèmes, comme la charge du système et le nombre total de tâches.
Vous pouvez facilement voir qu’il y a 1 processus en cours d’exécution et que 55 processus sont dormants (aka idle/n’utilisant pas les ressources du processeur).
La partie inférieure comporte les processus en cours d’exécution et leurs statistiques d’utilisation.
Vous disposez d’une version améliorée de top
, appelée htop
, dans les référentiels. Installez-la avec cette commande :
sudo apt-get install htop
Si nous exécutons la commande htop
, l’affichage qui apparaîtra sera plus convivial :
htop***
Mem[||||||||||| 49/995MB] Load average: 0.00 0.03 0.05 CPU[ 0.0%] Tasks: 21, 3 thr; 1 running Swp[ 0/0 Mo] Uptime: 00:58:11 PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 1259 root 20 0 25660 1880 1368 R 0.0 0.2 0:00.06 htop 1 root 20 0 24188 2120 1300 S 0.0 0.2 0:00.56 /sbin/init 311 root 20 0 17224 636 440 S 0.0 0.1 0:00.07 upstart-udev-brid 314 root 20 0 21592 1280 760 S 0.0 0.1 0:00.06 /sbin/udevd --dae 389 messagebu 20 0 23808 688 444 S 0.0 0.1 0:00.01 dbus-daemon --sys 407 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.02 rsyslogd -c5 408 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.00 rsyslogd -c5 409 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.00 rsyslogd -c5 406 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.04 rsyslogd -c5 553 root 20 0 15180 400 204 S 0.0 0.0 0:00.01 upstart-socket-br
Vous pouvez en savoir plus sur la manière d’utiliser top et htop ici.
top
et htop
vous offrent tous les deux une belle interface pour visualiser les processus en cours d’exécution, similaire à un gestionnaire de tâches graphique.
Cependant, ces outils ne sont pas toujours suffisamment flexibles pour couvrir tous les scénarios correctement. Une commande puissante appelée ps
est souvent utilisée pour répondre à ces problèmes.
Lorsque vous appelez une commande sans arguments, la sortie peut manquer d’éclat :
ps***
PID TTY TIME CMD 1017 pts/0 00:00:00 bash 1262 pts/0 00:00:00 ps
Cette sortie affiche tous les processus associés à l’utilisateur et la session du terminal actuels. Cela est logique, car nous exécutons actuellement bash
et ps
avec ce terminal.
Pour obtenir une image plus complète des processus sur ce système, nous pouvons exécuter ce qui suit :
ps aux***
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.2 24188 2120 ? Ss 14:28 0:00 /sbin/init root 2 0.0 0.0 0 0 ? S 14:28 0:00 [kthreadd] root 3 0.0 0.0 0 0 ? S 14:28 0:00 [ksoftirqd/0] root 6 0.0 0.0 0 0 ? S 14:28 0:00 [migration/0] root 7 0.0 0.0 0 0 ? S 14:28 0:00 [watchdog/0] root 8 0.0 0.0 0 0 ? S< 14:28 0:00 [cpuset] root 9 0.0 0.0 0 0 ? S< 14:28 0:00 [khelper] . . .
Ces options indiquent à ps
d’afficher les processus détenus par tous les utilisateurs (quel que soit le terminal auquel ils sont associés) sous un format convivial pour les utilisateurs.
Pour avoir une vue en arborescencee, dans laquelle les relations hiérarchiques sont illustrées, nous pouvons exécuter la commande avec les options suivantes :
ps axjf***
PPID PID PGID SID TTY TPGID STAT UID TIME COMMAND 0 2 0 0 ? -1 S 0 0:00 [kthreadd] 2 3 0 0 ? -1 S 0 0:00 \_ [ksoftirqd/0] 2 6 0 0 ? -1 S 0 0:00 \_ [migration/0] 2 7 0 0 ? -1 S 0 0:00 \_ [watchdog/0] 2 8 0 0 ? -1 S< 0 0:00 \_ [cpuset] 2 9 0 0 ? -1 S< 0 0:00 \_ [khelper] 2 10 0 0 ? -1 S 0 0:00 \_ [kdevtmpfs] 2 11 0 0 ? -1 S< 0 0:00 \_ [netns] . . .
Comme vous pouvez le voir, le processus kthreadd
est montré comme un parent du processus ksoftirqd/0
et les autres.
Dans les systèmes de type Linux et Unix, chaque processus se voit attribuer un process ID ou** PID**. Voici de quelle manière le système d’exploitation identifie et assure le suivi des processus.
Vous pouvez rapidement obtenir le PID d’un processus en utilisant la commande pgrep
:
pgrep bash***
1017
Cette commande interrogera simplement l’ID du processus et la renverra.
Le premier processus généré au démarrage, appelé init, se voit attribuer le PID de « 1 ».
pgrep init***
1
Ce processus est ensuite chargé de générer tout autre processus sur le système. Les processus suivants se voient attribuer des numéros PID plus grands.
Le parent d’un processus est le processus qui était chargé de le générer. Les processus parent disposent d’un PPID, que vous pouvez voir dans l’en-tête des colonnes dans de nombreuses applications de gestion de processus, dont top
, htop
et ps
.
Toute communication sur les processus entre l’utilisateur et le système d’exploitation implique, à un moment donné de l’opération, la traduction entre les noms de processus et le PID. C’est pour cette raison que les utilitaires vous indiquent le PID.
La création d’un processus enfant se fait en deux étapes : fork(), qui crée un nouvel espace d’adresse et copie les ressources que le parent possède via une copie-sur-écriture pour les rendre disponibles sur le processus enfant ; et exec(), qui charge un exécutable dans l’espace de l’adresse et l’exécute.
Dans le cas où un processus enfant meurt avant son parent, l’enfant devient un zombie jusqu’à que le parent collecte les informations le concernant ou indique au noyau qu’il n’a pas besoin de ces informations Les ressources du processus enfant seront ensuite libérées. Cependant, si le processus parent meurt avant l’enfant, l’enfant sera adopté par l’init, bien qu’il puisse également être réattribué à un autre processus.
Tous les processus de Linux répondent à des signals. Les signaux permettent d’indiquer aux programmes, au niveau du système d’exploitation, de s’arrêter ou de modifier leur comportement.
La façon la plus courante de transmettre des signaux à un programme consiste à utiliser la commande kill
.
Comme vous pouvez vous y attendre, la fonctionnalité par défaut de cet utilitaire consiste à tenter de tuer un processus :
<pre>kill <span class=“highlight”>PID_of_target_process</span></pre>
Cela envoie le signal TERM au processus. Le signal TERM indique au processus de bien vouloir se terminer. Cela permet au programme d’effectuer des opérations de nettoyage et de s’arrêter en douceur.
Si le programme se comporte mal et ne se ferme pas lorsque le signal TERM est actionné, nous pouvons escalader le signal en passant le signal KILL
:
<pre>kill -KILL <span class=“highlight”>PID_of_target_process</span></pre>
Il s’agit d’un signal spécial que n’est pas envoyé au programme.
Au lieu de cela, il est envoyé au noyau du système d’exploitation qui interrompt le processus. Vous pouvez l’utiliser pour contourner les programmes qui ignorent les signaux qui leur sont envoyés.
Chaque signal est associé à un numéro que vous pouvez passer à la place du nom. Par exemple, vous pouvez passer « -15 » au lieu de « -TERM » et « -9 » au lieu de « -KILL ».
Les signaux ne servent pas uniquement à fermer des programmes. Vous pouvez également les utiliser pour effectuer d’autres actions.
Par exemple, de nombreux démons redémarrent lorsqu’un signal HUP
ou de suspension leur est envoyé Apache est un programme qui fonctionne ainsi.
<pre>sudo kill -HUP <span class=“highlight”>pid_of_apache</span></pre>
La commande ci-dessus poussera Apache à recharger son fichier de configuration et à reprendre le contenu d’utilisation.
Vous pouvez répertorier tous les signaux que vous pouvez envoyer avec kill en saisissant :
kill -l***
1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP 6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1 11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM . . .
Bien que la façon classique d’envoyer des signaux consiste à utiliser des PIDS, il existe également des méthodes de le faire avec des noms de processus réguliers.
La commande pkill
fonctionne de manière pratiquement de la même manière que kill
, mais elle fonctionne plutôt avec le nom d’un processus :
pkill -9 ping
La commande ci-dessus est l’équivalent de :
kill -9 `pgrep ping`
Vous pouvez utiliser la commande killall
pour envoyer un signal à chaque instance d’un processus donné :
killall firefox
La commande ci-dessus enverra le signal TERM à chaque instance de firefox en cours d’exécution sur l’ordinateur.
Il vous arrivera souvent de vouloir ajuster la priorité donnée aux processus dans un environnement de serveur.
Certains processus peuvent être considérés comme critiques à la mission pour votre situation, tandis que d’autres peuvent être exécutés chaque fois qu’il y aura des ressources restantes.
Linux contrôle la priorité par le biais d’une valeur appelée niceness.
Les tâches hautement prioritaires sont considérées comme moins nice, car elles ne partagent pas également les ressources. Les processus faiblement prioritaires sont en revanche nice car ils insistent à prendre seulement des ressources minimales.
Lorsque nous avons exécuté top
au début de l’article, il y avait une colonne nommée « NI ». Il s’agit de la valeur nice du processus :
top***
Tasks: 56 total, 1 running, 55 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 1019600k total, 324496k used, 695104k free, 8512k buffers Swap: 0k total, 0k used, 0k free, 264812k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1635 root 20 0 17300 1200 920 R 0.3 0.1 0:00.01 top 1 root 20 0 24188 2120 1300 S 0.0 0.2 0:00.56 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:00.11 ksoftirqd/0
Les valeurs nice peuvent aller de « -19/-20 » (priorité la plus grande) à «19/20» (priorité la plus faible) en fonction du système.
Pour exécuter un programme avec une certaine nice valeur, nous pouvons utiliser la commande nice
:
<pre>nice -n 15 <span class=“highlight”>command_to_execute</span></pre>
Elle fonctionne uniquement au démarrage d’un nouveau programme.
Pour modifier la valeur nice d’un programme déjà en cours d’exécution, nous utilisons un outil
appelé renice :
<pre>renice 0 <span class=“highlight”>PID_to_prioritize</span></pre>
Remarque : bien que nice fonctionne avec un nom de commande par nécessité, renice fonctionne en appelant le PID de processus
La gestion de processus est un sujet parfois difficile à comprendre pour les nouveaux utilisateurs car les outils utilisés sont différents de leurs homologues graphiques.
Cependant, les idées sont familières et intuitives et deviennent naturelles avec un peu de pratique. Étant donné que les processus sont impliqués dans toutes les tâches que vous effectuez avec un système informatique, il est essentiel que vous appreniez à les contrôler efficacement.
<div class=“author”>Par Justin Ellingwood</div>
]]>Un servidor Linux, como cualquier otro equipo con el que pueda estar familiarizado, ejecuta aplicaciones. Para el equipo, estos se consideran “procesos”.
Mientras que Linux se encargará de la administración de bajo nivel, entre bastidores, en el ciclo de vida de un proceso, se necesitará una forma de interactuar con el sistema operativo para administrarlo desde un nivel superior.
En esta guía, explicaremos algunos aspectos sencillos de la administración de procesos. Linux proporciona una amplia colección de herramientas para este propósito.
Exploraremos estas ideas en un VPS de Ubuntu 12.04, pero cualquier distribución moderna de Linux funcionará de manera similar.
La forma más sencilla de averiguar qué procesos se están ejecutando en su servidor es ejecutar el comando top
:
top***
top - 15:14:40 up 46 min, 1 user, load average: 0.00, 0.01, 0.05 Tasks: 56 total, 1 running, 55 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 1019600k total, 316576k used, 703024k free, 7652k buffers Swap: 0k total, 0k used, 0k free, 258976k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1 root 20 0 24188 2120 1300 S 0.0 0.2 0:00.56 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:00.07 ksoftirqd/0 6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 7 root RT 0 0 0 0 S 0.0 0.0 0:00.03 watchdog/0 8 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 cpuset 9 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper 10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
La parte superior de la información ofrece estadísticas del sistema, como la carga del sistema y el número total de tareas.
Se puede ver fácilmente que hay 1 proceso en ejecución y 55 procesos en reposo (es decir, inactivos/que no usan los recursos de la CPU).
La parte inferior tiene los procesos en ejecución y las estadísticas de uso.
Una versión mejorada de top
, llamada htop
, está disponible en los repositorios. Instálelo con este comando:
sudo apt-get install htop
Si ejecutamos el comando htop
, veremos que hay una pantalla más fácil de usar:
htop***
Mem[||||||||||| 49/995MB] Load average: 0.00 0.03 0.05 CPU[ 0.0%] Tasks: 21, 3 thr; 1 running Swp[ 0/0MB] Uptime: 00:58:11 PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 1259 root 20 0 25660 1880 1368 R 0.0 0.2 0:00.06 htop 1 root 20 0 24188 2120 1300 S 0.0 0.2 0:00.56 /sbin/init 311 root 20 0 17224 636 440 S 0.0 0.1 0:00.07 upstart-udev-brid 314 root 20 0 21592 1280 760 S 0.0 0.1 0:00.06 /sbin/udevd --dae 389 messagebu 20 0 23808 688 444 S 0.0 0.1 0:00.01 dbus-daemon --sys 407 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.02 rsyslogd -c5 408 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.00 rsyslogd -c5 409 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.00 rsyslogd -c5 406 syslog 20 0 243M 1404 1080 S 0.0 0.1 0:00.04 rsyslogd -c5 553 root 20 0 15180 400 204 S 0.0 0.0 0:00.01 upstart-socket-br
Puede aprender más sobre cómo usar top y htop aquí.
top
y htop
proporcionan una buena interfaz para ver los procesos en ejecución, similar a la de un administrador de tareas gráfico.
Sin embargo, estas herramientas no siempre son lo suficientemente flexibles para cubrir adecuadamente todos los escenarios. Un poderoso comando llamado ps
generalmente es la respuesta a estos problemas.
Cuando se invoca sin argumentos, el resultado puede ser un poco escaso:
ps***
PID TTY TIME CMD 1017 pts/0 00:00:00 bash 1262 pts/0 00:00:00 ps
Este resultado muestra todos los procesos asociados con el usuario actual y la sesión de terminal. Eso tiene sentido porque, actualmente, solo estamos ejecutando bash
y ps
con este terminal.
Para obtener un panorama más completo de los procesos en este sistema, podemos ejecutar lo siguiente:
ps aux***
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.2 24188 2120 ? Ss 14:28 0:00 /sbin/init root 2 0.0 0.0 0 0 ? S 14:28 0:00 [kthreadd] root 3 0.0 0.0 0 0 ? S 14:28 0:00 [ksoftirqd/0] root 6 0.0 0.0 0 0 ? S 14:28 0:00 [migration/0] root 7 0.0 0.0 0 0 ? S 14:28 0:00 [watchdog/0] root 8 0.0 0.0 0 0 ? S< 14:28 0:00 [cpuset] root 9 0.0 0.0 0 0 ? S< 14:28 0:00 [khelper] . . .
Estas opciones ordenan a ps
que muestre los procesos de propiedad de todos los usuarios (independientemente de su asociación con el terminal) en un formato fácil de usar.
Para ver una vista de estructura jerárquica, en la que se ilustran las relaciones jerárquicas, podemos ejecutar el comando con las siguientes opciones:
ps axjf***
PPID PID PGID SID TTY TPGID STAT UID TIME COMMAND 0 2 0 0 ? -1 S 0 0:00 [kthreadd] 2 3 0 0 ? -1 S 0 0:00 \_ [ksoftirqd/0] 2 6 0 0 ? -1 S 0 0:00 \_ [migration/0] 2 7 0 0 ? -1 S 0 0:00 \_ [watchdog/0] 2 8 0 0 ? -1 S< 0 0:00 \_ [cpuset] 2 9 0 0 ? -1 S< 0 0:00 \_ [khelper] 2 10 0 0 ? -1 S 0 0:00 \_ [kdevtmpfs] 2 11 0 0 ? -1 S< 0 0:00 \_ [netns] . . .
Como puede ver, el proceso kthreadd
se muestra como proceso principal de kstadd/0
y los demás.
En Linux y sistemas tipo Unix, a cada proceso se le asigna un ID de proceso o PID. Esta es la forma en que el sistema operativo identifica y realiza un seguimiento de los procesos.
Una forma rápida de obtener el PID de un proceso es con el comando pgrep
:
pgrep bash***
1017
Esto simplemente consultará el ID del proceso y lo mostrará en el resultado.
El primer proceso que se generó en el arranque, llamado init, recibe el PID “1”.
pgrep init***
1
Entonces, este proceso es responsable de engendrar todos los demás procesos del sistema. Los procesos posteriores reciben números PID mayores.
Un proceso principal es el proceso que se encargó de generarlo. Los procesos principales tienen un PPID, que puede ver en los encabezados de las columnas en muchas aplicaciones de administración de procesos, incluidos top
, htop
y ps
.
Cualquier comunicación entre el usuario y el sistema operativo sobre los procesos implica traducir entre los nombres de procesos y los PID en algún momento durante la operación. Este es el motivo por el que las utilidades le indican el PID.
Para crear un proceso secundario se deben seguir dos pasos: fork(), que crea un nuevo espacio de direcciones y copia los recursos propiedad del principal mediante copy-on-write para que estén disponibles para el proceso secundario; y exec(), que carga un ejecutable en el espacio de direcciones y lo ejecuta.
En caso de que un proceso secundario muera antes que su proceso principal, el proceso secundario se convierte en un zombi hasta que el principal haya recopilado información sobre él o haya indicado al núcleo que no necesita esa información. Luego, los recursos del proceso secundario se liberarán. Sin embargo, si el proceso principal muere antes que el secundario, init adoptará el secundario, aunque también puede reasignarse a otro proceso.
Todos los procesos en Linux responden a señales. Las señales son una forma de decirle a los programas que terminen o modifiquen su comportamiento.
La forma más común de pasar señales a un programa es con el comando kill
.
Como es de esperar, la funcionalidad predeterminada de esta utilidad es intentar matar un proceso:
<pre>kill <span class=“highlight”>PID_of_target_process</span></pre>
Esto envía la señal TERM al proceso. La señal TERM indica al proceso debe terminar. Esto permite que el programa realice operaciones de limpieza y cierre sin problemas.
Si el programa tiene un mal comportamiento y no se cierra cuando se le da la señal TERM, podemos escalar la señal pasando la señal KILL
:
<pre>kill -KILL <span class=“highlight”>PID_of_target_process</span></pre>
Esta es una señal especial que no se envía al programa.
En su lugar, se envía al núcleo del sistema operativo, que cierra el proceso. Eso se utiliza para eludir los programas que ignoran las señales que se les envían.
Cada señal tiene un número asociado que puede pasar en vez del nombre. Por ejemplo, puede pasar “-15” en lugar de “-TERM” y “-9” en lugar de “-KILL”.
Las señales no solo se utilizan para cerrar programas. También pueden usarse para realizar otras acciones.
Por ejemplo, muchos demonios se reinician cuando reciben la señal HUP
o la señal de colgar. Apache es un programa que funciona así.
<pre>sudo kill -HUP <span class=“highlight”>pid_of_apache</span></pre>
El comando anterior hará que Apache vuelva a cargar su archivo de configuración y reanude sirviendo contenidos.
Puede enumerar todas las señales que puede enviar con kill escribiendo lo siguiente:
kill -l***
1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP 6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1 11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM . . .
Aunque la forma convencional de enviar señales es mediante el uso de PID, también hay métodos para hacerlo con nombres de procesos regulares.
El comando pkill
funciona casi exactamente igual que kill
, pero funciona con un nombre de proceso en su lugar:
pkill -9 ping
El comando anterior es el equivalente a:
kill -9 `pgrep ping`
Si quiere enviar una señal a todas las instancias de un determinado proceso, puede utilizar el comando killall
:
killall firefox
El comando anterior enviará la señal TERM a todas las instancias de firefox que se estén ejecutando en el equipo.
A menudo, querrá ajustar qué procesos reciben prioridad en un entorno de servidor.
Algunos procesos pueden considerarse como una misión crítica para su situación, mientras que otros pueden ejecutarse siempre que haya recursos sobrantes.
Linux controla la prioridad a través de un valor llamado niceness.
Las tareas de alta prioridad se consideran menos buenas porque no comparten los recursos tan bien. Por otro lado, los procesos de baja prioridad son buenos porque insisten en tomar solo los recursos mínimos.
Cuando ejecutamos top
al principio del artículo, había una columna marcada como “NI”. Este es el valor bueno del proceso:
top***
Tasks: 56 total, 1 running, 55 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 1019600k total, 324496k used, 695104k free, 8512k buffers Swap: 0k total, 0k used, 0k free, 264812k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1635 root 20 0 17300 1200 920 R 0.3 0.1 0:00.01 top 1 root 20 0 24188 2120 1300 S 0.0 0.2 0:00.56 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:00.11 ksoftirqd/0
Los valores buenos pueden oscilar entre “-19/-20” (máxima prioridad) y “19/20” (mínima prioridad) dependiendo del sistema.
Para ejecutar un programa con un determinado valor bueno, podemos usar el comando nice
:
<pre>nice -n 15 <span class=“highlight”>command_to_execute</span></pre>
Esto solo funciona cuando se inicia un nuevo programa.
Para alterar el valor bueno de un programa que ya se está ejecutando, usamos una herramienta llamada renice
:
<pre>renice 0 <span class=“highlight”>PID_to_prioritize</span></pre>
Nota: Mientras que nice funciona necesariamente con un nombre de comando, renice funciona invocando al PID del proceso
La administración de procesos es un tema que a veces resulta difícil de entender para los nuevos usuarios, debido a que las herramientas utilizadas son diferentes a sus contrapartes gráficas.
Sin embargo, las ideas son familiares e intuitivas, y, con un poco de práctica, se convertirá en algo natural. Dado que los procesos interviene en todo lo que se hace con un sistema informático, aprender a controlarlos de forma eficaz es una habilidad esencial.
<div class=“author”>Por Justin Ellingwood</div>
]]>Here is how you could do that!
]]>I ran the following commands yesterday to update the permissions on files and directories for my WordPress site and now I cannot connect to the parent directory (name.com - example directory name) using SSH or SFTP.
sudo find /var/www/name.com/ -type f -exec chmod 640 {} \;
sudo find /var/www/name.com/ -type d -exec chmod 750 {} \;
These commands were recommended on this tutorial article: https://www.digitalocean.com/community/tutorials/how-to-install-wordpress-with-lamp-on-ubuntu-18-04.
My FTP client (Transmit, macOS) displays this error when trying to connect: Could not change directory to “/var/www/callkneehill.ca/wp-content/themes”.
Could someone share what is going wrong and possibly let me know how best to set permissions so that SSH and SFTP work correctly.
Cheers,
]]>Here is how you could do that!
]]>Commands to be daily executed on night 11:25pm
sudo service apache2 restart
sudo apt update
sudo apt upgrade -y
sudo apt autoremove -y
sudo reboot
Please help me. Thanks in advance
]]>ERROR - Could not read the configuration file /opt/hivemq/conf/config.xml. Using default config
]]>I don’t know much about back-end, and other person built the system.
But all the images of my sites that are at port:8888 are not appearing anymore. I don’t see anything in my log error, the port is simply not responding.
The only tip I have is that I see nginx and redis at the system, and a phyton is what I have at port 8888.
(My system is linux 14)
Any tip?
Thanks.
]]>/
partition was running very low on disk space, but I also had an additional disk mounted at /home/
with plenty of disk space.
However as by default Docker stores everything at /var/lib/docker
my /
partition was nearly full.
To fix that I moved the default /var/lib/docker
to another directory on the /home
partition.
In case that you are in the same situation, here is how to do that on a Linux server!
]]>The Debian project maintains an official repository holding thousands of software packages which APT users can install via the apt
command-line program and a network connection. Users can also install packages from third-party repositories as well as locally-stored repositories.
For more information about APT and other package management utilities, see our guide on Package Management Basics.
]]>Does anyone have a command on hand which could show the remote URL of a specific local git repository?
I usually use the cat
command to check the content of the .git/config
and look for the remote origin
section in there.
But is there a better way of doing this?
Thanks!
]]>I’ve attempted to fix this by following a similar question posted on the community: [https://www.digitalocean.com/community/questions/set-up-django-nginx-and-ubuntu-18-04-welcome-to-nginx-landing-page-displays-when-www-prefix-is-loaded-with-domain-works-fine-without] without any success
/etc/nginx/sites-available/default:
server {
listen 80 default_server;
listen [::]:80 default_server;
# SSL configuration
#
# listen 443 ssl default_server;
# listen [::]:443 ssl default_server;
#
# Note: You should disable gzip for SSL traffic.
# See: https://bugs.debian.org/773332
#
# Read up on ssl_ciphers to ensure a secure configuration.
# See: https://bugs.debian.org/765782
#
# Self signed certs generated by the ssl-cert package
# Don't use them in a production server!
#
# include snippets/snakeoil.conf;
root /var/www/html;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
server_name _;
location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ =404;
}
# pass PHP scripts to FastCGI server
#
#location ~ \.php$ {
# include snippets/fastcgi-php.conf;
#
# # With php-fpm (or other unix sockets):
# fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
# # With php-cgi (or other tcp sockets):
# fastcgi_pass 127.0.0.1:9000;
#}
# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
# deny all;
#}
}
# Virtual Host configuration for example.com
#
# You can move that to a different file under sites-available/ and symlink that
# to sites-enabled/ to enable it.
#
#server {
# listen 80;
# listen [::]:80;
#
# server_name example.com;
#
# root /var/www/example.com;
# index index.html;
#
# location / {
# try_files $uri $uri/ =404;
# }
#}
/etc/nginx/sites-available/website:
server {
server_name website.com www.website.com;
location = /favicon.ico { access_log off; log_not_found off; }
location /static/ {
root /home/…/website;
}
location / {
include proxy_params;
proxy_pass http://unix:/run/gunicorn.sock;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/website.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/website.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
if ($host = www.website.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
if ($host = website.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name server_name website.com www.website.com;
return 404; # managed by Certbot
}
]]>In this tutorial you will use systemd
to configure MySQL to restart automatically after a reboot or crash.
This is the second half of a two-part series. Part One covers general Linux service management concepts like the init
daemon and runlevels. It ends with a demonstration of service management in systemd
. Here you will examine targets
, wants
, requires
, and unit
files. This part, part two, provides a practical example using the MySQL database.
Note: You might also consider reading our very popular tutorial on using systemctl
to control systemd services and units.
To complete this tutorial, you will need:
A server running CentOS 8, including a non-root user with sudo privileges. To set all this up, including a firewall, you can create a DigitalOcean Droplet running CentOS 8 and then follow our Initial Server Setup Guide.
MySQL installed. For detailed instructions, follow our tutorial, How To Install MySQL on CentOS 8.
systemd
With MySQL installed, check the status of your service:
- sudo systemctl status mysqld.service
The output should show that the service is running, but the daemon is disabled:
Output mysqld.service - MySQL 8.0 database server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; disabled; vendor preset: disabled)
Active: active (running) since Thu 2020-12-24 23:48:56 UTC; 1h 6min ago
Process: 30423 ExecStartPost=/usr/libexec/mysql-check-upgrade (code=exited, status=0/SUCCESS)
Process: 30294 ExecStartPre=/usr/libexec/mysql-prepare-db-dir mysqld.service (code=exited, status=0/SUCCESS)
Process: 30270 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
Main PID: 30378 (mysqld)
Status: "Server is operational"
Tasks: 40 (limit: 4763)
...
If the service is enabled, disable it. We want to first explore the disabled behavior before making our changes:
- sudo systemctl disable mysqld.service
Next, run this command to check if MySQL is wanted by multi-user.target:
- sudo systemctl show --property "Wants" multi-user.target | fmt -10 | grep mysql
Nothing will return. Now check if the symbolic link exists:
- sudo ls -l /etc/systemd/system/multi-user.target.wants/mysql*
A message appears stating that the symlink file does not exist:
Outputls: cannot access '/etc/systemd/system/multi-user.target.wants/mysql*': No such file or directory
Now, if you like, reboot the server and check the MySQL service. It should not be running.
Whether you rebooted or not, now re-enable the MySQL service:
- sudo systemctl enable mysqld.service
This time, the system will create a symbolic link under /etc/systemd/system/multi-user.target.wants/
:
OutputCreated symlink /etc/systemd/system/multi-user.target.wants/mysqld.service → /usr/lib/systemd/system/mysqld.service.
Run the ls command again to confirm this:
- sudo ls -l /etc/systemd/system/multi-user.target.wants/mysql*
You will receive an output like this:
Outputlrwxrwxrwx 1 root root 38 Aug 1 04:43 /etc/systemd/system/multi-user.target.wants/mysqld.service -> /usr/lib/systemd/system/mysqld.service
Enabling or disabling a systemd service creates or removes the symbolic link from the default target’s wants directory.
If you like, reboot the Droplet again, and when it comes back online run the ps -ef
command to check the service status.
- ps -ef | grep mysql
This command will provide information about MySQL, if it is running:
[secondary_label Output]\
mysql 851 1 2 04:26 ? 00:00:02 /usr/libexec/mysqld --basedir=/usr
You have now configured MySQL to restart after a reboot. Next you will account for crashes.
systemd
Being a modern application, MySQL already comes configured to auto-start after a crash. Let’s see how to disable that.
Open the MySQL service unit file in an editor:
- sudo vi /etc/systemd/system/multi-user.target.wants/mysqld.service
After the header information, the contents of the file looks like this:
[Unit]
Description=MySQL 8.0 database server
After=syslog.target
After=network.target
[Service]
Type=notify
User=mysql
Group=mysql
ExecStartPre=/usr/libexec/mysql-check-socke
ExecStartPre=/usr/libexec/mysql-prepare-db-dir %n
`# Note: we set --basedir to prevent probes that might trigger SELinux alarms,`
`# per bug #547485`
ExecStart=/usr/libexec/mysqld --basedir=/usr
ExecStartPost=/usr/libexec/mysql-check-upgrade
ExecStopPost=/usr/libexec/mysql-wait-stop
`# Give a reasonable amount of time for the server to start up/shut down`
TimeoutSec=300
`# Place temp files in a secure directory, not /tmp`
PrivateTmp=true
Restart=on-failure
RestartPreventExitStatus=1
`# Sets open_files_limit`
LimitNOFILE = 10000
`# Set enviroment variable MYSQLD_PARENT_PID. This is required for SQL restart command.`
Environment=MYSQLD_PARENT_PID=1
[Install]
WantedBy=multi-user.target
As you can see the value of the Restart parameter is set to on-failure. This means the MySQL service will restart for unclean exit codes or timeouts.
The man page for systemd service shows the following table for Restart parameters:
Restart settings/Exit causes | no | always | on-success | on-failure | on-abnormal | on-abort | on-watchdog |
---|---|---|---|---|---|---|---|
Clean exit code or signal | X | X | |||||
Unclean exit code | X | X | |||||
Unclean signal | X | X | X | X | |||
Timeout | X | X | X | ||||
Watchdog | X | X | X | X |
In a systemd service unit file, two parameters - Restart
and RestartSec
- control crash behavior. The first parameter specifies when the service should restart, and the second parameter defines how long it should wait before restarting.
To test the crash behavior, stop the MySQL process with a kill -9 signal. In our case the main PID is 851; replace the PID with your own:
ps -ef | grep mysql
- sudo kill -9 851
Wait for a few seconds and then check the status:
- sudo systemctl status mysqld.service
The output will show that MySQL has restarted with a new PID (in our case the new Process ID is 1513):
Output mysqld.service - MySQL 8.0 database server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2020-12-25 04:47:48 UTC; 55s ago
Process: 1420 ExecStopPost=/usr/libexec/mysql-wait-stop (code=exited, status=0/SUCCESS)
Process: 1559 ExecStartPost=/usr/libexec/mysql-check-upgrade (code=exited, status=0/SUCCESS)
Process: 1476 ExecStartPre=/usr/libexec/mysql-prepare-db-dir mysqld.service (code=exited, status=0/SUCCESS)Process: 1451 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
Main PID: 1513 (mysqld)
Status: "Server is operational"
...
Next, reopen the unit file:
- sudo vi /etc/systemd/system/multi-user.target.wants/mysqld.service
Comment out the Restart directive in the MySQL daemon’s unit file and save it. This will disable the restart behavior:
`# Restart=on-failure`
After this, reload the systemd daemon, followed by a restart of the mysqld service:
- sudo systemctl daemon-reload
- sudo systemctl restart mysqld.service
You can find the main PID of the service by running this command:
- sudo systemctl status mysqld.service
Output. . .
Main PID: 1895 (mysqld)
Using the kill -9
command, kill the main PID of the MySQL PID in your environment (we are using the PID in our test environment).
sudo kill -9 1895
Check the status for MySQL:
- sudo systemctl status mysqld.service
It will show that the service has failed:
Outputmysqld.service - MySQL 8.0 database server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
Active: **failed** (Result: signal) since Fri 2020-12-25 05:07:22 UTC; 1min 14s ago
Process: 1976 ExecStopPost=/usr/libexec/mysql-wait-stop (code=exited, status=0/SUCCESS)
Process: 1940 ExecStartPost=/usr/libexec/mysql-check-upgrade (code=exited, status=0/SUCCESS)
Process: 1895 ExecStart=/usr/libexec/mysqld --basedir=/usr (code=killed, signal=KILL)
Process: 1858 ExecStartPre=/usr/libexec/mysql-prepare-db-dir mysqld.service (code=exited, status=0/SUCCESS
Process: 1833 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
Main PID: 1895 (code=**killed**, signal=KILL)
...
Try to find the service status a few times. Each time the service will show as failed
.
So, we have emulated a crash where the service has stopped and hasn’t come back. This is because we have instructed systemd not to restart the service after an unclean stop. If you edit the mysqld.service unit file to uncomment the Restart parameter, save it, reload the systemctl daemon, and finally restart the service, that will restore normal function.
This is how you can configure a native systemd service to auto-start after a crash. All you have to do is add an extra directive for Restart
(and optionally RestartSec
) under the [Service]
section of the service unit file.
In this two-part series, you learned about the service management daemons used across the Linux ecosystem. You then explored the fundamentals of systemd and applied those fundamentals to a practical example: configuring a database to restart after a reboot or crash. If you wish to learn more about systemd, consider exploring our comprehensive tutorial on using systemctl
.
In this two-part tutorial, you will learn how to configure a Linux service to restart automatically after a reboot or crash using systemd
.
Part One covers general Linux service management concepts like the init
daemon and runlevels. It ends with a demonstration of service management in systemd
. Here you will examine targets
, wants
, requires
, and unit
files.
Part Two offers a step-by-step tutorial for completing a practical and common systemd
task. Specifically, you will configure a MySQL database server to automatically start after a crash or reboot.
Note: You might also consider reading our very popular tutorial on using systemctl
to control systemd services and units.
To complete this tutorial, you will need:
Linux services can be made self-healing largely by changing the way they are handled by the service management daemon, also known as the init
daemon.
init
is the first process that starts in a Linux system after the machine boots and the kernel loads into memory. Among other things, it decides how a user process or a system service should load, in what order, and whether it should start automatically.
As Linux has evolved, so has the behavior of the init
daemon. Originally, Linux started out with System V init
, the same that was used in Unix. Since then, Linux has implemented the Upstart init
daemon (created by Ubuntu), and now the systemd init
daemon (first implemented by Fedora).
Most modern Linux distributions have gradually migrated away from System V and currently using systemd. Older style init
(if used) is kept only for backward compatibility. FreeBSD, a variant of UNIX, uses a different System V implementation, known as BSD init
.
We will cover systemd in this article as this is the most recent and common service manager used in Linux distributions today. However, we will also talk about System V and Upstart when necessary and see how systemd evolved from there. To give you an idea:
init
system, used inTo understand the init
daemon, let’s start with something called runlevel.
A runlevel represents the current state of a Linux system. For example, a runlevel can be the shutdown state of a Linux server, a single-user mode, the restart mode, etc. Each mode will dictate what services can be running in that state.
Some services can run in one or more runlevel but not in others. Runlevels are denoted by a value between 0 and 6. The following list shows what each of these levels means:
Runlevels 2, 3, and 4 vary by distribution. For example, some Linux distributions don’t implement runlevel 4, while others do. Some distributions have a clear distinction between these three levels. In general, runlevel 2, 3, or 4 means a state where Linux has booted in multi-user, network-enabled, text mode.
When you enable a service to auto-start, Linux is actually adding it to a runlevel. In System V, for example, the OS will start with a particular runlevel; and, when it starts, it will try to start all the services that are associated with that runlevel. In systemd, runlevel has become target, and when a service is made to auto-start, it’s added to a target. We’ll discuss targets later in this article.
init
DaemonSystem V uses an inittab
file, which later init
methods have kept for backward compatibility. Let’s run through System V’s startup sequence:
init
daemon is created from the binary file /sbin/initinit
daemon reads is /etc/inittabinit
daemon looks further into the /etc/inittab file and reads what init
scripts it needs to run for that runlevelSo, when the init
daemon finds what init
scripts its needs to run for the given runlevel, it’s actually finding out what services it needs to start up. These init
scripts are where you can configure startup behavior for individual services.
An init
script is what controls a specific service in System V. init
scripts for services were either provided by the application’s vendor or came with the Linux distribution (for native services). You can also create your own init
script for a custom created service.
When a process or service such as MySQL Server starts under System V, its binary program file has to load into memory. Depending on how the service is configured, this program may continuously execute the background (and accept client connections). The job of starting, stopping, or reloading this binary application is handled by the service’s init
script. It’s called the init
script because it initializes the service.
In System V, an init
script is a shell script. They are also called rc
(run command) scripts. The scripts are located under the /etc/init.d
directory. These scripts are symlinked to the /etc/rc
directories. Within the /etc
directory, there are a number of rc
directories, each with a number in its name. The numbers represent different runlevels. So we have /etc/rc0.d
, /etc/rc1.d
, /etc/rc2.d
and so on.
To make a service restart after a crash or reboot, you can usually add a line like this to the init
script:
- ms:2345:respawn:/bin/sh /usr/bin/service_name
To enable a System V service to start at system boot time, run this command:
- sudo chkconfig service_name on
To disable it, run this command:
- sudo chkconfig service_name off
To check the status (running or stopped), run this command
- sudo service service_name status
As the serialized way of loading jobs and services became more time consuming and complex with System V init
, the Upstart daemon was introduced for faster loading of the OS, graceful clean-up of crashed services, and predictable dependency between system services.
Upstart init
was better than System V init
in a few ways:
To keep things simple, Upstart is backward-compatible with System V. The /etc/init.d/rc
script still runs to manage native System V services. Its main difference is the way it allows multiple events to be associated with a service. This event-based architecture allowed Upstart to be a flexible service manager. With Upstart, each event can fire off a shell script that takes care of that event. These events included:
In between these events, a service can be in a number of states, like waiting, pre-start, starting, running, pre-stop, stopping, etc. Upstart could take action for each of these states as well, creating a very flexible architecture.
At startup, Upstart will run any System V init
scripts normally. It will then look under the /etc/init
directory and execute the shell commands in each service configuration file. Among other things, these files controlled the service’s startup behavior.
The files have a naming style of service_name.conf
, and they have plain text content with different sections, called stanzas. Each stanza describes a different aspect of the service and how it should behave. To make a service start automatically after a crash or reboot, you can add the respawn
command in its service configuration files, as shown below for the cron service.
...
description "regular background program processing daemon"
start on runlevel [2345]
stop on runlevel [!2345]
expect fork
**respawn**
exec cron
systemd
DaemonThe latest in Linux init
daemons is systemd. In fact, it’s more than an init
daemon: systemd is a framework that encompasses many components of a modern Linux system.
One of its functions is to work as a system and service manager for Linux. In this capacity, systemd controls how a service should behave if it crashes or the machine reboots. You can read about systemd’s systemctl here.
systemd is backward-compatible with System V commands and initialization scripts. That means any System V service will also run under systemd. This is possible because most Upstart and System V administrative commands have been modified to work under systemd.
At the heart of systemd are unit files. Each unit file represents a specific system resource. Information about the resource is kept track of in the unit file. Service unit files are simple text files (like Upstart .conf files) with a declarative syntax. This makes the files easy to understand and modify.
The main difference between systemd and the other two init
methods is that systemd is responsible for the initialization of service daemons and other types of resources like device operating system paths, mount points, sockets, etc. The naming style for a unit file is service_name.unit_type
. So, you will see files like dbus.service
, sshd.socket
, or home.mount
.
In Red Hat-based systems like CentOS, unit files are located in two places. The main location is /lib/systemd/system/
. Custom-created unit files or existing unit files modified by system administrators will live under /etc/systemd/system
.
If a unit file with the same name exists in both locations, systemd will use the one under /etc
. Suppose a service is enabled to start at boot time or any other target/runlevel. In that case, a symbolic link will be created for that service unit file under appropriate directories in /etc/systemd/system
. Unit files under /etc/systemd/system
are actually symbolic links to the files with the same name under /lib/systemd/system
.
init
Sequence: Target UnitsA special type of unit file is a target unit.
A target unit filename is suffixed by .target. Target units are different from other unit files because they don’t represent one particular resource. Rather, they represent the state of the system at any one time. Target units do this by grouping and launching multiple unit files that should be part of that state. systemd targets can therefore be loosely compared to System V runlevels, although they are not the same.
Each target has a name instead of a number. For example, we have multi-user.target
instead of runlevel 3
or reboot.target
instead of runlevel 6
. When a Linux server boots with say, multi-user.target
, it’s essentially bringing the server to runlevel 2, 3, or 4
, which is the multi-user text mode with networking enabled.
How it brings the server up to that stage is where the difference lies. Unlike System V, systemd does not bring up services sequentially. Along the way, it can check for the existence of other services or resources and decide the order of their loading. This makes it possible for services to load in parallel.
Another difference between target units and runlevels is that in System V, a Linux system could exist in only one runlevel. You could change the runlevel, but the system would exist in that new runlevel only. With systemd, target units can be inclusive, which means when a target unit activates, it can ensure other target units are loaded as part of it.
For example, a Linux system that boots with a graphical user interface will have the graphical.target activated, which in turn will automatically ensure multi-user.target is loaded and activated as well. In System V terms, that would be like having runlevels 3 and 5 activated simultaneously.
The table below compares runlevels and targets:
Runlevel (System V init) | Target Units (Systemd) |
---|---|
runlevel 0 | poweroff.target |
runlevel 1 | resuce.target |
runlevel 2, 3, 4 | multi-user.target |
runlevel 5 | graphical.target |
runlevel 6 | reboot.target |
systemd default.target
is equivalent to the System V default runlevel.
System V had the default runlevel defined in a file called inittab. In systemd, that file is replaced by default.target. The default target unit file lives under /etc/systemd/system directory. It’s a symbolic link to one of the target unit files under /lib/systemd/system.
When you change the default target, you are essentially recreating that symbolic link and changing the system’s runlevel.
The inittab file in System V also specified which directory Linux will execute its init
scripts from: it could be any of the rcn.d directories. In systemd, the default target unit determines which resource units will be loaded at boot time.
As units are activated, they are activated all in parallel or all in sequence. How a resource unit loads may depend on other resource units it wants or requires.
systemd wants and requires control how systemd addresses dependency among service daemons.
As mentioned before, Upstart ensures parallel loading of services using configuration files. In System V, a service could start in a particular runlevel, but it also could be made to wait until another service or resource became available. In a similar fashion, systemd services can be made to load in one or more targets, or wait until another service or resource became active.
In systemd, a unit that requires another unit will not start until the required unit is loaded and activated. If the required unit fails for some reason while the first unit is active, the first unit will also stop.
This ensures system stability. A service that requires a particular directory to be present can thus be made to wait until the mount point to that directory is active. On the other hand, a unit that wants another unit will not impose such restrictions. It won’t stop if the wanted unit stops when the caller is active. An example of this would be the non-essential services that come up in graphical-target mode.
To understand service startup behavior under systemd, we are using a CentOS 8.3 Droplet. We will follow the .target rabbit-trail as far as we can. systemd’s startup sequence follows a long chain of dependencies.
First, let’s run this command to list the default target unit file:
- sudo ls -l /etc/systemd/system/default.target
This shows output like the following:
Outputlrwxrwxrwx. 1 root root 37 Dec 4 17:42 /etc/systemd/system/default.target -> /lib/systemd/system/multi-user.target
As you can see, the default target is actually a symbolic link to the multi-user target file under /lib/systemd/system/. So, the system is supposed to boot under multi-user.target, which is similar to runlevel 3 in System V init.
Next, let’s run the following command to check all the services the multi-user.target file wants:
- sudo ls -l /etc/systemd/system/multi-user.target.wants/*.service
This should show a list of symbolic link files, pointing back to actual unit files under /usr/lib/systemd/system/:
Outputlrwxrwxrwx. 1 root root 38 Dec 4 17:38 /etc/systemd/system/multi-user.target.wants/auditd.service -> /usr/lib/systemd/system/auditd.service
lrwxrwxrwx. 1 root root 39 Dec 4 17:39 /etc/systemd/system/multi-user.target.wants/chronyd.service -> /usr/lib/systemd/system/chronyd.service
lrwxrwxrwx. 1 root root 37 Dec 4 17:38 /etc/systemd/system/multi-user.target.wants/crond.service -> /usr/lib/systemd/system/crond.service
lrwxrwxrwx. 1 root root 42 Dec 4 17:39 /etc/systemd/system/multi-user.target.wants/irqbalance.service -> /usr/lib/systemd/system/irqbalance.service
lrwxrwxrwx. 1 root root 37 Dec 4 17:41 /etc/systemd/system/multi-user.target.wants/kdump.service -> /usr/lib/systemd/system/kdump.service
...
Other than multi-user.target
, there are different types of targets like system-update.target
or basic.target
. To see what targets the multi-user target depends on, run the following command:
- sudo systemctl show --property "Requires" multi-user.target | fmt -10
The output shows:
OutputRequires=basic.target
You can run the following command to see if there are any required units for basic.target:
- sudo systemctl show --property "Requires" basic.target | fmt -10
As it turns out, the basic.target requires sysinit.target:
OutputRequires=sysinit.target
-.mount
And it wants a few targets as well:
- sudo systemctl show --property "Wants" basic.target | fmt -10
The command will return the following:
OutputWants=slices.target
paths.target
timers.target
microcode.service
sockets.target
sysinit.target
Going recursively, you can see if the sysinit.target will require any other targets to be running as well:
- sudo systemctl show --property "Requires" sysinit.target | fmt -10
There will be none. However, there will be other targets wanted by sysinit.target:
- systemctl show --property "Wants" sysinit.target | fmt -10
An output like this will appear:
OutputWants=systemd-random-seed.service
dev-mqueue.mount
rngd.service
systemd-modules-load.service
proc-sys-fs-binfmt_misc.automount
local-fs.target
sys-fs-fuse-connections.mount
systemd-sysusers.service
systemd-update-done.service
systemd-update-utmp.service
systemd-journal-flush.service
dev-hugepages.mount
dracut-shutdown.service
swap.target
systemd-udevd.service
import-state.service
sys-kernel-debug.mount
nis-domainname.service
systemd-journald.service
selinux-autorelabel-mark.service
kmod-static-nodes.service
loadmodules.service
ldconfig.service
cryptsetup.target
systemd-sysctl.service
systemd-ask-password-console.path
systemd-journal-catalog-update.service
systemd-udev-trigger.service
systemd-tmpfiles-setup.service
systemd-hwdb-update.service
sys-kernel-config.mount
systemd-binfmt.service
systemd-tmpfiles-setup-dev.service
systemd-machine-id-commit.service
systemd-firstboot.service
systemd
Unit FileGoing a step further now, let’s look inside a service unit file, the one for sshd:
- sudo vi /etc/systemd/system/multi-user.target.wants/sshd.service
It looks like this:
[Unit]
Description=OpenSSH server daemon
Documentation=man:sshd(8) man:sshd_config(5)
After=network.target sshd-keygen.target
Wants=sshd-keygen.target
[Service]
Type=notify
EnvironmentFile=-/etc/crypto-policies/back-ends/opensshserver.config
EnvironmentFile=-/etc/sysconfig/sshd
ExecStart=/usr/sbin/sshd -D $OPTIONS $CRYPTO_POLICY
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartSec=42s
[Install]
WantedBy=multi-user.target
You can see the service unit file is clean and easy to understand.
The first important part is the After clause in the [Unit]
section. This says the sshd service needs to load after the network.target and the sshd-keygen.target are loaded.
The [Install]
section shows the service is wanted by the multi-user.target. This means multi-user.target will load the sshd daemon, but it won’t shut down or crash if sshd fails during loading.
Since multi-user.target
is the default target, sshd daemon is supposed to start at boot time. In the [Service]
section, the Restart
parameter has the value on-failure
. This setting allows the sshd daemon to restart if it crashes or has an unclean exit.
In this article you learned about System V, Upstart, and systemd service management daemons. You explored startup scripts and configuration files, important parameters, startup sequence, and the commands that control service startup behavior.
In part two of this article, we will apply these skills to a real-life example and use systemd to configure MySQL. Once completed, your MySQL instance will automatically restart after a reboot or crash. And while you will use MySQL as your example application, you can substitute any number of services, such as the Nginx or Apache web servers.
]]>I am working on a Bash script and I wanted to change the color of some of the output to emphasize on some specific words.
For example, I want to be able to print the text in green when the script is successful and in red when the script fails.
Does anyone know an easy way of doing so?
]]>I have created a user in linux and I’m trying to restrict them in creating sub folders in their home directory, but they should still able to upload or create files.
Hope someone can help me here.
]]>Ein Problem, auf das Benutzer stoßen, wenn sie zum ersten Mal mit einem Linux-Rechner arbeiten, ist die Frage, wie Sie die gesuchten Dateien finden können.
Dieser Leitfaden behandelt die Verwendung des treffend benannten Befehls find
Dies hilft Ihnen bei der Suche nach Dateien auf Ihrem VPS unter Verwendung einer Vielzahl von Filtern und Parametern. Wir werden auch kurz auf den Befehl locate
eingehen, mit dem Sie auf andere Weise nach Dateien suchen können.
Die offensichtlichste Möglichkeit der Suche nach Dateien ist nach Name.
Um eine Datei anhand ihres Namens zu finden, geben Sie Folgendes ein:
- find -name "query"
Dabei wird die Groß- und Kleinschreibung beachtet, d. h. eine Suche nach file
unterscheidet sich von einer Suche nach File
.
Um eine Datei nach dem Namen zu suchen, aber die Groß-/Kleinschreibung der Abfrage zu ignorieren, geben Sie Folgendes ein:
- find -iname "query"
Wenn Sie alle Dateien finden möchten, die sich nicht an ein bestimmtes Muster halten, können Sie die Suche mit -not
oder !
umkehren. Wenn Sie !
verwenden, müssen Sie das Zeichen Maskieren, damit die Bash nicht versucht, es zu interpretieren, bevor „find“ agieren kann:
- find -not -name "query_to_avoid"
Oder
- find \! -name "query_to_avoid"
Sie können den Typ der Dateien angeben, die Sie mit dem Parameter -type
finden möchten. Es funktioniert wie folgt:
- find -type type_descriptor query
Einige der gebräuchlichsten Deskriptoren, die Sie zur Angabe des Dateityps verwenden können, sind:
f: reguläre Datei
d: Verzeichnis
l: symbolischer Link
c: Zeichengeräte
b: Blockgeräte
Wenn wir beispielsweise alle Zeichengeräte in unserem System finden möchten, könnten wir diesen Befehl ausgeben:
- find / -type c
Output/dev/parport0
/dev/snd/seq
/dev/snd/timer
/dev/autofs
/dev/cpu/microcode
/dev/vcsa7
/dev/vcs7
/dev/vcsa6
/dev/vcs6
/dev/vcsa5
/dev/vcs5
/dev/vcsa4
. . .
Wir können nach allen Dateien suchen, die in .conf
enden:
- find / -type f -name "*.conf"
Output/var/lib/ucf/cache/:etc:rsyslog.d:50-default.conf
/usr/share/base-files/nsswitch.conf
/usr/share/initramfs-tools/event-driven/upstart-jobs/mountall.conf
/usr/share/rsyslog/50-default.conf
/usr/share/adduser/adduser.conf
/usr/share/davfs2/davfs2.conf
/usr/share/debconf/debconf.conf
/usr/share/doc/apt-utils/examples/apt-ftparchive.conf
. . .
„Find“ bietet Ihnen eine Vielzahl von Möglichkeiten, um Ergebnisse nach Größe und Zeit zu filtern.
Sie können mit dem Parameter -size
nach Größe filtern.
Wir fügen ein Suffix am Ende unseres Wertes hinzu, das angibt, wie wir zählen. Dies sind einige beliebte Optionen:
c: Bytes
k: Kilobytes
M: Megabytes
G: Gigabytes
b: 512-Byte-Blöcke
Um alle Dateien zu finden, die genau 50 Bytes groß sind, geben Sie Folgendes ein:
- find / -size 50c
Um alle Dateien zu finden, die kleiner als 50 Bytes sind, können wir stattdessen diese Form verwenden:
- find / -size -50c
Um alle Dateien zu finden, die größer als 700 Megabytes sind, können wir diesen Befehl verwenden:
- find / -size +700M
Linux speichert Zeitdaten über Zugriffszeiten, Modifizierungszeiten und Änderungszeiten.
Zugriffszeit: Letzter Zeitpunkt, in dem eine Datei gelesen oder geschrieben wurde.
Modifizierungszeit: Letzter Zeitpunkt, an dem der Inhalt der Datei modifiziert wurde.
Änderungszeit: Letzter Zeitpunkt, an dem die Inode-Metadaten der Datei geändert wurden.
Wir können diese mit den Parametern -atime
, -mtime
und -ctime
verwenden. Diese können die Plus- und Minus-Symbole verwenden, um größer als oder kleiner als anzugeben, wie wir es bei der Größe getan haben.
Der Wert dieses Parameters gibt an, wie viele zurückliegende Tage Sie suchen möchten.
Um Dateien zu finden, die eine Modifizierungszeit von vor einem Tag haben, geben Sie Folgendes ein:
- find / -mtime 1
Wenn wir Dateien möchten, auf die vor weniger als einem Tag zugegriffen wurden, können wir Folgendes eingeben:
- find / -atime -1
Um Dateien zu erhalten, deren letzte Änderung der Metadaten mehr als 3 Tage zurückliegt, geben Sie Folgendes ein:
- find / -ctime +3
Es gibt auch einige Begleitparameter, mir denen wir Minuten anstelle von Tagen angeben können:
- find / -mmin -1
Dadurch werden die Dateien angegeben, die in der letzten Minute im System modifiziert wurden.
„Find“ kann auch Vergleiche gegen eine Referenzdatei durchführen und diejenigen zurückgeben, die neuer sind:
- find / -newer myfile
Sie können auch nach Dateien anhand des Dateieigentümers oder des Gruppeneigentümers suchen.
Dazu verwenden Sie die Parameter -user
bzw. -group
. Suchen Sie eine Datei, deren Eigentümer der Benutzer „syslog“ ist, indem Sie Folgendes eingeben:
- find / -user syslog
Ähnlich können wir Dateien angeben, deren Eigentümer die Gruppe „shadow“ ist, indem wir Folgendes eingeben:
- find / -group shadow
Wir können auch nach Dateien mit bestimmten Berechtigungen suchen.
Wenn wir einen genauen Satz von Berechtigungen abgleichen möchten, verwenden wir diese Form:
- find / -perm 644
Dies gleicht Dateien mit genau den angegebenen Berechtigungen ab.
Wenn wir etwas mit zumindest diesen Berechtigungen angeben möchten, können Sie diese Form verwenden:
- find / -perm -644
Dies gleicht alle Dateien ab, die zusätzliche Berechtigungen aufweisen. Eine Datei mit der Berechtigung „744“ würde in diesem Fall gefunden werden.
Für diesen Abschnitt erstellen wir eine Verzeichnisstruktur in einem temporären Verzeichnis. Es enthält drei Verzeichnisebenen, mit zehn Verzeichnissen auf der ersten Ebene. Jedes Verzeichnis (einschließlich des temporären Verzeichnisses) enthält zehn Dateien und zehn Unterverzeichnisse.
Erstellen Sie diese Struktur durch Ausgabe der folgenden Befehle:
- cd
- mkdir -p ~/test/level1dir{1..10}/level2dir{1..10}/level3dir{1..10}
- touch ~/test/{file{1..10},level1dir{1..10}/{file{1..10},level2dir{1..10}/{file{1..10},level3dir{1..10}/file{1..10}}}}
- cd ~/test
Sie können sich die Verzeichnisstrukturen mit ls
und cd
ansehen, um einen Überblick über die Organisation zu erhalten. Wenn Sie fertig sind, kehren Sie in das Testverzeichnis zurück:
- cd ~/test
Wir werden uns damit beschäftigen, wie bestimmte Dateien aus dieser Struktur zurückgegeben werden. Versuchen wir zunächst ein Beispiel mit einer regulären Namenssuche, zum Vergleich:
- find -name file1
Output./level1dir7/level2dir8/level3dir9/file1
./level1dir7/level2dir8/level3dir3/file1
./level1dir7/level2dir8/level3dir4/file1
./level1dir7/level2dir8/level3dir1/file1
./level1dir7/level2dir8/level3dir8/file1
./level1dir7/level2dir8/level3dir7/file1
./level1dir7/level2dir8/level3dir2/file1
./level1dir7/level2dir8/level3dir6/file1
./level1dir7/level2dir8/level3dir5/file1
./level1dir7/level2dir8/file1
. . .
Es gibt eine Menge Ergebnisse. Wenn wir die Ausgabe in einen Zähler leiten, können wir sehen, dass es insgesamt 1111 Ergebnisse gibt:
- find -name file1 | wc -l
Output1111
Das sind wahrscheinlich zu viele Ergebnisse, um für Sie in den meisten Fällen nützlich zu sein. Versuchen wir, dies einzugrenzen.
Sie können die maximale Tiefe der Suche unter dem Suchverzeichnis auf höchster Ebene festlegen:
- find -maxdepth num -name query
Um „file1“ nur in den Verzeichnissen „level1“ und höher zu finden, können Sie eine maximale Tiefe von 2 (1 für das Verzeichnis auf höchster Ebene und 1 für die Verzeichnisse level1) festlegen:
- find -maxdepth 2 -name file1
Output./level1dir7/file1
./level1dir1/file1
./level1dir3/file1
./level1dir8/file1
./level1dir6/file1
./file1
./level1dir2/file1
./level1dir9/file1
./level1dir4/file1
./level1dir5/file1
./level1dir10/file1
Das ist eine wesentlich besser verwaltbare Liste.
Sie können auch ein Mindestverzeichnis angeben, wenn Sie wissen, dass alle Dateien ab einem bestimmten Punkt unter dem aktuellen Verzeichnis bestehen:
- find -mindepth num -name query
Wir können dies verwenden, um nur die Dateien am Ende der Verzeichniszweige zu finden:
- find -mindepth 4 -name file
Output./level1dir7/level2dir8/level3dir9/file1
./level1dir7/level2dir8/level3dir3/file1
./level1dir7/level2dir8/level3dir4/file1
./level1dir7/level2dir8/level3dir1/file1
./level1dir7/level2dir8/level3dir8/file1
./level1dir7/level2dir8/level3dir7/file1
./level1dir7/level2dir8/level3dir2/file1
. . .
Wiederum wird dies aufgrund unserer verzweigten Verzeichnisstruktur eine große Anzahl von Ergebnissen (1000) zurückgegeben.
Sie können die Parameter Min- und Max-Tiefe kombinieren, um sich auf einen engen Bereich zu konzentrieren:
- find -mindepth 2 -maxdepth 3 -name file
Output./level1dir7/level2dir8/file1
./level1dir7/level2dir5/file1
./level1dir7/level2dir7/file1
./level1dir7/level2dir2/file1
./level1dir7/level2dir10/file1
./level1dir7/level2dir6/file1
./level1dir7/level2dir3/file1
./level1dir7/level2dir4/file1
./level1dir7/file1
. . .
Sie können einen beliebigen Hilfsbefehl auf alles, was „find“ findet, ausführen, indem Sie den Parameter -exec
verwenden. Dieser wird wie folgt aufgerufen:
- find find_parameters -exec command_and_params {} \;
Die {}
werden als Platzhalter für die Dateien verwendet, die „find“ findet. Das \;
wird verwendet, damit „find“ weiß, wo der Befehl endet.
Wir könnten beispielsweise die Dateien im vorherigen Abschnitt finden, die 644
Berechtigungen aufwiesen, und sie so ändern, dass sie 664
Berechtigungen aufweisen:
- cd ~/test
- find . -type f -perm 644 -exec chmod 664 {} \;
Wir könnten dann die Verzeichnisberechtigungen wie folgt ändern:
- find . -type d -perm 755 -exec chmod 700 {} \;
Wenn Sie verschiedene Ergebnisse miteinander verketten möchten, können Sie die Befehle -and
oder -or
verwenden. Das -and
wird angenommen, wenn es weggelassen wird.
- find . -name file1 -or -name file9
Eine Alternative zur Verwendung von find
, ist der Befehl locate
. Dieser Befehl ist oft schneller und kann das gesamte Dateisystem mit Leichtigkeit durchsuchen.
Sie können den Befehl unter Debian oder Ubuntu mit apt
installieren:
- sudo apt install mlocate
Unter CentOS verwenden Sie stattdessen dnf
:
- sudo dnf install mlocate
Der Grund, warum locate
schneller ist als find
, ist, dass er sich auf eine Datenbank der Dateien im Dateisystem verlässt.
Die Datenbank wird normalerweise einmal täglich mit einem Cron-Skript aktualisiert, aber Sie können sie manuell aktualisieren, indem Sie Folgendes eingeben:
- sudo updatedb
Führen Sie diesen Befehl nun aus. Denken Sie daran, dass die Datenbank immer aktuell sein muss, wenn Sie kürzlich erworbene oder erstellte Dateien finden möchten.
Um Dateien mit „locate“ zu finden, verwenden Sie einfach diese Syntax:
- locate query
Sie können die Ausgabe auf einige Arten filtern.
Um beispielsweise nur Dateien zurückzugeben, die die Abfrage selbst enthalten, anstatt jede Datei zurückzugeben, die die Abfrage in den Verzeichnissen aufweist, die zu ihr führen, können Sie das -b
verwenden, um nur den „Basisnamen“ zu suchen:
- locate -b query
Wenn „locate“ nur Ergebnisse zurückgeben soll, die noch im Dateisystem vorhanden sind (die nicht zwischen dem letzten Aufruf von updatedb
und dem aktuellen Aufruf von locate
entfernt wurden), verwenden Sie das Flag -e
:
- locate -e query
Um eine Statistik über die Informationen anzuzeigen, die „locate“ katalogisiert hat, verwenden Sie die Option -S
:
- locate -S
OutputDatabase /var/lib/mlocate/mlocate.db:
3,315 directories
37,228 files
1,504,439 bytes in file names
594,851 bytes used to store database
Sowohl „find“ als auch „locate“ sind gute Methoden, um Dateien auf einem System zu finden. Es ist Ihnen überlassen zu entscheiden, welches dieser Tools in der jeweiligen Situation angemessen ist.
„Find“ und „locate“ sind leistungsfähige Befehle, die durch die Kombination mit anderen Dienstprogrammen über noch verstärkt werden können. Experimentieren Sie mit der Filterung durch Verwendung von Befehlen wie wc
, sort
und grep
.
Hope that this helps!
]]>Um problema comum de usuários ao utilizar uma máquina Linux pela primeira vez é como localizar os arquivos que eles estão procurando.
Este guia irá abordar como usar o comando find
nomeado apropriadamente. Isso irá ajudar a pesquisar arquivos em seu VPS usando uma variedade de filtros e parâmetros. Vamos também abordar brevemente o comando locate
, que pode ser usado para localizar arquivos de outra forma.
A maneira mais óbvia de localizar arquivos é pelo nome.
Para encontrar um arquivo pelo nome, digite:
- find -name "query"
O comando diferencia letras maiúsculas de minúsculas, o que significa que pesquisar por file
é diferente de pesquisar por File
.
Para encontrar um arquivo pelo nome, mas não diferenciar maiúsculas de minúsculas, digite:
- find -iname "query"
Se quiser encontrar todos os arquivos que não aderem a um padrão específico, inverta a pesquisa com -not
ou !
. Se você usar !
, é necessário adicionar um caractere de escape para que o bash não tente interpretá-lo antes do comando find agir:
- find -not -name "query_to_avoid"
ou
- find \! -name "query_to_avoid"
É possível especificar o tipo de arquivo que deseja encontrar com o parâmetro -type
. Funciona desta forma:
- find -type type_descriptor query
Alguns dos descritores mais comuns que você pode usar para especificar o tipo de arquivo estão aqui:
f: arquivo regular
d: diretório
l: link simbólico
c: dispositivos de caracteres
b: dispositivos de bloqueio
Por exemplo, se quisermos encontrar todos os dispositivos de caracteres em nosso sistema, emitimos este comando:
- find / -type c
Output/dev/parport0
/dev/snd/seq
/dev/snd/timer
/dev/autofs
/dev/cpu/microcode
/dev/vcsa7
/dev/vcs7
/dev/vcsa6
/dev/vcs6
/dev/vcsa5
/dev/vcs5
/dev/vcsa4
. . .
Podemos pesquisar todos os arquivos que terminam em .conf
desta forma:
- find / -type f -name "*.conf"
Output/var/lib/ucf/cache/:etc:rsyslog.d:50-default.conf
/usr/share/base-files/nsswitch.conf
/usr/share/initramfs-tools/event-driven/upstart-jobs/mountall.conf
/usr/share/rsyslog/50-default.conf
/usr/share/adduser/adduser.conf
/usr/share/davfs2/davfs2.conf
/usr/share/debconf/debconf.conf
/usr/share/doc/apt-utils/examples/apt-ftparchive.conf
. . .
O comando find oferece várias maneiras de filtrar resultados por tamanho e tempo.
É possível filtrar por tamanho usando o parâmetro -size
.
Adicionamos um sufixo no final do nosso valor que especifica como estamos contando. Estas são algumas opções populares:
c: bytes
k: Kilobytes
M: Megabytes
G: Gigabytes
b: blocos de 512 bytes
Para encontrar todos os arquivos que possuem exatamente 50 bytes, digite:
- find / -size 50c
Para encontrar todos os arquivos com menos de 50 bytes, usamos esta forma:
- find / -size -50c
Para encontrar todos os arquivos com mais de 700 Megabytes, usamos este comando:
- find / -size +700M
O Linux armazena dados de tempo sobre o horário de acesso, horário de modificação e horário de alteração.
Horário de acesso: horário da última vez que um arquivo foi lido ou escrito.
Horário de modificação: horário da última vez que o conteúdo do arquivo foi modificado.
Horário de alteração: horário da última vez que os meta-dados do inode do arquivo foram alterados.
Podemos usá-los com os parâmetros -atime
, -mtime
e -ctime
. Eles podem usar os símbolos + e - para especificar maior que ou menor que, como fizemos com o tamanho.
O valor deste parâmetro especifica quantos dias atrás você gostaria de pesquisar.
Para encontrar arquivos que possuem um horário de modificação de um dia atrás, digite:
- find / -mtime 1
Se quisermos arquivos que foram acessados há menos de um dia, digitamos:
- find / -atime -1
Para obter arquivos que tiveram seus metadados alterados há mais de 3 dias, digite:
- find / -ctime +3
Existem também alguns parâmetros complementares que podem ser usados para especificar minutos em vez de dias:
- find / -mmin -1
Isso irá encontrar os arquivos que foram modificados no sistema no último minuto.
O find também é capaz de fazer comparações com um arquivo de referência e retornar as que são mais novas:
- find / -newer myfile
Também é possível pesquisar arquivos pelo proprietário do arquivo ou proprietário do grupo.
Isso é feito usando os parâmetros -user
e -group
respectivamente. Encontre um arquivo de propriedade do usuário “syslog” digitando:
- find / -user syslog
Da mesma forma, é possível especificar arquivos de propriedade do grupo “shadow” digitando:
- find / -group shadow
Também podemos pesquisar arquivos com permissões específicas.
Se quisermos que haja correspondência com um conjunto específico de permissões, usamos esta forma:
- find / -perm 644
Isso irá encontrar arquivos com as permissões especificadas.
Se quisermos especificar qualquer coisa com pelo menos essas permissões, use esta forma:
- find / -perm -644
Isso irá encontrar qualquer arquivo correspondente que tenha permissões adicionais. Neste caso, um arquivo com permissões de “744” seria correspondente.
Para esta seção, iremos criar uma estrutura de diretório em um diretório temporário. Ela irá conter três níveis de diretórios, com dez diretórios no primeiro nível. Cada diretório (incluindo o diretório temporário) irá conter dez arquivos e dez subdiretórios.
Crie essa estrutura emitindo os seguintes comandos:
- cd
- mkdir -p ~/test/level1dir{1..10}/level2dir{1..10}/level3dir{1..10}
- touch ~/test/{file{1..10},level1dir{1..10}/{file{1..10},level2dir{1..10}/{file{1..10},level3dir{1..10}/file{1..10}}}}
- cd ~/test
Sinta-se livre para verificar as estruturas de diretório com ls
e cd
para entender melhor como as coisas estão organizadas. Quando terminar, retorne ao diretório de teste:
- cd ~/test
Iremos estudar como retornar arquivos específicos dessa estrutura. Vamos testar primeiro um exemplo com apenas uma pesquisa regular de nomes, para comparação:
- find -name file1
Output./level1dir7/level2dir8/level3dir9/file1
./level1dir7/level2dir8/level3dir3/file1
./level1dir7/level2dir8/level3dir4/file1
./level1dir7/level2dir8/level3dir1/file1
./level1dir7/level2dir8/level3dir8/file1
./level1dir7/level2dir8/level3dir7/file1
./level1dir7/level2dir8/level3dir2/file1
./level1dir7/level2dir8/level3dir6/file1
./level1dir7/level2dir8/level3dir5/file1
./level1dir7/level2dir8/file1
. . .
Há muitos resultados. Se canalizarmos o resultado para um contador, vemos que há um total de 111 resultados:
- find -name file1 | wc -l
Output1111
Provavelmente, esse número de resultados é grande de mais para lhe ser útil na maioria dos casos. Vamos tentar restringi-lo.
Especifique a profundidade máxima da pesquisa no diretório de pesquisa de nível superior:
- find -maxdepth num -name query
Para encontrar “file1” apenas nos diretórios de “nível 1”, especifique uma profundidade máxima de 2 (1 para o diretório de nível superior e 1 para os diretórios de nível 1):
- find -maxdepth 2 -name file1
Output./level1dir7/file1
./level1dir1/file1
./level1dir3/file1
./level1dir8/file1
./level1dir6/file1
./file1
./level1dir2/file1
./level1dir9/file1
./level1dir4/file1
./level1dir5/file1
./level1dir10/file1
Essa é uma lista muito mais gerenciável.
Também é possível especificar um diretório mínimo se souber que todos os arquivos existem depois de um determinado ponto abaixo do diretório atual:
- find -mindepth num -name query
Podemos usar isso para encontrar apenas os arquivos no final das ramificações de diretório:
- find -mindepth 4 -name file
Output./level1dir7/level2dir8/level3dir9/file1
./level1dir7/level2dir8/level3dir3/file1
./level1dir7/level2dir8/level3dir4/file1
./level1dir7/level2dir8/level3dir1/file1
./level1dir7/level2dir8/level3dir8/file1
./level1dir7/level2dir8/level3dir7/file1
./level1dir7/level2dir8/level3dir2/file1
. . .
Novamente, devido à nossa estrutura de diretórios ramificada, isso irá retornar um grande número de resultados (1000).
É possível combinar os parâmetros de profundidade mínima e máxima para se concentrar em um intervalo estreito:
- find -mindepth 2 -maxdepth 3 -name file
Output./level1dir7/level2dir8/file1
./level1dir7/level2dir5/file1
./level1dir7/level2dir7/file1
./level1dir7/level2dir2/file1
./level1dir7/level2dir10/file1
./level1dir7/level2dir6/file1
./level1dir7/level2dir3/file1
./level1dir7/level2dir4/file1
./level1dir7/file1
. . .
É possível executar um comando auxiliar arbitrário em tudo o que for encontrado pelo find usando o parâmetro -exec
. Isso é chamado desta forma:
- find find_parameters -exec command_and_params {} \;
O {}
é usado como espaço reservado para os arquivos que encontram correspondências. O \;
é usado para que o comando find saiba onde o comando termina.
Por exemplo, poderíamos encontrar os arquivos na seção anterior que possuía permissões 644
e modificá-los para possuir permissões 664
:
- cd ~/test
- find . -type f -perm 644 -exec chmod 664 {} \;
Em seguida, poderíamos alterar as permissões de diretório desta forma:
- find . -type d -perm 755 -exec chmod 700 {} \;
Se quiser encadear diferentes resultados juntos, use os comandos -and
ou -or
. O -and
é considerado caso seja omitido.
- find . -name file1 -or -name file9
Uma alternativa ao find
é o comando locate
. Esse comando é geralmente mais rápido e pode pesquisar todo o sistema de arquivos com facilidade.
É possível instalar o comando no Debian ou no Ubuntu com o apt
:
- sudo apt install mlocate
No CentOS, use o dnf
:
- sudo dnf install mlocate
A razão pela qual o locate
é mais rápido que o find
é porque ele depende de um banco de dados dos arquivos no sistema de arquivos.
O banco de dados é geralmente atualizado uma vez por dia com um script cron, mas é possível atualizá-lo manualmente digitando:
- sudo updatedb
Execute este comando agora. Lembre-se, o banco de dados deve estar sempre atualizado caso você queira encontrar arquivos recentemente adquiridos ou criados.
Para encontrar arquivos com o locate, use esta sintaxe:
- locate query
É possível filtrar o resultado de algumas maneiras.
Por exemplo, para retornar apenas arquivos que contêm a própria consulta, em vez de retornar cada arquivo que possui a consulta nos diretórios que levam a ele, é possível usar o -b
para pesquisar apenas o “nome base”:
- locate -b query
Para localizar apenas os resultados de retorno que ainda existem no sistema de arquivos (que não foram removidos entre a última chamada de updatedb
e a chamada locate
atual), utilize o sinalizador -e
:
- locate -e query
Para ver estatísticas sobre as informações que o locate catalogou, use a opção -S
:
- locate -S
OutputDatabase /var/lib/mlocate/mlocate.db:
3,315 directories
37,228 files
1,504,439 bytes in file names
594,851 bytes used to store database
Tanto o find quanto o locate representam boas maneiras de encontrar arquivos em seu sistema. Cabe a você decidir qual dessas ferramentas é apropriada em cada situação.
O find e o locate são comandos poderosos que podem ser fortalecidos combinando-os com outros utilitários ao longo das pipelines. Experimente filtrar usando comandos como wc
, sort
e grep
.
Un problema al que los usuarios se enfrentan cuando utilizan por primera vez una máquina Linux es cómo encontrar los archivos que buscan.
En esta guía, se explicará cómo utilizar el comando llamado find
de manera adecuada. Eso lo ayudará a buscar archivos en su VPS utilizando diversos filtros y parámetros. También abordaremos brevemente el comando locate
, que se puede utilizar para buscar archivos de distinta manera.
La forma más evidente de buscar archivos es por nombre.
Para encontrar un archivo por nombre, escriba lo siguiente:
- find -name "query"
Distingue entre mayúsculas y minúsculas, lo que significa que una búsqueda de file
será diferente a File
.
Para encontrar un archivo por nombre, pero ignorar mayúsculas o minúsculas en la consulta, escriba lo siguiente:
- find -iname "query"
Si desea encontrar todos los archivos que no se ajustan a un patrón específico, puede invertir la búsqueda con -not
o !
. Si utiliza !
, debe quitar el carácter de forma que bash no intente interpretarlo antes que find pueda actuar:
- find -not -name "query_to_avoid"
o
- find \! -name "query_to_avoid"
Puede especificar el tipo de archivos que desea encontrar con el parámetro -type
. Funciona de la siguiente manera:
- find -type type_descriptor query
A continuación, encontrará algunos de los descriptores más frecuentes que puede utilizar para especificar el tipo de archivo:
f: archivo normal
d: directorio
l: enlace simbólico
c: dispositivos de caracteres
b: dispositivos de bloque
Por ejemplo, si quisiéramos encontrar todos los dispositivos de caracteres en nuestro sistema, podríamos ejecutar el siguiente comando:
- find / -type c
Output/dev/parport0
/dev/snd/seq
/dev/snd/timer
/dev/autofs
/dev/cpu/microcode
/dev/vcsa7
/dev/vcs7
/dev/vcsa6
/dev/vcs6
/dev/vcsa5
/dev/vcs5
/dev/vcsa4
. . .
Podemos buscar todos los archivos que terminan en .conf
como se muestra a continuación:
- find / -type f -name "*.conf"
Output/var/lib/ucf/cache/:etc:rsyslog.d:50-default.conf
/usr/share/base-files/nsswitch.conf
/usr/share/initramfs-tools/event-driven/upstart-jobs/mountall.conf
/usr/share/rsyslog/50-default.conf
/usr/share/adduser/adduser.conf
/usr/share/davfs2/davfs2.conf
/usr/share/debconf/debconf.conf
/usr/share/doc/apt-utils/examples/apt-ftparchive.conf
. . .
Find le proporciona diversas maneras de filtrar resultados por tamaño y fecha.
Puede filtrar por tamaño utilizando el parámetro -size
.
Añadimos un sufijo al final de nuestro valor con el que se especifica cómo se cuenta. Estas son algunas opciones populares:
c: bytes
k: kilobytes
M: megabytes
G: gigabytes
b: bloques de 512 bytes
Para encontrar todos los archivos que son exactamente de 50 bytes, escriba lo siguiente:
- find / -size 50c
Para encontrar todos los archivos de menos de 50 bytes, podemos utilizar la siguiente forma:
- find / -size -50c
Para encontrar todos los archivos de más de 700 megabytes, podemos utilizar el siguiente comando:
- find / -size +700M
Linux almacena datos de fecha de las fechas de acceso, fechas de modificación y fechas de cambio.
Fechas de acceso: La última vez que se leyó o se escribió un archivo.
Fecha de modificación: La última vez que el contenido del archivo se modificó.
Fecha de cambio: La última vez que se modificaron los metadatos inode del archivo.
Podemos utilizarlos con los parámetros -atime
, -mtime
y -ctime
. Con estos, se pueden usar los signos de más y menos para especificar mayor o menor que, como hicimos con el tamaño.
Con el valor de este parámetro, se especifica cuántos días atrás le gustaría buscar.
Para encontrar archivos que tengan una fecha de modificación de un día atrás, escriba lo siguiente:
- find / -mtime 1
Si queremos archivos a los que se haya accedido hace menos de un día, podemos escribir lo siguiente:
- find / -atime -1
Para obtener archivos cuya metainformación se haya modificado hace más de 3 días, escriba lo siguiente:
- find / -ctime +3
También existen algunos parámetros complementarios que podemos usar para especificar minutos en vez de días:
- find / -mmin -1
Con esto, se mostrarán los archivos que se modificaron en el último minuto en el sistema.
Find también puede hacer comparaciones con un archivo de referencia y mostrar los que son más recientes:
- find / -newer myfile
También puede buscar archivos por su propietario o propietario del grupo.
Para hacerlo, utilice los parámetros -user
y -group
respectivamente. Busque un archivo que sea propiedad del usuario “syslog” ingresando lo siguiente:
- find / -user syslog
De igual manera, podemos especificar archivos que son propiedad del grupo “shadow” escribiendo lo siguiente:
- find / -group shadow
También podemos buscar archivos con permisos específicos.
Si queremos que coincida un conjunto exacto de permisos, utilizamos esta forma:
- find / -perm 644
Eso relacionará los archivos con los permisos especificados.
Si queremos especificar algún elemento con al menos esos permisos, puede utilizar esta forma:
- find / -perm -644
Eso relacionará cualquier archivo que tenga permisos adicionales. En este caso, se relacionaría un archivo con permisos de “744”.
Para esta sección, crearemos una estructura de directorio en un directorio temporal. Contendrá tres niveles de directorios, con diez directorios en el primer nivel. Cada directorio (incluido el directorio temporal) contendrá diez archivos y diez subdirectorios.
Cree esta estructura ejecutando los siguientes comandos:
- cd
- mkdir -p ~/test/level1dir{1..10}/level2dir{1..10}/level3dir{1..10}
- touch ~/test/{file{1..10},level1dir{1..10}/{file{1..10},level2dir{1..10}/{file{1..10},level3dir{1..10}/file{1..10}}}}
- cd ~/test
No dude en consultar las estructuras de directorios con ls
y cd
para entender cómo se organizan los elementos. Cuando termine, regrese al directorio de prueba:
- cd ~/test
Trabajaremos en cómo obtener archivos específicos con esta estructura. Para comparar, primero probaremos un ejemplo con una búsqueda de nombre normal:
- find -name file1
Output./level1dir7/level2dir8/level3dir9/file1
./level1dir7/level2dir8/level3dir3/file1
./level1dir7/level2dir8/level3dir4/file1
./level1dir7/level2dir8/level3dir1/file1
./level1dir7/level2dir8/level3dir8/file1
./level1dir7/level2dir8/level3dir7/file1
./level1dir7/level2dir8/level3dir2/file1
./level1dir7/level2dir8/level3dir6/file1
./level1dir7/level2dir8/level3dir5/file1
./level1dir7/level2dir8/file1
. . .
Aparecen muchos resultados. Si canalizamos el resultado en un contador, observamos que hay 1111 resultados en total:
- find -name file1 | wc -l
Output1111
Probablemente, sean demasiados resultados para que le resulte útil en la mayoría de los casos. Intentemos reducirlo.
Puede especificar la profundidad máxima de la búsqueda en el directorio de búsqueda de nivel superior:
- find -maxdepth num -name query
Para encontrar “file1” solo en los directorios de “level1” y superiores, puede especificar una profundidad máxima de 2 (1 para el directorio de nivel superior y 1 para los directorios de level1):
- find -maxdepth 2 -name file1
Output./level1dir7/file1
./level1dir1/file1
./level1dir3/file1
./level1dir8/file1
./level1dir6/file1
./file1
./level1dir2/file1
./level1dir9/file1
./level1dir4/file1
./level1dir5/file1
./level1dir10/file1
Esa lista es mucho más manejable.
También se puede especificar un directorio mínimo si sabe que todos los archivos existen a partir de cierto punto en el directorio actual:
- find -mindepth num -name query
Podemos utilizarlo para encontrar solo los archivos al final de las ramas de los directorios:
- find -mindepth 4 -name file
Output./level1dir7/level2dir8/level3dir9/file1
./level1dir7/level2dir8/level3dir3/file1
./level1dir7/level2dir8/level3dir4/file1
./level1dir7/level2dir8/level3dir1/file1
./level1dir7/level2dir8/level3dir8/file1
./level1dir7/level2dir8/level3dir7/file1
./level1dir7/level2dir8/level3dir2/file1
. . .
Una vez más, debido a nuestra estructura de directorio con ramificaciones, obtendremos un gran número de resultados (1000).
Puede combinar los parámetros de profundidad min y max para centrarse en un rango más pequeño:
- find -mindepth 2 -maxdepth 3 -name file
Output./level1dir7/level2dir8/file1
./level1dir7/level2dir5/file1
./level1dir7/level2dir7/file1
./level1dir7/level2dir2/file1
./level1dir7/level2dir10/file1
./level1dir7/level2dir6/file1
./level1dir7/level2dir3/file1
./level1dir7/level2dir4/file1
./level1dir7/file1
. . .
Puede ejecutar un comando auxiliar arbitrario en todo lo que se encuentre con find mediante el uso del parámetro -exec
. Eso se invoca de la siguiente manera:
- find find_parameters -exec command_and_params {} \;
El {}
se utiliza como un marcador de posición para los archivos que se encuentren con find. El \;
se utiliza para que find sepa dónde termina el comando.
Por ejemplo, podríamos encontrar los archivos en la sección anterior que tenían 644
permisos y modificarlos para que tengan 664
permisos:
- cd ~/test
- find . -type f -perm 644 -exec chmod 664 {} \;
Luego, podríamos cambiar los permisos del directorio de esta manera:
- find . -type d -perm 755 -exec chmod 700 {} \;
Si desea encadenar diferentes resultados, puede usar los comandos -and
o -or
. Si se omite, se asume el comando -and
.
- find . -name file1 -or -name file9
Una alternativa a utilizar find
es el comando locate
. A menudo, este comando es más rápido y puede realizar una búsqueda en todo el sistema de archivos con facilidad.
Puede instalar el comando en Debian o Ubuntu con apt
:
- sudo apt install mlocate
En CentOS, utilice dnf
en su lugar:
- sudo dnf install mlocate
La razón por la que locate
es más rápido que find
es porque utiliza una base de datos de los archivos en el sistema de archivos.
Normalmente, la base de datos se actualiza una vez al día con una secuencia de comandos de Cron, pero puede actualizarla de forma manual escribiendo lo siguiente:
- sudo updatedb
Ejecute ese comando ahora. Recuerde que la base de datos siempre debe estar actualizada si desea encontrar archivos recientemente adquiridos o creados.
Para encontrar archivos con locate, simplemente utilice la siguiente sintaxis:
- locate query
Puede filtrar el resultado de alguna manera.
Por ejemplo, para solo obtener archivos que contengan la consulta en sí, en vez de obtener cada archivo que contenga la consulta en los directorios en que se encuentre, puede utilizar el -b
solamente para buscar el “nombre de base”:
- locate -b query
Para que con locate solo se obtengan resultados que todavía existen en el sistema de archivos (que no se eliminaron entre la última invocación de updatedb
y la invocación actual de locate
), utilice el indicador -e
:
- locate -e query
Para ver estadísticas sobre la información que se catalogó con locate, utilice la opción -S
:
- locate -S
OutputDatabase /var/lib/mlocate/mlocate.db:
3,315 directories
37,228 files
1,504,439 bytes in file names
594,851 bytes used to store database
Find y locate son buenas maneras de encontrar archivos en su sistema. Le corresponde decidir cuál de estas herramientas es la adecuada para cada situación.
Find y locate son comandos potentes que se pueden reforzar combinándolos con otras utilidades mediante canalizaciones. Experimente con filtros utilizando comandos como wc
, sort
y grep
.
du
command is essential!
You can use the du
command which estimates the directory space usage.
For example, let’s say that we want to check the size of the directories located in the /home
directory. To do that let’s first cd
into the /home
folder:
- cd /home/
And then run the du
command:
- du -h --max-depth=1
[secondary_label Output]
16K ./demo
5.0G ./bobby
28K ./sammy
1.8G ./dev
7.1G .
I like running the du
command with the --max-depth=1
argument which only shows the size of the folders located inside the current directory and not the subfolders.
As you can see from the output the size is not sorted by size, here is how to do that!
]]>Linuxを始めて最初に突き当たる問題は、ファイルの検索方法です。
このガイドは、find
というコマンドの適切な使い方について説明します。これにより、さまざまなフィルターやパラメータを使用してVPS上のファイルを検索できます。また、別の方法でファイルを検索できるlocate
コマンドについても簡単に触れます。
ファイル検索の最も一般的な手段はファイル名の検索です。
名前からファイルを探すには、次のように入力します。
- find -name "query"
大文字小文字を区別します。つまり、file
の検索はFile
の検索とは異なります。
大文字小文字を無視して名前検索するには、次のように入力します。
- find -iname "query"
特定の文字列を除外してファイルを検索するには、-not
または!
で検索を反転させます。!
を使用する際は、findが実行される前にbashが解釈しないよう、この文字をエスケープしておく必要があります。
- find -not -name "query_to_avoid"
または次を入力します。
- find \! -name "query_to_avoid"
検索するファイルの種類を-type
パラメータで指定できます。次のようになります。
- find -type type_descriptor query
ファイルの種類を指定するのに最もよく使用される記述子にはこのようなものがあります。
f:通常のファイル
d:ディレクトリ
l:シンボルリンク
c:キャラクタデバイス
b:ブロックデバイス
たとえば、システム上のキャラクタデバイスをすべて表示するには、このコマンドを発行します。
- find / -type c
Output/dev/parport0
/dev/snd/seq
/dev/snd/timer
/dev/autofs
/dev/cpu/microcode
/dev/vcsa7
/dev/vcs7
/dev/vcsa6
/dev/vcs6
/dev/vcsa5
/dev/vcs5
/dev/vcsa4
. . .
.conf
で終わるファイルをすべて表示するには次のように入力します。
- find / -type f -name "*.conf"
Output/var/lib/ucf/cache/:etc:rsyslog.d:50-default.conf
/usr/share/base-files/nsswitch.conf
/usr/share/initramfs-tools/event-driven/upstart-jobs/mountall.conf
/usr/share/rsyslog/50-default.conf
/usr/share/adduser/adduser.conf
/usr/share/davfs2/davfs2.conf
/usr/share/debconf/debconf.conf
/usr/share/doc/apt-utils/examples/apt-ftparchive.conf
. . .
Findは、結果をサイズと時刻でフィルタリングするさまざまな方法を提供します。
サイズでフィルターするには-size
パラメータを使用します。
値の後ろにカウント方法を示す接尾辞を付けて単位を指定します。一般的なオプションは次のとおりです。
c: Byte
k: KByte
M: MByte
G: GByte
b: 512 Byteブロック
50Byteちょうどのサイズのファイルをすべて検索するには、次を入力します。
- find / -size 50c
50Byte未満のファイルをすべて検索するには、このフォームを使用します。
- find / -size -50c
700MByte以上のファイルをすべて検索するには、このコマンドを使用します。
- find / -size +700M
Linuxは、Access Time、Modification Time、Change Timeに関する時刻データを保存します。
Access Time: ファイルが最後に読み込み、書き込みされた時刻。
Modification Time: ファイルの内容が最後に変更された時刻。
Change Time: ファイルのinodeメタデータが最後に変更された時刻。
これらの検索には-atime
、-mtime
、-ctime
パラメータを使用します。サイズの場合と同様に、+/-記号を使用して、以前/以内を指定します。
このパラメータの値は、何日前に保存されたファイルを検索するかを指定します。
1日前に内容が変更されたファイルを検索するには、次のように入力します。
- find / -mtime 1
1日以内にアクセスしたファイルを検索するには、次のように入力します。
- find / -atime -1
メタ情報の最終変更日から3日以上経過したファイルを検索するには、次のように入力します。
- find / -ctime +3
類似のパラメータを使用して、日数ではなく分数を指定することもできます。
- find / -mmin -1
これにより、1分以内に内容が変更されたファイルがわかります。
また、Findを使用して参照ファイルとの比較を行い、新しい方のファイルを返すこともできます。
- find / -newer myfile
ファイルの所有者やグループ所有者でファイルを検索することもできます。
これを行うには、それぞれ-user
パラメータと-group
パラメータを使用します。「syslog」ユーザーが所有するファイルを検索するには次を入力します。
- find / -user syslog
同様に、「shadow」グループが所有するファイルを指定するには次のように入力します。
- find / -group shadow
また、特定のパーミッションでファイルを検索することもできます。
あるパーミッションに完全一致するファイルをマッチさせる場合は、次の体裁で入力します。
- find / -perm 644
この場合、指定されたパーミッションと完全に一致するファイルがマッチします。
これらのパーミッション以上のファイルを指定するには、次の体裁で入力します。
- find / -perm -644
この場合、指定より上のパーミッションを持つファイルがマッチします。このインスタンスでは、パーミッション「744」のファイルがマッチします。
このセクションでは、一時ディレクトリにディレクトリ構造を作成します。このディレクトリ構造に、3階層のディレクトリと、1階層目に10個のディレクトリを配置します。各ディレクトリ(一時ディレクトリを含む)には10個のファイルと10個のサブディレクトリを配置します。
次のコマンドでこの構造を作成します。
- cd
- mkdir -p ~/test/level1dir{1..10}/level2dir{1..10}/level3dir{1..10}
- touch ~/test/{file{1..10},level1dir{1..10}/{file{1..10},level2dir{1..10}/{file{1..10},level3dir{1..10}/file{1..10}}}}
- cd ~/test
ls
とcd
で各ディレクトリを自由に確認して回り、その構造を把握してください。終わったら、testディレクトリに戻ります。
- cd ~/test
この構造から特定のファイルを返す方法について見ていきます。比較のためにまず、通常の名前検索で例を試してみましょう。
- find -name file1
Output./level1dir7/level2dir8/level3dir9/file1
./level1dir7/level2dir8/level3dir3/file1
./level1dir7/level2dir8/level3dir4/file1
./level1dir7/level2dir8/level3dir1/file1
./level1dir7/level2dir8/level3dir8/file1
./level1dir7/level2dir8/level3dir7/file1
./level1dir7/level2dir8/level3dir2/file1
./level1dir7/level2dir8/level3dir6/file1
./level1dir7/level2dir8/level3dir5/file1
./level1dir7/level2dir8/file1
. . .
結果がたくさん表示されます。出力をカウンタにパイプすると、計1111個の結果があるのがわかります。
- find -name file1 | wc -l
Output1111
これでは結果が多すぎて、ほとんどの場合役に立たないでしょう。絞り込み検索をしてみましょう。
検索するディレクトリの最大の深さを指定できます。
- find -maxdepth num -name query
「レベル1」ディレクトリかそれ以上の階層でのみ「file1」を検索するには、最大深さ2(トップレベルディレクトリ1、レベル1ディレクトリ1)を指定します。
- find -maxdepth 2 -name file1
Output./level1dir7/file1
./level1dir1/file1
./level1dir3/file1
./level1dir8/file1
./level1dir6/file1
./file1
./level1dir2/file1
./level1dir9/file1
./level1dir4/file1
./level1dir5/file1
./level1dir10/file1
これでとても管理しやすい一覧になりました。
現在のディレクトリ配下の、一定の階層から先にすべてのファイルが存在するとわかっていれば、最小ディレクトリを指定することもできます。
- find -mindepth num -name query
これにより末端ディレクトリのファイルだけを検索できます。
- find -mindepth 4 -name file
Output./level1dir7/level2dir8/level3dir9/file1
./level1dir7/level2dir8/level3dir3/file1
./level1dir7/level2dir8/level3dir4/file1
./level1dir7/level2dir8/level3dir1/file1
./level1dir7/level2dir8/level3dir8/file1
./level1dir7/level2dir8/level3dir7/file1
./level1dir7/level2dir8/level3dir2/file1
. . .
ここでも、分岐ディレクトリ構造の為、多数の結果(1000)が返されます。
最小・最大階層パラメータを組み合わせれば、焦点を狭い範囲に絞り込めます。
- find -mindepth 2 -maxdepth 3 -name file
Output./level1dir7/level2dir8/file1
./level1dir7/level2dir5/file1
./level1dir7/level2dir7/file1
./level1dir7/level2dir2/file1
./level1dir7/level2dir10/file1
./level1dir7/level2dir6/file1
./level1dir7/level2dir3/file1
./level1dir7/level2dir4/file1
./level1dir7/file1
. . .
-exec
パラメータを使用すると、すべての検索コマンド上で任意のヘルパーコマンドが実行できます。このように呼び出します。
- find find_parameters -exec command_and_params {} \;
{}
は、findで合致したファイルのプレースホルダーです。\;
は、findにコマンドの終わりを示します。
たとえば、前のセクションで見たファイルのパーミッション644
を664
に変更します。
- cd ~/test
- find . -type f -perm 644 -exec chmod 664 {} \;
また、ディレクトリのパーミッションも次のように変更できます。
- find . -type d -perm 755 -exec chmod 700 {} \;
異なる結果を連結させるには、-and
または-or
コマンドを使用します。省略されていれば-and
と想定します。
- find . -name file1 -or -name file9
find
の代替コマンドとして、locate
コマンドがあります。このコマンドの方が早いことが多く、ファイルシステム全体の検索が容易になります。
このコマンドは、DebianまたはUbuntuにapt
でインストールできます。
- sudo apt install mlocate
CentOSでは、dnf
を使用します。
- sudo dnf install mlocate
locate
がfind
より早いのは、それがファイルシステムのファイルのデータベースに依存しているためです。
データベースは通常、cronスクリプトで1日に一度更新されますが、次を入力して手動でも更新できます。
- sudo updatedb
このコマンドを今すぐ実行します。最近取得したファイルや作成されたばかりのファイルを検索するなら、データベースを常に最新にしておく必要があります。
locateでファイルの検索に使用する構文はこれだけです。
- locate query
出力をフィルターする方法がいくつかあります。
たとえば、クエリ自体に含まれるファイルのみを返すなら、上位ディレクトリ付きでファイルを一つ一つ返すのではなく、-b
を使用して「basename」だけを検索できます。
- locate -b query
locateが、ファイルシステムにまだ存在する(直近のupdatedb
コールと現在のlocate
コール間で削除されなかった)結果のみを返すようにするには、-e
フラグを使用します。
- locate -e query
locateでカタログ化された情報を統計表示するには、-S
オプションを選択します。
- locate -S
OutputDatabase /var/lib/mlocate/mlocate.db:
3,315 directories
37,228 files
1,504,439 bytes in file names
594,851 bytes used to store database
find、locateの両コマンドとも、システム上のファイルの検索手段としてお勧めします。状況に応じてどちらのツールが適当かご自身で判断してください。
find、locateの両コマンドとも、他のユーティリティとパイプラインでつないで増強できる、パワフルなコマンドです。wc
、sort
、grep
などのコマンドを使用してフィルタリングを試しましょう。
Satu masalah yang dialami pengguna saat kali pertama berhadapan dengan mesin Linux adalah cara menemukan berkas yang mereka cari.
Panduan ini akan membahas cara menggunakan perintah aptly bernama find
. Ini akan membantu Anda mencari berkas di VPS menggunakan berbagai filter dan parameter. Kita juga akan membahas secara singkat perintah locate
, yang dapat digunakan untuk mencari berkas dengan cara lain.
Cara paling mudah dimengerti untuk mencari berkas adalah menurut namanya.
Untuk mencari berkas menurut nama, ketikkan:
- find -name "query"
Perintah ini peka huruf kapital, yang berarti mencari berkas
berbeda dengan mencari Berkas
.
Untuk mencari berkas menurut nama, tetapi mengabaikan huruf kapital kueri, ketikkan:
- find -iname "query"
Jika Anda ingin menemukan semua berkas yang tidak mematuhi pola tertentu, Anda dapat membalik pencarian dengan -not
atau!
. Jika menggunakan!
, Anda harus meloloskan karakter agar bash tidak mencoba menafsirkannya sebelum find bertindak:
- find -not -name "query_to_avoid"
Atau
- find \! -name "query_to_avoid"
Anda dapat menentukan tipe berkas yang ingin ditemukan dengan parameter -type
. Cara kerjanya seperti ini:
- find -type type_descriptor query
Beberapa deskriptor paling umum yang dapat Anda gunakan untuk menentukan tipe berkas adalah:
f: berkas reguler
d: direktori
l: tautan simbolis
c: perangkat karakter
b: perangkat blok
Misalnya, jika ingin menemukan semua perangkat karakter di sistem, kita dapat memberikan perintah ini:
- find / -type c
Output/dev/parport0
/dev/snd/seq
/dev/snd/timer
/dev/autofs
/dev/cpu/microcode
/dev/vcsa7
/dev/vcs7
/dev/vcsa6
/dev/vcs6
/dev/vcsa5
/dev/vcs5
/dev/vcsa4
. . .
Kita dapat mencari semua berkas yang berakhiran .conf
seperti ini:
- find / -type f -name "*.conf"
Output/var/lib/ucf/cache/:etc:rsyslog.d:50-default.conf
/usr/share/base-files/nsswitch.conf
/usr/share/initramfs-tools/event-driven/upstart-jobs/mountall.conf
/usr/share/rsyslog/50-default.conf
/usr/share/adduser/adduser.conf
/usr/share/davfs2/davfs2.conf
/usr/share/debconf/debconf.conf
/usr/share/doc/apt-utils/examples/apt-ftparchive.conf
. . .
Find memberi Anda beragam cara untuk memfilter hasil menurut ukuran dan waktu.
Anda dapat memfilter menurut ukuran dengan parameter -size
.
Kita menambahkan akhiran di belakang nilai yang menentukan cara perhitungan. Inilah beberapa opsi yang populer:
c: bita
k: Kilobita
M: Megabita
G: Gigabita
b: blok 512-bita
Untuk menemukan semua berkas yang persis 50 bita, ketikkan:
- find / -size 50c
Untuk menemukan semua berkas yang kurang dari 50 bita, kita dapat menggunakan bentuk ini sebagai gantinya:
- find / -size -50c
Untuk menemukan semua berkas yang lebih dari 700 Megabita, kita dapat menggunakan perintah ini:
- find / -size +700M
Linux menyimpan data waktu tentang waktu akses, waktu modifikasi, dan waktu perubahan.
Waktu Akses: Terakhir kali berkas dibaca atau ditulis.
Waktu Modifikasi: Terakhir kali isi berkas dimodifikasi.
Waktu Perubahan: Terakhir kali metadata inode berkas diubah.
Kita dapat menggunakannya dengan parameter -atime
, -mtime
, dan -ctime
. Simbol plus dan minus dapat digunakan untuk menentukan lebih dari atau kurang dari, seperti yang kita lakukan dengan ukuran.
Nilai parameter ini menentukan jumlah hari lampau yang ingin Anda cari.
Untuk menemukan berkas yang memiliki waktu modifikasi sehari yang lalu, ketikkan:
- find / -mtime 1
Jika kita ingin berkas yang diakses kurang dari sehari yang lalu, kita dapat mengetikkan:
- find / -atime -1
Untuk mendapatkan berkas dengan informasi meta yang diubah lebih dari 3 hari yang lalu, ketikkan:
- find / -ctime +3
Ada juga beberapa parameter pendamping yang dapat kita gunakan untuk menentukan dalam menit, bukan dalam hari:
- find / -mmin -1
Parameter ini akan memberikan berkas yang telah dimodifikasi di sistem dalam menit terakhir.
Find juga dapat melakukan perbandingan terhadap berkas referensi dan menampilkan berkas yang lebih baru:
- find / -newer myfile
Anda juga dapat mencari berkas menurut pemilik berkas atau pemilik grup.
Anda melakukannya masing-masing dengan menggunakan parameter -user
dan -group
. Temukan berkas yang dimiliki oleh pengguna “syslog” dengan memasukkan:
- find / -user syslog
Demikian pula, kita dapat menentukan berkas yang dimiliki oleh grup “shadow” dengan mengetikkan:
- find / -group shadow
Kita juga dapat mencari berkas dengan izin tertentu.
Jika kita ingin mencocokkan dengan seperangkat izin secara persis, kita gunakan bentuk ini:
- find / -perm 644
Perintah ini akan mencocokkan secara persis berkas dengan beberapa izin yang ditentukan.
Jika ingin menentukan sesuatu setidaknya dengan izin itu, Anda dapat menggunakan bentuk ini:
- find / -perm -644
Bentuk ini akan mencocokkan berkas yang memiliki izin tambahan. Berkas dengan izin “744” akan cocok dengan instans ini.
Untuk bagian ini, kita akan membuat struktur direktori dalam suatu direktori sementara. Isinya akan berupa tiga tingkat direktori, dengan sepuluh direktori di tingkat pertama.mprovemenimprovement. Setiap direktori (termasuk direktori sementara) akan berisi sepuluh berkas dan sepuluh subdirektori.
Buatlah struktur ini dengan mengeluarkan perintah berikut:
- cd
- mkdir -p ~/test/level1dir{1..10}/level2dir{1..10}/level3dir{1..10}
- touch ~/test/{file{1..10},level1dir{1..10}/{file{1..10},level2dir{1..10}/{file{1..10},level3dir{1..10}/file{1..10}}}}
- cd ~/test
Silakan periksa struktur direktori dengan ls
dan cd
untuk mengatur cara mengorganisirnya. Bila Anda selesai, kembalilah ke direktori pengujian:
- cd ~/test
Kita akan membahas cara menghasilkan berkas tertentu dari struktur ini. Mari kita coba suatu contoh terlebih dahulu dengan mencari nama reguler untuk perbandingan:
- find -name file1
Output./level1dir7/level2dir8/level3dir9/file1
./level1dir7/level2dir8/level3dir3/file1
./level1dir7/level2dir8/level3dir4/file1
./level1dir7/level2dir8/level3dir1/file1
./level1dir7/level2dir8/level3dir8/file1
./level1dir7/level2dir8/level3dir7/file1
./level1dir7/level2dir8/level3dir2/file1
./level1dir7/level2dir8/level3dir6/file1
./level1dir7/level2dir8/level3dir5/file1
./level1dir7/level2dir8/file1
. . .
Ada banyak hasilnya. Jika kita menyalurkan keluaran ke dalam penghitung, kita dapat melihat bahwa seluruhnya ada 1111 hasil:
- find -name file1 | wc -l
Output1111
Hasil ini mungkin terlalu banyak untuk Anda gunakan dalam keadaan umum. Mari kita coba mempersempitnya.
Anda dapat menentukan kedalaman maksimum pencarian di direktori pencarian tingkat teratas:
- find -maxdepth num -name query
Untuk menemukan direktori “file1” hanya dalam direktori “level1” ke atas, Anda dapat menentukan kedalaman maksimum sebesar 2 (1 untuk direktori tingkat teratas dan 1 untuk direktori level1):
- find -maxdepth 2 -name file1
Output./level1dir7/file1
./level1dir1/file1
./level1dir3/file1
./level1dir8/file1
./level1dir6/file1
./file1
./level1dir2/file1
./level1dir9/file1
./level1dir4/file1
./level1dir5/file1
./level1dir10/file1
Daftar itu menjadi jauh lebih mudah diatur.
Anda juga dapat menentukan direktori minimum jika mengetahui bahwa semua berkas yang ada melewati titik tertentu dalam direktori saat ini:
- find -mindepth num -name query
Kita dapat menggunakan ini untuk menemukan berkas yang ada di akhir cabang direktori saja:
- find -mindepth 4 -name file
Output./level1dir7/level2dir8/level3dir9/file1
./level1dir7/level2dir8/level3dir3/file1
./level1dir7/level2dir8/level3dir4/file1
./level1dir7/level2dir8/level3dir1/file1
./level1dir7/level2dir8/level3dir8/file1
./level1dir7/level2dir8/level3dir7/file1
./level1dir7/level2dir8/level3dir2/file1
. . .
Sekali lagi, karena struktur direktori pencabangan, ini akan mengembalikan hasil yang sangat banyak (1000).
Anda dapat mengombinasikan parameter kedalaman minimum dan maksimum untuk memfokuskan pada rentang yang sempit:
- find -mindepth 2 -maxdepth 3 -name file
Output./level1dir7/level2dir8/file1
./level1dir7/level2dir5/file1
./level1dir7/level2dir7/file1
./level1dir7/level2dir2/file1
./level1dir7/level2dir10/file1
./level1dir7/level2dir6/file1
./level1dir7/level2dir3/file1
./level1dir7/level2dir4/file1
./level1dir7/file1
. . .
Anda dapat mengeksekusi perintah pembantu arbiter atas segala sesuatu yang dianggap cocok menggunakan parameter -exec
. Parameter itu dipanggil seperti ini:
- find find_parameters -exec command_and_params {} \;
{}
digunakan sebagai penampung berkas yang dianggap cocok. \;
digunakan agar mengetahui akhir perintah.
Misalnya, kita dapat menemukan berkas di bagian sebelumnya yang memiliki 644
izin dan memodifikasinya agar memiliki 664
izin:
- cd ~/test
- find . -type f -perm 644 -exec chmod 664 {} \;
Kita nanti dapat mengubah izin direktori seperti ini:
- find . -type d -perm 755 -exec chmod 700 {} \;
Jika ingin merangkai hasil yang berbeda, Anda dapat menggunakan perintah -and
atau -or
. -and
dianggap ada jika tidak disertakan.
- find . -name file1 -or -name file9
Alternatif penggunaan find
adalah perintah locate
. Perintah ini sering kali lebih cepat dan dapat mencari ke seluruh sistem berkas dengan mudah.
Anda dapat menginstal perintah di Debian atau Ubuntu dengan apt
:
- sudo apt install mlocate
Di CentOS, gunakan dnf
sebagai gantinya:
- sudo dnf install mlocate
Alasan locate
lebih cepat daripada find
adalah karena mengandalkan basis data berkas di sistem berkas.
Basis data biasanya diperbarui sekali sehari dengan skrip cron, tetapi Anda dapat memperbaruinya secara manual dengan mengetikkan:
- sudo updatedb
Jalankan perintah ini sekarang. Ingat, basis data harus selalu diperbarui jika Anda ingin menemukan berkas yang baru-baru ini diperoleh atau dibuat.
Untuk menemukan berkas dengan locate, cukup gunakan sintaks ini:
- locate query
Anda dapat memfilter keluaran dalam beberapa cara.
Misalnya, agar hanya mengembalikan berkas yang berisi kueri itu sendiri, bukan mengembalikan setiap berkas yang memiliki kueri dalam direktori yang mengarah ke sana, Anda dapat menggunakan -b
untuk mencari "basename"saja:
- locate -b query
Agar locate hanya mengembalikan hasil yang masih ada di sistem berkas (yang belum dihapus di antara panggilan updatedb
dan panggilan locate
saat ini), gunakan bendera -e
:
- locate -e query
Untuk melihat statistik tentang informasi yang dikatalogkan oleh locate, gunakan opsi -S
:
- locate -S
OutputDatabase /var/lib/mlocate/mlocate.db:
3,315 directories
37,228 files
1,504,439 bytes in file names
594,851 bytes used to store database
Find dan locate adalah cara bagus untuk menemukan berkas di sistem Anda. Anda bebas memilih untuk menentukan alat yang sesuai dengan setiap situasi.
Find dan locate adalah perintah berguna yang dapat diperkuat dengan mengombinasikannya dengan utilitas lain melalui berbagai saluran. Bereksperimenlah dengan melakukan filter menggunakan perintah seperti wc
, sort
, dan grep
.
Here is how you could transfer a Docker image from one server to another without pushing the image to an external Docker repository!
]]>If you wanted to see the SSL certificate information for a specific website, you could do that via your browser, by clicking on the green padlock and then click on Certificate
which would open a modal with all of the information about the SSL certificate like the Common Names, the Organization that issued the certificate, the expiry date and etc.
Here’s how to do the same thing via your command line directly!
]]>Here is an example of how to generate a Unix timestamp using the date
command:
date +%s
This will output a long string of numbers like this:
1606319820
This number represents the seconds since 00:00:00 UTC on 1 January 1970.
So here’s how to convert Unix timestamps to human readable date format in Bash!
]]>Here is an example script, where we expect the user to enter a value when asked to do so:
#!/bin/bash
echo "Enter your name: "
read name
echo "Hello ${name}"
If you run the script, you will be able to press enter, and the output would be just Hello
.
Here’s how to check if a variable is empty in a Bash script!
]]>I need to commit an empty directory to my Git project, but when I create a new directory with:
- mkdir my_dir
And then check the status with:
- git status
Git says that there is nothing to commit, so running git add .
does not do anything.
How can I add an empty directory/folder to my Git repository?
]]>git checkout -b branch_name
and I would realize that I’ve made a typo or I would come up with a better name for the branch later on.
If I have just created the branch it is ok as I can create a new one, but sometimes I would notice this after a couple of commits.
So here’s how you could rename a local Git branch via your command line!
]]>The sar
command allows you to capture the utilization of your resources like RAM, CPU, Disk I/O and etc.
In this post, I will show you how to install and configure sar
!
cat
and tail
to check your server logs.
Here I will show you how to check the logs of your Kubernetes pods for both running and crashed pods using the kubectl
command.
sudo dnf install mariadb-server
command does not install the latest stable MariaDB version.
Here’s how you could install the latest MariaDB available on CentOS 8!
]]>Universally unique identifiers (UUIDs) are 128-bit numbers that are accepted as being unique on the local system they are created on as well as among the UUIDs created on other systems in the past as well as the future. Because of their uniqueness, they come in handy in situations where an auto incremented primary key can fall short.
Because of their uniqueness, UUIDs are well suited for generating test data. Need a random string? A UUID is fine. What about an email? UUID@UUID.com
is great. Need a bunch of random string? UUIDs will be unique, making them easy to track down as they move through a system.
To generate universally unique identifiers from the command-line interface, you can use the uuidgen
utility,
In this tutorial you’ll use uuidgen
and some shell scripting to generate UUIDs and some sample data.
The uuidgen
command is often already installed on Unix-like operating systems like Linux and macOS. If it’s not, you can install it through your package manager. On Ubuntu and Debian systems, install the uuid-runtime
package.
First, update your system’s list of available packages using the apt update
command:
- sudo apt update
Next install the uuid-runtime
package using the apt
package manager:
- sudo apt install uuid-runtime
To generate a single UUID, run the uuidgen
command without any arguments. Out of the box, uuidgen
will generate a random UUID each time it runs.
Execute the following command in your terminal:
- uuidgen
You’ll see output similar to the following, although your UUID will be different:
Outputa522f494-92ce-44e9-b1a3-f891baed8d60
Note: The macOS version of uuidgen
does function a bit differently than that of the Linux version in that is returns UUIDs in all capital letters.
You can also generate time-based and hash-based UUIDs, but generally speaking, the random values are sufficient for most cases.
You may want to generate multiple UUIDs at once, which you’ll explore next.
To generate a bunch of UUIDs at once, you’ll leverage a small bit of shell scripting using the for
loop to execute the uuidgen
command multiple times.
For example, to generate 10 UUIDs, execute the following command:
- for i in {1..10}; do uuidgen; done
You’ll see 10 UUIDs printed to the screen:
Output834efdb6-6044-4b44-8fcb-560710936f37
e8fa8d54-641a-4d7b-9422-91474d713c62
dff59ac0-4d80-4b96-85c4-14f3a118e7fe
511fea83-9f5f-4606-85ec-3d769da4bf63
3bc82ef7-1138-4f97-945a-08626a42a648
a33abc11-264e-4bbb-82e8-b87226bb4383
2a38839e-3b0d-47f0-9e60-d6b19c0978ad
74dca5e8-c702-4e70-ad16-0a16a64d55fa
cd13d088-21cf-4286-ae61-0643d321dd9e
9aec3d5a-a339-4f24-b5a3-8419ac8542f2
You can swap the 10
out for the number you’d like.
Based on the unique nature of UUIDs, you don’t have to worry about any duplicates in your generated data. Now let’s look at using UUIDs in different ways.
If you wanted to generate a list of comma-separated values (CSV) with 2 UUIDs per line, you would use the echo
command to print two UUIDs during each iteration of the for
loop.
Execute the following command:
- for i in {1..10}; do echo `uuidgen`,`uuidgen`; done
You’ll get this output:
Output63b1146f-9e7c-4e1f-82eb-3fe378e203df,ed9d6201-e5b2-4410-9ab1-35c8ca037994
8d3981b6-f112-4f21-ac4b-44791e279b2a,eb63310e-d436-44fa-80c6-65721a300a2b
0eddfe24-1c2e-43a1-b2c2-9d3af6bad837,62ef1782-76a2-4b3c-ac69-1c2d02f65789
29f18766-fc9d-46a4-a1d0-e112738edb30,b6bd303d-1148-4f46-bec7-d7e4cb6e4f03
865bcf30-6a8b-49d6-8b27-8dc51620adf7,972b0959-4270-4683-b19b-360b2605f2d0
0d82d54b-566a-45d1-b3a8-5da1a88bceb3,1c67a802-9647-46b1-bde4-3053699b27f9
778b5415-3e1f-4bc5-a349-499459ac4ab7,7e1a2081-c742-4882-9154-e5d2a4af630c
e6cc95bd-3ee1-43cb-bea1-51783de5fc57,5088d3a3-ab67-4684-8761-e48bb14596ec
a7453bc0-b5e5-41a3-9ed4-cf4d8e0908a2,957ef50f-7889-4335-9f40-17878e3d20fe
3689362d-588a-409e-bd2c-d6fdaa361574,9ffe7c8d-9afb-4b24-a5b7-b29a06f6fac7
Using the same approach, you can generate data that looks like email addresses by making a small tweak to the echo
statement:
- for i in {1..10}; do echo `uuidgen`@`uuidgen`.com; done
You’ll receive this output:
Output7dd44050-9df4-43aa-b3b4-3b47eff8fc31@3052e93c-95d1-40f5-b468-3d4e06dd208b.com
cca71187-f666-46ff-81c6-eb3b72ff6972@30f4c9a8-712e-4f4c-ad3a-4b55ef85eee0.com
6ff086ad-493d-4b3a-8ed1-970239d7125b@8302d772-4deb-43d1-8901-0a3b4f747b55.com
f9813daa-6a8e-4543-8708-d42cefdda20a@d586854c-7df9-4046-89f8-51f960973afb.com
a7e9e43b-d2b1-4415-b73d-ff72b713e45f@a7c56c2c-df25-44bc-872d-a893f750b54d.com
0d1d13fe-777d-44d8-b1b2-302ca1e48aa1@7c2d8e6a-fa8b-4fa3-a0ef-8360aa42e730.com
f85d0772-22d2-43d0-8d71-4e6714c2bb20@fb4f74fe-f9f9-4e86-b31d-f148344a97e0.com
f46eb868-0a99-4291-98f2-19d95f1e9fbb@37ef072d-c515-4145-8b8a-edf32ec18bd2.com
eaa4a63e-2646-427a-a892-f8027c2791ed@33daf102-2b5b-4070-88c5-261fe5d96cfa.com
d75f6720-b249-4395-bcc7-9ffe2b67cabb@457b04b4-3c15-4b77-aae2-9afd6803bcfe.com
These aren’t real email addresses that you can validate, but you can tweak the output one more time, and swap the second uuidgen
for a disposable email address domain, like [mailinator.com](https://mailinator.com], you will not only have a list of realistic-looking data, but it will be a list email addresses you could actually use or monitor in tests. Try the following command:
- for i in {1..10}; do echo `uuidgen`@mailinator.com; done
This time you’ll see output like this:
Output4ba50929-520b-49f7-996d-e369be5d6232@mailinator.com
16deaeae-64bd-45f0-9f73-b32d41ca1bfb@mailinator.com
743701e8-0dc5-4851-8fc4-24d155755bdc@mailinator.com
adff0015-c535-431a-970f-98ffd1fc21eb@mailinator.com
6516fcb3-e54f-4800-a6cc-11d50d756f28@mailinator.com
8a9c5252-bd0c-4c3b-a7c9-4b60ebcc4294@mailinator.com
eed94fd6-b075-493c-8d8e-3acae90d5629@mailinator.com
f4ab80d2-85ca-4722-a260-0f84c37051fd@mailinator.com
53ead1d0-cc70-410f-a91a-4a79b339fba2@mailinator.com
b208e103-d7f1-4f6d-838d-530d6339dce7@mailinator.com
Finally, to save the output of this command to a file, you can append > /path/to/some/file
to pipe the output:
- for i in {1..10}; do echo `uuidgen`@mailinator.com; done > /tmp/emails.txt
Then use the cat
command to view the file you just created:
- cat /tmp/emails.txt
The file is displayed on your screen:
Output826119d2-f590-4fa3-ba7e-0717869d40b1@mailinator.com
795fec1a-76fe-4fed-8a06-ed517c1a5e7d@mailinator.com
14a502ad-0aa9-40e5-a46f-5806264b5316@mailinator.com
c6c2a588-7cce-4675-a490-0101d7bcc614@mailinator.com
7346c15b-0c92-44c4-a854-5de18c0c202d@mailinator.com
c67a535a-e28d-43b1-b553-c203bc22a821@mailinator.com
76d22d18-0f09-405d-9903-eb44ec93b605@mailinator.com
2b631756-21e6-4d95-873b-3245797f9028@mailinator.com
aab686e8-540e-43e9-9e24-ca04fbf4d414@mailinator.com
a577e9c9-0ad1-4934-b5f1-17b68938fff8@mailinator.com
You can use the same approach to save any of the other examples in this tutorial to files.
Universally unique identifiers are more robust than a random number. Their uniqueness makes them quite powerful. Combined with some light shell scripting on the command-line interface, you’re able to generate useful data, without needing to load up your favorite programming language’s package repository.
Next time you’re in need of a UUID, save yourself the search for “online UUID generator” and use your system’s uuidgen
command. To learn more about your system’s particular implementation of uuidgen
, type man uuidgen
in your terminal to view its documentation…
Tools? Tips? Any help would be much appreciated.
]]>Here are the steps that you need to follow in order to get the newest changes from the original repository pulled into your fork!
]]>I am supporting a few projects on GitHub, I’m at a point where I have hundreds of branches across all projects and deleting the branches manually is not really an option.
Does anyone have a script on hand to delete all merged branches which do not have any new commits since a specific date?
Any help will be appreciated!
]]>ps: AWS doesn’t have this annoyance.
]]>lxc config device add store store90 proxy listen=tcp:0.0.0.0:80 connect=tcp:127.0.0.1:80
working, i can open defaut page, modified to verify… the host is in virtuelbox with ip 192.168.1.33 and http://192.168.1.33 work like a charm
so, installed proftpd server,and want to connect to it from my network
i was modified proftpd MasqueradeAddress 192.168.1.33 to rely on external host address, and defined
PassivePorts 49152 65534
tried some proxy setting, but connexion fail when passing to passive connection.
filezilla log:
Status: Connecting to 192.168.1.33:21...
Status: Connection established, waiting for welcome message...
Status: Insecure server, it does not support FTP over TLS.
Status: Logged in
Status: Retrieving directory listing...
Command: PWD
Response: 257 "/" is the current directory
Command: TYPE I
Response: 200 Type set to I
Command: PASV
Response: 227 Entering Passive Mode (192,168,1,33,216,239).
Command: LIST
Error: The data connection could not be established: ECONNREFUSED - Connection refused by server
Error: Connection timed out after 20 seconds of inactivity
Error: Failed to retrieve directory listing
in the host, the devices network are:
devices:
eth0:
name: eth0
nictype: bridged
parent: lxdbr0
type: nic
root:
path: /
pool: default
type: disk
store80:
connect: tcp:127.0.0.1:80
listen: tcp:0.0.0.0:80
type: proxy
storeftp21:
connect: tcp:127.0.0.1:21
listen: tcp:0.0.0.0:21
type: proxy
storeftp22:
connect: tcp:127.0.0.1:22
listen: tcp:0.0.0.0:22
type: proxy
storeftppassive:
connect: tcp:127.0.0.1:49152-65534
listen: tcp:0.0.0.0:49152-65534
type: proxy
any help ?? or link to real working solution to set proxy for lxd with proftpd ? i searched and founded very obscur solutions, nothing that i can understand now… please help…
]]>When I try to run “mysql” command through bash window I got the error below. Most people say it is because of missing configuration in SQL ports. But when I restart the droplet everything returns back to normal, at least for a while until another random crash occurs.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
I’ve also checked SQL logs. There is a reoccuring pattern. The below error is logged everytime I have this problem. Can anybody give me some pointers about what it means?
2020-08-10 4:13:38 0 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown
2020-08-10 4:13:38 0 [Note] Event Scheduler: Purging the queue. 0 events
2020-08-10 4:13:38 0 [Note] InnoDB: FTS optimize thread exiting.
2020-08-10 4:13:38 0 [Note] InnoDB: Starting shutdown...
2020-08-10 4:13:38 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2020-08-10 4:13:38 0 [Note] InnoDB: Buffer pool(s) dump completed at 200810 4:13:38
2020-08-10 4:13:40 0 [Note] InnoDB: Shutdown completed; log sequence number 231485352; transaction id 514172
2020-08-10 4:13:40 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2020-08-10 4:13:40 0 [Note] /usr/sbin/mysqld: Shutdown complete
]]>But, when I try to ‘ssh deploy@ipaddr’ I get this error message: Permission denied (publickey)
.
I’m guessing it’s trying to authenticate with my root key instead of asking me for the deploy users password. How can I get around this little issue?
]]>I have a website(example.com) already running in /var/www/html can i add another website (newdomain.com) by creating directory /var/www/newdomain.com and use it Without changing the location of example.com as you know it will take time to transfer from /var/www/html to /var/www/example.com ? If yes, How can i do that? Please guide will be thankfull.
I’m using Apache Web Server on Ubuntu 20.04 (LTS) x64
Please reply me ASAP
]]>There are times you’ll want to know how long it took for a command or script to execute. You could track the start and end times and calculate the difference to get the duration. You could even grab your trusty stopwatch app and track things that way. Or, you could save yourself all that trouble by leveraging the time
command.
The command-line is full of a bunch of small, single-purpose utilities that can help eliminate the need to write any additional code yourself. The time
command is one such command. The time
command is in no way related to the date
command, which provides the system’s date and time. Instead, it times a program or script’s execution and tells you how long it took.
In this tutorial, you’ll use the time
command and explore its output.
The examples in this tutorial (unless otherwise noted) will be using time
that’s built into the Bash shell on Linux.
To time the execution duration of a command, you prefix your command with time
.
However, how you execute time
depends on your operating system. time
exists both as a built-in command for some shells, like bash
and zsh
, as well as a stand-alone command, known as GNU time
which has different arguments than the time
built into shells.
Use the following command to see how time
works on your system:
- type -a time
You’ll see output like the following:
Outputtime is a shell keyword
time is /usr/bin/time
In this case, there’s both a built-in shell command called time
, and a version of time
installed at /usr/bin/time
.
If you want to use the GNU version of time
instead, you can prefix it with a backslash:
- \time
If you don’t, your shell will use its built-in version instead.
Note: the fish
shell doesn’t provide its own implementation of time
. If you happen to be using fish
, you will want to make sure you have the GNU time
command installed.
Both methods serve the same purpose, although the GNU version has more advanced formatting options.
Let’s explore timing program execution by using the tree
command on the root /
of your file system, which will list out a visual tree of all of your files and directories.
The tree
command is not always installed by default on many systems, but you can install it on Ubuntu and Debian systems with apt
:
- sudo apt install tree
On macOS, you can install tree
with Homebrew:
- brew install tree
Now that the tree
command is installed, use it to look at all the files on the system, but prefix it with time
to see how long it takes:
- time tree /
You’ll see the file information scroll through. Eventually, it’ll stop and present you with the time taken:
Output# The output of tree will scroll by here.
166590 directories, 1568127 files
tree / 12.24s user 10.37s system 66% cpu 33.827 total
Notice the command you executed, tree /
, is echoed back by the time
command as well. The output shows a few pieces of info, but for now, focus on the total
and the number just prior:
Outputtree / 12.24s user 10.37s system 66% cpu 33.827 total
That’s how long the command took to execute in seconds.
The time
command also works if you happen to cancel a command with CTRL+C
. If you were to run time tree /
again and quickly hit CTRL+C
, the tree
command will stop scrolling by, and you’ll be presented with the time
results for the time the command was executing before you stopped it.
The output from time
includes three values, in addition to the total duration.
Outputtree / 12.24s user 10.37s system 66% cpu 33.827 total
The first is the total amount of time (in CPU-seconds) that the command spent in user mode. That’s the value with user
after it.
The next, suffixed by system
, is the amount of time (again, in CPU-seconds) that the command spent in system, or kernel mode.
Finally, the percentage of the CPU that was allocated to the command, suffixed with cpu
.
The difference between user
and system
time is that CPU usage is broken down by access levels. When code is executed in user mode, it doesn’t have direct access to hardware or reference memory and must rely on APIs of the system for delegation. This is how most code runs on your system, and due to its isolation, crashes are always recoverable.
Kernel mode, on the other hand, is when code being executed has unrestricted access to the system hardware. This mode is pretty much reserved for the most trusted functions of the operating system. Because of having complete access, when things crash in kernel mode, they crash bad and tend to take the system down with them.
In this tutorial, you explored how to use the time
command to see how long commands and scripts take to run, and where that time goes. Your shell’s time
command gives you quick access to timing the execution duration of a command without any additional scripting.
Not all command-line utilities allow you to run a command against multiple files, but with the power of shell scripting and the for
loop, you can super charge any command you choose.
Looping is one of the most powerful things you can do in programming. It allows you to apply the same logic over and over to a group of items with minimal code.
In this tutorial you’ll iterate over files and apply commands using either the Bash or zsh shells.
You are going to use the for
shell scripting loop in this tutorial. This particular control flow method is baked into the Bash or zsh command-line shell.
You can apply the following commands to any directory of your choosing, but for this tutorial, create a directory and some files to play around with.
First, create the directory:
- mkdir looping
Then switch to the new directory:
- cd looping
Now use the touch
command to create a few text files:
- touch file-1.txt
- touch file-2.txt
- touch file-3.txt
- touch file-4.txt
- touch file-5.txt
You can also create these files quickly using brace expansion and a range:
- touch file-{1..5}.txt
To loop through a directory, and then print the name of the file, execute the following command:
- for FILE in *; do echo $FILE; done
You’ll see the following output:
Outputfile-1.txt
file-2.txt
file-3.txt
file-4.txt
file-5.txt
You probably noticed we’re using the wild card character, *
, in there. That tells the for
loop to grab every single file in the directory. You could change the wild card could to file-*
to target all of the files that started with file-
, or to *.txt
to grab just the text files.
Now that you know how to loop through the files in a directory and spit out the names, let’s use those files with a command.
The files you created in the previous section were all created with the touch
command and are empty, so showing out how to cat
each file wouldn’t be very useful.
Fortunately, using a similar for
loop, you can insert text into each file with a single command.
Execute the following command to insert the file’s name, followed by a newline, followed by the text Loops Rule!
into each file:
- for FILE in *; do echo -e "$FILE\nLoops Rule\!" > $FILE; done
The -e
flag on echo
ensures that newlines are preserved.
The exclamation point needs to be escaped with a backslash so the shell doesn’t interpret the character as a shell command.
Now iterate through each file and print its contents:
- for FILE in *; do cat $FILE; done
Now each file contains the name of the file on the first line, and a universal truth about loops on the second line.
Outputfile-1.txt
Loops Rule!
file-2.txt
Loops Rule!
file-3.txt
Loops Rule!
file-4.txt
Loops Rule!
file-5.txt
Loops Rule!
To take it a step further, you could combine these examples to first write to the file, then display its contents in a single loop:
- for FILE in *; do echo -e "$FILE\nLoops Rule\!" > $FILE; cat $FILE; done
You’ll see the following output:
Outputfile-1.txt
Loops Rule!
file-2.txt
Loops Rule!
file-3.txt
Loops Rule!
file-4.txt
Loops Rule!
file-5.txt
Loops Rule!
By separating the commands with a semi-colon, ;
, you can string together whichever commands you need.
Now let’s look at a practical example: making backup copies of files.
Now that you know how the for
loop works, let’s try something a bit more real world by taking a directory of files and making backups that are suffixed with the .bak
extension.
Execute the following command in your shell which creates a backup for each file:
- for FILE in *; do cp $FILE "$FILE.bak"; done;
Now use the ls
command to display each file:
- ls -l
You’ll see the following output:
Outputtotal 40K
-rw-r--r-- 1 sammy sammy 29 Nov 7 18:34 file-1.txt
-rw-r--r-- 1 sammy sammy 29 Nov 7 18:41 file-1.txt.bak
-rw-r--r-- 1 sammy sammy 29 Nov 7 18:34 file-2.txt
-rw-r--r-- 1 sammy sammy 29 Nov 7 18:41 file-2.txt.bak
-rw-r--r-- 1 sammy sammy 29 Nov 7 18:34 file-3.txt
-rw-r--r-- 1 sammy sammy 29 Nov 7 18:41 file-3.txt.bak
-rw-r--r-- 1 sammy sammy 29 Nov 7 18:34 file-4.txt
-rw-r--r-- 1 sammy sammy 29 Nov 7 18:41 file-4.txt.bak
-rw-r--r-- 1 sammy sammy 29 Nov 7 18:34 file-5.txt
-rw-r--r-- 1 sammy sammy 29 Nov 7 18:41 file-5.txt.bak
You now have exact copies of each of your files, with an extension that indicates that they are backup files.
You’re not limited to making copies in the same directory either; you can specify a new path for your backup files. The following command stores the backups in the folder /tmp/my-backups
, provided the directory already exists:
- for FILE in *; do cp $FILE "/tmp/my-backups/$FILE.bak"; done;
The backups are created in the new location.
In this tutorial you experimented with the for
loop in your shell and used it to iterate over files and apply commands.
What I want is, total 125GB storage accessible under root / how can I do that?
]]>But I cannot understand how each folder and file relates to the Tomcat server. Using Django seems to have the same idea with less files. Also, it uses python in the backend to modify settings and different virtual environments created by the user. The first step I took in installing tomcat on my Fedora 25 system was using this command.
dnf -y install tomcat
I then checked the java version I have with the command:
Java -version
[root@host-40 tomcat]# java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
I know that I can install a different instances of Tomcat per the book: I then checked what Tomcat version I had
[root@host-40 tomcat]# tomcat version
Server version: Apache Tomcat/8.0.47
Server built: Oct 25 2017 15:10:50 UTC
Server number: 8.0.47.0
OS Name: Linux
OS Version: 4.13.16-100.fc25.x86_64
Architecture: amd64
JVM Version: 1.8.0_151-b12
JVM Vendor: Oracle Corporation
In my system the webapps folder is /usr/share/tomcat
.
Using the tree
command I verified the folder structure like so:
[root@host-40 tomcat]# tree
.
├── bin
│ ├── bootstrap.jar
│ ├── catalina-tasks.xml
│ └── tomcat-juli.jar
├── conf -> /etc/tomcat
├── lib -> /usr/share/java/tomcat
├── logs -> /var/log/tomcat
├── temp -> /var/cache/tomcat/temp
├── webapps -> /var/lib/tomcat/webapps
└── work -> /var/cache/tomcat/work
7 directories, 3 files
I know it comes with the webapps folder and basic examples but I just can’t get around understanding how different instances can live together or what would be a good resource to understand the different folders and files that are important to Tomcat and Java.
]]>What is better for my droplet Ubuntu 18.04.3 LTS or Ubuntu 20.04 LTS because I have read something about Ubuntu 18 vs 20
and I found 20 better but I don’t know why Digitalocean select 18 by default
Thank you in advance
]]>CageFS:
https://docs.cloudlinux.com/cloudlinux_os_components/#cagefs
*"CageFS is a virtualized file system and a set of tools to contain each user in its own ‘cage’. Each customer will have its own fully functional CageFS, with all the system files, tools, etc.
The benefits of CageFS are:
At the same time, user’s environment will be fully functional, and user should not feel in any way restricted. No adjustments to user’s scripts are needed. CageFS will cage any scripts execution done via:
VS
DO’s VPC:
https://www.digitalocean.com/docs/networking/vpc/
“A Virtual Private Cloud (VPC) is a private network interface for collections of DigitalOcean resources. VPC networks are private networks that contain collections of resources that are isolated from the public internet and other VPC networks within your account, project or between teams in the same datacenter region. This means your resources, such as Droplets and databases, can reside in a network that is only accessible to other resources in the same network.”
Am I right?
]]>i try with this butnot working
sysctl net.ipv4.ip_forward = 1 iptables -A PREROUTING -t nat -i eth0 -p tcp –dport 8270 -j DNAT –to-destination 192.168.245.1:8270 iptables -A FORWARD -p tcp -d 192.168.245.2 –dport 8270 -j ACEPTA iptables -A POSTROUTING -t nat -s 192.168.245.2 -o eth0 -j MASQUERADE
intente de esta otra manera:
sysctl net.ipv4.ip_forward = 1 iptables -t nat -A PREROUTING -p tcp –dport 8270 -j DNAT –to-destination 192.168.245.2:8270 iptables -t nat -A POSTROUTING -j MASQUERADE
]]>intente via ip tables y no funciona
]]>ssmtp
.
Here is a mini-tutorial on how to install and configure ssmtp
and send emails directly from your command line or bash scripts.
I just installed Nginx but when I try to start it it fails with the following error:
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
If I check the status I get the following output:
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; disabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/nginx.service.d
└─override.conf
Active: failed (Result: exit-code) since Mon 2020-06-08 15:59:32 UTC; 5min ago
Docs: man:nginx(8)
Process: 3839 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=2)
Process: 3816 ExecStartPost=/bin/sleep 0.1 (code=exited, status=0/SUCCESS)
Process: 25445 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
Process: 25435 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 3813 (code=exited, status=0/SUCCESS)
Jun 08 15:59:30 ubuntu nginx[25445]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Jun 08 15:59:30 ubuntu nginx[25445]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Jun 08 15:59:31 ubuntu nginx[25445]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Jun 08 15:59:31 ubuntu nginx[25445]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Jun 08 15:59:31 ubuntu nginx[25445]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Jun 08 15:59:31 ubuntu nginx[25445]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Jun 08 15:59:32 ubuntu nginx[25445]: nginx: [emerg] still could not bind()
Jun 08 15:59:32 ubuntu systemd[1]: nginx.service: Control process exited, code=exited status=1
Jun 08 15:59:32 ubuntu systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 08 15:59:32 ubuntu systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Does anyone have any ideas why Nginx would not start?
Thanks!
]]>Linux utilities often follow the Unix philosophy of design. Tools are encouraged to be small, use plain text files for input and output, and operate in a modular manner. Because of this legacy, we have great text processing functionality with tools like sed and awk
.
awk
is both a programming language and text processor that you can use to manipulate text data in very useful ways. In this guide, you’ll explore how to use the awk
command line tool and how to use it to process text.
The awk
command is included by default in all modern Linux systems, so you do not need to install it to begin using it.
awk
is most useful when handling text files that are formatted in a predictable way. For instance, it is excellent at parsing and manipulating tabular data. It operates on a line-by-line basis and iterates through the entire file.
By default, it uses whitespace (spaces, tabs, etc.) to separate fields. Luckily, many configuration files on your Linux system use this format.
The basic format of an awk
command is:
- awk '/search_pattern/ { action_to_take_on_matches; another_action; }' file_to_parse
You can omit either the search portion or the action portion from any awk
command. By default, the action taken if the “action” portion is not given is “print”. This simply prints all lines that match.
If the search portion is not given, awk
performs the action listed on each line.
If both are given, awk
uses the search portion to decide if the current line reflects the pattern, and then performs the actions on matches.
In its simplest form, you can use awk
like cat
to print all lines of a text file out to the screen.
Create a favorite_food.txt
file which lists the favorite foods of a group of friends:
- echo "carrot sandy
- wasabi luke
- sandwich brian
- salad ryan
- spaghetti jessica" > favorite_food.txt
Now use the awk
command to print the file to the screen:
- awk '{print}' favorite_food.txt
You’ll see the file printed to the screen:
Outputcarrot sandy
wasabi luke
sandwich brian
salad ryan
spaghetti jessica
This isn’t very useful. Let’s try out awk
’s search filtering capabilities by searching through the file for the text “sand”:
- awk '/sand/' favorite_food.txt
Outputcarrot sandy
sandwich brian
As you can see, awk
now only prints the lines that have the characters “sand” in them.
Using regular expressions, you can target specific parts of the text. To display only the line that starts with the letters “sand”, use the regular expression ^sand
:
- awk '/^sand/' favorite_food.txt
This time, only one line is displayed:
Outputsandwich brian
Similarly, you can use the action section to specify which pieces of information you want to print. For instance, to print only the first column, use the following command:
- awk '/^sand/ {print $1;}' favorite_food.txt
Outputsandwich
You can reference every column (as delimited by whitespace) by variables associated with their column number. For example, the The first column is $1
, the second is $2
, and you can reference the entire line with $0
.
The awk
command uses some internal variables to assign certain pieces of information as it processes a file.
The internal variables that awk
uses are:
You can change the values of these variables at will to match the needs of your files. Usually you do this during the initialization phase of your processing.
This brings us to another important concept. The awk
syntax is slightly more complex than what you’ve used so far There are also optional BEGIN
and END
blocks that can contain commands to execute before and after the file processing, respectively.
This makes our expanded syntax look something like this:
- awk 'BEGIN { action; }
- /search/ { action; }
- END { action; }' input_file
The BEGIN
and END
keywords are specific sets of conditions, just like the search parameters. They match before and after the document has been processed.
This means that you can change some of the internal variables in the BEGIN
section. For instance, the /etc/passwd
file is delimited with colons (:
) instead of whitespace.
To print out the first column of this file, execute the following command:
- awk 'BEGIN { FS=":"; }
- { print $1; }' /etc/passwd
Outputroot
daemon
bin
sys
sync
games
man
. . .
You can use the BEGIN
and END
blocks to print information about the fields you are printing. Use the following command to transform the data from the file into a table, nicely spaced with tabs using \t
:
- awk 'BEGIN { FS=":"; print "User\t\tUID\t\tGID\t\tHome\t\tShell\n--------------"; }
- {print $1,"\t\t",$3,"\t\t",$4,"\t\t",$6,"\t\t",$7;}
- END { print "---------\nFile Complete" }' /etc/passwd
You’ll see this output:
OutputUser UID GID Home Shell
--------------
root 0 0 /root /bin/bash
daemon 1 1 /usr/sbin /bin/sh
bin 2 2 /bin /bin/sh
sys 3 3 /dev /bin/sh
sync 4 65534 /bin /bin/sync
. . .
---------
File Complete
As you can see, you can format things quite nicely by taking advantage of some of awk
’s features.
Each of the expanded sections are optional. In fact, the main action section itself is optional if another section is defined. For example, you can do things like this:
- awk 'BEGIN { print "We can use awk like the echo command"; }'
And you’ll see this output:
OutputWe can use awk like the echo command
Now let’s look at how to look for text within fields of the output.
In one of the previous examples, you printed the line in the favorite_food.txt
file that began with “sand”. This was easy because you were looking for the beginning of the entire line.
What if you wanted to find out if a search pattern matched at the beginning of a field instead?
Create a new version of the favorite_food.txt
file which adds an item number in front of each person’s food:
- echo "1 carrot sandy
- 2 wasabi luke
- 3 sandwich brian
- 4 salad ryan
- 5 spaghetti jessica" > favorite_food.txt
If you want to find all foods from this file that begin with “sa”, you might begin by trying something like this:
- awk '/sa/' favorite_food.txt
This shows all lines that contain “sa”:
Output1 carrot sandy
2 wasabi luke
3 sandwich brian
4 salad ryan
Here, you are matching any instance of “sa” in the word. This ends up including things like “wasabi” which has the pattern in the middle, or “sandy” which is not in the column you want. In this case you’re only interested in words beginning with “sa” in the second column.
You can tell awk
to only match at the beginning of the second column by using this command:
- awk '$2 ~ /^sa/' favorite_food.txt
As you can see, this allows us to only search at the beginning of the second column for a match.
The field_num ~
part specifies that awk
should only pay attention to the second column.
Output3 sandwich brian
4 salad ryan
You can just as easily search for things that do not match by including the “!” character before the tilde (~). This command will return all lines that do not have a food that starts with “sa”:
- awk '$2 !~ /^sa/' favorite_food.txt
Output1 carrot sandy
2 wasabi luke
5 spaghetti jessica
If you decide later on that you are only interested in lines that don’t start with “sa” and the item number is less than 5, you could use a compound expression like this:
- awk '$2 !~ /^sa/ && $1 < 5' favorite_food.txt
This introduces a few new concepts. The first is the ability to add additional requirements for the line to match by using the &&
operator. Using this, you can combine an arbitrary number of conditions for the line to match. In this case, you’re using this operator to add a check that the value of the first column is less than 5.
You’ll see this output:
Output1 carrot sandy
2 wasabi luke
You can use awk
to process files, but you can also work with the output of other programs.
You can use the awk
command to parse the output of other programs rather than specifying a filename. For example, you can use awk
to parse out the IPv4 address from the ip
command.
The ip a
command displays the IP address, broadcast address, and other information about all the network interfaces on your machine. To display the information for the interface called eth0
, use this command:
- ip a s eth0
You’ll see the following results:
Output2571: eth0@if2572: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:11:00:0b brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.17.0.11/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
You can use awk
to target the inet
line and then print out just the IP address:
- ip a s eth0 | awk -F '[\/ ]+' '/inet / {print $3}'
The -F
flag tells awk
to delimit by forward slashes or spaces using the regular expression [\/ ]+
. This splits the line inet 172.17.0.11/16
into separate fields. The IP address is in the third field because the spaces at the start of the line also count as a field, since you delimited by spaces as well as slashes. Note that awk
treated consecutive spaces as a single space in this case.
The output shows the IP address:
Output172.17.0.11
You’ll find many places where you can use awk
to search or parse the output of other commands.
By now, you should have a basic understanding of how you can use the awk
command to manipulate, format, and selectively print text files and streams of text. Awk is a much larger topic though, and is actually an entire programming language complete with variable assignment, control structures, built-in functions, and more. You can use it within your own scripts to format text in a reliable way.
To learn more about awk
, you can read the free public-domain book by its creators which goes into much more detail.
# yum update
DigitalOcean Agent 80 kB/s | 3.3 kB 00:00
Dovecot 2.3 CentOS 8 - x86_64 588 B/s | 162 B 00:00
Failed to download metadata for repo ‘dovecot-2.3-latest’
Error: Failed to download metadata for repo ‘dovecot-2.3-latest’
I use CentOS 8 with Open LightSpeed and CyberPanel.
I searched a lot on the internet, but I still couldn’t solve this problem. Can you help me?
]]>The sed
stream editor is a powerful editing tool that can make sweeping changes with very little input. In the previous article on sed
, you explored the basics of using sed to edit text.
This article will continue your introduction by examining some more advanced topics.
Note: This tutorial uses the GNU version of sed
found on Ubuntu and other Linux operating systems. If you’re using macOS, you’ll have the BSD version which has different options and arguments. You can install the GNU version of sed
with Homebrew using brew install gnu-sed
.
To complete this tutorial, you’ll need some files to manipulate, which you should have from the the first tutorial. If you don’t have them, you can recreate them with the following commands:
- cd
- cp /usr/share/common-licenses/BSD .
- echo "this is the song that never ends
- yes, it goes on and on, my friend
- some people started singing it
- not knowing what it was
- and they'll continue singing it forever
- just because..." > song.txt
In addition, you’ll use the GPL 3 license in this tutorial, so copy that file as well:
- cp /usr/share/common-licenses/GPL-3 .
If you don’t have it, you can download it with curl
:
- curl -o GPL-3 https://www.gnu.org/licenses/gpl-3.0.txt
Now that you have the files, you’ll explore using sed
with multiple commands.
There are quite a few instances where you might wish to pass multiple commands to sed
simultaneously. There are a few ways that this can be accomplished.
Since sed
operates through standard input and output, you can string different calls to sed
together through a pipeline. Execute this command to replace the word and
with an apersand (&
), and the word people
with horses
:
- sed 's/and/\&/' song.txt | sed 's/people/horses/'
Note that you need to escape the “&” since it means “the complete matched pattern” to sed
):
You’ll see the following output:
Outputthis is the song that never ends
yes, it goes on & on, my friend
some horses started singing it
not knowing what it was
& they'll continue singing it forever
just because...
This works, but it creates unnecessary overhead with multiple calls to sed
, requires more typing, and does not take advantage of sed
’s built-in capabilities.
You can string various commands to sed
by using the -e
option before each command. This is how you would rewrite the previous command:
- sed -e 's/and/\&/' -e 's/people/horses/' song.txt
Another approach to stringing commands together is using a semi-colon character (;
) to separate distinct commands. This works the same as the previous example, but the “-e
” is not required:
- sed 's/and/\&/;s/people/horses/' song.txt
Note how when using the -e
construct, you need separate single-quotation groups for the different commands. However, when separating commands with a semi-colon, all commands are placed within just one single-quoted command string. Although these two ways of expressing multiple commands is useful, there are times when the previous piping technique is still required.
Consider the =
operator. This operator inserts a line-number on a new line between each existing line. The output looks like this:
- sed '=' song.txt
Here’s the ouput you’ll see:
Output1
this is the song that never ends
2
yes, it goes on and on, my friend
3
some people started singing it
4
not knowing what it was
5
and they'll continue singing it forever
6
just because...
If you would like to change the formatting of the numbering by modifying the text, however, you’ll see that things do not work as expected.
To demonstrate, let’s look at the G
command, which by default, enters a blank line between each line (this actually is more complex, but you’ll explore that later):
- sed 'G' song.txt
Here’s the result:
Outputthis is the song that never ends
yes, it goes on and on, my friend
some people started singing it
not knowing what it was
and they'll continue singing it forever
just because...
If you combine these two commands, you might expect a space between each regular line and line-number line:
- sed '=;G' song.txt
However, you get something different:
Output1
this is the song that never ends
2
yes, it goes on and on, my friend
3
some people started singing it
4
not knowing what it was
. . .
. . .
This happens because the =
operator modifies the actual output stream directly. This means that you cannot use the results for more editing.
You can get around this by using two sed
calls, treating the first sed
modification as a simple stream of text for the second:
- sed '=' song.txt | sed 'G'
You now see the results you were expecting:
Output1
this is the song that never ends
2
yes, it goes on and on, my friend
3
some people started singing it
. . .
. . .
Keep in mind that some of the commands operate like this, especially if you are stringing multiple commands together and the output differs from what you were expecting.
One of the advantages of sed
’s addressable commands is that regular expressions can be used as selection criteria. This means that you are not limited to operating on known line values, like you saw previously:
- sed '1,3s/.*/Hello/' song.txt
OutputHello
Hello
Hello
not knowing what it was
and they'll continue singing it forever
just because...
You can, instead, use regular expressions to match only lines that contain a certain pattern. To do this, place the match pattern between two forward slashes (/) prior to giving the command strings:
- sed '/singing/s/it/& loudly/' song.txt
Outputthis is the song that never ends
yes, it goes on and on, my friend
some people started singing it loudly
not knowing what it was
and they'll continue singing it loudly forever
just because...
In this example, you’ve placed loudly
after the first occurrance of it
on every line that contains the string singing
. Notice that the second and fourth line are unaltered because they do not match the pattern.
The expressions for addressing can be arbitrarily complex. This provides a great deal of flexibility in executing commands.
This is not a complex example, but it demonstrates using regular expressions to generate addresses for other commands. The following command matches any blank lines (the start of a line followed immediately by the end of the line) and passes them to the delete command:
- sed '/^$/d' GPL-3
This is the output you’ll see:
Output GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc.
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU General Public License is a free, copyleft license for
. . .
. . .
Keep in mind that you can use regular expressions on either side of a range as well. For instance, you can delete lines starting at a line that only contains the word START
until a line reading END
,
For example, create a file called inputfile
:
- echo "This is an input file
- START
- this is the text we don't want
- END
- This is additional text" > inputfile
Now use sed
to delete the content between START
and END
:
- sed '/^START$/,/^END$/d' inputfile
You’ll see the following output:
This is an input file
This is additional text
One thing to note though, is that this will delete everything from the first START
to the first END
, and then restart the deletion if it comes across another START
marker.
If you want to invert an address (operate on any line that does not match a pattern), you can follow the pattern with an exclamation point (!
).
For example, you can delete any line that is not blank (not terribly useful, but just an example), with the following command:
- sed '/^$/!d' GPL-3
This results in a large blank output, as sed
still prints lines by default:
Output
The address does not need to be a complex expression to be inverted. Inversion works the same on regular numbered address too.
One piece of functionality that increases sed
’s ability perform multi-line aware edits is what is called the “hold buffer”. The hold buffer is an area of temporary storage that can be modified by certain commands.
The presence of this extra buffer means that you can store lines while working on other lines, and then operate on each buffer as necessary.
The following are the commands that affect the holding buffer:
The contents of the holding buffer cannot be operated on until it is moved to the pattern buffer in one way or another.
Let’s explore this idea with a complex example.
This is a procedural example of how to join adjacent lines (sed
actually has a built-in command that would take care of a lot of this for us. The N
command appends the next line to the current line. You are going to do things the hard way though for the sake of practice):
- sed -n '1~2h;2~2{H;g;s/\n/ /;p}' song.txt
Here’s the output you’ll see:
Outputthis is the song that never ends yes, it goes on and on, my friend
some people started singing it not knowing what it was
and they'll continue singing it forever just because...
This is a lot to digest, so let’s break it down.
The first thing to note is that the -n
option is used to suppress automatic printing. sed
will only print when you specifically tell it too.
The first part of the instruction is 1\~2h
. The beginning is an address specification meaning to perform the subsequent operation on the first line, and then on every other line afterwards (each odd numbered line). The h
part is the command to copy the matched line into the holding buffer.
The second half of the command is more complex. Again, it begins with an address specification. This time, it is referring to the even numbered lines (the opposite of the first command).
The rest of the command is enclosed in braces. This means that the rest of the commands will inherit the address that was just specified. Without the braces, only the “H” command would inherit the address, and the rest of the commands would be executed on every line.
The H
command copies a new-line character, followed by the current pattern buffer, onto the end of the current holding pattern.
This holding pattern (an odd numbered line, followed by a new-line character, followed by an even numbered line) is then copied back into the pattern buffer (replacing the previous pattern buffer) with the g
command.
Next, the new-line character is replaced with a space and the line is printed with the p
command.
If you are curious, using the N
command would shorten this considerably. The following command will produce the same results that you’ve just seen:
- sed -n 'N;s/\n/ /p' song.txt
As you begin to use more complex commands, it may be helpful to compose them in a text editor. This is also helpful if you have a large number of commands that you’d like to apply to a single target.
For example, if you like to compose messages in plain text, but you need to perform a set of standardized formatting before using the text, a sed
script would be useful.
Instead of typing each set of sed
calls, you can put the commands in a script and supply it as an argument to sed
. A sed
script is simply a list of raw sed
commands (the part normally between the single-quote characters).
To try this out, create a new file called sed_script
with the following contents:
- echo "s/this/that/g
- s/people/horses/g
- 1,5s/it/that/g" > sed_script
Save the file and exit the editor.
Now tell sed
to use the file by using the -f
switch:
- sed -f sed_script song.txt
The results will look like this:
Outputthat is the song that never ends
yes, that goes on and on, my friend
some horses started singing that
not knowing what that was
and they'll continue singing that forever
just because...
This allows you to put all of your edits in one file and execute it on arbitrary text files that need to conform to the format you’ve created.
Sed’s commands are not always easy to understand at first, and it often takes some actual experimentation to get an idea of their utility. For this reason, it’s recommended that you practice manipulating text before you actually need to. Have an end goal in mind and try to implement it only using sed
.
Hopefully, by this point, you are beginning to understand the power that a proper mastery of sed
can give you. The more comfortable you are with sed
, the less work you will have to do in the long run.
The sed
command, short for stream editor, performs editing operations on text coming from standard input or a file. sed
edits line-by-line and in a non-interactive way.
This means that you make all of the editing decisions as you are calling the command, and sed
executes the directions automatically. This may seem confusing or unintuitive, but it is a very powerful and fast way to transform text, especially as part of a script or automated workflow.
This tutorial will cover some basic operations and introduce you to the syntax required to operate this editor. You will almost certainly never replace your regular text editor with sed
, but it will probably become a welcomed addition to your text editing toolbox.
Note: This tutorial uses the GNU version of sed
found on Ubuntu and other Linux operating systems. If you’re using macOS, you’ll have the BSD version which has different options and arguments. You can install the GNU version of sed
with Homebrew using brew install gnu-sed
.
sed
operates on a stream of text that it reads from either a text file or from standard input (STDIN). This means that you can send the output of another command directly into sed for editing, or you can work on a file that you’ve already created.
You should also be aware that sed
outputs everything to standard out (STDOUT) by default. That means that, unless redirected, sed
will print its output to the screen instead of saving it in a file.
The basic usage is:
- sed [options] commands [file-to-edit]
In this tutorial, you’ll use a copy of the BSD Software License to experiment with sed
. On Ubuntu, execute the following commands to copy the BSD license file to your home directory so you can work with it:
- cd
- cp /usr/share/common-licenses/BSD .
If you don’t have a local copy of the BSD license, create one yourself with this command:
- cat << 'EOF' > BSD
- Copyright (c) The Regents of the University of California.
- All rights reserved.
-
- Redistribution and use in source and binary forms, with or without
- modification, are permitted provided that the following conditions
- are met:
- 1. Redistributions of source code must retain the above copyright
- notice, this list of conditions and the following disclaimer.
- 2. Redistributions in binary form must reproduce the above copyright
- notice, this list of conditions and the following disclaimer in the
- documentation and/or other materials provided with the distribution.
- 3. Neither the name of the University nor the names of its contributors
- may be used to endorse or promote products derived from this software
- without specific prior written permission.
-
- THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
- ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
- FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- SUCH DAMAGE.
- EOF
Let’s use sed
to view the contents of the BSD license file. sed
sends its results to the screen by default, which means you can use it as a file reader by passing it no editing commands. Try executing the following command:
- sed '' BSD
You’ll see the BSD license displayed to the screen:
OutputCopyright (c) The Regents of the University of California.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
...
...
The single quotes contain the editing commands you pass to sed
. In this case, you passed it nothing, so sed
printed each line it received to standard output.
sed
can use standard input rather than a file. Pipe the output of the cat
command into sed
to produce the same result:
- cat BSD | sed ''
You’ll see the output of the file:
OutputCopyright (c) The Regents of the University of California.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
. . .
. . .
As you can see, you can operate on files or streams of text, like the ones produced when piping output with the pipe (|)
character, just as easily.
In the previous example, you saw that input passed into sed
without any operations would print the results directly to standard output.
Let’s explore sed
’s explicit print
command, which you specify by using the p
character within single quotes.
Execute the following command:
- sed 'p' BSD
You’ll see each line of the BSD
file printed twice:
OutputCopyright (c) The Regents of the University of California.
Copyright (c) The Regents of the University of California.
All rights reserved.
All rights reserved.
Redistribution and use in source and binary forms, with or without
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
modification, are permitted provided that the following conditions
are met:
are met:
. . .
. . .
sed
automatically prints each line by default, and then you’ve told it to print lines explicitly with the “p” command, so you get each line printed twice.
If you examine the output closely, you’ll see that it has the first line twice, followed by the second line twice, etc, which tells you that sed
operates on data line by line. It reads a line, operates on it, and outputs the resulting text before repeating the process on the next line.
You can clean up the results by passing the -n
option to sed
, which suppresses the automatic printing:
- sed -n 'p' BSD
OutputCopyright (c) The Regents of the University of California.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
. . .
. . .
We now are back to printing each line once.
The examples so far can hardly be considered editing (unless you wanted to print each line twice…). Next you’ll explore how sed
can modify the output by targeting specific sections of the text data.
Addresses let you target specific parts of a text stream. You can specify a specific line or even a range of lines.
Let’s have sed
print the first line of the file. Execute the following command:
- sed -n '1p' BSD
The first line prints to the screen:
OutputCopyright (c) The Regents of the University of California.
By placing the number 1
before the print command, you told sed
the line number to operate on. You can just as easily print five lines (don’t forget the “-n”):
- sed -n '1,5p' BSD
You’ll see this output:
OutputCopyright (c) The Regents of the University of California.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
You’ve just given an address range to sed
. If you give sed
an address, it will only perform the commands that follow on those lines. In this example, you’ve told sed to print line 1 through line 5. You could have specified this in a different way by giving the first address and then using an offset to tell sed how many additional lines to travel, like this:
- sed -n '1,+4p' BSD
This will result in the same output, because you told sed
to start at line 1 and then operate on the next 4 lines as well.
If you want to print every other line, specify the interval after the ~
character. The following command prints every other line in the BSD
file, starting with line 1:
- sed -n '1~2p' BSD
Here’s the output you’ll see:
OutputCopyright (c) The Regents of the University of California.
modification, are permitted provided that the following conditions
1. Redistributions of source code must retain the above copyright
2. Redistributions in binary form must reproduce the above copyright
documentation and/or other materials provided with the distribution.
may be used to endorse or promote products derived from this software
. . .
. . .
You can use sed
to delete text from the output as well.
You can perform text deletion where you previously were specifying text printing by changing the p
command to the d
command.
In this case, you no longer need the -n
command because sed
will print everything that is not deleted. This will help you see what’s going on.
Modify the last command from the previous section to make it delete every other line starting with the first:
- sed '1~2d' BSD
The result is that you see every line you were not given last time:
OutputAll rights reserved.
Redistribution and use in source and binary forms, with or without
are met:
notice, this list of conditions and the following disclaimer.
notice, this list of conditions and the following disclaimer in the
3. Neither the name of the University nor the names of its contributors
without specific prior written permission.
. . .
. . .
It is important to note here that our source file is not being affected. It is still intact. The edits are output to our screen.
If we want to save our edits, we can redirect standard output to a file like so:
- sed '1~2d' BSD > everyother.txt
Now open the file with cat
:
- cat everyother.txt
You see the same output that you saw onscreen previously:
OutputAll rights reserved.
Redistribution and use in source and binary forms, with or without
are met:
notice, this list of conditions and the following disclaimer.
notice, this list of conditions and the following disclaimer in the
3. Neither the name of the University nor the names of its contributors
without specific prior written permission.
. . .
. . .
The sed
command does not edit the source file by default, but you can change this behavior by passing the -i
option, which means “perform edits in-place.” This will alter the source file.
Warning: Using the -i
switch will overwrite the original file, so you should use this with care. Perform the operations without the -i
switch first and then run the command again with -i
once you have what you want, create a backup of the original file, or redirect the output to a file. It’s very easy to accidentally alter the original file with the -i
switch.
Let’s try it by editing the everyother.txt
file you just created, in-place. Let’s further reduce the file by deleting every other line
again:
- sed -i '1~2d' everyother.txt
If you use cat
to display the file with cat everyother.txt
, you’ll see that the file has been edited.
The -i
option can be dangerous. Thankfully, sed
gives you the ability to create a backup file prior to editing.
To create a backup file prior to editing, add the backup extension directly after the “-i” option:
- sed -i.bak '1~2d' everyother.txt
This creates a backup file with the .bak
extension, and then edits the original file in-place.
Next you’ll look at how to use sed
to perform search and replace operations.
Perhaps the most well-known use for sed
is substituting text. sed
can search for text patterns using regular expressions, and then replace the found text with something else.
You can learn more about regular expressions by following the Using Grep Regular Expressions to Search for Text Patterns in Linux.
In its most basic form, you can change one word to another word using the following syntax:
's/old_word/new_word/'
The s
is the substitute command. The three slashes (/
) are used to separate the different text fields. You can use other characters to delimit the fields if it would be more helpful.
For instance, if you were trying to change a website name, using another delimiter would be helpful since URLs contain slashes.
Execute the following command to print a URL with echo
and modify it with sed
, using the underscore (_
) character as the delimiter:
- echo "http://www.example.com/index.html" | sed 's_com/index_org/home_'
This replaces com/index
with org/home
. The output shows the modifed URL:
Outputhttp://www.example.org/home.html
Do not forget the final delimiter, or sed
will complain. If you ran this command:
- echo "http://www.example.com/index.html" | sed 's_com/index_org/home'
You’d see this output:
Outputsed: -e expression #1, char 20: unterminated `s' command
Let’s create a new file to practice some substitutions. Execute the following command to create a new text file called song.txt
:
- echo "this is the song that never ends
- yes, it goes on and on, my friend
- some people started singing it
- not knowing what it was
- and they'll continue singing it forever
- just because..." > song.txt
Now let’s substitute the expression on
with forward
. Use the following command:
- sed 's/on/forward/' song.txt
The output looks like this:
Outputthis is the sforwardg that never ends
yes, it goes forward and on, my friend
some people started singing it
not knowing what it was
and they'll cforwardtinue singing it forever
just because...
You can see a few notable things here. First, is that sed
replaced patterns, not words. The on
within song
is changed to forward
.
The other thing to notice is that on line 2, the second on
was not changed to forward
.
This is because by default, the s
command operates on the first match in a line and then moves to the next line. To make sed
replace every instance of on
instead of just the first on each line, you must pass an optional flag to the substitute command.
Provide the g
flag to the substitute command by placing it after the substitution set:
- sed 's/on/forward/g' song.txt
You’ll see this output:
Outputthis is the sforwardg that never ends
yes, it goes forward and forward, my friend
some people started singing it
not knowing what it was
and they'll cforwardtinue singing it forever
just because...
Now the substitute command changes every instance.
If you only wanted to change the second instance of “on” that sed finds on each line, then you would use the number 2
instead of the g
:
- sed 's/on/forward/2' song.txt
This time the other lines are unchanged, as they don’t have a second occurrence:
Outputthis is the song that never ends
yes, it goes on and forward, my friend
some people started singing it
not knowing what it was
and they'll continue singing it forever
just because...
If you only want to see which lines were substituted, use the -n
option again to suppress automatic printing.
You can then pass the p
option to the substitute command to print lines where substitution took place.
- sed -n 's/on/forward/2p' song.txt
The line that changed prints to the screen:
Outputyes, it goes on and forward, my friend
As you can see, you can combine the flags at the end of the command.
If you want the search process to ignore case, you can pass it the “i” flag.
- sed 's/SINGING/saying/i' song.txt
Here’s the output you’ll see:
Outputthis is the song that never ends
yes, it goes on and on, my friend
some people started saying it
not knowing what it was
and they'll continue saying it forever
just because...
If you want to find more complex patterns with regular expressions, you have a number of different methods of referencing the matched pattern in the replacement text.
For instance, to match from the beginning of the line to at
, use the following command:
- sed 's/^.*at/REPLACED/' song.txt
You’ll see this output:
OutputREPLACED never ends
yes, it goes on and on, my friend
some people started singing it
REPLACED it was
and they'll continue singing it forever
just because...
You can see that the wildcard expression matches from the beginning of the line to the last instance of at
.
Since you don’t know the exact phrase that will match in the search string, you can use the &
character to represent the matched text in the replacement string.
Let’s put parentheses around the matched text:
- sed 's/^.*at/(&)/' song.txt
You’ll see this output:
Output(this is the song that) never ends
yes, it goes on and on, my friend
some people started singing it
(not knowing what) it was
and they'll continue singing it forever
just because...
A more flexible way of referencing matched text is to use escaped parentheses to group sections of matched text.
Every group of search text marked with parentheses can be referenced by an escaped reference number. For instance, the first parentheses group can be referenced with \1
, the second with \2
and so on.
In this example, we’ll switch the first two words of each line:
- sed 's/\([a-zA-Z0-9][a-zA-Z0-9]*\) \([a-zA-Z0-9][a-zA-Z0-9]*\)/\2 \1/' song.txt
You’ll see this output:
Outputis this the song that never ends
yes, goes it on and on, my friend
people some started singing it
knowing not what it was
they and'll continue singing it forever
because just...
As you can see, the results are not perfect. For instance, the second line skips the first word because it has a character not listed in our character set. Similarly, it treated they'll
as two words in the fifth line.
Let’s improve the regular expression to be more accurate:
- sed 's/\([^ ][^ ]*\) \([^ ][^ ]*\)/\2 \1/' song.txt
You’ll see this output:
Outputis this the song that never ends
it yes, goes on and on, my friend
people some started singing it
knowing not what it was
they'll and continue singing it forever
because... just
This is much better than last time. This groups punctuation with the associated word.
Notice how we repeat the expression inside the parentheses (once without the *
character, and then once with it). This is because the *
character matches the character set that comes before it zero or more times. This means that the match with the wildcard would be considered a “match” even if the pattern is not found.
To ensure that sed
finds the text at least once, you must match it once without the wildcard before employing the wildcard.
In this tutorial you explored the sed
command. You printed specific lines from the file, searched for text, deleted lines, overwrote the original file, and used regular expressions to replace text. You should be able to see already how you can quickly transform a text document using properly constructed sed commands.
In the next article in this series, you will explore some more advanced features.
]]>I’ve been asked this question a lot of times - ‘Can we restrict sudo users to only a handfull commands?’. In this mini tutorial, I’ll show you how you can do that.
]]>Is there an easy way to trim a string in bash?
For example if I wanted to trim a string with JS, I could simply do string.trim();
.
However as far as I can see there is no such command in bash:
- trim
The output that I get is:
- trim: command not found
Any suggestions would be appreciated!
]]>docker system df
command you would get a summary of your Docker usage including things like:
However, here’s how to check the size of each running container
]]>The prompt only seems to accept normal commands and filters out and really messes up some other characters. (supposedly those who have to be back-slashed ("") normally, while in a normal computer based terminal.)
I´m on MacOS, Yosemite, and using the terminal application. I cannot log in via the terminal and copy-paste from there because the SSH key is not currently in the Droplet list of approved and used SSH-keys. So catch 22 on trying this anywhere except for in the DO Online environmet´s “Console” window. That´s where I am, but I cannot import the text, it just messes up the pasted text.
This is my command that I am trying with:
echo "ssh-rsa AAAAB3... ...79akr/oys9 user@my-computer" \
>> ~/.ssh/authorized_keys
So this command should enter the ssh-rsa text line into the authorized_keys file and then I´ll open it in nano editor to have a look.
Also, the code was copied directly from one of the documentation pages here on DO:
digitalocean.com/docs/droplets/how-to/add-ssh-keys/to-existing-droplet/#manually
Perhaps I´d have an additional issue in the last part of my echo statement, the “user@my-computer”, in the post that is set to “user@00.000.00.000”, so the user and IP of the droplet, instead of my local machine. But still, the copy-paste is the issue here. It does not work.
I know what most would think - “why not just open up the nano editor and paste the code directly into the file?”
Well the nano editor messes up the code pasted into it as well. When I try to import a simple HTML tag into nano editor in the Console app, it fails too. Messes up the initial <
of the <html>
tags.
Surely I´m missing something?
Possible solutions I see: using terminal on my local computer to upload a txt file via scp (which is based on ssh, no?, so catch 22 again). Using an import command from the Console app online to do the same.
]]>Here’s how to do that!
]]>Accurate time keeping is critical for almost any service or software. Emails, loggers, event systems and schedulers, user authentication mechanisms, and services running on distributed platforms all need accurate timestamps to record events in chronological order. These services use the Network Time Protocol, or NTP, to synchronize the system clock with a trusted external source. This source can be an atomic clock, a GPS receiver, or another time server that already uses NTP.
This is where the NTP Pool Project project comes into play. It’s a huge worldwide cluster of time servers that provides easy access to known “good time” for tens of millions of clients around the world. It’s the default time server for Ubuntu and most of the other major Linux distributions, as well as many networked appliances and software applications.
In this guide, you will set up NTP on your server and configure it to be part of the NTP Pool Project, so it provides accurate time to other users of the NTP Pool Project. Providing your spare CPU cycles and unused bandwidth is a perfect way to give something back to the community.
The required bandwidth is relatively low and can be adjusted depending on the amount you can provide and where your server resides. Each client will only send a couple of UDP packets every 20 minutes, so most servers only receive about a dozen NTP packets per second, with spikes a couple of times a day of up to one hundred packets per second. This translates to bandwidth usage of 10-15Kb/sec with spikes of 50-120Kb/sec.
There are three basic requirements you must satisfy before joining the NTP Pool Project:
For most cloud-based servers, the first two requirements are usually met automatically. The third requirement emphasizes that joining the NTP Pool Project constitutes a long-term commitment. Of course, if your circumstances change, it’s fine to take a server out of the pool, but it will take a long time (mostly weeks, but sometimes months or even years) before the traffic completely vanishes.
To complete this tutorial, you will need:
The NTP package is not installed by default, so you’ll use the package manager to install it. First, update your packages:
- sudo yum update
Then install NTP:
- sudo yum install ntp
Once the installation completes, start the service and configure it so it starts automatically each time the server boots:
- sudo systemctl start ntpd
- sudo systemctl enable ntpd
If you’ve configured the firewall as specified in the prerequisites, you must allow UDP traffic for the NTP service in order to communicate with the NTP pool:
- sudo firewall-cmd --permanent --add-service=ntp
- sudo firewall-cmd --reload
For more on FirewallD, refer to How To Set Up a Firewall Using FirewallD on CentOS 7.
NTP is now installed, but it’s configured to use the default NTP pool time servers. Lets pick some specific time servers instead.
The NTP Pool project asks operators who want to join the pool to choose good network-local time servers rather than using the default pool.ntp.org
servers. This ensures that the NTP Pool Project remains reliable, fast, and healthy. When choosing your time source, you’ll want a stable network connection with no packet loss and as few hops as possible between the servers.
The multi-tiered and hierarchical NTP protocol separates the parties involved into primary servers, secondary servers, and clients. The primary servers are called Stratum 1 and are connected directly to the source of time, which is called Stratum 0. This source can be an atomic clock, a GPS receiver, or a radio navigation system. Secondary servers in the chain are called Stratum 2, Stratum 3 and so on.
Each server is also a client. A Stratum 2 client receives time from an upstream Stratum 1 server, and provides time to downstream Stratum 3 servers or other clients. For NTP Pool Project members to work properly, the NTP daemon needs at least three servers configured. The project recommends a minimum of four, and no more than seven sources.
The NTP Pool Project provides a list of public Stratum 1 and Startum 2 time servers. The lists designate the NTP time servers available for public access under stated restrictions. You’ll find three types:
Warning: Don’t use servers that are not listed as OpenAccess unless you’ve received approval to do so.
Visit the Stratum 1 Time Servers list. You’ll see a list like the following:
Sort the list by the ISO code column and find one or two servers that are geographically close to your server’s data center. When the server’s Access Policy column states OpenAccess, you can use it without issue. If it says “RestrictedAccess”, click to open the entry and read the instructions noted in the AccessDetails field. Often, you’ll find that NotificationMessage is set to Yes, which means you have to craft an informal email directed to the address provided in ServerContact, informing the server operator about your desire to use this time server as a time source for your NTP Pool Project member.
Once you’ve identified the servers you’d like to use, click the link for each server in the ISO column and copy its host name or IP address. You’ll use these addresses in Step 3.
Next, select three or four servers from the Stratum 2 list, following the same process.
Once you have selected your time servers, it’s time to configure your NTP client to use them.
To use your server with the NTP pool, and configure your new time servers, you’ll need to make some modifications to your NTP daemon’s configuration. To do so, edit the /etc/ntp.conf
file:
- sudo vi /etc/ntp.conf
First, make sure a driftfile is configured. A driftfile stores the frequency offset between the system clock running at its nominal frequency, and the frequency required to remain in synchronization with correct time. It helps to achieve a stable and accurate time. You should find this at the top of your configuration file on a default installation:
# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).
driftfile /var/lib/ntp/drift
...
Next, remove the default time source entries from the configuration. You’re looking for all lines which are of the pattern server 0.centos.pool.ntp.org iburst
. If you’re using a default configuration, remove the highlighted lines as shown in the following example:
...
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst
Replace the lines you removed with the hand-picked servers you selected in the previous step.
...
server ntp_server_hostname_1 iburst
server ntp_server_hostname_2 iburst
server ntp_server_hostname_3 iburst
server ntp_server_hostname_4 iburst
server ntp_server_hostname_5 iburst
...
We use the iburst
option for each servers, per the NTP Pool recommendations. That way, if the server is unreachable, this will send a burst of eight packets instead of the usual one packet. Using the burst
option in the NTP Pool Project is considered abuse as it will send those eight packets every poll interval, whereas iburst
sends the eight packets only the first time.
Next, make sure the default configuration does not allow management queries. If you don’t, your server could be used in NTP reflection attacks, or could be vulnerable to ntpq
and ntpdc
queries that attempt to modify the state of the server. Check that the noquery
option is added to the default restrict
lines. Also make sure you add the options kod
and limited
as they restrict too eagerly asking clients and enforce rate limiting.
...
# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default nomodify notrap nopeer noquery kod limited
# Permit all access over the loopback interface. This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict ::1
You can find more information about the other options in the official documentation.
Your NTP daemon configuration file now should look like the following, although your file may have additional comments, which you can safely disregard:
driftfile /var/lib/ntp/ntp.drift
restrict default nomodify notrap nopeer noquery kod limited
restrict 127.0.0.1
restrict ::1
server ntp_server_hostname_1 iburst
server ntp_server_hostname_2 iburst
server ntp_server_hostname_3 iburst
server ntp_server_hostname_4 iburst
server ntp_server_hostname_5 iburst
Save the file and exit the editor.
Now restart the NTP service and let your time server synchronize its clock to the upstream servers.
- sudo systemctl restart ntpd
After a few minutes, check the health of your time server with the ntpq
command:
- ntpq -p
The output should look similar to this:
Output remote refid st t when poll reach delay offset jitter
==============================================================================
mizbeaver.udel. .INIT. 16 u - 64 0 0.000 0.000 0.000
montpelier.ilan .GPS. 1 u 25 64 7 55.190 2.121 130.492
+nist1-lnk.binar .ACTS. 1 u 28 64 7 52.728 23.860 3.247
*ntp.okstate.edu .GPS. 1 u 31 64 7 19.708 -8.344 6.853
+ntp.colby.edu .GPS. 1 u 34 64 7 51.518 -5.914 6.669
The remote column tells you the hostname of the servers the NTP daemon is using, and the refid column tells you the source the servers are using. So for Stratum 1 servers, the refid field should show GPS, PPS, ACTS, or PTB, and Stratum 2 and higher servers will show the IP address of the upstream server. The st column shows the stratum, and delay, offset and jitter tell you about the quality of the time source. Lower values are better for these three fields.
Your time server is now able to serve time to the public. You can verify this by calling ntpdate
from another host:
- ntpdate -q your_server_ip
The output should look similar to this and it tells you it adjusted the time server and the offset:
Outputserver your_server_ip, stratum 2, offset 0.001172, delay 0.16428
2 Mar 23:06:44 ntpdate[18427]: adjust time server your_server_ip offset 0.001172 sec
You are now ready to register your NTP server with the NTP Pool Project so others can use it.
To add your server so others can use it, visit manage.ntppool.org and sign up for an account. You will receive an email from NTP Pool help@ntppool.org requesting that you verify your account. Confirm your account by following the instructions in the email, and then log in to manage.ntppool.org.
Once logged in, you’ll see the simple interface for adding servers:
Enter your server’s IP address and click Submit.
The next screen asks you to verify that it identified the region of your server. If it shows your server in a different region than you expect, use the Comment box to let them know.
If you are happy, confirm the entry by clicking Yes, this is my server, add it!
Your server is now part of the NTP Pool Project. Visit http://www.pool.ntp.org/scores/your_server_ip
to see information the NTP Pool’s monitoring system has collected about your server. It checks your server a few times per hour and displays offset data, alog with the score of your system. As long as your server is keeping good time and is reachable, the score will rise untill it reaches 20 points. Only servers with a score higher than 10 are used in the pool.
If you are having trouble getting your server to sync you might have a packet firewall in place dropping your outgoing packets on port 123
. Take a look at How To Set Up a Firewall Using FirewallD on CentOS 7 to learn how to check the status of the firewall.
If the NTP Pool Project’s monitoring station can’t reach your NTP server and your server score is going down, or you can’t use your server to sync some other clock, you might have a packet firewall in place dropping your incoming traffic on port 123
. Check your firewall status.
If you are certain that you have no firewall in place, or you have opened port 123
for both incoming and outgoing traffic, your server provider or another transit provider might be dropping your packets along the way. If you do not have the knowledge to solve those problems on your own, it’s best to turn to the community and reach for help. The NTP Pool Projects forum is a good place to start. You can also join the mailing list or send an emaill to the NTP Pool Project operator. Just be sure you can show all the steps you’ve already tried to resolve the issue before asking for help.
In this tutorial, you successfully set up your own time server and made it a member of the NTP Pool Project, serving time to the community. To keep in touch with the time-keeping community. join the NTP Pool Projects forum or the mailing list. Be sure to monitor your server’s score and make any adjustments necessary.
]]>Accurate time keeping is critical for almost any service or software. Emails, loggers, event systems and schedulers, user authentication mechanisms, and services running on distributed platforms all need accurate timestamps to record events in chronological order. These services use the Network Time Protocol, or NTP, to synchronize the system clock with a trusted external source. This source can be an atomic clock, a GPS receiver, or another time server that already uses NTP.
This is where the NTP Pool Project project comes into play. It’s a huge worldwide cluster of time servers that provides easy access to known “good time” for tens of millions of clients around the world. It’s the default time server for Ubuntu and most of the other major Linux distributions, as well as many networked appliances and software applications.
In this guide, you will set up NTP on your server and configure it to be part of the NTP Pool Project, so it provides accurate time to other users of the NTP Pool Project. Providing your spare CPU cycles and unused bandwidth is a perfect way to give something back to the community.
The required bandwidth is relatively low and can be adjusted depending on the amount you can provide and where your server resides. Each client will only send a couple of UDP packets every 20 minutes, so most servers only receive about a dozen NTP packets per second, with spikes a couple of times a day of up to one hundred packets per second. This translates to bandwidth usage of 10-15Kb/sec with spikes of 50-120Kb/sec.
There are three basic requirements you must satisfy before joining the NTP Pool Project:
For most cloud-based servers, the first two requirements are usually met automatically. The third requirement emphasizes that joining the NTP Pool Project constitutes a long-term commitment. Of course, if your circumstances change, it’s fine to take a server out of the pool, but it will take a long time (mostly weeks, but sometimes months or even years) before the traffic completely vanishes.
To complete this tutorial, you will need:
The NTP package is not installed by default, so you’ll use the package manager to install it. First, update your packages:
- sudo apt-get update
Then install NTP:
- sudo apt-get install ntp
If you’ve configured the firewall as specified in the prerequisites, you must allow UDP traffic on port 123
in order to communicate with the NTP pool:
- sudo ufw allow 123/udp
For more on UFW, refer to How To Set Up a Firewall with UFW on Ubuntu.
NTP is now installed, but it’s configured to use the default NTP pool time servers. Lets pick some specific time servers instead.
The NTP Pool project asks operators who want to join the pool to choose good network-local time servers rather than using the default pool.ntp.org
servers. This ensures that the NTP Pool Project remains reliable, fast, and healthy. When choosing your time source, you’ll want a stable network connection with no packet loss and as few hops as possible between the servers.
The multi-tiered and hierarchical NTP protocol separates the parties involved into primary servers, secondary servers, and clients. The primary servers are called Stratum 1 and are connected directly to the source of time, which is called Stratum 0. This source can be an atomic clock, a GPS receiver, or a radio navigation system. Secondary servers in the chain are called Stratum 2, Stratum 3 and so on.
Each server is also a client. A Stratum 2 client receives time from an upstream Stratum 1 server, and provides time to downstream Stratum 3 servers or other clients. For NTP Pool Project members to work properly, the NTP daemon needs at least three servers configured. The project recommends a minimum of four, and no more than seven sources.
The NTP Pool Project provides a list of public Stratum 1 and Startum 2 time servers. The lists designate the NTP time servers available for public access under stated restrictions. You’ll find three types:
Warning: Don’t use servers that are not listed as OpenAccess unless you’ve received approval to do so.
Visit the Stratum 1 Time Servers list. You’ll see a list like the following:
Sort the list by the ISO code column and find one or two servers that are geographically close to your server’s data center. When the server’s Access Policy column states OpenAccess, you can use it without issue. If it says “RestrictedAccess”, click to open the entry and read the instructions noted in the AccessDetails field. Often, you’ll find that NotificationMessage is set to Yes, which means you have to craft an informal email directed to the address provided in ServerContact, informing the server operator about your desire to use this time server as a time source for your NTP Pool Project member.
Once you’ve identified the servers you’d like to use, click the link for each server in the ISO column and copy its host name or IP address. You’ll use these addresses in Step 3.
Next, select three or four servers from the Stratum 2 list, following the same process.
Once you have selected your time servers, it’s time to configure your NTP client to use them.
To use your server with the NTP pool, and configure your new time servers, you’ll need to make some modifications to your NTP daemon’s configuration. To do so, edit the /etc/ntp.conf
file:
- sudo nano /etc/ntp.conf
First, make sure a driftfile is configured. A driftfile stores the frequency offset between the system clock running at its nominal frequency, and the frequency required to remain in synchronization with correct time. It helps to achieve a stable and accurate time. You should find this at the top of your configuration file on a default installation:
# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help
driftfile /var/lib/ntp/ntp.drift
...
Next, remove the default time source entries from the configuration. You’re looking for all lines which are of the pattern pool [0-3].ubuntu.pool.ntp.org iburst
or pool ntp.ubuntu.com
. If you’re using a default configuration, remove the highlighted lines:
# Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board
# on 2011-02-08 (LP: #104525). See http://www.pool.ntp.org/join.html for
# more information.
pool 0.ubuntu.pool.ntp.org iburst
pool 1.ubuntu.pool.ntp.org iburst
pool 2.ubuntu.pool.ntp.org iburst
pool 3.ubuntu.pool.ntp.org iburst
# Use Ubuntu's ntp server as a fallback.
pool ntp.ubuntu.com
Replace the lines you removed with the hand-picked servers you selected in the previous step, using the server
keyword instead of the pool
keyword.
...
server ntp_server_hostname_1 iburst
server ntp_server_hostname_2 iburst
server ntp_server_hostname_3 iburst
server ntp_server_hostname_4 iburst
server ntp_server_hostname_5 iburst
...
We use the iburst
option for each servers, per the NTP Pool recommendations. That way, if the server is unreachable, this will send a burst of eight packets instead of the usual one packet. Using the burst
option in the NTP Pool Project is considered abuse as it will send those eight packets every poll interval, whereas iburst
sends the eight packets only the first time.
Next, make sure the default configuration does not allow management queries. If you don’t, your server could be used in NTP reflection attacks, or could be vulnerable to ntpq
and ntpdc
queries that attempt to modify the state of the server. Check that the noquery
option is added to the default restrict
lines:
...
# By default, exchange time with everybody, but don't allow configuration.
restrict -4 default kod notrap nomodify nopeer noquery limited
restrict -6 default kod notrap nomodify nopeer noquery limited
# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict ::1
You can find more information about the other options in the official documentation.
Your NTP daemon configuration file now should look like the following, although your file may have additional comments, which you can safely disregard:
driftfile /var/lib/ntp/ntp.drift
server ntp_server_hostname_1 iburst
server ntp_server_hostname_2 iburst
server ntp_server_hostname_3 iburst
server ntp_server_hostname_4 iburst
server ntp_server_hostname_5 iburst
# By default, exchange time with everybody, but don't allow configuration.
restrict -4 default kod notrap nomodify nopeer noquery limited
restrict -6 default kod notrap nomodify nopeer noquery limited
# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict ::1
Save the file and exit the editor.
Now restart the NTP service and let your time server synchronize its clock to the upstream servers.
- sudo systemctl restart ntp.service
After a few minutes, check the health of your time server with the ntpq
command:
- ntpq -p
The output should look similar to this:
Output remote refid st t when poll reach delay offset jitter
==============================================================================
mizbeaver.udel. .INIT. 16 u - 64 0 0.000 0.000 0.000
montpelier.ilan .GPS. 1 u 25 64 7 55.190 2.121 130.492
+nist1-lnk.binar .ACTS. 1 u 28 64 7 52.728 23.860 3.247
*ntp.okstate.edu .GPS. 1 u 31 64 7 19.708 -8.344 6.853
+ntp.colby.edu .GPS. 1 u 34 64 7 51.518 -5.914 6.669
The remote column tells you the hostname of the servers the NTP daemon is using, and the refid column tells you the source the servers are using. So for Stratum 1 servers, the refid field should show GPS, PPS, ACTS, or PTB, and Stratum 2 and higher servers will show the IP address of the upstream server. The st column shows the stratum, and delay, offset and jitter tell you about the quality of the time source. Lower values are better for these three fields.
Your time server is now able to serve time to the public. You can verify this by calling ntpdate
from another host:
- ntpdate -q your_server_ip
The output should look similar to this and it tells you it adjusted the time server and the offset:
Outputserver your_server_ip, stratum 2, offset 0.001172, delay 0.16428
2 Mar 23:06:44 ntpdate[18427]: adjust time server your_server_ip offset 0.001172 sec
You are now ready to register your NTP server with the NTP Pool Project so others can use it.
To add your server so others can use it, visit manage.ntppool.org and sign up for an account. You will receive an email from NTP Pool help@ntppool.org requesting that you verify your account. Confirm your account by following the instructions in the email, and then log in to manage.ntppool.org.
Once logged in, you’ll see the simple interface for adding servers:
Enter your server’s IP address and click Submit.
The next screen asks you to verify that it identified the region of your server. If it shows your server in a different region than you expect, use the Comment box to let them know.
If you are happy, confirm the entry by clicking Yes, this is my server, add it!
Your server is now part of the NTP Pool Project. Visit http://www.pool.ntp.org/scores/your_server_ip
to see information the NTP Pool’s monitoring system has collected about your server. It checks your server a few times per hour and displays offset data, alog with the score of your system. As long as your server is keeping good time and is reachable, the score will rise untill it reaches 20 points. Only servers with a score higher than 10 are used in the pool.
If you are having trouble getting your server to sync you might have a packet firewall in place dropping your outgoing packets on port 123
. Take a look at How To Set Up a Firewall with UFW on Ubuntu to learn how to check the status of the firewall.
If the NTP Pool Project’s monitoring station can’t reach your NTP server and your server score is going down, or you can’t use your server to sync some other clock, you might have a packet firewall in place dropping your incoming traffic on port 123
. Check your firewall status.
If you are certain that you have no firewall in place, or you have opened port 123
for both incoming and outgoing traffic, your server provider or another transit provider might be dropping your packets along the way. If you do not have the knowledge to solve those problems on your own, it’s best to turn to the community and reach for help. The NTP Pool Projects forum is a good place to start. You can also join the mailing list or send an emaill to the NTP Pool Project operator. Just be sure you can show all the steps you’ve already tried to resolve the issue before asking for help.
In this tutorial, you successfully set up your own time server and made it a member of the NTP Pool Project, serving time to the community. To keep in touch with the time-keeping community. join the NTP Pool Projects forum or the mailing list. Be sure to monitor your server’s score and make any adjustments necessary.
]]>One of the most basic tasks that you should know how to do on a fresh Linux server is add and remove users. When you create a new system, you are often only given the root account by default. While running as the root user gives you a lot of power and flexibility, it is also dangerous and can be destructive. It is almost always a better idea to add an additional, unprivileged user to do common tasks. You should then create additional accounts for any other users you may have on your system.
You can still acquire administrator privileges when you need them through a mechanism called sudo
. In this tutorial, you’ll learn how to create user accounts, assign sudo
privileges, and delete users.
If you are signed in as the root user, you can create a new user at any time by typing:
- adduser sammy
If you are signed in as a non-root user who has been given sudo
privileges, as demonstrated in the initial server setup guide, you can add a new user by typing:
- sudo adduser sammy
Once you execute the command, you’ll see some output, followed by series of prompts asking you to assign and confirm a password for the new user. Then you’ll be asked to enter any additional information about the new user. This is entirely optional and can be skipped by hitting ENTER
if you don’t wish to enter information into these fields.
Finally, you’ll be asked to confirm that the information you provided was correct. Enter Y
to continue. The whole process looks like this:
OutputAdding user `sammy' ...
Adding new group `sammy' (1001) ...
Adding new user `sammy' (1001) with group `sammy' ...
Creating home directory `/home/<^>sammy' ...
Copying files from `/etc/skel' ...
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Changing the user information for sammy
Enter the new value, or press ENTER for the default
Full Name []: Sammy the Shark
Room Number []: 123
Work Phone []: 555-555-5555
Home Phone []:
Other []:
Is the information correct? [Y/n] y
The adduser
command created a user, a group, and a home directory for your user. Your new user is now ready for use! You can now log in using the password you set up.
Note: Continue if you need your new user to have access to administrative functionality.
If your new user needs to execute commands with root privileges, you will need to give the new user access to sudo
. Let’s examine two approaches to this problem: Adding the user to a pre-defined sudo
user group, and specifying privileges on a per-user basis in sudo
’s configuration.
By default, sudo
on Debian 8 systems is configured to extend full privileges to any user in the sudo group.
You can see what groups your new user is in with the groups
command:
- groups sammy
Outputsammy : sammy
By default, a new user is only in their own group, which is created at the time the account was created, and shares a name with the user. In order to add the user to a new group, use the usermod
command:
- usermod -aG sudo sammy
The -aG
option here tells usermod
to add the user to the listed groups.
Use the groups
command again to verify that your user is now a member of the sudo
group:
Outputsammy : sammy sudo
Now, your new user is able to execute commands with administrative privileges.
When signed in as the new user, you can execute commands as your regular user by typing commands as normal:
- ls ~
You can execute commands with administrative privileges by typing sudo
in front of the command:
- sudo ls /root
When prefixing a command with sudo
, you will be prompted to enter a password. Enter the password for the user account that issued the command, not the root user’s password.
As an alternative to putting your user in the sudo group, you can use the visudo
command, which opens a configuration file called /etc/sudoers
in the system’s default editor, and explicitly specify privileges on a per-user basis.
Editing the /etc/sudoers/
file offers more flexibility, but should only be used when you need this flexibility, as it requires more maintenance when managing user accounts.
Using visudo
is the only recommended way to make changes to /etc/sudoers
, because it locks the file against multiple simultaneous edits and performs a sanity check on its contents before overwriting the file. This helps to prevent a situation where you misconfigure sudo
and are prevented from fixing the problem because you have lost sudo
privileges.
If you are currently signed in as root, type:
- visudo
If you are signed in using a non-root user with sudo
privileges, type:
- sudo visudo
Traditionally, visudo
opened /etc/sudoers
in the vi
editor, which can be confusing for inexperienced users. By default on new Debian installations, it should instead use nano
, which provides a more familiar text editing experience. Use the arrow keys to move the cursor, and search for the line that looks like this:
root ALL=(ALL:ALL) ALL
Below this line, copy the format you see here, changing only the word “root” to reference the new user that you would like to give sudo privileges to:
root ALL=(ALL:ALL) ALL
sammy ALL=(ALL:ALL) ALL
You should add a new line like this for each user that should be given full sudo privileges. When you are finished, you can save and close the file by hitting CTRL-X
, followed by Y
, and then ENTER
to confirm.
In the event that you no longer need a user, it is best to delete the old account.
You can delete the user itself, without deleting any of their files, by typing this as root:
- deluser sammy
If you are signed in as another non-root user with sudo privileges, you could instead type:
- sudo deluser sammy
If you want to delete the user’s home directory when the user is deleted, issue the following command as root:
- deluser --remove-home sammy
If you’re running this as a non-root user with sudo privileges, you would instead type:
- sudo deluser --remove-home sammy
If you had previously configured sudo privileges for the user you deleted by editing the /etc/sudoers
file, you should remove the relevant line in the file by typing:
- visudo
Or use this if you are a non-root user with sudo privileges:
- sudo visudo
Then locate the line in the file associated with your user and remove it.
Outputroot ALL=(ALL:ALL) ALL
sammy ALL=(ALL:ALL) ALL # DELETE THIS LINE
This will prevent a new user created with the same name from being accidentally given sudo privileges.
You should now have a good handle on how to add and remove users from your Debian 8 system. Effective user management will allow you to separate users and give them only the access that they are required to do their job.
For more information about how to configure sudo
, check out our guide on how to edit the sudoers file here.
When you first create a new Debian 8 server, there are a few configuration steps that you should take early on as part of the basic setup. This will increase the security and usability of your server and will give you a solid foundation for subsequent actions.
To log into your server, you will need to know your server’s public IP address and the password for the “root” user’s account. If you have not already logged into your server, you may want to follow the first tutorial in this series, How to Connect to Your Droplet with SSH, which covers this process in detail.
If you are not already connected to your server, go ahead and log in as the root
user using the following command (substitute the highlighted word with your server’s public IP address):
- ssh root@SERVER_IP_ADDRESS
Complete the login process by accepting the warning about host authenticity, if it appears, then providing your root authentication (password or private key). If it is your first time logging into the server, with a password, you will also be prompted to change the root password.
The root user is the administrative user in a Linux environment that has very broad privileges. Because of the heightened privileges of the root account, you are actually discouraged from using it on a regular basis. This is because part of the power inherent with the root account is the ability to make very destructive changes, even by accident.
The next step is to set up an alternative user account with a reduced scope of influence for day-to-day work. We’ll teach you how to gain increased privileges during the times when you need them.
Once you are logged in as root
, we’re prepared to add the new user account that we will use to log in from now on.
This example creates a new user called “demo”, but you should replace it with a user name that you like:
- adduser demo
You will be asked a few questions, starting with the account password.
Enter a strong password and, optionally, fill in any of the additional information if you would like. This is not required and you can just hit “ENTER” in any field you wish to skip.
Now, we have a new user account with regular account privileges. However, we may sometimes need to do administrative tasks.
To avoid having to log out of our normal user and log back in as the root account, we can set up what is known as “super user” or root privileges for our normal account. This will allow our normal user to run commands with administrative privileges by putting the word sudo
before each command.
Debian 8 doesn’t come with sudo
installed, so let’s install it with apt-get.
First, update the apt package index:
- apt-get update
Then use this command to install sudo:
- apt-get install sudo
Now you are able to use the sudo
and visudo
commands.
To add these privileges to our new user, we need to add the new user to the “sudo” group. By default, on Debian 8, users who belong to the “sudo” group are allowed to use the sudo
command.
As root
, run this command to add your new user to the sudo group (substitute the highlighted word with your new user):
- usermod -a -G sudo demo
Now your user can run commands with super user privileges! For more information about how this works, check out this sudoers tutorial.
The next step in securing your server is to set up public key authentication for your new user. Setting this up will increase the security of your server by requiring a private SSH key to log in.
If you do not already have an SSH key pair, which consists of a public and private key, you need to generate one. If you already have a key that you want to use, skip to the Copy the Public Key step.
To generate a new key pair, enter the following command at the terminal of your local machine (ie. your computer):
- ssh-keygen
Assuming your local user is called “localuser”, you will see output that looks like the following:
ssh-keygen outputGenerating public/private rsa key pair.
Enter file in which to save the key (/Users/localuser/.ssh/id_rsa):
Hit return to accept this file name and path (or enter a new name).
Next, you will be prompted for a passphrase to secure the key with. You may either enter a passphrase or leave the passphrase blank.
Note: If you leave the passphrase blank, you will be able to use the private key for authentication without entering a passphrase. If you enter a passphrase, you will need both the private key and the passphrase to log in. Securing your keys with passphrases is more secure, but both methods have their uses and are more secure than basic password authentication.
This generates a private key, id_rsa
, and a public key, id_rsa.pub
, in the .ssh
directory of the localuser’s home directory. Remember that the private key should not be shared with anyone who should not have access to your servers!
After generating an SSH key pair, you will want to copy your public key to your new server. We will cover two easy ways to do this.
Note: The ssh-copy-id
method will not work on DigitalOcean if an SSH key was selected during Droplet creation. This is because DigitalOcean disables password authentication if an SSH key is present, and the ssh-copy-id
relies on password authentication to copy the key.
If you are using DigitalOcean and selected an SSH key during Droplet creation, use option 2 instead.
If your local machine has the ssh-copy-id
script installed, you can use it to install your public key to any user that you have login credentials for.
Run the ssh-copy-id
script by specifying the user and IP address of the server that you want to install the key on, like this:
- ssh-copy-id demo@SERVER_IP_ADDRESS
After providing your password at the prompt, your public key will be added to the remote user’s .ssh/authorized_keys
file. The corresponding private key can now be used to log into the server.
Assuming you generated an SSH key pair using the previous step, use the following command at the terminal of your local machine to print your public key (id_rsa.pub
):
- cat ~/.ssh/id_rsa.pub
This should print your public SSH key, which should look something like the following:
id_rsa.pub contentsssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBGTO0tsVejssuaYR5R3Y/i73SppJAhme1dH7W2c47d4gOqB4izP0+fRLfvbz/tnXFz4iOP/H6eCV05hqUhF+KYRxt9Y8tVMrpDZR2l75o6+xSbUOMu6xN+uVF0T9XzKcxmzTmnV7Na5up3QM3DoSRYX/EP3utr2+zAqpJIfKPLdA74w7g56oYWI9blpnpzxkEd3edVJOivUkpZ4JoenWManvIaSdMTJXMy3MtlQhva+j9CgguyVbUkdzK9KKEuah+pFZvaugtebsU+bllPTB0nlXGIJk98Ie9ZtxuY3nCKneB+KjKiXrAvXUPCI9mWkYS/1rggpFmu3HbXBnWSUdf localuser@machine.local
Select the public key, and copy it to your clipboard.
To enable the use of SSH key to authenticate as the new remote user, you must add the public key to a special file in the user’s home directory.
On the server, as the root
user, enter the following command to switch to the new user (substitute your own user name):
- su - demo
Now you will be in your new user’s home directory.
Create a new directory called .ssh
and restrict its permissions with the following commands:
- mkdir .ssh
- chmod 700 .ssh
Now open a file in .ssh called authorized_keys
with a text editor. We will use nano to edit the file:
- nano .ssh/authorized_keys
Now insert your public key (which should be in your clipboard) by pasting it into the editor.
Hit CTRL-X
to exit the file, then Y
to save the changes that you made, then ENTER
to confirm the file name.
Now restrict the permissions of the authorized_keys file with this command:
- chmod 600 .ssh/authorized_keys
Type this command once to return to the root
user:
- exit
Now you may SSH login as your new user, using the private key as authentication.
To read more about how key authentication works, read this tutorial: How To Configure SSH Key-Based Authentication on a Linux Server.
Now that we have our new account, we can secure our server a little bit by modifying its SSH daemon configuration (the program that allows us to log in remotely) to disallow remote SSH access to the root account.
Begin by opening the configuration file with your text editor as root:
- nano /etc/ssh/sshd_config
Here, we have the option to disable root login through SSH. This is generally a more secure setting since we can now access our server through our normal user account and escalate privileges when necessary.
To disable remote root logins, we need to find the line that looks like this:
#PermitRootLogin yes
You can modify this line to “no” like this if you want to disable root login:
PermitRootLogin no
Disabling remote root login is highly recommended on every server!
When you are finished making your changes, save and close the file using the method we went over earlier (CTRL-X
, then Y
, then ENTER
).
Now that we have made our changes, we need to restart the SSH service so that it will use our new configuration.
Type this to restart SSH:
- systemctl restart ssh
Now, before we log out of the server, we should test our new configuration. We do not want to disconnect until we can confirm that new connections can be established successfully.
Open a new terminal window. In the new window, we need to begin a new connection to our server. This time, instead of using the root account, we want to use the new account that we created.
- ssh demo@SERVER_IP_ADDRESS
You will be prompted for the new user’s password that you configured. After that, you will be logged in as your new user.
Remember, if you need to run a command with root privileges, type “sudo” before it like this:
- sudo command_to_run
If all is well, you can exit your sessions by typing:
- exit
At this point, you have a solid foundation for your Debian 8 server. You can install any of the software you need on your server now.
]]>In recent years, Linux distributions have increasingly transitioned from other init systems to systemd
. The systemd
suite of tools provides a fast and flexible init model for managing an entire machine from boot onwards.
In this guide, we’ll give you a quick run through of the most important commands you’ll want to know for managing a systemd
enabled server. These should work on any server that implements systemd
(any OS version at or above Ubuntu 15.04, Debian 8, CentOS 7, Fedora 15). Let’s get started.
The basic object that systemd
manages and acts upon is a “unit”. Units can be of many types, but the most common type is a “service” (indicated by a unit file ending in .service
). To manage services on a systemd
enabled server, our main tool is the systemctl
command.
All of the normal init system commands have equivalent actions with the systemctl
command. We will use the nginx.service
unit to demonstrate (you’ll have to install Nginx with your package manager to get this service file).
For instance, we can start the service by typing:
- sudo systemctl start nginx.service
We can stop it again by typing:
- sudo systemctl stop nginx.service
To restart the service, we can type:
- sudo systemctl restart nginx.service
To attempt to reload the service without interrupting normal functionality, we can type:
- sudo systemctl reload nginx.service
By default, most systemd
unit files are not started automatically at boot. To configure this functionality, you need to “enable” to unit. This hooks it up to a certain boot “target”, causing it to be triggered when that target is started.
To enable a service to start automatically at boot, type:
- sudo systemctl enable nginx.service
If you wish to disable the service again, type:
- sudo systemctl disable nginx.service
There is a great deal of information that we can pull from a systemd
server to get an overview of the system state.
For instance, to get all of the unit files that systemd
has listed as “active”, type (you can actually leave off the list-units
as this is the default systemctl
behavior):
- systemctl list-units
To list all of the units that systemd
has loaded or attempted to load into memory, including those that are not currently active, add the --all
switch:
- systemctl list-units --all
To list all of the units installed on the system, including those that systemd
has not tried to load into memory, type:
- systemctl list-unit-files
A systemd
component called journald
collects and manages journal entries from all parts of the system. This is basically log information from applications and the kernel.
To see all log entries, starting at the oldest entry, type:
- journalctl
By default, this will show you entries from the current and previous boots if journald
is configured to save previous boot records. Some distributions enable this by default, while others do not (to enable this, either edit the /etc/systemd/journald.conf
file and set the Storage=
option to “persistent”, or create the persistent directory by typing sudo mkdir -p /var/log/journal
).
If you only wish to see the journal entries from the current boot, add the -b
flag:
- journalctl -b
To see only kernel messages, such as those that are typically represented by dmesg
, you can use the -k
flag:
- journalctl -k
Again, you can limit this only to the current boot by appending the -b
flag:
- journalctl -k -b
While the above commands gave you access to the general system state, you can also get information about the state of individual units.
To see an overview of the current state of a unit, you can use the status
option with the systemctl
command. This will show you whether the unit is active, information about the process, and the latest journal entries:
- systemctl status nginx.service
To see all of the journal entries for the unit in question, give the -u
option with the unit name to the journalctl
command:
- journalctl -u nginx.service
As always, you can limit the entries to the current boot by adding the -b
flag:
- journalctl -b -u nginx.service
By now, you know how to modify a unit’s state by starting or stopping it, and you know how to view state and journal information to get an idea of what is happening with the process. However, we haven’t seen yet how to inspect other aspects of units and unit files.
A unit file contains the parameters that systemd
uses to manage and run a unit. To see the full contents of a unit file, type:
- systemctl cat nginx.service
To see the dependency tree of a unit (which units systemd
will attempt to activate when starting the unit), type:
- systemctl list-dependencies nginx.service
This will show the dependent units, with target
units recursively expanded. To expand all dependent units recursively, pass the --all
flag:
- systemctl list-dependencies --all nginx.service
Finally, to see the low-level details of the unit’s settings on the system, you can use the show
option:
- systemctl show nginx.service
This will give you the value of each parameter being managed by systemd
.
If you need to make a modification to a unit file, systemd
allows you to make changes from the systemctl
command itself so that you don’t have to go to the actual disk location.
To add a unit file snippet, which can be used to append or override settings in the default unit file, simply call the edit
option on the unit:
- sudo systemctl edit nginx.service
If you prefer to modify the entire content of the unit file instead of creating a snippet, pass the --full
flag:
- sudo systemctl edit --full nginx.service
After modifying a unit file, you should reload the systemd
process itself to pick up your changes:
- sudo systemctl daemon-reload
Another function of an init system is to transition the server itself between different states. Traditional init systems typically refer to these as “runlevels”, allowing the system to only be in one runlevel at any one time.
In systemd
, “targets” are used instead. Targets are basically synchronization points that the server can used to bring the server into a specific state. Service and other unit files can be tied to a target and multiple targets can be active at the same time.
To see all of the targets available on your system, type:
- systemctl list-unit-files --type=target
To view the default target that systemd
tries to reach at boot (which in turn starts all of the unit files that make up the dependency tree of that target), type:
- systemctl get-default
You can change the default target that will be used at boot by using the set-default
option:
- sudo systemctl set-default multi-user.target
To see what units are tied to a target, you can type:
- systemctl list-dependencies multi-user.target
You can modify the system state to transition between targets with the isolate
option. This will stop any units that are not tied to the specified target. Be sure that the target you are isolating does not stop any essential services:
- sudo systemctl isolate multi-user.target
For some of the major states that a system can transition to, shortcuts are available. For instance, to power off your server, you can type:
- sudo systemctl poweroff
If you wish to reboot the system instead, that can be accomplished by typing:
- sudo systemctl reboot
You can boot into rescue mode by typing:
- sudo systemctl rescue
Note that most operating systems include traditional aliases to these operations so that you can simply type sudo poweroff
or sudo reboot
without the systemctl
. However, this is not guaranteed to be set up on all systems.
By now, you should know the basics of how to manage a server that uses systemd
. However, there is much more to learn as your needs expand. Below are links to guides with more in-depth information about some of the components we discussed in this guide:
By learning how to leverage your init system’s strengths, you can control the state of your machines and more easily manage your services and processes.
]]>In the Linux ecosystem, installing, maintaining and upgrading software on a periodic basis is a must. However, tracking changes made to local configuration files is still necessary. As opposed to the old standby of making copies of configuration files before making changes, etckeeper lets you keep track of modifications using a Git, Mercurial, Bazaar, or Darcs repository, just like you would do with a software development project.
In addition, etckeeper integrates seamlessly with yum to automatically commit changes made to the contents of the /etc
directory when packages are upgraded. This will allow you to revert to previous versions of your configuration files should you want — or need — to do so.
To follow this tutorial, you will need:
etckeeper only tracks file permissions, metadata, and changes. It does not provide out-of-the-box tools to perform the restoration of files, so an understanding of the fundamentals of a revision control system is necessary.
In this article we will use Git, which is the default VCS that etckeeper uses. If you want to refresh your memory about Git and version control, you can check the this tutorial series. Although you will not use Git directly in this guide, you can run Git-specific commands through etckeeper.
In this step, we will install etckeeper.
First, you will need to enable EPEL (Extra Packages for Enterprise Linux) on your CentOS 7 server, because that is the repository which contains etckeeper.
sudo yum update && sudo yum install epel-release
Then install etckeeper.
sudo yum update && sudo yum install etckeeper
Git comes with CentOS 7 by default, so we don’t need to install it.
Once you have installed etckeeper, the next step is updating the /etc/etckeeper/etckeeper.conf
configuration file.
First, open the configuration file with Nano or your favorite text editor.
sudo nano /etc/etckeeper/etckeeper.conf
The following are the essential variables that you need to configure in order for etckeeper to work properly. Feel free to leave the rest of the settings commented out.
First, under the comment # The VCS to use.
, make sure the VCS="git"
is uncommented (i.e. there is no #
at the beginning of the line). By default, this option will already be uncommented as git is the default VCS used by etckeeper installations on CentOS 7.
If you want to prevent etckeeper from committing changes automatically once per day, make sure the AVOID_DAILY_AUTOCOMMITS=1
is uncommented. In order to decide if you want to set this, you should consider whether your system configuration files undergo frequent changes (e.g. testing environments often change every day). If so, you should comment out that line; otherwise, you can leave it commented.
If you want yum install to abort when there are uncommitted changes in /etc
, make sure to uncomment AVOID_COMMIT_BEFORE_INSTALL=1
. This will require a manual commit before using yum to install packages. Otherwise, leave it commented out and yum will automatically commit the updated files before running an installation. This choice is entirely up to you; it depends largely on your environment, and the quantity of changes. This is much like the previous example, except that this time it will depend on the frequency with which you install packages.
When you’re done updating the options, save and close the file.
In this step, we will initialize the Git repository in /etc
.
First, change to the /etc
directory.
cd /etc
Next, initialize the repository by running the following command.
sudo etckeeper init
You should get the following message:
Initialized empty Git repository in /etc/.git/
You should now see the .git
directory and the .gitignore
file inside /etc
. For example, if you run the following command:
ls -la /etc | grep git
You should see these lines included in the output:
drwx------. 7 root root 4096 Apr 2 21:42 .git
-rw-r--r--. 1 root root 874 Apr 2 21:42 .gitignore
Note: .git
must be protected in the local system (hence the read, write, and execute permissions for the superuser only); because version control systems don’t keep track of file permissions by themselves, etckeeper provides this feature instead.
The .git
directory contains several configuration and description files and other subdirectories intended for the use of Git itself. The .gitignore
file, which specifies explicitly untracked files that git should ignore, is intended to be managed by etckeeper in its entirety. It is not advisable to edit it by hand, with one exception.
If there are certain files that you don’t want to track using version control, you can add them to the .gitignore file manually. To stop tracking a file, first open .gitignore
for editing.
sudo nano .gitignore
The last line of the file will read # end section managed by etckeeper
. Add the filenames of the files you want to ignore, one per line, above this one.
file_to_ignore
another_file_to_ignore
# end section managed by etckeeper
Then save and close the file.
You additionally will need to remove those files from the cache that is currently being managed by git since you initialized the local repository earlier.
etckeeper vcs rm --cached file_to_ignore
Repeat the above command for as many files as you previously added to .gitignore
.
/etc
to the Git RepositoryIn this step, we will commit our initial /etc
.
Adding your first commit is easy; simply enter the following command. Although not strictly necessary, you should add a description to each commit to be able to easily identify them later.
sudo etckeeper commit "First commit of my /etc directory"
You should then see the outputted list of files being committed to your repository, like the below (truncated):
create mode 100644 selinux/targeted/modules/active/modules/dnsmasq.pp
create mode 100644 selinux/targeted/modules/active/modules/dnssec.pp
create mode 100644 selinux/targeted/modules/active/modules/docker.pp
create mode 100644 selinux/targeted/modules/active/modules/dovecot.pp
In this step, we will make some changes to a file in /etc
and commit them. In the next step, we will revert these changes.
First, make a change to the contents of a file of your choice. For example, you can add a new host to your local name resolution by adding a line consisting of an IP address and its associated hostname at the end of /etc/hosts
.
First, open the file.
sudo nano /etc/hosts
Then, add the following line to the end of the file.
192.168.0.2 node01
Save and close the file. Now, let’s commit this change.
sudo etckeeper commit "Added a line to the hosts file"
Finally, modify the file’s permissions.
sudo chmod 640 /etc/hosts
And modify its ownership (making sure to replace sammy with your own username).
sudo chown sammy:sammy /etc/hosts
You can check the current permissions and ownership of /etc/hosts
.
ls -l /etc/hosts
The output should look like this:
-rw-r----- 1 sammy sammy 675 Apr 17 15:01 /etc/hosts
Now let’s test the restoring capabilities of etckeeper — not just of the file and its contents, but also its permissions and ownership.
First, list the commits you’ve made so far.
sudo git log --pretty=oneline
The first column of the output is a SHA-1 hash that uniquely identifies the commit; the second is the description that you used when you submit the changes earlier.
Your output should look similar to this, with different hashes.
d0958fbe4d947a6a3ad98141f9fe89d1fd1a95c4 Added a line to the hosts file
76c193da740a3e137fa000773a79de8bb5c898b7 First commit of my /etc directory
Take note of the hash of each commit. You will use them to roll back to that previous state.
Let’s roll back /etc/hosts
to how it looked before we started this tutorial. Replace the characters in red with the SHA-1 hash that corresponds to your first commit. Note that you don’t need to specify the whole SHA-1 hash string; a few characters that uniquely identify it will do.
sudo etckeeper vcs checkout 76c19 /etc/hosts
Now we can check the contents, permissions, and ownership of /etc/hosts
to see that they’ve been changed back.
Look at the last few lines of the file.
tail /etc/hosts
Should give you an output like this, which is missing the 192.168.0.2 node01
line we added, as expected.
...
# The following lines are desirable for IPv6 capable hosts
::1 test-etckeeper test-etckeeper
::1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
Check the current permissions and ownership of /etc/hosts
again.
ls -l /etc/hosts
You’ll see an output that looks like this.
-rw-r--r-- 1 root root 704 Apr 17 15:01 /etc/hosts
The contents of the file were restored correctly, as well as the permissions and ownership.
In this tutorial we have explained how to use etckeeper, a great tool to store your /etc
directory in a Git repository. You can also use to a Bazaar, Mercurial, or Darcs repository.
Regardless of the VCS of your choice, etckeeper will help you keep on top of your configuration files and ensure that you can always roll back to a previous state if you need to undo changes or modify functionality.
]]><p>In this article, we will discuss how to use dig to verify your domain name settings and return data about how the internet sees your domain. We will also examine a few other related tools like “whois” and “ping”.</p>
<p>We will be using an Ubuntu 12.04 VPS to test the commands in this guide, but any modern Linux distribution should function in a similar way.</p>
<h2>How to Use Dig</h2>
<p>The most basic way to use dig is to specify the domain we wish to query:</p>
<pre>dig <span class=“highlight”>example.com</span></pre>
<p>We can test “duckduckgo.com” to find out what kind of information it returns:</p>
<pre>dig duckduckgo.com</pre>
<pre>; <<>> DiG 9.8.1-P1 <<>> duckduckgo.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64399 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION: ;duckduckgo.com. IN A
;; ANSWER SECTION: duckduckgo.com. 99 IN A 107.21.1.61 duckduckgo.com. 99 IN A 184.72.106.253 duckduckgo.com. 99 IN A 184.72.106.52 duckduckgo.com. 99 IN A 184.72.115.86
;; Query time: 33 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Fri Aug 23 14:26:17 2013 ;; MSG SIZE rcvd: 96</pre>
<p>This is a lot of information. Let’s examine it in smaller chunks:</p>
<pre>; <<>> DiG 9.8.1-P1 <<>> duckduckgo.com ;; global options: +cmd</pre>
<p>The lines above act as a header for the query performed. It is possible to run dig in batch mode, so proper labeling of the output is essential to allow for correct analysis.</p>
<pre>;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64399 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0</pre>
<p>The next section gives us a technical summary of our query results. We can see that the query was successful, certain flags were used, and that 4 “answers” were received.</p>
<pre>;; QUESTION SECTION: ;duckduckgo.com. IN A
;; ANSWER SECTION: duckduckgo.com. 99 IN A 107.21.1.61 duckduckgo.com. 99 IN A 184.72.106.253 duckduckgo.com. 99 IN A 184.72.106.52 duckduckgo.com. 99 IN A 184.72.115.86</pre>
<p>The above section of the output contains the actual results we were looking for. It restates the query and then returns the matching DNS records for that domain name.</p>
<p>Here, we can see that there are four “A” records for “duckduckgo.com”. By default, “A” records are returned. This gives us the IP addresses that the domain name resolves to.</p>
<p>The “99” is the TTL (time to live) before the DNS server rechecks the association between the domain name and the IP address. The “IN” means the class of the record is a standard internet class.</p>
<pre>;; Query time: 33 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Fri Aug 23 14:26:17 2013 ;; MSG SIZE rcvd: 96</pre>
<p>These lines simply provide some statistics about the actual query results. The query time can be indicative of problems with the DNS servers.</p>
<h2>How to Use Dig to Test DNS Records</h2>
<p>If you’ve <a href=“https://www.digitalocean.com/community/articles/how-to-set-up-a-host-name-with-digitalocean”>set up a domain name with DigitalOcean</a>, you can use dig to query the information.</p>
<p>To test that your “A” records are set correctly, type:</p>
<pre>dig <span class=“highlight”>your_domain_name.com</span></pre>
<p>If you would like to check if your mail servers are directing correctly, type:</p>
<pre>dig <span class=“highlight”>your_domain_name.com</span> MX</pre>
<p>In general, you can issue the type of record that you would like to query after the domain name in the query.</p>
<p>If you would like to receive information about all of the records, type:</p>
<pre>dig <span class=“highlight”>your_domain_name.com</span> ANY</pre>
<p>This will return any records that match the base domain you set up. These will include “SOA” records, “NS” records, “A” records, and “MX” records.</p>
<p><strong>Note: Due to the way that TTL and DNS works, it sometimes takes awhile for the changes you create to propagate to the name servers. If you have created a record and do not see it, wait until the TTL reaches 0 to see if your record shows up.</strong></p>
<p>If you would like to only return the actual IP the domain points to, you can type:</p>
<pre>dig <span class=“highlight”>your_domain_name.com</span> +short</pre>
<h3><strong>Using the “host” Command Instead of “dig”</strong></h3>
<p>An alternative to dig is a command called “<strong>host</strong>”. This command functions in a very similar way to dig, with many of the same options.</p>
<p>The basic syntax is:</p>
<pre>host <span class=“highlight”>domain_name_or_IP_address</span></pre>
<p>Notice how you do not need a flag to change the functionality from regular DNS lookup to reverse DNS lookup.</p>
<p>As with dig, you can specify the type of record you are interested in. This is done with the “-t” flag.</p>
<p>To return the mx records of google, you can type:</p>
<pre>host -t mx google.com</pre>
<pre>google.com mail is handled by 50 alt4.aspmx.l.google.com. google.com mail is handled by 10 aspmx.l.google.com. google.com mail is handled by 40 alt3.aspmx.l.google.com. google.com mail is handled by 30 alt2.aspmx.l.google.com. google.com mail is handled by 20 alt1.aspmx.l.google.com.</pre>
<p>Other types of records can be retrieved just as easily.</p>
<p>You can return all records with the “-a” flag. I will not post the output of this command, because it can be quite long:</p>
<pre>host -a google.com</pre>
<p>If you need additional information about the host, you can turn on verbose output with the “-v” flag:</p>
<pre>host -v google.com</pre>
<p>This will provide extended information.</p>
<h2>Using Other Tools to Query DNS Information</h2>
<h3><strong>Ping</strong></h3>
<p>A simple way to check if your domain name is resolving correctly is “ping”.</p>
<p>It’s usage is incredibly simple:</p>
<pre>ping <span class=“highlight”>your_domain_name.com</span></pre> <pre>PING your_domain_name.com (192.241.160.34) 56(84) bytes of data. 64 bytes from 192.241.160.34: icmp_req=1 ttl=64 time=0.026 ms 64 bytes from 192.241.160.34: icmp_req=2 ttl=64 time=0.038 ms 64 bytes from 192.241.160.34: icmp_req=3 ttl=64 time=0.037 ms . . .</pre>
<p>This will continue to output information until you type “CTRL-C”.</p>
<p>You can also tell the software to ping only a certain number of times. This will ping 3 times:</p>
<pre>ping -c 3 <span class=“highlight”>your_domain_name.com</span></pre>
<pre>PING your_domain_name.com (192.241.160.34) 56(84) bytes of data. 64 bytes from 192.241.160.34: icmp_req=1 ttl=64 time=0.027 ms 64 bytes from 192.241.160.34: icmp_req=2 ttl=64 time=0.059 ms 64 bytes from 192.241.160.34: icmp_req=3 ttl=64 time=0.042 ms
— your_domain_name.com ping statistics — 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 0.027/0.042/0.059/0.015 ms</pre>
<p>This command can be used to simply check that the domain name resolves to the IP address that you assigned.</p>
<h3><strong>Whois</strong></h3>
<p>The whois protocol returns information about registered domain names, including the name servers they are configured to work with.</p>
<p>While most of the information concerns the registration of the domain, it can be helpful to see that the name servers are returned correctly.</p>
<p>Run the command like this:</p>
<pre>whois <span class=“highlight”>your_domain_name.com</span></pre>
<p>This will return a long list of information. The formatting will vary based on the whois server that contains the information.</p>
<p>Towards the bottom, you can usually see the domain servers that provide the domain forwarding to the correct IP addresses.</p>
<h2>Conclusion</h2>
<p>While dig, ping, and whois are all simple tools that perform basic checks on your domain names, they can be incredibly useful. When you are setting up your domain names, having a terminal open with dig handy can save you a lot of guesswork.</p>
<div class=“author”>By Justin Ellingwood</div>
]]>