Tutorial

The Basics of Using the Sed Stream Editor to Manipulate Text in Linux

The Basics of Using the Sed Stream Editor to Manipulate Text in Linux

Introduction

The sed command, short for stream editor, performs editing operations on text coming from standard input or a file. sed edits line-by-line and in a non-interactive way.

This means that you make all of the editing decisions as you are calling the command, and sed executes the directions automatically. This may seem confusing or unintuitive, but it is a very powerful and fast way to transform text, especially as part of a script or automated workflow.

This tutorial will cover some basic operations and introduce you to the syntax required to operate this editor. You will almost certainly never replace your regular text editor with sed, but it will probably become a welcomed addition to your text editing toolbox.

Note: This tutorial uses the GNU version of sed found on Ubuntu and other Linux operating systems. If you’re using macOS, you’ll have the BSD version which has different options and arguments. You can install the GNU version of sed with Homebrew using brew install gnu-sed.

Basic Usage

sed operates on a stream of text that it reads from either a text file or from standard input (STDIN). This means that you can send the output of another command directly into sed for editing, or you can work on a file that you’ve already created.

You should also be aware that sed outputs everything to standard out (STDOUT) by default. That means that, unless redirected, sed will print its output to the screen instead of saving it in a file.

The basic usage is:

  1. sed [options] commands [file-to-edit]

In this tutorial, you’ll use a copy of the BSD Software License to experiment with sed. On Ubuntu, execute the following commands to copy the BSD license file to your home directory so you can work with it:

  1. cd
  2. cp /usr/share/common-licenses/BSD .

If you don’t have a local copy of the BSD license, create one yourself with this command:

  1. cat << 'EOF' > BSD
  2. Copyright (c) The Regents of the University of California.
  3. All rights reserved.
  4. Redistribution and use in source and binary forms, with or without
  5. modification, are permitted provided that the following conditions
  6. are met:
  7. 1. Redistributions of source code must retain the above copyright
  8. notice, this list of conditions and the following disclaimer.
  9. 2. Redistributions in binary form must reproduce the above copyright
  10. notice, this list of conditions and the following disclaimer in the
  11. documentation and/or other materials provided with the distribution.
  12. 3. Neither the name of the University nor the names of its contributors
  13. may be used to endorse or promote products derived from this software
  14. without specific prior written permission.
  15. THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
  16. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
  17. IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
  18. ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
  19. FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
  20. DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
  21. OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
  22. HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
  23. LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
  24. OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
  25. SUCH DAMAGE.
  26. EOF

Let’s use sed to view the contents of the BSD license file. sed sends its results to the screen by default, which means you can use it as a file reader by passing it no editing commands. Try executing the following command:

  1. sed '' BSD

You’ll see the BSD license displayed to the screen:

Output
Copyright (c) The Regents of the University of California. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. ... ...

The single quotes contain the editing commands you pass to sed. In this case, you passed it nothing, so sed printed each line it received to standard output.

sed can use standard input rather than a file. Pipe the output of the cat command into sed to produce the same result:

  1. cat BSD | sed ''

You’ll see the output of the file:

Output
Copyright (c) The Regents of the University of California. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. . . . . . .

As you can see, you can operate on files or streams of text, like the ones produced when piping output with the pipe (|) character, just as easily.

Printing Lines

In the previous example, you saw that input passed into sed without any operations would print the results directly to standard output.

Let’s explore sed’s explicit print command, which you specify by using the p character within single quotes.

Execute the following command:

  1. sed 'p' BSD

You’ll see each line of the BSD file printed twice:

Output
Copyright (c) The Regents of the University of California. Copyright (c) The Regents of the University of California. All rights reserved. All rights reserved. Redistribution and use in source and binary forms, with or without Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions modification, are permitted provided that the following conditions are met: are met: . . . . . .

sed automatically prints each line by default, and then you’ve told it to print lines explicitly with the “p” command, so you get each line printed twice.

If you examine the output closely, you’ll see that it has the first line twice, followed by the second line twice, etc, which tells you that sed operates on data line by line. It reads a line, operates on it, and outputs the resulting text before repeating the process on the next line.

You can clean up the results by passing the -n option to sed, which suppresses the automatic printing:

  1. sed -n 'p' BSD
Output
Copyright (c) The Regents of the University of California. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. . . . . . .

We now are back to printing each line once.

The examples so far can hardly be considered editing (unless you wanted to print each line twice…). Next you’ll explore how sed can modify the output by targeting specific sections of the text data.

Using Address Ranges

Addresses let you target specific parts of a text stream. You can specify a specific line or even a range of lines.

Let’s have sed print the first line of the file. Execute the following command:

  1. sed -n '1p' BSD

The first line prints to the screen:

Output
Copyright (c) The Regents of the University of California.

By placing the number 1 before the print command, you told sed the line number to operate on. You can just as easily print five lines (don’t forget the “-n”):

  1. sed -n '1,5p' BSD

You’ll see this output:

Output
Copyright (c) The Regents of the University of California. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions

You’ve just given an address range to sed. If you give sed an address, it will only perform the commands that follow on those lines. In this example, you’ve told sed to print line 1 through line 5. You could have specified this in a different way by giving the first address and then using an offset to tell sed how many additional lines to travel, like this:

  1. sed -n '1,+4p' BSD

This will result in the same output, because you told sed to start at line 1 and then operate on the next 4 lines as well.

If you want to print every other line, specify the interval after the ~ character. The following command prints every other line in the BSD file, starting with line 1:

  1. sed -n '1~2p' BSD

Here’s the output you’ll see:

Output
Copyright (c) The Regents of the University of California. modification, are permitted provided that the following conditions 1. Redistributions of source code must retain the above copyright 2. Redistributions in binary form must reproduce the above copyright documentation and/or other materials provided with the distribution. may be used to endorse or promote products derived from this software . . . . . .

You can use sed to delete text from the output as well.

Deleting Text

You can perform text deletion where you previously were specifying text printing by changing the p command to the d command.

In this case, you no longer need the -n command because sed will print everything that is not deleted. This will help you see what’s going on.

Modify the last command from the previous section to make it delete every other line starting with the first:

  1. sed '1~2d' BSD

The result is that you see every line you were not given last time:

Output
All rights reserved. Redistribution and use in source and binary forms, with or without are met: notice, this list of conditions and the following disclaimer. notice, this list of conditions and the following disclaimer in the 3. Neither the name of the University nor the names of its contributors without specific prior written permission. . . . . . .

It is important to note here that our source file is not being affected. It is still intact. The edits are output to our screen.

If we want to save our edits, we can redirect standard output to a file like so:

  1. sed '1~2d' BSD > everyother.txt

Now open the file with cat:

  1. cat everyother.txt

You see the same output that you saw onscreen previously:

Output
All rights reserved. Redistribution and use in source and binary forms, with or without are met: notice, this list of conditions and the following disclaimer. notice, this list of conditions and the following disclaimer in the 3. Neither the name of the University nor the names of its contributors without specific prior written permission. . . . . . .

The sed command does not edit the source file by default, but you can change this behavior by passing the -i option, which means “perform edits in-place.” This will alter the source file.

Warning: Using the -i switch will overwrite the original file, so you should use this with care. Perform the operations without the -i switch first and then run the command again with -i once you have what you want, create a backup of the original file, or redirect the output to a file. It’s very easy to accidentally alter the original file with the -i switch.

Let’s try it by editing the everyother.txt file you just created, in-place. Let’s further reduce the file by deleting every other line again:

  1. sed -i '1~2d' everyother.txt

If you use cat to display the file with cat everyother.txt, you’ll see that the file has been edited.

The -i option can be dangerous. Thankfully, sed gives you the ability to create a backup file prior to editing.

To create a backup file prior to editing, add the backup extension directly after the “-i” option:

  1. sed -i.bak '1~2d' everyother.txt

This creates a backup file with the .bak extension, and then edits the original file in-place.

Next you’ll look at how to use sed to perform search and replace operations.

Substituting Text

Perhaps the most well-known use for sed is substituting text. sed can search for text patterns using regular expressions, and then replace the found text with something else.

You can learn more about regular expressions by following the Using Grep Regular Expressions to Search for Text Patterns in Linux.

In its most basic form, you can change one word to another word using the following syntax:

's/old_word/new_word/'

The s is the substitute command. The three slashes (/) are used to separate the different text fields. You can use other characters to delimit the fields if it would be more helpful.

For instance, if you were trying to change a website name, using another delimiter would be helpful since URLs contain slashes.

Execute the following command to print a URL with echo and modify it with sed, using the underscore (_) character as the delimiter:

  1. echo "http://www.example.com/index.html" | sed 's_com/index_org/home_'

This replaces com/index with org/home. The output shows the modifed URL:

Output
http://www.example.org/home.html

Do not forget the final delimiter, or sed will complain. If you ran this command:

  1. echo "http://www.example.com/index.html" | sed 's_com/index_org/home'

You’d see this output:

Output
sed: -e expression #1, char 20: unterminated `s' command

Let’s create a new file to practice some substitutions. Execute the following command to create a new text file called song.txt:

  1. echo "this is the song that never ends
  2. yes, it goes on and on, my friend
  3. some people started singing it
  4. not knowing what it was
  5. and they'll continue singing it forever
  6. just because..." > song.txt

Now let’s substitute the expression on with forward. Use the following command:

  1. sed 's/on/forward/' song.txt

The output looks like this:

Output
this is the sforwardg that never ends yes, it goes forward and on, my friend some people started singing it not knowing what it was and they'll cforwardtinue singing it forever just because...

You can see a few notable things here. First, is that sed replaced patterns, not words. The on within song is changed to forward.

The other thing to notice is that on line 2, the second on was not changed to forward.

This is because by default, the s command operates on the first match in a line and then moves to the next line. To make sed replace every instance of on instead of just the first on each line, you must pass an optional flag to the substitute command.

Provide the g flag to the substitute command by placing it after the substitution set:

  1. sed 's/on/forward/g' song.txt

You’ll see this output:

Output
this is the sforwardg that never ends yes, it goes forward and forward, my friend some people started singing it not knowing what it was and they'll cforwardtinue singing it forever just because...

Now the substitute command changes every instance.

If you only wanted to change the second instance of “on” that sed finds on each line, then you would use the number 2 instead of the g:

  1. sed 's/on/forward/2' song.txt

This time the other lines are unchanged, as they don’t have a second occurrence:

Output
this is the song that never ends yes, it goes on and forward, my friend some people started singing it not knowing what it was and they'll continue singing it forever just because...

If you only want to see which lines were substituted, use the -n option again to suppress automatic printing.

You can then pass the p option to the substitute command to print lines where substitution took place.

  1. sed -n 's/on/forward/2p' song.txt

The line that changed prints to the screen:

Output
yes, it goes on and forward, my friend

As you can see, you can combine the flags at the end of the command.

If you want the search process to ignore case, you can pass it the “i” flag.

  1. sed 's/SINGING/saying/i' song.txt

Here’s the output you’ll see:

Output
this is the song that never ends yes, it goes on and on, my friend some people started saying it not knowing what it was and they'll continue saying it forever just because...

Replacing and Referencing Matched Text

If you want to find more complex patterns with regular expressions, you have a number of different methods of referencing the matched pattern in the replacement text.

For instance, to match from the beginning of the line to at, use the following command:

  1. sed 's/^.*at/REPLACED/' song.txt

You’ll see this output:

Output
REPLACED never ends yes, it goes on and on, my friend some people started singing it REPLACED it was and they'll continue singing it forever just because...

You can see that the wildcard expression matches from the beginning of the line to the last instance of at.

Since you don’t know the exact phrase that will match in the search string, you can use the & character to represent the matched text in the replacement string.

Let’s put parentheses around the matched text:

  1. sed 's/^.*at/(&)/' song.txt

You’ll see this output:

Output
(this is the song that) never ends yes, it goes on and on, my friend some people started singing it (not knowing what) it was and they'll continue singing it forever just because...

A more flexible way of referencing matched text is to use escaped parentheses to group sections of matched text.

Every group of search text marked with parentheses can be referenced by an escaped reference number. For instance, the first parentheses group can be referenced with \1, the second with \2 and so on.

In this example, we’ll switch the first two words of each line:

  1. sed 's/\([a-zA-Z0-9][a-zA-Z0-9]*\) \([a-zA-Z0-9][a-zA-Z0-9]*\)/\2 \1/' song.txt

You’ll see this output:

Output
is this the song that never ends yes, goes it on and on, my friend people some started singing it knowing not what it was they and'll continue singing it forever because just...

As you can see, the results are not perfect. For instance, the second line skips the first word because it has a character not listed in our character set. Similarly, it treated they'll as two words in the fifth line.

Let’s improve the regular expression to be more accurate:

  1. sed 's/\([^ ][^ ]*\) \([^ ][^ ]*\)/\2 \1/' song.txt

You’ll see this output:

Output
is this the song that never ends it yes, goes on and on, my friend people some started singing it knowing not what it was they'll and continue singing it forever because... just

This is much better than last time. This groups punctuation with the associated word.

Notice how we repeat the expression inside the parentheses (once without the * character, and then once with it). This is because the * character matches the character set that comes before it zero or more times. This means that the match with the wildcard would be considered a “match” even if the pattern is not found.

To ensure that sed finds the text at least once, you must match it once without the wildcard before employing the wildcard.

Conclusion

In this tutorial you explored the sed command. You printed specific lines from the file, searched for text, deleted lines, overwrote the original file, and used regular expressions to replace text. You should be able to see already how you can quickly transform a text document using properly constructed sed commands.

In the next article in this series, you will explore some more advanced features.

Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.

Learn more about our products


Tutorial Series: Using Sed

In this interactive tutorial series, you’ll explore basic and advanced operations with sed , the command line stream editor found in Linux. You will almost certainly never replace your regular text editor with sed, but it will certainly become a welcomed addition to your text editing toolbox.

About the authors


Still looking for an answer?

Ask a questionSearch for more help

Was this helpful?
 
3 Comments


This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

echo "The best tutorial I found so far for esd" | sed -n 's/es/se/2p'

Thanks!!

address ranges — this doesn’t work for me on Mac OS X Yosemite:

sed: 1: "1,+4p": expected context address
$ what /usr/bin/sed
/usr/bin/sed
	PROGRAM:sed  PROJECT:text_cmds-88```

But does work on Ubuntu 14.04
```# sed --version
sed (GNU sed) 4.2.2
Copyright (C) 2012 Free Software Foundation, Inc.

You can resolve the issue with Mac OS X by installing the GNU sed with MacPorts:

[...]
$ gsed -n '1~2p' BSD
Copyright (c) The Regents of the University of California.
[...]```

Nice tutorial. Did find a minor typo - annoying.text for annoying.txt

Daniel Goldman - http://www.sed-book.com/

Try DigitalOcean for free

Click below to sign up and get $200 of credit to try our products over 60 days!

Sign up

Join the Tech Talk
Success! Thank you! Please check your email for further details.

Please complete your information!

Featured on Community

Get our biweekly newsletter

Sign up for Infrastructure as a Newsletter.

Hollie's Hub for Good

Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.

Become a contributor

Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

Welcome to the developer cloud

DigitalOcean makes it simple to launch in the cloud and scale up as you grow — whether you're running one virtual machine or ten thousand.

Learn more