Sed: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
 
(61 intermediate revisions by the same user not shown)
Line 1: Line 1:
=External=
* https://www.gnu.org/software/sed/manual/sed.html
=Internal=
=Internal=


* [[Linux#Commands|Linux]]
* [[Linux#Commands|Linux]]
* [[Sed_Regular_Expressions#Overview|sed Regular Expressions]]
* [[awk#Overview|awk]]
=GNU sed vs. BSD sed=
<syntaxhighlight lang='bash'>
sed [options] -e [command] file1 file2 ...
</syntaxhighlight>
GNU sed supports extra functionality relative to the BDS sed - for example [[#Convert_a_Character_to_Upper_or_Lower_Case|\L and \U functions to turn a character to lower and upper cases]]. macOS comes with the BSD sed, so that is why it's probably a good idea to make [[Install_gnu_sed_on_Mac#Overview|upgrading to GNU sed part of the configuring a new Mac]].
This is how you know you're using a BSD sed: if no GNU sed was installed, 'man sed' displays in the header "BSD General Commands Manual". The default location is
This is how you know you're using a GNU sed: 'man sed'  lists Free Software Foundations in copyright. Also, 'which sed' that returns /usr/local/bin/sed is a good indicator.
==Programmatic Test==
<syntaxhighlight lang='bash'>
echo a | sed -e 's/\(.*\)/\U\1/'
</syntaxhighlight>
will return "Ua" for BDS sed and "A" for GNU sed.
=Install=
{{Internal|Install gnu sed on Mac|Install gnu sed on Mac}}
=Command Line Options=
==<tt>-i</tt>==
<syntaxhighlight lang='bash'>
sed -i <extension> <expression> <filename>
</syntaxhighlight>
Edit the file in-place, saving backups with the specified ''extension''. If a zero-length extension is given, no backup will be saved:
<syntaxhighlight lang='bash'>
sed -i -e <expression> <filename>
</syntaxhighlight>
Example:
<syntaxhighlight lang='bash'>
sed -i -e 's/a/b/' ./somefile.txt
</syntaxhighlight>
==<tt>-e</tt>==
Provides the command expression:
<syntaxhighlight lang='bash'>
sed -e 's/a/b/g'
</syntaxhighlight>
Multiple -e are allowed:
<syntaxhighlight lang='bash'>
sed -e 's/a/b/g' -e 's/x/y/g' -e ...
</syntaxhighlight>
=Commands=
==<tt>s</tt>==
Substitute according to regex.
The return code of the command is not influenced by whether the regex matched or not.
==<tt>p</tt>==
Print the current pattern space. Note that the pattern space is printed automatically, unless <tt>-n</tt> is used.
=<tt>sed</tt> Regular Expressions=
<blockquote style="background-color: #f9f9f9; border: solid thin lightgrey;">
:[[sed Regular Expressions]]
</blockquote>


=Insert a Line/Append in a Specific Position (line number) in a file=
=Insert a Line/Append in a Specific Position (line number) in a file=
Line 28: Line 93:
</pre>
</pre>


=Special Characters (need to be escaped in regular expressions)=
=Insert Multiple Lines at a Specific Position in a File=
 
Use the above recipe, and in the last phase, use:
 
cat ${f} | sed -e ${n}'a\
blah\
blah\
blah with variable substitution, etc' > ${dest}


<pre>
=Insert on a New Line=
    /
    "
    $ # unescaped signifies end of line
</pre>


=Non-Special Characters (do not need to be escaped in regular expressions)=
Assuming that blah is the last sequence on the line:


<pre>
cat ... | sed -e 's/blah/blah\n    on a new line/'
    <
    >
    (
    )
    !
    -
</pre>


=Deleting with <tt>sed</tt>=
=Deleting with <tt>sed</tt>=
Line 66: Line 127:
===Delete the third line of the file===
===Delete the third line of the file===


<pre>
sed -e '3d' a.txt  
sed '3d' a.txt > b.txt
 
</pre>
Delete lines 5 through 10 and 12:
 
sed -e '5,10d;12d' a.txt
 
Delete all lines below line 10:
 
sed -e '10,$d' a.txt


===Delete all lines that match a certain pattern===
===Delete all lines that match a certain pattern===


<pre>
<syntaxhighlight lang='bash'>
sed '/b..h/d' a.txt > b.txt
sed '/b..h/d' a.txt > b.txt
</pre>
</syntaxhighlight>
 
===Delete all lines that match a certain patter and then the next lines===
 
The solution uses pattern-matching deletion command 'd', and the command 'N' that loads the next pattern space.
 
This removes the pattern line and the next line:
 
sed -e '/something/{N;d}' ...
 
This removes the pattern line and the next two lines:
 
sed -e '/something/{N;N;d}' ...
 
Delete only the next line containing the pattern, but not the very line:
 
sed '/something/{N;s/\n.*//;}' file
 
Delete only the line containing the pattern, and the line before the pattern:
 
  sed -n '/something/{s/.*//;x;d;};x;p;${x;p;}' file | sed '/^$/d'
 
===Delete empty lines===
 
sed '/^$/d' file
 
===Delete all lines which are empty or which contains just some blank spaces===
 
sed '/^ *$/d' file


===Delete the last line===
===Delete the last line===


<pre>
$ means "the last line"
sed '$d' a.txt > b.txt
 
</pre>
sed -e '$d' a.txt


==Delete Lines Between Certain "Addresses"==
==Delete Lines Between Certain "Addresses"==
Line 86: Line 181:
<pre>
<pre>
sed '<n1>,<n2>d' <file-name>
sed '<n1>,<n2>d' <file-name>
sed '/<regex1/,/regex2/d' <file-name>
sed '/<regex1/,/regex2/d' <file-name>
sed '<n1>,/regex1/d' <file-name>
sed '/regex1/,<n2>d' <file-name>
</pre>
</pre>


The addresses can be line numbers or regular expressions. Line numbers are 1-based.
The addresses can be line numbers or regular expressions. Line numbers are 1-based.
<blockquote style="background-color: Gold; border: solid thin Goldenrod;">
:Line numbers and regular expressions can be combined, they're both treated as addresses.<br>
</blockquote>


Examples:
Examples:
Line 107: Line 211:
==Delete Negation==
==Delete Negation==


<tt>sed<tt> can be used to delete all lines that ''do not'' match a certain pattern.  
<tt>sed</tt> can be used to delete all lines that ''do not'' match a certain pattern.  


<pre>
<pre>
sed '/<regex>/!d' <file-name>
sed '/<regex>/!d' <file-name>
</pre>
</pre>
=Printing with <tt>sed</tt>=
==Print a Line Specified by Its Number==
Line numbering is 1-based. The following command suppresses the automatic display of the pattern space by specifying <tt>-n</tt>, then it loads the specified line in the pattern space and prints it:
sed -n <line_number>p <file>
The folding example prints line 12:
sed -n 12p a.txt
=Replacing First Occurrence In a File=
On Mac, use 1. I read that on other sed version I have to use 0:
<pre>
    sed -e '1,/pattern/s/pattern/replacement/' ${file} > ./tmp
</pre>
=Convert a Character to Upper or Lower Case=
This functionality is only available in the [[Sed#GNU_sed_vs._BSD_sed|GNU sed]].
Use \L and \U, as follows:
sed -e 's/^\(.\)/\U\1/'
Numbers are not changed.
To convert the entire string:
sed -e 's/^\(.*\)$/\U\1/')

Latest revision as of 18:41, 1 January 2024

External

Internal

GNU sed vs. BSD sed

sed [options] -e [command] file1 file2 ...

GNU sed supports extra functionality relative to the BDS sed - for example \L and \U functions to turn a character to lower and upper cases. macOS comes with the BSD sed, so that is why it's probably a good idea to make upgrading to GNU sed part of the configuring a new Mac.

This is how you know you're using a BSD sed: if no GNU sed was installed, 'man sed' displays in the header "BSD General Commands Manual". The default location is

This is how you know you're using a GNU sed: 'man sed' lists Free Software Foundations in copyright. Also, 'which sed' that returns /usr/local/bin/sed is a good indicator.

Programmatic Test

echo a | sed -e 's/\(.*\)/\U\1/'

will return "Ua" for BDS sed and "A" for GNU sed.

Install

Install gnu sed on Mac

Command Line Options

-i

sed -i <extension> <expression> <filename>

Edit the file in-place, saving backups with the specified extension. If a zero-length extension is given, no backup will be saved:

sed -i -e <expression> <filename>

Example:

sed -i -e 's/a/b/' ./somefile.txt

-e

Provides the command expression:

sed -e 's/a/b/g'

Multiple -e are allowed:

sed -e 's/a/b/g' -e 's/x/y/g' -e ...

Commands

s

Substitute according to regex.

The return code of the command is not influenced by whether the regex matched or not.

p

Print the current pattern space. Note that the pattern space is printed automatically, unless -n is used.

sed Regular Expressions

sed Regular Expressions

Insert a Line/Append in a Specific Position (line number) in a file

Figure out the line number:

# determine the last line that contains ^JAVA_OPTS
local n
n=$(cat ${f} | grep -n "^JAVA_OPTS=" | tail -1) || { echo "failed to determine the line number" 1>&2; exit 1; }
n=${n%%:*}

Insert a line at line 'n':

# insert at line "n":
cat ${f} | sed -e ${n}'a\
This line will be inserted at line number '"${n}"', and this '"${variable}"' will be substituted' > ${dest}

To append at a specific line number, determine the line number as before and effectively "substitute" (s) the line end with your addition:

cat ${f} | sed -e ${n}'s/$/this text will be appended at line number, and this '"${variable}"' will be substituted\n' > ${dest}

Insert Multiple Lines at a Specific Position in a File

Use the above recipe, and in the last phase, use:

cat ${f} | sed -e ${n}'a\
blah\
blah\
blah with variable substitution, etc' > ${dest}

Insert on a New Line

Assuming that blah is the last sequence on the line:

cat ... | sed -e 's/blah/blah\n    on a new line/'

Deleting with sed

Delete a Line that Matches a Certain Pattern

sed will delete a line identified by line number or if the line matches a regular expression pattern:

sed '{<n>|/<regex>/}d' <file-name>

where:

  • n is the 1-based line number.
  • /<regex>/ is the pattern.

Examples:

Delete the third line of the file

sed -e '3d' a.txt 

Delete lines 5 through 10 and 12:

sed -e '5,10d;12d' a.txt

Delete all lines below line 10:

sed -e '10,$d' a.txt

Delete all lines that match a certain pattern

sed '/b..h/d' a.txt > b.txt

Delete all lines that match a certain patter and then the next lines

The solution uses pattern-matching deletion command 'd', and the command 'N' that loads the next pattern space.

This removes the pattern line and the next line:

sed -e '/something/{N;d}' ...

This removes the pattern line and the next two lines:

sed -e '/something/{N;N;d}' ...

Delete only the next line containing the pattern, but not the very line:

sed '/something/{N;s/\n.*//;}' file

Delete only the line containing the pattern, and the line before the pattern:

 sed -n '/something/{s/.*//;x;d;};x;p;${x;p;}' file | sed '/^$/d'

Delete empty lines

sed '/^$/d' file

Delete all lines which are empty or which contains just some blank spaces

sed '/^ *$/d' file

Delete the last line

$ means "the last line"

sed -e '$d' a.txt

Delete Lines Between Certain "Addresses"

sed '<n1>,<n2>d' <file-name>

sed '/<regex1/,/regex2/d' <file-name>

sed '<n1>,/regex1/d' <file-name>

sed '/regex1/,<n2>d' <file-name>

The addresses can be line numbers or regular expressions. Line numbers are 1-based.

Line numbers and regular expressions can be combined, they're both treated as addresses.

Examples:

Delete all lines between line 2 and line 5 (inclusively)

sed '2,5d' a.txt > b.txt

Delete all lines between the occurrences of two regular expressions

sed '/red/,/blue/d' a.txt > b.txt

Delete Negation

sed can be used to delete all lines that do not match a certain pattern.

sed '/<regex>/!d' <file-name>

Printing with sed

Print a Line Specified by Its Number

Line numbering is 1-based. The following command suppresses the automatic display of the pattern space by specifying -n, then it loads the specified line in the pattern space and prints it:

sed -n <line_number>p <file>

The folding example prints line 12:

sed -n 12p a.txt

Replacing First Occurrence In a File

On Mac, use 1. I read that on other sed version I have to use 0:


    sed -e '1,/pattern/s/pattern/replacement/' ${file} > ./tmp

Convert a Character to Upper or Lower Case

This functionality is only available in the GNU sed.

Use \L and \U, as follows:
sed -e 's/^\(.\)/\U\1/'

Numbers are not changed.

To convert the entire string:

sed -e 's/^\(.*\)$/\U\1/')