Sed: Difference between revisions
(26 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
=External= | |||
* https://www.gnu.org/software/sed/manual/sed.html | |||
=Internal= | =Internal= | ||
Line 6: | Line 10: | ||
=GNU sed vs. BSD sed= | =GNU sed vs. BSD sed= | ||
<syntaxhighlight lang='bash'> | |||
sed [options] -e [command] file1 file2 ... | |||
</syntaxhighlight> | |||
GNU sed supports extra functionality relative to the BDS sed - for example [[#Convert_a_Character_to_Upper_or_Lower_Case|\L and \U functions to turn a character to lower and upper cases]]. macOS comes with the BSD sed, so that is why it's probably a good idea to make [[Install_gnu_sed_on_Mac#Overview|upgrading to GNU sed part of the configuring a new Mac]]. | GNU sed supports extra functionality relative to the BDS sed - for example [[#Convert_a_Character_to_Upper_or_Lower_Case|\L and \U functions to turn a character to lower and upper cases]]. macOS comes with the BSD sed, so that is why it's probably a good idea to make [[Install_gnu_sed_on_Mac#Overview|upgrading to GNU sed part of the configuring a new Mac]]. | ||
This is how you know you're using a BSD sed: if no GNU sed was installed, 'man sed' displays in the header "BSD General Commands Manual". | This is how you know you're using a BSD sed: if no GNU sed was installed, 'man sed' displays in the header "BSD General Commands Manual". The default location is | ||
This is how you know you're using a GNU sed: 'man sed' lists Free Software Foundations in copyright. Also, 'which sed' that returns /usr/local/bin/sed is a good indicator. | This is how you know you're using a GNU sed: 'man sed' lists Free Software Foundations in copyright. Also, 'which sed' that returns /usr/local/bin/sed is a good indicator. | ||
==Programmatic Test== | |||
<syntaxhighlight lang='bash'> | |||
echo a | sed -e 's/\(.*\)/\U\1/' | |||
</syntaxhighlight> | |||
will return "Ua" for BDS sed and "A" for GNU sed. | |||
=Install= | =Install= | ||
Line 18: | Line 32: | ||
=Command Line Options= | =Command Line Options= | ||
==<tt>-i</tt>== | |||
==-i== | <syntaxhighlight lang='bash'> | ||
sed -i <extension> <expression> <filename> | |||
</syntaxhighlight> | |||
Edit the file in-place, saving backups with the specified ''extension''. If a zero-length extension is given, no backup will be saved: | Edit the file in-place, saving backups with the specified ''extension''. If a zero-length extension is given, no backup will be saved: | ||
<syntaxhighlight lang='bash'> | |||
sed -i -e <expression> <filename> | |||
</syntaxhighlight> | |||
Example: | |||
<syntaxhighlight lang='bash'> | |||
sed -i -e 's/a/b/' ./somefile.txt | |||
</syntaxhighlight> | |||
==<tt>-e</tt>== | |||
Provides the command expression: | |||
<syntaxhighlight lang='bash'> | |||
sed -e 's/a/b/g' | |||
</syntaxhighlight> | |||
Multiple -e are allowed: | |||
<syntaxhighlight lang='bash'> | |||
sed -e 's/a/b/g' -e 's/x/y/g' -e ... | |||
</syntaxhighlight> | |||
=Commands= | =Commands= | ||
==<tt>s</tt>== | |||
Substitute according to regex. | |||
==p== | The return code of the command is not influenced by whether the regex matched or not. | ||
==<tt>p</tt>== | |||
Print the current pattern space. Note that the pattern space is printed automatically, unless <tt>-n</tt> is used. | Print the current pattern space. Note that the pattern space is printed automatically, unless <tt>-n</tt> is used. | ||
Line 68: | Line 97: | ||
Use the above recipe, and in the last phase, use: | Use the above recipe, and in the last phase, use: | ||
cat ${f} | sed -e ${n}'a\ | |||
cat ${f} | sed -e ${n}'a\ | blah\ | ||
blah\ | blah\ | ||
blah\ | blah with variable substitution, etc' > ${dest} | ||
blah with variable substitution, etc' > ${dest} | |||
=Insert on a New Line= | |||
Assuming that blah is the last sequence on the line: | |||
cat ... | sed -e 's/blah/blah\n on a new line/' | |||
=Deleting with <tt>sed</tt>= | =Deleting with <tt>sed</tt>= | ||
Line 94: | Line 127: | ||
===Delete the third line of the file=== | ===Delete the third line of the file=== | ||
sed -e '3d' a.txt | |||
sed '3d' a.txt | |||
Delete lines 5 through 10 and 12: | |||
sed -e '5,10d;12d' a.txt | |||
Delete all lines below line 10: | |||
sed -e '10,$d' a.txt | |||
===Delete all lines that match a certain pattern=== | ===Delete all lines that match a certain pattern=== | ||
< | <syntaxhighlight lang='bash'> | ||
sed '/b..h/d' a.txt > b.txt | sed '/b..h/d' a.txt > b.txt | ||
</ | </syntaxhighlight> | ||
===Delete all lines that match a certain patter and then the next lines=== | |||
The solution uses pattern-matching deletion command 'd', and the command 'N' that loads the next pattern space. | |||
This removes the pattern line and the next line: | |||
sed -e '/something/{N;d}' ... | |||
This removes the pattern line and the next two lines: | |||
sed -e '/something/{N;N;d}' ... | |||
Delete only the next line containing the pattern, but not the very line: | |||
sed '/something/{N;s/\n.*//;}' file | |||
Delete only the line containing the pattern, and the line before the pattern: | |||
sed -n '/something/{s/.*//;x;d;};x;p;${x;p;}' file | sed '/^$/d' | |||
===Delete empty lines=== | |||
sed '/^$/d' file | |||
===Delete all lines which are empty or which contains just some blank spaces=== | |||
sed '/^ *$/d' file | |||
===Delete the last line=== | ===Delete the last line=== | ||
$ means "the last line" | |||
sed '$d' a.txt | |||
sed -e '$d' a.txt | |||
==Delete Lines Between Certain "Addresses"== | ==Delete Lines Between Certain "Addresses"== | ||
Line 181: | Line 248: | ||
Numbers are not changed. | Numbers are not changed. | ||
To convert the entire string: | |||
sed -e 's/^\(.*\)$/\U\1/') |
Latest revision as of 18:41, 1 January 2024
External
Internal
GNU sed vs. BSD sed
sed [options] -e [command] file1 file2 ...
GNU sed supports extra functionality relative to the BDS sed - for example \L and \U functions to turn a character to lower and upper cases. macOS comes with the BSD sed, so that is why it's probably a good idea to make upgrading to GNU sed part of the configuring a new Mac.
This is how you know you're using a BSD sed: if no GNU sed was installed, 'man sed' displays in the header "BSD General Commands Manual". The default location is
This is how you know you're using a GNU sed: 'man sed' lists Free Software Foundations in copyright. Also, 'which sed' that returns /usr/local/bin/sed is a good indicator.
Programmatic Test
echo a | sed -e 's/\(.*\)/\U\1/'
will return "Ua" for BDS sed and "A" for GNU sed.
Install
Command Line Options
-i
sed -i <extension> <expression> <filename>
Edit the file in-place, saving backups with the specified extension. If a zero-length extension is given, no backup will be saved:
sed -i -e <expression> <filename>
Example:
sed -i -e 's/a/b/' ./somefile.txt
-e
Provides the command expression:
sed -e 's/a/b/g'
Multiple -e are allowed:
sed -e 's/a/b/g' -e 's/x/y/g' -e ...
Commands
s
Substitute according to regex.
The return code of the command is not influenced by whether the regex matched or not.
p
Print the current pattern space. Note that the pattern space is printed automatically, unless -n is used.
sed Regular Expressions
Insert a Line/Append in a Specific Position (line number) in a file
Figure out the line number:
# determine the last line that contains ^JAVA_OPTS local n n=$(cat ${f} | grep -n "^JAVA_OPTS=" | tail -1) || { echo "failed to determine the line number" 1>&2; exit 1; } n=${n%%:*}
Insert a line at line 'n':
# insert at line "n": cat ${f} | sed -e ${n}'a\ This line will be inserted at line number '"${n}"', and this '"${variable}"' will be substituted' > ${dest}
To append at a specific line number, determine the line number as before and effectively "substitute" (s) the line end with your addition:
cat ${f} | sed -e ${n}'s/$/this text will be appended at line number, and this '"${variable}"' will be substituted\n' > ${dest}
Insert Multiple Lines at a Specific Position in a File
Use the above recipe, and in the last phase, use:
cat ${f} | sed -e ${n}'a\ blah\ blah\ blah with variable substitution, etc' > ${dest}
Insert on a New Line
Assuming that blah is the last sequence on the line:
cat ... | sed -e 's/blah/blah\n on a new line/'
Deleting with sed
Delete a Line that Matches a Certain Pattern
sed will delete a line identified by line number or if the line matches a regular expression pattern:
sed '{<n>|/<regex>/}d' <file-name>
where:
- n is the 1-based line number.
- /<regex>/ is the pattern.
Examples:
Delete the third line of the file
sed -e '3d' a.txt
Delete lines 5 through 10 and 12:
sed -e '5,10d;12d' a.txt
Delete all lines below line 10:
sed -e '10,$d' a.txt
Delete all lines that match a certain pattern
sed '/b..h/d' a.txt > b.txt
Delete all lines that match a certain patter and then the next lines
The solution uses pattern-matching deletion command 'd', and the command 'N' that loads the next pattern space.
This removes the pattern line and the next line:
sed -e '/something/{N;d}' ...
This removes the pattern line and the next two lines:
sed -e '/something/{N;N;d}' ...
Delete only the next line containing the pattern, but not the very line:
sed '/something/{N;s/\n.*//;}' file
Delete only the line containing the pattern, and the line before the pattern:
sed -n '/something/{s/.*//;x;d;};x;p;${x;p;}' file | sed '/^$/d'
Delete empty lines
sed '/^$/d' file
Delete all lines which are empty or which contains just some blank spaces
sed '/^ *$/d' file
Delete the last line
$ means "the last line"
sed -e '$d' a.txt
Delete Lines Between Certain "Addresses"
sed '<n1>,<n2>d' <file-name> sed '/<regex1/,/regex2/d' <file-name> sed '<n1>,/regex1/d' <file-name> sed '/regex1/,<n2>d' <file-name>
The addresses can be line numbers or regular expressions. Line numbers are 1-based.
- Line numbers and regular expressions can be combined, they're both treated as addresses.
Examples:
Delete all lines between line 2 and line 5 (inclusively)
sed '2,5d' a.txt > b.txt
Delete all lines between the occurrences of two regular expressions
sed '/red/,/blue/d' a.txt > b.txt
Delete Negation
sed can be used to delete all lines that do not match a certain pattern.
sed '/<regex>/!d' <file-name>
Printing with sed
Print a Line Specified by Its Number
Line numbering is 1-based. The following command suppresses the automatic display of the pattern space by specifying -n, then it loads the specified line in the pattern space and prints it:
sed -n <line_number>p <file>
The folding example prints line 12:
sed -n 12p a.txt
Replacing First Occurrence In a File
On Mac, use 1. I read that on other sed version I have to use 0:
sed -e '1,/pattern/s/pattern/replacement/' ${file} > ./tmp
Convert a Character to Upper or Lower Case
This functionality is only available in the GNU sed.
Use \L and \U, as follows:
sed -e 's/^\(.\)/\U\1/'
Numbers are not changed.
To convert the entire string:
sed -e 's/^\(.*\)$/\U\1/')