#] #] ********************* #] "$d_SysMaint"'Linux/sed notes.txt' www.BillHowell.ca ?date? initial - maybe 2017or something? 24************************24 # Table of contents, generated with : # $ grep "^#] " "$d_SysMaint"'Linux/sed notes.txt' | sed 's/^#\]/ /' # ********************* "$d_SysMaint""Linux/sed notes.txt" 15Sep2023 pStrPAll_L - check to make sure that all intended strP are tabbed : 19Sep2023 sed -n '8000{p;q}' file - get nth line of a file 03Sep2023 search "linux sed and case insensitive option" 04Aug2023 [search. replace] general text - encode in alpha chrs, then process!! 03Aug2023 search "sed replace backslash with forward slash" 02Aug2023 problem of how to handle `& : & is special in the REPLACEMENT text, not in search 02Aug2023 strOld049 always gets tagged with chrSeq \' 02Aug2023 search "Linux sed replace - which characters must be escaped?" 01Aug2023 Paul Brown: Ampersands and File Descriptors in Bash 01Aug2023 15 Special Characters You Need to Know for Bash 01Aug2023 Escape a string for a sed replace pattern, includes & 29Jul2023 search "Linux sed replace - which characters must be escaped?" 29Jul2023 $'\x22'=", $'\x27'=', $'\x5C'=\, $'\7C'=|, $'\x2F'=/ 29Jul2023 search 'Linux bash and how do I put a tab into a string?' $'\t' as tabulator 15Mar2023 search 'Linux and how do I extract a sequence of lines in a file?' 10Mar2023 search 'Linux sed and how do I output a quote?' - [apo, quote] 12Mar2023 for "$d_Qmkts"'market symbol [regressSL, detrend].ndf' 08Mar2023 search 'Linux sed and how do I reference more than 9 items?' 01Mar2023 the address n~m addresses every m:th line starting at line n 20Feb2023 sed case insensitive sed -i 20Feb2023 reverse str - to get middle part 15Feb2023 sed-like simple string replacement (no regexpr?) 21Oct2021 add [result start, EOL paren] to a fault, using geany test editor : 14Oct2021 search "Linux sed and how do I extract characters by position?" regexp option for words (\b boundaries) sed problems with ['.] search "maximum length of a sed expression" ??? | tr \\n \\0 | sed 's/\\0 //' | tr \x0 \\n |... sed search \n -> use tr : might not work - see "$d_SysMaint""Linux/sed notes.txt" hdorio and others multi-line sed sed-cmakes - pull out my modifications to cmake files bash function for converting paths for use with SED - function sedPath Often-Used Commands - http://www.computerhope.com/unix/used.htm 08********08 #] ??Sep2023 08********08 #] ??Sep2023 08********08 #] ??Sep2023 08********08 #] ??Sep2023 08********08 #] ??Sep2023 08********08 #] ??Sep2023 08********08 #] 15Sep2023 pStrPAll_L - check to make sure that all intended strP are tabbed : 15Sep2023 simple manual example : $ sed s'|\x9|xyZyx|' "$d_webWork"'pStrPAll_L change.txt' | grep --invert-match 'xyZyx' <:5 paths with fileName 5:> /home/bill/web/ProjMini/Puetz & Borchardt/Howell - comments on Puetz UWS, the greatest of cycles, human implications.odt /home/bill/web/ProjMini/Puetz & Borchardt/Howell - possible phenomenological origins of Puetz Universal Wave Series (UWS).html <:5 bookmarks - hopefully these wont backfile 5:> <:5 here is where I start changes to dirs 5:> >> very easy to see the two legitimate lines that were NOT tabbed don't know yet if povrL_pStrP_replace will blow up with non-tabbed (incl empty) lines... 08********08 #] 19Sep2023 man sed -n, --quiet, --silent suppress automatic printing of pattern space q [exit-code] Immediately quit the sed script without processing any more input, except that if auto-print is not disabled the current pattern space will be printed. The exit code argument is a GNU extension. >> Howell l1ne333=$( sed -n '1{p;q}' "$pinn333" ) >> error : /home/bill/web/bin/fileops.sh: line 1374: warning: command substitution: ignored null byte in input /home/bill/web/bin/fileops.sh: line 1374: warning: command substitution: ignored null byte in input try : l1ne333=$( sed -n '1{p;q 0}' "$pinn333" ) 21:54$ bash "$d_bin"'webSite update local.sh' find all html files extract webPages from htmlL: they have strTst769 strTst333 = What special characters do I need to escape in the find field? Do I need to escape the same characters in the replace_with field? Is backslash the appropriate escape key? I will use the following method: find . -type f -name "*.php" -exec sed -i 's/"find"/"replace_with"/g' {} + That method has worked for simple replacements. I read "How can I replace a string in a file(s)?" (https://unix.stackexchange.com/questions/112023/how-can-i-replace-a-string-in-a-files), and other pages, but could not find the answer I need. edited Apr 13, 2017 at 12:36 CommunityBot asked Feb 11, 2016 at 22:14 Jonathon Neville +--+ sed "s^script>');} < /script^&--^g" – Costas Feb 11, 2016 at 22:24 +--+ you can use sed -i "s+< /script>+< /script-->+g" So final command would become find . -type f -name "*.php" -exec sed -i "s+< /script>+< /script-->+g" {} + answered Apr 14, 2016 at 10:43 Rahul 08********08 #] 01Aug2023 Paul Brown: Ampersands and File Descriptors in Bash https://www.linux.com/training-tutorials/ampersands-and-file-descriptors-bash/ Ampersands and File Descriptors in Bash Paul Brown - February 12, 2019 Another thing with the standard file descriptors is that, when you pipe from one to another, the order in which you do this is a bit counterintuitive. Take the command above, for example. It looks like it has been written the wrong way around. You may be reading it like this: “pipe the output to a file and then pipe errors to the standard output.” It would seem the error output comes to late and is sent when 1 is already done. But that is not how file descriptors work. A file descriptor is not a placeholder for the file, but for the input and/or output channel to the file. In this case, when you do 1> services.txt, you are saying “open a write channel to services.txt and leave it open“. 1 is the name of the channel you are going to use, and it remains open until the end of the line. If you still think it is the wrong way around, try this: find /etc -iname "*.service" 2>&1 1>services.txt And notice how it doesn’t work; notice how errors get piped to the terminal and only the non-erroneous output (that is stdout) gets pushed to services.txt. That is because Bash processes every result from find from left to right. Think about it like this: when Bash gets to 2>&1, stdout (1) is still a channel that points to the terminal. If the result that find feeds Bash contains an error, it is popped into 2, transferred to 1, and, away it goes, off to the terminal! Then at the end of the command, Bash sees you want to open stdout as a channel to the services.txt file. If no error has occurred, the result goes through 1 into the file. By contrast, in find /etc -iname "*.service" 1>services.txt 2>&1 1 is pointing at services.txt right from the beginning, so anything that pops into 2 gets piped through 1, which is already pointing to the final resting place in services.txt, and that is why it works. In any case, as mentioned above &> is shorthand for “both standard output and standard error“, that is, 2>&1. 08********08 #] 01Aug2023 15 Special Characters You Need to Know for Bash https://www.howtogeek.com/439199/15-special-characters-you-need-to-know-for-bash/#amp-background-process 15 Special Characters You Need to Know for Bash That sequence of strange symbols on the Bash command line must mean something, right? We're breaking down special characters and how to use them. Dave McKay Published Sep 18, 2019 simple introductory - handy for what I keep forgetting 08********08 #] 01Aug2023 Escape a string for a sed replace pattern, includes & +-----+ https://stackoverflow.com/questions/407523/escape-a-string-for-a-sed-replace-pattern (again - see next log below) Escape a string for a sed replace pattern Asked 14 years, 7 months ago Modified 9 months ago Viewed 388k times +--+ Warning: This does not consider newlines. For a more in-depth answer, see this SO-question instead. (Thanks, Ed Morton & Niklas Peter) Note that escaping everything is a bad idea. Sed needs many characters to be escaped to get their special meaning. For example, if you escape a digit in the replacement string, it will turn in to a backreference. As Ben Blank said, there are only three characters that need to be escaped in the replacement string (escapes themselves, forward slash for end of statement and & for replace all): ESCAPED_REPLACE=$(printf '%s\n' "$REPLACE" | sed -e 's/[\/&]/\\&/g') # Now you can use ESCAPED_REPLACE in the original sed statement sed "s/KEYWORD/$ESCAPED_REPLACE/g" If you ever need to escape the KEYWORD string, the following is the one you need: sed -e 's/[]\/$*.^[]/\\&/g' And can be used by: KEYWORD="The Keyword You Need"; ESCAPED_KEYWORD=$(printf '%s\n' "$KEYWORD" | sed -e 's/[]\/$*.^[]/\\&/g'); # Now you can use it inside the original sed statement to replace text sed "s/$ESCAPED_KEYWORD/$ESCAPED_REPLACE/g" Remember, if you use a character other than / as delimiter, you need replace the slash in the expressions above wih the character you are using. See PeterJCLaw's comment for explanation. Edited: Due to some corner cases previously not accounted for, the commands above have changed several times. Check the edit history for details. edited Oct 2, 2021 at 5:25 Cadoiz 1,4502121 silver badges3131 bronze badges answered Apr 24, 2010 at 18:35 Pianosaurus +--+ Based on Pianosaurus's regular expressions, I made a bash function that escapes both keyword and replacement. function sedeasy { sed -i "s/$(echo $1 | sed -e 's/$[[\/.*]\|\]$/\\&/g')/$(echo $2 | sed -e 's/[\/&]/\\&/g')/g" $3 } Here's how you use it: sedeasy "include /etc/nginx/conf.d/*" "include /apps/*/conf/nginx.conf" /etc/nginx/nginx.conf answered May 6, 2012 at 1:46 Gurpartap Singh 08********08 #] 29Jul2023 search "Linux sed replace - which characters must be escaped?" +-----+ https://stackoverflow.com/questions/407523/escape-a-string-for-a-sed-replace-pattern Escape a string for a sed replace pattern Asked 14 years, 7 months ago Modified 9 months ago Viewed 388k times +-----+ The only three literal characters which are treated specially in the replace clause are / (to close the clause), \ (to escape characters, backreference, &c.), and & (to include the match in the replacement). Therefore, all you need to do is escape those three characters: sed "s/KEYWORD/$(echo $REPLACE | sed -e 's/\\/\\\\/g; s/\//\\\//g; s/&/\\\&/g')/g" Example: $ export REPLACE="'\"|\\/><&!" $ echo fooKEYWORDbar | sed "s/KEYWORD/$(echo $REPLACE | sed -e 's/\\/\\\\/g; s/\//\\\//g; s/&/\\\&/g')/g" foo'"|\/><&!bar edited Jun 1, 2016 at 2:14 Mark Stosberg answered Jan 2, 2009 at 18:31 Ben Blank +--+ Warning: This does not consider newlines. For a more in-depth answer, see this SO-question instead. (Thanks, Ed Morton & Niklas Peter) Note that escaping everything is a bad idea. Sed needs many characters to be escaped to get their special meaning. For example, if you escape a digit in the replacement string, it will turn in to a backreference. As Ben Blank said, there are only three characters that need to be escaped in the replacement string (escapes themselves, forward slash for end of statement and & for replace all): ESCAPED_REPLACE=$(printf '%s\n' "$REPLACE" | sed -e 's/[\/&]/\\&/g') # Now you can use ESCAPED_REPLACE in the original sed statement sed "s/KEYWORD/$ESCAPED_REPLACE/g" If you ever need to escape the KEYWORD string, the following is the one you need: sed -e 's/[]\/$*.^[]/\\&/g' And can be used by: KEYWORD="The Keyword You Need"; ESCAPED_KEYWORD=$(printf '%s\n' "$KEYWORD" | sed -e 's/[]\/$*.^[]/\\&/g'); # Now you can use it inside the original sed statement to replace text sed "s/$ESCAPED_KEYWORD/$ESCAPED_REPLACE/g" Remember, if you use a character other than / as delimiter, you need replace the slash in the expressions above wih the character you are using. See PeterJCLaw's comment for explanation. Edited: Due to some corner cases previously not accounted for, the commands above have changed several times. Check the edit history for details. edited Oct 2, 2021 at 5:25 Cadoiz answered Apr 24, 2010 at 18:35 Pianosaurus +-----+ It turns out you're asking the wrong question. I also asked the wrong question. The reason it's wrong is the beginning of the first sentence: "In my bash script...". I had the same question & made the same mistake. If you're using bash, you don't need to use sed to do string replacements (and it's much cleaner to use the replace feature built into bash). Instead of something like, for example: function escape-all-funny-characters() { UNKNOWN_CODE_THAT_ANSWERS_THE_QUESTION_YOU_ASKED; } INPUT='some long string with KEYWORD that need replacing KEYWORD.' A="$(escape-all-funny-characters 'KEYWORD')" B="$(escape-all-funny-characters '')" OUTPUT="$(sed "s/$A/$B/g" <<<"$INPUT")" you can use bash features exclusively: INPUT='some long string with KEYWORD that need replacing KEYWORD.' A='KEYWORD' B='' OUTPUT="${INPUT//"$A"/"$B"}" edited Aug 18, 2016 at 18:07 answered Dec 29, 2015 at 22:51 Dennis Estenson 08********08 #] 29Jul2023 $'\x22'=", $'\x27'=', $'\x5C'=\, $'\7C'=|, $'\x2F'=/ see fileops.sh sedEscExpr='s#\^#\\^#g;s#\$#\\$#g;s#\&#\\&#g;s#\"#\\"#g;s#\x27#\\x27#g;s#\,#\\,#g;s#\*#\\*#g;s#\.#\\.#g;s#\[#\\[#g;s#\]#\\]#g;s#\-#\\-#g;s#\/#\\/#g' # still problems : {}\| # ;s#\\#\\\#g;s#\|#\\|#g # $ echo "$sedEscExpr" # $ echo '^$&"\x27*.[]{}-\|/' | sed "$sedEscExpr" # \^\$\&\"\x27\*\.\[\]{}\-\|\/ # >> almost +-----+ https://unix.stackexchange.com/questions/211834/slash-and-backslash-in-sed Slash and backslash in sed [duplicate] Asked 8 years, 1 month ago Modified 6 years, 6 months ago Viewed 246k times +--+ When in doubt, echo the command: echo sed "s/\//\\\//g" -> sed s/\//\\//g. Btw you can use something else for sed, like 's@/@\\/@g'. – Mingye Wang Jun 24, 2015 at 13:28 +--+ Use single quotes for the expression you used: sed 's=/=\\/=g' In double quotes, \ has a special meaning, so you have to backslash it: sed sed "s=/=\\\/=g" Share Improve this answer Follow answered Jun 24, 2015 at 13:24 choroba's user avatar choroba 08********08 #] 29Jul2023 search 'Linux bash and how do I put a tab into a string?' $'\t' as tabulator https://unix.stackexchange.com/questions/489150/how-to-print-strings-separated-by-tab-in-bash How to print strings separated by TAB in bash? Asked 4 years, 7 months ago Modified 4 years, 7 months ago Viewed 29k times Despite that fact that you already know that you have an issue with your terminal: Bash by itself interprets $'\t' as tabulator. So you can always concatenate strings like this - for example as assignment: v='This is'$'\t''a test' And the print it literally, e.g. printf '%s' "$v" – rexkogitans Dec 17, 2018 at 7:11 08********08 #] 15Mar2023 search 'Linux and how do I extract a sequence of lines in a file?' Howell : sed -n '$nHead,$nTail"p;"$nQuit"q"' filename > newfile +-----+ https://stackoverflow.com/questions/83329/how-can-i-extract-a-predetermined-range-of-lines-from-a-text-file-on-unix How can I extract a predetermined range of lines from a text file on Unix? Asked 14 years, 6 months ago Modified 5 months ago Viewed 618k times sed -n '16224,16482p;16483q' filename > newfile From the sed manual: p - Print out the pattern space (to the standard output). This command is usually only used in conjunction with the -n command-line option. n - If auto-print is not disabled, print the pattern space, then, regardless, replace the pattern space with the next line of input. If there is no more input then sed exits without processing any more commands. q - Exit sed without processing any more commands or input. Note that the current pattern space is printed if auto-print is not disabled with the -n option. and Addresses in a sed script can be in any of the following forms: number Specifying a line number will match only that line in the input. An address range can be specified by specifying two addresses separated by a comma (,). An address range matches lines starting from where the first address matches, and continues until the second address matches (inclusively). edited Jun 13, 2018 at 18:09 avandeursen answered Sep 17, 2008 at 13:42 boxxar 207 If, like me, you need to do this on a VERY large file, it helps if you add a quit command on the next line. Then it's sed -n '16224,16482p;16483q' filename. Otherwise sed will keep scanning till the end (or at least my version does). – wds, Feb 1, 2013 at 13:40 08********08 #] 10Mar2023 search 'Linux sed and how do I output a quote?' - [apo, quote] Yeesh - no notes (later) #] 12Mar2023 for "$d_Qmkts"'market symbol [regressSL, detrend].ndf' # +--+ # BEFORE running revampALL_CsvToTab, in "rawCsv files" : # clean up titleLine - remove commas within quotes # geany regexpr search : \,\"(.*)\,(.*)\"\, replace "\1\2" # geany can't specify `g in 's|searchExpr|replaceExpr|g' ? # # sed 's|\"$.*$\,$.*$\:$.*$\"|"\1\2:\3"|g' "$d_mktData"'rawCsv/rawCsv SPX 1872-now TradingView.txt' >"$d_temp"'rawCsv SPX 1872-now TradingView.txt' # sed 's|\"$.*$\,$.*$\:|"\1\2:|g' "$d_mktData"'rawCsv/rawCsv SPX 1872-now TradingView.txt' >"$d_temp"'rawCsv SPX 1872-now TradingView.txt' # sed 's|\,$.*$\:|\1:|g' "$d_mktData"'rawCsv/rawCsv SPX 1872-now TradingView.txt' >"$d_temp"'rawCsv SPX 1872-now TradingView.txt' # sed 's|[,]$.*$\:|\1:|g' "$d_mktData"'rawCsv/rawCsv SPX 1872-now TradingView.txt' >"$d_temp"'rawCsv SPX 1872-now TradingView.txt' # sed 's|[,]$.*$\:|\1[:]|g' "$d_mktData"'rawCsv/rawCsv SPX 1872-now TradingView.txt' >"$d_temp"'rawCsv SPX 1872-now TradingView.txt' # sed 's|\,||g' "$d_mktData"'rawCsv/rawCsv SPX 1872-now TradingView.txt' >"$d_temp"'rawCsv SPX 1872-now TradingView.txt' # sed 's|\"$.*$\,$.*$\:|\"\1,\2:|g' "$d_mktData"'rawCsv/rawCsv SPX 1872-now TradingView.txt' >"$d_temp"'rawCsv SPX 1872-now TradingView.txt' # blub='time,open,high,low,close,"FEDFUNDS, FRED: Open","FEDFUNDS, FRED: High","FEDFUNDS, FRED: Low","FEDFUNDS, FRED: Close","FDHBFRBN, FRED: Open","FDHBFRBN, FRED: High","FDHBFRBN, FRED: Low","FDHBFRBN, FRED: Close","WRBWFRBL, FRED: Open","WRBWFRBL, FRED: High","WRBWFRBL, FRED: Low","WRBWFRBL, FRED: Close","WRESBAL, FRED: Open","WRESBAL, FRED: High","WRESBAL, FRED: Low","WRESBAL, FRED: Close","USHMI, ECONOMICS: Open","USHMI, ECONOMICS: High","USHMI, ECONOMICS: Low","USHMI, ECONOMICS: Close","WLFN, FRED: Open","WLFN, FRED: High","WLFN, FRED: Low","WLFN, FRED: Close","CUS2!, CME: Open","CUS2!, CME: High","CUS2!, CME: Low","CUS2!, CME: Close","GOLD, TVC: Open","GOLD, TVC: High","GOLD, TVC: Low","GOLD, TVC: Close","USOIL, TVC: Open","USOIL, TVC: High","USOIL, TVC: Low","USOIL, TVC: Close","TNX, TVC: Open","TNX, TVC: High","TNX, TVC: Low","TNX, TVC: Close"' # echo "$blub" | sed 's|\"$.*$\,$.*$\:|\"\1,\2:|g' # echo "$blub" | sed 's|\"||g' # echo "$blub" | sed 's|\"$[A-Za-z0-9]\+$\,|"\1|g' >> FINALLY, this took me forever... # sed "s|NaN||g" "$d_mktData"'rawCsv/rawCsv SPX 1872-now TradingView.txt' >"$d_temp"'rawCsv SPX 1872-now TradingView.txt' # # the key in this example was to avoid the "greedy regexpr # by being much more specific on words in searchExpr : $ fNam='rawCsv SPX 1872-now TradingView.txt' # $ sed 's|\"$[A-Za-z0-9]\+$\,|"\1|g' "$d_mktData"'rawCsv/'"$fNam' | sed "s|NaN||g" >"$d_temp$fNam" # $ mv "$d_temp$fNam" "$d_mktData"'rawCsv/'"$fNam' # 08********08 #] 08Mar2023 search 'Linux sed and how do I reference more than 9 items?' +-----+ https://stackoverflow.com/questions/36972352/more-than-9-sed-groups-multiple-lines-still-using-sed more than 9 sed groups- multiple lines (still using SED) [duplicate] Asked 6 years, 10 months ago Modified 6 years, 8 months ago Viewed 1k times sed '/very-long-pattern/ { H s/.......(ten)(eleven).../\1 \2 .../ x s/(one)(two)...(nine).*/\1 \2 .../ G s/\n// }' Basically, you capture the last 9 (<=) groups, and store them in the hold buffer, then capture the first 9 groups, and put them together, at last, remove the newline character introduced by G. You can extend this script to support as many sed groups as you want. Jonathan Leffler answered Jun 14, 2016 at 3:13 pdg >> yuch! but this gives me an idea to sequentially do a sed.. see 08Mar2023 in "$d_bin"'port bash to python.txt' 08********08 #] 01Mar2023 the address n~m addresses every m:th line starting at line n +-----+ https://unix.stackexchange.com/questions/542611/bash-script-how-to-assign-each-line-of-command-output-to-values-in-an-array BASH script: How to assign each line of command output to values in an array? Asked 3 years, 5 months ago Modified 3 years, 5 months ago Viewed 6k times Using readarray in the bash shell, and GNU sed: readarray -t my_array < <( my_command | sed '1~2d' ) The built-in readarray reads the lines into an array. The lines are read from a process substitution. The sed command in the process substitution will only output every second line read from my_command (and could also be written sed '1!n;d', or as sed -n 'n;p' with standard sed). In GNU sed, the address n~m addresses every m:th line starting at line n. This is a GNU extension to standard sed, for convenience. The my_command command will only ever be called once. Testing: $ readarray -t my_array < <( seq 10 | sed '1~2d' ) $ printf '%s\n' "${my_array[@]}" 2 4 6 8 10 edited Sep 20, 2019 at 12:29 answered Sep 19, 2019 at 12:48 Kusalananda 08********08 #] 20Feb2023 sed case insensitive sed -i https://www.cyberciti.biz/faq/unixlinux-sed-case-insensitive-search-replace-matching/ sed Case Insensitive Search Matching and Replacement Author: Vivek Gite Last updated: April 8, 2021 8 comments sed -i 's/SEARCH-WORD/REPLACMENT-WORD/gi' input >> I don't use in-place edit, so sed 's/SEARCH-WORD/REPLACMENT-WORD/gi' input >> cat "$d_web"'webWork files/urls pLst base non-[webWork, conference guides].txt' | tr \\n \\0 | xargs -0 -IFILE grep -i ".*|\1|gi' | rev | sed 's|>$.*$\"|\1|gi' | rev | sort -u >"$d_web"'webWork files/webDirL from links.txt' 08********08 #] 20Feb2023 reverse str - to get middle part search 'Linux sed in reverse direction' (from end-to-start) +-----+ https://stackoverflow.com/questions/19035938/backward-search-and-replace-using-sed Backward search and replace using sed Asked 9 years, 4 months ago Modified 5 years, 11 months ago +--+ What if we reverse the string, change the FIRST occurrence in reversed string by sed, then reverse the result back to normal order: #!/bin/bash str="aaa bbb ccc bbb aaa" echo "${str}" | rev | sed 's/bbb/zzz/' | rev It works fine, if we need to replace just one symbol. For words, you need to reverse both, pattern and replacement words. answered Mar 1, 2017 at 19:23 paratrooper >> 20Feb2023 Great! worked for me, example cat "$d_web"'webWork files/urls pLst base non-[webWork, conference guides].txt' | tr \\n \\0 | xargs -0 -IFILE grep --with-filename -i '\[#=; backtrack ;=#\]' "FILE" | sed 's|$.*$:.*|\1|' | rev | sed 's|.*:||;s|$.*$\>".*|\1|' | rev | sort -u >"$d_web"'webWork files/urls pLst backtrack.txt' 08********08 #] 15Feb2023 sed-like simple string replacement (no regexpr?) +-----+ https://www.educba.com/bash-replace-string/ Bash Replace String By Alokananda Ghoshal #!/bin/bash string="abcPQRxYZabcxYZABCabc" string_to_replace="abc" replacing_string="AbcRstNopAbcVwx" echo "**Example of replacing the first matched string: $string_to_replace With: $replacing_string**" echo ${string/abc/AbcRstNopAbcVwx} echo "**Example of replacing all occurences of matched string: $string_to_replace With: $replacing_string**" echo ${string//abc/AbcRstNopAbcVwx} echo "**Example of replacing the first front end matched string: $string_to_replace With: $replacing_string**" echo ${string/#abc/AbcRstNopAbcVwx} echo "**Example of replacing the first back end matched string: $string_to_replace With: $replacing_string**" echo ${string/%abc/AbcRstNopAbcVwx} >> cool!! U have no clue about the syntax 08********08 #] 21Oct2021 add [result start, EOL paren] to a fault, using geany test editor : sed regexpr search "result := write '" replace "result := \n\t\t\t(fault '?" sed normal search "' ; " replace "') ; " 08********08 #] 14Oct2021 search "Linux sed and how do I extract characters by position?" https://www.unix.com/shell-programming-and-scripting/218415-extract-substring-specif-position-length-file-line.html Yoda Forum Staff, Moderator while read line do echo ${line:10:4} done < filename >> this didn't work for me https://stackoverflow.com/questions/24469977/replace-particular-string-at-fixed-position-using-sed asked Jun 28 '14 at 19:01, user333422 Now, I want to replace 12th and 13th character if only these two characters are 49 with characters 58. answered Jun 28 '14 at 19:04, Lev Levitsky sed -E 's/^(.{11})49/\158/' data.txt Howell below : worked echo "$date_ymdhm_new" | sed 's/^$.\{2\}$.*/\1/' didn't but why? echo "$date_ymdhm_new" | sed 's/^$[.]\{2\}$.*/\1/' 08********08 #] regexp option for words (\b boundaries) 11Sep2021 https://stackoverflow.com/questions/1032023/sed-whole-word-search-and-replace \b in regular expressions match word boundaries (i.e. the location between the first word character and non-word character): $ echo "bar embarassment" | sed "s/\bbar\b/no bar/g" no bar embarassment edited Jun 23 '09 at 11:54 answered Jun 23 '09 at 11:41 Joakim Lundborg 08********08 #] sed problems with ['.] 29Jun2021 https://stackoverflow.com/questions/4332392/how-to-remove-all-occurences-of-dot-in-a-string-in-a-shell-script A general way would be to pipe it to sed: sed -e 's/\.//g' On command prompt: $ echo $hostname // you type this test.test.test // this is the result $ echo $hostname | sed -e 's/\.//g' testtesttest edited Dec 2 '10 at 9:20 answered Dec 2 '10 at 6:46 darioo # $ sed "s/\'/\~/" "/media/bill/Dell2/Website - raw/Qnial/MY_NDFS/dictionaries/Dictionary.com clean.txt" | grep --invert-match "\~" >"$d_temp""grep period apo.txt" # $ sed "s/\'/\~/" "/media/bill/Dell2/Website - raw/Qnial/MY_NDFS/dictionaries/Dictionary.com clean.txt" >"$d_temp""grep period apo.txt" # $ sed s/\'/~/ "/media/bill/Dell2/Website - raw/Qnial/MY_NDFS/dictionaries/Dictionary.com clean.txt" >"$d_temp""grep period apo.txt" # $ sed s/\'/~/ "/media/bill/Dell2/Website - raw/Qnial/MY_NDFS/dictionaries/Dictionary.com clean.txt" | grep --invert-match "\~" >"$d_temp""grep period apo.txt" # $ sed s/\'/zzz/;s/\./zzz/ "/media/bill/Dell2/Website - raw/Qnial/MY_NDFS/dictionaries/Dictionary.com clean.txt" | grep --invert-match "zzz" >"$d_temp""grep period apo.txt" # $ sed s/\./zzz/ "/media/bill/Dell2/Website - raw/Qnial/MY_NDFS/dictionaries/Dictionary.com clean.txt" | grep --invert-match "zzz" >"$d_temp""grep period apo.txt" # $ sed -e 's/\./zzz/' "/media/bill/Dell2/Website - raw/Qnial/MY_NDFS/dictionaries/Dictionary.com clean.txt" | grep --invert-match "zzz" >"$d_temp""grep period apo.txt" # $ sed s/\'/zzz/ "/media/bill/Dell2/Website - raw/Qnial/MY_NDFS/dictionaries/Dictionary.com clean.txt" | sed -e 's/\./zzz/' | grep --invert-match "zzz" >"$d_temp""grep period apo.txt" solution in bash $ sed s/\'/zzz/ "pinn" | sed -e 's/\./zzz/' | grep --invert-match "zzz" >"pout" - [.'] split expr solution in QNial qnial> host link 'sed s/\' chr_apo '/zzz/ "' p_temp2 '" | sed -e "s/\./zzz/" | grep --invert-match "zzz" >"' p_temp1 '"' ; 08********08 #] search "maximum length of a sed expression" 18Jun2021 https://www.unix.com/shell-programming-and-scripting/263413-sed-255-character-limitation.html sed 255 Character Limitation 08********08 17Jun2021 Howell's guess #] ??? | tr \\n \\0 | sed 's/\\0 //' | tr \x0 \\n |... sed search \n -> use tr : #] might not work - see "$d_SysMaint""Linux/sed notes.txt" hdorio and others +-----+ https://stackoverflow.com/questions/1251999/how-can-i-replace-a-newline-n-using-sedHow can I replace a newline (\n) using sed? Asked 11 years, 10 months ago Active 14 days ago Viewed 1.9m times 1503 511 How can I replace a newline ("\n") with a space ("") using the sed command? I unsuccessfully tried: sed 's#\n# #g' file sed 's#^$# #g' file How do I fix it? sed Share Improve this question Follow edited Aug 12 '19 at 15:44 agc 7,02222 gold badges2525 silver badges4444 bronze badges asked Aug 9 '09 at 19:10 hhh 44.6k5656 gold badges155155 silver badges253253 bronze badges +--+ 38 tr is only the right tool for the job if replace a single character for a single character, while the example above shows replace newline with a space.. So in the above example, tr could work.. But would be limiting later on. – Angry 84 Dec 31 '15 at 2:47 12 tr in the right tool for the job because the questioner wanted to replace each newline with a space, as shown in his example. The replacement of newlines is uniquely arcane for sed but easily done by tr. This is a common question. Performing regex replacements is not done by tr but by sed, which would be the right tool... for a different question. – Mike S Dec 28 '16 at 15:01 3 "tr" can also just delete the newline ` tr -d '\n' ` however you may also like to delete returns to be more universal ` tr -d '\012\015' `. – anthony Feb 27 '17 at 23:44 2 WARNING: "tr" acts differently with regards to a character ranges between Linux and older Solaris machines (EG sol5.8). EG: ` tr -d 'a-z' ` and ` tr -d '[a-z]' `. For that I recommend you use "sed" which doesn't have that difference. – anthony Feb 27 '17 at 23:45 2 @MikeS Thanks for the answer. Follow tr '\012' ' ' with an echo. Otherwise the last linefeed in the file is deleted, too. tr '\012' ' ' < filename; echodoes the trick. – Bernie Reiter Dec 28 '17 at 23:30 Show 5 more comments 42 Answers 1 2 Next 1624 +--+ Use this solution with GNU sed: sed ':a;N;$!ba;s/\n/ /g' file This will read the whole file in a loop, then replaces the newline(s) with a space. Explanation: Create a label via :a. Append the current and next line to the pattern space via N. If we are before the last line, branch to the created label $!ba ($! means not to do it on the last line as there should be one final newline). Finally the substitution replaces every newline with a space on the pattern space (which is the whole file). Here is cross-platform compatible syntax which works with BSD and OS X's sed (as per @Benjie comment): sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/ /g' file As you can see, using sed for this otherwise simple problem is problematic. For a simpler and adequate solution see this answer. Share Improve this answer Follow edited Sep 23 '18 at 17:56 jpp 135k2929 gold badges200200 silver badges243243 bronze badges answered Aug 9 '09 at 20:26 Zsolt Botykai 46.4k1414 gold badges8181 silver badges102 +--+ Long answer from the sed FAQ 5.10 5.10. Why can't I match or delete a newline using the \n escape sequence? Why can't I match 2 or more lines using \n? The \n will never match the newline at the end-of-line because the newline is always stripped off before the line is placed into the pattern space. To get 2 or more lines into the pattern space, use the 'N' command or something similar (such as 'H;...;g;'). Sed works like this: sed reads one line at a time, chops off the terminating newline, puts what is left into the pattern space where the sed script can address or change it, and when the pattern space is printed, appends a newline to stdout (or to a file). If the pattern space is entirely or partially deleted with 'd' or 'D', the newline is not added in such cases. Thus, scripts like sed 's/\n//' file # to delete newlines from each line sed 's/\n/foo\n/' file # to add a word to the end of each line will NEVER work, because the trailing newline is removed before the line is put into the pattern space. To perform the above tasks, use one of these scripts instead: tr -d '\n' < file # use tr to delete newlines sed ':a;N;$!ba;s/\n//g' file # GNU sed to delete newlines sed 's/$/ foo/' file # add "foo" to end of each line Since versions of sed other than GNU sed have limits to the size of the pattern buffer, the Unix 'tr' utility is to be preferred here. If the last line of the file contains a newline, GNU sed will add that newline to the output but delete all others, whereas tr will delete all newlines. To match a block of two or more lines, there are 3 basic choices: (1) use the 'N' command to add the Next line to the pattern space; (2) use the 'H' command at least twice to append the current line to the Hold space, and then retrieve the lines from the hold space with x, g, or G; or (3) use address ranges (see section 3.3, above) to match lines between two specified addresses. Choices (1) and (2) will put an \n into the pattern space, where it can be addressed as desired ('s/ABC\nXYZ/alphabet/g'). One example of using 'N' to delete a block of lines appears in section 4.13 ("How do I delete a block of specific consecutive lines?"). This example can be modified by changing the delete command to something else, like 'p' (print), 'i' (insert), 'c' (change), 'a' (append), or 's' (substitute). Choice (3) will not put an \n into the pattern space, but it does match a block of consecutive lines, so it may be that you don't even need the \n to find what you're looking for. Since GNU sed version 3.02.80 now supports this syntax: sed '/start/,+4d' # to delete "start" plus the next 4 lines, in addition to the traditional '/from here/,/to there/{...}' range addresses, it may be possible to avoid the use of \n entirely. Share Improve this answer Follow edited Jun 20 '20 at 9:12 Community♦ 111 silver badge answered Oct 8 '11 at 14:55 hdorio 11.7k55 gold badges3131 silver badges3 08********08 #] multi-line sed 04Mar2021 +-----+ https://www.baeldung.com/linux/sed-replace-multi-line-string >> very powerful, so consider in the future, but next idea is simple +-----+ https://unix.stackexchange.com/questions/26284/how-can-i-use-sed-to-replace-a-multi-line-string I think, it's better to replace \n symbol with some other symbol, and then work as usual: e.g. not-worked source code: cat alpha.txt | sed -e 's/a test\nPlease do not/not a test\nBe/' can be changed to: cat alpha.txt | tr '\n' '\r' | sed -e 's/a test\rPlease do not/not a test\rBe/' | tr '\r' '\n' If anybody doesn't know, \n is UNIX line ending, \r\n - windows, \r - classic Mac OS. Normal UNIX text doesn't use \r symbol, so it's safe to use it for this case. You can also use some exotic symbol to temporarily replace \n. As an example - \f (form feed symbol). You can find more symbols here. cat alpha.txt | tr '\n' '\f' | sed -e 's/a test\fPlease do not/not a test\fBe/' | tr '\f' '\n' edited Dec 24 '17 at 19:34 user530873 24711 silver badge88 bronze badges answered Aug 27 '14 at 10:03 xara see "$d_bin""review form copy.sh" cat "$biblio" | tr '\n' '\f' | sed 's/\f$[^[]$/ \1/g;s/\f\[/\f\f\[/g' | tr '\f' '\n' >"$p_temp" echo >>"$p_temp" "" echo >>"$p_temp" "" echo >>"$p_temp" "References to authors own papers (check this!)" grep >"$p_tem2" "$authorsLastNames" "$p_temp" echo >>"$p_tem2" "" cat "$p_tem2" | tr '\n' '\f' | sed 's/$\f[^[].*\f$/\f/g;s/\f\[/\f\f[/g;s/\f\f/\f>> \f\f/g' | tr '\f' '\n' >"$p_tem3" ******** 22Oct2020 great example in great detail see link d_Qndfs 'website updates notes.txt' # cat "$p_webPage" | grep --invert-match '|[#\!: path_insertIn_fHand (link d_webWork \"\1\") fout |;s|> OK, this works for ONE ../, what about multiple? +-----+ Here are the guts and the pain... # p_temp="$d_temp""update_encoding temp.txt" # example with '[ \\t].*/[#!: path_insertIn_fHand (link d_webWork \1) fout/;s!"$p_temp" # $ cat "$p_webPage" | grep --invert-match '[ \\t].*/[#!: path_insertIn_fHand (link d_webWork \1) fout/' # $ cat "$p_webPage" | grep --invert-match '[ \\t].*/[#\!: path_insertIn_fHand (link d_webWork \1) fout/" # $ cat "$p_webPage" | grep --invert-match '/[#\!: path_insertIn_fHand (link d_webWork \1) fout/" >> Better, just need quotes around filename # $ cat "$p_webPage" | grep --invert-match '/[#\!: path_insertIn_fHand (link d_webWork \"\1\") fout /" >> Beautiful!! +-----+ Now for http://www.BillHowell.ca, replace ! by % as sed separator : # p_webPage="$d_webRaw""page projects.html" # $ cat "$p_webPage" | grep --invert-match '/[#\!: path_insertIn_fHand (link d_webWork \"\1\") fout /;s%> OK, albeit only one link had that There are no ../ to replace Pick another file with that : "$d_webRaw""Pandemics, health, and the Sun/influenza/Howell - influenza virus.html" # $ cat "$p_webPage" | grep --invert-match '/[#\!: path_insertIn_fHand (link d_webWork \"\1\") fout /;s%> wow, something seriusly wrong :

/[#\!: path_insertIn_fHand (link d_webWork \"\1\") fout /;s%> This is mangled too, in that it doesn't solve http: stuff, but at least doesn't corrupt them. >> Oh, BillHowell rather than billhowell, -e option # cat "$p_webPage" | grep --invert-match '/[#\!: path_insertIn_fHand (link d_webWork \"\1\") fout /;s%> Hmm, maybe the problem is the % separator # cat "$p_webPage" | grep --invert-match '/[#\!: path_insertIn_fHand (link d_webWork \"\1\") fout /;s#> oops, no http://www.billhowell # cat "$p_webPage" | grep --invert-match '/[#\!: path_insertIn_fHand (link d_webWork \"\1\") fout /;s#|[#\!: path_insertIn_fHand (link d_webWork \"\1\") fout |;s||[#\!: path_insertIn_fHand (link d_webWork \"\1\") fout |;s|> NYET - doesn't replace http://www.bill >> escape [], nyet, lowercase billhowell >> OK works! so the ignore case flag is wrong? use -e or -i and [i,i] after last separator >> OK, checks out +-----+ # Now, back to the ../ # p_webPage="$d_webRaw""Pandemics, health, and the Sun/influenza/Howell - influenza virus.html" # cat "$p_webPage" | grep --invert-match '|[#\!: path_insertIn_fHand (link d_webWork \"\1\") fout |;s|> OK, this works for ONE ../, what about multiple? # cat "$p_webPage" | grep --invert-match '|[#\!: path_insertIn_fHand (link d_webWork \"\1\") fout |;s|> Nope - just replaced one ../ # Brute force - just sequence to a max of 4 : # cat "$p_webPage" | grep --invert-match '|[#\!: path_insertIn_fHand (link d_webWork \"\1\") fout |;s|> this works but is ugly and probably inefficient >> there must be a way to cleanly get multiple-sequenced (repeated) strings # bash command : cat "$p_webPage" | grep --invert-match '|[#\!: path_insertIn_fHand (link d_webWork \"\1\") fout |;s||[#\!: path_insertIn_fHand (link d_webWork \"\1\") fout |;s|%20| |g;s||[#\!: path_insertIn_fHand (link d_webWork \"\1\") fout |;s|%20| |g;s|"' p_temp '" ' ; +-----+ Recover %20 in mailtos (IMPORTANT!) Use a QNial routine *********** 06Jan2020 WCCI mas email work search "Linux sed and how do I replace a comma?" https://unix.stackexchange.com/questions/36734/is-there-a-way-to-detect-null-bytes-nul-0-in-sed Yes, the pattern \x00 matches to the null byte. Example: $ printf "\0\n\0\n" > file $ sed -e 's/\x00/test/' -i file $ cat file test test $ edited Apr 18 '12 at 12:05 answered Apr 18 '12 at 9:34 jofel I'll try problematic email : $ echo '"Liao, Chien-Chih" ,' | tr "\x0" 'z' "Liao, Chien-Chih" , Manually replace the whole address : $ echo '"Liao, Chien-Chih" > Wow! I couldn't correct it! >> I simply removed the email from the list - a bit scary, perhaps this is intentional to defeat or mess up email servers?! *********** 15Aug2019 case-insensitive matching : https://stackoverflow.com/questions/16917623/use-sed-with-ignore-case-while-adding-text-before-some-pattern#16917665 You can use the following: sed 's/[Ff][Ii][Rr][Ss][Tt]/last/g' file Otherwise, you have the /I and n/i flags: sed 's/first/last/Ig' file From man sed: I i The I modifier to regular-expression matching is a GNU extension which makes sed match regexp in a case-insensitive manner. edited Jun 20 '16 at 11:24 answered Jun 4 '13 at 12:19 fedorqui ************* 14Apr2019 https://askubuntu.com/questions/408960/how-to-sed-and-replace-string-with-a-folder-path In circumstances where the replacement string or pattern string contain slashes, you can make use of the fact that GNU sed allows an alternative delimiter for the substitute command. Common choices for the delimiter are the pipe character | or the hash # - the best choice of delimiting character will often depend on the type of file being processed. In your case you can try sed -i "s#_HOME_DIR_#$HOME_DIR#" Also note that you mistyped your variable name $HOME_DIR (it does not start with an underscore). $ HOME_DIR=/opt/my_home $ echo "_HOME_DIR_/config/data/_DOMAIN_/users.conf" | sed "s#_HOME_DIR_#$HOME_DIR#" /opt/my_home/config/data/_DOMAIN_/users.conf Just fyi you may want to avoid using all-caps for user variables in your script - they should be reserved for system variables. Also the g switch is not necessary unless you want to make multiple replacements on a single line. answered Jan 22 '14 at 2:54 steeldriver ************** 13Apr2019 search "Linux and multiple text replacements in same sed" https://stackoverflow.com/questions/26568952/how-to-replace-multiple-patterns-at-once-with-sed#26568996 sed is a stream editor. It searches and replaces greedily. The only way to do what you asked for is using an intermediate substitution pattern and changing it back in the end. echo 'abcd' | sed -e 's/ab/xy/;s/cd/ab/;s/xy/cd/' edited Oct 26 '14 at 6:56 answered Oct 26 '14 at 1:43, kuriouscoder ************************** 30Aug2018 UTF-16 to UTF-8 https://stackoverflow.com/questions/2398393/identifying-and-removing-null-characters-in-unix 15 down vote A large number of unwanted NUL characters, say one every other byte, indicates that the file is encoded in UTF-16 and that you should use iconv to convert it to UTF-8. answered Mar 7 '10 at 23:16 Ignacio Vazquez-Abram I used: recode UTF-16..UTF-8 to get rid of zeroes in file. edited Dec 17 '16 at 13:00, kenorb answered Jun 22 '15 at 10:04, logisec *************************** 30Aug2018 problem replacing nulls in sed $ cat "/media/bill/PROJECTS/System_maintenance/Netflix/0_Netflix myViews.txt" | tr \\n \\0 | sed 's/Report a problem\\0//' | sed 's/⊘\\0//' | tr \\0 \\n $ cat "/media/bill/PROJECTS/System_maintenance/Netflix/0_Netflix myViews.txt" | tr \\n \\0 | sed 's/Report a problem//' | tr \\0 \\n $ cat "/media/bill/ramdisk/Netflix watch list temp1.txt" | tr \\n \\0 | sed 's/Report a problem//' | sed 's/⊘//' | tr \\0 \\n >"/media/bill/ramdisk/Netflix watch list temp2.txt" > No workee!? nulls cause problems even with tr $ cat "/media/bill/ramdisk/Netflix watch list temp1.txt" | sed 's/Report a problem//' | sed 's/⊘//' | tr \\n \\0 | sed 's/$2018.*$/\1 fuck you/' | tr \\0 \\n $ cat "/media/bill/ramdisk/Netflix watch list temp1.txt" | sed 's/Report a problem//' | sed 's/⊘//' | sed 's/$2018.*$/\1 blah you/' $ cat "/media/bill/ramdisk/Netflix watch list temp1.txt" | sed 's/Report a problem//' | sed 's/⊘//' | tr \\n \\0 | sed 's/$\\0\\0$/$\\0$/' | tr \\0 \\n https://www.unix.com/shell-programming-and-scripting/54417-replacing-nulls-using-sed.html $ cat "/media/bill/ramdisk/Netflix watch list temp1.txt" | sed 's/Report a problem//' | sed 's/⊘//' | tr \\n \\0 | sed 's/||//g;s/||//g' | tr \\0 \\n >> nyet https://superuser.com/questions/287997/how-to-use-sed-to-remove-null-bytes This is a solution that involves replacing into space characters which should work in all occasions: sed 's/\x0/ /g' file1 > file2 answered May 24 '11 at 16:25, Tom Wijsman $ cat "/media/bill/ramdisk/Netflix watch list temp1.txt" | sed 's/Report a problem//' | sed 's/⊘//' | tr \\n \\0 | sed 's/$2018\.*$\x0/\1 blah /g' | sed 's/\x0\x0//g' | tr \\0 \\n $ cat "/media/bill/ramdisk/Netflix watch list temp1.txt" | sed 's/Report a problem//' | sed 's/⊘//' | tr \\n \\0 | sed 's/$2018$$\.*$\x0/\1\2 blah /g' | sed 's/\x0\x0//g' | tr \\0 \\n $ cat "/media/bill/ramdisk/Netflix watch list temp1.txt" | sed 's/Report a problem//' | sed 's/⊘//' | tr \\n \\0 | sed 's/$2018$$.*$\x0/\1\2 blah /g' | sed 's/\x0\x0//g' | tr \\0 \\n $ cat "/media/bill/ramdisk/Netflix watch list temp1.txt" | sed 's/Report a problem//;s/⊘//' | tr \\n \\0 | sed 's/$2018$$.*$\x0/\1\2 blah /g' | sed 's/\x0\x0//g' | tr \\0 \\n *************************** 15Jul2018 using sed for RaspPi high-number search-replace! (when I don't have kwrite) create temporary file to preview changes : d_QNial_temp "Howell - Background math for Lucas Universal Force, Chapter 4 temp.txt" see "/media/bill/PROJECTS/Qnial/MY_NDFS/symbols grep-sed.sh" ... cat "$finn" | sed 's/cos(Aθoc(POIo)$[^)]$/cos(Aθoc(POIo))\1/g' \ | sed 's/sin(Aθoc(POIo)$[^)]$/sin(Aθoc(POIo))\1/g' \ >"$fout" ... *************************** 14Jul2018 from "/media/bill/PROJECTS/Qnial/MY_NDFS/symbols translate regexpr.sh" Lucas project f_trans() { cat "$d_Luca$1" | sed "s/Ro$[cd]$$[vsh]$(POIo$.\{0,5\}$)/Ro\1\2(POIo)/g" >"$d_temp""f_trans1.txt" cat "$d_temp""f_trans1.txt" | sed "s/Ro$[cd]$$[vsh]$(POIp$.\{0,5\}$)/Ro\1\2(POIp(t))/g" >"$d_temp""f_trans2.txt" cat "$d_temp""f_trans1.txt" | sed "s/Ro$[cd]$$[vsh]$(POIp$.\{0,5\}$)/Ro\1\2(POIp(t))/g" >"$d_temp""f_trans2.txt" cat "$d_temp""f_trans1.txt" | sed "s/Ro$[cd]$$[vsh]$(POIp$.\{0,5\}$)/Ro\1\2(POIp(t))/g" >"$d_temp""f_trans2.txt" cat "$d_temp""f_trans2.txt" | sed "s/EIo$[cd]$$[vsh]$(POIo$.\{0,5\}$)/EIo\1\2(POIo,t)/g" >"$d_temp""f_trans3.txt" cat "$d_temp""f_trans2.txt" | sed "s/EIo$[cd]$$[vsh]$(POIp$.\{0,5\}$)/EIo\1\2(POIp(t),t)/g" >"$d_temp""f_trans3.txt" cat "$d_temp""f_trans3.txt" | sed "s/EIp$[cd]$$[vsh]$(POIo$.\{0,5\}$)/EIp\1\2(POIo(t),t)/g" >"$d_temp""f_trans4.txt" cat "$d_temp""f_trans3.txt" | sed "s/EIp$[cd]$$[vsh]$(POIp$.\{0,5\}$)/EIp\1\2(POIp,t)/g" >"$d_temp""f_trans4.txt" ... ********************************* 15Apr2018 What characters do I need to escape when using sed in a sh script? https://unix.stackexchange.com/questions/32907/what-characters-do-i-need-to-escape-when-using-sed-in-a-sh-script # edited Oct 24 '17 at 12:20, Stéphane Chazelas # answered Feb 29 '12 at 1:06, Gilles In a nutshell, for sed 's/…/…/': Write the regex between single quotes. Use '\'' to end up with a single quote in the regex. Put a backslash before $.*/[\]^ and only those characters (but not inside bracket expressions). Inside a bracket expression, for - to be treated literally, make sure it is first or last ([abc-] or [-abc], not [a-bc]) Inside a bracket expression, for ^ to be treated literally, make sure it is not first (use [abc^], not [^abc]) To include ] in the list of characters matched by a bracket expression, make it the first character (or first after ^ for a negated set): []abc] or [^]abc] (not [abc]] nor [abc\]]). In the replacement text: & and \ need to be quoted, as do the delimiter (usually /) and newlines. \ followed by a digit has a special meaning. \ followed by a letter has a special meaning (special characters) in some implementations, and \ followed by some other character means \c or c depending on the implementation. With single quotes around the argument (sed 's/…/…/'), use '\'' to put a single quote in the replacement text. # from https://unix.stackexchange.com/questions/32907/what-characters-do-i-need-to-escape-when-using-sed-in-a-sh-script # edited Oct 24 '17 at 12:20, Stéphane Chazelas # answered Feb 29 '12 at 1:06, Gilles in the list of characters matched by a bracket expression '$.*/[\]^' In the replacement text: '&\/\n' ; # answered Feb 29 '12 at 2:56, R Perrin The problem you're experiencing isn't due to shell interpolating and escapes - it's because you're attempting to use extended regular expression syntax without passing sed the -r or --regexp-extended option. Change your sed line from sed 's/(127\.0\.1\.1)\s/\1/' [some file] to sed -r 's/(127\.0\.1\.1)\s/\1/' [some file] and it will work as I believe you intend. By default sed uses uses basic regular expressions (think grep style), which would require the following syntax: sed 's/$127\.0\.1\.1$[ \t]/\1/' [some file] # 15Apr2018 Currently I do this manually ********************************* 08Apr2018 Problems with sed (translate symbols.ndf with Lucas project) Problem here was including $.*$ at start & end of s-expression, so only one line occurence was picked out inspite of g-global option. Also - initially screwed up "^" - needs to be first character inside "[...]" # $ echo " = Vons(particle)^2/c*sin(AOpc(POIo,t))^2*cos(AOpc(POIo,t))*Rodh(Vonv_X_Rpcv(POIo(t),t))" | sed 's/AOpc(POIo,t)/Aθpc(POIo(t),t)/g' # $ echo " = Vons(particle)^2/c*sin(AOpc(POIo,t))^2*cos(AOpc(POIo,t))*Rodh(Vonv_X_Rpcv(POIo(t),t))" | sed "s/$.*$$^[A-Za-z0-9θφ]$$'AOpc(POIo,t)'$$^[A-Za-z0-9θφ]$$.*$/\1\2'Aθpc(POIo(t),t)'\4\5/g" # $ echo " = Vons(particle)^2/c*sin(AOpc(POIo,t))^2*cos(AOpc(POIo,t))*Rodh(Vonv_X_Rpcv(POIo(t),t))" | sed "s/$.*$$[^A-Za-z0-9θφ]$$AOpc(POIo,t)$$[^A-Za-z0-9θφ]$$.*$/\1\2Aθpc(POIo(t),t)\4\5/g" # $ echo " = Vons(particle)^2/c*sin(AOpc(POIo,t))^2*cos(AOpc(POIo,t))*Rodh(Vonv_X_Rpcv(POIo(t),t))" | sed 's/$.*$$[^A-Za-z0-9θφ]$$AOpc(POIo,t)$$[^A-Za-z0-9θφ]$$.*$/\1\2Aθpc(POIo(t),t)\4\5/g' # $ echo " = Vons(particle)^2/c*sin(AOpc(POIo,t))^2*cos(AOpc(POIo,t))*Rodh(Vonv_X_Rpcv(POIo(t),t))" | sed 's/$[^A-Za-z0-9θφ]$$AOpc(POIo,t)$$[^A-Za-z0-9θφ]$/\1Aθpc(POIo(t),t)\3/g' # $ echo " = Vons(particle)^2/c*sin(AOpc(POIo,t))^2*cos(AOpc(POIo,t))*Rodh(Vonv_X_Rpcv(POIo(t),t))" | sed "s/$[^A-Za-z0-9θφ]$$AOpc(POIo,t)$$[^A-Za-z0-9θφ]$/\1Aθpc(POIo(t),t)\3/g" >> OK # echo " = Vons(particle)^2/c*sin(AOpc(POIo,t))^2*cos(AOpc(POIo,t))*Rodh(Vonv_X_Rpcv(POIo(t),t))" | sed "s/$[^A-Za-z0-9θφ]$$AOpc(POIo,t)$$[^A-Za-z0-9θφ]$/\1Aθpc(POIo(t),t)\3/g" # sed '"s/$[^A-Za-z0-9θφ]$$AOpc(POIo,t)$$[^A-Za-z0-9θφ]$/\1Aθpc(POIo(t),t)\3/g"' '" = Vons(particle)^2/c*sin(AOpc(POIo,t))^2*cos(AOpc(POIo,t))*Rodh(Vonv_X_Rpcv(POIo(t),t))"' >> OK ************************** 02Feb2018 # /media/bill/PROJECTS/My Reviews/3_conference review setup.sh paper_tbk="$d_reviews""$confNmYr $paperNum ptbk $paperAuthTitl"".txt" revew_cmt="$d_tempory""$confNmYr $paperNum rcmt $paperAuthTitl"".txt" echo >"$revew_cmt" " " # extract multiline comments echo "1st sed" echo "" cat "$paper_tbk" | sed -n ' :main # if occurrence of ‘^message("#------’ print and load next line /||/ { p; n; bmain; } bprint_msg ' >>"$revew_cmt" ************************** 26Jan2018 # search "Linux rsync and how do I copy only the files in a directory?" # search "Linux rsync and how do I ignore directories?" # https://serverfault.com/questions/368370/how-do-i-exclude-directories-when-listing-files # ls -p | grep -v '/$' # answered Mar 10 '12 at 20:11, Sean O'Leary search "Linux ls and how do I use a variable in sed?" +--+ https://stackoverflow.com/questions/13055889/sed-with-literal-string-not-input-file You have a single quotes conflict, so use: echo "A,B,C" | sed "s/,/','/g" If using bash, you can do too (<<< is a here-string): sed "s/,/','/g" <<< "A,B,C" but not sed "s/,/','/g" "A,B,C" because sed expect file(s) as argument(s) EDIT: if you use ksh or any other ones : echo string | sed ... edited Jan 7 '14 at 16:57 answered Oct 24 '12 at 19:00, Gilles Quenot +--+ See Howell's "/media/bill/SWAPPER/bin/rsync quickie.sh" # rsync_WCCI2018_photos - don't want recursion, no subdirectory creation!!! # https://stackoverflow.com/questions/16647476/how-to-rsync-only-a-specific-list-of-files # Yes, and to reiterate: The filenames that are read from the FILE are all relative to the source dir. – ash Feb 12 '16 at 2:25 rsync_WCCI2018_photos() { becho "+---------------------------------------------------------+" becho "rsync_WCCI2018_photos() - no recursion, start of operation $dater" d_out="/media/bill/HOWELL_BASE/Website/Neural nets/2018 WCCI Rio de Janeiro/" becho "" d_src="/media/bill/SWAPPER/2018 WCCI Rio de Janiero/portraits/" becho "cd '$d_src'" cd "$d_src" becho "ls -1p '$d_src' | grep -v '/$' | rsync >'$p_log1' 2>&1 --files-from=- '$d_src' '$d_out' " ls -1p "$d_src" | grep -v "/$" | rsync >"$p_log1" 2>&1 --files-from=- "$d_src" "$d_out" greper becho "" d_src="/media/bill/SWAPPER/2018 WCCI Rio de Janiero/mass emails/images/" becho "cd '$d_src'" cd "$d_src" becho "ls -1p '$d_src' | grep -v '/$' | rsync >'$p_log1' 2>&1 --files-from=- '$d_src' '$d_out' " ls -1p "$d_src" | grep -v "/$" | rsync >"$p_log1" 2>&1 --files-from=- "$d_src" "$d_out" greper becho "" dater=$(date +"%y%m%d %kh%Mm") becho "rsync_WCCI2018_photos() - no recursion, end of operation $dater" becho "" } ************************ 13Jan2018 email address extraction from text files # tests with sed # email_test=" From: ""Saxena, Vishal (v-saxena@u-idaho.edu)"" " # echo "$email_test" | sed 's/$[A-Za-z]*@[A-Za-z]*$/\1/g' # echo "$email_test" | sed 's/$.*$$[A-Za-z0-9_]*@[A-Za-z0-9_]*$$.*$/\2/g' # echo "$email_test" | sed 's/$.*$$[A-Za-z0-9.\-_]*@[A-Za-z0-9.\-_]*$$.*$/\2/g' # echo "$email_test" | sed 's/$.*$$[A-Za-z0-9._]*@[A-Za-z0-9._]*$$.*$/\2/g' # @uidaho.edu # echo "$email_test" | sed 's/$.*$$[A-Za-z0-9._]*$$@[A-Za-z0-9._]*$$.*$/\2\3/g' # @uidaho.edu # echo "$email_test" | sed 's/$.*$$[A-Za-z0-9._]*$$@[A-Za-z0-9._]*$$.*$/\2\3/g' # email_test=" From: \"Saxena, Vishal (v-saxena@u-idaho.edu)\"" # echo "$email_test" | sed 's/$.*$$[A-Za-z0-9._]*$$@[A-Za-z0-9._]*$$.*$/\2\3/g' # @u # echo "$email_test" | sed 's/$.*$$[A-Za-z0-9.\\-_]*$$@[A-Za-z0-9.\\-_]*$$.*$/\2\3/g' # sed: -e expression #1, char 64: Invalid range end # How do I include the character `- ?? # echo "$email_test" | sed 's/$.*$$[-A-Za-z0-9._]*$$@$$[-A-Za-z0-9._]*$$.*$/\2\3\4/g' # @u-idaho.edu # OK the `- character problem is now OK, so why am I missing \2 in the output? # echo "$email_test" | sed 's/$.*$$[-A-Za-z0-9._]*$$.*$/\2/g' # gives nothing! # echo "$email_test" | sed 's/$.*$$v[-A-Za-z0-9._]*$$.*$/\2/g' # v-saxena # but for this, I specified a starting character! # echo "$email_test" | sed 's/$[-A-Za-z0-9._]*$$.*$/\2/g' # From: "Saxena, Vishal (v-saxena@u-idaho.edu)" # but this is the entire expression!! (idiot - I put \2!!) # email_test=" From: \"Saxena, Vishal (v-saxena@u-idaho.edu)\" " # echo "$email_test" | sed 's/$[-A-Za-z0-9._]*$$.*$/\1/g' # outputs nothing # echo "$email_test" | sed 's/$.*$$[-A-Za-z0-9._]*$/\2/g' # outputs nothing # echo "$email_test" | sed 's/$.*$$[-A-Za-z0-9._]?[-A-Za-z0-9._]*$$@$$[-A-Za-z0-9._]*$$.*$/\2\3\4/g' # echo "$email_test" | sed 's/$.*$$[-A-Za-z0-9._][-A-Za-z0-9._]*$$@$$[-A-Za-z0-9._]\+$$.*$/\2\3\4/g' # a@uidaho.edu # echo "$email_test" | sed 's/$.*$$[^-A-Za-z0-9._]\+$$[-A-Za-z0-9._]\+$$@$$[-A-Za-z0-9._]\+$$.*$/\3\4\5/g' # vsaxena@uidaho.edu # echo "$email_test" | sed 's/$[^-A-Za-z0-9._]\+$$[-A-Za-z0-9._]\+$$@$$[-A-Za-z0-9._]\+$$.*$/\2\3\4/g' # From: "Saxena, Vishalv-saxena@u-idaho.ed echo "$email_test" | sed 's/$.*$$[^-A-Za-z0-9._]\+$$[-A-Za-z0-9._]\+$$@$/\3\4/g' | \ sed 's/$@$$[-A-Za-z0-9._]\+$$.*$/\1\2/g' # From: "Saxena, Vishalv-saxena@u-idaho.edvsaxena # https://www.linuxquestions.org/questions/linux-software-2/sed-to-extract-multiple-matches-in-a-line-348417/ 07-31-2005, 12:28 PM , twsnnva Can you use 'grep -o'? Code: Thomas@lightning:~$ cat test.txt then the three of them went swimming until they were quite tired Thomas@lightning:~$ cat test.txt | grep -o the[a-z] then them they Thomas@lightning:~$ That will give you a list, if that's not the format you want you could run it through a for loop. Code: Thomas@lightning:~$ for i in `cat test.txt|grep -o the[a-z]`; do echo -n $i; done ; echo thenthemthey Thomas@lightning:~$ ************************* 13Jan2018 search "Linux and sed why am getting the last match?" https://www.linuxquestions.org/questions/linux-desktop-74/sed-last-occurence-of-a-match-704206/ 02-12-2009, 12:39 PM, jan61 Moin, regular expressions are greedy (they catch the longest part of a line, which matches the pattern), so this will work: Code: jan@jack:~/tmp> echo 'blah-blah-blah-123' | sed -r 's/(.*)-.*/\1/' blah-blah-blah Jan P.S.: The only one exception from the "greedy behaviour" is the "?" quantifier in Perl compatible regular expressions (not supported by grep and sed). search "Linux and sed how do I get the first match?" *********************** 13Jan2017 search "Linux and sed how do I include a - in a character set?" I think I remember a comment - something about putting a character FIRST between brackets # email_test=" From: \"Saxena, Vishal (v-saxena@u-idaho.edu)\"" # How do I include the character `- ?? # echo "$email_test" | sed 's/$.*$$[A-Za-z0-9._]*-[A-Za-z0-9._]*$$@[A-Za-z0-9._]*-[A-Za-z0-9._]*$$.*$/\2\3/g' # -saxena@u-idaho.edu # echo "$email_test" | sed 's/$.*$$[-A-Za-z0-9._]*@[-A-Za-z0-9._]*$$.*$/\2\3/g' # ****************************** /home/bill/OOPM builds/0_environment.sh #] sed-cmakes - pull out my modifications to cmake files sed-cmakes() { p_log="$d_log""5_cmakes-edits log.txt" p_out="$d_log""6_cmakes-flist in OPM.txt" p_mks="$d_log""7_cmakes edits in OPM $dater.txt" becho 'cd "/home/bill/OPM" ' cd "/home/bill/OPM" becho 'find-grep^2-sed^2-read -> see sed-makes() ' find -nowarn -name "?*cmake" | grep "?*Find?*" | grep "(copy)" | sed 's/^\.\//\/home\/bill\/OPM\//' | sed 's/ (copy)//' | while read LINE do sed >>"$p_mks" -n ' :main # if occurrence of ‘^message("#------’ print and load next line /^message("#------/ { bprint_msg; } n bmain # print the old line :print_msg p n /^message("#------/ { p; n; bmain; } bprint_msg ' $LINE echo >>"$p_mks" "" echo >>"$p_mks" "" done #| sed 's/^/"/' | sed >"$p_mks" 's/$/"/' #becho 'cat "$p_out" | sed >"$p_mks" "s/$?*$/"\1"/" ' # cat "$p_out" | cat >"/home/bill/OOPM builds/7_cmake file to process.txt" # cat "/home/bill/OOPM builds/7_cmake file to process.txt" | bash "/home/bill/OOPM builds/2_sed comment.sh" # | cat >"/home/bill/OOPM builds/7_cmake file to process.txt" | bash "/home/bill/OOPM builds/2_sed comment.sh" # | bash "/home/bill/OOPM builds/2_sed comment.sh" # | sed 's/$?*$/\1"/' # >"$p_mks" #| sed >\"$p_out\" 2>&1 becho ">> Results (manually copy-paste pertinent parts) : " becho "" } ****************************** # /home/bill/OOPM builds/2_sed comment.sh cat "/home/bill/OPM/opm-common/cmake/Modules/FindSuiteSparse.cmake" | sed -n ' :main # if occurrence of ‘^message("#------’ print and load next line /^message("#------/ { bprint_msg; } n bmain # print the old line :print_msg p n /^message("#------/ { p; n; bmain; } bprint_msg ' ****************************** # sed-cmakes - pull out my modifications to cmake files sed-cmakes() { p_log="$d_log""OPM build.txt" p_out="$d_log_grep""grep cmakes-Howell in OPM $dater.txt" becho 'cd "/home/bill/OPM" ' cd "/home/bill/OPM" becho 'find -nowarn -name -print ".cmake" ' find -nowarn -name "?*cmake" | grep "?*Find?*" | grep "(copy)" | sed 's/^\.\//\/home\/bill\/OPM\//' #| grep >"$p_out" 2>&1 --invert-match ":" # "/home/bill/OPM" #becho "sed -ri \"$3\" | grep >\"$p_out\" 2>&1 --invert-match \"Binary file \" " # grep -ri "$3" | grep >"$p_out" 2>&1 --invert-match "Binary file " becho ">> Results (manually copy-paste pertinent parts) : " becho "" } ******************** #!/bin/sh # /home/bill/OOPM builds/make/0_list of targets - bash transformations.sh # www.BillHowell.ca 13Apr2017 17:54 # /home/bill/OOPM builds/make/0_list of targets for initial file creation - template.txt # Template for the initial creation of target files, to avoid root ownership. # $ bash "0_list of targets - bash transformations.sh" cd "/home/bill/OOPM builds/make" cat "0_list of targets for initial file creation.txt" | \ sed 's/$.*$/\1 : \1_VAR : echo "0_dummy_test.txt"/' # sed >"0_list of targets for initial file creation - bash.sh" 's/$.*$/cp "0_list of targets for initial file creation - template.txt" "\1"/' ******************** #!/bin/sh # /home/bill/OOPM builds/make/0_header includes for make.sh # www.BillHowell.ca 14Apr2017 00:00 # $ bash "/home/bill/OOPM builds/make/0_header includes for make.sh" cd "/home/bill/OOPM builds/make" # Boost is in /usr/lib/x86_64-linux-gnu # 14Apr2017 00:15 As a first shot, I'll try shared libs ".so" rather than static ".a" ?????????? cat "lib-boost0-list from make.txt" | \ grep --invert-match "\-dev" | \ sed 's/$^Setting up $//' | sed 's/$:.*$//' | sed 's/$ .*$//' | sed 's/$[0-9].*$//' | \ sed 's/$/\.so \\/' | sed 's/$^.*$/\/usr\/lib\/x86_64-linux-gnu\/\1/' | sed $'s/^/\t/' | \ sort >"lib-boost0-list cleaned.txt" ******************** #!/bin/sh # /home/bill/OOPM builds/make/3_bash-it.sh # www.Billhowell.ca, 11Apr2017 initial # $ bash "/home/bill/OOPM builds/make/3_apt-it.sh" cd "/home/bill/OOPM builds/make" dater=$(date +"%y%m%d %kh%Mm") cmdtmp=$(cat "/home/bill/OOPM builds/make/9_command temp.txt") vartmp=$(cat "/home/bill/OOPM builds/make/9_variable temp.txt") f_log="$vartmp.txt" echo "vartmp= $vartmp" date_last=$(stat -c %y "$f_log" | sed 's/^20//; s/-//g; s/:/h/; s/:/m/' | cut -c 1-13) #echo "date_last= $date_last" cp "$f_log" "$vartmp log $date_last.txt" echo >"$f_log" "$vartmp >>\"$f_log\" 2>&1 " sudo $cmdtmp >>"$f_log" 2>&1 ******************** 27Jan2018 # $ bash "/media/bill/PROJECTS/My Reviews/3_conference review setup.sh" # www.BillHowell.ca 27Jan2018 initial d_reviews="/media/bill/PROJECTS/My Reviews/" setup_review() { revew_frm="$d_reviews""ISNN2018 0 review form.txt" paper_pdf="$d_reviews""$confNmYr $paperNum ppdf $paperAuthTitl"".pdf" paper_txt="$d_reviews""$confNmYr $paperNum ptxt $paperAuthTitl"".txt" revew_tmp="$d_reviews""$confNmYr $paperNum revt $paperAuthTitl"".txt" revew_txt="$d_reviews""$confNmYr $paperNum revw $paperAuthTitl"".txt" echo "revew_frm= $revew_frm" echo "paper_pdf= $paper_pdf" echo "paper_txt= $paper_txt" echo "revew_tmp= $revew_tmp" echo "revew_txt= $revew_txt" #cp "$revew_frm" "$revew_tmp" #pdftotext "$paper_pdf" "$paper_txt" echo "cat '$revew_tmp' | sed 's/$.*$1_Paper Review ISNN.txt$.*$/\1'$confNmYr' '$paperNum' revw '$paperAuthTitl'\2/' | sed 's/$^Paper : $/\1'$confNmYr' '$paperNum' revw '$paperAuthTitl'/' >'$revew_txt' " cat "$revew_tmp" | sed "s/$.*$1_Paper Review ISNN.txt$.*$/\1$confNmYr $paperNum revw $paperAuthTitl.txt/" | sed "s/$^Paper : $/\1$confNmYr $paperNum revw $paperAuthTitl/" >"$revew_txt" # test #echo "/media/bill/SWAPPER/My Reviews/1_Paper Review ISNN.txt" | sed "s/$.*$1_Paper Review ISNN.txt$.*$/\1$confNmYr $paperNum revw $paperAuthTitl.txt/" } ******************** 13Apr2017 17:52 My own example, created for the OPM project make system Notice the Perl (-P) search for a tab #!/bin/sh # /home/bill/OOPM builds/make/0_list of targets for initial file creation.sh # www.BillHowell.ca 13Apr2017 17:54 # /home/bill/OOPM builds/make/0_list of targets for initial file creation - template.txt # Template for the initial creation of target files, to avoid root ownership. # $ bash "/home/bill/OOPM builds/make/0_list of targets for initial file creation.sh" cd "/home/bill/OOPM builds/make" grep --invert-match -P '^\t' 0_Makefile | grep --invert-match ".*echo.*" | grep --invert-match ^[#] | grep --only-matching ".*.txt" | sort >"0_list of targets for initial file creation.txt" cat "0_list of targets for initial file creation.txt" | \ sed >"0_list of targets for initial file creation - bash.sh" 's/$.*$/cp "0_list of targets for initial file creation - template.txt" "\1"/' bash "0_list of targets for initial file creation - bash.sh" ~/Qnial/MY_NDFS $ ls -1 | sed 's/Toshiba/Dell64/' 0_Dell64 maintenance.txt 0_Dell64_maintenance notes.txt Dell64_clamscan_log.txt Dell64_clamscan.ndf Dell64_dirSizes_du.txt Dell64_dirSizes_log.txt Dell64_dirSizes.ndf Dell64_HD_backup_log.txt Dell64_HD_backup.ndf Dell64_maintenance_log.txt Dell64_maintenance.ndf http://www.computerhope.com/unix/used.htm Linux and Unix sed command http://www.grymoire.com/Unix/Sed.html#uh-8 Sed - An Introduction and Tutorial by Bruce Barnett good examples ********************* 07Apr2017 10:33 https://superuser.com/questions/318439/sed-error-sed-e-expression-1-char-extra-characters-after-command Sed error “sed: -e expression #1, char [#]: extra characters after command” You need to use a semicolon ; to separate sed commands on the same line. Specifically, between the N and s. No semicolon is needed when there is a newline which is why your script form works. change sed '/\r$/ {N s/\n//}' to sed '/\r$/ {N; s/\n//}' edited Nov 24 '14 at 15:00, fedorqui answered Aug 3 '11 at 10:03, jw013 ******************************************* ******************************************* SUMMARY *********************** 13Jan2017 search "Linux and sed how do I include a - in a character set?" I think I remember a comment - something about putting a character FIRST between brackets # email_test=" From: \"Saxena, Vishal (v-saxena@u-idaho.edu)\"" # How do I include the character `- ?? # echo "$email_test" | sed 's/$.*$$[A-Za-z0-9._]*-[A-Za-z0-9._]*$$@[A-Za-z0-9._]*-[A-Za-z0-9._]*$$.*$/\2\3/g' # -saxena@u-idaho.edu # echo "$email_test" | sed 's/$.*$$[-A-Za-z0-9._]*@[-A-Za-z0-9._]*$$.*$/\2\3/g' # +-----+ #] bash function for converting paths for use with SED - function sedPath https://unix.stackexchange.com/questions/32907/what-characters-do-i-need-to-escape-when-using-sed-in-a-sh-script Here is a bash function for converting paths for use with SED: function sedPath { path=$((echo $1|sed -r 's/([\$\.\*\/\[\\^])/\\\1/g'|sed 's/[]]/\[]]/g')>&1) } #Escape path for use with sed – user2428118 May 10 '16 at 12:57 ****************************** #] Often-Used Commands - http://www.computerhope.com/unix/used.htm If you use sed at all, you will probably want to know these commands. # (No addresses allowed with this command.) The # character begins a comment; the comment continues until the next newline. If you are concerned about portability, be aware that some implementations of sed (which are not POSIX conformant) may only support a single one-line comment, and then only when the very first character of the script is a #. Warning: if the first two characters of the sed script are #n, then the -n (no-autoprint) option is forced. If you want to put a comment in the first line of your script and that comment begins with the letter ‘n’ and you do not want this behavior, then be sure to either use a capital ‘N’, or place at least one space before the ‘n’. q [exit-code] This command only accepts a single address. Exit sed without processing any more commands or input. Note that the current pattern space is printed if auto-print is not disabled with the -n options. The ability to return an exit code from the sed script is a GNU sed extension. d Delete the pattern space; immediately start next cycle. p Print out the pattern space (to the standard output). This command is usually only used in conjunction with the -n command-line option. n If auto-print is not disabled, print the pattern space, then, regardless, replace the pattern space with the next line of input. If there is no more input then sed exits without processing any more commands. { commands } A group of commands may be enclosed between { and } characters. This is particularly useful when you want a group of commands to be triggered by a single address (or address-range) match. The s Command The syntax of the s command (which stands for "substitute") is: ‘s/regexp/replacement/flags’. The / characters may be uniformly replaced by any other single character within any given s command. The / character (or whatever other character is used in its stead) can appear in the regexp or replacement only if it is preceded by a \ character. The s command is probably the most important in sed and has a lot of different options. Its basic concept is simple: the s command attempts to match the pattern space against the supplied regexp; if the match is successful, then that portion of the pattern space which was matched is replaced with replacement. The replacement can contain \n (n being a number from 1 to 9, inclusive) references, which refer to the portion of the match which is contained between the nth $ and its matching $. Also, the replacement can contain unescaped & characters which reference the whole matched portion of the pattern space. Finally, as a GNU sed extension, you can include a special sequence made of a backslash and one of the letters L, l, U, u, or E. The meaning is as follows: \L Turn the replacement to lowercase until a \U or \E is found \l Turn the next character to lowercase \U Turn the replacement to uppercase until a \L or \E is found \u Turn the next character to uppercase \E Stop case conversion started by \L or \U To include a literal \, &, or newline in the final replacement, be sure to precede the desired \, &, or newline in the replacement with a \. The s command can be followed by zero or more of the following flags: g Apply the replacement to all matches to the regexp, not just the first. number Only replace the numberth match of the regexp. Note: the POSIX standard does not specify what should happen when you mix the g and number modifiers, and currently there is no widely agreed upon meaning across sed implementations. For GNU sed, the interaction is defined to be: ignore matches before the numberth, and then match and replace all matches from the numberth on. p If the substitution was made, then print the new pattern space. Note: when both the p and e options are specified, the relative ordering of the two produces very different results. In general, ep (evaluate then print) is what you want, but operating the other way round can be useful for debugging. For this reason, the current version of GNU sed interprets specially the presence of p options both before and after e, printing the pattern space before and after evaluation, while in general flags for the s command show their effect just once. This behavior, although documented, might change in future versions. w file If the substitution was made, then write out the result to the named file. As a GNU sed extension, two special values of file are supported: /dev/stderr, which writes the result to the standard error, and /dev/stdout, which writes to the standard output. e This command allows one to pipe input from a shell command into pattern space. If a substitution was made, the command that is found in pattern space is executed and pattern space is replaced with its output. A trailing newline is suppressed; results are undefined if the command to be executed contains a null character. This is a GNU sed extension. I, i The I modifier to regular-expression matching is a GNU extension which makes sed match regexp in a case-insensitive manner. M, m The M modifier to regular-expression matching is a GNU sed extension which causes ^ and $ to match respectively (in addition to the normal behavior) the empty string after a newline, and the empty string before a newline. There are special character sequences (\` and \') which always match the beginning or the end of the buffer. M stands for multi-line. Less Frequently-Used Commands Though perhaps less frequently used than those in the previous section, some very small yet useful sed scripts can be built with these commands. y/source-chars/dest-chars/ (The / characters may be uniformly replaced by any other single character within any given y command.) Transliterate any characters in the pattern space which match any of the source-chars with the corresponding character in dest-chars. Instances of the / (or whatever other character is used instead), \, or newlines can appear in the source-chars or dest-chars lists, provide that each instance is escaped by a \. The source-chars and dest-chars lists must contain the same number of characters (after de-escaping). a\ text As a GNU extension, this command accepts two addresses. Queue the lines of text which follow this command (each but the last ending with a \, which are removed from the output) to be output at the end of the current cycle, or when the next input line is read. Escape sequences in text are processed, so you should use \\ in text to print a single backslash. As a GNU extension, if between the a and the newline there is other than a whitespace-\ sequence, then the text of this line, starting at the first non-whitespace character after the a, is taken as the first line of the text block. (This enables a simplification in scripting a one-line add.) This extension also works with the i and c commands. i\ text As a GNU extension, this command accepts two addresses. Immediately output the lines of text which follow this command (each but the last ending with a \, which are removed from the output). c\ text Delete the lines matching the address or address-range, and output the lines of text which follow this command (each but the last ending with a \, which are removed from the output) in place of the last line (or in place of each line, if no addresses were specified). A new cycle is started after this command is done, since the pattern space will have been deleted. = As a GNU extension, this command accepts two addresses. Print out the current input line number (with a trailing newline). l n Print the pattern space in an unambiguous form: non-printable characters (and the \ character) are printed in C-style escaped form; long lines are split, with a trailing \ character to indicate the split; the end of each line is marked with a $. n specifies the desired line-wrap length; a length of 0 (zero) means to never wrap long lines. If omitted, the default as specified on the command line is used. The n parameter is a GNU sed extension. r filename As a GNU extension, this command accepts two addresses. Queue the contents of filename to be read and inserted into the output stream at the end of the current cycle, or when the next input line is read. Note that if filename cannot be read, it is treated as if it were an empty file, without any error indication. As a GNU sed extension, the special value /dev/stdin is supported for the file name, which reads the contents of the standard input. w filename Write the pattern space to filename. As a GNU sed extension, two special values of filename are supported: /dev/stderr, which writes the result to the standard error, and /dev/stdout, which writes to the standard output. The file will be created (or truncated) before the first input line is read; all w commands (including instances of the w flag on successful s commands) which refer to the same filename are output without closing and reopening the file. D If pattern space contains no newline, start a normal new cycle as if the d command was issued. Otherwise, delete text in the pattern space up to the first newline, and restart cycle with the resultant pattern space, without reading a new line of input. N Add a newline to the pattern space, then append the next line of input to the pattern space. If there is no more input then sed exits without processing any more commands. P Print out the portion of the pattern space up to the first newline. h Replace the contents of the hold space with the contents of the pattern space. H Append a newline to the contents of the hold space, and then append the contents of the pattern space to that of the hold space. g Replace the contents of the pattern space with the contents of the hold space. G Append a newline to the contents of the pattern space, and then append the contents of the hold space to that of the pattern space. x Exchange the contents of the hold and pattern spaces. Commands for sed gurus In most cases, use of these commands indicates that you are probably better off programming in something like awk or Perl. But occasionally one is committed to sticking with sed, and these commands can enable one to write quite convoluted scripts. : label [No addresses allowed with this command.] Specify the location of label for branch commands. In all other respects, a no-op (no operation performed). b label Unconditionally branch to label. The label may be omitted, in which case the next cycle is started. t label Branch to label only if there has been a successful substitution since the last input line was read or conditional branch was taken. The label may be omitted, in which case the next cycle is started. # enddoc