#] #] ********************* #] "$d_SysMaint"'Linux/read file & loops.txt' # www.BillHowell.ca 31May2018 initial # view in text editor, using constant-width font (eg courier), tabWidth = 3 Great blog examples at end of this file ... especially "...Reading from more than one file at a time..." #24************************24 #24************************24 # Table of Contents, generate with : # $ grep "^#]" "$d_SysMaint"'Linux/read file & loops.txt' | sed "s/^#\]/ /" # "$d_SysMaint"'Linux/read file & loops.txt' IFS='' to preserve leading [spaces,tabs] +-----+ 17Feb2023 leading [space, tab]s igonored by read, use IFS='' before read 17Feb2023 leading [space, tab]s igonored by read 10Jul2022 file read loops - eval to expand variables!!, see "$d_bin""fileops.sh" 07Jul2022 file read - failure to capture tabs 10Oct2019 leading spaces & tabs Multiple file reads (roster database-like) : from "$d_bin""email - remove emails from text file.sh" read tab-separated-columns of text file : while IFS=$'\t' read -r paperNumber copyright authorTitle; do 05Aug2018 calculations example 20Jul2018 nested read within while loop simply echo followed by read other cool examples To make the read in the loop body read from the terminal: https://stackoverflow.com/questions/9688371/how-to-read-from-user-within-while-loop-read-line You pipe data into your the while loops STDIN. So the read in get_ans is also taking data from that STDIN stream. My version in "/media/bill/PROJECTS/Qnial/MY_NDFS/symbols sort manual.sh" 20Jul2018 IFS stands for "internal field separator" IFS stands for "internal field separator". It is used by the shell to determine how to do word splitting, i. e. how to recognize word boundaries. 17Jul2018 see "/media/bill/PROJECTS/Qnial/MY_NDFS/symbols grep count.s" 31May2018 file read - conditional 31May2018 Howell - author provides MANY interesting cases! Howell - author provides MANY interesting cases! Reading from the output of another command, using process substitution Reading from more than one file at a time Reading a whole file into an array (Bash versions 4x and later) 23May2018 for loops : #24************************24 # Setup, ToDos, #] 29Aug2019 example $ bash "/media/bill/SWAPPER/bin/text scroll.sh" text_scroll() { while read -u 9 line; do echo "$line" sleep "$1"s done 9< "$2" } text_scroll 0.1 "/media/bill/RaspPi_ext4_32Gb/Lucas - Universal Force/Howell - math of Lucas Universal Force.txt" #24************************24 08********08 #] ??Feb2024 08********08 #] ??Feb2024 08********08 #] ??Feb2024 08********08 #] ??Feb2024 08********08 #] ??Feb2024 08********08 #] ??Feb2024 08********08 #] ??Feb2024 08********08 #] ??Feb2024 08********08 #] ??Feb2024 08********08 #] ??Feb2024 08********08 #] ??Feb2024 08********08 #] ??Feb2024 08********08 #] ??Feb2024 #08********08 #] 05Feb2024 Segmentation fault - exec 681<"$pinn681" must appear before each file-loop!! comment-out transcript_unite_each_paragraph() bash "$d_bin"'fileops run.sh' with pth_get_dir_run active 14:37$ bash "$d_bin"'fileops run.sh' /home/bill/web/Qnial/MY_NDFS/ >> OK, no segmentation fault now pth_get_dArchive_run : 14:47$ bash "$d_bin"'fileops run.sh' /home/bill/web/Qnial/MY_NDFS/z_Archive/ >> OK what is problem with transcript_unite_each_paragraph()??? comment-out : # proceed to start of transcript, writing lines along way exec 681<"$pinn681" while IFS='' read -u 681 line681; do if [ 'Transcript' == "$line681" ]; then break; fi done # consolidate each paragraph, remove [timestamp, newLine]s lines along way # notice that reads continue from line above bolInn=0 while IFS='' read -u 681 line681; do if [ '' == "$line681" ]; then if [ "$bolInn" == 1 ]; then echo ' ' >>"$pout681" # save paragraph cat "$ptmp681" | tr \\n ' ' | sed 's| \([0-9]*:[0-9]*\) | |g' >>"$pout681" rm "$ptmp681" bolInn=0 echo ' ' >>"$pout681" fi else echo "$line681" >>"$ptmp681" bolInn=1 fi done 14:49$ bash "$d_bin"'fileops run.sh' /home/bill/web/Qnial/MY_NDFS/z_Archive/ >> OK, no segmentation fault maybe I need another exec 681<"$pinn681"?? uncomment 1st use 14:52$ bash "$d_bin"'fileops run.sh' /home/bill/web/Qnial/MY_NDFS/z_Archive/ >> OK, works put in another exec 681<"$pinn681" 14:56$ bash "$d_bin"'fileops run.sh' /home/bill/web/Qnial/MY_NDFS/z_Archive/ >> works!!! 08********08 #] 26Dec2023 #] +-----+ #] force read from terminal, eg user input!!! or nested scripts fail!! #] read -p "Press [Enter] key to continue..." > good discussion, but I have the opposite problem +-----+ https://askubuntu.com/questions/1170557/command-does-not-work-in-script-file-but-works-ok-with-shell-command command does not work in script file, but works ok with shell command Asked 4 years, 3 months ago Modified 1 year, 1 month ago Viewed 20k times +-----+ https://www.unix.com/shell-programming-and-scripting/279698-bash-read-p-option.html Bash read with -p option annacreek read -p "Press [Enter] key to continue..." > Howell: I added to pinn_strOld_strNew_replace : IFS='' >> YEAH!! it worked, and I had solved this in the past but forgot it... +-----+ https://unix.stackexchange.com/questions/614708/linux-bash-shell-read-with-tab-support-t Linux Bash Shell read with tab support (\t) Asked 2 years, 4 months ago Modified 2 years, 4 months ago Viewed 834 times +--+ You can always use the ksh93-style $'...' form of quotes that understand those escape sequences: IFS= read -r -p $'\thello ' var (the IFS= and -r are not relevant, I'm just adding them here as calling read without them rarely makes sense). Note that -p is not a standard sh function. In ksh/zsh, -p is to read from the co-process and prompts are specified with read 'var?Prompt: '. It's unfortunate that bash chose to introduce an incompatible API here. You don't have to use -p though, you can just do portably: printf >&2 '\thello ' IFS= read -r var printf does recognise those escape sequences in its format argument, and arguments for %b specifiers. Whether echo recognises them (or accepts a -e option to recognise them) depends on the implementation and for many implementations (including bash's builtin) on build time and runtime settings, so is best avoided. edited Oct 16, 2020 at 7:50 answered Oct 15, 2020 at 16:03 Stéphane Chazelas >> Howell: -p option does NOT work in LMDE +-----+ https://stackoverflow.com/questions/29689172/bash-read-ignores-leading-spaces Bash read ignores leading spaces Asked 7 years, 10 months ago Modified 4 years ago Viewed 21k times +--+ This is covered in the Bash FAQ entry on reading data line-by-line. The read command modifies each line read; by default it removes all leading and trailing whitespace characters (spaces and tabs, or any whitespace characters present in IFS). If that is not desired, the IFS variable has to be cleared: # Exact lines, no trimming #] while IFS='' read -r line; do #] printf '%s\n' "$line" #] done < "$file" As Charles Duffy correctly points out (and I'd missed by focusing on the IFS issue); if you want to see the spaces in your output you also need to quote the variable when you use it or the shell will, once again, drop the whitespace. Notes about some of the other differences in that quoted snippet as compared to your original code. The use of the -r argument to read is covered in a single sentence at the top of the previously linked page. The -r option to read prevents backslash interpretation (usually used as a backslash newline pair, to continue over multiple lines). Without this option, any backslashes in the input will be discarded. You should almost always use the -r option with read. As to using printf instead of echo there the behavior of echo is, somewhat unfortunately, not portably consistent across all environments and the differences can be awkward to deal with. printf on the other hand is consistent and can be used entirely robustly. edited Apr 17, 2015 at 2:35 answered Apr 17, 2015 at 2:26 Etan Reisner 08********08 #] 10Jul2022 file read loops - eval to expand variables!!, see "$d_bin""fileops.sh" pthOptrNamL_pthNdfL_argOrderNum() { pthOptrNamL="$1" pthNdfL="$2" while read -u 9 optrNam; do while read -u 8 pthNdf; do # echo "pthNdf= $pthNdf" evalStr="echo ""$pthNdf" pthNdfFull=$( eval "$evalStr" ) # echo "pthNdfFull= $pthNdfFull" pth_optrNam_getArgOrderNum_dout "$pthNdfFull" "$optrNam" done 8< "$pthNdfL" done 9< "$pthOptrNamL" } 08********08 #] 07Jul2022 file read - failure to capture tabs Howell - I must have solved this problem in many of my bash scripts - can't remember how I did it use an easy short string to separate strPL!! : ' ~:~ ' ( enclosed by apos) >> works very nicely https://unix.stackexchange.com/questions/614708/linux-bash-shell-read-with-tab-support-t There is \t option for horizontal tab in echo wolf@linux:~$ echo hello hello wolf@linux:~$ wolf@linux:~$ echo -e '\thello' hello wolf@linux:~$ asked Oct 15, 2020 at 15:57 user avatar Wolf +--+ But yes, you can also get an actual tab character: read -p $'\t''hello ' Or read -p "$(echo -e '\thello ')" Or read -p "$(printf '\thello ')" edited Oct 15, 2020 at 16:21 answered Oct 15, 2020 at 16:02 user avatar terdon♦ $ echo "ary_compatible_idxary_idx_compatible" | sed 's|^\(.*\)\t\(.*\)|\1|' ary_compatible_idxary_idx_compatible ~ $ echo "ary_compatible_idx\x9ary_idx_compatible" | sed 's|^\(.*\)\t\(.*\)|\1|' ary_compatible_idx\x9ary_idx_compatible ~ $ echo "ary_compatible_idx\x9ary_idx_compatible" | sed 's|^\(.*\)\x9\(.*\)|\1|' ary_compatible_idx\x9ary_idx_compatible ~ $ echo "ary_compatible_idx\x9ary_idx_compatible" | sed 's|^\(.*\)\\x9\(.*\)|\1|' ary_compatible_idx $ echo "ary_compatible_idxary_idx_compatible" | sed 's|^\(.*\)\t\(.*\)|\1|' *********** #] 10Oct2019 leading spaces & tabs https://stackoverflow.com/questions/7314044/use-bash-to-read-line-by-line-and-keep-space IFS='' cat test.file | while read data do echo "$data" done I realize you might have simplified the example from something that really needed a pipeline, but before someone else says it: IFS='' while read data; do echo "$data" done < test.file answered Sep 6 '11 at 1:24 DigitalRoss *********** see "/media/bill/SWAPPER/bin/0_directory updates - bash & qnial.sh" process_dir() { # Create temporary [dir,subdirs] mkdir "$3" find "$1" -maxdepth 1 -type d | sed 's#'$1'##g' | xargs -Idiry mkdir "$3""diry" # list files find "$1" -maxdepth $2 -type f -name "$4" | sed 's#'$1'##g' >"$d_lst" # process files while read -u 9 line; do cat "$1""$line" | sed "$5" >"$3""$line" done 9< "$d_lst" } process_dir "$d_bin" 2 "$d_bin_tmp" "*.sh" "s#/PROJECTS/#/SWAPPER/#g" #process_dir "$d_ndf" 2 "$d_ndf_tmp" "*.ndf" "s#/PROJECTS/#/SWAPPER/#g;s#/media/bill/HOWELL_BASE/#/media/bill/SWAPPER/#g;s#/home/bill/Qnial/#/media/bill/SWAPPER/Qnial/#g;s#/home/bill/nial/#/media/bill/SWAPPER/Qnial/#g;s#/home/bill/Languages/#/media/bill/SWAPPER/Languages/#g;s#d_PROJECTS#d_SWAPPER#g" #process_dir "$d_qni" 2 "$d_qni_tmp" "*.ndf" "s#/PROJECTS/#/SWAPPER/#g;s#/home/bill/Qnial/#/media/bill/SWAPPER/Qnial/#g;s#d_PROJECTS#d_SWAPPER#g" +-----+ #] Multiple file reads (roster database-like) : IJCNN2019_INNS_IEEE_notification() { d_sftp="/media/bill/SWAPPER/2019 IJCNN Budapest/Publications/sftp access/" p_list_emails="$d_sftp""INNS and IEEE-CIS notification, list emails.txt" p_list_nameAffils="$d_sftp""INNS and IEEE-CIS notification, list nameAffils.txt" p_list_passwords="$d_sftp""INNS and IEEE-CIS notification, list pwds.txt" exec 7<"$p_list_nameAffils" exec 8<"$p_list_passwords" while read -u 9 z_email; do read -u 7 z_nameAffil read -u 8 z_password echo "z_email = $z_email" echo "z_nameAffil = $z_nameAffil" echo "z_password = $z_password" echo "" #email_compose_send done 9< "$p_list_emails" } +--+ #] from "$d_bin""email - remove emails from text file.sh" d_remove_emails() { if [ -d "$d_withEml" ]; then if [ -d "$d_noEmail" ]; then find "$d_withEml" -maxdepth 1 -type f -name "*" | sed "s#$d_withEml##" >"$d_temp""d_remove_emails files.txt" while read -u 9 line; do cat "$d_withEml""$line" | sed 's/[A-Za-z0-9._-]*@[A-Za-z0-9._-]*//g' >"$d_noEmail""$line" done 9< "$d_temp""d_remove_emails files.txt" else echo "directory doesnt exist : $d_noEmail" fi else echo "directory doesnt exist : $d_withEml" fi } +-----+ #] read tab-separated-columns of text file : process_one_of_several() { paperNumSearch="$1" let "i = 1" #] while IFS=$'\t' read -r paperNumber copyright authorTitle; do if [ "$paperNumSearch" == "$paperNumber" ]; then colour="red" d_paperDone="$d_paperProb" process_file colour="white" d_paperDone="$d_list""all - copyrighted/" process_file let "i += 1" break fi done < "$p_paperSpec" rm "$p_pdftk_logs" } ************************************************* **************************************** #] 05Aug2018 calculations example #!/bin/sh # /media/bill/SWAPPER/bin/dir_size sum, net.sh # www.BillHowell.ca 05Aug2018 initial calculations example see $ bash "/media/bill/SWAPPER/bin/dir_size sum, net.sh" d_data="/media/bill/SWAPPER/bin/" d_temp="/media/bill/ramdisk/" f_du_data="$d_data""dir_size sum, net test.txt" f_sizes="$d_temp""dir_size sizes.txt" cat "$f_du_data" | grep --only-matching "^[0-9]*" >"$f_sizes" exec 9<"$f_sizes" read -u 9 line let "total = line" let "residual = line" echo "total= $total" calculations() { while read -u 9 line; do let "residual = residual - line" echo "residual= $residual" done let "percent = residual / total * 100" } calculations echo "" echo "total(Mb)= $total, residual(Mb)= $residual, residual(%)= $percent" $ bash "/media/bill/SWAPPER/bin/dir_size sum, net.sh" total= 27714 residual= 25881 residual= 25872 residual= 24491 residual= 18671 residual= 18646 residual= 18550 residual= 18549 residual= 1363 residual= 839 total(Mb)= 27714, residual(Mb)= 839, residual(%)= 0 ************************ #] 20Jul2018 nested read within while loop https://stackoverflow.com/questions/18015137/linux-terminal-input-reading-user-input-from-terminal-truncating-lines-at-4095 #] simply echo followed by read +-----+ #] other cool examples https://unix.stackexchange.com/questions/423160/problem-using-read-command-within-while-read-loop #] To make the read in the loop body read from the terminal: exec 3<"somefile" while IFS= read -r words <&3; do read -r word1 word2 junk printf 'Got words = "%s"\n' "$words" printf 'Got word1 = "%s"\n' "$word1" printf 'Got word2 = "%s"\n' "$word2" done This makes the input to the first read come from the file somefile while the standard input stream of the second read will read from the terminal (or wherever the standard input stream of the script comes from). edited Feb 10 at 9:46 answered Feb 10 at 9:16 Kusalananda +-----+ #] https://stackoverflow.com/questions/9688371/how-to-read-from-user-within-while-loop-read-line #] You pipe data into your the while loops STDIN. So the read in get_ans is also taking data from that STDIN stream. You can pipe data into while on a different file descriptor to avoid the issue and stop bothering with temp files: while read -u 9 line; do NAME=$(get_ans Name "$line") done 9< list.txt get_ans() { local PROMPT=$1 DEFAULT=$2 ans read -p "$PROMPT [$DEFAULT]: " ans echo "${ans:-$DEFAULT}" } edited Mar 13 '12 at 17:09 answered Mar 13 '12 at 17:01 glenn jackman +-----+ #] My version in "/media/bill/PROJECTS/Qnial/MY_NDFS/symbols sort manual.sh" sort_manual_symbols() { rm "$f_tran" rm "$f_good" rm "$f_engl" rm "$f_eqtn" rm "$f_latr" rm "$f_ignr" while read -u 9 line; do echo "$line" read -p "[(t)ranslate,(g)ood,(e)nglish,equatio(n),(l)ater,(i)gnore] : " inn if [ "$inn" = t ];then echo >>"$f_tran" "\t""$line""\t\t\t""$line""\t\t\t""added $dater""\t\t\t" elif [ "$inn" = g ];then echo >>"$f_good" "$line" elif [ "$inn" = e ];then echo >>"$f_engl" "$line" elif [ "$inn" = n ];then echo >>"$f_eqtn" "$line" elif [ "$inn" = l ];then echo >>"$f_latr" "$line" elif [ "$inn" = i ];then echo >>"$f_ignr" "$line" else echo "Oops : error for $line" fi done 9<"$f_innn" } ***************************** #] 20Jul2018 IFS stands for "internal field separator" #] # https://unix.stackexchange.com/questions/423160/problem-using-read-command-within-while-read-loop IFS stands for "internal field separator". It is used by the shell to determine how to do word splitting, i. e. how to recognize word boundaries. Try this in a shell like bash (other shells may handle this differently, for example zsh): mystring="foo:bar baz rab" for word in $mystring; do echo "Word: $word" done The default value for IFS consists of whitespace characters (to be precise: space, tab and newline). Each character can be a word boundary. So, with the default value of IFS, the loop above will print: Word: foo:bar Word: baz Word: rab In other words, the shell thinks that whitespace is a word boundary. Now, try setting IFS=: before executing the loop. This time, the result is: Word: foo Word: bar baz rab Now, the shell splits mystring into words as well -- but now, it only treats a colon as the word boundary. The first character of IFS is special: It is used to delimit words in the output when using the special $* variable (example taken from the Advanced Bash Scripting Guide, where you can also find more information on special variables like that one): ************************ #] 17Jul2018 see "/media/bill/PROJECTS/Qnial/MY_NDFS/symbols grep count.s" grep_symbols() { echo >"$fot2" "" echo >"$fot2" "+----------+" echo >"$fot2" "grep_symbols() : $dater" rm "$fot1" while read -r line; do grep -cF "$line" "$finn" >>"$fot1" done < "$finn" paste "$fot1" "$finn" >"$fot2" } in grep : -c, --count Suppress normal output; instead print a count of matching lines for each input file. With the -v, --invert-match option (see below), count non-matching lines. (-c is specified by POSIX.) -F, --fixed-strings Interpret PATTERN as a list of fixed strings (rather than regular expressions), separated by newlines, any of which is to be matched. (-F is specified by POSIX.) paste merges files line-by-line ************************ #] 31May2018 file read - conditional /media/bill/PROJECTS/Qnial/code develop_test/test_summary_all.sh summary_write() { in_summary=0 echo >>"$p_log" " " echo >>"$p_log" "************************************* " echo >>"$p_log" "$1" while read -u 99 -r line; do if [ "$in_summary" == 1 ];then echo "$line" >>"$p_log"; fi; if [ "$line" == "Summary of test results :" ];then in_summary=1; fi; done 99< "$1" in_summary=0 } Doesn't work!! -> probably due to "************************************* " summary_write() { while read -u 3 -r line; do if [ "$line" == "************************************* " ];then in_summary=1; fi if [ "$in_summary" = 1 ];then echo "$line" >>"$p_log"; fi done 3< "$1" } ************************ #] 31May2018 Howell - author provides MANY interesting cases! https://stackoverflow.com/questions/1521462/looping-through-the-content-of-a-file-in-bash Howell - author provides MANY interesting cases! +-----+ One way to do it is: while read p; do echo $p done