#] 
#] *********************
#] "$d_bin"'fileops notes.txt'  
# www.BillHowell.ca  26Feb2023 initial, took older notes from 
#	"$d_SysMaint"'webSite/0_website bash notes.txt'  - etc
# view in text editor, using constant-width font (eg courier), tabWidth = 3

#48************************************************48




#24************************24
# Table of Contents, generate with :
# $ grep  "^#]"  "$d_bin"'fileops notes.txt' |  sed "s/^#\]/  /" 
#
   *********************
   "$d_bin"'fileops notes.txt'  
   16Mar2024 povrL_pRgxP_replace(): in future, add options to pth file
   15Mar2024 povrL_pRgxP_replace(): revamp  arrays nested- test ndfs
   15Mar2024 check  rm [povr_pStrP_replace, povr_pRgxP_replace, povrL_rgxP_replace]
   27Feb2024 change povrL_pStrP_replace because of bookmarks:  '^#] ' comment, not `#
   26Feb2024 fileops.sh add pth_get_ext()  - get extension without fname from pth
   24Feb2024 'pStrPAll_L change.txt' - are [^#, whitespace] lines ignred??
   16Feb2024 post changes, run dWeb_get_pHtmWebYesNonL
   14Feb2024 fileops.sh: rm function pHtml_extract_pTOC(pHtml pTblOfCont) 
   13Feb2024 correct [set, use] * [dir, pth] names
   05Feb2024 Segmentation fault - must be new code
   20Jan2024 
   19Jan2024 dWeb_get_pWebPageL
   12Nov2023 fix pOvrL_pStrP_pInsert_replace()  ALL webPages
   11Nov2023 fix pOvrL_pStrP_pInsert_replace() test[1, 5]
   30Sep2023 z_historic 'webSite update.sh'
   24Sep2023 webSite_getCheck_internalLinks() - remove sedExpr
   17Sep2023 create initial povrL_pTypRgxT_replace family of functions
   14Sep2023 now fixed: some are wrong!!  after combining [bolPovEs, bolPstrEs] -> bolChrCd
   14Sep2023 pOvrL_pStrP_replace - should either [en, de]code [neither, both] [pOvr, pStrP]
   14Sep2023 now fixed: some are wrong!! after combining [bolPovEs, bolPstrEs] -> bolChrCd
   14Sep2023 pOvrL_pStrP_replace - should either [en, de]code [neither, both] [pOvr, pStrP]
   +-----+
   see "$d_bin"'0_test/fileops/povrL_pStrP_replace/0_povrL_pStrP_replace_test notes.txt'
   see "$d_bin"'0_test/fileops/povr_strP_replace/0_povr_strP_replace_test notes.txt'
   31Jul2023 strSearch_strP_replace_test() - many examples
   31Jul2023 pStrP_sedEsc_test()
   31Jul2023 [str, povr] replace [strP, pStrP] in path[,L]  - collection of functions
   povr_sedEsc()  - escape ALL OR[sedEscSearch, sedEscApo] in povr
   28Jul2023 create fileops.sh sedEscape()
   +-----+
   16Mar2023 readarray -t fndL611 <"$pinn", want a txtL from readarray - NOT an evaluation 
   15Mar2023 
   15Mar2023 pinn_getLinseq_povr()  - extract a sequence of lines between [strt, endd] markers in pTxt
   15Mar2023 pinn_sedFndGetPut_pout()
   14Mar2023 povr_sedEach_sedAll()  - transforms matching sequences of lines 
   01Mar2023 clean up file_ops.sh: povr_strP_replace() etc, tests for wrd
   povr_strP_replace  - backs up povr, replaces strOld with strNew, backup flag 
   03Mar2023 develop "$d_bin"'port bash to python.sh'
   02Mar2023 cp file_ops.sh to python, same function names
   01Mar2023 more dirChanges, to make pths shorter
   01Mar2023 search 'bash read syntax error near unexpected token `newline'
   01Mar2023 pinn_pOld_pNew_replace() development, fixes
   01Mar2023 search 'Linux bash read and what is the maximum file indicator?' 1024
   01Mar2023 search 'Linux bash and how do I assign the lines of a file to an array?' readarray
   17Feb2023 pinn_strOld_strNew_replace - test on webwork files, use IFS='' before read!!
   16Feb2023 fileops.sh Library: key issue is practical use of library
   16Feb2023 fileops.sh Library: test str_inWebPages_pout() in "$d_bin"'fileops.sh'
   16Feb2023 search 'bash script to change strings in multiple files'


#24************************24
# Setup, ToDos,   

01Mar2023 
	[test, fix] code to replace Menus - pinnL_pStrPL_replace()
	create code to :
		insert pth in : '<!--  [#!:/home/bill/web/fname:!#]  -->'
	cp Howell's *.html to "$d_webWork"'webPagesLocal/', replacing URL with pth in 1st line
	test use of "$d_webWork"'webPagesLocal/'
08Mar2023 def regexstr_tests iterative problem : incOne_create_seq( 0 1 c=0; c< )  
	should be ( 0 1 c ), 2 other ';for's work well...
14Sep2023 pOvrL_pStrPL_replace - should either [en, de]code [neither, both] [pOvr, pStrP]
28Sep2023 can I get ?I forget - in notes of something?
16Mar2024 povrL_pRgxP_replace(): in future, add options to pth file




#08********08
#] ??Mar2024 




#08********08
#] ??Mar2024 




#08********08
#] ??Mar2024 




#08********08
#] ??Mar2024 




#08********08
#] ??Mar2024 




#08********08
#] ??Mar2024 




#08********08
#] ??Mar2024 




#08********08
#] ??Mar2024 




#08********08
#] ??Mar2024 




#08********08
#] 16Mar2024 povrL_pRgxP_replace(): in future, add options to pth file




#08********08
#] 15Mar2024 povrL_pRgxP_replace(): revamp  arrays nested- test ndfs
revamping : 
/home/bill/web/Qnial/code develop_test/arrays nested/arrays nested- data templates.ndf
/home/bill/web/Qnial/code develop_test/arrays nested/arrays nested- testGroupsCluster.ndf
/home/bill/web/Qnial/code develop_test/arrays nested/arrays nested- testOptrs.ndf

14:38$ bash  "$d_bin"'fileops run.sh'  
~
>> no changes!!???
oops, I used povrL_pStrP_replace
15:05$ bash  "$d_bin"'fileops run.sh'  
~
>> still no changes
15:23$ bash  "$d_bin"'fileops run.sh'  
/home/bill/web/Qnial/code develop_test/arrays nested/arrays nested- data templates.ndf
sed: -e expression #1, char 35: invalid reference \1 on `s' command's RHS
/home/bill/web/Qnial/code develop_test/arrays nested/arrays nested- testGroupsCluster.ndf
sed: -e expression #1, char 35: invalid reference \1 on `s' command's RHS
/home/bill/web/Qnial/code develop_test/arrays nested/arrays nested- testOptrs.ndf
sed: -e expression #1, char 35: invalid reference \1 on `s' command's RHS
~

Hah! was missing `\ in front of [()] - bash not geany
have to replace ndfs from archive
15:23$ bash  "$d_bin"'fileops run.sh'  
/home/bill/web/Qnial/code develop_test/arrays nested/arrays nested- data templates.ndf
sed: can't read : No such file or directory
/home/bill/web/Qnial/code develop_test/arrays nested/arrays nested- testGroupsCluster.ndf
sed: can't read : No such file or directory
/home/bill/web/Qnial/code develop_test/arrays nested/arrays nested- testOptrs.ndf
sed: can't read : No such file or directory
~
>> files were wiped out again!??

I put archive in povrL_pRgxP_replace, then call povr_rgxP_replace no archive : 
			if	[ -s "$povr982" ]; then 
				if	[ 1 -eq  "$bolArXiv982" ]; then
					pinn_archiveLocal  "$povr982" 
				fi
				while IFS='' read -r -u 983 rgxP982; do 
					povr_rgxP_replace  0  "$povr982"  "$rgxP982"
				done 983< "$pRgxPTmp982"

change : 
			sed  "s|$rgxOld448|$rgxNew448|g" "$pov1Tmp448"  >"$povrTmp448" 
to :
			sed  "s|$rgxOld448|$rgxNew448|g" "$povr____448"  >"$povrTmp448" 

15:53$ bash  "$d_bin"'fileops run.sh'  
/home/bill/web/Qnial/code develop_test/arrays nested/arrays nested- data templates.ndf
/home/bill/web/Qnial/code develop_test/arrays nested/arrays nested- testGroupsCluster.ndf
/home/bill/web/Qnial/code develop_test/arrays nested/arrays nested- testOptrs.ndf
~
>> now it runs without rm file contents
>> but does it work? checks : 
tmplt	grp	ops	
y						idx\([LA]*\)	ndx\1
n				n		ary\([LA]*\)	nry\1			often ary without [LA]
				n		slc\([LA]*\)	slcN\1
				n		ary\([A-Za-z]*\)_	nry\1_
				n		_slc\([LA_]*\)_	_slcN\1_
				?		_replace\([A-Za-z0-9]*\)_	_put\1_
>> none of these worked!!!!?

"$d_temp"'povrL_pRgxP_replace pRgxPTmp.txt' only has : 
_replace\([A-Za-z0-9]*\)_	_put\1_
>> oops! only one showing : 
_replace\([A-Za-z0-9]*\)_	_put\1_

move out of while loop!
		if  [ -f  "$pRgxPTmp982" ]; then
			rm  "$pRgxPTmp982"
		fi

16:26$ bash  "$d_bin"'fileops run.sh'  
/home/bill/web/Qnial/code develop_test/arrays nested/arrays nested- data templates.ndf
/home/bill/web/Qnial/code develop_test/arrays nested/arrays nested- testGroupsCluster.ndf
/home/bill/web/Qnial/code develop_test/arrays nested/arrays nested- testOptrs.ndf
~
tmplt	grp	ops	
						idx\([LA]*\)	ndx\1
						ary\([LA]*\)	nry\1			often ary without [LA]
						slc\([LA]*\)	slcN\1
						ary\([A-Za-z]*\)_	nry\1_
						_slc\([LA_]*\)_	_slcN\1_
						_replace\([A-Za-z0-9]*\)_	_put\1_

OK examples : 
nry_
_ndx_
_slcN_
>> actually, it looks like it mostly worked!

16:31 take a break, I'm tired


#08********08
#] 15Mar2024 check  rm [povr_pStrP_replace, povr_pRgxP_replace, povrL_rgxP_replace]
will no longer use! (error prone, excess [code, opNam])
I updated povrL_pRgxP_replace

+-----+
olde code

#  povr_pStrP_replace(bolArXiv bolChrCd povr pStrP) sed replace encoded (pStrP, povr)
#     pStrP : put dir changes AFTER pth changes - safer, [no-tab, empty] pStrP lines are OK
# example see : "$d_bin"'0_test/fileops test.sh' 
# tightly tied to function called, povr_strP_replace : easy to make symbol errors!!
# www.BillHowell.ca  01Jun2021 intitial, 03Mar2023 unfinished, 07Aug2023 revamped
# 16Sep2023 handle [no-tab, empty lines in pStrP] 

	povr_pStrP_replace()
{  
	bolArXiv607="$1"		# 1= archive povr
	bolChrCd607="$2"		# 1= escape (code) human <--> sed search-replace
	povr____607="$3"		# file to overwrite after sed search-replace 
	pStrP___607="$4"		# str[Old, New] search-replace pairs
	#	echo  >>"$pLog607"  "povr  = $povr____607" 
	#	echo  >>"$pLog607"  "pStrP = $pStrP___607" 

	povr_tmp607="$d_temp"'povr_pStrP_replace povr temp.txt'
	pStrPtmp607="$d_temp"'povr_pStrP_replace pStrP temp.txt'

	if		[[    -s  "$povr____607"  &&  -s "$pStrP___607" ]]; then 
		if	[ 1  -eq  "$bolArXiv607" ]; then
			pinn_archiveLocal			"$povr____607" 
		fi
		if [ 1  -eq  "$bolChrCd607" ]; then
			pHum_sed_pCde	"$povr____607"		"$povr_tmp607"
			pHum_sed_pCde	"$pStrP___607"		"$pStrPtmp607"
		else
			cp -p				"$povr____607"		"$povr_tmp607"
			cp					"$pStrP___607"		"$pStrPtmp607"
		fi

		while IFS='' read -r -u 607 strP607; do 
			povr_strP_replace_run  "$povr_tmp607"  "$strP607"
		done 607<  "$pStrPtmp607"  

		if [ 1 -eq "$bolChrCd607" ]; then
			pCde_sed_pHum	"$povr_tmp607"	"$povr____607"
		else
			
			mv					"$povr_tmp607"	"$povr____607"
		fi
	else 
		date_ymdhms=$(date +"%0y%0m%0d %0kh%0Mm%0Ss")
		echo  >>"$pLog384"  "$date_ymdhms povr_pStrP_replace error: OR[povr, pStrPL] doesnt exist :"
		echo  >>"$pLog384"  "povr  =  $povr____384"
		echo  >>"$pLog384"  "pStrP =  $pStrP___384"
		echo  >>"$pLog384"  ''  
	fi
}  


#  povrL_strP_replace(bolArXiv bolChrCd povrL strP) sed replace encoded (pStr, povrL) 
#     NO LONGER available - dangerous mixups, just use povrL_pStrP_replace


#  povr_pRgxP_replace(bolArXiv bolChrCd povr pRgxP) sed replace encoded (pRgxP, povr)
# www.BillHowell.ca 17Sep2023 initial copy of strP form, some adjustments

	povr_pRgxP_replace()
{  
	bolArXiv457="$1"		# 1= archive povr
	povr____457="$2"		# file to overwrite after sed search-replace 
	pRgxP___457="$3"		# str[Old, New] search-replace pairs
	#	echo  >>"$pLog982"  "povr  = $povr____457" 
	#	echo  >>"$pLog982"  "pRgxP = $pRgxP___457" 
	#pLog982 = 'povrL_pRgxP_replace funcFamily log.txt', defined in Setup section above 

	if		[[ -s  "$povr____457"  &&  -s  "$pRgxP___457" ]]; then 
		if	[ 1  -eq  "$bolArXiv457" ]; then
			pinn_archiveLocal	"$povr____457" 
		fi

		while IFS='' read -r -u 457 rgxP457; do 
			povr_rgxP_replace  0  "$povr____457"  "$rgxP457"
		done 457<  "$pRgxP___457"  

	else 
		date_ymdhms=$(date +"%0y%0m%0d %0kh%0Mm%0Ss")
		echo  >>"$pLog982"  "$date_ymdhms povr_pRgxP_replace error: OR[povr, pRgxPL] doesnt exist :"
		echo  >>"$pLog982"  "povr  =  $povr____457"
		echo  >>"$pLog982"  "pRgxP =  $pRgxP___457"
		echo  >>"$pLog982"  ''  
	fi
}  


#  povrL_rgxP_replace(bolArXiv bolChrCd povrL rgxP) sed replace encoded (pStr, povrL) UNTESTED!
#     [all povrL, rgxP] are escaped if p[ov, st]Esc961 == 1 
#     z_Archive each povrL (file listed), rgxP=(strOld, strNew) 
# tightly tied to function called, povr_rgxP_replace : easy to make symbol errors!!
# www.BillHowell.ca 17Sep2023 initial copy of strP form, some adjustments

	povrL_rgxP_replace()
{  
	bolArXiv961="$1"
	povrL___961="$2"
	rgxP____961="$3"
	#pLog982 = 'povrL_pRgxP_replace funcFamily log.txt', defined in Setup section above 

	if		[ -s  "$povrL961" ]; then 
		while  IFS=''  read -u 961 povr961; do 
			povr_rgxP_replace  "$bolArXiv961"  "$povr961"  "$rgxP____961"
		done 961<"$povrL___961"
	else 
		date_ymdhms=$(date +"%0y%0m%0d %0kh%0Mm%0Ss")
		echo  >>"$pLog982"  "$date_ymdhms povrL_rgxP_replace error: povrL doesnt exist : "
		echo  >>"$pLog982"  "povrL :  $povrL___961"
		echo  >>"$pLog982"  ''  
	fi
}  




#08********08
#] 27Feb2024 change povrL_pStrP_replace because of bookmarks:  '^#] ' comment, not `#
change: 
					chrOne=0
					chrOne=$(  echo  "$strP384"  |  cut  --bytes=1  )
					if  ! [ '#] '  ==  "$chrOne"  ]; then
to:
					commentChrs=0
					commentChrs=$(  echo  "$strP384"  |  cut  --bytes=1,2,3  )
					if  ! [ '#] '  ==  "$commentChrs"  ]; then

test : 
14:25$ strP384='#] 27Feb2024 OOPS!!! ^# makes function ignore line - changes wont be made'
~
14:25$ echo  "$strP384"  |  cut  --bytes=1,2,3
~
14:25$ strP384='# 27Feb2024 OOPS!!! ^# makes function ignore line - changes wont be made'
~
14:26$ echo  "$strP384"  |  cut  --bytes=1,2,3
# 2
~
>> OK, works


#08********08
#] 26Feb2024 fileops.sh add pth_get_ext()  - get extension without fname from pth

+-----+
olde code 

in [fileops.sh - none, fileops run.sh - none, 0_tests/fileops test.sh - none], 
	search  pth_get_fNoExt
	replace pth_get_fnmNoExt

#  pth_get_fNoExt(pth)  - get fNam without extension (NOT pth!)
# 23Sep2023 initial

	pth_get_fNoExt()
{  
	path551="$1"
	fNam551=$( pth_get_fnam  "$path551" )
	bolPth551=$(  echo  "$fNam551"  |  grep  "\/"  )
	if  [[ -z  "$bolPth551" ]]; then 
		fNoExt551=$(  echo  "$fNam551"  |  sed  's|\(.*\)\..*|\1|'  )
	else
		fNoExt551="fNam_get_fNoExt error, isPth : $fNam551"
	fi
	echo "$fNoExt551"
}



#08********08
#] 24Feb2024 'pStrPAll_L change.txt' - are [^#, whitespace] lines ignred??
				while IFS='' read -r -u 385 strP384; do 
					# screen out [comment, empty] lines 
					# tests if strP has ONE tab - 11Oct2023 needs fix : grep -c -o  'ztbz' or whatever
					#bolTabInStr=$( echo  "$strP384"  |  sed  's|\x9|ztbz|'  |  grep -c -o  'ztbz' )
					nTab=$( echo  "$strP384"  |  tr  \\t  \\n  |  wc -l ) 
					nTab=$(( $nTab - 1 ))
					if		[[ 1 = "$nTab" ]]; then  
						# both of the following are in Cde format (tab=\x9 is NOT translated!)
						 strFnd384=$( echo  "$strP384"  |  sed 's|\x9.*||' )
						nStrFnd384=$( grep -c  "$strFnd384"  "$povr_tmp384" ) 
						if		[[ 0 < "$nStrFnd384" ]]; then  
							echo  >>"$pLog384"  "$nStrFnd384 :  $strP384"
							povr_strP_replace_run  "$povr_tmp384"  "$strP384"
						fi 
					fi
				done 385< "$pStrPtmp384"
>> whitespace is taken care of
>> ^# is NOT, so add that 

changed to : 
				while IFS='' read -r -u 385 strP384; do 
					# screen out [comment, empty] lines 
					# tests if strP has ONE tab - 11Oct2023 needs fix : grep -c -o  'ztbz' or whatever
					# also ignores lines without a tab
					#bolTabInStr=$( echo  "$strP384"  |  sed  's|\x9|ztbz|'  |  grep -c -o  'ztbz' )
					nTab=$( echo  "$strP384"  |  tr  \\t  \\n  |  wc -l ) 
					nTab=$(( $nTab - 1 ))
					chrOne=$(  echo  "$strP384"  |  cut  --bytes=1  )
					if  [ '#'  ==  "$chrOne"  ]; then
						if		[[ 1 = "$nTab" ]]; then  
							# both of the following are in Cde format (tab=\x9 is NOT translated!)
							strFnd384=$( echo  "$strP384"  |  sed 's|\x9.*||' )
							nStrFnd384=$( grep -c  "$strFnd384"  "$povr_tmp384" ) 
							if		[[ 0 < "$nStrFnd384" ]]; then  
								echo  >>"$pLog384"  "$nStrFnd384 :  $strP384"
								povr_strP_replace_run  "$povr_tmp384"  "$strP384"
							fi 
						fi
					fi  
				done 385< "$pStrPtmp384"

+-----+
olde code

dangerous - just remove from fileops.sh!! : 

#  povrL_strP_replace(bolArXiv bolChrCd povrL strP) sed replace encoded (pStr, povrL) UNTESTED!
#     [all povrL, strP] are escaped if p[ov, st]Esc961 == 1 
#     z_Archive each povrL (file listed), strP=(strOld, strNew) 
# tightly tied to function called, povr_strP_replace : easy to make symbol errors!!
# www.BillHowell.ca  02Jun2021 intitial, 05Aug2023 upgrade with chrEncoding
# 14Sep2023 inefficient but simple: povrL_pStr[P]_replace keep encoding pStrP with each pOvr
# 16Sep2023 handle [no-tab, empty lines in pStrP] 
# 10Oct2023 pLog384="$d_bin"'bin logs/povrL_pStrP_replace funcFamily pLog.txt'


	povrL_strP_replace()
{
	date_ymdhms=$(date +"%0y%0m%0d %0kh%0Mm%0Ss")
	echo  >>"$p_log" "$date_ymdhms povrL_strP_replace"
  
	bolArXiv961="$1"
	bolChrCd961="$2" 
	povrL___961="$3"
	strP____961="$4"
#	pLog384 ='povrL_pStrP_replace funcFamily pLog.txt', defined in Setup section above  

# 11Oct2023 missing [check, code]s of [povr, strP, etc] 

	if		[ -s  "$povrL961" ]; then 
		while  IFS=''  read -u 961 povr961; do 
			povr_strP_replace_run  "$povr961"  "$strP____961"
		done 961<"$povrL___961"
	else 
		date_ymdhms=$(date +"%0y%0m%0d %0kh%0Mm%0Ss")
		echo  >>"$pLog384"  "$date_ymdhms povrL_strP_replace error: povrL doesnt exist : "
		echo  >>"$pLog384"  "povrL :  $povrL___961"
		echo  >>"$pLog384"  ''  
	fi
}  




#08********08
#] 16Feb2024 post changes, run dWeb_get_pHtmWebYesNonL

15:20$ bash  "$d_bin"'webSite update local.sh' 
find all html files
/home/bill/web/bin/fileops.sh: line 2914: : No such file or directory
extract webPages from htmlL: they have strTst
/home/bill/web/bin/fileops.sh: line 1325: : No such file or directory
diff: pth_get_dArchive error, OR[no-exist, isP] pth_get_dir error, OR[no-exist, isDir] :  /home/bill/web/webWork/pWebYesL.txtfNam_get_fNoExt error, isPth : pth_get_fnam error, OR[no-exist, isDir] /home/bill/web/webWork/pWebYesL.txt archive.fNam_get_ext error, isPth : pth_get_fnam error, OR[no-exist, isDir] /home/bill/web/webWork/pWebYesL.txt: No such file or directory
diff: /home/bill/web/webWork/pWebYesL.txt: No such file or directory
diff: pth_get_dArchive error, OR[no-exist, isP] pth_get_dir error, OR[no-exist, isDir] :  /home/bill/web/webWork/pWebNo_L.txtfNam_get_fNoExt error, isPth : pth_get_fnam error, OR[no-exist, isDir] /home/bill/web/webWork/pWebNo_L.txt archive.fNam_get_ext error, isPth : pth_get_fnam error, OR[no-exist, isDir] /home/bill/web/webWork/pWebNo_L.txt: No such file or directory
diff: /home/bill/web/webWork/pWebNo_L.txt: No such file or directory
grep: /home/bill/web/webWork/pWebNo_L.txt: No such file or directory
~

pinnL_lineIdx_strTst_split_pYes_pNo() : test
# 16Feb2024 test, see 'fileops notes.txt' 
#    20:39$ line333=$( sed -n '1{p;q}' "$pinn333" )
#	pinn333="$d_web"'home.html'
#	strTst_333='<!--  /home/bill/web/'
#	linIdx_333=1
#	strIdx333="$linIdx_333"'{p;q}'
#			l1ne333=$( sed -n "$strIdx333" "$pinn333" )
#			line333=$( echo  "$l1ne333"   |  cut -c 1-"$strTst_len333" )


#  pinnL_getBadLine1_poutL(pinnL poutL)  - 
#     list files with only one space 1st line '<!-- /home/bill/howell'

17:22$ bash  "$d_bin"'webSite update local.sh'  
/home/bill/web/Personal/240101 TimeLog of activities during 2023.html
/home/bill/web/bin/fileops.sh: line 1399: warning: command substitution: ignored null byte in input
/home/bill/web/bin/fileops.sh: line 1399: warning: command substitution: ignored null byte in input
/home/bill/web/webOther/Steven H Yaskell/240125 emto Steven Yaskell: reification of historical weather data from ancient sources.html
/home/bill/web/webOther/Steven H Yaskell/240125 emto Steven Yaskell: reification of historical weather data from ancient sources, URL.html

problem what that exclude file MIST hacve at least one entry, or nothing passes --invert-match

#08********08
#] 14Feb2024 fileops.sh: rm function pHtml_extract_pTOC(pHtml pTblOfCont) 
this is redundant, not so good as 
	htmlHeadings_to_TblOfContents(pHtml) - create a TblOfContents from headings in html file

#  pHtml_extract_pTOC(pHtml pTblOfCont) - create a TblOfContents from headings in html file
# 18Oct2023 initial from htmlHeadings_to_TblOfContents, added pTblOfCont


+-----+
olde code 

	pHtml_extract_pTOC()
{
	date_ymdhms=$(date +"%0y%0m%0d %0kh%0Mm%0Ss")
	echo  >>"$p_log" "$date_ymdhms htmlHeadings_to_TblOfContents"

	pHtml="$1"
	pTblOfCont="$2"
	grep  '<A id='  "$pHtml"  |  sed  's|<A id=["]\(.*\)["]>.*|<LI><A HREF="#\1">\n\t\t\t\t  \1</a>|'  >"$pTblOfCont"
}  


#08********08
#] 13Feb2024 correct [set, use] * [dir, pth] names
for current list, see :
	"$d_bin"'fileops header.sh'


#08********08
#] 05Feb2024 Segmentation fault - must be new code

comment-out transcript_unite_each_paragraph()
bash  "$d_bin"'fileops run.sh'  with pth_get_dir_run active
14:37$ bash  "$d_bin"'fileops run.sh'  
/home/bill/web/Qnial/MY_NDFS/
>> OK, no segmentation fault now

pth_get_dArchive_run : 
14:47$ bash  "$d_bin"'fileops run.sh'  
/home/bill/web/Qnial/MY_NDFS/z_Archive/
>> OK

what is problem with transcript_unite_each_paragraph()???
comment-out :
	# proceed to start of transcript, writing lines along way 
	exec 681<"$pinn681"
	while IFS='' read -u 681 line681; do
		if		[ 'Transcript'  ==  "$line681" ]; then  break;  fi
	done  

	# consolidate each paragraph, remove [timestamp, newLine]s lines along way 
	# notice that reads continue from line above 
	bolInn=0
	
	while IFS='' read -u 681 line681; do
		if		[ ''  ==  "$line681" ]; then 
			if  [ "$bolInn" == 1 ]; then
				echo  ' '  >>"$pout681"
				# save paragraph
				cat  "$ptmp681"  |  tr  \\n  ' '  |  sed  's| \([0-9]*:[0-9]*\) | |g'  >>"$pout681"
				rm   "$ptmp681"
				bolInn=0
				echo  ' '  >>"$pout681"
			fi
		else 
			echo  "$line681"  >>"$ptmp681"
			bolInn=1
		fi
	done

14:49$ bash  "$d_bin"'fileops run.sh'  
/home/bill/web/Qnial/MY_NDFS/z_Archive/
>> OK, no segmentation fault

maybe I need another exec 681<"$pinn681"??
uncomment 1st use 
14:52$ bash  "$d_bin"'fileops run.sh'  
/home/bill/web/Qnial/MY_NDFS/z_Archive/
>> OK, works

put in another exec 681<"$pinn681"
14:56$ bash  "$d_bin"'fileops run.sh'  
/home/bill/web/Qnial/MY_NDFS/z_Archive/
>> works!!!


#08********08
#] 20Jan2024 
create new 
	pinn_archiveLocal()
	pth_getLocal_pArchive()
modifications
	pinnL_idx_strTst_split_pYes_pNo()
	dWeb_get_pWebPageL()
must revamp webSite funcs due to pth changes in above files

NUTS!! - I forgot to :
	grep  --invert-match  -i  "z_Old\|z_Archive"  
	avoid current exclude files??!! maybe 'pHtmlPathWebPage no.txt' ??
		nyet! 'pHtmlPathWebPage no.txt' IS the exclude list?

change : 
	find  "$d_web" -type f  -name "*.html"								|  sort -u  >"$pAllL__769"
to : 
	find  "$d_web" -type f  -name "*.html"  |  grep  --invert-match -i  "z_Old\|z_Archive"  |  sort -u  >"$pAllL__769"

first, redo with new find, then work more
pinnL_idx_strTst_split_pYes_pNo() problem : 
		line333=$( sed -n '1{p;q}' "$pinn333" )
		line333=$( echo  "$line333"   |  cut -c 1-"$strTst_len333" )
>> 1st extracts line 1? 2nd cuts line to do test? 
+-----+
20:27$ pinn333="$d_web"'home.html' 
~
20:39$ echo "$pinn333"
/home/bill/web/home.html
~
20:39$ line333=$( sed -n '1{p;q}' "$pinn333" )
~
20:40$ echo "$line333"
<!--  /home/bill/web/home.html  -->
~
20:40$ line333=$( echo  "$line333"   |  cut -c 1-"$strTst_len333" )
~
20:40$ echo "$line333"
<!--  /home/bill/web/home.html  -->
~
+-----+
>> 1st command OK, 2nd fails!

+-----+
20:44$ echo "$strTst_769"
<!--  /home/bill/web/
~
20:45$ strTst333="$strTst_769"
~
20:46$ strTst_len333=${#strTst333}
~
20:46$ echo "$strTst_len333"
21
~
+-----+
20:46$ line333=$( echo  "$line333"   |  cut -c 1-"$strTst_len333" )
~
20:47$ echo "$line333"
<!--  /home/bill/web/
~
+-----+
>> OK!!!

not working : 
	grep  --invert-match  -i  "z_Old\|z_Archive"  
try 
		find  "$d_web" -type f  -name "*.html" grep  --invert-match  -i  "z_Old"  |  grep  --invert-match  -i  "z_Archive"  |  grep  --invert-match  -i  "$z_history"  |  sort -u  >"$pAllL__769"

20:54$ bash "$d_bin"'webSite update local.sh' 

find all html files
find: paths must precede expression: `grep'
extract webPages from htmlL: they have strTst769
~
>> ran fast, but didn't work at all!?!?!?
oops - missing a pipe
pHtmlPathAll_L.txt has NO content!

try : 
	find  "$d_web" -type f  -name "*.html"  |  grep  -i  --invert-match  "z_Old"  |  grep  -i  --invert-match  "z_Archive"  |  grep  -i  --invert-match  "$z_history"  |  sort -u  >"$pAllL__769"
>> still nothing in pHtmlPathAll_L.txt
ouch! errant $
grep  -i  --invert-match  "$z_history"

21:58$ bash "$d_bin"'webSite update local.sh' 
find all html files
extract webPages from htmlL: they have strTst769
/home/bill/web/bin/fileops.sh: line 1374: warning: command substitution: ignored null byte in input
/home/bill/web/bin/fileops.sh: line 1374: warning: command substitution: ignored null byte in input
~
>> 'pHtmlPathWebPage no.txt' has ALL files?

try line 1374 :
		l1ne333=$( sed '1p;q' "$pinn333" )
21:58$ bash "$d_bin"'webSite update local.sh' 
find all html files
extract webPages from htmlL: they have strTst769
/home/bill/web/bin/fileops.sh: line 1374: warning: command substitution: ignored null byte in input
/home/bill/web/bin/fileops.sh: line 1374: warning: command substitution: ignored null byte in input
~
>> nyet

see "$d_SysMaint"'Linux/sed notes.txt'  19Sep2023 

same problem - maybe I have to check for null files?
change to : 
	while IFS='' read -u 333 pinn333; do
		if  [ -s  "$pinn333" ]; then 
			# right now n fixed = 1, later make = idx
			l1ne333=$( sed '1p;q' "$pinn333" )
			line333=$( echo  "$line333"   |  cut -c 1-"$strTst_len333" )
			#	echo  "l1ne333 = $l1ne333,  line333 = $line333"
			if		 [ "$line333" == "$strTst333" ]; then 
				echo  "$pinn333"  >>"$pYes__333"
			else
				echo  "$pinn333"  >>"$pNo___333"
			fi
		fi
	done 333<"$pinnL_333" 
22:07$ bash "$d_bin"'webSite update local.sh' 
find all html files
extract webPages from htmlL: they have strTst769
/home/bill/web/bin/fileops.sh: line 1375: warning: command substitution: ignored null byte in input
/home/bill/web/bin/fileops.sh: line 1375: warning: command substitution: ignored null byte in input
~
>> nyet

pHtmlPathAll_L.txt look OK
pHtmlPathWebPage [yes, no].txt are FAR larger!?!?!

SHIT!!
change to : 
	if  [ -f "$pYes__333" ]; then
		pinn_archiveLocal	"$pYes__333"
		rm						"$pYes__333"
	fi
	if  [ -f "$pNo___333" ]; then
		pinn_archiveLocal	"$pNo___333"
		rm						"$pNo___333"
	fi
	if  [ -f "$pDifYs333" ]; then
		pinn_archiveLocal	"$pDifYs333"
		rm						"$pDifYs333"
	fi
	if  [ -f "$pDifNo333" ]; then
		pinn_archiveLocal	"$pDifNo333"
		rm						"$pDifNo333"
	fi

BINGO!
	pHtmlPathWebPage dif[Ys, No].txt are both empty (matches)
gives 270 webPages




#08********08
#] 19Jan2024 dWeb_get_pWebPageL
this looks wrong : 
	find  "$d_web" -type f  -name "*.html"  |  grep --file="$pExclL769"  --invert-match  |  sort -u  >"$pWebL_769"  
>> ?? grep --file="$pExclL769

20:50$ bash  "$d_bin"'webSite update local.sh'  
+-----+

find all html files
grep: /home/bill/web/webWork/pHtmlPathExclL webPage.txt:288: Invalid range end
grep: /home/bill/web/webWork/pHtmlPathExclL webPage.txt:328: Invalid range end
...
+-----+
>> not working: bunches too many terms?
	grep --file="$pExclL769"  --invert-match

change : 
	find  "$d_web" -type f  -name "*.html"  |  grep --file="$pExclL769"  --invert-match  |  sort -u  >"$pWebL_769"  
to : 
	find  "$d_web" -type f  -name "*.html"  |  sort -u  >"$pWebL_769"  
>> OK! works now?

uncomment : 
	#echo  'extract webPages from htmlL: they have strTst769'
	#index__769=1  
	#strTst_769='<!--  /home/bill/web/' 
	#pinnL_idx_strTst_split_pYes_pNo  "$pAllL__769"  "$index__769"  "$strTst_769"  "$pWebYes769"  "$pWebNo_769"  "$pDifYes769"  "$pDifNo_769"

'pHtmlPathWebPage yes.txt' unchanged? :
+-----+
extract webPages from htmlL: they have strTst769
/home/bill/web/bin/fileops.sh: line 1374: warning: command substitution: ignored null byte in input
/home/bill/web/bin/fileops.sh: line 1374: warning: command substitution: ignored null byte in input
~
+-----+
>> why the error? it works well from command line?

21:17$ bash "$d_bin"'webSite update local.sh' 
find all html files
extract webPages from htmlL: they have strTst769
strTst333 = <!--  /home/bill/web/
strTst_len333 = 21
/home/bill/web/bin/fileops.sh: line 1374: warning: command substitution: ignored null byte in input
/home/bill/web/bin/fileops.sh: line 1374: warning: command substitution: ignored null byte in input
~

+-----+
olde code

	dWeb_get_pWebPageL()
{  
	date_ymdhms=$(date +"%0y%0m%0d %0kh%0Mm%0Ss")
	echo  >>"$p_log" "$date_ymdhms dWeb_get_pWebPageL"
	echo  ''

	#d_webTrgt="$d_web"
	pWebL_769="$d_webWork"'pHtmlPathAll_L.txt'
	#pTrNNL769="$d_webTrgt"'webWork/pHtmlPathTrNNL.txt'
	pExclL769="$d_webWork"'pHtmlPathExclL webPage.txt'

	pWebPageLTmp769="$d_temp"'pWebPageLTmp.txt'

	if  [ -s "$pWebL_769" ]; then
		pInn_archiveLocal_pDateMod  "$pWebL_769"
	fi
	#if  [ -s "$pTrNNL769" ]; then
	#	pInn_archiveLocal_pDateMod  "$pTrNNL769"
	#fi

	idx769=1  
	strTst769='<!--  /home/bill/web/' 

	echo  'find all html files'
	find  "$d_web"      -type f  -name "*.html"  |  grep   --file="$pExclL769"   --invert-match  |  sort -u  >"$pWebL_769"  

	echo  'cull all webPages from htmlL'
	povrL_idx_strTst_cut  "$pWebL_769"  "$idx769"  "$strTst769"  "$pExclL769"
#	???? need to put in -f check on whether a file exists?

	# 19Jan2024 no longer using d_webTrgt??
	#find  "$d_webTrgt"  -type f  -name "*.html"  |  grep   --file="$pExclL769"   --invert-match  |  sort -u  >"$pTrNNL769"  
	#povrL_idx_strTst_cut  "$pTrNNL769"  "$idx769"  "$strTst769"  "$pExclL769"
	#???? need to put in -f check on whether a file exists?
}



#08********08
#] 12Nov2023 fix pOvrL_pStrP_pInsert_replace()  ALL webPages
see also : 
	"$d_bin"'0_test/fileops/0_fileops test notes.txt'
	"$d_bin"'fileops notes.txt' 
	"$d_bin"'webSite update notes.txt'  - LOCAL d_web update, notes
>> GREAT! it all worked, but it was tough


#08********08
#] 11Nov2023 fix pOvrL_pStrP_pInsert_replace() test[1, 5]
see 
	"$d_bin"'0_test/fileops/fileops test run.sh'  
	"$d_bin"'0_test/fileops/fileops test run.sh'  

12Nov2023 

pOvrL_pStrP_pInsert_replace() change : 
	if [[ -s  "$pOvL734"  &&   [ -s  "$pStP734" ]]; then  
to:
	if [[ -s  "$pOvL734" ] && [ -s  "$pStP734" ]]; then  

change : 
	if [[ -s  "$pOvL734" ] && [ -s  "$pStP734" ]]; then  
to :
	if [ -s  "$pOvL734"  &&   -s  "$pStP734" ]; then  
change : 
			if  [[ -n "$bolStrt734" ] && [ -n "$bolEndd734" ]]; then  
to : 
			if  [ -n "$bolStrt734"  &&  -n "$bolEndd734" ]; then  
change : 
	if [  -s  "$pOvL734"  &&   -s  "$pStP734"  ]; then  
			if  [  -n "$bolStrt734"  &&  -n "$bolEndd734"  ]; then  
to : 
	if [[ -s  "$pOvL734"  &&   -s  "$pStP734" ]]; then  
			if  [[ -n "$bolStrt734"  &&  -n "$bolEndd734" ]]; then  

grep: ^<![-]+[-]+Global site tag [(]gtag.js[)][-]+-[-]+Google Analytics[-]+[-]+>: No such file or directory
grep: ^  gtag('config', 'UA-136580492-1');: No such file or directory

change :
			bolStrt734=$( grep  "$pOvr734"  "$strt734" )
			bolEndd734=$( grep  "$pOvr734"  "$endd734" )
to : 
			bolStrt734=$( grep  "$strt734"  "$pOvr734" )
			bolEndd734=$( grep  "$endd734"  "$pOvr734" )

+-----+
olde code

#  pOvrL_pStrP_pInsert_replaceFix(pOvrL pStrP pInsert) replace ONE strPStrEnd in each pOvrL
#     pStrP MUST be properly escaped (problem for arbitrary pStrP -> need pHum_sed_pCde)
#     n[Strt, Endd] are set manually: to fix "addition" in prodn version instead of "replace"
# 10Nov2023 following failure of pOvrL_pStrP_pIns_replaceLines
# 11Nov2023 still has mistake!!! don't use until fixed!  


	pOvrL_pStrP_pInsert_replaceFix()
{
	# 11Nov2023 still has mistake!!! don't use until fixed!
	#    see "$d_SysMaint"'webSite/0_website bash notes.txt'  10Nov2023 manually fix
	pOvL734="$1"
	pStP734="$2"
	pIns734="$3"
	pTmp734="$d_temp"'pOvrL_pStrP_pInsert_replace temp.txt'

	pLog734="$d_bin"'bin logs/pOvrL_pStrP_pInsert_replace log.txt'
	if  [ -s "$pLog734" ]; then  pInn_archiveLocal_pDateMod  "$pLog734";  rm  "$pLog734";  fi


	if [[ -s  "$pOvL734"  &&   [ -s  "$pStP734" ]]; then  
		# 09Nov2023 auto-archive!  may want to make that a boolean later
		pInn_archiveLocal_pDateMod  "$pOvr734"

		# 11Nov2023 currently only one pStrP works in file
		exec 734<"$pStP734"
		read -u 734 strP734 
		strt734=$( echo  "$strP734"  |  sed 's|\x9.*||' )
		endd734=$( echo  "$strP734"  |  sed 's|.*\x9||' )
		#close 734  "command not found" how do I close a file? maybe bash does that...
			echo					  "strt734 = $strt734"
			echo					  "endd734 = $endd734"
			echo  >>"$pLog734"  "strt734 = $strt734"
			echo  >>"$pLog734"  "endd734 = $endd734"

		while IFS='' read -u 734 pOvr734; do
			echo >>"$pLog734"  "$pOvr734"
			if  [ -s  "$pTmp734" ]; then  rm  "$pTmp734";  fi  

			# 11Nov223 tempFix: find out if ^  gtag('config', 'UA-136580492-1'); is present
			bolStop734=$( grep  "$pOvr734"  "$endd734" )
			if	[[ -n "$bolStop734" ]]; then
			
				bolIn734=0 
				# initial read up to strt 
				while IFS='' read -u 584 line734; do
							bolStrt734=$( echo  "$line734"  |  grep  "$strt734" )
					if	[[ -n "$bolStrt734" ]]; then
						bolIn734=1 
						cat  "$pIns734"  >>"$pTmp734"
						# skip original insert
						while [[ 1  ==  "$bolIn734" ]]; do
							IFS='' read -u 584 line734
							bolEndd734=$( echo  "$line734"  |  grep  "$endd734" )
							#	echo  "line734    = $line734"
							#	echo  "bolEndd734 = $bolEndd734"
							if [[ -n "$bolEndd734" ]]; then  
								bolIn734=0;  
								# 11Nov2023 fixTmp: remove for normal cases when this function works!!
								read -u 584 line734  # get rid of extra </script>
							fi
						done
					elif  [[ 0 == "$bolIn734" ]]; then  echo  "$line734"  >>"$pTmp734"
					fi 
				done 584<"$pOvr734" 
				cp  "$pTmp734"  "$pOvr734"

			else
				echo >>"$pLog734"  'pOvrL_pStrP_pInsert_replace bolStop noexist : '
				echo >>"$pLog734"  "   $pOvr734" 
			fi
		done 734<"$pOvL734" 

	else
		echo >>"$pLog734"  'pOvrL_pStrP_pInsert_replace error, at least one noexist : '
		echo >>"$pLog734"  "   $pOvL734"  
		echo >>"$pLog734"  "   $pStP734"  
	fi
}  


#08********08
#] 30Sep2023 z_historic 'webSite update.sh'
see "$d_bin"'webSite update notes.txt'


#08********08
#] 24Sep2023 webSite_getCheck_internalLinks() - remove sedExpr
I removed :
	;s|<!\-\-[ ]*\([$chrL_pathCde901]+\)\.html.*|\1.html|Ig
Now "un-commented active part is : 
	chrL_pathCde901=$( strHum_sed_strCde  "$chrL_path" )
	sedExprCde901='s|.*<A HREF\=\"\([$chrL_pathCde901]+\)\">.*|\1|Ig;s|.*<IMG SRC\=\"\([$chrL_pathCde901]+\)\".*|\1|Ig;s|.*<IMG moz-do-not-send="true"[ ]*SRC\=\"\([$chrL_pathCde901]+\)\".*|\1|Ig' 
		echo  "chrL_pathCde901 = $chrL_pathCde901"

17:13$ bash  "$d_bin"'fileops run webSite.sh'  
chrL_pathCde901 = Azm@zZazm@zz0zm@z9zp@z_zm@zzrlzzrrz<>za@z%#zbszzfsz
sedExprCde901   = s|.*<A HREF\=\"\([$chrL_pathCde901]+\)\">.*|\1|Ig;s|.*<IMG SRC\=\"\([$chrL_pathCde901]+\)\".*|\1|Ig;s|.*<IMG moz-do-not-send="true"[ ]*SRC\=\"\([$chrL_pathCde901]+\)\".*|\1|Ig;s|<!\-\-[ ]*\([$chrL_pathCde901]+\)\.html.*|\1.html|Ig
/home/bill/web/bin/fileops.sh: line 2276: : No such file or directory
>> Hmm, I need something better - as \([$chrL_pathCde901]+\) is chr-by-chr match

change :
	\([$chrL_pathCde901]+\)
to : 
	\(.*\)
then process pWebLinkTemp901
This might mix "end quotes" and drag in a whole pile of junk - process pWebLinkTemp901 in another step? 

TTook out for now : 
|  grep  "^\/home\/bill\/web\/"  |  sed  's|#.*||;s|^\/home\/bill\/web\/$||'  |  grep --invert-match "^$"  


#08********08
#] 17Sep2023 create initial povrL_pTypRgxT_replace family of functions
# www.BillHowell.ca 17Sep2023 initial draft, based on attempts ?Jun2023? 
# 17Sep2023 still must fully translate!


#08********08
#] 14Sep2023 now fixed: some are wrong!!  after combining [bolPovEs, bolPstrEs] -> bolChrCd
"$d_bin"'0_test/fileops/fileops test.sh'
n		sedChrL_test						the \ test results always fail!!?
n		strHum_sed_strCde_test
OK		str_strP_replace_test
OK		pHum_sed_pCde_test
n		povr_pStrP_replace_test
n		povrL_pStrP_replace_test		1/3 wrong: povrL_pStrP_replace: bad - has  p_dif output
n		povrL_pStrP_fixLinks_test		real mess

+-----+
sedChrL_test() the \ test results always fail!!?
	sedCodeStd =  zbsz
	sedCodes   =  za@z
	sedHumToCde='s#\\#zbsz#g'		change to s#[\]#zbsz#g
	sedCdeToHum='s#zbsz#\\#g'		change to s#zbsz#\\\#g
>> still doesn't work!
$ bash  "$d_bin"'0_test/fileops/fileops test.sh' 
sed: -e expression #1, char 88: unknown option to `s'
>> with sedCdeToHum='s#zbsz#\\#g'		change to s#zbsz#\\\#g
try 
	sedHumToCde='s#\\#zbsz#g'		change to s#[\\]#zbsz#g
change 
	s#[\&]#za@z#g	s#[\\]#zbsz#g
to
	s#[&]#za@z#g	s#[\]#zbsz#g
>> YES! that did it - both the & and \ were replaced by the s#[\&]#za@z#g

+-----+
strHum_sed_strCde_test()
I fixed the test and its sub, putting in proper strHumToCdeStd (actually from results)

+-----+
str_strP_replace_test() looks OK, didn't look really hard
pHum_sed_pCde_sub() looks OK, didn't look really hard (complex, long files)

+-----+
povr_pStrP_replace_test() alway fails, key errors : 
	povr_pStrP_replace error: OR[povr, pStrPL] doesnt exist : 
	povr  :  1
change : 
			povr_pStrP_replace  0  1  1  "$p_res172"  "$pStrP172"  
to : 
			povr_pStrP_replace  0  1  "$p_res172"  "$pStrP172"  

changed : 
	sedChrL legend simple rfc.txt
to : 
	sedChrL legend simple inn.txt

now repeat error : 
	230914 13h32m22s
	povr_strP_replace error, could not get :
		0
povr_strP_replace()
		echo  >>"$pLog"  "povr_strP_replace error, could not get :"
		echo  >>"$pLog"  "   $povr__448"  
>> getting a zero - weird screwup
'fileops pStrP.txt' - very tough coding lines, great test!
ither files also tough
>> however, it doesn't look like any pStrP replacements have been made

povr_pStrP_replace()
change : 
			povr_strP_replace  0  0  0		"$povr_tmp607"  "$strP607"
to: 
			povr_strP_replace  0  0  "$povr_tmp607"  "$strP607"
>> but still the same error : 
	230914 13h32m22s
	povr_strP_replace error, could not get :
		0
>> get rid of the 230914 13h32m22s

I didn't escape ()??? - hah! I forgot!!
	sedHumToCde=	;s#[(]#zrlz#g;s#[)]#zrrz#g
	sedCdeToHum=	;s#zrlz#\(#g;s#zrrz#\)#g
Rerun sedChrL_test() 

I'm blind!

change : 
			povr_strP_replace  0  0  0		"$povr_tmp607"  "$strP607"
to :
			povr_strP_replace  0  0  "$povr_tmp607"  "$strP607"
>> FINALLY, it works

+-----+
povr_pStrP_replace_test()

$ bash  "$d_bin"'0_test/fileops/fileops test.sh' 
cp: cannot stat '/home/bill/web/bin/0_test/fileops/povr_pStrP_replace/sedChrL legend simple rfc.txt': No such file or directory
cp: cannot stat '/home/bill/web/bin/0_test/fileops/povr_pStrP_replace/urls interWebPage short-to-longs rfc.txt': No such file or directory
cp: cannot stat '/home/bill/web/bin/0_test/fileops/povr_pStrP_replace/fileops rfc.txt': No such file or directory

povrL_pStrP_replace_test()
change : 
	povrL_pStrP_replace  0  1  1  "$povrL047"  "$pStrP047"  
to : 
	povrL_pStrP_replace  0  1  "$povrL047"  "$pStrP047"  

change rfc to inn to cp files from "$bin"'0_test/fileops/povr_pStrP_replace/'

$ bash  "$d_bin"'0_test/fileops/fileops test.sh' 
/home/bill/web/bin/0_test/fileops/povrL_pStrP_replace/sedChrL legend simple res.txt
/home/bill/web/bin/0_test/fileops/povrL_pStrP_replace/urls interWebPage short-to-longs res.txt
/home/bill/web/bin/0_test/fileops/povrL_pStrP_replace/fileops res.txt
>> Awesome.  done

+-----+
olde code 

	echo >>"$pLog"  'the \ test result always fails, giving ampersand!!?' 


#08********08
#] 14Sep2023 pOvrL_pStrP_replace - should either [en, de]code [neither, both] [pOvr, pStrP]
change :
	"$d_bin"'fileops.sh' - library of handy bash functions for file needs
	"$d_webWork"'fileops run webSite.sh'  - process to update [, sub]webSites
	"$d_PROJECTS"'bin - secure/lftp update specified dir.sh'  - actual uploading of files
	... many other files that use pOvrL_pStrP_replace

geany regexpr [bolPovEs, bolPstrEs, bolStrEs, bolPStPEs] -> bolChrCd
to simplify - povrL_pStrP_replace() keeps encoding pStrP with each pOvr
	not as efficient, but code is MUCH simpler

Done - but usage will reveal bugs...

+-----+
olde code

dir_updateNonHtml_dWeb()
# dWeb is a SUB-dir - doesn't include [http://www.BillHowell.ca/, /home/bill/web/] 

	# there was a HUGE bug in str_strP_replace()

# bolPStPEs -> bol_pStrPEscape

povrL_pStrP_replace()
		if [ 1  -eq  "$bolChrCd384" ]; then
			pHum_sed_pCde	"$pStrP_384"	"$pStrpPtmp384"
		else 
			cp -p				"$pStrP_384"	"$pStrpPtmp384"
		fi

povrL_strP_replace()
		if [ 1  -eq  "$bolChrCd961" ]; then
			strP_tmp961=$( strHum_sed_strCde  "$strP_961" )
		else 
			strP_tmp961="$strP_961"
		fi

pHtmlL_putClass_pStrPL()
#	povrL_pStrP_replace  0 1   "$pHtmlL289"  "$pStrPL289"  


#08********08
#] +-----+
#] see "$d_bin"'0_test/fileops/povrL_pStrP_replace/0_povrL_pStrP_replace_test notes.txt'
#] see "$d_bin"'0_test/fileops/povr_strP_replace/0_povr_strP_replace_test notes.txt'
#] 31Jul2023 strSearch_strP_replace_test() - many examples
#] 31Jul2023 pStrP_sedEsc_test()
#] 31Jul2023 [str, povr] replace [strP, pStrP] in path[,L]  - collection of functions
#] povr_sedEsc()  - escape ALL OR[sedEscSearch, sedEscApo] in povr
#] 28Jul2023 create fileops.sh sedEscape()
#] +-----+

#08********08
#] 16Mar2023 readarray -t fndL611 <"$pinn", want a txtL from readarray - NOT an evaluation 
see "$d_SysMaint"'Linux/readarray notes.txt'  

#08********08
#] 15Mar2023 see '0_test/fileops/pinn_sedFndGetPut_pout/0_pinn_sedFndGetPut_pout notes.txt'

#08********08
#] 15Mar2023 pinn_getLinseq_povr()  - extract a sequence of lines between [strt, endd] markers in pTxt
see "$d_bin"'0_test/fileops/pinn_sedFndGetPut_pout/0_pinn_getLinseq_povr notes.txt'

#08********08
#] 15Mar2023 pinn_sedFndGetPut_pout()
see "$d_bin"'0_test/fileops/pinn_sedFndGetPut_pout/0_pinn_sedFndGetPut_pout notes.txt' 

#08********08
#] 14Mar2023 povr_sedEach_sedAll()  - transforms matching sequences of lines 
see "$d_bin"'0_test/fileops/pinn_sedFndGetPut_pout/0_pinn_sedFndGetPut_pout notes.txt' 

#08********08
#] 01Mar2023 clean up file_ops.sh: povr_strP_replace() etc, tests for wrd

+-----+
olde code

#] povr_strP_replace  - backs up povr, replaces strOld with strNew, backup flag 
# www.BillHowell.ca  01Jun2021 intitial
# 16Feb2023 initial independent of earlier version, code 448
# 17Feb2023 couldn't resolve leading [space, tab]s which are removed
#		see "$d_SysMaint"'Linux/bash notes.txt'
#		continue for now with butchered output - at least it will be backed up...

	povr_strP_replace()
{	
	flagBackup448="$1"
	povr448="$2"
	strOld448="$3"
	strNew448="$4"
	ptmp448="$d_temp"'povr_strOld_strNew_replace temp.txt'
	
	#	echo  "$flagBackup"
	#	echo  "$povr448"
	#	echo  "$strOld448"
	#	echo  "$strNew448"
	#	echo  "$ptmp448"

	if  [ -f  "$povr448" ]; then 
		if	[ "$flagBackup448" == "1" ]; then
			pinn_backupDatedTo_zArchive  "$povr448" 
		fi
		# make sure that ptmp is empty, backup pStrOld384
		if [ -f "$ptmp448" ]; then 
			rm  "$ptmp448"
		fi 
		
		IFS=''
		while read -r -u 7 line; do
			printf '%s\n'  "$line" |  sed "s|$strOld448|$strNew448|g"  >>"$ptmp448"
		done 7< "$povr448" 
		
		mv  "$ptmp448"  "$povr448"  
	else
		echo  "povr_strP_replace error: noExist  $povr448"
	fi
}  


#08********08
#] 03Mar2023 develop "$d_bin"'port bash to python.sh'
#] 02Mar2023 cp file_ops.sh to python, same function names
see "$d_bin"'0_test/fileops/port bash to python/0_port bash to python notes.txt'  


#08********08
#] 01Mar2023 more dirChanges, to make pths shorter
/Professional & Resume/	/Professional/
/Projects - major/	/ProjMajor/
/Projects - mini/	/ProjMini/
/Software programming & code/	/CompLangs/
/Fractal comptr languages/	/Fractals/
/LibreOffice macros/	/LibreOffice/
/PineScript for TradingView market charts/	/PineScript/
/Professional & Resume/	/Professional/

#08********08
#] 01Mar2023 search 'bash read syntax error near unexpected token `newline'

+-----+
https://stackoverflow.com/questions/5134399/bash-syntax-error-near-unexpected-token-newline
-bash: syntax error near unexpected token `newline'
Asked 12 years ago
Modified 1 year, 11 months ago
Viewed 403k times


#08********08
#] 01Mar2023 pinn_pOld_pNew_replace() development, fixes
both my functionsd work beautifully now, for ongoing work on webSite see : 
	"$d_web"'webWork/0_website bash notes.txt' 

+-----+
olde code
			elif	[ $bool146 == 1 ]; then
					# cat "$pTmp146"  "$linePinn146"  >>"$pRep146"
					# rm  "$pTmp146"
					idx146=0
					bool146=0

			#	pTmpAll197="$d_temp"'pinnL_pOldNewPL_replace pTmpAll.txt'
					#	mv  "$pTmpOne197"  "$pTmpAll197" 

#08********08
#] 01Mar2023 search 'Linux bash read and what is the maximum file indicator?' 1024

+-----+
https://stackoverflow.com/questions/24955883/what-is-the-max-opened-files-limitation-on-linux
What is the max opened files limitation on Linux?
Asked 8 years, 7 months ago
Modified 4 years ago
Viewed 39k times
You can check the Soft limits and hard limits of your system by ulimit -a command.
    soft limits are simply the currently enforced limits.
    hard limits mark the maximum value which cannot be exceeded by setting a soft limit.
Soft limits could be set by any user while hard limits are changeable only by root. Limits are a property of a process. They are inherited when a child process is created so system-wide limits should be set during the system initialization in init scripts and user limits should be set during user login for example by using pam_limits.

There are often defaults set when the machine boots. So, even though you may reset your ulimit in an individual shell, you may find that it resets back to the previous value on reboot. You may want to grep your boot scripts for the existence ulimit commands if you want to change the default.

If the limitation was 1024, means you/process can open maximum 1024 files. if you exceed this limit means open, pipe and dup system calls will fail:

    RLIMIT_NOFILE:

    Specifies a value one greater than the maximum file descriptor number that can be opened by this process. Attempts (open(2), pipe(2), dup(2), etc.) to exceed this limit yield the error EMFILE.

edited Jun 20, 2020 at 9:12
CommunityBot
111 silver badge
answered Jul 25, 2014 at 13:32
Sathish


#08********08
#] 01Mar2023 search 'Linux bash and how do I assign the lines of a file to an array?' readarray
see "$d_SysMaint"'Linux/readarray notes.txt'  


#08********08
#] 17Feb2023 pinn_strOld_strNew_replace - test on webwork files, use IFS='' before read!!

$ ls -1 "$d_web"'webWork files/'
see "$d_SysMaint"'webSite/0_website bash notes.txt' : 
	'16Feb2023 executeEmbed examples'

first test : 
pinn_strOld_strNew_replace -> active line : 
	pinn_strOld_strNew_replace  1  "$d_web"'webWork files/fin footer Neil Howell.html'  '[#=; backtrack ;=#]'  'http://www.BillHowell.ca/'
$ bash  "$d_bin"'fileops calls.sh'  
>> nothing happened to files, no pArc

try 
	pth_get_pArchive "$d_web"'webWork files/fin footer Neil Howell.html'  
$ bash  "$d_bin"'fileops calls.sh'  
/home/bill/web/webWork files/z_Archive/230217 12h54m fin footer Neil Howell.html
>> OK

>> substitutions are in every line, wrong place!!?? example : 
from :	<COL WIDTH=80%>
to:		<COLhttp://www.BillHowell.ca/WIDTH=20%>
>> must be `[] charaters? so backslash them
OK result is : 
from :
<IMG SRC="[#=; backtrack ;=#]Creative commons.png"  alt="Creative Commons License"  style="border-width:0"></a></TD>
to: 
MG SRC="http://www.BillHowell.ca/Creative commons.png"  alt="Creative Commons License"  style="border-width:0"></a></TD>
>> but leading [space, tab]s are lost!? comand line echo doesn't do that 
>> also - may have multiple strOld per line so I added tailing `g to sedExpr
>> leading empty line added 
>> subDir not added to local files - 
	CAN'T use relative addressing as [head, foot, Menu]s used widely. 
	Can local subDir be appended to 'http://www.BillHowell.ca/'?

for 'leading [space, tab]s are lost' : 
see "$d_SysMaint"'Linux/echo notes.txt'
>> I used printf but same problem

I will continue to test without leads...
>> oops problem is that working file lack formating : leading [space, tab]s!!

>> Howell: I added to pinn_strOld_strNew_replace : 
IFS='' 
>> YEAH!! it worked, and I had solved this in the past but forgot it...

26Feb2023 

+-----+
olde code 

		if		[ "$check_pOldNewPL" == 1 ]; then 
			# check that all listed pths exist 
			while  read -r -u 9 pPOldNew; do  
			done 88<"$pOldNewPL"

27Feb2023
		# set pL[Old, New] separate files to avoint repeated line-splits, start empty
		pLOld384="$d_temp"'pinnL_pOldNewPL_replace pLOld.txt'
		pLNew384="$d_temp"'pinnL_pOldNewPL_replace pLNew.txt'
		if  [ -f  "$pLOld384" ]; then 
				rm  "$pLOld384"
		fi
		
		if  [ -f  "$pLNew384" ]; then 
				rm  "$pLNew384"
		fi

					# not needed...open file "$pLNew197" to read ... 7< "$pLNew197"
							read -r -u 6 lineNew197


#08********08
#] 16Feb2023 fileops.sh Library: key issue is practical use of library
	reduced functionality, but [re-use , modularisation] of code
	randowm 3-digit integer to reduce "cross-ver" effects of using same variable name
>> going well, need to update webPages soon so it works!!! (broken 09Jun2021?)

	pinn_strOld_strNew_replace  1  "$d_web"'webWork files/fin footer Paul Vauhan.html'  '\[#=; backtrack ;=#\]'  'http://www.BillHowell.ca/'
>> same problem of leading [space, tab]s :  
see "$d_SysMaint"'Linux/echo notes.txt'


08********08
#] 16Feb2023 fileops.sh Library: test str_inWebPages_pout() in "$d_bin"'fileops.sh'
$ bash  "$d_bin"'website urls.sh'  - quick fixes to webSiteURLs
>> results show nasty problem with : code develop_test
>> add it to excludes

ran str_inWebPages_pout() with strEmbedExecute="\[#!: "
but first, I must clean up fin-[header, footer, menu]s so that they don't generate mire work by gaff!
do NOT touch conf[EmbedHead, Foot, Foot_authors, Status].ntml
	oops, I have to - they have [#=; backtrack ;=#]
so replace [#=; backtrack ;=#] with http://www.BillHowell.ca/ in d_webWork


#08********08
#] 16Feb2023 search 'bash script to change strings in multiple files'

+-----+
https://stackoverflow.com/questions/11392478/how-to-replace-a-string-in-multiple-files-in-linux-command-line
How to replace a string in multiple files in linux command line
Asked 10 years, 7 months ago
Modified 25 days ago
Viewed 882k times

+--+
	cd /path/to/your/folder
	sed -i 's/foo/bar/g' *
Occurrences of "foo" will be replaced with "bar".
On BSD systems like macOS, you need to provide a backup extension like -i '.bak' or else "risk corruption or partial content" per the manpage.
	cd /path/to/your/folder
	sed -i '.bak' 's/foo/bar/g' *
edited May 18, 2020 at 19:16
Frungi
answered Jul 9, 2012 at 9:39
kev
>> never saw the backup option before!!!

+--+
If you have list of files you can use
	replace "old_string" "new_string" -- file_name1 file_name2 file_name3
If you have all files you can use
	replace "old_string" "new_string" -- *
If you have list of files with extension, you can use
	replace "old_string" "new_string" -- *.extension
edited Jan 8, 2018 at 9:39
Jawahar Rakkiannan
answered May 8, 2014 at 19:10
Pawel Dubiel
>> wow!  I am stupid (but no archive), I don't have replace






# enddoc