#] 
#] *********************
#] "$d_bin"'fileops notes.txt'  
# www.BillHowell.ca  26Feb2023 initial, took older notes from 
#	"$d_SysMaint"'webSite/0_website bash notes.txt'  - etc
# view in text editor, using constant-width font (eg courier), tabWidth = 3

#48************************************************48




#24************************24
# Table of Contents, generate with :
# $ grep  "^#]"  "$d_bin"'fileops notes.txt' |  sed "s/^#\]/  /" 
#
   *********************
   "$d_bin"'fileops notes.txt'  
   14Sep2023 now fixed: some are wrong!! after combining [bolPovEs, bolPstrEs] -> bolChrCd
   14Sep2023 pOvrL_pStrP_replace - should either [en, de]code [neither, both] [pOvr, pStrP]
   +-----+
   see "$d_bin"'0_test/fileops/povrL_pStrP_replace/0_povrL_pStrP_replace_test notes.txt'
   see "$d_bin"'0_test/fileops/povr_strP_replace/0_povr_strP_replace_test notes.txt'
   31Jul2023 strSearch_strP_replace_test() - many examples
   31Jul2023 pStrP_sedEsc_test()
   31Jul2023 [str, povr] replace [strP, pStrP] in path[,L]  - collection of functions
   povr_sedEsc()  - escape ALL OR[sedEscSearch, sedEscApo] in povr
   28Jul2023 create fileops.sh sedEscape()
   +-----+
   16Mar2023 readarray -t fndL611 <"$pinn", want a txtL from readarray - NOT an evaluation 
   15Mar2023 
   15Mar2023 pinn_getLinseq_povr()  - extract a sequence of lines between [strt, endd] markers in pTxt
   15Mar2023 pinn_sedFndGetPut_pout()
   14Mar2023 povr_sedEach_sedAll()  - transforms matching sequences of lines 
   01Mar2023 clean up file_ops.sh: povr_strP_replace() etc, tests for wrd
   povr_strP_replace  - backs up povr, replaces strOld with strNew, backup flag 
   03Mar2023 develop "$d_bin"'port bash to python.sh'
   02Mar2023 cp file_ops.sh to python, same function names
   01Mar2023 more dirChanges, to make pths shorter
   01Mar2023 search 'bash read syntax error near unexpected token `newline'
   01Mar2023 pinn_pOld_pNew_replace() development, fixes
   01Mar2023 search 'Linux bash read and what is the maximum file indicator?' 1024
   01Mar2023 search 'Linux bash and how do I assign the lines of a file to an array?' readarray
   17Feb2023 pinn_strOld_strNew_replace - test on webwork files, use IFS='' before read!!
   16Feb2023 fileops.sh Library: key issue is practical use of library
   16Feb2023 fileops.sh Library: test str_inWebPages_pout() in "$d_bin"'fileops.sh'
   16Feb2023 search 'bash script to change strings in multiple files'


#24************************24
# Setup, ToDos,   

01Mar2023 
	[test, fix] code to replace Menus - pinnL_pStrPL_replace()
	create code to :
		insert pth in : '<!--  [#!:/home/bill/web/fname:!#]  -->'
	cp Howell's *.html to "$d_webWork"'webPagesLocal/', replacing URL with pth in 1st line
	test use of "$d_webWork"'webPagesLocal/'
08Mar2023 def regexstr_tests iterative problem : incOne_create_seq( 0 1 c=0; c< )  
	should be ( 0 1 c ), 2 other ';for's work well...
14Sep2023 pOvrL_pStrPL_replace - should either [en, de]code [neither, both] [pOvr, pStrP]




#08********08
#] ??Sep2023 




#08********08
#] ??Sep2023 




#08********08
#] ??Sep2023 




#08********08
#] 30Sep2023 z_historic 'webSite update.sh'
see "$d_bin"'webSite update notes.txt'


#08********08
#] 24Sep2023 webSite_getCheck_internalLinks() - remove sedExpr
I removed :
	;s|<!\-\-[ ]*\([$chrL_pathCde901]+\)\.html.*|\1.html|Ig
Now "un-commented active part is : 
	chrL_pathCde901=$( strHum_sed_strCde  "$chrL_path" )
	sedExprCde901='s|.*<A HREF\=\"\([$chrL_pathCde901]+\)\">.*|\1|Ig;s|.*<IMG SRC\=\"\([$chrL_pathCde901]+\)\".*|\1|Ig;s|.*<IMG moz-do-not-send="true"[ ]*SRC\=\"\([$chrL_pathCde901]+\)\".*|\1|Ig' 
		echo  "chrL_pathCde901 = $chrL_pathCde901"

17:13$ bash  "$d_bin"'fileops run webSite.sh'  
chrL_pathCde901 = Azm@zZazm@zz0zm@z9zp@z_zm@zzrlzzrrz<>za@z%#zbszzfsz
sedExprCde901   = s|.*<A HREF\=\"\([$chrL_pathCde901]+\)\">.*|\1|Ig;s|.*<IMG SRC\=\"\([$chrL_pathCde901]+\)\".*|\1|Ig;s|.*<IMG moz-do-not-send="true"[ ]*SRC\=\"\([$chrL_pathCde901]+\)\".*|\1|Ig;s|<!\-\-[ ]*\([$chrL_pathCde901]+\)\.html.*|\1.html|Ig
/home/bill/web/bin/fileops.sh: line 2276: : No such file or directory
>> Hmm, I need something better - as \([$chrL_pathCde901]+\) is chr-by-chr match

change :
	\([$chrL_pathCde901]+\)
to : 
	\(.*\)
then process pWebLinkTemp901
This might mix "end quotes" and drag in a whole pile of junk - process pWebLinkTemp901 in another step? 

TTook out for now : 
|  grep  "^\/home\/bill\/web\/"  |  sed  's|#.*||;s|^\/home\/bill\/web\/$||'  |  grep --invert-match "^$"  


#08********08
#] 17Sep2023 create initial povrL_pTypRgxT_replace family of functions
# www.BillHowell.ca 17Sep2023 initial draft, based on attempts ?Jun2023? 
# 17Sep2023 still must fully translate!


#08********08
#] 14Sep2023 now fixed: some are wrong!!  after combining [bolPovEs, bolPstrEs] -> bolChrCd
"$d_bin"'0_test/fileops/fileops test.sh'
n		sedChrL_test						the \ test results always fail!!?
n		strHum_sed_strCde_test
OK		str_strP_replace_test
OK		pHum_sed_pCde_test
n		povr_pStrP_replace_test
n		povrL_pStrP_replace_test		1/3 wrong: povrL_pStrP_replace: bad - has  p_dif output
n		povrL_pStrP_fixLinks_test		real mess

+-----+
sedChrL_test() the \ test results always fail!!?
	sedCodeStd =  zbsz
	sedCodes   =  za@z
	sedHumToCde='s#\\#zbsz#g'		change to s#[\]#zbsz#g
	sedCdeToHum='s#zbsz#\\#g'		change to s#zbsz#\\\#g
>> still doesn't work!
$ bash  "$d_bin"'0_test/fileops/fileops test.sh' 
sed: -e expression #1, char 88: unknown option to `s'
>> with sedCdeToHum='s#zbsz#\\#g'		change to s#zbsz#\\\#g
try 
	sedHumToCde='s#\\#zbsz#g'		change to s#[\\]#zbsz#g
change 
	s#[\&]#za@z#g	s#[\\]#zbsz#g
to
	s#[&]#za@z#g	s#[\]#zbsz#g
>> YES! that did it - both the & and \ were replaced by the s#[\&]#za@z#g

+-----+
strHum_sed_strCde_test()
I fixed the test and its sub, putting in proper strHumToCdeStd (actually from results)

+-----+
str_strP_replace_test() looks OK, didn't look really hard
pHum_sed_pCde_sub() looks OK, didn't look really hard (complex, long files)

+-----+
povr_pStrP_replace_test() alway fails, key errors : 
	povr_pStrP_replace error: OR[povr, pStrPL] doesnt exist : 
	povr  :  1
change : 
			povr_pStrP_replace  0  1  1  "$p_res172"  "$pStrP172"  
to : 
			povr_pStrP_replace  0  1  "$p_res172"  "$pStrP172"  

changed : 
	sedChrL legend simple rfc.txt
to : 
	sedChrL legend simple inn.txt

now repeat error : 
	230914 13h32m22s
	povr_strP_replace error, could not get :
		0
povr_strP_replace()
		echo  >>"$pLog"  "povr_strP_replace error, could not get :"
		echo  >>"$pLog"  "   $povr__448"  
>> getting a zero - weird screwup
'fileops pStrP.txt' - very tough coding lines, great test!
ither files also tough
>> however, it doesn't look like any pStrP replacements have been made

povr_pStrP_replace()
change : 
			povr_strP_replace  0  0  0		"$povr_tmp607"  "$strP607"
to: 
			povr_strP_replace  0  0  "$povr_tmp607"  "$strP607"
>> but still the same error : 
	230914 13h32m22s
	povr_strP_replace error, could not get :
		0
>> get rid of the 230914 13h32m22s

I didn't escape ()??? - hah! I forgot!!
	sedHumToCde=	;s#[(]#zrlz#g;s#[)]#zrrz#g
	sedCdeToHum=	;s#zrlz#\(#g;s#zrrz#\)#g
Rerun sedChrL_test() 

I'm blind!

change : 
			povr_strP_replace  0  0  0		"$povr_tmp607"  "$strP607"
to :
			povr_strP_replace  0  0  "$povr_tmp607"  "$strP607"
>> FINALLY, it works

+-----+
povr_pStrP_replace_test()

$ bash  "$d_bin"'0_test/fileops/fileops test.sh' 
cp: cannot stat '/home/bill/web/bin/0_test/fileops/povr_pStrP_replace/sedChrL legend simple rfc.txt': No such file or directory
cp: cannot stat '/home/bill/web/bin/0_test/fileops/povr_pStrP_replace/urls interWebPage short-to-longs rfc.txt': No such file or directory
cp: cannot stat '/home/bill/web/bin/0_test/fileops/povr_pStrP_replace/fileops rfc.txt': No such file or directory

povrL_pStrP_replace_test()
change : 
	povrL_pStrP_replace  0  1  1  "$povrL047"  "$pStrP047"  
to : 
	povrL_pStrP_replace  0  1  "$povrL047"  "$pStrP047"  

change rfc to inn to cp files from "$bin"'0_test/fileops/povr_pStrP_replace/'

$ bash  "$d_bin"'0_test/fileops/fileops test.sh' 
/home/bill/web/bin/0_test/fileops/povrL_pStrP_replace/sedChrL legend simple res.txt
/home/bill/web/bin/0_test/fileops/povrL_pStrP_replace/urls interWebPage short-to-longs res.txt
/home/bill/web/bin/0_test/fileops/povrL_pStrP_replace/fileops res.txt
>> Awesome.  done

+-----+
olde code 

	echo >>"$pLog"  'the \ test result always fails, giving ampersand!!?' 


#08********08
#] 14Sep2023 pOvrL_pStrP_replace - should either [en, de]code [neither, both] [pOvr, pStrP]
change :
	"$d_bin"'fileops.sh' - library of handy bash functions for file needs
	"$d_webWork"'fileops run webSite.sh'  - process to update [, sub]webSites
	"$d_PROJECTS"'bin - secure/lftp update specified dir.sh'  - actual uploading of files
	... many other files that use pOvrL_pStrP_replace

geany regexpr [bolPovEs, bolPstrEs, bolStrEs, bolPStPEs] -> bolChrCd
to simplify - povrL_pStrP_replace() keeps encoding pStrP with each pOvr
	not as efficient, but code is MUCH simpler

Done - but usage will reveal bugs...

+-----+
olde code

dir_updateNonHtml_dWeb()
# dWeb is a SUB-dir - doesn't include [http://www.BillHowell.ca/, /home/bill/web/] 

	# there was a HUGE bug in str_strP_replace()

# bolPStPEs -> bol_pStrPEscape

povrL_pStrP_replace()
		if [ 1  -eq  "$bolChrCd384" ]; then
			pHum_sed_pCde	"$pStrP_384"	"$pStrpPtmp384"
		else 
			cp -p				"$pStrP_384"	"$pStrpPtmp384"
		fi

povrL_strP_replace()
		if [ 1  -eq  "$bolChrCd961" ]; then
			strP_tmp961=$( strHum_sed_strCde  "$strP_961" )
		else 
			strP_tmp961="$strP_961"
		fi

pHtmlL_putClass_pStrPL()
#	povrL_pStrP_replace  0 1   "$pHtmlL289"  "$pStrPL289"  


#08********08
#] +-----+
#] see "$d_bin"'0_test/fileops/povrL_pStrP_replace/0_povrL_pStrP_replace_test notes.txt'
#] see "$d_bin"'0_test/fileops/povr_strP_replace/0_povr_strP_replace_test notes.txt'
#] 31Jul2023 strSearch_strP_replace_test() - many examples
#] 31Jul2023 pStrP_sedEsc_test()
#] 31Jul2023 [str, povr] replace [strP, pStrP] in path[,L]  - collection of functions
#] povr_sedEsc()  - escape ALL OR[sedEscSearch, sedEscApo] in povr
#] 28Jul2023 create fileops.sh sedEscape()
#] +-----+

#08********08
#] 16Mar2023 readarray -t fndL611 <"$pinn", want a txtL from readarray - NOT an evaluation 
see "$d_SysMaint"'Linux/readarray notes.txt'  

#08********08
#] 15Mar2023 see '0_test/fileops/pinn_sedFndGetPut_pout/0_pinn_sedFndGetPut_pout notes.txt'

#08********08
#] 15Mar2023 pinn_getLinseq_povr()  - extract a sequence of lines between [strt, endd] markers in pTxt
see "$d_bin"'0_test/fileops/pinn_sedFndGetPut_pout/0_pinn_getLinseq_povr notes.txt'

#08********08
#] 15Mar2023 pinn_sedFndGetPut_pout()
see "$d_bin"'0_test/fileops/pinn_sedFndGetPut_pout/0_pinn_sedFndGetPut_pout notes.txt' 

#08********08
#] 14Mar2023 povr_sedEach_sedAll()  - transforms matching sequences of lines 
see "$d_bin"'0_test/fileops/pinn_sedFndGetPut_pout/0_pinn_sedFndGetPut_pout notes.txt' 

#08********08
#] 01Mar2023 clean up file_ops.sh: povr_strP_replace() etc, tests for wrd

+-----+
olde code

#] povr_strP_replace  - backs up povr, replaces strOld with strNew, backup flag 
# www.BillHowell.ca  01Jun2021 intitial
# 16Feb2023 initial independent of earlier version, code 448
# 17Feb2023 couldn't resolve leading [space, tab]s which are removed
#		see "$d_SysMaint"'Linux/bash notes.txt'
#		continue for now with butchered output - at least it will be backed up...

	povr_strP_replace()
{	
	flagBackup448="$1"
	povr448="$2"
	strOld448="$3"
	strNew448="$4"
	ptmp448="$d_temp"'povr_strOld_strNew_replace temp.txt'
	
	#	echo  "$flagBackup"
	#	echo  "$povr448"
	#	echo  "$strOld448"
	#	echo  "$strNew448"
	#	echo  "$ptmp448"

	if  [ -f  "$povr448" ]; then 
		if	[ "$flagBackup448" == "1" ]; then
			pinn_backupDatedTo_zArchive  "$povr448" 
		fi
		# make sure that ptmp is empty, backup pStrOld384
		if [ -f "$ptmp448" ]; then 
			rm  "$ptmp448"
		fi 
		
		IFS=''
		while read -r -u 7 line; do
			printf '%s\n'  "$line" |  sed "s|$strOld448|$strNew448|g"  >>"$ptmp448"
		done 7< "$povr448" 
		
		mv  "$ptmp448"  "$povr448"  
	else
		echo  "povr_strP_replace error: noExist  $povr448"
	fi
}  


#08********08
#] 03Mar2023 develop "$d_bin"'port bash to python.sh'
#] 02Mar2023 cp file_ops.sh to python, same function names
see "$d_bin"'0_test/fileops/port bash to python/0_port bash to python notes.txt'  


#08********08
#] 01Mar2023 more dirChanges, to make pths shorter
/Professional & Resume/	/Professional/
/Projects - major/	/ProjMajor/
/Projects - mini/	/ProjMini/
/Software programming & code/	/CompLangs/
/Fractal comptr languages/	/Fractals/
/LibreOffice macros/	/LibreOffice/
/PineScript for TradingView market charts/	/PineScript/
/Professional & Resume/	/Professional/

#08********08
#] 01Mar2023 search 'bash read syntax error near unexpected token `newline'

+-----+
https://stackoverflow.com/questions/5134399/bash-syntax-error-near-unexpected-token-newline
-bash: syntax error near unexpected token `newline'
Asked 12 years ago
Modified 1 year, 11 months ago
Viewed 403k times


#08********08
#] 01Mar2023 pinn_pOld_pNew_replace() development, fixes
both my functionsd work beautifully now, for ongoing work on webSite see : 
	"$d_web"'webWork/0_website bash notes.txt' 

+-----+
olde code
			elif	[ $bool146 == 1 ]; then
					# cat "$pTmp146"  "$linePinn146"  >>"$pRep146"
					# rm  "$pTmp146"
					idx146=0
					bool146=0

			#	pTmpAll197="$d_temp"'pinnL_pOldNewPL_replace pTmpAll.txt'
					#	mv  "$pTmpOne197"  "$pTmpAll197" 

#08********08
#] 01Mar2023 search 'Linux bash read and what is the maximum file indicator?' 1024

+-----+
https://stackoverflow.com/questions/24955883/what-is-the-max-opened-files-limitation-on-linux
What is the max opened files limitation on Linux?
Asked 8 years, 7 months ago
Modified 4 years ago
Viewed 39k times
You can check the Soft limits and hard limits of your system by ulimit -a command.
    soft limits are simply the currently enforced limits.
    hard limits mark the maximum value which cannot be exceeded by setting a soft limit.
Soft limits could be set by any user while hard limits are changeable only by root. Limits are a property of a process. They are inherited when a child process is created so system-wide limits should be set during the system initialization in init scripts and user limits should be set during user login for example by using pam_limits.

There are often defaults set when the machine boots. So, even though you may reset your ulimit in an individual shell, you may find that it resets back to the previous value on reboot. You may want to grep your boot scripts for the existence ulimit commands if you want to change the default.

If the limitation was 1024, means you/process can open maximum 1024 files. if you exceed this limit means open, pipe and dup system calls will fail:

    RLIMIT_NOFILE:

    Specifies a value one greater than the maximum file descriptor number that can be opened by this process. Attempts (open(2), pipe(2), dup(2), etc.) to exceed this limit yield the error EMFILE.

edited Jun 20, 2020 at 9:12
CommunityBot
111 silver badge
answered Jul 25, 2014 at 13:32
Sathish


#08********08
#] 01Mar2023 search 'Linux bash and how do I assign the lines of a file to an array?' readarray
see "$d_SysMaint"'Linux/readarray notes.txt'  


#08********08
#] 17Feb2023 pinn_strOld_strNew_replace - test on webwork files, use IFS='' before read!!

$ ls -1 "$d_web"'webWork files/'
see "$d_SysMaint"'webSite/0_website bash notes.txt' : 
	'16Feb2023 executeEmbed examples'

first test : 
pinn_strOld_strNew_replace -> active line : 
	pinn_strOld_strNew_replace  1  "$d_web"'webWork files/fin footer Neil Howell.html'  '[#=; backtrack ;=#]'  'http://www.BillHowell.ca/'
$ bash  "$d_bin"'fileops calls.sh'  
>> nothing happened to files, no pArc

try 
	pth_get_pArchive "$d_web"'webWork files/fin footer Neil Howell.html'  
$ bash  "$d_bin"'fileops calls.sh'  
/home/bill/web/webWork files/z_Archive/230217 12h54m fin footer Neil Howell.html
>> OK

>> substitutions are in every line, wrong place!!?? example : 
from :	<COL WIDTH=80%>
to:		<COLhttp://www.BillHowell.ca/WIDTH=20%>
>> must be `[] charaters? so backslash them
OK result is : 
from :
<IMG SRC="[#=; backtrack ;=#]Creative commons.png"  alt="Creative Commons License"  style="border-width:0"></a></TD>
to: 
MG SRC="http://www.BillHowell.ca/Creative commons.png"  alt="Creative Commons License"  style="border-width:0"></a></TD>
>> but leading [space, tab]s are lost!? comand line echo doesn't do that 
>> also - may have multiple strOld per line so I added tailing `g to sedExpr
>> leading empty line added 
>> subDir not added to local files - 
	CAN'T use relative addressing as [head, foot, Menu]s used widely. 
	Can local subDir be appended to 'http://www.BillHowell.ca/'?

for 'leading [space, tab]s are lost' : 
see "$d_SysMaint"'Linux/echo notes.txt'
>> I used printf but same problem

I will continue to test without leads...
>> oops problem is that working file lack formating : leading [space, tab]s!!

>> Howell: I added to pinn_strOld_strNew_replace : 
IFS='' 
>> YEAH!! it worked, and I had solved this in the past but forgot it...

26Feb2023 

+-----+
olde code 

		if		[ "$check_pOldNewPL" == 1 ]; then 
			# check that all listed pths exist 
			while  read -r -u 9 pPOldNew; do  
			done 88<"$pOldNewPL"

27Feb2023
		# set pL[Old, New] separate files to avoint repeated line-splits, start empty
		pLOld384="$d_temp"'pinnL_pOldNewPL_replace pLOld.txt'
		pLNew384="$d_temp"'pinnL_pOldNewPL_replace pLNew.txt'
		if  [ -f  "$pLOld384" ]; then 
				rm  "$pLOld384"
		fi
		
		if  [ -f  "$pLNew384" ]; then 
				rm  "$pLNew384"
		fi

					# not needed...open file "$pLNew197" to read ... 7< "$pLNew197"
							read -r -u 6 lineNew197


#08********08
#] 16Feb2023 fileops.sh Library: key issue is practical use of library
	reduced functionality, but [re-use , modularisation] of code
	randowm 3-digit integer to reduce "cross-ver" effects of using same variable name
>> going well, need to update webPages soon so it works!!! (broken 09Jun2021?)

	pinn_strOld_strNew_replace  1  "$d_web"'webWork files/fin footer Paul Vauhan.html'  '\[#=; backtrack ;=#\]'  'http://www.BillHowell.ca/'
>> same problem of leading [space, tab]s :  
see "$d_SysMaint"'Linux/echo notes.txt'


08********08
#] 16Feb2023 fileops.sh Library: test str_inWebPages_pout() in "$d_bin"'fileops.sh'
$ bash  "$d_bin"'website urls.sh'  - quick fixes to webSiteURLs
>> results show nasty problem with : code develop_test
>> add it to excludes

ran str_inWebPages_pout() with strEmbedExecute="\[#!: "
but first, I must clean up fin-[header, footer, menu]s so that they don't generate mire work by gaff!
do NOT touch conf[EmbedHead, Foot, Foot_authors, Status].ntml
	oops, I have to - they have [#=; backtrack ;=#]
so replace [#=; backtrack ;=#] with http://www.BillHowell.ca/ in d_webWork


#08********08
#] 16Feb2023 search 'bash script to change strings in multiple files'

+-----+
https://stackoverflow.com/questions/11392478/how-to-replace-a-string-in-multiple-files-in-linux-command-line
How to replace a string in multiple files in linux command line
Asked 10 years, 7 months ago
Modified 25 days ago
Viewed 882k times

+--+
	cd /path/to/your/folder
	sed -i 's/foo/bar/g' *
Occurrences of "foo" will be replaced with "bar".
On BSD systems like macOS, you need to provide a backup extension like -i '.bak' or else "risk corruption or partial content" per the manpage.
	cd /path/to/your/folder
	sed -i '.bak' 's/foo/bar/g' *
edited May 18, 2020 at 19:16
Frungi
answered Jul 9, 2012 at 9:39
kev
>> never saw the backup option before!!!

+--+
If you have list of files you can use
	replace "old_string" "new_string" -- file_name1 file_name2 file_name3
If you have all files you can use
	replace "old_string" "new_string" -- *
If you have list of files with extension, you can use
	replace "old_string" "new_string" -- *.extension
edited Jan 8, 2018 at 9:39
Jawahar Rakkiannan
answered May 8, 2014 at 19:10
Pawel Dubiel
>> wow!  I am stupid (but no archive), I don't have replace






# enddoc