#] #] ********************* #] "$d_bin""backup notes.txt" www.BillHowell.ca 16Jul2018 initial # view in text editor, using constant-width font (eg courier), tabWidth = 3 #] 14Nov2021 no more [weekly, monthly] distinction - all are incremental, just rename with YYMMDD_HMS when time to archive dangerous to --delete[, -excluded] = delete [extraneous, excluded] files from dest dirs this would desty files involved in inter-computer transfers can do for [webSite, ssh] transfers (separate bash file for safety?) ********** excludes : $ find "/media/bill/SWAPPER/" -name "gvfs-metadata" $ find "gvfs-metadata" +-----+ Problems #] 14Nov2021 >> WHAT!? did not [create, backup] a dir, so I manually did so. Why did it fail???? : backer_rsync() - rsync PROJECTS to USBbackup, 211114 9h46m /media/bill/Dell2/PROJECTS/Personal & Events/Sony digicorder #] +-----+ #48************************************************48 #24************************24 # Table of Contents, generate with : # $ grep "^#]" "$d_bin"'backup notes.txt' | sed "s/^#\]/ /" # ********************* "$d_bin""backup notes.txt" 14Nov2021 no more [weekly, monthly] distinction - all are incremental, 14Nov2021 >> WHAT!? did not [create, backup] a dir, so I manually did so. Why did it fail???? : +-----+ 01Sep2022 'VIDEOS' to 'SG6' USB HDD : via "$d_bin""rsync directories.sh" 23Aug2022 ThinkPad new main computer - problems with backup 14Nov2021 clean up "$d_bin""backup.sh" 05Mar2021 change-over to new "virtual" drives 20Jan2021 problems with backup 07Oct2020 problems with backup 28Sep2019 Jump in size! : 12&13Aug2019 rsync d_travel="/media/bill/SAMSUNG_G4/IJCNN2019/" back to SWAPPER 11Aug2019 backup SWAPER got stuck : 03May2019 move PROJECTS contents to SWAPPER (only 1 main USB) 25Feb2019 Thunderbird remove from SWAPPER (it's been on LMDE2 for some time) 22Dec2018 Move videos to VIDEO Disk 20Nov2018 LMDE21 maintenance - updates : 08Oct2018 stats not producing files transferred output 07Sep2018 rsync to /media/bill/SWAPPER/Common_between_computers/ 16Jul2018 Backups were a pain : SWAPPER was read-only (again!) so I had to try 3 times #24************************24 # Setup, ToDos, #08********08 #] ??Jan2023 #08********08 #] ??Jan2023 #08********08 #] 20Jan2023 home dir backup halts, home backs up all! one or more of subDirs halts progress - /var/lib/samba/usershares - add usershares to excludes >> seems to work but may affect usershares in other dirs? due to new inclusion of these dirs in ~ home excludes : ramdisk, PROJECTS, web, evolution, Thunderbird] - done by : backer_home >> OK, now works!? +-----+ olde code if [ $flag_home == "true" ]; then # echo "p_excludes_home rsync = $p_excludes_home" p_excl="$p_excludes_home" elif [ $flag_home == "home" ]; then p_excl="$d_log""excludes null.txt" else p_excl="$d_log""excludes.txt" fi #08********08 #] 26Dec2022 upgrade backup.sh - mostly home dir didn't backup [home, evolution, >> must include hidden files! '0_bookmarks Howell.html' - don't need special function in script! +-----+ olde code # 26Dec2022 - actiually don't need the rest! : just make sure hidden [fil, dir] are backed up log_title="backer_rsync() - rsync .bashrc in $partition to USBbackup, " d_src="/home/bill/.bashrc" d_out="$d_backupDrive$partition/" backer_rsync log_title="backer_rsync() - rsync libreOffice macros in $partition to USBbackup, " d_src="/home/bill/.config/libreoffice/4/user/basic/" d_out="$d_backupDrive$partition/.config/libreoffice/4/user/basic/" backer_rsync log_title="backer_rsync() - rsync HTML bookmarks in $partition to USBbackup, " d_src="/home/bill/.mozilla/firefox/izgy9n6t.bill/bookmarkbackups/" d_out="$d_backupDrive$partition/.mozilla/firefox/izgy9n6t.bill/bookmarkbackups/" backer_rsync # backer_home() - trickey as [d_PROJECTS, d_web] are in d_home, must be LAST backup!! backer_home() { # trickey flag - must be LAST backup!! flag_home="true" # rebuild --exclude-from using "base" version p_excludes_home="$d_bin""backupper/excludes home.txt" # echo "p_excludes_home setup = $p_excludes_home" echo "web" >"$p_excludes_home" echo "PROJECTS" >>"$p_excludes_home" echo "Suse home" >>"$p_excludes_home" echo "evolution" >>"$p_excludes_home" echo "Thunderbird" >>"$p_excludes_home" echo "firefox" >>"$p_excludes_home" #echo "Thunderbird" >>"$p_excludes_home" #echo "Thunderbird" >>"$p_excludes_home" cat "$d_bin""backupper/excludes.txt" >>"$p_excludes_home" partition="Suse home" log_title="backer_rsync() - rsync $partition to USBbackup, " d_src="/home/bill/" d_out="$d_backupDrive$partition/" backer_rsync flag_home="home" log_title="backer_rsync() - rsync .bashrc in $partition to USBbackup, " d_src="/home/bill/.bashrc" d_out="$d_backupDrive$partition/" backer_rsync log_title="backer_rsync() - rsync libreOffice macros in $partition to USBbackup, " d_src="/home/bill/.config/libreoffice/4/user/basic/" d_out="$d_backupDrive$partition" backer_rsync log_title="backer_rsync() - rsync HTML bookmarks in $partition to USBbackup, " d_src="/home/bill/.mozilla/firefox/izgy9n6t.bill/bookmarkbackups/" d_out="$d_backupDrive$partition/" backer_rsync } # 03Dec2022 couldn't get above to work. # LMDE original has more stuff : see ~'.bashrc 02Dec2022 backup.txt' #export d_home='/home/bill/' #export d_meda='/media/bill/' #export d_temp='/home/bill/ramdisk/' #export EDITOR='/usr/bin/geany' Backup as protection against missing things "$d_bin"'backupper/excludes.txt' : .adobe .android .aspell.en.prepl .aspell.en.pws .audacity-data .bash_history .bash_logout .bashrc .cache .cinnamon .cmake .config .dbus .directory .dmrc .eclipse .electrum .emacs .emacs~ .emacs.d .esd_auth .face .fonts .fonts.conf .fseventsd .gconf .gimp-2.8 .gksu.lock .gnome2 .gnupg .gnuplot_history .gphoto .gtkrc-2.0 .i18n .ICEauthority .icons .inputrc .java .jigdo-lite .lesshst .linuxmint .local .macromedia .matlab .mozilla .mutt .mutt_certificates .muttrc .nano .oracle_jre_usage .pki .profile .putty .rnd .Scilab .selected_editor .ssh .subversion .swt .synaptic .texlive2016 .texmf-var .themes .thumbnails .thunderbird .urlview .viminfo .wget-hsts .Xauthority .xdvirc .xim.template .xinitrc.template .xinputrc .xsession-errors .xsession-errors.old .zoom #08********08 #] 01Sep2022 'VIDEOS' to 'SG6' USB HDD : via "$d_bin""rsync directories.sh" $ du_diff '/run/media/bill/VIDEOS/' '/home/bill/SG6/VIDEOS/' +-----+ 1,2d0 < 1.0T < 1.0T total 6a5 > 225G Neural Net videos 9,10d7 < 4.0G DVD_Geert_Wilders < 469G Neural Net videos 14a12,13 > 776G > 776G total +-----+ >> In excludes : DVD_Geert_Wilders I then created backer_NoExcludes(), and ran : backer_NoExcludes '/run/media/bill/VIDEOS/DVD_Geert_Wilders/' '/home/bill/SG6/VIDEOS/DVD_Geert_Wilders/' backer_NoExcludes '/run/media/bill/VIDEOS/Neural Net videos/' '/home/bill/SG6/VIDEOS/Neural Net videos/' $ bash "$d_bin""du_diff.sh" >> empty diff file - same [size, file, pth]s #08********08 #] 23Aug2022 ThinkPad new main computer - problems with backup many, many details changed, still not working right, but mostly OK PROJECTS, ?web? - web log cumulative.txt 23Aug2022 seems to be working - didn't backup SG6 : that's good, but I don't know why Suse home log cumulative.txt 23Aug2022 NUTS!!! 4 separate updates - repeated commands!! backer_rsync() - rsync Midas to USBbackup, 220823 18h32m only created 3 files! not working 08********08 #] 14Nov2021 clean up "$d_bin""backup.sh" no more [weekly, monthly] distinction - all are incremental, just rename with YYMMDD_HMS when time to archive +-----+ olde code # update_ymdhm - updates to Linux [Mint, Raspian], Apr2018 - do this manually!!! #becho 'sudo >>'$p_log' 2>&1 apt-get update' # sudo >>"$p_log" 2>&1 apt-get update # $ bash "" # update_ymdhm() #{ #if [ $wk_or_mnth = "weekly" ]; then # date_ymd=$(date +"%y%m%d weekly_backup") #else date_ymd=$(date +"%y%m%d monthly_backup") #fi # #partition="UPDATES" #p_log="$d_log""$partition $HOWELL_computer log cumulative.txt" ##p_log9="$d_log0""$partition log details.txt" #becho "" #becho "+---------------------------------------------------------+" #becho "Update_ymdhm $HOWELL_computer during $wk_or_mnth backup.sh, date : $date_ymd" #becho "log in to continue : " #beval 'sudo apt-get update' #becho "" #} computer -> USBbackup transfers use 'weekly'! (incremental backup) # [d_overwrites, last update varable] : I need to upgrade script to save potential conflicts # # 12. USBbackup -> computer transfers use 'weekly'! (incremental backup) # [d_overwrites, last update varable] : I need to upgrade script to save potential conflicts # Problem : old deleted directories get reincarnated (do test backup first and check) # backup_USBdrive_to_LMDE # backup_USBdrive_to_ThinkPad # backer_SAMSUNG_G4() - # 29Feb2020 DON'T use this! use "backup computer [to,from] TravelUSB.sh" # 16May2020 Reactivated - have permanent-resident files for Meth8-VL4 mini-project # 08Jul2021 needs update to correct handling of [d_out, partition]! backer_SAMSUNG_G4() { if [ -d "$d_meda""SAMSUNG_G4" ]; then partition="SAMSUNG_G4" log_title="backer_rsync() - rsync $partition to USBbackup, " d_src="$d_meda$partition/" d_out="$d_backupDrive$partition/" backer_rsync fi } # olde code # if [ $wk_or_mnth = "annual" ]; then # mkdir "$d_out" # while read -u 9 line; do # diry=$( echo "$line" | sed 's/.*\/\(.*\)/\1/' ) # dinn="$line" # beval 'rsync "'"$options"'" "'"$dinn"'" "'"$d_out"'" >>"'"$p_log"'" ' # becho "" # done 9< "$p_annu" # else # beval 'rsync --stats "'"$options"'" --exclude-from="'"$p_excl"'" "'"$d_src"'" "'"$d_out"'" >>"'"$p_log"'" ' # becho "" # dir_sizes # fi # 28Sep2021 Not used!!! -> one-way to USBbackup only! # partition="Suse home" # log_title="backer_rsync() - rsync USBbackup to d_$partition of $Howell_computer, " # d_src="$d_backupDrive$partition/" # d_out="$d_LMDE" # backer_rsync # No back-copy to home directory (dangerous!) # 28Sep2021 Not used!!! -> one-way to USBbackup only! # partition="LMDE home" # log_title="backer_rsync() - rsync USBbackup to d_$partition of $Howell_computer, " # d_src="$d_backupDrive$partition/" # d_out="$d_LMDE" # backer_rsync #if [ -d "$d_meda""WD_3.6Tb/201007 PROJECTS monthly_backup" ]; then # wk_or_mnth="special" # partition="PROJECTS" # d_src="$d_meda201013 WD_3.6Tb/201007 PROJECTS monthly_backup""/" # backer_rsync #fi # very simple - I created "SPECIAL excludes.txt" # 08********08 #] 05Mar2021 change-over to new "virtual" drives [PROJECTS, webRawe, webSite] delete [Ubuntu, Lenovo] For each computer, I want just ONE excludes file : $ cat "$d_bin""backupper/PROJECTS excludes.txt" | sort -u >"$d_bin""backupper/excludes.txt" backer_rsync() : Change : +.....+ p_excl="$d_log""$partition excludes.txt" +.....+ To : +.....+ p_excl="$d_log""excludes.txt" +.....+ Everthing ran well, backup sizes look good for drives [LMDE, ROJECTS, webRawe, webSite] I didn't put the results into the spreadsheet. That can be done next monthly backup. ******** #] 20Jan2021 problems with backup $ bash "$d_bin""du_sort.sh" +-----+ backer_rsync() - monthly rsync of /media/bill/HOWELL_BASE/, 210120 6h48m rsync --stats -rltgu --exclude-from="/media/bill/PROJECTS/bin/backupper/HOWELL_BASE excludes.txt" "/media/bill/HOWELL_BASE/" "/media/bill/WD_3.6Tb/210120 HOWELL_BASE monthly_backup" >>"/media/bill/PROJECTS/bin/backupper/HOWELL_BASE log cumulative.txt" rm: cannot remove '/media/bill/ramdisk/dir_size list.txt': No such file or directory HOWELL_BASE : dir_sizes du: cannot access '/media/bill/HOWELL_BASE/NN_conf_DVDs/2020 WCCI papers & videos/extras 200731': Structure needs cleaning >> what ??? I deleted from "$d_bin""backupper/HOWELL_BASE excludes.txt" : Neural Nets bag Purchases I added to excludes : Diversity - i9018xtp.default Diversity - ssh site +-----+ LMDE I moved /home/bill/backup USB drives/200911 PROJECTS To /media/bill/WD_3.6Tb No sense having it hang around. +-----+ PROJECTS OK for all drives !! ******** #] 07Oct2020 problems with backup massive shift of directories to webRaw changed : /media/bill/PROJECTS/bin/backupper/backups - directory sizes all drives.ods >> OK for now PROBLEM!! no 201007 (07Oct2020) backup directories for [SWAPPER, PROJECTS] on /media/bill/WD_3.6Tb/!!!??? what happened? the log for each doesn't show a problem? Yes it does! rsync --stats -rltgu --exclude-from="/media/bill/PROJECTS/bin/backupper/SWAPPER excludes.txt" "/media/bill/SWAPPER/" "201007 SWAPPER monthly_backup" >>"/media/bill/PROJECTS/bin/backupper/SWAPPER log cumulative.txt" rsync --stats -rltgu --exclude-from="/media/bill/PROJECTS/bin/backupper/PROJECTS excludes.txt" "/media/bill/PROJECTS/" "201007 PROJECTS monthly_backup" >>"/media/bill/PROJECTS/bin/backupper/PROJECTS log cumulative.txt" Both were backed up to /home/bil!!!! why? Move both misplaced directories to /media/bill/WD_3.6Tb/ Now - how to solve the problem? partition directory defaulted to /home/bill, so the d_[PROJECTS, SWAPPER] parts were not used maybe I accidntally deleted a line of code? maybe d_meda=""? $ echo "$d_meda" /media/bill/ >> nope, that doesn't explain it Idiot! - I hadn't uncommented : # d_backupDrive="$d_meda""WD_3.6Tb/" Lucky that the hard drive had the space! *********** #] 28Sep2019 Jump in size! : Jump in size! : 19May2019 5.3 Gb /media/bill/Seagate4Tb180804/190519 HOWELL_BASE monthly_backup/Software installations & maintenance/ 28Sep2019 20.5 Gb /media/bill/HOWELL_BASE/Software installations & maintenance/ >> OK - this is probably part of the change? ... >> EXCLUDE that directory!!! - add to "/media/bill/SWAPPER/bin/backupper/HOWELL_BASE excludes.txt" >> done *********** #] 12&13Aug2019 rsync d_travel="/media/bill/SAMSUNG_G4/IJCNN2019/" back to SWAPPER see ""/media/bill/SWAPPER/bin/backup stuff for travel.sh"" # AFTER all transfers complete & checked!! : (Note - I did tests, then superficial check on some transfers) # $ rm -r "/media/bill/SAMSUNG_G4/IJCNN2019/" # -> have to say "Yes" to special protected files to remove (only 3 for IJCNN2019) *********** #] 11Aug2019 backup SWAPER got stuck : backer_rsync() - monthly rsync of /media/bill/SWAPPER/, 190811 11h27m rsync --stats -rltgu --exclude-from='/media/bill/SWAPPER/bin/backupper/SWAPPER excludes.txt' '/media/bill/SWAPPER/' '/media/bill/SEAGATE3_4T/190811 SWAPPER monthly_backup' >>'/media/bill/SWAPPER/bin/backupper/SWAPPER log cumulative.txt' ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(644) [sender=3.1.2] rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(513) [generator=3.1.2] >> had to Ctrl-C to escape deadlock >> Why? Just in case, I copied to SWAPPER excludes : gvfs-metadata mwad0hks.default excludes : $ find "/media/bill/SWAPPER/" -name "gvfs-metadata" $ find "gvfs-metadata" ****************** #] 03May2019 move PROJECTS contents to SWAPPER (only 1 main USB) References : mostly overlap $ rsync -rltgu "/media/bill/PROJECTS/References/" "/media/bill/SWAPPER/References" ****************** #] 25Feb2019 Thunderbird remove from SWAPPER (it's been on LMDE2 for some time) $ mv "/media/bill/SWAPPER/190225 Thunderbird backup" "/media/bill/SEAGATE3_4T/190225 Thunderbird backup" $ mv "/media/bill/HOWELL_BASE/Climate" "/media/bill/SWAPPER/Climate" $ rm -rd "/media/bill/SEAGATE3_4T/190224 HOWELL_BASE monthly_backup" redo [Projects, LMDE2] - change HOWELL_BASE directory date $ mv "/media/bill/HOWELL_BASE/Projects/ATA_WDC_500G" "/media/bill/SEAGATE3_4T/141123 Lenovo ATA_WDC_500G" $ mv "/media/bill/HOWELL_BASE/Projects/ATA_WDC_500G/Ball, Gordon" "/media/bill/SEAGATE3_4T/160822 Ball, Gordon" test backer_rsync_excludes.sh nyet - maybe some other day : $ mv "/media/bill/HOWELL_BASE/Projects" "/media/bill/SWAPPER/Projects" ****************** #] 22Dec2018 Move videos to VIDEO Disk $ rsync 2>&1 -stats -rltgu "/home/bill/Videos of note/2017 EU Future Science conf/" "/media/bill/VIDEOS/Electric Universe/EU2017 Future Science conf/" >> too many duplicates (different names) ******************** #] 20Nov2018 LMDE21 maintenance - updates : To check authenticity, I web-searched "Linux and ???", where "???" is each library listed below. Warning You are about to install software that can't be authenticated! Doing this could allow a malicious individual to damage or take control of your system. OK curl (version 7.38.0-4+deb8u12) will be upgraded to version 7.38.0-4+deb8u13 OK firmware-atheros (version 0.43) will be upgraded to version 20161130-4~deb8u1 OK firmware-bnx2 (version 0.43) will be upgraded to version 20161130-4~deb8u1 OK firmware-bnx2x (version 0.43) will be upgraded to version 20161130-4~deb8u1 OK firmware-brcm80211 (version 0.43) will be upgraded to version 20161130-4~deb8u1 OK firmware-intelwimax (version 0.43) will be upgraded to version 20161130-4~deb8u1 OK firmware-ipw2x00 (version 0.43) will be upgraded to version 20161130-4~deb8u1 firmware-ivtv (version 0.43) will be upgraded to version 20161130-4~deb8u1 firmware-iwlwifi (version 0.43) will be upgraded to version 20161130-4~deb8u1 firmware-libertas (version 0.43) will be upgraded to version 20161130-4~deb8u1 firmware-linux (version 0.43) will be upgraded to version 20161130-4~deb8u1 firmware-linux-nonfree (version 0.43) will be upgraded to version 20161130-4~deb8u1 firmware-qlogic (version 0.43) will be upgraded to version 20161130-4~deb8u1 firmware-ralink (version 0.43) will be upgraded to version 20161130-4~deb8u1 firmware-realtek (version 0.43) will be upgraded to version 20161130-4~deb8u1 imagemagick (version 8:6.8.9.9-5+deb8u14) will be upgraded to version 8:6.8.9.9-5+deb8u15 OK imagemagick-6.q16 (version 8:6.8.9.9-5+deb8u14) will be upgraded to version 8:6.8.9.9-5+deb8u15 imagemagick-common (version 8:6.8.9.9-5+deb8u14) will be upgraded to version 8:6.8.9.9-5+deb8u15 libcurl3 (version 7.38.0-4+deb8u12) will be upgraded to version 7.38.0-4+deb8u13 libcurl3-gnutls (version 7.38.0-4+deb8u12) will be upgraded to version 7.38.0-4+deb8u13 libmagickcore-6.q16-2 (version 8:6.8.9.9-5+deb8u14) will be upgraded to version 8:6.8.9.9-5+deb8u15 libmagickwand-6.q16-2 (version 8:6.8.9.9-5+deb8u14) will be upgraded to version 8:6.8.9.9-5+deb8u15 libopenjp2-7 (version 2.1.0-2+deb8u4) will be upgraded to version 2.1.0-2+deb8u5 thunderbird (version 1:52.9.1-1~deb8u1) will be upgraded to version 1:60.3.0-1~deb8u1 thunderbird-l10n-en-gb (version 1:52.9.1-1~deb8u1) will be upgraded to version 1:60.3.0-1~deb8u1 firmware-amd-graphics (version 20161130-4~deb8u1) will be installed firmware-misc-nonfree (version 20161130-4~deb8u1) will be installed >> check the rest next month! ******************** #] 08Oct2018 stats not producing files transferred output - stats not producing files transferred output MAybe redirection is problem : rsync 2>&1 -stats - - Ubuntu not working - corrupted file OK rsync: opendir "/media/bill/Ubuntu/home/bill/.dbus" failed: Permission denied (13) OK? rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.1] - residual % is WRONG (always 0) total(Mb)= 6326, residual(Mb)= -1067, residual(%)= 0 ******************************* #] 07Sep2018 rsync to /media/bill/SWAPPER/Common_between_computers/ hasn't worked since January 2018!!!!???? NOTE : 07Sep2018 - backup did occur - why aren't stats reported? : [HOWELL_BASE, LMDE2, Midas, PROJECTS, SWAPPER, Common_to_SWAPPER] Error codes include : du: cannot read directory ‘/media/bill/Ubuntu/home/bill/.dbus’: Permission denied NOTE : 07Sep2018 - backup did occur - why aren't stats reported? : "/media/bill/SWAPPER/bin/commons_to_swapper.sh" doesn't work - why? COMMONS_TO_SWAPPER log cumulative.txt - no error entries Dates missing from log entries???? NO directory : /media/bill/Seagate4Tb180804/Common_between_computers/ **************************** #] 16Jul2018 Backups were a pain : SWAPPER was read-only (again!) so I had to try 3 times Backups were a pain : SWAPPER was read-only (again!) so I had to try 3 times. Very slow SWAPPER permissions : only solution seems to be eject -> mount SWAPPER USB Update Manager warning while doing updates following monthly backups WARNING : You are about to install software that can't be authenticated! Doing this could allow a malicious individual to damage or take control of your system. Checking LMDE2 "Software Sources" Mirrors that I have selected : Main (betsy) : http://muug.ca/mirror/linuxmint/packages Manitoba Unix Users' Group Base (jessie): ftp://debian.bhs.mirrors.ovh.net/debian OVH.com downloads? locations: Beauharnois, QC; France (3) There is no obvious way to check [reliability,honesy,vulnerability] of these sites? I did the Linux kernel update (need to reboot) # enddoc