#] #] ********************* #] "$d_SysMaint"'drives - disk & USB memory/0_[disk, USB] drive notes.txt' # www.BillHowell.ca 29Dec2015 initial #24************************24 #24************************24 # List of operators, generated with : # $ grep "^#]" "$d_SysMaint"'drives - disk & USB memory/0_[disk, USB] drive notes.txt' | sed "s/^#\] / /" ********************* "$d_SysMaint"'Disk drives/0_disk drive notes.txt' 24Jan2024 cp Seagate4Tb180804/ -> WD6Tb_240124 24Jan2024 [format, partition] new WD 6Tb USB HDD (WD4Tb_240124), cp old -> new 28Oct2023 cp to USBmemStick - failure of transfer or wayyyy tooo slow! 27Oct2023 cp to USBmemStick - failure of transfer 28Jul2022 Seagate 6Tb USB HDD - fstab 27Jul2022 setup new Seagate 6Tb USB HDD 27Jull2022 redo transfer of files to new Seagate_6Tb HDD drive backup_USBdrive_to_ThinkPad - 01Dec2021 cannot access 'USB VIDEO' drive - why? 26Mar2021 restructure webSite USB!! 21Jan2021 move [SWAPPER, PROJECTS] to 2nd internal drive 12Oct2020 13Oct2020 nyet - they magically re-appeared? 17Sep2020 PROJECTS USB frozen!?!?! 07Aug2020 special $d_HOWELL_BASE backup - find: "Structure needs cleaning" 31Oct2018 HOWELL_BASE Toshiba drive is getting slow!!! 17Dec2017 Corrupted USB Lexar drive 29Dec2015 Seagate Momentus 7200.1 60Gb, Old Dell Laptop Latitude D620 29Dec2015 Drive information (manufacturer, partitions etc) - what do I use? ~ #24************************24 Handy commands see Linux Desktop Reference Chapter 10 du 2>&1 --max-depth=1 / | grep --invert-match "cannot read" | grep --invert-match "cannot access" du -a /home/bill : sizes of all system files and directories df -h --print-type : display information about file system gparted (LMDE menu) : great for information (incl hardware) lsblk -l : lists all disks, not just USB drives ***** Disks ATA Hitachi HDS72168 75Gb, Lenovo desktop main drive ATA WDC WD5000AAKX-0 465Gb, Lenovo desktop 2nd internal drive Iomega , Lenovo desktop weekly backup ADATA 465Gb, Stalin-Hitler Seagate 2.6Tb, Both desktop & laptop backup TOSHIBA MK8034GSX 75Gb, Toshiba laptop - backup ToshLap_backup FUJITSU MHU2100AT 93Gb, Toshiba laptop - extra directories NOT to back up, take space (should put back with rsyn!) Quantum XP32150W 2Gb, old SCI drive Aug1995 ?never used? Seagate Momentus 7200.1 60Gb, Old Dell Laptop Latitude D620 ***** #24************************24 08********08 #] ??Jan2024 08********08 #] ??Jan2024 How did I virtually partition d_[web, PROJECTS]? 08********08 #] ??Jan2024 08********08 #] ??Jan2024 08********08 #] ??Jan2024 08********08 #] ??Jan2024 08********08 #] ??Jan2024 08********08 #] 26Jan2024 rm (not chmod) problem files [d_web, WD6Tb_240124] from yesterday : +-----+ "$d_web"'ProjMini/Kaal- Structured Atom Model/z_Archive' "$d_web"'ProjMini/Kaal- Structured Atom Model/240115 emto Steve: Sierpinski [triangles, tertrahedra]: Johannes Kepler, Edo Kaal, Pyramids [Bosnia, Egypt, Mesopotamia, Central America, China (dirt, not stone)].html' "$d_web"'ProjMini/Kaal- Structured Atom Model/HowellVoice' "$d_web"'ProjMini/Kaal- Structured Atom Model/Kaal SAM vs QM: deactivation.html' "$d_web"'ProjMini/Kaal- Structured Atom Model/vidAudios' "$d_web"'ProjMini/Kaal- Structured Atom Model/0_Kaal SAM notes.txt' "$d_web"'ProjMini/Kaal- Structured Atom Model/Kaal Structured Atom Model vs Quantum Mechanics.html' "$d_web"'ProjMini/Kaal- Structured Atom Model/Kaal: The Proton-Electron Atom, paragraphL.txt' "$d_web"'ProjMini/Kaal- Structured Atom Model/images' +-----+ wait!! - delete 'ProjMini/Kaal- Structured Atom Model/' !!! first : $ sudo rm -r "$d_web"'ProjMini/Kaal- Structured Atom Model/' $ sudo rm -r '/media/bill/WD6Tb_240124/web/ProjMini/Kaal- Structured Atom Model/' >> done 08********08 #] 24Jan2024 which: [cp, rsync, drive cp utility]? old -> new I feel most comfortable using rsync also, use a drive for "full backups" that ONLY has current drive contents. That way I can simply swap it if disaster hits. [incremental, occasional full] backups as usual see "$d_bin"'backup.sh' heavy_metal_backup() /media/bill/WD6Tb_240124 progress (Tb): 25Jan2024 26Jan2024 12:18 17:20 20:14 00:06 web 0 0 0 0.03 PROJECTS 0 0.46 0.46 0.46 LMDE home n/a n/a 0.02 0.02 VIDEOS n/a n/a 0.67 1.1 Vidoes n/a n/a 0 0.06 calcTotal n/a n/a 1.22 1.67 WD6Tb_240124 n/a n/a 1.1 1.1 461,023 items time (17:22 - 12:14) + (24:06 - 17h24m) = 05:08 + 06:42 = 11:50 hours >> Howell 40 Mb/s transfer rate - not bad, not as high as high speed internet download!! PROJECTS : Neural Nets n/a n/a n/a 0.18 NNvid n/a n/a 0.07 NNconf n/a n/a n/a 0.05 12:14$ >> oops, accidentally did Ctrl-C while typing note above - 12:14$ bash "$d_bin"'backup.sh' +---------------------------------------------------------+ 240125 12h18m start 240125 12h18m rsync --stats -rltgu --delete "/media/bill/9e05b040-df7b-4e71-936b-1f7c459b8842/" "/media/bill/WD6Tb_240124/" >>"/home/bill/web/bin/backupper/ log cumulative.txt" ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(503) [generator=3.2.3] rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(703) [sender=3.2.3] end 240125 17h22m ~ 17:22$ rsync: [receiver] write error: Broken pipe (32) >> restart : 17:22$ bash "$d_bin"'backup.sh' +---------------------------------------------------------+ heavy_metal_rsync(dinn, dout) - rsync a USB drive drop-in for current drive start 240125 17h24m rsync --stats -rltgu --delete "/media/bill/9e05b040-df7b-4e71-936b-1f7c459b8842/" "/media/bill/WD6Tb_240124/" >>"/home/bill/web/bin/backupper/heavy_metal_rsync log cumulative.txt" rsync: [sender] readlink_stat("/media/bill/9e05b040-df7b-4e71-936b-1f7c459b8842/web/ProjMini/Kaal- Structured Atom Model/z_Archive") failed: Permission denied (13) rsync: [sender] readlink_stat("/media/bill/9e05b040-df7b-4e71-936b-1f7c459b8842/web/ProjMini/Kaal- Structured Atom Model/240115 emto Steve: Sierpinski [triangles, tertrahedra]: Johannes Kepler, Edo Kaal, Pyramids [Bosnia, Egypt, Mesopotamia, Central America, China (dirt, not stone)].html") failed: Permission denied (13) rsync: [sender] readlink_stat("/media/bill/9e05b040-df7b-4e71-936b-1f7c459b8842/web/ProjMini/Kaal- Structured Atom Model/HowellVoice") failed: Permission denied (13) rsync: [sender] readlink_stat("/media/bill/9e05b040-df7b-4e71-936b-1f7c459b8842/web/ProjMini/Kaal- Structured Atom Model/Kaal SAM vs QM: deactivation.html") failed: Permission denied (13) rsync: [sender] readlink_stat("/media/bill/9e05b040-df7b-4e71-936b-1f7c459b8842/web/ProjMini/Kaal- Structured Atom Model/vidAudios") failed: Permission denied (13) rsync: [sender] readlink_stat("/media/bill/9e05b040-df7b-4e71-936b-1f7c459b8842/web/ProjMini/Kaal- Structured Atom Model/0_Kaal SAM notes.txt") failed: Permission denied (13) rsync: [sender] readlink_stat("/media/bill/9e05b040-df7b-4e71-936b-1f7c459b8842/web/ProjMini/Kaal- Structured Atom Model/Kaal Structured Atom Model vs Quantum Mechanics.html") failed: Permission denied (13) rsync: [sender] readlink_stat("/media/bill/9e05b040-df7b-4e71-936b-1f7c459b8842/web/ProjMini/Kaal- Structured Atom Model/Kaal: The Proton-Electron Atom, paragraphL.txt") failed: Permission denied (13) rsync: [sender] readlink_stat("/media/bill/9e05b040-df7b-4e71-936b-1f7c459b8842/web/ProjMini/Kaal- Structured Atom Model/images") failed: Permission denied (13) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1333) [sender=3.2.3] end 240126 0h06m ~ 00:06$ +---------------------------------------------------------+ heavy_metal_rsync(dinn, dout) - rsync a USB drive drop-in for current drive start 240125 17h24m rsync --stats -rltgu --delete "/media/bill/9e05b040-df7b-4e71-936b-1f7c459b8842/" "/media/bill/WD6Tb_240124/" >>"/home/bill/web/bin/backupper/heavy_metal_rsync log cumulative.txt" IO error encountered -- skipping file deletion Number of files: 522,949 (reg: 495,464, dir: 27,385, link: 100) Number of created files: 62,701 (reg: 61,056, dir: 1,644, link: 1) Number of deleted files: 0 Number of regular files transferred: 61,057 Total file size: 1,685,182,908,817 bytes Total transferred file size: 971,589,061,286 bytes Literal data: 971,589,061,286 bytes Matched data: 0 bytes File list size: 10,876,447 File list generation time: 0.045 seconds File list transfer time: 0.000 seconds Total bytes sent: 971,848,148,830 Total bytes received: 1,200,866 sent 971,848,148,830 bytes received 1,200,866 bytes 40,228,050.16 bytes/sec total size is 1,685,182,908,817 speedup is 1.73 end 240126 0h06m >> Howell 40 Mb/s transfer rate - not bad +----------+ !!!!!! screw all the stuff below - I'm not recreating the boot drive etc. just use rsync, and it can simply maintain the entire drive to be a ready sub-in for the current USB [disk, SSD] drive. search "Linux complete hard drive copy" +-----+ https://www.maketecheasier.com/clone-entire-hard-drive-linux/ 4 Ways to Clone an Entire Hard Drive on Linux Mike Tee Nov 3, 2021 >> image files? NYET!!! sucks +-----+ https://askubuntu.com/questions/640259/how-can-i-easily-clone-my-entire-system How can I easily clone my entire system? Asked 8 years, 7 months ago Modified 2 years, 7 months ago Viewed 130k times +--+ The easiest way to do this is by using Gparted from Live CD/USB as suggested above. To do this have a look at the "Copying and pasting a partition" section in the Gparted manual. To reproduce the steps given in the link: To copy a partition: Select an unmounted partition. See the section called “Selecting a Partition”. Choose: Partition → Copy. The application marks the partition as the source partition. To Paste a partition: Select an unallocated space on a disk device. See the section called “Selecting Unallocated space”. Choose: Partition → Paste. The application displays the Paste /path-to-partition dialog. If you want you can adjust the size and location of the partition. See the section called “Specifying Partition Size and Location”. If you want you can specify the alignment of partition. See the section called “Specifying Partition Alignment”. Click Paste. The application displays the copy partition operation in the Pending Operations pane. answered Jun 24, 2015 at 6:38 Ron 20.6k66 gold badges5757 silver badges7272 bronze badges Thank you for helping me with this, but i think i will use a manual simple way as @Frantique suggested below, sudo cat /dev/sda >/dev/sdb i know it's risky, but looks the easiest way to me tho. – Adascalitei Lucian Jun 24, 2015 at 6:49 Yes, it is your choice. May be some others will find this method easier :) Be careful with dd (and even cat) as they can mess up things if not used with caution. – Ron Jun 24, 2015 at 8:22 1 @AdascaliteiLucian don't do the cat (see comment above). If you want to speed up things, use a larger buffer size (64MB or so) with dd – Fabby Jun 24, 2015 at 8:53 cat is safer and simpler than dd. There's no reason at all not to use it. – terdon Jul 2, 2019 at 9:55 There is an e2fsck before the actual cloning that has no progress bar and might take hours .. – Wolfgang Fahl Nov 8, 2020 at 19:04 1 Gparted: Can someone clarify whether everything in the partitions is fully accessible(permissions of: Directories, files and Veracrypt containers); & if all partitions can be used to recreate the same PC by loading it to bare hardware? – CloudE11 Jan 11, 2021 at 0:46 If I do this will the TMP security chip drive-passwored-protected-and-locked-to-hardware thingy be copied too? I.e. the original drive can only be accessed in this computer, will that now also be true of the new drive? – Rabbit Jul 12, 2021 at 13:44 +--+ Boot from a live system, plug in the new HDD via USB (or another (s)ata cable, depending on your hardware), and start a 'Terminal'. Check what is the path for your old HDD (probably /dev/sda), and the new one as well (/dev/sdb), and issue this command: sudo dd if=/dev/sda of=/dev/sdb status=progress Note that it will take a long time. You can monitor the progress on the terminal. Another possibility is (even faster than dd): cat /dev/sda >/dev/sdb edited May 31, 2021 at 14:15 caffeineFiend answered Jun 24, 2015 at 6:40 Frantique 2, I gues this is the method i will choise, to be honest, i preffer the shell thing instead of Gparted thing. – , Adascalitei Lucian, Jun 24, 2015 at 6:47 2 , Lucian: da, e cea mai buna solutie. ;) – , Frantique, Jun 24, 2015 at 6:52 6, using dd with a larger block size will speed it up. e.g. sudo dd if=/dev/sda of=/dev/sdb bs=1024k – , David Purdue, Jun 24, 2015 at 6:59 David: yes, indeed, just since we don't know anything about the hardware, it is better to go slower. – Frantique, Jun 24, 2015 at 7:03 1, Buna asta :), nu m-as fi astepat :), oricum tin thread-ul deschis in caz de experientez ceva probleme, Teoretic eu vreau sa inlocuiesc un Hard Wd Blue 1Tb, care deja cred ca i se duc acele de pe platane, cu un Hard Wd Black 1Tb, si nu vreau sa reinstalez toata nebunia iarasi, imi ia o zi sa mi le pun toate la punct, si prefer sa fac treaba asta. Estimativ cred ca mi-ar lua in jur de 4 5 ore sa se copie. Multumesc inca o data. – , Adascalitei Lucian, Jun 24, 2015 at 7:06 Nu va fi problema. Acum 2 luni am facut la fel. Totul a fost OK. – , Frantique, Jun 24, 2015 at 7:49 1, @Frantique: the cat wouldn't preserve any security settings of any of the files... So it would work for root, but no other users! – , Fabby, Jun 24, 2015 at 8:51 ..plus it will not accept any block size (but +1 for cat though :)) – , Ron, Jun 24, 2015 at 8:54 2, This should work, but note that you are writing every block. If you're in a VM environment and are cloning a sparsely-allocated disk, you're going to copy everything, including the empty space, undermining the benefit of sparse allocation. For this situation, a gparted copy/paste (after shrinking the source to a near-minimum size) will be better. As would a file-based clone to a newly-created partition. – , David C., Mar 22, 2018 at 22:37 +--+ I created a bash script to automate this whole cloning process to another disk using rsync in the easiest way possible. The script takes care of everything like changing fstab file, installing the proper grub in your new disk and updating it to create a new grub menu. You just have to prepare your target disk with the appropriate partitions using gparted for example. For more details, please visit its repository at https://github.com/thiggy01/clone-ubuntu answered Jan 3, 2020 at 0:42 thiggy01 1 Good job on cloning the cloning script I wrote. Thanks for giving credit in your github. – WinEunuuchs2Unix Jan 3, 2020 at 12:18 Glad you liked. I just needed a tool to upgrade my computer from a hdd to a sdd. Then, I found your script and, as you didn't restrict its use, I decided to customize it to automate the whole process. – thiggy01 Jan 3, 2020 at 21:28 Can confirm the script works, have used it to clone an older HDD to a new SSD. The script did a great job in checking the prerequisites and I could boot from the SSD immediately. One caveat was, I had to change the UUID of the / partition in /etc/fstab by hand - for some reason this has not been changed. – Tyreal Apr 2, 2023 at 6:31 08********08 #] 24Jan2024 chown new drive, rsyc (not cp) Seagate4Tb180804/ -> WD6Tb_240124 is cp -rup reliable? test: 23:27$ cp -rup "$d_web"'webWork' '/media/bill/d_web' cp: cannot create directory '/media/bill/d_web/webWork': Permission denied >> forgot chown ... tired - do tomorrow morning +-----+ 25Jan2024 set ownership of new drive to bill (me) see "$d_SysMaint"'Linux/chown & chmod notes.txt' 08********08 #] 24Jan2024 NYET: =1 partition!: [format, partition] new WD 6Tb USB HDD (WD6Tb_240124) # d_backupDrive="$d_meda"'Seagate4Tb180804/' d_workinDrive="$d_meda"'Seagate6Tb working drive' d_workinDrive="$d_meda"'WD6Tb_240124/' I got rid of d_[web, PROJECTS] on new drive GParted Tb /dev/sda /dev/sda1/ 0.08 linux-swap linux-swap /dev/sda2/ 0.46 ext4 ext4 /dev/sdb/ 5.46 /media/bill/9e05b040-df7b-4e71-936b-1f7c459b8842 6.0 TB Volume - simply long thing above /dev/sdb2/ 5.46 Seagate_6Tbb ext4 /dev/sdc 5.46 /media/bill/4c09623b-6676-40a4-bb9e-4750f0dcff11 /media/bill/WD6Tb_240124 /dev/sdc2 5.46 WD6Tb_240124 ext4 Also - set label name +-----+ WD6Tb_240124 initial - wrong /dev/sdc 5.46 /dev/sdc1 0.99 d_web ext4 /dev/sdc2 5.46 d_PROJECTS ext4 d_web d_PROJECTS WD4Tb_240124/ 1 Tb 5 Tb (976 Gb, 4.5 Tb) +-----+ WD6Tb_240124 drive revert to one partition ONLY I delete d_web, and reesized d_PROJECTS should have deleted both and creaated a new -> much faster BUT, at least it looks like a way to retain old data in a partition? /dev/sdc 5.46 /dev/sdc2 5.46 WD6Tb_240124 ext4 +-----+ https://www.howtogeek.com/193669/whats-the-difference-between-gpt-and-mbr-when-partitioning-a-drive/ What's the Difference Between GPT and MBR When Partitioning a Drive? Need a hundred partitions on a single drive? GPT might be right for you. Chris Hoffman, Nick Lewis Updated Aug 26, 2023 Key Takeaways GPT is a newer partitioning standard that has fewer limitations than MBR, such as allowing for more partitions per drive and supporting larger drives. Both Windows and macOS, as well as other operating systems, can use GPT for partitioning drives. GPT is more robust and provides better data protection and recovery options compared to MBR, but MBR is still necessary for compatibility with older systems. 08********08 #] 28Oct2023 cp to USBmemStick - failure of transfer or wayyyy tooo slow! d_FECC="$d_PROJECTS"'9_My sports & clubs/FECC fusion/231024 FECC webinar/' d_USBB='/media/bill/HOWELL62GB/' cp "$d_FECC"'Lawrence Livermore NatlLab, NISA, laser + Xray implosion, He-filled hohlraum 231024.MTS' "$d_USBB" +-----+ https://www.howtogeek.com/226992/htg-explains-why-cant-i-copy-large-files-to-my-high-capacity-flash-drive/ Why Can't I Copy Large Files to My High Capacity Flash Drive? Jason Fitzpatrick Published Sep 21, 2016 You have a brand new high-capacity flash drive that can store more than the hard drive of your first three computers combined, but when you go to copy a large file it denies you. Why Do Removable Drives Still Use FAT32 or exFAT Instead of NTFS? Nearly every flash drive in the world ships formatted as with the FAT32 file system. FAT32 isn't the most robust file system around but it has been around for ages, it is widely supported, and most of the time the applications one would use a flash drive for do not require the advanced features found in other file systems (and, in fact, some of those advanced features could even shorten the life of the drive via increased read/writes). If you're curious as to why FAT32 is still so prevalent, check out our article HTG Explains: Why Are Removable Drives Still Using FAT32 Instead of NTFS? Despite the ubiquitous nature of FAT32, however, it has one big shortcoming in the age of large files: the maximum possible file size for FAT32 files is 4GB. (If we're being technical it's actually 4GB minus a single byte or 4,294,967,295 bytes). These days 4GB isn't exactly a whopper of a file and it's real easy, as you discovered, to exceed the 4GB barrier with large video files, DVD and Blu-ray ISOs, and other large files. Now that we know why you can't copy that DVD ISO file over to your nice spacious 64GB flash drive, what can we do about it? Fortunately the fix is very simple. We'll switch the file system to a file system which allows for larger-than-4GB file sizes. +-----+ https://unix.stackexchange.com/questions/743950/slow-file-transfer-to-usb-stick Slow file transfer to USB stick Asked 6 months ago Modified 6 months ago Viewed 976 times "480M" is typical of USB2.0 ports. What you're observing is normal and expected for that USB standard (the apparent high speed at the beginning is a sort of RAM buffering). – ChanganAuto Apr 24 at 23:05 OK thank you, does the lsusb output mean I'm supposed to have USB3 ports? How can I confirm this from the hardware and make those available in my system? – jpo38 Apr 25 at 5:17 1 3 MB/s can't be the limit of the bus, but it can be the writing speed of a cheap USB stick. It starts with 60 MB/s (which indicates USB3, because you won't get this as effective rate on USB2), until the stick's cache is filled, then it stalls to continue with the actual writing speed. What does the data sheet of the stick say? – Philippos Apr 25 at 8:28 1 There's no "cache" for 99.999% of USB flash sticks out there (only rare "SSD" USB flash sticks have it), you're solely looking at dirty buffers being filled up. – Artem S. Tashkinov Apr 25 at 9:16 1 @Philippos, you say it indicates "USB 3" but lsusb shows I'm on a USB 2 bus, doesn't it? I have no USB stick datasheet, it comes from Amazon... it's cheap. – jpo38 Apr 25 at 10:23 08********08 #] 27Oct2023 cp to USBmemStick - failure of transfer nemo doesn't work - gets stuck at small transfer, locks up >> perhaps it's the USB memoray stick, but many problems hit today! +-----+ try cp : d_FECC="$d_PROJECTS"'9_My sports & clubs/FECC fusion/231024 FECC webinar/' $ cp "$d_FECC"'Lawrence Livermore NatlLab, NISA, laser + Xray implosion, He-filled hohlraum 231024.MTS' '/media/bill/USB DISK/' Lawrence Livermore NatlLab, NISA, laser and Xray implosion, He-filled hohlraum 231024.MTS Lawrence Livermore NatlLab, NISA, laser + Xray implosion, He-filled hohlraum 231024.MTS +-----+ maybe its a bad USBmemStick - try "SWAPPER 124Gb" d_FECC="$d_PROJECTS"'9_My sports & clubs/FECC fusion/231024 FECC webinar/' $ cp "$d_FECC"'Lawrence Livermore NatlLab, NISA, laser + Xray implosion, He-filled hohlraum 231024.MTS' '/media/bill/USB_VIDEO/' 08********08 #] 28Jul2022 Seagate 6Tb USB HDD - fstab change entry to : UUID=9e05b040-df7b-4e71-936b-1f7c459b8842 /home/bill ext4 defaults 0 0 >> see if that works 08********08 #] 27Jul2022 setup new Seagate 6Tb USB HDD see also : /home/bill/Documents/0_notes ThinkPad.txt - mount SG6 under /home/bill/SG6 "$d_SysMaint"'Linux/rm notes.txt' - safe removal of [file, dir]s "$d_SysMaint""Disk drives/0_disk drive notes.txt" Modify for environmental variables : ~.bashrc >> so far, small struggles but doesn't look too bad an adjustment "$d_Qndfs"'QNial setup.ndf' - YES!!!! startup worked extremely well!!!! will bash "start" work? 08********08 #] 27Jull2022 redo transfer of files to new Seagate_6Tb HDD drive copy with dolphin puts ALL [file, dir]s to date-of-transfer (like cp usually does!!) only use rsync!! +-----+ Rational for new HDD : Gb transferred : PROJECTS 250.6 10:34 81.8; 11:17 167.0; 11:41 214.3; 11:55 241.1; 12:00E done webRawe 25.4 start ~12:15; 12:20E 12.6; Evolution 3.6 Thunderbird 10.0 ThinkPad '/' 177.6 (= 458.0-280.4) Total 467.2 Total is > ThinkPad HDD 458.0, ignoring new [software, project, etc]s However, /run/media/bill/ 154.1 Gb much of which may be redundant, old programs not used even if zero - still not much headspace I like : security advantage of a separate USB HDD easier to startup a new computer easier to switch to a new SSD can put VIDEOS on new Seagate_6Tb drive but can't use it on a different computer, even Linux +-----+ from: /run/media/bill/WD_3.6Tb/[PROJECTS, webRawe, Evolution, Thunderbird] to: /run/media/bill/9e05b040-df7b-4e71-936b-1f7c459b8842/ first delete all copy-pastes from yesterday (4 hours wasted!) $ rm -rf "/run/media/bill/9e05b040-df7b-4e71-936b-1f7c459b8842/PROJECTS/" $ rm -rf "/run/media/bill/9e05b040-df7b-4e71-936b-1f7c459b8842/Website - raw/" $ rm -rf "/run/media/bill/9e05b040-df7b-4e71-936b-1f7c459b8842/Evolution/" $ rm -rf "/run/media/bill/9e05b040-df7b-4e71-936b-1f7c459b8842/Thunderbird/" +-----+ now rsync!! - adapt core rsync command : done $ partition='PROJECTS' $ partition='webRawe' rename simpler for scripts! (was 'Website - raw') $ partition='Evolution' $ partition='Thunderbird' $ rsync --stats '-rltgu' '/run/media/bill/WD_3.6Tb/'"$partition/" '/run/media/bill/9e05b040-df7b-4e71-936b-1f7c459b8842/'"$partition/" >>'/run/media/bill/WD_3.6Tb/Website - raw/bin/backupper/'"$partition"' log cumulative.txt' +-----+ earlier attempts - no workee added [Evolution, Thhunderbird] to : #] backup_USBdrive_to_ThinkPad - backup_USBdrive_to_ThinkPad() { partition="webRawe" log_title="backer_rsync() - rsync USBbackup to $partition of $Howell_computer, " d_src="$d_backupDrive$partition/" d_out="$d_webRawe" backer_rsync partition="PROJECTS" log_title="backer_rsync() - rsync USBbackup to $partition of $Howell_computer, " d_src="$d_backupDrive$partition/" d_out="$d_PROJECTS" backer_rsync partition="Evolution" log_title="backer_rsync() - rsync USBbackup to $partition of $Howell_computer, " d_src="$d_backupDrive$partition/" d_out="$d_Evolution" backer_rsync partition="Thunderbird" log_title="backer_rsync() - rsync USBbackup to $partition of $Howell_computer, " d_src="$d_backupDrive$partition/" d_out="$d_Thunderbird" backer_rsync } >>> NYET!! log file won't work, let's see if it does the job $ bash '/run/media/bill/WD_3.6Tb/Website - raw/bin/backup.sh' had to temporarily reassign : $ d_bin='/run/media/bill/WD_3.6Tb/Website - raw/bin/' >> doesn't work without existing d_webRawe 08********08 #] 01Dec2021 cannot access 'USB VIDEO' drive - why? gparted no help fat32 (microshit) doesn't do same access control if I remember correctly I used another USB memory stick 08********08 #] 26Mar2021 restructure webSite USB!! edit fow directory structures : "$d_bin""rsync website.sh" - update from d_PROJECTS -> d_webRawe -> d_webSite -> d_webOnln to rsync to webSite see "$d_SysMaint""Linux/rsync notes.txt" 08********08 #] 21Jan2021 move [SWAPPER, PROJECTS] to 2nd internal drive I will use full backup drives to copy-over to [ThinkPad, RaspPi,etc] much more reliabave files on hand big issue with conflicting files!!! 1st - rename second intenal hard drive gparted change own etc $ sudo chown -R bill:bill "/media/bill/Dell2" Use rsync to avoid fuckups with [owner, date, etc] edite, run : $ bash "$d_bin""rsync directories.sh" $ ls -1 "$d_HOWELL_BASE" Charities Diversity - i9018xtp.default Diversity - ssh site DVD_BigBattles_WWII DVD_Geert_Wilders DVD_GreatBattles_WWII DVD_HitlerLost DVD_SovietStory DVD_SuvorovTalks Family Friends Icebreaker vids, music, anims Jobs johannesstoertterart.com.mp4 Jokes letters My Music My Music mp3 My Pictures Neural Nets bag NN_conf_DVDs NRCan OOPM builds OOPM builds old OPM Purchases QNial old junk SW install & maintain ToDos bag Website z_Old From this, a list of additional excludes : Diversity - i9018xtp.default Diversity - ssh site DVD_BigBattles_WWII DVD_Geert_Wilders DVD_GreatBattles_WWII DVD_HitlerLost DVD_SovietStory DVD_SuvorovTalks Icebreaker vids, music, anims NN_conf_DVDs OOPM builds OOPM builds old OPM SW install & maintain +-----+ one problem : rsync --stats --itemize-changes -rltgu --exclude-from="/media/bill/PROJECTS/bin/rsync directories excludes.txt" "/media/bill/HOWELL_BASE/" "/media/bill/Dell2/HOWELL_BASE/" >>"/media/bill/PROJECTS/bin/rsync directories log.txt" du: cannot access '/media/bill/HOWELL_BASE/NN_conf_DVDs/2020 WCCI papers & videos/extras 200731': Structure needs cleaning 2020 WCCI papers & videos : I lost everything!!??!?! 08********08 #] 12Oct2020 I lost : bin - secure!???? multifractal - analysis of SP500 1872-2020 What is the problem? a nemo glitch occurred (just after downloading bash toolsets) perhaps the problem persists? nemo is screwed up, or other ultilities from updates needing a cold reboot perhaps a [virus, worm, trojan horse]? I had downloaded some bash script systems, >> it's in my Todo list, so it won't get done, but now will have to my QNial programs are doing this? (links) my manual move of directories to webRaw directories were [moved, hidden] by [keyboard, mouse] sloppiness some other form of my stupidity Actions re-instate directories from back I have to cold reboot for some updates copy Web to HOWELL_BASE to prevent easy chances of mixing up directories when working inconsistenciescould be very costly +-----+ #] 13Oct2020 nyet - they magically re-appeared? I did a backup.sh check anyways on SWAPPER with option --itemize-changes >> errors, after changing backup.sh : $ bash "$d_bin""backup.sh" /media/bill/PROJECTS/bin/backup.sh: line 82: syntax error near unexpected token `done' /media/bill/PROJECTS/bin/backup.sh: line 82: ` done 9< "$p_annu"' $ bash "$d_bin""backup.sh" /media/bill/PROJECTS/bin/backup.sh: line 38: null: command not found /media/bill/PROJECTS/bin/backup.sh: line 44: [: missing `]' /media/bill/PROJECTS/bin/backup.sh: line 44: false=false: command not found /media/bill/PROJECTS/bin/backup.sh: line 38: null: command not found /media/bill/PROJECTS/bin/backup.sh: line 44: [: missing `]' /media/bill/PROJECTS/bin/backup.sh: line 44: false=false: command not found >> Yuch, I had the wrong formatting on multi-conditions. change "[" to "[[" etc : if [[ $flag_error="true" || $flag_error="false" ]]; then $ bash "$d_bin""backup.sh" +---------------------------------------------------------+ /media/bill/PROJECTS/bin/standard header.sh: line 31: /media/bill/PROJECTS/bin/backupper//media/bill/SWAPPER log cumulative.txt: No such file or directory backer_rsync() - special rsync of 201013 WD_3.6Tb/201012 SWAPPER monthly_backup/, 201013 10h24m >> oops changed : partition="$d_meda""SWAPPER" to : d_src="$d_meda201013 WD_3.6Tb/201012 SWAPPER monthly_backup""/" >> OK - nowit works, but I get no list of changes to bemade. Modify the "test" options for verbose change to : if [[ $flag_error="true" || $flag_error="false" ]]; then if [ $test="yes" ]; then options="-rltguv --itemize-changes" elif [ $test="no" ]; then options="-rltgu" fi +-----+ I don't know what the problem was, or what caused it. Possibly it was open errors and subsequent freezes of QNial? nemo somehow regeerated, even if I had several times done : Menu -> View -> ReLoad ******** #] 17Sep2020 PROJECTS USB frozen!?!?! Also - system responses are MUCH slower than with the Dell64 internal hard drive!!! +-----+ https://forums.linuxmint.com/viewtopic.php?t=255009 Re: An operation is already pending Post by WharfRat » Fri Nov 30, 2018 12:32 am Hello Don Image Welcome to Linux Mint and the Linux Mint forum :) It appears that the device was registered as /dev/sdb1 It should show with lsblk or blkid Did you try mounting it from the terminal :?: Re: An operation is already pending Post by darussell » Thu Mar 14, 2019 6:33 pm blkid /dev/sdb1 returns: /dev/sdb1: UUID="1085919e-7d3c-4a40-8508-ad4b7a81c738" TYPE="ext4" PARTUID="d8d859a1-01" lsblk /dev/sdb1 returns: NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdb1 8:17 1 59.6G 0 part Re: An operation is already pending Post by Hoser Rob » Fri Mar 15, 2019 8:52 am If it is file system damage, and it's formatted ntfs, don't use Linux recovery tools. Use Windows ones instead because that format has a few of those undocumented 'features' MS loves. I use Hiren's boot disk for this, here's a link for it: https://sourceforge.net/projects/hirenscd2bootableusb/ Re: An operation is already pending Post by BG405 » Fri Mar 15, 2019 1:33 pm darussell wrote: ↑ Thu Mar 14, 2019 6:33 pm blkid /dev/sdb1 returns: /dev/sdb1: UUID="1085919e-7d3c-4a40-8508-ad4b7a81c738" TYPE="ext4" PARTUID="d8d859a1-01" This is ext4 as shown in bold, so the Linux tools will hopefully be able to fix it. How about sudo fsck -y /dev/sdb1? :) You'll have to unmount it first. +-----+ Howell's efforts : GParted : USB is /dev/sdd1 >> I unmounted the drive $ lsblk /dev/sdd1 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdd1 8:49 1 57.7G 0 part /media/bill/PROJECTS $ blkid /dev/sdd1 /dev/sdd1: LABEL="PROJECTS" UUID="a89e16bd-d559-4cb8-8cfe-004e3eb19152" TYPE="ext4" PARTUUID="dde6becb-01" $ sudo fsck -y /dev/sdd1 fsck from util-linux 2.29.2 e2fsck 1.43.4 (31-Jan-2017) PROJECTS: clean, 13020/3784704 files, 3489124/15132416 blocks >> nothing special showing? >> pull USB & re-insert Now it's not even showing in nemo!? >> close & re-open nemo, close GParted >> OK - now PROJECTS shows! (whew!!!) >> Oops - very slow response?? I need to backup, and go buy a new USB drive ******** #] 07Aug2020 special $d_HOWELL_BASE backup - find: "Structure needs cleaning" Howell : does rsync do sector-level backups? - not that I can see https://www.ubuntupit.com/top-15-best-disk-cloning-software-for-linux/ dd is Linux disk cloning command most often used? GParted - clones !!???!! - I'll use the familiar, also lessens chance of big booboo # https://unix.stackexchange.com/questions/330742/cannot-remove-file-structure-needs-cleaning +-----+ That is strongly indicative of file-system corruption. You should unmount, make a sector-level backup of your disk, and then run e2fsck to see what is up. If there is major corruption, you may later be happy that you did a sector-level backup before letting e2fsck tamper with the data. share improve this answer answered Dec 16 '16 at 2:17 DepressedDaniel +--+ Did what you suggested. e2fsck fixed some fs errors. Seems like I'm back to normal. Thx =) – Rotareti Dec 20 '16 at 20:06 +--+ I strongly suggest people to use GUIs like e.g. gparted to do this. Gparted still wraps e2fsck (in the case of checking a partition), but the graphical interface makes it a lot easier to select the proper hard-disk, as well as ensuring that you run these tools with the correct arguments. With the wrong arguments you risk severe loss of data! – Qqwy Feb 7 '19 at 22:50 +-----+ If it helps anyone, I had a similar issue (rsync/rsnapshot backups, on an affected file). I posted my problem/solution, here: https://ubuntuforums.org/showthread.php?t=2348768&p=13627299#post13627299 SUMMARY: rsnapshot (rsync) backup error on arch linux x86_64 system; a corrupted, deeply-nested file was throwing that error, also shown when I tried to delete that file: $ sudo rm -fR hourly.5/ rm: cannot remove 'hourly.5/snapshot_root/mnt/Vancouver/temp/temp - old/temp - 09 (Dec 07, 2014 - Sep 02, 2015)/a_OLD-gmail/victoria.a.stuart@gmail.com/[Gmail]/LINUX/rsync, rsnapshot; Other backups/19.bak': Structure needs cleaning Here is the problem: $ cd mnt/Vancouver/temp/temp\ -\ old/temp\ -\ 09\ \(Dec\ 07\,\ 2014\ -\ Sep\ 02\,\ 2015\)/a_OLD-gmail/victoria.a.stuart@gmail.com/\[Gmail\]/LINUX/rsync\,\ rsnapshot\;\ Other\ backups/ ls -l ls: cannot access '19.bak': Structure needs cleaning total 0 -????????? ? ? ? ? ? 19.bak ## << THAT IS THE PROBLEM!! [ See also: https://www.reddit.com/r/linuxquestions/comments/4b47r2/has_anyone_ever_gotten_structure_needs_cleaning/ ] My backup drive is /dev/sda1. sudo umount /dev/sda1 sudo fsck.ext4 /dev/sda1 ## << accepted suggested fixes Rebooted: all seems fine. Went into backups drive, deleted that problematic file: /mnt/Backups/rsnapshot_backups/hourly.5/snapshot_root/mnt/Vancouver/temp/temp - old/temp - 09 (Dec 07, 2014 - Sep 02, 2015)/a_OLD-gmail/victoria.a.stuart@gmail.com/[Gmail]/LINUX/rsync, rsnapshot; Other backups/19.bak Q.E.D.?! [Update: yes; that worked: my backups are running normally, again! :-) ] share improve this answer edited Apr 1 '17 at 21:41 answered Mar 30 '17 at 21:30 Victoria Stuart +-----+ File systems sometimes are not in order and need cleaning. This can be done by fsck command. But remember, you must run fsck only to unmounted partitions to avoid risk of file corruption. If your file system is ext4, try running this command : fsck -AR -t ext4 -y answered Apr 3 '17 at 12:41 Ayush Goyal 41411 gold badge44 silver badges99 bronze badges a list of what options do is always welcome, as we should teach people to understand what they are running instead of just copy/paste some magic command. So -A is check all fstab devices, -R will exclude the root filesystem from -A, as it is mounted readwrite and usually can't be check at this time (can be check offline or if mounted read-only) , -t ext4 to limit the list to ext4 filesystems (ignoring the others that may exist) and -y to auto-answer yes to the repair questions – higuita May 25 at 14:02 ********************** #] 31Oct2018 HOWELL_BASE Toshiba drive is getting slow!!! $ cp -r "/media/bill/HOWELL_BASE/Thunderbird/." "/media/bill/SWAPPER/Thunderbird" >> NUTS! dates not preserved! should be $ cp -rup "/media/bill/HOWELL_BASE/Thunderbird/." "/media/bill/SWAPPER/Thunderbird" cp: cannot stat ‘/media/bill/HOWELL_BASE/Thunderbird/./n4caryuo.default/lock’: Input/output error cp: cannot stat ‘/media/bill/HOWELL_BASE/Thunderbird/./n4caryuo.default/extensions.json’: Input/output error u - is not important for a "new" cp, but is a good habit Pretty shaky - almost lost my shit! Next : config Thunderbird in LMDE2 ******************** #] 17Dec2017 Corrupted USB Lexar drive https://www.maketecheasier.com/repair-corrupted-usb-drive-linux/ Removing bad blocks from the USB drive with fsck An easy way to repair a flash drive, or any drive really, is to use the fsck tool. This tool is great for removing bad file blocks, as most (if not all) corruption and unreadability comes from problems like this. To remove the corrupted file blocks from the USB flash drive, open a terminal window and enter the following commands. Users must figure out what the drive designation is on the system before things can move forward. Do this by entering the lsblk command. This will list all of the attached disks on your system. Note: the lsblk command lists all disks, not just USB drives. Pay close attention to the output, as it is easy to mistake a hard drive for a flash drive. $ lsblk -> hows that Lexar is sdd1 To remove the bad file block, run the fsck command on either a specific partition (e.g. /dev/sdc1), or the entire disk (e.g. /dev/sdc). Once completed, the USB drive will have a healthy partition again and be fully operational on Linux. $ sudo fsck /dev/sdd1 >> This ran, but it did NOT recover memory Zeroing the USB drive Sometimes a USB drive can be totally unreadable to the point where it is no longer worth saving. When this happens the best route is often to just zero out the data and start over. The best tool for the job in this situation is dd, and it works quite well. Start by taking the drive label that was found earlier with the lsblk command, and apply the same logic (remember that /dev/sdc1 is a partition, and /dev/sdc is an entire device). $ sudo dd if=/dev/zero of=/dev/sdd Zeroing a USB drive (or any device for that matter) renders the data on it totally useless. This means that you’ll need to create a new data partition. Choose a file system, and then run the command! Ext4 sudo mkfs.ext4 -f /dev/sdc1 >> Can ZERO a drive recover iso image drives? ***************** #] 29Dec2015 Seagate Momentus 7200.1 60Gb, Old Dell Laptop Latitude D620 Data recovery attempt use old SATA <-> USB drive unit (and one other that I bought with power transformer to 5 & 12V 2Amp) >> terminal window ERROR messages : "/org/freedesktop/UDisks2/drives/ST96023AS_5MG03MA4" has new interfaces: ("org.freedesktop.UDisks2.Drive") "/org/freedesktop/UDisks2/block_devices/sdc" has new interfaces: ("org.freedesktop.UDisks2.Block", "org.freedesktop.UDisks2.PartitionTable") I searched : "/org/freedesktop/UDisks2/drives/ST96023AS_5MG03MA4" has new interfaces: ("org.freedesktop.UDisks2.Drive") http://www.seagate.com/docs/pdf/datasheet/disc/ds_momentus7200.pdf pdf file "Momentus 7200.1 2.5-inch storage for workstation-class notebook PCs" OK - this is the right drive I searched : /org/freedesktop/UDisks2/drives http://udisks.freedesktop.org/docs/latest/gdbus-org.freedesktop.UDisks2.Drive.html programming technical description gparted - seems to keep the drive light flickering, stuck on something try Toshiba laptop : >> same result. Will have to take it in to check ************************** #] 29Dec2015 Drive information (manufacturer, partitions etc) - what do I use? Linux Desktop Reference - simple commands du, df not enough LMDE Software manager : gnome-disk-utility - GNOME Disks is a tool to manage disk drives and media: * Format and partition drives. * Mount and unmount partitions. * Query S.M.A.R.T. attributes. It utilizes udisks. >> Reviews : seems can cause trouble for LMDE? gparted - GParted uses libparted to detect and manipulate devices and partition tables while several (optional) filesystem tools provide support for filesystems not included in libparted. >> I installed it >> YES!! It provides hardware infor via Menu -> View -> Device Information toggle enddoc