I want to be able to boot from a live Ubuntu USB stick, and then using a rsync script permanently on that USB stick, make incremental snapshots (backups) of my main Debian system to a 2nd hard drive.
One would think that making a persistent live Ubuntu USB stick would be so easy, but most live USB makers don't allow for persistence, and those that do, haven't worked. This includes UNetBootin, even after I locally compiled and debugged the missing dependencies. So I want a simple, terminal only, solution that I can understand and debug when necessary.
The simple dd from iso, and then add casper-rw file method had problems with partitioning tools getting confused by the non-standard efi partition image. (ref)
Some solutions, like UNetBootin got close to working, but when I tried to do a sudo -s to get a root shell, they would crash and dump, and destroy my live USB image.
Most of the solution (below) comes from this great article which comes close to working, but not quite, as it needs some updates a few places. But I suggest you look at it, as it's more complete in explanation of steps.
This is for my BIOS based laptop. I'm guessing this might not work for a EUFI boot machine.
TIPS:
Don't make the grub configuration file as that article above suggests, but rather as I show below.
Once you finally have it working below, you'll find that over time the file system will become full unexpectedly. I finally figured out that this occurs because Ubuntu is silently doing
apt-get updateand this will fill up all of your file space and then you'll get a warning message. (It took me a couple of years to figure out what was going on.)
The initial fix was to do apt-get clean to dump the apt cache, but this doesn't really solve the underlying problem. Then I tried simply disabling the wi-fi so it couldn't do an automatic update. This works great for me! Now my sticks last a long time without problems.
Steps carefully tested to work:
1. Download Ubuntu 16.04 ISO image here.
2. set some variables to use below:
iso=/path/to/isoimage #e.g. iso=~/Downloads/ubuntu-16.04-desktop-amd64.iso
s=/mnt/isoimage #Source mount point for ISO files (via loop file system)
t=/media/USBRoot #Target mount point for USB files in partition #1
!! Next, BE VERY CAREFUL to correctly point to the USB stick, and not to your hard drive, with this next step, as you can accidentally overwrite your hard disk. (Tip: lookup and confirm using
lsblkor the like)dev=/dev/sd? #set the "?" to your USB drive letter, e.g. /dev/sdb
3. Plugin and optionally erase If partitioning fails below the erase is recommended. (Before when I had dd'ed an iso image to the usb device, I had to zero it out before I could make fdisk work properly again.).
sudo dd if=/dev/zero of=$dev #bs=2048 is optional and doesn't seem to matter
You'll get a message like this when it's done:
dd: writing to ‘/dev/sdb’: No space left on device
30326785+0 records in
30326784+0 records out
15527313408 bytes (16 GB) copied, 4099.2 s, 3.8 MB/s
The erase step is because I've found that sometimes a quick format won't work. I don't know for sure, but I suspect that 2nd copies of the partition map, or the like are being found and are confusing the partitioning software. So writing all zeros to the USB stick before we begin seems to make sure that you're starting fresh. But, and yes, I know, it takes LONG time to complete.
4. Make partitions. Use a partitioning tool to put a msdos type partition map on the usb stick and partition it into two partitions as follows:
Partition 1) VFAT32 partition for the kernel, ramdisk, grub, and persistence via casper-rw file. Use all of the remaining space for this.
Partition 2) bootable partition for the linux iso image. Make it about 2G in size (because that's how big the .iso file is).
The size of the #2 partition needs to be about 2g, so subtract this from the size of the drive and then divide by 512 to get the size of the #1 partition in sectors. In my case that's 16gb - 2gb = 14gb / 512 = about 27343750 sectors.
Tip: unmount the device now if it is mounted or you will get an error when you are done with fdisk. Don't use the graphical unmount button, as it seems to make the device un-findable (until the USB drive is unplugged and then replugged back in), rather use the terminal command as follows:
? Question: I'm not sure why these partitions can't be swapped, with the boot partition as the first partition. But for what it's worth, I tried that and could not get it to work.
sudo umount ${dev}1
sudo umount ${dev}2
Here's how: You can use either fdisk, parted, gparted, or cfdisk but I like fdisk.
sudo fdisk $dev
o [create partition map], and then
n (p) (1) (2048) 27343750 [new partition #1]
t c [change partition type to type c, or W95 FAT32], and
n (p) (2) (default) (default) [new partition #2]
t 2 83 [type 83 or linux], then
a (2) [set [toggle] the bootable flag], and
p [check new table], and finally
It should look something like this:
Disk /dev/sdb: 14.5 GiB, 15527313408 bytes, 30326784 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xa42995f9
Device Boot Start End Sectors Size Id Type
/dev/sdb1 2048 27343750 27341703 13G c W95 FAT32 (LBA)
/dev/sdb2 * 27344896 30326783 2981888 1.4G 83 Linux
Then do this:
w [write the partition table to the usb drive]
You should get a response like this:
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
or if you forgot to unmount it first then you will get this:
The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Device or resource busy
The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).
In which case you simply go back, unmount it as described above, and then run fdisk again and only hit w, and you're done.
5. Write the iso image to the 2nd partition. I struggled with whether to include a block size parameter or not here, (e.g. bs=2048), and in the end it appears it doesn't matter or has little effect. Also note this partition doesn't need formatting, as the ISO will effectively format it when copied directly to the partition.
sudo dd if=$iso of=${dev}2
6. Format the first partition. Note, he formats this with VFAT32 (i.e. long file names version of FAT32). I'm guessing this is to make it easy to read that partition on any machine (i.e. portability). [I plan to see if ext3 might also work and will update this later w/ that test].
sudo mkfs.vfat -F 32 -c ${dev}1 # for vFAT32 file system
7. Mount partition #1 to be able to install files into it.
sudo mkdir -p $t
sudo mount ${dev}1 $t
8. Install grub into the first sector of the usb stick and into the root directory in partition #1.
sudo grub-install --no-floppy --root-directory=$t $dev
9. Copy the kernel and ram disk files. (NOTE, this is revised from the article. The kernel now comes as an *.efi version, (but it's still just a kernel), and initrd (the ram disk with all of the packages) is now compressed with lz rather than gz). First we mount the image file (as a loop device) where we will get these two files from:
sudo mkdir -p $s
sudo mount -o loop $iso $s
sudo cp $s/casper/{vmlinuz.efi,initrd.lz} $t/boot/
10. Create a file to hold the persistent file system. Adjust the size if you want. This is set as 1024x1mb=1gb. Note that inside the casper-rw file it's an ext3 file system. The name casper-rw is magic, so don't change it.
sudo dd if=/dev/zero of=$t/casper-rw bs=1M count=1024
sudo mkfs.ext3 -F $t/casper-rw #takes a long time
11. Create a simple grub configuration file. Be sure to give it a (possibly newer?) cfg not conf file name extension so grub can find it (ref). Use your favorite editor, or use:
sudo nano $t/boot/grub/grub.cfg
12. Paste the following grub commands in.
*Notes: you can use echo and read here for more debugging if necessary. Also the "ro" "splash" and "quiet" are optional (but suggested) kernel options with mostly self explanatory behaviors. See this quite helpful post on how to use the GRUB> prompt if necessary. I'm guessing that he had a .conf file because this might have been for GRUB 1.0, rather than GRUB 2.0, and this might also explain the commands he had not working. Also note that the root is partition 1 (i.e. msdos1).
echo LOADING USB DRIVE
echo
echo To continue press any key
echo To abort press ^-alt-delete
echo
read
echo Proceeding...
set default=0
set timeout=10
set title="Ubuntu (Live)"
set root=(hd0,msdos1)
linux /boot/vmlinuz.efi boot=casper file=/preseed/ubuntu.seed persistent ro splash quiet
echo vmlinuz.efi loaded
initrd /boot/initrd.lz
echo initrd.lz loaded
echo
echo Grub done. Booting...
boot
13. sync (recommended) and cleanup (optional):
sync
sudo umount $s; sudo rmdir $s
sudo umount $t; sudo rmdir $t
14. Backup your new USB drive.
dd if=$dev of=/your/backup/location #takes a long time
To later restore it use the reverse:
dd if=/your/backup/location of=$dev #takes a long time
15. Reboot and hit a key to boot Ubuntu USB stick.
16. Add your backup script to it. Here's my incremental script which deals with two laptops. FIRST CHECK IT CAREFULLY FOR YOUR SYSTEM AND ADJUST IT AS NEEDED. Name it mybackup and hard link myrestore to it (ln mybackup myrestore). Set execution permissions with chmod u+x my{backup,restore}. Run it with ./mybackup
#!/bin/bash
#Usage:
#
# mybackup - show list of current backups
# myrestore - ''
#
# mybackup- machine: love2d or sharon-pc
# myrestore- by convention name is 'nn-descriptiveName' (so it sorts by date)
#################################################################
#################################################################
### PARTITIONS ######################
#Partition labels (also used for mount point folder names):
# Note: use labels rather than UUID as they might be more controllable.
BackupDrive='Linux backup' # USB backup drive (I removed space from 'name' & it removed it from 'label')
BackupBase="$BackupDrive/Backups" # Backup base folder directory & name
SubDir="files"
### PARTITIONS LABEL HELP:
#lsblk -o +label gives (note older method was blkid, but this suggests we use lsblk):
#NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT LABEL
#sda 8:0 0 465.8G 0 disk
#├─sda1 8:1 0 199M 0 part SYSTEM
#├─sda2 8:2 0 288.1G 0 part
#├─sda3 8:3 0 1K 0 part
#├─sda4 8:4 0 29.3G 0 part Shared
#├─sda5 8:5 0 23.3G 0 part d8root
#├─sda6 8:6 0 119.5G 0 part d8home
#└─sda7 8:7 0 5.4G 0 part [SWAP]
#sdc 8:32 0 3.7T 0 disk
#├─sdc1 8:33 0 128M 0 part
#├─sdc2 8:34 0 2.7T 0 part /media/ubuntu/Seagate Backup Plus Drive Seagate Backup Plus Drive
#└─sdc3 8:35 0 976.6G 0 part Linux backup
### MOUNTING 1of2 ######################
sudo umount "/mnt/$BackupDrive">& /dev/null # --- cleanup from prior failed attempt:
sudo mkdir "/mnt/$BackupDrive">& /dev/null; sudo mount -L "$BackupDrive" "/mnt/$BackupDrive" -o defaults,suid >& /dev/null #Allow to set user owner of files
########################################################################
### FUNCTIONS #################################################################
####################
function usage { echo; echo "Usage: ${0##*/} [machine name: Love2d|Sharon-pc] [BackupFolderName]";echo;}
####################
####################
#If parameter just show dirs for that machine, else show for both
function myls {
echo -n "'$1' existing backups:"
if [ -d "$2" ]; then echo; ls -lFgG "$2" |grep -v ^total |grep ' [0-9][0-9]-' |sed 's/..................//'; else echo ' (none)'; fi;
}
##################
function currentbackups {
if [ "$1" ]; then
myls "$1" "/mnt/$BackupBase/$1/$SubDir"
else
myls 'Love2d' "/mnt/$BackupBase/Love2d/$SubDir"
echo
myls 'Sharon-pc' "/mnt/$BackupBase/Sharon-pc/$SubDir"
echo
fi
}
###################
function badmachine { echo "Machine type '$1' is invalid.";}
###################
function cleanup {
# echo "--- cleaning up --------------------------------------"
sudo umount "/mnt/$BackupDrive"
}
########################################################################
########################################################################
### CHECK INPUTS #######################################################
#Check if backup name paramter exists:
if [ $# = 0 ]; then usage; cleanup; exit; fi
if [ $# = 1 ]; then if [ "$1" != "Love2d" -a "$1" != "Sharon-pc" ]; then badmachine "$1"; usage; else usage; currentbackups "$1"; fi; cleanup; exit; fi
if [ $# = 2 ]; then if [ "$1" != "Love2d" -a "$1" != "Sharon-pc" ]; then badmachine "$1"; usage; cleanup; exit; fi; fi
### MOUNTING 2of2 ######################
if [ "$1" = 'Love2d' ]; then
MyHome='d8home' # Love2 Debian /home partition name
MyRoot='d8root' # Love2 Debian / (root) partition name
MyShared='Shared' # Love2 Debian Shared partition name
sudo umount "/mnt/$MyRoot" >& /dev/null # --- cleanup from prior failed attempt:
sudo umount "/mnt/$MyHome" >& /dev/null #
sudo umount "/mnt/$MyShared" >& /dev/null #
sudo mkdir "/mnt/$MyRoot" >& /dev/null; sudo mount -L "$MyRoot" "/mnt/$MyRoot" >& /dev/null
sudo mkdir "/mnt/$MyHome" >& /dev/null; sudo mount -L "$MyHome" "/mnt/$MyHome" >& /dev/null
sudo mkdir "/mnt/$MyShared" >& /dev/null; sudo mount -L "$MyShared" "/mnt/$MyShared" >& /dev/null
else
# MyHome='uhome' # Love2 Ubuntu /home partition name
# MyRoot='uroot' # Love2 Ubuntu / (root) partition name
MyHome='a41eaa3e-bd31-4ebc-86d4-cf8ed5f3e779' # Love2 Ubuntu /home partition name
MyRoot='f3b7424c-0144-42a6-8488-62fbee94d245' # Love2 Ubuntu / (root) partition name
sudo umount "/mnt/$MyRoot" >& /dev/null # --- cleanup from prior failed attempt:
sudo umount "/mnt/$MyHome" >& /dev/null #
#sudo mkdir "/mnt/$MyRoot" >& /dev/null; sudo mount -L "$MyRoot" "/mnt/$MyRoot" >& /dev/null
#sudo mkdir "/mnt/$MyHome" >& /dev/null; sudo mount -L "$MyHome" "/mnt/$MyHome" >& /dev/null
sudo mkdir "/mnt/$MyRoot" >& /dev/null; sudo mount -U "$MyRoot" "/mnt/$MyRoot" >& /dev/null
sudo mkdir "/mnt/$MyHome" >& /dev/null; sudo mount -U "$MyHome" "/mnt/$MyHome" >& /dev/null
fi
#=================================================================
BackupDir="$BackupBase/$1/$SubDir/$2" # /dir/BackupFolderName
#rSync stuff:
MyRsync="sudo rsync -aAXv --delete"
RootExclude=" --exclude={\"/dev/*\",\"/lost+found\",\"/media/*\",\"/mnt/*\",\"/proc/*\",\"/run/*\",\"/sys/*\",\"/tmp/*\"}"
HomeExclude=" --exclude='*cache*'" #this does not work
if [ "${0##*/}" = "mybackup" ]; then
echo backing up...
sudo mkdir -p "/mnt/$BackupDir/root" # Making directories to save backup to
sudo mkdir -p "/mnt/$BackupDir/home"
echo "--- Backing up: / -----------------------------------"
$MyRsync $RootExclude "/mnt/$MyRoot/" "/mnt/$BackupDir/root/"
echo "--- Backing up: /home -------------------------------"
$MyRsync $HomeExclude "/mnt/$MyHome/" "/mnt/$BackupDir/home/"
if [ "$MyShared" ]; then #no shared partion on Sharon's machine
sudo mkdir -p "/mnt/$BackupDir/shared"
echo "--- Backing up: Shared ------------------------------"
$MyRsync "/mnt/$MyShared/" "/mnt/$BackupDir/shared/"
fi
else
# Confirm
read -p "YOU ARE ABOUT TO OVERWRITE YOUR PARTITIONS - CONFIRM (y/N)?" -n 1 -r; echo if [[ ! $REPLY =~ ^[Yy]$ ]]; then echo "Aborting.";exit; fi
echo;
read -p "DANGER! Really overwrite your hard disk partitions? (y/N)?" -n 1 -r; echo if [[ ! $REPLY =~ ^[Yy]$ ]]; then echo "Aborting.";exit; fi
echo "--- Restoring: / -----------------------------------"
$MyRsync "/mnt/$BackupDir/root/" "/mnt/$MyRoot"
echo "--- Restoring: /home -------------------------------"
$MyRsync "/mnt/$BackupDir/home/" "/mnt/$MyHome"
if [ "$MyShared" ]; then #no shared partion on Sharon's machine
echo "--- Restoring: Shared ------------------------------"
$MyRsync "/mnt/$BackupDir/shared/" "/mnt/$MyShared"
fi
fi
cleanup
sudo umount "/mnt/$MyRoot" ;sudo rmdir "/mnt/$MyRoot"
sudo umount "/mnt/$MyHome" ;sudo rmdir "/mnt/$MyHome"
if [ "$MyShared" ]; then #no shared partion on Sharon's machine
sudo umount "/mnt/$MyShared" ;sudo rmdir "/mnt/$MyShared"
fi
echo "=== DONE. ============================================"
exit 0
No comments:
Post a Comment