Root Mirror
From UF HPC Wiki
If you are looking to mirror the entire system, you can instead look at the instructions for a System Mirror.
Mirroring the System Disk
This assumes you have a working machine with system and that the system disk has three partitions.
- sda1, /boot
- sda2, swap
- sda3 / (i.e. root)
We also assume there is a second drive, sdb, identical to the system disk.
Partition the Mirror (sdb)
sfdisk -d /dev/sda > /var/tmp/pt.dat sfdisk /dev/sdb < /var/tmp/pt.dat
The partition table for sdb should look something like
# sfdisk -d /dev/sdb # partition table of /dev/sdb unit: sectors /dev/sdb1 : start= 63, size= 530082, Id=fd, bootable /dev/sdb2 : start= 530145, size= 16771860, Id=fd /dev/sdb3 : start= 17302005, size=473033925, Id=fd /dev/sdb4 : start= 0, size= 0, Id= 0
Note that the partition type MUST be 0xfd (Linux Raid Autodetect). If it is not, simply edit /var/tmp/pt.dat, change the "id" field to "fd", then rerun "sfdisk /dev/sdb < /var/tmp/pt.dat"
Install a Boot Block on the Mirror
grub-install /dev/sdb
Create Degraded Mirror Devices
Now use mdadm to create the SW RAID devices. Initially, we will create them using only the sdb partitions. We'll add the sda partitions after the system is up and running on the degraded mirror devices (/dev/md0, /dev/md1, /dev/md2). Note that you can destroy in existing mdadm configuration data on the disk with mdadm --zero-superblock <dev>
mdadm --zero-superblock /dev/sdb1 mdadm --zero-superblock /dev/sdb2 mdadm --zero-superblock /dev/sdb3 mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdb1 missing mdadm --create /dev/md2 --level=1 --raid-devices=2 /dev/sdb2 missing mdadm --create /dev/md3 --level=1 --raid-devices=2 /dev/sdb3 missing
Create mdadm.conf
echo "MAILADDR hpc-logs@hpc.ufhpc" > /etc/mdadm.conf echo DEVICE /dev/sda1 >> /etc/mdadm.conf echo DEVICE /dev/sdb1 >> /etc/mdadm.conf echo DEVICE /dev/sda2 >> /etc/mdadm.conf echo DEVICE /dev/sdb2 >> /etc/mdadm.conf echo DEVICE /dev/sda3 >> /etc/mdadm.conf echo DEVICE /dev/sdb3 >> /etc/mdadm.conf mdadm --examine --scan --brief /dev/sdb1 >> /etc/mdadm.conf mdadm --examine --scan --brief /dev/sdb2 >> /etc/mdadm.conf mdadm --examine --scan --brief /dev/sdb3 >> /etc/mdadm.conf
The resulting mdadm.conf file should look something like...
MAILADDR hpc-logs@hpc.ufhpc DEVICE /dev/sda1 DEVICE /dev/sdb1 DEVICE /dev/sda2 DEVICE /dev/sdb2 DEVICE /dev/sda3 DEVICE /dev/sdb3 ARRAY /dev/md1 level=raid1 num-devices=2 UUID=733652cd:727abc7f:00020178:11e2b542 ARRAY /dev/md2 level=raid1 num-devices=2 UUID=f7931fdf:e4675373:f8da53ac:69e332ac ARRAY /dev/md3 level=raid1 num-devices=2 UUID=cec80bb8:68241c03:6b05f5ea:059b2dad
Format or Initialize Each of the MD Devices
mkfs.ext3 /dev/md1 mkswap /dev/md2 mkfs.ext3 /dev/md3
Modify /etc/fstab and /boot/grub/grub.conf
Edit /etc/fstab and /boot/grub/grub.conf so that you will boot from and mount the mirror devices. Relevant entries from each are below.
Fstab Entries
/dev/md3 / ext3 defaults 1 1 /dev/md1 /boot ext3 defaults 1 2 /dev/md2 swap swap defaults 0 0
Grub.conf Entries
title CentOS (2.6.18-92.1.17.el5) root (hd0,0) kernel /vmlinuz-2.6.18-92.1.17.el5 ro root=/dev/md3 initrd /initrd-2.6.18-92.1.17.el5.img
Make a new initrd Image
Make sure your initial ramdisk image contains the SW RAID driver, raid1.
mkinitrd -v -f --with=raid1 /boot/initrd-2.6.18-92.1.17.el5.img 2.6.18-92.1.17.el5
Copy / and /boot to the MD Devices
mkdir /mnt/root mkdir /mnt/boot mount /dev/md3 /mnt/root mount /dev/md1 /mnt/boot cd /mnt/root dump 0f - / | restore rvf - cd /mnt/boot dump 0f - /boot | restore rvf -
Reboot
Reboot the system. When it comes back up you should be running on the degraded mirrors. A "df" will show something like...
Filesystem 1K-blocks Used Available Use% Mounted on /dev/md3 232804824 5647872 215331112 3% / /dev/md1 256586 29274 214064 13% /boot tmpfs 8222212 0 8222212 0% /dev/shm
Add the sda Partitions to the MD Devices
sfdisk -d /dev/sdb > /var/tmp/pt.dat sfdisk /dev/sda < /var/tmp/pt.dat mdadm --add /dev/md1 /dev/sda1 mdadm --add /dev/md2 /dev/sda2 mdadm --add /dev/md3 /dev/sda3
Check Status
Use /proc/mdstat to check the status.
# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdb1[0] sda1[1]
264960 blocks [2/2] [UU]
md2 : active raid1 sdb2[0] sda2[2]
8385856 blocks [2/1] [U_]
resync=DELAYED
md3 : active raid1 sdb3[0] sda3[2]
236516864 blocks [2/1] [U_]
[===================>.] recovery = 99.3% (234933312/236516864) finish=0.5min speed=52653K/sec
unused devices: <none>
One Last Detail
Since we want to be able to boot the machine from either disk should one of them fail (assume it is removed), we need the grub-installed boot block to reference either/both disks as "hd0" i.e. the first/only disk in the system rather than as hd0 *and* hd1. To do this we first run grub-install on /dev/sda with a device map that lists sda as grubs hd0. We then want to run grub-install on sdb but with a device map that tells grub to tread sdb as "hd0". Watch closely...
cat << EOF > /boot/grub/device.map (hd0) /dev/sda (hd1) /dev/sdb EOF grub-install /dev/sda cat << EOF > /boot/grub/device.map (hd1) /dev/sda (hd0) /dev/sdb EOF grub-install /dev/sdb
