System mirror
From UF HPC Wiki
|
If you are only looking to mirror the root file system, you can do so on the Root Mirror page.
In order to create a RAID-1 Mirror of a system whose disk already exists, and you are now adding a new disk to the system, you need to do the following...
mdadm
The following assumes you have mdadm (the software raid tools) installed and working.
Create Partitions
The first thing to do is to create partitions on the new disk for the directory structure that you want in this new mirrored system. Remember, this is a chance for you to change some things if the structure of the old system was a bit off.
For instance, I have a system right now that would benefit from this because the root partition is a bit too small and I keep filling it. I would love to rebuild the machine so that the root partition is bigger so that this no longer happens.
So, create the new partition structure on the new drive, using whatever partitioning software you like. I tend to do things the old fashioned way and use fdisk, which for a lot of people is not easy to work with because there is so little to go on, but I am accustomed to it so that is what I use. The important thing here is that you get the new partition structure installed on the disk.
IMPORTANT: When setting the type of partitions on the new disk, you need to set them to "Linux raid autodetect", which is filesystem type "fd".
When you are done, the partition table should look something like this:
Command (m for help): p Disk /dev/sda: 160.0 GB, 160041885696 bytes 255 heads, 63 sectors/track, 19457 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 13 104391 fd Linux raid autodetect /dev/sda2 14 262 2000092+ fd Linux raid autodetect /dev/sda3 263 19457 154183837+ fd Linux raid autodetect
Create RAID device
Now it is time to create the RAID devices on the new drive. For the above partitions we have three different partitions that need to be created:
mdadm --create /dev/md0 --level=1 --raid-disks=2 missing /dev/sda1
The missing nomenclature in this command lets the mdadm command know that there is a drive missing in the array.
If you receive an error informing you that the /dev/md0 device is missing (or /dev/md1, etc.) then you will need to create this device. This is not really a problem, just an extra step. First you need to find out just what kind of device it is, and what the major and minor numbers are on the device:
[root@test ~]# file /dev/md0 /dev/md0: block special (9/0)
This shows us that it is a block special device with major number 9 and minor number 0. Looking at /dev/md1, we see the following:
[root@test etc]# file /dev/md1 /dev/md1: block special (9/1)
From this we can gather that as the number of the md device increases, so does the minor number. So, we now need to create that device:
[root@test etc]# mknod /dev/md0 b 9 0
Be sure to substitute the appropriate device number and minor number for your situation.
Format RAID Device
Next you need to format the RAID device. Depending on what you are using for a filesystem, this may differ. In our case we use ext3, so that is what we will use...
mkfs.ext3 /dev/md0
Also, if you have swap devices, don't forget to create those as well:
mkswap /dev/md1
Copy system
Create a mount point:
mkdir /mnt/md0
Mount the RAID device that you want to copy to:
mount /dev/md0 /mnt/md0
Copy the system files over:
cp -axu / /mnt/md0
Note that you will have to do this for each partition that you have on your original disk.
Edit /etc/fstab so that you mount your new RAID partition on boot:
/dev/md0 /mnt/md0 ext3 defaults 0 0
Reboot and check to see if the RAID partition mounted.
Reboot to RAID device
Build new RAID initrd
We need to create a new initrd image that has the md devices in it. To do this, we do the following:
mkinitrd -v -f --with=raid1 /boot/initrd-2.6.18-8.1.8.el5.img 2.6.18-8.1.8.el5
This overwrites the current initrd with a new version, includes the modules for raid1 and md in case they were not in there, and also writes into the initrd file the fact that we have an md device installed and running at this time. This is important because without the initrd knowing that there is an md device installed, it will fail to boot properly!
Update grub boot menu
Edit /boot/grub/menu.lst (or /boot/grub/grub.conf, depending on distribution. These two are usually symlinked to each other) and add a new entry:
title RAID Kernel root (hd0,0) kernel /boot/vmlinuz-2.6.18-8.1.8.el5 root=/dev/md0 ro initrd /boot/initrd-2.6.18-8.1.8.el5.img savedefault boot
This will provide you with a safety net in case something doesn't work. You will be able to reboot with the old entry without trouble.
Edit fstab
You also need to edit the new fstab on the new drive. If the original entry for the root device was something like the following:
/dev/sda1 / ext3 defaults 0 0
You will need to change it to something like this:
/dev/md0 / ext3 defaults 0 0
Remember that you need to edit the fstab on the new drive (mounted on /mnt/md0 or wherever) instead of the one on the current booting device. Reboot and verify that you are now mounting the md devices as the proper drives.
Reformat first drive
Now that we are booting off of the other drive, we can do one of the following:
- Reformat the old drive and image
- Replace the old drive and image
Either one works, so it makes little difference here.
First, wipe out the partition information that is currently on the old disk. It is probably easiest if you simply copy the partition data from the new disk to the old disk so that everything is the same:
sfdisk -d /dev/sda | sfdisk /dev/sdb
Prior to doing that, you may want to check and ensure that the output from sfdisk -d /dev/sda is exactly what you want to go on /dev/sdb. Wiping out your partition tables at this time would not be a good thing to do.
Add first disk to RAID
Now we need to add the first disk back into the RAID. To do this, we go back to mdadm:
mdadm --add /dev/md0 /dev/sda1
Go ahead and do this for each of the raid devices you are creating. You can monitor the status of the resync of the arrays by doing the following:
cat /proc/mdstat
Once the arrays are synced completely, rerun mkinitrd so that the system does not think that the drives are degraded:
mkinitrd -v -f --with=raid1 /boot/initrd-2.6.18-8.1.8.el5.img 2.6.18-8.1.8.el5
Reboot and check to ensure that the array is fully up.
Add grub to disks
You now need to add grub to the disks that do not have it installed:
grub grub> device (hd0) /dev/sda grub> root (hd0,0) grub> setup (hd0)
Debian
Its a good idea to actually copy the original images as backup first before changing anything and just simply modifying grubs menu.lst file as you go.
For a Debian system, you have to use initramfs-tools instead of mkinitrd. Additionally, make sure when you run the copy command above when copying the disks to do it in single user mode (otherwise /dev won't get copied; if this happens you can just stop udev, copy it over again, and restart it)
To use initramfs-tools with the raid/md modules, you have to specify it in the modules file in:
/etc/initramfs-tools/modules
So this should set it up so initramfs-tools uses the right modules when you make an image:
echo raid1 >> /etc/initramfs-tools/modules echo md >> /etc/initramfs-tools/modules
The mdadm.conf file should contain at the least the definitions of the arrays, i.e.:
# definitions of existing MD arrays ARRAY /dev/md0 level=raid1 num-devices=2 UUID=955fb73d:8be56149:671e4e06:cea18c64 ARRAY /dev/md1 level=raid1 num-devices=2 UUID=917601a6:49169563:cf605443:e1ec83a9 ARRAY /dev/md2 level=raid1 num-devices=2 UUID=5a26bcc2:c34f9039:671e4e06:cea18c64
This output is easily retrieved through the command:
mdadm --examine --scan
NOTE: When you check the status of the arrays during the setup like above, they won't say active until their first write, so for the swap, it may say read-only filesystem when you mdadm query the drives, because it hasn't written to it yet.
Like above, the fstab file should look something like this:
/dev/md0 /boot/ ext3 defaults 1 2 /dev/md1 swap swap defaults 0 0 /dev/md2 / ext3 defaults 1 1
When installing grub on the secondary HD the following commands are SUPPOSED to install it in the right directory but it may not:
grub-install --root-directory=/boot /dev/hda
Make sure to check that grub was installed in /boot and furthermore that the /boot/grub/device.map contains lines for both harddrives that match what you want to edit in the /boot/grub/menu.lst
GRUBs menu.lst file might look like this (notice the second entry contains md=0)
## ## End Default Options ## title Debian GNU/Linux, kernel 2.6.24-1-amd64 - RAID root (hd1,0) kernel /vmlinuz-2.6.24-1-amd64 root=/dev/md2 ro notsc numa=off initrd /initrd.img-2.6.24-1-amd64 savedefault boot title Debian GNU/Linux, kernel 2.6.24-1-amd64 - RAID Recovery root (hd1,0) kernel /vmlinuz-2.6.24-1-amd64 root=/dev/md2 ro md=0,/dev/hdc1 initrd /initrd.img-2.6.24-1-amd64 boot title Debian GNU/Linux, kernel 2.6.24-1-amd64 root (hd1,0) kernel /vmlinuz-2.6.24-1-amd64 root=UUID=5163917a-925a-480c-88f7-55f894505034 ro quiet initrd /initrd.img-2.6.24-1-amd64.premd title Debian GNU/Linux, kernel 2.6.24-1-amd64 (single-user mode) root (hd1,0) kernel /vmlinuz-2.6.24-1-amd64 root=UUID=5163917a-925a-480c-88f7-55f894505034 ro single initrd /initrd.img-2.6.24-1-amd64.premd
