How to Generalize a Linux VM Template

When building out an environment of any kind, you need to have a good starting base, a strong foundation and the same holds true for many things. It’s no different when building a virtual machine (VM) template to deploy in your environment in a repeatable fashion.

Recently I spent some time developing a generalized/sanitized VM template for Red Hat Enterprise Linux (RHEL) versions 6 and 7. This script should be run right before you shutdown the VM and template it for reuse.

#!/bin/bash
echo "Generalize the template..."
echo "Remove RHN system ID..."
rm -f /etc/sysconfig/rhn/systemid

echo "Create sanitized ifcfg-eth0..."
echo -en 'DEVICE=eth0\nTYPE=Ethernet\nONBOOT=yes\nBOOTPROTO=dhcp\n' > /etc/sysconfig/network-scripts/ifcfg-eth0

echo "Clear /etc/sysconfig/network file..."
cat > /etc/sysconfig/network < /dev/null

echo "Remove SHH keys..."
rm -f /etc/ssh/ssh_host_*

echo "Remove udev rules..."
rm -f /etc/udev/rules.d/70-*

echo "Remove fixed hostname..."
rm -f /etc/hostname

echo "Clear Machine ID (SID)..."
> /etc/machine-id

echo "Remove all logs from /var/log..."
rm -rf /var/log/*

echo "Remove all logs from /root..."
rm -rf /root/*
Advertisements

Things to understand about unRAID and XEN.

There seems to be a little misconception about the target audience for unRAID and XEN, so I think I’ll take a couple minutes and clarify.

unRAID is primarily aimed at the entry to mid level as far as difficulty is concerned. Also you need some technical know how and the capability to trouble shoot. This is not a Western Digital DUO or equivalent, it is not plug and play. This software is aimed at the DIY crowd, having said this you don’t need a lot of technical ability as there is a lot of documentation out there for unRAID and the community is one of the best I have ever had the privilege to be a part of. Also to note that unRAID 6 is in beta right now and as of yet it does not have any plugins written for it, much like version 4 and 5 the finished product will have these. If you do not want to use XEN, in the final product you won’t have to. Right now however since it is in beta if you want application on version 6 you need to run them in a VM such as Ubuntu and Arch Linux. unRAID can be run on very inexpensive hardware, and it can be scaled as required by the user. Some users run unRAID on Atom processors with 4GB of RAM. If you do this don’t expect stellar performance.

This is not FreeNAS. I found FreeNAS to be flaky at best, and the hardware requirements for that OS are very particular. ZFS has some nice features and it protects your data really well, however it requires ECC RAM and quite a bit of it too. Cost of FreeNAS can be greater than unRAID. I say “Can be..”, because you can throw any hardware at unRAID, be it low end or high end it really doesn’t matter. So in the end unRAID scales better and one can start out with cheap hardware, then slowly upgrade. Also with unRAID and XEN now you can have features such as the ones offered in ZFS to protect your critical data, it is just a matter of you willing to put the work in.

Each disk in the unRAID array is independent, two disks crashing will not bring down your entire array only the disks that crashed. The array is designed only for parity. Parity is used by unRAID to protect against data loss. If a drive in the array fails, the data on the other drives can be combined with the parity data to reconstruct the missing data.In general, a parity process is designed to detect a single bit change across a given set of bits, by setting the value of an added bit such that a summation across that set of bits is forced to a known value. The added bit is known as a parity bit.

Digital data is stored as a 1 or a 0. So, for example:

  • If you have 4 drives with bit values 1,1,1,1 the parity will be 0 (1+1+1+1+0=even).
  • If the bit values are 1,0,0,0 the parity will be 1 (1+0+0+0+1=even).

In unRAID, the parity bits are stored on a parity drive independent of the data drives. This parity bit works across the set of bits in the same relative bit position on each drive. So the 57th bit of the parity drive is the parity bit for the 57th bit of all of the data drives. A parity check of the 57th bit position therefore adds up all of the 57th bits of every drive including the parity drive, and checking to see total is an EVEN number. If the value returned is not EVEN, then the parity bit is toggled so that the parity check WILL return an EVEN number. unRAID uses ‘even parity’, which simply means that the summation process (using a mathematical operation called ‘exclusive OR’ or ‘XOR’) across that set of bits must return a value that is an EVEN number.

XEN hypervisor is a lot more difficult to use than any other, IMO. However the benefits of the hypervisor outweigh the effort necessary to get XEN going. If you do not need a hypervisor go with unRAID 5 or 4, the license is transferable to version 6. Why XEN? XEN is capable of running a Windows VM that you can game on, provided you have the necessary hardware and are running a HVM. So it is very robust and powerful.

Xen Hypervisor and unRAID server 6

For those of you that have home servers or are considering one, this might be of a little interest to you. Have a look at this quick post first.

First let me talk about Unraid server. Unraid server is a storage server used for media, documents and whatever else you need storage for. It’s essentially a NAS device/server. Disk access is fairly fast and generally faster than most RAID setups. Just like it’s name implies it does not use a RAID array/setup, instead it opts for an array of independently functioning drives with a parity disk. Each drive stores file data that spans only that particular drive. There are several ways that you can fill a drive up with data. When you read or write a file it does not span multiple disks. Check the gallery examples below to see the difference between a RAID5 and unRAID setup.

Images taken from Lime Tech website

What does that mean. Well for starters, unlike a RAID array if more than one drive fail you will not loose all your information. If one drive fails, with the assistance of the parity drive, much like a RAID setup you can still rebuild the information on said failed drive. If more than one drive fails however, for examples sake, if the parity drive and a single storage drive fail, you will only loose the information that was stored on said storage drive that failed. Parity holds no data only a checksum of bits from each drive in the array. Unraid can function without a parity drive. Unraid is also expandable and you can grow your array slowly over time. If you only need a 4 drive array with future expandability that’s fine, you can grow your storage pool up to 24 drives at a later date. You can increment your storage size as you please. Unraid is very flexible.

lime-logo-300x47

Unraid v5 has a repository of plugins. Plugins are applications which can be installed and configured to run on the server. Much like most NAS devices available on the market today, applications such as CrashPlan, BTsync, SabNZBD, Couch Potato, Headphones, Sick Bears, and many more can be installed in the unRAID5 environment. Give the software a go, it’s free to try and you don’t need a license for the basic version. Here is the link: http://lime-technology.com/ . The software runs off a flash drive and is accessed via a webpage interface.

So how does unRAID benefit from the Xen hypervisor? At the moment some community plugins for version 5 of unRAID are compiled with different versions of python, 2.6 and 2.7 respectively. Now if you install two of these plugins that have the varying versions of python they will conflict in the same environment. However, if you separate them in to different virtual machines this will not cause any issues. With unRAID6 one does not have to wait for a plugin to be created for the unRAID environment which is based on the slackware 14 linux distribution. Install a linux distribution of your choice or Windows if you’d like, and install the software of your choosing. Another benefit is memory allocation, it has been noted in some posts that PlexServer on unRAID does not have the best memory management around and is quite RAM hungry, sometimes crashing other applications in unRAID. With Xen that is no longer and issue, software will only crash anything inside the VM and the memory space that is allocated to it, anything outside of that VM will be untouched. It all comes down to stability and isolation.

Currently I am running Unraid Server 5.0.4 final. Over the last couple weeks I have put in a lot of research into Unraid6 with the Xen hypervisor, in the hopes of moving to the newer version. In my opinion the Xen hypervisor of Unraid 6 is the bread and butter of the new OS version. What is a Hypervisor? A Hypervisor or virtual machine monitor (VMM) is a piece of computer software, firmware or hardware that creates and runs virtual machines. A virtual machine (VM) is a software-based emulation of a computer. Why is this relevant? Well for starters you can isolate services and machines. You can have a test environment. You can consolidate your hardware into one machine. You can emulate and pass through hardware and even game on a Virtual Machine with the Xen Hypervisor. That’s right, an unRAID forum memeber, ironicbadger, successfully demonstrates a Windows VM with GPU passthrogh and the ability to game inside a Virtual Machine. Unfortunately it is not capable of multi GPU support yet.

At the moment in my home I have an unRAID5 server, a gaming PC, an HTPC, and a raspberry PI for indexing the SQL database used by XBMC. That’s 4 physical devices that could theoretically be consolidated into one piece of hardware through the use of hardware pass through and VMs. Mind you I would never emulate the Gaming PC as it is a beast, and I’m running eyefinity on it. However the potential is there. In order for hardware pass through to be possible your motherboard needs to be equipped with an input/output memory management unit or IOMMU. Essentially the motherboard chipset must support VT-d or AMD-Vi, also you need a CPU capable of handling the VT-x and AMD-V instruction sets.

The unRAID community is using Arch Linux for it’s VMs. Arch is a very minimal and light weight operating system, it can be run on one CPU and 512MB or RAM with some light applications in the background if desired. I also found that the Arch distribution has really amazing documentation and a very large community of contributors. If you want more reading on Arch go to ironicbadgers blog, he loves his Arch distribution, and no one can fault him for it, it’s a rock solid distro. http://blog.ktz.me/

There are two types of methods of virtualizing, PV and HVM. PV or a para virtualization does not require VT-x and AMD-V CPU hardware extensions and runs completely within a software driven environment. HVM or hardware assisted virtualization requires your CPU to be capable of the VT-x and AMD-V extensions. On top of this if you want to pass through any I/O hardware to the VM, like a VGA card or a PCi device, your motherboard needs to support IOMMU and have a chipset capable of the VT-d or AMD-Vi instruction set.

So if you’d like to learn some more about unRAID6 and Xen, head over to the Lime Tech website or the forums. Enjoy.