Home Media – Part 2 – The Setup

Part 1: The NAS Build

Part 3: The Rip

Part 4: OSMC/KODI/XBMC

So what do you do if you want to build an unRAID box?

unRAID boots from a USB flash drive, and the flash drive needs to have a GUID or Globally Unique Identifier. This is for the purposes of licensing. If you do end up loosing or something happens to the flash drive generally lime-tech is pretty good about it and all you need to do is email them, they might ask for a new GUID and send you a new key. Here is their policy on that. I’ve had two flash drives fail on my and they were pretty good about giving me a replacement key. Once unRAID boots from the flash drive it mostly runs in memory, and since it’s a stripped version of Slackware Linux it doesn’t really require a lot of memory to run.

First, grab a 4GB flash drive, for best results grab a flash drive from the Hardware Compatibility page. The key here is that your flash drive needs to have a GUID, some have it and some don’t. Speed is also a factor, as in read and write speed. Alternatively if you want to know if your drive is compatible and has a GUID grab any flash drive and quick format it to FAT32, use Volume Label: UNRAID, then unzip the contents of the ZIP file found here to said drive, and boot it on any networked machine. If you’re having trouble booting follow the drive preparation instructions. Once booted log into unRAID, username:root, password:<blank>. Type in ifconfig to obtain the IP address, then head over to another networked machine and type that IP into a browser, this should take you to the main unRAID page. Here is the getting started page from Lime-Technology which describes the same thing in greater detail. Alternatively by default you can access the unRAID web GUI by typing in http://tower instead of http://<unRAID IP address>.

There is also something called the GO file which is located in the /config folder on the root of the flash drive. The go file gets executed at boot and in here you can put any special instructions or drive maps that you’d like to execute at boot. In my case I mount a drive that is outside of the array, I call it the system drive and it holds all the configurations for my docker containers. This is not necessary as the same can be accomplished with the cache drive. Here’s what my go file looks like.

#!/bin/bash
# Start the Management Utility
/usr/local/sbin/emhttp &

# Make directory and Mount the system drive
mkdir /mnt/system
mount -t reiserfs /dev/disk/by-label/systemdisk /mnt/system
sleep 5

#Install Screen and Utemper, utemper is necessary for screen to work
installpkg /boot/packages/screen-4.0.3-x86_64-4.txz
installpkg /boot/packages/utempter-1.1.5-x86_64-1.txz

The free version of unRAID (uR) allows you to use 3 drives. The Hardware compatibility list also provides some standards and minimum requirements for the server hardware, usually an old desktop will do. If you decide to go with a 8 or 25 disk server, you will need to pickup a decent RAID card and a good enclosure.

There is also something called the GO file which is located in the /config folder on the root of the flash drive. The go file gets executed at boot and in here you can put any special instructions or drive maps that you’d like to execute at boot. In my case I mount a drive that is outside of the array, I call it the system drive and it holds all the configurations for my docker containers. This is not necessary as you can to the same for the cache drive.

Plugins vs Docker. In unRAID, plugins were the go to in version 5 and prior, this was the way to install applications on your NAS. They were easy to install and worked well, until you installed a new plugin which had, let’s say a newer version of Python in it. If this happened it would break all the plugins which relied on the previous version of Python. I ran into this issue multiple times in both version 5 and 4 of unRAID. There was no standard and people wrote plugins they way it suited them. At one point in version 5 I had to go into a the plugin and change the python version it pulled and installed, this broke a minor function of the plugin but made it sort of work. There was also an issue I had with sql lite, a plugin was pulling a newer version than was supported by another application, and finally some plugins would break the web interface. When version 6 of unRAID entered beta and docker was a possibility I jumped for joy. Yes by nature the docker equivalent is larger, however since these are self contained application on top of a OS layer, there is no chance that one application would break another. Also now with docker one could have multiple versions of python ans whatever the prerequisite for an application was. The only problem with docker is that it is not as easy to implement as the plugins. It takes a little know how to get it up and running. But, since I moved everything to docker I have less downtime on my server, and I’m not as often remoting into the server to administer it. That’s a win in my books.

Most applications that you would want on your NAS more than likely already have a docker container created for them. However if you want to create your own and if you’re curious about docker in general here is a great Docker 101 tutorial video by Ken Cochrane.

I’m not going to tell you how to setup docker in unRaid or how to put together your server hardware, but what I will do is point you to some really cool articles/blogs that describe how to do so. In my opinion there is no point in reinventing the wheel.

Update: In the final version 6 of UnRAID docker container manager is included and installing docker containers is as easy as point and click for the most part. Good place to get the containers is from http://linuxserver.io

Over at the Corsair Blog there is a really cool How to Build a PC section. I hope http://www.corsair.com doesn’t mind me linking to their site. It’s intended on teaching you how plan and build an gaming PC, but there are still articles relevant to building and putting together computer hardware regardless of what the desired purpose is. If you click the link make sure you sort by Date (Older – Newer), this way you will start on the first post and you can continue on in a logical order.

The Lime-technology website has a really good tutorial on how to get docker up and running on your server. Check out their Docker Guide over at http://lime-technology.com/ . Again I hope they don’t mind that I link to their content as this is a very well written and comprehensive docker guide.

And last but not least there is a really good plugin for Docker containers in unRAID, yes I said plugin, it’s ok to have one. This plugin allows to view / add / remove unRAID docker template repositories. The alternative is adding the repositories yourself, you can get these from this lime-tech forum post.

To install the plg you need to SSH into your server, I generally use PuTTY from a Windows machine. Once there navigate to the directory /boot/config/plugins by typing the following:

cd /boot/config/plugins

Any plg file that is located in this folder will get installed on server boot. Once there you’ll need to download the plg to your local machine. Do this by typing the following:

wget –no-check-certificate https://raw.githubusercontent.com/Squidly271/repo.update/master/plugins/community.repositories.plg

This will download the plg file. Now you can either reboot the server or install it manually by typing the following:

installplg community.repositories.plg

That’s all for now. You should now see a new tab under the Docker heading in the web GUI of unRAID.

If you have any questions or comments, post below.

Next: Part 3 – The Rip

Advertisements

Home Media – Part 1 – The NAS build

This is all part of a series.

Part 2: The Setup

Part 3: The Rip

Part 4: OSMC/KODI/XBMC

Digital media and distribution is the way of the future. Netflix is doing it, iTunes is doing it, and Steam is doing it. These are three very successful examples of cloud based streaming and digital distribution services. There are many more I could have used as an example, but in my opinion these are the most successful ones out there and the most popular. While iTunes and Steam charge on a per episode or per item basis, Netflix charges a flat monthly fee for unlimited viewing. I prefer the Netflix model, unfortunately with Netflix you don’t get the content right away, instead it usually arrives after the season has ended or the Blu-Ray has been released. However Netflix does have original programming such as Daredevil, Trailer Park Boys, Lilihammer, Arrested Development, Hemlock Grove, and House of Cards. Still this requires an internet connection, and if you want to watch in Super HD a somewhat fast one at that.

netflix-logo

What if you own hundreds of DVDs, HDDVDs, and Blu-Rays as it is in my case. What if you could access this content instantly on your television and reproduce them in 1080P with high fidelity multi-channel sound, and even in 3D if you would like. Well if you have the time, money, and would like to learn something new this is all possible. That and you will also have the satisfaction of completing something cool. You could have an instant on, Movie, Television, and Music jukebox library that you can share through out your entire house.Lime_Tech_Logo

The goal here is to put together a Network Attached Storage (NAS) device which is capable of serving up the information to your network and the devices on it. These devices could include a smartphone, HTPC, or a laptop. As far as NAS devices go you have two options, either buy a ready made NAS from manufacturers such as QNAP, Synology, ASUSTOR, Drobo, NETGEAR, or if you have an old computer kicking around you can purchase a registration key for unRAID and build your own solution. The benefit of going with your own build is that you can expand it as you please and you can add hardware and hard drives as you go along. You would be future proofing by going with a custom build, as you can tailor it to your needs. It can grow with you as you require. The ready made NAS devices come in pre-configured sizes and they take a limited number of Hard Drives. One thing to note about unRAID, there are three levels of licensing for unRAID, Basic which supports 4 drives, Plus which supports up to 8 drives, and Pro which supports up to 25 drives. Prices are $29, $69 and $119 respectively.

serverlevel

As a best practice, you always want to have a parity drive in your drive array. A parity drive is a Hard Drive in a RAID array to provide fault tolerance. This will ensure that is one of your drives fails all your data will be recoverable. If one of your drives does happen to fail you should immediately replace the parity drive and rebuild it. A good practice is to always have a spare drive on hand, or hot spare as some call it. Also when you are making the initial purchase and buying multiple drives of the same make for your array, another best practice is to buy drives from different stores. This way you can ensure that the drives came from different manufacturing lines and all of them will not fail at the same time. After all you’ll probably want to store your pictures, and home movies on this rig. Always have a hot spare ready to go, it’s an empty drive that sits in your array ready to be used if another one fails, this is important as it prevents any down time or a double drive failure.

unRAID is not a RAID array, hence the name. unRAID is an array of disks with a parity drive. The parity drive holds a bit sum of all the drives in the array, and if a single disks fails the parity can rebuild said failed disk. If two disks fail, then you are SOL. Having said this you will only lose the data on the failed disks, the rest will retain their data. This is why it is important to have a hot spare on hand at all times.

If you plan on going with a pre-built device might I recommend Synology or QNAP. In my opinion these are the most popular ones on the market and they both have good communities that are willing to help you out if you hit a wall with with any settings. Also they are one of the few NAS devices that have packages for SABnzbd, Sick Beard or Sonarr, and Couch Potato. I will not get into specifics of these applications but these are some of the more popular packages.docker-logo

As of Unraid version 6 most of the packages that you would require can be installed as a docker container. Docker is a sort of Virtual Machine that runs applications in a self contained package on top of an OS layer independent of the OS that the container is running on. I find this to be a better solution than the Unraid plugins in previous versions, however you can still opt in for plugins in Unraid 6 there just isn’t a lot of them. They’re a dying breed. I would strongly advise against plugins. Why? Well, I find that the plugins have a tendency to break too often, especially if a prerequisite is changed or altered on the server, or another plugin updates it’s version of a prerequisite. With docker this is not an issue as all the prerequisites are in the container itself, and migrating to a new version of unRAID will not break your applications. Docker is also OS agnostic, meaning the same docker container can run on Linux, Windows, and Unix operating system. If you want an idea of what kind of docker packages are available for Unraid 6 just head over to the Lime-Technology forum and have a look at some of the packages that the awesome developers have designed specifically for unRAID.

Another nice feature in unRAID is that it supports virtual machines via either XEN or KVM. In the final version of unRAID there is a possibility that XEN might be dropped in favour of KVM. KVM is native to the Linux kernel and allows for easy hardware I/O passthrough. I’m currently using KVM with the assistance of a couple of plugins. I’m running an Ubuntu server with murmur installed on it, but this might be punted in light of a developer creating a docker container for it. If you wish to run KVM on unRAID you’ll need to install the VM manager and the Libvirt plugin of the server. The instructions are available here in the second post. Or you can just grab them from here:

https://raw.githubusercontent.com/dmacias72/vmMan/master/vmMan.plg

https://raw.githubusercontent.com/dmacias72/virtMan/master/virtMan.plg

As for the hardware requirements, almost any computer will do, you’ll need a flash drive, a motherboard with some sata ports and some hard drives. Realistically you could be running unRAID on your laptop if you want to, but that would defeat the purpose. I would recommend at least a 4 core processor with 8 GB or RAM, less if you do not plan of creating any Virtual Machines and 4 Hard Drive slots.

As for other hardware I recommend the Supermicro AOC-SAS2LP-MV8 8 Channel 6GB/S SAS/SATA PCI-E SAS card, and the Norco RPC-4224 24 Bay Hot Swap SATA/SAS 4U Rackmount Server Case. If you already have a tower case then you could opt in for a 5 bay drive enclosure such as the Norco SS-500 5-bay SATA/SAS hot swap rack module. This bay enclosure fits 5 standard size Hard Drives and occupies 3 x 5.25″ bays in a case. It’s a perfect fit for the smaller size setups. If you do go the route of the Norco SS-500 bay and get a SAS card you will also need a Forward SAS to SATA breakout cable.

Yes you will spend money on something like this, however it will still be cheaper and more effective than purchasing a QNAP or Synology device. Having said this there are pros and cons to both.

An unRAID home build can be cheaper than the competition, you can upgrade it as you wish and you can do it slowly. With unRAID you can scale it up or down as you need it and it can be powerful enough to run VIrtual Machines. Unlike FreeNAS or others, unRAID is easier to setup than the competition. unRAID however is not a true RAID NAS. unRAID has an array of disks that can be of varying sizes provided the parity drive is as large or larger than the largest disk in the array. If more than one disk fail in the array the data on said disks is lost, but only data on the disks that failed.

With systems like QNAP and Synology they are ready out of the box, provided you insert some disk in them. These NAS systems are not as easily scalable, however some of them do offer expansions. They can provide greater fault tolerance. These systems can cost more than a mid range computer depending on the amount of bays that you require. The cost can be as much as $1000 for a 5 bay solution. Applications are very easy to install and configure you don’t need to be a tech wizard. These systems are small, quiet, and don’t draw a lot of power. You can hide one of these under your bed if you desire.

Both are good options however I’m here to guide you through some of the hurdles involved in getting an unRAID system off the ground.

Next: Part 2 – The Setup

Things to understand about unRAID and XEN.

There seems to be a little misconception about the target audience for unRAID and XEN, so I think I’ll take a couple minutes and clarify.

unRAID is primarily aimed at the entry to mid level as far as difficulty is concerned. Also you need some technical know how and the capability to trouble shoot. This is not a Western Digital DUO or equivalent, it is not plug and play. This software is aimed at the DIY crowd, having said this you don’t need a lot of technical ability as there is a lot of documentation out there for unRAID and the community is one of the best I have ever had the privilege to be a part of. Also to note that unRAID 6 is in beta right now and as of yet it does not have any plugins written for it, much like version 4 and 5 the finished product will have these. If you do not want to use XEN, in the final product you won’t have to. Right now however since it is in beta if you want application on version 6 you need to run them in a VM such as Ubuntu and Arch Linux. unRAID can be run on very inexpensive hardware, and it can be scaled as required by the user. Some users run unRAID on Atom processors with 4GB of RAM. If you do this don’t expect stellar performance.

This is not FreeNAS. I found FreeNAS to be flaky at best, and the hardware requirements for that OS are very particular. ZFS has some nice features and it protects your data really well, however it requires ECC RAM and quite a bit of it too. Cost of FreeNAS can be greater than unRAID. I say “Can be..”, because you can throw any hardware at unRAID, be it low end or high end it really doesn’t matter. So in the end unRAID scales better and one can start out with cheap hardware, then slowly upgrade. Also with unRAID and XEN now you can have features such as the ones offered in ZFS to protect your critical data, it is just a matter of you willing to put the work in.

Each disk in the unRAID array is independent, two disks crashing will not bring down your entire array only the disks that crashed. The array is designed only for parity. Parity is used by unRAID to protect against data loss. If a drive in the array fails, the data on the other drives can be combined with the parity data to reconstruct the missing data.In general, a parity process is designed to detect a single bit change across a given set of bits, by setting the value of an added bit such that a summation across that set of bits is forced to a known value. The added bit is known as a parity bit.

Digital data is stored as a 1 or a 0. So, for example:

  • If you have 4 drives with bit values 1,1,1,1 the parity will be 0 (1+1+1+1+0=even).
  • If the bit values are 1,0,0,0 the parity will be 1 (1+0+0+0+1=even).

In unRAID, the parity bits are stored on a parity drive independent of the data drives. This parity bit works across the set of bits in the same relative bit position on each drive. So the 57th bit of the parity drive is the parity bit for the 57th bit of all of the data drives. A parity check of the 57th bit position therefore adds up all of the 57th bits of every drive including the parity drive, and checking to see total is an EVEN number. If the value returned is not EVEN, then the parity bit is toggled so that the parity check WILL return an EVEN number. unRAID uses ‘even parity’, which simply means that the summation process (using a mathematical operation called ‘exclusive OR’ or ‘XOR’) across that set of bits must return a value that is an EVEN number.

XEN hypervisor is a lot more difficult to use than any other, IMO. However the benefits of the hypervisor outweigh the effort necessary to get XEN going. If you do not need a hypervisor go with unRAID 5 or 4, the license is transferable to version 6. Why XEN? XEN is capable of running a Windows VM that you can game on, provided you have the necessary hardware and are running a HVM. So it is very robust and powerful.

Xen Hypervisor and unRAID server 6

For those of you that have home servers or are considering one, this might be of a little interest to you. Have a look at this quick post first.

First let me talk about Unraid server. Unraid server is a storage server used for media, documents and whatever else you need storage for. It’s essentially a NAS device/server. Disk access is fairly fast and generally faster than most RAID setups. Just like it’s name implies it does not use a RAID array/setup, instead it opts for an array of independently functioning drives with a parity disk. Each drive stores file data that spans only that particular drive. There are several ways that you can fill a drive up with data. When you read or write a file it does not span multiple disks. Check the gallery examples below to see the difference between a RAID5 and unRAID setup.

Images taken from Lime Tech website

What does that mean. Well for starters, unlike a RAID array if more than one drive fail you will not loose all your information. If one drive fails, with the assistance of the parity drive, much like a RAID setup you can still rebuild the information on said failed drive. If more than one drive fails however, for examples sake, if the parity drive and a single storage drive fail, you will only loose the information that was stored on said storage drive that failed. Parity holds no data only a checksum of bits from each drive in the array. Unraid can function without a parity drive. Unraid is also expandable and you can grow your array slowly over time. If you only need a 4 drive array with future expandability that’s fine, you can grow your storage pool up to 24 drives at a later date. You can increment your storage size as you please. Unraid is very flexible.

lime-logo-300x47

Unraid v5 has a repository of plugins. Plugins are applications which can be installed and configured to run on the server. Much like most NAS devices available on the market today, applications such as CrashPlan, BTsync, SabNZBD, Couch Potato, Headphones, Sick Bears, and many more can be installed in the unRAID5 environment. Give the software a go, it’s free to try and you don’t need a license for the basic version. Here is the link: http://lime-technology.com/ . The software runs off a flash drive and is accessed via a webpage interface.

So how does unRAID benefit from the Xen hypervisor? At the moment some community plugins for version 5 of unRAID are compiled with different versions of python, 2.6 and 2.7 respectively. Now if you install two of these plugins that have the varying versions of python they will conflict in the same environment. However, if you separate them in to different virtual machines this will not cause any issues. With unRAID6 one does not have to wait for a plugin to be created for the unRAID environment which is based on the slackware 14 linux distribution. Install a linux distribution of your choice or Windows if you’d like, and install the software of your choosing. Another benefit is memory allocation, it has been noted in some posts that PlexServer on unRAID does not have the best memory management around and is quite RAM hungry, sometimes crashing other applications in unRAID. With Xen that is no longer and issue, software will only crash anything inside the VM and the memory space that is allocated to it, anything outside of that VM will be untouched. It all comes down to stability and isolation.

Currently I am running Unraid Server 5.0.4 final. Over the last couple weeks I have put in a lot of research into Unraid6 with the Xen hypervisor, in the hopes of moving to the newer version. In my opinion the Xen hypervisor of Unraid 6 is the bread and butter of the new OS version. What is a Hypervisor? A Hypervisor or virtual machine monitor (VMM) is a piece of computer software, firmware or hardware that creates and runs virtual machines. A virtual machine (VM) is a software-based emulation of a computer. Why is this relevant? Well for starters you can isolate services and machines. You can have a test environment. You can consolidate your hardware into one machine. You can emulate and pass through hardware and even game on a Virtual Machine with the Xen Hypervisor. That’s right, an unRAID forum memeber, ironicbadger, successfully demonstrates a Windows VM with GPU passthrogh and the ability to game inside a Virtual Machine. Unfortunately it is not capable of multi GPU support yet.

At the moment in my home I have an unRAID5 server, a gaming PC, an HTPC, and a raspberry PI for indexing the SQL database used by XBMC. That’s 4 physical devices that could theoretically be consolidated into one piece of hardware through the use of hardware pass through and VMs. Mind you I would never emulate the Gaming PC as it is a beast, and I’m running eyefinity on it. However the potential is there. In order for hardware pass through to be possible your motherboard needs to be equipped with an input/output memory management unit or IOMMU. Essentially the motherboard chipset must support VT-d or AMD-Vi, also you need a CPU capable of handling the VT-x and AMD-V instruction sets.

The unRAID community is using Arch Linux for it’s VMs. Arch is a very minimal and light weight operating system, it can be run on one CPU and 512MB or RAM with some light applications in the background if desired. I also found that the Arch distribution has really amazing documentation and a very large community of contributors. If you want more reading on Arch go to ironicbadgers blog, he loves his Arch distribution, and no one can fault him for it, it’s a rock solid distro. http://blog.ktz.me/

There are two types of methods of virtualizing, PV and HVM. PV or a para virtualization does not require VT-x and AMD-V CPU hardware extensions and runs completely within a software driven environment. HVM or hardware assisted virtualization requires your CPU to be capable of the VT-x and AMD-V extensions. On top of this if you want to pass through any I/O hardware to the VM, like a VGA card or a PCi device, your motherboard needs to support IOMMU and have a chipset capable of the VT-d or AMD-Vi instruction set.

So if you’d like to learn some more about unRAID6 and Xen, head over to the Lime Tech website or the forums. Enjoy.