About Nerd Drivel

Knowledge is power. Empower yourself! I've been a computer nerd since I was 7 years old. I started in the days of the Power PC with a Commodore 64. As I grew so did my knowledge and curiosity, anything I could take apart with a screw driver would be opened and investigated. Later on I went on to graduate from a post secondary Computer Engineering program. Today I work in an IT department for a mid sized company, I get to tinker and toy with gadgets of all types, fuelling my passion for technology and software. I understand computers more than I understand some people.

HDR is here, sigh…

First, here is a good quick article on the difference of 8-bit vs 10-bit colour. Give it a quick one, two. Thumbs up the guy that took the time to write that.

My opinion is as follows.

First HDR should not be called HDR. It should be called Colour+ or something. Because it’s not what traditional HDR is or does. There are 3 HDR formats as of this writing, HDR, HDR10+ and, Dolby Vision.

HDR offers a singular benefit which is the colour depth, but as is with Dolby and Samsungs HDR10+ everything else about it is a gimmick. Just like Curved screens, and the likes of 8K Televisions. Perhaps 8K is useful for larger cinema screens but to me as a consumer 8K won’t offer any value in the near future. HDR offers a singular benefit, this is the wider colour gamut or depth(10 bit vs 8 bit). This has been a long ways in the making and started in the days of DVD past.

While yes the capability of the wider colour gamut or depth is welcome, things such as changing meta data is riding on borderline gimmicky and stupid. Also I would prefer there be no changing meta data in my video streams, thank you very much. I’d rather my Television have less capability to spy on me than they already do. I’m looking at you Samsung. I don’t own any Samsung products by the way, and there is a good reason for that, explosions and spying aside, I don’t buy into what they sell. While LG has adopted HDR and Dolby Vision(Netflix), Samsung has opted for HDR and HDR10+(Amazon). Amazon opted to adopt the HDR10+ standard.

I have experienced both Netflix and Amazon HDR, except for HDR10+ on Amazon. Based on my experience I will say this; the quality of Netflix and Dolby Vision leaves something to be desired. While it can offer benefits for edge lit LCD panels I see no benefit for OLED TVs. The idea behind Dolby Vision is such that it adjusts the contrast and brightness levels so that one can see the darks or shadows in a scene better and with greater detail than before. Is this true? Yes, somewhat but it comes at a price. To me it reeks of gimmick that only Edge Lit LCD panels benefit from. Let me explain. The problem with edge lit LCDs is that if a single pixel on the screen needs to be lit, and depending on the TV weather it has local dimming zones or not, the entire screen or sections of it will be lit up with the back light to provide illumination to the single pixel. LCD pixels are a passive technology and do not emit light. That is not the case with OLED. This means that on LCD/LED/QLED panels you will see a beam of light dropping down from the top to the bottom for a single pixel on an edge lit LCD panel, or LED panel as the manufacturers like to call them. QLED does something similar as well. While Dolby Vision aims to minimize this. Testing DV(Dolby Vision) on the LG 65UH8500 which is an LED edge lit LCD TV, this effect is very noticeable. Dolby Vision somewhat corrected this problem, but I also noticed that the contrast, the whites and, overall brightness of the image suffered in DV. Having watched and tested many DV shows, it was all the same across the board. Most noticeably in Luke Cage, DV rendered the image and show unwatchable and I had to make adjustments to the DV settings. Note that this was a calibrated television screen. In Marco Polo it did improve the dark scenes, however during the bright scenes I noticed that the colours and whites were way off. The whites weren’t white anymore and colour had a dark brown tint to it. But then again Netflix streaming quality is nothing to rave about. While Amazon streaming quality is slightly better than Netflix, this is only true for their UHD stuff. Also Amazon’s HDR implementation is a little simpler than DV and having watched the Grand Tour in HDR I can say, it’s just ok. HDR is simple and does what it’s supposed to without gimmicks. It adds a larger colour gamut, while HDR10+ and Dolby Vision add changing meta data to the stream. Thanks but no thanks.

The likes of 4 HDR standards exist right now, or soon will. Each vying for TV supremacy.
This smells to me like the next format war, Beta Max vs VHS, HD-DVD vs Blu-Ray, well Dolby Vision vs HDR10+ is here. So get ready folks, get ready to stop buying new TVs because a new format war is about to come to your living room.

Also did you know that that UHD television in your living room will not display UHD Television content for a long time, if ever, except for a couple channels. For the most part TV producers opted for HDR instead of UHD resolution. On top of that, said live television HDR format is a new standard which has not been added to any TV manufacturers lineups yet. Planned obsolescence? Maybe. Who knows maybe it’s just a firmware update, but if you’re dreaming of watching sports in UHD, think again, it might not ever happen. Yes Direct TV streamed some UHD channels last year, but have a look around, most Television broadcasts are still in 720p or 1080i, nothing has changed in the last little while.

So as it stands right now the only benefit HDR offers is via UHD-BluRay. You get uncompressed HDR video with wider colour range and uncompressed audio. However, UHD-BluRays aren’t flying off the shelves either. I have a sneaking suspicion that UHD BluRay will go the way of BetaMax, HD-DVD, and the Dodo. As such I could not recommend a new LCD panel that supports HDR at the moment to anyone. Let alone UHD LCD panel, that is unless you plan to use it with a PC. There really is no benefit at the moment to the average consumer purchasing a UHD TV unless you are buying a large format screeen, 80″ + I’d say. Be patient, sit and wait, watch, and then make an informed purchase.

If you’re are going to upgrade your LCD panel do it because you’re upgrading it to an OLED TV. That is the single best upgrade you can make to your TV, you will be stunned at the difference between OLED and LCD. Once you look at OLED you will realize how trashy LCD, LED and, Samsung’s QLED panels really are. This is truly the future of Television displays, and Goggles rumored $800 Million investment in LGs OLED technologies is proof of that. Christ, go to any robotics hobby shops and they are stocking OLED monochrome displays, some even colour. OLED is the future display tech for the time being. QLED or Quantum dot Light Emitting Diodes are just LCD panels with a gimmicky name to try and fool you into buying Samsung’s panels which are LCD/LED edge lit displays of the last 10 years past.

OLED or go home I say. This is what the industry should be concentrating on.

References:

http://www.techhive.com/article/171223/10_bit_color.html

http://www.digitaltrends.com/home-theater/directv-4k-uhd-masters-broadcast/

http://www.cnbc.com/2017/04/10/google-offers-at-least-880-mln-to-lg-display-for-oled-investment-electronic-times.html

http://www.trustedreviews.com/opinions/hdr-tv-high-dynamic-television-explained

https://en.wikipedia.org/wiki/Ultra-high-definition_television#Americas

Whatcha talkin’ bout? FOO! Setting up proper fail over in a Cluster.

Here is what a 2 node fail over cluster should look like. Double network redundancy on the back end with each node and the SAN connecting to both back end switches.

selection_046

In order to setup proper cluster fail over the Server needs to be set to Fail Over Only (FOO). Remember that Windows has iSCSI volume size restrictions. So when you create volumes and LUNs on you SAN you need to limit the size. See link at end of article.

To do this you need a couple things, first you need to connect the iSCSI connection to both servers. Some SAN manufacturers have their own DMS drivers, which are usually a modified version of the Microsoft DSM driver. HP actually recommends using the Microsoft DSM.

The proper DSM is required in order to setup proper MPIO (Multipath Input and Output) on a cluster.

In short MPIO, is the multipath interconnect necessary for failover, it uses the DSM driver to achieve this. Generally the DSM driver is provided by the OS vendor, in this case Microsoft. This is also the HP recommended method of connecting to the SAN from Microsoft Windows, and most other manufacturers also use the Microsoft DSM driver.

Map iSCSI connections

First we need to properly map the iSCSI connections. Be aware that you will be mapping the same connection multiple times, this is necessary for failover. In the above example each server has 4 connections. 2 for the 20 subnet and 2 for the 30 subnet. Open up iSCSI Initiator and select the Discovery tab.

In the discovery tab add all 4 IP destinations, x.x.20.110, x.x.30.110, x.x.20.111, x.x.30.111.

Click the Discover Portal… button and add each one of those connections.

Next select the Targets tab, you should see the inactive iSCSI connection here.

Highlight the connection and click Connect. The HP SAN is setup with a single iSCSI connector and multiple LUNs. Some devices have multiple iSCSI connectors with a single LUN on each. Depending on the setup you might have to do this to each connector.

A Connect To Target window will pop up, check off Enable multi-path and click Advanced.

Under Local adapter select Microsoft iSCSI Initiator, for the Initiator IP select the IP for the Server, and the Target Portal IP should be one of the two IPs on the same subnet as the Initiator IP. It should look like the following.

selection_047

Now repeat these steps, highlight the same connection, click Connect, check of Enable multi-path, click Advanced…, rinse and repeat, this will map the other three connections.

10.10.20.4 -> 10.10.20.111, 10.10.30.6 ->  10.10.30.110, 10.10.30.6 -> 10.10.30.111.

If you click the Favourite targets tab you should see 4 similar targets. These are all the connections you just created for the one iSCSI target(iqn).

Set up Connection Fail Over

Next start the MPIO applet, Start > Run > mpiocpl your Vendor should be listed in the Devices: window. If it is not you will need to add it via the Discover Multi-Paths tab, others window. Highlight the Device Hardware and click Add. Say no to the reboot.  

selection_048

Next in the SPC-3 compliant window check off Add support for iSCSI devices and click Add. You will again be prompted to reboot. This time do so.

selection_049

If you run the command mpclaim -s -d in an admin CMD session you should see the connection now.

selection_050

Back in the iSCSC Initiator applet, if you highlight the iqn connection on the Targets tab and click Properties, Devices, and MPIO, you should see the Load Balance policy and all the paths that this connection can fail over to.

Your load balance policy will initially default to Round Robin change this to Fail Over Only. If you do this all but one connection should set to Active, all others will go into Standby. Click Apply.

Don’t worry if the connections don’t go into standby, just make sure that FOO is applied. Sometimes with multiple mapped disks this can happen.

selection_051

Now is you run the same mpclaim command your LB Policy should be changed to FOO (Fail Over Only). You will need to do this for each mapped disk.

selection_052

To change the Load Balancing policy to FOO run mpclaim with the -L and -M switch.

mpclaim.exe -L -M 1

The one at the end is indicative of a FOO LB policy, if a connection fails it will immediately fail over to the next one. This is for always on high demand systems.

Now if you run the -s -d switches you should see FOO under the LB policy.

selection_053

Now go into Control Panel > Administrative Tools > Computer Management and bring the iSCSI disk online and format them to NTFS. I had an instance where the disk wouldn’t come online even when I brought it online. If this is the case resize your LUN disks, they are too large.

Mpclaim determines the policy for the iSCSI connection. For more information on mpclaim go to Microsoft’s website and user the following reference https://technet.microsoft.com/en-us/library/ee619743(v=ws.10).aspx

https://technet.microsoft.com/en-us/library/dd851699(v=ws.11).aspx

iSCSI and VHD/VHDX volume size restrictions.

The age of telemetry and online tracking.

Telemetry has been around for a while, Windows 7 had it, and I believe even Vista might have had it.

Having said this Microsoft did not previously sell this information to third parties not did it previously include personal information. When I say personal information I mean your age gender, approximate location, and online habits. Legally they can not give your name, address, and other details that would allow someone to pin point you in real life. So instead they are selling as much as they can about you without actually telling them who you really are. Mind you Micrrosoft is not the only company that uses telemetry data, recently Nvidia got caught with their hand in the cookie jar as well. Scary, right?

Windows:

Fret not where there is a will there is a way. If like me you spend a lot of time connected to the online world there should be a few applications that you need to explore.

For Windows 10 there are a couple of Telemetry applications that you can install that will kill the services and registry entries that allow Windows to send your data to big brother.

O&O Software has an Antispy-Tool for Windows 10. This is a free download. It’s called O&O ShutUp10. With this tool you can pick and choose in great detail what you want to block and what you want to allow.

03

The folks over at Safer Networking that brought us Spybot Search and Destroy have also built a similar tool. They call their Telemetry removal tool Spybot Anti-Beacon, the link can be found on their download page. This has a portable version so you can run it off a flash drive if you need to.

capture

The first time you run this it will show you what you have blocked and what is still tracking you. If you click the “Show Options” button it will show you the registry keys that will be changed. Click Immunize and restart your computer.

Besides these tools, I have also edited my hosts file and added a bunch of domain names that direct to 0.0.0.0, essentially to nowhere. Spybot also adds a couple hosts entries into the file.

0.0.0.0 a.ads1.msn.com
0.0.0.0 a.ads2.msads.net
0.0.0.0 a.ads2.msn.com
0.0.0.0 a.rad.msn.com
0.0.0.0 a-0001.a-msedge.net
0.0.0.0 a-0002.a-msedge.net
0.0.0.0 a-0003.a-msedge.net
0.0.0.0 a-0004.a-msedge.net
0.0.0.0 a-0005.a-msedge.net
0.0.0.0 a-0006.a-msedge.net
0.0.0.0 a-0007.a-msedge.net
0.0.0.0 a-0008.a-msedge.net
0.0.0.0 a-0009.a-msedge.net
0.0.0.0 ac3.msn.com
0.0.0.0 ad.doubleclick.net
0.0.0.0 adnexus.net
0.0.0.0 adnxs.com
0.0.0.0 ads.msn.com
0.0.0.0 ads1.msads.net
0.0.0.0 ads1.msn.com
0.0.0.0 aidps.atdmt.com
0.0.0.0 aka-cdn-ns.adtech.de
0.0.0.0 a-msedge.net
0.0.0.0 az361816.vo.msecnd.net
0.0.0.0 az512334.vo.msecnd.net
0.0.0.0 b.ads1.msn.com
0.0.0.0 b.ads2.msads.net
0.0.0.0 b.rad.msn.com
0.0.0.0 bs.serving-sys.com
0.0.0.0 c.atdmt.com
0.0.0.0 c.msn.com
0.0.0.0 cdn.atdmt.com
0.0.0.0 cds26.ams9.msecn.net
0.0.0.0 choice.microsoft.com
0.0.0.0 choice.microsoft.com.nsatc.net
0.0.0.0 compatexchange.cloudapp.net
0.0.0.0 corp.sts.microsoft.com
0.0.0.0 corpext.msitadfs.glbdns2.microsoft.com
0.0.0.0 cs1.wpc.v0cdn.net
0.0.0.0 db3aqu.atdmt.com
0.0.0.0 df.telemetry.microsoft.com
0.0.0.0 diagnostics.support.microsoft.com
0.0.0.0 ec.atdmt.com
0.0.0.0 feedback.microsoft-hohm.com
0.0.0.0 feedback.search.microsoft.com
0.0.0.0 feedback.windows.com
0.0.0.0 flex.msn.com
0.0.0.0 g.msn.com
0.0.0.0 h1.msn.com
0.0.0.0 i1.services.social.microsoft.com
0.0.0.0 i1.services.social.microsoft.com.nsatc.net
0.0.0.0 lb1.www.ms.akadns.net
0.0.0.0 live.rads.msn.com
0.0.0.0 m.adnxs.com
0.0.0.0 msedge.net
0.0.0.0 msftncsi.com
0.0.0.0 msnbot-65-55-108-23.search.msn.com
0.0.0.0 msntest.serving-sys.com
0.0.0.0 oca.telemetry.microsoft.com
0.0.0.0 oca.telemetry.microsoft.com.nsatc.net
0.0.0.0 pre.footprintpredict.com
0.0.0.0 preview.msn.com
0.0.0.0 rad.live.com
0.0.0.0 rad.msn.com
0.0.0.0 redir.metaservices.microsoft.com
0.0.0.0 schemas.microsoft.akadns.net
0.0.0.0 secure.adnxs.com
0.0.0.0 secure.flashtalking.com
0.0.0.0 settings-sandbox.data.microsoft.com
0.0.0.0 settings-win.data.microsoft.com
0.0.0.0 sls.update.microsoft.com.akadns.net
0.0.0.0 sqm.df.telemetry.microsoft.com
0.0.0.0 sqm.telemetry.microsoft.com
0.0.0.0 sqm.telemetry.microsoft.com.nsatc.net
0.0.0.0 ssw.live.com
0.0.0.0 static.2mdn.net
0.0.0.0 statsfe1.ws.microsoft.com
0.0.0.0 statsfe2.ws.microsoft.com
0.0.0.0 telecommand.telemetry.microsoft.com
0.0.0.0 telecommand.telemetry.microsoft.com.nsatc.net
0.0.0.0 telemetry.appex.bing.net
0.0.0.0 telemetry.microsoft.com
0.0.0.0 telemetry.urs.microsoft.com
0.0.0.0 vortex.data.microsoft.com
0.0.0.0 vortex-bn2.metron.live.com.nsatc.net
0.0.0.0 vortex-cy2.metron.live.com.nsatc.net
0.0.0.0 vortex-sandbox.data.microsoft.com
0.0.0.0 vortex-win.data.microsoft.com
0.0.0.0 watson.live.com
0.0.0.0 http://www.msftncsi.com
0.0.0.0 apps.skype.com
0.0.0.0 fe2.update.microsoft.com.akadns.net
0.0.0.0 m.hotmail.com
0.0.0.0 pricelist.skype.com
0.0.0.0 reports.wes.df.telemetry.microsoft.com
0.0.0.0 s.gateway.messenger.live.com
0.0.0.0 s0.2mdn.net
0.0.0.0 services.wes.df.telemetry.microsoft.com
0.0.0.0 statsfe2.update.microsoft.com.akadns.net
0.0.0.0 survey.watson.microsoft.com
0.0.0.0 ui.skype.com
0.0.0.0 view.atdmt.com
0.0.0.0 watson.microsoft.com
0.0.0.0 watson.ppe.telemetry.microsoft.com
0.0.0.0 watson.telemetry.microsoft.com
0.0.0.0 watson.telemetry.microsoft.com.nsatc.net
0.0.0.0 wes.df.telemetry.microsoft.com

*Note not all of the above addresses belong to Microsoft. Most of those URLs preform some sort of tracking so probably a good idea to block them anyways.

Nvidia:

Recently it has been discovered that Nvidia included Telemetry in their driver software. Some believe it is only part of the GForce experience, but that is not the case, it is installed along side the driver.

In order to disable the Telemetry bundled with the Nvidia driver you need to download Autoruns from Sysinternals. Extract the package and run the executable as Administrator. Make sure the everything tab is selected and filter content with the keyword: nvidia. Then proceed to uncheck the following items and then just close the application, followed by a reboot.

capture2

Now Nvidia might not be tracking anything, but that is not the point. Also you will need to do this every time you reinstall the driver.

Web browser:

To hide yourself online, obviously the best technology is a VPN or a proxy server. However these can cost quite a bit of money.

My suggestion is use two of the better online extensions. Electronic Frontier Foundations Privacy badger and uBlockOrigin.

Privacy Badger can be installed right from the eff.org web page. To install uBlickOrigin go to your web browsers extension store to find plugin and install it. For Vivaldi go to the chrome store as it is based on Chromium and all chrome extensions are compatible with it.

Home Media – Part 3 – The Rip

It’s been a while but finally I got around to completing and old series.

Part 1: The NAS build, can be found here. 

Part 2 The Setup can be found here.

The whole idea of this was so you could have your media at your finger tips. Your movies, your TV shows and your Music. If you’re like me you have shelves and shelves full of Movies and Television shows, and plastic bins full of CDs.

I’m not condoning piracy, and personally I think you should have control over the media that you own, as long as you don’t distribute it or broadcast it for the masses. Having said this you should probably check with your Country’s copyright laws. Either way it is nice to have it all at your finger tips.

I’m finding less and less compatibility between software and HD DVD movies these days, some titles I am unable to convert into a digital format form HD DVD. Also there is a draw back with HD DVDs, they only support Dolby Digital and DTS audio, where as newer Blu-Ray movies support lossless audio such as Dolby True HD and DTS Master Audio formats. There are a few new ones on the horizon as well, these would be Dolby Atmos, DTS:X and, Auro-3D.

Couple of good sites where you can find a lot of information on ripping and appending videos are http://www.videohelp.com/ and http://www.doom9.org/ .

To rip movies/tv you will require a few pieces of software. Besides the necessary codecs only two, Make MKV and Handbrake. Make MKV strips the copy protection and rips 1:1 video and puts them in MKV containers. Handbrake compresses the videos to specific formats and output devices. These are both free, if you want to pay for a single piece of software I would recommend DVDFab Blu-Ray ripper.

K-Lite Codec Pack is used to decode the material and the different formats for converting the material you will be ripping.

Capture5

MakeMKV can be used to strip the AACS and BD+ protections from discs, however this software goes a step further and can also convert your Blu-Ray movie to a MKV container. Mind you the movie will be large and there will be no compression, so the files can turn out to be 20GB, great if you want a lossless 1:1 copy of your movies. This software is free while in beta and for full functionality. You must input their beta key in to the software to make if functional, the key can be found here on their forum. This software also allows you to rip 3D Blu-Ray Titles. You can actually get away using only this software for ripping content.

Capture

HandBrake requires you to have the disc already ripped and stripped, generally using MakeMKV or Fab Decrypter. I really like using the batch processing mode in HandBrake, it’s very useful for TV shows. Also with handbrake you can create a custom template so it’s nice and easy to work with TV shows or repetitive content. HandBrake is also a converter so it can convert content for portable devices, it can make the file sizes smaller and change the resolution of the source, etc.

Capture4

DVDFab HD Decrypter is used to copy movies into a file and remove the encryption on the disc. This will remove AACS and BD+ protection. This software is free and usually the first step to digitizing your library, uless you are using MakeMKV. Be aware though, some software is not able to create Rips from these backups there is something specific to this software that sometimes does not play well with other applications.

DVDFab Blu-Ray Ripper (3D Plus), I use this software to rip my 3D Bly-Rays. This software allows the movies to be ripped into a 3D half side by side MKV or a 3D half over under MKV file. It’s paid software, but worth the money. Having said this there are other options for ripping 3D movies, MakeMKV can do it as well. Nice thing about this application is that you can compress the file and there is no need for a 1:1 copy.

Pavtube ByteCopy, I’ve actually never used this software but in my research I found that people have good things to say about the software.

BDtoAVCHD is another tool that came to my attention recently, If you ever want to create discs from your ripped content this is the tool to use.

So now that you have the necessary software where do you go from here?

Note that if you are looking for menus etc, this is not the guide for you. I don’t really care for the extra content and only want Audio track/Subtitle options. Mind you with Make MKV you are able to rip the special content as well, but I reiterate there is no menu system. Also I will only cover the Make MKV how to here, the DVDFab method is a paid method and kind of an all in one package that is simple to use, so no need to get into the details of that.

Download, install, and get the beta Make MKV key from the forum. The key changes with each Beta release, and even though MakeMKV is in version 1.x it is still free/considered beta.

Capture2

Once you install the software start up Make MKV and input the key in Help > Register. If you like the software and want to purchase it go to Help > Purchase. Untitled5

Insert the disc with the material you want to rip, make sure you have the right optical drive selected in the drop down list, then press the Open Disc icon in the left pane of the application window. The icon looks like a DVD drive with a disc inside. Alternatively you can go to File > Open Disc and select the appropriate optical drive where your disc is. Untitled2

 

The icon will start to look like it’s spinning and the software will take a minute to read and decrypt the disc in the drive. Once this process completes you will be presented with a list of valid Video sources on the disc. Generally the large ones are the content that you want.

Untitled3

Note that with this particular disc you have 4 episodes/Titles and that the 5th 7.0GB file is all of the episodes in one seamless Title. When I was converting this I generally skipped this one Title. Select the titles you wish to rip. Change the output folder on the right side of the window to where you want to save the converted files to. Then click the Make MKV button to the right of that. Depending on the processing prowess of your PC this can take a bit, it will also prompt you to create the folder if it does not exist. Untitled6

Once the process completes go into the folder and rename the files to something meaningful.

Once you have your MKV, you can start to compress and convert the formats.

To do this use Handbrake, it has several presets available for you to use. However if you wish you can create  your own and save it. This is where you can take 1080p content and convert it to 720p, having said this this will take a long time if you have a slower computer.

handbrake

You select your source which is either a file or a folder, and you can set your resolution in the Picture tab. Notice the output settings, you can change them to MKV or MP4 in this example. This is something to keep in mind.

handbrake2

In the Video tab you can change the codec that the encoder will use to recode your video in, H.265 is an option, select the Quality and you can even do 2-Pass Encoding. While 2-Pass encoding does improve the quality you will be looking at a really long encode unless you have lots of RAM and a really fast processor.

handbrake3

Then you have your Audio section where you can downmix to stereo or upmix to a different codec all together. Note that you can have multiple audio tracks.

Once you have selected your desired settings you can save them as to use at a later time, just click the ADD button on the bottom right. If you have multiple videos to encode add them to the queue and keep going, nice thing about Handbrake is that it does batch conversions.

 

Debian 8 Jessie sources.list, missing installation sources.

debian8_with_release_date

Each time I install Debian Jessie or Debian 8 I notice that I can not install any software packages afterwards. I’m not sure if it is me, or the installation media, but the sources.list is missing entries. Recently I tried installing xrdp, which allows Windows OS machines to RDP into Linux OS machines.

Commands such as…

apt-get update
apt-get upgrade
apt-get install package_name

… yielded no results. When I navigate to /etc/apt/sources.list I noticed there were only 3 entries in the file.

deb cdrom:[Debian GNU/Linux....

deb http://security.debian.org/ jessie/updates main contrib
deb-src http://security.debian.org/ jessie/updates main contrib

However the Debian Wiki specifies two additional sources.

deb http://httpredir.debian.org/debian jessie main
deb-src http://httpredir.debian.org/debian jessie main

deb http://httpredir.debian.org/debian jessie-updates main
deb-src http://httpredir.debian.org/debian jessie-updates main

Once you add these to your sources list and run the above commands the
packages should update, and you can proceed with installing XRDP.

Converting Virtual Machine disk formats

There are many Virtual Disk formats, VDI, VMDK, VHD, VHDX, IMG, RAW, HDD, and many more. Unfortunately VHD and VHDX formats are amongst the least popular ones, but if you’re running a Hyper-V server these are the only formats you can work with.

Don’t fret, there is a way to convert a lot of these common formats to the VHD Micrsosoft disk image. Oracle includes a conversion tool with their Virtual Box application VBoxManage.Virtual box is a free tool available for download, and use. You can find the software over here: https://www.virtualbox.org/ .

I found that this works better than the actual tool that Microsoft offers and have since successfully converted many formats with Virtual Box to the VHD disk image. Usually I go from a IMG to VHD file. I capture the drive using dd from within a Linux Mint boot drive and then proceed to convert it. But I have sometimes setup test VMs in VirtualBox that I needed to run on a Hyper-V server after.

To perform the conversions have a look at the following two commands.

Install Virtual Box and use it by opening a command prompt and navigate to the VirtualBox installation directory. Usually located in C:\Program Files\Oracle\VirtualBox.

Usage:

VBoxManage clonehd  <uuid|inputfile>  <uuid|outputfile>
                    [--format VDI|VMDK|VHD|RAW|<other>]
                    [--variant Standard,Fixed,Split2G,Stream,ESX]
                    [--existing]

A baisc command and output of this would look something like this.

C:\Program Files\Oracle\VirtualBox>VBoxManage.exe clonehd "c:\VMs\Windows 10\Windows 10.vmdk" "f:\temp\Windows10.vhd" -format vhd
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Clone hard disk created in format 'vhd'. UUID: 1f6e118a-f0e2-49ed-a352-6b842791cdfa

C:\Program Files\Oracle\VirtualBox>

VHD is a Hyper-V generation 1 format, where as VHDX is a Hyper-V generation 2 format.

Alternatively if you have a DD captured raw IMG file you can convert it to VHD by using the following command first:

 C:\Program Files\Oracle\VirtualBox>VBoxManage.exe convertdd file.img file.vmdk

…once converted follow this up by the “clonehd” command which converts to vhd. Prior to converting to vhd make an attempt to boot the vmdk in VirtualBox. Attempt to boot it in to the OS, either normally or via safe mode. The reason for this is that sometimes the OS will need to run a chkdsk before booting into itself, you should let it run as this chkdsk will allow the vhd to properly mount in Hyper-V. It seems that either the VMDK format is more forgiving than VHD, or only VirtualBox can fix the conversion errors.

If you’re only looking to only mount a volume and not boot off the virtual disk and into an OS you can try a tool called Disk2vhd. Also since windows 7 the backup software built into the OS created vhd backup sets. That could be an option as well.

Hyper-V replication in a workgroup or across domains using a self signed certificate.

Why would you want a HyperV server in a workgroup environment?

Well if your Domain Controller is a VM you really don’t want to add the HyperV server to the domain as it will boot before the DC comes up. This type of setup is ripe for domain issues, so we’re left with a server that is only in a workgroup. Also if you are doing cross site replication, you might be replicating from/to different domains, this is where the self signed certificate authentication comes in to play as it is domain agnostic.

Kerberos authentication does not work in this setup, so we need to use a certificate authority as a means of authenticating the two servers with each other. The Primary server is where all the VMs are, and the Replica server is where the VMs will be copied to. HyperV replication is native and built into Server 2012 +, so there are no extra licenses necessary.

What are the steps involved?:

  1. Change the DNS suffix on both Primary and Replica servers.
  2. Reboot both servers.
  3. Create self signed certificates on both servers.
  4. Open the Certificate MMC snap-in on the Primary server and export the certificate to a .pfx file.
  5. Copy the export file and RootCA certificate from the Primary to the Replica server.
  6. Import the Primary RootCA certificate file on the Replica server.
  7. Import the .pfx file on the Replica server.
  8. Copy the RootCA certificate from the Replica to the Primary server and import it.
  9. Disable Certificate Revocation Check on both servers for replication and fail over replication.
  10. Setup the Replica server as a replica in HyperV.
  11. Start replication of a Server on the Primary server.

First we need to change the server names, or rather add a DNS suffix to them. Bring up System Properties in the Control Panel, under the Computer Name tab click change. In the Computer Name/Domain Changes window click More…

In the DNS Suffix and NetBIOS Computer Name add a Primary DNS suffix. Something along the lines of “hypervreplica.local”, it doesn’t matter call it what you will.

Click OK and save all the changes. Note that you will be required to reboot the server in order for changes to take effect. Do this to both the Primary and Replica server.

Primary is the server where your VMs reside, and Replica is where your VMs will be replicated or copied to.

Next we need to create a self signed certificate. For this you will either need Visual Studio or the Windwos SDK(https://www.microsoft.com/en-us/download/details.aspx?id=8442).

What we really need out of either of these is the makecert.exe file.

If you have VS installed the makecert.exe file is located under C:\Program Files (x86)\Windows Kits\8.1\bin\x64, or a similar path, the 8.1 will change depending on the version of Visual Studio you have installed.

Copy the makecert.exe file from here to the primary and the replica servers.

On both those servers create an empty directory somewhere, place the makecert file in there. This is also where we will create and store the self signed certificates.

On the Replica server open up an elevated command prompt and navigate to the directory where the “mekecert.exe” file is located and type in the following:

makecert -pe -n “CN=ReplicaRootCA” -ss root -sr LocalMachine -sky signature -r “ReplicaRootCA.cer”

The above command assigns a signature certificate issuer name to the replica server of “ReplicaRootCa”

Followed by:

makecert -pe -n “CN=replicahostname” -ss my -sr LocalMachine -sky exchange -eku 1.3.6.1.5.5.7.3.1,1.3.6.1.5.5.7.3.2 -in “ReplicaRootCA” -is root -ir LocalMachine -sp “Microsoft RSA SChannel Cryptographic Provider” -sy 12 ReplicaCert.cer

…where the replicahostname is replaced by the name of the server with the DNS suffix and all. Ex. hostname.domain.local.

Now move over to the Primary server, open up an elevated command prompt and navigate over to the folder where “makecert.exe” is located, and type the following:

makecert -pe -n “CN=PrimaryRootCA” -ss root -sr LocalMachine -sky signature -r “PrimaryRootCA.cer”

Followed by:

makecert -pe -n “CN=primaryhostname” -ss my -sr LocalMachine -sky exchange -eku 1.3.6.1.5.5.7.3.1,1.3.6.1.5.5.7.3.2 -in “PrimaryRootCA” -is root -ir LocalMachine -sp “Microsoft RSA SChannel Cryptographic Provider” -sy 12 PrimaryCert.cer

… where the primaryhostname reflects the name of the Primary server with the added DNS suffix. The above 4 commands will create two files on each server.

On the Primary server run > mmc, click File and select Add/Remove Snap-in…

Select the Certificates snap in and click Add>, on the next windows select Computer Account, click Next>, and then select Local computer:. Click Finish.

In the Certificates snap in, expand the Personal store and then click on Certificates.

 

You should have a certificate here with the Replica serve name and Issued by ReplicaRootCa.

Right click this certificate, select All Tasks and select Export…

Capture7

This will open the Certificate Export Wizard, when prompted select Yes, export the private key.

Export File Format, use Personal Information Exchange…. (.PFX), Include all certificates and the certification path if possible.

On the Security page check the password box, and input a password you will remember.

Click the Browse button to save the export in a *.pfx file format, give it a file name (PrimaryServer.pfx) and click save.

Double check all your settings on the final page and click Finish.

Copy the PrimaryRootCA.cer file and the PrimaryServer.pfx files to the Replica server. Put it in the folder where you created your Replica Server certificates.

On the Replica server we will now import the cer and pfx files. Open up an elevated command prompt and navigate to the file location. Type in the follwing:

certutil -addstore -f Root “PrimaryRootCA.cer”

The quotes are only necessary if you have spaces or special characters in the file name.

Open up MMC, expand the Personal section, right click on Certificates and select All Tasks > Import.

The Certificate Import Wizard will open up. Click Next. On the File to Import page click Browse…

You might have to change the file type to view the pfx file.

Navigate over to the location of your PrimaryServer.pfx file and select it, click Open. Click Next. On the next screen enter the password for the Private Key. You can mark the key as exportable if you’d like, this means you can export it at a later time if you do not keep a copy of it somewhere. Also check off Include all extended properties. Click Next.

Place the certificate in the Personal certificate store. Click Next. On the final page inspect all the details and make they are correct. Finally click Finish.

*Please be aware that for fail over replication you will more than likely need to export the certificate in the pfx format from both servers and then copy them over and import them on both servers as well. The reason for this is that replication is only one way, where as fail over replication goes both ways. Something to think about.

Now copy the ReplicaRootCA.cer file over to the primary server, place it in the folder with all the other certificate files. In and elevated command prompt add it to the certificate store.

certutil -addstore -f Root “ReplicaRootCA.cer”

Run the following two commands on both servers in an elevated command prompt.

reg add “HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization\FailoverReplication” /v DisableCertRevocationCheck /d 1 /t REG_DWORD /f

reg add “HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization\Replication” /v DisableCertRevocationCheck /d 1 /t REG_DWORD /f

Note that the only difference between the two registry commands is FailoverReplication and Replication. You shouldn’t need to restart either of the servers after these commands.

Next you need to enable the Replica server as the replica.

On the Replica server open the HyperV manager.

Select HyperV Replication Configuration, in the configuration window check off Enable this computer as a Replica server. Check off Use certificate-based Authentication (HTTPS). It should prompt you to select a certificate, if not click on Select Certificate… you should have the option to select the certificate you created on the Replica server. Select it and click Apply.

*Note, that you will get a prompt about the Windows Firewall, I have mine disabled on all servers so this message never applied to my setup. However if you have your fire wall turned on I would recommend adding a rule to it to allow the traffic on port 443 to pass.

Now the real test, go to your Primary HyperV server and attempt to enable replication. Open up HyperV manager, select a virtual machine, right click it and slect Enable Replication…

You will be presented with a Before You Begin page, click Next >.

On the Specify Replica Server page, type in the FQDN of the replica server, for example, replicahostname.domain.local, or whatever the hostname and dns suffix that you assigned to your replica server is. Click Next >.

On the Specify Connection Parameters, make sure that certificate-based authentication is selected. Kerberos authentication only works on a domain. You may need to select the proper certificate, this will be the Primary server certificate. Also check off Compress the data that is transmitted over the network. If you see a yellow exclamation sign with the text “Could not get configuration details of the specified server.” at the bottom, don’t worry about it, if everything is setup properly it should not impact the replication in any way, shape, or form. Click Next >.

Choose Replication VHDs, here you can pick and choose which storage attached to the server you want to replicate. Select the storage you want and click Next >.

Configure Replication Frequency. The options here are 30 seconds, 5 minutes, or 15 minutes. Depending on how mission critical your data is choose accordingly. Note that replication frequency differs from Server 2012 to Server 2012 R2.

Configure Additional Recovery Points. Depending on how many recovery points you require here is where you set that up. You can setup additional hourly recovery points and even use VSS for snapshots. Hourly recovery points provide granularity, no only can you recover form the last replication point, but with this option enabled you can go back hours. You also have the option of VSS snapshots which, from personal experience, can fail. I don’t have experience with VSS on replication, but VSS on backups and more often than not VSS was always the culprit for failed backups. VSS has a tendency to fail, not ofter but every once in a while. Either way I usually only maintain the latest recovery point. Again the number of recovery points differs from 2012 to 2012 R2, 15 vs 24.

Pick your poison and click Next >.

Choose Initial Replication Method. These options are self explanatory. Chose your replication method and when. I usually just send it over the network, I find that the impact is minimal. You can also start the initial replication at a defined time, perhaps when your system is not as busy at night etc.

 

*One thing to note about replication and this is important, replication creates an avhdx file. This is a HyperV change file, and during the initial replication this can grow quite large in size. On a normal active system I have observed that this file can grow to 33% size of the original VHDX/VHD file. So be careful and be warned, because if the storage medium runs out of space it will pause the VM.

Click Next >. Confirm your settings and click Finish. Your replication should now begin.