SBS 2003 Active Directory Restore

I was tasked a few months ago to P2V a few windows servers that were part of a small domain for a small doctors practice.  That practice had merged with the Hospital I work for a few years back and they kept their old Physical servers with their domain and EMR system on them to access the EMR for historical patient records.

Well... needless to say the hardware started approaching 8 years of age and failing on one of the servers.  We wanted to virtualize them into our VMware infrastructure in our secure datacenters.  Now I have done a lot of P2V's in my day of servers as VMware became the standard and old systems needed to be kept around, or otherwise simplified.  I have NEVER had a P2V corrupt Active Directory in a domain controller.

I proceeded to take an image of all three servers using an older version of Acronis True Image that I had hanging around.  The servers were a terminal server, the EMR server and the SBS 2003 server.  All are part of a domain that is controlled by the SBS 2003 server.  The users in this practice on the Hospital LAN access the terminal server which is where we installed their EMR application.  They are authenticating to that terminal server via the SBS 2003 server.  Needless to say, I took hot images of all three servers and everything went fine.  Even the P2V of the servers using VMware's converter tool worked like a champ.  The terminal server booted fine, and the EMR server booted fine.  However, the SBS 2003 server booted with the following: security accounts manager initialization failed directory service cannot start

I was a little worried at this point.  I tried numerous things that are pretty typical.  Booted into safe mode... same error.  I did a quick google search and everything came back in the results as a corrupted Active directory.  I was sitting OK however, because the old Physical servers were still running.  But they were ticking time bombs.

I thought: "perhaps I should take a cold image (boot to a disc so windows is not running) and take an image that way so the AD database is not changing like it can during a hot image.  I started this process only to have the power go out on us 2 hours into the 4 hour process.  So I put on my big boy boots and started to go down the road of restoring AD without a backup.

I spent several hours pouring through Microsoft documentation, tech net articles and forums to get the help I needed.  I came up with a mixture of a few solutions.  Here are the steps that ended up working for me, because I was getting errors in Active Directory restore mode pertaining to the Jet Database.

1. Restart the server and press F8 key, select Directory Services restore mode.
2. Log in with the local administrator username and password
3. Type: cd \windows\system32
4. Type: NTDSUTIL
5. Type: files
6. If you encounter an error stating that the Jet engine could not be initialized exit out of ntdsutil.
7. Type: cd\
8. Type: md backupad
9. Type: cd \windows\ntds
10. Type: copy ntds.dit c:\backupad
11. Type: cd \windows\system32
12. Type: esentutl /g c:\windows\ntds\ntds.dit
13. This will perform an integrity check, (the results indicate that the jet database is corrupt)
14. Type: esentutl /p   c:\windows\ntds\ntds.dit
15. Agree with the prompt
16. Type: cd \windows\ntds
17. Type: move *.log c:\backupad   (or just delete the log files)

This should complete the repair.  To verify that the repair has worked successfully:

1. Type: cd \windows\system32
2. Type: ntdsuti
3. Type: files   (you should no longer get an error when you do this)
4. Type: info   (file info should now appear correctly)

One final step, not sure if it’s required:

From the NTDSUTIL command prompt:

1. Type: Semantic Database Analysis
2. Type: Go

There you go!  Hopefully nobody has to use this article as server 2003 is almost at End of Life, but I know that there will be instances where some legacy applications will Keep Windows 2003 in datacenters for many more years.

Copy & Paste in VCenter Vsphere

I recently read a post on a blog by Francesco at thevirtualway.it  It tells how to configure your VM's in vcenter for copy & paste.  This can be super annoying in some regards if you use the Vcenter to go to the console of a VM.  It was one of those things that bugs me when I need to do it and can't, but know its possible.  I just had it way far down on my "to do list" and it never moves up on that list because its such a low priority.  Because of this I never had time to dig into how to fix this.  Thanks to Francesco posting at a online group I subscribe to, I saw it come up in my email and now its a quick fix for me and one worth passing along!

To do this you need to SSH access to the host and add these entries to the file /etc/vmware/config 

vmx.fullpath = "/bin/vmx"
isolation.tools.copy.disable="FALSE"
isolation.tools.paste.disable="FALSE"

You will need to Shutdown and power-on the virtual machine for changes to take effect.

VCenter 5.1 SQL Database install/setup

Per my previous post pertaining to losing my VCenter in our data center this week I had a crash course in SQL database setup for VCenter 5.1.  I come from working 10 years in the VAR business which was all SMB IT support so any VCenter setups I had done in the past I was able to get away with SQL Express databases for VCenter.  With VCenter 5.1 there are now two databases needed.  One for the SSO and obviously one for the VCenter.  My previous post outlines how to install the database for the SSO and this step is required before you can install VCenter 5.1 so if you have not done that yet you need to!

If you have or are planning to have more than 5 hosts and 50 VMs than you cannot use the SQL Express version for VCenter.  I will out line what I did to install a full SQL 2008 r2 database in preparation for your VCenter install.  Keep in mind that as of this writing VMware does not yet support SQL 2012 or Server 2012 for its SSO or VCenter products.

 - First: Create an individual role account called "SQL" or "SQLDB".  I am very anal and can't stand it when people tie all their services to the domain administrator account, or worse... the user account of the Network Admin.  (Its one of my huge pet peeves).

- Make sure you have .NET installed and updated.  On server 2008 this is installed by installing the Application server role in the server manager.  Make sure to run windows updates after that to patch it.

- Launch your SQL 2008 r2 install and click on "Installation" and "New installation" which is the first option.

There will be some checks that run.

- The next screen is your product key screen so what you need here and click next, accept the terms and click next.

- On the Setup Support Files screen you should keep the defaults and click next.

- If you run into any warnings on the next screen be sure to address them as neccesary.

- Next, leave the default selected: "SQL Server Feature Installation".

- Now we are going to check the following: Database Engine Services, Client Tools Connectivity, Integration Services, Management Tools - Basic, Management Tools - Complete.  Change your directory paths as you wish.  So it should look like this:

- Click next until you get to the instance configuration screen.  Here you can get fancy and change the default settings to what you want but I choose to keep it simple and not create headaches and leave it all default.

Click next until you get to the Server Configuration screen.

- Here select the "Use the same account for all SQ: Server services" and put in your SQL account you created in AD earlier.

Make sure to change the start-up type to "automatic" for the SQL Server Agent before clicking next.

- On the "Account Provisioning" screen, select "Mixed Mode (SQL Server Authentication and Windows Authentication).  Set a password for the built in SQL Server system administrator account.  This is the "sa" or Super User account for the database.

- Under "Specify SQL Server Administrators" you can add any additional users or groups that will need to maintain the database.  I added a group that our SQL folks are a part of.

- That's it.  Keep moving through the rest of the screens and you are done!  Make sure to run windows updates again after words.

Database Configuration

OK... now you got SQL installed so we need a database for the Vcenter.

- Start the SQL Server Management Studio which is in the default area of the program files under the start menu.  Log in with the "sa" user account.

- Right click on the topmost SQL Server object which is usually name after the server you are on or "localhost" depending on how you logged in.

- Choose "properties", then the "Memory" page.  I configured mine like this for my environment to start.  You can always change it later and increase the memory.  For now I set the maximum server memory to 50% of the memory the server actually has on the host.

- Click OK and your back at the home screen.  Now right click the databases folder and select "new database".  

- Under "General" i set my database name to "VCDB", but this you can safely use what you want.  Under the "options" set the recovery mode to "simple".  Click OK and you now have a new database.

- Right click the "security" folder.  Pick "new", "Login".

- Under "general" enter the username "vpxuser", select "SQL Server Authentication"  and enter a password.  I uncheck the "enforce password policy", "enforce password expiration", and "user must change password at next login".  Set the default database to "VCDB" and language to English.

- Click the "user mappings" page on the right.  Check the "Map" box for VCDB. then choose db_owner from the role membership list below.  Then check the "map" box for msdb, and choose db_owner.  I also select the "..." area in the "Default Schema Column" for both msdb and VCDB and set the default schema to "dbo".  Click OK.

- Now we need to configure the TCP/IP options.  Vmware has good documentation on how to do this.  It cam be found here and I really recommend following this guideline.

- Next the "local launch" permissions needs to be set.  Go to Administrative tools and "Component Services".  Go to "Component Services", "Computers", "My Computer", "DCOM Config".  Right click "MsDtsServer100" and choose properties.  Under the Security tab and "Launch and Activation Permissions", select "Customize", then "Edit".  Add the "SQLDB" user account you created.  This will prevent some common errors that pop up in logs and ensures scheduled jobs run.

- Lastly we need to create an OSBC data source.  This is one where again, VMware's library notes are very good and easy to follow.  You can find them here.

That is it!  Your set to install VCenter now using your own SQL database!

Happy virtualizing!

VCenter 5.1 SSO Install

It has been a whirlwind of a week for me as a Datacenter admin.  It started with a Monday off for vacation, a Tuesday off for a stomach bug, and I wasn't in my chair at the office for no more than 15 minutes Wednesday before the VCenter in our datacenter crashed hard.  The memory and processor was tacked at 100% and would not budge due to java.exe.  The best I could tell it was the VMware update manager causing this.  The VMware forums were full of other users with this issue and they all said to uninstall and re-install update manager, but for me it made the problem worse.  I made the decision to rebuild the Vcenter from scratch.  This meant my first run-in with the new and now required VMware SSO, as well as my first time installing full SQL instead of SQL express for the Vcenter database.  I inherited this Vcenter and have only done lots of small deployments of Vcenters in years past.  So, I thought I would pass along my findings in hopes it makes other Admins lives a little less stressful as I spent a total of 2 days rebuilding the Vcenter and our Veeam backup server to work with the rebuilt Vcenter.

As I already mentioned, VMware SSO is now a requirement before installing VCenter server.  I, along with many, many other admins out there do not think this should be a requirement.  It seems to be a push for VMware to cram security down our throats.  While I agree security should be at every admin's top priority, there is such thing as going overboard.  I think back to my days of supporting SMB's where 2 or 3 esxi hosts were all that needed to be managed for a Vcenter and this SSO requirement just adds a huge, thick layer into those installs.  Read more here about it to understand it.

The easiest way to do the SSO in a large environment is as follows:

- Install SQL server 2008r2 on server 2008r2.  (Vmware does not support SQL 2012 or server 2012 for Vcenter or SSO as of this writing).  Configure your instance how you want.  (I am assuming you know how to do this already).

- Open up SQL Server Management Studio.

- Go the following location on the Vcenter Server install Disc: \VCenter 5.1\Single Sign On\DBScripts\SSOServer\schema\mssql

- Click and launch the Script: rsaIMSLiteMSSQLSetupTablespaces

- You should now see the script in SQL Management studio like this:

- This script will create the SSO database for you.  I strongly recommend you don't be like me and try to get all fancy and name the database the way you want because your anal and want to be able to see a database name that makes sense to you down the road.  I spent many hours doing this... just stick to the defaults and what I write here and you will be off and on your way.

- You need to change three lines in this script for the location of the database files.  If you read the green text its pretty clear how to do this.  The lines you need to change all have C:\Change Me\.... marked in red.  Just put your path to where you want the database files in place of those lines.

So go from this:

To this:

***IT IS VERY IMPORTANT TO NOT CHANGE THE FILE NAMES IN THE LINE***  Your database location may vary on your setup.

- Run and execute that script and your SSO database named "RSA" will be created.

- Now you have to setup the user and DBA user for that database.  Thankfully VMware provided another script in that same location of the VCenter installer you just lifted the last one from!  With your SQL management studio still open, launch this script: rsaIMSLiteMSSQLSetupUsers

- The only thing you will want to change in this one is the passwords for the users its going to create.  Leave everything else the default and make sure to record the user names "RSA_DBA" and "RSA_USER" with the passwords you give them in this script.  Give them good strong passwords.  The lines you want to change are in red:

Make sure to leave the ' marks and put your password in between them for each user.

- Execute that script and your are done.  Now you can install the VMware SSO from the installer disc.

- During the install select "Create the primary Node...." because you would not have needed to do any of the above otherwise.  :-)

- At the next options menu select "create the primary node..."

- type in a new password and confirm it for the new admin sso account.

- Now choose "use an existing supported database" because you just created one.

- The next screen is where you are going to put in your information for the "RSA" database you created as follows:

The JDBC stuff is for certificates which I am not covering here, and most VMware admins of small or medium datacenters like myself don't use them anyway.

- After that the final "next" will connect to the database, create that sso admin user account and your off and running.  The install takes a little bit of time so go grab a cup-o-coffee.  :-)

Now you can move on and install Vcenter.  I will post soon on how to install a full SQL database to prep for VCenter to install to.  There most likely are other VMware admins out there like me who don't have a clue on SQL.

Happy Virtualizing!

Unmount VMtools Installer with Orchestrator

Here is a super-saucy Vmware Orchestrator workflow to un-mount the VMtools installer from all the VMs in a cluster you choose.  I have been working tirelessly on it these last couple of days and got it to a point where it is successful!  I needed this workflow because I manage 140 VMs and when I patch or update the 9 hosts in the cluster, the VMs start complaining about their tools being out of date.  I created another script for updating all the VMs tools in the clusters (here), but as many VMware admins will attest to, the Linux VMs don't always unmount after upgrading.  Thats where this workflow will come in!

The workflow is here for you to download.  ***RUN AT YOUR OWN RISK***

Here is a snapshot of the workflow:

Happy Virtualizing!

VM Tools upgrade with Orchestrator

Unless your have been living under a rock or in a video game virtual world in your parents basement for the last year, you already know that Automation in IT is the big buzz word.  Further more, you probably already know that Vmware comes bundled with Orchestrator when you install your VCenter.  Orchestrator comes with a lot of canned Workflows (scripts) that allow you to automate tasks in your Vcenter which when you couple with scheduling these tasks should make the life of an Enterprise IT admin much easier and free up some evening and weekend hours!

While Orchestrator comes with some good workflows to start you off, right off the bat I noticed there was one in particular that I was looking for that was not already a part of Orchestrator.  For the record, in order to create your own workflows, you need to have VCenter standard or higher licensing.  You may still use Orchestrator with the Essentials or Essentials plus license but you are limited to the built in workflows or importing them.

I set out on a hardy adventure to create my own workflow.  Don't worry... Orchestrator is heavy on the Gui side, but will require some scripting and further research on your part to figure it out... be patient.  The script in particular I was looking for was to update the VM tools on all the VM's in a cluster that I set.  I was able to do this successfully and have attached the workflow to this post.  Feel free to download it and import it into Orchestrator for your own use.  ***USE AT YOUR OWN RISK***

Here is a snapshot of the workflow:

Happy Virtualizing!

Veeam impressions

I was a part of a project that implemented Veeam 6.5 on our production network.  We were desperately in need of a new backup as the old one was out of space out of date, and simply not good for disaster recoery in our environment

A quick over view of our environment:

- 10 ESXi Hosts managed by Vsphere 5. 4 hosts are esxi 4.1, 5 are esxi 5, and 1 is a stand alone host with esxi 5.

- 4 SAN's of various sizes.  2 Dell EqualLogics, 1 EMC VNX 5300 and an older HP I canot recall the model of.

- 110 virtual machines varying between Windows server 2003 & 2008r2 and many different flavors of Linux.

- 2 data-centers connected by 1gb fiber (soon to be 10gb).

I implemented Veeam 6.5 about a week ago now.  I installed the software on a single VM.  (Server 2008 r2, 2 quad core CPU's and 12gb RAM).  We also got a new HP P2000 SAN.  I created 3 Luns on it.  One 12TB, another 8TB and the last 3TB.  I presented the 12TB and 3 TB Luns to the Veeam VM.  The configuration of Veeam is very straight forward and their website has great videos that walk you through setting up the software.  I found Veeam to be much easier to setup than Symantec Backup Exec but not quite as easy as Acronis VM protect 7 or 8.  But the performance of Veeam blows either of those other two products out of the pond!

Veeam is all about its built in deduplication and Vsphere or Hyper-V integration.  We have roughly 10TB in use on all of our SAN's combined.  Veeam was able to take all this data and dedup it down to 4TB on the first set of backups of all the VM's!  That is very impressive!  In order to get the best deduplication results with Veeam, you have to group as many similar VM's in a job as possible.  I have done my best to group our SQL servers in one job, file servers in another etc...  Currently I have a total of 11 jobs running at various times.  I have a few jobs that I set to only run once a week on Sunday.  (Terminal servers, domain controllers, print servers etc...).

Another nice feature of Veeam is that it does what is called a "synthetic full backup".  This means that rather than taking a full backup every so often as you set it to, it will do forever incremental backups.  On a day that you specify, you can Veeam run its synthetic full backup.  This will take all the incremental backups that you have based on the period of your retention and compile them into a full backup.  This takes the work load off of your production environment of having to run a full backup!  In our case our file servers would run for 12 hours to do a full backup.  Instead the Veeam server itself does this job with the backups it has for the VM's in its repository.  Its really slick!

I will follow up after I have run this in production for a while but I already have plans to look at setting up a Second VM to act as a proxy for Veeam to put some of the work loads onto as I have had some performance issues already on the single VM I have for Veeam.  Also, I just read that Server 2012 has a dedup feature for its volumes you can setup!  This would mean after Veeam's deduplication, windows could take my iscsi volume for my Veeam repository and dedup it even further!  Veeam has been boasting impressive dedup results with this type of setup!

Cheers!

Turn off ESXi 4.1 SSH warnings

After patching the hosts on our VCenter last night I noticed the hosts we have that are still running esxi 4.1 were now showing a warning stating: "SSH for the Host has been enabled".

I don't know why this is not fixed in a patch from VMware since they now support SSH being turned on but it is what it is.  The way to turn this off is to go into the configuration of each host, click on advanced settings, UserVars and scroll down and find "UserVars.SuppressShellWarning".  Change the value to 1.

Happy Virtualizing!

VMware P2V Best Practices

In prepping to do a few more P2V conversions of servers in my datacenter, I realized its been a while since I have done one.  It was almost a weekly occurrence when I was an Engineer/Consultant in the VAR industry.  Its amazing how quickly things slip my mind the less frequently I do them... perhaps its age!  I found a great resource at blog run by Lieven D'hoore. (www.ituda.com)

The best/easiest approach to converting a Windows operating system from a physical machine to a virtual machine is to perform a hot migration with VMware Converter installed locally on the source (physical machine) operating system. Below an overview of the different steps involved in a P2V (Physical to Virtual) conversion. Note that these steps only apply to Windows operating systems.

 BEFORE CONVERSION

• Confirm that the source server has at least 200 MB free disk space on its system volume. This space is required to operate the disk snapshot features in VMware Converter
• Confirm that the source machine has at least 364 MB RAM
• If you have software mirroring, break the mirror (but not the data) since VMware converter does not support software mirrors.
• Clean-up any temporary files and un-needed data
• Change all hardware related services to “disabled” startup mode
• Download the following utilities/scripts and install them to the directory c:\Temp\P2Von the source server. The following files are included:
  • p1-HWPhysical.bat
  • p2-installP2VConverter.bat
  • p3-SystemConfigUtil.bat
  • v1-HWVirtual.bat
  • v2-vmprofile.bat
  • v3-enablehdwacc.bat
  • v3-enablehdwacc.vbs
  • v4-renameNICs.bat
  • v4-renameNICs.vbs
  • v5-setip.bat
  • v6-uninstallP2Vconverter.bat
  • v7-SystemConfigUtil.bat
  • v8-hiddendevices.bat
  • v9-HALupdate.bat
  • PSPCleaner.exe
  • comm.exe
  • libiconv2.dll
  • libintl3.dll
  • devcon.exe
  • VMware vCenter Converter Standalone 4.01 build 161434 (See also http://www.vmware.com/download/converter/)
• Log on to the source machine (mstsc -v: servername /F -console) with a local administrator account and open a command window
• Run c:\Temp\P2V\p1-HWPhysical.bat
  •  The script creates a list of all devices of the physical machine (including non-present devices)
• Install the VMware vCenter Converter Standalone software by executing c:\Temp\P2V\p2-installP2VConverter.bat
• Run the System Configuration Utility on the source server by executing c:\Temp\P2V\p3-SystemConfigUtil.bat to reduce the number of services and applications running on startup (all software except for all Microsoft Services and VMware Converter Service).
  • On the General tab
    • Select “Selective Startup
    • Uncheck “Load Startup Items”
  • On the Services tab
    • Select “Hide All Microsoft Services”
    • Click “Disable All”
    • Mark the VMware vCenter Converter Agent
    • Mark the VMware vCenter Converter Server
  • Click “Apply”
  • Click “Close”
  • Click “Restart

CONVERSION

• Log on to the source machine with a local administrator account
• Start the VMware vCenter Converter Standalone application (A shortcut should be available on the desktop)
• Select “Convert Machine”
• Specify Source
  • Select source Type: Powered-on machine
  • Specify the powered-on machine: This local machine
  • Click NEXT
• Specify Destination
  •  Destination Type
    • Select destination type: VMware Infrastructure virtual machine
    • VMware Infrastructure details
      • Server: type in the IP address of the ESX server you want to convert to (or of your Virtual Center server)
      • User name: type the user name of an administrative account for the above server
      • Password: type the password of the above user name
      • Click NEXT
  • Host/Resource
    • Select the ESX host/group you want to convert to
    • Virtual machine name: type the name of the destination virtual machine (this is normally already set)
    • Datasource: select the datastore in which to place the destination virtual machine
    • Click NEXT
• View/Edit Options
  • Data to copy
    • Keep the defaults unless you need to re-size the partitions
    • Select “Ignore page file and hibernation file”
  • Devices
    • Select the number of processors: adapt if needed (Remember to revert the HAL after conversion if you are changing from a multi-processor to a uni-processor machine)
    • Disk controller: select SCSI LSI logic for Windows 2003 and beyond
    • Memory for this virtual machine: adapt if needed
  • Network adapters
    • Choose the number of network adapters you need
    • Select the appropriate VLAN
    • Select Connect at power-on
  • Services
    • Go to tab “Destination Services”
    • Change starting mode to “Disabled” for all services you will not need in the virtual machine
  • Advanced options
    • De-select “Synchronize changes that occur to the source during cloning”
    • De-select “Power on target machine”
    • De-select “Install VMware Tools on the imported machine”
    • Select “Remove System Restore checkpoints on destination”
    • Select “Reconfigure destination virtual machine”
  • Click NEXT
• Ready to Complete
  • Review the summary information
  • Click NEXT

AFTER CONVERSION

• Shut down the physical machine
• Use the VI client to log on to your virtual center server or to your ESX server
• Review and adjust the virtual hardware settings:
  • Adjust the number of NICs, CPUs, RAM, …
  • Remove any unnecessary devices such as serial ports, USB controllers, COM ports, floppy drives, …
• Start the virtual machine
• Log on to the virtual machine
• Run c:\Temp\P2V\v1-HWVirtual.bat
  • Creates a list of all devices that matches the virtual machine
  • Compares the list of all devices of the physical machine (created in the step “before conversion” with the list of all devices that matches the virtual machine (created in the previous step)
  • Removes all phantom hardware
  • Rescans hardware
  • Reboots the  server
• Change registry key regarding profile problem in VMware (see VMware article) by executing c:\Temp\P2V\v2-vmprofile.bat
• Enable Video Hardware Acceleration by executing c:\Temp\P2V\v3-enablehdwacc.bat (this command calls the vbs script c:\Temp\P2V\enablehdwacc.vbs)
• Rename your network connections by executing c:\Temp\P2V\v4-renameNICs.bat (this command calls the vbs script c:\Temp\P2V\renameNICs.vbs)
• Set IP information on the first network connection by executing c:\Temp\P2V\v5-setip.bat  %1 %2 %3 %4 %5 %6 %7
• From this step on you should be able to connect to the virtual server via RDP
• Uninstall VMware vCenter Converter Standalone by executing c:\Temp\P2V\v6-uninstallP2Vconverter.bat
• If you are converting from HP Proliant hardware you can clean up the HP hardware related drivers, utilities, agents using the HP Proliant Support Pack Cleaner from Guillermo Musumeci. Executec:\Temp\P2V\PSPCleaner.exe
• Run the System Configuration Utility on the virtual server by executing c:\Temp\P2V\v7-SystemConfigUtil.bat, select the “normal startup” option and reboot your server
• Show all hidden devices and uninstall any unused devices by executing c:\Temp\P2V\v8-hiddendevices.bat
  • Select show hidden devices
  • Check if there are still unused devices present and uninstall them manually
• Update the HAL if you changed from multi to uniprocessor by executing the script c:\Temp\P2V\v9-HALupdate.bat
• Reboot your server
• Install VMware Tools

Cheers!