Copy & Paste in VCenter Vsphere

I recently read a post on a blog by Francesco at thevirtualway.it  It tells how to configure your VM's in vcenter for copy & paste.  This can be super annoying in some regards if you use the Vcenter to go to the console of a VM.  It was one of those things that bugs me when I need to do it and can't, but know its possible.  I just had it way far down on my "to do list" and it never moves up on that list because its such a low priority.  Because of this I never had time to dig into how to fix this.  Thanks to Francesco posting at a online group I subscribe to, I saw it come up in my email and now its a quick fix for me and one worth passing along!

To do this you need to SSH access to the host and add these entries to the file /etc/vmware/config 

vmx.fullpath = "/bin/vmx"
isolation.tools.copy.disable="FALSE"
isolation.tools.paste.disable="FALSE"

You will need to Shutdown and power-on the virtual machine for changes to take effect.

Access Terminal Server Farm behind firewall WITHOUT RD GATEWAY!!!

In my last post I showed how easy it is to setup an RDS (terminal services) Farm.  You also will find out just how well it works... on your internal network.  The problem I quickly discovered is what seemingly almost every other person discovered when they setup a RDS Farm... accessing it from outside is not as simple as using an RDP client and logging into one of the terminal servers.  The behavior you will get is that it may or may not connect you.  When it does connect you to a RDS server it connects you exactly to the one you specified and other times it will not connect you.  This may lead you to believe that the Broker is not doing its job, but in fact it is.  If you look at the log files for the broker sessions on the Broker server at: EventViewer > Applications and Services Logs > Microsoft > TerminalServices-RemoteConnectionManager > Operational > TerminaServices-SessionBroker you will see that the broker is indeed doing its job and passing you to the RDS server it believes is less utilized based on load and connections.

Great... so you know its not the broker so what is it?  Well... you are outside the network and your Broker as well as your RDS servers are all addressed for your internal network.  Quick logic tells you to simply create a NAT rule on your firewall for all your RDS servers.  Unfortunately that won't work because your Broker is still telling your RDP client to go to another server whose IP address is  on your internal network.  When you are outside your network your RDP client is not going to find that internal network.

I cannot tell you how many things I have tried based on logic and what other users posted in forums.  The one trend you will see with 98% of the answers to solve this problem is to use the RD Gateway.  I agree that is the best solution especially because its more secure.  I would even suggest to anyone to do this.  However, on the network I work on we have over 20,000 devices on our network and a lot of them are Mac OS X devices.  The Microsoft RDP client for Mac OS X does not work with TS Gateway.  I know there are other RDP clients out there for the Mac OS that do work with TS Gateways but the good ones cost some coin so we wanted to avoid that if possible.

So here we go... To make your RDS Farm work for users outside your network you need to configure a NAT rule for each terminal server on your firewall.  They have to be separate outside IP's that you own and can put in your firewall and NAT'd to the internal IP of an RDS server.  So these will be a one-to-one NAT not a one-to-many... if you do a one-to-many you will basically create a loop for your broker.

Next, on each RDS server add a second NIC. (Easy for me because I run all VM's on VMware).  For our environment we created a new Vlan for these NIC's to be sure they did not talk to the firewall and have ARP freak out over a conflicted IP address because of those NAT rules we just created.  Address each of the second NIC's with the outside IP address that corresponds to the NAT rule you created.  (i.e. Use the outside IP from the rule for TS01's because thats NAT'd to the internal IP of the same hostname).

Next be sure to go into each RDS servers and add that second address and NIC to be used in the Farm.  Open up Server Manager and go to: Roles > Remote Desktop Services > RD Session Host Configuration: SERVERNAME.  Under the "edit settings" section open up "Member of farm in RD Connection Broker".  The window where it says "Select IP Addresses to be used for reconnection" you should now see that second network adapter with that outside IP you gave it.  Select it and "OK" your way out.  Be sure to do this on each RDS server only.  YOU DON'T NEED TO DO THIS STEP ON THE BROKER!!!

Thats it!  Now your RDS servers are sending the login requests to your broker with each of its IP addresses,  and in turn your broker will send the RDP client your connecting from two IP addresses for the RDS server in the farm it wants you to connect to, the internal IP and the external IP!

See the diagram below for the visual representation.

I will note that the IP's in the diagram are not my own but used for the purposes of visualizing the solution.

Some notes:

- You do not need to add any additional DNS entries to your internal DNS servers for the RDS servers or farm name that contain the outside IP addresses.

- For us we made the Vlan the second nics are connected to a "dead" Vlan.  Meaning nothing routes or from it.  We had to do this so that all the ESXi hosts on our network could Vmotion the RDS server VM's to each other and have them still communicate on those second NICs.

- Nothing on the firewall is routed to those second NICs, nor should there be.  You don't want a direct connection to the second NIC of your RDS servers from outside if you can help it.  (I read a post where someone solved this problem by doing just that).  That's why we NAT the outside IP to the internal IP of each server.

- Don't believe any post that talks about using a Microsoft Loop Back adapter on the RDS servers... it does not work.

- Don't even begin to think you can simply edit the local "hosts" file on these servers to accomplish this task.  It won't work.

There you go.  This was a 3 day long, brain-bending problem for me.  In the end it was one of the most gratifying solutions I have done in IT!

Cheers!

RDS Farm setup (formerly Terminal Services Farm).

Its been a few months since my last post but that time has not seen me idle by any means.  ;)

Today is going to be the start of a lengthy one followed but another, but one that definitely warrants it because it appears there are a lot of other Admins who ran into my situation but could not figure out how to overcome the final hurdle.

(From here on out I will refer to terminal services as RDS which is Microsoft's new name for that role... Remote Desktop Services).

Our network consists of 150ish servers, 8,000ish users and about 250ish vlan's.  We are a big organization to say the least.  We have never really had a widespread need for users to use RDS.  That changed recently when the software company that does our payroll system (which does timekeeping, vacation time, sick time and of course payroll decided to move to a cloud solution).  Most end users (such as myself) only use a web portal to request time off, enter hours or view their pay stubs.  Other users such as managers, payroll staff and the accounting staff have to have an application installed to do their managerial and payroll tasks in the system.  The problem being is that we have a lot of staff that are on Mac's and the client application for this system is not compatible with Macs.  To top it off, the team that installs applications on the users' computers does not like the idea of having to go to a bunch of PC's and install this application.  So the idea of RDS is brought up to me.  Which of course prompts me to think of the amount of users that would be using this RDS so my thoughts turn to high availability and future growth/needs of RDS.  Enter and RDS farm project!

Setting up the RDS farm was very simple.  I already had one RDS VM for my departments use.  I simply cloned it twice and changed the SID on it after installing the new client application for our new payroll system.  After renaming the clones and joining them to the domain, I setup the farm.

I do not yet have a need for the Published web apps side of RDS so I will focus here on the install of the RDS farm utilizing only the Session host service so users can log into the RDS servers the old school traditional way we think of a "Terminal Server".

Let me quickly explain the role of the "Broker" in an RDS Farm.  The broker is basically the traffic cop for RDS connections.  It keeps the RDS logins between the servers in the farm balanced based on connection and load.  Some people in smaller environments will put this service on one of their RDS servers but in larger environments it will be best to put it on its own server.  I setup three RDS servers and wanted the capability to grow the farm, so I want this service to be separated from a server that end users will log into.

For the record, as of this post the Microsoft RDP application for Mac OSX is not compatible with Server 2012 RDS so I used Server 2008r2 for all the servers in the farm.  I know there are very good RDP apps for MACOSX that will work with server 2012 (iTap) but the good ones cost money which our organization is not favorable of.

To start, create your broker VM or server.  Its as easy as installing the Remote desktop services role and choosing only the "Remote Desktop Connection Broker" service.

Click next all the way through accepting any dependencies that need to be installed.

Once that role is installed on the broker you need to go into the local users and groups on that server and add all the RDS servers that will be in your farm to the group named "Session Broker Computers".

 Before adding the Servers you will need to select "object types" and select "computers" so the names or IP's of the servers resolve.

Now type in the name or IP of your servers you plan to use in the RDS farm to be added to the group.  ***NOTE: YOU DO NOT NEED TO PUT THE SERVER WITH THE BROKER ROLE IN THIS GROUP UNLESS IT TOO IS A RDS SERVER YOUR USERS WILL USE".

Thats it for the Broker! There is nothing else to do here!

Now we will install the roles and configure the servers that the actual users will be using.

On each RDS server you want the users to use you need to install the "Remote Desktop Services" role with just the "Remote Desktop Session Host" service.

Accept all the defaults that follow.

Now we need to configure each server for the farm!

In the server manager drill down to the "RD Session Host Configuration".

Now we will be working under the "Edit Settings" window.  In licensing you need to configure your licensing.  I have user CAL's for our environment and have the licensing server setup on another server in our domain.  On your terminal servers you just have to specify that server that is doing the licensing service.

Next, in this same window we need to open "Member of farm in RD Connection Broker".

The properties window that opens up needs the following: The name of your server you setup for "RD Connection Broker" and a name for your RDS Farm in "Farm Name".  This is the first time you have named the farm so call it what you wish, but you will need to make sure that you use the same farm name in all the terminal servers with the session host service.  This is also how you would create multiple farms with the same broker server managing them.  So each server with the same farm name is in the same farm.  (I hope that explains it well).  "Check Participate in Connection Broker Load-Balancing"  Also, in the window for "Select IP address to be used..." you want to select your NIC that should have your static IP of the server listed.  Leave the setting for "Relative Weight of this server in the farm" to its default.  This is the actual connections it will allow on this server.  If you have older hardware or are worried about the performance of the RDS server you would drop this down so it would eventually deny connections to keep performance stable.  Im running on Virtual machines so Im not worried about this as I can give the VM's more memory or processors if needed.

OK... final step.  Your going to need to create a DNS record for each IP address of your RDS session host servers with the name you gave your farm.  For example: if you have two RDS Session servers with the IP's of 192.168.1.10 and 192.168.1.11, and you named your Farm "TS-Farm" on each RDS Server, you need to create two DNS A records on your internal DNS server for each of those IP's with the name "TS-Farm".  This is so when the Broker sends your users out to find a RDS server to log onto it knows what servers are in the farm.  It uses DNS round-robin which by default is on for server 2008 and 2012 domain controllers.  I will assume you know how to do this step since its a common admin task and I would be blacking out a ton of stuff in the pictures at this point.  :)

To use the farm simply have all your users log into the same RDS session host server.  You don't have to give your users all the names or IP's of the servers in the farm.  Better yet, make an A record on your internal DNS server called something like: remote.yourdomain.com and point it to just one of the IP's of a session host in your RDS farm.  What happens is when a user uses an RDP client with that address, the server will pass the request to the broker and the broker will send the client to any of session host servers in your farm based on load and/or session count.  So they won't always end up on that one server you have all the users logging into based on the IP or CNAME record.

Thats it!  You know have a terminal server farm.  If you want to have fun, watch the logs on the broker to see the broker do its work.  For each session request you should see up to 4 records.  You will see how the broker handles it and ultimately puts the user on the selected destination session host.  The ones you want to watch for fun are found in server manager at: EventViewer > Applications and Services Logs > Microsoft > TerminalServices-RemoteConnectionManager > Operational

***Note*** If a user disconnects a session instead of logging off, the broker will always put them back to the same session host they left their disconnected session at.  Its good to setup a good Group Policy for your terminal server so users are locked down from doing damage to the entire server.  I even put things in my policy to terminate sessions that have been disconnected and idle for 8 hours.

Like I said, this part was very easy.  I spent three days with our Network Administrator working out getting the farm to work from outside our network.  What you do here with these instructions will work flawlessly on your internal network.  Everything else you read in forums at Microsoft and other places on the net from other admins is that they finally settled on a RD Gateway to get the farm to work outside their network.  YOU CAN GET THE FARM TO WORK OUTSIDE YOUR NETWORK WITHOUT A RD GATEWAY!!!  I will show you how to do this in the next post.  We spent many, many hours troubleshooting with trial and error but we did get it working and anyone can do it.  I will try and get this posted for you next week.

Cheers!

VCenter 5.1 SQL Database install/setup

Per my previous post pertaining to losing my VCenter in our data center this week I had a crash course in SQL database setup for VCenter 5.1.  I come from working 10 years in the VAR business which was all SMB IT support so any VCenter setups I had done in the past I was able to get away with SQL Express databases for VCenter.  With VCenter 5.1 there are now two databases needed.  One for the SSO and obviously one for the VCenter.  My previous post outlines how to install the database for the SSO and this step is required before you can install VCenter 5.1 so if you have not done that yet you need to!

If you have or are planning to have more than 5 hosts and 50 VMs than you cannot use the SQL Express version for VCenter.  I will out line what I did to install a full SQL 2008 r2 database in preparation for your VCenter install.  Keep in mind that as of this writing VMware does not yet support SQL 2012 or Server 2012 for its SSO or VCenter products.

 - First: Create an individual role account called "SQL" or "SQLDB".  I am very anal and can't stand it when people tie all their services to the domain administrator account, or worse... the user account of the Network Admin.  (Its one of my huge pet peeves).

- Make sure you have .NET installed and updated.  On server 2008 this is installed by installing the Application server role in the server manager.  Make sure to run windows updates after that to patch it.

- Launch your SQL 2008 r2 install and click on "Installation" and "New installation" which is the first option.

There will be some checks that run.

- The next screen is your product key screen so what you need here and click next, accept the terms and click next.

- On the Setup Support Files screen you should keep the defaults and click next.

- If you run into any warnings on the next screen be sure to address them as neccesary.

- Next, leave the default selected: "SQL Server Feature Installation".

- Now we are going to check the following: Database Engine Services, Client Tools Connectivity, Integration Services, Management Tools - Basic, Management Tools - Complete.  Change your directory paths as you wish.  So it should look like this:

- Click next until you get to the instance configuration screen.  Here you can get fancy and change the default settings to what you want but I choose to keep it simple and not create headaches and leave it all default.

Click next until you get to the Server Configuration screen.

- Here select the "Use the same account for all SQ: Server services" and put in your SQL account you created in AD earlier.

Make sure to change the start-up type to "automatic" for the SQL Server Agent before clicking next.

- On the "Account Provisioning" screen, select "Mixed Mode (SQL Server Authentication and Windows Authentication).  Set a password for the built in SQL Server system administrator account.  This is the "sa" or Super User account for the database.

- Under "Specify SQL Server Administrators" you can add any additional users or groups that will need to maintain the database.  I added a group that our SQL folks are a part of.

- That's it.  Keep moving through the rest of the screens and you are done!  Make sure to run windows updates again after words.

Database Configuration

OK... now you got SQL installed so we need a database for the Vcenter.

- Start the SQL Server Management Studio which is in the default area of the program files under the start menu.  Log in with the "sa" user account.

- Right click on the topmost SQL Server object which is usually name after the server you are on or "localhost" depending on how you logged in.

- Choose "properties", then the "Memory" page.  I configured mine like this for my environment to start.  You can always change it later and increase the memory.  For now I set the maximum server memory to 50% of the memory the server actually has on the host.

- Click OK and your back at the home screen.  Now right click the databases folder and select "new database".  

- Under "General" i set my database name to "VCDB", but this you can safely use what you want.  Under the "options" set the recovery mode to "simple".  Click OK and you now have a new database.

- Right click the "security" folder.  Pick "new", "Login".

- Under "general" enter the username "vpxuser", select "SQL Server Authentication"  and enter a password.  I uncheck the "enforce password policy", "enforce password expiration", and "user must change password at next login".  Set the default database to "VCDB" and language to English.

- Click the "user mappings" page on the right.  Check the "Map" box for VCDB. then choose db_owner from the role membership list below.  Then check the "map" box for msdb, and choose db_owner.  I also select the "..." area in the "Default Schema Column" for both msdb and VCDB and set the default schema to "dbo".  Click OK.

- Now we need to configure the TCP/IP options.  Vmware has good documentation on how to do this.  It cam be found here and I really recommend following this guideline.

- Next the "local launch" permissions needs to be set.  Go to Administrative tools and "Component Services".  Go to "Component Services", "Computers", "My Computer", "DCOM Config".  Right click "MsDtsServer100" and choose properties.  Under the Security tab and "Launch and Activation Permissions", select "Customize", then "Edit".  Add the "SQLDB" user account you created.  This will prevent some common errors that pop up in logs and ensures scheduled jobs run.

- Lastly we need to create an OSBC data source.  This is one where again, VMware's library notes are very good and easy to follow.  You can find them here.

That is it!  Your set to install VCenter now using your own SQL database!

Happy virtualizing!

VCenter 5.1 SSO Install

It has been a whirlwind of a week for me as a Datacenter admin.  It started with a Monday off for vacation, a Tuesday off for a stomach bug, and I wasn't in my chair at the office for no more than 15 minutes Wednesday before the VCenter in our datacenter crashed hard.  The memory and processor was tacked at 100% and would not budge due to java.exe.  The best I could tell it was the VMware update manager causing this.  The VMware forums were full of other users with this issue and they all said to uninstall and re-install update manager, but for me it made the problem worse.  I made the decision to rebuild the Vcenter from scratch.  This meant my first run-in with the new and now required VMware SSO, as well as my first time installing full SQL instead of SQL express for the Vcenter database.  I inherited this Vcenter and have only done lots of small deployments of Vcenters in years past.  So, I thought I would pass along my findings in hopes it makes other Admins lives a little less stressful as I spent a total of 2 days rebuilding the Vcenter and our Veeam backup server to work with the rebuilt Vcenter.

As I already mentioned, VMware SSO is now a requirement before installing VCenter server.  I, along with many, many other admins out there do not think this should be a requirement.  It seems to be a push for VMware to cram security down our throats.  While I agree security should be at every admin's top priority, there is such thing as going overboard.  I think back to my days of supporting SMB's where 2 or 3 esxi hosts were all that needed to be managed for a Vcenter and this SSO requirement just adds a huge, thick layer into those installs.  Read more here about it to understand it.

The easiest way to do the SSO in a large environment is as follows:

- Install SQL server 2008r2 on server 2008r2.  (Vmware does not support SQL 2012 or server 2012 for Vcenter or SSO as of this writing).  Configure your instance how you want.  (I am assuming you know how to do this already).

- Open up SQL Server Management Studio.

- Go the following location on the Vcenter Server install Disc: \VCenter 5.1\Single Sign On\DBScripts\SSOServer\schema\mssql

- Click and launch the Script: rsaIMSLiteMSSQLSetupTablespaces

- You should now see the script in SQL Management studio like this:

- This script will create the SSO database for you.  I strongly recommend you don't be like me and try to get all fancy and name the database the way you want because your anal and want to be able to see a database name that makes sense to you down the road.  I spent many hours doing this... just stick to the defaults and what I write here and you will be off and on your way.

- You need to change three lines in this script for the location of the database files.  If you read the green text its pretty clear how to do this.  The lines you need to change all have C:\Change Me\.... marked in red.  Just put your path to where you want the database files in place of those lines.

So go from this:

To this:

***IT IS VERY IMPORTANT TO NOT CHANGE THE FILE NAMES IN THE LINE***  Your database location may vary on your setup.

- Run and execute that script and your SSO database named "RSA" will be created.

- Now you have to setup the user and DBA user for that database.  Thankfully VMware provided another script in that same location of the VCenter installer you just lifted the last one from!  With your SQL management studio still open, launch this script: rsaIMSLiteMSSQLSetupUsers

- The only thing you will want to change in this one is the passwords for the users its going to create.  Leave everything else the default and make sure to record the user names "RSA_DBA" and "RSA_USER" with the passwords you give them in this script.  Give them good strong passwords.  The lines you want to change are in red:

Make sure to leave the ' marks and put your password in between them for each user.

- Execute that script and your are done.  Now you can install the VMware SSO from the installer disc.

- During the install select "Create the primary Node...." because you would not have needed to do any of the above otherwise.  :-)

- At the next options menu select "create the primary node..."

- type in a new password and confirm it for the new admin sso account.

- Now choose "use an existing supported database" because you just created one.

- The next screen is where you are going to put in your information for the "RSA" database you created as follows:

The JDBC stuff is for certificates which I am not covering here, and most VMware admins of small or medium datacenters like myself don't use them anyway.

- After that the final "next" will connect to the database, create that sso admin user account and your off and running.  The install takes a little bit of time so go grab a cup-o-coffee.  :-)

Now you can move on and install Vcenter.  I will post soon on how to install a full SQL database to prep for VCenter to install to.  There most likely are other VMware admins out there like me who don't have a clue on SQL.

Happy Virtualizing!

Unmount VMtools Installer with Orchestrator

Here is a super-saucy Vmware Orchestrator workflow to un-mount the VMtools installer from all the VMs in a cluster you choose.  I have been working tirelessly on it these last couple of days and got it to a point where it is successful!  I needed this workflow because I manage 140 VMs and when I patch or update the 9 hosts in the cluster, the VMs start complaining about their tools being out of date.  I created another script for updating all the VMs tools in the clusters (here), but as many VMware admins will attest to, the Linux VMs don't always unmount after upgrading.  Thats where this workflow will come in!

The workflow is here for you to download.  ***RUN AT YOUR OWN RISK***

Here is a snapshot of the workflow:

Happy Virtualizing!

VM Tools upgrade with Orchestrator

Unless your have been living under a rock or in a video game virtual world in your parents basement for the last year, you already know that Automation in IT is the big buzz word.  Further more, you probably already know that Vmware comes bundled with Orchestrator when you install your VCenter.  Orchestrator comes with a lot of canned Workflows (scripts) that allow you to automate tasks in your Vcenter which when you couple with scheduling these tasks should make the life of an Enterprise IT admin much easier and free up some evening and weekend hours!

While Orchestrator comes with some good workflows to start you off, right off the bat I noticed there was one in particular that I was looking for that was not already a part of Orchestrator.  For the record, in order to create your own workflows, you need to have VCenter standard or higher licensing.  You may still use Orchestrator with the Essentials or Essentials plus license but you are limited to the built in workflows or importing them.

I set out on a hardy adventure to create my own workflow.  Don't worry... Orchestrator is heavy on the Gui side, but will require some scripting and further research on your part to figure it out... be patient.  The script in particular I was looking for was to update the VM tools on all the VM's in a cluster that I set.  I was able to do this successfully and have attached the workflow to this post.  Feel free to download it and import it into Orchestrator for your own use.  ***USE AT YOUR OWN RISK***

Here is a snapshot of the workflow:

Happy Virtualizing!