Let's talk about Cisco UCS - Unified Computing System.

I help stand up new IT infrastructure all over the world, and I have been seeing a lot more of these lately. It's a pretty impressive system. In most small to mid-size shops you tend to see an onsite server closet or maybe a small cage in a datacenter full of 2 or 3 generations old HP Proliants and Dell Poweredges. But for the largest scale enterprise operations, nothing beats the density and manageability of blades. (And their ability to lock you in with a particular vendor. ;)) Blade systems essentially do for hardware what hypervisors did for operating systems. Not only are you packing more into less and increasing your compute density, you're centralizing the management of your entire datacenter and simplifying the deployment process by orders of magnitude. What do I mean by that? Well have some pictures worth a thousand words:

(Try right-clicking the images and opening in a new tab and you might get a better view.)

Turn this...

... into this.

(The above image is courtesy of dalgeek - knightfoo.wordpress.com.)

Turn this...

... into this.

(I took that picture on the left myself a couple years ago, from a place I used to work.)

Now, when we talk about Cisco UCS, we're actually talking about a few discrete components that come together to form the UCS. First, we have the fabric interconnect. We'll use the Cisco UCS 6120XP 20-Port Fabric Interconnect as an example.

It's a specialized 1U 10Gb (ten gigabit) switch that supports up to 160 servers or 20 chassis as a single, seamless system. (And remember each "server" can have dozens of VMs on it.) This particular switch is capable of 520Gbps of throughput. (I keep feeling like I'm making typos when I type numbers that large.)

The next piece is the blade chassis itself. Take the Cisco UCS 5108 Blade Server Chassis for example. This thing is 6 rack units, making the entire solution so far 7U for what could potentially house hundreds of VMs. Those smaller ports on the bottom of the chassis are for power supplies. Note that you can cram either half-width blades or full-width blades into this chassis. A full-width blade would look a little more like the traditional pizza box that we're used to, and has room for more stuff in it obviously, but I think the extra agility offered by half-width blades is probably the reason that they're the only ones I really see out in the wild.

And lastly we have the blades themselves. Take the Cisco UCS B200 M2, a half-width blade, for example:

*Why yes, that is 192GB of DDR3 RAM, thanks for noticing*

And here's a little artist's depiction of what an entirely fleshed out "Unified Computing System" would look like. Note that you'd probably want some SAN storage somewhere for this to be considered a complete solution, beyond just the couple of disks that you can stick into each blade. I wonder how much storage you could get up there in the top 4 to 6 U of each cabinet...

Hardware characteristics such as MAC addresses are configured at the chassis slot level, so if a blade fails you can swap in a new blade and not have to reconfigure anything. You can also do things like automatically reboot a host onto another blade if one fails, etc.

And lastly - it is all managed from a single web interface. (I hope you like Java.)

So that all looks pretty amazing, right? There may be a couple of cons to going with Cisco UCS however, and there are alternative blade systems to consider as well. You just have to weigh these pros and cons for yourself and your enterprise's situation. One of these possible cons is cost. The old adage goes that nobody in IT ever got in trouble for buying Cisco. They do make great stuff, but they also make practically the most expensive equipment in existence. Exact pricing is complicated and of course depends on exactly how you configure your equipment, but list price is somewhere in the ballpark of $20,000 per blade. Don't worry though, no one pays list price. Especially if you were to make a huge order like this, Cisco would be expected to have their discount pen at the ready. $10,000 - $12,000 per blade might be a more realistic figure. I count 288 blades in the picture above, putting your budgetary needs at somewhere around $2.9 - $3.46 million USD. (And we still don't have storage or networking yet... but you are well on your way to having one of the densest datacenters in the world.)

In the types of environments that I'm most used to, I see one to two UCS chassis per datacenter, each with two clustered interconnects for redundancy. In contrast, you might decide to go with HP c7000 blade servers filled with BL465c's. I see some of these as well, especially as things like cloud technologies cause aggressive IT expansion and thus the need to do more with your budget. You would almost certainly save a substantial amount of cash if you did go with HP or Dell; however, I think Cisco still has a very compelling price-per-blade as the solution scales out extremely well, and you only pay for the management stack once. (Or twice if you're really reaching for the stars like we did above.)

So in conclusion, I'll just leave you with a couple last things. Here is Cisco's UCS In A Nutshell documentation if I've whet your appetite and you want more information. And here is a Cisco UCS emulator, if you'd like to play around with what it feels like to administer one of these things. And lastly, here are some tutorials to go along with that emulator software.

'Till next time!

Alright, so this blog has been up for a few months now, and so far I've only blogged about IT topics, Windows Server, networking... those sorts of things.  I've not made a single post yet that hints at what a life-long video game junkie I am.  I'm about to remedy that with this post by going over some of the video games that I've been playing recently.

Battlefield 3

BF3 is my bread and butter. It seems like I play it almost every day. I've been an avid fan of the Battlefield franchise since Battlefield 1942... I thought it was great, and its Desert Combat mod made it legendary, so when Battlefield 2 came out, I was right there and played it regularly for years. Then I skipped Bad Company 1, but I did play a lot of Bad Company 2. Now we're at Battlefield 3. Every game in this franchise is amazing - at least from an online multiplayer standpoint. On the other hand, the single player campaign in most all of the Battlefield games that I've played is nothing spectacular. BF3 is no exception. I played through the single player campaign once and then never looked back. But no one should be buying a Battlefield game for the single player. For multiplayer tactical military warfare that will have you shooting people in the face with buckshot, dogfighting with other fighter jets, blowing holes in buildings with your grenade launcher to get at that sniper, and ripping your victim's dogtags off with your combat knife, all while coordinating with your team to ensure victory... there is simply no better game than Battlefield 3 right now. The graphics are unparalleled, the controls are responsive, the ballistics and hitboxes feel right... it's just a great game. One of my favorite things is that even though I've been playing the game for months, it seems like I'm still hearing new sound clips from the people around me that I've never heard before.  Things like "there's a machine gunner settin' up over there!" and "holy shit grenaaaaaade!" There are lot of those little voice clips that add to the immersion.

There are only two things that I don't like about this game and its publishers - bring back mod support, and kill Origin. (I know neither of those will ever happen, but I can dream.) I hate Origin. I am seriously considering not playing Mass Effect 3 (even though I want to) simply because Origin disgusts me.

Modern Warfare 3

I bought Call of Duty: Modern Warfare 3 because one of my coworkers talked me into it. The single player campaign in MW3 is pretty good. Better than Battlefield 3's, for sure. It's action packed and entertaining, but mostly in the sense that an action movie is action packed and entertaining. I still would not consider it to have any replay value. The graphics don't appear to have advanced at all from the last Modern Warfare game. The multiplayer is absolutely no comparison to BF3. Modern Warfare's multiplayer game is arcade-like and twitch-based... it still feels like you're playing Quake III Arena, running down halls and hoping you pull the trigger faster than the other guy. There are no vehicles to drive, no open maps to roam around in... I don't know - it's not a bad game, but it just doesn't hold nearly as much interest for me as BF3. Also, it suffers from this horrible multiplayer synchronization bug where I keep getting "out of sync" with the other person that I'm playing with. E.g., it will appear to me like he has died, so I run over to go revive him, but then on voice chat he says "what are you talking about dude, I'm fine!" And it doesn't seem like the problem is getting patched. It still happens.  Inexcusable.

Assassin's Creed Revelations

I have strong feelings about this game and franchise. I think the Assassin's Creed games are fantastic games, and a pretty damn good story to boot. I didn't play the first Assassin's Creed when it came out. But Assassin's Creed II went on a Steam sale so I picked it up. I was floored. What a great game! (Seriously, Leonardo da Vinci building contraptions for me? How much cooler can you get?) After finishing it, I immediately got back on Steam and grabbed AC:Brotherhood. Again, extremely fun and interesting. Then because I was loving the storyline and the style so much, I grabbed the first Assassin's Creed game and tried to play through it. Unfortunately I wasn't able to make it through because AC:II and AC:B were so much better than the first one, they had spoiled me. I waited eagerly for months for AC:Revelations to come out. I watched the trailer at least a dozen times.

I bought it on day one. It's still good, but I don't think it's as good as ACII or AC:B. It's short. The whole Lucy/Shawn/Rebecca/real life part of the story is gone. The characters are not as interesting or as vivid as the characters from Italy like Rodrigo Borgia and Caterina Sforza. There are no more beautifully rendered chapels full of artwork. (They tried, and some scenes from AC:Rev do look really awesome, but there's only so much to work with in the Middle East.) And when it ended, I was left with a bitter taste in my mouth. I literally sat there watching the credits roll, waiting for something else to happen. I felt cheated by the lack of resolution in the ending. I guess I expected that a game called Revelations was going to offer some... revelations, or tell us something that we didn't already know. Honestly, it left me feeling like maybe Ubisoft actually wasn't concerned about telling this story in a rich and artistic way as much as just making those yearly releases on time so they can make some more money.

But, a few months after finishing it, I've pretty much come to terms with it. It's still a good addition to the franchise, and I've come to accept that all of Ezio's life and part in this story is pretty small in the grand scheme of things. I was expecting too much out of Revelations and out of Ezio himself; as if I expected everything to be resolved and wrapped up right there and then by this game.

I'm ready to see the next assassin - a whole new generation, a whole new time and place. I hear it's going to be taking place during the American Revolution, with some Native American influences... but it's going to be hard to outdo Ezio. Ezio was an incredibly memorable character, and we got to know him literally from the moment of his birth, until his death.

Ezio lives to a ripe old age and apparently dies a natural death from a heart attack on a public bench in Florence, Italy, with his wife and daughter nearby. Not at all a bad way to go for someone who spent their entire life as an international assassin.

Ubisoft - if you're listening, I just want to say this: Remember that you have a chance to tell an amazing story here. A work of art. It can be more than just a video game. It can live on long after you and the game's profits are gone. Please don't ruin that opportunity.

Batman: Arkham City

Not a whole lot to say about this game other than "it rocks." I got Batman Arkham Asylum bundled with a video card upgrade a couple years ago, and I freakin' loved it. Extremely nice graphics, combat that made you feel like a real badass, and more gadgets than you knew what to do with.  So I knew that Arkham City - the sequel to Arkham Asylum - was going to be an instant buy for me.

Arkham City is just more of what made Arkham Asylum great. The combat is even better, the gadgets are even cooler, and the world is more open and free-roam-able. Plus, you can play as Catwoman, who is even more agile than Batman. Talk about kicking some major ass, ninja style.

The only downside I can think of is that there is a bit of a DirectX 11 stuttering problem, but you can easily switch DX11 off.

The only thing that gets me about Batman, is that doesn't he realize that he's actually putting more people in more danger by keeping criminal masterminds like Joker alive? I get that there is sort of a dynamic between Joker and Batman where they subconsciously keep each other alive because without each other, they would both lose their sense of purpose, even though they'll never admit that... but still, Batman's absolute refusal to kill anyone, even when it results in putting everyone else in mortal danger, annoys me.

Dead Island

This game is interesting in that it doesn't seem like it was very well received. When the trailer was first released, everyone gawked at how awesome it was. You know, that video where everything happens backwards to the tune of a melodramatic piano and it turns out that it's a family fighting off a zombie invasion? It's here by the way. You owe it to yourself to watch it if you've not seen it. It's probably one of the best video game trailers ever made.

Alright, well the actual game didn't turn out much like that trailer, and I think that disappointed a lot of people.

However, I still really enjoyed Dead Island. (My current love of the TV show "The Walking Dead" may be related.) It's got that sort of multiplayer where teammates are free to come and go sort of like in Left 4 Dead. The acting and story line is pretty bad, but the scenery is beautiful, the zombies just keep coming, ammo is scarce so you're always feeling desperate, and the game world is very big and you're free to roam around in it.

If you had a small group of dedicated friends with voice chat - this game could simply be an amazing multiplayer experience. In that sort of "OK, you go to the lifeguard station and I will go get some gas from the gas station, and then we'll meet back here. Watch out for zombies" kind of way. However, just dropping in on a random game with one or two people you don't know... may not be so great.

Warhammer 40K: Space Marine

I love the Warhammer 40k universe. It would make a fantastic MMO universe... just sayin'. It's got pure testosterone-based badassery like 8-foot tall genetically engineered space marines that destroy all of the emperor's enemies... and it also has a touch of comic relief like an "Ork" race that is ostensibly stupid, but apparently so prolific that there is no end to their numbers. When Warhammer 40K: Dawn of War came out years ago, it was a real-time strategy game like Starcraft, and I absolutely loved it. Dawn of War II took that similar RTS concept and boiled it down to where you only controlled one or two squads at a time, basically took away all of the base-building, and it was up to you to tactically maneuver those one or two squads around the terrain. That was met with mixed reactions. Some people enjoyed the more personal, tactical aspect... but some people preferred the more large-scale war-like feel of throwing hundreds of troops against one another in an epic battle.

Warhammer 40k Space Marine distills it even further to where you're actually just controlling one single space marine in a 3rd person over the shoulder type of game. There are a couple of other people with you at various parts of the game but of course they're largely useless. It's up to you to blow the heads off of endless hordes of Orks and Chaos Marines. I still enjoyed the game for what it was - stomping the heads of Orks, mowing them down with your auto-bolter, etc... the graphics are nice, the controls are nice, multiplayer is fun, but it does get old after a while.

Nothing too out of the ordinary.

Skyrim

Skyrim. This game was hugely popular. I played Morrowind in 2001. I played Oblivion a few years later. They're OK... but none of these games have ever really knocked my socks off. I can't just complete the main questline without feeling like I'm missing out on all those side-quests and free-roaming, yet at the same time, if I just go off and do my own thing, then I feel like I have no sense of direction or purpose. I can't just do the side missions without feeling like I'm letting the dragons/demons/bad guys rape and pillage in the meantime.

Anyway, the graphics are fantastic, obviously. The combat is fine. I hate fighting trolls. I'd rather face a dragon than a troll I think.

Anyway, I finished the main story line, but I'm just not as excited about it as most people are. I might have put 40 hours into it. I still prefer modern or sci-fi themes... see above with the Warhammer 40k MMO idea.

All that said, I will enjoy coming back to this game in a year or so, after the modders have had some time to create some really cool mods for this game.

Dustforce

I've been playing quite a bit of this game. It's very fun. It's addictive, and possibly most importantly - it has a fantastic soundtrack. Seriously. Just do a Youtube search for "Dustforce OST". It's good stuff.

It's 2012, and ISPs are still slow to adopt IPv6. It seems like very few of us can say that we have globally-accessible IPv6 addresses. Myotherpcisacloud.com doesn't even have an IPv6 address yet... and that makes me a very sad panda. But there is something I can do about it right now, without the help of my ISP.

If you have a Cisco router, I can show you how to create an IPv6 tunnel that will you have dual-stacked and on the IPv6 Internet in no time! This article assumes that you cannot use native IPv6 out to the Internet, and that you already have the router properly set up and in use in an IPv4 network.

The router I will use in this example is a 2621XM; I bought it for $150 on eBay. It has two FastEthernet ports. It was manufactured in 1999. So any model at least as recent as that should be able to handle this just fine. I do IPv4 NAT between the two FE ports so that the rest of my home network served by my AT&T U-Verse Residential Gateway stays separate from my lab network, but the lab still has to go through the U-Verse gateway to reach the Internet. (U-Verse claims that they'll push an IPv6 firmware upgrade out automatically to all their customers sometime in 2012, but I'll believe it when I see it.)

*There's still some juice left in this crusty old thing*

For this to work for me, I needed to configure my U-Verse Gateway to put my Cisco router in "DMZ+" mode, and allow the outside interface of my Cisco router to receive a DHCP address. This allows my U-Verse gateway to assign my router the same public IPv4 address as itself, and forward all unspecified traffic to it.

We’re going to utilize the free service at Hurricane Electric for this. Follow that link and sign up. It’s their "Tunnel Broker" service that you’re after. After a short quiz, they will give you your very own IPv6 tunnel and your very own IPv6 address space! For free!

All you need to do now is configure your router. If you've never used Cisco IOS, these commands might look weird to you. They're shorthand for things like "enable" - enter "enable" mode which allows us privileged access so that we can make configuration changes to the router. "conf t" is shorthand for "configure terminal" - meaning "I wish to make configuration changes to this router from the terminal."

Router>en
Router#conf t
Router(config)#ipv6 unicast-routing
Router(config)#exit
Router#copy run start

At this point you have enabled ipv6 routing globally on your router. "Copy run start" is shorthand for "copy the running configuration to the startup configuration, effectively making these changes permanent."

Next, create a tunnel on your router like this:

Router>en
Router#conf t
interface Tunnel0
description Hurricane Electric IPv6 Tunnel Broker
no ip address
ipv6 enable
ipv6 address 2001:470:1f0e:5a4::2/64 (Use your side of the endpoint that Hurricane electric gave you!)
tunnel source 75.32.98.76 (Your public IPv4 address)
tunnel destination 216.218.224.42 (Hurricane Electric’s IPv4 endpoint for this tunnel)
tunnel mode ipv6ip
ipv6 route ::/0 Tunnel0
end
write

And you’re pretty much done! Configure your clients with an IPv6 address in that space, and you now have IPv6 connectivity all the way to the Internet. Google has a public DNS server at 2001:4860:4860::8888. Test out your tunnel by trying to ping that address. Remember that IPv6 and IPv4 are quite different. There is no NAT in IPv6. (Let's not talk about NAT64 yet.) Internet communication is the way it was truly meant to be – end to end. That also means the need to protect yourself with firewalls will become more important than ever, since you can’t hide behind a NAT anymore!

Now you can surf the web with a “dual-stack,” meaning that you’re runnnig both IPv4 and IPv6 — and your IPv4 packets will take their normal route, while your IPv6 packets will be diverted through your new tunnel. Seamlessly. Pretty neat huh? Try to ping ipv6.google.com and see what happens!

I guess that’ll have to do until ISPs catch up with IPv6 technology.

The 2012 Scripting Games won't be starting for over a month, but I'm excited about them. I'm going to give it a try, and hopefully I don't get stomped! If you're interested in participating, click the image to go to the 2012 Scripting Games FAQ.

I'm back, finally.

I was recently challenged with trying to not only enumerate all the network adapters on a system across dozens of different operating system versions and hardware platforms, but also to try to figure out where they are physically in the machine, remotely, without being able to see the actual hardware.

The short answer is you can't.

The long answer is you can't... do it scriptomatically without the assistance of vendor-specific software, such as the HP network configuration software and maybe an API or WBEM queries... but that's only going to cover one specific hardware platform. I need to consistently gather this data across not only Proliants, but Poweredges, VMs, desktop workstations, anything that runs Windows. Windows doesn't know where in space your network adapters are. By that I mean Windows doesn't know which physical port on your 4-port NIC is the third one from the left, etc. This would be why there is seemingly no rhyme or reason as to which network adapter Windows assigns "Local Area Network", "Local Area Network #2", "Local Area Network #3", etc. The installed NICs are enumerated randomly, as evidenced by the fact that you may get different results for which NIC port is assigned to which network connection every time you re-install Windows on a multi-NIC machine. I have heard that some particularly anal administrators even go so far as to install Windows, then delete all the Network Connections that are out of order, and continue removing and letting Windows reinstall them until they are all in the "correct" order. There is also a theory that NIC manufacturers of multi-port NICs should give each port on the card sequential MAC addresses, starting from the port closest to the PCI bus. So you might be able to infer something from that, but that's not something I would put money on for thousands of NICs with dozens of manufacturers.

Furthermore, "NIC teaming" throws yet another wrench into this, as now you can no longer rely on what Windows thinks the MAC address of a teamed adapter is, or what the cabinet switch thinks the MAC address is on a given switch port that has a teamed NIC plugged in to it.

I can get you all the information that Windows does have though, including (apparent) MAC addresses, IPs, and "Location Information" as read from the registry. This is that "Bus 0, Device 8, Function 25" stuff that you might have seen in Device Manager. It might be useful in drawing some correlations, but it's still not going to tell you much about physically where all these NICs are.

So without further ado, here are the scripts. The first one is Powershell. The second one is the exact same but ported to VB Script, for compatibility with older versions of Windows. Note the operating system version check in the VB Script.

Powershell:

$ErrorActionPreference = 'Stop'
$nics = Get-WmiObject Win32_NetworkAdapter
$cfgs = Get-WmiObject Win32_NetworkAdapterConfiguration

Write-Host "`nPhysical NICs In No Particular Order"
Write-Host "------------------------------------`n"
foreach ($_ in $nics)
{
	Try
	{
		if($_.PNPDeviceID.StartsWith('PCI'))
		{
			$registryKey = Get-Item HKLM:\System\CurrentControlSet\Enum\$($_.PNPDeviceID)
			$keyValues   = Get-ItemProperty $registryKey.PSPath
			$regSplit    = $keyValues.LocationInformation.Split(";") 
			$location    = $regSplit[2].Replace('(','').Replace(')','')
			$locSplit    = $location.Split(",")			
			
			Write-Host "Name    : $($_.Name)"
			Write-Host "MAC     : $($_.MACAddress)"
			Write-Host "Location: Bus $($locSplit[0]), Device $($locSplit[1])`, Function $($locSplit[2])"
			$mac = $_.MACAddress
			foreach ($cfg in $cfgs)
			{
				if($cfg.MACAddress -eq $mac -And $cfg.IPAddress)
				{
					Write-Host "IP      : $($cfg.IPAddress)"
				}
			}
			Write-Host " "	
		}	
	}
	Catch {	}
}

VB Script:

Option Explicit
const HKEY_LOCAL_MACHINE = &H80000002

Dim nic, objNICs, objCfgs, objWMIService, objReg, objOSVer
Dim strWMIQuery, strRegistryKey, strValue, strLocInfo, strBus, strDevice, strFunction, strOSMajor
Dim arrSplitKey, arrSplitLoc, arrOSBuild
Dim mac, cfg, ip, v

strWMIQuery = "SELECT * FROM Win32_NetworkAdapter"
Set objWMIService = GetObject("winmgmts:\\.\root\CIMv2")
Set objNICs = objWMIService.ExecQuery(strWMIQuery)
strWMIQuery = "SELECT MACAddress,IPAddress FROM Win32_NetworkAdapterConfiguration"
Set objCfgs = objWMIService.ExecQuery(strWMIQuery)
strWMIQuery = "SELECT Version FROM Win32_OperatingSystem"
Set objOSVer = objWMIService.ExecQuery(strWMIQuery)

For Each v in objOSVer
	arrOSBuild = Split(v.Version,".")
Next

strOSMajor = arrOSBuild(0)

Wscript.Echo "Physical NICs In No Particular Order"
Wscript.Echo "------------------------------------"

For Each nic In objNICs
	If StrComp(Left(nic.PNPDeviceID,3),"PCI",1) = 0 Then
		Set objReg = GetObject("winmgmts:{impersonationLevel=impersonate}!\\.\root\default:StdRegProv")
		strRegistryKey = "System\CurrentControlSet\Enum\" & nic.PNPDeviceID				
		objReg.GetStringValue HKEY_LOCAL_MACHINE,strRegistryKey,"LocationInformation",strValue
		If CInt(strOSMajor) >= 6 Then
			arrSplitKey = Split(strValue,";")
			strLocInfo = arrSplitKey(2)
			strLocInfo = Replace(strLocInfo,"(","")
			strLocInfo = Replace(strLocInfo,")","")
			arrSplitLoc = Split(strLocInfo,",")
		End If
		
		Wscript.Echo "Name    : " & nic.Name
		Wscript.Echo "MAC     : " & nic.MACAddress
		
		If CInt(strOSMajor) >= 6 Then
			Wscript.Echo "Location: Bus " & arrSplitLoc(0) & ", Device " & arrSplitLoc(1) & ", Function " & arrSplitLoc(2)
		Else
			Wscript.Echo "Location: " & strValue
		End If
		
		mac = nic.MACAddress
		For Each cfg In objCfgs
			If StrComp(cfg.MACAddress,mac) = 0 And isNull(cfg.IPAddress) = False Then
				For Each ip In cfg.IPAddress
					Wscript.Echo "IP      : " & ip
				Next				
			End If
		Next
		Wscript.Echo " "
		If isObject(objReg) Then Set objReg = Nothing
	End If
Next

The output looks like this:

The IPs are not shown on the second adapter because it's switched off right now and thus doesn't have any IPs. My first idea for improvement of the Powershell version (I don't invest much time into improving VBS,) is making custom objects out of the output instead of just doing Write-Hosts. The power of Powershell is in its ability to deal with objects, and so you should try to keep everything as objects for as long as possible. Once you've spit it out on the screen in a Write-Host statement for example, you can no longer pass it along the pipeline, etc.

Thanks to Kelvin Wong and Server Fault for helping me research this.

It's been a while since I posted, so I figured I'd show you a little something I whipped out a few days ago. The script is a sort of "domain health report," and it sends out a nicely-formatted email with its findings. I have the script set in a scheduled task to run nightly. Every morning when I wake up, the email is there waiting for me in my inbox. The script uses the Active Directory Powershell module to get a list of all the computer accounts and user accounts in your domain. After displaying some general domain stats, based on the enabled computer accounts that it finds, it then attempts to find information from all of those machines. The information will be highlighted in red and bold if it falls below a certain threshold, e.g. disk space below 10%, an SSI below 7, etc.

So without further ado:

# DomainReport.ps1
# Emails a report of various metrics collected from every computer in the domain.
# This script is intended to be run automatically, on a schedule of once a day or so,
# to let us know how our domain is doing.

[string]$senderName   = "Domain Health Report"
[string]$senderAddr   = "dc1@domain.myotherpcisacloud.com"
[string]$recptName    = "Ryan Ries"
[string]$recptAddr    = "ryanries09@gmail.com"
[string]$emailSubject = "Domain Health Report"
[string]$smtpServer   = "smtp.domain.myotherpcisacloud.com"
[string]$emailBody    = ""
[int]$staleCompAcctDays = 60
[int]$staleUserAcctDays = 60
[int]$diskFreePercentThreshold = 10
[int]$SSIIndexThreshold = 7

Import-Module ActiveDirectory	# It will not hurt if the module is already loaded.

$localhost = Get-Content env:Computername
$domain = Get-ADDomain
$forest = Get-ADForest
$allComputerAccts = Get-ADComputer -Filter * -Properties *
$enabledComputerAccts = Get-ADComputer -Filter 'Enabled -eq $true' -Properties *
$allUserAccts = Get-ADUser -Filter * -Properties *
$enabledUserAccts = Get-ADUser -Filter 'Enabled -eq $true' -Properties *

Function Ping-Server
{
    param($hostName)
    trap
    {
        $false; continue
    }
	$object = New-Object System.Net.NetworkInformation.Ping
	$object.Send($hostName, 2000) #2000ms ping timeout
}

$emailBody += "<FONT STYLE=`"font-size:30px;`">"
$emailBody += "Domain Health Report"            
$emailBody += "</FONT>"
$emailBody += "<FONT STYLE=`"font-size:9px;`">"
$emailBody += "<BR/>Report executed from $localhost at $(Get-Date)<HR/>"
$emailBody += "</FONT>"
$emailBody += "<FONT STYLE=`"font-family:Monospace;font-size:13px`"><BR/>"
$emailBody += "<strong>Forest Root Domain:</strong> $($forest.RootDomain) ($($forest.ForestMode))<BR/>"
$emailBody += "<strong>Current Domain:</strong> $($domain.Name), NetBIOS $($domain.NetBIOSName) ($($domain.DomainMode))<BR/>"
$emailBody += "<BR/>"
$emailBody += "<strong>Domain Controllers:</strong> $($domain.ReplicaDirectoryServers.Count) Writable, $($domain.ReadOnlyReplicaServers.Count) RODCs, $($forest.GlobalCatalogs.Count) Global Catalogs<BR/>"
$emailBody += "<strong>Schema Master:</strong> $($forest.SchemaMaster)<BR/>"
$emailBody += "<strong>Domain Naming Master:</strong> $($forest.DomainNamingMaster)<BR/>"
$emailBody += "<strong>Infrastructure Master:</strong> $($domain.InfrastructureMaster)<BR/>"
$emailBody += "<strong>RID Master:</strong> $($domain.RIDMaster)<BR/>"
$emailBody += "<strong>PDC Emulator:</strong> $($domain.PDCEmulator)<BR/>"
$emailBody += "<strong>Sites:</strong> $($forest.Sites)<BR/>"
$emailBody += "<BR/>"
$emailBody += "<strong>Computer Accounts:</strong> $($allComputerAccts.Count) found, $($enabledComputerAccts.Count) enabled<BR/>"
if($allComputerAccts.Count -gt $enabledComputerAccts.Count)
{
	$emailBody += "<strong>Disabled Computer Accounts:</strong> "
	ForEach($_ in $allComputerAccts)
	{
		if($_.Enabled -eq $false)
		{
			$emailBody += "$($_.CN)`, " 
		}
	}
	$emailBody = $emailBody -Replace "..$" # Trim off the last two characters
	$emailBody += "<BR/>"
}
$emailBody += "<strong>Stale Computer Accounts<sup>*</sup>: </strong> "
ForEach($_ in $allComputerAccts)
{
	if($_.PasswordLastSet -lt $((Get-Date).AddDays(-$($staleCompAcctDays))))
	{
		$emailBody += "$($_.CN)`, "
	}
}
$emailBody += "<BR/><BR/>"
$emailBody += "<strong>User Accounts: </strong> $($allUserAccts.Count) found, $($enabledUserAccts.Count) enabled<BR/>"
if($allUserAccts.Count -gt $enabledUserAccts.Count)
{
	$emailBody += "<strong>Diabled User Accounts:</strong> "
	ForEach($_ in $allUserAccts)
	{
		if($_.Enabled -eq $false)
		{
			$emailBody += "$($_.SAMAccountName)`, "
		}		
	}
	$emailBody = $emailBody -Replace "..$"
	$emailBody += "<BR/>"
}
$emailBody += "<strong>Stale User Accounts<sup>*</sup>: </strong> "
ForEach($_ in $enabledUserAccts)
{
	$lastLogon = [DateTime]::FromFileTime($_.LastLogonTimeStamp)
	if($lastLogon -lt $((Get-Date).AddDays(-$($staleUserAcctDays))))
	{
		$emailBody += "$($_.SAMAccountName)`, "
	}
}
$emailBody += "</FONT><BR/><BR/>"
$emailBody += "<FONT STYLE=`"font-size:9px;`">* A `"stale`" computer account is one that has not updated its machine password with AD in $staleCompAcctDays days."
$emailBody += "<BR/>* A `"stale`" user account is not disabled but has not logged on to the domain in $staleUserAcctDays days."
$emailBody += "</FONT>"
$emailBody += "<HR/><BR/>"

ForEach($_ in $enabledComputerAccts)
{
	$emailBody += "<FONT STYLE=`"font-size:16px;`">"
	$emailBody += "<strong>$($_.CN)</strong> <BR/>"
	$emailBody += "</FONT>"
	$emailBody += "<div style=`"border-width:1px;border-style:solid;margin:2px;padding:2px;`">"
	$emailBody += "<FONT STYLE=`"font-family:Monospace;font-size:13px`">"
	$pingNode = Ping-Server $($_.CN)
	$emailBody += "<strong>Ping:</strong> "
	if($pingNode.Status -ne "Success")
	{
		$emailBody += "<FONT STYLE=`"color:red;`"><strong>NO REPLY!</strong></FONT><BR/>"
	}
	else
	{
		$emailBody += "$($pingNode.RoundTripTime) ms reply from $($pingNode.Address)<BR/>"
	}
	$computerSystem = Get-WmiObject Win32_ComputerSystem -ComputerName $($_.CN)
	$emailBody += "<strong>System: </strong> $($computerSystem.Manufacturer) $($computerSystem.Model)<BR/>"
	$latestStabilityIndex = Get-WmiObject Win32_ReliabilityStabilityMetrics -ComputerName $($_.CN) | Select-Object -First 1 | ForEach {$_.SystemStabilityIndex}
	$emailBody += "<strong>Latest SSI<sup>*</sup>: </strong>"
	if($latestStabilityIndex -gt 0 -and $latestStabilityIndex -le 10)
	{
		if($latestStabilityIndex -lt $SSIIndexThreshold)
		{
			$emailBody += "<FONT STYLE=`"color:red;`"><strong>$latestStabilityIndex<BR/></strong></FONT>"
		}
		else
		{
			$emailBody += "$latestStabilityIndex<BR/>"
		}		
	}
	else
	{
		$emailBody += "<FONT STYLE=`"color:red;`"><strong>NO DATA!</strong></FONT><BR/>"
	}
	
	## Don't want to use $log.Count here because it seems to be implemented inconsistently in the Get-Eventlog cmdlet,
	## e.g. sometimes it is null when it should be zero, and vice versa, and still other times it throws an exception
	## for no matches found.

	$emailBody += "<strong>Application Log Errors Last 24hrs: </strong>"
	Try
	{
		$appLogErrors = Get-EventLog -Log Application -EntryType Error -After $(Get-Date).AddHours(-24) -ComputerName $($_.CN)
		if($appLogErrors -eq $null)
		{
			$emailBody += "0<BR/>"
		}
		else
		{
			## This technique doesn't work if $log is null. $counter goes to 1 when it should stay at 0.
			$counter = 0
			$appLogErrors | ForEach-Object { $counter++ }
			$emailBody += "$counter<BR/>"
		}
	}
	Catch 
	{ 
		$emailBody += "<FONT STYLE=`"color:red;`"><strong>$($_.Exception.Message.ToString())</strong></FONT><BR/>" 
	}
	
	$emailBody += "<strong>System Log Errors Last 24hrs: </strong>"
	Try
	{
		$sysLogErrors = Get-EventLog -Log System -EntryType Error -After $(Get-Date).AddHours(-24) -ComputerName $($_.CN)
		if($sysLogErrors -eq $null)
		{
			$emailBody += "0<BR/>"
		}
		else
		{
			$counter = 0
			$sysLogErrors | ForEach-Object { $counter++ }
			$emailBody += "$counter<BR/>"
		}

	}
	Catch
	{
		$emailBody += "<FONT STYLE=`"color:red;`"><strong>$($_.Exception.Message.ToString())</strong></FONT><BR/>"
	}
	
	$emailBody += "<strong>Security Audit Failures Last 24hrs: </strong>"
	Try
	{
		$secLogErrors = Get-EventLog -Log Security -EntryType FailureAudit -After $(Get-Date).AddHours(-24) -ComputerName $($_.CN)
		if($secLogErrors -eq $null)
		{
			$emailBody += "0<BR/>"
		}
		else
		{
			$counter = 0
			$secLogErrors | ForEach-Object { $counter++ }
			$emailBody += "$counter<BR/>"
		}

	}
	Catch
	{
		$emailBody += "<FONT STYLE=`"color:red;`"><strong>$($_.Exception.Message.ToString())</strong></FONT><BR/>"
	}

	$emailBody += "<strong>Total RAM: </strong>$([math]::Round($computerSystem.TotalPhysicalMemory/1GB,0)) GB <BR/>"
	$emailBody += "<strong>Logical Disks:</strong>"
	$emailBody += "<div style=`"border-width:1px;border-style:dashed;margin:8px;padding:8px;background-color:`#dddddd`">"
	$computer = $($_.CN)
	ForEach($_ in $(Get-WMIObject -Query "SELECT DeviceID FROM Win32_Logicaldisk WHERE DriveType=3" -Computer $computer | ForEach { $_.DeviceID }))
	{
		$logicalDisk = Get-WMIObject -Query "SELECT * FROM Win32_Logicaldisk WHERE DeviceID='$_'" -Computer $computer
		$freespace = [math]::Round($logicalDisk.FreeSpace/1GB,0)
		$totalSize = [math]::Round($logicalDisk.Size/1GB,0)
		if((($freespace/$totalSize)*100) -lt $diskFreePercentThreshold)
		{
			$emailBody += "<strong><FONT STYLE=`"color:red;`">$($logicalDisk.DeviceID) ($($logicalDisk.VolumeName)) $freespace GB free out of $totalSize GB </FONT></strong><BR/>"
		}
		else
		{
			$emailBody += "$($logicalDisk.DeviceID) ($($logicalDisk.VolumeName)) $freespace GB free out of $totalSize GB <BR/>"
		}	
	}
	$emailBody += "</DIV>"
	$emailBody += "</FONT></DIV><BR/><BR/>"
}

$emailBody += "</FONT>"
$emailBody += "<FONT STYLE=`"font-size:9px;`">* SSI = Windows System Stability Index. Configure WMI Reliability Providers across your domain via Group Policy and ensure that the RAC scheduled task is running on the machines in order to gather this data.<BR/>"
$emailBody += "* The Remote Registry service must be running on remote computers in order to gather event log data."
$emailBody += "</FONT>"

Send-MailMessage -From "$senderName <$senderAddr>" -To "$recptName <$recptAddr>" -Subject "$emailSubject" -Body $emailBody -SMTPServer $smtpServer -BodyAsHTML

I really like the Win32_Reliability classes, Win32_ReliabilityRecords and Win32_ReliabilityStabiltyMetrics. I used one of them in a previous post. They basically hold records of all the useful system events that relate to system configuration and stability, such as unexpected shutdown events, application errors and software installs/uninstalls, etc. To boot, Windows uses all those events to calculate a System Stability Index. Some people might think the SSI is unnecessary, but I personally really like it as a quick at-a-glance number that I can use to give me an idea of overall system health when I have a thousand machines to look at. It's basically an index from 0 to 10 that fluctuates based on the aforementioned system stability events. Machines with an SSI below a certain number need to be looked at more closely, you get the idea.

The difference is in my previous post, I didn't realize that the Win32_Reliability classes are not enabled by default on Windows 2008 R2 servers. On Windows 7 they are enabled by default, and on the one Windows 2008 Server (non-R2) on which I used them, they were functioning, which means that they're either enabled on 2008 Server by default or someone had turned them on previously.

You can, of course, access both these WMI classes in Powershell with the good old Get-WMIObject that we all know and love, like this:

Get-WMIObject win32_reliabilityrecords
Get-WMIObject win32_reliabilitystabilitymetrics

On a Windows 2008 R2 server that does not have these two classes enabled, you will get the error

Get-WmiObject : Provider load failure

whether you are executing the Powershell cmdlet locally or remotely. So as I started to research this problem, it seemed to be a simple matter of enabling the GPO setting "Configure Reliability WMI Providers." (This article from The Scripting Guy is pretty much all you need for that.) So I did that and applied it to all of my servers. And then I waited. I waited for 24 hours. Still nothing. I got onto one of the servers and ran gpupdate /force. Then I waited some more. (Maybe it needs time to gather the data, right?) 24 hours later, nothing. Rebooted the server. Nothing.

OK, that GPO setting is obviously not the only piece of the puzzle here. I researched a little more and The Scripting Guy showed up yet again!

So there is a Scheduled Task named "RacTask" in Scheduled Tasks -> Task Scheduler Library -> Microsoft -> Windows -> RAC. (Make sure you are set to view hidden tasks, just in case.) That task has two triggers - one that only fires when a new Application log event 1007 from Customer Experience Improvement Program shows up, and another that runs indefinitely every hour. On Server 2008 R2, by default, the first trigger is enabled while the latter trigger is disabled. (On client OSes like Win7, both triggers are enabled by default.) So the GPO setting alone would have worked, except that I had not gotten an event ID 1007 from CEIP in three days. Event 1007 from CEIP is "Successfully sent CEIP data to Microsoft." I have only gotten Error 1008s (Failure to send data to Microsoft) in the past three days. I'm choosing that to mean there's something wrong with Microsoft's SQM servers at the moment. Maybe they're down for maintenance or just too busy...

Needless to say, you'd never get event 1007s at all if you opted out of the Customer Experience Improvement Program, in which case simply changing that GPO setting would definitely not be enough. I'm not saying that you have to participate in CEIP on your servers if you want to use the Win32_Reliability monitors. But you do need to enable that second trigger on the scheduled task. Enable the trigger, run the task, and then you'll be able to access the WMI classes immediately, locally and remotely.

$latestStabilityIndex = Get-WmiObject Win32_ReliabilityStabilityMetrics -ComputerName $server | Select-Object -First 1 | ForEach {$_.SystemStabilityIndex}

That's how you kick it off manually. I should note that I received a 1007 (data sent successfully) on one of my servers the next day, which enabled the monitors as expected. (The CEIP uploader is set to attempt to collect and upload data every 19 hours by default.)

So the moral of the story is enabling the GPO setting "Configure Reliability WMI Providers" in the Computer Config -> Administrative Templates area is enough to enable the use of the Win32_Reliability WMI classes on your Win2K8R2 servers if they are participating in CEIP and you are willing to wait until they are able to successfully upload CEIP data, which could take one to several days. Otherwise, you're going to have to find a way to also kick off that scheduled task on all your servers, be it manually or scriptomatically.

I don't feel like this was altogether implemented that well in that regard. I do like the reliability data, but I don't feel like it should be related to or dependent on CEIP events at all. Also, while trying to come up with hypothetical ways to automate the enabling of this so that I wouldn't have to log on to every server:

Come on Microsoft, get it together!

Hey guys. No interesting and deeply technical documentation today. I've been pretty busy with work, and also going over Powershell material in preparation of teaching a Powershell Boot Camp at work. In the mean time though, here's something interesting I read today. It's a quote from Nikola Tesla written in the New York Times on the subject of Thomas Edison, the day after Edison died:

"He had no hobby, cared for no sort of amusement of any kind and lived in utter disregard of the most elementary rules of hygiene  ... His method was inefficient in the extreme, for an immense ground had to be covered to get anything at all unless blind chance intervened and, at first, I was almost a sorry witness of his doings, knowing that just a little theory and calculation would have saved him 90 percent of the labor. But he had a veritable contempt for book learning and mathematical knowledge, trusting himself entirely to his inventor's instinct and practical American sense."

Reading that instantly made me think of the more popular, and now more amusing quotes from Edison himself:

"Genius is one per cent inspiration and ninety-nine per cent perspiration."
"I have not failed. I've just found 10,000 ways that won't work."

An interesting perspective on the two diametric inventors, is it not?

Work smarter, not harder, folks.

For the first installment of this series, click here. This is yet another post that required hours of research and testing and resulted in me learning way more about various tangential things than I realized I wanted to know.

Ready to wrap this up with some real security goodness? Me too.

When you operate a small network or LAN, passing your credentials over the wire via easy-to-crack hashes and sending other traffic as clear text might be acceptable to you - or more accurately, you don't spend any time thinking about it. But when your network is large, heterogeneous and spans dozens of cities and continents, and has hundreds of internet-facing nodes and thousands of employees and external users... you need to start paying more attention to security. Well actually you needed to have started paying more attention to security way before you got to that point, but you get my meaning. You never know who might be listening with an intent to uncover privileged information - whether outside hackers or internal employees.

When I first set out writing this post, I didn't realize that all WinRM traffic is already encrypted by default. Even when sent over the HTTP protocol and even when Negotiate/NTLM authentication is used. It's still encrypted. But I don't mean to conflate authentication and data encryption right off the bat. Let's start with encryption. Here is an example of the header of an HTTP packet sent to a WinRM server:

POST /wsman?PSVersion=2.0 HTTP/1.1
Connection: Keep-Alive
Content-Type: multipart/encrypted;protocol="application/HTTP-SPNEGO-session-encrypted";boundary="Encrypted Boundary"
User-Agent: Microsoft WinRM Client
Content-Length: 1662
Host: server1:5985

SPNEGO is used to negotiate the strongest authentication type available to both nodes, and then everything in the payload of the packet between the "--Encrypted Boundary--" tags... is gibberish. There is no clear text in the packets. I've looked. The only bit of useful information I could find so far while sniffing on the wire was the domain\username used for authentication. I'm guessing that this encrypted data sent over HTTP must be new to WinRM 2.0, because this Technet article implies that data sent over HTTP was clear text at some point.

*Click for larger*

Since the nodes are using the Negotiate protocol, they will settle on the best authentication mechanism that they can both agree upon. (Kerberos > NTLMv2 > NTLMv1, etc.) In this case it's going to be NTLMv2 authentication since they can't use Kerberos since they aren't in the same AD domain. Windows machines haven't used the old easy-to-crack LAN Manager or NTLMv1 password hashes in over 10 years. If you search through your packet capture, (use the "ntlmssp" display filter in Wireshark to easily find the relevant authentication traffic,) you will find several hashes of varying lengths, including the server challenge, client response, client challenge, HMAC, session key, MIC, etc. What's important though is that nothing that directly exposes the actual password is sent over the wire. If the machine has already been compromised, the attacker could access the local SAM and get an unsalted password hash that, using something like Rainbow Tables, could eventually be decoded into the original password. But that's only if the Security Accounts Manager on the local machine has already been compromised. The hash that's stored there is not sent over the wire.

This guy right here has an ongoing, amazing 6-part (and growing) exposé on getting at those infamous NTLM hashes. It always involves already having access to the machine though, and almost always involves old versions of Windows and exploiting flaws that have since been fixed.

In the first part of this tutorial, we did a basic set up of the WinRM service on a standalone computer named SERVER1. Then we connected to that computer from a domain-joined machine named DC01 to demonstrate using the Negotiate authentication protocol to connect to machines that are outside of your Active Directory trust boundary. By the way:

It should be noted that even within a domain, for Kerberos authentication to work when using WinRM, an SPN for the service must be registered in AD. As an example, you can find all of the "WSMAN" SPNs currently registered in your forest with this command:

setspn -T yourForest -F -Q WSMAN/*

SPN creation for this should be taken care of automatically, but you know something is wrong (and Kerberos will not be used) if there is no WSMAN SPN for the device that is hosting the WinRM service.

As with last time, when I execute the command on DC01:

$creds = Get-Credential
$server1 = New-PSSession -ComputerName SERVER1 -Credential $creds -Authentication "Negotiate"

A persistent remote connection is made from DC01 to SERVER1 via WinRM.

The following excerpt from this MSDN article is a pretty good description of what's going on. I'll recap, poorly:

1. The nodes do a classic TCP/IP handshake. (ACK, SYN-ACK, NICE-TO-MEET-YOU-ACK)
2. Client (DC01) does an HTTP Get for the resource. (Hey can I just access you?)
3. Server (SERVER1) says "No way, I'm not that easy! (Returns 401 Unauthorized) ... but I am willing to negotiate, here's what I can do..."
4. Client evaluates the authentication methods on offer, and sends new token with new base64-encoded authentication junk in it.
5. Server accepts the challenge response by checking the hash against what it has in its own SAM, and allows Client to connect. (Returns HTTP 200)

So like I said, this communication is already pretty well protected. But maybe you want more. You can either set up an IPsec tunnel between the nodes, or you can enable certificate-based SSL encryption between the two nodes if you feel the need to further wrap all your packets in a warm blanket of security. We'll discuss certificate-based SSL encryption here.

First off, remember from Part I that WinRM just won't work with self-signed certificates. So we need a Certificate Authority. Luckily I've got one, but it's an ECA that belongs to a domain of which SERVER1 is not a member. No matter - to issue certificates to non-trusted parties, you simply need to add the web-enrollment junk to your ECA:

Add all of the web role services

I chose "Username and Password" authentication, but Integrated Windows Authentication might have worked as well, since the two computers are on the same network.

The next thing I'm going to do is install the certificate from the ECA into the Trusted Root CA store on SERVER1, so that SERVER1 will implicitly trust any certificate issued by my ECA. Export the certificate from the ECA, move the certificate to SERVER1, then Import the certificate to the correct store, you know the drill. You could also just browse to http://your-ECA/certsrv and download the CA certificate from there. Whatever works for you.

This next part gave me so much heartburn, and I'm hoping I can now save you some. You might remember from the first part of this tutorial that WinRM needs a certificate for "Server Authentication" purposes. The "Web Server" template in a basic set up of Certificate Services is for Server Authentication purposes. So we should just be able to use that one, right?

No one tells you this, but for WinRM to use the certificate, the private key must be marked as exportable when you request it from the CA. However, the original "Web Server" certificate template does not allow exporting private keys. So on your ECA, copy the Web Server template. Make sure you choose the "Windows Server 2003" option and not the "Server 2008" option, or else the new template will not show up in the drop-down menu on your Certsrv webpage. Name the copy something like "Web Server With Private Key." Modify the new template, and on the "Request Handling" tab, click "Allow private key to be exported." Nothing else needs to be changed unless you need to. Then, on the "Certificate Templates" node under your ECA, right click it, select New -> Certificate Template to Issue, and choose the new "Web Server with Private Key" template that you just created.

This article came in handy for me.

Next, on SERVER1, launch Internet Explorer. Make sure that the URL http://your-ECA/certsrv is added to SERVER1's Trusted Sites list, and modify your Internet Explorer security settings as needed so that you're able to run any and all ActiveX scripts that the ECA wants you to run, etc. Now browse to http://your-ECA/certsrv. You should need to provide credentials. (I bet it's through the same SPNEGO process that we witnessed earlier!) Make sure that the credentials you log in with have permissions to enroll in the certificate you will be requesting. You should get a pretty plain web page where you can request a certificate. If you don't, you've already gone astray.

Click on Request a Certificate, then submit an advanced certificate request. Then click Create and submit a request to this CA. Choose the "Web Server With Private Key" Certificate Template from the drop down menu that you created earlier. Yes, I know this is not a web server, but recall from part one of this tutorial that we need a certificate for "Server Authentication" purposes, and this certificate will give us that. You can spend a bunch of time playing around with the certrqtp.inc file on your CA and customizing the certificate request webpage to provide new templates if you want, but I don't really care about that right now. Also pay attention that as you change the certificate request type in the drop-down menu, you should be seeing the text boxes on the web page change around. This means the ActiveX junk is running successfully in the background.

So submit that request, and the website will either tell you that your certificate request requires approval (go approve it,) or it'll just give it to you right away, depending on the policy you set up on your CA. When you click the link to install your new certificate, the webpage will automatically install your new cert in your Current User > Personal store. Go look at it. It's important that it has the correct name on it (the subject and "CN=" part of the certificate needs to say SERVER1, etc.) and that there are no other validation errors. It should have "Server Authentication" in the Enhanced Key Usage field. Now they say that WinRM should be able to use this certificate whether it resides in the current user store or in the local computer store, but I had to export the certificate from there (including private key!) and then import it into the Local Computer > Personal store to get it to work. Finally, while you're here, open the properties of the certificate, and copy the Thumbprint. You'll need that in a second. (Reference)

Next, delete your old HTTP listener on SERVER1 with this command:

winrm delete winrm/config/Listener?Address=*+Transport=HTTP

You can only have one listener per network interface, from what I understand. Better we keep the config as simple as possible in any case. By the way, you can execute non-Powershell commands like this from within Powershell if you start the command with an ampersand. (& winrm delete winrm/config...)

Create your new HTTPS listener, configured to use your specified certificate, like this:

winrm create winrm/config/Listener?Address=*+Transport=HTTPS @{Hostname="SERVER1";CertificateThumbprint="1d9256aea461788764cec1904463120f094393f9"}

Where CertificateThumbprint is the thumbprint you copied off of the certificate a minute ago. If all goes well, you will get a "ResourceCreated" response from WinRM. Otherwise, the errors you are likely to see include "Cannot create a WinRM listener on HTTPS because this machine does not have an appropriate certificate. To be used for SSL, a certificate must have a CN matching the hostname, be appropriate for Server Authentication, and not be expired, revoked, or self-signed" or "A specified logon session does not exist. It may already have been terminated." I wrestled with both of those errors, and the only way I've gotten past them is to follow my above steps to the letter.

SERVER1 is now listening on port 5986, waiting for an SSL-protected connection. Keep in mind that since you didn't use quickconfig, it didn't automatically create a Windows Firewall rule for TCP 5986.

And finally - the fruit of our labor... So I've moved back to DC01. Keep in mind that DC01 is already configured to trust certificates that were issued by my ECA. I run this in Powershell:

$creds = Get-Credential
$server1 = New-PSSession SERVER1 -UseSSL -Credential $creds

And magic happens. And by "magic," I mean "no errors!" The connection established successfully with no fuss. And here's the cool part: This is a packet capture of the exact moment when the two nodes start communicating:

I don't even get to see the headers any more. It's all 100% complete gibberish, impervious to prying eyes, thanks to SSL encryption.

One last thing as a bonus for making it all the way through this article: In case you didn't know, Powershell creates "pseudo-drives," for lack of a better term, for certain repositories on your computer. For instance, in Powershell, type "cd cert:" to navigate around your Certificate Stores like a normal directory, and type "cd wsman:" to navigate around your WinRM configuration as if it were a regular file structure! Pretty cool, eh?

Hey guys. I should have called this post "Monitoring with Windows Remote Management (WinRM), and Powershell, and maybe a Certificate Services tutorial too," but then the title would have definitely been too long. In any case, I poured many hours of effort and research into this one. Lots of trial and error. And whether it helps anyone else or not, I definitely bettered myself through the creation of this post.

I'm pretty excited about this topic. This foray into WinRM and Powershell Remoting was sparked by a conversation I had with a coworker the other day. He's a senior Unix engineer, so he obviously enjoys *nix and when presented with a problem, naturally he approaches it with the mindset of someone very familiar with and ready to use Unix/Linux tools.

I'm the opposite of that - I feel like Microsoft is the rightful king of the enterprise and usually approach problems with Windows-based solutions already in mind. But what's important is that we're both geeks and we'll both still happily delve into either realm when it presents an interesting problem that needs solving. There's a mutual respect there, even though we don't play with the same toys.

The Unix engineer wants to monitor all the systems using SNMP because it's tried and true and it's been around forever, and it doesn't require an agent or expensive third-party software. SNMP wasn't very secure or feature-rich at first so now they're on SNMPv3. Then there's WBEM. Certain vendors like HP have their own implementations of WBEM. I guess Microsoft wasn't in love with either and so decided to go their own way, as Microsoft is wont to do, hence why you won't find an out of the box implementation of SNMPv3 from Microsoft.

One nice thing about SNMP though, is that it uses one static, predictable port.

In large enterprise IT infrastructures, you're likely to see dozens of sites, hundreds (if not thousands,) of subnets, sprinklings of Windows and Unix devices all commingled together... and you can't swing a dead cat without hitting a firewall which may or may not have some draconian port restrictions on it. Furthermore, in a big enterprise you're likely to see the kind of bureaucracy and separation of internal organizations such that server infrastructure guys can't just go and reconfigure firewalls on their own, network guys can't just make changes without running it by a "change advisory board" first, and it all basically just makes you want to pull your hair out while you wait... and wait, and wait some more. You just want to be able to communicate with your other systems, wherever they are.

Which brings us to WinRM and Powershell Remoting. WinRM, a component of Windows Hardware Management, is Microsoft's implementation of the multi-platform, industry-standard WS-Management protocol. (Like WMI is Microsoft's implementation of WBEM. Getting tired of the acronym soup yet? We're just getting started. You might also want to review WMI Architecture.) I used WinRM in a previous post, but only used the "quickconfig" option. Seems like most people rarely go any deeper than the quickconfig parameter.

Here's an excerpt from a Technet doc:

"WinRM is Microsoft's implementation of the WS-Management protocol, a standard Simple Object Access Protocol (SOAP)-based, firewall-friendly protocol that enables hardware and operating systems from different vendors to interoperate. You can think of WinRM as the server side and WinRS the client side of WS-Management."

I bolded the phrase that especially made my ears perk up. You see, Windows has a long history with things like RPC and DCOM. Those protocols have been instrumental in many awesome distributed systems and tool sets throughout Microsoft's history. But it just so happens that these protocols are also probably the most complex, and most firewall unfriendly protocols around. It's extremely fortuitous then that Ned over at AskDS just happened to write up a magnificent explication of Microsoft RPC. (Open that link in a background tab and read it after you're done here.)

Here's the thing - what if I want to remotely monitor or interact with a machine in another country, or create a distributed system that spans continents? There are dozens of patchwork networks between the systems. Each packet between the systems traverses firewall after firewall. Suddenly, protocols such as RPC are out the window. How am I supposed to get every firewall owner from here to Timbuktu to let my RPC and/or DCOM traffic through?

That's why monitoring applications like SCOM or NetIQ AppManager require the installation of agents on the machines. They collect the data locally and then ship it to a central management server using just one or two static ports. Well, they do other more complex stuff too that requires software be installed on the machine, but that's beside the point.

Alright, enough talk. Let's get to work on gathering performance metrics remotely from a Windows server. There are a few scenarios to test here. One is communications within the boundaries of an Active Directory domain, and the other is communications with an external, non-domain machine. Then, exploring SSL authentication and encryption.

winrm get winrm/config

to see the existing default WinRM configuration:

WinRM originally used ports 80 for HTTP and 443 for HTTPS. With Win7 and 2k8R2, it has changed to use ports 5985 and 5986 respectively. But those are just defaults and you can change the listener(s) back to the old ports if you want. Or any port for that matter. Run:

winrm enumerate winrm/config/listener

to list the WinRM listeners that are running. You should get nothing, because we haven't configured any listeners yet. WinRM over SSL will not work with a self-signed certificate. It has to be legit. From support.microsoft.com:

"WinRM HTTPS requires a local computer "Server Authentication" certificate with a CN matching the hostname, that is not expired, revoked, or self-signed to be installed."

To set up a WinRM listener on your machine, you can run

winrm quickconfig

or

winrm quickconfig -transport:HTTPS

or even

winrm create winrm/config/listener?Address=*+Transport=HTTPS @{Port="443"}

Use "set" instead of "create" if you want to modify an existing listener. The @{} bit at the end is called a hash table and can be used to pass multiple values. The WinRM.cmd command line tool is actually just a wrapper for winrm.vbs, a VB script. The quickconfig stuff just runs some script that configures and starts the listener, starts and sets the WinRM service to automatic, and creates some Windows Firewall exceptions. What is more is that Powershell has many cmdlets that use WinRM, and the entire concept of Powershell Remoting uses WinRM. So now that you know the fundamentals of WinRM and what's going on in the background, let's move ahead into using Powershell. In fact, you can emulate all of the same behavior of "winrm quickconfig" by instead running 

Configure-SMRemoting.ps1

from within Powershell to set up the WinRM service. Now from another machine, fire up Powershell and try to use the WinRM service you just set up:

$dc01 = New-PSSession -ComputerName DC01
Invoke-Command -Session $dc01 -ScriptBlock { gwmi win32_computersystem }

Returns:

You just pulled some data remotely using WinRM! The difference between using a "session" in Powershell, and simply executing cmdlets using the -ComputerName parameter, is that a session persists such that you can run multiple different sets of commands that all share the same data. If you try to run New-PSSession to connect to a computer on which you have not configured the WinRM service, you will get a nasty red error. You can also run a command on many machines simultaneously, etc. Hell, it's Powershell. You can do anything.

Alright so that was simple, but that's because we were operating within the safe boundaries of our Active Directory domain and all the authentication was done in the background. What about monitoring a standalone machine, such as SERVER1?

My first test machine:

• Hostname: SERVER1 
• IP: 192.168.1.10 
• OS: Windows 2008 R2 SP1, fully patched, Windows Firewall is on
• It's not a member of any domain

First things first: Launch Powershell on SERVER1. Run:

Set-ExecutionPolicy Unrestricted

Then set up your WinRM service and listener by running

Configure-SMRemoting.ps1

and following the prompts. If the WinRM server (SERVER1) is not in your forest (it's not) or otherwise can't use Kerberos, then HTTPS/SSL must be used, or the destination machine must be added to the TrustedHosts configuration setting. Let's try the latter first. On your client, add the WinRM server to the "Trusted Hosts" list:

We just authenticated and successfully created a remote session to SERVER1 using the Negotiate protocol! Negotiate is basically "use Kerberos if possible, fall back to NTLM if not." So the credentials are passed via NTLM, which is not clear text, but it's not awesome either. You can find a description of the rest of the authentication methods here, about halfway down the page, if you need a refresher.

Edit 1/29/2012: It should be noted that even within a domain, for Kerberos authentication to work when using WinRM, an SPN for the service must be registered in AD. As an example, you can find all of the "WSMAN" SPNs currently registered in your forest with this command:

setspn -T yourForest -F -Q WSMAN/*

SPN creation for this should have been taken care of automatically, but you know something is wrong (and Kerberos will not be used) if there is no WSMAN SPN for the device that is hosting the WinRM service.

OK, I am pooped. Time to take a break. Next time in Part II, we're going to focus on setting up SSL certificates to implement some real security to wrap up this experiment!