I was in Salt Lake City most of this week. Being surrounded by stark snow-covered mountains made for some wonderful scenery... it could not be more different than it is here in Texas. Plus I got to meet and greet with a bunch of Novell and NetIQ people. And eat an enormous bone-in ribeye that no human being has any business eating in one sitting.

But anyway, here's a little AD mystery I ran in to a couple weeks ago, and it may not be as simple as you first think.

As Active Directory admins, something we're probably all familiar with is member servers authenticating with the "wrong" domain controller. By wrong, I mean a DC that is in a different site than the member server, when there's a perfectly fine DC right there in the same site as the member server, and so the member server is incurring cross-site communication when it doesn't need to be. Everything might still function well enough as long as the communication between DC and member server is successful, but now you're saturating your slower inter-site WAN links with AD traffic when you don't need to be. You should want your AD replication, group policy application, DFS referrals, etc., to run like a well-oiled machine.

I often work in a huge environment with AD sites in many countries and on multiple continents, and thousands of little /26 subnets that can't always be easily grouped into a predictable supernet for the purposes of linking subnets to sites in AD Sites & Subnets. So I'm always alert to the fact that if I log on to a server, and I notice that logon takes an abnormally long time, I very well could be logging on to the wrong DC. First, I run set log to see which DC I have logged on to:

*DC01 is in Amsterdam*

So in this case, I noticed that while I had logged on to a member server in Dallas, that server's logon server was a DC in Europe. :(

You immediately think "The server's IP subnet isn't defined in AD Sites & Services or is associated to the wrong site," don't you?  Yeah, me too. So I went and checked. Lo and behold, the server's IP subnet was properly defined and associated to the correct site in AD.

Now we have a puzzle. Back on the member server, I run nltest /dsgetsite to verify that the domain member does know to which site it belongs. (Which the domain member's NetLogon service stores in the registry in the DynamicSiteName value once it's discovered.)

I also ran nltest /dsgetdc:domain.com /Account:server01$ to essentially emulate the DC locator and selection process for that server, which basically just confirmed what we already knew:

C:\Users\Administrator>nltest /dsgetdc:domain.com /Account:server01$ 
           DC: \\DC01.DOMAIN.COM (In Amsterdam) 
      Address: \\10.0.2.55 
     Dom Guid: blah-blah-blah 
     Dom Name: DOMAIN.COM 
  Forest Name: DOMAIN.COM 
 Dc Site Name: Amsterdam 
Our Site Name: Arlington 
        Flags: GC DS LDAP KDC TIMESERV WRITABLE DNS_DC DNS_DOMAIN DNS_FOREST FUL
L_SECRET WS 
The command completed successfully

So where do we look next if there's no problem with the IP subnets in AD Sites & Services?  I'm going with DNS. We know that domain controllers register site-specific SRV records so that clients who know to which site they belong will know what DNS query to make to find domain controllers specific to their own site.  So what DNS records did we find for the Arlington site?

Forward Lookup Zones
    _msdcs.domain.com
        dc
            _sites
                Arlington
                    _kerberos SRV NewYorkDC
                    _kerberos SRV SanDiegoDC
                    _kerberos SRV MadridDC
                    _kerberos SRV ArlingtonDC
                    _ldap     SRV NewYorkDC
                    _ldap     SRV SanDiegoDC
                    _ldap     SRV MadridDC
                    _ldap     SRV ArlingtonDC

OK, now things are getting weird.  All of these other domain controllers that are not part of the Arlington site have registered their SRV records in the Arlington site.  The only way I can imagine that happening is because of Automatic Site Coverage, whereby domain controllers will register their own SRV records into sites where it is detected that the site has no domain controllers of its own... combined with the fact that scavenging is turned off for the DNS server, including the _msdcs zone.  So someone, once upon a time, must have created the Arlington site in AD before the actual domain controllers for Arlington were ready.  What's more is that Automatic Site Coverage is supposed to intelligently use site link costing so that only the domain controllers in the next closest site provide "coverage" for the site with no DCs, not every DC in the domain. Turns out the domain did not have a site link strategy either - it used DEFAULTIPSITELINK for everything - the entire global infrastructure. So even after Arlington did get some domain controllers, the SRV records from all the other DCs stayed there because of no scavenging.

Here's the thing though - did you notice that almost every other domain controller in the domain had SRV records registered in the Arlington site, except for the domain controller in Amsterdam that our member server actually authenticated to!?

This is getting kinda' nuts.  So what else, besides the DNS query, does a member server perform in order to locate a suitable domain controller?

So after a client does a DNS query for _ldap._tcp.SITENAME._sites.ForestDnsZones.domain.com, and gets a response, the client then begins to do LDAP queries against the DCs given in the DNS response to make sure that the DCs are alive and servicing requests. If you want to see this for yourself, I recommend starting Wireshark, and then restarting the NetLogon service while the capture is running. If it turns out that none of the DCs in the list that was returned by the site-specific DNS query is responding to your LDAP queries, then the client has to back up and try again with a domain-wide query.

And that is what was happening. The client, server01, was getting a list of DCs for its site, even ones that were erroneously there, but I confirmed that it was unable to contact any of those domain controllers over port 389. So after that failed, the server was forced to try again with a domain-wide query, where it finally found one domain controller that it could perform an LDAP query on... a domain controller in Amsterdam.

Moral of the story: Always blame the network guys.

*This image copyright NASA*

Metal whiskers, also referred to as tin whiskers or zinc whiskers, are something that I've read about with some curiosity before. Science does not currently fully understand why many metals and alloys form these tiny whiskers over time. The phenomenon has been known since the early 20th century, but we still don't know much about why it happens or how to effectively stop it.  It is still being studied today, and you can find metal whiskers in the news being blamed for things like fires aboard aircraft.  Obviously, these whiskers can wreak havoc in an electronic system whose components are packed tightly together. These whiskers can grow out of the solder used to manufacture electronic equipment, and they can also grow out of other non-electrified pieces of metal like server rack rails and the metal parts of datacenter raised flooring tiles. These tiny little metal whiskers can then be shaken loose or scraped off by such actions as lifting the floor tile and sliding it across the surface of an adjacent tile, then blown into the air by the datacenter ventilation system, and then subsequently sucked into the power supplies of the computers housed within the datacenter.  Resulting short-circuits can cause electronic component failure, and even fire.

This question was asked today by someone on ServerFault, which rekindled my interest in the subject.  I also recommend reading the Wikipedia on it.  And I highly recommend visiting this NASA page - an entire page devoted to the phenomenon of metal whiskers.

From that page, if you just watch or read one thing from it, I specifically recommend this video, which is specifically about the damage metal whiskers can do in a datacenter environment.

In my last post, I showed how to get RDP/TS session information from a local or remote computer, including ClientName, on the command line using Powershell. (But it's not really Powershell. It's a PS wrapper around some C# that P/Invokes a native API. Which is pretty nuts but it works.)  Since I can't think of any other way to get this information on the command line, I thought it would be useful to convert the thing to native code. I named the program users.exe. It's written in C. Here is what it looks like:

C:\Users\joe\Desktop>users /?

USAGE: users.exe [hostname]

Not specifying a hostname implies localhost.
This command will return information about users currently logged onto
a local or remote computer, including the client's hostname and IP.
Users.exe was written by Ryan Ries.

C:\Users\joe\>users SERVER01

Session ID  : 0
Domain\User : System
Client Name : Local
Net Address : n/a
Conn. State : Disconnected

Session ID  : 1
Domain\User : System
Client Name : Local
Net Address : n/a
Conn. State : Connected

Session ID  : 25
Domain\User : DOMAIN\joe
Client Name : JOESLAPTOP
Net Address : 10.122.124.21 (AF_INET)
Conn. State : Active

I had a pretty rough time shaking the rust off of my native/unmanaged code skills, but I pulled it off.  That said, if you wanna give it a try, I would very much appreciate any bug reports/feature requests.

users.exe (63.50 kb)

I came across this question on ServerFault today, which I thought was quite interesting.  And so I spent all morning formulating a solution to it.  How do you go about getting the list of currently connected users of a Windows machine including the remote client's machine name, on the command line? You can type query session, but that doesn't tell you the client's computer name or IP address.  The various Terminal Services and Remote Desktop-related Windows event logs are of very limited help. The Users tab in Task Manager tells you the usernames and their session IDs, but not their computer name.  You've got a bunch of WMI classes like Win32_LoggedOnUser and Win32_LogonSession, but none of them seem to contain any data about the connected client's machine name or IP address.

I wanted to come up with a Powershell solution.  Something that could also be run over the network and not just on the local machine.  So after a couple hours of research, finding the WTSQuerySessionInformation documentation, and spending some quality time on pinvoke.net, here is the "Powershell" solution that I came up with:

(I put "Powershell" in quotes because there's one single actual Powershell command in the whole thing.  Add-Type.  It's really all .NET code, which P/Invokes an unmanaged DLL.  Nevertheless, it's pretty sweet that you can do all that from within Powershell.)

# QuerySessionInformation.ps1
# Written by Ryan Ries, Jan. 2013, with help from MSDN and Stackoverflow.

$Code = @'
using System;
using System.Collections.Generic;
using System.Text;
using System.Runtime.InteropServices;
public class RDPInfo
{
    [DllImport("wtsapi32.dll")]
    static extern IntPtr WTSOpenServer([MarshalAs(UnmanagedType.LPStr)] String pServerName);

    [DllImport("wtsapi32.dll")]
    static extern void WTSCloseServer(IntPtr hServer);

    [DllImport("wtsapi32.dll")]
    static extern Int32 WTSEnumerateSessions(
        IntPtr hServer,
        [MarshalAs(UnmanagedType.U4)] Int32 Reserved,
        [MarshalAs(UnmanagedType.U4)] Int32 Version,
        ref IntPtr ppSessionInfo,
        [MarshalAs(UnmanagedType.U4)] ref Int32 pCount);

    [DllImport("wtsapi32.dll")]
    static extern void WTSFreeMemory(IntPtr pMemory);

    [DllImport("Wtsapi32.dll")]
    static extern bool WTSQuerySessionInformation(System.IntPtr hServer, int sessionId, WTS_INFO_CLASS wtsInfoClass, out System.IntPtr ppBuffer, out uint pBytesReturned);

    [StructLayout(LayoutKind.Sequential)]
    private struct WTS_SESSION_INFO
    {
        public Int32 SessionID;
        [MarshalAs(UnmanagedType.LPStr)]
        public String pWinStationName;
        public WTS_CONNECTSTATE_CLASS State;
    }

    public enum WTS_INFO_CLASS
    {
        WTSInitialProgram,
        WTSApplicationName,
        WTSWorkingDirectory,
        WTSOEMId,
        WTSSessionId,
        WTSUserName,
        WTSWinStationName,
        WTSDomainName,
        WTSConnectState,
        WTSClientBuildNumber,
        WTSClientName,
        WTSClientDirectory,
        WTSClientProductId,
        WTSClientHardwareId,
        WTSClientAddress,
        WTSClientDisplay,
        WTSClientProtocolType
    }

    public enum WTS_CONNECTSTATE_CLASS
    {
        WTSActive,
        WTSConnected,
        WTSConnectQuery,
        WTSShadow,
        WTSDisconnected,
        WTSIdle,
        WTSListen,
        WTSReset,
        WTSDown,
        WTSInit
    }

    public static IntPtr OpenServer(String Name)
    {
        IntPtr server = WTSOpenServer(Name);
        return server;
    }

    public static void CloseServer(IntPtr ServerHandle)
    {
        WTSCloseServer(ServerHandle);
    }

    public static List<string> ListUsers(String ServerName)
    {
        IntPtr serverHandle = IntPtr.Zero;
        List<String> resultList = new List<string>();
        serverHandle = OpenServer(ServerName);

        try
        {
            IntPtr SessionInfoPtr = IntPtr.Zero;
            IntPtr userPtr = IntPtr.Zero;
            IntPtr domainPtr = IntPtr.Zero;
            IntPtr clientNamePtr = IntPtr.Zero;
            Int32 sessionCount = 0;
            Int32 retVal = WTSEnumerateSessions(serverHandle, 0, 1, ref SessionInfoPtr, ref sessionCount);
            Int32 dataSize = Marshal.SizeOf(typeof(WTS_SESSION_INFO));
            Int32 currentSession = (int)SessionInfoPtr;
            uint bytes = 0;
            if (retVal != 0)
            {
                for (int i = 0; i < sessionCount; i++)
                {
                    WTS_SESSION_INFO si = (WTS_SESSION_INFO)Marshal.PtrToStructure((System.IntPtr)currentSession, typeof(WTS_SESSION_INFO));
                    currentSession += dataSize;

                    WTSQuerySessionInformation(serverHandle, si.SessionID, WTS_INFO_CLASS.WTSUserName, out userPtr, out bytes);
                    WTSQuerySessionInformation(serverHandle, si.SessionID, WTS_INFO_CLASS.WTSDomainName, out domainPtr, out bytes);
                    WTSQuerySessionInformation(serverHandle, si.SessionID, WTS_INFO_CLASS.WTSClientName, out clientNamePtr, out bytes);

                    if(Marshal.PtrToStringAnsi(domainPtr).Length > 0 && Marshal.PtrToStringAnsi(userPtr).Length > 0)
                    {
                        if(Marshal.PtrToStringAnsi(clientNamePtr).Length < 1)                       
                            resultList.Add(Marshal.PtrToStringAnsi(domainPtr) + "\\" + Marshal.PtrToStringAnsi(userPtr) + "\tSessionID: " + si.SessionID + "\tClientName: n/a");
                        else
                            resultList.Add(Marshal.PtrToStringAnsi(domainPtr) + "\\" + Marshal.PtrToStringAnsi(userPtr) + "\tSessionID: " + si.SessionID + "\tClientName: " + Marshal.PtrToStringAnsi(clientNamePtr));
                    }
                    WTSFreeMemory(clientNamePtr);
                    WTSFreeMemory(userPtr);
                    WTSFreeMemory(domainPtr);
                }
                WTSFreeMemory(SessionInfoPtr);
            }
        }
        catch(Exception ex)
        {
            Console.WriteLine("Exception: " + ex.Message);
        }
        finally
        {
            CloseServer(serverHandle);
        }
        return resultList;
    }
}
'@

Add-Type $Code

Copy all of that into a file named QuerySessionInformation.ps1. Now launch the 32 bit version of Powershell in C:\Windows\SysWOW64\WindowsPowershell\v1.0. The code above uses pointers that will not work in a native 64 bit environment.

Now run the script. If you've never run the 32 bit version of Powershell on that server before, you will need to modify the script execution policy with Set-ExecutionPolicy, as 32 bit and 64 bit Powershell have separate execution policies. Note that there should be no output from the script itself, as all it is doing is compiling the .NET code and adding it to the current environment. Also note that once a type is added with Add-Type, you can not unload it without exiting that Powershell session... AFAIK. It makes debugging this sort of stuff really annoying as you have to restart Powershell every time you modify the code.

Now that the code is loaded, type this:

PS C:\> [RDPInfo]::ListUsers("REMOTESERVER") 

If there are any active user sessions on REMOTESERVER, the output will look like this:

DOMAIN\UserName SessionID: 2 ClientName: RYAN-PC 

This will work on remote computers as well as the local computer, but beware that if the user running this does not have sufficient permissions to the remote computer, it will fail silently (no output.)

PS: There are other bits of info in WTS_INFO_CLASS that may be of interest to you, such as WTSConnectState and WTSClientAddress. All you have to do is query for them with WTSQuerySessionInformation().

First, wooo I hit 5K on ServerFault today.

I'm embarrassed to say that something I read about recently but didn't pay enough attention to at the time officially just bit me in the butt.  

A significant change occurred in January 2012 in the way that DFS Replication behaves.  Windows Server 2008 R2 SP1 post KB2663685 and Windows Server 2012 have changed the default behavior of DFSR.  Auto-recovery of DFSR replicated folders after unexpected shutdown is now disabled.  In other words, if a computer that hosts a DFSR replicated folder experiences an unexpected shutdown, DFSR will not automatically resume upon reboot.  (This includes Sysvol!) 

On older versions of Windows, DFSR auto-recovery was enabled.  I'm sure the reason for this change involves auto-recovery leading to unexpected rollbacks and unauthoritative conflict resolutions between replication partners, especially in wide-spread domains with high end-to-end replication latency and frequent changes... but even though the news was published, I for one didn’t pay enough attention to it and it has a very real effect on the way we manage our Windows systems that utilize DFSR going forward. 

So what if a domain controller or a file server with a DFSR share on it running 2008R2 or 2012 crashes unexpectedly, leaving the DFS database and the NTFS USN journal out of sync?  Then Sysvol no longer receives updates on that DC.  The DFSR file share no longer receives updates on that file server.  It's up to you to manually restart replication, and to resolve any conflicts with replication partners if changes took place during the time that the crashed server wasn't replicating. 

Luckily that is easy to do, and it's also possible to set the behavior back to auto-recovery if that is what you wish. 

How will I know if this effects my server?

While this first example is just a symptom of the problem, here is how it first came to my attention, triggering the investigation:

(Click on images for a better view.)

Application of Group Policy was failing, but only on DC02 and servers which were using DC02 as a domain controller.  Not DC01 or any server logged on by DC01.  As it turns out, the GPO referenced by that error event, a new GPO that I had just created on DC01, didn’t exist on DC02, hence the errors.  Sysvol did not seem to be replicating anymore.

Here is the actual event log event to let you know that DFS Replication has stopped on one or more volumes:

Luckily, starting replication back up again is easy and the command to do it with your actual GUID, is right there in the event:

wmic /namespace:\\root\microsoftdfs path dfsrVolumeConfig where volumeGuid="12345678-ABCD-1234-EFGH-1A2B3C4E5F" call ResumeReplication

 You can also turn auto-recovery back on with wmic or by modifying the registry if you don’t have the time to be bothered by this:

HKLM\System\CurrentControlSet\Services\DFSR\Parameters\StopReplicationOnAutoRecovery = 0

Just be aware that auto-recovery can lead to unwanted rollbacks of DFSR data in some circumstances.

D-Root changed its IPv4 address back on January 3rd.  The old one is still up for now, but will be phased out in the future.  The old IPv4 address is 128.8.10.90.  The new address is 199.7.91.13. (d.root-servers.net IPv6 address is not changing.)

Today, someone asked me how to figure out, on the command-line, whether their Windows DNS server had the new updated d-root IP address or not.

After about 10 minutes of fiddling, here is what I came up with:

PS C:\> If($(Get-WmiObject -Namespace root\MicrosoftDNS MicrosoftDNS_ResourceRecord | ? DomainName -Match "root-servers.net" | ? OwnerName -Match "d.root-servers.net." | ? ContainerName -Match "..RootHints").IPAddress -ne "199.7.91.13") { Write-Host "YOUR D-ROOT IS OLD!" }

If your DNS server is still configured with the old root hints address for d-root, the command will notify you.

Well I did something quite embarrassing the other day. But then I did something pretty cool to make up for it. I'll call it a Christmas miracle, because it involves me not knowing the domain admin password, having no other normal way of logging in, despairing for a couple minutes, then getting back in anyway. And also because I'm histrionic like that. It's a real story of tribulation and triumph.

A couple days ago, the domain admin password on one of my lab domains expired unexpectedly, and so I was prompted to reset it upon RDPing to a server. So I did. As a matter of habit, I saved the new password in my password vault software. (I'm one of those people that has a different password for every single account I have, but as a result I can't remember them all.)

At least, I thought I saved the new password.

Fast forward to a week later. I wanted to do some work in that lab again, and as I began the login process, I realized that I had no idea what I had set the password to. The domain admin password. The only Administrative account in that domain. "No problem," I thought. It's in my password vault...

It wasn't.

I went through several backup copies of the password file, thinking that it must have been overwritten. Nope. Dreadful thoughts of spending the whole weekend rebuilding the lab flashed before my eyes.

Then I had an idea.

I switched my monitor over to the physical domain controller, and plugged in a keyboard.

It's a very calm and soothing login screen, isn't it? I like it.

Anyway, there's the ever-present "Accessibility" or "Ease of Access" icon down there in the corner. (I know many of you already know where I'm going with this, and you're right. But if you don't know, read on. Otherwise, you may want to skip to the "Afterthoughts" section below.) This domain controller is running Server 2012 Core. That accessibility button in the corner doesn't do anything. I can click it, but it doesn't do anything. Maybe it's because this is a Server Core installation with no GUI elements. Hm.

I had a Windows installation image on a bootable USB thumb drive handy, so I plugged that in to the domain controller and rebooted the machine (using the power button on the chassis obviously, since I wasn't going to be logging in.)

Once the machine had booted off the installation media, instead of choosing to install Windows, I chose "Repair your computer." From there, I chose the option to launch a Command Prompt. Voila. Sweet. I now had a shell on the computer.

Next I navigated on over to where everything interesting happens, \Windows\System32. (The drives had different drive letters than normal, but who cares.) Now, under regular circumstances, that little Accessibility button on the logon screen launches a program called Utilman.exe. From there you can get all the gizmos like the on-screen keyboard and magnifier and things like that to help you log in, in case your eyesight's not so good or you have no hands or something. But there was no Utilman.exe in the System32 folder. I guess that's why the Accessibility icon didn't do anything when I clicked it. My domain controllers are very inconsiderate of the handicapped.

So I just decided to try the command:

copy X:\Windows\System32\cmd.exe X:\Windows\System32\Utilman.exe

Then I rebooted. Once I was back at the Windows login screen, I hit the Accessibility icon one last time. Command Prompt. And not just any Command Prompt, but a Command Prompt running as Local System! Since this was a normal boot of the domain controller and therefore Active Directory was online, I just did:

net user DomainAdmin *

And I was greeted with a prompt to reset the domain administrator password to whatever I wanted.

This time I made damn sure that the new password was saved and backed up in my password vault.

Afterthoughts

So it's great that I was able to reset a domain administrator password with nothing but a bootable image and physical access to the machine, but, one can't help but wonder about the glaring security implications of this. Once a person has a Local System shell on a domain controller, they own the domain.

Furthermore, today's enterprise environments are more complex than ever. Having physical access to the console of the machine doesn't necessarily mean you have to actually be able to touch the machine, like I was able to do here. These days, "physical access" can mean logging in to an ILO or accessing a virtual machine management console from a thousand miles away. And are we, as enterprise administrators and IT directors and CTOs, sure that everyone in our organization with access to those things are people we also trust as potential domain admins? It's not just about the threat of outside attackers. We may be unwittingly exposing a massive attack surface to all of our front-line support personnel should they ever decide to go rogue.

Alright, so what should be done about this little "feature" of Windows? First off, Microsoft needs to not launch Utilman.exe as Local System. Launch it as an unprivileged account that doesn't have the ability to do things like reset Administrator passwords. I'm sure there's a technical reason why they think they "have" to run the Ease of Access stuff as Local System, so that it can simulate the secure attention sequence, etc., but I'm not buying it.  Just because it's hard doesn't mean they should get away with letting this gaping security hole go unpatched.  They could also include a hash check in the code behind that little icon so it would at least be a little more difficult to replace the real Utilman.exe with, say, a copy of Cmd.exe that was renamed to Utilman.exe.

To make matters worse, there is no "official" Group Policy setting to disable this button. You might be able to mitigate the risk somewhat by using Group Policy to do something like run a script that cacls Utilman.exe at startup and denies Everyone all access, or having Group Policy push out your own version of Utilman.exe at every boot of the computer... neither of which are ideal solutions nor are they likely to be bulletproof.

Oh well. I guess that goes along with that immutable law of computer security, that if someone else has physical access to your computer, then it's not your computer anymore.

Accept isn't an accepted verb in Powershell (Get-Verb,) but I don't really care.  Someone asked if there was a way, using Powershell, to log in to an Exchange mailbox, find all the meeting invites and accept them.  Piece of cake:

[Reflection.Assembly]::LoadWithPartialname("Microsoft.Office.Interop.Outlook") | Out-Null
$Folders = "Microsoft.Office.Interop.Outlook.OlDefaultFolders" -As [Type]
$Outlook = New-Object -ComObject Outlook.Application
$Namespace = $Outlook.GetNameSpace("MAPI")
$Inbox = $Namespace.getDefaultFolder($Folders::olFolderInbox)
ForEach ($_ In $Inbox.Items)
{
    If ($_.MessageClass -eq "IPM.Schedule.Meeting.Request") 
    {
        $AppointmentItem = $_.GetAssociatedAppointment($true)       
        $Response = $AppointmentItem.Respond(3,$True,$False)
        $Response.Send()
    }
}

Of course you need Outlook installed on the machine where you want to run this script, or otherwise have access to that assembly. You'd likely want to spruce the script up a bit to do things like log on to other mailboxes, have error handling, etc., but this is just a proof-of-concept that you can certainly build yourself a complete alternative Outlook client in Powershell if you wanted to.

Sorry for making a post consisting of nothing but a link, but this is a seriously cool video.  This guy is approaching Russinovich levels.  It's about some of the advances in Windows 8/Server 2012 at the kernel code level.

Inside Windows 8: Pedro Teixeira - Thread pools.

I haven't posted in a while; been busy both studying for Windows Server 2012 stuff and also preparing for a possible slight career shift.  But I do want to put this up here, because it's one of my answers to a Serverfault  question that I'm a little proud of.  Nevertheless, it's a deep enough topic that I expect someone who knows more about it than me to come along and correct me.  Which I welcome.  That's how science works.  I'm not learning if I'm not wrong.

Here was the question:

In order to assess performance monitoring accuracy on virtualization platforms, the CPU steal time has become an increasingly relevant metric - see EC2 monitoring: the case of stolen CPU for an instructive summary in the context of Amazon EC2 and IBM's paper on CPU time accounting for a more in-depth technical explanation (including illustrations) of the concept:

Steal time is the percentage of time a virtual CPU waits for a real CPU while the hypervisor is servicing another virtual processor.

Accordingly, it is exposed in most related Unix/Linux monitoring tools nowadays - see e.g. columns %steal or st in sar or top:

st -- Steal Time
The amount of CPU 'stolen' from this virtual machine by the hypervisor for other tasks (such as running another virtual machine).

I've been unable to figure out how to capture the same metric on Windows though, is this possible already? (Ideally for the Windows 2008 Server R2 AMIs on EC2 and via a respective Windows Performance Counters of course.)

 And here was my answer:

Let me preface by saying that I am coming from the point of view of Hyper-V as a virtualization platform because that is where I have the most experience. Even though there may be certain tenets of virtualization, as we know it, that cannot be deviated from, Microsoft and VMware and Xen all have different strategies for how they design their hypervisors.

That's the first thing that makes your question challenging. You pose your question as if it were hypervisor-agnostic, when in truth it is not. Amazon EC2, for example, uses the Xen hypervisor, and the "CPU Steal Time" metric that you see in the output of a top command issued from within a Linux VM running on that hypervisor is a result of the integration services installed on that guest OS (or virtualization-aware tools on the guest) in conjunction with data provided by that specific hypervisor.

First off let me just answer your question straight up: There is no way to see from inside a virtual machine running Windows how much time the processors belonging to the physical machine on which the hypervisor runs spends doing other things, unless the particular virtual tools/services or virtualization-aware tools for your particular hypervisor are installed in the guest VM and the particular hypervisor on which the guest is running exposes that data. Even a Windows guest running on a Hyper-V hypervisor will not have immediate access to information regarding the time spent that the physical processors on the hypervisor were doing other things. (To quote voretaq7, something that "breaks the fourth wall.") Even though Windows client and server operating systems running as virtualized guests in Hyper-V with the correct integration services/tools installed make use of "enlightenments" (which are literally kernel code alterations made especially for VMs) that significantly increase their performance in using the resources of a physical host, the bottom line is that the hypervisor does not have to give any more information to the guest OS than it wants to. That means the hypervisor does not have to tell a guest VM what else it is doing besides servicing that VM... unless it wants to. And that information about what else the physical processors are doing is necessary for deriving a metric from the perspective of the VM such as "CPU Steal Time: the percentage of time the vCPU waits for a physical CPU."

How could the guest OS know that, if it didn't even realize that it was actually virtualized? It's like The Truman Show... for computers.

In other words, without the right integration tools installed on the guest, the guest OS won't even know that its CPU is actually a *v*CPU. It won't even know that there is another force outside of itself "stealing" CPU cycles from it, therefore that metric will not exist on the guest VM.

That's why I don't even like the phrase "CPU Steal Time." The word steal just puts everybody in the wrong frame of mind from the get-go.

A hypervisor such as Hyper-V does not give guests direct access to physical resources such as physical processors or processor cores. Instead the hypervisor gives them vDevs - virtual devices - such as vCPUs.

A prime example of why: Say a virtual machine guest OS makes the call to flush the TLB (translation look-aside buffer) which is a physical component of a physical CPU. If the guest OS was allowed to clear the entire TLB on a physical processor, that would have negative performance effects for all the other VMs that were also sharing that same physical TLB. In the case of Windows, that call in the guest OS is translated into a "hypercall" or "enlightened" call which is interpreted by the hypervisor so that only the section of the TLB that is relevant to that virtual machine is flushed.

(Interestingly, that hints to me that guest VMs that do not have the proper integration tools and/or services could have the ability to impact the performance of all the other VMs on the same host, but that is completely outside the scope of this topic.)

All that to say that you can still detect in a Hyper-V host the time that a virtual processor spent waiting for a real processor to become available so that it could scheduled to run. But you can only see that data on a Windows Hyper-V hypervisor. If it is possible to see this in other hypervisors, I urge others to tell us how to see this in that hypervisor and also if it is exposed to the guests. And that is before we even get to whether that data is exposed to the guest OS or not.

My test machine was Hyper-V Server 2012, which is the free edition of Server 2012 that only runs Core and the Hyper-V role. It's effectively the same as any Windows Server 2012 running Hyper-V.

Fire up Perfmon on your parent partition, aka physical host. Load this counter:

Hyper-V Hypervisor Virtual Processor\CPU Wait Time Per Dispatch\* 

You will notice that there will be an instance of that counter for each virtual machine on that hypervisor, as well as _Total. The Microsoft definition of that Perfmon counter is:

The average time (in nanoseconds) spent waiting for a virtual processor to be dispatched onto a logical processor.

Obviously, you want that number to be as low as possible. For computers, waiting is almost never a good thing.

Other performance counters on the hypervisor that you will want to investigate are Hyper-V Hypervisor Root Virtual Processor\% Guest Run Time, % Hypervisor Run Time, and % Total Run Time. These counters provide you with the percentages that could be used to determine facts such as how much time the "real" processors spend doing things other than servicing a VM or all VMs.

So in conclusion, the metric that you are looking for in a guest virtual machine depends on the hypervisor that it is running on, whether that hypervisor chooses to provide the data about how it spends its time other than servicing that VM, and if the guest OS has the right virtualization integration tools/services/drivers to be aware enough to realize that the hypervisor is making that data available.

I know of no way on a Windows guest, integration tools installed or not, to see how much time, in terms of seconds or percentage, that VM's host has spent servicing it or not servicing it respective to the total physical processor time.