The "CPU Steal Time" Metric in Unix/Linux Virtual Machines and a Windows Counterpart

by Ryan 10. December 2012 19:37

I haven't posted in a while; been busy both studying for Windows Server 2012 stuff and also preparing for a possible slight career shift.  But I do want to put this up here, because it's one of my answers to a Serverfault  question that I'm a little proud of.  Nevertheless, it's a deep enough topic that I expect someone who knows more about it than me to come along and correct me.  Which I welcome.  That's how science works.  I'm not learning if I'm not wrong.

Here was the question:

In order to assess performance monitoring accuracy on virtualization platforms, the CPU steal time has become an increasingly relevant metric - see EC2 monitoring: the case of stolen CPU for an instructive summary in the context of Amazon EC2 and IBM's paper on CPU time accounting for a more in-depth technical explanation (including illustrations) of the concept:

Steal time is the percentage of time a virtual CPU waits for a real CPU while the hypervisor is servicing another virtual processor.

Accordingly, it is exposed in most related Unix/Linux monitoring tools nowadays - see e.g. columns %steal or st in sar or top:

st -- Steal Time
The amount of CPU 'stolen' from this virtual machine by the hypervisor for other tasks (such as running another virtual machine).

I've been unable to figure out how to capture the same metric on Windows though, is this possible already? (Ideally for the Windows 2008 Server R2 AMIs on EC2 and via a respective Windows Performance Counters of course.)

 And here was my answer:

Let me preface by saying that I am coming from the point of view of Hyper-V as a virtualization platform because that is where I have the most experience. Even though there may be certain tenets of virtualization, as we know it, that cannot be deviated from, Microsoft and VMware and Xen all have different strategies for how they design their hypervisors.

That's the first thing that makes your question challenging. You pose your question as if it were hypervisor-agnostic, when in truth it is not. Amazon EC2, for example, uses the Xen hypervisor, and the "CPU Steal Time" metric that you see in the output of a top command issued from within a Linux VM running on that hypervisor is a result of the integration services installed on that guest OS (or virtualization-aware tools on the guest) in conjunction with data provided by that specific hypervisor.

First off let me just answer your question straight up: There is no way to see from inside a virtual machine running Windows how much time the processors belonging to the physical machine on which the hypervisor runs spends doing other things, unless the particular virtual tools/services or virtualization-aware tools for your particular hypervisor are installed in the guest VM and the particular hypervisor on which the guest is running exposes that data. Even a Windows guest running on a Hyper-V hypervisor will not have immediate access to information regarding the time spent that the physical processors on the hypervisor were doing other things. (To quote voretaq7, something that "breaks the fourth wall.") Even though Windows client and server operating systems running as virtualized guests in Hyper-V with the correct integration services/tools installed make use of "enlightenments" (which are literally kernel code alterations made especially for VMs) that significantly increase their performance in using the resources of a physical host, the bottom line is that the hypervisor does not have to give any more information to the guest OS than it wants to. That means the hypervisor does not have to tell a guest VM what else it is doing besides servicing that VM... unless it wants to. And that information about what else the physical processors are doing is necessary for deriving a metric from the perspective of the VM such as "CPU Steal Time: the percentage of time the vCPU waits for a physical CPU."

How could the guest OS know that, if it didn't even realize that it was actually virtualized? It's like The Truman Show... for computers.

In other words, without the right integration tools installed on the guest, the guest OS won't even know that its CPU is actually a *v*CPU. It won't even know that there is another force outside of itself "stealing" CPU cycles from it, therefore that metric will not exist on the guest VM.

That's why I don't even like the phrase "CPU Steal Time." The word steal just puts everybody in the wrong frame of mind from the get-go.

A hypervisor such as Hyper-V does not give guests direct access to physical resources such as physical processors or processor cores. Instead the hypervisor gives them vDevs - virtual devices - such as vCPUs.

A prime example of why: Say a virtual machine guest OS makes the call to flush the TLB (translation look-aside buffer) which is a physical component of a physical CPU. If the guest OS was allowed to clear the entire TLB on a physical processor, that would have negative performance effects for all the other VMs that were also sharing that same physical TLB. In the case of Windows, that call in the guest OS is translated into a "hypercall" or "enlightened" call which is interpreted by the hypervisor so that only the section of the TLB that is relevant to that virtual machine is flushed.


(Interestingly, that hints to me that guest VMs that do not have the proper integration tools and/or services could have the ability to impact the performance of all the other VMs on the same host, but that is completely outside the scope of this topic.)


All that to say that you can still detect in a Hyper-V host the time that a virtual processor spent waiting for a real processor to become available so that it could scheduled to run. But you can only see that data on a Windows Hyper-V hypervisor. If it is possible to see this in other hypervisors, I urge others to tell us how to see this in that hypervisor and also if it is exposed to the guests. And that is before we even get to whether that data is exposed to the guest OS or not.

My test machine was Hyper-V Server 2012, which is the free edition of Server 2012 that only runs Core and the Hyper-V role. It's effectively the same as any Windows Server 2012 running Hyper-V.

Fire up Perfmon on your parent partition, aka physical host. Load this counter:

Hyper-V Hypervisor Virtual Processor\CPU Wait Time Per Dispatch\* 

You will notice that there will be an instance of that counter for each virtual machine on that hypervisor, as well as _Total. The Microsoft definition of that Perfmon counter is:

The average time (in nanoseconds) spent waiting for a virtual processor to be dispatched onto a logical processor.

Obviously, you want that number to be as low as possible. For computers, waiting is almost never a good thing.

Other performance counters on the hypervisor that you will want to investigate are Hyper-V Hypervisor Root Virtual Processor\% Guest Run Time, % Hypervisor Run Time, and % Total Run Time. These counters provide you with the percentages that could be used to determine facts such as how much time the "real" processors spend doing things other than servicing a VM or all VMs.

So in conclusion, the metric that you are looking for in a guest virtual machine depends on the hypervisor that it is running on, whether that hypervisor chooses to provide the data about how it spends its time other than servicing that VM, and if the guest OS has the right virtualization integration tools/services/drivers to be aware enough to realize that the hypervisor is making that data available.

I know of no way on a Windows guest, integration tools installed or not, to see how much time, in terms of seconds or percentage, that VM's host has spent servicing it or not servicing it respective to the total physical processor time.

Best-Practices Remediation Tips for Server 2012 Pt II

by Ryan 30. November 2012 17:36

Part I is here.

Enable Large Send Offload (LSO) on a network adapter.

Before: 

PS C:\Users\Administrator> Get-NetAdapterLso

Name                           Version         V1IPv4Enabled  IPv4Enabled  IPv6Enabled
----                           -------         -------------  -----------  -----------
Ethernet 2                     LSO Version 2   True           False        True
Ethernet                       LSO Version 2   True           False        True
Team                           LSO Version 2   False          True         True

 Now type this (it will interrupt network connectivity, but it should come back): 

PS C:\Users\Administrator> Enable-NetAdapterLso -Name *
WARNING: The network connection to DC02 has been interrupted. Attempting to reconnect for up to 4 minutes...
WARNING: Attempting to reconnect to DC02 ...
WARNING: The network connection to DC02 has been restored.

Now, let's look again:

PS C:\Users\Administrator> Get-NetAdapterLso

Name                           Version         V1IPv4Enabled  IPv4Enabled  IPv6Enabled
----                           -------         -------------  -----------  -----------
Ethernet 2                     LSO Version 2   True           True         True
Ethernet                       LSO Version 2   True           True         True
Team                           LSO Version 2   False          True         True

 

Enable Receive Side Scaling (RSS) on a network adapter.

Receive side scaling is a nice technology to have on servers with multiple processors and lots of network traffic. It basically spreads out processing load of network traffic across all your cores, instead of just piling it all on core 0. A good cmdlet to see what network adapters on your machine are capable of RSS:

PS C:\Windows\system32> Get-SmbServerNetworkInterface

Scope Name          Interface Index     RSS Capable         RDMA Capable        Speed               IpAddress
----------          ---------------     -----------         ------------        -----               ---------
*                   12                  True                False               1 Gbps              192.168.1.9
*                   12                  True                False               1 Gbps              fe80::2d14:f5e1:...
*                   12                  True                False               1 Gbps              fd58:2c98:ee9c:2...

To enable RSS across all your network adapters, simply do:

PS C:\Windows\system32> Enable-NetAdapterRss -Name *

Just like you did before with LSO.

 

Enable IPsec Task Offload v2 (TOv2) on a network adapter.

One more. Works the same way. Try Get-NetAdapterIPsecOffload to see the status of that feature on your network adapters. If the cmdlet returns nothing, that means the feature is not available on any of your network adapters. If it is available, but not enabled, then just do Enable-NetAdapterIPsecOffload -Name *.

Cyber Monday Sale - Half Off eBooks from O'Reilly

by Ryan 26. November 2012 13:03

50% off all eBooks today from O'Reilly Media. Man I love eBooks. This is probably my Generation Y showing, but most times I'll take an eBook over print. (I know that makes most book lovers cringe.)

My shopping cart keeps growing...

EventLogClearer v1.1.3.22

by Ryan 19. November 2012 20:45

I have released an updated version of my EventLogClearer, bringing it up to version 1.1.3.22. For the original release, see this post.

EventLogClearer 1.1.3.22

Improvements made in this version include:

  • Fixed a bug where the application acted weird if you ran the log clearing procedure two or more times in a row.
  • Added a new mechanism for supplying alternate credentials, instead of only being able to run as the currently logged on user. This applies to both auto-populating the list of computers from AD, and running the event log clearing procedure. If you leave the credentials blank or as the default, "username," the current user will be used.
  • Added the ability to clear a ton more Applications and Services logs than before, due to me realizing the potential of the EventLogSession class.

As before, .NET 4.5 is required to run the application. The project was built in Visual Studio 2012.

Here is the executable: EventLogClearer-1.1.3.22-exe.zip (68.71 kb)

Here is the source code: EventLogClearer-1.1.3.22-source.zip (308.11 kb)

Sometimes I Can Access the WebDAV Share, Sometimes I Can't!

by Ryan 13. November 2012 10:19

You probably already know that all of the Sysinternals tools, such as Process Monitor, Process Explorer, Autoruns, and much more, can be accessed via "shared folder" from any computer connected to the internet by navigating to \\live.sysinternals.com\.  This isn't the same kind of share you'd create if you just shared a folder on your PC.  It's a WebDAV share, and is accessed over HTTP.

Sometimes though, I feel the need to access this share from the command line, either in the Cmd shell or Powershell.  Alas, here's what I see:

Network path not found*Path not found.*

I get the same result with Powershell. Bummer. Well I know I can access the path with Explorer when I type that same UNC into the address bar, or if I just type the UNC into the Run dialog box, so this must just be a limitation of those command-line tools, right?

It works in Explorer*Works fine in Explorer*

Oh well... but wait. Now having successfully accessed the network path with Explorer, let me now immediately go back to the Cmd shell and try it again:

 

Now it works in Cmd too!*Now it works in Cmd too!*

OK, now accessing the network path works fine from the Cmd shell and from Powershell, even though all I did was access it through Explorer first, and then try again. Now I just have to know what the heck is going on... and to do that, I need to use Process Monitor. Which, amusingly, is in the WebDAV share I'm trying to access. But I'll run a local copy.

I started the trace. Here's my first attempt to access the network path with Cmd.exe, which failed:

Cmd.exe network path not found*Network path not found*

This was the very first time in the Process Monitor trace when the string "live.sysinternals.com" appeared in the Path field. It's also the first time the Cmd.exe process shows up in the trace. It's currently filtered to only include events where the Path field contains the string live.sysinternals.com. The really interesting part about this is that it appears the moment I pressed Enter on the command line, Explorer.exe was the first process to be involved, not the process I was interacting with! That's odd. Maybe a file system filter driver intercepted the call and notified Explorer? It looks like Explorer is looking for something related to named pipes and the Workstation Service (wkssvc) on the remote server, but it doesn't find it.  Then Cmd.exe first checked my local file system for a file in Windows\CSC\ directory, which it didn't find, and then it tried to access the network path that I actually asked for, which resulted in "Bad network path." Then it apparently tries again with the same local file system path, and then again with the network directory instead of the specific executable name.  All failed. "Network path not found," my command prompt tells me. But with no further input from me, Explorer takes off doing its own thing, calling cscapi.dll and loading things in the background and sending things over network. All I did was hit enter in the Command Prompt above.

So what is this CSC directory? Googling the term led me to an old post on Raymond Chen's blog. Client Side Caching. OK, so apparently both processes are looking for a cached or offline version of the network path.

Then I move over to the Explorer.exe window and type the path into the address bar. Explorer looks for some more CSC stuff first, and then svchost.exe starts communicating with the remote server over TCP. There's a lot of loading of WebDAVRedirector stuff. Finally, after a lot of work, I start seeing events like these from Explorer:

Explorer finds it, finally*Explorer starts finding it, finally*

Notice that Explorer also seems to be storing the autoruns executable in a temporary "Tfs_DAV" directory on my workstation.

Finally, after having success with Explorer, I go right back to the Command Prompt and try it again. This time, the trace looks like this:

Works in cmd.exe now too

Now I see svchost.exe stepping in with a WebDavRedirector, and cmd.exe getting some successful returns from its IRPs. Finally, after playing around in that Tfs_DAV directory and some more intermingling of svchost.exe and the System process both helping out, the process autoruns.exe finally launches.

So that's a pretty fast and loose overview of what is actually going on. The entire trace was a beast to wade through, and there is obviously a lot of orchestration and cooperation required between many different Windows components required to allow you to access a WebDAV share from within Cmd.exe and I don't fully understand all of it... but the bottom line is that at least on my Windows 7 SP1 x64 workstation, it looks like Explorer.exe is smart enough to read from a WebDAV share and cache the data locally, whereas Cmd.exe is only smart enough to read the data locally, if and only if it's already cached locally... or perhaps the redirector had to be "woken up" by Explorer first, before Cmd.exe was able to use it.

Finally, I'll leave off with a bit about the WebDAV Mini-Redirector from Wikipedia:

"In Windows XP, Microsoft added the Web Client service is also known as the WebDAV mini-redirector[11] which is preferred by default over the old Web folders client. This newer client works as a system service at the network-redirector level (immediately above the file-system), allowing WebDAV shares to be assigned to a drive letter and used by any software. The redirector also allows WebDAV shares to be addressed via UNC paths (e.g. http://host/path/ is converted to\\host\path\) for compatibility with Windows filesystem APIs."

Blog Posts You Must Read

by Ryan 12. November 2012 08:43

The PFE Platforms team has published another blog post in their MCM: Active Directory series, which was such a fantastic post, it inspired me to create a "Blog Posts You Must Read" section over there on the side bar. It will be for blog posts and/or blog post series' that are so good that I find myself going back and reading them multiple times, or even going back to use them for reference material. I think that's a lot more meaningful than just a gigantic generic list of every website I know of.

More to come as I finish trawling through my bookmarks or stumble across new ones.

Best-Practices Remediation Tips for Server 2012 Pt I.

by Ryan 11. November 2012 15:19

I'm calling this Part 1 because I realized as I started writing that this is a lot of work, and can easily be split into 2 or more articles.

Like most of the IT Pro community, I've been getting comfortable with Server 2012 the past several weeks now, and the journey is still ongoing. As I talked about last time, I do like those Best Practices Analyzers for Windows Server. Here's me running it in Server Manager:

Best Practices Analyzer Server Manager 2012*BPA in Server Manager 2012*

Getting any of these results back means that I have some work to do in remediating them. It's not uncommon for a Server 2012 system that was just built fresh with no applications loaded or configuration changes to still have one or two compliance issues in the Best Practices Analyzers. There is a balance to be maintained between compatibility and performance optimization. Also, many of these issues that popped up for me personally were not role-specific, but rather apply to a base component of the OS. Now I'll go over some of the interesting ones I've gotten and how I fixed them:

 

[Hyper-V]: Avoid storing Smart Paging files on a system disk.

Smart paging is new with Hyper-V 2012. Read about it here. Basically, you now enter a minimum amount of RAM a virtual machine can have, you enter a maximum amount of RAM a virtual machine can have, and you also now enter the amount of startup RAM a virtual machine can have. The Windows OS can boot up more comfortably with a larger amount of RAM, but once it reaches cruising altitude and is idle, the RAM requirements go back down, which will allow the Hyper-V host to gradually start reclaiming memory from the VM. With all this dynamic shrinking and growing of the memory on all your VMs, that's where the "smart paging file" comes in. And just like you can improve performance by putting your traditional Windows paging file on its own disk, the same goes with the Smart Paging file.

[Hyper-V]: Use RAM that provides error correction.

Microsoft doesn't support Hyper-V environments on hardware that isn't using ECC RAM. This is just a lab using desktop-grade hardware, so there's nothing I can really do about this. If you're using real server gear, this should not be an issue for you.

[Hyper-V]: Virtual machines should be backed up at least once every week.

This one I still don't understand. I have all the guest operating systems backing up nightly, and then I am backing them up again through the backup of the Hyper-V host. So go away, error.

[Windows]: Short file name creation should be disabled.

Then why is it enabled by default? Oh, I know why... it's because of your crappy line-of-business apps that were written back in 1998 that you can't get rid of, right? Well I'm in a pure 2012 environment right now, so I have no such worries. Good bye 8.3 filenames. Change this registry value to 1 to disable short file name creation on all volumes: HKLM\System\CurrentControlSet\Control\FileSystem\NtfsDisable8dot3NameCreation.

[Windows]: IrpStackSize should have the recommended value.

Irp stands for I/O Request Packet. Mark Russinovich did a great job of explaining IRPs and their role in the Windows I/O system in his book Windows Internals. All the various components or layers that an I/O packet traverses on its way to and from a disk, for example, are collectively referred to as "a stack." Each filter driver you add to the file system means that the IrpStackSize needs to be increased in order to accommodate it. A common example of this is when you install an antivirus product that uses a file system filter driver. If your IrpStackSize is set too small, certain operations might fail, such as attempting to access that machine's file system remotely. Conversely, it doesn't need to be set too high, either.  It was at 11 by default on my 2012 systems. The Best Practices Analyzer says it should be at 15, so I'll set it to 15. HKLM\System\CurrentControlSet\Services\LanmanServer\Parameters\IRPStackSize.

 

Alright, that's where I'll leave off for part 1. Part 2 will focus on some more network-specific optimizations, so check back soon!

Best Practices Analyzer in Server Core 2012

by Ryan 6. November 2012 19:31

I always liked the Best Practices Analyzers in Windows Server, where you could scan most of the installed roles for "best practices" and it would inform you if you had any errors or warnings in your setup.  It's just a nice little sanity check to make sure you didn't forget any major steps when installing AD Directory Services, or a DNS server, etc.  Traditionally, you'd initiate the Best Practices Analyzer from within Server Manager.

So what about Server Core, when you have no GUI? Well I'm glad you asked.

Get-BPAModel

That command will list all of the available Best Practices Analyzer modules that are installed on the current system. You use the module ID, such as "Microsoft/Windows/DirectoryServices" to run the BPA, like so:

Invoke-BPAModel Microsoft/Windows/DNSServer

The above command will run the Best Practices Analyzer on the specified role. The results from the BPA are written to disk, so to retrieve the results from your last BPA scan at any time, you do:

Get-BPAResult Microsoft/Windows/Hyper-V

If you wanted to only see the results where a potential problem was found, try this:

Get-BPAResult Microsoft/Windows/DirectoryServices | ? {$_.Problem -ne $null}

 

Native NIC Teaming In Server 2012

by Ryan 4. November 2012 14:18

Built-in NIC teaming was one of my personally most anticipated features of Server 2012.  NIC teaming, whether for redundancy or for more bandwidth, has always been a cool concept and one of the foundations of highly-available systems, but it has historically required 3rd party vendor software to enable.  Probably the most popular example I can think of is the HP Network Configuration Utility software:

HP Network Configuration Utility

Almost every IT pro is going to be familiar with that screen.  Up until now, to team network adapters, one had to use vendor software such as the HP software pictured above. But starting with Windows Server 2012, the ability is built right in to the operating system, bringing the feature to new sets of hardware and without the need for any 3rd party vendor drivers or software! (Also of note is that Microsoft supports their NIC teaming, whereas they do not support the HP Network Configuration Utility.)

You can use the graphical Server Manager to configure NIC teams, but you can also do it all right from within Powershell. And since I typically prefer to keep my servers in straight-up Server Core mode, I wanted to figure out how to do it all from Powershell. My test machine for this experiment is a SuperMicro SYS-5015A-H 1U. It has two embedded GbE adapters (Realtek based.) Before Server 2012, I always just kept one of the NICs disabled since I had no use for it, and no teaming software. But now, I've installed a fresh copy of Windows Server 2012 Standard edition on it. 

Get-NetAdapter*Get-NetAdapter*

To make a team out of these two network adapters, simply do

New-NetLbfoTeam -Name Team -TeamMembers Ethernet,"Ethernet 2"

That's it! (Just put quotes around 'Ethernet 2' because it contains a space.) Now keep in mind that you'll probably have to re-do the IP configuration for your new NIC team now, so you'll want physical or DRAC/ILO access to the machine so you can do that. (Or do it via script. I set the IP configuration on my new NIC team via sconfig.) Here is what the new team looks like in Powershell: 

Get-NetLbfoTeam*Get-NetLbfoTeam*

The TeamingMode and LoadBalancingAlgorithm default to SwitchIndependent and TransportPorts, respectively, but of course can be configured to whatever you want as you create the team with the New-NetLbfoTeam command. Check this Technet article for explanations on the different options and what they do. If you later want to add another NIC to the existing team, you can use the Add-NetLbfoTeamMember command and specify the NIC you want to add.

 

Get-NetLbfoTeamMember*Get-NetLbfoTeamMember*

Beautiful.

Log Parser 2.2 and Log Parser Studio

by Ryan 31. October 2012 19:56

At first I thought to title this post the same as the catchphrase of Log Parser: "The Whole World Is Your Database!"

But then I decided that was a bit too exciting for what I actually wanted to talk about.

So I just discovered Log Parser Studio a few days ago. LPS is a graphical frontend to Log Parser; quite similar to how SQL Management Studio is a GUI frontend to interacting with SQL Server.  I am, quite frankly, ashamed that I didn't already know about it. It's fantastic.

The thing is... Log Parser is a command-line utility that uses a very SQL-esque language to interact with logs. What kind of logs, you ask?  Any kind of logs! That's right... you can use it to query the Windows Security Event Log, or you can use it to query a folder full of IIS web server logs, or you can use it to query a log full of your own personal electric utility bills from last year!

However, Log Parser itself is a very complex, albeit powerful and flexible, command-line utility. Maybe you want something a little more user-friendly to get you started. That's exactly where Log Parser Studio, the GUI frontend, comes in to play.

As a little demonstration, I installed Log Parser 2.2 on my workstation. Then I downloaded Log Parser Studio to my workstation. I fired it up as a Windows application, and I pointed it to the remote IIS logs directory of this very web server. I then right-clicked on "IIS: Request per Hour" and chose "Run report now." As if I had just run a SQL query in SQL Management Studio, this window popped up:

 

Log Parser Studio Query*Click for Bigger*

This data is probably every single HTTP GET request made per hour, rather than a count of hits made by unique IP addresses, but the point is you now have this amazing utility that will parse practically any amount of data you can think of from any source of data you can think of. Go check it out and see how Log Parser is even capable of generating pie charts and bar charts and all sorts of crazy things using this data!

About Me

Name: Ryan Ries
Location: Texas, USA
Occupation: Systems Engineer 

I am a Windows engineer and Microsoft advocate, but I can run with pretty much any system that uses electricity.  I'm all about getting closer to the cutting edge of technology while using the right tool for the job.

This blog is about exploring IT and documenting the journey.


Blog Posts (or Vids) You Must Read (or See):

Pushing the Limits of Windows by Mark Russinovich
Mysteries of Windows Memory Management by Mark Russinovich
Accelerating Your IT Career by Ned Pyle
Post-Graduate AD Studies by Ned Pyle
MCM: Active Directory Series by PFE Platforms Team
Encodings And Character Sets by David C. Zentgraf
Active Directory Maximum Limits by Microsoft
How Kerberos Works in AD by Microsoft
How Active Directory Replication Topology Works by Microsoft
Hardcore Debugging by Andrew Richards
The NIST Definition of Cloud by NIST


MCITP: Enterprise Administrator

VCP5-DCV

Profile for Ryan Ries at Server Fault, Q&A for system administrators

LOPSA

GitHub: github.com/ryanries

 

I do not discuss my employers on this blog and all opinions expressed are mine and do not reflect the opinions of my employers.