Windows Emergency Management Services

by Ryan 12. November 2013 20:37

BSODToday we're going to talk about one of the more esoteric features of Windows.  A feature that even some seasoned sysadmins don't know about, and that almost nobody outside of kernel debuggers and device driver writers in Redmond ever use...

Emergency Management Services!

Imagine you have a Windows computer that has suffered a blue screen of death. If you want to sound more savvy, you might call it a STOP error or a bug check. Pictured is a very old example of a BSoD, but it's just so much more iconic than the pretty new Win8 one with the giant frowny face on it.

So you're sitting there staring at a blue screen on the computer's console... can you still reboot the machine gracefully?  Or even crazier, could you still run, for example, Powershell scripts on this machine even after it has suffered some massive hardware failure?

Don't reach for that power button just yet, because yes you can!

You might have thought that once a Windows computer has blue-screened, then it's done. It's stopped forever and it cannot execute any more code, period.  I thought that myself for a long time. But lo and behold, there's still a little juice left even after you've blue-screened, and all you need is a serial or USB cable.  That's where Emergency Management Services comes in.

As the name implies, EMS is typically there for when all else fails. For when your computer has already gone to hell in a handbasket. You could consider it an out-of-band management solution.

Of course you need to have already enabled it beforehand, not after a bug check has already occurred. You'd enable it on Vista/2008 and above like so:

Bcdedit.exe /EMS ON /EMSSETTINGS BIOS

If using a USB port, or

Bcdedit.exe /EMS ON /EMSSETTINGS EMSPORT:COM2 EMSBAUDRATE:9600

If using an RS-232 serial port. (How quaint.)

Now that it's enabled, you can connect to the Special Administration Console (SAC.)

SAC Special Administration Console

From here, you can launch a command prompt (Cmd.exe,) and from there, you can launch Powershell.exe!  All over a serial or USB cable connection. If the regular SAC mode cannot be entered for some reason, then EMS will put you in !SAC mode, where you can still at least read the event logs and reboot the server in a more graceful manner than just pulling the plug.

Mark Russinovich has this to say about the Windows boot up process as it concerns EMS:

"At this point, InitBootProcessor enumerates the boot-start drivers that were loaded by Winload and calls DbgLoadImageSymbols to inform the kernel debugger (if attached) to load symbols for each of these drivers. If the host debugger has configured the break on symbol load option, this will be the earliest point for a kernel debugger to gain control of the system. InitBootProcessor now calls HvlInit System, which attempts to connect to the hypervisor in case Windows might be running inside a Hyper-V host system’s child partition. When the function returns, it calls HeadlessInit to initialize the serial console if the machine was configured for Emergency Management Services (EMS)."
Mark Russinovich, David Solomon, Alex Ionescu, Windows Internals 6th Ed.

So there you have it. Even when faced with a BSoD, if you have an opportunity to shut down or reboot the machine in a more graceful manner than just pulling the electricity from it, then you should do it.

My Powershell Profile Just Went Full-Glitz

by Ryan 24. August 2013 14:48

My cat woke me up extremely early this Saturday morning with the incessant meowing and carpet-scratching that signals either her boredom, or an empty food dish.

So I got up, made some coffee, put some meat crackers into kitty's bowl, and then started tinkering with my Powershell profile... now it looks like this every time I launch PS: 

Powershell Profile

It all started a couple weeks ago when I watched a Channel9 video where Jeffrey Snover was playing with Powershell, and I noticed that he had changed his error text color to green. I'm guessing like so:

$Host.PrivateData.ErrorForegroundColor = "Green"

I don't know why he configured his error messages to be green. Maybe it's just because it's easier to see than the default red.  But I like to imagine the idea is to promote positive feedback... like elementary school teachers marking their student's incorrect homework answers with another color of pen besides a red pen... because red ink makes the kids feel bad.

Anyway, as I started playing with text colors and title bar text and whatnot, it occured to me that all these settings would just revert to defaults after I closed this Powershell session. So how do we make such changes permanent?

The Powershell Profile!

Just type $Profile into Powershell right now, and it will tell you the full path of your very own PS profile. It should be something like this:

C:\Users\Ryan\Documents\WindowsPowerShell\Microsoft.PowerShell_profile.ps1

That script gets executed first thing every time you launch PS. It may not exist yet - you have to create it.  Just type Notepad $Profile and Notepad will open that file up, or prompt you to create it if it doesn't already exist.

I'm still thinking of more neat gizmos to throw in here, but this is good for now. The  weather information comes from the Yahoo Weather web API, and the ServerFault rep information comes from the StackExchange API. *Swoon...* REST APIs are so dreamy...

The StackExchange API gives you 300 anonymous calls per day per IP (more if you authenticate.)  There is a basic amount of error handling so that if you can't connect to one or the other of the web APIs to get the data for whatever reason, it will just replace the appropriate string with [Error connecting to weather API], and so on. You'd want to put a short timeout on the API calls too... Powershell doesn't need any help being slow to load!

And without further ado, here's the code:

Set-StrictMode -Version Latest
[String]$WOEID           = "2355944" # Where on earth ID for Arlington TX
[String]$WelcomeName     = "Ryan"
[Xml]$WeatherAPIResponse = $Null
$StackExAPIResponse      = $Null
[String]$WelcomeBanner   = [String]::Empty
[String]$WeatherString   = [String]::Empty
[String]$StackExString   = [String]::Empty

Try
{
    $WeatherAPIResponse = Invoke-WebRequest http://weather.yahooapis.com/forecastrss?w=$WOEID -TimeoutSec 3 -ErrorAction Stop
    If($WeatherAPIResponse -NE $Null -AND $WeatherAPIResponse.PSObject.Properties.Match('rss').Count)
    {
        $WeatherString = "Current weather in $($WeatherAPIResponse.rss.channel.location.city), $($WeatherAPIResponse.rss.channel.location.Region): $($WeatherAPIResponse.rss.channel.item.condition.temp)°, $($WeatherAPIResponse.rss.channel.item.condition.text), $($WeatherAPIResponse.rss.channel.atmosphere.humidity)% humidity"
    }
    Else
    {
        Throw
    }
}
Catch
{
    $WeatherString = "[Error connecting to weather service.]"
}

Try
{
    $StackExAPIResponse = Invoke-WebRequest https://api.stackexchange.com/users/104624?site=serverfault -TimeoutSec 3 -ErrorAction Stop
    If($StackExAPIResponse -NE $Null -AND $StackExAPIResponse.PSObject.Properties.Match('Content'))
    {
        $StackExString = "Current ServerFault rep: $($(ConvertFrom-Json $StackExAPIResponse.Content).Items.Reputation) total,  $($(ConvertFrom-Json $StackExAPIResponse.Content).Items.reputation_change_day) today, $($(ConvertFrom-Json $StackExAPIResponse.Content).Items.reputation_change_week) this week"
    }
    Else
    {
        Throw
    }
}
Catch
{
    $StackExString = "[Error connecting to StackExchange.  ]"
}


$WelcomeBanner      = @"
            .ooooooo            Welcome back, $WelcomeName!
          oooooooooooo          $WeatherString
        ooooo      ooooo        $StackExString
       oooo          oooo       
       ooo            .oo       
   oooooooooo          ooo      
  ooooooo.oooo.        oo.      
 ooo        .o.        ooooo    
ooo                    ooooooo  
oo                    .oooooooo 
oo                    oo     ooo
ooo                           oo
.oo                          ooo
 oooo                        oo.
  .oo myotherpcisacloud.com oo 
    .oooooooooooooooooooooooo  

"@

Write-Host $WelcomeBanner -ForegroundColor Cyan

So Long, TechNet Subscription.

by Ryan 1. July 2013 18:45

It's been fun.  Got the email this afternoon.  I'm not sure that I'll be able to do much lab stuff any more.  Which means less content for this blog.  Less ability to answer questions on Server Fault through having the ability to quickly verify things.  Less ability for me take the things I've learned and tested and use them for the benefit of my employers.  Less ability to try out the extremely atypical scenarios that I'd get asked in the usual tricky Microsoft exam yet never see in a production environment.

I guess I can still get stuff from TechNet Evaluation Center, but as far as I can tell I'll have to promptly rebuild my entire lab every 6 months, which makes me less inclined.

I'll think of something.  Man, I never thought I'd be saying this, but sometimes I feel like things would be a lot easier on me if I just specialized in Linux stuff.

Psst, You Want A Script To Backup Your Lab VMs?

by Ryan 23. June 2013 09:33

I can hook you up...

So I'm always doing a lot of lab work with Hyper-V virtual machines. Every once in a while I want to just save the state of the entire lab all at once and back it up to a safe storage volume.  I suppose I could set up Windows Server Backup on each of the VMs, and find some disk to use as a pass-through disk for one of the virtual machines and then share that so that the VMs could back up to the network share... but that's a ton of hassle.

How about I just save the state of all the VMs, and export them directly to my backup volume, then resume the VMs, all from the hypervisor?  As a scheduled task, perhaps?

About 10 minutes in the Powershell ISE and I've done just that.  A couple things to be warned of - First, you can't do this in production. The virtual machines are frozen while they're being exported, and it can take several minutes to export a VM. Secondly, make sure you are running with full Administrator privileges, or else cmdlets such as Get-VM will silently return nothing.

 

# Ryan Ries, 2013
# Backs up some lab VMs. Takes several minutes at least.

[String]$BackupPath = "D:\Backups\Hyper-V"
[String]$ErrorLog   = "D:\Backups\Hyper-VBackups.log"
$Start = Get-Date
"$(Get-Date) - Hyper-V Backups Starting." | Out-File $ErrorLog -Append
Try
{
    Get-Childitem $BackupPath -Recurse -Force | Remove-Item -Recurse -Force -ErrorAction Stop
}
Catch
{
    "$(Get-Date) - Error during Get-Childitem or Remove-Item: $($_.Exception.Message)" | Out-File $ErrorLog -Append
    Return
}
Try
{
    Get-VM -ErrorAction Stop | Stop-VM -ErrorAction Stop -Save
}
Catch
{
    "$(Get-Date) - Error during Get-VM Stop-VM -Save: $($_.Exception.Message)" | Out-File $ErrorLog -Append
    Return
}
Try
{
    Get-VM -ErrorAction Stop | Export-VM -ErrorAction Stop -Path $BackupPath
}
Catch
{
    "$(Get-Date) - Error during Export-VM: $($_.Exception.Message)" | Out-File $ErrorLog -Append
    Return
}
Try
{
    Get-VM -ErrorAction Stop | Start-VM -ErrorAction Stop
}
Catch
{    
    "$(Get-Date) - Error during Start-VM: $($_.Exception.Message)" | Out-File $ErrorLog -Append
    Return
}
$End = Get-Date
"$(Get-Date) - Hyper-V Backups completed in $([Math]::Round((New-TimeSpan $Start $End).TotalMinutes)) minutes." | Out-File $ErrorLog -Append

Neat Windows Tricks (Or Back When I Was Young and Foolish Pt. III)

by Ryan 5. March 2013 18:29

I work with Windows a lot. Almost every day of my life since Windows 3.1, I've been submersed in Windows for both work and play. Getting to know it inside and out. Learning new Windows applications. Being excited for new releases of Windows because I know it'll bring big changes to the operating system which will make me have to learn new things. Every now and then, I even start to think I'm pretty knowledgable about Windows...

Which is why I'm always flabbergasted when someone non-chalantly shows me a relatively mundane Windows trick I never knew about, yet it's been under my nose the whole time!  And so, I bring you two Windows tricks that I learned about today. If you already knew about them, you can just think of me as an amateur and move on.  But if you didn't already know about them, they might just change the way you do your daily work!

C:\> SomeCommand | clip

 I never knew about this! On the command line, you can pipe the output of any program or command to your Windows clipboard and Ctrl+V it anywhere!

Clip

You can also do something like

C:\> clip < textFile.txt to quickly copy the contents of the file to your clipboard.

Alright, trick #2:

Shift + Right-Click to Run as Different User

This one was down-right embarrassing for me to have not known. Maybe I did know it at one time, but then I used it so infrequently that I forgot about it... I don't know. Back in the XP/2003 days, you could right-click on an application and choose "Run as..." which would prompt you for the credentials of another user under which to to run that process. But then in Vista onwards it disappeared. Sure, there's still "Run as administrator," but sometimes you need to run a process as someone else besides Administrator. Well, I always just got around it by launching a command prompt and doing something like runas /user:BobMarley@domain.com notepad.exe.

But I completely forgot that it's still there in the GUI.  All you have to do is Shift+Right-click:

Shift+Right-click

It's a Christmas Miracle!

by Ryan 25. December 2012 11:06

Well I did something quite embarrassing the other day. But then I did something pretty cool to make up for it. I'll call it a Christmas miracle, because it involves me not knowing the domain admin password, having no other normal way of logging in, despairing for a couple minutes, then getting back in anyway. And also because I'm histrionic like that. It's a real story of tribulation and triumph.

A couple days ago, the domain admin password on one of my lab domains expired unexpectedly, and so I was prompted to reset it upon RDPing to a server. So I did. As a matter of habit, I saved the new password in my password vault software. (I'm one of those people that has a different password for every single account I have, but as a result I can't remember them all.)

At least, I thought I saved the new password.

Fast forward to a week later. I wanted to do some work in that lab again, and as I began the login process, I realized that I had no idea what I had set the password to. The domain admin password. The only Administrative account in that domain. "No problem," I thought. It's in my password vault...

It wasn't.

I went through several backup copies of the password file, thinking that it must have been overwritten. Nope. Dreadful thoughts of spending the whole weekend rebuilding the lab flashed before my eyes.

Then I had an idea.

I switched my monitor over to the physical domain controller, and plugged in a keyboard.

Server 2012 Login

It's a very calm and soothing login screen, isn't it? I like it.

Anyway, there's the ever-present "Accessibility" or "Ease of Access" icon down there in the corner. (I know many of you already know where I'm going with this, and you're right. But if you don't know, read on. Otherwise, you may want to skip to the "Afterthoughts" section below.) This domain controller is running Server 2012 Core. That accessibility button in the corner doesn't do anything. I can click it, but it doesn't do anything. Maybe it's because this is a Server Core installation with no GUI elements. Hm.

I had a Windows installation image on a bootable USB thumb drive handy, so I plugged that in to the domain controller and rebooted the machine (using the power button on the chassis obviously, since I wasn't going to be logging in.)

Once the machine had booted off the installation media, instead of choosing to install Windows, I chose "Repair your computer." From there, I chose the option to launch a Command Prompt. Voila. Sweet. I now had a shell on the computer.

Next I navigated on over to where everything interesting happens, \Windows\System32. (The drives had different drive letters than normal, but who cares.) Now, under regular circumstances, that little Accessibility button on the logon screen launches a program called Utilman.exe. From there you can get all the gizmos like the on-screen keyboard and magnifier and things like that to help you log in, in case your eyesight's not so good or you have no hands or something. But there was no Utilman.exe in the System32 folder. I guess that's why the Accessibility icon didn't do anything when I clicked it. My domain controllers are very inconsiderate of the handicapped.

So I just decided to try the command:

copy X:\Windows\System32\cmd.exe X:\Windows\System32\Utilman.exe

Then I rebooted. Once I was back at the Windows login screen, I hit the Accessibility icon one last time. Command Prompt. And not just any Command Prompt, but a Command Prompt running as Local System! Since this was a normal boot of the domain controller and therefore Active Directory was online, I just did:

net user DomainAdmin *

And I was greeted with a prompt to reset the domain administrator password to whatever I wanted.

This time I made damn sure that the new password was saved and backed up in my password vault.

Afterthoughts

So it's great that I was able to reset a domain administrator password with nothing but a bootable image and physical access to the machine, but, one can't help but wonder about the glaring security implications of this. Once a person has a Local System shell on a domain controller, they own the domain.

Furthermore, today's enterprise environments are more complex than ever. Having physical access to the console of the machine doesn't necessarily mean you have to actually be able to touch the machine, like I was able to do here. These days, "physical access" can mean logging in to an ILO or accessing a virtual machine management console from a thousand miles away. And are we, as enterprise administrators and IT directors and CTOs, sure that everyone in our organization with access to those things are people we also trust as potential domain admins? It's not just about the threat of outside attackers. We may be unwittingly exposing a massive attack surface to all of our front-line support personnel should they ever decide to go rogue.

Alright, so what should be done about this little "feature" of Windows? First off, Microsoft needs to not launch Utilman.exe as Local System. Launch it as an unprivileged account that doesn't have the ability to do things like reset Administrator passwords. I'm sure there's a technical reason why they think they "have" to run the Ease of Access stuff as Local System, so that it can simulate the secure attention sequence, etc., but I'm not buying it.  Just because it's hard doesn't mean they should get away with letting this gaping security hole go unpatched.  They could also include a hash check in the code behind that little icon so it would at least be a little more difficult to replace the real Utilman.exe with, say, a copy of Cmd.exe that was renamed to Utilman.exe.

To make matters worse, there is no "official" Group Policy setting to disable this button. You might be able to mitigate the risk somewhat by using Group Policy to do something like run a script that cacls Utilman.exe at startup and denies Everyone all access, or having Group Policy push out your own version of Utilman.exe at every boot of the computer... neither of which are ideal solutions nor are they likely to be bulletproof.

Oh well. I guess that goes along with that immutable law of computer security, that if someone else has physical access to your computer, then it's not your computer anymore.

Passive-Aggressive Configuration Management FTW

by Ryan 13. August 2012 15:52

I was doing a little deep investigation of the Windows DNS Cache service today, and I discovered that the process checks for the existence of this registry value upon startup:

HKLM\System\CurrentControlSet\services\Dnscache\Parameters\DowncaseSpnCauseApiOwnerIsTooLazy

Needless to say, the configuration setting doesn't appear to be publicly documented.  Whatever it does though, I sense some latent hostility toward some API owner. It wouldn't be the first time Microsoft has let Registry settings with silly names slip through the cracks.

From the "No Not All My Stuff Is Patched Yet" Dept.

by Ryan 8. May 2012 19:35

This is a public service announcement.

If you have a Windows 2008 R2 server without SP1 or without this hotfix, and you also have a monitoring application or script that uses WMI to query for service information from the Win32_Service WMI class on a regular basis, you might just get burned really badly by a memory leak that will have the WMI service gobbling up >500MB of memory until it crashes.  You might then have to roll back hours of work and hope the Microsoft bug didn't cause any cluster failovers or dropped Exchange mailbox stores across a couple dozen servers.

ORS - Office Rageface Sender

by Ryan 2. April 2012 22:33

freddie

If you work in an office in a corporate environment like I do, you're probably familiar with Microsoft Office Communicator, often referred to as OCS. These days they call it Lync -- but it's still most widely known as OCS. Anyway, it's an IM client that you can use to communicate with your bosses... and for them to see when you're taking a bit too long of a lunch break. The one thing it doesn't do, however, is quickly paste pictures for others to see. A picture is worth a thousand words, right? So I set out this last weekend to remedy this situation.

What I ended up with a couple days later is what I'm calling ORS - or Office Rageface Sender. A coworker of mine is very fond of those ragefaces that seem to be all the... rage... lately, but it's difficult to make those jokes without actually being able to show the relevant picture to go with it. 

Currently, ORS is a network application, but only works within your current subnet, because it uses UDP broadcasts as a "discovery" mechanism to discover peers on the network (i.e. other people also running ORS) to populate your contact list. In addition, TCP port 9068 is used for direct communication. 

Upon launching ORS for the first time, you will be asked for your nickname. You can change it at any time by clicking the status bar at the bottom of the main window. Your nickname will be saved in the registry so it won’t ask you every time you launch the app. The effects of duplicate nicknames on the network hasn’t been thoroughly explored, (hey I’m only one guy) but they should be minimal as communications are typically IP-based. 

Also when you launch ORS for the first time, it will create an Images folder at the same location where the executable is running. Dump all your favorite images here. Optimally, they should be as close to 512x512 as possible, as they will be displayed in a 512x512 window. However, images larger than that will be automatically scaled down to fit. Images smaller than that will be centered (not stretched.) 

When you right-click on a person’s name in the main window, a context menu will pop up which contains a list of all the images currently in your Images directory. This list and context menu is dynamic, so you don’t need to relaunch the app every time you modify the contents of your images directory. By clicking an image name over a contact, that image will be displayed on their screen in real-time. 

If the recipient does not currently have the image that you are trying to send them, you will automatically send it to them over TCP, it will be saved to their own Images directory, and then displayed normally. If they already have the same image (as determined by name,) that local image will be displayed. If two users have the same filename in their images directory but are actually different pictures, then the recipient will see a different image than the one you intended. 

The application minimizes to the system tray. You can right-click the icon to exit the app, or just close the form. 

If you’d like to give this a try when you’re in the same broadcast domain with one or two other people, you can download the program at the very end of this post. I very much welcome bug reports, feature requests, etc. You probably don't want to run this with a bunch of people you don't trust, as it would be possible for them to flash pictures of boobs on your screen if they wanted to.

ors

Finally, here are some stats on how much broadcast traffic each client sends, just to prove how nominal it is. About 1 packet every 10 seconds. 

Avg packets/sec 0.128
Avg packet size 75 bytes
Avg bytes/sec   9.587

ORS.exe (93.00 kb)

Stack Exchange An Early Adopter For SQL 2012?

by Ryan 9. March 2012 12:26

I just coincidentally happened to catch this on Server Fault a couple days ago. A couple minutes after my comment, he added the bolded edit at the bottom:

serverfault

Ironically, I saw it fit to place this post in both the "IT Pro" and "IT (Not Very) Pro" categories simultaneously. Not sure how you mistake "posting on a public web forum" with "sending a private email."  Oh well. I got a chuckle out of it.

About Me

Name: Ryan Ries
Location: Texas, USA
Occupation: Systems Engineer 

I am a Windows engineer and Microsoft advocate, but I can run with pretty much any system that uses electricity.  I'm all about getting closer to the cutting edge of technology while using the right tool for the job.

This blog is about exploring IT and documenting the journey.


Blog Posts (or Vids) You Must Read (or See):

Pushing the Limits of Windows by Mark Russinovich
Mysteries of Windows Memory Management by Mark Russinovich
Accelerating Your IT Career by Ned Pyle
Post-Graduate AD Studies by Ned Pyle
MCM: Active Directory Series by PFE Platforms Team
Encodings And Character Sets by David C. Zentgraf
Active Directory Maximum Limits by Microsoft
How Kerberos Works in AD by Microsoft
How Active Directory Replication Topology Works by Microsoft
Hardcore Debugging by Andrew Richards
The NIST Definition of Cloud by NIST


MCITP: Enterprise Administrator

VCP5-DCV

Profile for Ryan Ries at Server Fault, Q&A for system administrators

LOPSA

GitHub: github.com/ryanries

 

I do not discuss my employers on this blog and all opinions expressed are mine and do not reflect the opinions of my employers.