I was discussing with some fellow IT admins, the topic of blocking certain websites so that employees or students couldn't access them from the work or school network.  This is a pretty common topic for IT in most workplaces.  However, I personally don't want to be involved in it.  I realize that at some places, like schools for instance, filtering of some websites may be a legal or policy requirement.  But at the workplace, if an employee wants to waste company time on espn.com, that is an issue for HR and management to take up with that employee.  And again in my opinion, it's not about how much time an employee spends on ESPN or Reddit either, but simply whether that employee delivers satisfactory results.  I don't want to handle a people problem with a technical solution.  I don't want to be the IT guy that derives secret pleasure from blocking everyone from looking up their fantasy football scores.  (Or whatever it is people do on espn.com.)  I could spend my entire career until I retire working on a web proxy, blocking each and every new porn site that pops up.  If there's one thing the internet has taught me, it's that there will always be an infinite number of new porn sites.

On the other extreme of black listing, someone then suggested white listing.  Specifically, implementing "DNS white listing" in their environment for the purpose of restricting what internet sites users were allowed to access to only a handful of internet sites.  Well that is a terrible idea.  The only proper way of doing this in my opinion is to use a real web proxy, such as ISA or TMG or Squid.  But I could not help but imagine how I might implement such a system, and then how I might go about circumventing it from the perspective of a user.

OK, well for my first half-baked idea, I can imagine standing up a DNS server, disabling recursion/forwarders on that DNS server, and putting my "white list" of records on that DNS server.  Then, by way of firewall, block all port 53 access to any other IP except my special DNS server.  Congratulations, you just made your users miserable, and have done almost nothing to actually improve the security of your network or prevent people from accessing other sites.  Now the users just have to find another way of acquiring IP addresses for sites that aren't on your white list.

Well how do I get name resolution back if I can't use my DNS server?  I have an idea... DNS over HTTP!

The guys at StatDNS have already thought about this.  And what's awesome, is that they've created a web API for resolving names to IPs over HTTP.  Here's what I did in 5 minutes of Powershell:

PS C:\> Function Get-ARecordOverHTTP([string]$Query) { $($($(Invoke-WebRequest http://api.statdns.com/$Query/a).Content | ConvertFrom-Json).Answer).rdata }

PS C:\> Get-ARecordOverHTTP google.com
173.194.70.101
173.194.70.100
173.194.70.138
173.194.70.102
173.194.70.139
173.194.70.113

PS C:\> Get-ARecordOverHTTP myotherpcisacloud.com
168.61.52.184

Simple as that. How cool is Powershell, seriously?  One line to create a function that accepts a name and returns a list of IPs by interacting with an internet web service.  Pretty awesome if you ask me.

As long as you have port 80 open to StatDNS, you have internet name resolution.  Now, to wrap this into a .NET-based Windows service...

I was in Salt Lake City most of this week. Being surrounded by stark snow-covered mountains made for some wonderful scenery... it could not be more different than it is here in Texas. Plus I got to meet and greet with a bunch of Novell and NetIQ people. And eat an enormous bone-in ribeye that no human being has any business eating in one sitting.

But anyway, here's a little AD mystery I ran in to a couple weeks ago, and it may not be as simple as you first think.

As Active Directory admins, something we're probably all familiar with is member servers authenticating with the "wrong" domain controller. By wrong, I mean a DC that is in a different site than the member server, when there's a perfectly fine DC right there in the same site as the member server, and so the member server is incurring cross-site communication when it doesn't need to be. Everything might still function well enough as long as the communication between DC and member server is successful, but now you're saturating your slower inter-site WAN links with AD traffic when you don't need to be. You should want your AD replication, group policy application, DFS referrals, etc., to run like a well-oiled machine.

I often work in a huge environment with AD sites in many countries and on multiple continents, and thousands of little /26 subnets that can't always be easily grouped into a predictable supernet for the purposes of linking subnets to sites in AD Sites & Subnets. So I'm always alert to the fact that if I log on to a server, and I notice that logon takes an abnormally long time, I very well could be logging on to the wrong DC. First, I run set log to see which DC I have logged on to:

*DC01 is in Amsterdam*

So in this case, I noticed that while I had logged on to a member server in Dallas, that server's logon server was a DC in Europe. :(

You immediately think "The server's IP subnet isn't defined in AD Sites & Services or is associated to the wrong site," don't you?  Yeah, me too. So I went and checked. Lo and behold, the server's IP subnet was properly defined and associated to the correct site in AD.

Now we have a puzzle. Back on the member server, I run nltest /dsgetsite to verify that the domain member does know to which site it belongs. (Which the domain member's NetLogon service stores in the registry in the DynamicSiteName value once it's discovered.)

I also ran nltest /dsgetdc:domain.com /Account:server01$ to essentially emulate the DC locator and selection process for that server, which basically just confirmed what we already knew:

C:\Users\Administrator>nltest /dsgetdc:domain.com /Account:server01$ 
           DC: \\DC01.DOMAIN.COM (In Amsterdam) 
      Address: \\10.0.2.55 
     Dom Guid: blah-blah-blah 
     Dom Name: DOMAIN.COM 
  Forest Name: DOMAIN.COM 
 Dc Site Name: Amsterdam 
Our Site Name: Arlington 
        Flags: GC DS LDAP KDC TIMESERV WRITABLE DNS_DC DNS_DOMAIN DNS_FOREST FUL
L_SECRET WS 
The command completed successfully

So where do we look next if there's no problem with the IP subnets in AD Sites & Services?  I'm going with DNS. We know that domain controllers register site-specific SRV records so that clients who know to which site they belong will know what DNS query to make to find domain controllers specific to their own site.  So what DNS records did we find for the Arlington site?

Forward Lookup Zones
    _msdcs.domain.com
        dc
            _sites
                Arlington
                    _kerberos SRV NewYorkDC
                    _kerberos SRV SanDiegoDC
                    _kerberos SRV MadridDC
                    _kerberos SRV ArlingtonDC
                    _ldap     SRV NewYorkDC
                    _ldap     SRV SanDiegoDC
                    _ldap     SRV MadridDC
                    _ldap     SRV ArlingtonDC

OK, now things are getting weird.  All of these other domain controllers that are not part of the Arlington site have registered their SRV records in the Arlington site.  The only way I can imagine that happening is because of Automatic Site Coverage, whereby domain controllers will register their own SRV records into sites where it is detected that the site has no domain controllers of its own... combined with the fact that scavenging is turned off for the DNS server, including the _msdcs zone.  So someone, once upon a time, must have created the Arlington site in AD before the actual domain controllers for Arlington were ready.  What's more is that Automatic Site Coverage is supposed to intelligently use site link costing so that only the domain controllers in the next closest site provide "coverage" for the site with no DCs, not every DC in the domain. Turns out the domain did not have a site link strategy either - it used DEFAULTIPSITELINK for everything - the entire global infrastructure. So even after Arlington did get some domain controllers, the SRV records from all the other DCs stayed there because of no scavenging.

Here's the thing though - did you notice that almost every other domain controller in the domain had SRV records registered in the Arlington site, except for the domain controller in Amsterdam that our member server actually authenticated to!?

This is getting kinda' nuts.  So what else, besides the DNS query, does a member server perform in order to locate a suitable domain controller?

So after a client does a DNS query for _ldap._tcp.SITENAME._sites.ForestDnsZones.domain.com, and gets a response, the client then begins to do LDAP queries against the DCs given in the DNS response to make sure that the DCs are alive and servicing requests. If you want to see this for yourself, I recommend starting Wireshark, and then restarting the NetLogon service while the capture is running. If it turns out that none of the DCs in the list that was returned by the site-specific DNS query is responding to your LDAP queries, then the client has to back up and try again with a domain-wide query.

And that is what was happening. The client, server01, was getting a list of DCs for its site, even ones that were erroneously there, but I confirmed that it was unable to contact any of those domain controllers over port 389. So after that failed, the server was forced to try again with a domain-wide query, where it finally found one domain controller that it could perform an LDAP query on... a domain controller in Amsterdam.

Moral of the story: Always blame the network guys.

You probably already know that all of the Sysinternals tools, such as Process Monitor, Process Explorer, Autoruns, and much more, can be accessed via "shared folder" from any computer connected to the internet by navigating to \\live.sysinternals.com\.  This isn't the same kind of share you'd create if you just shared a folder on your PC.  It's a WebDAV share, and is accessed over HTTP.

Sometimes though, I feel the need to access this share from the command line, either in the Cmd shell or Powershell.  Alas, here's what I see:

*Path not found.*

I get the same result with Powershell. Bummer. Well I know I can access the path with Explorer when I type that same UNC into the address bar, or if I just type the UNC into the Run dialog box, so this must just be a limitation of those command-line tools, right?

*Works fine in Explorer*

Oh well... but wait. Now having successfully accessed the network path with Explorer, let me now immediately go back to the Cmd shell and try it again:

*Now it works in Cmd too!*

OK, now accessing the network path works fine from the Cmd shell and from Powershell, even though all I did was access it through Explorer first, and then try again. Now I just have to know what the heck is going on... and to do that, I need to use Process Monitor. Which, amusingly, is in the WebDAV share I'm trying to access. But I'll run a local copy.

I started the trace. Here's my first attempt to access the network path with Cmd.exe, which failed:

*Network path not found*

This was the very first time in the Process Monitor trace when the string "live.sysinternals.com" appeared in the Path field. It's also the first time the Cmd.exe process shows up in the trace. It's currently filtered to only include events where the Path field contains the string live.sysinternals.com. The really interesting part about this is that it appears the moment I pressed Enter on the command line, Explorer.exe was the first process to be involved, not the process I was interacting with! That's odd. Maybe a file system filter driver intercepted the call and notified Explorer? It looks like Explorer is looking for something related to named pipes and the Workstation Service (wkssvc) on the remote server, but it doesn't find it.  Then Cmd.exe first checked my local file system for a file in Windows\CSC\ directory, which it didn't find, and then it tried to access the network path that I actually asked for, which resulted in "Bad network path." Then it apparently tries again with the same local file system path, and then again with the network directory instead of the specific executable name.  All failed. "Network path not found," my command prompt tells me. But with no further input from me, Explorer takes off doing its own thing, calling cscapi.dll and loading things in the background and sending things over network. All I did was hit enter in the Command Prompt above.

So what is this CSC directory? Googling the term led me to an old post on Raymond Chen's blog. Client Side Caching. OK, so apparently both processes are looking for a cached or offline version of the network path.

Then I move over to the Explorer.exe window and type the path into the address bar. Explorer looks for some more CSC stuff first, and then svchost.exe starts communicating with the remote server over TCP. There's a lot of loading of WebDAVRedirector stuff. Finally, after a lot of work, I start seeing events like these from Explorer:

*Explorer starts finding it, finally*

Notice that Explorer also seems to be storing the autoruns executable in a temporary "Tfs_DAV" directory on my workstation.

Finally, after having success with Explorer, I go right back to the Command Prompt and try it again. This time, the trace looks like this:

Now I see svchost.exe stepping in with a WebDavRedirector, and cmd.exe getting some successful returns from its IRPs. Finally, after playing around in that Tfs_DAV directory and some more intermingling of svchost.exe and the System process both helping out, the process autoruns.exe finally launches.

So that's a pretty fast and loose overview of what is actually going on. The entire trace was a beast to wade through, and there is obviously a lot of orchestration and cooperation required between many different Windows components required to allow you to access a WebDAV share from within Cmd.exe and I don't fully understand all of it... but the bottom line is that at least on my Windows 7 SP1 x64 workstation, it looks like Explorer.exe is smart enough to read from a WebDAV share and cache the data locally, whereas Cmd.exe is only smart enough to read the data locally, if and only if it's already cached locally... or perhaps the redirector had to be "woken up" by Explorer first, before Cmd.exe was able to use it.

Finally, I'll leave off with a bit about the WebDAV Mini-Redirector from Wikipedia:

"In Windows XP, Microsoft added the Web Client service is also known as the WebDAV mini-redirector^[11] which is preferred by default over the old Web folders client. This newer client works as a system service at the network-redirector level (immediately above the file-system), allowing WebDAV shares to be assigned to a drive letter and used by any software. The redirector also allows WebDAV shares to be addressed via UNC paths (e.g. http://host/path/ is converted to\\host\path\) for compatibility with Windows filesystem APIs."

I'm calling this Part 1 because I realized as I started writing that this is a lot of work, and can easily be split into 2 or more articles.

Like most of the IT Pro community, I've been getting comfortable with Server 2012 the past several weeks now, and the journey is still ongoing. As I talked about last time, I do like those Best Practices Analyzers for Windows Server. Here's me running it in Server Manager:

*BPA in Server Manager 2012*

Getting any of these results back means that I have some work to do in remediating them. It's not uncommon for a Server 2012 system that was just built fresh with no applications loaded or configuration changes to still have one or two compliance issues in the Best Practices Analyzers. There is a balance to be maintained between compatibility and performance optimization. Also, many of these issues that popped up for me personally were not role-specific, but rather apply to a base component of the OS. Now I'll go over some of the interesting ones I've gotten and how I fixed them:

[Hyper-V]: Avoid storing Smart Paging files on a system disk.

Smart paging is new with Hyper-V 2012. Read about it here. Basically, you now enter a minimum amount of RAM a virtual machine can have, you enter a maximum amount of RAM a virtual machine can have, and you also now enter the amount of startup RAM a virtual machine can have. The Windows OS can boot up more comfortably with a larger amount of RAM, but once it reaches cruising altitude and is idle, the RAM requirements go back down, which will allow the Hyper-V host to gradually start reclaiming memory from the VM. With all this dynamic shrinking and growing of the memory on all your VMs, that's where the "smart paging file" comes in. And just like you can improve performance by putting your traditional Windows paging file on its own disk, the same goes with the Smart Paging file.

[Hyper-V]: Use RAM that provides error correction.

Microsoft doesn't support Hyper-V environments on hardware that isn't using ECC RAM. This is just a lab using desktop-grade hardware, so there's nothing I can really do about this. If you're using real server gear, this should not be an issue for you.

[Hyper-V]: Virtual machines should be backed up at least once every week.

This one I still don't understand. I have all the guest operating systems backing up nightly, and then I am backing them up again through the backup of the Hyper-V host. So go away, error.

[Windows]: Short file name creation should be disabled.

Then why is it enabled by default? Oh, I know why... it's because of your crappy line-of-business apps that were written back in 1998 that you can't get rid of, right? Well I'm in a pure 2012 environment right now, so I have no such worries. Good bye 8.3 filenames. Change this registry value to 1 to disable short file name creation on all volumes: HKLM\System\CurrentControlSet\Control\FileSystem\NtfsDisable8dot3NameCreation.

[Windows]: IrpStackSize should have the recommended value.

Irp stands for I/O Request Packet. Mark Russinovich did a great job of explaining IRPs and their role in the Windows I/O system in his book Windows Internals. All the various components or layers that an I/O packet traverses on its way to and from a disk, for example, are collectively referred to as "a stack." Each filter driver you add to the file system means that the IrpStackSize needs to be increased in order to accommodate it. A common example of this is when you install an antivirus product that uses a file system filter driver. If your IrpStackSize is set too small, certain operations might fail, such as attempting to access that machine's file system remotely. Conversely, it doesn't need to be set too high, either.  It was at 11 by default on my 2012 systems. The Best Practices Analyzer says it should be at 15, so I'll set it to 15. HKLM\System\CurrentControlSet\Services\LanmanServer\Parameters\IRPStackSize.

Alright, that's where I'll leave off for part 1. Part 2 will focus on some more network-specific optimizations, so check back soon!

Built-in NIC teaming was one of my personally most anticipated features of Server 2012.  NIC teaming, whether for redundancy or for more bandwidth, has always been a cool concept and one of the foundations of highly-available systems, but it has historically required 3rd party vendor software to enable.  Probably the most popular example I can think of is the HP Network Configuration Utility software:

Almost every IT pro is going to be familiar with that screen.  Up until now, to team network adapters, one had to use vendor software such as the HP software pictured above. But starting with Windows Server 2012, the ability is built right in to the operating system, bringing the feature to new sets of hardware and without the need for any 3rd party vendor drivers or software! (Also of note is that Microsoft supports their NIC teaming, whereas they do not support the HP Network Configuration Utility.)

You can use the graphical Server Manager to configure NIC teams, but you can also do it all right from within Powershell. And since I typically prefer to keep my servers in straight-up Server Core mode, I wanted to figure out how to do it all from Powershell. My test machine for this experiment is a SuperMicro SYS-5015A-H 1U. It has two embedded GbE adapters (Realtek based.) Before Server 2012, I always just kept one of the NICs disabled since I had no use for it, and no teaming software. But now, I've installed a fresh copy of Windows Server 2012 Standard edition on it. 

*Get-NetAdapter*

To make a team out of these two network adapters, simply do

New-NetLbfoTeam -Name Team -TeamMembers Ethernet,"Ethernet 2"

That's it! (Just put quotes around 'Ethernet 2' because it contains a space.) Now keep in mind that you'll probably have to re-do the IP configuration for your new NIC team now, so you'll want physical or DRAC/ILO access to the machine so you can do that. (Or do it via script. I set the IP configuration on my new NIC team via sconfig.) Here is what the new team looks like in Powershell: 

*Get-NetLbfoTeam*

The TeamingMode and LoadBalancingAlgorithm default to SwitchIndependent and TransportPorts, respectively, but of course can be configured to whatever you want as you create the team with the New-NetLbfoTeam command. Check this Technet article for explanations on the different options and what they do. If you later want to add another NIC to the existing team, you can use the Add-NetLbfoTeamMember command and specify the NIC you want to add.

*Get-NetLbfoTeamMember*

Beautiful.

It's 2012, and ISPs are still slow to adopt IPv6. It seems like very few of us can say that we have globally-accessible IPv6 addresses. Myotherpcisacloud.com doesn't even have an IPv6 address yet... and that makes me a very sad panda. But there is something I can do about it right now, without the help of my ISP.

If you have a Cisco router, I can show you how to create an IPv6 tunnel that will you have dual-stacked and on the IPv6 Internet in no time! This article assumes that you cannot use native IPv6 out to the Internet, and that you already have the router properly set up and in use in an IPv4 network.

The router I will use in this example is a 2621XM; I bought it for $150 on eBay. It has two FastEthernet ports. It was manufactured in 1999. So any model at least as recent as that should be able to handle this just fine. I do IPv4 NAT between the two FE ports so that the rest of my home network served by my AT&T U-Verse Residential Gateway stays separate from my lab network, but the lab still has to go through the U-Verse gateway to reach the Internet. (U-Verse claims that they'll push an IPv6 firmware upgrade out automatically to all their customers sometime in 2012, but I'll believe it when I see it.)

*There's still some juice left in this crusty old thing*

For this to work for me, I needed to configure my U-Verse Gateway to put my Cisco router in "DMZ+" mode, and allow the outside interface of my Cisco router to receive a DHCP address. This allows my U-Verse gateway to assign my router the same public IPv4 address as itself, and forward all unspecified traffic to it.

We’re going to utilize the free service at Hurricane Electric for this. Follow that link and sign up. It’s their "Tunnel Broker" service that you’re after. After a short quiz, they will give you your very own IPv6 tunnel and your very own IPv6 address space! For free!

All you need to do now is configure your router. If you've never used Cisco IOS, these commands might look weird to you. They're shorthand for things like "enable" - enter "enable" mode which allows us privileged access so that we can make configuration changes to the router. "conf t" is shorthand for "configure terminal" - meaning "I wish to make configuration changes to this router from the terminal."

Router>en
Router#conf t
Router(config)#ipv6 unicast-routing
Router(config)#exit
Router#copy run start

At this point you have enabled ipv6 routing globally on your router. "Copy run start" is shorthand for "copy the running configuration to the startup configuration, effectively making these changes permanent."

Next, create a tunnel on your router like this:

Router>en
Router#conf t
interface Tunnel0
description Hurricane Electric IPv6 Tunnel Broker
no ip address
ipv6 enable
ipv6 address 2001:470:1f0e:5a4::2/64 (Use your side of the endpoint that Hurricane electric gave you!)
tunnel source 75.32.98.76 (Your public IPv4 address)
tunnel destination 216.218.224.42 (Hurricane Electric’s IPv4 endpoint for this tunnel)
tunnel mode ipv6ip
ipv6 route ::/0 Tunnel0
end
write

And you’re pretty much done! Configure your clients with an IPv6 address in that space, and you now have IPv6 connectivity all the way to the Internet. Google has a public DNS server at 2001:4860:4860::8888. Test out your tunnel by trying to ping that address. Remember that IPv6 and IPv4 are quite different. There is no NAT in IPv6. (Let's not talk about NAT64 yet.) Internet communication is the way it was truly meant to be – end to end. That also means the need to protect yourself with firewalls will become more important than ever, since you can’t hide behind a NAT anymore!

Now you can surf the web with a “dual-stack,” meaning that you’re runnnig both IPv4 and IPv6 — and your IPv4 packets will take their normal route, while your IPv6 packets will be diverted through your new tunnel. Seamlessly. Pretty neat huh? Try to ping ipv6.google.com and see what happens!

I guess that’ll have to do until ISPs catch up with IPv6 technology.