I learned something today. It's something that made me feel stupid for not knowing. Something that seemed elemental and trivial - yet, I did not know it. So please, allow me to relay my somewhat embarrassing learning experience in the hopes that it will save someone else from the same embarrassment.

I did know what DNS round robin was. Or at least, I would have said that I did.

Imagine you configure DNS1, as a DNS server, to use round robin. Then, you create 3 host (A or AAAA) records for the same host name, using different IPs. Let's say we create the following A records on DNS1:

server01 - A 10.0.0.4
server01 - A 10.0.0.5
server01 - A 10.0.0.6

Then on a workstation which is configured to use DNS1 as a DNS server, you ping server01. You receive 10.0.0.4 as a reply. You ping server01 again. With no hesitation, you get a reply from 10.0.0.4 again. We assume that your local workstation has cached 10.0.0.4 locally and will reuse that IP for server01 until the entry either expires, or we flush the DNS cache on the workstation with a command like ipconfig/flushdns.

I run ipconfig/flushdns. Then I ping server01 again.

This time I receive a response from 10.0.0.5. Now I assume DNS round robin is working perfectly. I go home for the day feeling like I know everything there is to know about DNS.

But was it that the DNS server is responding to DNS queries with the single next A/AAAA record that it has on file, in a round-robin type sequential fashion to every DNS query that it receives? That is what I assumed.

But the fact of the matter is that DNS servers, when queried for a host name, actually return a list of all A/AAAA records associated with that host name, every time that host name is queried for. (To a point - the list must fit within a UDP packet, and some firewalls/filters don't let UDP packets longer than 512 bytes through. That's changing though. Our idea of how big data is and should be allowed to be is always growing.)

I assume that www.google.com, being one of the busiest websites in the world, has not only some global load balancing and other advanced load balancing techniques employed, but probably also has more than one host record associated with it. To test my theory, I fire up Wireshark and start a packet capture. I then flush my local DNS cache with ipconfig/flushdns and then ping www.google.com.

Notice how I pinged it, got one IP address in response (.148), then flushed my DNS cache, pinged it again and got another different IP address (.144)? But despite what it may look like, that name server is not returning just one A/AAAA record each time I query it:

*Click for Larger*

My workstation is ::9. My workstation's DNS server is ::1. The DNS server is configured to forward DNS requests for zones for which it is not authoritative on to yet another DNS server. So I ask for www.google.com, my DNS server doesn't know, so it forwards the request. The forwardee finally finds out and reports back to my DNS server, which in turn relays back to me a list of all the A records for www.google.com. I get a long list containing not only a mess of A records, but a CNAME thrown in there too, all from a single DNS query! (We're not worried about the subsequent query made for an AAAA record right now. Another post perhaps.)

I was able to replicate this same behavior in a sanitary lab environment running a Windows DNS server and confirmed the same behavior. (Using the server01 example I mentioned earlier.)

Where round robin comes in is that it rotates the order of the list given to each subsequent client who requests it. Keep in mind that while round robin-ing the A records in your DNS replies does supply a primitive form of load distribution, it's a pretty poor substitute for real load balancing, since if one of the nodes in the list goes down, the DNS server will be none the wiser and will continue handing out the list with the downed node's IP address on it.

Lastly, since we know that our client is receiving an entire list of A records for host names which have many IP addresses, what does it actually do with the list?  Well, the ping utility doesn't do much. If the first IP address on the list is down, you get a destination unreachable message and that's it. (Leading to a lot of people not realizing they have a whole list of IPs they could try.) Web browsers however, have a nifty feature known as "browser retry" or "client retry," where they will continue trying the other IPs in the list until they find a working one. Then they will cache the working IP address so that the user does not continue to experience the same delay in web page loading as they did the first time. Yes, there are exploits concerning this feature, and yes it's probably a bad idea to rely on this since browser retry is implemented differently across every different browser and operating system. It's a relatively new mechanism actually, and people may not believe you if you tell them. To prove it to them, find (or create) a host name which has several bad IPs and one or two good ones. Now telnet to that hostname. Even telnet (a modern version from a modern operating system) will use getaddrinfo() instead of gethostbyname() and if it fails to connect the first IP, you can watch it continue trying the next IPs in the list.

More info here, here and here. That last link is an MSDN doc on getaddrinfo(). Notice that it does talk about different implementations on different operating systems, and that ppResult is "a pointer to a linked list of one or more addrinfo structures that contains response information about the host."

I am absolutely a Windows engineer and an extremely avid advocate of most everything Microsoft, but more importantly I'm an enthusiast of all forms of technology that help to achieve business goals. Whatever it takes to further the state of the art. That means I occasionally enjoy dabbling in Linux too. Whatever gets me closer to the bleeding edge of technology. Not to mention that the vast majority of enterprises have some sort of mixture of both operating systems.

But it's rare to see a deployment in which the Unix/Linux servers participate in Active Directory. Yes, Active Directory is a Microsoft technology and *nix isn't just ready to jump into domain membership right out of the box, but I strongly believe that AD is the mortar that glues any corporate IT environment together. Let us not think Linux vs. Windows... but Linux and Windows!

So what are the ways *nix could benefit from Active Directory?

• Secure, central management:
  No more maintaining a separate list of local user accounts and passwords on each and every machine. Why not keep just one database of users and machines in your Active Directory that is guaranteed to stay consistent and secure among every single member server forever?
• Authentication:
  The main mode of authentication in an Active Directory domain is Kerberos. It was invented by some nerds at MIT. Kerberos is Greek for the three-headed hound that guards the gates of hell. (Cerberus in Latin.) This name is apt, because Kerberos is an authentication system that requires three parties. This authentication system involving a "trusted third party" has proven to be secure and trustworthy in any enterprise environment. And the best part? Kerberos is an open protocol that both Microsoft and *nix can both enjoy.
• As if that wasn't enough:
  Authenticate from machine to machine to machine, without having to re-type your password; without any user intervention at all even! Use one account to run a service on every machine. Active Directory-integrated machines can securely and dynamically update their own DNS records. Log on to a freshly-built machine with domain credentials, without ever needing to manage the local accounts on each and every box. The list goes on and on...

As any IT company grows, it becomes increasingly important that they maintain a cohesive, easily manageable structure that includes all of their devices. So, to that end, I took the time to replicate in my personal lab the steps necessary to join a Linux machine to my existing Windows Active Directory domain. And I've documented the journey. So without further ado:

As you can see, I've created a virtual machine and installed Linux on it. My domain is at the 2008 R2 forest and domain functional levels. It's pretty much the best domain ever. I'd put my AD architecting skills artistry up against anyone's.

Here I am on said virtual machine, downloading the Likewise (free edition) client. I was planning on doing it all the long, complex, hard way. This software saved me a lot of time.

I created a basic user, and delegated domain-joining permissions to him, but nothing else. I'm going to use this service account for the sole purpose of joining *nix machines to my domain.

Here's where the hair on the back of the neck of any real nerd would start standing up. See what I did there? I just joined my Linux machine to my Active Directory domain, using my specified service account. "SUCCESS" it says. I shall stand for nothing less.

Now we rush off to look at the security log on our domain controller. And what else do we see there but zero audit failures, and a handful of beautiful Kerberos ticket requests and grants. The machine account even popped up in my AD Users & Computers!

And finally - the one screenshot to rule them all - here I am SSH'ing into my Linux box for the first time using domain creds! Kerberos wins the day.

So, that's all I've got for now. I haven't really done any more in-depth research into this than what you've just seen. You're probably already wondering if I can make it do smartcards next, aren't you?