Powershell Dynamic Arrays Will Murder You, Also... A Pretty Picture! (Part 1 of X)

by Ryan 5. March 2014 20:31

I was browsing the web a couple days ago and I saw this post, which I thought was a fun idea.  I didn't want to look at his code though, because, kind of like reading movie spoilers, I wanted to see if I could do something similar myself first before spoiling the fun of figuring it out on my own. The idea is that you create an image that contains every RGB color, using each color only once.

Plus I decided that I wanted to do it in Powershell, because I'm a masochist.

There are 256 x 256 x 256 RGB colors (in a 24-bit spectrum.) That equals 16777216 colors. Since each color will be used only once, if I only use 1 pixel per color, a 4096 x 4096 image would be capable of containing exactly 16777216 different colors.

First I thought to just generate a random color, draw the pixel in that color, then add it to a list of "already used" colors, and then on the next iteration, just keep generating random colors in a loop until I happen upon one that wasn't in my list of already used colors. But I quickly realized this would be horribly inefficient and slow.  Imagine: to generate that last pixel, I'd be running through the loop all day hoping for the 1 in ~16.7 million chance that I got the last remaining color that hadn't already been used. Awful idea.

So instead let's just generate a big fat non-random array of all 16777216 RGB colors:

$AllRGBCombinations = @()
For ([Int]$Red = 0; $Red -LT 256; $Red++)
{
    For ([Int]$Green = 0; $Green -LT 256; $Green++)
    {
        For ([Int]$Blue = 0; $Blue -LT 256; $Blue++)
        {
            $AllRGBCombinations += [System.Drawing.Color]::FromArgb($Red, $Green, $Blue)
        }
    }
}

That does generate an array of 16777216 differently-colored and neatly-ordered pixel objects... but guess how long it takes?

*... the following day...*

Well, I wish I could tell you, but I killed the script after it ran for about 20 hours. I put a progress bar in just to check that it wasn't hung or in an endless loop, and it wasn't... the code is just really that slow.  It starts out at a decent pace and then gradually slows to a crawl.

Ugh, those dynamic arrays and the += operator strike again. I suspect it's because the above method recreates and resizes the array every iteration... like good ole' ReDim back in the VBscript days.  It may be handy for small bits of data, but if you're dealing with large amounts of data, that you want processed this decade, you better strongly type your stuff and use lists.  Let's try the above code another way:

$AllRGBCombinations = New-Object 'System.Collections.ObjectModel.Collection[System.Drawing.Color]'
For ([Int]$Red = 0; $Red -LT 256; $Red++)
{
    For ([Int]$Green = 0; $Green -LT 256; $Green++)
    {        
        For ([Int]$Blue = 0; $Blue -LT 256; $Blue++)
        {
            $AllRGBCombinations.Add([System.Drawing.Color]::FromArgb($Red, $Green, $Blue))
        }
    }
    $PixelsGenerated += 65536
    Write-Progress -Activity "Generating Pixels..." -Status "$PixelsGenerated / 16777216" -PercentComplete (($PixelsGenerated / 16777216) * 100)
}

Only 5.2 seconds in comparison, including the added overhead of writing the progress bar. Notice how I only update the progress bar once every 256 * 256 pixels, because it will slow you down a lot if you try to update the progress bar after every single pixel is created.

Now I can go ahead and generate an image containing exactly one of every color that looks like this:

Yep, there really are 16.7 million different colors in there, which is why even a shrunken PNG of it is 384KB.  Hard to compress an image when there are NO identical colors! The original 4096x4096 bitmap is ~36MB.  And I ended up loading a shrunken and compressed JPG for this blog post, because I didn't want every page hit consuming a meg of bandwidth.

It kinda' makes you think about how limited and narrow a human's vision is, doesn't it?  24-bit color seems to be fine for us when watching movies or playing video games, but that image doesn't seem to capture how impressive our breadth of vision should be.

Next time, we try to randomize our set of pixels a little, and try to make a prettier, less computerized-looking picture... but still only using each color only once.  See you soon.

Verifying RPC Network Connectivity Like A Boss

by Ryan 16. February 2014 10:02

Aloha.  I did some fun Powershelling yesterday and now it's time to share.

If you work in an IT environment that's of any significant size, chances are you have firewalls.  Maybe lots and lots of firewalls. RPC can be a particularly difficult network protocol to work with when it comes to making sure all the ports necessary for its operation are open on your firewalls. I've found that firewall guys sometimes have a hard time allowing the application guy's RPC traffic through their firewalls because of its dynamic nature. Sometimes the application guys don't really know how RPC works, so they don't really know what to ask of the firewall guys.  And to make it even worse, RPC errors can be hard to diagnose.  For instance, the classic RPC error 1722 (0x6BA) - "The RPC server is unavailable" sounds like a network problem at first, but can actually mean access denied, or DNS resolution failure, etc.

MSRPC, or Microsoft Remote Procedure Call, is Microsoft's implementation of DCE (Distributed Computing Environment) RPC. It's been around a long time and is pervasive in an environment containing Windows computers. Tons of Windows applications and components depend on it.

A very brief summary of how the protocol works: There is an "endpoint mapper" that runs on TCP port 135. You can bind to that port on a remote computer anonymously and enumerate all the various RPC services available on that computer.  The services may be using named pipes or TCP/IP.  Named pipes will use port 445.  The services that are using TCP are each dynamically allocated their own TCP ports, which are drawn from a pool of port numbers. This pool of port numbers is by default 1024-5000 on XP/2003 and below, and 49152-65535 on Vista/2008 and above. (The ephemeral port range.) You can customize that port range that RPC will use if you wish, like so:

reg add HKLM\SOFTWARE\Microsoft\Rpc\Internet /v Ports /t REG_MULTI_SZ /f /d 8000-9000
reg add HKLM\SOFTWARE\Microsoft\Rpc\Internet /v PortsInternetAvailable /t REG_SZ /f /d Y
reg add HKLM\SOFTWARE\Microsoft\Rpc\Internet /v UseInternetPorts /t REG_SZ /f /d Y

And/Or

netsh int ipv4 set dynamicport tcp start=8000 num=1001
netsh int ipv4 set dynamicport udp start=8000 num=1001
netsh int ipv6 set dynamicport tcp start=8000 num=1001
netsh int ipv6 set dynamicport udp start=8000 num=1001

This is why we have to query the endpoint mapper first, because we can't just guess exactly which port we need to connect to for a particular service.

So, I wrote a little something in Powershell that will test the network connectivity of a remote machine for RPC, by querying the endpoint mapper, and then querying each port that the endpoint mapper tells me that it's currently using.


#Requires -Version 3
Function Test-RPC
{
    [CmdletBinding(SupportsShouldProcess=$True)]
    Param([Parameter(ValueFromPipeline=$True)][String[]]$ComputerName = 'localhost')
    BEGIN
    {
        Set-StrictMode -Version Latest
        $PInvokeCode = @'
        using System;
        using System.Collections.Generic;
        using System.Runtime.InteropServices;

        public class Rpc
        {
            // I found this crud in RpcDce.h

            [DllImport("Rpcrt4.dll", CharSet = CharSet.Auto)]
            public static extern int RpcBindingFromStringBinding(string StringBinding, out IntPtr Binding);

            [DllImport("Rpcrt4.dll")]
            public static extern int RpcBindingFree(ref IntPtr Binding);

            [DllImport("Rpcrt4.dll", CharSet = CharSet.Auto)]
            public static extern int RpcMgmtEpEltInqBegin(IntPtr EpBinding,
                                                    int InquiryType, // 0x00000000 = RPC_C_EP_ALL_ELTS
                                                    int IfId,
                                                    int VersOption,
                                                    string ObjectUuid,
                                                    out IntPtr InquiryContext);

            [DllImport("Rpcrt4.dll", CharSet = CharSet.Auto)]
            public static extern int RpcMgmtEpEltInqNext(IntPtr InquiryContext,
                                                    out RPC_IF_ID IfId,
                                                    out IntPtr Binding,
                                                    out Guid ObjectUuid,
                                                    out IntPtr Annotation);

            [DllImport("Rpcrt4.dll", CharSet = CharSet.Auto)]
            public static extern int RpcBindingToStringBinding(IntPtr Binding, out IntPtr StringBinding);

            public struct RPC_IF_ID
            {
                public Guid Uuid;
                public ushort VersMajor;
                public ushort VersMinor;
            }

            public static List QueryEPM(string host)
            {
                List ports = new List();
                int retCode = 0; // RPC_S_OK                
                IntPtr bindingHandle = IntPtr.Zero;
                IntPtr inquiryContext = IntPtr.Zero;                
                IntPtr elementBindingHandle = IntPtr.Zero;
                RPC_IF_ID elementIfId;
                Guid elementUuid;
                IntPtr elementAnnotation;

                try
                {                    
                    retCode = RpcBindingFromStringBinding("ncacn_ip_tcp:" + host, out bindingHandle);
                    if (retCode != 0)
                        throw new Exception("RpcBindingFromStringBinding: " + retCode);

                    retCode = RpcMgmtEpEltInqBegin(bindingHandle, 0, 0, 0, string.Empty, out inquiryContext);
                    if (retCode != 0)
                        throw new Exception("RpcMgmtEpEltInqBegin: " + retCode);
                    
                    do
                    {
                        IntPtr bindString = IntPtr.Zero;
                        retCode = RpcMgmtEpEltInqNext (inquiryContext, out elementIfId, out elementBindingHandle, out elementUuid, out elementAnnotation);
                        if (retCode != 0)
                            if (retCode == 1772)
                                break;

                        retCode = RpcBindingToStringBinding(elementBindingHandle, out bindString);
                        if (retCode != 0)
                            throw new Exception("RpcBindingToStringBinding: " + retCode);
                            
                        string s = Marshal.PtrToStringAuto(bindString).Trim().ToLower();
                        if(s.StartsWith("ncacn_ip_tcp:"))                        
                            ports.Add(int.Parse(s.Split('[')[1].Split(']')[0]));
                        
                        RpcBindingFree(ref elementBindingHandle);
                        
                    }
                    while (retCode != 1772); // RPC_X_NO_MORE_ENTRIES

                }
                catch(Exception ex)
                {
                    Console.WriteLine(ex);
                    return ports;
                }
                finally
                {
                    RpcBindingFree(ref bindingHandle);
                }
                
                return ports;
            }
        }
'@
    }
    PROCESS
    {
        ForEach($Computer In $ComputerName)
        {
            If($PSCmdlet.ShouldProcess($Computer))
            {
                [Bool]$EPMOpen = $False
                $Socket = New-Object Net.Sockets.TcpClient
                
                Try
                {                    
                    $Socket.Connect($Computer, 135)
                    If ($Socket.Connected)
                    {
                        $EPMOpen = $True
                    }
                    $Socket.Close()                    
                }
                Catch
                {
                    $Socket.Dispose()
                }
                
                If ($EPMOpen)
                {
                    Add-Type $PInvokeCode
                    $RPCPorts = [Rpc]::QueryEPM($Computer)
                    [Bool]$AllPortsOpen = $True
                    Foreach ($Port In $RPCPorts)
                    {
                        $Socket = New-Object Net.Sockets.TcpClient
                        Try
                        {
                            $Socket.Connect($Computer, $Port)
                            If (!$Socket.Connected)
                            {
                                $AllPortsOpen = $False
                            }
                            $Socket.Close()
                        }
                        Catch
                        {
                            $AllPortsOpen = $False
                            $Socket.Dispose()
                        }
                    }

                    [PSObject]@{'ComputerName' = $Computer; 'EndPointMapperOpen' = $EPMOpen; 'RPCPortsInUse' = $RPCPorts; 'AllRPCPortsOpen' = $AllPortsOpen}
                }
                Else
                {
                    [PSObject]@{'ComputerName' = $Computer; 'EndPointMapperOpen' = $EPMOpen}
                }
            }
        }
    }
    END
    {

    }
}

And the output will look a little something like this:


You can also query the endpoint mapper with PortQry.exe -n server01 -e 135, but I was curious about how it worked at a deeper level, so I ended up writing something myself. There weren't many examples of how to use that particular native API, so it was pretty tough.

Redirecting HTTP to HTTPS with the IIS URL Rewrite Module

by Ryan 1. February 2014 15:02

I should use TLS

Hi folks. I've been working on my HTTP web server the past couple of weeks. It's pretty good so far. It uses the managed thread pool, does asynchronous request handling, tokenization (like %CURRENTDATE%, etc.) server-side includes, SSL/TLS support, can run multiple websites simultaneously, etc. It seems really fast, but I haven't put it through any serious load tests yet.  My original goal was to use it to completely replace this blog and content management system (IIS and BlogEngine.NET,) but now I'm starting to think that won't happen. Mainly because I would need to convert years worth of content to a new format, and I would want to preserve the URLs so that if anyone on the internet had ever linked to any of my blog posts, they wouldn't all turn to 404s when I made the switch. That's actually more daunting to me than writing a server and a CMS.



So I'm taking a break from that and instead improving what I've already got here. One thing I've always wanted to do but never got around to was redirecting HTTP to HTTPS. We should be using SSL/TLS to encrypt our web traffic whenever possible. Especially when it's this simple:

First, open IIS Manager, and use Web Platform Installer to install the URL Rewrite 2.0 module:

Install URL Rewrite


Now at the server level, open URL Rewrite and add a new blank rule:

Add Blank Rule


Next you want to match (.*) regex pattern:

Edit the rule


And finally add the following condition:


Condition


And that's all there is to it!  This is of course assuming that you've got an SSL certificate bound to the website already, and that you're listening on both ports 80 and 443.

Note that you can also do this exact same thing by editing your web.config file by hand:


    
<system.WebServer>
   ...
   <rewrite>
      <globalrules>
        <rule name="Redirect to HTTPS" stopprocessing="true" enabled="true">
          <match url="(.*)"></match>
          <action type="Redirect" url="https://{HTTP_HOST}/{R:1}"></action>
          <conditions>
            <add pattern="^OFF$" input="{HTTPS}"></add>
          </conditions>
        </rule>
      </globalrules>
    </rewrite>
</system.WebServer>

And finally, a tip: Don't put resources such as images in your website that are sourced from HTTP, regardless if they're from your own server or someone else's. Use relative paths (or at least the https prefix) instead. Web browsers typically complain or even refuse to load "insecure" content on a secure page.

Tired of the NSA Seeing What Model of Plug and Play Mouse You're Using?

by Ryan 14. January 2014 13:40

Not long ago, the story broke that the NSA was capturing internet traffic generated by Windows crash dumps, driver downloads from Windows Updates, Windows Error Reporting, etc.

As per Microsoft's policy, this information, when it contains sensitive or personally identifiable data, is encrypted.

Encryption: All report data that could include personally identifiable information is encrypted (HTTPS) during transmission. The software "parameters" information, which includes such information as the application name and version, module name and version, and exception code, is not encrypted.

While I'm not saying that SSL/TLS poses an impenetrable obstacle for the likes of the NSA, I am saying that Microsoft is not just sending full memory dumps across the internet in clear text every time something crashes on your machine.  But if you were to, for instance, plug in a new Logitech USB mouse, your computer very well could try to download drivers for it from Windows Update automatically, and when that happens, it sends a few details about your PC and the device you just plugged in, in clear text.

Here is where you can read more about that.

So let's say you're an enterprise administrator, and you want to put an end to all this nonsense for all the computers in your organization, such that your computers no longer attempt to contact Microsoft or send data to them when an application crashes or someone installs a new device.  Aside from setting up your own internal corporate Windows Error Reporting server, (who does that?) you can disable the behavior via Group Policy. There are a surprising number of policy settings that should be disabled so that you're not leaking data all over the web:

  • The system will be configured to prevent automatic forwarding of error information.

Configure the policy value for Computer Configuration -> Administrative Templates -> System -> Internet Communication Management -> Internet Communication settings-> “Turn off Windows Error Reporting” to “Enabled”.

  • An Error Report will not be sent when a generic device driver is installed.

Configure the policy value for Computer Configuration -> Administrative Templates -> System -> Device Installation -> "Do not send a Windows error report when a generic driver is installed on a device" to "Enabled".

  • Additional data requests in response to Error Reporting will be declined.

Configure the policy value for Computer Configuration -> Administrative Templates -> Windows Components -> Windows Error Reporting -> "Do not send additional data" to "Enabled".

  • Errors in handwriting recognition on Tablet PCs will not be reported to Microsoft.

Configure the policy value for Computer Configuration -> Administrative Templates -> System -> Internet Communication Management -> Internet Communications settings “Turn off handwriting recognition error reporting” to “Enabled”.

  • Windows Error Reporting to Microsoft will be disabled.

Configure the policy value for Computer Configuration -> Administrative Templates -> Windows Components -> Windows Error Reporting “Disable Windows Error Reporting” to “Enabled”.

  • Windows will be prevented from sending an error report when a device driver requests additional software during installation.

Configure the policy value for Computer Configuration -> Administrative Templates -> System -> Device Installation -> “Prevent Windows from sending an error report when a device driver requests additional software during installation” to “Enabled”.

  • Microsoft Support Diagnostic Tool (MSDT) interactive communication with Microsoft will be prevented.

Configure the policy value for Computer Configuration -> Administrative Templates -> System -> Troubleshooting and Diagnostics -> Microsoft Support Diagnostic Tool -> “Microsoft Support Diagnostic Tool: Turn on MSDT interactive communication with Support Provider” to “Disabled”.

  • Access to Windows Online Troubleshooting Service (WOTS) will be prevented.

Configure the policy value for Computer Configuration -> Administrative Templates -> System -> Troubleshooting and Diagnostics -> Scripted Diagnostics -> “Troubleshooting: Allow users to access online troubleshooting content on Microsoft servers from the Troubleshooting Control Panel (via Windows Online Troubleshooting Service - WOTS)” to “Disabled”.

  • Responsiveness events will be prevented from being aggregated and sent to Microsoft.

Configure the policy value for Computer Configuration -> Administrative Templates -> System -> Troubleshooting and Diagnostics -> Windows Performance PerfTrack -> “Enable/Disable PerfTrack” to “Disabled”.

  • The Application Compatibility Program Inventory will be prevented from collecting data and sending the information to Microsoft.

Configure the policy value for Computer Configuration -> Administrative Templates -> Windows Components -> Application Compatibility -> “Turn off Program Inventory” to “Enabled”.

  • Device driver searches using Windows Update will be prevented.

Configure the policy value for Computer Configuration -> Administrative Templates -> System -> Device Installation -> “Specify Search Order for device driver source locations” to “Enabled: Do not search Windows Update”.

  • Device metadata retrieval from the Internet will be prevented.

Configure the policy value for Computer Configuration -> Administrative Templates -> System -> Device Installation -> “Prevent device metadata retrieval from internet” to “Enabled”.

  • Windows Update will be prevented from searching for point and print drivers.

Configure the policy value for Computer Configuration -> Administrative Templates -> Printers -> “Extend Point and Print connection to search Windows Update” to “Disabled”.

Website Upgrade, Coding, and Dealing with NTFS ACLs on Server Core

by Ryan 9. January 2014 15:20

I apologize in advance - this blog post is going to be all over the place.  I haven't posted in a while, mainly because I've been engrossed in a personal programming project. Part of which includes a multithreaded web server I wrote over the weekend that I'm kind of proud of. My ShareDiscreetlyWebServer is single-threaded, because when I wrote it, I had not yet grasped the awesome power of the async and await keywords in C#.  They're très sexy. Right now the only verb I support is GET (because it's all I need for now,) but it's about as fast as you could hope a server written in managed code could be.

Secondly, I just upgraded this site to Blogengine.NET v2.9.  The motivation behind it was that today, I got this email from a visitor to the site:

Hi,
I tried to leave you a comment but it didnt work.
Can you please go through steps you took to migrate your blog to Azure as I am interested in doing the same thing.
Did you set it up as a Azure Web Site or use an Azure VM and deployed it that way?
Are you using BlogEngine or some other blog publishing tool.

Wait, my comments weren't working? Damnit. I tried to post a comment myself and sure enough, commenting on this blog was busted. It was working fine but it just stopped working some time in the last few weeks. And so, I figured that if I was going to go to the trouble of debugging it, I'd just go ahead and upgrade Blogengine.NET while I was at it.

But first, to answer the guy's question above, my blog migration was simple. I used to host this blog out of my house on a Windows Server running in my home office. I signed up for a Server 2012 Azure virtual machine, RDP'ed to it, installed the IIS role, robocopy'd my entire C:\inetpub directory to the new VM, and that was that.

So version 2.9 so far is a little lackluster so far.  They updated most of the UI to the simplistic, sleek "modern" look that's all the rage these days, especially on tablets and phones.  But in the process it appears they've regressed to the point where the editor is no longer compatible with Internet Explorer 11, 10, or 9. (Not that it worked perfectly before either.)  It's annoying as hell. I'm writing this post right now in IE with compatibility mode turned on, and still half of the buttons don't work.  It's crippled compared to the version 2.8 that I was on this morning.

That's ironic that the developers who wrote a CMS entirely in .NET, in Visual Studio, couldn't be bothered to test it on any version of IE.  Guess I'll wait patiently for version 3.0.  Or maybe write my own CMS after I get finished writing the web server to run it on.

But even after the upgrade, and after fixing all the little miscellaneous bugs that the upgrade introduced, it still didn't fix my busted comment system. So I had to dig deeper. I logged on to the server, fired up Process Monitor while I attempted to post a comment: 

w3wp.exe gets an Access Denied error right there, clear as day.  (Thanks again, ProcMon.)

If you open the properties of w3wp.exe, you'll notice that it runs in the security context of an application pool, e.g. "IIS APPPOOL\Default Web Site". So just give that security principal access to that App_Data directory.  Only one problem...

Server Core.

No right-clicking our way out of this one.  Of course we could have done this with cacls.exe or something, but you know I'm all about the Powershell.  So let's do it in PS.

$Acl = Get-Acl C:\inetpub\wwwroot\App_Data
$Ace = New-Object System.Security.AccessControl.FileSystemAccessRule("IIS APPPOOL\Default Web Site", "FullControl", "ContainerInherit, ObjectInherit", "None", "Allow")
$Acl.AddAccessRule($Ace)
Set-Acl C:\inetpub\wwwroot\App_Data $Acl

Permissions, and commenting, are back to normal.

About Me

Ryan Ries
Texas, USA
Systems Engineer
ryan@myotherpcisacloud.com

I am a systems engineer with a focus on Microsoft tech, but I can run with pretty much any system that uses electricity.  I'm all about getting closer to the cutting edge of technology while using the right tool for the job.

This blog is about exploring IT and documenting the journey.


Blog Posts (or Vids) You Must Read (or See):

Pushing the Limits of Windows by Mark Russinovich
Mysteries of Windows Memory Management by Mark Russinovich
Accelerating Your IT Career by Ned Pyle
Post-Graduate AD Studies by Ned Pyle
MCM: Active Directory Series by PFE Platforms Team
Encodings And Character Sets by David C. Zentgraf
Active Directory Maximum Limits by Microsoft
How Kerberos Works in AD by Microsoft
How Active Directory Replication Topology Works by Microsoft
Hardcore Debugging by Andrew Richards
The NIST Definition of Cloud by NIST



MCITP: Enterprise Administrator

VCP5-DCV

Profile for Ryan Ries at Server Fault, Q&A for system administrators

LOPSA

GitHub: github.com/ryanries

 

I do not discuss my employers on this blog and all opinions expressed are mine and do not reflect the opinions of my employers.