New-DepthGaugeFile.ps1: The Powershell Pipeline Is Neat, But It's Also Slow

by Ryan 20. December 2014 14:12

If you know me or this blog at all, you know that first and foremost, I think Powershell is awesome.  It is essential to any Windows system administrator or I.T. pro's success.  The good news is that there are a dozen ways to accomplish any given task in Powershell.  The bad news is that eleven of those twelve techniques are typically as slow as a three-legged tortoise swimming through a vat of cold Aunt Jemima.

This is not the first, or the second, blog post I've made about Powershell performance pitfalls.  One of the fundamental problems with super-high-level languages such as Powershell, C#, Java, etc., is that they take raw performance away from you and give you abstractions in return, and you then have to fight the language in order to get your performance back.

Today I ran across another such example.

I'm creating a program that reads from files.  I need to generate a file that has "markers" in it that tell me where I am within that file at a glance.  I'll call this a "depth gauge."  I figured Powershell would be a simple way to create such a file.  Here is an example of what I'm talking about:

Depth Gauge or Yardstick File


The idea being that I'd be able to tell my program "show me what's at byte 0xFFFF of file.txt," and I'd be able to easily visually verify the answer because of the byte markers in the text file.  The random characters after the byte markers are just gibberish to take up space.  In the above example, each line takes up exactly 64 bytes - 62 printable characters plus \r\n.  (In ASCII.)

I reach for Powershell whenever I want to whip up something in 5 minutes that accomplishes a simple task.  And voila:

Function New-DepthGaugeFile
{
    [CmdletBinding()]
    Param([Parameter(Mandatory=$True)]
          [ValidateScript({Test-Path $_ -IsValid})]
          [String]$FilePath, 
          [Parameter(Mandatory=$True)]
          [Int64]$DesiredSize, 
          [Parameter(Mandatory=$True)]
          [ValidateRange(20, [Int32]::MaxValue)]
          [Int32]$BytesPerLine)
    Set-StrictMode -Version Latest
    [Int64]$CurrentPosition = 0
    $ValidChars = @('a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z',
                    'A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','W','X','Y','Z',
                    '0','1','2','3','4','5','6','7','8','9',
                    '!','@','#','$','%','^','&','*','(',')','_','+','-','=','?','<','>','.','[',']',';','`','{','}','\','/')
    
    Try
    {
        New-Item -Path $FilePath -ItemType File -Force -ErrorAction Stop | Out-Null
    }
    Catch
    {
        Write-Error $_
        Return
    }    
    
    [Int32]$BufferMaxLines = 64
    [Int32]$BufferMaxBytes = $BufferMaxLines * $BytesPerLine

    If ($DesiredSize % $BytesPerLine -NE 0)
    {
        Write-Warning 'BytesPerLine is not a multiple of DesiredSize.'
    }


    $LineBuffer = New-Object 'System.Collections.Generic.List[System.String]'
        
    While ($DesiredSize -GT $CurrentPosition)
    {
        [System.Text.StringBuilder]$Line = New-Object System.Text.StringBuilder
        
        # 17 bytes
        [Void]$Line.Append($("{0:X16} " -F $CurrentPosition))
        
        # X bytes
        1..$($BytesPerLine - 19) | ForEach-Object { [Void]$Line.Append($(Get-Random -InputObject $ValidChars)) }        
        
        # +2 bytes (`r`n)        

        [Void]$LineBuffer.Add($Line.ToString())
        $CurrentPosition += $BytesPerLine

        # If we're getting close to the end of the file, we'll go line by line.
        If ($CurrentPosition -GE ($DesiredSize - $BufferMaxBytes))
        {
            Add-Content $FilePath -Value $LineBuffer -Encoding Ascii
            [Void]$LineBuffer.Clear()
        }

        # If the buffer's full, and we still have more than a full buffer's worth left to write, then dump the
        # full buffer into the file now.
        If (($LineBuffer.Count -GE $BufferMaxLines) -And ($CurrentPosition -LT ($DesiredSize - $BufferMaxBytes)))
        {
            Add-Content $FilePath -Value $LineBuffer -Encoding Ascii
            [Void]$LineBuffer.Clear()
        }
    }
}

Now I can create a dummy file of any size and dimension with a command like this:

New-DepthGaugeFile -FilePath 'C:\testfolder1\largefile2.log' `
                   -DesiredSize 128KB -BytesPerLine 64

I thought I was being really clever by creating an internal "buffering" system, since I instinctively knew that performing a file write operation (Add-Content) on each and every loop iteration would slow me down.  I also knew from past experience that overuse of "array arithmetic" like $Array += $Element would slow me down because of the constant cloning and resizing of the array.  I also remembered that in .NET, strongly-typed lists are faster than ArrayLists because we want to avoid boxing and unboxing.

Despite all these little optimizations, here is the performance while writing a 1MB file:

Measure-Command { New-DepthGaugeFile -FilePath C:\testfolder1\largefile.log `
                                     -DesiredSize 1MB -BytesPerLine 128 }

TotalSeconds    : 103.8428624

Over 100 seconds to generate 1 megabyte of data.  I'm running on an SSD that is capable of copying hundreds of megabytes per second of data, so the storage is not the bottleneck.

To try to speed things up, I decided to focus on the line that appears to be doing the most amount of work:

1..$($BytesPerLine - 19) | ForEach-Object { 
             [Void]$Line.Append($(Get-Random -InputObject $ValidChars)) }

The idea behind this line of code is that we add some random characters to each line.  If we want each line to take up 64 characters, then we would add (64 - 19) characters to the line, because the byte marker at the beginning of the line, plus a space character, takes up 17 bytes.  Then then the newline and carriage return takes up 2 bytes.

My first instinct was that the Get-Random action was taking all the CPU cycles.  So I replaced it with static characters... and it made virtually no difference.  Maybe it's the pipe and Foreach-Object?  Let's change it to this:

For ($X = 0; $X -LT ($BytesPerLine - 19); $X++)
{
    [Void]$Line.Append($(Get-Random -InputObject $ValidChars))
}

And now the results:

Measure-Command { New-DepthGaugeFile -FilePath C:\testfolder1\largefile.log `
                                     -DesiredSize 1MB -BytesPerLine 128 }

TotalSeconds    : 61.0464638

Exact same output, almost twice as fast.

NTHashTickler_CUDA v1.0

by Ryan 24. October 2014 11:10

I've ported my NTHashTickler program to NVIDIA CUDA... poorly.

As before, my motivation for writing this program is not because I really care about finding NT hashes per se, but just teaching myself more about multithreaded programming and in this case, NVIDIA CUDA. If you know anything at all about CUDA, please feel free to criticize my work and tell me what I'm doing wrong. (Source code is on Github.)

NTHashTickler_CUDA

NTHashTickler_CUDA

So CUDA is basically like standard C, but with a few extensions.  The challenge is that you have "host" code, and you have "device" code comingling, with host code only being able to run on your CPU and access your main memory, while "device" code can only run on your GPU and only access GPU memory.  What this means is that you are totally on your own when you're writing device code to run on your GPU. You can't even use C standard library functions like memcmp or strtol, etc.  And you definitely cannot call something as luxurious as a Windows API function from device code. The sole exception as far as I know is that CUDA does allow you to use printf from within device code for debugging purposes...

Traditionally, you could only transfer data from the host to the device through CUDA API calls such as cudaMalloc, etc., but in CUDA compute capability 3.0 they introduced what I like to call "CUDA Easy-Mode," or essentially, variables that can be seamlessly accessed by both host and device simultaneously, making your code look a lot cleaner and simpler.  These are the __device__ __managed__ variables.  They provide what CUDA calls a "unified" view of memory.  I'm guessing it's probably just the CUDA runtime doing the dirty work for you under the hood that you used to have to do yourself.  You can still cause the same kind of problems reading and writing to the variables from different threads that applies to any multithreaded code, but it's still so much easier to work with than cudaMalloc, cudaFree, etc.

So the idea is that if you have a problem that can be parallelized sufficiently, it can take advantage of your GPU's ability to run thousands of parallel threads, instead of your CPU's mere 4 or 8 simultaneous threads.

In practice though, when I finally got the code working, I get about 750,000 hashes per second of throughput, which is slower than the ~ 5 million hashes/second of throughput by my previous version in C that just ran on the CPU!

So I'm definitely doing something wrong.  I think I should be seeing a billion hashes/sec of throughput or better.  I have a lot more work to do.

Luckily, the CUDA Toolkit comes with some really amazing profiling and tracing tools, and I think that's a good place to start looking for optimization opportunities.

(Source code is on Github.)

Supersymmetry Outlook Add-In v1.1.3.17

by Ryan 4. October 2014 19:10

You can see the original Supersymmetry 1.0 post here.

The Supersymmetry Outlook Add-In has been upgraded to version 1.1.3.17.  (Similarity to "31337" or "1337" is coincidental!) But this release is significantly cooler than 1.0 anyway.

In case you're not familiar, the purpose of the Supersymmetry Outlook Add-In is to prevent you from accidentally sending email messages in Outlook if they contain an uneven or unmatched pair of quotation marks, parentheses, curly braces or square brackets.  Read more about it and see more early screenshots in the 1.0 post I linked to earlier.

Improvements in this release include:

  • A new text file named ignore.txt should be placed at %USERPROFILE%\Supersymmetry\ignore.txt.  In this text file, the user can put character sequences ("tokens") for Supersymmetry to ignore in a message.  This is very handy for ignoring things like emoticons, because emoticons usually have parentheses in them, and the use of emoticons can make the message as a whole appear as though it contains an uneven number of parentheses, when the message is actually fine if you ignore the emoticons. A sample ignore.txt is included in the download package.
  • A new text file named divider.txt should be placed at %USERPROFILE%\Supersymmetry\divider.txt.  In this text file, the user can put a character or string token that splits or divides an email thread into pieces, and Supersymmetry will only scan up to the first occurrence of that token. The token or divider can be a single character, or a special string.  This is very useful so that Supersymmetry does not include all of the previous messages in the email thread during its scan.  I recommend putting your special delimiter character or string cleverly in your email signature for both new messages and replies, so that Outlook automatically inserts the token into every message you draft.  I'll leave that up to you.  A sample divider.txt is included in the download package.
  • If either of the two aforementioned files does not exist in the right place on the file system, the add-in will still work, but you will really miss those features.  Additionally, you will see some warning notifications pop up when Outlook loads the add-in:

Supersymmetry cannot locate its files

  • Notice that the recommended installation directory has changed from %APPDATA%\Supersymmetry to %USERPROFILE%\Supersymmetry, just because it's a little simpler.  If the add-in is installed correctly, and the configuration files were read successfully, you'll see these popups instead:

Supersymmetry loaded correctly

Installation

Download the package here:
Supersymmetry-v1.1.3.17.zip (296.8KB)

Make sure to Uninstall any old version of Supersymmetry first, by going to Programs and Features in your Control Panel and uninstalling Supersymmetry.  Also, make sure you've closed Outlook.  Then, unzip the package into your user profile directory, so that %USERPROFILE%\Supersymmetry exists and contains a file named setup.exe.  Next, run setup.exe.  In theory, that should help you download any required prerequisites such as .NET 4.5, and Visual Studio Tools for Office, and then install the add-in.  It's a "ClickOnce" deployment.  A great idea, when it works.

Uninstallation

Simply go to Programs and Features (aka Add/Remove Programs) in Control Panel, find Supersymmetry, and uninstall it.

Have fun!

Supersymmetry in action!

Supersymmetry Outlook Add-In v1.0

by Ryan 25. September 2014 08:09

Update Oct. 4th, 2014: You want the updated version of this addin, v1.1.3.17!

Like millions of others, I use Outlook as an email client, especially at work.  I was drafting an email at work the other day, and after quickly proofreading it, I sent it out.  Only after sending it, of course, did I spot an error.  I had used a parenthesis to start a parenthetical clause (like this,) only I forgot to use the accompanying closing parenthesis at the end of the statement, so it came out (like this.

I realized that I do this quite a bit in my writing, particularly when I'm rapid-firing work emails.  And not just with parentheses, but also with quotation marks, and occasionally curly braces and square brackets.  There are no red squiggly underlines for this and spellcheck won't help you here.

So I wrote an Outlook 2013 Add-In that will catch me if I attempt to send an email that contains an unmatched set of quotation marks, parentheses, curly braces or square brackets.  Notice the popup when I hit the Send button:

Email draft with a mistake in it


It requires Outlook 2013, .NET 4.5, and Windows Vista or later. It should work on both 32-bit and 64-bit machines, though I didn't test it on 32-bit. You may need to install Visual Studio 2010 Tools for Office Runtime, depending on whether you already installed it when you installed Microsoft Office or not.  If you download the package, and your computer already recognizes the *.vsto file extension, then you probably already have the necessary VSTO runtime installed.  On my development machine, I had to uninstall VSTO, delete the "C:\Program Files\Common Files\Microsoft Shared\VSTO" directory, then reinstall VSTO, or else I got an error when trying to install the add-in.  However, on a fresh test machine that never had Visual Studio installed and only had MS Office installed, I did not get the error and only needed to double-click the *.vsto file and everything worked.

Installation

Download the ZIP archive here:

Supersymmetry-v1.0.zip (49.5KB)

Unpack the ZIP archive somewhere... I chose %APPDATA%\Supersymmetry because that's a good place to put per-user add-ins that doesn't require administrator privileges to write to.  Once you have unzipped the files to a directory, double-click the Supersymmetry.vsto file.

I signed the manifest using a code signing certificate that chains up to the Baltimore CyberTrust Root CA.

Publisher Has Been Verified

You may or may not have the certificate chain in your trusted CAs store.  If you would rather compile the code yourself, send me an email and I will just send you the source code.  The source code is so stupid-simple that I don't feel it deserves a Github repo.  Getting Visual Studio set up just right and figuring out the idiosyncrasies of "ClickOnce" deployment was way more involved than actually writing the code.

Uninstallation

Just go to "Programs and Features" in the Control Panel and click Supersymmetry from the list and click Uninstall.

Limitations

When you click the Send button on an email, the add-in currently scans the entire previous thread embedded with the message, not just the part that you just typed.  That means that the add-in will catch quotation mark and parentheses mistakes that other people made earlier on in the email thread, in addition to your own.  When I think of the best way to filter out these older original messages, I will add that to version 1.1.

Update Oct. 4th, 2014: You want the updated version of this addin, v1.1.3.17!

NTHashTickler_C v1.0

by Ryan 13. September 2014 17:09

Last time, I unveiled my multithreaded brute force NTLM hash cracker written in C#.  However, I was not very happy with the performance... specifically the scalability... it has some pretty massive concurrency issues that are apparent when executing on a machine with lots of processors.  For instance, the program barely ran any faster on a machine with 8 CPUs than it did on a machine with 4 relatively slower CPUs.  There is obviously a big threading issue there causing a problem with diminishing returns. 

Now don't get me wrong - I love C#, but I had a hunch that if I wanted to be serious about performance I needed to go native.  So I spent about a day porting it to standard C, and now both single-threaded and mult-threaded performance is about seven times better than in C#.  Again, I'm not bashing C#, it's just that different types of problems call for different types of tools.  And like I said last time, I realize there already several other apps out there that do this, but I've taught myself a lot of C doing this, which makes it worth it to me.

As usual, you can view the source on Github if you're interested.


About Me

Ryan Ries
Texas, USA
Systems Engineer
ryan@myotherpcisacloud.com

I am a systems engineer with a focus on Microsoft tech, but I can run with pretty much any system that uses electricity.  I'm all about getting closer to the cutting edge of technology while using the right tool for the job.

This blog is about exploring IT and documenting the journey.


Blog Posts (or Vids) You Must Read (or See):

Pushing the Limits of Windows by Mark Russinovich
Mysteries of Windows Memory Management by Mark Russinovich
Accelerating Your IT Career by Ned Pyle
Post-Graduate AD Studies by Ned Pyle
MCM: Active Directory Series by PFE Platforms Team
Encodings And Character Sets by David C. Zentgraf
Active Directory Maximum Limits by Microsoft
How Kerberos Works in AD by Microsoft
How Active Directory Replication Topology Works by Microsoft
Hardcore Debugging by Andrew Richards
The NIST Definition of Cloud by NIST



MCITP: Enterprise Administrator

VCP5-DCV

Profile for Ryan Ries at Server Fault, Q&A for system administrators

LOPSA

GitHub: github.com/ryanries

 

I do not discuss my employers on this blog and all opinions expressed are mine and do not reflect the opinions of my employers.