Tag Archives: Export-Clixml

Part V: Splunk, HEC, Indexer Acknowledgement, and PowerShell

In the last four parts of this series (Part I, Part II, Part III, Part IV),  we discussed sending telemetry data to Splunk using HEC (HTTP Event Collector). This requires no software to be installed. We can send data for ingestion to Splunk using REST and a REST endpoint. In the previous four parts, we’ve included Indexer Acknowledgement. This set up uses a random GUID we create and send to Splunk in our initial connection using Invoke-RestMethod.

Then, in additional requests we use the GUID to continually poll Splunk to determine if the data has been more than just received, but that it’s being processed. There was much too much delay for me to consider its use and so I disabled Indexer Acknowledgment. In this post, we’ll take our final code for Part IV, below, and remove all the parts that were in place for Indexer Acknowledgment. This greatly reduces the amount of code and overall complexity seen in the previous parts of this series. Compare the two code examples below as we wrap up the series. Hopefully, if you were looking for ways to send data to Splunk using PowerShell that you found this series of articles and in time that they were helpful for you. If you’ve got Splunk available to you, then don’t forget that you have a place where data can be sent and stored after it’s been collected with PowerShell. And it’s not just about data storage. In fact, that’s such a small portion of the benefits to Splunk. If you’re collecting good data, then there’s nothing you can’t find by searching the data.

With Indexer Acknowledgement

#region: Read .env file and create environment variables.
$FilterScript = {$_ -ne '' -and $_ -notmatch '^#'}
$Content = Get-Content -Path 'C:\Users\tommymaynard\Documents\_Repos\code\functiontemplate\random\telemetry\.env' | Where-Object -FilterScript $FilterScript
If ($Content) {
    Foreach ($Line in $Content) {
        $KVP = $Line -split '=',2; $Key = $KVP[0].Trim(); $Value = $KVP[1].Trim()
        Write-Verbose -Message "Adding an environment variable: `$env`:$Key."
        [Environment]::SetEnvironmentVariable($Key,$Value,'Process')
    } # End Foreach.
} Else {
    '$Content variable was not set. Do things without the file/the environment variables.'
}   # End If.
#endregion.
 
#region: Read clixml .xml file and obtain hashtable.
$HashTablePath = 'C:\Users\tommymaynard\Documents\_Repos\code\functiontemplate\random\telemetry\eventhashtable.xml'
$EventHashTable = Import-Clixml -Path $HashTablePath
#endregion.
 
#region: Create Splunk / Invoke-RestMethod variables.
$EventUri = $env:SplunkUrl + '/services/collector/event'
$AckUri = $env:SplunkUrl + '/services/collector/ack'
$ChannelIdentifier = (New-Guid).Guid
$Headers = @{Authorization = "Splunk $env:SplunkHECToken"; 'X-Splunk-Request-Channel' = $ChannelIdentifier}
$Body = ConvertTo-Json -InputObject $EventHashTable
$HttpRequestEventParams = @{URI = $EventUri; Method = 'POST'; Headers = $Headers; Body = $Body}
#endregion.
 
#region: Make requests to Splunk REST web services.
$Ack = Invoke-RestMethod @HttpRequestEventParams -StatusCodeVariable StatusCode -Verbose
$StatusCode
$AckBody = "{`"acks`": [$($Ack.ackId)]}"
$HttpRequestAckParams = @{URI = $AckUri; Method = 'POST'; Headers = $Headers; Body = $AckBody}
 
Measure-Command -Expression {
    Do {
        $AckResponse = Invoke-RestMethod @HttpRequestAckParams -Verbose
        $AckResponse.acks.0
        Start-Sleep -Seconds 30
    } Until ($AckResponse.acks.0 -eq $true)
} # End Measure-Command
#endregion.

Without Indexer Acknowledgement

#region: Read .env file and create environment variables.
$FilterScript = {$_ -ne '' -and $_ -notmatch '^#'}
$Content = Get-Content -Path 'C:\Users\tommymaynard\Documents\_Repos\code\functiontemplate\random\telemetry\.env' | Where-Object -FilterScript $FilterScript
If ($Content) {
    Foreach ($Line in $Content) {
        $KVP = $Line -split '=',2; $Key = $KVP[0].Trim(); $Value = $KVP[1].Trim()
        Write-Verbose -Message "Adding an environment variable: `$env`:$Key."
        [Environment]::SetEnvironmentVariable($Key,$Value,'Process')
    } # End Foreach.
} Else {
    '$Content variable was not set. Do things without the file/the environment variables.'
}   # End If.
#endregion.
 
#region: Read clixml .xml file and obtain hashtable.
$HashTablePath = 'C:\Users\tmaynard\Documents\_Repos\code\functiontemplate\random\telemetry\eventhashtable.xml'
$EventHashTable = Import-Clixml -Path $HashTablePath
#endregion.
 
#region: Create Splunk / Invoke-RestMethod variables.
$EventUri = $env:SplunkUrl + '/services/collector/event'
$Headers = @{Authorization = "Splunk $env:SplunkHECToken"}
$Body = ConvertTo-Json -InputObject $EventHashTable
$HttpRequestEventParams = @{URI = $EventUri; Method = 'POST'; Headers = $Headers; Body = $Body}
#endregion.
 
#region: Make requests to Splunk REST web services.
Invoke-RestMethod @HttpRequestEventParams -StatusCodeVariable StatusCode -Verbose
$StatusCode

Part IV: Splunk, HEC, Indexer Acknowledgement, and PowerShell

And now, Part IV! Like I’ve mentioned previously, please do yourself a favor and read the previous parts to this series of posts: Part I, Part II, and Part III. This won’t make sense without it. It may not make sense with it, and if that turns out to be true, then let me know by leaving a comment, or reaching me on Twitter. I know it’s possible that Splunk feels like a foreign language. What I know, I learned over the course of a few weeks, and it doesn’t feel like much. I probably shouldn’t be surprised that there’s so much to cover; so much was learned. Maybe you’ll have to do it too: I read the same articles over and over and over. I did have some help from a colleague, but if you don’t have that, then you can consider it me, up to a point,

The next thing we’ll look at is our first POST request to Splunk. This is us, sending in our JSON payload. Notice our first below Invoke-RestMethod command. First, it includes our $HttpRequestEventParams parameter hash table. This includes the Uri, Method, Headers, and the Body parameter and their corresponding parameter values. Here’s a reminder of what’s in that parameter hash table.

URI = @{$EventUri; Method = 'POST'; Headers = $Headers; Body = $Body}

As a part of this Invoke-RestMethod command, we also included the StatusCodeVariable parameter. When our command completes, it will have created the $StatusCode variable which should contain a 200 response, if our message was received by Splunk. Additionally, we have this command writing any output of the command (from Splunk) into the $Ack variable.

#region: Make requests to Splunk REST web services.
$Ack = Invoke-RestMethod @HttpRequestEventParams -StatusCodeVariable StatusCode -Verbose 
$StatusCode
$AckBody = "{`"acks`": [$($Ack.ackId)]}"
$HttpRequestAckParams = @{URI = $AckUri; Method = 'POST'; Headers = $Headers; Body = $AckBody}

Measure-Command -Expression {
    Do {
        $AckResponse = Invoke-RestMethod @HttpRequestAckParams -Verbose
        $AckResponse.acks.0
        Start-Sleep -Seconds 30
    } Until ($AckResponse.acks.0 -eq $true)
} # End Measure-Command
#endregion.

Keep in mind here, that if we weren’t using indexer acknowledgment, we’d probably be done. But now, we’ve got to create a second POST request to Splunk, so we can determine when the data that we know was received (200 response), is headed into the Splunk pipeline for processing. It’s a guarantee of data ingestion (not indigestion). But, since we’re going along down the indexer acknowledgment path (for now and only now), let’s walk through what else is happening here.

First, we create $AckBody. It uses the PowerShell object in $Ack and turns it back into JSON. Invoke-RestMethod has this helpful feature of turning JSON into a PowerShell object, so we’ve got to reverse that so we can send it back to Splunk. Once $AckBody is done, we’ll use it as a parameter value in $HttpRequestAckParams. About $AckBody, be sure to read this page (it’s been linked before), under the “Query for indexing status” section. Splunk sends us an ack ID and then we need to send it back with the same headers as in the first Invoke-RestMethod request. Remember, this includes the HEC token (pulled out of our .env file forever ago), and the channel identifier we created as a random GUID.

Following? Again, reach out to me if you need this information and I’m failing at getting it across to you. I’m not too big to help/rewrite/clarify. I get it, this has the potential to make someone launch their laptop out of a second-story window. Luckily, I work on the first floor of my house. Also, print out the Splunk articles I’ve linked and make them a home on the drink table next to your couch (like I did).

Okay, we’re on the last step. Remember, we’re still working as those indexer acknowledgement is enabled. This whole second POST request is irrelevant if you’re not going to use it. As is the channel identifier. Again, I’ll post modified code just as soon as my indexer acknowledgment is disabled.

Measure-Command -Expression {
    Do {
        $AckResponse = Invoke-RestMethod @HttpRequestAckParams -Verbose
        $AckResponse.acks.0
        Start-Sleep -Seconds 30
    } Until ($AckResponse.acks.0 -eq $true)
} # End Measure-Command

I mentioned that I’m not going to be using indexer acknowledgment because of the time it takes; I simply don’t have that. I’m in the business of automating. I record the duration of every function I invoke. It isn’t going to work for me. Anyway, I have my second Invoke-RestMethod request inside a Do-Until loop. This loop continues to run the command every 30 seconds until I get a $true response from Splunk (that means that Splunk is for sure going to index my data). For fun, that’s inside of a Measure-Command Expression parameter. This, is how I determined it took too much time for me. Below I’ll include the entire code as one block from all three posts. In the fifth post, yet to be written, I’ll include the entire code as one block, too. And remember, that post won’t have any requirements on an enabled indexer acknowledgment, the second POST request, the channel identifier, etc.

Thank you for hanging in for this one. Hopefully, it proves helpful for someone. Oh, that reminds me. There were two Splunk functions written to do all this when I started looking around. Maybe you found them too. They made little sense to me until I really took the time to learn what’s happening. Now that you’ve read this series, read over those functions; see what you understand about them now that you wouldn’t have been able to before. It might be much more than you thought it would be.

@torggler https://www.powershellgallery.com/packages/Send-SplunkEvent

@halr9000 https://gist.github.com/halr9000/d7bce26533db7bca1746

#region: Read .env file and create environment variables.
$FilterScript = {$_ -ne '' -and $_ -notmatch '^#'}
$Content = Get-Content -Path 'C:\Users\tommymaynard\Documents\_Repos\code\functiontemplate\random\telemetry\.env' | Where-Object -FilterScript $FilterScript
If ($Content) {
	Foreach ($Line in $Content) {
		$KVP = $Line -split '=',2; $Key = $KVP[0].Trim(); $Value = $KVP[1].Trim()
		Write-Verbose -Message "Adding an environment variable: `$env`:$Key."
		[Environment]::SetEnvironmentVariable($Key,$Value,'Process')
	} # End Foreach.
} Else {
	'$Content variable was not set. Do things without the file/the environment variables.'
}	# End If.
#endregion.

#region: Read clixml .xml file and obtain hashtable.
$HashTablePath = 'C:\Users\tommymaynard\Documents\_Repos\code\functiontemplate\random\telemetry\eventhashtable.xml'
$EventHashTable = Import-Clixml -Path $HashTablePath
#endregion.

#region: Create Splunk / Invoke-RestMethod variables.
$EventUri = $env:SplunkUrl + '/services/collector/event'
$AckUri = $env:SplunkUrl + '/services/collector/ack'
$ChannelIdentifier = (New-Guid).Guid
$Headers = @{Authorization = "Splunk $env:SplunkHECToken"; 'X-Splunk-Request-Channel' = $ChannelIdentifier}
$Body = ConvertTo-Json -InputObject $EventHashTable
$HttpRequestEventParams = @{URI = $EventUri; Method = 'POST'; Headers = $Headers; Body = $Body}
#endregion.

#region: Make requests to Splunk REST web services.
$Ack = Invoke-RestMethod @HttpRequestEventParams -StatusCodeVariable StatusCode -Verbose 
$StatusCode
$AckBody = "{`"acks`": [$($Ack.ackId)]}"
$HttpRequestAckParams = @{URI = $AckUri; Method = 'POST'; Headers = $Headers; Body = $AckBody}

Measure-Command -Expression {
	Do {
		$AckResponse = Invoke-RestMethod @HttpRequestAckParams -Verbose
		$AckResponse.acks.0
		Start-Sleep -Seconds 30
	} Until ($AckResponse.acks.0 -eq $true)
} # End Measure-Command
#endregion.

Part III: Splunk, HEC, Indexer Acknowledgement, and PowerShell

This is part three of a continuation of posts on Splunk, HEC, and sending data using PowerShell to Splunk for consumption. Make sure you’ve read the first two parts (Part I and Part II), as I’m going to assume you have!

Now that we have our data ready to be sent to Splunk in our $EventHashTable variable, we need to create some URLs using the environment variables we created in Part I. The first two lines create two URIs. In the first one, $EventUri, combines the value stored in $env:SplunkUrl with the string ‘/services/collector/event’. The second one, $AckUri, combines the same $env:SplunkUrl with the string ‘/services/collector/ack’. When completed, these two URIs are two different Splunk REST endpoints.

#region: Create Splunk / Invoke-RestMethod variables.
$EventUri = $env:SplunkUrl + '/services/collector/event'
$AckUri = $env:SplunkUrl + '/services/collector/ack'
$ChannelIdentifier = (New-Guid).Guid
$Headers = @{Authorization = "Splunk $env:SplunkHECToken"; 'X-Splunk-Request-Channel' = $ChannelIdentifier}
$Body = ConvertTo-Json -InputObject $EventHashTable
$HttpRequestEventParams = @{URI = $EventUri; Method = 'POST'; Headers = $Headers; Body = $Body}
#endregion.

We’re only going to need the second URI — $AckUri (see the above code) — if our HEC token requires indexer acknowledgment. For now, we’ll assume that it does. Beneath the two lines that create and assign the $EventUri and $AckUri variable is the creation of the $ChannelIdentifier variable. As the HEC has indexer acknowledgment enabled (a checkmark in the GUI), you’re required to make more than one POST request to Splunk. You send your first one, which we’ll see momentarily, and then you send a second one, as well. The first one gets your data to Splunk. The second (or third, or fourth, or fifth, etc.) lets you know if your data is being processed by Splunk. It may take more than just a second POST request to get back an acknowledgment. It’s a nice feature — who wouldn’t want to know!? Not only do you get your 200 status response letting you know Splunk received your data in the first request, but you can also know when Splunk is actually doing something with it!

I loved the idea at first; however, I was quick to discover it was taking anywhere from two to four, and sometimes five, minutes to return “true.” Once it returned true, I could rest assure that the data was being indexed by Splunk. Nice feature, but for me, that’s too much time to spend waiting. If I get a 200 back from my first POST, then I’d rather just carry on without concerning myself with whether or not my data actually hit the Splunk processing pipeline. From Part I through here, we’re going to continue as though you want to use indexer acknowledgment. Toward the end, we’ll see the code changes I implement to remove it.

I should say this, however, it’s in here, but the idea behind indexer acknowledgment is not about security. It’s about, and I quote, “…to prevent a fast client from impeding the performance of a slow client.” You see, index acknowledgment slows things down. So much so, I couldn’t use it. Lovely idea, but not for my purposes. It also uses channels, as you might’ve gathered from our $ChannelIdentifier variable. To create a channel, you create a random GUID and send that in with your initial POST request. Then you use the same channel identifier to obtain the acknowledgment. As I write this and read this, I realize that you really ought to read through the document I linked above. Here it is again: https://docs.splunk.com/Documentation/Splunk/8.1.0/Data/AboutHECIDXAck. It’s going to do this process more justice than I have. That said, I feel confident that what I’ve had to say might actually prove helpful when combined with the Splunk documentation.

After our $ChannelIdentifier variable is created and assigned, we’re going to create a $Headers (hash table) variable. It will contain two headers: one for authorization that contains our HEC token and one for our channel identifier used for indexer acknowledgment. The below code has been copied from the above code example.

$Headers = @{
    Authorization = "Splunk $env:SplunkHECToken"
    'X-Splunk-Request-Channel' = $ChannelIdentifier
}

Following that variable creation, we’re going to create two more. One will be our $Body variable and the other a hash table of parameters and their values that we’ll end up including with Invoke-RestMethod in Part IV. As a part of creating the $Body variable, we’re going to take our Hash table (with its nested hash table) and convert it all to JSON. This is what we want to send to Splunk. The below code has also been copied from the above code example.

$Body = ConvertTo-Json -InputObject $EventHashTable
$HttpRequestEventParams = @{
    URI = $EventUri
    Method = 'POST'
    Headers = $Headers
    Body = $Body
}

I think it’s important to see the JSON representation of our $EventHashTable variable because you’ll end up seeing this in the Splunk documentation. Notice the placement of the source, host, and time event metadata versus the (nested) event data.

$Body = Convertto-Json -InputObject $EventHashTable
$EventHashTable
{
  "source": "PowerShellFunctionTemplate",
  "host": "mainpc.tommymaynard.com\\TOMMYCOMPUTER",
  "event": {
    "OperatingSystemPlatform": "Win32NT",
    "PSHostProgram": "ConsoleHost",
    "PSVersionActive": "PowerShell 7.0.1"
  },
  "time": 1608148655,
  "sourcetype": "Telemetry"
}

The final line in our code example is the creation of a parameter hash table we’ll use — in a later part — to send our data to Splunk using Invoke-RestMethod.

$HttpRequestEventParams = @{
    URI = $EventUri
    Method = 'POST'
    Headers = $Headers
    Body = $Body
}

Until next time. … Part IV is now available.

Part II: Splunk, HEC, Indexer Acknowledgement, and PowerShell

In the first part of this series, we discussed using a .env file to obtain a Splunk HEC token and our Splunk URL. These two pieces of information are going to need to make their way into our POST request to Splunk. Be sure to read Part I if you haven’t already.

The next thing we need to do is pull in a hash table from the disk. The reason I’ve done it this way is to ensure I had a hash table to use without having to create it. It was a quicker way to work with my code, as I was still in the development and testing phase. In production, the hash table would be created not from the disk/stale data, but from real-time data collection.

#region: Read clixml .xml file and obtain hashtable.
$HashTablePath = 'C:\Users\tommymaynard\Documents\_Repos\code\functiontemplate\random\telemetry\eventhashtable.xml'
$EventHashTable = Import-Clixml -Path $HashTablePath
#endregion.

The above code does two things: One, it sets the $HashTablePath variable to an XML file. This file was created previously using Export-Clixml. The second thing it does is import the XML file using Import-Clixml. So, what’s in the file and what are we doing here? As we’ll see later, we’re going to use a hash table—an in-memory data structure to store key-value pairs—and get that converted (to something else) and sent to Splunk.

This isn’t just any hash table, so let’s take a look at what’s stored in the $EventHashTable variable once these two lines of code are complete.

$EventHashTable
Name                           Value
----                           -----
event                          {OperatingSystemPlatform, IPAddressPublic, Command:0, PSVersionOther…}
sourcetype                     Telemetry
host                           mainpc.tommymaynard.com\TOMMYCOMPUTER
time                           1608148655
source                         PowerShellFunctionTemplate

If you’ve worked with PowerShell hash tables before, then you’ll probably recognize that there’s a nested hash table inside this hash table. That, or at minimum, that one of the values doesn’t seem right. This nested hash table is stored as the value for the event key. The four other keys — host, time, source, and sourcetype — are standard key-value pairs that have values that are not nested hash tables. Let’s take a look at the key-value pairs inside the nested hash table. We can use dotted notation to view the values stored in each key.

$EventHashTable.host
mainpc.tommymaynard.com\TOMMYCOMPUTER
$EventHashTable.time; $EventHashTable.source
1608148655
PowerShellFunctionTemplate
$EventHashTable.sourcetype
Telemetry

Now, let’s view the key-value pairs stored inside the nested hash table. Then we’ll run through how we’re able to nest a hash table inside another hash table before it was saved to disk using Export-Clixml.

 $EventHashTable.event  
Name                           Value
----                           -----
OperatingSystemPlatform        Win32NT
IPAddress                      64.204.192.66
Command:0:CommandName          New-FunctionTemplate
PSVersionAdditional            Windows PowerShell 5.1.19041.610
DateTime                       Wed. 12/16/2020 12:57:35 PM     
Command:0:Parameter:1          PassThru:True
Template                       FunctionTemplate 3.4.7
Command:0:CommandType          Function
IPAddressAdditional            192.168.86.127
Command:0:Parameter:0          Log:ToScreen
Duration                       0:00:00:00.9080064
Domain\Username                mainpc.tommymaynard.com\tommymaynard
CommandName                    New-FunctionTemplate
ModuleName
Domain\Computer                mainpc.tommymaynard.com\TOMMYCOMPUTER
OperatingSystem                Microsoft Windows 10.0.19041
PSVersion                      PowerShell 7.1.0
PSHostProgram                  ConsoleHost

Look at all that data! It’s beautiful (although not completely accurate, so that it may be posted here). When it’s not pulled from the disk/stale data, such as in this example, it’s being gathered by a function during its invocation, and then it’s sent off to Splunk! Because Splunk is involved, it’s important to understand why we’ve set up the hash tables the way we have. The host, time, source, and sourcetype keys in the top-level hash table are what Splunk calls event metadata (scroll down to Event Metadata). It’s optional to send in this data, but for me, it made sense. I wanted to guarantee that I had control over these values versus what their default values might be. The nested hash table is the event data. It’s not data about data, such as metadata is; it’s the data we collected and want Splunk to retain.

We’re going to wrap this part up, but I do want to provide how I got a hash table to nest inside of another hash table. We won’t bother working with the complete information above, but I’ll explain what you need to know. Here’s our downsized event hash table.

$Event = @{
    OperatingSystemPlatform = 'Win32NT'
    PSVersion = 'PowerShell 7.0.1'
    PSHostProgram = 'ConsoleHost'
}

And here it is, stored in the $Event variable and being used as a value in the parent, or previously used term, top-level hash table.

$EventHashTable = @{
    event = $Event
    host = 'mainpc.tommymaynard.com\TOMMYCOMPUTER'
    time = [int](Get-Date -Date (Get-Date) -UFormat %s)
    source = 'PowerShellFunctionTemplate'
    sourcetype = 'Telemetry'
}
$EventHashTable
Name                           Value
----                           -----
source                         PowerShellFunctionTemplate
sourcetype                     Telemetry
host                           mainpc.tommymaynard.com\TOMMYCOMPUTER
event                          {OperatingSystemPlatform, PSHostProgram, PSVersion}
time                           1608148655

If you opt to include the time event metadata, do recognize that it uses, and expects, epoch time. This is the number of seconds after epoch, or  June 1, 1970, 00:00:00 UTC. The -UFormat parameter with the %s parameter value will return this value. I believe this was added in Windows PowerShell 3.0, so if you have a client with an earlier version, first, I’m sorry, and second, there’s another way to obtain this value without the UFormat parameter: think datetime subtraction.

Okay, check back for more; there’s still plenty left to cover. Part III will be up soon!

Edit: Part III is now available.

Part I: Splunk, HEC, Indexer Acknowledgement, and PowerShell

A few weeks ago and I only knew one of the four words that make up this post’s title: PowerShell. I mean, I had heard of Splunk. But realistically, the other two words, which pertain to Splunk, I didn’t know. They weren’t a part of November’s vocabulary. Hopefully, you ended up here by doing some of the same searches I did. The same ones that didn’t really net me as much as I had hoped they would. I’m still no Splunk expert — I could see that taking some time and consistent interaction with the product — but, I have sent data to Splunk using PowerShell via REST. Who would have thought!? Perhaps that’s what you need to do, too.

It was a few weeks ago when I was wrestling with how I should handle collecting telemetry data, about users and their environment, when they used a function written with my function template. If just maybe you wondered, my function template is derived using some of the same techniques I wrote about in The PowerShell Conference Book: Volume 3.This book can be obtained from Leanpub, as an eBook. A print copy can be acquired from Amazon.

The initial idea was to use AWS API Gateway with DynamoDB. Lambda wouldn’t have been required, as those first two mentioned services integrate without it. I had a colleague mention Splunk though, as it’s available to me. With that, I quickly changed direction, and it’s worked out well. I didn’t get to checkoff those two AWS services from my I’ve-done-this-in-production list, but I’m nearing it for Splunk. I had wanted a reason to work with Splunk and up until that moment I didn’t have a project that offered that. At least I didn’t think I did.

There was at least one upfront requirement: I couldn’t install software. My PowerShell functions could be running anywhere, and so a REST endpoint was a requirement. Lucky for me, I had an option (that I was going to need to learn a bit about). I needed a term with which I could begin my searching, and reading, and research, and eventually my coding and integration. It was HEC. That’s pronounced “heck.” I know because I watched at least one Splunk education video. Sadly, it was later in the overall development process.

HEC, or HTTP Event Collector, is a means by which data can be securely sent to Splunk for ingestion. Read over the included link for a 20 second lesson. Seriously, it’s short and worth it! Using HEC allowed me to use a token as authentication and use REST to get my PowerShell function acquired telemetry data into Splunk. Let’s get started with some PowerShell already.

First things first, we need to protect our HEC token. You’ll need to get it from your Splunk administrator. There’s a few things out of my hands here and that, is one of them. I don’t have administrative permissions to Splunk and that’s been just fine for me. The HEC token will allow our PowerShell to authenticate to Splunk without the need for any other type of credentials. The problem was, that under no circumstances did I want to embed my HEC token in my function. Therefore, I opted to use a .env file (for now?). We’ll start with that concept. This isn’t 100% foolproof, but again, it’s much, much better than dropping my HEC token into my code. Here’s the first section of the code I’m going to share.

#region: Read .env file and create environment variables.
$FilterScript = {$_ -ne '' -and $_ -notmatch '^#'}
$Path = 'C:\Users\tommymaynard\Documents\_Repos\code\functiontemplate\random\telemetry\.env'
$Content = Get-Content -Path $Path | Where-Object -FilterScript $FilterScript
If ($Content) {
	Foreach ($Line in $Content) {
		$KVP = $Line -split '=',2; $Key = $KVP[0].Trim(); $Value = $KVP[1].Trim()
		Write-Verbose -Message "Adding an environment variable: `$env`:$Key."
		[Environment]::SetEnvironmentVariable($Key,$Value,'Process')
	} # End Foreach.
} Else {
	'$Content variable was not set. Do things without the file/the environment variables.'
}	# End If.
#endregion.

This has nothing to really do with Splunk so far, but it may help you protect your HEC token. I’d like to get the values in this file stored in Azure Key Vault someday. This is just for now… to see that everything else is working. Let me give you an example of what the .env file looks like, and then we’ll breakdown what the previously included code is doing.

# FunctionTemplate Environment Variables.

## Splunk Environment Variables.
SplunkUrl=https://splunkhectest.prod.splunk.tommymaynard.com:8088
SplunkHECToken=88a11c44-3a1c-426c-bb04-390be8b32287

The farther above code example creates and stores a script block inside the $FilterScript variable. This will exclude lines in our .env file that are empty or begin with a hash (#) character. It’s used as a part of the Get-Content command, which reads in the file’s lines that we actually want and assigns them to the $Content variable. As you can see, each line we want includes a string on the left, an equals sign (=) in the middle, and a string on the right. The remainder of the code — the Foreach loop — creates a process environment variable for each usable line during each iteration. When it’s completed, we’ll have $env:SplunkUrl and $env:SplunkHECToken and both will be populated with their respective, right side of the equals sign, value. It’s not perfect, but it’s 10x better than storing these values in the actual function template.

I’m going to stop here for now, but we’ll continue very soon with Part II. We haven’t gotten into Splunk yet, but we’re now set to do just that. With these two values (in variables), we can start working toward making our POST request to Splunk and passing along the data we want indexed.

Part II is now available.

Comfortably Save a Pester Test

I’ve spent some time with Pester this week. While exploring the OutputFile parameter, it quickly became clear that the best I seemed to be able to output was some form of XML, and honestly, that wasn’t good enough for me in my moment of discovery. While I intend to make myself a Pester expert in the coming year (2018), there’s somethings I don’t know 100% yet, and so I understand it’s possible that I sound like an idiot, as I may not know about x in regard to Pester.

While working with Invoke-Pester, I came across the -PassThru parameter. Its purpose in life, in relation to the Invoke-Pester command, is to create a PSCustomObject. Now that’s, something I can work with. The idea here is to invoke Pester against an AWS instance, export my object (to an XML standard I can deal with [thank you Export-Clixml]), and write it to S3. Then, I can download the file to a computer that’s not my newly configured AWS instance, and check for what tests passed and failed. This, without the need to RDP (Remote Desktop), to the instance, and visually and manually check to see what did and didn’t work from there. We’re getting closer and closer to RDP being a security incident.

My example is not going to include an actual Pester run, so instead we’ll jump directly to the Invoke-Pester command and what we’ll do after that command has executed. This first example is a fairly standard way of invoking Pester (with parameters) and creating an output file. Again, this isn’t the output I’m after.

PS > Invoke-Pester -Script @{Path = 'C:\WkDir\HostAcceptance.Tests.ps1'; Parameters = @{Project = 'trailking'; Environment = 'tst'}} -OutputFile 'C:\WkDir\PesterOutputXml.xml'

Instead we’re going to include some other parameters, to include PassThru, Show, and OutVariable. PassThru will provide us an object that contains all of the Pester results, Show with the None value will hide the Pester test results, and OutVariable will get that object (that again, contains all of the Pester results) into the PesterResults variable.

PS > Invoke-Pester -Script @{Path = 'C:\WkDir\HostAcceptance.Tests.ps1'; Parameters = @{Project = 'trailking'; Environment = 'tst'}} -PassThru -Show None -OutVariable PesterResults

I was mainly okay with the above command; however, it was still writing my object to the host program, and I wanted nothing to be displayed, at all. It’s too bad that option isn’t in there by default, but I’m okay with improvisation. Again, I know PowerShell better than Pester, so there is the possibility I just don’t know enough at the immediate moment to not have done this a potentially better way. Anyway, the below example removes all the output to the host, but still stuffs my results into the PesterResults variable.

PS > [System.Void](Invoke-Pester -Script @{Path = 'C:\WkDir\HostAcceptance.Tests.ps1'; Parameters = @{Project = 'trailking'; Environment = 'tst'}} -PassThru -Show None -OutVariable PesterResults)

Okay, now what? Everything I need and more is now in the $PesterResults variable. Next, we’ll export it into an XML format I (as in Import-Clixml), can deal with.

PS > $PesterResults | Export-Clixml -Path 'C:\WkDir\PesterExport.xml'

Now that it’s in a usable format, I’m going to read it back in. You can go ahead and pretend that I’ve moved my exported XML file from the computer on which it was created, and now I’m about to read it in on a different computer. This visualization is as though it was uploaded to AWS S3, and downloaded from S3 on a different device. We’ll say it’s on my Desktop.

$PesterResults = Import-Clixml -Path 'C:\Users\tommymaynard\Desktop\PesterStuff\PesterExport.xml'
Foreach ($Result in $PesterResults.TestResult) {
    [System.Array]$Object += [PSCustomObject]@{
        Describe = $Result.Describe.TrimEnd(':')
        Context = $Result.Context.TrimEnd(':')
        It = $Result.Name.TrimEnd(':')
        Should = $Result.Result
        Time = $Result.Time
    }
} 

$Object
# $Object | Format-Table -Autosize
# $Object | Export-Csv -Path 'C:\Users\tommymaynard\Desktop\PesterStuff\PesterExport.xml' -NoTypeInformation

In the above example, we do a few things. We import the file we created out in AWS into a variable on my local computer, and then begin to iterate through its contents. For each entry, we add other object to my Object variable. Each object will contain a Describe, Context, It, Should, and Time property which is all obtained from the TestResult property of my PesterResults variable.

The last three lines are a few ways to handle the output (2 and 3 are commented out): (1) display it in the host, (2) display it in the host in an easier to read format, and (3) write the objects to CSV. Neat, right. As I continue to better learn Pester… I might just be back here. I can’t be 100% that this is the best way to save off and deal with the results of the Pester tests, but we’ll see!