This is part three of a continuation of posts on Splunk, HEC, and sending data using PowerShell to Splunk for consumption. Make sure you’ve read the first two parts (Part I and Part II), as I’m going to assume you have!

Now that we have our data ready to be sent to Splunk in our $EventHashTable variable, we need to create some URLs using the environment variables we created in Part I. The first two lines create two URIs. In the first one, $EventUri, combines the value stored in $env:SplunkUrl with the string ‘/services/collector/event’. The second one, $AckUri, combines the same $env:SplunkUrl with the string ‘/services/collector/ack’. When completed, these two URIs are two different Splunk REST endpoints.

#region: Create Splunk / Invoke-RestMethod variables.
$EventUri = $env:SplunkUrl + '/services/collector/event'
$AckUri = $env:SplunkUrl + '/services/collector/ack'
$ChannelIdentifier = (New-Guid).Guid
$Headers = @{Authorization = "Splunk $env:SplunkHECToken"; 'X-Splunk-Request-Channel' = $ChannelIdentifier}
$Body = ConvertTo-Json -InputObject $EventHashTable
$HttpRequestEventParams = @{URI = $EventUri; Method = 'POST'; Headers = $Headers; Body = $Body}
#endregion.

We’re only going to need the second URI — $AckUri (see the above code) — if our HEC token requires indexer acknowledgment. For now, we’ll assume that it does. Beneath the two lines that create and assign the $EventUri and $AckUri variable is the creation of the $ChannelIdentifier variable. As the HEC has indexer acknowledgment enabled (a checkmark in the GUI), you’re required to make more than one POST request to Splunk. You send your first one, which we’ll see momentarily, and then you send a second one, as well. The first one gets your data to Splunk. The second (or third, or fourth, or fifth, etc.) lets you know if your data is being processed by Splunk. It may take more than just a second POST request to get back an acknowledgment. It’s a nice feature — who wouldn’t want to know!? Not only do you get your 200 status response letting you know Splunk received your data in the first request, but you can also know when Splunk is actually doing something with it!

I loved the idea at first; however, I was quick to discover it was taking anywhere from two to four, and sometimes five, minutes to return “true.” Once it returned true, I could rest assure that the data was being indexed by Splunk. Nice feature, but for me, that’s too much time to spend waiting. If I get a 200 back from my first POST, then I’d rather just carry on without concerning myself with whether or not my data actually hit the Splunk processing pipeline. From Part I through here, we’re going to continue as though you want to use indexer acknowledgment. Toward the end, we’ll see the code changes I implement to remove it.

I should say this, however, it’s in here, but the idea behind indexer acknowledgment is not about security. It’s about, and I quote, “…to prevent a fast client from impeding the performance of a slow client.” You see, index acknowledgment slows things down. So much so, I couldn’t use it. Lovely idea, but not for my purposes. It also uses channels, as you might’ve gathered from our $ChannelIdentifier variable. To create a channel, you create a random GUID and send that in with your initial POST request. Then you use the same channel identifier to obtain the acknowledgment. As I write this and read this, I realize that you really ought to read through the document I linked above. Here it is again: https://docs.splunk.com/Documentation/Splunk/8.1.0/Data/AboutHECIDXAck. It’s going to do this process more justice than I have. That said, I feel confident that what I’ve had to say might actually prove helpful when combined with the Splunk documentation.

After our $ChannelIdentifier variable is created and assigned, we’re going to create a $Headers (hash table) variable. It will contain two headers: one for authorization that contains our HEC token and one for our channel identifier used for indexer acknowledgment. The below code has been copied from the above code example.

$Headers = @{
    Authorization = "Splunk $env:SplunkHECToken"
    'X-Splunk-Request-Channel' = $ChannelIdentifier
}

Following that variable creation, we’re going to create two more. One will be our $Body variable and the other a hash table of parameters and their values that we’ll end up including with Invoke-RestMethod in Part IV. As a part of creating the $Body variable, we’re going to take our Hash table (with its nested hash table) and convert it all to JSON. This is what we want to send to Splunk. The below code has also been copied from the above code example.

$Body = ConvertTo-Json -InputObject $EventHashTable
$HttpRequestEventParams = @{
    URI = $EventUri
    Method = 'POST'
    Headers = $Headers
    Body = $Body
}

I think it’s important to see the JSON representation of our $EventHashTable variable because you’ll end up seeing this in the Splunk documentation. Notice the placement of the source, host, and time event metadata versus the (nested) event data.

$Body = Convertto-Json -InputObject $EventHashTable
$EventHashTable
{
  "source": "PowerShellFunctionTemplate",
  "host": "mainpc.tommymaynard.com\\TOMMYCOMPUTER",
  "event": {
    "OperatingSystemPlatform": "Win32NT",
    "PSHostProgram": "ConsoleHost",
    "PSVersionActive": "PowerShell 7.0.1"
  },
  "time": 1608148655,
  "sourcetype": "Telemetry"
}

The final line in our code example is the creation of a parameter hash table we’ll use — in a later part — to send our data to Splunk using Invoke-RestMethod.

$HttpRequestEventParams = @{
    URI = $EventUri
    Method = 'POST'
    Headers = $Headers
    Body = $Body
}

Until next time. … Part IV is now available.