Tag Archives: background jobs

Background Jobs and Recursive Functions

I recently had a thought: I have not written much on background jobs over the last nearly eight years. I should do more of that. Part of it stemmed from recently preparing an old post that was published elsewhere to be brought back to life here on tommymaynard.com. That one was Keeping a Continuous Total.

In that post, I wrote, “Next up is likely putting this code into a background job; it’s not the quickest thing I’ve written (although I blame that on the speed of the PowerShell Gallery lookup process, perhaps). Maybe a background job that runs in, or starts at, the end of the profile script. This, in order that these slow-to-obtain results are available sooner and with minimal impact.”

Well, I did that. While the function to do lookups in the PowerShell Gallery already existed in my profile script, I added a new function to my profile script that includes various background job commands that is invoked by my profile script. And, it turns out I was able to make it a recursive function, as well. You never think you will need one of those until the opportunity presents itself. I am so glad I noticed this opportunity.

Let’s start with my Show-PSGalleryProject function. You are welcome to use this as well. This function goes out to the PowerShell Gallery and determines the download count of each of my scripts and modules published there. This function takes some time to run, therefore, I want it to run in the background, so the results are available quicker than they would be otherwise.

Function Show-PSGalleryProject {
    [CmdletBinding()]
    Param (
        [System.Array]$Projects = ('TMOutput','Start-1to100Game3.0','Get-TMVerbSynonym',
            'SinkProfile','Show-PSDriveMenu','Switch-Prompt')
    )

    Foreach ($Project in $Projects) {
        If (Find-Module -Name $Project -ErrorAction SilentlyContinue) {
            $TempVar = Find-Module -Name $Project; $Type = 'Module'
        } ElseIf (Find-Script -Name $Project) {
            $TempVar = Find-Script -Name $Project; $Type = 'Script'
        }
        $TotalDownloads = [int]$TotalDownloads + [int]$TempVar.AdditionalMetadata.downloadCount
        [PSCustomObject]@{
            Name = $TempVar.Name
            Type = $Type
            Version = $TempVar.Version
            Downloads = $TempVar.AdditionalMetadata.downloadCount
            TotalDownloads = $TotalDownloads
        }
    } # End Foreach.
} # End Function: Show-PSGalleryProject.

The invocation of the above function is controlled by the below function. So, the Show-PSGalleryProject function goes out to the gallery to collect information, and the Show-PSGalleryProjectJob function orchestrates this process. Let me explain what this section function does. But first, take a look at it and see what you can extract yourself.

Set-Alias -Name psgal-Value Show-PSGalleryProjectJob
Function Show-PSGalleryProjectJob {
	$FunctionName = $MyInvocation.MyCommand.Name
	if (-Not(Get-Job -Name $FunctionName -ErrorAction SilentlyContinue)) {
		Start-Job -Name $FunctionName -ScriptBlock ${Function:Show-PSGalleryProject} | Out-Null
	} else {
		if ((Get-Job -Name $FunctionName).State -eq 'Completed') {
			$JobEndTime = (Get-Job -Name $FunctionName).PSEndTime
			Receive-Job -Name $FunctionName |
				Select-Object -Property *,@{Name='EndTime'; Expression={$JobEndTime}} -ExcludeProperty RunspaceId |
				Format-Table -AutoSize
			Remove-Job -Name $FunctionName
			& $FunctionName
		} else {
			Write-Warning -Message "Please wait. The $FunctionName background job is $((Get-Job -Name Show-PSGalleryProjectJob).State.ToLower()) (Id: $((Get-Job -Name Show-PSGalleryProjectJob).Id))."
		}
	}
}
psgal

Buckle up; here we go.

Line 1: Create the psgal alias for the function.
Line 2 (and 18): Define the Show-PSGalleryProjectJob function.
Line 3: Create the $FunctionName variable and assign it the name of this function using the $MyInvocation variable. The function’s name is used repeatedly throughout the function, so it made sense to store it in a variable.
Line 4: Include an if-else language construct. The if portion determines if there is a background job called Show-PSGalleryProjectJobor not.
Line 5: If that background job does not exist, it should be created and started. The Start-Job cmdlet’s ScriptBlock property invokes the Show-PSGalleryProject function.
Line 6: This is the beginning of else portion. The lines beneath it will run if there is already a background job running called Show-PSGalleryProjectJob.
Line 7 – 13: Nested in the else portion is another if-else construct. If the job is complete we run the commands in the if portion. If the job is not yet complete, we run the commands in the else portion below. The if portion does all of the following: it collects the end time of the job and stores it in $JobEndTime, it uses Receive-Job to collect the results of the completed background job, it pipes those results to Select-Object and displays all the default properties, as well as the end time we add using a calculated property. It takes those results and pipes them to Format-Table -AutoSize . Once it is done with those steps, it uses Remove-Job to remove/delete the background job.

Now, for the recursion. The final step inside the if portion of this nested if-else is to invoke the Show-PSGalleryProjectJob function. That is right. The function invokes or calls itself. It starts this whole process over again. It does this using the call operator (&) and the $FunctionName variable. Remember, that variable holds the name of the function. Without the call operator, it would just echo the value in the variable. That operator also called the invocation operator, invokes the function again. Every time the job is completed and the values are returned, the process starts over. On a side note, I have written about recursive functions once before.

Line 14 – 16: The else portion issues a Write-Warning message indicating that the job is not yet complete.
Line 19: The alias invokes the Show-PSGalleryProjectJob function.

The below results show the functions working together. Consider background jobs for longer running tasks and consider recursive functions when a situation presents itself where a function should call, or invoke, itself.

[PS7.2.1][C:\] psgal
The Show-PSGalleryProjectJob background job is running (Id: 1).
[PS7.2.1][C:\] 
[PS7.2.1][C:\] psgal
The Show-PSGalleryProjectJob background job is running (Id: 1).
[PS7.2.1][C:\] 
[PS7.2.1][C:\] # Here is where I waited from some time to pass...
[PS7.2.1][C:\] 
[PS7.2.1][C:\] psgal

Name                Type   Version Downloads TotalDownloads EndTime
----                ----   ------- --------- -------------- -------
TMOutput            Module 1.1     2994                2994 2/10/2022 5:07:50 PM
Start-1to100Game3.0 Script 3.0     266                 3260 2/10/2022 5:07:50 PM 
Get-TMVerbSynonym   Script 1.4     293                 3553 2/10/2022 5:07:50 PM 
SinkProfile         Module 1.0     304                 3857 2/10/2022 5:07:50 PM 
Show-PSDriveMenu    Script 1.1     186                 4043 2/10/2022 5:07:50 PM 
Switch-Prompt       Script 1.2.0   236                 4279 2/10/2022 5:07:50 PM 

[PS7.2.1][C:\] psgal
The Show-PSGalleryProjectJob background job is running (Id: 3).
[PS7.2.1][C:\] 
[PS7.2.1][C:\] psgal
The Show-PSGalleryProjectJob background job is running (Id: 3).
[PS7.2.1][C:\]

Saving Time with Background Jobs

If you’re like me, there’s something you know a decent amount about regarding PowerShell, but you just don’t get to use it much. Today, it’s PowerShell background jobs. If you’ve been reading my blog currently, then you know I’m right in the middle of a series regarding Splunk. In the series, I’m sending telemetry data from my function template to Splunk. The problem, although slight, is that it’s increased the duration, or length of time, the function takes to complete. No surprise. It’s running several additional commands where it polls the user and system for information. It’s only adding maybe a second more of time to the duration of the function. Still, why not determine if it’s time that can be reclaimed. Enter background jobs.

If I can collect my telemetry data in the background, while the function is doing whatever it’s supposed to be doing, then I can potentially remove any additional time added to the invocation of the function due to collecting telemetry data. Let’s take a look a couple code examples to begin.

This first function is called Start-SleepWITHOUTBackgroundJob. Notice the “without” in the name. This function will run Start-Sleep twice within the function: once for five seconds and then once for three seconds. Therefore, we’d expect the function to take around eight seconds to complete. The five second section is standing in for where we’d run our standard function code. The three second section is standing in for where we’d collect our telemetry data.

Function Start-SleepWITHOUTBackgroundJob {
    Start-Sleep -Seconds 5

    Start-Sleep -Seconds 3
} # End Function: Start-SleepWITHOUTBackgroundJob.

Measure-Command -Expression {
    Start-SleepWITHOUTBackgroundJob
}

Let’s run it a few times. As you’ll see, and just as we’d suspected, it comes in at right around the 8 second mark. If you’ve seen the output of Measure-Command then you can tell I’ve removed several of the time properties; they weren’t necessary.

Seconds           : 8
Milliseconds      : 16

Seconds           : 8
Milliseconds      : 26

Seconds           : 8
Milliseconds      : 22

The second function is called Start-SleepWITHBackgroundJob. We’ve swapped our Start-Sleep commands because we want what takes less time to happen first. It has to be what happens inside the background job. I suspect that gathering telemetry data is most always going to take less time than whatever else the function is doing. That may not always be the case, but it’s a safe choice.

Function Start-SleepWITHBackgroundJob {
    Start-Job -ScriptBlock {
        Start-Sleep -Seconds 3
    } | Out-Null

    Start-Sleep -Seconds 5
} # End Function: Start-SleepWITHBackgroundJob.

Get-Job | Remove-Job
Measure-Command -Expression {
    Start-SleepWITHBackgroundJob
}

And, look at that. We’ve shaved off three seconds from our function invocation by placing those three seconds inside of a background job. Our three seconds are running in a separate PowerShell process that executes at the same time the function sleeps for five seconds. This is going to work great for me.

Seconds           : 5
Milliseconds      : 596

Seconds           : 5
Milliseconds      : 795

Seconds           : 5  
Milliseconds      : 417

Now that we’ve proved we can use PowerShell background jobs to save time and avoid some unnecessary top-to-bottom/sequential programming, let’s do this while actually gathering some telemetry data. We’ll do two things at once and shave off some time from the overall time. The time difference may not be as dramatic as the above examples, but I’ll take anything. In fact, watch this first.

Do you see the delay? There’s a moment where my telemetry data is being gathered and sent to Splunk, before the prompt reappears. The idea is to get those milliseconds back — they add up!

As you can see below, we have another code example. This will run without a background job. It’ll sleep for five seconds (as thought it’s fulfilling its purpose), and then collect some telemetry data and display that on the screen. I’ll share the code in between each of the below regions at the end of this post in case someone finds themself interested.

Function Start-SleepWITHOUTBackgroundJob {
    Start-Sleep -Seconds 5

    #region: Obtain telemetry.
	New-Variable -Name FuncTmplHash -Value @{}
	New-Variable -Name TelemetryHashBonus -Value @{}
        #region: Determine PowerShell version.
        #endregion.
        #region: Check for other version: Windows PowerShell|PowerShell.
        #endregion.
        #region: Determine IP address(es).
        #endregion.
        #region: Determine Operating System.
        #endregion.
        #region: Determine computer tier.
        #endregion.
    $TelemetryHashBonus
    #endregion.
} # End Function: Start-SleepWITHOUTBackgroundJob.

Measure-Command -Expression {
    Start-SleepWITHOUTBackgroundJob | Out-Default
}

While the time difference isn’t too dramatic (roughly 750 milliseconds), it’s something. Something of which I want to partially reclaim. This is exactly why you see the hesitation/pause before PowerShell rewrites a fresh prompt in the above GIF. Now, let’s get this corrected.

Function Start-SleepWITHBackgroundJob {
    Start-Job -ScriptBlock {
        #region: Obtain telemetry.
        New-Variable -Name FuncTmplHash -Value @{}
        New-Variable -Name TelemetryHashBonus -Value @{}
        #region: Determine PowerShell version.
        #endregion.
        #region: Check for other version: Windows PowerShell|PowerShell.
        #endregion.
        #region: Determine IP address(es).
        #endregion.
        #region: Determine Operating System.
        #endregion.
        #region: Determine computer tier.
        #endregion.
        $TelemetryHashBonus
        #endregion.
     } -OutVariable Job | Out-Null

    Start-Sleep -Seconds 5
    Receive-Job -Id $Job.Id
} # End Function: Start-SleepWITHBackgroundJob.

Measure-Command -Expression {
    Start-SleepWITHBackgroundJob | Out-Default
}

If we take a look a the below results versus the run without the background job we can see that we’ve saved roughly 500 milliseconds, or a 1/2 a second. That’s not much; I’d agree, even though it feels like an eternity when I’m waiting for my prompt to be rewritten. I guess I should consider that this isn’t the full telemetry gathering code I use. Still, for every two invocations, I save a single second. One hundred and twenty invocations saves me a minute. If my tools are far reaching, then there’s definitely time to be saved.

It does take time to create the job and receive its data once it’s complete, so perhaps that’s eating into my return on time, as well. That makes me think of one more thing worth sharing. If you find yourself interested in implementing something like this, then it’s probably wise to not assume the background job is complete, as I’ve done in these examples. Instead of running Receive-Job, first run Get-Job and ensure your job’s State property is “Completed,” and not still “Running.” It would probably be best to put this inside a Do-Until language construct, so it can loop until you can be certain the job is completed, before receiving its data.

I said I share the telemetry gathering code, so that’s been included below. I make no guarantees that it’ll work or make sense for you, but there it is.

#region: Obtain telemetry.
New-Variable -Name FuncTmplHash -Value @{}
New-Variable -Name TelemetryHashBonus -Value @{}

#region: Determine PowerShell version.
$FuncTmplHash.Add('PSVersion',"$(If ($PSVersionTable.PSVersion.Major -lt 6) {"Windows PowerShell $($PSVersionTable.PSVersion.ToString())"} Else {
	"PowerShell $($PSVersionTable.PSVersion.ToString())"})")
$TelemetryHashBonus.Add('PSVersion',$FuncTmplHash.PSVersion)
#endregion.

#region: Check for other version: Windows PowerShell|PowerShell.
If ($FuncTmplHash.PSVersion -like 'PowerShell*') {
	$TelemetryHashBonus.Add('PSVersionAdditional',
		"$(try {powershell.exe -NoLogo -NoProfile -Command {"Windows PowerShell $($PSVersionTable.PSVersion.ToString())"}} catch {})")
} ElseIf ($FuncTmplHash.PSVersion -like 'Windows PowerShell*') {
	$TelemetryHashBonus.Add('PSVersionAdditional',
		"$(try {pwsh.exe -NoLogo -NoProfile -Command {"PowerShell $($PSVersionTable.PSVersion.ToString())"}} catch {})")
} # End If-Else.
#endregion.

#region: Determine IP address(es).
$ProgressPreference = 'SilentlyContinue'
$TelemetryHashBonus.Add('IPAddress',(Invoke-WebRequest -Uri 'http://checkip.dyndns.com' -Verbose:$false).Content -replace "[^\d\.]")
$TelemetryHashBonus.Add('IPAddressAdditional',@(Get-NetIPAddress | Where-Object -Property AddressFamily -eq 'IPv4' |
	Where-Object -FilterScript {$_ -notlike '169.*' -and $_ -notlike '127.*'}).IPAddress)
$ProgressPreference = 'Continue'
#endregion.

#region: Determine Operating System.
If ($FuncTmplHash.PSVersion -like 'Windows PowerShell*' -and $FuncTmplHash.PSVersion.Split(' ')[-1] -lt 6) {
	$TelemetryHashBonus.Add('OperatingSystem',"Microsoft Windows $((Get-CimInstance -ClassName Win32_OperatingSystem -Verbose:$false).Version)")
	$TelemetryHashBonus.Add('OperatingSystemPlatform','Win32NT') 
} Else {$TelemetryHashBonus.Add('OperatingSystem',"$($PSVersionTable.OS)")
	$TelemetryHashBonus.Add('OperatingSystemPlatform',"$($PSVersionTable.Platform)")} # End If-Else.
#endregion.

#region: Determine computer tier.
Switch ($FuncTmplHash.'Domain\Computer') {{$_ -like '*-PT0-*'} {$TelemetryHashBonus.Add('ComputerTier','T0'); break} 
{$_ -like '*-PT1-*'} {$TelemetryHashBonus.Add('ComputerTier','T1'); break}
default {$TelemetryHashBonus.Add('ComputerTier','Unknown')}} # End Switch.
#endregion.
$TelemetryHashBonus
#endregion.