More PowerShell 7.2 and Azure Automation troubles…

Last week I had blogged about ExchangeOnlineManagement and Az module troubles with PowerShell 7.2. This week I ran into another issue as I moved more Runbooks over to PowerShell 7.2.

Some of them started failing for no reason. It happened when I’d do a Connect-PnPOnline to connect to a SharePoint site, and the error was: Host not reachable.

Such a weird one, coz if I try and connect to the site from the Hybrid Runbook Worker this Runbook runs on, I can connect to the site. Moreover, most of my Runbooks work fine – even though they all connect to the same site and run from the same HRW – just a few failed. This stumped me for a bit.

Then I realized the ones that fail were using this throttling function I had created. It basically checks if there’s another instance of the Runbook already running, and if so quits or waits. Hmm, why was that causing things to fail?

Yes the throttling function connects to Azure and does some stuff, but I was connecting to Azure in all the other runbooks anyway (to read Key Vaults and such) and that had no issue. Digging more, I realized the issue was with the Az.Resources module. The cmdlets used by that function make use of this module, and looks like that conflicts with PnP.PowerShell. Eugh.

Looks like this is fixed in the upcoming 2.3.0 release of PnP.PowerShell (still at 2.2.0 as of writing) – that doesn’t help me currently. I can’t update my production Runbooks to using nightly versions of the module just to fix this issue. I could, of course, remove the throttling function – which is what I did in the interim – but I wasn’t happy with that. I can’t have these Runbooks running concurrently.

An update on the throttling function

Last we met my throttling function it looked like this:

# This is a Function I created (from various Google results) to throttle a Runbook.
# It will either wait or quit the runbook. 
function Throttle-AzRunbook {
    param(
        [switch]$quitRatherThanWait,
        [int]$numberOfInstances = 1
    )

    # Connect to Azure. With a Managed Identity in this case as that's what I use. 
    # It's like I am already connectes but I can't assume that within this function. 
    # Must connect to Azure before running Get-AzAutomationJob or Get-AzResource
    try {
        # From https://docs.microsoft.com/en-us/azure/automation/enable-managed-identity-for-automation#authenticate-access-with-system-assigned-managed-identity
        # Ensures you do not inherit an AzContext in your runbook
        Disable-AzContextAutosave -Scope Process | Out-Null

        # Connect to Azure with system-assigned managed identity
        Connect-AzAccount -Identity | Out-Null

    } catch {
        Write-Error "Runbook could not connect to Azure: $($_.Exception.Message)"
        exit
    }

    # Get the Job ID from PSPrivateMetadata. That's the only thing it contains!
    $automationJobId = $PSPrivateMetadata.JobId.Guid

    # Get all Runbooks in the current subscription
    $allAutomationAccounts = Get-AzResource -ResourceType Microsoft.Automation/automationAccounts

    $automationAccountName = $null
    $resourceGroupName = $null
    $runbookName = $null

    foreach ($automationAccount in $allAutomationAccounts) {
        $runbookJob = Get-AzAutomationJob -AutomationAccountName $automationAccount.Name `
                                    -ResourceGroupName $automationAccount.ResourceGroupName `
                                    -Id $automationJobId `
                                    -ErrorAction SilentlyContinue

        if (!([string]::IsNullOrEmpty($runbookJob))) {
            $automationAccountName = $runbookJob.AutomationAccountName
            $resourceGroupName = $runbookJob.ResourceGroupName
            $runbookName = $runbookJob.RunbookName
        }
    }

    # At this point I'll have the Automation Account Name, Runbook Name, Job ID and Resource Group Name, 
    # Find all other active jobs of this Runbook.

    $allActiveJobs = Get-AzAutomationJob -AutomationAccountName $automationAccountName `
                                    -ResourceGroupName $resourceGroupName `
                                    -RunbookName $runbookName | 
                    Where-Object { ($_.Status -eq "Running") -or ($_.Status -eq "Starting") -or ($_.Status -eq "Queued")}  

    if ($quitRatherThanWait.IsPresent -and $allActiveJobs.Count -gt $numberOfInstances) {
        Write-Output "Exiting as another job is already running"
        exit

    } else {
        $oldestJob = $AllActiveJobs | Sort-Object -Property CreationTime  | Select-Object -First 1

        # If this job is not the oldest created job we will wait until the existing jobs complete or the number of jobs is less than numberOfInstances
        while (($AutomationJobID -ne $oldestJob.JobId) -and ($allActiveJobs.Count -ge $numberOfInstances)) {
            Write-Output "Waiting as there are currently running $($allActiveJobs.Count) active jobs for this runbook already. Sleeping 30 seconds..."
            Write-Output "Oldest Job is $($oldestJob.JobId)"
        
            Start-Sleep -Seconds 30
        
            $allActiveJobs = Get-AzAutomationJob -AutomationAccountName $automationAccountName `
                                        -ResourceGroupName $resourceGroupName `
                                        -RunbookName $runbookName | 
                        Where-Object { ($_.Status -eq "Running") -or ($_.Status -eq "Starting") -or ($_.Status -eq "Queued")}  
        
            $oldestJob = $allActiveJobs | Sort-Object -Property CreationTime | Select-Object -First 1
        } 
        
        Write-Output "Job can continue..."
    }
}

# This is a Function I created (from various Google results) to throttle a Runbook.

# It will either wait or quit the runbook.

function Throttle-AzRunbook {

param(

[switch]$quitRatherThanWait,

[int]$numberOfInstances = 1

)

# Connect to Azure. With a Managed Identity in this case as that's what I use.

# It's like I am already connectes but I can't assume that within this function.

# Must connect to Azure before running Get-AzAutomationJob or Get-AzResource

try {

# From https://docs.microsoft.com/en-us/azure/automation/enable-managed-identity-for-automation#authenticate-access-with-system-assigned-managed-identity

# Ensures you do not inherit an AzContext in your runbook

Disable-AzContextAutosave -Scope Process | Out-Null

# Connect to Azure with system-assigned managed identity

Connect-AzAccount -Identity | Out-Null

} catch {

Write-Error "Runbook could not connect to Azure: $($_.Exception.Message)"

exit

}

# Get the Job ID from PSPrivateMetadata. That's the only thing it contains!

$automationJobId = $PSPrivateMetadata.JobId.Guid

# Get all Runbooks in the current subscription

$allAutomationAccounts = Get-AzResource -ResourceType Microsoft.Automation/automationAccounts

$automationAccountName = $null

$resourceGroupName = $null

$runbookName = $null

foreach ($automationAccount in $allAutomationAccounts) {

$runbookJob = Get-AzAutomationJob -AutomationAccountName $automationAccount.Name `

-ResourceGroupName $automationAccount.ResourceGroupName `

-Id $automationJobId `

-ErrorAction SilentlyContinue

if (!([string]::IsNullOrEmpty($runbookJob))) {

$automationAccountName = $runbookJob.AutomationAccountName

$resourceGroupName = $runbookJob.ResourceGroupName

$runbookName = $runbookJob.RunbookName

}

# At this point I'll have the Automation Account Name, Runbook Name, Job ID and Resource Group Name,

# Find all other active jobs of this Runbook.

$allActiveJobs = Get-AzAutomationJob -AutomationAccountName $automationAccountName `

-ResourceGroupName $resourceGroupName `

-RunbookName $runbookName |

Where-Object { ($_.Status -eq "Running") -or ($_.Status -eq "Starting") -or ($_.Status -eq "Queued")}

if ($quitRatherThanWait.IsPresent -and $allActiveJobs.Count -gt $numberOfInstances) {

Write-Output "Exiting as another job is already running"

exit

} else {

$oldestJob = $AllActiveJobs | Sort-Object -Property CreationTime | Select-Object -First 1

# If this job is not the oldest created job we will wait until the existing jobs complete or the number of jobs is less than numberOfInstances

while (($AutomationJobID -ne $oldestJob.JobId) -and ($allActiveJobs.Count -ge $numberOfInstances)) {

Write-Output "Waiting as there are currently running $($allActiveJobs.Count) active jobs for this runbook already. Sleeping 30 seconds..."

Write-Output "Oldest Job is $($oldestJob.JobId)"

Start-Sleep -Seconds 30

$allActiveJobs = Get-AzAutomationJob -AutomationAccountName $automationAccountName `

-ResourceGroupName $resourceGroupName `

-RunbookName $runbookName |

Where-Object { ($_.Status -eq "Running") -or ($_.Status -eq "Starting") -or ($_.Status -eq "Queued")}

$oldestJob = $allActiveJobs | Sort-Object -Property CreationTime | Select-Object -First 1

}

Write-Output "Job can continue..."

}

Turns out this doesn’t work with PowerShell 7.2 and HRWs as the PSPrivateMetadata variable is not present in 7.2 + HRWs. (It is present in 5.x + HRWs and even 7.2 running on Azure – so it’s one of those things that will appear in the future I guess).

This means I can’t extract the JobId and use it to search other jobs. What can I do here? After some tinkering I realized I can cheat and extract the JobId from one of the trace log files. You see, every HRW Runbook writes to this path:

The highlighted bit varies per runbook.

The file there looks like this:

Orchestrator.Sandbox.Diagnostics Critical: 0 : [2023-09-26T09:46:51.8046468Z]  Starting sandbox process. [sandboxId=1a16c23f-90f5-46de-a7a8-213eed634246]
Orchestrator.Sandbox.Diagnostics Critical: 0 : [2023-09-26T09:46:52.0546465Z]  Hybrid Sandbox
Orchestrator.Sandbox.Diagnostics Critical: 0 : [2023-09-26T09:46:52.6485351Z]  First Trace Log.
Orchestrator.Sandbox.Diagnostics Critical: 0 : [2023-09-26T09:46:52.8984865Z]  Sandbox Recieving Job. [sandboxId=1a16c23f-90f5-46de-a7a8-213eed634246][jobId=b20c76be-b132-4679-84e2-17b244734f65]
Orchestrator.Sandbox.Diagnostics Critical: 0 : [2023-09-26T09:48:23.0674290Z]  Sandbox close request. The sandbox will exit immediately. [sandboxId=1a16c23f-90f5-46de-a7a8-213eed634246]
Orchestrator.Sandbox.Diagnostics Critical: 0 : [2023-09-26T09:48:23.0674290Z]  Leaving sandbox process. [sandboxId=1a16c23f-90f5-46de-a7a8-213eed634246]

Orchestrator.Sandbox.Diagnostics Critical: 0 : [2023-09-26T09:46:51.8046468Z] Starting sandbox process. [sandboxId=1a16c23f-90f5-46de-a7a8-213eed634246]

Orchestrator.Sandbox.Diagnostics Critical: 0 : [2023-09-26T09:46:52.0546465Z] Hybrid Sandbox

Orchestrator.Sandbox.Diagnostics Critical: 0 : [2023-09-26T09:46:52.6485351Z] First Trace Log.

Orchestrator.Sandbox.Diagnostics Critical: 0 : [2023-09-26T09:46:52.8984865Z] Sandbox Recieving Job. [sandboxId=1a16c23f-90f5-46de-a7a8-213eed634246][jobId=b20c76be-b132-4679-84e2-17b244734f65]

Orchestrator.Sandbox.Diagnostics Critical: 0 : [2023-09-26T09:48:23.0674290Z] Sandbox close request. The sandbox will exit immediately. [sandboxId=1a16c23f-90f5-46de-a7a8-213eed634246]

Orchestrator.Sandbox.Diagnostics Critical: 0 : [2023-09-26T09:48:23.0674290Z] Leaving sandbox process. [sandboxId=1a16c23f-90f5-46de-a7a8-213eed634246]

Neat, so line 4 has the JobId.

What can I do to find this path to this file? Turns out $PSScriptRoot has it. Split its path to get the parent, tack on "\diags\trace.log" and that’s my file. I can essentially do something like this to get the Id if it’s not found:

if (!$automationJobId) {
    Write-Output "Unable to find JobID from PSPrivateMetadata"
    if ($PWD -match "HybridWorker") {
        Write-Output "Trying a workaround to find JobID as this is an HRW"

        $parentPath =  Split-Path -Parent $PSScriptRoot
        $fullPath =  $parentPath + "\diags\trace.log"

        try {
            $automationJobId = ((Get-Content $fullPath -ErrorAction Stop | Select-String "jobId") -split 'jobId=')[1] -replace ']',''

        } catch {
            $automationJobId = $null
        }
    }
}

if (!$automationJobId) {

Write-Output "Unable to find JobID from PSPrivateMetadata"

if ($PWD -match "HybridWorker") {

Write-Output "Trying a workaround to find JobID as this is an HRW"

$parentPath = Split-Path -Parent $PSScriptRoot

$fullPath = $parentPath + "\diags\trace.log"

try {

$automationJobId = ((Get-Content $fullPath -ErrorAction Stop | Select-String "jobId") -split 'jobId=')[1] -replace ']',''

} catch {

$automationJobId = $null

}

With this in hand my throttling function now looks like this:

function Throttle-AzRunbook {
    param(
        [switch]$quitRatherThanWait,
        [int]$numberOfInstances = 1
    )

    # Connect to Azure. With a Managed Identity in this case as that's what I use. 
    # It's like I am already connectes but I can't assume that within this function. 
    # Must connect to Azure before running Get-AzAutomationJob or Get-AzResource
    try {
        # From https://docs.microsoft.com/en-us/azure/automation/enable-managed-identity-for-automation#authenticate-access-with-system-assigned-managed-identity
        # Ensures you do not inherit an AzContext in your runbook
        Disable-AzContextAutosave -Scope Process | Out-Null

        # Connect to Azure with system-assigned managed identity
        Connect-AzAccount -Identity | Out-Null

    } catch {
        Write-Error "Runbook could not connect to Azure: $($_.Exception.Message)"
        exit
    }

    # Get the Job ID from PSPrivateMetadata. That's the only thing it contains!
    $automationJobId = $PSPrivateMetadata.JobId.Guid

    # A workaround for PowerShell 7.x and HRW where $PSPrivateMetadata is missing
    # I extract it from the trace.log file instead
    if (!$automationJobId) {
        Write-Output "Unable to find JobID from PSPrivateMetadata"
        if ($PWD -match "HybridWorker") {
            Write-Output "Trying a workaround to find JobID as this is an HRW"

            $parentPath =  Split-Path -Parent $PSScriptRoot
            $fullPath =  $parentPath + "\diags\trace.log"
    
            try {
                $automationJobId = ((Get-Content $fullPath -ErrorAction Stop | Select-String "jobId") -split 'jobId=')[1] -replace ']',''

            } catch {
                $automationJobId = $null
            }
        }
    }

    if ($automationJobId) {
        Write-Output "JobID is $automationJobId"
        # Get all Runbooks in the current subscription
        $allAutomationAccounts = Get-AzResource -ResourceType Microsoft.Automation/automationAccounts

        $automationAccountName = $null
        $resourceGroupName = $null
        $runbookName = $null

        foreach ($automationAccount in $allAutomationAccounts) {
            $runbookJobParams = @{
                "AutomationAccountName" = $automationAccount.Name
                "ResourceGroupName" = $automationAccount.ResourceGroupName
                "Id" = $automationJobId
                "ErrorAction" = "SilentlyContinue"
            }

            $runbookJob = Get-AzAutomationJob @runbookJobParams

            if (!([string]::IsNullOrEmpty($runbookJob))) {
                $automationAccountName = $runbookJob.AutomationAccountName
                $resourceGroupName = $runbookJob.ResourceGroupName
                $runbookName = $runbookJob.RunbookName
            }
        }

        # At this point I'll have the Automation Account Name, Runbook Name, Job ID and Resource Group Name, 
        # Find all other active jobs of this Runbook.
        $runbookJobParams = @{
            "AutomationAccountName" = $automationAccountName
            "ResourceGroupName" = $resourceGroupName
            "RunbookName" = $runbookName
            "ErrorAction" = "SilentlyContinue"
        }

        $allActiveJobs = Get-AzAutomationJob @runbookJobParams | Where-Object { ($_.Status -eq "Running") -or ($_.Status -eq "Starting") -or ($_.Status -eq "Queued") -or ($_.Status -eq "Activating") -or ($_.Status -eq "Resuming") }

        if ($allActiveJobs.Count -gt $numberOfInstances) {
            if ($quitRatherThanWait.IsPresent) {
                Write-Output "Exiting as another job is already running"
                exit
    
            } else {
                $oldestJob = $AllActiveJobs | Sort-Object -Property CreationTime | Select-Object -First 1
    
                # If this job is not the oldest created job we will wait until the existing jobs complete or the number of jobs is less than numberOfInstances
                while (($AutomationJobID -ne $oldestJob.JobId) -and ($allActiveJobs.Count -ge $numberOfInstances)) {
                    Write-Output "Waiting as there are currently running $($allActiveJobs.Count) active jobs for this runbook already. Sleeping 30 seconds..."
                    Write-Output "Oldest Job is $($oldestJob.JobId)"
                
                    Start-Sleep -Seconds 30
                
                    $allActiveJobs = Get-AzAutomationJob @runbookJobParams | Where-Object { ($_.Status -eq "Running") -or ($_.Status -eq "Starting") -or ($_.Status -eq "Queued") -or ($_.Status -eq "Activating") -or ($_.Status -eq "Resuming") }
                    $oldestJob = $allActiveJobs | Sort-Object -Property CreationTime | Select-Object -First 1
                } 
                
                Write-Output "Job can continue..."
            }

        } else {
            Write-Output "No other concurrent jobs found..."
        }
        
    } else {
        Write-Warning "Unable to find JobID. Poceeding with job, this might result in concurrent executions"
        if ($PSVersionTable.PSVersion.Major -eq 7 -and $PWD -match "HybridWorker") {
            Write-Output "This is PowerShell 7.x in HRW - that explains it!"
        }
    }
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

function Throttle-AzRunbook {

param(

[switch]$quitRatherThanWait,

[int]$numberOfInstances = 1

)

# Connect to Azure. With a Managed Identity in this case as that's what I use.

# It's like I am already connectes but I can't assume that within this function.

# Must connect to Azure before running Get-AzAutomationJob or Get-AzResource

try {

# From https://docs.microsoft.com/en-us/azure/automation/enable-managed-identity-for-automation#authenticate-access-with-system-assigned-managed-identity

# Ensures you do not inherit an AzContext in your runbook

Disable-AzContextAutosave -Scope Process | Out-Null

# Connect to Azure with system-assigned managed identity

Connect-AzAccount -Identity | Out-Null

} catch {

Write-Error "Runbook could not connect to Azure: $($_.Exception.Message)"

exit

}

# Get the Job ID from PSPrivateMetadata. That's the only thing it contains!

$automationJobId = $PSPrivateMetadata.JobId.Guid

# A workaround for PowerShell 7.x and HRW where $PSPrivateMetadata is missing

# I extract it from the trace.log file instead

if (!$automationJobId) {

Write-Output "Unable to find JobID from PSPrivateMetadata"

if ($PWD -match "HybridWorker") {

Write-Output "Trying a workaround to find JobID as this is an HRW"

$parentPath = Split-Path -Parent $PSScriptRoot

$fullPath = $parentPath + "\diags\trace.log"

try {

$automationJobId = ((Get-Content $fullPath -ErrorAction Stop | Select-String "jobId") -split 'jobId=')[1] -replace ']',''

} catch {

$automationJobId = $null

}

if ($automationJobId) {

Write-Output "JobID is $automationJobId"

# Get all Runbooks in the current subscription

$allAutomationAccounts = Get-AzResource -ResourceType Microsoft.Automation/automationAccounts

$automationAccountName = $null

$resourceGroupName = $null

$runbookName = $null

foreach ($automationAccount in $allAutomationAccounts) {

$runbookJobParams = @{

"AutomationAccountName" = $automationAccount.Name

"ResourceGroupName" = $automationAccount.ResourceGroupName

"Id" = $automationJobId

"ErrorAction" = "SilentlyContinue"

}

$runbookJob = Get-AzAutomationJob @runbookJobParams

if (!([string]::IsNullOrEmpty($runbookJob))) {

$automationAccountName = $runbookJob.AutomationAccountName

$resourceGroupName = $runbookJob.ResourceGroupName

$runbookName = $runbookJob.RunbookName

}

# At this point I'll have the Automation Account Name, Runbook Name, Job ID and Resource Group Name,

# Find all other active jobs of this Runbook.

$runbookJobParams = @{

"AutomationAccountName" = $automationAccountName

"ResourceGroupName" = $resourceGroupName

"RunbookName" = $runbookName

"ErrorAction" = "SilentlyContinue"

}

$allActiveJobs = Get-AzAutomationJob @runbookJobParams | Where-Object { ($_.Status -eq "Running") -or ($_.Status -eq "Starting") -or ($_.Status -eq "Queued") -or ($_.Status -eq "Activating") -or ($_.Status -eq "Resuming") }

if ($allActiveJobs.Count -gt $numberOfInstances) {

if ($quitRatherThanWait.IsPresent) {

Write-Output "Exiting as another job is already running"

exit

} else {

$oldestJob = $AllActiveJobs | Sort-Object -Property CreationTime | Select-Object -First 1

# If this job is not the oldest created job we will wait until the existing jobs complete or the number of jobs is less than numberOfInstances

while (($AutomationJobID -ne $oldestJob.JobId) -and ($allActiveJobs.Count -ge $numberOfInstances)) {

Write-Output "Waiting as there are currently running $($allActiveJobs.Count) active jobs for this runbook already. Sleeping 30 seconds..."

Write-Output "Oldest Job is $($oldestJob.JobId)"

Start-Sleep -Seconds 30

$oldestJob = $allActiveJobs | Sort-Object -Property CreationTime | Select-Object -First 1

}

Write-Output "Job can continue..."

}

} else {

Write-Output "No other concurrent jobs found..."

}

} else {

Write-Warning "Unable to find JobID. Poceeding with job, this might result in concurrent executions"

if ($PSVersionTable.PSVersion.Major -eq 7 -and $PWD -match "HybridWorker") {

Write-Output "This is PowerShell 7.x in HRW - that explains it!"

}

Getting PnP PowerShell working with this

Ok, so what can I do to fix PnP PowerShell? Can’t I just unload the Az.Resources module after its done? Yes, I can (Remove-Module) but that doesn’t unload any of the loaded assemblies, and since those are usually the source of conflict Remove-Module can’t help us.

What can I do regarding assemblies? In my previous post I had alluded to this very informative article from Microsoft. It suggests three ways to work around this issue:

Start PowerShell as a sub-process – I didn’t try that, wasn’t sure if it would work
Use the job system – this is what I tried
Use PowerShell remoting – won’t work with Runbooks

With the job system you start the function as a separate job basically. And since it runs independent of the main script, the modules & assemblies it loads too are independent. When the job exits these are removed. Awesome!

Tyically the solution is simple:

$result = Start-Job { Invoke-ConflictingCommand } | Receive-Job -Wait

1	$result = Start-Job { Invoke-ConflictingCommand } \| Receive-Job -Wait

In my case this is a function. How the heck do I get that in there? I could of course define the function within the Start-Job, but I don’t want that. I want to keep my code consistent across Runbooks. Thanks to a helpful StackOverflow post I learnt I can do the following:

function FOO { "HEY" }

Start-Job -ScriptBlock { 

  # Redefine function FOO in the context of this job.
  $function:FOO = "$using:function:FOO" 
  
  # Now FOO can be invoked.
  FOO

} | Receive-Job -Wait -AutoRemoveJob

function FOO { "HEY" }

Start-Job -ScriptBlock {

# Redefine function FOO in the context of this job.

$function:FOO = "$using:function:FOO"

# Now FOO can be invoked.

FOO

} | Receive-Job -Wait -AutoRemoveJob

So all I have to do is:

Start-Job {
    ${function:Throttle-AzRunbook} = "${using:function:Throttle-AzRunbook}"

    Throttle-AzRunbook -ScriptRoot $ScriptRoot

} | Receive-Job -Wait -AutoRemoveJob

Start-Job {

${function:Throttle-AzRunbook} = "${using:function:Throttle-AzRunbook}"

Throttle-AzRunbook -ScriptRoot $ScriptRoot

} | Receive-Job -Wait -AutoRemoveJob

I need to use the curly braces because of the dash in the name, else it complains.

I didn’t know of this function name space. That’s useful.

Two issues with this.

One: my function complains that it can’t find PSScriptRoot any more. Apparently that’s how it is. So I modified the function to take this as an input parameter:

function Throttle-AzRunbook {
    param(
        [Parameter(Mandatory=$false)]
        [switch]$quitRatherThanWait,

        [Parameter(Mandatory=$false)]
        [int]$numberOfInstances = 1,

        # If invoked from Start-Job pass the $PSScriptRoot as $ScriptRoot
        [Parameter(Mandatory=$false)]
        [string]$ScriptRoot
    )

    # Connect to Azure. With a Managed Identity in this case as that's what I use. 
    # I could be already connected but I can't assume that within this function. 
    # Must connect to Azure before running Get-AzAutomationJob or Get-AzResource
    # Note that this loads the Az.Resources module.
    try {
        # From https://docs.microsoft.com/en-us/azure/automation/enable-managed-identity-for-automation#authenticate-access-with-system-assigned-managed-identity
        # Ensures you do not inherit an AzContext in your runbook
        Disable-AzContextAutosave -Scope Process | Out-Null

        # Connect to Azure with system-assigned managed identity
        Connect-AzAccount -Identity | Out-Null

    } catch {
        Write-Error "Runbook could not connect to Azure: $($_.Exception.Message)"
        exit
    }

    Write-Output "Checking whether to throttle..."
    # Get the Job ID from PSPrivateMetadata. That's the only thing it contains!
    $automationJobId = $PSPrivateMetadata.JobId.Guid

    # A workaround for PowerShell 7.x and HRW where $PSPrivateMetadata is missing
    # I extract it from the trace.log file instead
    if (!$automationJobId) {
        Write-Output "Unable to find JobID from PSPrivateMetadata"
        if ($PWD -match "HybridWorker") {
            Write-Output "Trying a workaround to find JobID as this is an HRW"

            if ($ScriptRoot) {
                $parentPath =  Split-Path -Parent $ScriptRoot

            } elseif ($PSScriptRoot) {
                $parentPath =  Split-Path -Parent $PSScriptRoot

            } else {
                $parentPath = $null
            }

            if ($parentPath) {
                $fullPath =  $parentPath + "\diags\trace.log"
        
                try {
                    $automationJobId = ((Get-Content $fullPath -ErrorAction Stop | Select-String "jobId") -split 'jobId=')[1] -replace ']',''
    
                } catch {
                    $automationJobId = $null
                }
            }
        }
    }

    if ($automationJobId) {
        Write-Output "JobID is $automationJobId"
        # Get all Runbooks in the current subscription
        $allAutomationAccounts = Get-AzResource -ResourceType Microsoft.Automation/automationAccounts

        $automationAccountName = $null
        $resourceGroupName = $null
        $runbookName = $null

        foreach ($automationAccount in $allAutomationAccounts) {
            $runbookJobParams = @{
                "AutomationAccountName" = $automationAccount.Name
                "ResourceGroupName" = $automationAccount.ResourceGroupName
                "Id" = $automationJobId
                "ErrorAction" = "SilentlyContinue"
            }

            $runbookJob = Get-AzAutomationJob @runbookJobParams

            if (!([string]::IsNullOrEmpty($runbookJob))) {
                $automationAccountName = $runbookJob.AutomationAccountName
                $resourceGroupName = $runbookJob.ResourceGroupName
                $runbookName = $runbookJob.RunbookName
            }
        }

        # At this point I'll have the Automation Account Name, Runbook Name, Job ID and Resource Group Name, 
        # Find all other active jobs of this Runbook.
        $runbookJobParams = @{
            "AutomationAccountName" = $automationAccountName
            "ResourceGroupName" = $resourceGroupName
            "RunbookName" = $runbookName
            "ErrorAction" = "SilentlyContinue"
        }

        $allActiveJobs = Get-AzAutomationJob @runbookJobParams | Where-Object { ($_.Status -eq "Running") -or ($_.Status -eq "Starting") -or ($_.Status -eq "Queued") -or ($_.Status -eq "Activating") -or ($_.Status -eq "Resuming") }

        if ($allActiveJobs.Count -gt $numberOfInstances) {
            if ($quitRatherThanWait.IsPresent) {
                Write-Output "Exiting as another job is already running"
                exit
    
            } else {
                $oldestJob = $AllActiveJobs | Sort-Object -Property CreationTime | Select-Object -First 1
    
                # If this job is not the oldest created job we will wait until the existing jobs complete or the number of jobs is less than numberOfInstances
                while (($AutomationJobID -ne $oldestJob.JobId) -and ($allActiveJobs.Count -ge $numberOfInstances)) {
                    Write-Output "Waiting as there are currently running $($allActiveJobs.Count) active jobs for this runbook already. Sleeping 30 seconds..."
                    Write-Output "Oldest Job is $($oldestJob.JobId)"
                
                    Start-Sleep -Seconds 30
                
                    $allActiveJobs = Get-AzAutomationJob @runbookJobParams | Where-Object { ($_.Status -eq "Running") -or ($_.Status -eq "Starting") -or ($_.Status -eq "Queued") -or ($_.Status -eq "Activating") -or ($_.Status -eq "Resuming") }
                    $oldestJob = $allActiveJobs | Sort-Object -Property CreationTime | Select-Object -First 1
                } 
                
                Write-Output "Job can continue..."
            }

        } else {
            Write-Output "No other concurrent jobs found..."
        }
        
    } else {
        Write-Warning "Unable to find JobID. Poceeding with job, this might result in concurrent executions"
        if ($PSVersionTable.PSVersion.Major -eq 7 -and $PWD -match "HybridWorker") {
            Write-Output "This is PowerShell 7.x in HRW - that explains it!"
        }
    }
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

function Throttle-AzRunbook {

param(

[Parameter(Mandatory=$false)]

[switch]$quitRatherThanWait,

[Parameter(Mandatory=$false)]

[int]$numberOfInstances = 1,

# If invoked from Start-Job pass the $PSScriptRoot as $ScriptRoot

[Parameter(Mandatory=$false)]

[string]$ScriptRoot

)

# Connect to Azure. With a Managed Identity in this case as that's what I use.

# I could be already connected but I can't assume that within this function.

# Must connect to Azure before running Get-AzAutomationJob or Get-AzResource

# Note that this loads the Az.Resources module.

try {

# From https://docs.microsoft.com/en-us/azure/automation/enable-managed-identity-for-automation#authenticate-access-with-system-assigned-managed-identity

# Ensures you do not inherit an AzContext in your runbook

Disable-AzContextAutosave -Scope Process | Out-Null

# Connect to Azure with system-assigned managed identity

Connect-AzAccount -Identity | Out-Null

} catch {

Write-Error "Runbook could not connect to Azure: $($_.Exception.Message)"

exit

}

Write-Output "Checking whether to throttle..."

# Get the Job ID from PSPrivateMetadata. That's the only thing it contains!

$automationJobId = $PSPrivateMetadata.JobId.Guid

# A workaround for PowerShell 7.x and HRW where $PSPrivateMetadata is missing

# I extract it from the trace.log file instead

if (!$automationJobId) {

Write-Output "Unable to find JobID from PSPrivateMetadata"

if ($PWD -match "HybridWorker") {

Write-Output "Trying a workaround to find JobID as this is an HRW"

if ($ScriptRoot) {

$parentPath = Split-Path -Parent $ScriptRoot

} elseif ($PSScriptRoot) {

$parentPath = Split-Path -Parent $PSScriptRoot

} else {

$parentPath = $null

}

if ($parentPath) {

$fullPath = $parentPath + "\diags\trace.log"

try {

$automationJobId = ((Get-Content $fullPath -ErrorAction Stop | Select-String "jobId") -split 'jobId=')[1] -replace ']',''

} catch {

$automationJobId = $null

}

if ($automationJobId) {

Write-Output "JobID is $automationJobId"

# Get all Runbooks in the current subscription

$allAutomationAccounts = Get-AzResource -ResourceType Microsoft.Automation/automationAccounts

$automationAccountName = $null

$resourceGroupName = $null

$runbookName = $null

foreach ($automationAccount in $allAutomationAccounts) {

$runbookJobParams = @{

"AutomationAccountName" = $automationAccount.Name

"ResourceGroupName" = $automationAccount.ResourceGroupName

"Id" = $automationJobId

"ErrorAction" = "SilentlyContinue"

}

$runbookJob = Get-AzAutomationJob @runbookJobParams

if (!([string]::IsNullOrEmpty($runbookJob))) {

$automationAccountName = $runbookJob.AutomationAccountName

$resourceGroupName = $runbookJob.ResourceGroupName

$runbookName = $runbookJob.RunbookName

}

# At this point I'll have the Automation Account Name, Runbook Name, Job ID and Resource Group Name,

# Find all other active jobs of this Runbook.

$runbookJobParams = @{

"AutomationAccountName" = $automationAccountName

"ResourceGroupName" = $resourceGroupName

"RunbookName" = $runbookName

"ErrorAction" = "SilentlyContinue"

}

if ($allActiveJobs.Count -gt $numberOfInstances) {

if ($quitRatherThanWait.IsPresent) {

Write-Output "Exiting as another job is already running"

exit

} else {

$oldestJob = $AllActiveJobs | Sort-Object -Property CreationTime | Select-Object -First 1

# If this job is not the oldest created job we will wait until the existing jobs complete or the number of jobs is less than numberOfInstances

while (($AutomationJobID -ne $oldestJob.JobId) -and ($allActiveJobs.Count -ge $numberOfInstances)) {

Write-Output "Waiting as there are currently running $($allActiveJobs.Count) active jobs for this runbook already. Sleeping 30 seconds..."

Write-Output "Oldest Job is $($oldestJob.JobId)"

Start-Sleep -Seconds 30

$oldestJob = $allActiveJobs | Sort-Object -Property CreationTime | Select-Object -First 1

}

Write-Output "Job can continue..."

}

} else {

Write-Output "No other concurrent jobs found..."

}

} else {

Write-Warning "Unable to find JobID. Poceeding with job, this might result in concurrent executions"

if ($PSVersionTable.PSVersion.Major -eq 7 -and $PWD -match "HybridWorker") {

Write-Output "This is PowerShell 7.x in HRW - that explains it!"

}

And I will pass that as an input.

The second issue was that none of the Write-Output output from the function was appearing. I got it working by changing things a bit so here’s what my Start-Job looks like now (this includes the change to pass PSScriptRoot to the function; I make use of $using for that):

$job = Start-Job {
    ${function:Throttle-AzRunbook} = "${using:function:Throttle-AzRunbook}"
    $ScriptRoot = $using:PSScriptRoot
    Throttle-AzRunbook -ScriptRoot $ScriptRoot
}

Receive-Job -Wait $job

$job = Start-Job {

${function:Throttle-AzRunbook} = "${using:function:Throttle-AzRunbook}"

$ScriptRoot = $using:PSScriptRoot

Throttle-AzRunbook -ScriptRoot $ScriptRoot

}

Receive-Job -Wait $job

For some reason having Receive-Job separately got it to show the output.

And that’s it! Now I have throttling working with PowerShell 7.2 and HRWs. I also hopefully know how to tackle any further conflicts between these various modules.

Update (22nd July 2024): Looks like I don’t need the above workaround for PowerShell 7.2. I wrote a new blog post.