{"id":6998,"date":"2023-03-04T08:13:48","date_gmt":"2023-03-04T08:13:48","guid":{"rendered":"https:\/\/rakhesh.com\/?p=6998"},"modified":"2023-09-29T11:46:51","modified_gmt":"2023-09-29T10:46:51","slug":"throttling-concurrent-azure-automation-runbooks","status":"publish","type":"post","link":"https:\/\/rakhesh.com\/azure\/throttling-concurrent-azure-automation-runbooks\/","title":{"rendered":"Throttling concurrent Azure Automation Runbooks"},"content":{"rendered":"

I don’t take any credit for this post.<\/p>\n

It’s mostly stuff I learnt from Google searches, and a script I put together for my own use from other places. I should have credited the sources, but all this was put together over time and today I had to do it again and that’s when I went through my past snippets etc. and created something in a formal way.<\/p>\n

Anyways. Here’s the issue. I have a bunch of Azure Automation Runbooks that run on a schedule. Thing is, sometimes a Runbook takes longer than expected and so while it is running another one launches. This leads to both of them slowing down or conflicting with each others tasks and soon I have a whole bunch of Runbooks launching and making a mess of things. I need a way of ensuring only one Runbook is active at a time.<\/p>\n

There’s a variable within Runbooks called PSPrivateMetadata<\/code>. It doesn’t have anything except the JobId. You can extract this via $PSPrivateMetadata.JobId.Guid<\/code>. Then you must search all running jobs to find the Automation Account name, Runbook name, and Resource Group name of this Runbook; and using that find what other jobs are running. And then decide what to do – wait for quit. Simple really. Wish the variable had the various names so I can save some trouble finding them… but oh well.<\/p>\n

Here’s what I came up with:<\/p>\n

# This is a Function I created (from various Google results) to throttle a Runbook.\r\n# It will either wait or quit the runbook. \r\nfunction Throttle-AzRunbook {\r\n    param(\r\n        [switch]$quitRatherThanWait,\r\n        [int]$numberOfInstances = 1\r\n    )\r\n\r\n    # Connect to Azure. With a Managed Identity in this case as that's what I use. \r\n    # It's like I am already connectes but I can't assume that within this function. \r\n    # Must connect to Azure before running Get-AzAutomationJob or Get-AzResource\r\n    try {\r\n        # From https:\/\/docs.microsoft.com\/en-us\/azure\/automation\/enable-managed-identity-for-automation#authenticate-access-with-system-assigned-managed-identity\r\n        # Ensures you do not inherit an AzContext in your runbook\r\n        Disable-AzContextAutosave -Scope Process | Out-Null\r\n\r\n        # Connect to Azure with system-assigned managed identity\r\n        Connect-AzAccount -Identity | Out-Null\r\n\r\n    } catch {\r\n        Write-Error \"Runbook could not connect to Azure: $($_.Exception.Message)\"\r\n        exit\r\n    }\r\n\r\n    # Get the Job ID from PSPrivateMetadata. That's the only thing it contains!\r\n    $automationJobId = $PSPrivateMetadata.JobId.Guid\r\n\r\n    # Get all Runbooks in the current subscription\r\n    $allAutomationAccounts = Get-AzResource -ResourceType Microsoft.Automation\/automationAccounts\r\n\r\n    $automationAccountName = $null\r\n    $resourceGroupName = $null\r\n    $runbookName = $null\r\n\r\n    foreach ($automationAccount in $allAutomationAccounts) {\r\n        $runbookJob = Get-AzAutomationJob -AutomationAccountName $automationAccount.Name `\r\n                                    -ResourceGroupName $automationAccount.ResourceGroupName `\r\n                                    -Id $automationJobId `\r\n                                    -ErrorAction SilentlyContinue\r\n\r\n        if (!([string]::IsNullOrEmpty($runbookJob))) {\r\n            $automationAccountName = $runbookJob.AutomationAccountName\r\n            $resourceGroupName = $runbookJob.ResourceGroupName\r\n            $runbookName = $runbookJob.RunbookName\r\n        }\r\n    }\r\n\r\n    # At this point I'll have the Automation Account Name, Runbook Name, Job ID and Resource Group Name, \r\n    # Find all other active jobs of this Runbook.\r\n\r\n    $allActiveJobs = Get-AzAutomationJob -AutomationAccountName $automationAccountName `\r\n                                    -ResourceGroupName $resourceGroupName `\r\n                                    -RunbookName $runbookName | \r\n                    Where-Object { ($_.Status -eq \"Running\") -or ($_.Status -eq \"Starting\") -or ($_.Status -eq \"Queued\")}  \r\n\r\n    if ($quitRatherThanWait.IsPresent -and $allActiveJobs.Count -gt $numberOfInstances) {\r\n        Write-Output \"Exiting as another job is already running\"\r\n        exit\r\n\r\n    } else {\r\n        $oldestJob = $AllActiveJobs | Sort-Object -Property CreationTime  | Select-Object -First 1\r\n\r\n        # If this job is not the oldest created job we will wait until the existing jobs complete or the number of jobs is less than numberOfInstances\r\n        while (($AutomationJobID -ne $oldestJob.JobId) -and ($allActiveJobs.Count -ge $numberOfInstances)) {\r\n            Write-Output \"Waiting as there are currently running $($allActiveJobs.Count) active jobs for this runbook already. Sleeping 30 seconds...\"\r\n            Write-Output \"Oldest Job is $($oldestJob.JobId)\"\r\n        \r\n            Start-Sleep -Seconds 30\r\n        \r\n            $allActiveJobs = Get-AzAutomationJob -AutomationAccountName $automationAccountName `\r\n                                        -ResourceGroupName $resourceGroupName `\r\n                                        -RunbookName $runbookName | \r\n                        Where-Object { ($_.Status -eq \"Running\") -or ($_.Status -eq \"Starting\") -or ($_.Status -eq \"Queued\")}  \r\n        \r\n            $oldestJob = $allActiveJobs | Sort-Object -Property CreationTime | Select-Object -First 1\r\n        } \r\n        \r\n        Write-Output \"Job can continue...\"\r\n    }\r\n}<\/pre>\n
Will update this in case of any bugs later.<\/p>\n
I put a line like this before my main script in the Runbook:<\/p>\n
# === [ Main Script ] ===\r\n# Quit if another instance is active\r\nThrottle-AzRunbook -quitRatherThanWait<\/pre>\n
And that’s it.<\/p>\n
Update (17th July 2023)<\/strong>: Looks like PowerShell 7.1 and 7.2 Runbooks dont have the $PSPrivateMetadata<\/code> variable. Here’s the output of Get-Variable<\/code> on a PowerShell 7.2 HRW:<\/p>\n
\"\"<\/p>\n
But PowerShell 5.1 has it.<\/p>\n
\"\"<\/p>\n
So the above function won’t work on 7.x for now.<\/p>\n
Update (2nd Sept 2023)<\/strong>: Noticed that $PSPrivateMetadata<\/code> is now available in PowerShell 7.2 (must have been for a while, I just noticed today).<\/p>\n
Update (26th Sept 2023)<\/strong>: It is however missing in Hybrid Runbook Workers!<\/p>\n
Update (29th Sept 2023)<\/strong>: See also this post<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
I don’t take any credit for this post. It’s mostly stuff I learnt from Google searches, and a script I put together for my own use from other places. I should have credited the sources, but all this was put together over time and today I had to do it again and that’s when I … Continue reading Throttling concurrent Azure Automation Runbooks<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[887],"tags":[383,227,1047,1044],"jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts\/6998"}],"collection":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/comments?post=6998"}],"version-history":[{"count":5,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts\/6998\/revisions"}],"predecessor-version":[{"id":7402,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts\/6998\/revisions\/7402"}],"wp:attachment":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/media?parent=6998"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/categories?post=6998"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/tags?post=6998"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}