Was stuck with an irritating problem today. Installed PowerShell 7.4 on one of my Hybrid Runbook Workers (HRW) but the Runbooks refuse to run using it! Looking at the logs I see entries like this:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
Orchestrator.Sandbox.Diagnostics Critical: 0 : [2023-12-20T15:05:20.4533023Z] Starting sandbox process. [sandboxId=ecdbfee3-902a-44c8-b936-d51c8180190c] Orchestrator.Sandbox.Diagnostics Critical: 0 : [2023-12-20T15:05:22.4845620Z] Hybrid Sandbox Orchestrator.Sandbox.Diagnostics Critical: 0 : [2023-12-20T15:05:23.1564394Z] First Trace Log. Orchestrator.Sandbox.Diagnostics Critical: 0 : [2023-12-20T15:05:23.4220674Z] Sandbox Recieving Job. [sandboxId=ecdbfee3-902a-44c8-b936-d51c8180190c][jobId=0dfa6479-ffca-4f40-a54e-ee353f57734a] Orchestrator.Sandbox.Diagnostics Critical: 0 : [2023-12-20T15:05:23.6251949Z] An unhandled exception was encountered while handling the job action. The sandbox will terminate immediately. [jobId=0dfa6479-ffca-4f40-a54e-ee353f57734a][source=Maintainer][exceptionMessage=System.InvalidOperationException: Runbook type '17' not supported. at Orchestrator.Runtime.Account.CreateRunbook(RunbookKey runbookKey, RunbookData runbookData) in C:\__w\1\s\src\Shared\Orchestrator.Runtime\Account.cs:line 336 at Orchestrator.Runtime.Account.DefaultLoadRunbook(CompositeKey`2 compositeKey) in C:\__w\1\s\src\Shared\Orchestrator.Runtime\Account.cs:line 233 at Orchestrator.Shared.Shared.CacheWithLocking`2.LoadAndAddValue(TKey key) in C:\__w\1\s\src\Shared\Orchestrator.Shared\Shared\CacheWithLocking.cs:line 609 at Orchestrator.Shared.Shared.CacheWithLocking`2.GetEntry(TKey key) in C:\__w\1\s\src\Shared\Orchestrator.Shared\Shared\CacheWithLocking.cs:line 582 at Orchestrator.Shared.Shared.CacheWithLocking`2.ActWithGuardedValue[TResult](TKey key, Func`2 action) in C:\__w\1\s\src\Shared\Orchestrator.Shared\Shared\CacheWithLocking.cs:line 352 at Orchestrator.Sandbox.SandboxJobActionHandler.HandleJobActionWithGuardedAccount(Account account, JobData jobData, JobMessageSource source) in C:\__w\1\s\src\prod\Orchestrator.Sandbox\SandboxJobActionHandler.cs:line 605 at Orchestrator.Shared.Shared.CacheWithLocking`2.<>c__DisplayClass30_0.<ActWithGuardedValue>b__0(TValue value) in C:\__w\1\s\src\Shared\Orchestrator.Shared\Shared\CacheWithLocking.cs:line 319 at Orchestrator.Shared.Shared.RefCounted`1.UsingValue[TResult](Func`2 action) in C:\__w\1\s\src\Shared\Orchestrator.Shared\Shared\RefCounted.cs:line 153 at Orchestrator.Shared.Shared.CacheWithLocking`2.ActWithGuardedValue[TResult](TKey key, Func`2 action) in C:\__w\1\s\src\Shared\Orchestrator.Shared\Shared\CacheWithLocking.cs:line 352 at Orchestrator.Sandbox.SandboxJobActionHandler.HandleJobActionSync(JobData jobData, JobMessageSource source) in C:\__w\1\s\src\prod\Orchestrator.Sandbox\SandboxJobActionHandler.cs:line 329 at Orchestrator.Sandbox.SandboxJobActionHandler.HandleJobActionThread(JobData jobData, JobMessageSource source) in C:\__w\1\s\src\prod\Orchestrator.Sandbox\SandboxJobActionHandler.cs:line 680][sandboxId=ecdbfee3-902a-44c8-b936-d51c8180190c] |
Tried rebooting, updating the OS, downgrading it to PowerShell 7.2 just in case… nothing helped. Double checked the steps in the docs (there isn’t much) and all looked fine. But the Runbooks just wouldn’t run. They’d stay “Queued” with lot of logs like above being generated, and eventually move to a “Suspended” state.
I had two automation accounts, one of them linked to two HRWs; another to one HRW. Both had the same issue.
Luckily I also had another automation account + HRW where things worked, so I compared the two and noticed that under the VM in Azure > “Extensions + applications” there’s a difference:
Version was 1.1.12.
While on the problem HRW it was:
Older version. Same on all three non-working HRWs! There were some of my older HRWs so I must have been on a preview version or something which never updated.
So I enabled automatic upgrade. Then waited… but that didn’t update anything. Googled, and came across this doc. Looks like the updates only happen for minor versions, while I need to jump from 0.x to 1.x.
Btw, good instructions on how to enable/ disable automatic upgrades when needed.
|
1 2 3 4 |
$extensionType = "HybridWorkerForLinux/HybridWorkerForWindows" $extensionName = "HybridWorkerExtension" $publisher = "Microsoft.Azure.Automation.HybridWorker" Set-AzVMExtension -ResourceGroupName <RGName> -Location <Location> -VMName <vmName> -Name $extensionName -Publisher $publisher -ExtensionType $extensionType -TypeHandlerVersion 1.1 -Settings $settings -EnableAutomaticUpgrade $true/$false |
I learnt how to do a manual upgrade:
|
1 |
Set-AzVMExtension -ResourceGroupName <VMResourceGroupName> -Location <VMLocation> -VMName <VMName> -Name "HybridWorkerExtension" -Publisher "Microsoft.Azure.Automation.HybridWorker" -ExtensionType HybridWorkerForWindows -TypeHandlerVersion 1.1 -EnableAutomaticUpgrade $true/$false |
That worked!
And that did the trick! Now the Runbooks are no longer “Queued”. :)