{"id":7455,"date":"2023-12-20T19:10:09","date_gmt":"2023-12-20T19:10:09","guid":{"rendered":"https:\/\/rakhesh.com\/?p=7455"},"modified":"2024-01-20T09:40:59","modified_gmt":"2024-01-20T09:40:59","slug":"runbook-type-17-not-supported","status":"publish","type":"post","link":"https:\/\/rakhesh.com\/azure\/runbook-type-17-not-supported\/","title":{"rendered":"Runbook type ’17’ not supported. (or: Runbooks stuck in a Queued state)"},"content":{"rendered":"

Was stuck with an irritating problem today. Installed PowerShell 7.4 on one of my Hybrid Runbook Workers (HRW) but the Runbooks refuse to run using it! Looking at the logs <\/a>I see entries like this:<\/p>\n

Orchestrator.Sandbox.Diagnostics Critical: 0 : [2023-12-20T15:05:20.4533023Z]  Starting sandbox process. [sandboxId=ecdbfee3-902a-44c8-b936-d51c8180190c]\r\nOrchestrator.Sandbox.Diagnostics Critical: 0 : [2023-12-20T15:05:22.4845620Z]  Hybrid Sandbox\r\nOrchestrator.Sandbox.Diagnostics Critical: 0 : [2023-12-20T15:05:23.1564394Z]  First Trace Log.\r\nOrchestrator.Sandbox.Diagnostics Critical: 0 : [2023-12-20T15:05:23.4220674Z]  Sandbox Recieving Job. [sandboxId=ecdbfee3-902a-44c8-b936-d51c8180190c][jobId=0dfa6479-ffca-4f40-a54e-ee353f57734a]\r\nOrchestrator.Sandbox.Diagnostics Critical: 0 : [2023-12-20T15:05:23.6251949Z]  An unhandled exception was encountered while handling the job action.  The sandbox will terminate immediately. [jobId=0dfa6479-ffca-4f40-a54e-ee353f57734a][source=Maintainer][exceptionMessage=System.InvalidOperationException: Runbook type '17' not supported.\r\n   at Orchestrator.Runtime.Account.CreateRunbook(RunbookKey runbookKey, RunbookData runbookData) in C:\\__w\\1\\s\\src\\Shared\\Orchestrator.Runtime\\Account.cs:line 336\r\n   at Orchestrator.Runtime.Account.DefaultLoadRunbook(CompositeKey`2 compositeKey) in C:\\__w\\1\\s\\src\\Shared\\Orchestrator.Runtime\\Account.cs:line 233\r\n   at Orchestrator.Shared.Shared.CacheWithLocking`2.LoadAndAddValue(TKey key) in C:\\__w\\1\\s\\src\\Shared\\Orchestrator.Shared\\Shared\\CacheWithLocking.cs:line 609\r\n   at Orchestrator.Shared.Shared.CacheWithLocking`2.GetEntry(TKey key) in C:\\__w\\1\\s\\src\\Shared\\Orchestrator.Shared\\Shared\\CacheWithLocking.cs:line 582\r\n   at Orchestrator.Shared.Shared.CacheWithLocking`2.ActWithGuardedValue[TResult](TKey key, Func`2 action) in C:\\__w\\1\\s\\src\\Shared\\Orchestrator.Shared\\Shared\\CacheWithLocking.cs:line 352\r\n   at Orchestrator.Sandbox.SandboxJobActionHandler.HandleJobActionWithGuardedAccount(Account account, JobData jobData, JobMessageSource source) in C:\\__w\\1\\s\\src\\prod\\Orchestrator.Sandbox\\SandboxJobActionHandler.cs:line 605\r\n   at Orchestrator.Shared.Shared.CacheWithLocking`2.<>c__DisplayClass30_0.<ActWithGuardedValue>b__0(TValue value) in C:\\__w\\1\\s\\src\\Shared\\Orchestrator.Shared\\Shared\\CacheWithLocking.cs:line 319\r\n   at Orchestrator.Shared.Shared.RefCounted`1.UsingValue[TResult](Func`2 action) in C:\\__w\\1\\s\\src\\Shared\\Orchestrator.Shared\\Shared\\RefCounted.cs:line 153\r\n   at Orchestrator.Shared.Shared.CacheWithLocking`2.ActWithGuardedValue[TResult](TKey key, Func`2 action) in C:\\__w\\1\\s\\src\\Shared\\Orchestrator.Shared\\Shared\\CacheWithLocking.cs:line 352\r\n   at Orchestrator.Sandbox.SandboxJobActionHandler.HandleJobActionSync(JobData jobData, JobMessageSource source) in C:\\__w\\1\\s\\src\\prod\\Orchestrator.Sandbox\\SandboxJobActionHandler.cs:line 329\r\n   at Orchestrator.Sandbox.SandboxJobActionHandler.HandleJobActionThread(JobData jobData, JobMessageSource source) in C:\\__w\\1\\s\\src\\prod\\Orchestrator.Sandbox\\SandboxJobActionHandler.cs:line 680][sandboxId=ecdbfee3-902a-44c8-b936-d51c8180190c]<\/pre>\n
Tried rebooting, updating the OS, downgrading it to PowerShell 7.2 just in case… nothing helped. Double checked the steps in the docs <\/a>(there isn’t much) and all looked fine. But the Runbooks just wouldn’t run. They’d stay “Queued” with lot of logs like above being generated, and eventually move to a “Suspended” state.<\/p>\n
I had two automation accounts, one of them linked to two HRWs; another to one HRW. Both had the same issue.<\/p>\n
Luckily I also had another automation account + HRW where things worked, so I compared the two and noticed that under the VM in Azure > “Extensions + applications” there’s a difference:<\/p>\n
\"\"<\/p>\n
Version was 1.1.12.<\/p>\n
\"\"<\/p>\n
While on the problem HRW it was:<\/p>\n
\"\"<\/p>\n
Older version. Same on all three non-working HRWs! There were some of my older HRWs so I must have been on a preview version or something which never updated.<\/p>\n
So I enabled automatic upgrade. Then waited… but that didn’t update anything. Googled, and came across this doc<\/a>. Looks like the updates only happen for minor versions, while I need to jump from 0.x to 1.x.<\/p>\n
Btw, good instructions on how to enable\/ disable automatic upgrades when needed.<\/p>\n
$extensionType = \"HybridWorkerForLinux\/HybridWorkerForWindows\"\r\n$extensionName = \"HybridWorkerExtension\"\r\n$publisher = \"Microsoft.Azure.Automation.HybridWorker\"\r\nSet-AzVMExtension -ResourceGroupName <RGName> -Location <Location>  -VMName <vmName> -Name $extensionName -Publisher $publisher -ExtensionType $extensionType -TypeHandlerVersion 1.1 -Settings $settings -EnableAutomaticUpgrade $true\/$false<\/pre>\n
I learnt how to do a manual upgrade:<\/p>\n
Set-AzVMExtension -ResourceGroupName <VMResourceGroupName> -Location <VMLocation> -VMName <VMName> -Name \"HybridWorkerExtension\" -Publisher \"Microsoft.Azure.Automation.HybridWorker\" -ExtensionType HybridWorkerForWindows -TypeHandlerVersion 1.1 -EnableAutomaticUpgrade $true\/$false<\/pre>\n
\"\"<\/p>\n
That worked!<\/p>\n
\"\"<\/p>\n
And that did the trick! Now the Runbooks are no longer “Queued”. :)<\/p>\n","protected":false},"excerpt":{"rendered":"
Was stuck with an irritating problem today. Installed PowerShell 7.4 on one of my Hybrid Runbook Workers (HRW) but the Runbooks refuse to run using it! Looking at the logs I see entries like this: Orchestrator.Sandbox.Diagnostics Critical: 0 : [2023-12-20T15:05:20.4533023Z] Starting sandbox process. [sandboxId=ecdbfee3-902a-44c8-b936-d51c8180190c] Orchestrator.Sandbox.Diagnostics Critical: 0 : [2023-12-20T15:05:22.4845620Z] Hybrid Sandbox Orchestrator.Sandbox.Diagnostics Critical: 0 : … Continue reading Runbook type ’17’ not supported. (or: Runbooks stuck in a Queued state)<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[887],"tags":[1043,383,227],"jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts\/7455"}],"collection":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/comments?post=7455"}],"version-history":[{"count":2,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts\/7455\/revisions"}],"predecessor-version":[{"id":7479,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts\/7455\/revisions\/7479"}],"wp:attachment":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/media?parent=7455"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/categories?post=7455"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/tags?post=7455"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}