{"id":7367,"date":"2023-08-13T23:23:58","date_gmt":"2023-08-13T22:23:58","guid":{"rendered":"https:\/\/rakhesh.com\/?p=7367"},"modified":"2023-08-13T23:23:58","modified_gmt":"2023-08-13T22:23:58","slug":"notes-on-event-hubs","status":"publish","type":"post","link":"https:\/\/rakhesh.com\/azure\/notes-on-event-hubs\/","title":{"rendered":"Notes on Event Hubs"},"content":{"rendered":"

I had been using Event Hubs + Azure Functions pretty naively for the past few months. Mainly coz I just assumed how some of the things work, and also coz I guess when working with the cloud<\/em> you have this mindset that things just work and don’t really care about the details.<\/p>\n

Anyways.<\/p>\n

The first thing is that I have this Function that does some processing, and if it fails I was pushing the item to an event hub thus:<\/p>\n

try {\r\n  Push-OutputBinding -Name eventHubMessages -Value $body -ErrorAction Stop\r\n} catch {\r\n  Write-Host \"=== Error pushing ===\"\r\n  # do something about it...\r\n}<\/pre>\n
The expectation being that if the push fails I can output it and also do something like email me the item for instance. But this doesn’t work coz you can’t put the Push-OutputBinding<\/code> in a try\/ catch<\/code> block. I never tested whether this works or not, and always assumed it does, until I was testing something this weekend and realized the exceptions when pushing weren’t being caught. That’s because all output bindings are executed after a Function exits<\/a> and is done by the Function host\/ worker, not the Function itself.<\/p>\n
The way I encountered this was because I copy pasted some event hub bindings between two of my Functions without realizing I was copying the wrong code. The Function I was copying from had event hub triggers, while the Function I copied to had it as output and as you can see they have differences:<\/p>\n\n\n\n\n
Output binding<\/td>\nInput binding<\/td>\n<\/tr>\n
\n
\"bindings\": [\r\n  {\r\n    \"name\": \"eventHubMessagesOut\",\r\n    \"direction\": \"out\",\r\n    \"type\": \"eventHub\",\r\n    \"connection\": \"xxx_Function_EVENTHUB\",\r\n    \"eventHubName\": \"yyyy\"\r\n  }\r\n]<\/pre>\n
 <\/td>\n
\n
\"bindings\": [\r\n  {\r\n    \"type\": \"eventHubTrigger\",\r\n    \"name\": \"eventHubMessages\",\r\n    \"direction\": \"in\",\r\n    \"eventHubName\": \"yyyy\",\r\n    \"connection\": \"xxx_Function_EVENTHUB\",\r\n    \"cardinality\": \"many\",\r\n    \"consumerGroup\": \"$Default\"\r\n  },\r\n]<\/pre>\n
 <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
Because of this mismatch I was getting an error: No binding found for attribute 'Microsoft.Azure.WebJobs.EventHubTriggerAttribute'.<\/code><\/p>\n
I couldn’t find out why this was so until I realized the mistake I made.<\/p>\n
The biggest thing I learnt though was about retries. To begin with, check out this link<\/a> on how Azure Functions consumes Event Hubs. I am going to copy paste it here.<\/p>\n
Azure Functions consumes Event Hub events while cycling through the following steps:<\/span><\/p>\n
    \n
  1. A pointer is created and persisted in Azure Storage for each partition of the event hub.<\/span><\/li>\n
  2. When new messages are received (in a batch by default), the host attempts to trigger the function with the batch of messages.<\/span><\/li>\n
  3. If the function completes execution (with or without exception) the pointer advances and a checkpoint is saved to the storage account.<\/span><\/li>\n
  4. If conditions prevent the function execution from completing, the host fails to progress the pointer. If the pointer isn’t advanced, then later checks end up processing the same messages.<\/span><\/li>\n
  5. Repeat steps 2\u20134<\/span><\/li>\n<\/ol>\n
    This behavior reveals a few important points:<\/span><\/p>\n