Reading an Event Hub from Azure Functions

Some 10 days ago I had posted about reading from an Event Hub using Azure Functions or Logic Apps. I thought I had it nailed then, but quickly realized it wasn’t working as I expected.

I have since spent a lot of time with Functions and am in a better place. Of course I am by no means an expert, and what I’ve done is probably scratch the barest surface… but I know enough to get this working and to understand why it wasn’t working before.

I touch upon this briefly in a prior post of today.

When you create an Azure Function that’s triggered by an Event Hub the default code looks like this:

param($eventHubMessages, $TriggerMetadata)

Write-Host "PowerShell eventhub trigger function called for message array: $eventHubMessages"

$eventHubMessages | ForEach-Object { Write-Host "Processed message: $_" }

param($eventHubMessages, $TriggerMetadata)

Write-Host "PowerShell eventhub trigger function called for message array: $eventHubMessages"

$eventHubMessages | ForEach-Object { Write-Host "Processed message: $_" }

The first thing to keep in mind is that messages in the Event Hub are stored as JSON and (more importantly) when an Azure Function reads from a Event Hub it automatically converts the JSON to PowerShell native constructs. That is to say if I have an array of JSON objects, these are converted to an array of Hash Tables. If you try to view these via something like Write-Host "$_" it will fail because you can’t simple write a Hash Table, you need to enumerate its keys or values.

The second thing to keep in mind is to never click the Test/ Run button on an Event Hub triggered function and expect it to do anything. Once you create the function, if it has anything to process from the Event Hub it will simply do so… it doesn’t need a nudge from you. In fact, if you click the button the Function will run… not get any input, and it will complain about JSON serialization etc and you could go down a wrong path trying to figure it.

If your Function isn’t picking anything from an Event Hub I have found this usually to be because something else is tapping into the same Event Hub. Especially when fiddling around, it is quite possible you have another Logic App or something that is reading from that Event Hub and so the Function simply has nothing to work with. And if that’s not the case, I’ve usually seen that rotate the Shared Access policy keys that the Function uses to talk to the Event Hub gives it the needed kick (possibly because regenerating those stops anything else you might have forgotten about from reading the Event Hub).

So – $eventHubMessages is an array. Thus you can do $eventHubMessages | ForEachObject { ... }. Each $_ instance inside that is a Hash Table. It is possible the Hash Table itself contains further arrays of JSON. For instance, when reading Security Events from Event Hubs the $_ instance is a Hash Table with name “records” which in turn has an array of JSON (now Hash Table) entries. So the final way to process messages from Event Hubs is something like this:

$eventHubMessages | ForEach-Object { 
    foreach ($record in $_.records) {
	# Process the record - you can see the keys as $record.Keys for instance
	Write-Host $record.Keys
	}
}

$eventHubMessages | ForEach-Object {

foreach ($record in $_.records) {

# Process the record - you can see the keys as $record.Keys for instance

Write-Host $record.Keys

}

If you then want to push a record via HTTP someplace convert it back from a Hash Table to JSON:

$bodyJSON = ($record | ConvertTo-JSON -Depth 10 -EscapeHandling EscapeNonAscii -Compress)

1	$bodyJSON = ($record \| ConvertTo-JSON -Depth 10 -EscapeHandling EscapeNonAscii -Compress)

I have found that without the -Compress the output could have some \r (carriage return) characters. By adding -Compress everything is put on one line with no extra encoding. I don’t know if -EscapeHandling EscapeNonAscii is required – I added it as part of troubleshooting and then didn’t both removing. If it works, why break?

This latter part of trying to figure out how to convert the Hash Table back to JSON took a while to figure out.. mainly coz of the \r characters as they were messing up the output JSON.

Lastly, here’s how I push the JSON to a remote HTTP endpoint and in case of any errors add it back to the Event Hub so I can process it later:

try {
    $request = Invoke-WebRequest -Uri $APIUrl -Method "POST" -Headers $headers -ContentType $contentType -Body $bodyJSON
} catch {
    # In case of any exceptions with the above call (coz of port exhaustion for instance) I hit this block
    Write-Error "**Error** $($_.Exception)"
    Write-Host "Pushing the message back to the Event Hub"
    Push-OutputBinding -Name outputEventHubMessage -Value $bodyJSON
}

if ($request.StatusCode -eq 200) { 
    Write-Host "Successfully pushed to API" 
} elseif ($null -ne $request) {
    # I only hit this section if $request is not null (i.e. the Invoke-WebRequest succeeded) but we got some error from API
    Write-Error "**Error** HTTP Status code " $request.StatusCode 
    Write-Host "Pushing the message back to the Event Hub"
    Push-OutputBinding -Name outputEventHubMessage -Value $bodyJSON
}

try {

$request = Invoke-WebRequest -Uri $APIUrl -Method "POST" -Headers $headers -ContentType $contentType -Body $bodyJSON

} catch {

# In case of any exceptions with the above call (coz of port exhaustion for instance) I hit this block

Write-Error "**Error** $($_.Exception)"

Write-Host "Pushing the message back to the Event Hub"

Push-OutputBinding -Name outputEventHubMessage -Value $bodyJSON

}

if ($request.StatusCode -eq 200) {

Write-Host "Successfully pushed to API"

} elseif ($null -ne $request) {

# I only hit this section if $request is not null (i.e. the Invoke-WebRequest succeeded) but we got some error from API

Write-Error "**Error** HTTP Status code " $request.StatusCode

Write-Host "Pushing the message back to the Event Hub"

Push-OutputBinding -Name outputEventHubMessage -Value $bodyJSON

}

This whole thing runs within the loop I showed earlier. To get this working I had to add the Event Hub as an output binding too with name outputEventHubMessage and so I can refer to it via Push-OutputBinding.