Reading an Event Hub from Azure Functions and/ or Logic Apps

Continuing from the previous post where I exported data from Log Analytics to an Event Hub I now want to capture events from an Event Hub and do an HTTP trigger to an external REST API. I can do this via Logic Apps or Azure Functions (and others too I suppose but these are the two I tried).

Update (19 Oct 2021): The Functions way doesn’t really work so if you are reading this post for that feel free to skip this one and read my later post instead. Of course this one’s a good read to know what stupid mistakes I made. 🤦‍♂️

Logic Apps

I started with this as I figure this would be easier. I am going to skip the basic steps here like how to create a Logic App etc.

First get stuff from the Event Hub. For this we need to create a connection to it. While doing this the first time I thought I could get away with an access policy that has only the Listen claim but that didn’t help. Looks like you need an access policy with all three rights, similar to the default one. So that’s something to bear in mind.

When creating the trigger initially I left things at the default and just ran it to see what the output looks like. It was like this:

That seems to be some base64 encoded data. Then I realized I can change the content type in the trigger to application/json.

Did that and I get better results.

The output is now JSON. It contains a single “record” property that is an array. Each element of the array is the following:

I copied everything in that Content box in the output. Then I went back to the Logic App and added a Parse JSON action. I pasted the text I copied above into the sample payload section and so Logic Apps helpfully generated a schema.

Here’s the schema just in case anyone’s following along:

Next I added a For each action. I know the records property is an array so I added that as the input to the For each (that is the only option anyways).

Now I want to parse the individual entry. So I added an action for that too. For the schema I copy pasted an individual entry from the earlier output I captured from the Content box above and pasted that in. This gave me a schema:

 

Right ho! Let’s run that to see how things look.

Perfect! Now I can bung this off in an HTTP Request:

And that’s it!

(In reality I added an additional step to get the API key from a Key Vault and also some filtering of the JSON to skip records I am not interested in… but that doesn’t matter here).

Azure Function

Create a new function app. PowerShell’s all I know so that’s what I am going to use:

 

Go with the serverless plan.

Then create a new function. Select the Event Hub trigger template. Add a connection to the Event Hub (a policy using just the Listen claim is sufficient). Put in the Event Hub name.

You now end up with a function like this but it doesn’t work:

The output is like this:

Having no idea what’s wrong here, but knowing the objects seem to be hash tables I tried the following:

That gave the following error:

Trying to access a single property didn’t help either:

Hmm.

Googling on this latest error got me to this GitHub issue though. Looks like I should specify the binding as string.

That’s under the “Integration” section, so I did just that:

Select “String”:

I reverted the code back to the default:

Any better now? Yes:

I truncated the output as there’s a lot. As before though I was looking at a records property that was an array. Thing is, since I said this was a string I figured I’ll have to convert it to JSON first and then try to manipulate. So I did the following:

That works!

I added those extra text markers in there so I can easily identify the individual JSON pieces. In the final code I can remove those:

All I need to do now is post this to a REST API. That’s easy stuff.

Update (19 Oct 2021): Turns out I might have mistaken coz when I got to the supposedly easy stuff of posting this via a REST API I realized that the output isn’t formatted well. Here’s an example of what it looked like on the receiving side:

When I output it to the logs it looks fine, but when POSTing it it’s messed up. I think something about the conversion from JSON to hash table and back is messing things up. I tried alternatives like ConvertFrom-JSON -AsHashTable  (because the default is to convert it to a PSCustomObject) but that didn’t help either.

Another thing I noticed is that each $eventHubMessages object itself could contain many records JSON arrays. So I tried the following but no luck:

I am converting the incoming object from a JSON array to an array of hash tables and then iterating through it. It works fine on paper – I can get the Computer name and EventID etc. but when I push it out it’s f**ed up. :-/

Anyways, I don’t have too much time to spend on this (already wasted a day and weekend) so I think I’ll stick to Logic Apps for now.

Update (20 Oct 2021): I wonder if it’s related to this PowerShell Core issue on GitHub. There’s a SO post too I found from the GitHub issue.

Update (25 Oct 2021): I now have this working. It was just a case of me not understanding how Functions work. I first talk about my folly in this blog post and later in this blog post.