rakhesh sasidharan's mostly techie somewhat purpley blog

We use Cribl at work and my colleague had a requirement where we wanted to lookup users based on certain properties and filter them accordingly. Cribl has something called Lookup which can do that – it can lookup CSV files (and a few others) to enrich information.

Uploading the CSV file to Cribl is easy – do via the GUI – but we wanted a way to keep it up to date as the contents change pretty regularly. Enter automation! And since I am a PoSH kid, that’s how I set about doing it.

Here are some useful docs I came across while doing this:

API Docs – an intro guide, gives you info on how to do authentication; this one is specifically about lookups
API Reference – a reference to all the endpoints
A nice blog post on intro to the API
The steps you need to do to create/ update a lookup – the URLs need adjusting, but you get the idea.

I am not too familiar with Cribl, so what follows is going to be kind of vague. We use Cribl cloud. There’s a leader node and some worker nodes, and from what I understand the worker nodes are where I must update the lookups.

Authentication

First things first is the authentication. A colleagues with access to Cribl generated a client id and secret for me. I assume they followed the steps here. Armed with those, what I had to do was generate a token (valid for 24 hours) so I can call the API.

$authBody = @{
    "grant_type" = "client_credentials"
    "client_id" = "===replace me ===="
    "client_secret" = "=== replace me too ==="
    "audience" = "https://api.cribl.cloud"
} | ConvertTo-Json -Depth 5

$tokenObj = Invoke-RestMethod -Method "POST" -Uri "https://login.cribl.cloud/oauth/token" -ContentType "application/json" -Body $authBody -ErrorAction Stop

$criblHeaders = @{
    "Authorization" = "Bearer $($tokenObj.access_token)" 
}

$authBody = @{

"grant_type" = "client_credentials"

"client_id" = "===replace me ===="

"client_secret" = "=== replace me too ==="

"audience" = "https://api.cribl.cloud"

} | ConvertTo-Json -Depth 5

$tokenObj = Invoke-RestMethod -Method "POST" -Uri "https://login.cribl.cloud/oauth/token" -ContentType "application/json" -Body $authBody -ErrorAction Stop

$criblHeaders = @{

"Authorization" = "Bearer $($tokenObj.access_token)"

}

That’s it, straight forward! Now you have the headers to use in all subsequent calls.

Next you need your Cribl instance URL.

$criblBaseUrl = "https://main-sad-einstein-22kxv8a.cribl.cloud/api/v1"

1	$criblBaseUrl = "https://main-sad-einstein-22kxv8a.cribl.cloud/api/v1"

Here’s some of the things I did.

Getting a list of worker groups

$urlSnippet = '/master/groups?product=stream'
$criblUrl = "${criblBaseUrl}${urlSnippet}"

$workerGroupsObj = Invoke-RestMethod -Uri $criblUrl -Method "GET" -Headers $criblHeaders -ContentType "application/json"

$urlSnippet = '/master/groups?product=stream'

$criblUrl = "${criblBaseUrl}${urlSnippet}"

$workerGroupsObj = Invoke-RestMethod -Uri $criblUrl -Method "GET" -Headers $criblHeaders -ContentType "application/json"

Getting a list of lookup across all worker groups

I get the worker groups, and then call an API endpoint against each of them to get the list of lookups. Key thing is the URL varies per worker group.

$urlSnippet = '/master/groups?product=stream'
$criblUrl = "${criblBaseUrl}${urlSnippet}"

$workerGroupsObj = Invoke-RestMethod -Uri $criblUrl -Method "GET" -Headers $criblHeaders -ContentType "application/json"

foreach ($workerGroupObj in $workerGroupsObj.items) {
    $workerGroup = $workerGroupObj.id
    Write-Output "Worker group $workerGroup"

    $urlSnippet = "/m/$workerGroup/system/lookups"
    $criblUrl = "${criblBaseUrl}${urlSnippet}"

    $lookupsObj = Invoke-RestMethod -Uri $criblUrl -Method "GET" -Headers $criblHeaders -ContentType "application/json"
    $lookupsObj.items

}

$urlSnippet = '/master/groups?product=stream'

$criblUrl = "${criblBaseUrl}${urlSnippet}"

$workerGroupsObj = Invoke-RestMethod -Uri $criblUrl -Method "GET" -Headers $criblHeaders -ContentType "application/json"

foreach ($workerGroupObj in $workerGroupsObj.items) {

$workerGroup = $workerGroupObj.id

Write-Output "Worker group $workerGroup"

$urlSnippet = "/m/$workerGroup/system/lookups"

$criblUrl = "${criblBaseUrl}${urlSnippet}"

$lookupsObj = Invoke-RestMethod -Uri $criblUrl -Method "GET" -Headers $criblHeaders -ContentType "application/json"

$lookupsObj.items

}

Uploading a lookup file

$urlSnippet = "/m/$workerGroup/system/lookups?filename=${lookupFileName}"
$criblUrl = "${criblBaseUrl}${urlSnippet}"

$uploadResponse = Invoke-RestMethod -Method "PUT" -Uri $criblUrl -Headers $criblHeaders -ContentType "text/csv" -InFile $lookupFileName

$urlSnippet = "/m/$workerGroup/system/lookups?filename=${lookupFileName}"

$criblUrl = "${criblBaseUrl}${urlSnippet}"

$uploadResponse = Invoke-RestMethod -Method "PUT" -Uri $criblUrl -Headers $criblHeaders -ContentType "text/csv" -InFile $lookupFileName

This only uploads the file to the worker group. It does not actually create or update it. That’s the next step.

By the way, $lookupFileName would be something like UserMappings.csv. It has to end with CSV for a few other extensions.

It’s important to capture the response as it contains a temporary file name. We need that in the next step as the creation or updating of the lookup is based on this temporary name.

Creating or Updating a lookup

Different REST methods depending on whether you are creating or updating. So what I do is see if the $lookupObjs from above has the name of the lookup. The $lookupObjs.items contains is an array of all the lookups. The id property is the name of the lookup.

If we have to update it’s a PATCH method. If we have to create it’s a POST method. The body is the same for both, but I wasn’t sure initially so added it to both use cases. Also, “fileInfo” is case-sensitive. I used “fileinfo” initially and it kept giving an error (didn’t capture it so I don’t have the error, but it was misleading in that it said the file doesn’t exist).

if ($lookupsObj.items.id -contains $lookupFileName) {
    Write-Output "Lookup object needs updating"

    $method = "PATCH"
    $urlSnippet = "/m/$workerGroup/system/lookups/${lookupFileName}"

    $requestBody = @{
        "id" = "$lookupFileName"
        "fileInfo" = @{
            "filename" = $uploadResponse.filename
        }
    } | ConvertTo-Json -Depth 5

} else {
    Write-Output "Lookup object needs creating"

    $method = "POST"
    $urlSnippet = "/m/$workerGroup/system/lookups/"

    $requestBody = @{
        "id" = "$lookupFileName"
        "fileInfo" = @{
            "filename" = $uploadResponse.filename
        }
    } | ConvertTo-Json -Depth 5
    
}

$criblUrl = "${criblBaseUrl}${urlSnippet}"
$createResponseObj = Invoke-RestMethod -Uri $criblUrl -Method $method -Headers $criblHeaders -ContentType "application/json" -Body $requestBody

if ($lookupsObj.items.id -contains $lookupFileName) {

Write-Output "Lookup object needs updating"

$method = "PATCH"

$urlSnippet = "/m/$workerGroup/system/lookups/${lookupFileName}"

$requestBody = @{

"id" = "$lookupFileName"

"fileInfo" = @{

"filename" = $uploadResponse.filename

}

} | ConvertTo-Json -Depth 5

} else {

Write-Output "Lookup object needs creating"

$method = "POST"

$urlSnippet = "/m/$workerGroup/system/lookups/"

$requestBody = @{

"id" = "$lookupFileName"

"fileInfo" = @{

"filename" = $uploadResponse.filename

}

} | ConvertTo-Json -Depth 5

}

$criblUrl = "${criblBaseUrl}${urlSnippet}"

$createResponseObj = Invoke-RestMethod -Uri $criblUrl -Method $method -Headers $criblHeaders -ContentType "application/json" -Body $requestBody

It’s important to capture the response, because we use it in the next step.

Commiting the change

Now we need to commit the above change.

$urlSnippet = "/m/$workerGroup/version/commit"
$criblUrl = "${criblBaseUrl}${urlSnippet}"
$commitBody = @{
    "group" = $workerGroup
    "message" = "updating $lookupFile via automation"
}  | ConvertTo-Json -Depth 5

$commitObj = Invoke-RestMethod -Method POST -Uri $criblUrl -Headers $criblHeaders -ContentType "application/json" -Body $commitBody -ErrorAction Stop

$urlSnippet = "/m/$workerGroup/version/commit"

$criblUrl = "${criblBaseUrl}${urlSnippet}"

$commitBody = @{

"group" = $workerGroup

"message" = "updating $lookupFile via automation"

} | ConvertTo-Json -Depth 5

$commitObj = Invoke-RestMethod -Method POST -Uri $criblUrl -Headers $criblHeaders -ContentType "application/json" -Body $commitBody -ErrorAction Stop

Deploying the commit

And finally, we deploy the commit.

if ($commitId.Length -ne 0) {
    $urlSnippet = "/master/groups/$workerGroup/deploy"
    $criblUrl = "${criblBaseUrl}${urlSnippet}"
    $deployBody = @{
        "version" = $commitId
    } | ConvertTo-Json -Depth 5

    $deployObj = Invoke-RestMethod -Method PATCH -Uri $criblUrl -Headers $criblHeaders -ContentType "application/json" -Body $deployBody -ErrorAction Stop
}

if ($commitId.Length -ne 0) {

$urlSnippet = "/master/groups/$workerGroup/deploy"

$criblUrl = "${criblBaseUrl}${urlSnippet}"

$deployBody = @{

"version" = $commitId

} | ConvertTo-Json -Depth 5

$deployObj = Invoke-RestMethod -Method PATCH -Uri $criblUrl -Headers $criblHeaders -ContentType "application/json" -Body $deployBody -ErrorAction Stop

}

And that’s it, we are done!