Serverless Slack notifications for dbt Cloud Webhooks

Introduction

Managing data workflows in a modern data stack can be challenging, but with the right tools, it becomes a breeze. Today, we're going to walk you through how to integrate fal-serverless and dbt Cloud webhooks to send Slack notifications for dbt Cloud jobs. By the end of this post, you'll have a seamless, automated system to keep your team informed about the status of your dbt jobs.

Prerequisites

Before we dive in, make sure you have the following set up:

1. A dbt Cloud account and a configured project
2. A fal-serverless account
3. A Slack Bot token
4. A Slack Channel ID

Creating a fal-serverless Function to Handle dbt Webhook Events

Let's start by creating a new file hook.py and writing a fal-serverless function that will handle the incoming dbt webhook events and send a Slack message accordingly. Here's the code for the isolated function:

from fal_serverless import isolated

@isolated(requirements=["slack-sdk"], serve=True)
def slack_hook(accountId, eventId, timestamp, eventType, webhookName, data, *args):
    import json
    import os
    from slack_sdk import WebClient
    from slack_sdk.errors import SlackApiError
    CHANNEL_ID = os.environ["SLACK_CHANNEL_ID"]
    SLACK_TOKEN = os.environ["SLACK_TOKEN"]
    slack_client = WebClient(token=SLACK_TOKEN)
    message = f":tada: *dbt-cloud webhook {webhookName}: {eventType} *\\n\\n"
    if data:
        run_url = f"<https://cloud.getdbt.com/deploy/{accountId}/projects/{data.get('projectId')}/runs/{data.get('runId')}>"
        message += (
            f":file_folder: Project: *{data.get('projectName')}*\\n\\n"
            f":id: Project ID: *{data.get('projectId')}*\\n\\n"
            f":hammer_and_wrench: Job: *{data.get('jobName')}*\\n\\n"
            f":running_shirt_with_sash: Run ID: *{data.get('runId')}*\\n\\n"
            f":white_check_mark: Run Status: *{data.get('runStatus')}*\\n\\n"
            f":speech_balloon: Run Reason: *{data.get('runReason')}*\\n\\n"
            f":alarm_clock: Started: *{data.get('runStartedAt')}*\\n\\n"
            f":checkered_flag: Finished: *{data.get('runFinishedAt')}*\\n\\n"
            f":link: {run_url}\\n\\n"
        )
    else:
        message += ":warning: webhook request contains no data :warning:"
    try:
        response = slack_client.chat_postMessage(
            channel=CHANNEL_ID,
            text=message,
            mrkdwn=True
        )
    except Exception as e:
        return str(e)
    return "OK"

This function utilizes the @isolated decorator along with serve argument, in order to serve the function with the required slack-sdk package. It receives the dbt webhook payload, extracts relevant information, and formats a Slack message to send to the specified channel.

Securing the Slack App Token and Channel ID

To ensure the security of your Slack Client Token and channel ID, set them as secrets using the fal-serverless CLI commands. This way, they're securely stored and can be accessed as environment variables within the isolated function.

Here's how to set your secrets:

fal-serverless secrets set SLACK_TOKEN your_slack_token_here
fal-serverless secrets set SLACK_CHANNEL_ID your_slack_channel_id_here

Now that your Slack Token and channel ID are securely stored, let's move on to deploying your serverless function.

Deploying the Serverless Function

To deploy the function, use the fal-serverless CLI command.

fal-serverless function serve hook.py slack_hook --alias slack_hook

After deploying, you'll receive a URL that should have this structure: https://<userid>-slack_hook.gateway.alpha.fal.ai.

In order to be let dbt Cloud webhooks call this URL, you need to generate key-based credentials:

fal-serverless key generate

And then edit the URL to incorporate the generated key id and secret: https://<userid>-slack_hook.gateway.alpha.fal.ai?fal_key_id=<your-key-id>&fal_key_secret=<your-key-secret>. Make sure to keep your key id and secret secure, as access to them lets anyone run serverless functions on your behalf.

Configuring dbt Cloud Webhooks to Send Events

Head over to dbt Cloud Account Settings and navigate to the Webhooks section. Click on “Create New Webhook”, fill out the form and specify the serverless function URL (from the deployment step) as the webhook endpoint.

Choose the desired event triggers, such as Run Started, Run Completed, or Run Errored. Also choose which Jobs should trigger the webhook.

Testing the Integration

Once you save the new webhook and click the edit button for this endpoint, the same form should open but now with the active “Test Endpoint” button at the top of the form. Clicking it should send a POST request to the deployed serverless function and result in a message sent to your Slack channel.

Now that everything is set up, your can further test the integration by triggering a dbt Cloud job. If everything is configured correctly, you should see a notification message in your designated Slack channel, providing details about the dbt Cloud job.

Conclusion

We've successfully integrated fal-serverless and dbt Cloud to send Slack notifications for dbt Cloud jobs. This automation should keep your team informed and improve your data workflow management. Feel free to explore other ways to improve this integration, such as adding custom message formatting or extending the functionality to support other messaging platforms.

For more information on fal-serverless, check out our documentation and our examples. Feel free to reach out to our team on Discord and in dbt Slack community. 🚀