fal and dbt-fal: a magic combination

Since our release of dbt-fal, we understand that there has been some confusion about fal and it's relationship to dbt-fal. We want to clear up this confusion here. At the same time, we will aim to provide some guidelines on which tool should be used for which task.

fal and dbt-fal are two powerful tools that make working with Python and dbt easier and more efficient. dbt-fal is a dbt adapter that allows developers to easily run dbt models that are written in Python. fal, on the other hand, is a separate tool that enables developers to write Python pre-hooks and post-hooks for their dbt models. This means that you can add custom logic to your dbt runs, such as handling notification and sending API requests. Together, dbt-fal and fal provide a powerful and flexible solution for working with dbt and Python. In this post, we'll take a closer look at these tools and show you how to use them in your projects.

What is fal?

One of the main motivations behind fal is to provide an easier and more streamlined way to integrate Python into dbt runs without the need for complex and time-consuming setup of Apache Airflow DAGs. With fal, developers can easily define Python pre-hooks and post-hooks for their dbt runs, such as sending a Slack message, sending API requests and more. This allows developers to focus on the business logic of their code, rather than worrying about setting up and maintaining complex Airflow pipelines. Here's a simplified example of a fal script:

from my_lib import send_slack_message

model_name = context.current_model.name
model_status = context.current_model.status

df = ref(model_name)
message = f"Model name: {model_name}, status: {model_status}"

send_slack_message(message)

See a more detailed example in our docs.

Additionaly, fal supports it's own dialect of Python models. This feature was developed before dbt 1.3 and dbt Python models and therefore these models cannot be computed by using the usual dbt run command. Since the release of dbt 1.3, we recommend our users to migrate their fal Python models to dbt native Python models.

What is dbt-fal?

dbt-fal is our dbt adapter that we recently released. It is built with one single purpose: running dbt Python models. This means that your SQL models are still calculated in your SQL adapter but your Python models are calculated by dbt-fal.

dbt-fal provides a straightforward and intuitive way to connect to your SQL adapters. It works together with your main data warehouse as seen in this example profiles.yml:

my_profile:
  target: my_target
  outputs:
    my_target:
      type: fal
      db_profile: my_db_target # <-- data warehouse adapter!
    my_db_target:
      type: postgres
      host: localhost
      port: 5439
      user: pguser
      password: mypassword
      dbname: mydbname
      schema: myschema
      threads: 4

With dbt-fal, you can define dbt models using the full power and flexibility of Python. Your model transformations can now include matrix algebra, forecasts, natural language processing, and more.

What makes dbt-fal easy to use is that it doesn't require setup of a compute instance that is dedicated to Python models. Instead, it runs models locally on the same computer that is running dbt. This allows dbt-fal to work with all major dbt-adapters, including adapters that do not natively support Python models, such as dbt-postgres and dbt-redshift.

In the coming weeks, we will be publishing more dbt-fal examples, so stay tuned.

When should you use fal vs dbt-fal?

A good rule of thumb for choosing the right tool is this:

If your Python script writes back to the data warehouse, you should use a Python model and therefore dbt-fal. For everything else, you should use fal.

This means that sending notifications should be done with fal, but running predictions against a sentiment model and writing the results back to a table should be done with dbt-fal. Sending data to external APIs should be done with fal, but adding an is_public_holiday column to a table should be done with dbt-fal.

Does fal work with dbt-fal?

Absolutely! dbt-fal lets you work with Python models and fal lets you add custom Python logic to your dbt pipelines. When used together, fal and dbt-fal provide a seamless and efficient way to make dbt work with Python. Check out fal flow.

Summary

fal and dbt-fal provide a powerful and flexible solution for working with Python and dbt. As a result, they make it easy to have a dbt-centric toolkit that can handle Python models and custom Python logic. Just make sure to use dbt-fal if your Python script writes in your data warehouse.

Have a use-case that you're not sure if it's for fal or dbt-fal? Reach out to us on our Discord server or raise an issue in our Github repository. If you're a member of dbt Slack community, you can always find us at the #tools-fal channel.