We started the
fal project with a simple objective; we wanted to build a developer tool that enables running python scripts downstream of dbt models. Since we launched the initial capabilities of
fal, we have received an overwhelming amount of requests to run python scripts before, after and in between different
dbt runs. Last month we enabled running python scripts before a
dbt run, and today we are announcing our biggest update yet.
fal flow is a new CLI command that enhances the dbt graph to include python nodes, and enables familiar dbt operations on the whole graph.
Consider the example DAG below. Triggering a run that includes all the downstream dependencies of the
load_data.py script requires three different commands to be invoked separately. Determining the exact commands also involved some manual work, as we need to figure out the descendants of the origin node.
fal flow provides the same delightful developer experience as
dbt, but now for the whole DAG. Running the same workflow above is a single
fal flow command with familiar
dbt graph selector operations.
Our new release opens the door for single command workflows that previously required several different tools. Triggering an EL run followed by a mix of dbt nodes and python scripts is now a single fal command. Complicated polyglot workflows with Python nodes in between
dbt nodes are also made simple with
fal flow. We are going to explore the capabilities of
fal flow for the rest of our launch week. Stay tuned!