What’s new in fal 0.2.0

We’re constantly working on improving fal for our users by looking into issues and feature requests. You can create a Github issue, if you also would like something in fal to be improved. Today we are releasing a new version of fal with a set of features that add flexibility to how you can run Python scripts on your dbt models. New features include:
- Model specification
- Script specification
- Access to dbt test status
- Running scripts before dbt runs
Model specification
With fal you can specify which scripts should run after which models in a schema.yml
file and then, when you run fal run
, fal will automatically find the models that were recently ran and run Python scripts assigned to them. In order for this to work fal needs access to dbt run artifacts, which are altered with each dbt run. But what if you want to have more control over which models should be run? Or maybe you don't have access to dbt artifacts in a fal runtime and want to trigger a specific run manually. It is now possible to do that with a selection syntax that was rolled out in the latest version of fal. It’s very similar to the dbt selection syntax.
Example schema.yml
Here's the models section of schema.yml in our example project:
models:
- name: boston
description: Ozone levels
config:
materialized: table
meta:
owner: "@meder"
fal:
scripts:
- fal_scripts/slack.py
- fal_scripts/send_datadog_event.py
- name: zendesk_ticket_metrics
description: Zendesk ticket metrics
config:
materialized: table
meta:
owner: "@gorkem"
fal:
scripts:
- fal_scripts/forecast_slack.py
- name: lombardia_covid
description: Lombardia (IT) Covid19 Cases
config:
materialized: table
meta:
owner: "@omer"
fal:
scripts:
- fal_scripts/anomaly_detection.py
As you can see, each model in this schema.yml has different fal scripts associated with them.
select
and models
options
The new select
flag lets users choose which models to run fal scripts on:
fal run --select boston
fal run -s boston
This example will run fal scripts associated only with boston
model, namely slack.py
and send_datadog_event.py
. This means that the magic variables used in these scripts will refer to the boston
model and the associated data. Specifying a model like this guarantees that fal scripts run regardless of dbt run. You can also select multiple models:
fal run --select boston zendesk_ticket_metrics
The models
flag works exactly the same way:
fal run --models boston
fal run --models boston zendesk_ticket_metrics
exclude
option
As the name suggests, exclude will run all the model scripts except the one specified by this option:
fal run --exclude stg_zendesk_ticket_data
this will work as regular fal run
except the scripts that are associated with stg_zendesk_ticket_data
, which will be ignored.
selector
option
If you want to persist your custom selections, you can use the selector
option and specify the selection in a selectors.yml file:
fal run --selector my_selector
Find out more about writing your own selector here.
Script specification
Similar to model specification, sometimes you might want to disregard the scripts defined in schema.yml
and specify which script to run via a command line argument. This is now possible with --scripts
option:
fal run --scripts fal_scripts/slack.py
This is will run fal_scripts/slack.py
on all the models that we calculated in the latest dbt run.
You can also specify multiple scripts:
fal run --scripts fal_scripts/slack.py fal_scripts/send_datadog_event.py
It is now possible to specify both models and scripts:
fal run --select boston --scripts fal_scripts/slack.py
dbt test
results
In the earlier versions of fal
only the artifacts produced by a dbt run
were taken into account, in version 0.2.0 fal
now works with dbt tests:
$ dbt test --select modela
$ fal run # runs the scripts that belongs to modela
In the fal script,context
object has now access to test information related to the current model. If the previous dbt command was either test
or build
, the context.current_model.test
property is populated with a list of tests:
context.current_model.tests
#= [CurrentTest(name='not_null', modelname='boston', column='ds', status='Pass')]
So far, this only works with generic tests that can be associated with dbt models.
before
flag
The new --before
flag lets users run scripts before their dbt runs. Given the following schema.yml:
models:
- name: boston
description: Ozone levels
config:
materialized: table
meta:
owner: "@meder"
fal:
scripts:
before:
- fal_scripts/postgres.py
after:
- fal_scripts/slack.py
fal run --before
will run fal_scripts/postgres.py
script regardless if dbt has calculated the boston
model or not. Regular fal run
will run fal_scripts/slack.py
, but only if boston
model is already calculated by dbt.
Summary
It is now possible to run fal with more control. Here’s what you can do now:
- Use the same flags as dbt and specify the models that should be selected or excluded
- Specify scripts
- Keep track of your dbt tests
- Run scripts before your dbt runs with
--before
flag.
For examples of fal + dbt usage check out fal repo or join our Discord. And let us know if you have any ideas on how we could further improve fal.