Today we are releasing fal v0.4.0, which introduces some backwards-incompatible changes. The focus of this release is to make our experimental features generally available, simplifying
fal flow run .
If you have been using
fal flow run, you probably want Python scripts running in the middle of your dbt models. We introduced this some time ago with the
fal flow --experimental-flow flag, followed by the recent Python Data models with
fal flow --experimental-models. We have heard from many of our users that these two features are their default, so we are making them the default behavior from now on!
By adding the line
from fal.typing import * to your fal scripts, you will get correct typing (and no more errors in VS Code) for
write_to_model and other fal native functions.
Starting Dec 3, 2021, dbt Labs dropped
v0.* support. Based on our usage data, most fal users (i.e. 99%) are now on dbt
dbt-core introduced significant changes in their codebase for the
v1.0 release, dropping
v0.* support makes adding those improvements to fal much easier for our team.
We have had some requests to always run
before-scripts of a model when it is run. This was not the default behavior, so we introduced
pre-hook. This is the new recommended way to add scripts around your dbt (or Python) models. If the model runs, the hooks will run too!
before-scripts are deprecated now, and we would like your feedback to know if this means you would be missing them or if hooks cover your needs! You can always open an issue, message us in the #tools-fal dbt Slack channel or in Discord (you can even send us an email!).
We are thinking of ways of adding non-data-generating models, that means a Python script that is part of the DAG but does not create a table in the data warehouse. If you need something similar, leave us a comment letting us know about your use-case and how you would imagine it working.
You may notice that now the dbt calls are done 1 model at a time and with
--threads 1. Don’t worry, we are not making things slower! Since Python scripts are also running in parallel, we wanted to maximize starting to work on the next node of your DAG as soon as possible. This should make your
fal flow run faster in general (some use-cases may not benefit.)
This great contribution was introduced by our newly arrived, @isidentical!
Other fixes and improvements
- Change typing file generating tool to generate fal.typing module (#469)
- Handle partially-applied write_to_model before scripts in typing module (#472)
- Ensure logbook doesn't crash on startup (#477)
- Move away from multiprocessing pool on executions (#478)
- Accept keyword arguments in not_allowed_function for hooks (#480)
- Wrap BigQuery jobs in exception handler and fix tests (#479)
- Offer type annotations for exposed functions. (#466)
- Make before/after scripts happen right before/after models (#460)
- Implement pre-hooks (#471)
- Remove warning about mutiple select flags being passed (#465)
- Remove experimental flags (#470)
- Remove support for dbt 0.X (#464)
- Make fal run use new threading (#462)
- Use experimental threads, flow and models mode by default (#463)