Notebooks as fal scripts and more
Today we're presenting more new features that our users might like: Jupyter notebooks as fal scripts, execute_sql magic function, dbt source test results.
This is the day five of our launch week. We have already announced a couple of important updates, including Python Data Models and support for more databases. Today we're presenting more new features that our users might like.
Jupyter Notebooks as fal scripts
A lot of our users rely on Jupyter notebooks in order to quickly iterate on their Python code snippets and build their workflows. It’s possible to make your notebook aware of your dbt project by using the FalDbt
object, that has similar methods and properties as the ones we provide inside fal scripts.
One pain point in this case is the fact that you had to convert your notebook into a .py
file in order to be able to use it as a fal script in your dbt project. Not anymore. Now you can just use your notebook.
But what about magic functions? We have a solution for that. In order to use fal magic functions such as ref
and write_to_model
while iterating on your notebook, you can use the new init_fal
magic command.
First, you need to import init_fal
within a Python cell:
from faldbt.magics import init_fal
Then, you need to initialize fal in your notebook:
%init_fal project_dir=project_dir profiles_dir=profiles_dir default_model_name=my_model
Now you can access the magic functions, as if you were inside a fal script:
my_df = ref('some_model')
# ...
write_to_model(my_predictions)
Note that in fal runtime, the init_fal
line is ignored, and the rest of the file will be parsed as a regular fal script
You can specify an .ipynb file the same way as a regular Python file:
models:
- name: some_model
meta:
fal:
scripts:
- my_notebook.ipynb
execute_sql
magic function
Recently we ran into a case, where a fal user wanted to do a simple SQL calculation but didn’t want it to be stored as a dbt model. In order to meet this use case, we implemented a new magic function: execute_sql
.
execute_sql
takes a single string, your SQL query, and returns the query result as a pandas DataFrame:
my_df = execute_sql('SELECT * FROM {{ ref("my_model") }}')
No need to worry about profile names or the data warehouse connections, fal will figure this out for you. What’s more, execute_sql
supports Jinja. This means that you can use ref
to refer to your models by name and use your macros.
One thing to note is that the use of ref
inside execute_sql
does not create a node in a dbt DAG. So, if you’re using this function inside a Python model, you need to specify dependencies in a docstring at the top of the file.
Sources and tests
Until now fal has supported generic test information, when tests are associated with models. Each model has a status
property that can tell our users whether or not the model was tested. Each model also has a tests
property that provides a list of tests with more detailed test information.
This week we’re expanding the above functionality to dbt sources. Starting with version 0.3.0, when calling list_sources
function, our users will receive a list of DbtSource
objects, each of which will contain a status
property and a tests
property, that work exactly the same way as analogous properties work in models.
Conclusion
For more information about fal
, see our docs. Feedback and comments are always welcome on our Discord server. Also, check out our Github repository, give us a star or raise an issue.