Notebooks as fal scripts and more

This is the day five of our launch week. We have already announced a couple of important updates, including Python Data Models and support for more databases. Today we're presenting more new features that our users might like.

Jupyter Notebooks as fal scripts

A lot of our users rely on Jupyter notebooks in order to quickly iterate on their Python code snippets and build their workflows. It’s possible to make your notebook aware of your dbt project by using the FalDbt object, that has similar methods and properties as the ones we provide inside fal scripts.

One pain point in this case is the fact that you had to convert your notebook into a .py file in order to be able to use it as a fal script in your dbt project. Not anymore. Now you can just use your notebook.

But what about magic functions? We have a solution for that. In order to use fal magic functions such as ref and write_to_model while iterating on your notebook, you can use the new init_fal magic command.

First, you need to import init_fal within a Python cell:

from faldbt.magics import init_fal

Then, you need to initialize fal in your notebook:

%init_fal project_dir=project_dir profiles_dir=profiles_dir default_model_name=my_model

Now you can access the magic functions, as if you were inside a fal script:

my_df = ref('some_model')

# ...

write_to_model(my_predictions)

Note that in fal runtime, the init_fal line is ignored, and the rest of the file will be parsed as a regular fal script

You can specify an .ipynb file the same way as a regular Python file:

models:
  - name: some_model
    meta:
      fal:
        scripts:
            - my_notebook.ipynb

execute_sql magic function

Recently we ran into a case, where a fal user wanted to do a simple SQL calculation but didn’t want it to be stored as a dbt model. In order to meet this use case, we implemented a new magic function: execute_sql.

execute_sql takes a single string, your SQL query, and returns the query result as a pandas DataFrame:

my_df = execute_sql('SELECT * FROM {{ ref("my_model") }}')

No need to worry about profile names or the data warehouse connections, fal will figure this out for you. What’s more, execute_sql supports Jinja. This means that you can use ref to refer to your models by name and use your macros.

One thing to note is that the use of ref inside execute_sql does not create a node in a dbt DAG. So, if you’re using this function inside a Python model, you need to specify dependencies in a docstring at the top of the file.

Sources and tests

Until now fal has supported generic test information, when tests are associated with models. Each model has a status property that can tell our users whether or not the model was tested. Each model also has a tests property that provides a list of tests with more detailed test information.

This week we’re expanding the above functionality to dbt sources. Starting with version 0.3.0, when calling list_sources function, our users will receive a list of DbtSource objects, each of which will contain a status property and a tests property, that work exactly the same way as analogous properties work in models.

Conclusion

For more information about fal, see our docs. Feedback and comments are always welcome on our Discord server. Also, check out our Github repository, give us a star or raise an issue.