Today's release introduces parallelization of Python scripts in your fal runs. During a run, fal creates a directed acyclic graph (DAG) of links between models and scripts. The number of threads represents the maximum number of tasks fal handles simultaneously.
When fal is running, it will read the configuration for threads the same way dbt does it. We can set it in the profiles.yml file like this:
Or we can pass a --threads n flag:
In both cases, we see that fal communicated the number of threads with a line Concurrency: 1 threads. Notice that the order of logging changes between the two example runs: the 1 thread example logs in the same order they were announced before (<agent_wait_time, test.py>, <zendesk_ticket_data, test.py>, <john_table, john_test.py>), while 4 threads example logs in a different order; which is an expected race condition of threads.
The profiles.yml and --threads n flag settings also work for the fal flow run command. And it sets up the threads to use for the dbt run section.
Notice the 2 fal lines and 1 dbt line with threads information: Concurrency: 2 threads for fal and Concurrency: 2 threads (target='dev') for dbt.
The next step for parallelization is making logging for each different script clearer.
To start working with parallel Python workflows, get the latest version of fal at the fal-ai/fal repository!