Build and Deploy Machine Learning Models from Jupyter Notebooks with fal and dbt

Introduction Machine Learning (ML) is increasingly important in data-driven decision making, so it's important to use modern tools and techniques to streamline the machine learning workflows. This is where dbt and fal can come in - together they make it easy to manage and deploy machine learning models in a

Composable OLAP

Separation of compute and storage was initially an architectural choice by database vendors to make large scale data warehousing cheaper. Data platform vendors didn’t foresee that years later this choice will force them to be more open about where companies store their data and how they query it. Fast

Building language model powered pipelines with dbt

Yes, we know… Generative AI is all the hype right now. Whether you are an Analytics Engineer or Data Engineer, you might be wondering if this stuff is only useful for AI generated profile pics, or can I actually use it to help me with my day to day work.

Populate dbt models with CSV data. Part 2: the power of dbt-fal

In this blog post, we will explore how dbt-fal and dbt can be combined to create a streamlined and efficient process for loading CSV data into a data warehouse, even when dealing with large datasets.

fal and dbt-fal: a magic combination

When should you use fal vs dbt-fal? If your Python script writes back to the data warehouse, you should use dbt-fal.

dbt-fal v1.3.12 – Finding the cause of a subtle bug

We accidentally slowed down dbt seed for everyone using the dbt-fal adapter and now its fast again.

Featured

Announcing dbt-fal adapter

Today we are excited to announce the release of dbt-fal adapter. This is our first official release since we have been teasing it the last couple of months. dbt-fal allows you to develop your dbt Python models locally, seamlessly graduate them to the cloud and combine the scalability of your

Changelog

v0.7 — Playing nice with dbt Python models

dbt recently introduced native Python models support! At fal, we believe this should be the way to run your Python models, so we have been working on dbt-fal, an adapter that enables you to run dbt Python models with any datawarehouse (including Redshift, Postgres). dbt-fal can be used to run

Changelog

v0.5 — Adapter-specific dependencies, hooks with arguments, and --full-refresh support

Make the fal package lighter by installing only the dependencies needed for your adapter.

Changelog

v0.4 — moving forward

The focus of this release is to make our experimental features generally available, simplifying fal flow run.

Notebooks as fal scripts and more

Today we're presenting more new features that our users might like: Jupyter notebooks as fal scripts, execute_sql magic function, dbt source test results.

Introducing Post Hooks

fal post hooks let you define Python scripts that don't need to write to the data warehouse.

Introducing fal Python Models

fal enables you to run Python with your dbt project. Today we are enabling Python-only models anywhere in your dbt project: Python Data Models.

Supporting Athena and DuckDB

We’re constantly working on making it as easy as possible to integrate fal into our users’ projects. So far we have supported PostgreSQL, Google BigQuery, Snowflake and AWS Redshift. Today we’re adding two more: AWS Athena and DuckDB. Star us on GitHub [https://github.com/fal-ai/fal]Amazon

Announcing fal Launch Week #2

We are excited to bring you our second Launch Week! Over the past couple of months we have been working on exciting new features to make running Python even more delightful. Similar to the previous launch week [https://blog.fal.ai/announcing-fal-launch-week-1/], each day of the week we are going

How-to

Use fal to integrate SageMaker with dbt

We discuss a straightforward way to integrate your dbt project with SageMaker by using fal. We will train a SageMaker model, store model data in a dbt model, use a SageMaker model to make some test predictions and store prediction results in another dbt model.

Python or SQL? Why not both?

Python vs SQL is a frequent debate in the data community and it usually revolves around which language is better for data transformations. At Features and Labels [https://fal.ai/], we think about it a little differently. In our opinion, today, all transformations that are supported by SQL should be

Parallelizing Python Workflows in fal

Today's release introduces parallelization of Python scripts in your fal runs. During a run, fal creates a directed acyclic graph (DAG) of links between models and scripts. The number of threads represents the maximum number of tasks fal handles simultaneously. When fal is running, it will read the configuration for

Introducing fal flow

We started the fal project with a simple objective; we wanted to build a developer tool that enables running python scripts downstream of dbt models. Since we launched the initial capabilities of fal, we have received an overwhelming amount of requests to run python scripts before, after and in between

Announcing fal Launch Week #1

After our Unbundling of Airflow [https://blog.fal.ai/the-unbundling-of-airflow-2/] article, it has been a wild few weeks here at Features & Labels. Gorkem’s [https://twitter.com/gorkemyurt] now-viral post resonated with tens of thousands of data people around the globe. Meanwhile, we have been hard at work cooking up

How-to

Going from dbt models to pandas DataFrames

One of our favorite features of dbt is the ability to build dependencies between data models using the ref function. With ref, you can build pipelines all in SQL, simply by referring to other tables in your select statements. But what if you want to refer to these models in

How-to

Populate dbt models with CSV data

A common source of raw data in ELT pipelines are CSV files. These text files hold data in multiple lines with headers and need to be parsed and loaded into data warehouses, ideally in an automated process. One way to load CSV data is by using the dbt seed command.

What’s new in fal 0.2.0

We’re constantly working on improving fal for our users by looking into issues and feature requests. You can create a Github issue [https://github.com/fal-ai/fal/issues], if you also would like something in fal to be improved. Today we are releasing a new version of fal with

Featured

The Unbundling of Airflow

Star us on GitHub In each subdomain of software, we have seen the same story over and over again. Products start small, in time, add adjacent verticals and functionality to their offerings and end up becoming a platform. Once these platforms become big enough, people start to figure out how

How-to

How to run Python with dbt Cloud using Github Actions and dbt Cloud API

dbt [https://www.getdbt.com/] is our favorite tool to build data pipelines. It allows us to skip boilerplate data engineering code, focus only on SQL and helps us build with software engineering practices like reusability, auto generating docs and tests. We recently introduced our open source project fal [https: