Supporting Athena and DuckDB

We’re constantly working on making it as easy as possible to integrate fal into our users’ projects. So far we have supported PostgreSQL, Google BigQuery, Snowflake and AWS Redshift. Today we’re adding two more: AWS Athena and DuckDB.

Amazon Athena

Amazon Athena is a query service that allows users to analyze data stored in their S3 buckets. Athena supports standard SQL and has a community supported dbt adapter so you can turn data stored in S3 into dbt models. After some contributions to the dbt-athena adapter, we now support Athena in fal as well.

Given a profiles.yml like this:

fal_test:
  target: dev
  outputs:
    dev:
      type: athena
      s3_staging_dir: s3://my_bucket/my_staging_dir
      region_name: my_region
      schema: my_schema
      database: my_athena_catalog
      num_retries: 3

You can use all the magic functions, including ref and write_to_model, in order to interact with your Athena database from a Python context.

DuckDB

DuckDB is an in-process DB that lets you query Parquet and CSV files using SQL. It designed to support analytical queries without setting up a database instance and has been gaining popularity. The newest release of fal supports DuckDB out of the box. In order to use it, you must already have dbt-duckdb adapter installed, and have a profiles.yml file that looks like this:

fal_test:
  target: dev
  outputs:
    dev:
      type: duckdb
      path: path_to_my_duckdb

fal will then run scripts against DuckDB and make all the magic functions and variables available, with the exception of write_to_source.