Supporting Athena and DuckDB
We’re constantly working on making it as easy as possible to integrate fal into our users’ projects. So far we have supported PostgreSQL, Google BigQuery, Snowflake and AWS Redshift. Today we’re adding two more: AWS Athena and DuckDB.
Amazon Athena is a query service that allows users to analyze data stored in their S3 buckets. Athena supports standard SQL and has a community supported dbt adapter so you can turn data stored in S3 into dbt models. After some contributions to the dbt-athena adapter, we now support Athena in fal as well.
profiles.yml like this:
fal_test: target: dev outputs: dev: type: athena s3_staging_dir: s3://my_bucket/my_staging_dir region_name: my_region schema: my_schema database: my_athena_catalog num_retries: 3
You can use all the magic functions, including ref and write_to_model, in order to interact with your Athena database from a Python context.
DuckDB is an in-process DB that lets you query Parquet and CSV files using SQL. It designed to support analytical queries without setting up a database instance and has been gaining popularity. The newest release of fal supports DuckDB out of the box. In order to use it, you must already have dbt-duckdb adapter installed, and have a
profiles.yml file that looks like this:
fal_test: target: dev outputs: dev: type: duckdb path: path_to_my_duckdb
fal will then run scripts against DuckDB and make all the magic functions and variables available, with the exception of write_to_source.