Querying Deltalake From Apache Superset Dashboard

Setup

This assumes the Docker Compose setup from the Superset repo. Superset needs a few extra packages: duckdb-engine so it can talk to DuckDB at all, deltalake to read the Delta log, minio to presign the URLs, and python-dotenv for the credentials. In the Docker setup, extra packages go in docker/requirements-local.txt, and then you rebuild the stack.

duckdb-engine
deltalake
minio
python-dotenv

Next, put the MinIO credentials in a .env file next to the config (docker/pythonpath_dev/.env).

AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
AWS_ENDPOINT_URL=http://minio:9000

Then the config itself. I originally edited superset_config.py directly, but the Superset Docker docs say to put local overrides in docker/pythonpath_dev/superset_config_docker.py (it’s git-ignored and gets loaded on top), so that’s where this goes.

from datetime import timedelta
from pathlib import Path

from deltalake import DeltaTable
from dotenv import dotenv_values
from minio import Minio

REQUIRED_KEYS = (
    "AWS_ACCESS_KEY_ID",
    "AWS_SECRET_ACCESS_KEY",
    "AWS_ENDPOINT_URL",
)


def load_minio_env() -> dict[str, str]:
    """Read MinIO credentials from the .env next to this file."""
    env = dotenv_values(Path(__file__).with_name(".env"))
    missing = [key for key in REQUIRED_KEYS if not env.get(key)]
    if missing:
        raise ValueError(f"missing in .env: {missing}")
    return {k: v for k, v in env.items() if v is not None}


def delta_table(bucket: str, table_path: str) -> list[str]:
    """Return presigned URLs for the active files of a Delta table."""
    env = load_minio_env()
    table = DeltaTable(
        f"s3://{bucket}/{table_path}",
        storage_options={
            "AWS_ACCESS_KEY_ID": env["AWS_ACCESS_KEY_ID"],
            "AWS_SECRET_ACCESS_KEY": env["AWS_SECRET_ACCESS_KEY"],
            "AWS_ENDPOINT_URL": env["AWS_ENDPOINT_URL"],
            "AWS_REGION": env.get("AWS_REGION", "us-east-1"),
            "AWS_ALLOW_HTTP": "true",
        },
    )
    # The MinIO client wants host:port, not a URL.
    endpoint = env["AWS_ENDPOINT_URL"].split("//", 1)[1]
    client = Minio(
        endpoint,
        access_key=env["AWS_ACCESS_KEY_ID"],
        secret_key=env["AWS_SECRET_ACCESS_KEY"],
        secure=False,
    )
    prefix = f"s3://{bucket}/"
    return [
        client.presigned_get_object(
            bucket,
            uri.removeprefix(prefix),
            expires=timedelta(days=1),
        )
        for uri in table.file_uris()
    ]


FEATURE_FLAGS = {"ENABLE_TEMPLATE_PROCESSING": True}
JINJA_CONTEXT_ADDONS = {"delta_table": delta_table}

Two things here are easy to miss. First, Jinja does nothing until the ENABLE_TEMPLATE_PROCESSING feature flag is on. Without it the {{ ... }} just goes to the database as literal text. (If you already set other FEATURE_FLAGS, add it to that dict rather than replacing it.) Second, the storage_options keys. The delta-rs docs use AWS_ALLOW_HTTP for a MinIO that serves plain HTTP, which is what you get from a default local MinIO on port 9000. That’s also why the MinIO client has secure=False. If your MinIO sits behind HTTPS, flip it to True.

Finally, add a database in Superset with the SQLAlchemy URI duckdb:///:memory:. DuckDB doesn’t need to store anything here, it only reads the Parquet files on each query.

Querying It

Then in SQL Lab, against the DuckDB database, you call the function like any other Jinja macro.

SELECT test_id, test_actual_duration
FROM read_parquet({{ delta_table('spatch', 'path/to/table') }})
LIMIT 7;

The reason this works at all is kind of lucky. delta_table returns a Python list of strings, and Jinja renders the list with Python’s own formatting, ['http://...', 'http://...']. That happens to be valid DuckDB list syntax, and read_parquet accepts a list of files.

Where the Trick Falls Apart

This is a hack, and DuckDB only ever sees Parquet files. So you lose anything that lives only in the Delta log, such as these three I know about.

• Deletion vectors. Rows that were soft-deleted are still in the Parquet files, so they come back.
• Partition columns. Delta stores partition values in the log and usually not in the data files, so those columns are missing from read_parquet unless you recover them from the file paths.
• Column mapping. If the table renamed columns using column mapping, you get the physical column names.

Also be careful who can use this. delta_table takes the bucket and path straight from the SQL, so anyone with SQL Lab access can presign any file the MinIO credentials can read, and those URLs end up in the query history. Give Superset a MinIO user that can only read the tables you actually want on dashboards.

The presigned URLs also expire after one day, which is fine for SQL Lab since the function runs on every query, but every query also re-reads the Delta log and presigns every active file. The whole lakehouse is multiple terabytes spread over millions of files, so pointing this at a raw table would be hopeless, since every one of those URLs ends up pasted into the SQL text. The dashboards only read the nightly aggregate tables, though, and those charts took 5 to 10 seconds to load. Not fast, but fine for numbers that change once a day.