Moustafa Elzeftawy

Work · Data platforms

Project Bifrost: an API without an API

A metadata-driven integration on AWS S3, Airflow and Snowflake. INX's customers can import and export their own data safely, without anyone babysitting the process. Presented at the Snowflake User Group Perth.

Organisation
INX × DataDivers
My role
Technical lead and solution architect (DataDivers side)
When
2023 – 2024
  • ~0.6 FTEsaved on manual integration work
  • Self-servicecustomer-defined mappings with automatic checks and alerts
  • STUGpresented at Snowflake User Group Perth
The integration pattern
  1. Request filecustomer drops a request into AWS S3
  2. Airflowpicks up the request and the customer's stored mapping
  3. Snowflakeloads and runs dynamically defined checks
  4. Upsert or exportin the customer's required format
  5. Notifyalerts on anything that fails
Snowflake User Group Perth meetup card, 16 April 2024, "Building an API-like integration… that is not an API"
Photo: Snowflake Community / INX

The problem

INX’s customers needed to move data in and out of the INX Core platform. Every integration was a hand-built SQL Server ETL job checked by hand. That was slow to set up and risky for INX’s database, and someone always had to watch it run.

The solution

We designed a parameterised, metadata-driven integration that behaves like an API without needing an API endpoint.

  1. Each customer defines its own data mappings for import and export.
  2. Dropping a request file into AWS S3 triggers an Airflow workflow.
  3. Airflow uses the stored mapping to pull the data into Snowflake, where dynamically defined checks validate it.
  4. Valid records are upserted, or the export is produced in the customer’s required format.
  5. Anything that breaks or fails a check sends a notification, so INX’s database is never put at risk and nobody needs to babysit the process.

I led the platform evaluation that chose Snowflake over Databricks and Informatica. The deciding factors were INX’s in-house capability, build and maintenance cost, and long-term integration needs.

My role

Technical lead and solution architect on the DataDivers side, leading a team of two to three. We planned the workflow and built the Airflow and Snowflake layers; INX owned the Terraform infrastructure.

Outcome

  • About 0.6 FTE saved, measured against INX’s actual manual integration effort before and after go-live.
  • I co-presented it with INX’s Head of Data at the Snowflake Technical User Group Perth on 16 April 2024: “Building an API-like integration… that is not an API”.