Building Event-Driven Data Pipelines With Airflow 3 at Astrafy with Andrea Bombino

Real-time data expectations are reshaping how modern data teams think about orchestration and dependencies. As event-driven architectures become more common, teams need to rethink how pipelines react to data changes, rather than schedules.

In this episode, Andrea Bombino, Co-Founder and Head of Analytics Engineering at Astrafy, joins us to discuss how event-driven scheduling in Airflow is evolving and how Astrafy applies it to deliver faster, more responsive data pipelines.

Key Takeaways:

00:00 Introduction.

02:02 Astrafy’s role in guiding clients across the modern data stack.

03:15 Strong DAG dependencies create challenges for time-based scheduling.

04:48 Event-driven pipelines respond to increasing real-time data demands.

05:30 Airflow 3 introduces native support for event-driven orchestration.

06:27 Sensor-based workflows reveal scalability and efficiency limitations.

11:32 Event-driven assets improve efficiency and pipeline elegance.

14:45 Governance and cross-instance coordination emerge as ongoing challenges.

Resources Mentioned:

Andrea Bombino

https://www.linkedin.com/in/andrea-bombino/

Astrafy | LinkedIn

https://www.linkedin.com/company/astrafy/

Astrafy | Website

https://www.astrafy.io

Apache Airflow

https://airflow.apache.org/

Google Cloud

https://cloud.google.com/

Google Pub/Sub

https://cloud.google.com/pubsub

Google BigQuery

https://cloud.google.com/bigquery

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow