Since Airflow 2.0, the makes handling data between tasks much cleaner. When you return a value from a @task decorated function, it is automatically pushed as an XCom.
Demystifying Apache Airflow XComs: The Exclusive, Definitive Guide
This is where (cross-communication) comes in. XComs are the primary mechanism within Airflow for sharing small amounts of data between tasks within a DAG run. airflow xcom exclusive
: Tasks retrieve data using xcom_pull() , which can be filtered by task_ids , dag_id , or a specific key . Advanced "Exclusive" Strategies
: Keep default XCom payloads under a few kilobytes. Since Airflow 2
An XCom is explicitly defined by a DAG ID, a Task ID, a execution/logical date, and a unique key.
By combining XComs with Airflow's trigger rules (e.g., one_success , all_failed , one_failed ), you can build sophisticated conditional workflows where the data itself determines which tasks run. XComs are the primary mechanism within Airflow for
For true exclusivity and performance, many teams use a . This allows you to: Store the actual data in S3, GCS, or Azure Blob Storage . Only store the reference (the URI) in the Airflow database. Implement lifecycle policies to auto-delete old XCom data.
Uploads the heavy object to an external bucket (such as AWS S3, Google Cloud Storage, or Azure Blob). Generates a unique string URI pointing to that object.
Airflow converts Python objects into JSON format to write them to the metadata database. If your task returns a non-serializable object (such as an open file handle, a live database connection pool, or a complex custom class instance), Airflow will throw an explicit serialization exception and fail the task run. Advanced Architecture: Custom XCom Backends
In the world of Apache Airflow, task isolation is a core principle. Tasks are designed to run independently, potentially on different machines, ensuring that a failure in one node doesn’t bring down the entire workflow. However, data pipelines frequently require tasks to "talk" to each other—passing filenames, configuration parameters, or execution status.