Sunday, 4 June 2023

Airflow dags template for everyone

airflow dags practice, had some time to waste, through of creating this template so that I can use it later in live projects or walk it through during an interview. 



 from airflow import DAG

from airflow.operators.bash import BashOperator

from datetime import datetime

with DAG('group_dag', start_date=datetime(2022, 1, 1), schedule_interval='@daily', catchup=False) as dag:
         
         
         download_a=BashOperator(
             task_id='download_a',
             bash_command='sleep 10'
         )
         download_b=BashOperator(
             task_id='download_b',
             bash_command='sleep 10'
         )
         download_c=BashOperator(
             task_id='download_c',
             bash_command='sleep 10'
         )

         check_files = BashOperator(
             task_id = 'check_files',
             bash_command='sleep 10'
         )

         transform_a=BashOperator(
             task_id='transform_a',
             bash_command='sleep 10'
         )
         transform_b=BashOperator(
             task_id='transform_b',
             bash_command='sleep 10'
         )
         transform_c=BashOperator(
             task_id='transform_c',
             bash_command='sleep 10'
         )

         [download_a, download_b, download_c] >> check_files >> [transform_a, transform_b, transform_c]


This is how the dag will look like in graph







No comments:

Post a Comment