Apache Airflow Check Previous Run Status for a DAG
The Problem
We encountered a scenario where if any previous DAG run fails the next scheduled DAG run should not proceed. Checked for the solution and did not find a robust solution. So I created one working with Shahed Munir
The Solution
I came up with a very generic approach that can be applied to any DAG that is created within Apache Airflow. Here is a sample DAG that uses and applies this concept in practice.
To enable the capability of checking any of the previous DAG run (self references) the first task needs to be created on the lines provided in the sample DAG within GitHub https://github.com/deliverbi/Airflow_Code.git
This solution relies on a Airflow Connection a mysql database used to store the Airflow Metadata.
In the sample DAG code this Airflow connection is called deliverbi_mysql_airflow
Then place the sample DAG in the dags directory.
Notes
All previous DAG runs must be set to success for the current DAG run to proceed. This task should be the first in any DAG that is created to enable this functionality
Here is the link to GitHub Repo Click Here to go to GitHub for DeliverBI
Over and Out. Krishna
5 comments
Write commentsAndroid Apps Download
ReplyWindows Software Download
Mac Software Download
Software Free Download
Crack Software Download
Premium Software Free Download
Crack Software Site
Your style is so unique compared to other people I’ve read stuff from. Thank you for posting when you’ve got the opportunity, Guess I’ll just book mark this web site.Wondershare PDFelement Pro Crack
ReplyI am very impressed with your post because this post is very beneficial for and provide a new knowledge to me. Thanks for sharing.Folder Lock Crack
ReplyWell done! Your explanation way is very attractive for me and I like more tips from your great post. Surely I will share your great post for my friends and this is very useful for freshers. Keeping the great work!
Replyreflector mac crack
Amazing blog nice article
ReplyDLL Files Fixer 2021 Crack
Sylenth1 Crack
Apeaksoft DVD Creator Crack