Translate

Friday, 1 November 2019

Krishna

Apache Airflow Check Previous Run Status for a DAG

Apache Airflow Check Previous Run Status for a DAG


The Problem

We encountered a scenario where if any previous DAG run fails the next scheduled DAG run should not proceed. Checked for the solution and did not find a robust solution. So I created one working with Shahed Munir

The Solution

I came up with a very generic approach that can be applied to any DAG that is created within Apache Airflow. Here is a sample DAG that uses and applies this concept in practice. 

To enable the capability of checking any of the previous DAG run (self references) the first task needs to be created on the lines provided in the sample DAG within GitHub https://github.com/deliverbi/Airflow_Code.git

This solution relies on a Airflow Connection a mysql database used to store the Airflow Metadata.
In the sample DAG code this Airflow connection is called deliverbi_mysql_airflow

Then place the sample DAG in the dags directory.

Notes

All previous DAG runs must be set to success for the current DAG run to proceed. This task should be the first in any DAG that is created to enable this functionality

Here is the link to GitHub Repo Click Here to go to GitHub for DeliverBI

Over and Out. Krishna

8 comments

Write comments
Unknown
AUTHOR
2 November 2020 at 06:45 delete

Your style is so unique compared to other people I’ve read stuff from. Thank you for posting when you’ve got the opportunity, Guess I’ll just book mark this web site.Wondershare PDFelement Pro Crack

Reply
avatar
Unknown
AUTHOR
5 November 2020 at 11:21 delete

I am very impressed with your post because this post is very beneficial for and provide a new knowledge to me. Thanks for sharing.Folder Lock Crack

Reply
avatar
Anonymous
AUTHOR
22 November 2020 at 01:57 delete

Well done! Your explanation way is very attractive for me and I like more tips from your great post. Surely I will share your great post for my friends and this is very useful for freshers. Keeping the great work!
reflector mac crack

Reply
avatar
Cracked Apps
AUTHOR
3 June 2021 at 05:42 delete

Useful information.It’s truly a great and useful piece of info. Lucky me I came across your web site by accident, I bookmarked it.
PowerISO Crack
HitmanPro Crack
iMazing Crack
NSB Appstudio Crack
TeamViewer Crack
Avast Premium Crack
AVG Internet Security Crack
Movavi Video Converter Crack

Reply
avatar
19 October 2021 at 15:47 delete

I like your all post. You have done really good work. Thank you for the information you provide, it helped me a lot. I hope to have many more entries or so from you.
Very interesting blog.
howcrack.org
Apeaksoft DVD Creator Crack

Reply
avatar



About Authors

Shahed Munir

Krishna Udathu

Shahed and Krishna are Oracle / Big Data Experts