Apache Airflow

Apache Airflow is like cron jobs on steroids.

Here are some of its benefits over cron jobs.

• Task retries
• Dependencies between tasks
• Logs and success/fail metrics that go back in time
• Automatic parallelization of tasks that don’t depend on each other
• Centralized credential store so you don’t have to update each script when a password changes.
• Failure emails (to be fair, cron has this too, but it’s a bit clunky to set up)

Leave a Comment