In this article, I would like to give you a jump-start tutorial to understand the basic concepts and create a workflow pipeline from scratch.
So let's get started!
Apache Airflow is an orchestration tool that helps you to programmatically create and handle task execution into a single workflow. It then handles monitoring its progress and takes care of scheduling future workflows depending on the schedule defined.
Workflows are created using python scripts, which define how your tasks are executed. They are usually defined as Directed Acyclic Graphs(DAG).
In this blog, I take a practical approach to API first design with an example using Open API 3 specification.
Firstly, I describe how I created the API definition, then how I created the server and the client code from the API definition. Then I will talk about some of the problems I faced.
As we are adopting microservice-based architectures, API first approach have been gaining some traction. There are quite many advantages to using API first approach and I will discuss a few of them.
Clear Contract Definition With API first approach, you can create a concrete contract with…
I was facing a problem where I wanted to insert millions of records into the database which needed to be imported from the file.
So I did some research around this and I would like to share with you what I found which helped me improve the insert records throughput by nearly 100 times.
Initially when I was just trying to do bulk insert using spring JPA’s
saveAll method, I was getting a performance of about 185 seconds per 10,000 records . After doing the following changes below, the performance to insert 10,000 records was just in 4.3 seconds .
This article is about how you can handle AWS RDS secrets change or rotation without restarting your Spring Boot application.
I had this problem wherein I had to update my database connection whenever the database password was updated for my AWS RDS instance. This can be because of a monthly password rotation policy or may be the database credentials got compromised and you want all your running applications to keep running even when the database password are changed.
To solve this kind of a problem, AWS provides a library that will handle this updating of the database connection without even…