For those who already played with NiFi do know what it is capable of. So this first post is targeting the people who work in the software industry but doesn't have a clue about NiFi.
Read these tweets before get started :).
After using @apachenifi, it feels like I was cutting steak with a butter knife for years.— Manish Gupta (@manishpedia) October 29, 2015
If you're not using @ApacheNifi, you're probably doing too much work. #TheresANarForThat— Brian Ghigiarelli (@brianghig) September 4, 2015
Interesting right? Well simply if you are a software developer and if you are not aware of NiFi, then you are probably doing too much of work.
How did I get to know about this?
The first project I got with my new job was a Big data project. We had to accept a huge amount of data from different sources and move them into different places and process those using programs running on Hadoop. Not only that the data delivery guarantee should be there. Once the customer gives us the data, no data can be lost.
So simply we had following requirements to be implemented in our system.
1. Accept data from various sources
2. Write the original content of the data to hdfs
3. Write some of the selected content to Hbase
4. Write some other content to MongoDb
5. Those data should be processed using scheduled jobs
6. Any data must not be lost.
So we tried to implement simple java programs(Modules) to accept data from sources and route those data in given paths according to the requirement. So we realized that if we did that in that manner that would take months. So my project architect came up with this nice little solution called Apache NiFi.
So we were able to complete whole data ingestion flow within a couple of days. All we had to do left was to create some jobs to process those data( Later we realized that even processing the data can be done by using Nifi).
So months of workes narrowed down to days. What we did was just dragging and dropping some icons and configuring some parameters. The full data flow was completed and the data delivery was also guaranteed.
I'll write more about Apache nifi in future. So meanwhile anybody who wants to get rid of data ingestion hassles try nifi!
https://nifi.apache.org/
0 comments:
Post a Comment