Apache Oozie: The Workflow Scheduler for Hadoop

Download Full Version of the eBook "Apache Oozie: The Workflow Scheduler for Hadoop"

Apache Oozie: The Workflow Scheduler for Hadoop by Mohammad Kamrul Islam

Download - Apache Oozie: The Workflow Scheduler for Hadoop by Mohammad Kamrul Islam - PDF 

Hadoop is fast becoming the de facto big data platform across all industries. An entire ecosystem of tools, products, and services targeting every functionality and requirement have sprung up around Hadoop. Apache Oozie occupies an important space in this ever-expanding ecosystem. Since Hadoop’s early days at Yahoo!, it has been a natural platform for Extract, Transform, and Load (ETL) and other forms of data pipelines. Without a mature workflow management and scheduling system, implementing such pipelines can be a challenge. Oozie satisfies these requirements and provides a viable tool to implement complex, real-world data pipelines. In this book, we have tried our best to introduce readers to all the facets of Oozie and walk them through the intricacies of this rather powerful and flexible platform.


Software workflow systems are ubiquitous and each system has its own idiosyncrasies. But Oozie is a lot more than just another workflow system. One of Oozie’s strengths is that it was custom built from the ground up for Hadoop. This not only means that Oozie works well on Hadoop, but that the authors of Oozie had an opportunity to build a new system incorporating much of their knowledge about other legacy workflow systems. Although some users view Oozie as just a workflow system, it has evolved into something more than that. The ability to use data availability and time-based triggers to schedule workflows via the Oozie coordinator is as important to today’s users as the workflow. The higher-level concept of bundles, which enable users to package multiple coordinators into complex data pipelines, is also gaining a lot of traction as applications and pipelines moving to Hadoop are getting more complicated.


We are both very lucky to have been involved in Oozie’s journey from its early days. We have played several roles in its evolution, ranging from developer, architect, open source committer, Project Management Committee (PMC) member, product manager, and even demanding customer. We have tried to leverage all of that perspective to present a comprehensive view of Oozie in this book. We strongly believe in the vision of Oozie and its potential to make Hadoop a more powerful platform. Hadoop’s use is expanding and we notice that users want to use it in smarter and more interesting ways. We have seen many projects in the past getting bogged down with writing, operating, and debugging the workflow system meant to manage the business application. By delegating all of the workflow and scheduling complexities to Oozie, you can focus on developing your core business application.


This book attempts to explain all the technical details of Oozie and its various features with specific, real-world examples. The target audience for this book is Oozie users and administrators at all levels of expertise. Our only requirement for the reader is a working knowledge of Hadoop and the ecosystem tools. We are also very aware of the challenges of operating a Hadoop cluster in general and Oozie in particular, and have tried our best to cover operational issues and debugging techniques in depth. Last but not the least, Oozie is designed to be very flexible and extensible and we want to encourage users to get comfortable with the idea of becoming an Oozie developer if they so desire. We would love to grow the Oozie community and continue the innovation in this part of the Hadoop ecosystem. While it would be nice to achieve all of these goals with this book, the most fundamental hope is that readers find it helpful in using Oozie and Hadoop more effectively every day in their jobs.


116
Views
0
Likes

Licenses:

  • CC BY-NC-SA 3.0 PH
  • The author's reference is not required

Share on networks

eBooks Details:

Comments (0) Add

Кликните на изображение чтобы обновить код, если он неразборчив
No comments yet. Your comment will be the first!