Enhancing Data Pipeline Efficiency in Large-Scale Data Engineering Projects

Authors

  • Naveen Bagam Author

Keywords:

Data Pipeline, Data Engineering, Pipeline Optimization, Scalability, Fault Tolerance, Big Data, ETL, Cloud Computing, Distributed Systems.

Abstract

Data pipelining is a basic component in managing and processing data at scale, especially in large organizations. Optimal utilization of the pipeline must encompass all aspects that ensure scalability, cost effectiveness, and reliability. It is against this background that this research paper takes a central focus on the strategies and best practices for improving pipeline efficiency through design principles, optimization techniques, management of resources, automation, and security. We base our work on the recent works and industrial and academic frameworks to examine the impact of emergence technologies and suggest how pipeline performance may be measured and benchmarked, with respect to operational improvements and data-driven decision-making.

Downloads

Published

21.07.2019

How to Cite

Enhancing Data Pipeline Efficiency in Large-Scale Data Engineering Projects. (2019). International Journal of Open Publication and Exploration, ISSN: 3006-2853, 7(2), 44-57. https://ijope.com/index.php/home/article/view/166

Most read articles by the same author(s)

1 2 3 4 5 6 7 8 9 10 > >>