ii.To analyse and handle large volumes of data and utilize frameworks like Apache Hadoop and Apache Spark to distribute data processing tasks across multiple nodes;

iii.Designing and maintaining robust Extract, Transform, Load (ETL) pipelines to ensure smooth data flow and integration from various sources;

iv.To optimize data processing pipelines for performance and cost-effectiveness, utilizing technologies such as Hadoop, Spark and other Open Source technologies;

v.To integrate disparate datasets from different sources, formats and schemas, maintaining data lineage and metadata management;

vi.Apply the use of appropriate Machine Learning algorithms and models for extraction of useful information from large datasets to identify patterns, trends and relationships;

vii.To collaborate with cross-functional teams including data analysts, and business stakeholders to understand data requirements and ensure data accessibility and usability;

viii.To design and implement scalable data architectures and storage solutions to accommodate the volume, variety, and velocity of big data, leveraging technologies such as HDFS and OLAP (Online Analytical Processing) databases;

ix.To define data partitioning, indexing, and compression strategies to optimize storage efficiency and query performance;

x.To establish and enforce data governance policies, standards, and best practices to ensure data privacy, security, and compliance with Laws and regulations;

xi.To implement access controls, encryption, and auditing mechanisms to protect sensitive data and mitigate risks of data breaches or unauthorized access;

xii.To monitor data pipelines and systems for performance, availability, and reliability, proactively identifying and resolving issues to minimize downtime and data loss;

xiii.To conduct regular maintenance tasks such as data backups, system upgrades, and capacity planning to ensure the stability and scalability of the infrastructure;

xiv.To assist in developing and update technical documentation;

xv.To perform other related duties as may be assigned by the Supervisor.