About the role
We are seeking a highly skilled and experienced data lake cloud engineer with a proven track record of designing, implementing, and maintaining large-scale cloud-based data lake platforms. This role requires a professional who can take ownership of our current data lake ecosystem, optimize its performance, and drive future enhancements with minimal oversight. The ideal candidate will have at least 5 years of hands-on experience in building enterprise-grade data lakes, strong cloud architecture expertise, and the ability to work with cutting-edge data ingestion, processing, and analytics tools.
Key responsibilities
Take ownership of the existing enterprise data lake platform, ensuring scalability, reliability, and performance.
Lead the design, architecture, and implementation of cloud-native data lake solutions and integrations.
Manage and optimize data ingestion pipelines on Oracle OCI, using tools such as Apache NiFi, Kafka, batch processing of data, data captures, and CSV.
Design and implement pipelines for network data ingestion and file formats (e.g., Parquet, Avro, ORC, etc.), ensuring efficient storage, processing, and retrieval.
Build, configure, and tune query engines such as Trino (Presto), Spark, and Hive for efficient analytics and reporting.
Implement and maintain metadata management, data governance, and security frameworks.
Monitor and troubleshoot system performance, ensuring SLAs are met for ingestion, processing, and query workloads.
Automate platform deployment, monitoring, and maintenance with infrastructure-as-code (Terraform, CloudFormation, etc.).
Collaborate with data engineers, analysts, and business teams to understand data requirements and deliver solutions that maximize data accessibility and usability.
Keep the data platform up to date with the latest open-source and cloud-agnostic technologies, implementing upgrades and enhancements where needed.
Requirements
5+ years of proven, hands-on experience implementing and managing large-scale data lakes in the cloud (OCI).
Strong expertise in:
- Data ingestion & orchestration: Apache NiFi, Apache Kafka, CSV, and others
- Data processing frameworks: Apache Spark, PySpark, Trino (Presto), Hive, Flink
- Storage & lakehouse architectures: Delta Lake, Apache Hudi, Iceberg, and cloud-native object storage (S3)
- Query & analytics tools: Trino/Presto, SparkSQL, Metabase, or Apache Superset
Experience with data lake file formats such as Apache Parquet, Avro, ORC, CSV, etc., including ingestion, parsing, and analytics within a data lake.
Solid understanding of data governance, lineage, cataloging, and security frameworks (Apache Atlas).
Experience with CI/CD and IaC (ArgoCD, Terraform, Ansible) for automated deployments.
Hands-on experience with cloud security best practices, including IAM, encryption, and network security.
Strong proficiency in Python or Java for data engineering and automation tasks.
Proven ability to work independently, quickly understand existing environments, and deliver results without extensive training.
Preferred skills
Exposure to machine learning workflows integrated with data lakes.
Experience with real-time streaming data pipelines.
Familiarity with containerization and orchestration (Docker, Kubernetes).
Knowledge of cost optimization strategies in cloud-based data platforms.