Data Engineer

RoleOverview

Weare looking for a hands-on Data Engineer to design, build, and optimizescalable data pipelines on the Databricks Lakehouse platform. The role involvescollaborating with analysts, data scientists, and business teams to deliverreliable, analytics-ready data assets.

KeyResponsibilities

• Build and maintain batch and streaming data pipelines using PySpark and Databricks .
• Implement medallion architecture (Bronze, Silver, and Gold layers).
• Optimize Spark jobs for performance and cost efficiency.
• Ensure data quality, lineage, and governance using Unity Catalog and automated validation.
• Integrate data from APIs, databases, files, and event streams.
• Implement CI/CD, version control, and workflow orchestration.
• Monitor, troubleshoot, and document data pipelines and data models.
• Collaborate with stakeholders to translate business requirements into technical solutions.

RequiredQualifications

• 3-5 years of hands-on data engineering experience.
• Strong PySpark skills, including DataFrames, Spark SQL, UDFs, and performance tuning .
• Proven experience building production pipelines on Databricks.
• Working knowledge of Delta Lake, including ACID transactions, time travel, merge/upsert, and schema evolution .
• Advanced SQL and strong Python programming skills .
• Experience with Azure, AWS, or GCP cloud platforms.
• Understanding of data modelling, Lakehouse architecture, and ETL/ELT patterns.
• Familiarity with Git and CI/CD practices.

PreferredQualifications

• Databricks Data Engineer Associate or Professional certification.
• Experience with Unity Catalog, Auto Loader, and Structured Streaming.
• Familiarity with Airflow, Azure Data Factory, or Databricks Workflows.
• Knowledge of Terraform and testing frameworks such as pytest.
• Exposure to Kafka, Event Hubs, dbt, or MLOps workflows.
• Experience working in regulated or enterprise environments.

KeySkills

• PySpark and Python
• Databricks and Delta Lake
• Advanced SQL
• Data Pipeline Development
• Medallion Architecture
• Cloud Data Engineering
• Data Quality and Governance
• CI/CD and Git
• Spark Performance Tuning

DesiredAttributes

• Strong problem-solving and debugging skills.

EA Number: 11C4879