Senior Data Engineer

• Design, develop, and optimize data solutions using Azure Databricks, Python/PySpark, Spark SQL, and Delta Lake.
• Develop and maintain data pipelines following the Medallion Architecture (Bronze, Silver, Gold) to deliver scalable and trusted data products.
• Implement ingestion frameworks for structured and semi-structured data from APIs, JSON files, databases, and other enterprise sources.
• Manage and optimize data storage solutions using Azure Data Lake Storage Gen2 (ADLS Gen2) and associated Azure services.
• Build, schedule, and support workloads using Databricks Workflows/Jobs, with experience in Delta Live Tables (DLT) or Lakeflow considered a strong advantage.
• Develop and maintain data quality controls, validation frameworks, reconciliation processes, and automated testing.
• Design and implement data transformation, cleansing, and standardization processes to support business reporting and analytics.
• Establish metadata management, data lineage, and governance capabilities using Unity Catalog and related technologies.
• Develop API-based ingestion and export integrations for upstream and downstream systems.
• Support solutions involving document metadata management and references to PDF and eLabel assets.
• Drive performance tuning, partitioning strategies, and platform optimization to improve scalability and efficiency.
• Implement CI/CD practices for notebooks, code deployments, and automated testing throughout the development lifecycle.
• Develop monitoring, alerting, error-handling, and reprocessing capabilities to ensure operational stability and reliability.
• Support regulated data environments with a strong focus on traceability, auditability, reproducibility, and controlled release management.
  

Key Responsibilities

• Design, develop, and optimize data solutions using Azure Databricks, Python/PySpark, Spark SQL, and Delta Lake.
• Develop and maintain data pipelines following the Medallion Architecture (Bronze, Silver, Gold) to deliver scalable and trusted data products.
• Implement ingestion frameworks for structured and semi-structured data from APIs, JSON files, databases, and other enterprise sources.
• Manage and optimize data storage solutions using Azure Data Lake Storage Gen2 (ADLS Gen2) and associated Azure services.
• Build, schedule, and support workloads using Databricks Workflows/Jobs, with experience in Delta Live Tables (DLT) or Lakeflow considered a strong advantage.
• Implement schema management practices, including schema enforcement and schema evolution.
• Develop and maintain data quality controls, validation frameworks, reconciliation processes, and automated testing.
• Design and implement data transformation, cleansing, and standardization processes to support business reporting and analytics.
• Establish metadata management, data lineage, and governance capabilities using Unity Catalog and related technologies.
• Develop API-based ingestion and export integrations for upstream and downstream systems.
• Support solutions involving document metadata management and references to PDF and eLabel assets.
• Drive performance tuning, partitioning strategies, and platform optimization to improve scalability and efficiency.
• Implement CI/CD practices for notebooks, code deployments, and automated testing throughout the development lifecycle.
• Develop monitoring, alerting, error-handling, and reprocessing capabilities to ensure operational stability and reliability.
• Support regulated data environments with a strong focus on traceability, auditability, reproducibility, and controlled release management.
  

Desired Skills and Experience

Required Qualifications

• Strong hands-on experience with Azure Databricks, Python/PySpark, Spark SQL, and Delta Lake.
• Deep understanding of modern data lakehouse architectures, including the Bronze-Silver-Gold (Medallion) design pattern.
• Experience developing enterprise-scale ETL/ELT pipelines and data integration solutions.
• Strong knowledge of Azure storage technologies, particularly ADLS Gen2.
• Experience implementing data governance, lineage, and access control frameworks.
• Proven experience with data quality, validation, monitoring, and operational support.
• Experience working in regulated environments where compliance, auditability, and data traceability are key requirements.

Preferred Qualifications

• Experience with Delta Live Tables (DLT), Lakeflow, and advanced Databricks orchestration capabilities.
• Familiarity with document-centric data solutions, including PDF and eLabel metadata management.
• Experience implementing DevOps and CI/CD best practices within data development teams.
• Experience in pharmaceutical, healthcare, life sciences, or other highly regulated industries.