ETL Test Automation Engineer – Azure Data Platform

The QA & Automation Engineer will be responsible for validating ETL pipelines, data migration processes, and cloud data platform integrations across Azure services. This role involves hands‑on testing of large-scale data platforms, automation development, advanced SQL validation, and ensuring end‑to‑end data quality across enterprise data engineering initiatives.

Key Responsibilities

• Perform end‑to‑end testing of ETL pipelines built using Azure Data Factory, Azure Databricks, Azure Synapse, and SSIS.
• Validate data transformations, mappings, data quality rules, and data lineage.
• Conduct source‑to‑target (S2T) reconciliation, data profiling, completeness, and accuracy checks.
• Verify schema changes, incremental loads, delta loads, and historical load processes.

Data Migration Testing

• Design and execute test strategies for large‑scale on‑prem to cloud or cross‑cloud data migration projects.
• Validate ETL/ELT processes and post‑migration data accuracy.
• Perform count checks, checksum validation, CDC validation, duplicate checks, and referential integrity validation.
• Review and validate data mapping documents, business rules, and acceptance criteria.

Data Lake & Cloud Platform Testing

• Test ingestion pipelines into Azure Data Lake Storage (ADLS Gen2) from multiple source systems.
• Validate partitioning, folder structures, file formats (Parquet, CSV, JSON), and governance standards.
• Conduct performance and scalability testing for large data workloads.
• Validate integration flows across Data Lake, Synapse, ADF, and Power BI.

Automation Testing

• Develop automated data validation and regression frameworks using Python, PySpark, and ADF automated validation frameworks.
• Integrate automated tests within CI/CD pipelines using Azure DevOps.
• Build reusable automation templates, accelerators, and validation utilities.
• Create and maintain test plans, scenarios, and test cases in Azure DevOps or Jira.
• Log, track, triage, and validate defects in collaboration with development and data engineering teams.
• Publish daily/weekly QA health metrics, defect reports, coverage summaries, and quality dashboards.

Experience and Qualifications

• 4–10 years of experience in Data Warehouse / Data Engineering QA.
• Strong hands‑on expertise with ETL tools such as ADF, SSIS, Informatica, and Databricks.
• Expert‑level SQL for complex data validation and reconciliation.
• Strong understanding of data warehousing concepts (SCDs, facts/dimensions, star schema).
• Strong analytical, debugging, and problem‑solving skills.

Preferred Skills

• Quality management and defect lifecycle management.
• Integration of automated tests into CI/CD pipelines using Azure DevOps.