Position Overview
We are seeking an experienced Data QA Engineer to join our team and support Databricks development and migration projects. The ideal candidate will have a minimum of 5 years of hands‑on experience working with Databricks on cloud platforms such as AWS and Azure. In this role, you will collaborate closely with data engineering, analytics, and migration teams to ensure the highest quality standards for our data solutions and migration processes.
Key Responsibilities
- Design, develop, and execute comprehensive test strategies for Databricks-based data pipelines, ETL processes, and migration solutions on AWS and Azure.
- Develop and maintain automated test scripts to validate data accuracy, transformation logic, and system performance within Databricks.
- Work closely with developers, data engineers, and business analysts to define test requirements, acceptance criteria, and data validation protocols.
- Analyze data migration results, identify discrepancies, and drive resolution of data quality issues.
- Perform root cause analysis of defects and provide detailed reports to stakeholders.
- Contribute to the continuous improvement of QA processes, test automation frameworks, and best practices for Databricks on cloud platforms.
- Participate in code reviews, design discussions, and sprint planning activities to advocate for quality standards.
Must-Have Skills and Experience
- 5+ years of experience working with Databricks on cloud platforms (AWS, Azure).
- Strong expertise in data quality assurance, testing methodologies, and test automation for big data and cloud‑based environments.
- Hands‑on experience in writing and executing SQL queries for data validation and analysis.
- Proficiency with PySpark, Python, and Databricks notebooks for test automation and data validation.
- Experience in validating ETL pipelines, data transformations, and large‑scale data migrations.
- Solid understanding of cloud storage solutions (e.g., AWS S3, Azure Data Lake) and data integration patterns.
- Proven ability to troubleshoot and resolve data quality issues in complex environments.
- Excellent communication and collaboration skills.
Good‑to‑Have Skills
- Experience with CI/CD tools for data pipeline deployment and automated testing.
- Familiarity with other data platforms and tools such as Snowflake, Redshift, or Synapse Analytics.
- Knowledge of data governance, data lineage, and metadata management.
- Experience with performance testing and optimization of big data workloads.
- Exposure to scripting languages such as Scala or Shell scripting.
- Relevant certifications (e.g., Databricks Certified Data Engineer, AWS Certified Solutions Architect, Azure Data Engineer Associate).
Educational Qualifications
Bachelor’s or Master’s degree in Computer Science, Information Systems, Engineering, or a related field.
Seniority level
Mid‑Senior level
Employment type
Full‑time
Job function
Information Technology
Industries
IT Services and IT Consulting
Location
Miguel Hidalgo, Mexico City, Mexico
Salary
MX$15,000.00 per month