Job DescriptionJob Description
Astyra’s client is seeking a hands-on Senior Data Engineer to support a large-scale data and technology modernization initiative. This position will focus on modernizing legacy ETL workloads, developing Spark-based data pipelines, and ensuring data accuracy and consistency as workloads transition to modern platforms.
The ideal candidate has deep hands-on experience with Spark and Scala and a strong background designing, building, optimizing, and validating enterprise data pipelines.
What You’ll Do
- Design, build, and maintain scalable data pipelines using Spark and Scala
- Support the migration of legacy ETL workloads into a modern Spark-based data framework
- Develop and enhance data transformation and processing solutions
- Perform data comparison, reconciliation, and validation between legacy and target platforms
- Identify and troubleshoot data discrepancies throughout the migration process
- Develop data quality checks and controls to ensure completeness and accuracy
- Optimize Spark jobs and pipelines for performance, scalability, and reliability
- Work closely with data engineers, architects, and other technical teams throughout the modernization effort
- Support testing, deployment, troubleshooting, and ongoing maintenance of production data pipelines
- Contribute to broader feature engineering and data modernization initiatives
What We’re Looking For
- 7+ years of experience in data engineering or a related technical discipline
- Strong hands-on experience designing and developing enterprise data pipelines
- Advanced experience with Apache Spark
- Strong development experience with Scala
- Experience with ETL development, modernization, and migration initiatives
- Strong understanding of data transformation, data quality, and reconciliation
- Experience comparing and validating data across legacy and modernized systems
- Experience developing and supporting production data-processing workloads
- Familiarity with modern CI/CD and software development practices
- Strong troubleshooting and problem-solving skills
Qualifications
- Experience migrating legacy ETL platforms to Spark-based architectures
- Experience supporting large-scale technology or data modernization initiatives
- Experience within financial services or other highly regulated environments
- Python experience
- Exposure to Generative AI or AI-assisted development and modernization tools
- Previous experience within a large enterprise engineering environment
Proper email communication will only be done to and from @astyra.com email addresses. Please ensure you are communicating with approved Astyra recruiters by checking this point when receiving offers and messages from us. Please ensure you are communicating within these guidelines and proper channels for the quickest possible interview consideration!
#AC