Data Engineer
HCLTech · Netherlands ·
- Employment
- Full time
- Category
- Data engineering
Builds and migrates Python/PySpark data pipelines on Databricks for corporate credit-risk models, converting SAS code and ensuring data quality and performance.
Duration: Permanent Position
Work Location: Amsterdam, Netherlands
Job Description:
Seeking a highly skilled Python Data Engineer with expertise in Finance and Risk domain to join our dynamic team. The ideal candidate will have extensive experience in analyzing business requirements, designing technical solutions, implementing complex business logic, ensuring seamless transition and optimization of activities within the migration process. The position requires strong proficiency in Python, PySpark, and Databricks, experience with analytical models, and a solid understanding of data structures and quality. The candidate will oversee end-to-end development, from requirement analysis to deployment, while ensuring robust testing and validation processes.
Key Responsibilities:
- Requirement analysis: Analyse and interpret business documents related to Finance and Risk domain, especially corporate credit risk analytical models.
- Collaborate with stakeholders to translate business needs into clear technical requirements.
- Data analysis and understanding: Analyse and interpret data, understand data structures and concepts, assess data quality, identify inconsistencies, and resolve issues as needed.
- Design, build, and maintain: Scalable data pipelines, workflows using Python, PySpark, and Databricks. Enhance performance by leveraging the capabilities of Python and PySpark.
- Testing and Validation: Conduct thorough testing and validation of implementation to ensure they meet performance and functionality standards, ensuring comprehensive test coverage.
- Perform review, debugging, troubleshooting, and issue resolution.
- Deployment and Maintenance: Deploy solutions using Azure Pipeline and monitor and maintain deployed solutions.
- Code and Data Conversion: Convert SAS code of analytical assets to Python or PySpark, ensuring accuracy and efficiency of the new programs.
- Migration Planning: Contribute to developing a comprehensive migration plan, including timelines, resource allocation, and risk management.
- Evaluate all work to ensure compliance with internal Privacy and Security Policies and Procedures and migration project practices.
- Meet timelines and milestones by monitoring deliverables; identify, report, and help solve potential risks and issues.
- Documentation: Maintain detailed documentation of migration processes, including code changes, testing procedures, and performance metrics.
- Collaboration: Work closely with business analysts, modelling team, SAS developers, data scientists, data engineers, and other stakeholders to ensure successful migration and integration of programs.
- Training and Support: Provide training and support to team members on the new Databricks, Python/PySpark-based programs.
Seniority Level
Mid-Senior level
Employment Type
Full-time
Job Function
Information Technology
Industries
IT Services and IT Consulting