Data Engineer with PySpark
base camp digital pte. ltd. · Singapore ·
- Category
- Data engineering
Job responsibilities
Deliver end-to-end data pipeline solutions on cloud infrastructure leveraging the latest technologies and the best industry practices.
Use domain modelling techniques to allow us to build best in class business products.
Structure software so that it is easy to understand, test and evolve.
Build solutions that avoid single points of failure, using scalable architectural patterns.
Develop secure code so that our customers and ourselves are protected from malicious actors.
Promptly investigate and fix issues and ensure they do not resurface in the future.
Make sure our releases happen with zero downtime for our end-users.
See that our data is written and read in a way that's optimized for our needs.
Keep an eye on performance, making sure we use the right approach to identify and solve problems.
Ensure our systems are reliable and easy to operate.
Keep us up to date by continuously updating our technologies and patterns.
Support the products you've built through their entire lifecycle, including in production and during incident management
Required qualifications, capabilities & skills
Formal training or certification on data engineering concepts and applied experience
Recent hands-on professional experience as a data engineer
Experience in coding in a recent version of the
Python
programming language Experience in designing and implementing effective tests (unit, component, integration, end-to-end, performance, etc.) Excellent written and verbal communication skills in English Experience with
AWS
cloud technologies and distributed systems Experience with data transformation frameworks and data pipeline orchestration tools Experience with managing large volumes of data and optimizing data processing Understanding of event-base architecture, data streaming and messaging frameworks Coach other team members on coding practices, design principles, and implementation patterns that lead to high-quality maintainable solutions. Manage stakeholders and effectively prioritize work across multiple work streams. Preferred qualifications, capabilities & skills Experience in working in a highly regulated environment / industry Experience with AWS cloud technologies Experience with data governance frameworks Understanding of incremental data processing and versioning Understanding of RESTful APIs and web technologies
Python
programming language Experience in designing and implementing effective tests (unit, component, integration, end-to-end, performance, etc.) Excellent written and verbal communication skills in English Experience with
AWS
cloud technologies and distributed systems Experience with data transformation frameworks and data pipeline orchestration tools Experience with managing large volumes of data and optimizing data processing Understanding of event-base architecture, data streaming and messaging frameworks Coach other team members on coding practices, design principles, and implementation patterns that lead to high-quality maintainable solutions. Manage stakeholders and effectively prioritize work across multiple work streams. Preferred qualifications, capabilities & skills Experience in working in a highly regulated environment / industry Experience with AWS cloud technologies Experience with data governance frameworks Understanding of incremental data processing and versioning Understanding of RESTful APIs and web technologies