Data Pipeline Development & Management
.Design, develop, and maintain robust ETL/ELT pipelines to process large volumes of structured data
.Implement batch data processing solutions using Databricks
.Monitor data pipeline performance and troubleshoot issues to ensure reliable data flow
.Optimize existing pipelines for improved performance and cost efficiency
Data Architecture & Infrastructure
.Collaborate with senior engineers to design and implement scalable data architectures
.Build and maintain data warehouses, data lakes, and cloud-based storage solutions
.Ensure data quality, consistency, and reliability across all systems and datamarts
.Implement data governance best practices and security protocols defined by MOH HIM policies and DGPO
Technical Implementation
.Develop and maintain notebooks for data access and integration
.Write clean, efficient, and well-documented code following best practices
.Implement automated testing and deployment processes for data pipelines
Collaboration & Communication
.Partner with data scientists, analysts, clinical and business stakeholders to understand data requirements
.Provide technical guidance on data availability, limitations, and best practices to stakeholders
.Document data transformation processes, onboarding processes and automation processes
.Participate in code reviews and contribute to team knowledge sharing
Requirements
Bachelor's degree in Computer Science, Information Systems, Analytics, or a relevant field of study
1-3 years of experience in data engineering or a related role
Proficient in SQL and at least one programming language (Python, Java, Scala)
Decent understanding of data warehousing concepts and ETL processes
Familiarity with cloud platforms such as AWS and Databricks
Familiarity with Git commands and Unix/Linux system experience
Experience with data visualization tools (Tableau, Power BI)
Strong analytical and problem-solving skills
Familiar with database management, architecture design, system integration, security and data governance
Good interpersonal skills, a detail-oriented and flexible person who can work across different areas within the team as well as align cross functionally outside the team
Ability to deliver clear, concise reports and presentations/dashboards and effectively articulate observations and recommendations