The Data Engineer will be responsible for designing, developing, and maintaining scalable and reliable data pipelines on Databricks and cloud platforms. The role requires integrating diverse data sources, ensuring high-quality data processing, and supporting analytics, reporting, and machine learning workloads. The role involves collaborating closely with analytics, product, and infrastructure teams to enhance the company's data platform while adhering to best practices for governance, monitoring, and reliability.
What will you do
Develop and maintain ETL pipelines for centralized data storage systems (e.g. Delta Lake).
Integrate data from databases, APIs, log files, streaming platforms, and external providers
Develop data transformation routines to clean, normalize, and aggregate data
Apply data processing techniques to handle complex or inconsistent datasets
Contribute to frameworks and best practices for code development and deployment
Implement data governance in alignment with company standards
Partner with analytics and product leaders to design and operationalize pipelines
Collaborate with infrastructure leaders to advance cloud-based data platforms
Explore new tools and techniques leveraging Azure, Databricks, or related platforms
Monitor data pipelines to detect and resolve issues promptly
Develop monitoring tools, alerts, and automated error-handling mechanisms
Analyse business requirements and identify data extraction requirements
Attend requirement grooming and refinement sessions with users
Develop and maintain ETL pipelines for ingestion, transformation, validation, and loading
Optimise performance and batch scheduling
Develop dashboards, reports, scorecards, and data visualizations
Perform SIT, data validation, data profiling and confirm data accuracy
Validate completeness and consistency of ETL Loads
Support UAT and production implementation
Qualifications:
The proposed data engineer (or equivalent) shall have minimally the following qualifications, experience, and knowledge:
Relevant educational credentials in a quantitative discipline (e.g. statistics, computer science, data analytics, operations research, or robotics) or equivalent practical experience
At least two (2) years of experience in building and optimising data pipelines, architectures, and data sets and
At least two (2) years of experience in supporting data transformation, data structures, metadata, dependency, and workload management.
To apply,simply click the
'Apply'
button or send your updated profile to (HIDDEN TEXT)
EA Licence No.:18S9405 / EA Reg. No.:R1330864
Percept Solutions is expanding and actively seeking talented individuals. We encourage applicants to follow Percept Solutions on LinkedIn at
https://www.linkedin.com/company/percept-solutions/
to stay informed about new opportunities and events.