Senior Data Engineer
Kuala Lumpur Kepong Berhad Group · Petaling Jaya, Malaysia ·
This posting has been closed.
- Seniority
- Senior
- Category
- Data engineering
- Experience
- 5+ years
aianalyticsavrobigquerydata-analyticsdata-governancedata-ingestiondata-lineagedata-pipelinesdata-qualitydata-warehousingdatabricksdelta-lakedesign-patternsdimensional-modelingduckdbiceberglakehouseparquetpostgresqlpowerbipowershellsapsap-s4hanasnowflakesqlsql-serverssasssis
Senior Data Engineer owning end-to-end management of MS SQL Server, PostgreSQL and DuckDB data warehouse/lakehouse systems, extracting data from SAP, building and tuning T-SQL/SSIS pipelines, and migrating SSIS workloads to a modern data lakehouse while partnering with business stakeholders.
Role and Responsibilities
A.DATA MANAGEMENT & TECHNICAL DEVELOPMENT
- Lead the end-to-end management of MS SQL Server, PostgreSQL and DuckDB (DuckLake) database systems, ensuring their stability, scalability and reliability.
- Apply best practices to design and implement high quality, low latency and scalable data solutions and scripts to extract data from source system (SAP ECC6).
- Implement data ingestion strategies (batch and stream) by factoring in source-side constraints and data availability.
- Write, optimize and troubleshoot complex T-SQL queries, stored procedures, views, functions and triggers to support data transformation, loading, and validation logic within MS SQL Server, in line with established coding and naming standards.
- Orchestrate and schedule complex data workflows, ensuring timely and reliable data delivery.
- Implement and manage monitoring solutions to proactively address and prevent performance issues. Monitor daily job statuses and perform troubleshoot and recovery when necessary.
- Develop and maintain staging structures and data models (star and snowflake schemas) within MS SQL Server, ensuring data is structured for efficient querying and reliable analytical consumption.
- Contribute to data quality validation, reconciliation checks, and exception handling across data pipelines, ensuring completeness, accuracy, and consistency of data from source to destination.
- Document pipeline logic, data lineage, transformation rules, table definitions and data dictionary entries, ensuring technical artefacts are kept current and accessible to the team.
B.OPTIMIZATION & PLATFORM TRANSFORMATION
- Deep understanding and assessment of SSIS data pipelines to refactor SQL scripts and migrate them to the new data lakehouse environment.
- Lead in schema design, table optimization (clustering, partitioning, indexing) and other optimization strategies to handle growth.
- Conduct in-depth performance tuning activities, optimizing SQL queries, and database configurations for overall efficiency.
- Identify opportunities to optimize data processes, reduce complexities and costs.
- Participate in technical discussions, debates, peer reviews and knowledge‑sharing sessions on topics related to the data warehouse.
- Contribute to the establishment of a data governance framework. Drive the standards and strategy for maintaining a proper and traceable governance model for all end‑to‑end data assets.
C.BUSINESS PARTNERSHIP & PROJECT MANAGEMENT
- Collaborate with stakeholders to understand business requirements and translate them into technical solutions.
- Assess and evaluate Business Requirement Study (BRS) to ensure that they are thoroughly scoped, covering objectives, business value, data sourcing implications and delivery timeline.
- Serve as the primary liaison for users when it comes to data related issues. Troubleshoot and debug issues arising from user endpoints.
Job Success Requirements
- Minimum of 5 years' experience in business solutioning, data / analytics consulting or a data warehouse delivery role, with a track record of partnering directly with business stakeholders.
- Understanding of data warehousing concepts, including dimensional modelling (star & snowflake schemas), slowly changing dimensions (SCD), and staging/ODS design patterns, with the ability to apply these principles to schema design and pipeline architecture decisions.
- Proficiency in MS SQL Server, including database design, T‑SQL development (stored procedures, views, functions, triggers), query optimisation, performance tuning, developing and maintaining SSIS packages for data pipelines.
- Good to have: SAP ECC6 or SAP S/4HANA, table formats (e.g. Delta Lake, Apache Iceberg, Apache Hudi, Ducklake), file formats (e.g. Parquet, Avro), data platforms (e.g. Snowflake, Databricks, BigQuery, Synapse), columnar serialization interchange formats (e.g. Apache Arrows)
Technology Stack We Use
- Language: SQL (mandatory), PowerShell
- Storage: SQL Server, ADLS2
- Orchestration: SSIS, ADF
- Modelling: SSAS
- Transformation: SQL, SSIS
- Visualization: Excel, Power BI
Qualification
- Bachelor's degree in Information Technology, Computer Science, Data Analytics, Engineering or a related STEM field.
Additional Notes
Maintains awareness of current developments in data and AI technologies, and able to evaluate their potential application within the organisation