Senior Data Engineer
KLK OLEO · Selangor, Malaysia ·
- Seniority
- Senior
- Category
- Data engineering
- Experience
- 5+ years
KLK OLEO · Selangor, Malaysia ·
KLA · Chennai, India
Ztek Consulting · Bangalore Bazaar, India
Snowflake · New York, New York
Snowflake · US-NY-New York
The Senior Data Engineer will manage and optimize database systems including MS SQL Server and PostgreSQL, while leading the migration of legacy SSIS pipelines to a modern data lakehouse environment. The role involves designing scalable data solutions, ensuring data governance, and collaborating with business stakeholders to translate requirements into technical data architectures.
Lead the end-to-end management of MS SQL Server, PostgreSQL and DuckDB (DuckLake) database systems, ensuring their stability, scalability and reliability.
Monitor daily job statuses and perform troubleshoot and recovery when necessary.
Apply best practices to design and implement high quality, low latency and scalable data solutions and scripts to extract data from source system (SAP ECC6), using appropriate connectors, aligning with business requirements.
Implement data ingestion strategies (batch and stream) by factoring in source-side constraints and data availability.
Write, optimize and troubleshoot complex T-SQL queries, stored procedures, views, functions and triggers to support data transformation, loading, and validation logic within MS SQL Server, in line with established coding and naming standards.
Orchestrate and schedule complex data workflows, ensuring timely and reliable data delivery.
Implement and manage monitoring solutions to proactively address and prevent performance issues.
Implement and enforce data governance and security policies, such as RBAC and data masking.
Develop and maintain staging structures and data models (star and snowflake schemas) within MS SQL Server, ensuring data is structured for efficient querying and reliable analytical consumption.
Contribute to data quality validation, reconciliation checks, and exception handling across data pipelines, ensuring completeness, accuracy, and consistency of data from source to consumption layer in alignment with the team's Single Source of Truth principles.
Document pipeline logic, data lineage, transformation rules, table definitions and data dictionary entries, ensuring technical artefacts are kept current and accessible to the team.
Deep understanding and assessment of SSIS data pipelines to refactor SQL scripts and migrate them to the new data lakehouse environment.
Lead discussions and debates on technical and architectural topics that are related to the data warehouse.
Participate in peer reviews and knowledge-sharing sessions, actively contributing to a culture of continuous improvement and cross-training across both legacy and modern data tooling within the team.
Identify opportunities to optimize data processes, reduce complexities and costs.
Conduct in-depth performance tuning activities, optimizing SQL queries, and database configurations for maximum efficiency.
Optimize storage usage, query performance and overall data platform efficiency.
Lead in schema design, table optimization (clustering, partitioning, indexing) and other optimization strategies to handle growth
Contribute to the establishment of a data governance framework. Drive the standards and strategy for maintaining a proper and traceable governance model for all end-to-end data assets.
Collaborate with stakeholders to understand business requirements and translate them into technical solutions.
Assess, evaluate and prioritize new and change requests to ensure they are evaluated for strategic fit and feasibility before assigning them to the development pipeline.
Assess and evaluate Business Requirement Study (BRS) to ensure that they are thoroughly scoped, covering objectives, business value, data sourcing implications and delivery timeline.
Build and maintain strong relationships with departmental stakeholders to ensure alignment of objectives and a shared understanding of requirements.
Provide clear and timely communication on project status, risks and progress, keeping business stakeholders and ITBS management well-informed throughout the delivery lifecycle.
Serve as the primary liaison for users when it comes to data related issues. Troubleshoot and debug issues arising from user endpoints.
Lead engagement with business and operational units - including roadshows, workshops and discovery sessions.
Champion the adoption of the data and self-service analytics tools.
Minimum of 5 years' experience in business solutioning, data / analytics consulting or a data warehouse delivery role, with a track record of partnering directly with business stakeholders.
Working knowledge of SAP ERP systems (SAP ECC6 or SAP S/4HANA), including familiarity with key functional modules (e.g. FI, CO, MM, SD), ABAP data extraction approaches (BAPI, IDOC, Open SQL, ABAP reports), and the ability to navigate SAP table structures for data sourcing purposes.
Experience in developing and maintaining a data lakehouse Table Formats (e.g. Delta Lake, Apache Iceberg, Apache Hudi, Ducklake), File Formats (e.g. Parquet, Avro), Catalogues (e.g. Unity Catalog, Hive, Ducklake) and Platforms (e.g. Snowflake, Databricks, BigQuery, Synapse).
Experience in advanced system design and data optimization using in-memory columnar formats such as Apache Arrows. Understanding of vectorized execution and query plans dependencies (e.g. scanning volume, predicate pushdowns, column pruning and the execution engine).
Understanding of data warehousing concepts, including dimensional modelling (star & snowflake schemas), slowly changing dimensions (SCD), and staging/ODS design patterns, with the ability to apply these principles to schema design and pipeline architecture decisions.
Proficiency in MS SQL Server, including database design, T-SQL development (stored procedures, views, functions, triggers), query optimisation and performance tuning.
Hands-on experience developing and maintaining SSIS packages for ETL processes, covering data extraction from heterogeneous sources, transformation logic, error handling, logging, and loading into SQL Server targets — with the ability to debug and tune existing packages independently.
Detail-oriented and methodical, with a structured approach to documentation, data quality validation, and issue escalation, combined with the ability to manage multiple concurrent deliverables and work effectively in a fast-paced, cross-functional team environment.
Language: SQL (mandatory), PowerShell, Python (bonus)
Orchestration: SSIS, ADF
Transformation: SQL, SSIS
Storage: SQL Server, ADLS2
Bachelor's degree in Information Technology, Computer Science, Data Analytics, Engineering or a related STEM field.
Maintains awareness of current developments in data and AI technologies, and able to evaluate their potential application within the organisation.
mufgub · Jersey City, NJ