Data Engineering Consultant

Role Purpose

The Data Engineering Consultant designs, builds and supports reliable data pipelines, integrations and services for client data and analytics solutions. The role converts approved architecture, business requirements and governance controls into tested technical components that move, transform and deliver data with appropriate quality, metadata, lineage, security and operational monitoring. The consultant works with architects, analysts, platform teams, security specialists and client system owners throughout design, development, testing and transition. The role may implement within client environments under authorised access, but production ownership and operational authority remain with the client. The expected outcome is maintainable engineering work with clear source-to-target logic, automated controls, deployment evidence, runbooks and knowledge transfer suitable for client operation after handover.

Key Responsibilities

Analyse approved requirements, source systems, target models, data volumes, schedules, controls and service expectations. Design and build batch, streaming or API-based data ingestion and transformation pipelines. Implement source-to-target mappings, business transformations, reference-data logic and data-standardisation rules. Embed validation, reconciliation, exception handling and data-quality controls within engineering workflows. Capture technical metadata, lineage, schedules, dependencies, ownership and operational information for delivered pipelines. Apply approved security, privacy, classification, access, encryption and logging requirements. Develop automated unit, integration, regression and data-validation tests and retain execution evidence. Optimise performance, scalability, reliability and cost within the approved architecture and platform constraints. Implement deployment automation, version control, configuration management and controlled release practices. Monitor pipeline operation, investigate failures and support defect resolution during implementation and transition. Prepare technical specifications, code documentation, support procedures, runbooks and handover records. Walk client engineers through design decisions, operating procedures and known limitations before transition.

Minimum Requirements

Education

Bachelor's degree in computer science, software engineering, data engineering or a related field. An equivalent combination of relevant education and directly applicable consulting or implementation experience may be considered.

Professional Experience

Typically 3-8 years in data engineering, integration, ETL/ELT, software or cloud data platforms. Experience in data engineering, integration, ETL/ELT, software development or cloud data delivery. Experience building and testing production-grade pipelines, APIs or data-processing services. Experience with version control, automated deployment, monitoring and operational support. Experience working from architecture, security and data-quality requirements in multidisciplinary teams.

Technical and Domain Knowledge

Data ingestion, transformation and orchestration patterns. SQL and one or more relevant programming or scripting languages. Batch, streaming, API and integration technologies. Cloud or enterprise data platforms and storage patterns. Data quality, reconciliation, exception handling and test automation. Metadata, lineage, source-to-target mapping and documentation. Security, access control, encryption, logging and secrets management. Version control, CI/CD, configuration and operational monitoring.

Core Competencies

SQL and programming. Data pipelines. APIs and integration. Cloud/data platforms. Testing and observability. Security and quality controls. Engineering discipline. Problem solving. Reliability focus. Automation mindset. Clarifying technical requirements and identifying missing decisions early. Estimating engineering effort, dependencies and delivery risk. Explaining technical designs and defects to mixed audiences. Producing maintainable documentation and auditable test evidence. Collaborating with client engineers, vendors and multidisciplinary teams. Working within controlled client access and change processes. Transferring engineering knowledge and supporting operational handover.

Preferred Certifications

Google Cloud Professional Data Engineer or comparable cloud data credential CDMP Associate or Practitioner

Focus areas

SQL and programming Data pipelines APIs and integration