Data Engineer Consultant
Total-TECH Co. · Jeddah, Saudi Arabia ·
- Category
- Data engineering
- Experience
- 5+ years
Total-TECH Co. · Jeddah, Saudi Arabia ·
Build and maintain scalable GCP-based data pipelines (batch/streaming) using BigQuery, Dataflow, and Composer, plus optional Informatica IDMC, while ensuring reliability, performance, and cost efficiency.
Enterprise Data Ingestion and Data Engineering (1-): Design and build scalable, reusable ingestion pipelines (realtime and batch) on GCP (GCS, BigQuery, Dataflow). Develop parameterized pipelines using Google-native services and/or Informatica IDMC mappings and taskflows. Implement CDC patterns, idempotent loads, late‑arriving data handling, and schema evolution. Optimize BigQuery ingestion strategies (batch vs. streaming, partitioning, clustering). Establish version control, CI/CD, and environment promotion (dev/test/prod).
Profile source data and analyze data quality and patterns. Design field‑level mappings, business rules, joins, aggregations, and derivations. Implement transformations using BigQuery SQL, Dataflow, or IDMC transformation logic.
Build and parameterize end‑to‑end workflows using Cloud Composer (Airflow) and/or IDMC taskflows. Define job dependencies, schedules, SLAs, and failure handling strategies. Implement retries, backoff strategies, checkpoints, and restartability. Integrate monitoring, logging, and alerting using Cloud Monitoring, Cloud Logging, and ChatOps tools.
Monitor pipeline health, data freshness, volumes, and anomaly patterns. Support and monitor data pipelines during off‑hours and weekends, troubleshooting and resolving issues to ensure SLA compliance. Track and manage SLAs related to runtime, failures, cost, and data latency. Optimize BigQuery performance (query refactoring, partition pruning, materialized views). Manage and optimize GCP costs (storage lifecycle, slot usage, query optimization, caching).