Data Engineer
HCL Technologies Limited · Vancouver, Canada ·
- Category
- Data engineering
- Company size
- 1000+
HCL Technologies Limited · Vancouver, Canada ·
Pyramid Consulting, Inc · Jalisco, Mexico
Soroc Technology · Toronto, Canada
Sequoia Connect · Ciudad de México, CDMX
D L RESOURCES PTE LTD · Singapore, Singapore
Data Engineer at HCL Technologies in Vancouver building and maintaining large-scale ETL/ELT pipelines on big data platforms like Databricks, Cloudera, and Hadoop. Day-to-day involves developing PySpark/Spark and Python jobs, optimizing pipeline performance and cost, and contributing to CI/CD for data workflows.
Work with cutting-edge big data platforms (e.g., Databricks, Apache Spark) at large scale, pushing the boundaries of data processing and model enablement.
Build and maintain robust ETL/ELT pipelines for ingestion, transformation, and aggregation of large-scale datasets on Hadoop and enterprise data platforms.
Develop high-performance data processing jobs using PySpark/Spark, Python on data platforms such as cloudera and databricks.
Optimize pipeline performance and cost through partitioning, file formats, compute tuning, and efficient query patterns
Contribute to CI/CD for data workflows (testing, code reviews, deployment automation), promoting engineering best practices and maintainable codebases.
Partner with Product Managers to develop a deep understanding of users and use cases and apply that knowledge to scoping and building new modules and features
Ideal Candidate Qualifications:
Strong hands‑on experience in data engineering building production‑grade pipelines on big data platforms (Hadoop ecosystem and cloud data platforms - databricks).
High proficiency in using Python, Spark, Hadoop platforms & tools (Hive, Impala, Airflow, NiFi), SQL to build Big Data products.
Hands‑on experience with cloud data platforms such as databricks, snowflake (databricks preferred)
Experience with orchestration/integration tools such as Apache Airflow, Apache NiFi, or Talend.
Working knowledge of DevOps/CI-CD practices: version control (Git), automated testing, release pipelines, and observability.
Strong problem‑solving skills with the ability to debug complex data issues and communicate clearly with technical and non‑technical stakeholders.
Experience developing Java based applications is an added advantage.
LONGDESCRIPTION section. 2 of 6.
Work with cutting-edge big data platforms (e.g., Databricks, Apache Spark) at large scale, pushing the boundaries of data processing and model enablement.
Build and maintain robust ETL/ELT pipelines for ingestion, transformation, and aggregation of large-scale datasets on Hadoop and enterprise data platforms.
Develop high-performance data processing jobs using PySpark/Spark, Python on data platforms such as cloudera and databricks.
Optimize pipeline performance and cost through partitioning, file formats, compute tuning, and efficient query patterns
Contribute to CI/CD for data workflows (testing, code reviews, deployment automation), promoting engineering best practices and maintainable codebases.
Partner with Product Managers to develop a deep understanding of users and use cases and apply that knowledge to scoping and building new modules and features
Ideal Candidate Qualifications:
Strong hands‑on experience in data engineering building production‑grade pipelines on big data platforms (Hadoop ecosystem and cloud data platforms - databricks).
High proficiency in using Python, Spark, Hadoop platforms & tools (Hive, Impala, Airflow, NiFi), SQL to build Big Data products.
Hands‑on experience with cloud data platforms such as databricks, snowflake (databricks preferred)
Experience with orchestration/integration tools such as Apache Airflow, Apache NiFi, or Talend.
Working knowledge of DevOps/CI-CD practices: version control (Git), automated testing, release pipelines, and observability.
Strong problem‑solving skills with the ability to debug complex data issues and communicate clearly with technical and non‑technical stakeholders.
Experience developing Java based applications is an added advantage.
LONGDESCRIPTION section. 3 of 6.
Work with cutting-edge big data platforms (e.g., Databricks, Apache Spark) at large scale, pushing the boundaries of data processing and model enablement.
Build and maintain robust ETL/ELT pipelines for ingestion, transformation, and aggregation of large-scale datasets on Hadoop and enterprise data platforms.
Develop high-performance data processing jobs using PySpark/Spark, Python on data platforms such as cloudera and databricks.
Optimize pipeline performance and cost through partitioning, file formats, compute tuning, and efficient query patterns
Contribute to CI/CD for data workflows (testing, code reviews, deployment automation), promoting engineering best practices and maintainable codebases.
Partner with Product Managers to develop a deep understanding of users and use cases and apply that knowledge to scoping and building new modules and features
Ideal Candidate Qualifications:
Strong hands‑on experience in data engineering building production‑grade pipelines on big data platforms (Hadoop ecosystem and cloud data platforms - databricks).
High proficiency in using Python, Spark, Hadoop platforms & tools (Hive, Impala, Airflow, NiFi), SQL to build Big Data products.
Hands‑on experience with cloud data platforms such as databricks, snowflake (databricks preferred)
Experience with orchestration/integration tools such as Apache Airflow, Apache NiFi, or Talend.
Working knowledge of DevOps/CI-CD practices: version control (Git), automated testing, release pipelines, and observability.
Strong problem‑solving skills with the ability to debug complex data issues and communicate clearly with technical and non‑technical stakeholders.
Experience developing Java based applications is an added advantage.
SKILL section. 4 of 6.
SKILL section. 5 of 6.
LONGDESCRIPTION section. 6 of 6.
Work with cutting-edge big data platforms (e.g., Databricks, Apache Spark) at large scale, pushing the boundaries of data processing and model enablement.
Build and maintain robust ETL/ELT pipelines for ingestion, transformation, and aggregation of large-scale datasets on Hadoop and enterprise data platforms.
Develop high-performance data processing jobs using PySpark/Spark, Python on data platforms such as cloudera and databricks.
Optimize pipeline performance and cost through partitioning, file formats, compute tuning, and efficient query patterns
Contribute to CI/CD for data workflows (testing, code reviews, deployment automation), promoting engineering best practices and maintainable codebases.
Partner with Product Managers to develop a deep understanding of users and use cases and apply that knowledge to scoping and building new modules and features
Ideal Candidate Qualifications:
Strong hands-on experience in data engineering building production-grade pipelines on big data platforms (Hadoop ecosystem and cloud data platforms - databricks).
High proficiency in using Python, Spark, Hadoop platforms & tools (Hive, Impala, Airflow, NiFi), SQL to build Big Data products.
Hands‑on experience with cloud data platforms such as databricks, snowflake (databricks preferred)
Experience with orchestration/integration tools such as Apache Airflow, Apache NiFi, or Talend.
Working knowledge of DevOps/CI-CD practices: version control (Git), automated testing, release pipelines, and observability.
Strong problem‑solving skills with the ability to debug complex data issues and communicate clearly with technical and non‑technical stakeholders.
Experience developing Java based applications is an added advantage
Turtle Trax S.A. · Jalisco, Mexico