Big Data Engineer
Helius Technologies Pte Ltd · Singapore ·
- Category
- Data engineering
- Experience
- 5+ years
Build and optimize big-data pipelines and analytics using Hadoop, PySpark, Scala, and Python to process large-scale datasets and integrate data from multiple sources.
Big Data Engineer
Job Requirement: 5 years of hands on work experience in Hadoop, Scala, Pyspark and Python. Work with Hadoop Distributed File System (HDFS) to manage and store large datasets Utilize tools like Hive, HBase, and Pig to process data in Hadoop Set up and manage data pipelines to extract, transform, and load (ETL) large datasets Develop and optimize PySpark jobs and Scala-based Spark applications to process and analyze large-scale data Integrate data from various sources (relational databases, flat files, APIs, etc.) using Python and Spark. Build data models and perform advanced analytics using PySpark, Python libraries and Scala.
Job Requirement: 5 years of hands on work experience in Hadoop, Scala, Pyspark and Python. Work with Hadoop Distributed File System (HDFS) to manage and store large datasets Utilize tools like Hive, HBase, and Pig to process data in Hadoop Set up and manage data pipelines to extract, transform, and load (ETL) large datasets Develop and optimize PySpark jobs and Scala-based Spark applications to process and analyze large-scale data Integrate data from various sources (relational databases, flat files, APIs, etc.) using Python and Spark. Build data models and perform advanced analytics using PySpark, Python libraries and Scala.