Big Data Engineer

About the job Join our Team - Your career journey starts here - not just a job, but a future. Our Potential Opportunity INTECH Automation Intelligence is seeking a Big Data Engineer to join our dynamic team. This role is pivotal in supporting Industrial Digitalization Solutions across industries such as Oil & Gas, Petrochemicals, Utilities, and Manufacturing. Design and operate end-to-end big data and analytics solutions building scalable ETL/ELT pipelines, integrating industrial and enterprise data, optimizing Hadoop/Cloudera platforms, and delivering real-time, predictive dashboards that drive operational efficiency, safety, compliance, and data-driven decision-making. INTECH's commitment to providing cutting-edge Digital Solutions ensures you will contribute to transformative projects that enhance operational efficiency and productivity. Our Digital Solutions are a testament to our dedication to driving innovation and optimizing industrial processes, reflecting our focus on excellence, reliability, and overcoming industry challenges.

What You’ll Lead and Deliver

• Design, develop, and maintain robust ETL/ELT pipelines for both pre-production and production environments.Ingest data from diverse sources, including industrial sensors (pumps, boilers, storage tanks), relational databases (SQL, Oracle), and enterprise applications.
• Implement data integration and transformation logic using Scala, PySpark, Sqoop, and Apache NiFi within the Cloudera/Hadoop ecosystem.
• Manage and schedule data workflows using Oozie or similar orchestration tools.
• Optimize data storage and query performance using Hive, Impala, and HDFS for efficient analytics.
• Develop interactive and real-time dashboards in Power BI and Tableau to monitor critical business and operational KPIs.
• Translate complex technical datasets into user-friendly visualizations for stakeholders, enabling data-driven decision-making.
• Design dashboards for specific use cases such as:
• i) Real-time monitoring of sensor data (pumps, boilers) to improve plant uptime and save costs.
• ii) Track safety permits (ESWP), monitor workplace hazards (SHER, risk scores), and perform root-cause analysis on incidents.
• Support the development of predictive models, such as Hazard Prediction dashboards, by preparing and engineering the necessary data features.
• Perform data analysis and ad-hoc reporting to answer complex business questions related to plant manufacturing, energy, and safety.
• Ensure data accuracy, consistency, and reliability across all reporting and analytics platforms.
• Assist in the maintenance, configuration, and performance tuning of the Big Data platform (Cloudera) and BI servers (Tableau/Power BI).
• Collaborate on the integration of industrial systems (e.g., OSI PI System) with the big data lake for advanced analytics.