Senior Data Engineer - Databricks
Texplorers Inc · Hyderabad, India ·
- Seniority
- Senior
- Category
- Data engineering
Texplorers Inc · Hyderabad, India ·
HTEC Group · Bosnia and Herzegovina, North Macedonia, Croatia, Romania, Hungary, Spain, Latin America, Czech Republic, RS
State Street · Bangalore, Karnātaka
High Tech Engineering Center Kft. · Budapest, Szervita tér .
Citi · Pune Maharashtra India
Senior Data Engineer building and optimizing big-data ETL pipelines in Databricks and Apache Spark on cloud platforms (especially Azure), including performance tuning, CI/CD for data workflows, and Delta Lake/Lakehouse work. Collaboration with data scientists and analysts to support analytics at scale.
We are looking for 20 skilled Data Engineering (DE) professionals with expertise in Databricks, Apache Spark, and Azure cloud to design, develop, and optimize big data solutions. The ideal candidates will have strong experience in data engineering, ETL pipeline development, and performance optimization.
Key Responsibilities:
Design, develop, and optimize ETL pipelines in Databricks.
Build data ingestion, transformation, and processing workflows using Apache Spark.
Work with structured and unstructured data from various sources.
Optimize Spark jobs for performance, cost efficiency, and scalability.
Implement CI/CD for Databricks workflows and manage version control.
Work closely with Data Engineers, DataScientists, and Business Analysts to support analytics and reporting.
Implement security best practices andmanage access controls in Databricks.
Troubleshoot performance and scalability issues in big data processing.
Required Skills:
Strong hands-on experience with Databricks and Apache Spark (PySpark/Scala/Java).
Proficiency in SQL for data analysis and transformation.
Experience in cloud platforms (Azure, AWS,or GCP), especially Azure Data Lake, AWS S3, or GCP BigQuery.
Hands-on experience with ETL tools, DataPipelines, and Data Warehousing.
Strong knowledge of CI/CD, Git, and DevOpstools for data workflows.
Knowledge of Delta Lake, Lakehouse architecture, and performance tuning.
Experience in handling large-scale datasets and distributed computing.
Preferred Skills:
Experience with Databricks MLflow and machine learning pipelines.
Knowledge of Kafka, Airflow, or Snowflake.
Understanding of data governance,compliance, and security policies.
Education & Experience:
Bachelor’s or Master’s degree in ComputerScience, Data Engineering, or a related field.
Experience in Databricks development
NCS PTE. LTD. · Singapore