This role is for professionals seeking a long-term career with us; subcontractor applications will not be considered.
We are seeking an experiencedSenior Data Engineerto design, build, and optimize modern
data platforms using theDatabricks Lakehouse Platform. The ideal candidate will have10+
years of data engineering experience
years of data engineering experience
and deep expertise inLakehouse
Architecture,Medallion Architecture, cloud-native data solutions, and enterprise-scale data
governance practices.
Key Responsibilities
Data Platform Architecture & Engineering
- Design and implement scalable, high-performance data platforms using the Databricks Lakehouse architecture.
- Lead the development of enterprise data solutions leveraging Bronze, Silver, and Gold layers following Medallion Architecture principles.
- Build and maintain batch and streaming data pipelines for large-scale data processing.
- Develop reusable data frameworks, accelerators, and engineering best practices.
- Optimize data storage, processing, and query performance across cloud environments.
Databricks Development
- Design, develop, and maintain Databricks workflows using PySpark, SQL, and Python.
- Implement and manage Delta Live Tables (DLT) pipelines.
- Utilize Unity Catalog for centralized governance, metadata management, access controls, and lineage tracking.
- Support advanced analytics, machine learning, and AI workloads on the Databricks platform.
- Monitor and troubleshoot production data pipelines and platform performance.
Data Governance & Quality
- Establish and enforce enterprise data governance standards and policies.
- Implement data quality frameworks, validation rules, reconciliation processes, and monitoring solutions.
- Ensure compliance with security, privacy, regulatory, and audit requirements.
- Drive metadata management, lineage, cataloging, and data stewardship initiatives.
- Collaborate with business and governance teams to improve trust and usability of enterprise data assets.
- Design and deploy cloud-native data solutions on Azure, AWS, or GCP.
- Implement scalable and secure cloud architectures aligned with enterprise standards.
- Optimize cloud resource utilization, performance, and cost management.
- Integrate cloud-native services with Databricks-based solutions.
DevOps & Automation
- Automate deployment, testing, monitoring, and operational processes.
- Apply Infrastructure as Code (IaC) practices using tools such as Terraform or similar.
- Promote engineering excellence through code reviews, testing strategies, and release management processes.
- Provide technical leadership and mentorship to junior and mid-level engineers.
- Collaborate with architects, analysts, data scientists, product owners, and business stakeholders.
- Contribute to platform roadmaps, architectural decisions, and technology evaluations.
- Lead technical discussions and establish best practices across the data engineering organization.
Required Qualifications
- Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or a related field.
- 10+ years of experiencein Data Engineering, Data Warehousing, or Big Data platforms.
- 5+ years of hands-on experience with Databricksin enterprise environments.
- Strong expertise in:
- Lakehouse Architecture
- Medallion Architecture
- Unity Catalog
- Data Governance and Data Quality frameworks
- Data Modeling and Data Warehousing concepts
- Extensive experience with at least one major cloud platform:
- Amazon Web Services (AWS)
- Advanced programming experience with:
- Python
- SQL
- PySpark
- Strong understanding of CI/CD methodologies and DevOps practices.
- Experience implementing Infrastructure as Code (Terraform).
- Expertise in performance tuning, optimization, and troubleshooting of large-scale data pipelines.
- Strong understanding of security, access controls, and compliance requirements in cloud environments.
Preferred Qualifications
- Databricks certifications such as 'Databricks Certified Data Engineer Professional'
- Cloud certifications (Azure, AWS, or GCP).
- Experience with real-time streaming technologies and event-driven architectures.
- Experience supporting AI/ML and Generative AI data platforms.
- Familiarity with data observability tools and modern data quality frameworks.
- Experience implementing enterprise metadata management and data lineage solutions.
Hiring Process
We aim to complete the interview process in October-November. The targeted onboarding is January 2027 or earlier.