This is a Cactus International job and is located in Abu Dhabi, UAE.
Hybrid: 4 days in office; 1 day working from home
Job Summary
The Data Platform Lead serves as the hands-on technical owner of the company's Azure Databricks platform. This role combines platform engineering, data engineering leadership, governance implementation, production operations, and people leadership.
The successful candidate will build, operate, secure, and continuously improve the enterprise. The role owns Databricks platform standards, pipeline architecture, data observability, access governance, performance, cost management, and operational readiness.
This is a highly hands-on leadership role. Approximately 60-70% of the position will be spent designing, building, troubleshooting, configuring, and optimizing the platform and its pipelines. The remaining time will focus on mentoring engineers, reviewing designs and code, establishing practical standards, supporting delivery, and helping shape future data, analytics, AI, data product, and application product capabilities.
What Success Looks Like
- The Databricks platform is secure, governed, observable, cost-conscious, and ready to support business-critical workloads.
- Production pipelines are reliable, monitored, documented, recoverable, and owned.
- Data Engineers receive strong technical direction, design support, coaching, and clear engineering standards.
- Certified datasets and reusable data products accelerate reporting, automation, AI, and digital product delivery.
- Leadership has a trusted technical partner who can translate priorities into practical platform and engineering decisions.
Essential Functions, Roles and Responsibilities
Databricks Platform Ownership and Governance
- Own the technical configuration and ongoing evolution of the Azure Databricks platform, including workspace and environment structure, catalogs, schemas, storage, compute, and workload separation.
- Implement and maintain practical Unity Catalog standards for access control, lineage, data classification, ownership, catalog and schema organization, and least-privilege access.
- Define and maintain Bronze, Silver, and Gold layer standards, including naming conventions, table ownership, audit fields, refresh patterns, lifecycle expectations, and production-readiness criteria.
- Partner with cybersecurity, infrastructure, and DevOps teams on identity, networking, secrets management, private connectivity, monitoring, compliance, and deployment controls.
- Establish and maintain cluster policies, compute standards, job standards, cost controls, capacity practices, and platform usage monitoring.
- Maintain clear technical standards and guardrails without creating unnecessary process or slowing delivery.
Data Engineering and Pipeline Delivery
- Design, build, troubleshoot, and optimize production-grade pipelines using Azure Databricks, Apache Spark, Python/PySpark, SQL, Delta Lake, and approved orchestration patterns.
- Lead ingestion from ERP systems, PostgreSQL and other application databases, SaaS platforms, APIs, files, and enterprise data sources into the Lakehouse.
- Define reusable engineering patterns for full loads, incremental loads, change data capture where appropriate, schema evolution, reprocessing, error handling, logging, reconciliation, and recovery.
- Ensure production pipelines include source-to-target mapping, ownership, data quality rules, monitoring, alerting, operational documentation, and support handover.
- Perform technical design reviews, code reviews, performance tuning, and hands-on problem solving for complex or high-risk pipelines.
- Review vendor and contractor deliverables for maintainability, security, performance, documentation, supportability, and production readiness.
Platform Operations, Reliability and Observability
- Own the operational health and reliability of the Databricks platform and its production workloads.
- Implement monitoring and alerting for pipeline failures, job duration, data freshness, data quality, service-level expectations, compute health, platform usage, and cost anomalies.
- Lead root cause analysis for production incidents and ensure corrective and preventive actions are completed.
- Establish runbooks, escalation paths, recovery procedures, support expectations, and operational dashboards.
- Continuously improve platform performance, resilience, scalability, supportability, and cost efficiency.
Data Quality and Production Readiness
- Implement practical controls for completeness, uniqueness, validity, freshness, referential integrity, and reconciliation to source systems.
- Ensure datasets have clear owners, classifications, lineage, refresh schedules, quality expectations, and required documentation before production release.
- Work with business, ERP, product, and application teams to understand source-system meaning, business rules, schema changes, and downstream impact.
- Enable trusted Silver and Gold datasets that support analytics, executive reporting, operational dashboards, automation, AI/ML, and digital applications.
- Define and enforce clear acceptance criteria for platform changes, pipelines, and data products.
Team Leadership and Engineering Excellence
- Lead, mentor, and support two Data Engineers, providing clear priorities, technical guidance, coaching, and timely feedback.
- Develop the team's capability in Databricks, data modeling, pipeline engineering, testing, observability, troubleshooting, documentation, and production support.
- Promote an engineering culture built on ownership, collaboration, quality, reuse, automation, and continuous improvement.
- Support hiring, onboarding, role development, and future growth of the data platform and engineering team.
- Coordinate technical work across internal engineers, contractors, consultants, and vendors while maintaining clear accountability.
Data Products, AI and Digital Product Enablement
- Partner with Digital Solutions, analytics, ERP, and business teams to translate future product needs into scalable platform capabilities and reusable data assets.
- Help design and deliver governed data products with clear ownership, interfaces, quality expectations, documentation, and supported consumption patterns.
- Enable application products and customer-facing digital solutions that require reliable operational and analytical data.
- Prepare trusted and appropriately governed data for AI, machine learning, GenAI, automation, and advanced analytics use cases.
- Evaluate relevant Databricks capabilities and recommend practical adoption based on business value, security, operational maturity, and total cost.
Technical Partnership and Delivery
- Serve as the primary technical owner for Azure Databricks and data engineering decisions.
- Act as a trusted technical partner to IT and Digital leadership on platform strategy, delivery priorities, architecture decisions, production risk, investment choices, and vendor acceptance.
- Collaborate with analytics, business applications, ERP, product engineering, cybersecurity, infrastructure, DevOps, and business stakeholders.
- Communicate technical risks, options, trade-offs, dependencies, and recommendations clearly to both technical and non-technical audiences.
- Help evolve the data platform roadmap, engineering standards, and operating model as the organization grows.
Education, Training and Experience
Required Experience
- 7+ years of relevant experience in data engineering, data platforms, cloud data architecture, or enterprise data integration.
- Demonstrated hands-on experience designing, building, troubleshooting, and supporting production data pipelines in cloud environments.
- Strong production experience with Azure Databricks, Delta Lake, and modern Lakehouse architecture.
- Experience leading or mentoring Data Engineers and providing technical direction across multiple initiatives.
- Experience integrating enterprise sources such as ERP, CRM, SaaS applications, PostgreSQL, SQL Server, Oracle, APIs, and files.
- Experience reviewing vendor or contractor deliverables and enforcing technical and operational standards.
Required Technical Skills
- Strong expertise in Azure Databricks, Apache Spark, Python/PySpark, advanced SQL, Delta Lake, and Lakehouse design patterns.
- Hands-on knowledge of Unity Catalog, role-based access control, lineage, data classification, metadata, and access governance.
- Experience with Azure Data Lake Storage, GitHub or Azure DevOps, CI/CD, secrets management, and cloud integration patterns.
- Strong understanding of data modeling, dimensional and normalized models, medallion architecture, data quality, reconciliation, and schema evolution.
- Ability to design ingestion and processing patterns for ERP data, application databases, APIs, files, streaming or incremental changes, and CDC where appropriate.
- Experience implementing operational monitoring, pipeline observability, alerting, incident response, and performance optimization.
Preferred Qualifications
- Experience with Databricks Workflows, Lakeflow Declarative Pipelines, SQL Warehouses, Serverless compute, MLflow, Databricks Asset Bundles, or infrastructure-as-code.
- Experience designing reusable data products, semantic models, governed self-service datasets, or analytics-ready data layers.
- Exposure to Power BI, Tableau, enterprise reporting migration, or analytics product delivery.
- Exposure to ML/AI pipelines, feature engineering, GenAI, Vector Search, retrieval-augmented generation, automation, or AI-ready data product development.
- Experience in manufacturing, oil and gas, field services, industrial operations, or ERP-intensive environments.
- Databricks, Microsoft Azure Data Engineer, Azure Solutions Architect, or related cloud/data certifications are preferred but not required.
Knowledge, Skills and Abilities
- Ability to operate effectively as both a hands-on senior engineer and a people leader.
- Strong judgment and confidence to challenge designs that are not secure, scalable, observable, documented, supportable, or production ready.
- Ability to balance platform governance with delivery speed and practical business outcomes.
- Strong communication and stakeholder-management skills across engineering teams, vendors, business partners, and leadership.
- Ability to translate business and product needs into scalable data platform and engineering solutions.
- Strong ownership mindset, curiosity, documentation discipline, and comfort working in a growing environment with evolving standards.
Role Profile
This role is best suited to a senior Databricks practitioner who still enjoys building and troubleshooting, can lead a small engineering team, and wants meaningful influence over the future of an enterprise data, AI, and digital product platform.