Location: Bangalore (Yamlur)
Experience: 6-10 Years
Role: Principal Data Engineer
About Styli Marketplace
Launched in 2019 by Landmark Group, Styli Marketplace is the first e-commerce venture of the group, quickly becoming a leading online destination for fashion and lifestyle across the GCC, including Saudi Arabia, the UAE, Kuwait, Bahrain, and beyond. Styli connects global sellers and creators with millions of fashion-forward customers, offering the latest trends, exceptional value, and convenient services like same-day to 48-hour delivery and flexible payment options. Our mission is to make style accessible, aspirational, and exciting for all, backed by a passionate team fostering a culture of creativity and innovation.
Role Overview
We are looking for a Principal Data Engineer to take technical ownership of Styli's Data Platform and to lead its evolution from a BigQuery-centric warehouse to a modern Lakehouse on Databricks. This is a hands-on architecture and leadership role: you will set the technical direction for the platform, drive the migration end-to-end (design, staffing, partner management, cutover), and mentor a team of senior and staff data engineers. You will work closely with Platform/DevOps, Security, ML, and business stakeholders across the GCC and India to ensure the platform scales reliably, cost-efficiently, and securely as Styli grows.
What You'll Do
Data Platform Architecture & Migration Leadership
- Own the end-to-end target-state architecture for Styli's data platform, leading the migration from BigQuery to a Databricks Lakehouse built on Delta Lake / Apache Iceberg, Unity Catalog, and GCS.
- Lead build-vs-buy and platform evaluations (e.g., Databricks vs. Snowflake), and translate the recommendation into an executable, phased migration roadmap with clear cutover and rollback criteria.
- Define and defend architecture decision records (ADRs) covering table format strategy, catalog ownership, and cross-engine interoperability between Databricks and existing warehouses via the Iceberg REST catalog.
- Own the staffing and delivery model for the migration — partnering with Databricks implementation/SI partners, structuring hybrid onsite/offshore teams, and tracking cost, timeline, and risk.
- Drive Unity Catalog adoption for centralized governance, fine-grained access control, and data lineage across workspaces and markets.
Data Pipeline Development
- Design, build, and maintain batch and real-time pipelines migrating from Airflow/BigQuery patterns to Databricks Workflows and Delta Live Tables, alongside Apache Airflow and dbt where appropriate.
- Handle ingestion from diverse sources: MySQL/PostgreSQL, Kafka event streams, S3/GCS data lakes, REST APIs, and SaaS platforms.
- Ensure pipelines are fault-tolerant, recoverable, and instrumented with data quality checks at every stage, with clear migration parity/reconciliation testing against legacy BigQuery pipelines.
Data Modelling & Lakehouse Engineering
- Design and own the medallion (bronze/silver/gold) data architecture on Delta Lake, including dimensional and analytical models (star schema, OBT, wide tables) for BI and ML consumption.
- Own warehouse/lakehouse performance engineering — partitioning, Z-ordering/clustering, Photon acceleration, and cost-efficient query and compute (cluster policy) patterns on Databricks, alongside the legacy BigQuery estate during transition.
- Set and enforce standards for dbt models, tests, documentation, and lineage across a well-governed transformation layer.
Streaming & Real-Time Data
- Build and maintain real-time pipelines for live inventory updates, order event processing, and customer behavior streams, using Databricks Structured Streaming, Apache Flink, Spark Streaming, or ksqlDB.
- Design Kafka topic schemas, partition strategies, and consumer group management for high-throughput e-commerce event flows.
Data Lake & Cloud Infrastructure
- Build and manage a well-structured lakehouse on GCS, with Databricks running natively on GCP alongside existing cloud-native data services.
- Partner with the Platform/DevOps team on infrastructure-as-code (Terraform) for data infrastructure provisioning and containerized data workloads on Kubernetes.
Data Quality & Observability
- Set the data quality framework (Great Expectations, dbt tests, or Monte Carlo) to catch anomalies, nulls, duplicates, and schema drift before they reach downstream consumers — with particular rigor during migration cutover windows.
- Own pipeline SLAs, freshness SLOs, and schema-change alerting as first-class platform metrics; own incident response for data pipeline failures with clear runbooks and escalation paths.
ML & Analytics Enablement
- Build and maintain feature pipelines for ML models powering personalisation, recommendations, fraud detection, and demand forecasting, migrating toward Databricks Feature Store / MLflow where it improves training–serving consistency.
- Enable self-serve analytics by maintaining clean semantic layers and well-documented data marts for BI tools (Looker, Metabase, Superset).
Team Leadership & Mentorship
- Set technical standards and best practices for the data engineering team; review designs and code for scalability, cost, and reliability.
- Mentor senior and mid-level data engineers, including team members based in Bengaluru, and act as the technical escalation point for the platform.
- Partner directly with Platform Engineering, Security, and business stakeholders to align the migration roadmap with compliance and cost objectives.
What We're Looking For
Required
- 6-10 years of hands-on data engineering experience, including demonstrated ownership of platform-level architecture, ideally in a high-transaction e-commerce, fintech, or consumer tech environment.
- Proven, hands-on experience leading a large-scale migration onto Databricks — including Delta Lake, Unity Catalog, Databricks Workflows/Delta Live Tables, and cluster/cost optimization.
- Strong proficiency in SQL — complex analytical queries, window functions, CTEs, query optimisation, and warehouse-specific dialects (BigQuery / Databricks SQL / Redshift).
- Deep experience architecting lakehouse platforms on GCP, including evaluating and integrating open table formats (Apache Iceberg or Delta Lake) and cross-engine interoperability patterns.
- Hands-on experience with Apache Airflow for orchestration and dbt for transformation layer management, at scale — handling millions of events per day with reliability and observability.
- Working knowledge of Apache Kafka or equivalent event streaming platforms for real-time data ingestion.
- Proficiency in Python for pipeline development, data transformation, and automation, including PySpark for distributed processing on Databricks.
- Track record of technical mentorship and cross-functional stakeholder management, including partner/vendor (SI) management for large migration programs.
- Understanding of data modelling principles: normalisation, dimensional modelling, medallion architecture, and analytical patterns.
Good to Have
- Experience running Databricks and BigQuery (or Snowflake) in parallel during a phased migration, including reconciliation and cutover strategy design.
- Familiarity with data cataloguing and governance tooling (DataHub, Amundsen, Collibra, or Alation) beyond Unity Catalog.
- Knowledge of data mesh or data product principles and federated ownership models.
- Experience with real-time analytics databases: ClickHouse, Druid, or Pinot for high-throughput OLAP workloads.
- Familiarity with infrastructure-as-code (Terraform) and deploying data workloads on Kubernetes.
- Relevant certifications: Databricks Certified Data Engineer Professional, Databricks Lakehouse Architect, Google Professional Data Engineer, or dbt Analytics Engineering.
The Data Problems You'll Solve at Styli
- Leading the BigQuery-to-Databricks lakehouse migration — architecting and executing the move for a live, high-traffic e-commerce platform with zero-downtime tolerance.
- Personalisation at scale — building event pipelines that capture every click, view, and purchase to power real-time recommendations for millions of customers.
- Inventory & demand forecasting — reliable pipelines feeding ML models that optimise stock levels across hundreds of SKUs and multiple markets.
- Campaign & marketing analytics — ingesting data from paid channels, CRM, and app attribution to give growth teams a single source of truth.
- Order & payments intelligence — real-time event streams for order lifecycle tracking, fraud signals, and settlement reconciliation.
- Cross-market reporting — unified data models serving multiple country markets (GCC and India) with different currencies, catalogues, and fulfilment providers.