Data Platform Engineer

Lantern’s data organization is moving from a centralized model — where the platform team owns most pipeline delivery — toward one where domain teams (Data Engineering, Data Ops, Analytics Engineering) build and own their own data products, and the platform team’s job is to make that possible. This role is core to that shift: building and administering the self-serve infrastructure, governance automation, and access controls that domain teams build on top of, rather than building domain pipelines directly.

This is a hands-on administration and enablement role. Domain teams own their data products; the data architecture committee owns the standards. This role administers the Databricks platform and the Lakebase (managed Postgres) estate, holds the elevated privileges that go with that, and enforces our standards through platform controls, automation, and review — across Azure infrastructure and IAM, CI/CD, observability, and the governance layer of access control, audit, and lineage. It is also the platform’s front door for domain teams, from project kickoffs through day-to-day unblocking, and works in close partnership with the Cloud Platform, Azure SQL DBA, and Security teams on the shared foundations underneath.

Location: Hybrid - at least 3 days/week in our Dallas, TX offices

Responsibilities:

• Platform Administration: Administer the Azure Databricks and Unity Catalog platform against the architecture standards set by the data architecture committee — workspace and metastore configuration, catalog and schema structure, cluster policies, and compute cost management, including spend-by-team visibility and chargeback reporting. Feed back to the standards owners where a standard is blocking teams or no longer fits how the platform is being used.
• Self-Serve Enablement: Build the self-serve infrastructure layer — provisioning workflows, templates, and access patterns that let domain teams stand up and manage their own data products without depending on the platform team for every change.
• Postgres Database Administration (Lakebase): Act as the DBA for the platform’s Postgres estate — provisioning and decommissioning of Lakebase instances, user and role management, connection configuration, query and index performance tuning, version upgrades, backup and point-in-time recovery, and database incident response. Maintain the sync patterns between Lakebase and the lakehouse.
• Infrastructure-as-Code: Build and maintain Infrastructure-as-Code for the platform using Terraform against Azure — resource groups, networking and private endpoints, Entra ID app registrations, service principals, and Key Vault — so platform changes are reviewable, repeatable, and auditable.
• Access & Governance Model: Implement and enforce the platform’s approved access model across the lakehouse and the Postgres layer — Unity Catalog grants, OBO (on-behalf-of) authentication patterns, ABAC, row-level security, data masking, and role-based access across engineering, ops, and analytics personas.
• Audit & Lineage: Build the audit and lineage layer that makes decentralized ownership auditable — audit logging, lineage capture, sensitivity classification tagging, and the custom event-sourced audit layer needed where Databricks cannot natively capture app-layer events, including incident response when this layer breaks.
• Observability: Operate platform observability across Databricks, Lakebase, and supporting Azure services (Datadog integration, metrics and log export, alerting, SLOs) so domain teams get reliability guarantees without building their own monitoring.
• Reliability & Disaster Recovery: Execute the infrastructure lifecycle — provisioning, decommissioning, backup and restore validation, and disaster recovery testing for platform and database workloads.
• Support Coverage: There is no overnight on-call rotation for this role. Occasional availability outside core hours is expected to support scheduled deployments and major incidents.
• CI/CD: Support CI/CD for data workloads across domain teams — deployment pipelines for notebooks, jobs, DLT pipelines, dbt projects, Databricks Asset Bundles (DABs), and Lakebase schema migrations.
• GitHub Integration: Maintain the integration between the platform and GitHub / GitHub Actions — repo structure, branch policies, and CI/CD workflows that domain teams build their pipelines on top of.
• Cloud Platform Collaboration: Work with the Cloud Platform team on the shared Azure foundation — networking, Entra ID, Key Vault, policy, and cloud cost — with a clear boundary for where platform responsibility ends, and triage and hand off Azure infrastructure issues accordingly.
• Azure SQL DBA Collaboration: Coordinate with the Azure SQL DBA team, who own the Azure SQL estate independently — integration points between the two estates, shared practice on backup, HA/DR, and PHI handling, and input on workload placement.
• Cross-Team Liaison: Maintain a recurring liaison relationship with Security, InfoSec, and Core Apps on cross-team platform dependencies as they surface, and ensure platform capabilities meet HIPAA, SOC 2, and other regulatory requirements.
• Project Engagement: Engage early in new domain-team project kickoffs to advise on platform capabilities, access patterns, and the standards a design must meet, before teams build around constraints they did not know existed.
• Unblocking: Serve as the escalation point for domain engineers stuck on cluster policies, permissions boundaries, database performance, or broken CI steps.
• Reporting & Roadmap: Produce platform KPI reporting (usage, uptime, deployment velocity, spend) and contribute to the platform-as-product roadmap and backlog.
• Documentation: Capture reusable patterns from kickoffs, incidents, and liaison work in a shared knowledge base so domain teams do not have to rediscover solutions.

Requirements:

• A minimum of 4 years in a platform or infrastructure engineering role supporting a data or analytics organization.
• Deep hands-on experience with Databricks (workspace administration, Unity Catalog, Delta Lake) and Azure (Entra ID, networking, Key Vault, Infrastructure-as-Code via Terraform or Bicep).
• Hands-on PostgreSQL administration experience, as this role is the DBA of record for the platform’s Postgres estate — user and role management, connection configuration, query and index performance tuning, backup and restore, point-in-time recovery, and HA/DR.
• Real experience with identity and access architecture — service principals vs. user-authorization (OBO) patterns, and the tradeoffs between them.
• Practical experience building and maintaining CI/CD pipelines with GitHub Actions and/or Azure DevOps.
• Proficiency with Python or another scripting language for platform automation and tooling.
• Experience using AI tools (e.g., GitHub Copilot, ChatGPT, Claude) to accelerate engineering work.
• Comfort owning the infrastructure lifecycle: provisioning, decommissioning, and disaster recovery.
• Experience working in a regulated environment, ideally healthcare, with HIPAA and/or SOC 2 requirements.
• Comfort operating with elevated privileges in a regulated, PHI-bearing environment, and the judgment to escalate rather than grant exceptions unilaterally.
• Ability to hold a standard without owning it — enforcing architecture and security standards set elsewhere in the organization through influence and automation rather than authority.
• Strong problem-solving and troubleshooting ability, including under incident conditions.
• Excellent communication and collaboration skills, and the ability to work effectively with cross-functional engineering teams.

Strong Candidates Will:

• Have exposure to data mesh or domain-oriented data architecture, and the instinct that platform teams exist to make other teams self-sufficient rather than being the bottleneck.
• Have hands-on experience with Databricks Lakebase or another cloud-native Postgres service (Azure Database for PostgreSQL, Neon, Aurora), and with Postgres tooling such as psql, pgAdmin, and migration frameworks.
• Have enough familiarity with Azure SQL Database to coordinate effectively with the DBA team on integration points and workload placement, and experience integrating OLTP systems with a lakehouse.
• Have healthcare data environment experience (HIPAA-driven access controls, PHI segregation), and demonstrate the ability to learn new business models and technologies.
• Have experience with observability tooling such as Datadog or OpenTelemetry-based pipelines.
• Have experience with Databricks Asset Bundles, DLT, or dbt in a multi-team deployment model.
• Hold relevant certifications (e.g., Databricks Data Engineer Professional, Azure Administrator, etc.).

Benefits

• Medical Insurance
• Dental Insurance
• Vision Insurance
• Short & Long Term Disability
• Life Insurance
• 401k with company match
• Flexible Time Off
• Paid Parental Leave