Senior Data Engineer, Databricks

MISSION

Colas is one of North America’s largest road and infrastructure groups, and Colas ISS provides the technology and data services that keep the business running. Databricks is our long-term data platform, and we are looking for a Senior Data Engineer to build and own it. Our current major initiative is ERDW.Next, the migration of our Enterprise Reporting & Data Warehouse onto Databricks; when that lands, this role continues as the technical owner of the platform and of every new data domain we bring onto it.

This is a hands-on, build-first role, and deliberately more than a tools role. We want an engineer who brings architect-level thinking to enterprise data—target-state design, modeling and governance standards, and a sequenced migration path off our legacy SQL Server / Synapse / SSRS estate—and who then writes the production code that makes it real. Design authority here is earned by building: most of your time will be spent engineering, not advising.

You own the data platform’s architecture and are accountable for it working in production, within the enterprise direction set by our Enterprise Architect. This is a senior individual-contributor role with no direct reports; delivery priorities and partner direction sit with the Manager, Application Development & Data Engineering. You will work directly with our Power BI developers, Data Engineers, business users and citizen developers across finance, construction, equipment, and safety, and our offshore delivery partners.

COMPENSATION & BENEFITS

Compensation*: $119,340.00 – $145,860.00 USD

*Estimated target starting compensation; final compensation will be determined based on an assessment of an individual’s qualifications, education, and experience relevant to the role.

Benefits: Full-time salaried exempt employees are eligible for Colas Group benefits, as outlined in the plan documents. Available benefits include medical, dental, and vision insurance, term life insurance, supplemental life insurance, 401(K) with company match, short- and long-term disability. Other benefits include paid holidays, paid sick leave, accrued vacation time off, and paid parental leave.

Other Compensation: This position may be considered for annual bonus compensation.

RESPONSIBILITIES

Platform Architecture & Design

• Own the technical design of the ERDW.Next lakehouse: medallion layering, Delta table design, and the gold layer that feeds Power BI.
• Set the platform roadmap and defend pattern-level decisions — batch vs. streaming, Delta Live Tables vs. Workflows, where transformation logic belongs.
• Define enterprise data modeling standards: conformed dimensions, gold-layer star schemas, slowly changing dimensions, grain, and naming.
• Partner with the Enterprise Architect on the data governance framework, JD Edwards data strategy, and enterprise integration patterns.

Build & Migration

• Stay hands-on in the codebase, writing and reviewing production pipelines as the primary way you set technical standards.
• Build and optimize pipelines in PySpark, Spark SQL, and Delta Live Tables, orchestrated through Databricks Workflows.
• Engineer ingestion from JD Edwards, BMS, CMS, Cority, HCSS, Intelex, and Anaplan using CDC, Auto Loader, and MERGE.
• Drive migration of legacy workloads (SQL Server procedures, SSIS, Synapse, ADF, Qlik Replicate) with a defensible parallel-run and cutover approach.

Running the Platform

• Own technical delivery of the data platform workstream: sequence the build, surface dependencies and risk early, and land your half to plan.
• Tune performance and cost: liquid clustering, OPTIMIZE / Z-ORDER, Photon, and right-sized clusters, with clear accountability for consumption.
• Engineer reliability: data quality expectations, restartable pipelines, monitoring, and dependable SLAs through month-end close.
• Establish and maintain CI/CD: Git-based development, Databricks Asset Bundles, promotion across environments, and automated testing.

Governance & Trusted Data

• Design and implement the Unity Catalog governance model: catalog structure, entitlements, lineage, classification, and audit.
• Build semantic models with rich metadata, business definitions, and synonyms, so data is trustworthy for reporting and AI consumption alike.
• Capture design rationale in architecture decision records and runbooks, so the platform never depends on one person or partner.

Self-Service & AI Enablement

• Curate Databricks Genie Agents over the gold layer: datasets, example queries, business semantics, and verified answers.
• Work directly with business users and citizen developers across finance, construction, equipment, and safety.
• Make certified data the easiest path to use and raise their capability, rather than gatekeeping behind a request queue.
• Partner with Power BI developers on the serving layer beneath their semantic models, deciding what belongs where.

AI-Assisted Engineering

• Use agentic coding tools as a regular part of your workflow to accelerate pipeline development, legacy code migration, and testing.
• Own the correctness of AI-generated output: you review it, verify it, and remain accountable for what reaches production.

Legacy Estate, Partners & Communication

• Provide senior-level support and change-request delivery on the legacy ERDW estate (T-SQL, SSRS, Synapse, ADF) while it remains in service.
• Hold the engineering quality bar across offshore partner work (Datavail, Birlasoft) through design and code reviews and a clear definition of done.
• Communicate architecture, risk, and cost trade-offs to IT leadership and business stakeholders in plain language.

EXPERIENCE

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field; or equivalent practical experience.
• 5+ years of hands-on experience building production data platforms on Databricks / Apache Spark, backed by 8+ years of overall experience in data engineering, data warehousing, and business intelligence.
• Demonstrated ownership of the technical design of at least one enterprise data warehouse or lakehouse platform end to end—from target-state design through production delivery—rather than solely executing a design set by others.
• Currently hands-on in code and intending to remain so. This is not a design-only, advisory, or oversight position; candidates seeking to move away from day-to-day engineering will not be a fit.
• Proven track record migrating legacy on-premises or first-generation cloud warehouse workloads (SQL Server, SSIS, Synapse, ADF) to a modern lakehouse architecture.
• Experience sourcing data from ERP and operational systems (JD Edwards strongly preferred) and delivering finance and operational reporting at enterprise scale.
• Experience enabling a self-service analytics community—business users, analysts, or citizen developers—through curated data and guardrails rather than a ticket queue.
• Agentic coding tools are part of your regular workflow, and you verify and take ownership of what they produce.
• Experience reviewing the work of offshore or partner development teams and holding a consistent quality bar across them.
• Preferred certifications: Databricks Certified Data Engineer Professional and / or Databricks Certified Data Architect; Azure DP-203 or AZ-305 a plus.

PROFESSIONAL SKILLS

• Platform Design & Data Strategy: Able to define a target-state design and a sequenced migration roadmap, and to defend the trade-offs to both engineers and executives (must-have).
• Databricks Platform: Expert across Databricks SQL, Workflows / Jobs, Delta Live Tables, Unity Catalog, Auto Loader, Photon, and cluster / SQL warehouse configuration (must-have).
• Spark & Python: Advanced PySpark and Spark SQL, writing modular, tested, reusable code rather than one-off notebooks (must-have).
• Delta Lake: Deep proficiency—MERGE / upserts, time travel, OPTIMIZE, Z-ORDER, liquid clustering, partitioning, VACUUM, and schema evolution.
• Enterprise & Dimensional Data Modeling: Star schema, conformed dimensions, slowly changing dimensions, and grain design—including a gold layer purpose-built for a Power BI semantic layer (must-have).
• SQL: Expert SQL / T-SQL, including the ability to read and re-engineer large, complex legacy stored procedures.
• Data Governance: Unity Catalog, access control and entitlements, lineage, data classification, data quality frameworks, and master data / metric definition.
• Conversational & Agentic Analytics: Curating natural-language query layers over governed data—dataset selection, business semantics, synonyms, verified answers, and evaluation. Databricks Genie experience a plus.
• Self-Service Enablement: Making certified data usable by business users and citizen developers through documentation, guardrails, and coaching rather than gatekeeping.
• Azure Data Platform: ADLS Gen2, Azure Data Factory, Azure Synapse, Key Vault, Entra ID, and the networking fundamentals behind secure data services.
• Ingestion & Replication: CDC and replication tooling (Qlik Replicate or equivalent), API and file-based ingestion, and Structured Streaming concepts.
• DevOps & CI/CD: Git, Azure DevOps or GitHub Actions, Databricks Asset Bundles / Repos, automated testing, and infrastructure as code (Terraform a plus).
• Performance & Cost Optimization: Job and query tuning, cluster right-sizing, workload isolation, and demonstrable cloud cost management.
• BI Integration: Solid understanding of Power BI semantic models and of Direct Lake / DirectQuery / import trade-offs, plus Microsoft Fabric and OneLake.
• Legacy Estate: Working capability across SQL Server, SSIS, SSRS / paginated reporting, Synapse, and ADF to support the existing ERDW platform during transition.
• AI-Assisted Delivery: Fluent use of agentic coding assistants in day-to-day engineering, paired with the judgment to know when the output is wrong.
• Business Domain: Finance / P&L reporting logic; construction, equipment, or safety domains a plus. JD Edwards source-system knowledge is highly valued.

SOFT SKILLS

• Communication Skills: Ability to convey complex technical concepts and architectural trade-offs to technical and non-technical stakeholders, including leadership.
• Collaboration: Working effectively with cross-functional teams including developers, project managers, business analysts, citizen developers, and offshore delivery partners.
• Problem-Solving: Analyzing issues and developing effective solutions while considering constraints and trade-offs.
• Technical Influence: Influencing technical decision-making through demonstrated expertise rather than formal authority, and raising the capability of the engineers around you.
• Ownership & Bias for Action: Comfortable acting as the go-to technical authority and driving decisions to closure while the platform is still being defined.
• Adaptability: Openness to change and emerging technologies.
• Critical Thinking: Evaluating options and making informed decisions.
• Time Management: Prioritizing effectively across new platform build, legacy support, and governance work.
• Empathy: Understanding team and stakeholder perspectives to foster collaboration.

DISCLAIMER

This job description is not intended to be an exhaustive list of duties, responsibilities, or qualifications and may be amended to meet business needs.