Databricks Data Engineer

About the job

Job Title: Data Engineer/RSA (Databricks Focused)

Location: Remote

Employment Type: W2 Contract (No C2C and H1B candidates)

About the Role

We are seeking an experienced Data Engineer/ RSA with 10–12+ years of experience to act as the primary technical bridge between our data platforms and enterprise clients. This is a high-velocity role for an architect who thrives in ambiguity and can seamlessly manage multiple concurrent client engagements.

Unlike traditional architecture roles, you will be embedded directly within client delivery teams to lead the architecture, design, implementation, and optimization of enterprise Databricks Lakehouse platforms. You will translate complex business requirements into scalable, production-ready solutions while providing technical leadership, architectural guidance, and engineering best practices.

Required Experience

• 10–12+ years of experience in Data Engineering, Cloud Engineering, or Solution Architecture.
• 5+ years of hands-on experience with Databricks Lakehouse Platform.
• Experience leading enterprise-scale cloud data platform implementations.
• Strong client-facing experience working directly with technical and business stakeholders.
• Experience mentoring engineering teams and leading technical design discussions.

Key Responsibilities

● Lead technical priorities across multiple enterprise client engagements simultaneously, balancing rapid delivery with scalable architecture and long-term platform strategy.

● Architect and maintain automated CI/CD pipelines (GitHub Actions, Azure DevOps, Jenkins) for data engineering workflows, ensuring robust version control, automated testing, and seamless deployment of Databricks assets.

● Serve as the primary technical advisor within client delivery teams, translating business challenges into scalable, production-ready data solutions while mentoring engineering teams and driving architectural decisions.

● Architect, build, and optimize enterprise-scale Databricks Lakehouse platforms using Medallion Architecture (Bronze/Silver/Gold), supporting multiple business domains and large-scale production workloads.

● Own the full development lifecycle, from technical design and ETL/ELT pipeline construction (Databricks Workflows, SQL Warehouses, Spark) to deployment, monitoring, and production support.

● Implement governed Lakehouse patterns using Unity Catalog, RBAC/ABAC, lineage, and compliance controls while balancing security with development agility.

● Design distributed integration patterns across cloud-native services (AWS / Azure / GCP) and enterprise systems (CRM, ERP, Kafka/Kinesis streams).

● Lead architecture reviews, mentor engineering teams, establish platform best practices, and drive cloud cost optimization across storage, compute, and enterprise data platforms.

Technical Qualifications

● Strong hands-on experience with Unity Catalog, Delta Live Tables (DLT), Photon Engine, and Lakehouse Federation.

● Demonstrated experience implementing CI/CD pipelines for data platforms.

● Proficiency with Terraform and version control workflows.

● Proficiency in Spark (PySpark & Scala), complex query tuning, performance optimization, and pipeline automation (Airflow, dbt).

● Hands-on experience with cloud infrastructure (AWS / Azure / GCP), containerization (Docker), and building/debugging data-connected applications (FastAPI, Flask, Node.js).

● Experience with real-time streaming technologies (Kafka, Kinesis, Pub/Sub, Auto Loader).

Interview & Placement Process

Our hiring process is designed for speed and precision, matching your unique skill set with fast-moving, high-impact client projects. The typical timeline is one to a few weeks, depending on current project allocations.

1. Initial Screening: A quick background and qualification screening with the CloudTech Innovations team.

2. Client Introduction: A brief introductory phone call with the client to align on high-level goals.

3. Technical Deep-Dive: A one-round technical interview held by our Implementation Partner. This is where we analyze your depth of knowledge and technical versatility. Based on this assessment, we match your core strengths with the specific end-client requirements best suited to your profile.

Note on Flexibility: If you are a strong candidate but not the right fit for the specific project discussed, we pivot to match you with other open end-client projects that utilize your skills.

4. Final Interview: A final discussion with the end client to confirm project fit. There may be multiple rounds based on project needs and client requirements.

Ideal Candidate Profile

The Adaptive Builder: You are comfortable jumping into unfamiliar codebases or project structures and immediately adding value. You do not just support projects; you drive them.

The High-Pace Operator: You thrive in an environment where speed-to-market is critical. You balance the need for quick wins with long-term architectural stability.

The Client Partner: You possess the interpersonal skills to act as a technical advisor to stakeholders, managing expectations, translating technical trade-offs into business value, and maintaining trust across multiple concurrent workstreams.

Ownership Mindset: You take complete pride in the quality of your code, the stability of your deployments, and the measurable success of your clients.

Certifications: Certifications in Databricks, AWS/GCP/Azure Solution Architecture, or TOGAF are a plus.