Solution Architect - Data Engineering
Avisoft · India ·
- Category
- Data engineering
- Experience
- 7+ years
Avisoft · India ·
Experience :
- 10+ years of overall IT / Consulting experience.
- 7+ years of experience in Data Engineering, Big Data, Analytics, or related technologies.
- 68+ end-to-end Databricks implementation projects.
- Proven experience in Solution Architecture with strong client-facing exposure.
Mandatory Technical Skills :
- Databricks: Strong hands-on experience and solution architecture expertise.
- Apache Spark / PySpark: Deep expertise, including Spark internals and performance tuning.
- Delta Lake / Lakehouse: Strong understanding of Lakehouse architecture and design patterns.
- Unity Catalog: Security, governance, access control, and data management.
- Databricks Platform: Workflows, Auto Loader, Databricks SQL, batch and streaming pipelines.
- Spark Performance Optimization: Partitioning, shuffle optimization, data skew, join optimization, AQE, and Catalyst.
- Databricks Compute: Cluster sizing, autoscaling, Photon, and workload optimization.
- Programming: Python / PySpark and Advanced SQL.
- Data Engineering: Large-scale ETL/ELT pipeline design and implementation.
- Cloud: Deep expertise in at least one cloud platform (AWS / Azure / GCP) and working exposure to a second cloud.
- CI/CD & DevOps: Git, Terraform, Databricks Asset Bundles, and at least one of Azure DevOps, GitHub, GitLab, or Jenkins.
- Security & Governance: Databricks security, governance, access management, and enterprise data controls.
- MLflow / MLOps: Working knowledge of MLflow and MLOps concepts.
- Certification: Databricks Certified Data Engineer Professional Preferred / Highly Preferred.
Solution Architecture & Client-Facing Skills:
- Strong experience conducting architecture workshops and requirement-gathering sessions.
- Ability to translate business requirements into scalable Databricks / Lakehouse solutions.
- Experience in solution design, architecture documentation, technical presentations, and client discussions.
- Strong troubleshooting, problem-solving, and stakeholder management skills.
- Ability to lead technical discussions with both business and engineering stakeholders.