Senior data engineer/architect role designing and maintaining data analytics infrastructure on AWS, Databricks, and Informatica IDMC. Day to day: building ETL/ELT pipelines and data lakes, leading cloud data migration and modernization, automating cloud services, and ensuring security and governance.
Responsibilities
Define, develop, and maintain blueprints, roadmaps, and reference architectures for data analytics infrastructure and services.
Analyze new requirements, develop solutions, and manage solution delivery through acquisition or change control.
Enhance cloud capability by designing and implementing cloud-based data analytics architectures and patterns.
Lead data migration and modernization initiatives by leveraging AWS-native services, Databricks, and IDMC.
Work closely with business leads and system owners to understand solution requirements and identify architectural patterns.
Develop and implement automation playbooks for managing and scaling cloud services, containers, and applications.
Ensure compliance with industry best practices, governance, and security guidelines for cloud-based analytics solutions.
Collaborate with DevOps and other Data Engineering teams to define, implement, and optimize data pipelines, ETL/ELT processes, and data lakes.
Assist in vendor management to ensure that contracted vendors deliver architecturally scalable and sustainable solutions.
Requirements
Education & Experience:
Degree/Master's in Computer Science, Information Technology, Computer Engineering, or equivalent.
Minimum 10 years of experience in data architect, data warehousing, big data, or advanced analytics solutions.
Technical Skills:
Databases & Data Management:
Experience with databases (e.g., Oracle, MS SQL, MySQL, Teradata, Databricks).
Expertise in data repository design (e.g., operational data stores, data marts, data lakes).
Proficiency in data query techniques (e.g., SQL, NoSQL, Spark SQL).
Hands-on experience with Databricks (Delta Lake, MLflow, Spark).
Experience with Informatica Data Management Cloud (IDMC) for data integration, transformation, and governance.
Cloud Data & Analytics:
Must-have: Strong knowledge of AWS cloud services (e.g., AWS Glue, Redshift, S3, Lambda, Kinesis, Athena, EMR).
Experience in building and optimizing ETL/ELT workflows using AWS-native tools, Databricks, or IDMC.
Understanding of event-driven architectures and microservices.
Data Analytics & Machine Learning:
Data modeling experience (e.g., Star Schema, Snowflake Schema).
Proficiency in Python/R for data transformation, analytics, and statistical computing.
Hands-on experience with ML and AI frameworks for predictive modeling and healthcare analytics.
Experience in data visualization tools (e.g., Power BI, Tableau).
DevOps & Security:
Infrastructure as Code (IaC): Terraform, CloudFormation.
CI/CD & DevOps best practices for data pipelines and cloud infrastructure.
Identity and Access Management (IAM), security best practices, and data governance.
Soft Skills:
Strong problem-solving and critical thinking skills.
Ability to communicate complex technical solutions to non-technical stakeholders.
Proven experience in working with cross-functional teams and managing multiple stakeholders.
Healthcare data governance and compliance knowledge is a plus.
Certifications (Preferred but not mandatory)
AWS Certified Solutions Architect – Associate or Professional
Databricks Certified Data Engineer
Informatica IDMC Certification