DevOps Engineer (Senior)

We are seeking an experienced Databricks Infrastructure and DevOps Engineer to support the MSF DataSphere platform, a comprehensive data lakehouse solution built on AWS Government Commercial Cloud (GCC) environment. The successful candidate will be responsible for managing and optimising the Databricks infrastructure, implementing robust CI/CD pipelines using government-approved tools, and ensuring compliance with stringent security and governance requirements.

Key Responsibilities

Infrastructure Management:

• Design, deploy, and maintain Databricks clusters and workspaces on AWS GCC environment

• Manage Infrastructure-as-Code (IaC) using Terraform OR Databricks components using Databricks Asset Bundle (DAB)

• Configure and optimise AWS services including S3, IAM, VPC, and security groups specifically for Databricks

• Ensure compliance with Confidential Cloud Eligible classification and Sensitive High data requirements

CI/CD Pipeline Management:

• Develop and maintain CI/CD pipelines using SHIP-HATS tools ecosystem

• Configure GitLab for version control and automated deployment workflows

• Implement code quality gates using SonarQube for static code analysis

• Manage security scanning using Fortify on Demand (FOD) for vulnerability assessment

• Oversee dependency scanning and management using Nexus IQ

• Coordinate with CloudSCAPE for security compliance scanning and remediation

Security and Compliance:

• Implement and maintain Automated Baseline Log Review (ABLR) systems

• Coordinate with GovTech Cyber Security Operations Centre (GCSOC) for security monitoring

• Manage Central Accounts Management (CAM) integration for user access control

• Coordinate and resolve Vulnerability Assessment and Penetration Testing (VAPT) findings.

• Ensure compliance with Personal Data Protection Impact Assessment (PDPIA) requirements

• Implement and maintain Vulnerability Management System (VMS)

Platform Operations:

• Monitor system performance and implement optimization strategies

• Manage secrets and encryption key rotation for data protection

• Configure email notifications for data processing status and error alerts

• Implement and maintain data retention policies

• Support SaaS whitelisting processes and approvals

• Coordinate System Security Acceptance Testing (SSAT) activities

Collaboration and Support:

• Work closely with Data Engineers, Data Analysts and security technical team.

• Support multiple datamarts

• Provide technical guidance for Raw-to-Tech and Tech-to-Business data pipelines

• Collaborate with various government divisions and stakeholders

• Support training environment setup for user onboarding and skill development

Required Qualifications

Technical Skills:

• 3+ years of experience with Databricks platform administration and optimization

• Strong expertise in AWS services, particularly in GCC environment

• Proficiency in Infrastructure-as-Code tools, preferably Terraform

• Experience with GitLab CI/CD pipeline development and management

• Knowledge of security scanning tools (SonarQube, Fortify, Nexus IQ)

• Understanding of government security frameworks and compliance requirements

• Experience with containerization and orchestration technologies

• Strong scripting skills in Python, Bash, or PowerShell

Security and Compliance:

• Experience with government security clearance processes

• Knowledge of data classification and handling procedures (Sensitive High)

• Understanding of encryption mechanisms and key management

• Familiarity with vulnerability management and remediation processes

• Experience with security monitoring and incident response procedures

Certifications (Preferred):

• Databricks Certified Associate or higher

• AWS Certified DevOps Engineer

Required Experience

• Minimum 4-5 years of DevOps/Infrastructure engineering experience

• Experience working in government environments, esp in GCC.

• Experience in agile development methodologies

Personal Attributes

• Good communication and collaboration abilities

• Attention to detail and commitment to security best practices

• Strong documentation and knowledge sharing practices

• Proactive approach to system monitoring and maintenance