Software Engineer - Cloud Data Pipelines
INSIGHT INTERNATIONAL Limited · Stevenage, Hertfordshire ·
- Category
- Software engineering
A software engineer role building and maintaining scalable cloud data pipelines that run large-scale R and Python workloads. Day-to-day work spans containerization (Docker, Kubernetes), workflow orchestration (Airflow, Prefect), managed cloud services on GCP or Azure, and relational/non-relational databases.
Job Summary (List Format): - Design, build, and maintain scalable data pipelines for executing large-scale R and Python scripts. - Develop, containerize, and deploy custom scripts to run reliably in cloud-native (serverless or containerized) environments. - Integrate and automate data workflows using managed cloud services on Google Cloud Platform (GCP) or Microsoft Azure. - Implement and manage workflow orchestration tools (e.g., Apache Airflow, Prefect) for scheduling and monitoring. - Ensure…