Data Engineer
Viable Solutions Pty Ltd · Sydney, Australia ·
- Category
- Data engineering
airflowawsdata-qualitydbtecsevent-driven-architectureeventbridgeiceberglambdapythons3snowflakesqlsqsyaml
A Sydney-based data engineer building and maintaining dbt transformation models on Snowflake, developing Airflow DAGs on AWS ECS, and writing Python Lambda ingestion pipelines. Core stack: dbt Core, Snowflake SQL, Apache Airflow, and AWS services (S3, Lambda, ECS, SQS, EventBridge).
- Building and maintaining dbt transformation models across multiple pipeline layers — from raw ingestion through to governed, consumer-ready datasets
- Writing Snowflake SQL for complex transformations: type casting, deduplication, incremental load logic, surrogate key generation
- Developing and debugging Apache Airflow DAGs on AWS ECS
- Contributing to AWS Lambda-based ingestion( Python) — file validation, control file generation, S3 event handling
- Maintaining source-to-target mapping documentation and aligning models to business requirements and data specs
- Supporting onboarding of downstream consumers via data sharing mechanisms and event-driven notifications
- Participating in code reviews, writing YAML-based data quality tests, and owning tickets from design through to production
- Proficiency with dbt Core
- Apache Iceberg Table format (Nice to have)
- AWS - Working Knowledge of S3, Lambda, MWAA, ECS, Glue Catalogs, Lake Formation, Lambda, SQS, EventBridge
- Experience with fixed-format or structured financial data files