Data Engineer

The Role: Builds performant data processing pipelines, manages relational databases, and constructs cloud-portable data lake environments using modern open storage formats and embedded analytical engines.

Responsibilities: Pipeline Design: Construct resilient data processing pipelines in Python using Medallion architecture principles (Bronze/Silver/Gold) to transform raw data into analytics-ready assets.

Open Storage Management: Standardize data storage using open formats including Parquet, Apache Iceberg, and Delta Lake to ensure cross-cloud compatibility.

Embedded Analytics: Leverage DuckDB integrated with Python for fast, lightweight, and performant localized data processing within ETL workflows.

Database Operations: Architect and optimize production PostgreSQL schemas while utilizing SQLite for embedded or lightweight localized storage needs.

Distributed Compute: Scale heavy ETL, data transformation, and analytics workloads using Databricks or Apache Spark

Requirements:

Advanced SQL proficiency with deep PostgreSQL expertise and working knowledge of SQLite.

Hands-on experience building robust analytical pipelines using Python and DuckDB.

Mastery of open table and storage formats (Parquet, Apache Iceberg, Delta Lake).

Proven track record designing data lakes using the Medallion data processing architecture.

Familiarity with the Databricks / PySpark ecosystem for distributed processing.

AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards.

WHY JOIN US

If you're looking for a place to grow, make an impact, and work with people who care, we'd love to meet you!

ABOUT THE ROLE

We are looking for a Senior Data Engineer to replace a third-party data ingestion tool with an owned ETL pipeline.

MUST HAVES

- 4+ years of strong SQL and large-scale data ingestion/pipeline experience (batch ETL, orchestration, schema design).

- Working knowledge of AWS data services (RDS/Postgres, S3, ECS) sufficient to build within an existing cloud environment.

- Practical experience with Claude Code or similar tools, and agents or MCP in real work.

- Ability to contribute to application and backend work beyond pure ETL to support broader platform needs.

- Familiarity with, or ability to quickly evaluate, managed ELT tools (such as Fivetran) against custom builds.

- Ability to ramp into existing attribution and scoring logic so ingestion output aligns with downstream dashboard needs.

NICE TO HAVES

- Direct or adjacent experience with digital advertising / DSP data (e.g., Amazon DSP, Amazon Advertising API, or comparable programmatic platforms).

- Experience replacing legacy third-party data pipelines with owned, custom infrastructure.

- Familiarity with MLOps pipelines and how data feeds predictive models.

- Experience working alongside part-time technical advisory leads to validate architecture decisions.

WHAT YOU WILL DO

- Document current ingestion logic inside the legacy tool, including sources, endpoints, schemas, and transformation logic.

- Build the ETL pipeline to replace the legacy system, handling ingestion from the Amazon API, other DSP platforms, and related digital data sources.

- Evaluate and decide between off-the-shelf solutions (e.g., Fivetran) and owned ETL systems alongside the Infrastructure Lead.

- Design the pipeline as a standalone, owned system to power client-facing dashboards.

- Provide flexible engineering support for application and backend work within the AWS environment when directed.

- Coordinate with data science teams to ensure ingestion output aligns with the attribution and scoring logic of downstream models.

- Work seamlessly within the existing AWS environment (ECS, S3, RDS) rather than standing up parallel infrastructure.

PERKS AND BENEFITS

- Professional growth: Mentorship, TechTalks, and personalized growth roadmaps.

- Competitive compensation: USD-based pay with education, fitness, and team activity budgets.

- Exciting projects: Modern solutions with Fortune 500 and top product companies.

- Flextime: Flexible schedule with remote and office options.

Job Description

AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards.

WHY JOIN US

If you're looking for a place to grow, make an impact, and work with people who care, we'd love to meet you!

ABOUT THE ROLE

We are looking for a Senior Data Engineer to operate and improve a Snowflake-based enterprise data platform in a regulated healthcare environment.

WHAT YOU WILL DO

- Provide senior technical ownership for the Data Platform service tower during the LatAm coverage window, including day-to-day operations, complex troubleshooting, and L2/L3 escalation.

- Operate and improve Snowflake production and non-production environments, including warehouse configuration and sizing, performance and consumption monitoring, object lifecycle, environment hygiene, and support for production changes.

- Administer data access within established controls, including users, roles, service accounts, secrets, and credential rotation, while maintaining least-privilege and audit-ready practices.

- Operate and improve data ingestion across Fivetran, HVR where applicable, and custom pipelines, including connector configuration, scheduling, source onboarding, schema-change coordination, failure recovery, backfills, and dependency management.

- Design, build, and maintain reliable pipelines and dbt models across RAW, CURATED, and CONSUMPTION layers, with appropriate testing, documentation, lineage, version control, and CI/CD practices.

- Support AWS S3 data-lake operations, including raw and landing-zone workflows, lifecycle and retention controls, access patterns, logging, ingestion failures, and coordination with downstream Snowflake workloads.

- Support Argo Workflows and Kubernetes-hosted data workloads in close coordination with the Cloud / DevOps team, including scheduling, troubleshooting, deployment, recovery, and capacity dependencies.

- Define and improve data quality and observability standards, including freshness, zero-row, row-count growth, null, duplicate, schema-drift, and referential-integrity checks.

- Expand end-to-end monitoring and lineage using tools such as SYNQ, dbt, Snowflake audit data, Splunk, and the agreed alerting stack, linking actionable alerts to evidence and runbooks.

- Lead or support major data incidents, root-cause analysis, post-incident reviews, and preventive actions across ingestion, orchestration, Snowflake, and downstream Tableau dependencies.

- Support Tableau Cloud operations where upstream data, connectivity, permissions, extracts, or refresh failures require Data Platform investigation.

- Identify and deliver standardization, automation, reliability, performance, and cost improvements, including migration of suitable legacy or custom extraction patterns toward agreed golden paths.

- Execute work through controlled incident, request, access, change, and release processes using established service-management workflows.

- Create and maintain runbooks, operating procedures, architecture context, ownership information, recovery procedures, and knowledge-transfer materials.

- Mentor Middle-level engineers, review technical work, improve team practices, and ensure effective handoffs across the distributed service team.

- Participate in the Data Platform on-call rotation for critical incidents outside staffed service hours.

MUST HAVES

- 5+ years of professional experience in Data Engineering or Data Platform Engineering.

- Strong hands-on experience operating and developing solutions on Snowflake, including data-layer design, warehouse performance, access patterns, and production troubleshooting.

- Advanced SQL skills and strong experience with dbt for transformation, testing, documentation, lineage, and controlled deployment.

- Experience operating managed ingestion tools such as Fivetran or HVR and supporting custom data-ingestion pipelines.

- Hands-on experience with AWS data services, particularly S3 and event-driven or file-based ingestion patterns.

- Experience orchestrating and troubleshooting data workloads with Argo Workflows on Kubernetes, or comparable workflow-orchestration technologies.

- Proficiency in Python or a comparable language for data engineering, automation, and operational tooling.

- Strong understanding of data modeling, pipeline dependencies, schema evolution, backfills, data validation, and production data quality.

- Experience with observability, logging, alerting, and incident-management practices for production data platforms.

- Demonstrated ability to lead complex incident resolution, perform root-cause analysis, and convert findings into preventive improvements.

- Ability to make well-reasoned technical decisions, identify tradeoffs, estimate work, and guide improvements across a complex platform.

- Experience mentoring engineers and collaborating effectively with Cloud / DevOps, Analytics, Security, Governance, and business stakeholders.

- Strong written and verbal English communication skills, with the ability to work directly with client stakeholders.

- Full availability to work from 9:00 AM to 6:00 PM Pacific Time and participate in an agreed on-call rotation.

NICE TO HAVES

- Hands-on experience with a data-specific observability platform.

- Experience supporting Tableau Cloud administration, data-source connectivity, extracts, scheduled refreshes, or production dashboard dependencies.

- Familiarity with Snowflake cost optimization, audit logging, tasks, stored procedures, and environment rationalization.

- Experience migrating legacy ingestion or orchestration patterns such as Boomi or AWS Data Pipeline to modern managed or Kubernetes-based solutions.

- Experience with Terraform, CI/CD pipelines, and Infrastructure as Code practices supporting data platforms.

- Experience with service-management and change-control tools such as Freshservice and Jira.

- Familiarity with HIPAA, GDPR, FDA-related controls, least-privilege access, separation of duties, and audit-ready operational practices.

PERKS AND BENEFITS

- Professional growth: Accelerate your professional journey with mentorship, TechTalks, and personalized growth roadmaps.

- Competitive compensation: We match your ever-growing skills, talent, and contributions with competitive USD-based compensation and budgets for education, fitness, and team activities.

- A selection of exciting projects: Join projects with modern solutions development and top-tier clients that include Fortune 500 enterprises and leading product brands.

- Flextime: Flexible schedule with remote and office options.

BigQuery