Job Description:
Overview of the Role
The Senior Data & AI Platform Engineer will own the end-to-end data pipelines supporting the business, bringing client data, documents, call transcripts, and other information into a centralized PostgreSQL platform on GCP. The role also involves supporting Document AI and LLM solutions that process over 90,000 client files. This is a hands‑on Data Engineering position focused on building, operating, troubleshooting, and improving reliable production pipelines rather than data warehousing, big data, or ML model development.
Key Responsibilities:
- Own end-to-end production data pipelines, from design and development through deployment, monitoring, troubleshooting, and ongoing maintenance.
- Build and maintain Python-based data ingestion pipelines that pull information from third-party APIs and other sources such as client data, documents, and call transcripts.
- Consolidate data into a centralized PostgreSQL environment, ensuring data is reliable and accessible to downstream business systems.
- Design, develop, and maintain cloud-based data solutions using GCP and/or AWS, with GCP technologies including: Cloud Run, Cloud Functions, Cloud SQL, Pub/Sub, BigQuery, Cloud Scheduler, and Document AI
- Take primary responsibility for investigating and resolving production pipeline failures, data issues, and integration problems.
- Develop and maintain advanced PostgreSQL solutions, including complex/large views, JSONB structures, migrations, roles, grants, and database-level access controls.
- Build and support document-processing and AI/LLM pipelines, including systems capable of processing large volumes of client documents.
- Implement production LLM features and establish methods for evaluating the quality and accuracy of AI-generated outputs where applicable.
- Build or maintain MCP servers/tools that enable AI applications to interact with business data and answer meaningful business questions.
- Design secure, scoped, fail-closed APIs for accessing sensitive financial information, including parameterized queries, server-side identity verification, SQL-enforced authorization, and versioned API contracts.
- Apply appropriate privacy, security, access controls, and auditability when handling sensitive Australian client financial data.
- Work directly with Pivot Wealths Sydney-based stakeholders, advisers, brokers, and engineering team to understand business requirements and translate them into reliable technical solutions.
- Operate with a high degree of technical ownership and accountability, particularly in a small engineering environment.
- 5+ years of Python experience is preferred, but strong hands‑on skills and relevant Data Engineering experience matter more than tenure. We’re looking for someone with a solid technical foundation and a strong willingness to learn.
- Strong professional experience in Data Engineering, particularly the development and ownership of production data pipelines.
- Experience developing data ingestion & integrations using third-party APIs, rather than relying primarily on low-code/no-code integration tools.
- Hands‑on cloud experience with GCP and AWS.
- Strong experience with PostgreSQL, ideally including: JSONB, Complex or large views, Database migrations, Roles and grants, and SQL/database access controls.
- Demonstrated experience owning, troubleshooting, and resolving data pipeline issues in production.
- Experience building and maintaining complex end-to-end data pipelines, including API, document, or similar ingestion workflows.
- Strong understanding of data security, privacy, authentication, authorization, and access control, particularly when dealing with sensitive information.
- 500K per incident HMO coverage + Dental & Optical benefits
- 2-week paid Christmas vacation
- Electricity and Data subsidies
- 25K Educational Assistance
- Training and equipment will be provided
- Fixed Schedule of Mon-Fri from 7 AM to 4 PM