Senior Python ETL Developer - Data Engineering & Database Migration

Senior Python ETL Developer

Data Engineering & Database Migration

Overall Experience :

10+ years

Primary Focus :

Python, Pandas & ETL/ELT

Target Platform :

PostgreSQL on GCP

JRD Systems is seeking a Senior Python ETL Developer with 10+ years of overall IT experience and strong hands-on expertise in Python, Pandas DataFrames, ETL pipeline development, and relational databases. The candidate will design and implement reliable data extraction, transformation, and loading pipelines to migrate data from legacy Microsoft SQL Server databases to PostgreSQL on Google Cloud Platform (GCP). DuckDB experience for local analytical processing is an advantage, but it is not mandatory.

Key Responsibilities :

- Design, develop, and maintain end-to-end ETL/ELT pipelines using Python and Pandas; build reusable components for extraction, transformation, validation, and loading.

- Use Pandas DataFrames and Series for cleansing, filtering, aggregation, joins, merging, reshaping, and complex business transformations.

- Process large datasets efficiently using vectorized operations, chunking, and memory optimization.

- Extract data from Microsoft SQL Server and migrate transformed data to PostgreSQL, using PostgreSQL COPY and appropriate database drivers for bulk loading.

- Develop file-based workflows for CSV, JSON, and other structured data formats.

- Implement source-to-target mappings, schema transformations, data type conversions, and data quality checks.

- Build reconciliation checks for row counts, duplicates, nulls, aggregate comparisons, and business-rule validation.

- Create restartable and idempotent pipelines with batch processing, logging, error handling, transactions, and recovery mechanisms.

- Optimize Python code, Pandas operations, SQL queries, and database interactions for performance and scalability.

- Integrate migration workflows with GCP services, including Cloud SQL for PostgreSQL, Google Cloud Storage, Secret Manager, and Cloud Logging.

- Write automated tests, maintain technical documentation, participate in code reviews, and mentor developers on Python, ETL architecture, data processing, and database migration.

Required Skills - Python & Pandas :

- 10+ years of overall IT experience with substantial hands-on Python development and data-processing experience.

- Strong Python 3.x skills, object-oriented programming, modular design, and exception handling.

- Extensive practical experience with Pandas DataFrames and Series, including merge, join, concat, groupby, pivot, melt, filtering, and aggregation.

- Understanding of vectorized operations, data type conversions, missing-value handling, and date/time transformations.

- Experience optimizing DataFrame operations for large datasets and managing memory efficiently; familiarity with NumPy and related Python data-processing libraries.

ETL & Data Engineering :

- Strong understanding of ETL/ELT architecture and reusable, configurable, maintainable data pipeline frameworks.

- Experience with data extraction, cleansing, transformation, normalization, and loading; full and incremental loads, batch processing, and data synchronization.

- Knowledge of data validation, source-to-target reconciliation, audit trails, and data quality management.

- Understanding of idempotency, transaction management, restartability, and failure recovery.

SQL, Databases & Migration :

- Strong SQL skills, including joins, CTEs, aggregations, subqueries, and query optimization.

- Hands-on experience with Microsoft SQL Server and PostgreSQL, including schemas, indexes, constraints, transactions, and data types.

- Experience with Python database connectivity libraries such as pyodbc, psycopg, or SQLAlchemy.

- Experience with database migration and bulk data loading.

Cloud & Engineering Practices :

- Experience with Google Cloud SQL for PostgreSQL and Google Cloud Storage; familiarity with IAM, Secret Manager, and Cloud Logging.

- Experience with Docker, Git, automated testing using pytest, and CI/CD practices.

- Ability to troubleshoot data discrepancies, pipeline failures, and performance bottlenecks.

Good-to-Have Skills (Not Mandatory) :

- Basic or working knowledge of DuckDB for analytical processing and SQL-based transformations, including querying CSV and Parquet files directly.

- Familiarity with using DuckDB alongside Python and Pandas for data transformation and validation.

- Exposure to PyArrow, Apache Spark, or other data-processing technologies.

- Familiarity with Apache Airflow, Prefect, or similar workflow orchestration tools.

- Experience with Cloud Run Jobs or other GCP-based execution environments.

Expected Candidate Profile :

- 10+ years of overall IT experience with strong Python development and data engineering capabilities.

- Demonstrated hands-on expertise in Pandas and DataFrame-based transformations, with experience building and troubleshooting production ETL pipelines.

- Ability to independently drive technical design, development, testing, and deployment.

- Strong analytical, problem-solving, and communication skills.

- DuckDB is desirable but must not be treated as a screening requirement.

Core Technology Stack :

Python, Pandas, DataFrames, NumPy, ETL/ELT, SQL, Microsoft SQL Server, PostgreSQL, CSV, JSON, pyodbc, psycopg, SQLAlchemy, GCP Cloud SQL, Google Cloud Storage, Secret Manager, Cloud Logging, Docker, pytest, and Git. DuckDB, PyArrow, Apache Spark, Airflow/Prefect, and Cloud Run Jobs are advantageous.