data engineer for AI and ML projects

Описание

Inetum Polska is part of the global Inetum Group and drives the digital transformation of businesses and public institutions through consulting, IT infrastructure and application management, software implementation, and custom software development.

Задачи

• Design, develop, and deploy Python-based ETL/ELT pipelines to migrate data from on-premises MS SQL Server into Databricks
• Ensure efficient ingestion of historical Parquet datasets into Databricks
• Implement validation, reconciliation, and quality assurance checks for migrated data
• Handle schema mapping, field transformations, and metadata enrichment to standardize datasets
• Integrate data governance, quality assurance, and compliance into migration activities
• Tune pipelines for speed and efficiency using Databricks capabilities such as Delta Lake when appropriate
• Manage resource usage and scheduling for large dataset transfers
• Work with AI engineers, data scientists, and business stakeholders to define data access patterns for upcoming AI POCs
• Partner with infrastructure teams to ensure secure connections between legacy systems and Databricks
• Maintain technical documentation for all data pipelines
• Follow data governance, compliance, and security best practices throughout the migration process

Требования

• Proven experience with Python for data engineering tasks, including PySpark and Pandas
• Hands-on experience with Databricks and the Spark ecosystem
• Solid understanding of ETL/ELT concepts, data modeling, and pipeline orchestration
• Experience with Microsoft SQL Server, including direct database connections
• Practical experience ingesting Parquet data and managing large historical datasets
• Familiarity with secure data transfer protocols between on-premises environments and cloud platforms
• Strong problem-solving skills and ability to work independently
• Nice to have: Knowledge of Delta Lake and structured streaming in Databricks, experience with AI/ML data preparation workflows, understanding of data governance and compliance requirements related to customer and contract data, familiarity with Databricks Workflows or Airflow, experience setting up Databricks environments from first use

Условия

• Training, certifications, and participation in technology conferences are fully funded
• Flexible working hours
• Cafeteria benefits system
• Referral bonuses of up to PLN6,000
• Additional revenue-sharing opportunities for initiating partnerships with new clients
• Dedicated Team Manager guidance
• Technical mentoring from an assigned technical leader
• Team-building budget for online and on-site team events
• Opportunities to participate in charitable initiatives and local sports programs
• Supportive and inclusive work culture