Data Engineer at Yirifi | Jobyssey

This is a remote position.

We are seeking a Data Engineer to build and maintain our cutting-edge ELT pipelines that transform raw, diverse data sources into reliable, structured datasets powering our analytics.

You will play a pivotal role in transforming unstructured data into structured, actionable formats, enabling efficient and reliable data processing across Yirifi's operations. This includes designing and maintaining data architecture, building robust pipelines, and collaborating with cross-functional teams to deliver scalable, high-quality data solutions.

Requirements

Experience:
- Strong proficiency in Python, including libraries such as Pandas, NumPy, and data parsing frameworks for unstructured data.
- Proficiency in designing and implementing ETL/ELT workflows using dbt and Airflow.
- Experience with SQL and relational databases to build structured data repositories.
- In-depth experience with AWS services like S3, Glue, Lambda, and CloudWatch.
Operational Competencies:
- Analytical Thinking: Strong analytical and problem-solving skills, particularly in working with unstructured data sources.
- Adaptability: Ability to work effectively in a fast-paced, dynamic environment with competing priorities.
- Communication: Excellent communication skills to partner with data scientists, analysts, and stakeholders.

Job Responsibilities
Design and Build Scalable Data Pipelines

Develop scalable ETL/ELT pipelines to process and transform unstructured data into structured formats using Airflow, AWS Glue, Python, and dbt.
Ensure pipelines handle diverse data sources (e.g., JSON, XML, text, and raw logs) and support structured outputs like relational databases or parquet files.

Data Quality and Validation

Implement automated validation checks for data consistency, completeness, and accuracy using dbt tests, AWS Glue Data Quality, or custom Python scripts.
Implement logic that ensures consistency and completeness across all data ingestion points.
Build a reporting dashboard to monitor data quality metrics and pipeline health.

Metadata Management

Enhance and maintain metadata in AWS Glue Catalog, ensuring all datasets have clear descriptions, schema definitions, and lineage tracking.
Create a centralized metadata repository for easy data discovery and governance.

Pipeline Reliability and Performance Optimization

Establish monitoring mechanisms (e.g., AWS CloudWatch) to track pipeline performance and detect bottlenecks or failures proactively.
Optimize data workflows for speed and cost efficiency, particularly for high-volume unstructured data processing.

Collaborate with Stakeholders

Translate business and compliance requirements into technical specifications for data ingestion and transformation.
Partner with data scientists and analysts to create tailored data structures for machine learning models and analytics.

Emerging Technologies

Identify and pilot emerging tools and frameworks for handling unstructured data (e.g., text parsing with NLP libraries or distributed processing with Spark).

Benefits

Be part of a rapidly growing company revolutionizing crypto risk and compliance. As a Data Engineer at Yirifi, you will have the opportunity to build innovative data solutions that directly impact our mission. Work with cutting-edge technologies, solve complex challenges, and drive meaningful change in the digital assets space. Join us to accelerate your career and make a significant impact!

Explore more jobs

Requirements

Benefits