Data Engineer

The data engineer is responsible for designing, developing, and maintaining the infrastructure and systems required for data storage, processing, and analysis for AI initiatives. They play a crucial role in building and managing the data pipelines that enable efficient and reliable data integration, transformation, and delivery for all Al projects.

Key Responsibilities:

• Designs and develops data pipelines that extract data from various sources, transform it into the desired format, and load it into the appropriate data storage systems
• Collaborates with AI developers, data scientists and analysts to optimize models and algorithms for data quality, security, and governance
• Integrates data from different sources, including databases, data warehouses, APIs, and external systems
• Ensures data consistency and integrity during the integration process, performing data validation and cleaning as needed
• Transforms raw data into a usable format by applying data cleansing, aggregation, filtering, and enrichment techniques
• Optimizes data pipelines and data processing workflows for performance, scalability, and efficiency
• Monitors and tunes data systems, identifies and resolves performance bottlenecks, and implements caching and indexing strategies to enhance query performance
• Implements data quality checks and validations within data pipelines to ensure the accuracy, consistency, and completeness of data
• Supports the governance of data and algorithms used for analysis, analytical applications, and automated decision making

Desired Candidate Profile

• A bachelor's degree in computer science, data science, software engineering, information systems, or related quantitative field; master's degree preferred
• At least six years of work experience in data management disciplines, including data integration, modeling, optimization and data quality, or other areas directly relevant to data engineering responsibilities and tasks
• Proven project experience developing and maintaining data pipelines for AI projects

Technical Skills:

• Expert knowledge in Apache technologies such as Kafka, Airflow, and Spark to build scalable and efficient data pipelines
• Ability to design, build, and deploy data solutions that capture, explore, transform, and utilize data to support AI, ML, and Bl
• Strong ability in programming languages such as Java, Python, and C/C++
• Ability in data science languages/tools such as SQL, R, SAS, or Excel
• Proficiency in the design and implementation of modern data architectures and concepts such as cloud services (AWS, Azure, GCP) and modern data warehouse tools (Snowflake, Databricks)
• Experience with database technologies such as SQL, NoSQL, Oracle, Hadoop, or Teradata
• Ability to collaborate within and across teams of different technical knowledge to support delivery and educate end users on data products
• Expert problem-solving skills, including debugging skills, allowing the determination of sources of issues in unfamiliar code or systems, and the ability to recognize and solve repetitive problems
• Excellent business acumen and interpersonal skills; able to work across business lines at a senior level to influence and effect change to achieve common goals.
• Ability to describe business use cases/outcomes, data sources and management concepts, and analytical approaches/options
• Ability to translate among the languages used by executive, business, IT, and quant stakeholders.