Data Engineer

Data Engineer Job Description

The data engineer is responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow and collection for cross-functional teams. The Data Engineer will support software developers, database architects, data warehouse managers, data analysts, and data scientists on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects.

Duties & Responsibilities

Key Responsibilities:

• Create and maintain optimal data pipeline architecture.
• Assemble large, complex data sets that meet functional/non-functional business requirements.
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and Azure ‘big data’ technologies.
• Work with stakeholders including the Executive, Service, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
• Create data tools for analytics and data science teams.
• Work with data and analytics experts to strive for greater functionality in our data systems.
• Map data between source systems, data warehouses, and data marts.
• Keep our data separated and secure across national boundaries through multiple data centers and Cloud regions.
• Implement business rules via stored procedures, middleware, or other technologies.
• Develop and implement data extraction procedures from other systems, such as call centers, HR, or external systems.
• Create supporting documentation, such as metadata and diagrams of entity relationships, business processes, and process flow.
• Create plans, test files, and scripts for database testing, ranging from unit to integration testing.
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, redesigning infrastructure for greater scalability, etc.
• Select methods, techniques, or criteria for database evaluative procedures.
• Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency, and other key business performance metrics.
• Design and execute reporting systems for processes.
• Analyze system requirements and gather all business information.
• Research and implement appropriate technologies to ensure that the data warehouse is always operating optimally.

Desired Experience & Qualification

Experience, Knowledge and Qualifications:

• The ideal candidate is currently a Database Administrator with 3 years of experience in writing queries, extracting data, data scrubbing, and managing large volumes of records.
• Previous experience with Relational and/or NoSQL databases is essential.
• Previous experience with ETL processes is essential.
• A qualification in Data Informatics or at least 3 years of relevant experience in large data environments.
• Experience managing a team of analysts and demonstrable technical expertise in writing complex database queries.