Reporting to the Manager, BI Development, the Data Engineer will be responsible for integrating enterprise wide data from Cofense’s Cyber Security based Applications, Micro-services and other disparate data sources.
Essential Duties/Responsibilities
- Work with Architects & Cloud Systems Engineers in designing Data Platform and Architecture
- Substantial experience with SQL and no-SQL OLTP databases and OLAP data warehousing technologies, especially AWS Aurora
- Experience data modeling and building data pipelines for multi-product and/or multi-department organizations
- Develop Data Pipelines on Cloud Technologies like Azure/AWS with well-defined tool frameworks
- Able to develop ETL code to stream data from disparate (structured and semi-structured) SaaS product data stores to Data Lake/Data Warehouse using Python, Azure/AWS Data Lake services
- Ability to write complex SQL scripts and automate them using Python
- Develop test cases and unit tests for key implementations of Data Platform by adhering to software engineering best practices and standards
- Secure data end-to-end by complying data privacy rules while developing processes to move data across Applications/Data Sources and Data Lake/Data Warehouse, as well as while delivering data through SQL clients and BI tools.
- Experience building adhoc reporting APIs and service layer on top of underlying OLTP and OLAP databases
- Help integrate Data Platform with BI tools like Power BI, Tableau, Splunk etc.
- Ability to develop and interpret Entity Relationship Diagrams (ERD) across data sets in relational database systems as well as non-relational Data stores
- Able to do Data Mining and Identify trends, patterns, anomalies in complex data sets across multiple data sources/systems and present results without ambiguity
- Develop data transformations to generate Facts, Summaries, Key metrics by applying business rulesets and aggregations using Python, SQL and other transformation tools
- Able to review current processes related to data ingestion, transformation and statistical analysis and re-engineer them
- Collaborate with business users across Cofense’s departments in defining requirements, prioritize project work and deliver them timely
- Other duties as assigned
Knowledge, Skills and Abilities Required
- Expertise in SQL skills for data transformations, statistical analysis, and troubleshooting across more than one Database Platforms (MySQL, PostgreSQL, Redshift, Azure SQL Warehouse etc.)
- Expert in writing complex SQL scripts and automate them using Python
- Knowledge of Data management on NoSQL DBs like DynamoDB, Mongo, and know-how of Big Data tools Hadoop, Spark, Kafka/Kinesis/SQS/Azure Queues or other messaging tools is huge plus
- Analytical skills, with good at finding data trends/outliers, anomalies, and articulate complex information or data points with Business Users, Management, and individuals
- Enthusiasm to work with lot of data across disparate data sources and Databases
- Has strong sense of engineering craftsmanship, takes pride in the code they write.
- Has a sense of intellectual curiosity and a burning desire to learn is self-driven, actively looks for ways to contribute, and knows how to get things done
- Is deliriously customer-focused both internal and external customers
- Sees big picture impact and relationships among and across work units
- Identifies complex technical problems and tries to resolve with minimal help
Education and/or Experience:
- Bachelor’s degree in Computer Science or Math, Data Analytics, Data Sciences, BI or demonstrated industry experience preferred
- Over 5 years of proven experience in data architecture, data modelling, and lifecycle management
- Hands-on experience with relational and cloud databases, including Azure SQL Database, Microsoft SQL Server, Amazon Aurora, and Amazon Redshift
- Strong background in developing Python and data technologies
- Practical experience designing and developing ETL data pipelines and applications using SQL and Python
- Strong expertise in writing complex SQL queries for data transformation and automating processes using Python
- Experience building and consuming RESTful APIs using Python libraries
- Proficient in integrating data platforms with BI tools such as Power BI and Splunk, including dashboard and report development
- Solid experience working with Unix/Linux environments, including SSH tunnelling and writing/interpreting Bash scripts
- Advanced proficiency in Python 3, with hands-on experience using NumPy, SciPy, Scikit-learn, and Pandas
- Experience leveraging APIs to extract data and load it into databases using Python
- Disclaimer-
Cofense is committed to equal employment opportunity. We will not discriminate against employees or applicants for employment on any legally recognized basis [protected class] including, but not limited to: veteran status, uniform service member status, race, color, religion, sex (including pregnancy), gender identity, sexual orientation, national origin, age, physical or mental disability, marital status, genetic information or any other status or characteristic protected by applicable national, federal, state or local laws and ordinances. We adhere to these commitments in all aspects of employment, including recruitment, hiring, training, compensation, promotion, benefits, and discipline.