Job Summary
Design, develop, and maintain scalable data infrastructure and pipelines to support data-driven products. Collaborate cross-functionally to ensure data accuracy, integration, and compliance in a cloud-based big data environment.
Responsibilities
Design, develop, and deploy data tables, views, and marts across data warehouses, operational data stores, data lakes, and data virtualization platforms to support business needs
Extract, clean, transform, and manage data flows, including web scraping techniques, to ensure high data quality and availability
Build, launch, and maintain efficient large-scale batch and real-time data pipelines using data processing frameworks for reliable data delivery
Integrate and consolidate disparate data silos in scalable and compliant ways to enable unified data access
Collaborate with project managers, data architects, business analysts, frontend developers, designers, and data analysts to build scalable, data-driven products
Develop backend APIs and manage databases to support application functionality and performance
Apply Agile methodologies with continuous integration and delivery to accelerate development cycles
Engage in pair programming and code reviews to maintain code quality and knowledge sharing
Required competencies and certifications
Proficient in data cleaning and transformation using tools such as SQL, pandas, or R to ensure data accuracy and consistency
Skilled in building ETL pipelines with technologies like SQL Server Integration Services (SSIS), AWS Database Migration Services (DMS), Python, AWS Lambda, ECS Container tasks, Event bridge, AWS Glue, or Spring
Experienced in database design and management across various systems including SQL, PostgreSQL, AWS S3, Athena, MongoDB, Postgres/GIS, MySQL, SQLite, Volt DB, and Cassandra
Knowledgeable in cloud platforms such as AWS, Azure, or Google Cloud for big data engineering
Experienced in building production-grade data pipelines and ETL/ELT data integration workflows
Understanding of system design, data structures, and algorithms to optimize data processing
Familiar with data modeling, data access methods, and storage infrastructures like Data Mart, Data Lake, Data Virtualization, and Data Warehouse for efficient data retrieval
Familiar with REST APIs and web protocols to support data integration and application communication
Familiar with big data frameworks and tools such as Hadoop, Spark, Kafka, and RabbitMQ
Experienced with web scraping technologies including Beautiful Soup, Casper.JS, Phantom.JS, Selenium, and Node.js for customized data extraction
Knowledgeable about data governance policies, access control, and security best practices to ensure data compliance
Comfortable scripting in languages such as SQL and Python for automation and data manipulation
Proficient in both Windows and Linux development environments for versatile deployment and testing
Demonstrates interest in bridging engineering and analytics teams to enhance data-driven decision-making