Job Title: Senior Data Engineer
About Luupli:
Luupli is a social media app that has equity, diversity, and equality at its heart. We believe that social media can be a force for good, and we are committed to creating a platform that maximise the value that creators and businesses can gain from it, while making a positive impact on society and the planet. Luupli started internal testing since June 2024 and getting ready for a commercial BETA testing from December 2024, with the hope of launching fully summer of 2025.Our team is made up of passionate and dedicated individuals who are committed to making Luupli a success.
About the Role:
We’re seeking a skilled Senior Data Engineer with experience in recommendation systems to join our team. This role is pivotal in enhancing our backend systems, managing databases, and optimizing recommendation algorithms. You’ll play a critical role in the architecture and performance of our recommendation infrastructure, ensuring accurate, relevant, and efficient recommendations for our users.
Responsibilities:
Data and Database Management:
- Design, optimize, and maintain tables and data structures to support recommendation and trending content data.
- Work with structured data storage solutions, including PostgreSQL and JSONB, to manage recommendation and interaction data.
Recommendation Systems:
- Implement and refine recommendation algorithms (e.g., collaborative filtering, content-based, and hybrid approaches) to enhance relevancy.
- Use similarity search libraries like Annoy or Faiss to optimize recommendation speed and accuracy.
- Continuously evaluate recommendation logic to better serve user preferences, ensuring real-time delivery.
Data Aggregation and Analysis:
- Aggregate, analyze, and process user interaction data to support recommendations and trending content.
- Design efficient queries and implement aggregation methods to capture relevant data and insights for recommendations.
Debugging and Optimisation:
- Identify, troubleshoot, and resolve data handling issues to ensure accurate recommendation delivery.
- Optimise queries, processing workflows, and containerised services for high performance and scalability within AWS ECS.
Requirements:
- Proven experience as a Data Engineer or Backend Engineer, with a focus on recommendation systems.
- Proficient in SQL and database management, especially with PostgreSQL and JSONB for structured data handling.
- Solid understanding of recommendation algorithms (collaborative filtering, content-based, hybrid approaches).
- Experience with similarity search libraries such as Annoy or Faiss.
- Strong programming skills in Python, with experience in building backend logic for data-intensive applications in a containerised environment.
- Familiarity with AWS ECS for container management, including task scheduling and scaling.
- Experience using AWS EventBridge to trigger workflows or automate tasks in response to application events.
- Analytical skills for data aggregation, querying, and insights generation.
- Strong debugging and optimisation skills for handling large-scale data processing in cloud-based environments.
Preferred Qualifications:
- Knowledge of data aggregation pipelines, ETL processes, and data handling at scale.
- Familiarity with additional AWS services (e.g., S3, Lambda) for data storage and event-driven architectures.
- Experience with machine learning libraries or tools used in recommendation systems.
Compensation:
This is an equity-only position, offering a unique opportunity to gain a stake in a rapidly growing company and contribute directly to its success.