Big Data Engineer (Spark/Scala) Spark, Scala, Python, Hadoop, SQL, AWS Warsaw, Łódź, Gdańsk, Gdynia

As a recruitment company, DCG understands that every business is powered by experienced professionals. Our management style and partnership approach enable us to meet your needs and provide continuous support. Due to our ongoing growth and the large number of recruitment projects we undertake for our partners, we are currently looking for:

Responsibilities:

• Building distributed and highly parallelized BigData processing pipeline which process massive amount of data (both structured and unstructured) in near real-time
• Leverage Spark to enrich and transform corporate data to enable searching, data visualization, and advanced analytics
• Requirement analysis and questioning requirements when they are not clear
• Work closely with DevOps, QA and Product Management teams in a Continuous Delivery environment

Requirements:

• 7+ years of experience in Spark & Scala
• 5+ years of development experience with Python
• 3+ years experience with Robot Framework
• 3+ years experience with CI/CD tools such as Jenkins
• Experience with distributed data processing engines like Spark
• Experience with Hadoop and (Hive, Oozie, map reduce, ect)
• Strong SQL skills with previous experience in creating data flows
• BitBucket and GIT expert
• AWS knowledge about architectural components ( Lamba, step functions, state machine, ect)
• Unit testing Experience (Junit 5, Mockito, Spark testing)
• Code versioning strategy & Branching strategy
• Advanced English skills
• Strong working experience with Linux and bash
• Experience using following tools: Jenkins or Bamboo, Jira, bitbucket
• Demonstrate a collaborative mindset, achieving goals through effective communication and teamwork within an Agile (SAFe) environment
• Show curiosity and responsiveness, with the ability to understand others’ needs to ensure delivery of the expected outcomes
• Maintain a strong focus on quality, continuously striving to improve ways of working and delivered solutions
• Exhibit a high level of self‑motivation and ownership in day‑to‑day work

Nice to have:

• Experience with containerization using Docker
• Knowledge of streaming and queues technologies e.g. KAFKA, IBM MQ etc
• Working knowledge of DevOps

Offer:

• Private medical care
• Co‑financing for the sports card
• Constant support of dedicated consultant

Personal Consulting Agency (License No. 4642)