The Data Engineering team is developing the backbone of Wunderkind solutions. To do this, the team is using cutting-edge technologies and frameworks such as BigQuery, Spanner, Bigtable, has &Dataproc to process and analyze data, to help create the pipelines required to handle millions of events daily, and collaborate with the data science team to enable the innovative work in machine learning and AI to be utilized in our production environment.
Responsibilities:
- Contribute to the development of Wunderkind Micro Batch/Real Time streaming pipelines (lambda architecture)
- Work closely with the data science team to build our machine learning platform infrastructure
- Work closely with the engineering team on data preprocessing and data structure
- Help mentoring the data engineering team and leading the data engineering initiatives
- Participate in architecture reviews
Qualifications:
- BA/BS or MS degree in Computer Science preferred, or equivalent work experience
- Understanding of distributed data systems
- Previous experience building lambda architecture
- Previous experience working with big data tools such as Bigtable, Apache Spark, Apache Spark Streaming, Apache Heron, BigQuery
- Preferred:
- Google Cloud, Kubernetes
- Experience running production Golang systems, Infrastructure background, Low Latency / High Throughput Architectural System Designs
- Strong interest in data engineering
- Eagerness to learn new technologies on the fly and ship to production
- Knowledge in data science is a plus
Data source
Stack Overflow