Responsibilities
- Implement a large scale data warehouse in AWS
- Implement high-performance data pipelines that can be scaled to process petabytes of data on a daily basis
- Design, implement and maintain ETL processes
- Implement Direct Acyclic Graphs (DAGs) in Apache Airflow to programmatically author, schedule and monitor workflows
- Design and build rest APIs using Python Flask framework
- Work with data scientists and productionize machine learning algorithms for real-time fraud detection
- Work with data analysts to automate and optimize reporting and BI infrastructure
Requirements
- At least 5 years of experience as a data engineer or a back end developer
- Proficient in programming in Java and Python
- Proficient in Apache Spark and Airflow
- Proficient in writing and optimizing SQL statements
- Proficient with AWS and/or Cloud Computing
- Experienced with Data Engineering services like Athena, Redshift, Sagemaker, Kineses etc.
- Experienced with SQL and NoSQL Databases like DynamoDB, RDS Aurora, MySQL, ElasticSearch, Solr, etc.
- Experienced with BI tools such as Tableau, AWS Quicksight etc
- Experienced in using monitoring tools and instrumentation to ensure optimum platform and application performance
- Experienced in both streaming and batch data processing
- Knowledge of machine learning concepts will be an advantage
- Knowledge of Scala will be an advantage
- Prior experience in working with cross-functional data and tech teams will be an advantage
Data source
Stack Overflow