About Us
At Spider.com, we are building an intuitive &powerful API company. Spider.com is focused on easy to read but hard to collect data and simplify the complexity of crawling logic so customers can easily incorporate the data with a simple API call. We provide an API for Amazon product information, travel pricing, real estate data, search engine SERP results, and other structured data found on big sites. As a Residential Proxy we don’t get blocked when crawling the web because of Spider's home grown “Spider” Income™ client that connects us with millions of devices. Anyone can download and install our Spider Income™ desktop software and start collecting passive income from Spider for being part of our network.
The Role You will be responsible for leading the architecture, design and development of the data pipelines. You will work with tools, write excellent SQL scripts, build automated web scraping solutions and partner with external and internal customers.
You will be responsible for streamlining, automating and integrating the delivery of structured and unstructured data that will drive a variety of projects which range from researching individual candidates to analyzing macro market trends. In this role, you will have ownership of end-to-end development of data engineering solutions. You'll play an integral role in strategic decision-making to solve difficult web scraping challenges.
Most of our work is in GoLang because of its scaling but compared with Python. Those with a strong background in Python should apply.
Requirements You will be responsible for streamlining, automating and integrating the delivery of structured and unstructured data that will drive a variety of projects which range from researching individual candidates to analyzing macro market trends. In this role, you will have ownership of end-to-end development of data engineering solutions. You'll play an integral role in strategic decision-making to solve difficult web scraping challenges.
Most of our work is in GoLang because of its scaling but compared with Python. Those with a strong background in Python should apply.
- Based in either the U.S.A.
- Bachelor’s Degree in Computer Science, Software Engineering, or related Engineering fields
- 4-5+ years experience full-stack development
- 4-5+ years experience with Python or GoLang development
- 3+ years experience with the Django web framework (Flask ok)
- Experience with advanced web scraping utilizing real-browsers, proxies, and high concurrency
- Familiarity of Agile best practices
Other optional, but beneficial experience
- Working with .NET.
- 4-5+ years experience in Dev-Ops.
Data source
We Work Remotely