At Netflix, the big data platform is the core foundation in driving all of our product decisions that directly impact their customer experience. It is leveraged across the company to build data models for recommendations, analyze customer streaming experience, content take-rate, plus more. It is critical to the success of our business as they heavily rely on analyzing data to revolutionize internet TV!
They are the data infrastructure team that builds an ecosystem of microservices that expose the big data platform as a robust and highly available service to the rest of Netflix. They are constantly evolving their infrastructure. They are designing new microservices to take our architecture to the next level to keep pace with our increasing job execution and data needs.
Here is the vision of the data infrastructure team that is responsible for the orchestration layer.
Specifically, they are working on
- Transitioning their data infrastructure into a more scalable and sustainable event-driven architecture. For example, dropping a table in Metacat (their metadata service) would trigger an event to their S3 deletion service to clean up data. Finishing a job in Genie (their job management service) would trigger an event to their scheduler to start the next dependent job.
- Building an IFTTT service (called Microbots) to help them expand and scale on the usage pattern of event-driven architecture.
- Elevating the architecture and functionalities of our workflow scheduling engine (called Meson) so they could continue to grow its functionality and scalability to match their growing use cases.
- Integrating Genie with their virtual execution environment Titus so it can evolve into a distributed big data job execution environment.
- Building their first version of Data access service to allow different ML applications to access data directly in their data warehouse.
- And many more …
- If this aligns well with your passion, they would love to hear from you!
This would be your dream job if you enjoy
- Working with a massive amount of data (100+ PB) is growing rapidly.
- Understanding and solving real business needs at a large scale by applying your software engineering and analytical problem-solving skills.
- Architecting and building a robust, scalable, and highly available distributed infrastructure.
- Leading cross-functional initiatives and collaborating with engineers across teams.
- Sharing their experiences with the open source communities and contributing to Netflix OSS.
About You
- You have 5+ years of experience in building large scale distributed applications.
- You are proficient in design and development of RESTful web services.
- You are an expert in Java. Python or Scala expertise is a plus.
- You have a BS/MS/PhD in Computer Science or related field.
- Most importantly, you are thrilled to help solve our big data challenges while revolutionizing internet TV!
