Machine Learning System Design

Just about two years ago, I started to research and read more about the applications of machine learning and deep learning techniques beyond the textbooks. Initially, it was just an enthusiasm, but soon it became a habit! I started by reading blog posts and engineering blogs published by big tech companies to have a better understanding about what is actually happening behind the scenes and how the basic ideas fit into the software engineering design components including web servers, data bases, load balancers, etc.

In this post, I gathered a growing list of such materials, where a great deal of this effort has already been done in other posts that I cited them at the end. I tried to organize these materials and posts in a logical manner that is easier to follow for somebody who already have some knowledge in ML/DL field and is interested to know more about the implementation and deployment of these concepts.

I have also prepared a similar categorization for the state of the art (SOTA) papers presented for the popular applications. Hopefully, I can continue and keep it updated.

Last update: January 2022

Table of Contents

Part-1: Systems and Concepts by Companies

• Google
• Facebook
• Instagram
• Twitter
• Uber
• Netflix
• LinkedIn
• Airbnb
• Spotify

Part-2: Concepts Reviews and Surveys

• Recommendation
• Deep Learning
• Natural Language Processing
• Computer Vision
• Vision and Language
• Reinforcement Learning
• Graph
• Embeddings
• Meta-learning and Few-shot Learning
• Others

Part-3: Concepts In Practic

1. Data Quality
2. Data Engineering
3. Data Discovery
4. Feature Stores
5. Classification
6. Regression
7. Forecasting
8. Recommendation
9. Search & Ranking
10. Embeddings
11. Natural Language Processing
12. Sequence Modelling
13. Computer Vision
14. Reinforcement Learning
15. Anomaly Detection
16. Graph
17. Optimization
18. Information Extraction
19. Weak Supervision
20. Generation
21. Audio
22. Validation and A/B Testing
23. Model Management
24. Efficiency
25. Ethics
26. Infra
27. MLOps Platforms
28. Practices
29. Team Structure
30. Fails

Part-1 Machine Learning System Desing Patterns in Tech Companies

Google Machine Learning Courses 🔝

1. Google’s fast-paced, practical introduction to machine learning
2. Recommendation systems
   general information regarding candidate generation (content-based and collaborative filtering), retrieval, scoring, and re-ranking
3. Google Research Publications | Data Mining and Modeling
4. Google Research Publications | Machine Intelligence
5. Tensorflow embedding projector
6. Jeff Dean On Large-Scale Deep Learning At Google

Facebook (Meta) Machine Learning Contents 🔝

1. Field Guide to Machine Learning video series
2. Fighting Abuse @Scale
3. Preventing abuse using unsupervised learning
4. Community Standards report
5. Scalable data classification for security and privacy
6. Unicorn: A System for Searching the Social Graph
7. Hive – A Petabyte Scale Data Warehouse using Hadoop
8. Nemo: Data discovery at Facebook
9. Embedding-based Retrieval in Facebook Search (Paper)
10. How machine learning powers Facebook’s News Feed ranking algorithm
11. Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective (Paper)
12. Neural Code Search: ML-based code search using natural language queries
13. AI advances to better detect hate speech
14. Leveraging online social interactions for enhancing integrity at Facebook
15. Scalable data classification for security and privacy
16. Powered by AI: Advancing product understanding and building new shopping experiences
17. Here’s how we’re using AI to help detect misinformation
18. GrokNet: Unified Computer Vision Model Trunk and Embeddings For Commerce

Instagram 🔝

1. Powered by AI: Instagram’s Explore recommender system
2. Lessons Learned at Instagram Stories and Feed Machine Learning
3. Instagram Engineering

Twitter Engineering 🔝

1. Embeddings@Twitter
2. Using Deep Learning at Scale in Twitter’s Timelines
3. Improving engagement on digital ads with delayed feedback

Uber 🔝

1. Forecasting at Uber: An Introduction
2. Applying Customer Feedback: How NLP & Deep Learning Improve Uber’s Maps
3. Food Discovery with Uber Eats: Building a Query Understanding Engine
4. Food Discovery with Uber Eats: Recommending for the Marketplace
5. Food Discovery with Uber Eats: Using Graph Learning to Power Recommendations

Netflix 🔝

1. [Video] A Multi-Armed Bandit Framework for Recommendations at Netflix | Netflix

LinkedIn 🔝

1. An Introduction to AI at LinkedIn
2. Communities AI: Building Communities Around Interests on LinkedIn
3. Fairness, Privacy, and Transparency by Design in AI/ML Systems

Airbnb 🔝

1. Using Machine Learning to Predict Value of Homes On Airbnb
2. Listing Embeddings in Search Ranking
3. Learning Market Dynamics for Optimal Pricing
4. Categorizing Listing Photos at Airbnb
5. Applying Deep Learning To Airbnb Search
6. Discovering and Classifying In-app Message Intent at Airbnb
7. Machine Learning-Powered Search Ranking of Airbnb Experiences

Spotify 🔝

1. Personalized music recommendations
2. For Your Ears Only: Personalizing Spotify Home with Machine Learning

Part-2: Machine Learning Desing Concept | Reviews and Surveys

Recommendation 🔝

• Algorithms: Recommender systems survey (2013)
• Algorithms: Deep Learning based Recommender System: A Survey and New Perspectives (2019)
• Algorithms: Are We Really Making Progress? An Analysis of Neural Recommendation Approaches (2019)
• Serendipity: A Survey of Serendipity in Recommender Systems (2016)
• Diversity: Diversity in Recommender Systems – A survey (2017)
• Explanations: A Survey of Explanations in Recommender Systems (2007)

Deep Learning 🔝

• Architecture: A State-of-the-Art Survey on Deep Learning Theory and Architectures (2019)
• Knowledge distillation: Knowledge Distillation: A Survey (2021)
• Model compression: Compression of Deep Learning Models for Text: A Survey (2020)
• Transfer learning: A Survey on Deep Transfer Learning (2018)
• Neural architecture search: A Comprehensive Survey of Neural Architecture Search (2021)
• Neural architecture search: Neural Architecture Search: A Survey (2019)

Natural Language Processing 🔝

• Deep Learning: Recent Trends in Deep Learning Based Natural Language Processing (2018)
• Classification: Deep Learning Based Text Classification: A Comprehensive Review (2021)
• Generation: Survey of the SOTA in Natural Language Generation: Core tasks, applications and evaluation (2018)
• Generation: Neural Language Generation: Formulation, Methods, and Evaluation (2020)
• Transfer learning: Exploring Transfer Learning with T5: the Text-To-Text Transfer Transformer (2020)
• Transformers: Efficient Transformers: A Survey (2020)
• Metrics: Beyond Accuracy: Behavioral Testing of NLP Models with CheckList (2020)
• Metrics: Evaluation of Text Generation: A Survey (2020)

Computer Vision 🔝

• Object detection: Object Detection in 20 Years (2019)
• Adversarial attacks: Threat of Adversarial Attacks on Deep Learning in Computer Vision (2018)
• Autonomous vehicles: Computer Vision for Autonomous Vehicles: Problems, Datasets and SOTA (2021)
• Image Captioning: A Comprehensive Survey of Deep Learning for Image Captioning (2018)

Vision and Language 🔝

• Trends: Trends in Integration of Vision and Language Research: Tasks, Datasets, and Methods (2021)
• Trends: Multimodal Research in Vision and Language: Current and Emerging Trends (2020)

Reinforcement Learning 🔝

• Algorithms: A Brief Survey of Deep Reinforcement Learning (2017)
• Transfer learning: Transfer Learning for Reinforcement Learning Domains (2009)
• Economics: Review of Deep Reinforcement Learning Methods and Applications in Economics (2020)
• Discovery: Deep Reinforcement Learning for Search, Recommendation, and Online Advertising (2018)

Graph 🔝

• Survey: A Comprehensive Survey on Graph Neural Networks (2019)
• Survey: A Practical Guide to Graph Neural Networks (2020)
• Fraud detection: A systematic literature review of graph-based anomaly detection approaches (2020)
• Knowledge graphs: A Comprehensive Introduction to Knowledge Graphs (2021)

Embeddings 🔝

• Text: From Word to Sense Embeddings:A Survey on Vector Representations of Meaning (2018)
• Text: Diachronic Word Embeddings and Semantic Shifts (2018)
• Text: Word Embeddings: A Survey (2019)
• Text: A Reproducible Survey on Word Embeddings and Ontology-based Methods for Word Similarity (2019)
• Graph: A Comprehensive Survey of Graph Embedding: Problems, Techniques and Applications (2017)

Meta-learning and Few-shot Learning 🔝

• NLP: Meta-learning for Few-shot Natural Language Processing: A Survey (2020)
• Domain Agnostic: Learning from Few Samples: A Survey (2020)
• Neural Networks: Meta-Learning in Neural Networks: A Survey (2020)
• Domain Agnostic: A Comprehensive Overview and Survey of Recent Advances in Meta-Learning (2020)
• Domain Agnostic: Baby steps towards few-shot learning with multiple semantics (2020)
• Domain Agnostic: Meta-Learning: A Survey (2018)
• Domain Agnostic: A Perspective View And Survey Of Meta-learning (2002)

Others 🔝

• Transfer learning: A Survey on Transfer Learning (2009)

Part-3: Machine Learning Desing Concepts | By Topics

Data Quality 🔝

1. Reliable and Scalable Data Ingestion at Airbnb ➖ Airbnb 2016
2. Monitoring Data Quality at Scale with Statistical Modeling ➖ Uber 2017
3. Data Management Challenges in Production Machine Learning (Paper) ➖ Google 2017
4. Automating Large-Scale Data Quality Verification (Paper)Amazon 2018
5. Meet Hodor — Gojek’s Upstream Data Quality Tool ➖ Gojek 2019
6. An Approach to Data Quality for Netflix Personalization Systems ➖ Netflix 2020
7. Improving Accuracy By Certainty Estimation of Human Decisions, Labels, and Raters (Paper) ➖ Facebook 2020

Data Engineering 🔝

1. Zipline: Airbnb’s Machine Learning Data Management Platform ➖ Airbnb 2018
2. Sputnik: Airbnb’s Apache Spark Framework for Data Engineering ➖ Airbnb 2020
3. Unbundling Data Science Workflows with Metaflow and AWS Step Functions ➖ Netflix 2020
4. How DoorDash is Scaling its Data Platform to Delight Customers and Meet Growing Demand ➖ DoorDash 2020
5. Revolutionizing Money Movements at Scale with Strong Data Consistency ➖ Uber 2020
6. Zipline - A Declarative Feature Engineering Framework ➖ Airbnb 2020
7. Automating Data Protection at Scale, Part 1 (Part 2) ➖ Airbnb 2021
8. Real-time Data Infrastructure at Uber ➖ Uber 2021
9. Introducing Fabricator: A Declarative Feature Engineering Framework ➖ Doordash 2022
10. Functions & DAGs: introducing Hamilton, a microframework for dataframe generation ➖ Stitch Fix 2021

Data Discovery 🔝

1. Apache Atlas: Data Goverance and Metadata Framework for Hadoop (Code) ➖ Apache
2. Collect, Aggregate, and Visualize a Data Ecosystem’s Metadata (Code) ➖ WeWork
3. Discovery and Consumption of Analytics Data at Twitter ➖ Twitter 2016
4. Democratizing Data at Airbnb ➖ Airbnb 2017
5. Databook: Turning Big Data into Knowledge with Metadata at Uber ➖ Uber 2018
6. Metacat: Making Big Data Discoverable and Meaningful at Netflix (Code) ➖ Netflix 2018
7. Amundsen — Lyft’s Data Discovery & Metadata Engine ➖ Lyft 2019
8. Open Sourcing Amundsen: A Data Discovery And Metadata Platform (Code) ➖ Lyft 2019
9. DataHub: A Generalized Metadata Search & Discovery Tool (Code) ➖ LinkedIn 2019
10. Amundsen: One Year Later ➖ Lyft 2020
11. Using Amundsen to Support User Privacy via Metadata Collection at Square ➖ Square 2020
12. Turning Metadata Into Insights with Databook ➖ Uber 2020
13. DataHub: Popular Metadata Architectures Explained ➖ LinkedIn 2020
14. How We Improved Data Discovery for Data Scientists at Spotify ➖ Spotify 2020
15. How We’re Solving Data Discovery Challenges at Shopify ➖ Shopify 2020
16. Nemo: Data discovery at Facebook ➖ Facebook 2020
17. Exploring Data @ Netflix (Code) ➖ Netflix 2021

Feature Stores 🔝

1. Distributed Time Travel for Feature Generation ➖ Netflix 2016
2. Building the Activity Graph, Part 2 (Feature Storage Section) ➖ LinkedIn 2017
3. Fact Store at Scale for Netflix Recommendations ➖ Netflix 2018
4. Zipline: Airbnb’s Machine Learning Data Management Platform ➖ Airbnb 2018
5. Introducing Feast: An Open Source Feature Store for Machine Learning (Code) ➖ Gojek 2019
6. Michelangelo Palette: A Feature Engineering Platform at Uber ➖ Uber 2019
7. The Architecture That Powers Twitter’s Feature Store ➖ Twitter 2019
8. Accelerating Machine Learning with the Feature Store Service ➖ Condé Nast 2019
9. Feast: Bridging ML Models and Data ➖ Gojek 2020
10. Building a Scalable ML Feature Store with Redis, Binary Serialization, and Compression ➖ DoorDash 2020
11. Rapid Experimentation Through Standardization: Typed AI features for LinkedIn’s Feed ➖ LinkedIn 2020
12. Building a Feature Store ➖ Monzo Bank 2020
13. Butterfree: A Spark-based Framework for Feature Store Building (Code) ➖ QuintoAndar 2020
14. Building Riviera: A Declarative Real-Time Feature Engineering Framework ➖ DoorDash 2021
15. Optimal Feature Discovery: Better, Leaner Machine Learning Models Through Information Theory ➖ Uber 2021
16. ML Feature Serving Infrastructure at Lyft ➖ Lyft 2021

Classification 🔝

1. Prediction of Advertiser Churn for Google AdWords (Paper) ➖ Google 2010
2. High-Precision Phrase-Based Document Classification on a Modern Scale (Paper) ➖ LinkedIn 2011
3. Chimera: Large-scale Classification using Machine Learning, Rules, and Crowdsourcing (Paper) ➖ Walmart 2014
4. Large-scale Item Categorization in e-Commerce Using Multiple Recurrent Neural Networks (Paper) ➖ NAVER 2016
5. Learning to Diagnose with LSTM Recurrent Neural Networks (Paper) ➖ Google 2017
6. Discovering and Classifying In-app Message Intent at Airbnb ➖ Airbnb 2019
7. Teaching Machines to Triage Firefox Bugs ➖ Mozilla 2019
8. Categorizing Products at Scale ➖ Shopify 2020
9. How We Built the Good First Issues Feature ➖ GitHub 2020
10. Testing Firefox More Efficiently with Machine Learning ➖ Mozilla 2020
11. Using ML to Subtype Patients Receiving Digital Mental Health Interventions (Paper) ➖ Microsoft 2020
12. Scalable Data Classification for Security and Privacy (Paper) ➖ Facebook 2020
13. Uncovering Online Delivery Menu Best Practices with Machine Learning ➖ DoorDash 2020
14. Using a Human-in-the-Loop to Overcome the Cold Start Problem in Menu Item Tagging ➖ DoorDash 2020
15. Deep Learning: Product Categorization and Shelving ➖ Walmart 2021
16. Large-scale Item Categorization for e-Commerce (Paper) ➖ DianPing, eBay 2021

Regression 🔝

1. Using Machine Learning to Predict Value of Homes On Airbnb ➖ Airbnb 2017
2. Using Machine Learning to Predict the Value of Ad Requests ➖ Twitter 2020
3. Open-Sourcing Riskquant, a Library for Quantifying Risk (Code) ➖ Netflix 2020
4. Solving for Unobserved Data in a Regression Model Using a Simple Data Adjustment ➖ DoorDash 2020

Forecasting 🔝

1. Engineering Extreme Event Forecasting at Uber with RNN ➖ Uber 2017
2. Forecasting at Uber: An Introduction ➖ Uber 2018
3. Transforming Financial Forecasting with Data Science and Machine Learning at Uber ➖ Uber 2018
4. Under the Hood of Gojek’s Automated Forecasting Tool ➖ Gojek 2019
5. BusTr: Predicting Bus Travel Times from Real-Time Traffic (Paper, Video) ➖ Google 2020
6. Retraining Machine Learning Models in the Wake of COVID-19 ➖ DoorDash 2020
7. Automatic Forecasting using Prophet, Databricks, Delta Lake and MLflow (Paper, Code) ➖ Atlassian 2020
8. Introducing Orbit, An Open Source Package for Time Series Inference and Forecasting (Paper, Video, Code) ➖ Uber 2021
9. Managing Supply and Demand Balance Through Machine Learning ➖ DoorDash 2021
10. Greykite: A flexible, intuitive, and fast forecasting library ➖ LinkedIn 2021

Recommendation 🔝

1. Amazon.com Recommendations: Item-to-Item Collaborative Filtering (Paper) ➖ Amazon 2003
2. Netflix Recommendations: Beyond the 5 stars (Part 1 (Part 2) ➖ Netflix 2012
3. How Music Recommendation Works — And Doesn’t Work ➖ Spotify 2012
4. Learning to Rank Recommendations with the k -Order Statistic Loss (Paper) ➖ Google 2013
5. Recommending Music on Spotify with Deep Learning ➖ Spotify 2014
6. Learning a Personalized Homepage ➖ Netflix 2015
7. Session-based Recommendations with Recurrent Neural Networks (Paper) ➖ Telefonica 2016
8. Deep Neural Networks for YouTube Recommendations ➖ YouTube 2016
9. E-commerce in Your Inbox: Product Recommendations at Scale (Paper) ➖ Yahoo 2016
10. To Be Continued: Helping you find shows to continue watching on Netflix ➖ Netflix 2016
11. Personalized Recommendations in LinkedIn Learning ➖ LinkedIn 2016
12. Personalized Channel Recommendations in Slack ➖ Slack 2016
13. Recommending Complementary Products in E-Commerce Push Notifications (Paper) ➖ Alibaba 2017
14. Artwork Personalization at Netflix ➖ Netflix 2017
15. A Meta-Learning Perspective on Cold-Start Recommendations for Items (Paper) ➖ Twitter 2017
16. Pixie: A System for Recommending 3+ Billion Items to 200+ Million Users in Real-Time (Paper) ➖ Pinterest 2017
17. How 20th Century Fox uses ML to predict a movie audience (Paper) ➖ 20th Century Fox 2018
18. Calibrated Recommendations (Paper) ➖ Netflix 2018
19. Food Discovery with Uber Eats: Recommending for the Marketplace ➖ Uber 2018
20. Explore, Exploit, and Explain: Personalizing Explainable Recommendations with Bandits (Paper) ➖ Spotify 2018
21. Behavior Sequence Transformer for E-commerce Recommendation in Alibaba (Paper) ➖ Alibaba 2019
22. SDM: Sequential Deep Matching Model for Online Large-scale Recommender System (Paper) ➖ Alibaba 2019
23. Multi-Interest Network with Dynamic Routing for Recommendation at Tmall (Paper) ➖ Alibaba 2019
24. Personalized Recommendations for Experiences Using Deep Learning ➖ TripAdvisor 2019
25. Powered by AI: Instagram’s Explore recommender system ➖ Facebook 2019
26. Marginal Posterior Sampling for Slate Bandits (Paper) ➖ Netflix 2019
27. Food Discovery with Uber Eats: Using Graph Learning to Power Recommendations ➖ Uber 2019
28. Music recommendation at Spotify ➖ Spotify 2019
29. Using Machine Learning to Predict what File you Need Next (Part 1) ➖ Dropbox 2019
30. Using Machine Learning to Predict what File you Need Next (Part 2) ➖ Dropbox 2019
31. Learning to be Relevant: Evolution of a Course Recommendation System ➖ LinkedIn 2019
32. Temporal-Contextual Recommendation in Real-Time (Paper) ➖ Amazon 2020
33. P-Companion: A Framework for Diversified Complementary Product Recommendation (Paper) ➖ Amazon 2020
34. Deep Interest with Hierarchical Attention Network for Click-Through Rate Prediction (Paper) ➖ Alibaba 2020
35. TPG-DNN: A Method for User Intent Prediction with Multi-task Learning (Paper) ➖ Alibaba 2020
36. PURS: Personalized Unexpected Recommender System for Improving User Satisfaction (Paper) ➖ Alibaba 2020
37. Controllable Multi-Interest Framework for Recommendation (Paper) ➖ Alibaba 2020
38. MiNet: Mixed Interest Network for Cross-Domain Click-Through Rate Prediction (Paper) ➖ Alibaba 2020
39. ATBRG: Adaptive Target-Behavior Relational Graph Network for Effective Recommendation (Paper) ➖ Alibaba 2020
40. For Your Ears Only: Personalizing Spotify Home with Machine Learning ➖ Spotify 2020
41. Reach for the Top: How Spotify Built Shortcuts in Just Six Months ➖ Spotify 2020
42. Contextual and Sequential User Embeddings for Large-Scale Music Recommendation (Paper) ➖ Spotify 2020
43. The Evolution of Kit: Automating Marketing Using Machine Learning ➖ Shopify 2020
44. A Closer Look at the AI Behind Course Recommendations on LinkedIn Learning (Part 1) ➖ LinkedIn 2020
45. A Closer Look at the AI Behind Course Recommendations on LinkedIn Learning (Part 2) ➖ LinkedIn 2020
46. Building a Heterogeneous Social Network Recommendation System ➖ LinkedIn 2020
47. How TikTok recommends videos #ForYou ➖ ByteDance 2020
48. Zero-Shot Heterogeneous Transfer Learning from RecSys to Cold-Start Search Retrieval (Paper) ➖ Google 2020
49. Improved Deep & Cross Network for Feature Cross Learning in Web-scale LTR Systems (Paper) ➖ Google 2020
50. Mixed Negative Sampling for Learning Two-tower Neural Networks in Recommendations (Paper) ➖ Google 2020
51. Future Data Helps Training: Modeling Future Contexts for Session-based Recommendation (Paper) ➖ Tencent 2020
52. A Case Study of Session-based Recommendations in the Home-improvement Domain (Paper) ➖ Home Depot 2020
53. Balancing Relevance and Discovery to Inspire Customers in the IKEA App (Paper) ➖ Ikea 2020
54. How we use AutoML, Multi-task learning and Multi-tower models for Pinterest Ads ➖ Pinterest 2020
55. Multi-task Learning for Related Products Recommendations at Pinterest ➖ Pinterest 2020
56. Improving the Quality of Recommended Pins with Lightweight Ranking ➖ Pinterest 2020
57. Personalized Cuisine Filter Based on Customer Preference and Local Popularity ➖ DoorDash 2020
58. How We Built a Matchmaking Algorithm to Cross-Sell Products ➖ Gojek 2020
59. Lessons Learned Addressing Dataset Bias in Model-Based Candidate Generation (Paper) ➖ Twitter 2021
60. Self-supervised Learning for Large-scale Item Recommendations (Paper) ➖ Google 2021
61. Deep Retrieval: End-to-End Learnable Structure Model for Large-Scale Recommendations (Paper) ➖ ByteDance 2021
62. Using AI to Help Health Experts Address the COVID-19 Pandemic ➖ Facebook 2021
63. Advertiser Recommendation Systems at Pinterest ➖ Pinterest 2021
64. On YouTube’s Recommendation System ➖ YouTube 2021

Search & Ranking 🔝

1. Amazon Search: The Joy of Ranking Products (Paper, Video, Code) ➖ Amazon 2016
2. How Lazada Ranks Products to Improve Customer Experience and Conversion ➖ Lazada 2016
3. Ranking Relevance in Yahoo Search (Paper) ➖ Yahoo 2016
4. Learning to Rank Personalized Search Results in Professional Networks (Paper) ➖ LinkedIn 2016
5. Using Deep Learning at Scale in Twitter’s Timelines ➖ Twitter 2017
6. An Ensemble-based Approach to Click-Through Rate Prediction for Promoted Listings at Etsy (Paper) ➖ Etsy 2017
7. Powering Search & Recommendations at DoorDash ➖ DoorDash 2017
8. Applying Deep Learning To Airbnb Search (Paper) ➖ Airbnb 2018
9. In-session Personalization for Talent Search (Paper) ➖ LinkedIn 2018
10. Talent Search and Recommendation Systems at LinkedIn (Paper) ➖ LinkedIn 2018
11. Food Discovery with Uber Eats: Building a Query Understanding Engine ➖ Uber 2018
12. Globally Optimized Mutual Influence Aware Ranking in E-Commerce Search (Paper) ➖ Alibaba 2018
13. Reinforcement Learning to Rank in E-Commerce Search Engine (Paper) ➖ Alibaba 2018
14. Semantic Product Search (Paper) ➖ Amazon 2019
15. Machine Learning-Powered Search Ranking of Airbnb Experiences ➖ Airbnb 2019
16. Entity Personalized Talent Search Models with Tree Interaction Features (Paper) ➖ LinkedIn 2019
17. The AI Behind LinkedIn Recruiter Search and recommendation systems ➖ LinkedIn 2019
18. Learning Hiring Preferences: The AI Behind LinkedIn Jobs ➖ LinkedIn 2019
19. The Secret Sauce Behind Search Personalisation ➖ Gojek 2019
20. Neural Code Search: ML-based Code Search Using Natural Language Queries ➖ Facebook 2019
21. Aggregating Search Results from Heterogeneous Sources via Reinforcement Learning (Paper) ➖ Alibaba 2019
22. Cross-domain Attention Network with Wasserstein Regularizers for E-commerce Search ➖ Alibaba 2019
23. Understanding Searches Better Than Ever Before (Paper) ➖ Google 2019
24. How We Used Semantic Search to Make Our Search 10x Smarter ➖ Tokopedia 2019
25. Query2vec: Search query expansion with query embeddings ➖ GrubHub 2019
26. MOBIUS: Towards the Next Generation of Query-Ad Matching in Baidu’s Sponsored Search ➖ Baidu 2019
27. Why Do People Buy Seemingly Irrelevant Items in Voice Product Search? (Paper) ➖ Amazon 2020
28. Managing Diversity in Airbnb Search (Paper) ➖ Airbnb 2020
29. Improving Deep Learning for Airbnb Search (Paper) ➖ Airbnb 2020
30. Quality Matches Via Personalized AI for Hirer and Seeker Preferences ➖ LinkedIn 2020
31. Understanding Dwell Time to Improve LinkedIn Feed Ranking ➖ LinkedIn 2020
32. Ads Allocation in Feed via Constrained Optimization (Paper, Video) ➖ LinkedIn 2020
33. Understanding Dwell Time to Improve LinkedIn Feed Ranking ➖ LinkedIn 2020
34. AI at Scale in Bing ➖ Microsoft 2020
35. Query Understanding Engine in Traveloka Universal Search ➖ Traveloka 2020
36. Bayesian Product Ranking at Wayfair ➖ Wayfair 2020
37. COLD: Towards the Next Generation of Pre-Ranking System (Paper) ➖ Alibaba 2020
38. Shop The Look: Building a Large Scale Visual Shopping System at Pinterest (Paper, Video) ➖ Pinterest 2020
39. Driving Shopping Upsells from Pinterest Search ➖ Pinterest 2020
40. GDMix: A Deep Ranking Personalization Framework (Code) ➖ LinkedIn 2020
41. Bringing Personalized Search to Etsy ➖ Etsy 2020
42. Building a Better Search Engine for Semantic Scholar ➖ Allen Institute for AI 2020
43. Query Understanding for Natural Language Enterprise Search (Paper) ➖ Salesforce 2020
44. Things Not Strings: Understanding Search Intent with Better Recall ➖ DoorDash 2020
45. Query Understanding for Surfacing Under-served Music Content (Paper) ➖ Spotify 2020
46. Embedding-based Retrieval in Facebook Search (Paper) ➖ Facebook 2020
47. Towards Personalized and Semantic Retrieval for E-commerce Search via Embedding Learning (Paper) ➖ JD 2020
48. QUEEN: Neural query rewriting in e-commerce (Paper) ➖ Amazon 2021
49. Using Learning-to-rank to Precisely Locate Where to Deliver Packages (Paper) ➖ Amazon 2021
50. Seasonal relevance in e-commerce search (Paper) ➖ Amazon 2021
51. Graph Intention Network for Click-through Rate Prediction in Sponsored Search (Paper) ➖ Alibaba 2021
52. How We Built A Context-Specific Bidding System for Etsy Ads ➖ Etsy 2021
53. Pre-trained Language Model based Ranking in Baidu Search (Paper) ➖ Baidu 2021
54. Stitching together spaces for query-based recommendations ➖ Stitch Fix 2021
55. Deep Natural Language Processing for LinkedIn Search Systems (Paper) ➖ LinkedIn 2021
56. Siamese BERT-based Model for Web Search Relevance Ranking Evaluated on a New Czech Dataset (Paper, Code) ➖ Seznam 2021

Embeddings 🔝

1. Vector Representation Of Items, Customer And Cart To Build A Recommendation System (Paper) ➖ Sears 2017
2. Billion-scale Commodity Embedding for E-commerce Recommendation in Alibaba (Paper) ➖ Alibaba 2018
3. Embeddings@Twitter ➖ Twitter 2018
4. Listing Embeddings in Search Ranking (Paper) ➖ Airbnb 2018
5. Understanding Latent Style ➖ Stitch Fix 2018
6. Towards Deep and Representation Learning for Talent Search at LinkedIn (Paper) ➖ LinkedIn 2018
7. Personalized Store Feed with Vector Embeddings ➖ DoorDash 2018
8. Should we Embed? A Study on Performance of Embeddings for Real-Time Recommendations(Paper) ➖ Moshbit 2019
9. Machine Learning for a Better Developer Experience ➖ Netflix 2020
10. Announcing ScaNN: Efficient Vector Similarity Search (Paper, Code) ➖ Google 2020
11. Embedding-based Retrieval at Scribd ➖ Scribd 2021

Natural Language Processing 🔝

1. Abusive Language Detection in Online User Content (Paper) ➖ Yahoo 2016
2. Smart Reply: Automated Response Suggestion for Email (Paper) ➖ Google 2016
3. Building Smart Replies for Member Messages ➖ LinkedIn 2017
4. How Natural Language Processing Helps LinkedIn Members Get Support Easily ➖ LinkedIn 2019
5. Gmail Smart Compose: Real-Time Assisted Writing (Paper) ➖ Google 2019
6. Goal-Oriented End-to-End Conversational Models with Profile Features in a Real-World Setting (Paper) ➖ Amazon 2019
7. Give Me Jeans not Shoes: How BERT Helps Us Deliver What Clients Want ➖ Stitch Fix 2019
8. DeText: A deep NLP Framework for Intelligent Text Understanding (Code) ➖ LinkedIn 2020
9. SmartReply for YouTube Creators ➖ Google 2020
10. Using Neural Networks to Find Answers in Tables (Paper) ➖ Google 2020
11. A Scalable Approach to Reducing Gender Bias in Google Translate ➖ Google 2020
12. Assistive AI Makes Replying Easier ➖ Microsoft 2020
13. AI Advances to Better Detect Hate Speech ➖ Facebook 2020
14. A State-of-the-Art Open Source Chatbot (Paper) ➖ Facebook 2020
15. A Highly Efficient, Real-Time Text-to-Speech System Deployed on CPUs ➖ Facebook 2020
16. Deep Learning to Translate Between Programming Languages (Paper, Code) ➖ Facebook 2020
17. Deploying Lifelong Open-Domain Dialogue Learning (Paper) ➖ Facebook 2020
18. Introducing Dynabench: Rethinking the way we benchmark AI ➖ Facebook 2020
19. How Gojek Uses NLP to Name Pickup Locations at Scale ➖ Gojek 2020
20. The State-of-the-art Open-Domain Chatbot in Chinese and English (Paper) ➖ Baidu 2020
21. PEGASUS: A State-of-the-Art Model for Abstractive Text Summarization (Paper, Code) ➖ Google 2020
22. Photon: A Robust Cross-Domain Text-to-SQL System (Paper) (Demo) ➖ Salesforce 2020
23. GeDi: A Powerful New Method for Controlling Language Models (Paper, Code) ➖ Salesforce 2020
24. Applying Topic Modeling to Improve Call Center Operations ➖ RICOH 2020
25. WIDeText: A Multimodal Deep Learning Framework ➖ Airbnb 2020
26. Dynaboard: Moving Beyond Accuracy to Holistic Model Evaluation in NLP (Code) ➖ Facebook 2021
27. How we reduced our text similarity runtime by 99.96% ➖ Microsoft 2021
28. Textless NLP: Generating expressive speech from raw audio (Part 1) (Part 2) (Part 3) (Code and Pretrained Models) ➖ Facebook 2021

Sequence Modelling 🔝

1. Doctor AI: Predicting Clinical Events via Recurrent Neural Networks (Paper) ➖ Sutter Health 2015
2. Deep Learning for Understanding Consumer Histories (Paper) ➖ Zalando 2016
3. Using Recurrent Neural Network Models for Early Detection of Heart Failure Onset (Paper) ➖ Sutter Health 2016
4. Continual Prediction of Notification Attendance with Classical and Deep Networks (Paper) ➖ Telefonica 2017
5. Deep Learning for Electronic Health Records (Paper) ➖ Google 2018
6. Practice on Long Sequential User Behavior Modeling for Click-Through Rate Prediction (Paper)Alibaba 2019
7. Search-based User Interest Modeling with Sequential Behavior Data for CTR Prediction (Paper) ➖ Alibaba 2020
8. How Duolingo uses AI in every part of its app ➖ Duolingo 2020
9. Leveraging Online Social Interactions For Enhancing Integrity at Facebook (Paper, Video) ➖ Facebook 2020
10. Using deep learning to detect abusive sequences of member activity (Video) ➖ LinkedIn 2021

Computer Vision 🔝

1. Creating a Modern OCR Pipeline Using Computer Vision and Deep Learning ➖ Dropbox 2017
2. Categorizing Listing Photos at Airbnb ➖ Airbnb 2018
3. Amenity Detection and Beyond — New Frontiers of Computer Vision at Airbnb ➖ Airbnb 2019
4. How we Improved Computer Vision Metrics by More Than 5% Only by Cleaning Labelling Errors ➖ Deepomatic
5. Making machines recognize and transcribe conversations in meetings using audio and video ➖ Microsoft 2019
6. Powered by AI: Advancing product understanding and building new shopping experiences ➖ Facebook 2020
7. A Neural Weather Model for Eight-Hour Precipitation Forecasting (Paper) ➖ Google 2020
8. Machine Learning-based Damage Assessment for Disaster Relief (Paper) ➖ Google 2020
9. RepNet: Counting Repetitions in Videos (Paper) ➖ Google 2020
10. Converting Text to Images for Product Discovery (Paper) ➖ Amazon 2020
11. How Disney Uses PyTorch for Animated Character Recognition ➖ Disney 2020
12. Image Captioning as an Assistive Technology (Video) ➖ IBM 2020
13. AI for AG: Production machine learning for agriculture ➖ Blue River 2020
14. AI for Full-Self Driving at Tesla ➖ Tesla 2020
15. On-device Supermarket Product Recognition ➖ Google 2020
16. Using Machine Learning to Detect Deficient Coverage in Colonoscopy Screenings (Paper) ➖ Google 2020
17. Shop The Look: Building a Large Scale Visual Shopping System at Pinterest (Paper, Video) ➖ Pinterest 2020
18. Developing Real-Time, Automatic Sign Language Detection for Video Conferencing (Paper) ➖ Google 2020
19. Vision-based Price Suggestion for Online Second-hand Items (Paper) ➖ Alibaba 2020
20. New AI Research to Help Predict COVID-19 Resource Needs From X-rays (Paper, Model) ➖ Facebook 2021
21. An Efficient Training Approach for Very Large Scale Face Recognition (Paper) ➖ Alibaba 2021
22. Identifying Document Types at Scribd ➖ Scribd 2021
23. Semi-Supervised Visual Representation Learning for Fashion Compatibility (Paper) ➖ Walmart 2021

Reinforcement Learning 🔝

1. Deep Reinforcement Learning for Sponsored Search Real-time Bidding (Paper) ➖ Alibaba 2018
2. Budget Constrained Bidding by Model-free Reinforcement Learning in Display Advertising (Paper) ➖ Alibaba 2018
3. Reinforcement Learning for On-Demand Logistics ➖ DoorDash 2018
4. Reinforcement Learning to Rank in E-Commerce Search Engine (Paper) ➖ Alibaba 2018
5. Dynamic Pricing on E-commerce Platform with Deep Reinforcement Learning (Paper) ➖ Alibaba 2019
6. Productionizing Deep Reinforcement Learning with Spark and MLflow ➖ Zynga 2020
7. Deep Reinforcement Learning in Production Part1 Part 2 ➖ Zynga 2020
8. Building AI Trading Systems ➖ Denny Britz 2020

Anomaly Detection 🔝

1. Detecting Performance Anomalies in External Firmware Deployments ➖ Netflix 2019
2. Detecting and Preventing Abuse on LinkedIn using Isolation Forests (Code) ➖ LinkedIn 2019
3. Deep Anomaly Detection with Spark and Tensorflow (Hopsworks Video) ➖ Swedbank, Hopsworks 2019
4. Preventing Abuse Using Unsupervised Learning ➖ LinkedIn 2020
5. The Technology Behind Fighting Harassment on LinkedIn ➖ LinkedIn 2020
6. Uncovering Insurance Fraud Conspiracy with Network Learning (Paper) ➖ Ant Financial 2020
7. How Does Spam Protection Work on Stack Exchange? ➖ Stack Exchange 2020
8. Auto Content Moderation in C2C e-Commerce ➖ Mercari 2020
9. Blocking Slack Invite Spam With Machine Learning ➖ Slack 2020
10. Cloudflare Bot Management: Machine Learning and More ➖ Cloudflare 2020
11. Anomalies in Oil Temperature Variations in a Tunnel Boring Machine ➖ SENER 2020
12. Using Anomaly Detection to Monitor Low-Risk Bank Customers ➖ Rabobank 2020
13. Fighting fraud with Triplet Loss ➖ OLX Group 2020
14. Facebook is Now Using AI to Sort Content for Quicker Moderation (Alternative) ➖ Facebook 2020
15. How AI is getting better at detecting hate speech Part 1, Part 2, Part 3, Part 4 ➖ Facebook 2020

Graph 🔝

1. Building The LinkedIn Knowledge Graph ➖ LinkedIn 2016
2. Scaling Knowledge Access and Retrieval at Airbnb ➖ Airbnb 2018
3. Graph Convolutional Neural Networks for Web-Scale Recommender Systems (Paper)Pinterest 2018
4. Food Discovery with Uber Eats: Using Graph Learning to Power Recommendations ➖ Uber 2019
5. AliGraph: A Comprehensive Graph Neural Network Platform (Paper) ➖ Alibaba 2019
6. Contextualizing Airbnb by Building Knowledge Graph ➖ Airbnb 2019
7. Retail Graph — Walmart’s Product Knowledge Graph ➖ Walmart 2020
8. Traffic Prediction with Advanced Graph Neural Networks ➖ DeepMind 2020
9. SimClusters: Community-Based Representations for Recommendations (Paper, Video) ➖ Twitter 2020
10. Metapaths guided Neighbors aggregated Network for Heterogeneous Graph Reasoning (Paper) ➖ Alibaba 2021
11. Graph Intention Network for Click-through Rate Prediction in Sponsored Search (Paper) ➖ Alibaba 2021
12. JEL: Applying End-to-End Neural Entity Linking in JPMorgan Chase (Paper) ➖ JPMorgan Chase 2021

Optimization 🔝

1. Matchmaking in Lyft Line (Part 1) (Part 2) (Part 3) ➖ Lyft 2016
2. The Data and Science behind GrabShare Carpooling (Part 1) ➖ Grab 2017
3. How Trip Inferences and Machine Learning Optimize Delivery Times on Uber Eats ➖ Uber 2018
4. Next-Generation Optimization for Dasher Dispatch at DoorDash ➖ DoorDash 2020
5. Optimization of Passengers Waiting Time in Elevators Using Machine Learning ➖ Thyssen Krupp AG 2020
6. Think Out of The Package: Recommending Package Types for E-commerce Shipments (Paper) ➖ Amazon 2020
7. Optimizing DoorDash’s Marketing Spend with Machine Learning ➖ DoorDash 2020

Information Extraction 🔝

1. Unsupervised Extraction of Attributes and Their Values from Product Description (Paper) ➖ Rakuten 2013
2. Using Machine Learning to Index Text from Billions of Images ➖ Dropbox 2018
3. Extracting Structured Data from Templatic Documents (Paper) ➖ Google 2020
4. AutoKnow: self-driving knowledge collection for products of thousands of types (Paper, Video) ➖ Amazon 2020
5. One-shot Text Labeling using Attention and Belief Propagation for Information Extraction (Paper) ➖ Alibaba 2020
6. Information Extraction from Receipts with Graph Convolutional Networks ➖ Nanonets 2021

Weak Supervision 🔝

1. Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale (Paper) ➖ Google 2019
2. Osprey: Weak Supervision of Imbalanced Extraction Problems without Code (Paper) ➖ Intel 2019
3. Overton: A Data System for Monitoring and Improving Machine-Learned Products (Paper) ➖ Apple 2019
4. Bootstrapping Conversational Agents with Weak Supervision (Paper) ➖ IBM 2019

Generation 🔝

1. Better Language Models and Their Implications (Paper)OpenAI 2019
2. Image GPT (Paper, Code) ➖ OpenAI 2019
3. Language Models are Few-Shot Learners (Paper) (GPT-3 Blog post) ➖ OpenAI 2020
4. Deep Learned Super Resolution for Feature Film Production (Paper) ➖ Pixar 2020
5. Unit Test Case Generation with Transformers ➖ Microsoft 2021

Audio 🔝

1. Improving On-Device Speech Recognition with VoiceFilter-Lite (Paper)Google 2020
2. The Machine Learning Behind Hum to Search ➖ Google 2020

Validation and A/B Testing 🔝

1. Overlapping Experiment Infrastructure: More, Better, Faster Experimentation (Paper) ➖ Google 2010
2. The Reusable Holdout: Preserving Validity in Adaptive Data Analysis (Paper) ➖ Google 2015
3. Twitter Experimentation: Technical Overview ➖ Twitter 2015
4. It’s All A/Bout Testing: The Netflix Experimentation Platform ➖ Netflix 2016
5. Building Pinterest’s A/B Testing Platform ➖ Pinterest 2016
6. Experimenting to Solve Cramming ➖ Twitter 2017
7. Building an Intelligent Experimentation Platform with Uber Engineering ➖ Uber 2017
8. Scaling Airbnb’s Experimentation Platform ➖ Airbnb 2017
9. Meet Wasabi, an Open Source A/B Testing Platform (Code) ➖ Intuit 2017
10. Analyzing Experiment Outcomes: Beyond Average Treatment Effects ➖ Uber 2018
11. Under the Hood of Uber’s Experimentation Platform ➖ Uber 2018
12. Constrained Bayesian Optimization with Noisy Experiments (Paper) ➖ Facebook 2018
13. Reliable and Scalable Feature Toggles and A/B Testing SDK at Grab ➖ Grab 2018
14. Modeling Conversion Rates and Saving Millions Using Kaplan-Meier and Gamma Distributions (Code) ➖ Better 2019
15. Detecting Interference: An A/B Test of A/B Tests ➖ LinkedIn 2019
16. Announcing a New Framework for Designing Optimal Experiments with Pyro (Paper) (Paper) ➖ Uber 2020
17. Enabling 10x More Experiments with Traveloka Experiment Platform ➖ Traveloka 2020
18. Large Scale Experimentation at Stitch Fix (Paper) ➖ Stitch Fix 2020
19. Multi-Armed Bandits and the Stitch Fix Experimentation Platform ➖ Stitch Fix 2020
20. Experimentation with Resource Constraints ➖ Stitch Fix 2020
21. Computational Causal Inference at Netflix (Paper) ➖ Netflix 2020
22. Key Challenges with Quasi Experiments at Netflix ➖ Netflix 2020
23. Making the LinkedIn experimentation engine 20x faster ➖ LinkedIn 2020
24. Our Evolution Towards T-REX: The Prehistory of Experimentation Infrastructure at LinkedIn ➖ LinkedIn 2020
25. How to Use Quasi-experiments and Counterfactuals to Build Great Products ➖ Shopify 2020
26. Improving Experimental Power through Control Using Predictions as Covariate ➖ DoorDash 2020
27. Supporting Rapid Product Iteration with an Experimentation Analysis Platform ➖ DoorDash 2020
28. Improving Online Experiment Capacity by 4X with Parallelization and Increased Sensitivity ➖ DoorDash 2020
29. Leveraging Causal Modeling to Get More Value from Flat Experiment Results ➖ DoorDash 2020
30. Iterating Real-time Assignment Algorithms Through Experimentation ➖ DoorDash 2020
31. Spotify’s New Experimentation Platform (Part 1) (Part 2) ➖ Spotify 2020
32. Interpreting A/B Test Results: False Positives and Statistical Significance ➖ Netflix 2021
33. Interpreting A/B Test Results: False Negatives and Power ➖ Netflix 2021
34. Running Experiments with Google Adwords for Campaign Optimization ➖ DoorDash 2021
35. The 4 Principles DoorDash Used to Increase Its Logistics Experiment Capacity by 1000% ➖ DoorDash 2021
36. Experimentation Platform at Zalando: Part 1 - Evolution ➖ Zalando 2021
37. Designing Experimentation Guardrails ➖ Airbnb 2021
38. Network Experimentation at Scale(Paper] Facebook 2021
39. Universal Holdout Groups at Disney Streaming ➖ Disney 2021

Model Management 🔝

1. Operationalizing Machine Learning—Managing Provenance from Raw Data to Predictions ➖ Comcast 2018
2. Overton: A Data System for Monitoring and Improving Machine-Learned Products (Paper) ➖ Apple 2019
3. Runway - Model Lifecycle Management at Netflix ➖ Netflix 2020
4. Managing ML Models @ Scale - Intuit’s ML Platform ➖ Intuit 2020
5. ML Model Monitoring - 9 Tips From the Trenches ➖ Nubank 2021

Efficiency 🔝

1. GrokNet: Unified Computer Vision Model Trunk and Embeddings For Commerce (Paper) ➖ Facebook 2020
2. How We Scaled Bert To Serve 1+ Billion Daily Requests on CPUs ➖ Roblox 2020
3. Permute, Quantize, and Fine-tune: Efficient Compression of Neural Networks (Paper) ➖ Uber 2021

Ethics 🔝

1. Building Inclusive Products Through A/B Testing (Paper) ➖ LinkedIn 2020
2. LiFT: A Scalable Framework for Measuring Fairness in ML Applications (Paper) ➖ LinkedIn 2020

Infra 🔝

1. Reengineering Facebook AI’s Deep Learning Platforms for Interoperability ➖ Facebook 2020
2. Elastic Distributed Training with XGBoost on Ray ➖ Uber 2021

MLOps Platforms 🔝

1. Meet Michelangelo: Uber’s Machine Learning Platform ➖ Uber 2017
2. Operationalizing Machine Learning—Managing Provenance from Raw Data to Predictions ➖ Comcast 2018
3. Big Data Machine Learning Platform at Pinterest ➖ Pinterest 2019
4. Core Modeling at Instagram ➖ Instagram 2019
5. Open-Sourcing Metaflow - a Human-Centric Framework for Data Science ➖ Netflix 2019
6. Managing ML Models @ Scale - Intuit’s ML Platform ➖ Intuit 2020
7. Real-time Machine Learning Inference Platform at Zomato ➖ Zomato 2020
8. Introducing Flyte: Cloud Native Machine Learning and Data Processing Platform ➖ Lyft 2020
9. Building Flexible Ensemble ML Models with a Computational Graph ➖ DoorDash 2021
10. LyftLearn: ML Model Training Infrastructure built on Kubernetes ➖ Lyft 2021
11. “You Don’t Need a Bigger Boat”: A Full Data Pipeline Built with Open-Source Tools (Paper) ➖ Coveo 2021
12. MLOps at GreenSteam: Shipping Machine Learning ➖ GreenSteam 2021
13. Evolving Reddit’s ML Model Deployment and Serving Architecture ➖ Reddit 2021

Practices 🔝

1. Practical Recommendations for Gradient-Based Training of Deep Architectures (Paper) ➖ Yoshua Bengio 2012
2. Machine Learning: The High Interest Credit Card of Technical Debt (Paper) (Paper) ➖ Google 2014
3. Rules of Machine Learning: Best Practices for ML Engineering ➖ Google 2018
4. On Challenges in Machine Learning Model Management ➖ Amazon 2018
5. Machine Learning in Production: The Booking.com Approach ➖ Booking 2019
6. 150 Successful Machine Learning Models: 6 Lessons Learned at Booking.com (Paper) ➖ Booking 2019
7. Successes and Challenges in Adopting Machine Learning at Scale at a Global Bank ➖ Rabobank 2019
8. Challenges in Deploying Machine Learning: a Survey of Case Studies (Paper) ➖ Cambridge 2020
9. Reengineering Facebook AI’s Deep Learning Platforms for Interoperability ➖ Facebook 2020
10. The problem with AI developer tools for enterprises ➖ Databricks 2020
11. Continuous Integration and Deployment for Machine Learning Online Serving and Models ➖ Uber 2021
12. Tuning Model Performance ➖ Uber 2021
13. Maintaining Machine Learning Model Accuracy Through Monitoring ➖ DoorDash 2021
14. Building Scalable and Performant Marketing ML Systems at Wayfair ➖ Wayfair 2021
15. Our approach to building transparent and explainable AI systems ➖ LinkedIn 2021
16. 5 Steps for Building Machine Learning Models for Business ➖ Shopify 2021

Team structure 🔝

1. Engineers Shouldn’t Write ETL: A Guide to Building a High Functioning Data Science Department ➖ Stitch Fix 2016
2. Building The Analytics Team At Wish ➖ Wish 2018
3. Beware the Data Science Pin Factory: The Power of the Full-Stack Data Science Generalist ➖ Stitch Fix 2019
4. Cultivating Algorithms: How We Grow Data Science at Stitch Fix ➖ Stitch Fix
5. Analytics at Netflix: Who We Are and What We Do ➖ Netflix 2020
6. Building a Data Team at a Mid-stage Startup: A Short Story ➖ Erikbern 2021

Fails 🔝

1. When It Comes to Gorillas, Google Photos Remains Blind ➖ Google 2010
2. 160k+ High School Students Will Graduate Only If a Model Allows Them to ➖ International Baccalaureate 2020
3. An Algorithm That ‘Predicts’ Criminality Based on a Face Sparks a Furor ➖ Harrisburg University 2020
4. It’s Hard to Generate Neural Text From GPT-3 About Muslims ➖ OpenAI 2020
5. A British AI Tool to Predict Violent Crime Is Too Flawed to Use ➖ United Kingdom 2020
6. More in awful-ai

References

[1] Machine Learning System Design

[2] Machine Learning Surveys 【forked from eugeneyan/ml-surveys】

[3] Applied Machine Learning 【forked from eugeneyan/applied-ml】