ML monitoring with Evidently. A tutorial from CS 329S: Machine Learning Systems Design.

00:00 Introduction to the Tutorial 00:30 Speaker Introduction 01:05 What ML monitoring setup depends on 02:40 How to design ML monitoring 03:58 Toy example: bike demand monitoring 05:08 Code example starts 06:02 Dataset preparation 06:29 Model training 07:45 Model validation. Regression performance dashboard. 13:18 Production model training. Shorter version of the report. 14:55 Week 1 of production use. 17:30 Week 2 of production use. Choice of widgets. 19:08 Week 3 of production use. Model quality drop. 20:10 Quality drop debugging. Data drift dashboard. 24:43 Dashboard customization. Statistical tests, bins, tabs. 31:33 How to automate batch monitoring. MLflow example. 36:43 Q: How can I share reports with my coworker? 37:43 Q: What other features are most requested? 38:44 Q: Are standard deviations useful only for normal distributions? Code example: https://github.com/evidentlyai/evidently/blob/main/examples/data_stories/bicycle_demand_monitoring_setup.ipynb Information about the course: https://stanford-cs329s.github.io/syllabus.html