Find resources related to teaching and research on how to build, deploy, assure, and maintain software products with machine-learned models. For example, how to integrate a voice-to-text model and an LLM into a video conferencing product to create automated meeting summaries. We cover the entire lifecycle from a prototype ML model to an entire product deployed in the real world, not just models or notebooks. Covers also the responsible ML engineering of such systems (safety, security, fairness, transparency) and MLOps.
All materials (book, slides, assignments, bibliography) are released under creative commons licenses. We hope that this fosters teaching and research on these topics.
Maintained by Christian Kaestner.
The following talk motivates the entire endeavor, explains the need to focus on engineering the entire system, not just the model, and runs through what this means from the lens of quality assurance (model testing to system testing):
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/G-N1iBYYXVo" frameborder="0" allowfullscreen="true"></iframe>Print & Ebook version Online version
A book has been published MIT Press as open access. All author proceeds are donated to Evidence Action. The complete book is available online under a creative common license here.
We teach a 12-unit course at Carnegie Mellon University on this topic, open to undergraduates and graduate students. We expect some minimal machine learning background and some programming skills, but no prior software engineering research. The course is always offered in the spring semester and often also in the fall.
- Fall 2025: F2025 website and F2025 on GitHub
- Spring 2025: S2025 website and S2025 on GitHub
- Fall 2024: F2024 website and F2024 on GitHub
- Spring 2024: S2024 website and S2024 on GitHub
- Spring 2023: S2023 website and S2023 on GitHub
- Fall 2022: F2022 website and F2022 GitHub branch
- Spring 2022: S2022 website and S2022 GitHub branch
- Spring 2021: S2021 website and S2021 GitHub branch
- Fall 2020: F2020 website and F2020 GitHub branch
- Summer 2020 (with video recordings): S2020 website and S2020 GitHub branch
- Fall 2019: F2019 website and F2019 GitHub branch.
For a description of topics covered and course structure, see learning goals.
An (opinionated) annotated bibliography of academic papers in this space, covering a wide range of topics from research on testing to requirements to notebooks: https://github.com/ckaestne/seaibib
A curated set of open source products that use machine learning. These are all end-user products that incorporate machine learning models, not libraries, research prototypes, or notebooks. We hope that this lists facilitates research on building products beyond the model-centric view of analyzing ML components: https://github.com/mlip-cmu/awesome-ml-products. For more applications also see this list of Open Source Android Apps with LLMs.
- 17-649 Artificial Intelligence for Software Engineering: This course focuses on how AI techniques can be used to build better software engineering tools and goes into more depth with regard to specific AI techniques, whereas we focus on how software engineering techniques can be used to build AI-enabled systems. Our application scenarios are typical web-based systems for end users, rather than tools for software developers.
- 05-318 Human-AI Interaction: Focuses on the HCI angle on designing AI-enabled products. Overlaps in some coverage on fairness, covers in much more detail user interface design and how to involving humans in ML-supported decisions, whereas this course focuses more on architecture design, requirements engineering, and deploying systems in production. Both courses are complementary.
- 17-646 DevOps: Modern Deployment, 17-647 Engineering Data Intensive Scalable Systems, and similar: These course cover techniques to build scalable, reactive, and reliable systems in depth. We will survey DevOps, and big data systems in the context of designing and deploying systems, but will not explore them in as much detail as a dedicated course can. We will look at MLOps as a ML-specific variant of DevOps.
- 10-601 Machine Learning, 15-381 Artificial Intelligence: Representation and Problem Solving, 05-834 Applied Machine Learning, 95-865 Unstructured Data Analytics, 10-718: Machine Learning in Practice, and many others: CMU offers many course that teach how machine learning and artificial intelligence techniques work internally or how to apply them to specific problems (including feature engineering and model evaluation), often on static data sets. We assume a basic understanding of such techniques and processes (see prerequisites) but focus on the engineering process for production ML systems.
- 17-691 Machine Learning in Practice: Applied machine learning with an focus on deployment and monitoring of models.
- 15-884 Machine Learning Systems, 10-714 Deep Learning Systems: Courses that focus on the systems aspect of building ML libraries, such as distributed learning and using hardware acceleration.
- 17-630 Prompt Engineering: Course focused specifically on prompt engineering for large language models.
- 10-613 Machine Learning, Ethics and Society, 16-735 Ethics and Robotics, [05-899 Fairness, Accountability, Transparency, & Ethics (FATE) in Sociotechnical Systems], and others dive much deeper into ethical issues and fairness in machine learning, in some cases diving deeper into statistical notions or policy. We will cover these topics in a two-week segment among many others.