Building projects is the best way both to test the skills you have acquired and to demonstrate your newfound abilities to future employers.

This course trains an individual on how to set up, manage, and monitor an end to end ML pipeline in AWS. The course covers the following topics in depth. The course assumes that a student already has partial knowledge of ML AWS (as covered in the Full Stack ML in AWS course). As such - this course does not delve into algorithms and their use in Sagemaker, but rather into the set up, configuration, scale management, and monitoring of these pipelines.
- Overview of MLOps. What is MLOps? Why is it needed? How has it evolved over the last several years? What areas comprise effective MLOps and what does the tools landscape look like?
- Overview of AWS toolchain and facilities (Accounts, Authentication and Roles, Storage in S3, Sagemaker - with and without custom code, Serverless infrastructure, Security, CloudWatch, Billing, Instance management, Model Monitoring).
- S3 - Configuration and management.
- Cloudwatch - Configuration and management. This is covered early since Cloudwatch is the primary source of log management in AWS and as such is critical to the management of all other tools covered below.
- Sagemaker - Configuration and management of training jobs. Instances, Scale, Regions. Tracking and monitoring Sagemaker jobs via Cloudwatch. Cost and cost management of Sagemaker. Bringing in external code (custom algorithms) as Docker containers.
- Serverless infrastructure. How Lambdas work. How to configure them and use them at various stages of the ML pipeline. Value and limitations of lambdas in an ML pipeline. Monitoring and tracking lambdas in AWS.
- Model Monitoring. Unlike other aspects of ML - Model Monitoring is a technology and theory that is unique to MLOps. As such - we cover Model Drift and other monitoring techniques at both an algorithmic and tools level. Students will learn what Model Drift is, the latest in technical solutions available to detect drift, state of research in this area, and how to apply these algorithms inside AWS to monitor model performance and behaviour.
- Model Monitoring tools available in AWS’ MLOps framework. Configuring for alerts and reports.
- Billing. Tracking and managing your AWS usage. Projecting AWS cost futures.
- Advanced topics - Security - an introduction to ML Security - open issues and how to apply the knowledge to safeguard your pipeline,, Versioning, Latest in MLOps technologies beyond AWS.

Each student will also have the opportunity to apply the MLOps toolchain to a custom ML project of their choice. Any student who completes such a project will have the opportunity to present it at the last session.

---

We start with an overview of MLOps. What is MLOps? Why is it needed? How has it evolved over the last several years? What areas comprise effective MLOps and what does the tools landscape look like?
This session also provides an overview of AWS toolchain and facilities (Accounts, Authentication and Roles, Storage in S3, Sagemaker - with and without custom code, Serverless infrastructure, Security, CloudWatch, Billing, Instance management, Model Monitoring).
Students will log into their AWS account and explore the interfaces to several of the above tools - in particular Accounts, Authentication and roles, and S3. They will learn how to set up and configure S3 in a hands-on code lab.

Cloudwatch - Configuration and management. This is covered early since Cloudwatch is the primary source of log management in AWS and as such is critical to the management of all other tools covered below.
In this session, students will learn how Cloudwatch works and what it can be used for. They will experiment with Cloudwatch configurations in a hands-on code lab.

Sagemaker - Configuration and management of training jobs. Instances, Scale, Regions. Tracking and monitoring Sagemaker jobs via Cloudwatch. Cost and cost management of Sagemaker. Bringing in external code (custom algorithms) as Docker containers.
Students will set up training jobs with Sagemaker in a hands-on code lab. They will also connect their Sagemaker instances to the S3 and Cloudwatch configurations that they have learned about in Sessions 1 and 2.

Serverless infrastructure. How Lambdas work. How to configure them and use them at various stages of the ML pipeline. Value and limitations of lambdas in an ML pipeline. Monitoring and tracking lambdas in AWS.
Students will build a lambda to perform inference connectivity for their AWS Sagemaker jobs created in Session 3. They will learn how to monitor, deploy and upgrade their lambdas as well as monitor them in Cloudwatch.

Model Monitoring. Unlike other aspects of ML - Model Monitoring is a technology and theory that is unique to MLOps. As such - we cover Model Drift and other monitoring techniques at both an algorithmic and tools level. Students will learn what Model Drift is, the latest in technical solutions available to detect drift, the state of research in this area, and how to apply these algorithms inside AWS to monitor model performance and behavior.
Students will experiment with drift detection techniques and drift detection algorithms implemented natively in python. They will have the opportunity to experiment with drift detectors for various datasets.

Once the structural basics of Model Monitoring are covered in Session 5 - Session 6 is where students will learn how to configure and use the Model Monitoring tools available in AWS’ MLOps framework. Configuring for alerts and reports.
Students will configure AWS Model Monitoring in a code lab and connect it to the models and AI services built in earlier sessions.
Note - in the last three sessions, we encourage students to put the entire MLOps lifecycle into place in a custom ML project of their choice. We will provide an overview of possible projects in this session.

Each AWS service is billed independently and rates vary depending on patterns of usage. In this session we cover billing. tracking and managing your AWS usage, and projecting AWS cost futures. Students will experiment with reviewing their AWS usage, costs and ensuring that they understand how to find the rates for particular services and regions. We will also cover the AWS free tier and how it applies to different resources.

- Security - an introduction to ML Security - open issues and how to apply the knowledge to safeguard your pipeline.
- Versioning of models and pipelines. This is needed for Governance. Governance as a topic will not be covered in this course but the versioning methods will be.
- An overview of the latest advances in MLOps tools and workflows outside of AWS.
Students will also have the opportunity to present their projects.

Swami&Yamuna

Swami Sundararaman
Senior ML Engineer in Pyxeda.
Sindhu Ghanta
Head of Machine Learning in Pyxeda. She was a Post-Doctoral Fellow with BIDMC and the Department of Pathology, Harvard Medical School, where she was involved in detection and classification of features from histopathological (breast cancer) images. She worked as a research scientist with Parallel Machines on monitoring the health of machine learning algorithms in production and has many publications on ML innovations.

Nisha Talagala
Founder of Pyxeda AI. Previously, Nisha co-founded ParallelM which pioneered the MLOps practice of managing machine learning in production.

Yamuna Dulanjani
Senior Machine Learning Engineer at Pyxeda AI.