Space X Falcon Super Heavy rocket launch symbolizing deploy a Machine Learning Model into production

Deploy and operationalize machine learning solutions

This Study Guide describes how to deploy a Machine Learning Model into the production environment and to monitor it once it is deployed. The foundations of a reliable production environment are good Software Management and Software Engineering. The emerging job role of ML Ops, which is derived from Dev Ops, is focused on delivering the operational architecture for Machine Learning. The AWS White paper Machine Learning Lens has a section on Operational Excellence on pages 35 – 45: 

Questions

To confirm your understanding scroll to the bottom of the page for questions and answers.

Curated videos

This Study Guide covers sub-domain 4.4, Deploy and operationalize machine learning solutions. A description of all the knowledge domains in the exam is in this article: AWS Machine Learning exam syllabus

Video: An introduction to MLOps on Google Cloud

This is a 23.55 minute video from Google Cloud Platform (GCP). The first 17.30 minutes covers conceptual subjects for MLOps without referring to GCP services. This video is a good introduction to MLOps and covers many of the subjects expanded below.

A 23.55 minute video from GCP (Google Cloud Platform).

Software Engineering and Management for Machine Learning

A production Machine Learning Model needs managing the same as any software system. The Machine Learning function may be part of a larger system and will need to integrate with pre-existing systems and business processes such as:

  • Security
  • Logging and Monitoring
  • API versioning

Security

Security concerns both the security of the Machine Learning system and the data. If the data is anonymised it may need no more security than provided by the system in which it is used. However if the data contains Personal Identifiable Information (PII) or financial data it may require higher levels of security. Security for Machine Learning environments is discussed in this Study Guide:

Logging and Monitoring

Logging and Monitoring in the AWS environment is achieved by CloudWatch and CloudTrail. Both are discussed later in these revision notes.

Task management

With large, complex systems and teams of engineers you need to track the tasks and changes they make to the system. This enables Change Management and promotes engineering responsibility to the changes made. There are many commercial Task Management systems on the market. One popular choice is JIRA from Atassian which has a free version that can be used for small projects:

Version control

Machine Learning models will need to be improved, replaced or retrained. Multiple versions of models and configurations will grow and you will need a repository to manage version control. Most popular repositories are based on Git. AWS CodeCommit is the AWS git based service for version control. SageMaker model artifacts can be stored in the CodeCommit repository. Atlassian’s repository, BitBucket ,is another popular choice. BitBucket can be installed locally on AWS or used as a cloud based service.

AWS CodeCommit
Atlassian Bitbucket

Testing

End to end testing

End to end testing is a testing method that aims to run the system’s entire workflow from beginning to end. This aims to reproduce expected scenarios and exercise all systems and services that the Machine Learning system integrates with.

A/B testing

In Machine Learning A/B testing is used to compare the performance of a new model variant with the current one. In SageMaker the proportion of traffic split between the two model variants is configurable using the variant weight. Using this feature more than two variants could be tested at the same time if desired. See Deployment section below.

API versioning

SageMaker endpoints allow multiple production model variants to be deployed. Traffic can be shared between the variants depending on a configurable weighting parameter. This capability allows A/B testing, Blue Green and Canary deployments to be performed.

Reliability and failover

Reliability is the ability of systems to recover from disruptions. For Machine Learning systems inside the SageMaker managed environment much of the work is already done for you with just a little tweaking needed to maximise reliability. For example, when specifying SageMaker managed instances if you select the minimum number as two SageMaker will automatically provision each instance in different Availability Zones (AZ). This means that if there is a problem in one AZ the instance in the second AZ will still operate. SageMaker, on detecting one instance is unhealthy, will provision a new one in a different AZ. This is called failover.

If you are deploying your Machine Learning Model outside SageMaker you will have to take on more responsibility for Reliability yourself.

ML Ops

Machine Learning Operations (ML Ops) is a relatively new discipline descended from DevOps and Model Ops. It seeks to improve the quality of Production Machine Learning management by establishing an operational architecture to manage Machine Learning Model deployment, updates and operation.

ML Ops is concerned with:

  • Model generation, SDLC, CI/CD
  • Orchestration
  • Deployment
  • System health
  • Diagnostics
  • Governance
  • Business metrics

The foundation of the Machine Learning Software Development Life Cycle (SLDC) is a repository to store and version control:

  1. Model and configuration source code
  2. Model data
  3. Model artefacts

The repository provides the opportunity to enforce Change Control. This provides traceback for debugging investigations and the ability to roll back to a previous version if needed.

The Machine Learning SDLC is summarised in a diagram in the AWS White Paper,  Machine Learning Lens, on page 40:

The AWS tools that assist in this process are:

CodeBuild

AWS CodeBuild provides Source code compilation, for example, for supporting Lambda.

CodePipeline

AWS CodePipeline is used to Orchestrate workload movements from the Machine Learning system. CodePipeline provides a continuous delivery service that can trigger each stage of the MachineLearning pipeline in the correct order.

CodeCommit

AWS CodeCommit is a repository based on git.

AWS CloudFormation

AWS CloudFormation automates Infrastructure as Code and Configuration as Code to promote:

  • Consistency
  • Enable process to be created across environments
  • Automated mechanism to orchestrate workload movements

SageMaker deployment

When a model is ready to go into production there are a number of deployment options, one of which is to use Amazon SageMaker as the target deployment environment. SageMaker has a rich set of features to support Machine Learning models in production. An Amazon SageMaker endpoint is a fully managed service that allows you to make real-time inferences via a REST API. Behind the scenes SageMaker has provisioned SageMaker EC2 instances on which run Docker containers that host your model. You can configure the type and number of instances. So when we talk about a SageMaker endpoint we are often referring to the substantial compute power that SageMaker has created for us and our Model that is running on it.

There are three steps to creating a SageMaker Endpoint:

  1. Create the model
  2. Create the endpoint config
  3. Create HTTPS endpoint

Create a model

Once you have created the Model the following information is required for subsequent steps:

  • The Amazon S3 path where the model artifacts are stored. This must be in the same AWS Region as the SageMaker service you are deploying to.
  • The Docker registry path for the image that contains the inference code.
  • A name that you can use as an identifier.

Create the endpoint configuration

When creating the Endpoint configuration you will specify:

  • The Model to be hosted. You can specify more than one Model and identify the specific variants.
  • The SageMaker Managed EC2 instances that will host the Docker containers that will host your Model. You can specify the EC2 instance type and the number of instances.

Create HTTPS endpoint

Using the endpoint configuration SageMaker provisions the SageMaker EC2 instances and all the necessary infrastructure to expose an HTTPS REST API. Your Model is now ready for inferencing.

Video: Deploy Your ML Models to Production at Scale with Amazon SageMaker

This is a 7.52 minute video from AWS by Emily Webber. The first two and a half minutes are most relevant to SageMaker endpoints. The timestamps are:

  • 0 – 2.31: SageMaker endpoint
  • 2.32 – 5.26: Example using the Blazing Text algorithm in a Jupyter Notebook.
  • 5.26 – 7.52: Pro tips

Autoscaling

The SageMaker endpoint you have just created can also scale depending on the workload. So as more production inferencing requests are received SageMaker will provision more SageMaker EC2 instances and deploy more copies of the Model to process the requests. The parameters for autoscaling, such as the maximum number of instances can be configured. Scaling can be controlled by using CloudWatch metrics to trigger the scaling action.

Monitoring

Video: Inawisdom: Machine Learning and Automated Model Retraining with SageMaker

This is a 7.56 minute video from AWS with Robin Meehan and Shafreen Sayyed.

Why monitor models?

The performance of Models can change overtime due to:

  • a change in the data – data drift
  • a change in the context of the target variable – concept drift
  • unexpected factors

Monitoring is cumbersome, but critical. The outcome is model re-training.

What are concept and data drift?

When data changes in general, this problem is called data drift whereas the changes in the context of the target variable are called concept drift.

With Concept Drift the statistical properties of the target variable change over time in ways that could not be predicted when the model was designed. Data Drift occurs when the data has changed overtime, however the statistical properties of the target variable are unchanged. This leads to degrading performance of the model that may be corrected by re-training the model. For Concept Drift simple re-training may be insufficient and may require revisiting the preparation steps such as Feature Engineering.

What should be monitored?

Defining what needs to be monitored is still an evolving area in Data Science. The following monitoring techniques were mentioned in the AWS Exam Readiness course.

Data change tracking

Tracking the input data can be used to identify when the data has changed significantly when compared to the data on which the Model was trained.

Error rate

An increase in the error rate can indicate Data Drift and Concept Drift. Errors in the input data and output inferences have to be defined and monitored.

Error class proportion

Whilst the error rate may be unchanged, what about the composition of the errors? Data and Concept drift can be detected by comparing the change of the proportion error classes.

How are Models monitored?

The key AWS services for monitoring Machine Learning Models are CloudWatch and CloudTrail. SageMaker also has a built in feature to assist on Monitoring: Amazon SageMaker Model Monitor.

CloudWatch

Four important features of Amazon CloudWatch are: Logs, Metrics, Events and Alarms.

CloudWatch Logs

CloudWatch Log Groups enable you to monitor, store and access logs ordered by time. Sources generate Log Events which are presented as a Log Stream in time order. A Log Stream contains Log Events that originated from the same Source. Log Streams are grouped together into Log Groups. This enables Log Streams to share the same:

  • Retention
  • Monitoring
  • Access control

Sources can be:

  • EC2, using the CloudWatch log agent
  • AWS Services, some will need a CloudWatch Logs resource policy

CloudWatch Metrics

CloudWatch Metrics are created from Metric Filters that filter Log Events in a Log Stream. CloudWatch Metrics can be displayed in a graph, or used by a CloudWatch Alarm.

CloudWatch Events and EventBridge

CloudWatch Events are now being superseded by Amazon EventBridge. Currently the functionality is the same, so you can use either. CloudWatch Events comprise of three features:

  1. Events are produced by Supported Services. Unsupported Services can be tracked via CloudTrail.
  2. Rules route events from Sources to Targets.
  3. Targets process incoming events which are formatted as JSON documents.

CloudWatch Alarms

There are two types of CloudWatch Alarms:

  • Metric Alarm which watches CloudWatch Metrics
  • Composite Alarm which watches other Alarms

CloudWatch Alarms are triggered by considering three properties:

  1. Period is the length of time for a data point
  2. Evaluation periods are the number of periods to check to see if the alarm criteria are met.
  3. Data points to alarm are the number of data points that must be in error to trigger an alarm.
Missing data

Alarms can be configured to treat missing data in one of four ways:

  1. Not breaching – good
  2. Breaching – bad
  3. Ignore – current state maintained
  4. Missing – Insufficient Data state adopted
Maths expressions

CloudWatch Alarms are able to handle mathematical expressions with up to ten metrics. If you need to test more Metrics you could try a composite alarm (an alarm of alarms) or use a Lambda.

Percentile and low data alarms

When percentile alarms do not receive enough data the default action can  be set to evaluate anyway, or wait until enough good data arrives before evaluating.

CloudTrail

CloudTrail collects data from AWS API calls which can then be stored in an S3 bucket for analysis. CloudTrail can also be monitored from CloudWatch by configuring the Trail to send data to a CloudWatch Log.

SageMaker Model Monitor

By using SageMaker Model Monitor you can automatically monitor your model in production and be alerted when Data Drift or Concept Drift occurs. The alerting feature can also be used to automatically start retraining your model. Data Drift and Concept Drift are detected by monitoring the quality of the model based on the inputs to an ML model, which are features or independent variables and the outputs of the model which are dependent variables.

What does SageMaker Model Monitor do?

SageMaker Model Monitor emits per-feature metrics which are passed to CloudWatch by monitoring models in production and detecting errors. This means you do not have to develop yourself.

What types of drift are monitored by Model Monitor?

There are four types of drift monitored by Model Monitor:

  1. Data quality
  2. Model quality
  3. Model bias (SageMaker Clarify)
  4. Feature attribution drift (SageMaker Clarify)

Model Monitor compares the production data quality to the training data by using rules to detect Data Drift and alerting you when this happens.

Model quality is measured by comparing the models predictions with the actual labels the model has been trained to predict. This is done by merging production data with the actual labels and measuring the predictions.

Amazon SageMaker Clarify

Amazon SageMaker Clarify provides bias monitoring. Model Bias occurs when training and production data differ. This difference may be temporary, or a permanent shift in the real world data. 

Amazon SageMaker Clarify provides feature attribution drift monitoring. Feature attribution drift is where data drift causes changes in the attribution of features. When this drift breaches a configured threshold alerts are raised via CloudWatch.

Introducing Amazon SageMaker Clarify, part 1 – Bias detection – AWS re:Invent 2020

Model deployment approaches

When deploying to production there are two risks:

  1. There may be a disruption in the service which means that users cannot use the Model for a while whilst resources are being provisioned.
  2. The new Model may not perform as well as the one that it is replacing.

These risks can be mitigated by using Blue/Green and Canary deployment strategies. Both of these strategies make use of built in SageMaker endpoint features. In a SageMaker endpoint you can deploy more than one production variant and specify what proportion of traffic is routed to each production variant. The proportions can be changed instantly.

The AWS White paper Machine Learning Lens has a section on Model Deployment Approaches on pages 29 – 34: 

Blue Green

Blue Green deployments have two phases. In the first Phase the new variant is deployed in an identical environment to the production variant. It is then fed synthetic data and monitoring metrics are checked. In the second phase live traffic is switched to the new variant and the metrics are compared with those produced by the current production variant. If a problem is identified all live traffic is switched back to the production variant, otherwise the new variant becomes the new production variant and the old one is removed.

This infographic describes a blue green deployment using Amazon SageMaker
Add this revision card to your Pinterest account

Canary

A Canary release is a very risk averse deployment strategy. It involves directing a small proportion of the live traffic to the new production variant and checking that everything works as expected. The proportion of live traffic is gradually increased until all the traffic is being directed to the new production variant at which point the previous version can be removed. If any issues are identified live traffic can be switched back to the original production variant.

Summary

If you are progressing through the exam structure in order, this is the last sub-domain to be studied. Congratulations on finishing!

This Study Guide has described what is needed to deploy and operationalize a Machine Learning Model in a production environment. Many of the features and capabilities draw on existing Software Management and Engineering that is used for non-Machine Learning systems. Machine Learning Models in production have their own emphasis and priorities which the emerging role of ML Ops seeks to satisfy.

Notes:

  • The AWS Machine Learning Exam Readiness course refers to API versioning. I have taken this to mean SageMaker endpoints.
  • The AWS Machine Learning exam readiness course can usually be relied upon to identify areas of a broad subject that need to be focused on for the exam. Unfortunately for this sub-domain it provides little direction and lists a vast, almost random, list of software management and engineering terms.
  • Using Lambda with Amazon SageMaker has not been included yet.
Credits

Photo by Bill Jelen on Unsplash


AWS Certified Machine Learning Study Guide: Specialty (MLS-C01) Exam

This study guide provides the domain-by-domain specific knowledge you need to build, train, tune, and deploy machine learning models with the AWS Cloud. The online resources that accompany this Study Guide include practice exams and assessments, electronic flashcards, and supplementary online resources. It is available in both paper and kindle version for immediate access. (Vist Amazon books)


10 questions and answers

13
Created on By Michael Stainsbury

4.4 Deploy and operationalize machine learning solutions (full)

10 test style questions that cover subdomain 4.4, Deploy and operationalize machine learning solutions

1 / 10

Why can the performance of Models can change over time?

2 / 10

What Amazon SageMaker service can be alerted when Data Drift or Concept Drift occurs?

3 / 10

With <–?–> Drift the statistical properties of the target variable change over time in ways that could not be predicted when the model was designed.

4 / 10

A <–?–> release is a very risk averse deployment strategy. It involves directing a small proportion of the live traffic to the new production variant and checking that everything works as expected.

5 / 10

What are the types of drift monitored by Model Monitor?

6 / 10

What are the AWS services that provide SageMaker with logging and monitoring in the AWS environment?

7 / 10

What are the steps to creating a SageMaker Endpoint?

8 / 10

What is the first phase of Blue Green deployments?

The new variant is deployed in an identical environment to the production variant. It is then fed synthetic data.

9 / 10

More than one model can be hosted in a SageMaker Endpoint.

10 / 10

<–?–> is the AWS git based service for version control. SageMaker model artifacts can be stored in the <–?–> repository.

Your score is

The average score is 65%

0%


Amazon Study Guide for the AWS Machine Learning Speciality exam
Reviews
Amazon Study Guide review – AWS Certified Machine Learning Specialty

This Amazon Study Guide review is a review of the official Amazon study guide to accompany the exam. The study guide provides the domain-by-domain specific knowledge you need to build, train, tune, and deploy machine learning models with the AWS Cloud. The online resources that accompany this Study Guide include practice exams and assessments, electronic…

Pluralsight AWS Certified Machine Learning web page screen shot
Reviews
Pluralsight review – AWS Certified Machine Learning Specialty

Contains affiliate links. If you go to Whizlab’s website and make a purchase I may receive a small payment. The purchase price to you will be unchanged. Thank you for your support. The AWS Certified Machine Learning Specialty learning path from Pluralsight has six high quality video courses taught by expert instructors. Two are introductory…


Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *