NEW
Font size
WorksheetsMLOps Workshop Assessment – AWS SageMaker MLOps Pipeline
Total questions: 29
Worksheet time: 15mins
Architecture & Pipeline Design: Your medical AI team needs to implement an automated retraining pipeline that triggers when new hospital data arrives. Which AWS services combination provides the most robust event-driven MLOps architecture?
CloudWatch Events → Lambda → SageMaker Training → Manual Model Registry
S3 Event Notifications → Lambda → SageMaker Pipeline → Model Registry → Conditional Deployment
API Gateway → EC2 → Manual Training → S3 Storage
SageMaker Studio → Manual Triggers → Direct Endpoint Deployment
Model Performance & Quality Gates: Your cancer detection model achieves 94% accuracy, but your quality gate is set to 95%. What should happen in an automated MLOps pipeline?
Deploy the model since 94% is close to 95%
The pipeline should automatically reject the model
Manually override the quality gate for this deployment
Deploy only to development environment
Training Job Troubleshooting: Your SageMaker training job fails with "AlgorithmError: ExecuteUserScriptError: ExitCode 2". The CloudWatch logs show "ModuleNotFoundError: No module named 'tensorflow'". What is the most likely root cause?
Insufficient IAM permissions
Wrong instance type selected
Training script dependencies not properly configured
S3 bucket permissions issue
Medical AI Compliance & Governance: For healthcare applications, which practice is MOST critical for regulatory compliance?
High accuracy metrics and fast inference times
Model interpretability and audit logs
Cost optimization and resource scaling
Automated deployment without human oversight
Model Registry & Versioning: You have deployed cancer-detection-model-v1.2 to production. A new model version v1.3 achieves higher accuracy but shows concerning behavior on some test cases. What's the best MLOps strategy?
Immediately replace v1.2 with v1.3 due to higher accuracy
Keep v1.2 in production and investigate v1.3 before deployment
Deploy both models and let users choose
Deploy v1.3 immediately to get the accuracy benefits
Data Pipeline & Storage: Your medical imaging data is stored in S3 with the structure: "s3://bucket/data/train/benign/" and "s3://bucket/data/train/malignant/". What's the recommended approach for SageMaker training?
Download all images to local storage before training
Use S3 as direct input with SageMaker's built-in data loading
Convert images to a different format first
Stream images through API Gateway during training
Endpoint Configuration & Scaling: Your cancer detection endpoint needs to handle varying traffic throughout the day. What's the most appropriate scaling approach?
Use a fixed large instance that can handle peak traffic
Use auto-scaling to adjust capacity based on demand
Manually scale up and down based on time of day
Use the smallest instance to minimize costs
MLOps Monitoring: Which metric is MOST important to monitor for a production medical AI model?
Cost per prediction
Model accuracy over time
Server CPU utilization
Number of API calls per day
Security & IAM Troubleshooting: Your Lambda function fails to trigger SageMaker training with error: "User is not authorized to perform: sagemaker:CreateTrainingJob". What's the most likely issue?
The Lambda needs SageMaker permissions in its execution role
S3 bucket policy needs to allow Lambda access
SageMaker service is down
The training data is corrupted
Model Evaluation & Testing: During model evaluation, you discover your cancer detection model performs differently across different hospitals. What should you do first?
Deploy the model anyway since overall accuracy is good
Investigate the data differences between hospitals
Train separate models for each hospital
Ignore the differences and focus on overall performance
Pipeline Orchestration: Your MLOps pipeline includes: Data Validation → Training → Evaluation → Deployment. The evaluation step fails due to missing test data. How should the pipeline handle this?
Skip evaluation and proceed to deployment
Stop the pipeline and alert the team
Use training data for evaluation instead
Deploy without evaluation this time
Cost Optimization: Your training jobs are taking longer than expected and costing more than budgeted. What's the first thing you should investigate?
Switch to cheaper instance types
Reduce the dataset size
Check if the training script is efficient
Train less frequently
Real-time Inference: Your cancer detection API needs to respond quickly for real-time diagnosis support. The current response time is too slow. What should you try first?
Use a faster instance type for the endpoint
Reduce the model size
Cache previous predictions
Use batch processing instead
Model Rollback: Your production cancer detection model starts showing problems after deployment. You need to rollback to the previous version quickly. What's the fastest approach?
Retrain the previous model version
Update the endpoint to use the previous model
Delete everything and start over
Wait for the issues to resolve themselves
MLOps Best Practices: You're setting up a complete MLOps solution for medical image classification. Which component is most essential for getting started?
Advanced experiment tracking with multiple frameworks
Automated model deployment with proper testing
Complex multi-region disaster recovery
Integration with different hospital systems
Model Performance & Quality Gates: In a medical AI pipeline with safety thresholds, what should happen when a model fails to meet the defined threshold to prevent deployment of substandard models?
Automatically reject the model
Deploy the model with enhanced monitoring
Proceed to manual review before deployment
Delay deployment but keep the model in production
Training Job Troubleshooting: A TensorFlow training job fails with an import error indicating missing packages. What is the most likely root cause to address first?
Insufficient GPU memory on the instance
Training script dependencies not properly configured
Incorrect S3 bucket permissions
Model hyperparameters causing overfitting
Medical AI Compliance & Governance: Which capability is most critical to satisfy healthcare regulations beyond performance metrics?
Batch inference throughput reports
Model interpretability and audit logs
Continuous hyperparameter tuning
Multi-region endpoint failover
Model Registry & Versioning: A medical AI team sees concerning behavior in candidate version v1.3 while v1.2 is stable in production. What should they do before deploying v1.3?
Promote v1.3 immediately to production
Keep v1.2 in production and investigate v1.3 before deployment
Retire v1.2 and run only v1.3 in staging
Rollback v1.2 to v1.1 for safety
Data Pipeline & Storage: For training data access in SageMaker, what is the recommended approach to optimize loading and caching?
Use a self-managed NFS mounted on the training containers
Use S3 as direct input with SageMaker’s built-in data loading
Copy data to EBS volumes pre-training
Stream data from a relational database via JDBC
Endpoint Configuration & Scaling: To improve cost efficiency and performance based on actual demand patterns, what scaling approach should be used for the endpoint?
Provision a fixed number of instances based on peak load
Use auto-scaling to adjust capacity based on demand
Manually scale instances daily
Disable scaling and rely on caching
MLOps Monitoring: In medical applications, which metric is most critical to monitor in production ML systems to detect model drift and degradation?
CPU utilization of endpoints
Model accuracy over time
Number of API requests per minute
Average inference latency only
Security & IAM Troubleshooting: A Lambda function needs to create SageMaker training jobs but fails with AccessDenied. What is the most likely permission issue?
The Lambda needs SageMaker permissions in its execution role
The Lambda must run in a VPC
The training script lacks S3 read access
The endpoint does not allow public access
Model Evaluation & Testing: Performance differs across hospitals during validation. What is the priority action before deployment?
Deploy and monitor in production
Investigate the data differences between hospitals
Increase training epochs to improve accuracy
Reduce the validation dataset to speed evaluation
Pipeline Orchestration: The evaluation step cannot find required data. What should an MLOps pipeline do to ensure safe operation?
Skip the evaluation step and continue
Stop the pipeline and alert the team
Retry silently without notification
Proceed to deploy with the last successful model
Cost Optimization: Training is slow and expensive without clear infrastructure bottlenecks. What should be checked first?
Move training to larger GPU instances
Switch storage from S3 to EFS
Check if the training script is efficient
Increase batch size dramatically
Real-time Inference: Immediate latency improvements are needed for an endpoint. What is the most straightforward action?
Use a faster instance type for the endpoint
Retrain the model with quantization
Enable multi-AZ deployment
Add asynchronous inference
Model Rollback: A newly deployed model underperforms. What is the fastest way to restore the previous stable version?
Recreate all endpoint infrastructure from scratch
Update the endpoint to use the previous model
Roll back the training dataset
Disable auto-scaling temporarily
MLOps Best Practices: For reliable releases, which foundational capability should be implemented first?
Advanced feature engineering pipeline
Automated model deployment with proper testing
Canary traffic shifting across regions
Real-time drift remediation
