MLA-C01 Deployment and Orchestration of ML Workflows — All Questions With Answers

Refer to the exhibit.

Error log from SageMaker endpoint creation:
```
ResourceLimitExceeded: An error occurred (ResourceLimitExceeded) when calling the CreateEndpointConfig operation: The account-level service limit for 'ml.c5.large for real-time endpoints' is 5. You have requested 6 instances. Please use AWS Service Quotas to request an increase.
```

Refer to the exhibit.

SageMaker Model Monitor schedule configuration:
```
{
  "ScheduleConfig": {
    "ScheduleExpression": "cron(0 * * * ? *)",
    "DataAnalysisStartTime": "2023-01-01T00:00:00Z",
    "DataAnalysisEndTime": "2023-01-01T23:59:00Z"
  },
  "JobDefinition": {
    "Environment": {
      "output_path": "s3://my-bucket/reports/"
    }
  },
  "MonitoringType": "DataQuality"
}
```

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sagemaker:DescribeEndpoint",
        "sagemaker:ListEndpoints"
      ],
      "Resource": "*"
    }
  ]
}

[ERROR] 2024-03-15 10:23:45,234 - botocore.exception.ConnectTimeoutError: Connect timeout on endpoint URL: "https://database.example.com:5432"

TrainingStep(
    name="TrainModel",
    step_args=train_args,
    depends_on=[tuning_step]
)
tuning_step = TuningStep(...) # produces multiple artifacts

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sagemaker:CreateModel",
      "Resource": "arn:aws:sagemaker:us-east-1:123456789012:model/*"
    },
    {
      "Effect": "Deny",
      "Action": "sagemaker:CreateEndpointConfig",
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "sagemaker:VpcSubnets": "subnet-0123456789abcdef0"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": "sagemaker:CreateEndpoint",
      "Resource": "*"
    }
  ]
}

{
  "PipelineExperimentConfig": {
    "ExperimentName": "my-experiment",
    "TrialName": "my-trial"
  },
  "Parameters": {
    "TrainingInstanceType": "ml.m5.large",
    "TrainingInstanceCount": 2,
    "MaxRuntimeInSeconds": 86400
  },
  "Steps": [
    {
      "Name": "Preprocess",
      "Type": "Processing",
      "ProcessingJobName": "preprocess-job",
      "ProcessingResources": {
        "ClusterConfig": {
          "InstanceCount": 1,
          "InstanceType": "ml.m5.large",
          "VolumeSizeInGB": 30
        }
      }
    },
    {
      "Name": "Train",
      "Type": "Training",
      "TrainingJobName": "train-job",
      "AlgorithmSpecification": {
        "TrainingImage": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-algo:latest",
        "TrainingInputMode": "File"
      },
      "ResourceConfig": {
        "InstanceCount": 2,
        "InstanceType": "ml.m5.large",
        "VolumeSizeInGB": 30
      }
    }
  ]
}

{
  "MonitoringScheduleName": "model-quality-monitor",
  "EndpointName": "my-endpoint",
  "MonitoringType": "DataQuality",
  "MonitoringScheduleConfig": {
    "ScheduleExpression": "cron(0 * * * ? *)",
    "MonitoringJobDefinition": {
      "BaselineConfig": {
        "BaseliningJobName": "baseline-job",
        "ConstraintsResource": {
          "S3Uri": "s3://my-bucket/baseline/constraints.json"
        },
        "StatisticsResource": {
          "S3Uri": "s3://my-bucket/baseline/statistics.json"
        }
      },
      "MonitoringInputs": [
        {
          "EndpointInput": {
            "EndpointName": "my-endpoint",
            "LocalPath": "/opt/ml/processing/input/endpoint",
            "S3DataDistributionType": "FullyReplicated",
            "S3InputMode": "File"
          }
        }
      ],
      "MonitoringOutputConfig": {
        "MonitoringOutputs": [
          {
            "S3Output": {
              "S3Uri": "s3://my-bucket/monitor-output",
              "LocalPath": "/opt/ml/processing/output",
              "S3UploadMode": "Continuous"
            }
          }
        ]
      },
      "MonitoringResources": {
        "ClusterConfig": {
          "InstanceCount": 1,
          "InstanceType": "ml.m5.large",
          "VolumeSizeInGB": 20
        }
      },
      "RoleArn": "arn:aws:iam::123456789012:role/SageMakerMonitorRole"
    }
  }
}

Refer to the exhibit.

{
  "EndpointName": "my-endpoint",
  "EndpointConfigName": "my-endpoint-config-v1",
  "ProductionVariants": [
    {
      "VariantName": "v1",
      "DeployedImages": [
        {
          "SpecifiedImage": "123456789012.dkr.ecr.us-west-2.amazonaws.com/my-model:1",
          "ResolvedImage": "123456789012.dkr.ecr.us-west-2.amazonaws.com/my-model:1@sha256:abc123"
        }
      ],
      "CurrentWeight": 1.0,
      "DesiredWeight": 1.0,
      "CurrentInstanceCount": 2,
      "DesiredInstanceCount": 2
    }
  ],
  "EndpointStatus": "InService"
}

Refer to the exhibit.

[ERROR] 2022-12-01 10:15:30,123 – model_server – ModelLoadFailed: Unable to load model from /opt/ml/model. Parsed error: FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/model/classes.txt'

This log is from a SageMaker endpoint instance. The model was packaged as a tar.gz containing model.pth, classes.txt, and inference.py. The Docker container uses the SageMaker inference toolkit.

Refer to the exhibit.

aws sagemaker create-training-job \
    --training-job-name my-training-job \
    --algorithm-specification 'TrainingImage=123456789012.dkr.ecr.us-west-2.amazonaws.com/my-custom-training:latest,TrainingInputMode=File' \
    --role-arn arn:aws:iam::123456789012:role/SageMakerExecutionRole \
    --input-data-config '[{"ChannelName":"train","DataSource":{"S3DataSource":{"S3Uri":"s3://my-bucket/train/","S3DataType":"S3Prefix"}},"ContentType":"text/csv"}]' \
    --output-data-config '{"S3OutputPath":"s3://my-bucket/output/"}' \
    --resource-config '{"InstanceType":"ml.m5.large","InstanceCount":1,"VolumeSizeInGB":30}' \
    --vpc-config '{"SecurityGroupIds":["sg-12345678"],"Subnets":["subnet-12345678"]}'

Refer to the exhibit.
```
Pipeline definition snippet:
{
  "Steps": [
    {
      "Name": "Preprocess",
      "Type": "Processing",
      "Arguments": {
        "ProcessingResources": {
          "ClusterConfig": {
            "InstanceCount": 1,
            "InstanceType": "ml.m5.large",
            "VolumeSizeInGB": 10
          }
        }
      }
    },
    {
      "Name": "Train",
      "Type": "Training",
      "DependsOn": ["Preprocess"],
      "Arguments": {
        "AlgorithmSpecification": {
          "TrainingImage": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-training:latest",
          "TrainingInputMode": "File"
        },
        "ResourceConfig": {
          "InstanceCount": 2,
          "InstanceType": "ml.p3.2xlarge",
          "VolumeSizeInGB": 30
        }
      }
    }
  ]
}
```

Refer to the exhibit.
```
CloudWatch Logs from a SageMaker endpoint:
[ERROR] Runtime.ImportModuleError: Unable to import module 'inference': No module named 'transformers'
```

Refer to the exhibit.

{
  "ModelName": "my-model",
  "PrimaryContainer": {
    "Image": "763104351884.dkr.ecr.us-west-2.amazonaws.com/tensorflow-inference:2.11-cpu",
    "Environment": {
      "SAGEMAKER_PROGRAM": "inference.py",
      "SAGEMAKER_SUBMIT_DIRECTORY": "/opt/ml/model/code"
    }
  },
  "ExecutionRoleArn": "arn:aws:iam::123456789012:role/SageMakerRole"
}