Testing Validation And Troubleshooting practice questions

A NAT Gateway with an EIP.

Why wrong: While this enables access, it's not the most secure or recommended approach compared to Interface Endpoints.

A

Why wrong: A

An Internet Gateway attached to the VPC.

Why wrong: Internet Gateways provide public access, which may violate security compliance.

A public subnet in the VPC configuration.

Why wrong: VPC endpoints do not require public subnets.

A

Why wrong: A

Disable the Guardrail and re-run the test to see if performance improves.

Why wrong: This would expose more PII, not help troubleshoot.

Route all traffic through a WAF to block PII patterns.

Why wrong: WAF is for HTTP/HTTPS filtering, not internal PII content within an LLM inference payload.

Increase the PII sensitivity threshold in the IAM policy.

Why wrong: Guardrail settings are configured in the Guardrail resource, not IAM.

Use AWS Step Functions to iterate through a test dataset and store results in S3.

Why wrong: This does not provide a built-in comparison interface for model performance.

Write a custom Python script to invoke both models and print the output to stdout.

Why wrong: While possible, it is not the most efficient or recommended AWS-managed way.

A

Why wrong: A

Deploy two separate endpoints and use Amazon CloudWatch dashboard for visual comparison.

Why wrong: CloudWatch is for monitoring metrics, not for evaluating model text quality.

A

Why wrong: A

Latency

Why wrong: This is a measurement of the overall duration, but monitoring it alone doesn't pinpoint the cause as context size.

Invocations

Why wrong: This measures the number of requests, not the size of the prompt.

OutputTokenCount

Why wrong: This measures the generated response, which is a result of the processing but not the cause of latency due to context.

Amazon SageMaker Model Monitor.

Why wrong: SageMaker Model Monitor is for tabular/ML models, not specifically for LLM text toxicity evaluation.

A

Why wrong: A

AWS Glue to sanitize data.

Why wrong: Glue is for data transformation, not model output evaluation.

AWS Lambda to call the model and manually inspect outputs.

Why wrong: Manual inspection is not automated validation.

Reduce the number of examples in the few-shot prompt.

Why wrong: Few-shot examples generally improve instruction adherence, not degrade it.

Change the model to a smaller parameter version.

Why wrong: Smaller models typically have worse instruction following capabilities.

A

Why wrong: A

Update the model's training data using a fine-tuning job.

Why wrong: Fine-tuning is a heavy-handed approach and not the first step for prompt instruction following issues.

Increase the top-p value.

Why wrong: Top-p also increases randomness.

A

Why wrong: A

Increase the maximum token length.

Why wrong: This only limits the output size, not the randomness.

Enable streaming in the API call.

Why wrong: Streaming changes the delivery method, not the generation behavior.

The IAM policy duration.

Why wrong: IAM policies do not have duration limits on individual requests.

The Bedrock base model throughput limit.

Why wrong: This causes 429 errors, not timeouts.

The VPC route table propagation.

Why wrong: This affects connectivity, not specific request timeouts.

A

Why wrong: A

AWS X-Ray service map.

Why wrong: X-Ray shows service latency and connectivity, not the internal decision logic of the model.

Amazon CloudWatch Logs for the Lambda function.

Why wrong: This shows the function's execution, but not the agent's logic for invoking it.

The Agent's IAM execution role policy.

Why wrong: An incorrect role would cause permission errors, not logical reasoning failures.

A

Why wrong: A

A

Why wrong: A

AWS WAF.

Why wrong: WAF does not natively inspect text payloads for PII.

Amazon Comprehend.

Why wrong: Comprehend can detect PII, but it requires manual integration into the pipeline.

Amazon Macie.

Why wrong: Macie is for S3 bucket data discovery, not real-time prompt filtering.

Deploy the model inside a VPC.

Why wrong: VPC deployment affects network isolation, not training data policy.

A

Why wrong: A

Enable data logging in CloudWatch.

Why wrong: Logging has no impact on model training practices.

Use a Provisioned Throughput model.

Why wrong: Provisioned Throughput is for capacity planning, not data privacy settings.

AWS Cloud9.

Why wrong: Cloud9 is an IDE, not a model prompt testing tool.

A

Why wrong: A

AWS Step Functions.

Why wrong: Step Functions is for workflow orchestration.

Amazon SageMaker Studio.

Why wrong: While it can be used, it is not specifically focused on prompt iteration for Bedrock models.

A

Why wrong: A

Use a smaller model for the agent.

Why wrong: Smaller models are generally worse at following complex JSON structures.

Increase the model's temperature.

Why wrong: Increasing temperature makes output less structured.

Add 'You must output JSON' to the system prompt.

Why wrong: System prompts are suggestions; a schema constraint is a technical enforcement.

The VPC endpoint policy is blocking traffic to the Bedrock runtime service.

Why wrong: VPC endpoint issues typically result in connectivity timeouts or 403 errors.

The Lambda function timeout is set too low for the model response time.

Why wrong: This would result in a 504 Gateway Timeout error.

The IAM role is missing the bedrock:InvokeModelWithResponseStream permission.

Why wrong: This would result in a 403 Forbidden error, not a 429.

Increase the temperature setting to 1.0 to encourage more creative responses.

Why wrong: Higher temperature increases variability and likely increases hallucinations.

Use Amazon CloudWatch logs to calculate the latency of every inference call.

Why wrong: Latency does not measure accuracy or hallucination metrics.

Run a standard BLEU score evaluation against the model's training data.

Why wrong: BLEU is insufficient for semantic reasoning and context-grounded tasks.

Model Training Loss

Why wrong: This is for model fine-tuning performance, not RAG evaluation.

Inference Latency

Why wrong: This is a system performance metric, not a RAG-specific evaluation metric.

AWS X-Ray service map.

Why wrong: While useful, it is not a direct way to monitor core model usage statistics.

Amazon S3 bucket logging.

Why wrong: Not a primary way to monitor model-level service usage.

Restart the Bedrock service.

Why wrong: Users cannot restart managed services.

Increase the model's temperature to 1.0.

Why wrong: This is not a diagnostic step for logical failures.

Re-train the underlying base model.

Why wrong: Re-training is not a valid troubleshooting step for agent logic.

Increase the temperature.

Why wrong: This affects randomness, not verbosity.

Enable Guardrails to truncate output.

Why wrong: Guardrails are for content safety, not length control.

Use a different model family.

Why wrong: While different models have different tendencies, parameters and prompting are the primary tools.

Insecure API endpoint access.

Why wrong: While this is a general security issue, it is not specific to the LLM model itself.

Excessive API latency.

Why wrong: Latency is an availability issue, not a direct security threat.