CCNA AI Auditing Tools And Techniques Questions

Access control, encryption at rest, and data provenance are standard security controls for AI data pipelines.

The 'Counterfactuals' component in the Azure Responsible AI dashboard allows auditors to see how input changes impact predictions.

A central, versioned store (like MLflow or Azure Model Registry) for managing trained models.

Epsilon is the privacy budget; lower values provide stronger privacy guarantees but potentially lower utility.

The inference ID links the specific request to its model output, facilitating trace-ability during audits.

LIME creates a local surrogate model to explain individual predictions, helping the auditor understand why a specific decision was made.

AWS SageMaker Model Monitor and Google Cloud Vertex AI Model Monitoring are industry standard tools for drift detection.

Lack of version control or undocumented model changes are classic audit findings in AI systems.

Documenting the issue, notifying stakeholders, and recommending mitigation (re-sampling or retraining) are the appropriate steps.

Adversarial vulnerability, data drift, and bias are three primary risks requiring documentation in an AI audit report.

When data distribution changes (Concept Drift), the model's accuracy typically degrades over time.

Global importance explains the model overall, but individual predictions (local) may be driven by different factors.

Hashing files and maintaining lineage logs are standard technical methods to verify data provenance.

Logs, configuration files, and model artifacts are the three core pieces of evidence needed for an audit.

Vertex AI ML Metadata tracks the lineage of artifacts and executions within the pipeline.

SageMaker Clarify provides specific bias detection capabilities for both training data and model predictions.

The Model Card is a standardized document that lists intended use, limitations, and performance metrics.

PSI measures how much the distribution of a variable has changed between the training set and the production dataset.

Cross-validation using temporal splits (rather than random splits) prevents training on future information.

Accuracy, fairness metrics, and latency/throughput are standard KPIs for production readiness.

Data drift and prediction accuracy are the two most critical metrics for a model monitor.

Version control information and the exact dataset version are essential for reproducing the audit results.

Class imbalance can cause the model to be biased toward the majority class; the auditor should check for techniques like re-sampling or cost-sensitive learning.

Analyzing data provenance and lineage ensures that only trusted sources were used for the training set.

It should contain the model binary, the environment definition (requirements.txt), and the experiment metadata (the 'Run' ID).

Precision and Recall are fundamental metrics for assessing the quality of classification models.

Shadow mode allows running the new model in production alongside the old one to compare outputs without affecting users.

CleverHans is a library for benchmarking the robustness of neural networks against adversarial examples.

SHAP (SHapley Additive exPlanations) is used to quantify the contribution of each feature to a specific prediction, aiding interpretability.

PII scrubbers remove or anonymize Personally Identifiable Information from training sets to ensure privacy compliance.

Red Teaming involves manual and automated attempts to bypass safety filters and trigger malicious outputs.

Red teaming with adversarial prompts (jailbreaks) is the standard method for testing safety filter resilience.

Faithfulness measures whether the explanation accurately reflects the model's internal decision process rather than just being a plausible simplification.

The condition, the root cause, and the recommendation are essential components of an audit finding.

Model Cards and technical documentation provide the necessary governance and implementation context for an auditor.

Kubeflow Pipelines (KFP) ensure reproducibility through containerized steps and pipeline definitions stored in YAML.

Adversarial testing involves small perturbations to inputs to identify instability or security weaknesses.

Containers (e.g., Docker) encapsulate code, libraries, and dependencies, ensuring that the model runs the same way in audit as it did in training.

The F1-score balances precision and recall, providing a single metric for classification performance.

Apache Airflow provides DAG (Directed Acyclic Graph) visualization, which auditors use to map data flow and lineage.

HITL ensures that sensitive or critical decisions made by AI are reviewed by human experts.

Immutable model repositories and versioning control (e.g., Git LFS, MLflow) ensure the exact state can be audited.

Watson OpenScale features a 'Fairness' monitor that specifically tracks Disparate Impact ratios.

The MLflow Run object stores the code version (git commit), parameters, and metrics for a training session.

The first step in Aequitas is to define the bias metrics (e.g., False Positive Rate Parity) relevant to the context.

LIME, SHAP, and Partial Dependence Plots (PDPs) are model-agnostic methods that work with any ML model.

Stratified sampling and fairness-aware loss functions are common techniques to address and audit bias.

Assertions (e.g., in Great Expectations) check if data meets predefined schemas and constraints before ingestion.

Validation confirms that the model performs as expected against independent, unseen data.

Training logs, environment requirements (dependencies), and the training script are essential to reconstruct the training process.

Data Scientists and Model Owners have the necessary technical and business context for the AI system.

Model Inversion is an attack where an adversary reconstructs training data from model outputs.

Demographic Parity requires that the probability of a positive outcome be equal across different demographic groups.

Target leakage and future data leakage are primary types of data leakage in AI pipelines.