Lab 08 • Responsible AI & LLM Evaluation
AI Evaluation Casebook & Faithfulness Metrics
Annotate 24 authored responses from two systems against reference facts. Undefined ratios (no positive predictions/actuals) display N/A rather than 0%.
System A — Confusion Matrix
TP: 0
FP: 0
FN: 0
TN: 0
Precision: N/ARecall: N/ASkipped: 24
System B — Confusion Matrix
TP: 0
FP: 0
FN: 0
TN: 0
Precision: N/ARecall: N/ASkipped: 24
Annotation Grid (24 cases)
C1Return window for electronics?
A:B:
C2Refurb laptop warranty?
A:B:
C3Enterprise 24/7 phone support?
A:B:
C4BigQuery export supported?
A:B:
C5EU data hosting location?
A:B:
C6Max file upload size?
A:B:
C7HIPAA BAA available?
A:B:
C8Seat billing during mid-cycle expansion?
A:B:
C9Free accounts get custom subdomains?
A:B:
C10Is MFA mandatory?
A:B:
C11REST API rate limit?
A:B:
C12Can deleted workspaces be restored?
A:B:
C13What export formats are supported?
A:B:
C14Password reset link rules?
A:B:
C15SSO providers supported?
A:B:
C16Backup frequency and retention?
A:B:
C17Team roles and billing access?
A:B:
C18Data retention after cancellation?
A:B:
C19Custom domain SSL cost?
A:B:
C20Audit log retention by tier?
A:B:
C21Current API version status?
A:B:
C22Support SLA by tier?
A:B:
C23Data encryption approach?
A:B:
C24Free trial length and extensions?
A:B:
Inspection: C1
Return window for electronics?
Returnable within 30 days with receipt.
Returnable within 30 days with a receipt.
Electronics are non-returnable once opened.
Error Categories by System
Precision / Recall / F1 / Coverage (%)
Model Governance & Deployment Gate
1. Validity
Inputs Supported
All configuration parameters are valid.
2. Correctness
Objective: Demonstrated
Ready for verification run
3. Constraints
Within Limits
Budget, SLA, and capacity limits respected.
4. Trade-offs
Analysis
Inspect scenario parameters to evaluate trade-offs.