The DriftRail knowledge base
Understand the system.
Know what to watch.
Field notes on observing, evaluating, and protecting AI in production. Clear answers, practical checks, and the limits that matter.
The reading room / 05
Start with a question.
What is a Transformer?
Learn what transformer architecture is and how it powers modern LLMs like GPT-5, Claude 4, and Gemini. Understand the foundation of generative AI.
What is a Vector Database?
Learn what vector databases are and how they enable semantic search for RAG and AI applications. Understand embeddings and similarity search.
What is a Vision Model?
Learn what vision models are and how LLMs process images. Understand GPT-5 Vision, Claude 4 Vision, and Gemini 3 multimodal capabilities.
What is Zero-Shot Learning?
Learn what zero-shot learning is and how LLMs perform tasks without training examples. Understand the implications for AI monitoring and safety.
Trace a multi-step LLM request across services
Find broken trace continuity across a gateway, retrieval service, and model call using parent IDs and a reproducible orphan-span example.
Detect missing LLM events after a serverless response
Separate an HTTP response, collector acknowledgment, and durable event record when investigating telemetry lost around serverless completion.
Distinguish a model retry from duplicate ingestion
Count model attempts and repeated telemetry deliveries separately using request, attempt, and event identities.
Instrument tool failures without storing secrets
Keep an actionable tool-error record using an allowlist of fields instead of serializing credentials, request bodies, or arbitrary exceptions.
Track prompt versions through a production release
Resolve the prompt revision actually used by each request, including rollbacks and in-flight requests, without retaining raw prompt text.
Count missing and sampled events in an AI report
Report expected, sampled, observed, and reviewed event counts separately so a quality rate does not silently change denominators.
Trace cancellations in a streaming AI request
Follow cancellation from the user connection through model and tool work, preserving partial output and the final request state.
Build a held-out evaluation set from support tickets
Split support evaluation data by customer issue or conversation family so paraphrases and follow-ups do not leak into both tuning and held-out tests.
Calibrate an LLM judge against human review
Use an adjudicated sample and disagreement matrix to distinguish false alarms from missed failures in automated LLM grading.
Evaluate answers when source documents conflict
Test whether an assistant identifies conflicting policy evidence, applies a documented precedence rule, or escalates unresolved contradictions.
Evaluate abstention when evidence is missing
Measure unsupported answers and unnecessary abstentions separately using answerable, unanswerable, and ambiguous questions.
Compare release quality without changing the task mix
Separate within-task quality changes from changes in traffic composition using a fixed evaluation mix and explicit cohort counts.
Report small-sample LLM failure rates honestly
Show failure counts, sampling assumptions, and an illustrative uncertainty interval instead of presenting a tiny sample as a stable quality benchmark.
Investigate a valid JSON answer with wrong field values
Diagnose extraction errors by separating parseability, schema shape, domain rules, and agreement with the source document.
Find the cause of a sudden AI refusal increase
Distinguish a stricter model or policy from a change in request mix, retrieval coverage, or refusal labeling.
Diagnose a missing citation in a RAG answer
Locate whether a citation failed in corpus availability, retrieval, context assembly, generation, or rendering.
Investigate a fast first token but slow completed task
Separate first-token latency, complete-answer latency, and usable task completion before optimizing a streaming LLM workflow.
Find retry loops that inflate LLM spend
Identify repeated failures, nested retry amplification, and attempts that cannot change the outcome before reducing model prices.
Diagnose a model change after an alias update
Distinguish the model name requested by an application from the resolved revision reported for the inference.
Verify document deletion across a RAG index
Track a deleted source through chunks, vector records, search caches, and citations using lineage IDs and a synthetic deletion inventory.