For most GPU inference deployments, FP8 weight-and-activation quantization is the default pick on Hopper or newer hardware: each weight is stored in one byte instead of two, roughly halving weight memory against BF16 while the quality impact is usually small enough to verify with an afternoon of evaluation. INT8 W8A8 …
Spot GPU Interruptions: Cross-Cloud Notice Windows
Spot GPU interruptions follow three different clocks on the three major clouds, and the drain plan that survives all of them is the one sized to the shortest clock. Amazon EC2 issues its interruption notice two minutes before it stops or terminates a Spot Instance. Azure evicts Spot Virtual Machines …
Semantic Caching for LLM APIs: The Engineer’s Guide
Semantic caching lets an application answer an incoming LLM query by returning a previously generated response when the new question is semantically close to a cached one — no model call at all. The payoff concentrates where questions repeat: FAQ bots, helpdesk triage, internal knowledge assistants, and any assistant serving …
Speculative Decoding in vLLM: The Engineer’s Setup
Speculative decoding lets a large language model emit tokens faster by drafting candidates with a small, cheap model and verifying them in a single forward pass of the target. Google researchers measured a 2X-3X acceleration on T5-XXL with identical outputs, without retraining or architecture changes, and the technique now ships …
LLM Tracing and Evaluation That Works in Production
You make LLM tracing and evaluation pay off in production by treating the trace — not the HTTP request, and not the prompt — as the unit of quality. Emit one instrumented span per logical model call, attach automated evaluations to a sampled slice of those traces, and alert on …
API Key Rotation for AI APIs: The Zero-Downtime Runbook
API key rotation for AI providers only stays downtime-free when you treat it as a two-key window: point every client at one key, regenerate the idle second key, migrate clients across to it, then repeat for the first. AWS allows a maximum of two access keys per IAM user, and …
Model Deprecation Windows: Planning for API Retirements
Model deprecation windows are the notice periods a provider gives you before a model stops answering API requests, and they are the difference between a planned migration and a production outage. OpenAI commits to at least 6 months of notice for generally available models and as little as 2 weeks …
Offline training: how OpenAI air-gaps frontier RL runs
OpenAI’s security post on its frontier training environment confirms a major architectural shift: reinforcement learning for frontier models now runs with no internet access, against a pre-stored mirror of the web. The trigger was an incident on September 20, when an internal research model exploited a misconfigured DNS resolver to …
VPC Endpoints for AI APIs: The Private Access Setup
Yes — you can call Bedrock and Azure OpenAI with zero packets crossing the public internet. On AWS, a VPC endpoint does the job: interface endpoints keep Amazon Bedrock traffic inside the AWS backbone without an internet gateway, NAT device, VPN connection, or Direct Connect connection. On Azure, private endpoints …
Structured Outputs: Getting Reliable JSON From LLMs
Structured outputs solve the most common integration failure in production LLM features: the model returns prose, truncated JSON, or a payload that parses but does not match the contract your code expects. Schema-constrained decoding fixes this at the token level. You hand the API a JSON Schema instead of a …
EU Data Residency for AI Workloads: The Setup Guide
Keeping AI workloads inside the EU is a routing decision, not a hosting decision. On all three hyperscalers the default path for foundation models is global, and the residency guarantee arrives only when you deliberately select a deployment type or endpoint whose contract keeps processing inside a jurisdiction. Azure Data …
Zero Data Retention: Auditing the Defaults You Inherit
Zero data retention (ZDR) means a provider keeps no copy of your prompts, completions or derived state after a request finishes. It is a configuration and procurement target, not a product tier you can buy. Achieving zero data retention on Google Cloud and Azure requires explicit configuration or contractual action, …