Cloud AI
Featured
Artificial Intelligence
KV Cache Memory Sizing for Long-Context LLM Serving
KV cache memory — not model weights — is usually the constraint that decides how many long-context requests a GPU can serve concurrently. The key-value cache for each request is huge and grows and...
Artificial Intelligence
Cloud Run GPU Inference: Limits, Cold Starts, Costs
Artificial Intelligence
Confidential Computing Attestation for AI Inference
Insights for your day
Artificial Intelligence
OpenAI agent pipeline: 948 Linux root exploits in 30h
OpenAI has published the details of an agentic security pipeline that, in a single 30-hour October run using an upcoming cyber-focused model, triaged 2,308 candidate Linux kernel findings, dynamically reproduced 1,724 vulnerabilities, and produced...
Cloud Computing
AWS Training in Brazil: What Reddit Reveals About the Real Experience
Brazilian cloud engineers and DevOps practitioners have turned Reddit into one of the most honest feedback loops on AWS training available anywhere.
Fresh perspectives
Artificial Intelligence
Streaming LLM Tokens: SSE vs WebSocket in Production
For streaming LLM tokens from a hosted model API to a client, server-sent events over plain HTTPS is the right default — it is the transport the providers themselves ship. Anthropic’s Messages API streams...
Artificial Intelligence
Counting Tokens Before Inference: Three Provider APIs
Every LLM request draws down two budgets at once: the model’s context window and the provider’s tokens-per-minute quota. Amazon Bedrock, Anthropic and Google Vertex AI each expose a token-counting endpoint that measures the exact...
Artificial Intelligence
INT8 vs FP8 vs INT4: Selecting Weight Quantization
For most GPU inference deployments, FP8 weight-and-activation quantization is the default pick on Hopper or newer hardware: each weight is stored in one byte instead of two, roughly halving weight memory against BF16 while...
Artificial Intelligence
Spot GPU Interruptions: Cross-Cloud Notice Windows
Spot GPU interruptions follow three different clocks on the three major clouds, and the drain plan that survives all of them is the one sized to the shortest clock. Amazon EC2 issues its interruption...
Keep exploring
Artificial Intelligence
Semantic Caching for LLM APIs: The Engineer’s Guide
Artificial Intelligence
Speculative Decoding in vLLM: The Engineer’s Setup
Artificial Intelligence
LLM Tracing and Evaluation That Works in Production
Artificial Intelligence
API Key Rotation for AI APIs: The Zero-Downtime Runbook
Artificial Intelligence
Model Deprecation Windows: Planning for API Retirements
Artificial Intelligence
Offline training: how OpenAI air-gaps frontier RL runs
Artificial Intelligence
VPC Endpoints for AI APIs: The Private Access Setup
Artificial Intelligence