Category: Artificial Intelligence
Artificial Intelligence
Prompt Injection Guardrails: AWS vs Azure Compared
Prompt injection guardrails from the two biggest clouds work only when they are wired the way each platform expects, and both hide a default that silently weakens protection: Amazon Bedrock will not filter prompt...
Artificial Intelligence
GPU Autoscaling by Queue Depth: The Full KEDA Setup
GPU autoscaling for asynchronous inference should key off queue depth, not CPU utilization. KEDA’s aws-sqs-queue trigger reads an Amazon SQS queue and turns its backlog into a replica-count metric, using a per-replica queue target...
Artificial Intelligence
Multi-LoRA Serving: Sharing One GPU Across Adapters
Multi-LoRA serving answers a capacity question that hits every team running fine-tuned variants: if each customer needs its own adapted model, do you deploy a full model copy per customer? With adapter-based serving you...
Artificial Intelligence
HNSW Tuning in Practice: m, ef_construction, ef_search
HNSW tuning for vector search comes down to three settings: m, ef_construction and ef_search. The first two shape the graph you build; the third shapes every query you run. Raise any of them and...
Artificial Intelligence
KV Cache Memory Sizing for Long-Context LLM Serving
KV cache memory — not model weights — is usually the constraint that decides how many long-context requests a GPU can serve concurrently. The key-value cache for each request is huge and grows and...
Artificial Intelligence
Cloud Run GPU Inference: Limits, Cold Starts, Costs
Cloud Run GPU attaches an NVIDIA accelerator to a fully managed container service: no drivers to install, no reservations to book, and a service that can scale down to zero between requests. For bursty...
Artificial Intelligence
Confidential Computing Attestation for AI Inference
Confidential computing attestation is the cryptographic handshake that lets a key service decide whether the code requesting your model weights or customer data is the exact workload you built. AWS Nitro Enclaves issue signed...
Artificial Intelligence
OpenAI agent pipeline: 948 Linux root exploits in 30h
OpenAI has published the details of an agentic security pipeline that, in a single 30-hour October run using an upcoming cyber-focused model, triaged 2,308 candidate Linux kernel findings, dynamically reproduced 1,724 vulnerabilities, and produced...
Artificial Intelligence
Streaming LLM Tokens: SSE vs WebSocket in Production
For streaming LLM tokens from a hosted model API to a client, server-sent events over plain HTTPS is the right default — it is the transport the providers themselves ship. Anthropic’s Messages API streams...
Artificial Intelligence
Counting Tokens Before Inference: Three Provider APIs
Every LLM request draws down two budgets at once: the model’s context window and the provider’s tokens-per-minute quota. Amazon Bedrock, Anthropic and Google Vertex AI each expose a token-counting endpoint that measures the exact...
Artificial Intelligence
INT8 vs FP8 vs INT4: Selecting Weight Quantization
For most GPU inference deployments, FP8 weight-and-activation quantization is the default pick on Hopper or newer hardware: each weight is stored in one byte instead of two, roughly halving weight memory against BF16 while...
Artificial Intelligence
Spot GPU Interruptions: Cross-Cloud Notice Windows
Spot GPU interruptions follow three different clocks on the three major clouds, and the drain plan that survives all of them is the one sized to the shortest clock. Amazon EC2 issues its interruption...