What is Horizontal Pod Autoscaler? Meaning, Architecture, Examples, Use Cases, and How to Measure It (2026 Guide)

---

Quick Definition (30–60 words)

Horizontal Pod Autoscaler (HPA) automatically scales the number of pod replicas in a Kubernetes Deployment or other controller based on observed metrics. Analogy: HPA is like a smart thermostat that adds or removes heaters as room load changes. Formal: HPA watches metrics and adjusts replicas to meet target utilization.

---

What is Horizontal Pod Autoscaler?

Horizontal Pod Autoscaler (HPA) is a Kubernetes control-loop that automatically adjusts pod replica counts for scalable workloads. It is not a vertical resizer, not a node autoscaler, and not a replacement for capacity planning. HPA acts at the workload layer, translating observed telemetry into replica decisions within constraints you configure.

Key properties and constraints

• Works at the Horizontal scaling layer: adjusts replica counts of supported controllers.
• Uses metrics from Metrics API, Custom Metrics API, or External Metrics API.
• Subject to minReplicas and maxReplicas bounds.
• Decision frequency configurable by controller manager flags and Kubernetes version.
• Scaling effect is eventual; scaling cannot instantly change capacity.
• Pod startup, readiness, and termination behavior affects effective capacity.
• HPA does not directly provision nodes; relies on Cluster Autoscaler or cloud autoscaling.

Where it fits in modern cloud/SRE workflows

• Application level: ensures service capacity tracks demand.
• Observability: integrated with metrics pipelines for targets and alerts.
• CI/CD: HPA config is part of manifest and GitOps flows.
• Incident response: acts as automated mitigation for load spikes, but requires runbooks for mis-scaling.
• Cost management: helps match compute spend to demand but can also increase cost if targets are misconfigured.

Diagram description (text-only)

• Metrics sources (app metrics, node metrics, external) feed into Metrics API.
• HPA controller polls Metrics API at intervals.
• HPA evaluates target vs current; computes desired replicas.
• HPA writes new replica count to controller (Deployment/ReplicaSet).
• Controller creates or deletes pods; Pod lifecycle and readiness probes determine traffic routing.
• Cluster Autoscaler or cloud provider adjusts nodes if needed.

Horizontal Pod Autoscaler in one sentence

HPA is an automated controller that scales Kubernetes pod replicas based on telemetry-driven targets to maintain application performance and efficiency.

Horizontal Pod Autoscaler vs related terms (TABLE REQUIRED)

ID	Term	How it differs from Horizontal Pod Autoscaler	Common confusion
T1	Vertical Pod Autoscaler	Adjusts resource requests of containers not replicas	People think VPA scales pod count
T2	Cluster Autoscaler	Scales cluster nodes not pods	Confused as HPA auto-provisioning nodes
T3	KEDA	Event-driven scaler for external triggers	Often used interchangeably with HPA
T4	PodDisruptionBudget	Controls voluntary pod evictions, not scaling	Mistaken for scaling restraint
T5	Horizontal Pod Autoscaler V2	Supports custom/external metrics vs V1 static CPU only	Confused as different product
T6	Metrics Server	Provides CPU/memory metrics only	Believed to replace full metrics pipeline
T7	Custom Metrics API	Exposes app metrics for HPA	Users assume automatic setup
T8	VerticalScaling	Generic term for resizing resources	Misread as same as HPA
T9	AutoscalingPolicy	Policy frameworks around scaling	Mistaken as the scaler itself

Row Details (only if any cell says “See details below”)

• None

---

Why does Horizontal Pod Autoscaler matter?

Business impact

• Revenue: Automatic scaling reduces capacity-related outages during traffic surges, preventing revenue loss.
• Trust: Consistent performance improves customer trust and reduces churn.
• Risk: Misconfiguration can cause runaway costs or unstable service.

Engineering impact

• Incident reduction: HPA reduces incidents tied to predictable load patterns by auto-adjusting capacity.
• Velocity: Teams deliver features without manual scaling ops.
• Complexity: Requires observability and testing to avoid systemic failures.

SRE framing

• SLIs: Latency, error rate, and request success rate are primary service SLIs impacted by HPA behavior.
• SLOs and error budgets: HPA helps meet SLOs by adding capacity, but improper targets may deplete error budgets.
• Toil: HPA reduces manual scaling toil but adds operational surface for telemetry and tuning.
• On-call: On-call playbooks must include HPA health checks and rollback steps.

What breaks in production (3–5 realistic examples)

1. Rapid traffic spike triggers scale-up, but Cluster Autoscaler is slow causing pending pods and increased latency.
2. HPA misconfigured to scale on an unreliable custom metric, causing oscillations and repeated restarts.
3. Overconservative maxReplicas leads to saturation and SLO violations during peak events.
4. Attack or traffic spike causes runaway auto-scaling and high cloud costs.
5. Readiness probes misconfigured; HPA scales but pods aren’t serving traffic due to probe failures.

---

Where is Horizontal Pod Autoscaler used? (TABLE REQUIRED)

ID	Layer/Area	How Horizontal Pod Autoscaler appears	Typical telemetry	Common tools
L1	Edge	Scales ingress controllers and edge proxies based on request rate	Requests per second and latency	Ingress controller metrics, Prometheus
L2	Network	Scales sidecars and network policy agents by throughput	Network bytes and connections	CNI metrics, Prometheus
L3	Service	Scales stateless microservices by CPU or request latency	CPU, request latency, RPS	Prometheus, Metrics API
L4	App	Scales frontends and APIs using custom app metrics	Error rate, latency, queue depth	Custom Metrics API, Prometheus
L5	Data	Limited use for stateless data jobs; careful with stateful sets	Job queue length, consumer lag	Kafka metrics, Prometheus
L6	Kubernetes layer	Scales controllers or adapters handling events	Event processing lag	KEDA, controllers metrics
L7	IaaS/PaaS/SaaS	Operates on PaaS Kubernetes or managed clusters	Same as service layer	Cloud managed HPA integrations
L8	CI/CD	Used in pipelines for test environments to simulate scale	Synthetic load metrics	CI tooling, Prometheus
L9	Incident response	Auto-mitigation for load incidents	Spike detection metrics	Alert systems, Prometheus
L10	Observability	Feeds into dashboards for autoscaling decisions	Replica counts and metrics	Grafana, Prometheus

Row Details (only if needed)

• None

---

When should you use Horizontal Pod Autoscaler?

When it’s necessary

• Workloads are stateless or handle idempotent requests.
• Demand varies significantly over time.
• You have reliable metrics that reflect capacity needs.

When it’s optional

• Low-traffic internal tools with stable load.
• Systems where cost predictability outweighs elasticity.

When NOT to use / overuse it

• StatefulSets with strict affinity and single-writer constraints.
• Workloads dependent on local ephemeral storage per pod.
• When metrics are noisy or missing and cause oscillation.

Decision checklist

• If workload is stateless AND traffic varies -> Use HPA.
• If stateful AND per-pod state matters -> Avoid HPA; consider manual scaling or VPA.
• If you need event-driven scaling from external queues -> Use KEDA or External Metrics with HPA.
• If cluster node provisioning is slow -> Ensure Cluster Autoscaler configured before aggressive HPA targets.

Maturity ladder

• Beginner: Scale by CPU/memory with Metrics Server and basic targets.
• Intermediate: Use custom metrics (latency/queue depth), configure buffer and cooldown.
• Advanced: Combine HPA with predictive autoscaling, KEDA, Node autoscaling policies, and cost-aware controls; incorporate ML anomaly detection for scale events.

---

How does Horizontal Pod Autoscaler work?

Components and workflow

1. Metrics sources: Metrics Server for CPU/memory, Prometheus-adapter or custom metrics for app metrics, external metrics via External Metrics API.
2. HPA controller: Periodically fetches metrics and current replica count, calculates desired replicas using target formulas, applies stabilization and scaling policies.
3. Controller update: HPA writes new replica count to target controller (Deployment, ReplicaSet, StatefulSet if supported).
4. Controller reconciliation: Deployment/ReplicaSet creates or deletes pods.
5. Pod lifecycle: Scheduler places pods on nodes; readiness probes gate traffic.

Data flow and lifecycle

• Polling interval -> metrics fetch -> desiredReplica calculation -> apply min/max and policies -> update scale target -> observe effect over next cycles.

Edge cases and failure modes

• Missing metrics: HPA cannot compute targets and may not scale.
• Pods pending due to node shortage: HPA increases replicas but pods stay pending.
• Rapid oscillation: Frequent up/down causing instability.
• Unbalanced distribution: Pods scheduled on nodes lacking resources leading to pod eviction.

Typical architecture patterns for Horizontal Pod Autoscaler

• Basic HPA: CPU-based scaling using Metrics Server. Use for simple stateless services.
• Custom-metrics HPA: Latency or application queue-based scaling using Prometheus Adapter. Use when CPU not correlated with load.
• KEDA-based event scaling: HPA triggered via KEDA for event sources like queues and streams.
• External metrics HPA: Uses external cloud metrics like SNS queue size or custom cloud metrics.
• Combined HPA + Predictive autoscaler: Uses ML models to forecast demand and pre-scale pods.
• Burstable scaling with cooldowns: HPA tuned with stabilization windows to avoid oscillation.

Failure modes & mitigation (TABLE REQUIRED)

ID	Failure mode	Symptom	Likely cause	Mitigation	Observability signal
F1	Missing metrics	HPA reports unknown or no scaling	Metrics API unavailable	Fix metrics pipeline or add fallback metric	Metrics API errors
F2	Pending pods	New pods stuck Pending	Cluster lacks nodes	Configure Cluster Autoscaler or increase node pool	Pending pod count
F3	Oscillation	Rapid up/down scaling	Tight targets or noisy metrics	Add stabilization window and larger targets	Replica churn rate
F4	Slow scale-up	Latency during spike	Pod startup time or readiness issues	Optimize startup, warm pools, pre-scale	High latency and low ready pod ratio
F5	Over-scaling cost	Unexpected high costs	Aggressive targets or traffic spikes	Add budget caps and scale-down policies	Cost reports rising with replicas
F6	Wrong metric	SLOs degrade despite scaling	Metric not representative of load	Use SLI-aligned metric like latency	SLI-SLO mismatch signals
F7	Scale-down kills work	Jobs lost on scale-down	Non-idempotent processing or improper grace periods	Use job queues and safe shutdown hooks	Error spikes on pod termination

Row Details (only if needed)

• None

---

Key Concepts, Keywords & Terminology for Horizontal Pod Autoscaler

Below is a compact glossary of 40+ terms and short context for each.

Autoscaler — Controller that adjusts capacity — Central concept for HPA — Confused with node autoscaler ReplicaSet — Kubernetes controller for pods — HPA targets replicas — Not all controllers are scalable Deployment — Declarative app controller — Common HPA target — Ensure selector stability StatefulSet — Controller for stateful pods — HPA limited use — Scaling may break state HPA controller — Kubernetes control-loop — Implements scaling logic — Needs metrics Metrics Server — Provides CPU/memory metrics — Basic HPA source — Not for app metrics Custom Metrics API — Exposes app metrics to HPA — Enables latency-based scaling — Requires adapter External Metrics API — Exposes external metrics — For cloud queues and services — Adapter complexity Prometheus Adapter — Bridges Prometheus to Custom Metrics API — Common integration — Adapter config complexity KEDA — Event-driven scaling framework — Scales based on external events — Sometimes replaces HPA Cluster Autoscaler — Scales nodes based on pending pods — Works with HPA — Mis-tuned can delay pods Node Pool — Group of nodes with similar config — Important for scheduling — Hotspot risk if unbalanced Scale-up / Scale-down — Actions to add or remove replicas — Core operations — Can oscillate MinReplicas — Lower bound for replicas — Prevents scale to zero — Must be set correctly MaxReplicas — Upper bound for replicas — Cost control — Too low causes saturation Target metric — Value HPA tries to hit — E.g., 50% CPU — Should reflect SLOs Utilization — Ratio of used to requested resource — Often CPU utilization — Misleading if requests wrong Stabilization window — Time HPA waits to change scale — Avoids thrashing — Too long delays reaction Cooldown — Post-scale wait to avoid immediate reversal — Similar to stabilization — Needs tuning Scale policy — Rules around scaling increments — Controls velocity — Complex policies can hide issues Readiness probe — Indicates pod can serve traffic — Affects effective capacity — Misconfigurations hide readiness Liveness probe — Detects unhealthy pods — Ensures restart — Can cause disruption during scaling Pod Disruption Budget — Limits voluntary evictions — Protects availability during scale-down — May prevent scaling PriorityClass — Pod scheduling priority — Affects which pods evicted — Useful in mixed workloads Graceful termination — Time given for cleanup on termination — Important for stateful work — Too short causes errors PreStop hook — Lifecycle hook before termination — Useful to drain work — Not always reliable Burstable load — Short spikes in traffic — HPA should handle with headroom — Too aggressive policies harm cost Predictive autoscaling — Forecasting demand to pre-scale — Reduces cold-start latency — Requires training data Anomaly detection — Detects abnormal metrics — Can trigger protective behavior — False positives cause actions Scale-to-zero — Reducing to zero replicas for cost savings — Useful for dev workloads — Cold starts risk Cost-aware scaling — Balances performance and spend — Requires cost signals — Tradeoff analysis needed SLO — Service Level Objective — Target service behavior — Use as HPA alignment metric SLI — Service Level Indicator — Measurable metric for SLO — HPA must consider SLI alignment Error budget — Allowable SLO breach margin — Use before aggressive scaling — Misuse can mask faults Pod startup time — Time to become Ready — Critical for scaling speed — Measure and optimize Warm pools — Pre-warmed pods to reduce cold start — Improves response time — Adds baseline cost Throttling — Rate limiting at service or infra level — Can confuse HPA metrics — Observe throttling signals Backpressure — Upstream telling clients to slow down — Prefer over uncontrolled scaling — Application design issue Horizontal vs Vertical — Scaling across vs within instances — HPA is horizontal — Both may be needed Telemetry quality — Accuracy and latency of metrics — Critical for correct scaling — Poor telemetry causes false actions Autoscaling budget — Constraints to limit autoscaling costs — Protects cloud spend — Needs governance Admission controller — Kubernetes extension that can mutate HPA manifests — Used for policy — Misconfiguration can block deploys GitOps — Managing HPA via Git — Enables auditability — Drift must be handled Chaos testing — Inject failures to validate scaling — Ensures resilience — Needs controlled environment Runbook — Procedures for operators — Includes HPA operations — Essential for on-call

---

How to Measure Horizontal Pod Autoscaler (Metrics, SLIs, SLOs) (TABLE REQUIRED)

ID	Metric/SLI	What it tells you	How to measure	Starting target	Gotchas
M1	Replica count	Current scaled replica count	Kubernetes API replicas field	N/A use monitored baseline	Rapid changes may signal issues
M2	Desired replicas	HPA desired replicas value	HPA status.desiredReplicas	N/A for drift detection	Diff vs actual indicates failures
M3	CPU utilization	CPU usage vs request	Pod CPU / requested CPU via Metrics API	50%–70% as start	Wrong requests skew value
M4	Request latency SLI	End-to-end response latency	P95 request latency from app metrics	SLO dependent, e.g., 300ms	Tail latency hidden by P50
M5	Request rate (RPS)	Incoming traffic intensity	Aggregated RPS from ingress metrics	Use historical peaks	Bursts require headroom
M6	Queue depth	Backlog for async processing	Queue length metric from queue system	Keep below processing capacity	Inconsistent queue metrics
M7	Pending pods	Pods Pending state count	Kubernetes API pod status.phase	0 ideal	Pending indicates resource shortage
M8	Pod startup time	Time between pod creation and Ready	Container start to readiness event	<30s preferred	Image pulls and init containers lengthen it
M9	Pod readiness ratio	Ready pods / desired pods	Kubernetes pod conditions	>=95%	Readiness probes false negatives
M10	Scale latency	Time from metric trigger to ready capacity	Measure from spike to restored SLI	As low as possible	Depends on cloud and startup time
M11	Oscillation rate	Frequency of replica changes	Count of scaling events per window	<1 per 5m	Higher means unstable metrics
M12	Cost per request	Cloud cost relative to throughput	Cost / number of requests	Business-defined budget	Cost attribution complexity
M13	Error rate	Application errors during scaling	5xx rate from app logs	Keep below SLO error budget	Errors may be unrelated
M14	Node provisioning time	Time to add nodes when needed	Cloud node lifecycle times	Keep low for fast scale-ups	Cloud limits add variability
M15	Scale-to-zero events	Count of zero-replica states	HPA minReplicas metric	Controlled for dev only	Cold-start and readiness issues

Row Details (only if needed)

• None

Best tools to measure Horizontal Pod Autoscaler

Tool — Prometheus

• What it measures for Horizontal Pod Autoscaler: Replica counts, custom app metrics, pod resource usage
• Best-fit environment: Kubernetes-native observability stacks
• Setup outline:
• Scrape app and Kubernetes metrics
• Configure Prometheus Adapter for HPA
• Define recording rules for SLIs
• Create alerts for scaling failures
• Strengths:
• Flexible query language and ecosystem
• Good for custom metrics
• Limitations:
• Operational overhead at scale
• Adapter configuration complexity

Tool — Grafana

• What it measures for Horizontal Pod Autoscaler: Dashboards for HPA metrics and SLOs
• Best-fit environment: Teams needing visualization and alerting
• Setup outline:
• Connect to Prometheus
• Build HPA dashboards
• Configure alerts and notification channels
• Strengths:
• Rich visualization
• Alerting integration
• Limitations:
• Requires data source tuning
• Dashboard sprawl risk

Tool — Kubernetes Metrics Server

• What it measures for Horizontal Pod Autoscaler: CPU and memory usage per pod
• Best-fit environment: Basic CPU/memory HPA use cases
• Setup outline:
• Deploy metrics-server in cluster
• Ensure kubelet metrics available
• Use HPA with CPU-based targets
• Strengths:
• Lightweight
• Native integration
• Limitations:
• Not for custom application metrics
• Aggregation limitations

Tool — Prometheus Adapter

• What it measures for Horizontal Pod Autoscaler: Exposes Prometheus metrics as Custom Metrics API
• Best-fit environment: Prometheus-backed HPA with custom metrics
• Setup outline:
• Install adapter
• Map PromQL queries to metric names
• Test HPA behavior with custom metrics
• Strengths:
• Enables app-metric-driven scaling
• Flexible query mapping
• Limitations:
• Mapping complexity
• Can stress Prometheus with expensive queries

Tool — KEDA

• What it measures for Horizontal Pod Autoscaler: External event sources like queues, streams, and cron
• Best-fit environment: Event-driven workloads and queue consumers
• Setup outline:
• Install KEDA operator
• Configure ScaledObject referencing trigger
• Tune scale thresholds and cooldown
• Strengths:
• Native event-driven scaling
• Supports many external triggers
• Limitations:
• Additional operator to manage
• Some triggers require credentials

Tool — Cloud provider autoscaling (managed)

• What it measures for Horizontal Pod Autoscaler: Integrations exposing cloud metrics to HPA
• Best-fit environment: Managed Kubernetes offerings
• Setup outline:
• Enable provider metrics adapter
• Configure HPA to use external metrics
• Strengths:
• Works with provider-specific metrics
• Limitations:
• Varies by provider and account permissions

Recommended dashboards & alerts for Horizontal Pod Autoscaler

Executive dashboard

• Panels:
• Overall replica counts across services (why: show resource footprint)
• Cost per service (why: show financial impact)
• SLO compliance summary (why: business health)
• Top 10 services by scale events (why: highlight volatile apps)

On-call dashboard

• Panels:
• Desired vs actual replicas per target (why: quick detection of scale failures)
• Pending pods and node availability (why: identify node constraints)
• Recent scale events timeline (why: context during incident)
• SLI latency and error rate panels with annotations for scale events (why: causal link)

Debug dashboard

• Panels:
• Pod startup time distribution (why: root cause of slow scale-up)
• Readiness probe failures by pod (why: identify misconfigured probes)
• HPA metrics including raw metric time series (why: validate metric correctness)
• Prometheus query latency and adapter errors (why: ensure metrics pipeline health)

Alerting guidance

• Page vs ticket:
• Page on SLO breach affecting users, large sustained pending pods, or cluster node exhaustion.
• Create ticket for replica drift, minor scale anomalies, or cost alerts that do not impact SLIs.
• Burn-rate guidance:
• Use error budget burn rate thresholds (e.g., 3x burn in 5 minutes => page).
• Adjust for seasonal traffic; align with SLO policy.
• Noise reduction tactics:
• Group alerts by service and root cause.
• Suppress alerts during planned deployments.
• Deduplicate alerts by correlating replica spikes with known events.

---

Implementation Guide (Step-by-step)

1) Prerequisites – Kubernetes cluster with version supporting the intended HPA API. – Metrics pipeline (Metrics Server for CPU/memory; Prometheus + adapter for custom metrics). – Cluster Autoscaler or node provisioning strategy. – Team agreement on SLOs and cost constraints.

2) Instrumentation plan – Identify SLIs that map to user experience (latency, error rate). – Expose application metrics for queue depth, processing latency, and request rate. – Ensure metrics are tagged by service and environment.

3) Data collection – Deploy Prometheus or managed metrics solution. – Configure scraping targets and retention. – Deploy Prometheus Adapter or other API adapters.

4) SLO design – Define SLOs for each service (e.g., 99.9% success under 500ms). – Determine acceptable error budgets and burn-rate policies. – Map HPA targets to SLOs (e.g., scale on P95 latency).

5) Dashboards – Create executive, on-call, and debug dashboards. – Add annotations for deployments and scale events.

6) Alerts & routing – Alert on SLO breach, pending pods, and adapter errors. – Route critical pages to on-call, non-critical to inbox.

7) Runbooks & automation – Document steps to inspect HPA status, metrics, and pod states. – Automate common fixes: restart adapter, increase maxReplicas temporarily. – Consider automated rollback on repeated failures.

8) Validation (load/chaos/game days) – Run load tests to validate scaling behavior and node provisioning. – Chaos test node and metric failures to validate runbooks. – Perform game days simulating traffic spikes and observe behavior.

9) Continuous improvement – Review postmortems and adjust stabilization windows and targets. – Refine metrics and instrumentation based on incidents. – Periodically analyze cost vs performance tradeoffs.

Pre-production checklist

• Metrics pipeline validated with synthetic metrics.
• HPA manifests reviewed and in GitOps.
• Min/max replicas set and reasonable.
• Readiness and liveness probes tested.
• Cluster Autoscaler validated.

Production readiness checklist

• Observability dashboards and alerts in place.
• Cost controls and budgets defined.
• Runbooks and playbooks available to on-call.
• RBAC for HPA management restricted.

Incident checklist specific to Horizontal Pod Autoscaler

• Check HPA status and events.
• Verify metrics source health and adapter logs.
• Inspect pending pods and node pool capacity.
• Temporarily set replicas manually if needed.
• Escalate to infra team if nodes are unavailable.

---

Use Cases of Horizontal Pod Autoscaler

1) Public API under variable load – Context: Customer-facing REST API with diurnal traffic. – Problem: Peak spikes cause latency. – Why HPA helps: Scales replicas to match demand. – What to measure: P95 latency, request rate, replica count. – Typical tools: Prometheus, Grafana, Metrics Server

2) Background workers consuming queues – Context: Asynchronous job processing with variable queue depth. – Problem: Queue backlog increases during batch arrivals. – Why HPA helps: Scale workers based on queue depth. – What to measure: Queue length, processing latency. – Typical tools: KEDA, Prometheus, messaging metrics

3) Ingress controllers – Context: Edge proxies receiving global traffic. – Problem: Sudden traffic bursts cause proxy saturation. – Why HPA helps: Scale ingress pods to maintain throughput. – What to measure: RPS per pod, healthy connections, latency. – Typical tools: NGINX metrics, Prometheus, Cluster Autoscaler

4) Batch processing with time windows – Context: Nightly ETL jobs execute heavy work. – Problem: Need higher parallelism at night. – Why HPA helps: Scale workers for batch window then scale down. – What to measure: Job completion time, replica utilization. – Typical tools: CronJobs, custom metrics, Prometheus

5) Blue/green test environments – Context: Staging load tests after deployment. – Problem: Need temporary capacity during tests. – Why HPA helps: Automatically scale staging apps to match test load. – What to measure: Test RPS, replica count. – Typical tools: CI/CD tools, Prometheus

6) Cost optimization for dev environments – Context: Development namespaces idle outside working hours. – Problem: Wasteful always-on replicas. – Why HPA helps: Scale-to-zero or low baseline when idle. – What to measure: Active requests, idle time. – Typical tools: KEDA, Metrics Server

7) Event-driven microservices – Context: Microservices triggered by external events like webhooks. – Problem: Bursty event traffic needs quick scaling. – Why HPA helps: Scales based on event queue metrics. – What to measure: Event backlog, processing latency. – Typical tools: KEDA, Prometheus

8) ML inference services – Context: Model inference under spiky usage. – Problem: Latency-sensitive predictions require headroom. – Why HPA helps: Scale replicas to meet latency SLI. – What to measure: P95 latency, concurrency, GPU utilization. – Typical tools: Custom metrics, Prometheus

---

Scenario Examples (Realistic, End-to-End)

Scenario #1 — Kubernetes web service with latency SLO

Context: Public API served by a Kubernetes Deployment.
Goal: Maintain P95 latency under 300ms during traffic spikes.
Why Horizontal Pod Autoscaler matters here: HPA scales pods when latency rises to keep SLO.
Architecture / workflow: App exports P95 latency to Prometheus; Prometheus Adapter exposes custom metric; HPA uses custom metric to scale Deployment; Cluster Autoscaler manages nodes.
Step-by-step implementation:

1. Expose P95 latency metric.
2. Deploy Prometheus and Adapter.
3. Create HPA target using custom metric with minReplicas 3 and maxReplicas 50.
4. Configure stabilization window of 2 minutes.
5. Add dashboards and alerts for SLO and pending pods. What to measure: P95 latency, desired vs actual replicas, pending pods.
   Tools to use and why: Prometheus (metrics), Prometheus Adapter (custom metrics), Grafana (dashboards), Cluster Autoscaler (node scaling).
   Common pitfalls: Latency metric noisy at low traffic; adapter query too expensive.
   Validation: Run load test with spike pattern; verify latency maintained and nodes provisioned.
   Outcome: SLO met during spikes with controlled cost.

Scenario #2 — Serverless-like scale-to-zero for dev environments (managed PaaS)

Context: Managed Kubernetes offering with ability to scale to zero for non-production apps.
Goal: Reduce cost by scaling dev services to zero during off-hours.
Why Horizontal Pod Autoscaler matters here: HPA combined with scale-to-zero control reduces baseline cost.
Architecture / workflow: Metrics server or external metric signals idle state; controller scales replicas to zero; warmup job triggers pre-scale before work.
Step-by-step implementation:

1. Define minReplicas 0 and maxReplicas 5.
2. Use external metrics to detect active usage.
3. Configure warmup job to pre-scale before scheduled tests. What to measure: Scale-to-zero events, cold-start latency, cost savings.
   Tools to use and why: Managed HPA support in cloud provider, external metrics adapter.
   Common pitfalls: Cold starts violate SLOs; missing external metric authentication.
   Validation: Schedule off-hours test and measure cost reduction.
   Outcome: Lower cost for dev resources with targeted warmups.

Scenario #3 — Incident response: HPA failure during traffic surge (postmortem)

Context: Sudden traffic spike uncovered HPA misconfiguration causing pending pods.
Goal: Restore service and fix root cause to prevent recurrence.
Why Horizontal Pod Autoscaler matters here: HPA was first line of defense but failed due to missing metrics adapter.
Architecture / workflow: HPA using custom metrics; adapter crashed; HPA could not compute desired replicas.
Step-by-step implementation:

1. On-call inspects HPA status and adapter logs.
2. Manually scale replicas to restore capacity.
3. Restart Prometheus Adapter.
4. Update runbook to include adapter health checks. What to measure: Adapter uptime, pending pods, SLOs during incident.
   Tools to use and why: Prometheus logs, Kubernetes events, Grafana.
   Common pitfalls: No alert for adapter down; missing manual scale fallback.
   Validation: Inject adapter failure in staging and confirm runbook actions.
   Outcome: Incident resolved; postmortem identifies monitoring gap and adds automated restart.

Scenario #4 — Cost vs performance tuning

Context: E-commerce app where scaling aggressively is costly during marketing campaigns.
Goal: Balance cost with latency SLOs during promotions.
Why Horizontal Pod Autoscaler matters here: HPA controls replicas; needs budget-aware limits.
Architecture / workflow: HPA scales on RPS and latency; cost controller monitors spend; autoscaling budget applied.
Step-by-step implementation:

1. Define SLO tiers with degraded mode for low-priority features.
2. Set maxReplicas based on cost budgets per environment.
3. Implement alerting when cost per request exceeds thresholds. What to measure: Cost per request, SLO compliance, replica usage.
   Tools to use and why: Cost analytics, Prometheus, Grafana.
   Common pitfalls: MaxReplicas too low causing SLO breaches; too high causing overspend.
   Validation: Simulate promotion spike with cost constraints; observe tradeoffs.
   Outcome: Budget controls with acceptable SLO degradation during cost peaks.

---

Common Mistakes, Anti-patterns, and Troubleshooting

1. Symptom: HPA not scaling -> Root cause: Metrics API unreachable -> Fix: Check adapter logs and API access.
2. Symptom: Pods Pending -> Root cause: Insufficient nodes -> Fix: Configure Cluster Autoscaler and node pools.
3. Symptom: Replica oscillation -> Root cause: No stabilization window -> Fix: Set stabilization window and scale policies.
4. Symptom: SLO breaches despite scaling -> Root cause: Wrong metric (CPU vs latency) -> Fix: Align HPA metric with SLI.
5. Symptom: High cost after scaling -> Root cause: Unbounded maxReplicas -> Fix: Set sensible maxReplicas or cost caps.
6. Symptom: Slow recovery after spike -> Root cause: Long pod startup time -> Fix: Optimize startup, use warm pools.
7. Symptom: HPA shows desired>actual -> Root cause: Pod scheduling failure -> Fix: Inspect taints, node selectors, resource requests.
8. Symptom: False scaling on synthetic traffic -> Root cause: Test traffic not isolated -> Fix: Tag test traffic or use separate namespace.
9. Symptom: Adapter query timeouts -> Root cause: Expensive PromQL queries -> Fix: Use recording rules and optimized queries.
10. Symptom: Scale-down kills in-flight work -> Root cause: Non-idempotent processing -> Fix: Drain logic and safe shutdown hooks.
11. Symptom: No alert on adapter failure -> Root cause: No observability on adapter -> Fix: Add adapter health probes and alerts.
12. Symptom: HPA stuck due to RBAC -> Root cause: Adapter lacks permissions -> Fix: Grant necessary roles for metrics API.
13. Symptom: Readiness probes failing after scale -> Root cause: Probe depends on external service -> Fix: Make probe local or mock dependencies.
14. Symptom: Scale-to-zero causes long cold starts -> Root cause: heavy initialization -> Fix: Reduce init work or keep minimal warm replicas.
15. Symptom: Inconsistent metrics across replicas -> Root cause: Non-uniform instrumentation -> Fix: Standardize metrics and labels.
16. Symptom: Alerts noise during deployments -> Root cause: Deployment-induced traffic patterns -> Fix: Suppress alerts during deployment windows.
17. Symptom: Cluster Autoscaler interference -> Root cause: Autoscaler removes nodes too aggressively -> Fix: Node autoscaler tuning and pod priority.
18. Symptom: Incorrect SLI mapping -> Root cause: Measuring wrong latency dimension -> Fix: Re-evaluate SLI mapping to user experience.
19. Symptom: Pod churn increases latency -> Root cause: frequent restarts from liveness probes -> Fix: Adjust probe thresholds.
20. Symptom: HPA ignores custom metric -> Root cause: Metric name mismatch -> Fix: Verify metric names and API mapping.
21. Symptom: Observability blind spots -> Root cause: missing logs/metrics during scaling -> Fix: Ensure high-cardinality telemetry retention around events.
22. Symptom: Scaling performed by multiple systems -> Root cause: HPA and KEDA conflict -> Fix: Consolidate to one scaler or coordinate policies.
23. Symptom: RBAC prevents manual override -> Root cause: Overrestrictive permissions -> Fix: Review escalation path for on-call.
24. Symptom: Debugging slow due to sprawling dashboards -> Root cause: Too many metrics without tagging -> Fix: Standardize labels and minimal necessary dashboards.
25. Symptom: Autoscaler triggers by attacker traffic -> Root cause: No rate limiting -> Fix: Add WAF or rate-limiting and protection rules.

Observability pitfalls (at least 5 included above)

• Missing adapter health metrics.
• No recording rules leading to heavy PromQL queries.
• Lack of readiness probe telemetry when scaling.
• Low retention of telemetry around incidents.
• No correlation between scale events and SLIs.

---

Best Practices & Operating Model

Ownership and on-call

• Ownership: Application team owns HPA configuration and SLO alignment; platform owns metrics pipeline and node autoscaling.
• On-call: App on-call should be primary for SLO breaches; infra on-call supports node or adapter failures.

Runbooks vs playbooks

• Runbooks: Step-by-step actions for common HPA incidents (adapter restart, manual scale).
• Playbooks: Broader incident plans including stakeholders, escalation, and communication templates.

Safe deployments

• Use canary or progressive rollout for HPA changes via GitOps.
• Test HPA changes in staging with synthetic load.
• Have rollback manifests and validate min/max values.

Toil reduction and automation

• Automate detection of misconfigured HPAs (e.g., minReplicas 0 for critical services).
• Auto-remediate transient adapter failures with restart policies.
• Use Autonomous test scenarios to validate scaling post-deploy.

Security basics

• Limit RBAC for HPA and metrics adapters.
• Secure metric endpoints and adapters with TLS and least privilege.
• Rotate credentials used by external metric adapters.

Weekly/monthly routines

• Weekly: Review top services by scale events and costs.
• Monthly: Audit HPA manifests and align with updated SLOs.
• Quarterly: Load-test and validate predictive autoscaling models.

Postmortem reviews

• Review HPA configuration and metric alignment in every scaling-related postmortem.
• Check for missing alerts or runbook gaps and update documentation.

---

Tooling & Integration Map for Horizontal Pod Autoscaler (TABLE REQUIRED)

ID	Category	What it does	Key integrations	Notes
I1	Metrics provider	Collects and stores metrics	Prometheus, Metrics Server	Core for HPA decisions
I2	Metrics adapter	Exposes metrics to HPA API	Prometheus Adapter, External Adapter	Maps queries to custom metrics
I3	Event scaler	Event-driven triggers for scale	KEDA	Supports queues and cron triggers
I4	Node autoscaler	Adjusts node pool size	Cluster Autoscaler, cloud autoscaler	Works with HPA to satisfy pod scheduling
I5	Visualization	Dashboards and alerts	Grafana	Visualize HPA metrics and SLOs
I6	CI/CD	Manage HPA manifests and GitOps	GitOps tools	Ensures config drift control
I7	Cost analytics	Attribute cost to scaling events	Cost monitoring tools	Useful for cost-aware scaling
I8	Service mesh	Adds observability and traffic metrics	Istio, Linkerd	Provides advanced metrics for HPA
I9	Policy engine	Enforce scale constraints	OPA/Gatekeeper	Prevent unsafe HPA changes
I10	Chaos testing	Validate scaling resilience	Chaos engineering tools	Simulate failure modes

Row Details (only if needed)

• None

---

Frequently Asked Questions (FAQs)

H3: What metrics can HPA use?

HPA can use CPU and memory via Metrics Server, custom application metrics via Custom Metrics API, and external metrics via External Metrics API. Availability depends on adapters and setup.

H3: Can HPA scale StatefulSets?

StatefulSets can be scaled, but caution is needed due to per-pod identity and storage. Evaluate impact on state and ordering guarantees before use.

H3: How fast does HPA react?

Reaction time depends on polling interval, stabilization windows, pod startup time, and node provisioning delay. Exact timing varies.

H3: Does HPA provision nodes?

HPA does not provision nodes directly; Cluster Autoscaler or cloud autoscaling must provision nodes for pods to schedule.

H3: How to prevent oscillation?

Use stabilization windows, scale policies with limited increments, and choose stable metrics aligned to SLOs.

H3: Is HPA secure?

HPA itself follows Kubernetes RBAC, but metrics adapters and external metrics must be secured with TLS and least privilege.

H3: Should I scale on CPU?

Only if CPU correlates with user-visible SLOs. Prefer latency or queue depth when those represent user experience.

H3: Can HPA scale to zero?

Yes if minReplicas is set to 0 and external metrics or events permit. Consider cold-start implications.

H3: How to test HPA changes?

Use synthetic load tests in staging and chaos experiments to validate behavior under failure modes.

H3: What causes HPA to stop scaling?

Common causes: metrics unavailability, adapter RBAC issues, API errors, or controller misconfiguration.

H3: How are scale policies configured?

Policies are defined on the HPA resource specifying type (percent or absolute) and periods for stabilization.

H3: Can HPA use Prometheus directly?

Not directly; use Prometheus Adapter to expose Prometheus metrics to the Custom Metrics API for HPA consumption.

H3: How to align HPA with SLOs?

Choose SLI-based metrics (e.g., latency P95) as HPA targets or combine RPS with latency to influence replica counts.

H3: What about cost controls?

Set maxReplicas, use cost-aware controllers, and monitor cost per request to enforce budgets.

H3: Does HPA work with serverless platforms?

Managed platforms may provide autoscaling primitives; HPA concepts apply when underlying container orchestration is Kubernetes-based.

H3: How to debug HPA unexpected behavior?

Check HPA status, events, adapter logs, pod conditions, pending pods, and correlating metrics for root cause.

H3: Are there built-in safety mechanisms?

HPA has min/max bounds, stabilization windows, and scale policies to prevent extreme actions.

H3: What metrics indicate healthy scaling?

Stable desired vs actual replicas, low pending pods, maintained SLOs, and reasonable scale latency are indicators.

---

Conclusion

Horizontal Pod Autoscaler is a foundational automation for Kubernetes workloads, enabling responsive capacity management when paired with robust metrics, node autoscaling, and operational runbooks. Properly implemented, it reduces incidents and operational toil while aligning system capacity to business SLOs. Misconfigured or unsupported telemetry can cause failures or cost overruns; invest in observability, testing, and governance.

Next 7 days plan

• Day 1: Inventory services and identify candidates for HPA based on statelessness and traffic patterns.
• Day 2: Ensure Metrics Server and Prometheus are deployed and healthy; validate example metrics.
• Day 3: Create HPA manifests for a low-risk service and deploy to staging.
• Day 4: Run load tests to validate scaling and pod startup time; adjust stabilization windows.
• Day 5: Add dashboards and alerts for HPA metrics and adapter health.
• Day 6: Write or update runbooks for HPA-related incidents and train on-call.
• Day 7: Roll out HPA configuration to production services incrementally with monitoring.

---

Appendix — Horizontal Pod Autoscaler Keyword Cluster (SEO)

• Primary keywords
• Horizontal Pod Autoscaler
• Kubernetes HPA
• HPA scaling
• Horizontal scaling Kubernetes

• Kubernetes autoscaler

• Secondary keywords

• HPA metrics
• custom metrics HPA
• Prometheus HPA
• KEDA vs HPA

• HPA best practices

• Long-tail questions

• How does Horizontal Pod Autoscaler work in Kubernetes
• How to configure HPA for latency-based scaling
• Why is HPA not scaling pods
• HPA vs VPA differences and use cases

• How to prevent HPA oscillation

• Related terminology

• metrics server
• custom metrics API
• external metrics
• cluster autoscaler
• Prometheus adapter
• stabilization window
• scale policy
• minReplicas
• maxReplicas
• readiness probe
• liveness probe
• pod startup time
• replica set
• deployment scaling
• scale-to-zero
• warm pool
• event-driven scaling
• KEDA triggers
• cost-aware autoscaling
• predictive autoscaling
• anomaly detection autoscale
• node provisioning time
• pending pods
• replica churn
• SLI SLO mapping
• error budget
• runbook for autoscaling
• autoscaling governance
• RBAC for HPA
• adapter health checks
• recording rules for HPA
• PromQL for HPA metrics
• GitOps HPA
• canary HPA rollout
• chaos testing scaling
• scale down policies
• scale up policies
• observability for autoscaling
• dashboard for HPA
• alerting for scale events
• debug autoscaling
• API server HPA events
• HPA v2 features
• external metrics adapter
• Prometheus metrics for pods
• Kubernetes autoscaling ecosystem
• autoscaler incident postmortem
• HPA configuration checklist