Chapter 4

Troubleshooting and Optimization

AWS Certified Developer – Associate (DVA-C02) Study GuideBased on the official DVA-C02 exam objectives

Picture this: it is 2:13 AM, and your phone is buzzing. The checkout API for your company's online store is failing — one in every five orders is dying with an error, customers are complaining on social media, and the on-call dashboard is a wall of red. You open your laptop, still half asleep, and stare at the question every developer dreads: where do I even start? You start with Amazon CloudWatch , and within minutes the metrics tell a story: Lambda error rates spiked at 1:47 AM, right when a deployment finished. You switch to AWS X-Ray , and the service map shows the failing requests all dying inside one downstream call — a DynamoDB table whose read latency shot through the roof. The traces reveal the table's hottest partition key is getting hammered by a flash sale, and the Lambda functions are timing out waiting for it. Forty minutes later you have throttled the hot key, warmed a DAX cache, and rolled back the deployment. Orders are flowing again. That night contained this entire chapter. The DVA-C02 exam devotes 18% of its questions to what happens after code ships: how you observe it, how you find what broke, how you make it faster, how you make it cheaper, and how you design it so the next 2 AM page never comes. This chapter builds those instincts.

Seeing What Happened: Amazon CloudWatch

Amazon CloudWatch is AWS's monitoring service, and the first thing to understand is that it speaks two languages: metrics (numbers over time) and logs (text events). Metrics answer "how much, how fast, how often" — CPU utilization, request counts, error rates. Logs answer "what exactly happened" — the stack trace, the request ID, the line of code that threw. A metric lives in a namespace (like AWS/Lambda or AWS/DynamoDB), has a name (like Errors or Duration), and is sliced by dimensions — key-value pairs like FunctionName=my-checkout that let you filter. When you graph a metric you pick a statistic (Average, Sum, Maximum, p99) and a period (1 minute, 5 minutes). The exam loves the distinction between Sum of Errors (total failures — what you alert on) and Average of Duration (typical latency — what you optimize). Choose the wrong statistic and your alarm fires on noise or sleeps through a real incident. You can also publish your own numbers with custom metrics — say, OrdersPlaced per minute from your application code. The modern way to do it from Lambda is Embedded Metric Format (EMF) : you log a specially structured JSON blob, and CloudWatch extracts metrics from it automatically, without a separate API call. The exam may contrast EMF (cheap, asynchronous, no extra latency) with the PutMetricData API (direct but costs per call and adds latency to your code path). Metric math lets you combine metrics in the console — error rate is Errors / Invocations, not just raw errors. And composite alarms combine multiple alarms with AND/OR logic, so you can say "page me only when error rate is high AND traffic is normal" — suppressing the false alarm that fires during a quiet Sunday when three errors look like a 100% failure rate.

Alarms: Teaching CloudWatch to Wake You

A CloudWatch alarm watches one metric (or a math expression) and flips between three states: OK , ALARM , and INSUFFICIENT_DATA . That third state surprises people: it means there was not enough data to decide — common right after a deployment, or for a metric that only appears when errors happen. An alarm on a rarely-firing error metric will sit in INSUFFICIENT_DATA most of its life, and that is normal. Alarms evaluate over a number of datapoints within an evaluation period — the classic exam phrasing is "M out of N": alarm when 3 out of 5 datapoints breach the threshold. This is how you ignore a single blip but catch a real trend. When the state changes, the alarm can notify an SNS topic, trigger an Auto Scaling action, or even stop or reboot an EC2 instance.

Logs and Logs Insights

CloudWatch Logs collects log streams from Lambda, API Gateway, ECS, and anything using the agent. Logs live in log groups , and each group has a retention setting — this is a cost lever the exam tests: keeping every log forever is expensive, so production groups often retain 30 days while audit logs keep a year. CloudWatch Logs Insights is the query engine: a pipe-delimited language (fields, filter, stats, sort, limit) that scans log groups in seconds. A typical error-hunting query:

fields @timestamp, @message
| filter @message like /ERROR/
| stats count() by bin(5m)
| sort @timestamp desc

Following the Request: AWS X-Ray

Metrics told you something is wrong; AWS X-Ray tells you where. X-Ray is distributed tracing : it follows a single request as it hops from API Gateway to Lambda to DynamoDB to SQS, recording how long each hop took. The result is a trace — one request's full journey — made of segments (one per service) broken into subsegments (one per downstream call within a service). Open a trace and you see a waterfall timeline: the Lambda segment took 2,900 ms, and inside it the DynamoDB subsegment took 2,700 ms. You have just found your bottleneck without guessing. The service map zooms out further: a live diagram of all your services as nodes, with edges colored by health — red edges scream "the failures are between these two services." Two details the exam probes constantly. First, annotations vs. metadata : annotations are simple key-value pairs you add to a segment and they are indexed — you can search and filter traces by them:

from aws_xray_sdk.core import xray_recorder

subsegment = xray_recorder.begin_subsegment("dynamodb-query")
xray_recorder.put_annotation("customerId", "cust-4821")  # searchable
xray_recorder.put_metadata("query_plan", {"index": "GSI1"})  # not indexed
xray_recorder.end_subsegment()

When Lambda Misbehaves: Debugging Serverless

Lambda failures cluster around a handful of causes, and the exam tests each one. Timeouts. Every function has a timeout setting (maximum 15 minutes). When the code runs longer, Lambda kills it — and for asynchronous invocations, it retries twice. The classic scenario: a function calls a slow downstream API and the API Gateway integration timeout (29 seconds by default) fires first. The debugging rule: the timeout that fires is the shortest one in the chain. If API Gateway returns 504 after 29 seconds but your Lambda timeout is 60, the Lambda setting is irrelevant — API Gateway gave up first.

API Gateway: 429s, 502s, and 504s

API Gateway errors have distinct fingerprints. 429 Too Many Requests means throttling . Every API has throttle limits (default 10,000 requests/second steady-state with bursts) and you can tighten them per method with usage plans (rate + burst + quota per API key). When clients exceed the limit, they get 429 and should retry with exponential backoff . The exam scenario is almost always the same: "clients report intermittent 429s during a traffic spike" -> the answer involves throttling/usage plans, and the client-side fix is backoff, not "increase the limit" (though you can request a limit increase, the well-designed answer is usually throttling by design plus client retries). Caching sits in front of the integration: API Gateway caches responses per method, and clients can bust the cache by sending Cache-Control: max-age=0. Cache invalidation is per-stage and costs extra — the exam may ask what happens when cached data goes stale (answer: flush or wait for TTL). 502 Bad Gateway means the integration returned something malformed — the classic case is a Lambda proxy integration returning a response that is not the required {statusCode, headers, body} JSON shape. Debugging a 502? Check the Lambda's return format first. 504 we covered: the integration took longer than the timeout (29 seconds by default). And for access logging vs execution logging: execution logs (detailed, per-request, expensive — CloudWatch Logs) vs access logs (one line per request, like a web server log). Enable execution logging to debug; use access logs for audit.

DynamoDB Under Pressure

DynamoDB is fast until your access pattern fights its architecture. Hot partitions. Data is spread across partitions by partition key . If one key gets most of the traffic — the classic example is a status=ACTIVE key or a celebrity's user ID — that partition throttles while the rest sit idle. Symptoms: ProvisionedThroughputExceededException despite low average utilization. Fixes: choose a high-cardinality partition key, add a random suffix (write sharding), or use DynamoDB Accelerator (DAX) — an in-memory cache in front of the table that absorbs read heat. Note DAX's limits, which the exam tests: it caches reads (not writes), and DAX itself serves eventually consistent reads — a strongly consistent read bypasses DAX and goes straight to the table. Consistency models. Eventually consistent reads (the default, cheaper, half the RCUs) may return slightly stale data. Strongly consistent reads always return the latest write but cost double and are not supported on global secondary indexes. The exam scenario: "the application must read its own write immediately" -> strongly consistent reads. Adaptive capacity smooths uneven traffic automatically, but it is not magic — sustained hot-key traffic still needs a better key design. And on-demand vs. provisioned : on-demand absorbs spikes without planning (great for unpredictable traffic, pricier per request); provisioned with auto scaling is cheaper for steady, predictable workloads.

Caching Beyond DAX: ElastiCache Strategies

DAX only helps DynamoDB. For everything else — session data, rendered pages, leaderboard scores, database query results — the exam expects Amazon ElastiCache , the managed in-memory cache (Redis or Memcached). The troubleshooting angle is always the same: the database is the bottleneck, so put a fast layer in front of it. But which caching strategy matters, because the exam tests the tradeoffs. Lazy loading (cache-aside) is the default pattern: the application checks the cache first; on a miss, it reads the database, then writes the result into the cache for next time. Simple and the cache only holds what is actually requested — but the first request for each key always pays the database latency, and stale data sits in the cache until its TTL (time-to-live) expires. Write-through flips it: every write goes to the cache and the database together. Reads are always fast and never stale — but every write pays the cost of two operations, and the cache fills with data nobody may ever read. The exam scenario almost writes itself: "data changes infrequently but is read heavily" -> lazy loading with a TTL. "Data must never be stale on read" -> write-through. And when the question mentions a cache that must survive failover or persist, that is Redis (replication, persistence) over Memcached (simpler, multithreaded, no persistence).

CloudFront: The Developer's CDN

When your users are far from your origin — or when your origin is a Lambda function that should not serve the same image a million times — Amazon CloudFront sits in front as the CDN. For the developer-focused exam, three CloudFront concepts matter. Origins and behaviors. An origin is where CloudFront fetches content: an S3 bucket, an ALB, an API Gateway endpoint, even a Lambda function URL. Cache behaviors route by path pattern — /images/ can come from S3 with a long TTL while /api/ goes to API Gateway with caching disabled. The exam loves this split: static content cached at the edge, dynamic API calls passed through. Caching and invalidation. CloudFront caches at edge locations worldwide; the TTL controls how long. When content changes, you invalidate (/* for everything — the exam notes the first 1,000 invalidation paths per month are free, then it costs). Stale-content bugs in exam scenarios are almost always "the TTL is too long" or "they forgot to invalidate after deploying." Signed URLs and signed cookies. When content must be private — a paid video course, a pre-release download — CloudFront signed URLs (one URL, one user) or signed cookies (whole site section, one login) grant time-limited access without making the S3 bucket public. The origin stays private (via Origin Access Control); CloudFront checks the signature at the edge. The exam pattern: "restrict access to premium content to paying users" -> signed URLs/cookies, not bucket policies alone.

RDS Proxy: Connection Pooling for Lambda

Lambda's scaling model breaks traditional databases. Each concurrent Lambda execution opens its own database connection, and a sudden spike — 1,000 concurrent functions — means 1,000 connections hitting RDS at once. RDS has a max_connections limit; exceed it and new connections are refused. Worse, opening a connection is slow (TLS handshake, authentication), adding latency to every cold function. Amazon RDS Proxy sits between your functions and the database as a managed connection pool . Your Lambdas connect to the proxy; the proxy multiplexes those onto a small pool of real database connections, reusing them across executions. Two exam-tested benefits: it absorbs connection storms (thousands of Lambda connections become dozens of database connections), and it reduces failover time (the proxy holds the pool while RDS fails over, instead of every function reconnecting). The exam scenario is distinctive: "Lambda functions intermittently fail to connect to RDS during traffic spikes" or "database shows too many connections errors after moving to serverless" -> RDS Proxy. It works with RDS and Aurora (MySQL and PostgreSQL engines), and as a bonus it can enforce IAM authentication to the database instead of passwords in Secrets Manager.

SQS: The Quiet Workhorse's Failure Modes

SQS problems are almost always about the visibility timeout . When a consumer receives a message, it becomes invisible for the timeout duration (maximum 12 hours). If the consumer finishes processing, it deletes the message. If processing takes longer than the timeout, the message becomes visible again and another consumer picks it up — the same work happens twice. The rule: visibility timeout must exceed your worst-case processing time. The exam's favorite symptom: "messages are processed twice" -> timeout too short. When a message fails repeatedly, a redrive policy moves it to a dead-letter queue after maxReceiveCount attempts. DLQs are the answer to "poison messages must not block the queue forever." Long polling vs. short polling. Short polling returns immediately, even if empty — you pay for empty receives. Long polling waits up to 20 seconds for a message to arrive — fewer empty responses, lower cost, lower latency. Long polling is almost always the right answer; short polling exists mainly for legacy reasons. FIFO queues preserve ordering (within a message group) and deduplicate (5-minute deduplication window), at lower throughput than standard queues. If the scenario demands "exactly-once processing in order," FIFO is the answer — with the caveat the exam sometimes probes: FIFO gives exactly-once deduplication, and your consumer should still be idempotent.

Spending Less: Cost Optimization for Developers

Developers control more of the AWS bill than they think. Lambda right-sizing we covered under memory tuning: the cheapest configuration is rarely the smallest memory, because CPU scales with memory. Measure with Power Tuning, then pick the knee of the curve. S3 storage tiers are the exam's favorite cost topic for developers storing objects: Standard (frequent access), Intelligent-Tiering (unknown patterns — moves objects automatically), Standard-IA / One Zone-IA (infrequent), Glacier Instant/Flexible/Deep Archive (archival, retrieval in ms/minutes/hours). Lifecycle policies automate the transitions — "move to Glacier after 90 days, expire after 7 years" — and the exam tests that lifecycle rules, not manual moves, are the answer to "automatically archive old objects." Also remember: S 3 data transfer out to the internet costs money; keeping traffic inside AWS (same region) avoids it. Log retention is a sleeper cost: CloudWatch Logs charges per GB ingested and per GB-month stored. Never-log-everything-forever is a cost decision, not just hygiene.

Building for Failure: Resiliency Patterns

The last piece of the domain is designing so failures do not cascade. Retries with exponential backoff and jitter. When a call fails transiently (throttling, brief network blip), retry — but wait longer each time (1s, 2s, 4s…), and add jitter (randomness) so a thousand clients do not all retry in the same millisecond and recreate the spike. AWS SDKs do this by default; the exam expects you to know the pattern name and why jitter matters. Idempotency. Because retries can deliver twice (remember at-least-once delivery), the receiving code must tolerate duplicates: processing the same request twice must have the same effect as once. Techniques: conditional writes ("only write if this order ID is not already recorded"), idempotency keys, FIFO deduplication. The exam scenario: "duplicate deliveries must not double-charge the customer" -> idempotent design. Circuit breaker. When a downstream service is failing, stop calling it for a while instead of piling timeouts onto an outage. The breaker has three states — closed (normal), open (failing fast without calling), half-open (testing recovery with a trickle). The exam contrasts this with retries: retries help with transient blips; circuit breakers protect against sustained outages. Step Functions bakes all of this in: per-state Retry (with backoff config) and Catch (fallback paths) — the exam's answer to "orchestrate multi-step workflows with automatic retry handling."

[/EXAM]

End-of-Chapter Test

1. A developer's API receives 50,000 requests per minute. CloudWatch shows the Lambda Errors metric spiking, but the team cannot tell whether it is a real incident or normal background failures. Which approach gives the most reliable alarm? A) Alarm on the raw Errors metric with a threshold of 10 B) Alarm on a metric-math expression of Errors / Invocations with M-out-of-N evaluation C) Create a CloudWatch dashboard showing the Errors graph D) Enable X-Ray tracing on the function

2. An X-Ray trace shows a request spending 2,400 ms in a DynamoDB subsegment out of 2,600 ms total. The developer needs to find all other traces from the same customer to check whether the problem is widespread. Which X-Ray feature makes this possible? A) Annotations, because they are indexed and searchable B) Metadata, because it stores rich nested objects C) The service map, because it shows all services D) Sampling rules, because they capture every trace

3. A company sells premium video courses. Videos are stored in a private S3 bucket, and only paying customers may watch them. The site must handle global viewers with low latency. Which design meets all requirements? A) Make the S3 bucket public and rely on unguessable object URLs B) Generate S3 presigned URLs and serve traffic directly from the bucket C) Put CloudFront in front of the bucket with Origin Access Control and use signed URLs for paying customers D) Replicate the bucket to every region with Cross-Region Replication

4. After migrating an order API to Lambda, the team sees intermittent too many connections errors from RDS during traffic spikes. Each Lambda execution opens its own database connection. What is the best fix? A) Increase the RDS instance size to allow more connections B) Reduce Lambda reserved concurrency so fewer functions run at once C) Store database credentials in Secrets Manager with rotation D) Place RDS Proxy between the Lambda functions and the database

5. A DynamoDB table using provisioned capacity throws ProvisionedThroughputExceededException during a flash sale. Table-level CloudWatch metrics show only 40% average utilization. What is the most likely cause and the best fix? A) A hot partition key is throttling; add DAX or redesign the partition key B) The table needs more RCUs; increase provisioned read capacity C) Switch the table to on-demand capacity mode D) Enable DynamoDB Streams on the table

6. A Lambda function writes 2 GB of logs per day to CloudWatch, and the team keeps all logs indefinitely. The AWS bill shows growing log storage charges. What is the most appropriate cost action that preserves debuggability? A) Disable CloudWatch Logs for the function entirely B) Delete the log group every night with a scheduled Lambda C) Set a log retention policy (e.g., 30 days) on the log group D) Move the logs to S3 Standard storage

7. A product catalog changes once a day but is read thousands of times per second. The team adds an ElastiCache Redis cluster. Which caching strategy fits best, and why? A) Write-through, because every write must reach the database instantly B) No caching; the database can handle the read load directly C) Cache the entire database on every deployment D) Lazy loading with a TTL, because reads dominate and brief staleness is acceptable

8. An API Gateway REST API returns 502 errors for one endpoint backed by a Lambda proxy integration. Other endpoints work fine. CloudWatch Logs show the Lambda executed successfully. What is the most likely cause? A) The Lambda function's IAM role is missing DynamoDB permissions B) The Lambda response is not in the required proxy-integration format (statusCode, headers, body) C) API Gateway throttling is rejecting the requests D) The Lambda function's timeout is too short

9. A developer enables X-Ray tracing on a high-traffic API. Sampling is set to 5%, but the team notices they still see traces for every failed request. Why? A) The sampling rule is misconfigured and recording 100% of traffic B) The X-Ray reservoir guarantees a fixed number of traces per second are always recorded, capturing the errors C) Failed requests bypass sampling rules automatically D) Annotations force all error traces to be recorded

10. An SQS standard queue consumer processes messages that occasionally take 45 seconds. The visibility timeout is set to the default 30 seconds, and the team notices some orders are processed twice. What is the fix? A) Enable FIFO on the queue for exactly-once processing B) Increase the visibility timeout to exceed the worst-case processing time (e.g., 60+ seconds) C) Decrease the visibility timeout so messages return to the queue faster D) Add a dead-letter queue with maxReceiveCount of 1

11. A team creates a CloudWatch alarm on a Lambda error metric that only receives data points when errors occur. Most of the time the alarm sits in INSUFFICIENT_DATA state. A junior developer proposes treating INSUFFICIENT_DATA as an alarm condition so the team is notified. What is the correct response? A) Agree — INSUFFICIENT_DATA means the monitoring is broken and needs attention B) Disagree — INSUFFICIENT_DATA means there was not enough data to evaluate, which is normal for sparse metrics; only the ALARM state triggers actions by default C) Agree — but only if the metric has fewer than 3 data points in the evaluation period D) Disagree — the alarm should be deleted and replaced with a composite alarm

Answer Key

1. B — Error rate (Errors / Invocations) normalizes against traffic, so a spike means something real regardless of volume; M-out-of-N evaluation ignores single-blip noise. Why not A? A raw count threshold cannot distinguish 10 background errors from 10 errors out of 10 requests — it pages on noise or sleeps through incidents depending on traffic. Why not C? Dashboards visualize; they do not evaluate or notify. Why not D? X-Ray traces individual requests — the right tool for diagnosis, not for alarming. 2. A — Annotations are indexed key-value pairs, so traces can be searched and filtered by them (e.g., customerId). Why not B? Metadata holds richer data but is not indexed — it cannot be used to find traces. Why not C? The service map shows service topology and health, not per-request attributes. Why not D? Sampling rules control which traces are recorded, not how they are searched. 3. C — CloudFront caches the videos at edge locations for global low latency; Origin Access Control keeps the bucket private; signed URLs grant time-limited access to paying customers only. Why not A? A public bucket exposes the videos to everyone — security by obscurity is not access control. Why not B? Presigned URLs solve the access problem but traffic still comes from one regional bucket — no global caching. Why not D? Replication copies data but does not cache at the edge or restrict access to paying users. 4. D — RDS Proxy pools the Lambda functions' connections, multiplexing thousands of short-lived function connections onto a small set of real database connections. Why not A? A bigger instance raises the connection ceiling but does not fix the architectural problem — Lambda's scale will still exhaust it. Why not B? Throttling Lambda concurrency protects the database but sacrifices the API's ability to scale. Why not C? Secrets rotation is good hygiene but has nothing to do with connection counts. 5. A — Low average utilization with throttling errors is the textbook hot-partition signature: one partition key absorbs the flash-sale traffic and hits its per-partition limit. DAX absorbs the read heat, or a better key design spreads it. Why not B? More total RCUs do not raise the per-partition limit — the hot key still throttles. Why not C? On-demand also enforces per-partition limits and would not fix a hot-key access pattern. Why not D? DynamoDB Streams captures change records; it does nothing for read throttling. 6. C — A retention policy (e.g., 30 days) automatically expires old logs, capping storage cost while keeping recent logs debuggable. Why not A? Disabling logs entirely destroys all observability — an unacceptable tradeoff. Why not B? Manually deleting the group nightly is fragile operational overhead; retention policies do this natively. Why not D? CloudWatch Logs cannot be "moved" to S3 Standard as a storage tier — you can export to S3, but the native answer is retention. 7. D — With reads dominating and daily changes, lazy loading fills the cache only with requested keys and a TTL bounds staleness to an acceptable window; writes stay cheap. Why not A? Write-through is for data that must never be stale on read — overkill here, and every catalog update would pay double. Why not B? Thousands of reads per second against the database is exactly the bottleneck caching solves. Why not C? Pre-loading the whole catalog on every deployment wastes cache space on unrequested keys and adds deploy-time complexity. 8. B — With a Lambda proxy integration, API Gateway requires the specific {statusCode, headers, body} response shape; anything else yields a 502 even when Lambda "succeeded." Other endpoints working points to this endpoint's response format. Why not A? Missing IAM permissions would cause the Lambda itself to fail (and CloudWatch would show the error), not a 502 after success. Why not C? Throttling returns 429, not 502. Why not D? A short timeout would produce 504s, and the logs show successful completion. 9. B — The reservoir records a guaranteed number of traces per second regardless of the percentage rate — so rare errors are never missed even at low sampling rates. Why not A? The 5% rate applies to additional traffic beyond the reservoir; it is working as designed. Why not C? There is no automatic bypass — the reservoir mechanism is what captures the errors. Why not D? Annotations make traces searchable; they do not affect sampling decisions. 10. B — When processing exceeds the visibility timeout, the message becomes visible again and another consumer picks it up — the classic duplicate-processing cause. The timeout must exceed worst-case processing time. Why not A? FIFO gives ordering and deduplication, but the root cause is the timeout — and FIFO alone does not fix a too-short visibility window. Why not C? A shorter timeout makes duplicates more frequent, not less. Why not D? A DLQ catches poison messages after repeated failures; it does not prevent timeout-driven duplicates. 11. B — INSUFFICIENT_DATA is a normal state meaning "I can't tell" — common for error metrics that only appear when errors happen. Only the ALARM state triggers actions by default. Why not A? Treating "no data" as an error would page the team constantly during healthy periods — the opposite of useful alarming. Why not C? The data-point count does not change what the state means. Why not D? A composite alarm combines multiple alarms with AND/OR logic; it does not fix a misunderstanding of alarm states.