Chapter 4
Troubleshooting and Optimization
definitions, Step Functions state machines) and applies them directly through service APIs, skipping
CloudFormation. The trade-off is drift from CloudFormation's recorded state, which is why it is a development-only tool. --method=direct still performs a full CloudFormation deployment, sam deploy does not deploy CDK apps, and bootstrapping is a one-time environment setup.
6. C - Only a manual approval action actually pauses a pipeline execution until a person approves or rejects it
(and it can notify via SNS). The AI summary helps the reviewer but enforces nothing; a test action cannot judge review status, rollback undoes a deployment rather than gating it, and an SNS subscription by itself does not block the stage transition.
7. B - Feature flags are built for exactly this: gradual exposure via a deployment strategy, and alarm-driven automatic rollback (plus the ability to flip the flag off immediately). A freeform profile with AllAtOnce gives no gradual rollout, manual parameter edits have no rollout or rollback machinery, and secret rotation is about credentials, not feature exposure.
8. C - Equal increments at fixed intervals is the definition of a linear deployment, and the Lambda-scoped name matches the target platform. Options A and D describe canary (single step) and the ECS-scoped linear configuration respectively; option B shifts everything immediately.
Domain 4 grew from 18% of the exam in DVA-C02 to 22% in DVA-C03, and that shift tells you something about how AWS sees the developer role: writing the code is only half the job. The other half is proving what the code did in production, finding the hop where a request failed, and tuning the system once you understand it. This chapter builds those skills around the three pillars of AWS observability - CloudWatch for logs and metrics, X-Ray for traces, CloudTrail for the audit record - then works through the failure patterns the exam returns to again and again, and finishes with the optimization levers and the AI-assisted tooling now layered on top.
A note on method: troubleshooting here is mostly correlation - one request leaves a log line, metrics, a trace, and a CloudTrail entry for any API call it made. Read them in order: symptom (metrics), request path (traces), detail (logs), "who changed what" (audit trail).
The Observability Toolkit at a Glance
| Tool | CloudWatch Logs | CloudWatch Metrics | CloudWatch Logs Insights |
|---|---|---|---|
| What it records | Log events your code and AWS services emit | Time-series numbers (counts, latencies, errors) | Interactive queries over log data |
| Question it answers | "What did the application say happened?" | "Is something wrong, and since when?" | "Show me the slowest / failing / most frequent pattern" |
| Retention to know | You set it per log group: fixed periods from 1 day up to 10 years, or never expire | 15 months, at progressively coarser resolution | Queries run over whatever logs the retention policy keeps |
| Tool | What it records | Question it answers | Retention to know | |||
|---|---|---|---|---|---|---|
| AWS X-Ray | Traces: segments and | "Where did the request spend its | Traces are kept for 30 days | |||
| subsegments for each request | time, and which hop failed?" | |||||
| AWS | API activity in the account (who, | "Who or what called this API?" | Event history covers 90 days; trails | |||
| CloudTrail | what, when, from where) | and CloudTrail Lake extend this |
Hold onto that table's core distinction: why did the application behave this way - logs and traces; who performed this action - CloudTrail. Scenarios profit from confusing the two.
Amazon CloudWatch Logs
Log groups, log streams, and retention
Every Lambda function automatically gets a log group named /aws/lambda/<function-name>, with log streams separated by date and execution environment. Other workloads - containers using the awslogs driver, applications using the CloudWatch agent or an SDK - follow the same model: a log group is the container you apply settings to, a log stream is a sequence of events from one source, and a log event is a single timestamped record (up to 1 MB).
Retention is set on the log group, and the default is never expire - which is why "log group retention" is the cost-control lever in exam questions. Fixed periods run from 1 day to 10 years. Three operational facts matter for troubleshooting:
- Changing retention affects existing data immediately: shortening it deletes older events, so "last month's
logs are gone" is often a retention change, not an outage. And log groups are Region-scoped - "no logs"
frequently means the wrong Region or account.
- Lambda writes through its execution role. A missing
logs:CreateLogGroup,logs:CreateLogStream, or
logs:PutLogEvents permission means the function still runs - just silently. Check the execution role whena function produces no output.
For moving log data elsewhere, subscription filters stream events out of a log group in near real time to a
Lambda function, a Kinesis data stream, or Amazon Data Firehose (for delivery to S3, OpenSearch, and other destinations). A log group can have up to two subscription filters.
Metric filters: turning log events into metrics
A metric filter watches a log group, matches incoming events against a pattern, and publishes a CloudWatch metric on each match - how you alarm on something no AWS service measures for you, such as occurrences of a specific application error code.
Key behaviors to internalize:
- Filters come in two dialects. Term/space-delimited patterns match plain text (the word
ERROR, or
positional fields such as a 5xx status code). JSON patterns match properties inside JSON log events, for example { $.errorCode = "AccessDenied" }.
- A filter can extract a numeric value from the event - a latency field, a byte count - and publish that as the
metric value rather than a simple count of 1.
- You can assign a default value published when the pattern does not match, so the metric reports
continuously (zero errors produces a 0 data point instead of a gap).
- Filters apply only to events ingested after the filter is created; they never backfill. If a scenario needs last
week's error counts from a filter created yesterday, the answer is a Logs Insights query over the retained logs.
- The output is an ordinary CloudWatch metric, so dashboards, alarms, and anomaly detection all work on it
unchanged.
"Create an alarm when the application logs a specific error" always decomposes into two steps: a metric filter to convert matching log events into a metric, then a CloudWatch alarm on that metric. An alarm cannot read log text directly.
Metric filter. A CloudWatch Logs pattern rule that watches a log group and publishes a CloudWatch metric whenever an incoming event matches the pattern. It matters because alarms can only act on metrics, so a metric filter is the bridge that turns application log text - an error string, a status code, a latency value - into something you can alarm and dashboard on.
CloudWatch Logs Insights queries
Logs Insights is the interactive query engine over log groups - the tool for ad-hoc investigation: aggregating, filtering, parsing fields out of unstructured text, and ranking results. A query runs across up to 50 log groups at a time and returns up to 1,000 rows by default (configurable to 10,000).
The core commands, in the order you will usually write them:
| Command | fields |
|---|---|
| filter | parse |
| stats | Purpose |
| Choose which fields to display or work with | Keep only events matching a condition (like, comparisons, regex) |
| Extract new fields from unstructured text using a pattern or regex | Aggregate - count(), sum(), avg(), max(), percentile functions such as pct() - optionally with by grouping and bin() time buckets |
| Command | Purpose | |
|---|---|---|
| sort | Order results, e.g. sort @duration desc | |
| limit | Cap the number of rows returned | |
| dedup / display | Remove duplicate rows / shape which fields are shown |
Lambda log groups get discovered fields such as @timestamp, @message, @requestId, and on the REPORT line, @duration, @billedDuration, @memorySize, and @maxMemoryUsed. A canonical troubleshooting query - the slowest recent invocations - looks like this:
fields @timestamp, @requestId, @duration, @maxMemoryUsed, @memorySize
| filter @type = "REPORT"
| sort @duration desc
| limit 20Two habits separate effective Logs Insights work from flailing: bound the time range and start from an aggregate (stats - for example, error counts with stats count(*) by bin(5m)) before drilling into raw events; and use parse when a needed field is trapped inside a text message, since parsed fields can then be filtered and aggregated like native ones.
The embedded metric format (EMF)
Metric filters extract metrics from logs after the fact. The embedded metric format is the opposite arrangement: your application deliberately writes a structured JSON log line containing both the log data and a metrics directive, and CloudWatch Logs extracts the metric automatically on ingestion. There is no
PutMetricData call in the code path - the log line is the metric publication.An EMF event carries a _aws metadata block naming the timestamp, the namespace, the dimension sets, and which top-level properties are metrics:
{
"_aws": {
"Timestamp": 1718042400000,
"CloudWatchMetrics": [
{
"Namespace": "CheckoutService",
"Dimensions": [["Service", "Endpoint"]],
"Metrics": [{ "Name": "OrderLatency", "Unit": "Milliseconds" }]
}
]
},
"Service": "checkout",
"Endpoint": "/orders",
"OrderLatency": 187
}Facts the exam leans on:
- A single EMF event can carry up to 100 metrics, and each dimension set can include up to 30 dimension
keys - making EMF the natural fit for high-cardinality dimensions (per tenant, per endpoint, per feature flag) that would be impractical to publish one API call at a time.
- Properties named in the directive become metrics or dimensions; every other property stays in the log
event as searchable context, so you get the metric and the log line behind it.
- EMF works anywhere you can write to CloudWatch Logs - the AWS Embedded Metrics libraries (and
Powertools for AWS Lambda's Metrics feature) generate the format for you - and because publication rides on log ingestion, there is no separate metrics API to throttle and no added call in a hot code path.
Choosing between the two: metric filters retrofit metrics onto logs you already emit; EMF is a design-time choice for code you own, when you want rich custom metrics with dimensions from day one.
Common trap: An EMF event with a malformed _aws block is stored as an ordinary log event and silently produces no metric - suspect a wrong namespace, or a directive metric name that matches no property in the event.
CloudWatch Metrics and Alarms
Logs tell you what happened; metrics tell you whether it is still happening and how bad it is. Every service here publishes metrics under a namespace - AWS/Lambda, AWS/ApiGateway, AWS/DynamoDB, AWS/SQS - and the troubleshooting vocabulary starts with knowing the key metric names cold:
| Service | Lambda |
|---|---|
| Metrics that diagnose most incidents | Invocations, Errors, Duration (avg/max), Throttles, ConcurrentExecutions, IteratorAge for stream triggers |
| Service | |
|---|---|
| API Gateway | DynamoDB |
| SQS | |
| Metrics that diagnose most incidents | Count, Latency, IntegrationLatency, 4XXError, 5XXError, CacheHitCount / CacheMissCount |
| ConsumedReadCapacityUnits, ConsumedWriteCapacityUnits, ReadThrottleEvents, WriteThrottleEvents, SuccessfulRequestLatency | ApproximateNumberOfMessagesVisible, ApproximateAgeOfOldestMessage, NumberOfMessagesSent, NumberOfEmptyReceives |
The most useful diagnostic habit here is comparing the pair that splits a problem at the service boundary. For
API Gateway, Latency is end-to-end client time while IntegrationLatency is only the wait for the backend: if the two track together, the backend is slow; if Latency soars while IntegrationLatency stays flat, the problem sits in API Gateway or the network path. (The Lambda playbook below leans on the same kind of split, between Errors and Throttles.)
An alarm watches one metric (or a metric math expression, or a Logs Insights query result) over an evaluation window and transitions between OK, ALARM, and INSUFFICIENT_DATA. The vocabulary that appears in scenarios:
- Period, statistic, threshold, and datapoints to alarm - for example, average
Durationover 5 minutes
exceeding 3,000 ms, breaching in 3 of the last 5 periods (which smooths over single spikes; "1 out of 1"
pages you on noise). Percentile statistics (p90, p99) are usually fairer than averages for latency, because averages hide a suffering minority of requests.
- Treat missing data - what an absent data point means:
breaching(gaps are bad - right for
heartbeat-style metrics), notBreaching (gaps are fine), ignore (keep the previous state), or missing (fall back to INSUFFICIENT_DATA). Scenarios about alarms that "never fire even when the service stops reporting" test exactly this setting.
- Composite alarms - combine other alarms with AND/OR logic so a page means "errors high AND latency
high," not two noisy alerts.
- Anomaly detection - CloudWatch models a metric's expected range (accounting for daily/weekly
seasonality) and alarms when it leaves the band; useful when nobody knows a sensible static threshold.
Practically, an alarm is the start of the workflow below: it fixes when the problem began, the timestamp every log query and trace search should anchor to. (Its actions route to SNS, Lambda, Auto Scaling, or Systems
Manager.)
1. A service publishes a heartbeat metric every minute. It crashes overnight, the metric stops arriving - and the alarm never leaves OK, so nobody is paged. Which alarm setting is wrong?
2. A metric filter created today shows no error counts for last week, even though the log group holds a month of events. Why - and what produces last week's counts instead?
Answers: 1. Treat missing data is set to notBreaching, so the gaps count as healthy; set it to breaching for heartbeat-style metrics. 2. Metric filters never backfill - they only match events ingested after creation; run a
Logs Insights query over the retained logs for the historical counts.
AWS X-Ray: Distributed Tracing
Metrics can tell you the checkout service got slow at 14:03; they cannot tell you the slowness lives in a sub-call three services downstream. That is the job of distributed tracing - AWS X-Ray.
Segments and subsegments
A trace is the end-to-end record of one request as it crosses services. Each service contributes a segment: a
JSON document recording the service name, start and end times, inbound HTTP details, AWS resource details, and three flags that drive most analysis - error (4xx-class), fault (5xx-class), and throttle.
Within a segment, subsegments break the work down further. The SDK automatically creates subsegments for downstream calls it instruments - HTTP requests, AWS SDK calls to DynamoDB, SQS, and so on - and you can open custom subsegments around your own code blocks to time a pricing calculation or a call to a legacy system. When the exam shows a slow request and asks where the time went, picture the trace timeline: a segment with nested subsegments, each carrying its own duration.
Supporting facts worth knowing:
- API Gateway supports active tracing (it samples and starts traces itself) and passive tracing (it only
participates when an upstream service already started a trace).
- Lambda does not run the X-Ray daemon - the Lambda service hosts the listener. The execution role needs
X-Ray write permissions, and tracing is enabled per function.
- On EC2, ECS, and similar compute, the X-Ray daemon (UDP port 2000) or the AWS Distro for OpenTelemetry
(ADOT) collector receives segment data from your application and forwards it to X-Ray; ADOT is the modern path.
- A missing local segment raises a context-missing error; setting
AWS_XRAY_CONTEXT_MISSINGto
LOG_ERROR converts that crash into a log line - the classic fix when the SDK throws exceptions in a servicereceiving untraced requests.
Segment (X-Ray). The JSON record one service contributes to a trace, capturing its name, start and end times, inbound request details, and the error, fault, and throttle flags. It matters because the segment timeline
- with nested subsegments for each downstream call - is where you see exactly which hop consumed the time or returned the error.
Sampling rules
Tracing everything is expensive and mostly unnecessary, so X-Ray samples: it records a subset of requests chosen by rules. Every rule combines a reservoir - a guaranteed number of requests per second that are always traced - with a fixed rate applied to the remainder. The default rule is a reservoir of 1 request per second plus
5% of additional requests, which guarantees at least one trace per second from any active service while keeping volume low.
Custom rules target a subset of traffic by attributes such as service name, service type, host, HTTP method, and
URL path, and are evaluated in priority order (lower number wins, 1 to 9,999, default rule last). Two behaviors matter operationally:
- Rules are managed centrally in X-Ray and picked up by SDKs without redeploying code - raising the rate
on a failing endpoint mid-incident is a console change, not a release.
- Sampling is parent-based: the decision is made once, by the first instrumented service to see the request,
and downstream services honor it. A 100% rule on an internal service never fires for requests whose upstream service already declined to sample. Put investigative rules on the root service - or on genuine new trace roots such as asynchronous workers.
The sampling decision is made once, by the first instrumented service that sees the request, and every downstream service honors it. Students often put a 100% sampling rule on the internal service they suspect, then wonder why no traces appear - if the upstream root already declined to sample those requests, the downstream rule never gets a chance to fire.
Annotations versus metadata
Both attach key-value data to segments and subsegments, and the exam tests the difference relentlessly:
| Annotations | Metadata | |||
|---|---|---|---|---|
| Indexed for | Yes - filter expressions can query them | No - not searchable | ||
| search |
| Allowed types | ||
|---|---|---|
| Intended use | Limit discipline | |
| Annotations | Strings, numbers, booleans only | The few values you find traces by: order ID, tenant, outcome |
| Keep them few (max 50 per segment) | Metadata | |
| Arbitrary JSON, including nested objects | Rich context read after finding the trace: payloads, stack traces | Can be large and free-form |
Trace search uses filter expressions over indexed fields and annotations - all faulted traces for one service, or
annotation.orderId = "A123" for every trace touching one order. The rule: "search traces by X" means X is anannotation; "store the full payload for later inspection" means metadata.
The service map and trace analysis
The service map renders every traced service as a node and every observed call path as an edge, annotated with latency and error/throttle rates - in incident terms, "which node turned red." The workflow: start at the map, find the faulting node, open its traces filtered to faults, and read the segment timeline down to the failing subsegment, which typically names the exact downstream call and error. Groups save a filter expression as a named lens over traces; X-Ray Insights raises events with a root-cause summary when trace behavior deviates from baseline. Traces are retained for 30 days; older comparisons belong in logs and metrics.
Common trap: X-Ray shows sampled requests only. A clean service map does not prove an error is absent - at a
5% rate, a one-in-a-thousand failure may never have been sampled. Raise the rate (or check Logs Insights, which sees everything) before concluding the problem is elsewhere.
AWS CloudTrail: Auditing API Activity
CloudTrail records the API calls made in your account - by users, roles, and AWS services - with caller identity, source IP, timestamp, request parameters, and any error code returned. It is the tool that converts "the table is gone" or "this started right after a deploy" into a specific principal and a specific call.
Events come in three kinds:
- Management events - control-plane operations such as
CreateFunction,CreateTable,
UpdateFunctionConfiguration, and PutBucketPolicy. These are recorded by default and are the firstplace to look for configuration changes.
- Data events - object- and item-level operations such as S3
GetObject/PutObject, LambdaInvoke, and
DynamoDB item activity. These are not recorded by default; you opt in per resource because the volume
(and cost) is high. This is the most tested CloudTrail fact in developer scenarios: "which process read this S3 object" requires data events to have been enabled in advance - they cannot be reconstructed afterwards.
- Insights events - when enabled, CloudTrail analyzes management events for unusual API call volume or
error rates and records an insight: an early warning for runaway automation or a bad deployment.
Access paths matter too. The event history (console, or LookupEvents) covers the last 90 days of management events with no setup. A trail delivers events to S3 for durable retention, can be organization-wide and all-Regions, and can stream to CloudWatch Logs for alarming on specific calls (alert whenever DeleteTable occurs, say); log file validation adds digest files proving delivered logs were not altered. CloudTrail Lake is the managed alternative: an event data store with configurable retention (up to 10 years), queried with SQL.
In practice, developers use it three ways: blast-radius checks (filter event history by the deploy window to see every configuration change, in order, with its identity); explaining an AccessDenied (the denied call appears with the exact error code, principal, and resource - far faster than guessing at policies); and correlation (take a timestamp from a log entry, then find that role's API calls in the same minute).
Common trap: CloudTrail records API calls, not application behavior. A function that miscalculates a price leaves no CloudTrail trace of the miscalculation - only of the Invoke and the API calls it made. The calculation itself is in CloudWatch Logs.
CloudTrail and CloudWatch Logs answer different questions, and scenarios profit from blurring them.
CloudTrail records API calls - who did what to a resource, and when - while logs and traces record what the application itself did with a request. A bug inside your code leaves evidence in logs; a configuration change leaves evidence in CloudTrail; most real incidents need both, correlated by timestamp.
Root-Cause Analysis Playbooks
The sections above are instruments; this section is the flying - the five failure patterns that dominate real incidents and Domain 4 scenarios alike. Each follows the same discipline: read the symptom precisely, check the cheapest discriminating metric first, confirm in logs or traces, then change something.
Playbook: Lambda timeouts and throttling
Lambda produces two similar complaints - "requests are failing" - with opposite fixes, so separating them is step one. A timeout means the invocation started and ran past the configured timeout (maximum 15 minutes); a throttle means Lambda refused to start it at all because a concurrency limit was reached, returning a 429 with a TooManyRequestsException ("Rate Exceeded").
Symptom Likely cause Fix
| Symptom | Log line Task timed out after X.XX seconds; Duration metric hugs the timeout value | |
|---|---|---|
| Errors metric flat but Throttles climbing; callers receive 429s | Sudden 429s during a traffic spike, average concurrency low | Latency and timeouts only on the first request after idle periods |
| Timeouts clustered on calls to one database or HTTP API, code unchanged | Likely cause | |
| Function genuinely runs long - slow query, unresponsive downstream call, too little memory/CPU. X-Ray subsegments show where the time went | Concurrency limit reached: the account limit, or reserved concurrency set too low. Compare ConcurrentExecutions with the quota | Lambda's scaling rate: each function can add up to 1,000 execution environments every 10 seconds, so a near-vertical ramp can briefly outrun it |
| Cold starts. Signature: high Init Duration on the REPORT line, initialization subsegment in X-Ray | Downstream degradation, or connection setup overhead (new connections per invocation). X-Ray shows the time inside the downstream subsegment | |
| Fix | Keep downstream client timeouts below the function timeout, optimize the slow segment, raise memory for CPU-bound work, or raise the function timeout if the work is legitimately long | Request an account limit increase; raise or remove the reserved cap; smooth traffic with an SQS queue in front |
| Provision concurrency ahead of predictable peaks; buffer with SQS | Reduce package size and init work; SnapStart for supported Java runtimes, or provisioned concurrency | Enable keep-alive / connection reuse, create clients outside the handler, add retries with backoff and jitter, set per-call timeouts |
Behind API Gateway, a function with a 60-second timeout can still produce client failures at ~29 seconds: the
REST API integration timeout defaults to 29 seconds, so the client gets a 504 while the function runs on.
(Raising it via Service Quotas works for Regional/private APIs, not edge-optimized ones; the alternative is an asynchronous pattern returning 202.)
Reserved concurrency. A slice of the account concurrency pool set aside for one function, which doubles as that function's maximum. It matters in troubleshooting because it cuts both ways: set deliberately, it fences a runaway function away from the rest of the Region; set too low, it is the hidden cause of 429 throttles while the account pool sits idle.
Playbook: DynamoDB hot partitions and throttling
DynamoDB spreads a table's provisioned capacity across its partitions, and each partition has hard physical ceilings: roughly 3,000 read capacity units and 1,000 write capacity units per second per partition.
Throughput trouble therefore comes in two flavors that look identical at the API -
ProvisionedThroughputExceededException - but differ completely in distribution:- Table-level throttling: total demand exceeds the table's (or index's) provisioned capacity, or bursts outrun
the roughly 5 minutes of unused capacity DynamoDB retains for bursting. Consumed capacity sits at or above provisioned levels across the board.
- Hot partition throttling: the table has spare capacity, but traffic concentrates on one partition key value -
a celebrity user, a single "today" partition. One partition hits its ceiling while the rest idle. This is a key design problem, not a capacity problem, and raising table capacity does nothing for it.
Discrimination and fixes:
| Observation | ReadThrottleEvents / WriteThrottleEvents > 0 with consumed capacity near provisioned | Throttle events with consumed capacity well below provisioned |
|---|---|---|
| Writes throttle when a GSI is added or a GSI's keys are skewed | Throttling on a recently switched on-demand table during a spike | Read latency and cost high for a few extremely popular items |
| Interpretation | Genuine under-provisioning or a burst | Hot key or hot partition |
| GSI back-pressure: if a global secondary index cannot absorb the write rate, writes to the base table throttle too | On-demand adapts instantly up to about double the previous peak; faster growth can throttle until capacity catches up | Read hot spot |
| Fix | Raise provisioned capacity / enable auto scaling, or switch to on-demand | Redesign the partition key for spread; add a random/write-sharding suffix to hot keys and scatter-gather reads; use Contributor Insights to identify the offending key values |
| Provision the GSI for the table's write rate; choose GSI partition keys with high cardinality | Ramp load gradually before launch events; consider provisioned with auto scaling for extreme, abrupt spikes | DAX (below) or application-level caching absorbs the repeated reads |
The diagnostic tool built for this is CloudWatch Contributor Insights for DynamoDB, which reports the most accessed and most throttled partition keys. "Throttling with low overall utilization" plus "one key dominating in
Contributor Insights" is the complete hot-partition signature; the answer is a key-design change (distribution, sharding), never merely "increase provisioned capacity."
Common trap: Adaptive capacity - DynamoDB's automatic rebalancing toward busy partitions - helps with moderate imbalance, but it is neither instant nor unlimited and does not repeal the per-partition ceilings. A single key value receiving thousands of writes per second still throttles; answers relying on adaptive capacity alone for an extreme hot key are wrong.
Playbook: API Gateway errors - 403 vs 429 vs 5xx
API Gateway failures are classified by who is rejecting the request, and the status code tells you which layer that is.
Reading the code correctly is most of the battle:
| Sta- tus | 401 | 403 |
|---|---|---|
| 429 | 500 | 502 |
| 503 | 504 | Layer responsible |
| Authentication | Authorization / policy | Throttling |
| API Gateway / integration configuration | Backend integration | Backend reachability |
| Backend too slow | Typical causes | Missing or invalid token presented to a Lambda authorizer or Cognito authorizer |
| Authorizer returned an explicit Deny; resource policy conditions unmet (for example, wrong aws:SourceVpce for a private API); AWS WAF rule blocking; missing or invalid API key where one is required | Account-level per-Region rate exceeded (default 10,000 requests/second steady state, 5,000 burst, adjustable), stage/method-level throttle settings, or a usage plan's per-key rate or quota | Non-proxy integration response mapping failures; integration misconfiguration - check execution logs and integration response settings |
| Lambda proxy integration returned a malformed response, or the function errored without producing a valid proxy response; "Malformed Lambda proxy response" in the logs is the signature | VPC link target (Network Load Balancer) has no healthy targets - check target group health | Integration timeout exceeded - REST APIs default to 29 seconds (raisable for Regional/private APIs via Service Quotas, not for edge-optimized); HTTP APIs default to 30 seconds. Debug the backend, not the API |
The workflow that resolves most scenarios:
1. Separate the classes with metrics: graph 4XXError and 5XXError against Count. A wall of 403s starting exactly at a deploy points at policy or authorizer changes; 429s tracking total traffic points at throttling;
5xx on one method points at that method's integration.
2. Enable (or read) execution logging for the stage and access logs carrying $context variables such as
$context.authorizer.error and $context.integrationErrorMessage. The authorizer error variablealone resolves most 401/403 mysteries.
3. Correlate the API Gateway request ID with the backend logs (the integration playbook below generalizes this) to join "API Gateway returned 502" to the exact function exception.
Common trap: A Lambda error behind a proxy integration surfaces as 502, not 500 - from API Gateway's perspective the integration failed to produce a valid response. The crash is real, but the code it produces is
502. Conversely, a function that catches its errors and returns a well-formed 500 response object passes that
500 through untouched.
1. Clients receive 502 errors and the API Gateway execution log says "Malformed Lambda proxy response," yet the function completes without crashing. Where is the failure?
2. A private API starts returning 403 for calls arriving from an approved VPC shortly after the VPC endpoint was recreated. What do you check first?
Answers: 1. In the response shape, not the function's logic - a proxy integration requires a well-formed response object (statusCode, headers, body); fix what the function returns. 2. The API's resource policy - its
aws:SourceVpce condition still names the old endpoint ID, so the recreated endpoint no longer matches.Playbook: SQS poison messages and DLQ redrive
A poison message can never be processed successfully - malformed payload, reference to a deleted record, a payload that deterministically crashes the consumer. Unprotected, the consumer receives it, fails, the visibility timeout expires, and it receives it again: an infinite loop that burns compute and blocks progress behind it (in
FIFO, every later message in the same message group; in a Lambda batch, the whole batch retries).
The standard defense is a redrive policy on the source queue: after maxReceiveCount failed receives, SQS moves the message to a dead-letter queue (DLQ) instead of offering it again. Operational essentials:
- Set the visibility timeout with the consumer in mind. For Lambda event source mappings, AWS
recommends at least six times the function's timeout, so a message in active processing is never redelivered mid-flight; too-short timeouts create the duplicate-processing version of this incident.
- For Lambda consumers, enable partial batch responses (
ReportBatchItemFailuresin the event source
mapping's function response types). Without it, one bad message fails the entire batch and every healthy message retries too; with it, only failed IDs return.
- Alarm on the DLQ itself (
ApproximateNumberOfMessagesVisiblegreater than zero;
ApproximateAgeOfOldestMessage on the source queue) so a poison-message stall pages someoneinstead of sitting silent.
- Diagnose before redriving: read the message in the DLQ, reproduce the failure, fix the code or the data.
Redrive then moves messages back to the source queue (or a chosen destination) via the console's Start
DLQ redrive or the StartMessageMoveTask API; redriving before fixing just re-poisons the queue. (If a redrive is refused, check the DLQ's redrive allow policy, which controls which source queues may redrive into it.) Set maxReceiveCount modestly - commonly 3-5.
Dead-letter queue (DLQ). A queue that receives the messages a source queue gives up on after they exceed the redrive policy's maxReceiveCount. It matters because it converts an infinite poison-message retry loop into a parked, inspectable message: you diagnose and fix the cause first, then redrive the messages back.
Playbook: integration failures - finding the failing hop
The generic multi-service incident - "the order flow is broken somewhere" - yields to a fixed sequence rather than intuition:
1. Bound the blast radius with metrics. Which services show elevated error metrics, and since when? The earliest timestamp usually names patient zero; the IntegrationLatency vs Latency split and per-function Errors narrow the field.
2. Walk the path with the service map. Faults cluster on the failing node's inbound edges; latency bulges on the slow node's outbound edges. Open traces filtered to faults there.
3. Read the failing segment's timeline, then join to logs. The subsegment tree shows the exact downstream call - the DynamoDB Query, the HTTP call, the SQS SendMessage - with its error and cause;
the segment's request ID and trace ID then locate the full stack trace and offending input in that service's
CloudWatch logs.
4. Interrogate the change record. Check CloudTrail history and recent deployments for the window. Many
"mystery" failures are a rotated secret, a tightened security group, a new IAM policy version, or an environment variable changed by the last deploy.
5. Reproduce and verify. Re-run the failing request shape with a test invocation or a CloudWatch Synthetics canary, and confirm the trace goes green end to end - not merely that the first hop responds.
Keep a checklist of mundane causes, because they account for most real cases: expired or rotated secrets;
security group or NACL changes; an endpoint or environment variable pointing at the wrong Region or the pre-migration resource; TLS issues; and a downstream service hitting its own limits - DynamoDB throttling and
Lambda concurrency both masquerade as "the integration is down."
Scenarios that hand you a service map description ("the payment node shows faults; its subsegments show the call to the orders table faulting") want you to name the downstream resource as the root cause. The node reporting the error is the victim; the faulting subsegment's target is the culprit.
Optimization
Optimization questions here are never "make it faster" in the abstract; they present a measured symptom - a cost line, a latency percentile, a throttle count - and four levers, three of which address a different symptom.
Anchor every choice to a metric from the troubleshooting half of this chapter.
Lambda memory and the Power Tuning approach
Lambda's memory setting is secretly three settings in one: it allocates memory, CPU, and network bandwidth in proportion, from 128 MB to 10,240 MB. At about 1,769 MB a function gets the equivalent of one full vCPU;
multi-threaded or CPU-bound code keeps gaining beyond that, single-threaded code mostly does not. Billing is per GB-second, so memory and cost are not monotonic: doubling memory doubles the per-second price but can cut duration by more than half on CPU-bound work, making the function both faster and cheaper. I/O-bound functions gain almost nothing - duration barely moves while the price rises.
Because the sweet spot depends on the workload, guessing is a losing strategy, and the canonical tool is AWS
Lambda Power Tuning, an open-source project (maintained by Alex Casalboni and referenced in AWS's own
Lambda documentation) that you deploy from the AWS Serverless Application Repository. It is a Step Functions state machine that works like this:
1. You give it a function ARN, the power values to test (for example 128, 256, 512, 1,024, 1,536, 3,008 MB), a representative payload, and the invocations to run per configuration.
2. It creates temporary versions of the function at each memory size, invokes each the requested number of times, and collects actual billed durations from the logs.
3. Its analyzer applies a strategy - cost (lowest average cost per invocation), speed (lowest average duration), or balanced (a weighted compromise) - and returns the optimal value plus a visualization plotting cost and duration against memory. It can optionally apply the winner to an alias automatically.
The approach generalizes: measure with production-shaped payloads and re-tune when the workload changes character.
| Lever | Increase memory | Graviton (arm64) architecture |
|---|---|---|
| Helps when | CPU-bound work; duration falls faster than price rises | Most workloads - typically better price-performance |
| Watch out for | I/O-bound functions just cost more | Native dependencies must ship arm64 builds |
| Lever | Helps when | Watch out for | ||
|---|---|---|---|---|
| Smaller package / less init code | Cold-start latency and init duration | Layers move bytes; they do not remove them | ||
| from init entirely | ||||
| Client/connection reuse | Connection setup dominating short | Shared state must tolerate reuse across | ||
| outside the handler | invocations | warm environments |
Lambda's memory slider is also its CPU and network slider - all three scale in proportion to the memory you choose. That is why raising memory can make a CPU-bound function both faster and cheaper per invocation:
the higher per-second price is outrun by the shorter duration. Students who treat memory as "just RAM" pick the cheapest setting and pay more for slower runs.
Concurrency and provisioned concurrency
Lambda concurrency has three flavors, and exam questions hinge on which problem each solves:
- Unreserved (account) concurrency - the shared pool (default 1,000 per Region, raisable via Service
Quotas). Exhaust it, and every function in the Region can throttle.
- Reserved concurrency - a slice of the pool guaranteed to one function, doubling as its cap. Its purpose is
protective: stopping a runaway function from starving the others, and capping downstream load (how hard a function can hit a database, say).
- Provisioned concurrency - environments initialized in advance and kept warm, assigned to a version or
alias. Invocations within the provisioned count skip initialization entirely - the direct cure for cold-start latency. Requests beyond it spill over to on-demand environments, and provisioned capacity can auto-scale via Application Auto Scaling.
Choose by symptom: throttling under load -> raise limits or reserve deliberately; cold-start latency -> provisioned concurrency or SnapStart; one function endangering the rest -> reserved concurrency as a fence.
Cold start. The latency added when Lambda must create a new execution environment - fetching the code, starting the runtime, and running initialization - before the handler can begin. It matters because it strikes only the first request to a fresh environment, so it appears as intermittent tail latency (a high Init Duration on the
REPORT line), not as uniformly slow responses.
DynamoDB capacity modes and DAX
DynamoDB offers two capacity postures. Provisioned capacity with auto scaling suits steady, predictable traffic and is cheaper at sustained volume. On-demand suits spiky, unpredictable, or new workloads: you pay per request, and capacity adapts instantly up to roughly double your previous peak (beyond that, ramping can briefly throttle - see the hot-partition playbook). The exam frames the choice as an operations-versus-cost trade;
both modes share the same per-partition ceilings.
For read performance, Amazon DynamoDB Accelerator (DAX) is a managed in-memory cache that sits in front of a table inside your VPC and answers reads in microseconds instead of milliseconds. Its exam-relevant properties:
- DAX maintains an item cache (results of
GetItem/BatchGetItem) and a query cache (results ofQuery/
Scan), each with a TTL defaulting to 5 minutes.- It is a write-through cache: writes go to DynamoDB and update the cache; read misses are fetched and
cached. Items that do not exist are negatively cached, so repeated lookups of absent keys also stop hitting the table.
- DAX serves eventually consistent reads from the cache; strongly consistent reads pass through to
DynamoDB itself.
- It requires the DAX client rather than the standard SDK DynamoDB client, and it runs as a cluster in your
VPC - real operational surface area.
- It fixes read hot spots (a few items read relentlessly). It does nothing for write hot partitions, and it is the
wrong answer for write-dominated workloads or when every read must be strongly consistent. When a stem demands microsecond DynamoDB reads with minimal code change, DAX is the intended answer.
Exam tip. DAX questions turn on what DAX cannot do: it serves eventually consistent reads only, and it does nothing for write hot partitions. If a stem demands strongly consistent reads on every call, or the bottleneck is writes to one hot key, DAX is the distractor - the answer is a key-design change or a different data path.
API Gateway caching
REST APIs can cache responses at the stage level, absorbing repeated reads before they reach your backend:
enable caching per stage, choose a cluster size, and set a time-to-live (default 300 seconds, maximum 3,600 seconds). Behavior details that show up in scenarios:
- Caching is overridden per method - enable it for safe, slowly changing GETs, disable it elsewhere. Cached
responses are capped at 1 MB, and the cache can be encrypted.
- The cache key is built from the request parameters you designate (selected query string parameters,
headers, path parameters). Two requests differing in an undesignated parameter return the same cached response - the classic source of "users see each other's data" bugs, fixed by including the discriminating parameter in the cache key or not caching that method at all.
- Clients can force a refresh of one entry with
Cache-Control: max-age=0, but only when you authorize
that behavior; otherwise invalidation is an operator action (flush the stage cache), with a repopulation cost.
- HTTP APIs do not offer this built-in caching - the standard pattern there is CloudFront in front of the API.
Caching fits when identical repeated reads crush the backend and the data tolerates seconds of staleness - not for personalized, rapidly changing, or write-path problems. Any scenario mentioning data leaking between users is testing the cache-key bullet above.
Exam tip. When a scenario says API Gateway caching made users see each other's data, do not reach for
"disable caching entirely" first - the tested fix is the cache key: include the parameter that distinguishes the responses (a header, query string, or path parameter) in the key, or stop caching that one method.
RDS Proxy
Lambda and relational databases have a structural mismatch: every execution environment opens its own connections, setup is slow, and a concurrency burst becomes a connection storm that can exhaust the database.
Amazon RDS Proxy sits between application and database as a managed connection pool:
- It multiplexes many application connections onto a smaller pool of database connections, so a thousand
concurrent functions no longer means a thousand database connections.
- Failovers complete faster because the proxy keeps its position in the network path and re-points
connections - AWS quotes failover times reduced by up to roughly two-thirds versus direct reconnects.
- Authentication can use IAM authentication plus credentials in AWS Secrets Manager, removing
hard-coded database passwords from function configuration - the property that makes Proxy the answer in combined security-plus-performance scenarios.
- Caveat: certain session state -
SETstatements, temporary tables, advisory locks, some uses of prepared
statements - pins a client connection to one database connection for its lifetime, quietly defeating multiplexing. A workload that pins everything gains failover benefits but little pooling benefit. (Proxy targets RDS and Aurora; applications switch to the proxy endpoint, but query code is unchanged.)
"Lambda functions intermittently fail with too many connections errors against RDS during traffic spikes" is
RDS Proxy's home scenario. ElastiCache does not fix connection counts, and a bigger instance only moves the ceiling.
1. A table's reads concentrate on a few extremely popular items; write traffic is healthy, and eventually consistent reads are acceptable. Which lever absorbs the read hot spot?
2. During traffic spikes, hundreds of concurrent Lambda functions intermittently fail with "too many connections" errors against an RDS database. What fixes the connection count itself - not just the ceiling?
Answers: 1. DAX in front of the table - its item cache serves the repeated reads in microseconds. 2. RDS Proxy, which multiplexes the functions' connections onto a smaller pool of database connections.
AI-Assisted Troubleshooting
DVA-C03 threads AI assistance through every domain, and Domain 4's share is operational: using Amazon Q
Developer and CloudWatch's AI features to shorten the distance between symptom and root cause - knowing what they do and where their output must be verified.
Amazon Q Developer in the AWS console and IDE can reason over your actual environment: ask why a function is erroring, or what an error code means for a specific resource, and get an explanation plus candidate fixes grounded in AWS documentation and the resource's configuration. Treat its answers as a strong draft hypothesis - it can be confidently wrong about your configuration, and it acts within your IAM permissions, so least-privilege discipline applies to it as to any other tool.
CloudWatch investigations formalize that assistance around an incident. Starting from an alarm, a metric anomaly, or a log pattern, an investigation sweeps related telemetry - metrics, logs, traces, recent changes - and produces observations (supporting evidence) and root-cause hypotheses in natural language; lightweight, session-based investigations can also be launched from the console's operational troubleshooting views with no setup. In exam terms, it automates the correlation steps of the integration-failure playbook while the human still decides and acts.
Natural-language querying removes the syntax barrier in this chapter's tools. The Logs Insights query generator turns a plain-English request ("show me the 10 slowest Lambda invocations in the last day") into a correct query, explains it line by line, and refines it iteratively; the same capability generates Metrics Insights queries, and Logs Insights can produce a natural-language summary of query results. These features accelerate the authoring of queries, not the evidence itself - a generated query deserves the same read-before-you-run scrutiny as any generated code.
A closing note the exam's scenario writers share: AI assistance optimizes the hypothesis step, not the verification step. Every fix above still ends with a metric, a trace, or a log line proving the system now behaves.
When a scenario asks for help interpreting telemetry - generating a query from a description, summarizing results, forming a root-cause hypothesis - the answer is the AI feature. When it asks to collect telemetry, the answer is still the underlying feature: metric filters, EMF, X-Ray instrumentation, CloudTrail configuration. Keep the two layers separate and most of these questions resolve themselves.
End-of-Chapter Test
1. A team wants an alarm whenever their application logs the string "PaymentDeclined" more than ten times in five minutes. The application already writes plain-text logs to CloudWatch Logs. What should they configure first?
- A CloudWatch Logs Insights scheduled query that emails the team
- A metric filter matching PaymentDeclined that publishes a metric, plus a CloudWatch alarm on that metric
- An EMF block in the application emitting a PaymentDeclined metric
- A subscription filter to a Lambda function that increments a custom metric
2. Support staff must be able to find all traces for a specific order, and the full order payload must be stored with each trace for later inspection. How should the two pieces of data be attached?
- Both as annotations
- Both as metadata
- The order ID as an annotation; the payload as metadata
- The order ID as metadata; the payload as an annotation
3. An API serves 2,000 requests per second. Roughly how many requests per second does the default X-Ray sampling rule record?
- About 100 per second
- About 21 per second
- About 5 per second
- 2,000 per second at first, dropping after a quota is reached
4. Clients of a Regional REST API receive 504 responses after about 29 seconds on one endpoint. The backing Lambda function has a 60-second timeout and its logs show it completing successfully at around 40 seconds. What is the root cause and the cleanest fix?
- The Lambda function is throttling; add reserved concurrency
- The API Gateway integration timeout defaults to 29 seconds; request a Service Quotas increase for the integration timeout, or convert the operation to an asynchronous pattern
- The Lambda function memory is too low; increase it so the function finishes faster
- API Gateway caching is serving stale timeouts; flush the stage cache
5. A DynamoDB table shows heavy WriteThrottleEvents, yet consumed write capacity averages only 30% of provisioned capacity. Contributor Insights shows one partition key value receiving most writes. What is the correct diagnosis and remedy?
- Under-provisioned table; enable auto scaling
- Hot partition: a single key value exceeds the per-partition write ceiling; redesign the key - for example, write-sharding with a suffix - to spread writes
- GSI misconfiguration; delete and recreate the indexes
- DAX write-through failure; increase the DAX cluster size
6. A message with a malformed payload fails processing every time in a queue feeding a Lambda function, and each failure retries the entire batch of 10 messages. Which combination resolves both the poison message and the batch amplification?
- Increase the Lambda function timeout and the queue visibility timeout
- Configure a DLQ with a maxReceiveCount, and enable partial batch responses ( ReportBatchItemFailures) on the event source mapping
- Convert the queue to FIFO and enable content-based deduplication
- Reduce the batch size to 1 and delete the message manually
7. A security review asks who read a specific object from an S3 bucket last week. The account has CloudTrail enabled with default settings, but the review finds no record of the reads. Why?
- CloudTrail does not record S3 activity at all
- The event history only covers management events from the last 24 hours
- S3 object reads are data events, which are not recorded unless explicitly enabled for the bucket
- The trail must be organization-wide before any events are recorded
8. A CPU-bound Lambda function runs at 512 MB, averaging 8 seconds per invocation. The team wants to lower cost per invocation and suspects a higher memory setting may help. What is the most reliable way to choose the new setting?
- Set memory to the maximum 10,240 MB, since more memory is always cheaper per invocation
- Run AWS Lambda Power Tuning with representative payloads across several memory values and compare measured cost and duration under the cost strategy
- Halve the memory to 256 MB so each GB-second costs less
- Enable provisioned concurrency to reduce the billed duration
Answer key
1. B — Metric filters retrofit metrics onto logs you already emit, and alarms act on metrics - EMF (C) requires changing the application, and Insights queries (A) are interactive analysis, not alarming.
2. C — Annotations are indexed and searchable through filter expressions, so anything the team must find traces by belongs there; metadata is unindexed but accepts arbitrary JSON - the right home for a payload you only read after locating the trace.
3. A — The default rule records the first request each second (the reservoir) plus 5% of the rest: 1 + (0.05 × 1,999) ~ 101 traces per second.
4. B — The failure occurs at the integration timeout while the function itself succeeds - the signature of this mismatch. For Regional and private APIs the timeout can be raised via Service Quotas; otherwise return 202 and process asynchronously.
5. B — Throttling with abundant table-level headroom is the hot-partition signature: one partition is capped at about 1,000 WCU per second regardless of total capacity, so auto scaling (A) adds capacity the hot partition cannot use.
6. B — The redrive policy parks the message in the DLQ after a bounded number of attempts, for inspection and redrive after a fix; partial batch responses return only the failed item instead of the whole batch.
7. C — CloudTrail records management events by default; object-level operations such as S3 GetObject are data events, opted into per resource because of their volume - and they cannot be captured retroactively.
8. B — Because CPU scales with memory, CPU-bound functions often cost less at higher memory - but the optimum is workload-specific and must be measured, which is what the Power Tuning state machine automates.
Want the full AWS Developer – Associate (DVA-C03) package?
This guide is free to read. Get the downloadable PDF guidebook plus the full DVA-C03 practice exam with 3 months of online access.
Get DVA-C03 Full Access — $10