Chapter 4

Troubleshooting and Optimization

AWS Certified Developer – Associate (DVA-C03) Study GuideDomain 4 · 22% of the examBased on the official DVA-C03 exam objectives

definitions, Step Functions state machines) and applies them directly through service APIs, skipping

CloudFormation. The trade-off is drift from CloudFormation's recorded state, which is why it is a development-only tool. --method=direct still performs a full CloudFormation deployment, sam deploy does not deploy CDK apps, and bootstrapping is a one-time environment setup.

6. C - Only a manual approval action actually pauses a pipeline execution until a person approves or rejects it

(and it can notify via SNS). The AI summary helps the reviewer but enforces nothing; a test action cannot judge review status, rollback undoes a deployment rather than gating it, and an SNS subscription by itself does not block the stage transition.

7. B - Feature flags are built for exactly this: gradual exposure via a deployment strategy, and alarm-driven automatic rollback (plus the ability to flip the flag off immediately). A freeform profile with AllAtOnce gives no gradual rollout, manual parameter edits have no rollout or rollback machinery, and secret rotation is about credentials, not feature exposure.

8. C - Equal increments at fixed intervals is the definition of a linear deployment, and the Lambda-scoped name matches the target platform. Options A and D describe canary (single step) and the ECS-scoped linear configuration respectively; option B shifts everything immediately.

Domain 4 grew from 18% of the exam in DVA-C02 to 22% in DVA-C03, and that shift tells you something about how AWS sees the developer role: writing the code is only half the job. The other half is proving what the code did in production, finding the hop where a request failed, and tuning the system once you understand it. This chapter builds those skills around the three pillars of AWS observability - CloudWatch for logs and metrics, X-Ray for traces, CloudTrail for the audit record - then works through the failure patterns the exam returns to again and again, and finishes with the optimization levers and the AI-assisted tooling now layered on top.

A note on method: troubleshooting here is mostly correlation - one request leaves a log line, metrics, a trace, and a CloudTrail entry for any API call it made. Read them in order: symptom (metrics), request path (traces), detail (logs), "who changed what" (audit trail).

The Observability Toolkit at a Glance

ToolCloudWatch LogsCloudWatch MetricsCloudWatch Logs Insights
What it recordsLog events your code and AWS services emitTime-series numbers (counts, latencies, errors)Interactive queries over log data
Question it answers"What did the application say happened?""Is something wrong, and since when?""Show me the slowest / failing / most frequent pattern"
Retention to knowYou set it per log group: fixed periods from 1 day up to 10 years, or never expire15 months, at progressively coarser resolutionQueries run over whatever logs the retention policy keeps
ToolWhat it recordsQuestion it answersRetention to know
AWS X-RayTraces: segments and"Where did the request spend itsTraces are kept for 30 days
subsegments for each requesttime, and which hop failed?"
AWSAPI activity in the account (who,"Who or what called this API?"Event history covers 90 days; trails
CloudTrailwhat, when, from where)and CloudTrail Lake extend this

Hold onto that table's core distinction: why did the application behave this way - logs and traces; who performed this action - CloudTrail. Scenarios profit from confusing the two.

Amazon CloudWatch Logs

Log groups, log streams, and retention

Every Lambda function automatically gets a log group named /aws/lambda/<function-name>, with log streams separated by date and execution environment. Other workloads - containers using the awslogs driver, applications using the CloudWatch agent or an SDK - follow the same model: a log group is the container you apply settings to, a log stream is a sequence of events from one source, and a log event is a single timestamped record (up to 1 MB).

Retention is set on the log group, and the default is never expire - which is why "log group retention" is the cost-control lever in exam questions. Fixed periods run from 1 day to 10 years. Three operational facts matter for troubleshooting:

logs are gone" is often a retention change, not an outage. And log groups are Region-scoped - "no logs"

frequently means the wrong Region or account.

logs:PutLogEvents permission means the function still runs - just silently. Check the execution role when

a function produces no output.

For moving log data elsewhere, subscription filters stream events out of a log group in near real time to a

Lambda function, a Kinesis data stream, or Amazon Data Firehose (for delivery to S3, OpenSearch, and other destinations). A log group can have up to two subscription filters.

Metric filters: turning log events into metrics

A metric filter watches a log group, matches incoming events against a pattern, and publishes a CloudWatch metric on each match - how you alarm on something no AWS service measures for you, such as occurrences of a specific application error code.

Key behaviors to internalize:

positional fields such as a 5xx status code). JSON patterns match properties inside JSON log events, for example { $.errorCode = "AccessDenied" }.

metric value rather than a simple count of 1.

continuously (zero errors produces a 0 data point instead of a gap).

week's error counts from a filter created yesterday, the answer is a Logs Insights query over the retained logs.

unchanged.

Exam tip

"Create an alarm when the application logs a specific error" always decomposes into two steps: a metric filter to convert matching log events into a metric, then a CloudWatch alarm on that metric. An alarm cannot read log text directly.

Definition

Metric filter. A CloudWatch Logs pattern rule that watches a log group and publishes a CloudWatch metric whenever an incoming event matches the pattern. It matters because alarms can only act on metrics, so a metric filter is the bridge that turns application log text - an error string, a status code, a latency value - into something you can alarm and dashboard on.

CloudWatch Logs Insights queries

Logs Insights is the interactive query engine over log groups - the tool for ad-hoc investigation: aggregating, filtering, parsing fields out of unstructured text, and ranking results. A query runs across up to 50 log groups at a time and returns up to 1,000 rows by default (configurable to 10,000).

The core commands, in the order you will usually write them:

Commandfields
filterparse
statsPurpose
Choose which fields to display or work withKeep only events matching a condition (like, comparisons, regex)
Extract new fields from unstructured text using a pattern or regexAggregate - count(), sum(), avg(), max(), percentile functions such as pct() - optionally with by grouping and bin() time buckets
CommandPurpose
sortOrder results, e.g. sort @duration desc
limitCap the number of rows returned
dedup / displayRemove duplicate rows / shape which fields are shown

Lambda log groups get discovered fields such as @timestamp, @message, @requestId, and on the REPORT line, @duration, @billedDuration, @memorySize, and @maxMemoryUsed. A canonical troubleshooting query - the slowest recent invocations - looks like this:

fields @timestamp, @requestId, @duration, @maxMemoryUsed, @memorySize
| filter @type = "REPORT"
| sort @duration desc
| limit 20

Two habits separate effective Logs Insights work from flailing: bound the time range and start from an aggregate (stats - for example, error counts with stats count(*) by bin(5m)) before drilling into raw events; and use parse when a needed field is trapped inside a text message, since parsed fields can then be filtered and aggregated like native ones.

The embedded metric format (EMF)

Metric filters extract metrics from logs after the fact. The embedded metric format is the opposite arrangement: your application deliberately writes a structured JSON log line containing both the log data and a metrics directive, and CloudWatch Logs extracts the metric automatically on ingestion. There is no

PutMetricData call in the code path - the log line is the metric publication.

An EMF event carries a _aws metadata block naming the timestamp, the namespace, the dimension sets, and which top-level properties are metrics:

{
"_aws": {
"Timestamp": 1718042400000,
"CloudWatchMetrics": [
{
"Namespace": "CheckoutService",
"Dimensions": [["Service", "Endpoint"]],
"Metrics": [{ "Name": "OrderLatency", "Unit": "Milliseconds" }]
}
]
},
"Service": "checkout",
"Endpoint": "/orders",
"OrderLatency": 187
}

Facts the exam leans on:

keys - making EMF the natural fit for high-cardinality dimensions (per tenant, per endpoint, per feature flag) that would be impractical to publish one API call at a time.

event as searchable context, so you get the metric and the log line behind it.

Powertools for AWS Lambda's Metrics feature) generate the format for you - and because publication rides on log ingestion, there is no separate metrics API to throttle and no added call in a hot code path.

Choosing between the two: metric filters retrofit metrics onto logs you already emit; EMF is a design-time choice for code you own, when you want rich custom metrics with dimensions from day one.

Common trap: An EMF event with a malformed _aws block is stored as an ordinary log event and silently produces no metric - suspect a wrong namespace, or a directive metric name that matches no property in the event.

CloudWatch Metrics and Alarms

Logs tell you what happened; metrics tell you whether it is still happening and how bad it is. Every service here publishes metrics under a namespace - AWS/Lambda, AWS/ApiGateway, AWS/DynamoDB, AWS/SQS - and the troubleshooting vocabulary starts with knowing the key metric names cold:

ServiceLambda
Metrics that diagnose most incidentsInvocations, Errors, Duration (avg/max), Throttles, ConcurrentExecutions, IteratorAge for stream triggers
Service
API GatewayDynamoDB
SQS
Metrics that diagnose most incidentsCount, Latency, IntegrationLatency, 4XXError, 5XXError, CacheHitCount / CacheMissCount
ConsumedReadCapacityUnits, ConsumedWriteCapacityUnits, ReadThrottleEvents, WriteThrottleEvents, SuccessfulRequestLatencyApproximateNumberOfMessagesVisible, ApproximateAgeOfOldestMessage, NumberOfMessagesSent, NumberOfEmptyReceives

The most useful diagnostic habit here is comparing the pair that splits a problem at the service boundary. For

API Gateway, Latency is end-to-end client time while IntegrationLatency is only the wait for the backend: if the two track together, the backend is slow; if Latency soars while IntegrationLatency stays flat, the problem sits in API Gateway or the network path. (The Lambda playbook below leans on the same kind of split, between Errors and Throttles.)

An alarm watches one metric (or a metric math expression, or a Logs Insights query result) over an evaluation window and transitions between OK, ALARM, and INSUFFICIENT_DATA. The vocabulary that appears in scenarios:

exceeding 3,000 ms, breaching in 3 of the last 5 periods (which smooths over single spikes; "1 out of 1"

pages you on noise). Percentile statistics (p90, p99) are usually fairer than averages for latency, because averages hide a suffering minority of requests.

heartbeat-style metrics), notBreaching (gaps are fine), ignore (keep the previous state), or missing (fall back to INSUFFICIENT_DATA). Scenarios about alarms that "never fire even when the service stops reporting" test exactly this setting.

high," not two noisy alerts.

seasonality) and alarms when it leaves the band; useful when nobody knows a sensible static threshold.

Practically, an alarm is the start of the workflow below: it fixes when the problem began, the timestamp every log query and trace search should anchor to. (Its actions route to SNS, Lambda, Auto Scaling, or Systems

Manager.)

Check yourself

1. A service publishes a heartbeat metric every minute. It crashes overnight, the metric stops arriving - and the alarm never leaves OK, so nobody is paged. Which alarm setting is wrong?

2. A metric filter created today shows no error counts for last week, even though the log group holds a month of events. Why - and what produces last week's counts instead?

Answers: 1. Treat missing data is set to notBreaching, so the gaps count as healthy; set it to breaching for heartbeat-style metrics. 2. Metric filters never backfill - they only match events ingested after creation; run a

Logs Insights query over the retained logs for the historical counts.

AWS X-Ray: Distributed Tracing

Metrics can tell you the checkout service got slow at 14:03; they cannot tell you the slowness lives in a sub-call three services downstream. That is the job of distributed tracing - AWS X-Ray.

Segments and subsegments

A trace is the end-to-end record of one request as it crosses services. Each service contributes a segment: a

JSON document recording the service name, start and end times, inbound HTTP details, AWS resource details, and three flags that drive most analysis - error (4xx-class), fault (5xx-class), and throttle.

Within a segment, subsegments break the work down further. The SDK automatically creates subsegments for downstream calls it instruments - HTTP requests, AWS SDK calls to DynamoDB, SQS, and so on - and you can open custom subsegments around your own code blocks to time a pricing calculation or a call to a legacy system. When the exam shows a slow request and asks where the time went, picture the trace timeline: a segment with nested subsegments, each carrying its own duration.

Supporting facts worth knowing:

participates when an upstream service already started a trace).

X-Ray write permissions, and tracing is enabled per function.

(ADOT) collector receives segment data from your application and forwards it to X-Ray; ADOT is the modern path.

LOG_ERROR converts that crash into a log line - the classic fix when the SDK throws exceptions in a service

receiving untraced requests.

Definition

Segment (X-Ray). The JSON record one service contributes to a trace, capturing its name, start and end times, inbound request details, and the error, fault, and throttle flags. It matters because the segment timeline

- with nested subsegments for each downstream call - is where you see exactly which hop consumed the time or returned the error.

Sampling rules

Tracing everything is expensive and mostly unnecessary, so X-Ray samples: it records a subset of requests chosen by rules. Every rule combines a reservoir - a guaranteed number of requests per second that are always traced - with a fixed rate applied to the remainder. The default rule is a reservoir of 1 request per second plus

5% of additional requests, which guarantees at least one trace per second from any active service while keeping volume low.

Custom rules target a subset of traffic by attributes such as service name, service type, host, HTTP method, and

URL path, and are evaluated in priority order (lower number wins, 1 to 9,999, default rule last). Two behaviors matter operationally:

on a failing endpoint mid-incident is a console change, not a release.

and downstream services honor it. A 100% rule on an internal service never fires for requests whose upstream service already declined to sample. Put investigative rules on the root service - or on genuine new trace roots such as asynchronous workers.

Core concept

The sampling decision is made once, by the first instrumented service that sees the request, and every downstream service honors it. Students often put a 100% sampling rule on the internal service they suspect, then wonder why no traces appear - if the upstream root already declined to sample those requests, the downstream rule never gets a chance to fire.

Annotations versus metadata

Both attach key-value data to segments and subsegments, and the exam tests the difference relentlessly:

AnnotationsMetadata
Indexed forYes - filter expressions can query themNo - not searchable
search
Allowed types
Intended useLimit discipline
AnnotationsStrings, numbers, booleans onlyThe few values you find traces by: order ID, tenant, outcome
Keep them few (max 50 per segment)Metadata
Arbitrary JSON, including nested objectsRich context read after finding the trace: payloads, stack tracesCan be large and free-form

Trace search uses filter expressions over indexed fields and annotations - all faulted traces for one service, or

annotation.orderId = "A123" for every trace touching one order. The rule: "search traces by X" means X is an

annotation; "store the full payload for later inspection" means metadata.

The service map and trace analysis

The service map renders every traced service as a node and every observed call path as an edge, annotated with latency and error/throttle rates - in incident terms, "which node turned red." The workflow: start at the map, find the faulting node, open its traces filtered to faults, and read the segment timeline down to the failing subsegment, which typically names the exact downstream call and error. Groups save a filter expression as a named lens over traces; X-Ray Insights raises events with a root-cause summary when trace behavior deviates from baseline. Traces are retained for 30 days; older comparisons belong in logs and metrics.

Common trap: X-Ray shows sampled requests only. A clean service map does not prove an error is absent - at a

5% rate, a one-in-a-thousand failure may never have been sampled. Raise the rate (or check Logs Insights, which sees everything) before concluding the problem is elsewhere.

AWS CloudTrail: Auditing API Activity

CloudTrail records the API calls made in your account - by users, roles, and AWS services - with caller identity, source IP, timestamp, request parameters, and any error code returned. It is the tool that converts "the table is gone" or "this started right after a deploy" into a specific principal and a specific call.

Events come in three kinds:

UpdateFunctionConfiguration, and PutBucketPolicy. These are recorded by default and are the first

place to look for configuration changes.

DynamoDB item activity. These are not recorded by default; you opt in per resource because the volume

(and cost) is high. This is the most tested CloudTrail fact in developer scenarios: "which process read this S3 object" requires data events to have been enabled in advance - they cannot be reconstructed afterwards.

error rates and records an insight: an early warning for runaway automation or a bad deployment.

Access paths matter too. The event history (console, or LookupEvents) covers the last 90 days of management events with no setup. A trail delivers events to S3 for durable retention, can be organization-wide and all-Regions, and can stream to CloudWatch Logs for alarming on specific calls (alert whenever DeleteTable occurs, say); log file validation adds digest files proving delivered logs were not altered. CloudTrail Lake is the managed alternative: an event data store with configurable retention (up to 10 years), queried with SQL.

In practice, developers use it three ways: blast-radius checks (filter event history by the deploy window to see every configuration change, in order, with its identity); explaining an AccessDenied (the denied call appears with the exact error code, principal, and resource - far faster than guessing at policies); and correlation (take a timestamp from a log entry, then find that role's API calls in the same minute).

Common trap: CloudTrail records API calls, not application behavior. A function that miscalculates a price leaves no CloudTrail trace of the miscalculation - only of the Invoke and the API calls it made. The calculation itself is in CloudWatch Logs.

Core concept

CloudTrail and CloudWatch Logs answer different questions, and scenarios profit from blurring them.

CloudTrail records API calls - who did what to a resource, and when - while logs and traces record what the application itself did with a request. A bug inside your code leaves evidence in logs; a configuration change leaves evidence in CloudTrail; most real incidents need both, correlated by timestamp.

Root-Cause Analysis Playbooks

The sections above are instruments; this section is the flying - the five failure patterns that dominate real incidents and Domain 4 scenarios alike. Each follows the same discipline: read the symptom precisely, check the cheapest discriminating metric first, confirm in logs or traces, then change something.

Playbook: Lambda timeouts and throttling

Lambda produces two similar complaints - "requests are failing" - with opposite fixes, so separating them is step one. A timeout means the invocation started and ran past the configured timeout (maximum 15 minutes); a throttle means Lambda refused to start it at all because a concurrency limit was reached, returning a 429 with a TooManyRequestsException ("Rate Exceeded").

Symptom Likely cause Fix

SymptomLog line Task timed out after X.XX seconds; Duration metric hugs the timeout value
Errors metric flat but Throttles climbing; callers receive 429sSudden 429s during a traffic spike, average concurrency lowLatency and timeouts only on the first request after idle periods
Timeouts clustered on calls to one database or HTTP API, code unchangedLikely cause
Function genuinely runs long - slow query, unresponsive downstream call, too little memory/CPU. X-Ray subsegments show where the time wentConcurrency limit reached: the account limit, or reserved concurrency set too low. Compare ConcurrentExecutions with the quotaLambda's scaling rate: each function can add up to 1,000 execution environments every 10 seconds, so a near-vertical ramp can briefly outrun it
Cold starts. Signature: high Init Duration on the REPORT line, initialization subsegment in X-RayDownstream degradation, or connection setup overhead (new connections per invocation). X-Ray shows the time inside the downstream subsegment
FixKeep downstream client timeouts below the function timeout, optimize the slow segment, raise memory for CPU-bound work, or raise the function timeout if the work is legitimately longRequest an account limit increase; raise or remove the reserved cap; smooth traffic with an SQS queue in front
Provision concurrency ahead of predictable peaks; buffer with SQSReduce package size and init work; SnapStart for supported Java runtimes, or provisioned concurrencyEnable keep-alive / connection reuse, create clients outside the handler, add retries with backoff and jitter, set per-call timeouts
Exam tip

Behind API Gateway, a function with a 60-second timeout can still produce client failures at ~29 seconds: the

REST API integration timeout defaults to 29 seconds, so the client gets a 504 while the function runs on.

(Raising it via Service Quotas works for Regional/private APIs, not edge-optimized ones; the alternative is an asynchronous pattern returning 202.)

Definition

Reserved concurrency. A slice of the account concurrency pool set aside for one function, which doubles as that function's maximum. It matters in troubleshooting because it cuts both ways: set deliberately, it fences a runaway function away from the rest of the Region; set too low, it is the hidden cause of 429 throttles while the account pool sits idle.

Playbook: DynamoDB hot partitions and throttling

DynamoDB spreads a table's provisioned capacity across its partitions, and each partition has hard physical ceilings: roughly 3,000 read capacity units and 1,000 write capacity units per second per partition.

Throughput trouble therefore comes in two flavors that look identical at the API -

ProvisionedThroughputExceededException - but differ completely in distribution:

the roughly 5 minutes of unused capacity DynamoDB retains for bursting. Consumed capacity sits at or above provisioned levels across the board.

a celebrity user, a single "today" partition. One partition hits its ceiling while the rest idle. This is a key design problem, not a capacity problem, and raising table capacity does nothing for it.

Discrimination and fixes:

ObservationReadThrottleEvents / WriteThrottleEvents > 0 with consumed capacity near provisionedThrottle events with consumed capacity well below provisioned
Writes throttle when a GSI is added or a GSI's keys are skewedThrottling on a recently switched on-demand table during a spikeRead latency and cost high for a few extremely popular items
InterpretationGenuine under-provisioning or a burstHot key or hot partition
GSI back-pressure: if a global secondary index cannot absorb the write rate, writes to the base table throttle tooOn-demand adapts instantly up to about double the previous peak; faster growth can throttle until capacity catches upRead hot spot
FixRaise provisioned capacity / enable auto scaling, or switch to on-demandRedesign the partition key for spread; add a random/write-sharding suffix to hot keys and scatter-gather reads; use Contributor Insights to identify the offending key values
Provision the GSI for the table's write rate; choose GSI partition keys with high cardinalityRamp load gradually before launch events; consider provisioned with auto scaling for extreme, abrupt spikesDAX (below) or application-level caching absorbs the repeated reads

The diagnostic tool built for this is CloudWatch Contributor Insights for DynamoDB, which reports the most accessed and most throttled partition keys. "Throttling with low overall utilization" plus "one key dominating in

Contributor Insights" is the complete hot-partition signature; the answer is a key-design change (distribution, sharding), never merely "increase provisioned capacity."

Common trap: Adaptive capacity - DynamoDB's automatic rebalancing toward busy partitions - helps with moderate imbalance, but it is neither instant nor unlimited and does not repeal the per-partition ceilings. A single key value receiving thousands of writes per second still throttles; answers relying on adaptive capacity alone for an extreme hot key are wrong.

Playbook: API Gateway errors - 403 vs 429 vs 5xx

API Gateway failures are classified by who is rejecting the request, and the status code tells you which layer that is.

Reading the code correctly is most of the battle:

Sta- tus401403
429500502
503504Layer responsible
AuthenticationAuthorization / policyThrottling
API Gateway / integration configurationBackend integrationBackend reachability
Backend too slowTypical causesMissing or invalid token presented to a Lambda authorizer or Cognito authorizer
Authorizer returned an explicit Deny; resource policy conditions unmet (for example, wrong aws:SourceVpce for a private API); AWS WAF rule blocking; missing or invalid API key where one is requiredAccount-level per-Region rate exceeded (default 10,000 requests/second steady state, 5,000 burst, adjustable), stage/method-level throttle settings, or a usage plan's per-key rate or quotaNon-proxy integration response mapping failures; integration misconfiguration - check execution logs and integration response settings
Lambda proxy integration returned a malformed response, or the function errored without producing a valid proxy response; "Malformed Lambda proxy response" in the logs is the signatureVPC link target (Network Load Balancer) has no healthy targets - check target group healthIntegration timeout exceeded - REST APIs default to 29 seconds (raisable for Regional/private APIs via Service Quotas, not for edge-optimized); HTTP APIs default to 30 seconds. Debug the backend, not the API

The workflow that resolves most scenarios:

1. Separate the classes with metrics: graph 4XXError and 5XXError against Count. A wall of 403s starting exactly at a deploy points at policy or authorizer changes; 429s tracking total traffic points at throttling;

5xx on one method points at that method's integration.

2. Enable (or read) execution logging for the stage and access logs carrying $context variables such as

$context.authorizer.error and $context.integrationErrorMessage. The authorizer error variable

alone resolves most 401/403 mysteries.

3. Correlate the API Gateway request ID with the backend logs (the integration playbook below generalizes this) to join "API Gateway returned 502" to the exact function exception.

Common trap: A Lambda error behind a proxy integration surfaces as 502, not 500 - from API Gateway's perspective the integration failed to produce a valid response. The crash is real, but the code it produces is

502. Conversely, a function that catches its errors and returns a well-formed 500 response object passes that

500 through untouched.

Check yourself

1. Clients receive 502 errors and the API Gateway execution log says "Malformed Lambda proxy response," yet the function completes without crashing. Where is the failure?

2. A private API starts returning 403 for calls arriving from an approved VPC shortly after the VPC endpoint was recreated. What do you check first?

Answers: 1. In the response shape, not the function's logic - a proxy integration requires a well-formed response object (statusCode, headers, body); fix what the function returns. 2. The API's resource policy - its

aws:SourceVpce condition still names the old endpoint ID, so the recreated endpoint no longer matches.

Playbook: SQS poison messages and DLQ redrive

A poison message can never be processed successfully - malformed payload, reference to a deleted record, a payload that deterministically crashes the consumer. Unprotected, the consumer receives it, fails, the visibility timeout expires, and it receives it again: an infinite loop that burns compute and blocks progress behind it (in

FIFO, every later message in the same message group; in a Lambda batch, the whole batch retries).

The standard defense is a redrive policy on the source queue: after maxReceiveCount failed receives, SQS moves the message to a dead-letter queue (DLQ) instead of offering it again. Operational essentials:

recommends at least six times the function's timeout, so a message in active processing is never redelivered mid-flight; too-short timeouts create the duplicate-processing version of this incident.

mapping's function response types). Without it, one bad message fails the entire batch and every healthy message retries too; with it, only failed IDs return.

ApproximateAgeOfOldestMessage on the source queue) so a poison-message stall pages someone

instead of sitting silent.

Redrive then moves messages back to the source queue (or a chosen destination) via the console's Start

DLQ redrive or the StartMessageMoveTask API; redriving before fixing just re-poisons the queue. (If a redrive is refused, check the DLQ's redrive allow policy, which controls which source queues may redrive into it.) Set maxReceiveCount modestly - commonly 3-5.

Definition

Dead-letter queue (DLQ). A queue that receives the messages a source queue gives up on after they exceed the redrive policy's maxReceiveCount. It matters because it converts an infinite poison-message retry loop into a parked, inspectable message: you diagnose and fix the cause first, then redrive the messages back.

Playbook: integration failures - finding the failing hop

The generic multi-service incident - "the order flow is broken somewhere" - yields to a fixed sequence rather than intuition:

1. Bound the blast radius with metrics. Which services show elevated error metrics, and since when? The earliest timestamp usually names patient zero; the IntegrationLatency vs Latency split and per-function Errors narrow the field.

2. Walk the path with the service map. Faults cluster on the failing node's inbound edges; latency bulges on the slow node's outbound edges. Open traces filtered to faults there.

3. Read the failing segment's timeline, then join to logs. The subsegment tree shows the exact downstream call - the DynamoDB Query, the HTTP call, the SQS SendMessage - with its error and cause;

the segment's request ID and trace ID then locate the full stack trace and offending input in that service's

CloudWatch logs.

4. Interrogate the change record. Check CloudTrail history and recent deployments for the window. Many

"mystery" failures are a rotated secret, a tightened security group, a new IAM policy version, or an environment variable changed by the last deploy.

5. Reproduce and verify. Re-run the failing request shape with a test invocation or a CloudWatch Synthetics canary, and confirm the trace goes green end to end - not merely that the first hop responds.

Keep a checklist of mundane causes, because they account for most real cases: expired or rotated secrets;

security group or NACL changes; an endpoint or environment variable pointing at the wrong Region or the pre-migration resource; TLS issues; and a downstream service hitting its own limits - DynamoDB throttling and

Lambda concurrency both masquerade as "the integration is down."

Exam tip

Scenarios that hand you a service map description ("the payment node shows faults; its subsegments show the call to the orders table faulting") want you to name the downstream resource as the root cause. The node reporting the error is the victim; the faulting subsegment's target is the culprit.

Optimization

Optimization questions here are never "make it faster" in the abstract; they present a measured symptom - a cost line, a latency percentile, a throttle count - and four levers, three of which address a different symptom.

Anchor every choice to a metric from the troubleshooting half of this chapter.

Lambda memory and the Power Tuning approach

Lambda's memory setting is secretly three settings in one: it allocates memory, CPU, and network bandwidth in proportion, from 128 MB to 10,240 MB. At about 1,769 MB a function gets the equivalent of one full vCPU;

multi-threaded or CPU-bound code keeps gaining beyond that, single-threaded code mostly does not. Billing is per GB-second, so memory and cost are not monotonic: doubling memory doubles the per-second price but can cut duration by more than half on CPU-bound work, making the function both faster and cheaper. I/O-bound functions gain almost nothing - duration barely moves while the price rises.

Because the sweet spot depends on the workload, guessing is a losing strategy, and the canonical tool is AWS

Lambda Power Tuning, an open-source project (maintained by Alex Casalboni and referenced in AWS's own

Lambda documentation) that you deploy from the AWS Serverless Application Repository. It is a Step Functions state machine that works like this:

1. You give it a function ARN, the power values to test (for example 128, 256, 512, 1,024, 1,536, 3,008 MB), a representative payload, and the invocations to run per configuration.

2. It creates temporary versions of the function at each memory size, invokes each the requested number of times, and collects actual billed durations from the logs.

3. Its analyzer applies a strategy - cost (lowest average cost per invocation), speed (lowest average duration), or balanced (a weighted compromise) - and returns the optimal value plus a visualization plotting cost and duration against memory. It can optionally apply the winner to an alias automatically.

The approach generalizes: measure with production-shaped payloads and re-tune when the workload changes character.

LeverIncrease memoryGraviton (arm64) architecture
Helps whenCPU-bound work; duration falls faster than price risesMost workloads - typically better price-performance
Watch out forI/O-bound functions just cost moreNative dependencies must ship arm64 builds
LeverHelps whenWatch out for
Smaller package / less init codeCold-start latency and init durationLayers move bytes; they do not remove them
from init entirely
Client/connection reuseConnection setup dominating shortShared state must tolerate reuse across
outside the handlerinvocationswarm environments
Core concept

Lambda's memory slider is also its CPU and network slider - all three scale in proportion to the memory you choose. That is why raising memory can make a CPU-bound function both faster and cheaper per invocation:

the higher per-second price is outrun by the shorter duration. Students who treat memory as "just RAM" pick the cheapest setting and pay more for slower runs.

Concurrency and provisioned concurrency

Lambda concurrency has three flavors, and exam questions hinge on which problem each solves:

Quotas). Exhaust it, and every function in the Region can throttle.

protective: stopping a runaway function from starving the others, and capping downstream load (how hard a function can hit a database, say).

alias. Invocations within the provisioned count skip initialization entirely - the direct cure for cold-start latency. Requests beyond it spill over to on-demand environments, and provisioned capacity can auto-scale via Application Auto Scaling.

Choose by symptom: throttling under load -> raise limits or reserve deliberately; cold-start latency -> provisioned concurrency or SnapStart; one function endangering the rest -> reserved concurrency as a fence.

Definition

Cold start. The latency added when Lambda must create a new execution environment - fetching the code, starting the runtime, and running initialization - before the handler can begin. It matters because it strikes only the first request to a fresh environment, so it appears as intermittent tail latency (a high Init Duration on the

REPORT line), not as uniformly slow responses.

DynamoDB capacity modes and DAX

DynamoDB offers two capacity postures. Provisioned capacity with auto scaling suits steady, predictable traffic and is cheaper at sustained volume. On-demand suits spiky, unpredictable, or new workloads: you pay per request, and capacity adapts instantly up to roughly double your previous peak (beyond that, ramping can briefly throttle - see the hot-partition playbook). The exam frames the choice as an operations-versus-cost trade;

both modes share the same per-partition ceilings.

For read performance, Amazon DynamoDB Accelerator (DAX) is a managed in-memory cache that sits in front of a table inside your VPC and answers reads in microseconds instead of milliseconds. Its exam-relevant properties:

Scan), each with a TTL defaulting to 5 minutes.

cached. Items that do not exist are negatively cached, so repeated lookups of absent keys also stop hitting the table.

DynamoDB itself.

VPC - real operational surface area.

wrong answer for write-dominated workloads or when every read must be strongly consistent. When a stem demands microsecond DynamoDB reads with minimal code change, DAX is the intended answer.

Exam tip

Exam tip. DAX questions turn on what DAX cannot do: it serves eventually consistent reads only, and it does nothing for write hot partitions. If a stem demands strongly consistent reads on every call, or the bottleneck is writes to one hot key, DAX is the distractor - the answer is a key-design change or a different data path.

API Gateway caching

REST APIs can cache responses at the stage level, absorbing repeated reads before they reach your backend:

enable caching per stage, choose a cluster size, and set a time-to-live (default 300 seconds, maximum 3,600 seconds). Behavior details that show up in scenarios:

responses are capped at 1 MB, and the cache can be encrypted.

headers, path parameters). Two requests differing in an undesignated parameter return the same cached response - the classic source of "users see each other's data" bugs, fixed by including the discriminating parameter in the cache key or not caching that method at all.

that behavior; otherwise invalidation is an operator action (flush the stage cache), with a repopulation cost.

Caching fits when identical repeated reads crush the backend and the data tolerates seconds of staleness - not for personalized, rapidly changing, or write-path problems. Any scenario mentioning data leaking between users is testing the cache-key bullet above.

Exam tip

Exam tip. When a scenario says API Gateway caching made users see each other's data, do not reach for

"disable caching entirely" first - the tested fix is the cache key: include the parameter that distinguishes the responses (a header, query string, or path parameter) in the key, or stop caching that one method.

RDS Proxy

Lambda and relational databases have a structural mismatch: every execution environment opens its own connections, setup is slow, and a concurrency burst becomes a connection storm that can exhaust the database.

Amazon RDS Proxy sits between application and database as a managed connection pool:

concurrent functions no longer means a thousand database connections.

connections - AWS quotes failover times reduced by up to roughly two-thirds versus direct reconnects.

hard-coded database passwords from function configuration - the property that makes Proxy the answer in combined security-plus-performance scenarios.

statements - pins a client connection to one database connection for its lifetime, quietly defeating multiplexing. A workload that pins everything gains failover benefits but little pooling benefit. (Proxy targets RDS and Aurora; applications switch to the proxy endpoint, but query code is unchanged.)

Exam tip

"Lambda functions intermittently fail with too many connections errors against RDS during traffic spikes" is

RDS Proxy's home scenario. ElastiCache does not fix connection counts, and a bigger instance only moves the ceiling.

Check yourself

1. A table's reads concentrate on a few extremely popular items; write traffic is healthy, and eventually consistent reads are acceptable. Which lever absorbs the read hot spot?

2. During traffic spikes, hundreds of concurrent Lambda functions intermittently fail with "too many connections" errors against an RDS database. What fixes the connection count itself - not just the ceiling?

Answers: 1. DAX in front of the table - its item cache serves the repeated reads in microseconds. 2. RDS Proxy, which multiplexes the functions' connections onto a smaller pool of database connections.

AI-Assisted Troubleshooting

DVA-C03 threads AI assistance through every domain, and Domain 4's share is operational: using Amazon Q

Developer and CloudWatch's AI features to shorten the distance between symptom and root cause - knowing what they do and where their output must be verified.

Amazon Q Developer in the AWS console and IDE can reason over your actual environment: ask why a function is erroring, or what an error code means for a specific resource, and get an explanation plus candidate fixes grounded in AWS documentation and the resource's configuration. Treat its answers as a strong draft hypothesis - it can be confidently wrong about your configuration, and it acts within your IAM permissions, so least-privilege discipline applies to it as to any other tool.

CloudWatch investigations formalize that assistance around an incident. Starting from an alarm, a metric anomaly, or a log pattern, an investigation sweeps related telemetry - metrics, logs, traces, recent changes - and produces observations (supporting evidence) and root-cause hypotheses in natural language; lightweight, session-based investigations can also be launched from the console's operational troubleshooting views with no setup. In exam terms, it automates the correlation steps of the integration-failure playbook while the human still decides and acts.

Natural-language querying removes the syntax barrier in this chapter's tools. The Logs Insights query generator turns a plain-English request ("show me the 10 slowest Lambda invocations in the last day") into a correct query, explains it line by line, and refines it iteratively; the same capability generates Metrics Insights queries, and Logs Insights can produce a natural-language summary of query results. These features accelerate the authoring of queries, not the evidence itself - a generated query deserves the same read-before-you-run scrutiny as any generated code.

A closing note the exam's scenario writers share: AI assistance optimizes the hypothesis step, not the verification step. Every fix above still ends with a metric, a trace, or a log line proving the system now behaves.

Exam tip

When a scenario asks for help interpreting telemetry - generating a query from a description, summarizing results, forming a root-cause hypothesis - the answer is the AI feature. When it asks to collect telemetry, the answer is still the underlying feature: metric filters, EMF, X-Ray instrumentation, CloudTrail configuration. Keep the two layers separate and most of these questions resolve themselves.

End-of-Chapter Test

1. A team wants an alarm whenever their application logs the string "PaymentDeclined" more than ten times in five minutes. The application already writes plain-text logs to CloudWatch Logs. What should they configure first?

  1. A CloudWatch Logs Insights scheduled query that emails the team
  2. A metric filter matching PaymentDeclined that publishes a metric, plus a CloudWatch alarm on that metric
  3. An EMF block in the application emitting a PaymentDeclined metric
  4. A subscription filter to a Lambda function that increments a custom metric

2. Support staff must be able to find all traces for a specific order, and the full order payload must be stored with each trace for later inspection. How should the two pieces of data be attached?

  1. Both as annotations
  2. Both as metadata
  3. The order ID as an annotation; the payload as metadata
  4. The order ID as metadata; the payload as an annotation

3. An API serves 2,000 requests per second. Roughly how many requests per second does the default X-Ray sampling rule record?

  1. About 100 per second
  2. About 21 per second
  3. About 5 per second
  4. 2,000 per second at first, dropping after a quota is reached

4. Clients of a Regional REST API receive 504 responses after about 29 seconds on one endpoint. The backing Lambda function has a 60-second timeout and its logs show it completing successfully at around 40 seconds. What is the root cause and the cleanest fix?

  1. The Lambda function is throttling; add reserved concurrency
  2. The API Gateway integration timeout defaults to 29 seconds; request a Service Quotas increase for the integration timeout, or convert the operation to an asynchronous pattern
  3. The Lambda function memory is too low; increase it so the function finishes faster
  4. API Gateway caching is serving stale timeouts; flush the stage cache

5. A DynamoDB table shows heavy WriteThrottleEvents, yet consumed write capacity averages only 30% of provisioned capacity. Contributor Insights shows one partition key value receiving most writes. What is the correct diagnosis and remedy?

  1. Under-provisioned table; enable auto scaling
  2. Hot partition: a single key value exceeds the per-partition write ceiling; redesign the key - for example, write-sharding with a suffix - to spread writes
  3. GSI misconfiguration; delete and recreate the indexes
  4. DAX write-through failure; increase the DAX cluster size

6. A message with a malformed payload fails processing every time in a queue feeding a Lambda function, and each failure retries the entire batch of 10 messages. Which combination resolves both the poison message and the batch amplification?

  1. Increase the Lambda function timeout and the queue visibility timeout
  2. Configure a DLQ with a maxReceiveCount, and enable partial batch responses ( ReportBatchItemFailures) on the event source mapping
  3. Convert the queue to FIFO and enable content-based deduplication
  4. Reduce the batch size to 1 and delete the message manually

7. A security review asks who read a specific object from an S3 bucket last week. The account has CloudTrail enabled with default settings, but the review finds no record of the reads. Why?

  1. CloudTrail does not record S3 activity at all
  2. The event history only covers management events from the last 24 hours
  3. S3 object reads are data events, which are not recorded unless explicitly enabled for the bucket
  4. The trail must be organization-wide before any events are recorded

8. A CPU-bound Lambda function runs at 512 MB, averaging 8 seconds per invocation. The team wants to lower cost per invocation and suspects a higher memory setting may help. What is the most reliable way to choose the new setting?

  1. Set memory to the maximum 10,240 MB, since more memory is always cheaper per invocation
  2. Run AWS Lambda Power Tuning with representative payloads across several memory values and compare measured cost and duration under the cost strategy
  3. Halve the memory to 256 MB so each GB-second costs less
  4. Enable provisioned concurrency to reduce the billed duration

Answer key

1. B — Metric filters retrofit metrics onto logs you already emit, and alarms act on metrics - EMF (C) requires changing the application, and Insights queries (A) are interactive analysis, not alarming.

2. C — Annotations are indexed and searchable through filter expressions, so anything the team must find traces by belongs there; metadata is unindexed but accepts arbitrary JSON - the right home for a payload you only read after locating the trace.

3. A — The default rule records the first request each second (the reservoir) plus 5% of the rest: 1 + (0.05 × 1,999) ~ 101 traces per second.

4. B — The failure occurs at the integration timeout while the function itself succeeds - the signature of this mismatch. For Regional and private APIs the timeout can be raised via Service Quotas; otherwise return 202 and process asynchronously.

5. B — Throttling with abundant table-level headroom is the hot-partition signature: one partition is capped at about 1,000 WCU per second regardless of total capacity, so auto scaling (A) adds capacity the hot partition cannot use.

6. B — The redrive policy parks the message in the DLQ after a bounded number of attempts, for inspection and redrive after a fix; partial batch responses return only the failed item instead of the whole batch.

7. C — CloudTrail records management events by default; object-level operations such as S3 GetObject are data events, opted into per resource because of their volume - and they cannot be captured retroactively.

8. B — Because CPU scales with memory, CPU-bound functions often cost less at higher memory - but the optimum is workload-specific and must be measured, which is what the Power Tuning state machine automates.

Want the full AWS Developer – Associate (DVA-C03) package?

This guide is free to read. Get the downloadable PDF guidebook plus the full DVA-C03 practice exam with 3 months of online access.

Get DVA-C03 Full Access — $10