chore: make integration tests local

.github/workflows/integration_tests_local.yml

@@ -0,0 +1,43 @@
+name: local integration tests
+
+# Runs the docker-only integration test harness in integration_tests_local/
+# against the real AWS Lambda base images + the Runtime Interface Emulator
+# (RIE). No AWS account or credentials required.
+#
+# One job per Node major version; each job builds and tests both container
+# variants (cjs, esm), which share the same ~1 GB base image, so each job
+# pulls it exactly once. If public.ecr.aws anonymous pull limits ever bite,
+# the fix is to mirror the base images to GHCR (same pattern dd-trace-js
+# uses in its mirror-image.yml workflow).
+
+on:
+  pull_request:
+  push:
+    branches:
+      - "main"
+  workflow_dispatch:
+
+jobs:
+  container:
+    runs-on: ubuntu-latest
+
+    strategy:
+      fail-fast: false
+      matrix:
+        node_major: ["18", "20", "22", "24"]
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744 # v3.6.0
+
+      - name: Set up Node 18.12
+        uses: actions/setup-node@3235b876344d2a9aa001b8d1453c930bba69e610 # v3.9.1
+        with:
+          node-version: 18.12
+
+      - name: Run local container integration tests (nodejs${{ matrix.node_major }}.x)
+        # run.sh packs the local library (yarn build + npm pack), downloads
+        # the RIE binary, builds both variant images, invokes every input
+        # event, and diffs normalized logs against integration_tests_local/
+        # snapshots. PLATFORM selects the amd64 RIE binary and base images.
+        run: PLATFORM=linux/amd64 RUNTIME_PARAM=${{ matrix.node_major }} ./integration_tests_local/run.sh

.gitignore

@@ -17,3 +17,6 @@ package-lock.json
 
 # Local datadog-lambda-js tarballs produced for container integration tests
 integration_tests/container/*/datadog-lambda-js-local.tgz
+
+# Local integration test harness binaries
+integration_tests_local/bin/

integration_tests_local/.gitignore

@@ -0,0 +1 @@
+bin/

integration_tests_local/README.md

@@ -0,0 +1,135 @@
+# Local integration tests (docker + AWS RIE, no AWS account required)
+
+This directory contains a **local** integration test harness for
+datadog-lambda-js. It runs the container-image handler variants
+(`integration_tests/container/cjs` and `integration_tests/container/esm`)
+inside Docker against the
+[AWS Lambda Runtime Interface Emulator (RIE)](https://github.com/aws/aws-lambda-runtime-interface-emulator),
+invokes them with the same input events as the AWS-based suite, captures
+logs from `docker logs`, normalizes them with the **same** pipeline as
+`scripts/run_integration_tests.sh`, and diffs them against **local**
+snapshots in `./snapshots/`.
+
+Nothing here touches AWS, and nothing here touches
+`integration_tests/snapshots/` (the AWS suite's snapshots).
+
+## Prerequisites
+
+- Docker (tested with colima on macOS arm64)
+- `node`, `yarn`, `perl`, `sed`, `curl` (all already used by the AWS suite)
+- Network access for the first run (pulls `public.ecr.aws/lambda/nodejs:*`
+  base images, ~1 GB each, and downloads the RIE binary once into `./bin/`)
+
+## Running
+
+```bash
+# Everything: nodejs 18/20/22/24 x {cjs, esm}, diff against local snapshots
+./integration_tests_local/run.sh
+
+# One runtime / one variant
+RUNTIME_PARAM=18 VARIANT_PARAM=esm ./integration_tests_local/run.sh
+
+# (Re)generate local snapshots
+UPDATE_SNAPSHOTS=true ./integration_tests_local/run.sh
+
+# Skip repacking the library (reuse container/*/datadog-lambda-js-local.tgz)
+SKIP_PACK=true RUNTIME_PARAM=18 VARIANT_PARAM=esm ./integration_tests_local/run.sh
+
+# Force amd64 images instead of arm64
+PLATFORM=linux/amd64 ./integration_tests_local/run.sh
+```
+
+Unless `SKIP_PACK=true` is set, each run repacks the library under test
+(`yarn install --frozen-lockfile && yarn build && npm pack`) into
+`integration_tests/container/{cjs,esm}/datadog-lambda-js-local.tgz`, exactly
+like `scripts/run_integration_tests.sh` does, so the containers always test
+the working tree.
+
+## Proactive-initialization simulation
+
+```bash
+SIMULATE_PROACTIVE_INIT=true RUNTIME_PARAM=18 VARIANT_PARAM=esm ./integration_tests_local/run.sh
+```
+
+The library stamps `initTime = Date.now()` at wrapper-module load
+(`src/index.ts`) and marks the sandbox as proactively initialized when the
+first invocation starts more than 10 s after that
+(`src/utils/cold-start.ts`). On real Lambda, AWS sometimes runs the init
+phase well before the first invoke (proactive initialization), flipping that
+flag.
+
+Reproducing this under RIE takes two ingredients, because **the classic RIE
+runs the init phase lazily on the first invocation** — a post-start `sleep`
+alone creates no init→invoke gap (verified: with only a sleep, the logs are
+byte-identical to an immediate run):
+
+1. `run.sh` sets `AWS_LAMBDA_MAX_CONCURRENCY=1`, which switches the same RIE
+   binary into its managed-instances path; that path performs the init phase
+   (bootstrap + wrapper module load) **eagerly at container start**. Side
+   effects are contained: the function env gains
+   `AWS_LAMBDA_INITIALIZATION_TYPE=lambda-managed-instances`, which the
+   library only uses to gate cold-start tracing spans (already disabled here
+   via `DD_COLD_START_TRACING=false`), and `AWS_LAMBDA_LOG_FORMAT=text` is
+   pinned to keep runtime logs in the classic text format.
+2. `run.sh` then sleeps 15 s before the first invocation, producing an
+   init→invoke gap > 10 s.
+
+The first invocation then emits, exactly as in the CI failure on
+`container-esm_node18`:
+
+- `"proactive_initialization": 1` in the `aws.lambda` span's `metrics`,
+- `"proactive_initialization:true"` in the `aws.lambda.enhanced.invocations`
+  metric tags,
+- `cold_start:false` (instead of `true`) on the first invocation.
+
+A `SIMULATE_PROACTIVE_INIT=true` run is therefore **expected to fail** the
+diff against the immediate-run snapshots, with precisely those lines added
+(plus RIE-mode platform-log noise: `INIT/REPORT` line shapes and the emulated
+region/account in ARNs differ between the two RIE paths). The platform-side
+half of the CI diff, `END Duration: XXXX ms (init: XXXX ms)`, is emitted by
+the real Lambda platform and does not appear under RIE in either mode.
+
+## Files
+
+- `run.sh` — the runner (build images, run under RIE, invoke, diff snapshots)
+- `normalize.sh` — the log-normalization pipeline, factored out of
+  `scripts/run_integration_tests.sh` (lines 212-248) so both suites
+  normalize identically. Reads stdin, writes stdout; honors `RUN_ID` for
+  the AWS suite's per-run ID stripping.
+- `bin/` — downloaded RIE binary (gitignored)
+- `snapshots/logs/` — normalized log snapshots per variant+runtime
+- `snapshots/return_values/` — per-event handler return-value snapshots
+
+## Comparison with the AWS-based suite
+
+| | AWS suite (`scripts/run_integration_tests.sh`) | this harness |
+|---|---|---|
+| handlers | 9 variants (layers + container images) | container images only (cjs, esm) |
+| infra | real Lambda via serverless, CloudWatch logs | docker + RIE, `docker logs` |
+| snapshots | `integration_tests/snapshots/` | `integration_tests_local/snapshots/` |
+| credentials | AWS account + DD_API_KEY | none |
+| cost/wait | deploy + invoke + 20 s log wait | image build + invoke |
+
+Local snapshots legitimately differ from the AWS ones (fake account/region
+context, no real API Gateway IDs, RIE-formatted `START`/`END`/`REPORT`
+lines, no platform `init:` duration suffix). Do not diff one suite's output
+against the other's snapshots.
+
+## Known emulation gaps (RIE vs real Lambda)
+
+- The classic RIE runs the init phase **lazily on the first invocation**, so
+  init→invoke timing behavior (proactive initialization) cannot be observed
+  in the default mode; use `SIMULATE_PROACTIVE_INIT=true`, which switches
+  RIE to its eager-init managed-instances path (see above).
+- No platform `INIT_START` / `END ... (init: N ms)` lines — those come from
+  the Lambda platform, not the runtime.
+- No real AWS service context: API Gateway/DynamoDB/S3/SNS/SQS resource ARNs,
+  account IDs, and inferred-span metadata are derived only from the event
+  payloads, so they differ from the AWS snapshots.
+- `AWS_REGION` is faked to `eu-west-1` to keep the enhanced-metric region
+  tag stable.
+- Enhanced metrics that depend on platform-provided values (e.g. real
+  memory size / billed duration) may be absent or differ.
+- Layer-based handler variants are not exercised here (they need the built
+  layer zip and a zip-based deployment model; the container variants cover
+  the same npm-installed library code path).

integration_tests_local/normalize.sh

@@ -0,0 +1,81 @@
+#!/bin/bash
+
+# Shared log-normalization pipeline for integration test snapshots.
+#
+# This is the exact filter chain used by scripts/run_integration_tests.sh
+# (lines 212-248) to replace invocation-specific data (timestamps, IDs,
+# durations, ...) with XXXX before diffing logs against snapshots.
+# It is factored out here so both the AWS-based suite and the local
+# RIE-based harness (integration_tests_local/run.sh) normalize identically.
+#
+# Usage:
+#   some-log-producer | ./normalize.sh
+#   RUN_ID=abcdef12 ./normalize.sh < raw.log > normalized.log
+#
+# Reads from stdin, writes to stdout.
+# Optional env:
+#   RUN_ID - random per-run ID embedded in deployed function names; stripped
+#            to XXXX when set (AWS suite only; the local harness uses static
+#            function names and leaves this unset).
+
+set -e
+
+script_path=${BASH_SOURCE[0]}
+local_dir=$(dirname "$script_path")
+repo_dir=$(dirname "$local_dir")
+
+run_id_filter='s/$/^/' # no-op by default (matches nothing useful, harmless)
+if [ -n "$RUN_ID" ]; then
+    run_id_filter="s/${RUN_ID}/XXXX/g"
+fi
+
+node "$repo_dir/integration_tests/parse-json.js" |
+    # Filter serverless cli errors
+    sed '/Serverless: Recoverable error occurred/d' |
+    # Normalize Lambda runtime report logs
+    perl -p -e 's/(RequestId|TraceId|init|SegmentId|Duration|Memory Used|"e"):( )?[a-z0-9\.\-]+/\1:\2XXXX/g' |
+    # Normalize DD APM headers and AWS account ID
+    perl -p -e "s/(x-datadog-parent-id:|x-datadog-trace-id:|account_id:)[0-9]+/\1XXXX/g" |
+    # Strip API key from logged requests
+    perl -p -e "s/(api_key=|'api_key': ')[a-z0-9\.\-]+/\1XXXX/g" |
+    # Normalize log timestamps
+    perl -p -e "s/[0-9]{4}\-[0-9]{2}\-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]+( \(\-?\+?[0-9:]+\))?/XXXX-XX-XX XX:XX:XX.XXX/" |
+    # Normalize DD trace ID injection
+    perl -p -e "s/(dd\.trace_id=)[0-9]+ (dd\.span_id=)[0-9]+/\1XXXX \2XXXX/" |
+    # Normalize execution ID in logs prefix
+    perl -p -e $'s/[0-9a-z]+\-[0-9a-z]+\-[0-9a-z]+\-[0-9a-z]+\-[0-9a-z]+\t/XXXX-XXXX-XXXX-XXXX-XXXX\t/' |
+    # Normalize minor package version tag so that these snapshots aren't broken on version bumps
+    perl -p -e "s/(dd_lambda_layer:datadog-nodev[0-9]+\.)[0-9]+\.[0-9]+/\1XX\.X/g" |
+    perl -p -e 's/"(span_id|apiid|runtime-id|record_ids|parent_id|trace_id|start|duration|tcp\.local\.address|tcp\.local\.port|dns\.address|request_id|function_arn|x-datadog-trace-id|x-datadog-parent-id|datadog_lambda|dd_trace|process_id)":\ ("?)[a-zA-Z0-9\.:\-]+("?)/"\1":\2XXXX\3/g' |
+    # Strip out run ID (from function name, resource, etc.)
+    perl -p -e "$run_id_filter" |
+    # Normalize line numbers in stack traces
+    perl -p -e 's/(.js:)[0-9]*:[0-9]*/\1XXX:XXX/g' |
+    # Remove metrics and metas in logged traces (their order is inconsistent)
+    perl -p -e 's/"(meta|metrics)":{(.*?)}/"\1":{"XXXX": "XXXX"}/g' |
+    # Normalize enhanced metric datadog_lambda tag
+    perl -p -e "s/(datadog_lambda:v)[0-9\.]+/\1X.X.X/g" |
+    # Normalize lookup resource
+    perl -p -e "s/(\"resource\":\"169.)[0-9\.]+/\1X.X.X/g" |
+    # Normalize Axios version
+    perl -p -e "s/User-Agent:axios\/\d+\.\d+\.\d+/User-Agent:axios\/X\.X\.X/g" |
+    # Remove init start line
+    perl -p -e "s/INIT_START.*//g" |
+    # Drop proactive-initialization markers only: whether a sandbox was
+    # proactively initialized (>10s between init and first invoke, see
+    # src/utils/cold-start.ts) is platform scheduling, not code behavior.
+    # cold_start values are NOT normalized — the cold->warm transition
+    # (invoke #1 cold_start:true, #2..N false) is deliberate coverage,
+    # and is deterministic locally since proactive init cannot happen
+    # unless SIMULATE_PROACTIVE_INIT=true.
+    sed '/proactive_initialization/d' |
+    perl -p -e 's/ \(init: [^)]*\)//g' |
+    # Normalize RIE platform log lines (local harness only; no-op on AWS logs):
+    # "28 Jul 2026 19:42:34,536 [INFO] (rapid) ..." timestamps, request ids,
+    # and init/invoke durations vary run to run
+    perl -p -e 's/^[0-9]{2} \w{3} [0-9]{4} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3} (\[INFO\] \(rapid\))/XXXX \1/' |
+    perl -p -e 's/(requestId: )[0-9a-f-]+/\1XXXX/g' |
+    perl -p -e 's/(duration(Ms)?: )[0-9.]+/\1XXXX/g' |
+    sed -E "s/(tracestate\:)([A-Za-z0-9\-\=\:\;].+)/\1XXX/g" |
+    sed -E "s/(\"_dd.p.tid\"\: \")[a-z0-9\.\-]+/\1XXXX/g" |
+    sed -E "s/(_dd.p.tid=)[a-z0-9\.\-]+/\1XXXX/g"