Skip to content

test(php): cover OpenFeature in all weblog environments - #7393

Draft
leoromanovsky wants to merge 4 commits into
mainfrom
leo.romanovsky/php-openfeature-all-environments
Draft

test(php): cover OpenFeature in all weblog environments#7393
leoromanovsky wants to merge 4 commits into
mainfrom
leo.romanovsky/php-openfeature-all-environments

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Motivation

A customer found that an application error handler can convert a provider warning into an exception. OpenFeature then returns the code default with a GENERAL error.

The existing PHP FFE tests called the native Datadog client. They did not cross the OpenFeature provider and logger boundary where this failure occurred.

flowchart LR
    App["Application"] --> Client["OpenFeature client"]
    Client --> Provider["Datadog provider"]
    Provider --> Evaluator["Datadog evaluator"]
    Provider -. "warning" .-> Handler["Application error handler"]
    Handler --> Exception["Exception"]
    Exception --> Error["Code default plus GENERAL error"]

    Test["Previous FFE test"] --> Endpoint["POST /ffe"]
    Endpoint --> Native["Native Datadog client"]
    Native --> Evaluator
    Test -. "missed this boundary" .-> Provider
Loading

The OpenFeature PHP SDK requires PHP 8 or later. System tests still support PHP 7. PHP 7 must keep the /ffe contract when the SDK is unavailable.

Changes and Decisions

The /ffe endpoint now uses OpenFeature by default. Tests that need the native client can set evaluationApi to native.

The generic test omits evaluationApi. An available provider must return the configured value. An unavailable provider must return the supplied default with ERROR and PROVIDER_NOT_READY.

flowchart TD
    Test["Generic FFE test"] --> Endpoint["POST /ffe"]
    Endpoint --> Default["Default to OpenFeature"]
    Default --> Available{"Provider available?"}
    Available -->|"Yes"| Success["Configured value and no error"]
    Available -->|"No"| Fallback["Supplied default<br/>ERROR<br/>PROVIDER_NOT_READY"]
    Success --> Contract["Contract passes"]
    Fallback --> Contract
Loading

The shared PHP installer adds the OpenFeature SDK on PHP 8 and later. Laravel and Symfony use the shared /ffe handler. PHP 7 marks provider-dependent tests as unavailable but keeps the generic contract test enabled.

The PHP 8 success path depends on DataDog/dd-trace-php#4071.

Validation

These local runs used this PR at 0dad5d48bec63cffdd206b724013d26f497c0c8c. They used the CI-built ARM64 package from dd-trace-php bf360ef80839f483179e37215536d1713116167c.

TEST_LIBRARY=php WEBLOG_VARIANT=php-fpm-8.2 ./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION
37 passed, 9 skipped, 2735 deselected in 153.98s

TEST_LIBRARY=php WEBLOG_VARIANT=laravel11x ./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION tests/ffe/test_dynamic_evaluation.py::Test_FFE_OpenFeature_Evaluation
1 passed in 34.02s

TEST_LIBRARY=php WEBLOG_VARIANT=symfony7x ./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION tests/ffe/test_dynamic_evaluation.py::Test_FFE_OpenFeature_Evaluation
1 passed in 34.07s

TEST_LIBRARY=php WEBLOG_VARIANT=php-fpm-7.4 ./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION
3 passed, 9 skipped, 29 xfailed, 5 xpassed, 2735 deselected in 210.78s

The PHP-FPM 8.2 image used PHP 8.2.32. All four images reported tracer version 1.24.0+dev.bf360ef80839f483179e37215536d1713116167c.

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

CODEOWNERS have been resolved as:

docs/understand/weblogs/end-to-end_weblog.md                            @DataDog/system-tests-core
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
tests/ffe/test_dynamic_evaluation.py                                    @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
utils/build/docker/php/apache-mod/build.sh                              @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/common/ffe.php                                   @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/common/install_ddtrace.sh                        @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/weblogs/laravel11x/routes/web.php                @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/weblogs/symfony7x/src/Controller/AppController.php  @DataDog/apm-php @DataDog/system-tests-core

@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Pipelines  Tests

Unblock PR with BitsAI

⚠️ Warnings

🚦 40 Pipeline jobs failed

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-7.2-zts 1   View in Datadog   GitHub Actions

🧪 1 Test failed

tests.appsec.smoke_tests.test_apm_standalone.Test_AppSecAPMStandalone_Telemetry.test_telemetry_smoke[apache-mod-7.2-zts] from system_tests_suite   View in Datadog
assert False

self = &lt;tests.appsec.smoke_tests.test_apm_standalone.Test_AppSecAPMStandalone_Telemetry object at 0x7fe7d52fc3e0&gt;

    def test_telemetry_smoke(self) -&gt; None:
        telemetry_data = list(interfaces.agent.get_telemetry_data(flatten_message_batches=False))
    
        assert telemetry_data, &#34;Agent should forward telemetry data from the library&#34;
    
        found_metrics = False
...

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-7.4 1   View in Datadog   GitHub Actions

🧪 1 Test failed

tests.appsec.smoke_tests.test_apm_standalone.Test_AppSecAPMStandalone_Telemetry.test_telemetry_smoke[apache-mod-7.4] from system_tests_suite   View in Datadog
assert False

self = &lt;tests.appsec.smoke_tests.test_apm_standalone.Test_AppSecAPMStandalone_Telemetry object at 0x7f2242a5f1d0&gt;

    def test_telemetry_smoke(self) -&gt; None:
        telemetry_data = list(interfaces.agent.get_telemetry_data(flatten_message_batches=False))
    
        assert telemetry_data, &#34;Agent should forward telemetry data from the library&#34;
    
        found_metrics = False
...

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-8.2-zts 1   View in Datadog   GitHub Actions

🧪 1 Test failed

tests.appsec.smoke_tests.test_apm_standalone.Test_AppSecAPMStandalone_Telemetry.test_telemetry_smoke[apache-mod-8.2-zts] from system_tests_suite   View in Datadog
assert False

self = &lt;tests.appsec.smoke_tests.test_apm_standalone.Test_AppSecAPMStandalone_Telemetry object at 0x7f07ecf15be0&gt;

    def test_telemetry_smoke(self) -&gt; None:
        telemetry_data = list(interfaces.agent.get_telemetry_data(flatten_message_batches=False))
    
        assert telemetry_data, &#34;Agent should forward telemetry data from the library&#34;
    
        found_metrics = False
...

View all 40 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🔄 Datadog auto-retried 9 jobs - 2 passed on retry View in Datadog

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 0dad5d4 | Docs | Datadog PR Page | Give us feedback!

@leoromanovsky

Copy link
Copy Markdown
Contributor Author

Validated against the current PHP fix build.

  • system-tests head: 0dad5d48bec63cffdd206b724013d26f497c0c8c
  • dd-trace-php fix head: bf360ef80839f483179e37215536d1713116167c
  • CI-built ARM64 PHP package: 1.24.0+dev.bf360ef80839f483179e37215536d1713116167c
  • PHP-FPM runtime: PHP 8.2.32

Local results:

php-fpm-8.2, complete FEATURE_FLAGGING_AND_EXPERIMENTATION scenario
37 passed, 9 skipped, 2735 deselected in 153.98s

laravel11x, Test_FFE_OpenFeature_Evaluation
1 passed in 34.02s

symfony7x, Test_FFE_OpenFeature_Evaluation
1 passed in 34.07s

php-fpm-7.4, complete FEATURE_FLAGGING_AND_EXPERIMENTATION scenario
3 passed, 9 skipped, 29 xfailed, 5 xpassed, 2735 deselected in 210.78s

The current GitHub Actions run does not use this PHP build:

  • All 12 production PHP jobs use php@1.23.1. Their only failed step is the FFE scenario. These jobs need a release that contains the PHP fix.
  • The 12 PHP 8 development jobs use php@1.24.0+dev.5bf45ce4ececff3bc5c046464636ce758c839a3c. That build predates the fix. Their FFE failures should stop after [java] Update mssql-jdbc version #4071 merges and the development package refreshes. They do not need a public release.
  • All 27 development PHP jobs also fail unrelated AppSec telemetry scenarios. The 15 PHP 7 development jobs do not fail the FFE step. Their red status comes only from those AppSec failures.

Therefore, the PHP fix will unblock the FFE failures. It will not make every PHP job green because the AppSec failures are separate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant