Skip to content

fix(pandas): version-conditional timestamp resolution for to_pandas() (ns on v2, ms on v3) - #2724

Open
haakonvt wants to merge 15 commits into
masterfrom
fix-pandas-v3-new
Open

fix(pandas): version-conditional timestamp resolution for to_pandas() (ns on v2, ms on v3)#2724
haakonvt wants to merge 15 commits into
masterfrom
fix-pandas-v3-new

Conversation

@haakonvt

@haakonvt haakonvt commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Pandas v3 removed the implicit nanosecond default for datetimes, so to_pandas() output would get arbitrary precision depending on input. This pins resolution explicitly: ns on pandas v2 (matching old behavior), ms on pandas v3 (matching CDF's native resolution), via shared helpers in _pandas_helpers.py used across every timestamp-producing code path.

"Single" resource objects (aka not the list version), now delegate to the list-class to_pandas method to avoid having duplicated logic. This relies on _LIST_CLASS / _RESOURCE back-references staying in sync, so test_meta.py now enforces it: every CogniteResourceList._RESOURCE must point to a resource class whose _LIST_CLASS points right back. A class added without wiring up its counterpart now fails CI instead of silently duplicating conversion logic later. Also adds a test_standalone_to_pandas_allowlist test documenting the deliberate exceptions (e.g. Datapoint, RowCore) that don't follow this pattern.

For the pandas versioning itself, added tests/tests_unit/test_utils/test_pandas_helpers.py with a force_pandas_major_version fixture that monkeypatches the version check, so both the ns and ms branches run in every CI invocation regardless of which pandas is actually installed.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request standardizes the to_pandas export behavior across the SDK to use millisecond precision (datetime64[ms]) for timestamp and datetime columns, aligning with the Cognite API and ensuring compatibility with pandas v3. To eliminate duplicate conversion logic, CogniteResource.to_pandas now delegates to its corresponding list class via a new _LIST_CLASS class variable, with a fallback helper for resource types without a list counterpart. Custom to_pandas overrides (such as in Group and Instances) have been simplified or removed in favor of this unified delegation pattern, and meta-tests have been added to enforce the back-references between resource and list classes. No review comments were provided, so there is no feedback to assess.

@haakonvt
haakonvt force-pushed the fix-pandas-v3-new branch 2 times, most recently from 797e971 to cf6d97b Compare July 20, 2026 14:01
@codecov

codecov Bot commented Jul 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.16318% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.69%. Comparing base (8b99b98) to head (cf6d97b).

Files with missing lines Patch % Lines
cognite/client/utils/_pandas_helpers.py 90.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2724      +/-   ##
==========================================
- Coverage   93.71%   93.69%   -0.03%     
==========================================
  Files         504      504              
  Lines       51294    51476     +182     
==========================================
+ Hits        48070    48229     +159     
- Misses       3224     3247      +23     
Files with missing lines Coverage Δ
cognite/client/data_classes/_base.py 93.70% <100.00%> (-0.07%) ⬇️
cognite/client/data_classes/agents/agent_tools.py 98.67% <100.00%> (+0.01%) ⬆️
cognite/client/data_classes/agents/agents.py 100.00% <100.00%> (ø)
cognite/client/data_classes/agents/chat.py 97.11% <100.00%> (+0.01%) ⬆️
cognite/client/data_classes/aggregations.py 89.32% <100.00%> (+0.04%) ⬆️
cognite/client/data_classes/annotations.py 97.89% <100.00%> (+0.04%) ⬆️
cognite/client/data_classes/assets.py 94.47% <100.00%> (+0.02%) ⬆️
cognite/client/data_classes/capabilities.py 97.01% <100.00%> (+<0.01%) ⬆️
cognite/client/data_classes/contextualization.py 91.24% <100.00%> (+0.05%) ⬆️
...te/client/data_classes/data_modeling/containers.py 97.70% <100.00%> (+0.01%) ⬆️
... and 59 more

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@haakonvt
haakonvt force-pushed the fix-pandas-v3-new branch 2 times, most recently from 1d686d4 to ec5f01a Compare August 10, 2026 14:06
@haakonvt haakonvt changed the title fix(pandas v3): support pandas v3 by enforcing millisecond timestamp precision fix(pandas): version-conditional timestamp resolution for to_pandas() (ns on v2, ms on v3) Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant