[core] Support arbitrary time granularity for chain table delta computation#8185
[core] Support arbitrary time granularity for chain table delta computation#8185juntaozhang wants to merge 4 commits into
Conversation
| String fingerprint = DateTimeFormatter.ofPattern(formatter).format(FINGERPRINT); | ||
|
|
||
| checkArgument( | ||
| fingerprint.length() == formatter.length(), |
There was a problem hiding this comment.
This rejects valid partition.timestamp-formatter values that use DateTimeFormatter quoting, for example yyyyMMdd'T'HHmm. PartitionTimeExtractor and calPartValues can handle a pattern such as $dtT$hm with that formatter, but this check compares the formatted output length (20260609T1150) with the raw formatter pattern length including quote characters, so chain reads fail before delta partitions are enumerated. Please parse formatter fields without relying on raw pattern positions, or validate/reject these formatter patterns consistently before they can be used.
|
|
||
| private static final LocalDateTime FINGERPRINT = LocalDateTime.of(2026, 6, 9, 11, 50, 58); | ||
| private static final String TIME_UNIT_CHARS = "yMdHhmsS"; | ||
| private static final Pattern VARIABLE = Pattern.compile("\\$[a-zA-Z_]+"); |
There was a problem hiding this comment.
This variable regex stops before digits, so a valid partition column like dt1 is parsed as variable $dt plus a constant 1. For partition.timestamp-pattern = "$dt1" and formatter yyyyMMdd, the extractor then treats the day digit as a fixed constant and returns a monthly step, causing getDeltaPartitions to generate only month boundaries and miss daily delta partitions. Please include the same identifier characters that partition fields can use, e.g. digits after the first character, when splitting timestamp-pattern variables.
| public class ChainPartitionStepExtractor { | ||
|
|
||
| private static final LocalDateTime FINGERPRINT = LocalDateTime.of(2026, 6, 9, 11, 50, 58); | ||
| private static final String TIME_UNIT_CHARS = "yMdHhmsS"; |
There was a problem hiding this comment.
S is advertised as a time unit here, but resolveField cannot map fractional seconds: the fingerprint has zero nanos, so a formatter containing SSS produces 000, and Long.parseLong("000") does not match any year/month/day/hour/minute/second value. A valid Java formatter such as yyyyMMddHHmmssSSS will therefore throw Unknown time unit value: 000 during chain delta planning. Please either implement a fractional-second step or remove/reject S consistently instead of accepting it in TIME_UNIT_CHARS.
| .collectAsList().stream() | ||
| .map(Row::toString) | ||
| .collect(Collectors.toList())) | ||
| .containsExactlyInAnyOrder( |
There was a problem hiding this comment.
This new minute-level scenario currently fails locally with mvn -pl paimon-spark/paimon-spark-ut -am -Pfast-build -DfailIfNoTests=false -Dtest=SparkChainTableITCase#testChainTableWithMinuteLevelPartitions test: the query for dt=20250810 and hr_min=03:40 also returns [7,1,7,20250810,03:40] from the 03:45 delta partition. That means the chain planner is still reading delta partitions after the requested chain partition, so this feature can expose future delta rows when a partition filter is present. Please tighten the delta partition selection to (anchor, requested] for the actual requested partition.
Purpose
#8184
Tests