[core] introducing row-range-keyed DeletionVectors for DataEvolution tables

[steFaiz]

Purpose

The first part of #8322

This is a very BIG pr, I tried to split it into several sub PRs, but found it very hard.

The PR only focuses on core module, leaving Spark/Flink module untouched, mainly includes:

1. RowRange DeletionFiles are fully decoupled with data files, they are just to split one big deletion vector into many small files.
2. Introduce a DeletionFileKey to replace current filename as the unique identifier of each DeletionFile
3. Introduce a DataEvolutionApplyDvReader to filter out deleted row ids. This class refers to IndexedSplitRecordReader: First enrich readType by _ROW_ID, then filter rows by RowRange DeletionVectors.
4. Slightly changed SnapshotReaderImpl, assigning overlapping row range DeletionVectors to each DataSplit.

Besides, most changes are about changing many String fields to DeletionFileKey.

Tests

Unit Tests only.

[JingsongLi]

This reader only evaluates the first row-range DV whose end has not passed the current row id. If two row-range keyed DVs overlap, for example [0, 10] and [5, 15], rows 5-10 are never checked against the second DV, so deletions recorded only there will be returned. I don't see the writer/manifest combiner rejecting overlapping RowIdRangeKey values; they only deduplicate exact keys. Please either enforce non-overlapping row-range DVs when writing/combining metadata, or make this reader evaluate all DVs that may contain the row.

[steFaiz]

Thanks for your remind! I've added the check in GlobalCombiner and added a test

[JingsongLi]

For row-range keyed DVs this renders the compatibility marker row instead of the real row-id ranges, because metasToRowArrayData(..., FILE_NAME) intentionally fast-fails old readers. That makes table_indexes.dv_ranges misleading for data-evolution tables. Can we expose the row-range DV schema here as well, or render the ranges in a separate column, instead of showing the legacy marker?

[steFaiz]

This is very helpful! I've unified FileNameKey and RowRangeKey:

1. Reusing the same field
2. For FileNames, display the file name
3. For RowRanges, display the formated range. e.g. [0, 100]

[steFaiz]

I thought about the design and I think that making row ranged-DV aligned with normal file ranges is much more simple and easy to manage, compared to fully decoupled mode. Just like filename-based implementation.
The main logic:

1. At first DV generation, generate DVs aligned with existing normal data file ranges
2. During compaction, if some data files triggered row-level compaction and are merge into larger data file range, we merge related DVs too.

This is because it's complicated to update DVs if we let DVs can be fully decouple, consider this scenario:
Current DV ranges:

// Current DV:
[0, 100] , [101, 200], [230, 300]

Update a new DV range [50, 250]
This will involves:

1. among all DVs, binary search to find all overlapping DVs. here we find: [0, 100] , [101, 200] , [230, 300]
2. check each DV:
   a. For [0, 100], it's only partial overlapping with [50, 250], so we may have to split the new DV into [50, 100], [101, 250]
   b. For [101, 200] and [230, 300], we need to merge it with current splitted new DV: [101, 250], but how to merge them? Merge into a single big range [101, 300]? or split it into some selected size?

If we don't design sophiscated merge/split mechanism, there might be some DVs' size too big or too small.

I think we could:

1. At read path: capable of reading any decoupled DV ranges (already implemented in this PR)
2. At write/compaction path: force DV ranges be aligned with existing normal file ranges.

@JingsongLi Could you please review this proposal and share your thoughts? Any feedback would be greatly appreciated!