J-HowHuang opened a new pull request, #18998:
URL: https://github.com/apache/pinot/pull/18998
## Description
Before #17965, we might see a raw column is unsorted according to
`metadata.properties` but is actually sorted (false negative). For example:
```
column.xxx.cardinality = -2147483648
column.xxx.totalDocs = 48
column.xxx.dataType = LONG
column.xxx.bitsPerElement = 31
column.xxx.lengthOfEachEntry = 0
column.xxx.columnType = DATE_TIME
column.xxx.isSorted = false
column.xxx.hasDictionary = false
column.xxx.isSingleValues = true
column.xxx.totalNumberOfEntries = 48
column.xxx.minValue = 0
column.xxx.maxValue = 0
```
So when the table config enables dictionary on this column, during the
segment load/reload, the forward index rebuild in `updateIndicies`. Yet the
forward index creator is constructed using an
`AbstractColumnStatisticsCollector`, which actually read through the column and
report sorted.
Thus a `SingleValueSortedForwardIndexCreator` would be created here, write
index to a file with a different file extension `.sv.sorted.fwd`. Later segment
writer would try to read from `.sv.unsorted.fwd` because it determines the
extension by metadata's sortedness.
So we see
```
java.io.FileNotFoundException:
.../xxx.sv.unsorted.fwd (No such file or directory)
at java.base/java.io.RandomAccessFile.open0(Native Method)
at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:356)
at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:273)
at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:223)
at
org.apache.pinot.segment.spi.memory.PinotByteBuffer.readFrom(PinotByteBuffer.java:295)
at
org.apache.pinot.segment.local.segment.index.loader.LoaderUtils.writeIndexToV3Format(LoaderUtils.java:57)
```
## Change
We should follow the metadata as the source of truth. IMO the easiest change
here is to override the sortedness in `IndexCreationContext` with the value in
segment metadata.
## Test
Add an unit test to reproduce the failure scenario and verified it fails
without the change, and pass with the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
