mcvsubbu commented on a change in pull request #6710:
URL: https://github.com/apache/incubator-pinot/pull/6710#discussion_r603479529
##########
File path:
pinot-common/src/main/java/org/apache/pinot/common/utils/DataTable.java
##########
@@ -46,8 +52,120 @@
String RESIZE_TIME_MS_METADATA_KEY = "resizeTimeMs";
String EXECUTION_THREAD_CPU_TIME_NS_METADATA_KEY =
"executionThreadCpuTimeNs";
+ /* The MetadataKeys is used in V3, where we present metadata as
Map<MetadataKeys, String>
+ * ATTENTION:
+ * - Don't add new key which has same id/name with existing keys. Duplicate
id/name is not allowed.
+ * - Don't change id/name of existing keys.
+ * Otherwise, backward compatibility will be broken.
+ */
+ enum MetadataKeys {
+ UNKNOWN(0, "unknown"),
+ TABLE_KEY(1, "table"), // NOTE: this key is only used in
PrioritySchedulerTest
+ EXCEPTION_METADATA_KEY(2, "Exception"),
+ NUM_DOCS_SCANNED_METADATA_KEY(3, "numDocsScanned"),
+ NUM_ENTRIES_SCANNED_IN_FILTER_METADATA_KEY(4, "numEntriesScannedInFilter"),
+ NUM_ENTRIES_SCANNED_POST_FILTER_METADATA_KEY(5,
"numEntriesScannedPostFilter"),
+ NUM_SEGMENTS_QUERIED(6, "numSegmentsQueried"),
+ NUM_SEGMENTS_PROCESSED(7, "numSegmentsProcessed"),
+ NUM_SEGMENTS_MATCHED(8, "numSegmentsMatched"),
+ NUM_CONSUMING_SEGMENTS_PROCESSED(9, "numConsumingSegmentsProcessed"),
+ MIN_CONSUMING_FRESHNESS_TIME_MS(10, "minConsumingFreshnessTimeMs"),
+ TOTAL_DOCS_METADATA_KEY(11, "totalDocs"),
+ NUM_GROUPS_LIMIT_REACHED_KEY(12, "numGroupsLimitReached"),
+ TIME_USED_MS_METADATA_KEY(13, "timeUsedMs"),
+ TRACE_INFO_METADATA_KEY(14, "traceInfo"),
+ REQUEST_ID_METADATA_KEY(15, "requestId"),
+ NUM_RESIZES_METADATA_KEY(16, "numResizes"),
+ RESIZE_TIME_MS_METADATA_KEY(17, "resizeTimeMs"),
+ EXECUTION_THREAD_CPU_TIME_NS_METADATA_KEY(18, "executionThreadCpuTimeNs"),
+ RESPONSE_SERIALIZATION_CPU_TIME_NS_METADATA_KEY(19,
"responseSerializationCpuTimeNs"),
+ ;
+
+ private static final Map<Integer, MetadataKeys> _idToEnumKeyMap = new
HashMap<>();
+ private static final Map<String, MetadataKeys> _nameToEnumKeyMap = new
HashMap<>();
+ // _intValueTrailerKeys contains all metadata keys which has value of int
type.
+ private static final Set<MetadataKeys> _intValueMetadataKeys =
ImmutableSet.of(
+ MetadataKeys.NUM_SEGMENTS_QUERIED,
+ MetadataKeys.NUM_SEGMENTS_PROCESSED,
+ MetadataKeys.NUM_SEGMENTS_MATCHED,
+ MetadataKeys.NUM_RESIZES_METADATA_KEY,
+ MetadataKeys.NUM_CONSUMING_SEGMENTS_PROCESSED,
+ MetadataKeys.NUM_RESIZES_METADATA_KEY
+ );
+ // _longValueTrailerKeys contains all metadata keys which has value of
long type.
+ private static final Set<MetadataKeys> _longValueTrailerKeys =
ImmutableSet.of(
Review comment:
let us keep trailer/metadata consistent.
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/common/datatable/DataTableBuilder.java
##########
@@ -91,9 +107,17 @@
private ByteBuffer _currentRowDataByteBuffer;
public DataTableBuilder(DataSchema dataSchema) {
+ CURRENT_VERSION = VERSION_3;
Review comment:
I think we should make this configurable for until we migrate to v3 and
remove v2 code. Default should be V3 (so that if anyone upgrades broker first.
they are fine)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
