siddharthteotia commented on a change in pull request #7830: URL: https://github.com/apache/pinot/pull/7830#discussion_r764330022
########## File path: pinot-core/src/main/java/org/apache/pinot/core/operator/filter/BlockDrivenAndFilterOperator.java ########## @@ -0,0 +1,126 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.core.operator.filter; + +import java.util.ArrayList; +import java.util.List; +import javax.annotation.Nullable; +import org.apache.pinot.core.common.BlockDocIdSet; +import org.apache.pinot.core.common.Operator; +import org.apache.pinot.core.operator.VisitableOperator; +import org.apache.pinot.core.operator.blocks.FilterBlock; +import org.apache.pinot.core.operator.blocks.TransformBlock; +import org.apache.pinot.core.operator.dociditerators.ArrayBasedDocIdIterator; +import org.apache.pinot.core.operator.docidsets.AndDocIdSet; +import org.apache.pinot.core.operator.docidsets.ArrayBasedDocIdSet; +import org.apache.pinot.core.operator.docidsets.BitmapDocIdSet; +import org.apache.pinot.core.operator.docidsets.FilterBlockDocIdSet; +import org.apache.pinot.segment.spi.Constants; +import org.roaringbitmap.buffer.ImmutableRoaringBitmap; + +/** + * Performs an AND operation on top of a Filter Block DocIDSet + * and a block from the given filter operator. + */ +public class BlockDrivenAndFilterOperator extends BaseFilterOperator + implements VisitableOperator { + private static final String OPERATOR_NAME = "BlockDrivenAndFilterOperator"; + + private final BaseFilterOperator _filterOperator; + private FilterBlockDocIdSet _filterBlockDocIdSet; + private final int _numDocs; + + public BlockDrivenAndFilterOperator(BaseFilterOperator filterOperator, int numDocs) { + _filterOperator = filterOperator; + _numDocs = numDocs; + } + + @Override + public FilterBlock getNextBlock() { + + if (_filterBlockDocIdSet != null) { + List<FilterBlockDocIdSet> filterBlockDocIdSets = new ArrayList<>(2); + + filterBlockDocIdSets.add(_filterBlockDocIdSet); + filterBlockDocIdSets.add(_filterOperator.nextBlock().getBlockDocIdSet()); Review comment: The current (Existing) mechanism is to invoke the filter first completely. So the nextBlock() on root FilterOperator is called exactly once by DocIdSetOperator. It creates an iterator and then every call (per 10K records) from Transform to Project to DocIdSet operator iterates over the already processed filtered block docIDIterator. I think the processing done over the child operators in this new operator in the if block is same as being done currently in the AndFilterOperator::getNextBlock (which is called exactly once). But are you expecting getNextBlock() of BlockDrivenAndFilterOperator to be called multiple times ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
