On Tue, May 18, 2010 at 2:15 AM, Renaud Delbru <[email protected]> wrote:
> Hi,
>
> after some tuning, like increasing the hfile block size to 128KB, I have
> noticed that the storefileIndexSize is now half of what it was before
> (~250). Do storefileIndexSize is the size of the in-memory hfile block index
> ?
Yes.
So, yes, doubling the block size should halve the index size.
How come your index is so big? Do you have big keys? Lots of data?
Lots of storefiles?
Looking in HRegionServer I see that its calculated so:
storefileIndexSizeMB = (int)(store.getStorefilesIndexSize()/1024/1024);
In the Store, we do this:
/**
* @return The size of the store file indexes, in bytes.
*/
long getStorefilesIndexSize() {
long size = 0;
for (StoreFile s: storefiles.values()) {
Reader r = s.getReader();
if (r == null) {
LOG.warn("StoreFile " + s + " has a null Reader");
continue;
}
size += r.indexSize();
}
return size;
}
The indexSize is out of the HFile metadata.
St.Ack
>
> Thanks
> --
> Renaud Delbru
>
> On 17/05/10 15:27, Renaud Delbru wrote:
>>
>> Hi,
>>
>> I would like to understand the meaning of the storefileIndexSize metric,
>> could someone point me to a definition or explain me what does that mean ?
>>
>> Also, we are performing a large table import (90M rows, size of the row
>> varying between hundreds of kb to 8 MB), and we are encountering memory
>> problem (OOME). My observation is that it always happens after a while, when
>> the storefileIndexSize starts to be large (> 500). Is there a way to reduce
>> it ?
>>
>> Thanks,
>
>
