Yes I am waiting long periods between invocation of next. I didn't know that I
am fetching too much data at once.
I am using HBase 0.20.3. This is my code:
scan.setTimeRange(fromDate.getTime(), toDate.getTime());
ResultScanner scanner = table.getScanner(scan);
while( (result = scanner.next()) != null) {
channelRow = getChannelDeliveryRow(Bytes.toString(result.getRow()));
channelRowList.add(channelRow);
}
This is some of the output from the log file:
2010-05-04 15:27:44,546 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache:
Block cache LRU eviction started. Attempting to free 62791520 bytes
2010-05-04 15:27:44,552 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache:
Block cache LRU eviction completed. Freed 62797944 bytes. Priority Sizes:
Single=279.4997MB (293076672), Multi=224.35243MB (235250576),Memory=0.0MB (0)
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Stack
Sent: 04 May 2010 03:55 PM
To: [email protected]
Subject: Re: Improving HBase scanner
Are you waiting too long between invocations of next? (i.e. > the
scanner lease period?) Or, perhaps you are fetching too much in the
one go. If you fetch 1000 at a time -- scanner caching -- and you
don't get the next batch within the scanner lease period, again you
will timeout.
St.Ack
On Tue, May 4, 2010 at 1:46 AM, Michelan Arendse <[email protected]> wrote:
> Hi
>
> I would like to know how configure HBase to improve the scanner fetching data
> from the table or another method of using scanner, as my database is very
> large and scanner times out.
>
> Kind Regards,
>
> Michelan Arendse
> Junior Developer | AD:DYNAMO // happy business ;-)
> Office 0861 Dynamo (0861 396266) | Fax +27 (0) 21 465 2587
>
> Advertise Online Instantly - www.addynamo.com<http://www.addynamo.com>
> <http://www.addynamo.com>
>
>
