Re: Strange Performance Issue with Large number Region Creation

[Udo Kohlmeyer]

+1

Maybe we can create a JIRA for this.

On 11/22/16 11:56, Dan Smith wrote:

Seems like fixing that StatisicsMonitor to use a ConcurrentHashSet is a
good fix, there's no reason why it should be a list that I can see.

As other folks mentioned, there is significant overhead involved in
creating that many regions in terms of memory, messaging, and disk
metadata. Especially considering you are creating partitioned regions,
because each bucket has some overhead and you are creating some 170000
buckets if you are using the default number of buckets.

-Dan



On Mon, Nov 21, 2016 at 12:45 AM, Avinash Dongre <adon...@apache.org> wrote:

Hi All,

Thanks @Mike. Unfortunately I have situation where-in I need to create
those many number of regions.

I need to understand why in StatisticsMonitor and StatMonitorHandler
List is used to store monitors, listeners and statisticIds  , is there any
particular reason for this.

When I replace the List to com.gemstone.gemfire.internal.concurrent.
ConcurrentHashSet
I see significant improvement in creating large number of region creation.
( from ~7hrs to ~26 minutes)


Thanks
Avinash


On Thu, Nov 17, 2016 at 3:27 AM, Michael Stolz <mst...@pivotal.io> wrote:

I can't think of any reason why any use case could need 1500 Regions.

Regions are heavyweight constructs more similar in nature to Unix mounts
than Unix directories.

We usually use simple naming conventions for keys to simulate directory
structures.

So I would recommend that you create 1 Region, and store 1500 named hash
maps into it.

Make sense?

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771
On Nov 16, 2016 11:15 AM, "Avinash Dongre" <adon...@apache.org> wrote:

Hi,

I am seeing strange performance issue when I want to create large

number

of

regions.
Test I am doing with 1500 Regions.

~ 2 minutes for creating first 1130 regions
~7 minutes for creating remaining 370 regions.

Following is example code :

Cache geodeCache = this.createCache();
final String REGION_NAME = "Region_Name";
for ( int i = 0; i < 1500; i++) {
   RegionFactory<Object, Object> rf =
geodeCache.createRegionFactory(RegionShortcut.PARTITION_
PERSISTENT_OVERFLOW);
   rf.setDiskSynchronous(true);
   rf.setEvictionAttributes(EvictionAttributes.

createLIFOEntryAttributes(1,

EvictionAction.OVERFLOW_TO_DISK));
   Region<Object, Object> region = rf.create(REGION_NAME + "_" + i);
}
geodeCache.close();


If I remove following code from createVMRegion ( GemFireCacheImpl.java

),

then I could create Regions in 2 minutes.

if (!rgn.isInternalRegion()) {
   system.handleResourceEvent(ResourceEvent.REGION_CREATE, rgn);
}

Any help or pointers why this is happening ?


Thanks

Avinash