Thanks for your reply David. After doing more testing I found that overlapping latitudes or longitudes was not the issue as you point out.
The values presented are extreme but they are correct, our solution should allow a user to define a box in a map, a box crossing the 180 meridian is also valid. I know those are very extreme (and rare) scenarios but technically still valid ones. I understand your workaround and seems feasible but unfortunately I don't see any example on http://wiki.apache.org/solr/SolrAdaptersForLuceneSpatial4on how to index a multivalued field. Is there any other documentation that I am missing? If not could you please shed some light on the syntax to index to boxes (rectangles) on a field? With regards to search, will an Intersects function behaviour will be OR, that is match any rectangle in my example on that multivalued field that intersects with the given area? Thanks, Javier On 22 January 2013 05:43, David Smiley (@MITRE.org) <dsmi...@mitre.org>wrote: > Javier, > > Your minX is slightly greater than maxX, which is interpreted as a line > that > wraps nearly the entire globe. Is that what you intended? > > If this is what you intended, then you got bitten by this unfixed bug: > https://issues.apache.org/jira/browse/LUCENE-4550 > As a work-around, you could split that horizontal line into two equal > pieces > and index them as separate values for the document. > > ~ David > > > Javier Molina wrote > > Hi, > > > > I have been struggling the past days trying to spot what was causing my > > solr 4.0 to go Out Of Memory when doing a dataimport full-import > > > > After troubleshooting I found that the problem is not related with data > > volume but instead with one particular record in my DB. > > > > The offending record has a location value of 147.33767500000002 > -42.88601 > > 147.337675 -42.88601 > > > > As you can see, the min and max latitude have the same value, that is the > > rectangle is indeed a line. > > > > The location field uses the default definition for field type > > location_rpt, > > extract from schema.xml shown below. > > > > > > <fieldType name="location_rpt" > > class="solr.SpatialRecursivePrefixTreeFieldType"^M > > geo="true" distErrPct="0.025" maxDistErr="0.000009" > > units="degrees" > > /> > > > > We have configured the delta settings for dataimport to take a particular > > id from our db records, and what it is interesting is that if instead of > > doing a full-import I issue a delta-import the operation succeed. Note > > that > > in both cases I am just importing the same particular record, in the case > > of full-import I specify start and rows parameters, as seen in the log. > > > > My understanding is that defining a rectangle as a line it is still a > > valid > > area (so to speak) so the operation should succeed, nevertheless if those > > values are not a valid there should be a validation in place in order to > > prevent the OutOfMemoryError. > > > > > > See log below for an example of the run leading to the Out Of Memory > > Error. > > > > Thanks in advance for your feedback. > > > > Regards, > > Javier > > > > > > 21/01/2013 12:00:38 PM org.apache.solr.core.SolrCore execute > > INFO: [coreDap] webapp=/solr path=/dataimport > > > params={optimize=false&clean=true&commit=true&start=1995&verbose=true&command=full-import&rows=1} > > status=0 QTime=10 > > 21/01/2013 12:00:38 PM org.apache.solr.handler.dataimport.DataImporter > > doFullImport > > INFO: Starting Full Import > > 21/01/2013 12:00:38 PM > > org.apache.solr.handler.dataimport.SimplePropertiesWriter > > readIndexerProperties > > INFO: Read dataimport.properties > > 21/01/2013 12:00:38 PM org.apache.solr.core.SolrCore execute > > INFO: [coreDap] webapp=/solr path=/dataimport params={command=status} > > status=0 QTime=1 > > 21/01/2013 12:00:38 PM org.apache.solr.core.SolrDeletionPolicy onInit > > INFO: SolrDeletionPolicy.onInit: commits:num=1 > > > > commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@ > /DMS_dev/r01/solr/coreDap/data/index > > lockFactory=org.apache.lucene.store.NativeFSLockFactory@616181be; > > maxCacheMB=48.0 > > > maxMergeSizeMB=4.0),segFN=segments_mq,generation=818,filenames=[_12y_Lucene40_0.tip, > > _12w_Lucene40_0.prx, _12w_nrm.cfe, _12z_Lucene40_0.tim, > > _12z_Lucene40_0.frq, _12z.si, _12y_Lucene40_0.prx, _12z_nrm.cfe, > > _12w_Lucene40_0.frq, _12w.fdx, segments_mq, _12y_nrm.cfs, _12w.fnm, > > _12w_Lucene40_0.tip, _12w.si, _12w_Lucene40_0.tim, _12w.fdt, > _12z_nrm.cfs, > > _12z.fnm, _12y.fdx, _12y.fnm, _12z.fdx, _12z_Lucene40_0.prx, _12y.fdt, > > _12z.fdt, _12y_nrm.cfe, _12y.si, _12z_Lucene40_0.tip, > _12y_Lucene40_0.frq, > > _12w_nrm.cfs, _12y_Lucene40_0.tim] > > 21/01/2013 12:00:38 PM org.apache.solr.core.SolrDeletionPolicy > > updateCommits > > INFO: newest commit = 818 > > 21/01/2013 12:00:38 PM org.apache.solr.search.SolrIndexSearcher > > <init> > > INFO: Opening Searcher@70f4d063 realtime > > 21/01/2013 12:00:38 PM > org.apache.solr.handler.dataimport.JdbcDataSource$1 > > call > > INFO: Creating a connection for entity dataCollection with URL: > > jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST= > > myserver.mydomain.com)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME= > > myserver.mydomain.com))) > > 21/01/2013 12:00:39 PM > org.apache.solr.handler.dataimport.JdbcDataSource$1 > > call > > INFO: Time taken for getConnection(): 40 > > 21/01/2013 12:00:39 PM > org.apache.solr.handler.dataimport.JdbcDataSource$1 > > call > > INFO: Creating a connection for entity topic with URL: jdbc:oracle:thin:@ > > (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=myserver.mydomain.com > > )(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=myserver.mydomain.com))) > > 21/01/2013 12:00:39 PM > org.apache.solr.handler.dataimport.JdbcDataSource$1 > > call > > INFO: Time taken for getConnection(): 47 > > 21/01/2013 12:00:39 PM > org.apache.solr.handler.dataimport.JdbcDataSource$1 > > call > > INFO: Creating a connection for entity usergroup with URL: > > jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST= > > myserver.mydomain.com)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME= > > myserver.mydomain.com))) > > 21/01/2013 12:00:39 PM > org.apache.solr.handler.dataimport.JdbcDataSource$1 > > call > > INFO: Time taken for getConnection(): 34 > > 21/01/2013 12:00:39 PM > org.apache.solr.handler.dataimport.JdbcDataSource$1 > > call > > INFO: Creating a connection for entity usergroupDNs with URL: > > jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST= > > myserver.mydomain.com)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME= > > myserver.mydomain.com))) > > 21/01/2013 12:00:39 PM > org.apache.solr.handler.dataimport.JdbcDataSource$1 > > call > > INFO: Time taken for getConnection(): 39 > > 21/01/2013 12:00:39 PM > org.apache.solr.handler.dataimport.JdbcDataSource$1 > > call > > INFO: Creating a connection for entity person with URL: > jdbc:oracle:thin:@ > > (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=myserver.mydomain.com > > )(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=myserver.mydomain.com))) > > 21/01/2013 12:00:39 PM > org.apache.solr.handler.dataimport.JdbcDataSource$1 > > call > > INFO: Time taken for getConnection(): 33 > > 21/01/2013 12:00:39 PM > > org.apache.solr.schema.AbstractSpatialPrefixTreeFieldType > > newSpatialStrategy > > INFO: > > > location_rpt{class=org.apache.solr.schema.SpatialRecursivePrefixTreeFieldType,analyzer=org.apache.solr.schema.FieldType$DefaultAnalyzer,args={distErrPct=0.025, > > geo=true, maxDistErr=0.000009, units=degrees}} strat: > > > RecursivePrefixTreeStrategy(prefixGridScanLevel:7,SPG:(GeohashPrefixTree(maxLevels:11,ctx:SpatialContext{geo=true, > > calculator=Haversine, > > worldBounds=Rect(minX=-180.0,maxX=180.0,minY=-90.0,maxY=90.0)}))) > > maxLevels: 11 > > 21/01/2013 12:00:40 PM org.apache.solr.core.SolrCore execute > > INFO: [coreDap] webapp=/solr path=/dataimport params={command=status} > > status=0 QTime=0 > > 21/01/2013 12:00:53 PM org.apache.solr.update.DirectUpdateHandler2 commit > > INFO: start > > > commit{flags=0,_version_=0,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false} > > 21/01/2013 12:00:54 PM org.apache.solr.core.SolrDeletionPolicy onCommit > > INFO: SolrDeletionPolicy.onCommit: commits:num=2 > > > > commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@ > /DMS_dev/r01/solr/coreDap/data/index > > lockFactory=org.apache.lucene.store.NativeFSLockFactory@616181be; > > maxCacheMB=48.0 > > > maxMergeSizeMB=4.0),segFN=segments_mq,generation=818,filenames=[_12y_Lucene40_0.tip, > > _12w_Lucene40_0.prx, _12w_nrm.cfe, _12z_Lucene40_0.tim, > > _12z_Lucene40_0.frq, _12z.si, _12y_Lucene40_0.prx, _12z_nrm.cfe, > > _12w_Lucene40_0.frq, _12w.fdx, segments_mq, _12y_nrm.cfs, _12w.fnm, > > _12w_Lucene40_0.tip, _12w.si, _12w_Lucene40_0.tim, _12w.fdt, > _12z_nrm.cfs, > > _12z.fnm, _12y.fdx, _12y.fnm, _12z.fdx, _12z_Lucene40_0.prx, _12y.fdt, > > _12z.fdt, _12y_nrm.cfe, _12y.si, _12z_Lucene40_0.tip, > _12y_Lucene40_0.frq, > > _12w_nrm.cfs, _12y_Lucene40_0.tim] > > > > commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@ > /DMS_dev/r01/solr/coreDap/data/index > > lockFactory=org.apache.lucene.store.NativeFSLockFactory@616181be; > > maxCacheMB=48.0 > > > maxMergeSizeMB=4.0),segFN=segments_mr,generation=819,filenames=[segments_mr] > > 21/01/2013 12:00:54 PM org.apache.solr.core.SolrDeletionPolicy > > updateCommits > > INFO: newest commit = 819 > > 21/01/2013 12:00:54 PM org.apache.solr.update.DirectUpdateHandler2 commit > > INFO: end_commit_flush > > > > 21/01/2013 12:05:11 PM > org.apache.solr.update.processor.LogUpdateProcessor > > finish > > INFO: [coreDap] webapp=/solr path=/dataimport > > > params={optimize=false&clean=true&commit=true&start=1995&verbose=true&command=full-import&rows=1} > > status=0 QTime=10 {deleteByQuery=*:* (-1424731709158457344)} 0 10 > > 21/01/2013 12:05:11 PM org.apache.solr.common.SolrException log > > SEVERE: Full Import failed:java.lang.RuntimeException: > > java.lang.RuntimeException: > > org.apache.solr.handler.dataimport.DataImportHandlerException: > > java.lang.OutOfMemoryError: Java heap space > > at > > > org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:273) > > at > > > org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:382) > > at > > > org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:448) > > at > > > org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:429) > > Caused by: java.lang.RuntimeException: > > org.apache.solr.handler.dataimport.DataImportHandlerException: > > java.lang.OutOfMemoryError: Java heap space > > at > > > org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:413) > > at > > > org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:326) > > at > > > org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:234) > > ... 3 more > > Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: > > java.lang.OutOfMemoryError: Java heap space > > at > > > org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:542) > > at > > > org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:411) > > ... 5 more > > Caused by: java.lang.OutOfMemoryError: Java heap space > > at > > > com.spatial4j.core.context.SpatialContext.makeRectangle(SpatialContext.java:172) > > at > > com.spatial4j.core.io.GeohashUtils.decodeBoundary(GeohashUtils.java:150) > > at > > > org.apache.lucene.spatial.prefix.tree.GeohashPrefixTree$GhCell.getShape(GeohashPrefixTree.java:142) > > at > > org.apache.lucene.spatial.prefix.tree.Node.getSubCells(Node.java:155) > > at > > > org.apache.lucene.spatial.prefix.tree.SpatialPrefixTree.recursiveGetNodes(SpatialPrefixTree.java:157) > > at > > > org.apache.lucene.spatial.prefix.tree.SpatialPrefixTree.recursiveGetNodes(SpatialPrefixTree.java:175) > > at > > > org.apache.lucene.spatial.prefix.tree.SpatialPrefixTree.recursiveGetNodes(SpatialPrefixTree.java:175) > > at > > > org.apache.lucene.spatial.prefix.tree.SpatialPrefixTree.recursiveGetNodes(SpatialPrefixTree.java:175) > > at > > > org.apache.lucene.spatial.prefix.tree.SpatialPrefixTree.recursiveGetNodes(SpatialPrefixTree.java:175) > > at > > > org.apache.lucene.spatial.prefix.tree.SpatialPrefixTree.recursiveGetNodes(SpatialPrefixTree.java:175) > > at > > > org.apache.lucene.spatial.prefix.tree.SpatialPrefixTree.recursiveGetNodes(SpatialPrefixTree.java:175) > > at > > > org.apache.lucene.spatial.prefix.tree.SpatialPrefixTree.recursiveGetNodes(SpatialPrefixTree.java:175) > > at > > > org.apache.lucene.spatial.prefix.tree.SpatialPrefixTree.recursiveGetNodes(SpatialPrefixTree.java:175) > > at > > > org.apache.lucene.spatial.prefix.tree.SpatialPrefixTree.recursiveGetNodes(SpatialPrefixTree.java:175) > > at > > > org.apache.lucene.spatial.prefix.tree.SpatialPrefixTree.getNodes(SpatialPrefixTree.java:142) > > at > > > org.apache.lucene.spatial.prefix.tree.GeohashPrefixTree.getNodes(GeohashPrefixTree.java:98) > > at > > > org.apache.lucene.spatial.prefix.PrefixTreeStrategy.createIndexableFields(PrefixTreeStrategy.java:126) > > at > > > org.apache.lucene.spatial.prefix.PrefixTreeStrategy.createIndexableFields(PrefixTreeStrategy.java:121) > > at > > > org.apache.solr.schema.AbstractSpatialFieldType.createFields(AbstractSpatialFieldType.java:110) > > at > > > org.apache.solr.schema.AbstractSpatialFieldType.createFields(AbstractSpatialFieldType.java:55) > > at > > org.apache.solr.update.DocumentBuilder.addField(DocumentBuilder.java:193) > > at > > > org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:270) > > at > > > org.apache.solr.update.AddUpdateCommand.getLuceneDocument(AddUpdateCommand.java:73) > > at > > > org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:208) > > at > > > org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:61) > > at > > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51) > > at > > > org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:432) > > at > > > org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:557) > > at > > > org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:325) > > at > > > org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100) > > at > > org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:70) > > at > > > org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:233) > > > > 21/01/2013 12:05:11 PM org.apache.solr.update.DirectUpdateHandler2 > > rollback > > INFO: start rollback{flags=0,_version_=0} > > 21/01/2013 12:05:11 PM org.apache.solr.update.DefaultSolrCoreState > > newIndexWriter > > INFO: Creating new IndexWriter... > > 21/01/2013 12:05:11 PM org.apache.solr.update.DefaultSolrCoreState > > newIndexWriter > > INFO: Waiting until IndexWriter is unused... core=coreDap > > 21/01/2013 12:05:11 PM org.apache.solr.update.DefaultSolrCoreState > > newIndexWriter > > INFO: Rollback old IndexWriter... core=coreDap > > 21/01/2013 12:05:11 PM org.apache.solr.core.CachingDirectoryFactory get > > INFO: return new directory for /DMS_dev/r01/solr/coreDap/data/index > > forceNew:true > > 21/01/2013 12:05:11 PM org.apache.solr.core.SolrDeletionPolicy onInit > > INFO: SolrDeletionPolicy.onInit: commits:num=1 > > > > commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@ > /DMS_dev/r01/solr/coreDap/data/index > > lockFactory=org.apache.lucene.store.NativeFSLockFactory@7ab9fcb0; > > maxCacheMB=48.0 > > > maxMergeSizeMB=4.0),segFN=segments_mr,generation=819,filenames=[segments_mr] > > 21/01/2013 12:05:11 PM org.apache.solr.core.SolrDeletionPolicy > > updateCommits > > INFO: newest commit = 819 > > 21/01/2013 12:05:11 PM org.apache.solr.update.DefaultSolrCoreState > > newIndexWriter > > INFO: New IndexWriter is ready to be used. > > 21/01/2013 12:05:11 PM org.apache.solr.update.DirectUpdateHandler2 > > rollback > > INFO: end_rollback > > > > > > ----- > Author: > http://www.packtpub.com/apache-solr-3-enterprise-search-server/book > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Spatial-Dataimport-full-import-results-in-OutOfMemory-for-a-rectangle-defining-a-line-tp4034928p4035163.html > Sent from the Solr - User mailing list archive at Nabble.com. > -- Javier Molina MACS IT Consultant/Java Developer (M) 0449 640 386 (e) javier.mol...@acsmail.net.au
