Question about Map indexes
Hi, We have observed that creating an index on a Map field causes the creation of an index entry for every entry created in the region containing the Map, no matter if the Map field contained the key used in the index. Nevertheless, we would expect that only entries whose Map field contain the key used in the index would have the corresponding index entry. With this behavior, the memory consumed by the index could be much higher than needed depending on the percentage of entries whose Map field contained the key in the index. --- Example: We have a region with entries whose key type is a String and the value type is an object with a field called "field1" of Map type. We expect to run queries on the region like the following: SELECT * from /example-region1 p WHERE p.field1['mapkey1']=$1" We create a Map index to speed up the above queries: gfsh> create index --name=myIndex --expression="r.field1['mapkey1']" --region="/example-region1 r" We do the following puts: - Put entry with key="key1" and with value= - Put entry with key="key2" and with value= The observation is that Geode creates two index entries for each entry. For the first entry, the internal indexKey is "key1" and for the second one, the internal indexKey is null. These are the stats shown by gfsh after doing the above puts: gfsh>list indexes --with-stats=yes Member Name |Member ID| Region Path| Name | Type | Indexed Expression |From Clause | Valid Index | Uses | Updates | Update Time | Keys | Values --- | --- | | | - | - | -- | --- | | --- | --- | | -- server1 | 192.168.0.26(server1:1109606):41000 | /example-region1 | mapIndex | RANGE | r.field1['mapkey1'] | /example-region1 r | true| 1 | 1 | 0 | 1| 1 server2 | 192.168.0.26(server2:1109695):41001 | /example-region1 | mapIndex | RANGE | r.field1['mapkey1'] | /example-region1 r | true| 1 | 1 | 0 | 1| 1 --- Is there any reason why Geode would create an index entry for the second entry given that the Map field does not contain the key in the Map index? I have created a draft pull request changing the behavior of Geode to not create the index entry when the Map field does not contain the key used in the index. Only two Unit test cases had to be adjusted. Please see: https://github.com/apache/geode/pull/6028 With this change and the same scenario as the one in the example, only one index entry is created. The stats shown by gfsh after the change are the following: gfsh>list indexes --with-stats=yes Member Name |Member ID| Region Path| Name | Type | Indexed Expression |From Clause | Valid Index | Uses | Updates | Update Time | Keys | Values --- | --- | | | - | - | -- | --- | | --- | --- | | -- server1 | 192.168.0.26(server1:1102192):41000 | /example-region1 | mapIndex | RANGE | r.field1['mapkey1'] | /example-region1 r | true| 2 | 1 | 0 | 0| 0 server2 | 192.168.0.26(server2:1102279):41001 | /example-region1 | mapIndex | RANGE | r.field1['mapkey1'] | /example-region1 r | true| 2 | 1 | 0 | 1| 1 Could someone tell if the current behavior is not correct or if I am missing something and with the change I am proposing something else will stop working? Thanks in advance, /Alberto G.
Need Concourse access, please
Hello, may I please have access to the Geode Concourse? My GitHub username is pdxcodemonkey Thanks, Blake
Canceled: apache/geode-examples#513 (support/1.12 - 10fa613)
Build Update for apache/geode-examples - Build: #513 Status: Canceled Duration: ? Commit: 10fa613 (support/1.12) Author: Owen Nichols Message: temporarily point to staging repo for CI purposes View the changeset: https://github.com/apache/geode-examples/compare/745588975fcd...10fa613f076b View the full build log and details: https://travis-ci.com/github/apache/geode-examples/builds/216847192?utm_medium=notification&utm_source=email Restart your build: https://travis-ci.com/github/apache/geode-examples/builds/216847192?utm_medium=notification&utm_source=email -- You can unsubscribe from build emails from the apache/geode-examples repository going to https://travis-ci.com/account/preferences/unsubscribe?repository=16807635&utm_medium=notification&utm_source=email. Or unsubscribe from *all* email updating your settings at https://travis-ci.com/account/preferences/unsubscribe?utm_medium=notification&utm_source=email. Or configure specific recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications.
Re: Question about Map indexes
Hi Alberto, I haven't checked the PR yet, just read through the email. The first thought that comes to mind is when someone does a != query. The index still has to supply the correct answer to the query (all entries with null or undefined values possibly) I'll try to think of other cases where it might matter. There may be other ways to execute the query but it would probably take a bit of reworking.. (ill check your pr to see if this is already addressed. Sorry if it is!) -Jason On 2/11/21, 8:28 AM, "Alberto Gomez" wrote: Hi, We have observed that creating an index on a Map field causes the creation of an index entry for every entry created in the region containing the Map, no matter if the Map field contained the key used in the index. Nevertheless, we would expect that only entries whose Map field contain the key used in the index would have the corresponding index entry. With this behavior, the memory consumed by the index could be much higher than needed depending on the percentage of entries whose Map field contained the key in the index. --- Example: We have a region with entries whose key type is a String and the value type is an object with a field called "field1" of Map type. We expect to run queries on the region like the following: SELECT * from /example-region1 p WHERE p.field1['mapkey1']=$1" We create a Map index to speed up the above queries: gfsh> create index --name=myIndex --expression="r.field1['mapkey1']" --region="/example-region1 r" We do the following puts: - Put entry with key="key1" and with value= - Put entry with key="key2" and with value= The observation is that Geode creates two index entries for each entry. For the first entry, the internal indexKey is "key1" and for the second one, the internal indexKey is null. These are the stats shown by gfsh after doing the above puts: gfsh>list indexes --with-stats=yes Member Name |Member ID| Region Path| Name | Type | Indexed Expression |From Clause | Valid Index | Uses | Updates | Update Time | Keys | Values --- | --- | | | - | - | -- | --- | | --- | --- | | -- server1 | 192.168.0.26(server1:1109606):41000 | /example-region1 | mapIndex | RANGE | r.field1['mapkey1'] | /example-region1 r | true| 1 | 1 | 0 | 1| 1 server2 | 192.168.0.26(server2:1109695):41001 | /example-region1 | mapIndex | RANGE | r.field1['mapkey1'] | /example-region1 r | true| 1 | 1 | 0 | 1| 1 --- Is there any reason why Geode would create an index entry for the second entry given that the Map field does not contain the key in the Map index? I have created a draft pull request changing the behavior of Geode to not create the index entry when the Map field does not contain the key used in the index. Only two Unit test cases had to be adjusted. Please see: https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fgeode%2Fpull%2F6028&data=04%7C01%7Cjhuynh%40vmware.com%7C0957cc0ef91b4b23116408d8ceaa0a8d%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637486577011301177%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=2WDUj6NPEnfX3AXH72MTZYx%2FbXMPJQlVZeKq7KsJDTw%3D&reserved=0 With this change and the same scenario as the one in the example, only one index entry is created. The stats shown by gfsh after the change are the following: gfsh>list indexes --with-stats=yes Member Name |Member ID| Region Path| Name | Type | Indexed Expression |From Clause | Valid Index | Uses | Updates | Update Time | Keys | Values --- | --- | | | - | - | -- | --- | | --- | --- | | -- server1 | 192.168.0.26(server1:1102192):41000 | /example-region1 | mapIndex | RANGE | r.field1['mapkey1'] | /example-region1 r | true| 2 | 1 | 0 | 0| 0 server2 | 192.168.0.26(server2:1102279):41001 | /example-region1 | mapIndex | RANGE | r.field1['mapkey1'] | /example-region1 r | true| 2 | 1 | 0 | 1| 1 Could someone tell if the current behavior is not correct or if I am missing something and with the change I am proposing something else will stop working? Thanks in advance, /Alberto G.
Errored: apache/geode-examples#515 (rel/v1.12.1.RC1 - f097cb0)
Build Update for apache/geode-examples - Build: #515 Status: Errored Duration: 1 min and 9 secs Commit: f097cb0 (rel/v1.12.1.RC1) Author: Owen Nichols Message: Bumping version to 1.12.1 View the changeset: https://github.com/apache/geode-examples/compare/rel/v1.12.1.RC1 View the full build log and details: https://travis-ci.com/github/apache/geode-examples/builds/216847221?utm_medium=notification&utm_source=email -- You can unsubscribe from build emails from the apache/geode-examples repository going to https://travis-ci.com/account/preferences/unsubscribe?repository=16807635&utm_medium=notification&utm_source=email. Or unsubscribe from *all* email updating your settings at https://travis-ci.com/account/preferences/unsubscribe?utm_medium=notification&utm_source=email. Or configure specific recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications.
Canceled: apache/geode-examples#516 (support/1.12 - 2737dc2)
Build Update for apache/geode-examples - Build: #516 Status: Canceled Duration: ? Commit: 2737dc2 (support/1.12) Author: Owen Nichols Message: temporarily point to staging repo for CI purposes View the changeset: https://github.com/apache/geode-examples/compare/d235195a4dd4...2737dc2ff7f4 View the full build log and details: https://travis-ci.com/github/apache/geode-examples/builds/216862069?utm_medium=notification&utm_source=email Restart your build: https://travis-ci.com/github/apache/geode-examples/builds/216862069?utm_medium=notification&utm_source=email -- You can unsubscribe from build emails from the apache/geode-examples repository going to https://travis-ci.com/account/preferences/unsubscribe?repository=16807635&utm_medium=notification&utm_source=email. Or unsubscribe from *all* email updating your settings at https://travis-ci.com/account/preferences/unsubscribe?utm_medium=notification&utm_source=email. Or configure specific recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications.
Errored: apache/geode-examples#518 (rel/v1.12.1.RC2 - b88adb4)
Build Update for apache/geode-examples - Build: #518 Status: Errored Duration: 1 min and 8 secs Commit: b88adb4 (rel/v1.12.1.RC2) Author: Owen Nichols Message: Revert "temporarily point to staging repo for CI purposes" View the changeset: https://github.com/apache/geode-examples/compare/rel/v1.12.1.RC2 View the full build log and details: https://travis-ci.com/github/apache/geode-examples/builds/216862087?utm_medium=notification&utm_source=email -- You can unsubscribe from build emails from the apache/geode-examples repository going to https://travis-ci.com/account/preferences/unsubscribe?repository=16807635&utm_medium=notification&utm_source=email. Or unsubscribe from *all* email updating your settings at https://travis-ci.com/account/preferences/unsubscribe?utm_medium=notification&utm_source=email. Or configure specific recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications.
Passed: apache/geode-examples#514 (support/1.12 - d235195)
Build Update for apache/geode-examples - Build: #514 Status: Passed Duration: 25 mins and 57 secs Commit: d235195 (support/1.12) Author: Owen Nichols Message: Bumping version to 1.12.1 View the changeset: https://github.com/apache/geode-examples/compare/10fa613f076b...d235195a4dd4 View the full build log and details: https://travis-ci.com/github/apache/geode-examples/builds/216847219?utm_medium=notification&utm_source=email -- You can unsubscribe from build emails from the apache/geode-examples repository going to https://travis-ci.com/account/preferences/unsubscribe?repository=16807635&utm_medium=notification&utm_source=email. Or unsubscribe from *all* email updating your settings at https://travis-ci.com/account/preferences/unsubscribe?utm_medium=notification&utm_source=email. Or configure specific recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications.
Passed: apache/geode-examples#517 (support/1.12 - c4cf1c4)
Build Update for apache/geode-examples - Build: #517 Status: Passed Duration: 25 mins and 18 secs Commit: c4cf1c4 (support/1.12) Author: Owen Nichols Message: Bumping version to 1.12.1 View the changeset: https://github.com/apache/geode-examples/compare/2737dc2ff7f4...c4cf1c4ef9fc View the full build log and details: https://travis-ci.com/github/apache/geode-examples/builds/216862080?utm_medium=notification&utm_source=email -- You can unsubscribe from build emails from the apache/geode-examples repository going to https://travis-ci.com/account/preferences/unsubscribe?repository=16807635&utm_medium=notification&utm_source=email. Or unsubscribe from *all* email updating your settings at https://travis-ci.com/account/preferences/unsubscribe?utm_medium=notification&utm_source=email. Or configure specific recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications.
Canceled: apache/geode-examples#519 (support/1.12 - 552a4b6)
Build Update for apache/geode-examples - Build: #519 Status: Canceled Duration: ? Commit: 552a4b6 (support/1.12) Author: Owen Nichols Message: temporarily point to staging repo for CI purposes View the changeset: https://github.com/apache/geode-examples/compare/c4cf1c4ef9fc...552a4b6d5317 View the full build log and details: https://travis-ci.com/github/apache/geode-examples/builds/216865777?utm_medium=notification&utm_source=email Restart your build: https://travis-ci.com/github/apache/geode-examples/builds/216865777?utm_medium=notification&utm_source=email -- You can unsubscribe from build emails from the apache/geode-examples repository going to https://travis-ci.com/account/preferences/unsubscribe?repository=16807635&utm_medium=notification&utm_source=email. Or unsubscribe from *all* email updating your settings at https://travis-ci.com/account/preferences/unsubscribe?utm_medium=notification&utm_source=email. Or configure specific recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications.
Errored: apache/geode-examples#521 (rel/v1.12.1.RC3 - f27cc8d)
Build Update for apache/geode-examples - Build: #521 Status: Errored Duration: 1 min and 11 secs Commit: f27cc8d (rel/v1.12.1.RC3) Author: Owen Nichols Message: Revert "temporarily point to staging repo for CI purposes" View the changeset: https://github.com/apache/geode-examples/compare/rel/v1.12.1.RC3 View the full build log and details: https://travis-ci.com/github/apache/geode-examples/builds/216865791?utm_medium=notification&utm_source=email -- You can unsubscribe from build emails from the apache/geode-examples repository going to https://travis-ci.com/account/preferences/unsubscribe?repository=16807635&utm_medium=notification&utm_source=email. Or unsubscribe from *all* email updating your settings at https://travis-ci.com/account/preferences/unsubscribe?utm_medium=notification&utm_source=email. Or configure specific recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications.
[VOTE] Apache Geode 1.12.1.RC3
Hello Geode Dev Community, One year ago this month, Geode adopted a plan to maintain N-2 support branches [1] and support/1.12 was created. Over 140 fixes have now been backported to support/1.12 since 1.12.0 was released, so I’d like to propose a maintenance release! My apologies, RC1 and RC2 were DOA so I did not send an email for them. Please ignore any RC1/RC2 tags and focus your review on RC3. This is a release candidate for Apache Geode version 1.12.1.RC3. Thanks to all the community members for their contributions to this release! Please do a review and give your feedback, including the checks you performed. Voting deadline: 3PM PST Tue, February 16 2021. Please note that we are voting upon the source tag: rel/v1.12.1.RC3 Release notes: https://cwiki.apache.org/confluence/display/GEODE/Release+Notes#ReleaseNotes-1.12.1 Source and binary distributions: https://dist.apache.org/repos/dist/dev/geode/1.12.1.RC3/ Maven staging repo: https://repository.apache.org/content/repositories/orgapachegeode-1074 GitHub: https://github.com/apache/geode/tree/rel/v1.12.1.RC3 https://github.com/apache/geode-examples/tree/rel/v1.12.1.RC3 https://github.com/apache/geode-native/tree/rel/v1.12.1.RC3 https://github.com/apache/geode-benchmarks/tree/rel/v1.12.1.RC3 Pipelines: https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-support-1-12-main https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-support-1-12-rc Geode's KEYS file containing PGP keys we use to sign the release: https://github.com/apache/geode/blob/develop/KEYS Command to run geode-examples: ./gradlew -PgeodeReleaseUrl=https://dist.apache.org/repos/dist/dev/geode/1.12.1.RC3 -PgeodeRepositoryUrl=https://repository.apache.org/content/repositories/orgapachegeode-1074 build runAll Regards Owen Nichols [1] https://www.cwiki.us/display/GEODE/Shipping+patch+releases
Passed: apache/geode-native#3024 (rel/v1.12.1.RC1 - 21fcdb8)
Build Update for apache/geode-native - Build: #3024 Status: Passed Duration: 1 hr, 15 mins, and 59 secs Commit: 21fcdb8 (rel/v1.12.1.RC1) Author: Owen Nichols Message: Bumping copyright year to 2021 View the changeset: https://github.com/apache/geode-native/compare/rel/v1.12.1.RC1 View the full build log and details: https://travis-ci.com/github/apache/geode-native/builds/216847237?utm_medium=notification&utm_source=email -- You can unsubscribe from build emails from the apache/geode-native repository going to https://travis-ci.com/account/preferences/unsubscribe?repository=16807653&utm_medium=notification&utm_source=email. Or unsubscribe from *all* email updating your settings at https://travis-ci.com/account/preferences/unsubscribe?utm_medium=notification&utm_source=email. Or configure specific recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications.
Passed: apache/geode-native#3026 (rel/v1.12.1.RC2 - 21fcdb8)
Build Update for apache/geode-native - Build: #3026 Status: Passed Duration: 1 hr, 17 mins, and 5 secs Commit: 21fcdb8 (rel/v1.12.1.RC2) Author: Owen Nichols Message: Bumping copyright year to 2021 View the changeset: https://github.com/apache/geode-native/compare/rel/v1.12.1.RC2 View the full build log and details: https://travis-ci.com/github/apache/geode-native/builds/216862093?utm_medium=notification&utm_source=email -- You can unsubscribe from build emails from the apache/geode-native repository going to https://travis-ci.com/account/preferences/unsubscribe?repository=16807653&utm_medium=notification&utm_source=email. Or unsubscribe from *all* email updating your settings at https://travis-ci.com/account/preferences/unsubscribe?utm_medium=notification&utm_source=email. Or configure specific recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications.
Passed: apache/geode-native#3028 (rel/v1.12.1.RC3 - ba8fd9b)
Build Update for apache/geode-native - Build: #3028 Status: Passed Duration: 1 hr, 16 mins, and 33 secs Commit: ba8fd9b (rel/v1.12.1.RC3) Author: Owen Nichols Message: Bumping copyright year to 2021 View the changeset: https://github.com/apache/geode-native/compare/rel/v1.12.1.RC3 View the full build log and details: https://travis-ci.com/github/apache/geode-native/builds/216865809?utm_medium=notification&utm_source=email -- You can unsubscribe from build emails from the apache/geode-native repository going to https://travis-ci.com/account/preferences/unsubscribe?repository=16807653&utm_medium=notification&utm_source=email. Or unsubscribe from *all* email updating your settings at https://travis-ci.com/account/preferences/unsubscribe?utm_medium=notification&utm_source=email. Or configure specific recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications.
Errored: apache/geode-examples#523 (develop - d8b16a6)
Build Update for apache/geode-examples - Build: #523 Status: Errored Duration: 1 min and 9 secs Commit: d8b16a6 (develop) Author: Owen Nichols Message: pair develop examples with 1.15.0 now that support/1.14 has been created View the changeset: https://github.com/apache/geode-examples/compare/15e25f7c6a0e...d8b16a64f4f5 View the full build log and details: https://travis-ci.com/github/apache/geode-examples/builds/216875208?utm_medium=notification&utm_source=email -- You can unsubscribe from build emails from the apache/geode-examples repository going to https://travis-ci.com/account/preferences/unsubscribe?repository=16807635&utm_medium=notification&utm_source=email. Or unsubscribe from *all* email updating your settings at https://travis-ci.com/account/preferences/unsubscribe?utm_medium=notification&utm_source=email. Or configure specific recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications.
Passed: apache/geode-examples#522 (support/1.14 - 12a3c52)
Build Update for apache/geode-examples - Build: #522 Status: Passed Duration: 23 mins and 38 secs Commit: 12a3c52 (support/1.14) Author: Owen Nichols Message: Bumping copyright year to 2021 View the changeset: https://github.com/apache/geode-examples/commit/12a3c52b75d7 View the full build log and details: https://travis-ci.com/github/apache/geode-examples/builds/216874013?utm_medium=notification&utm_source=email -- You can unsubscribe from build emails from the apache/geode-examples repository going to https://travis-ci.com/account/preferences/unsubscribe?repository=16807635&utm_medium=notification&utm_source=email. Or unsubscribe from *all* email updating your settings at https://travis-ci.com/account/preferences/unsubscribe?utm_medium=notification&utm_source=email. Or configure specific recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications.
