Re: Data ingestion with predefined buckets

[Udo Kohlmeyer]

Hi there Steve,

Firstly, you are correct, the pattern you are describing is not recommended and 
possibly not even correctly supported. I've seen many implementations of Geode 
systems and none of them ever needed to do what you are intending to do. Seems 
like you are will to go through A LOT of effort for a benefit I don't 
immediately realize.

Also, I'm confused on what part of the "hashing" you are trying to avoid.
You will ALWAYS  have the hashing overhead. At the very least the key will have 
to be hashed for put() and later on get().
As for the "file-per-bucket" request, there will always be some form of bucket 
resolution that needs to happen. Be it a custom PartitionResolution or default 
partition bucket resolver.

In the code that Dan provided, you now have to manage the bucket number 
explicitly in the client. When you insert data, you have to provide the correct 
bucket number and if you retrieve the data, you have to provide the correct 
bucket number, otherwise you will get "null" back. So this means your client 
has to manage the bucket numbers. Because every subsequent put/get that does 
not provide the bucket number, will possibly result in some failure. In short, 
EVERY key operation (put/get) will require a bucketNumber to function 
correctly, as the PartitionResolver is used.

Maybe we can aid you better in suitable solution by understanding WHAT you are 
trying to achieve and WHAT you are trying to avoid.

So in short, you will NOT avoid hashing, as a Region will always hash the key, 
regardless of how you load your data. Think of a Region as a big distributed 
HashMap. Hashing is in its DNA and inner workings. The only thing step you'd 
avoid is the bucket allocation calculation, which tbh, is lightweight

`bucketNumber = (hashcode % totalNumberBuckets) + 1`

--Udo

On 4/10/20, 3:52 PM, "steve mathew"  wrote:

Thanks Dan for your quick response.

Though, This may not be a recommended pattern, Here, I am targeting a
bucket specific putAll and want to exclude hashing as it turn out as an
overhead in my scenario.
Is this achievable...? How should I define a PartionResolver that works
generically and returns a respective bucket for specific file.
What will get impacted if I opt this route (Fix partitioning per file), can
think of horizontal scalability as buckets made fix .. thoughts?


-Steave M.


On Sat, Apr 11, 2020, 1:54 AM Dan Smith  wrote:

> Hi Steve,
>
> The bucket that data goes into is generally determined by the key. So for
> example if your data in File-0 is all for customer X, you can include
> Customer X in your region key and implement a PartitionResolver that
> extracts the customer from your region key and returns it. Geode will then
> group all of the data for Customer X into a single bucket.
>
> You generally shouldn't have to target a specific bucket number (eg bucket
> 0). But technically you can just by returning an integer from your
> PartitionResolver. If you return the integer 0, your data will go into
> bucket 0. Usually it's just better to return your partition key (eg
> "Customer X") and let geode hash that to some bucket number.
>
> -Dan
>
> On Fri, Apr 10, 2020 at 11:04 AM steve mathew 
> wrote:
>
> > Hello Geode devs and users,
> >
> > I have a set of files populated with data, fairly distributed, I want to
> > put each file's data in a specific bucket,
> > like PutAll File-0 data into Geode bucket B0
> >   PutAll File-1 data into Geode bucket B1
> >
> >   and so on...
> >
> > How can i achieve this using geode client...?
> >
> > Can i achieve this using PartitonResolver or some other means...?
> >
> > Thanks in advance
> >
> > -Steve M.
> >
>

Help with Concourse-CI redis integration test failing

[Alberto Gomez]

Hi,

I am seeing the following integration test case fail in the Concourse-CI when 
executed on my pull request (https://github.com/apache/geode/pull/4928):

org.apache.geode.redis.SetsIntegrationTest > 
testSRem_should_notRemoveMembersOfSetNotSpecified FAILED

Interestingly, if I run the test locally it passes. Also, the line of code 
where the failure occurs does not point to a valid line in the code I see:

at 
org.apache.geode.redis.SetsIntegrationTest.testSRem_should_notRemoveMembersOfSetNotSpecified(SetsIntegrationTest.java:784

Any idea on what could be happening?

Thanks in advance,

-Alberto G.

Re: Help with Concourse-CI redis integration test failing

[Owen Nichols]

Hi Alberto, it looks like Integration tests have been broken since PR#4937 was 
merged at the end of the day Friday.  However this PR had passed all its PR 
checks (as of 5 hours before it was merged) so I suspect this was just an 
unexpected interaction with the PR#4912 that came in just 3 hours before it).

In general there is an expectation in the Geode community that when a commit 
breaks the pipeline, it should be reverted first, then fixed.  

Alberto, since I don’t think you’re a committer, how about you initiate the 
revert and I’ll approve it?  GitHub makes this pretty easy: go to 
https://github.com/apache/geode/pull/4937 and click Revert near the 
bottom-right of the page then click Create pull request.

> On Apr 11, 2020, at 10:01 AM, Alberto Gomez  wrote:
> 
> Hi,
> 
> I am seeing the following integration test case fail in the Concourse-CI when 
> executed on my pull request (https://github.com/apache/geode/pull/4928):
> 
> org.apache.geode.redis.SetsIntegrationTest > 
> testSRem_should_notRemoveMembersOfSetNotSpecified FAILED
> 
> Interestingly, if I run the test locally it passes. Also, the line of code 
> where the failure occurs does not point to a valid line in the code I see:
> 
> at 
> org.apache.geode.redis.SetsIntegrationTest.testSRem_should_notRemoveMembersOfSetNotSpecified(SetsIntegrationTest.java:784
> 
> Any idea on what could be happening?
> 
> Thanks in advance,
> 
> -Alberto G.

Re: Help with Concourse-CI redis integration test failing

[Alberto Gomez]

Thanks Owen.

It seems you already initiated the revert.

Best regards,

-Alberto

From: Owen Nichols 
Sent: Saturday, April 11, 2020 8:04 PM
To: dev@geode.apache.org 
Subject: Re: Help with Concourse-CI redis integration test failing

Hi Alberto, it looks like Integration tests have been broken since PR#4937 was 
merged at the end of the day Friday.  However this PR had passed all its PR 
checks (as of 5 hours before it was merged) so I suspect this was just an 
unexpected interaction with the PR#4912 that came in just 3 hours before it).

In general there is an expectation in the Geode community that when a commit 
breaks the pipeline, it should be reverted first, then fixed.

Alberto, since I don’t think you’re a committer, how about you initiate the 
revert and I’ll approve it?  GitHub makes this pretty easy: go to 
https://github.com/apache/geode/pull/4937 and click Revert near the 
bottom-right of the page then click Create pull request.

> On Apr 11, 2020, at 10:01 AM, Alberto Gomez  wrote:
>
> Hi,
>
> I am seeing the following integration test case fail in the Concourse-CI when 
> executed on my pull request (https://github.com/apache/geode/pull/4928):
>
> org.apache.geode.redis.SetsIntegrationTest > 
> testSRem_should_notRemoveMembersOfSetNotSpecified FAILED
>
> Interestingly, if I run the test locally it passes. Also, the line of code 
> where the failure occurs does not point to a valid line in the code I see:
>
> at 
> org.apache.geode.redis.SetsIntegrationTest.testSRem_should_notRemoveMembersOfSetNotSpecified(SetsIntegrationTest.java:784
>
> Any idea on what could be happening?
>
> Thanks in advance,
>
> -Alberto G.

Re: Help with Concourse-CI redis integration test failing

[Owen Nichols]

Looks like GitHub creates a branch for the revert as soon as someone clicks 
Revert (I didn’t mean to, I just wanted to see what the next screen looks like, 
I didn’t click Create Pull Request).

If I create the pull request, then we need to find a 3rd person to approve it, 
so hoping you can still do it...

Alberto, will it let you click "Create pull request" from here:
https://github.com/apache/geode/compare/revert-4937-feature_parity_redis_srem_command

> On Apr 11, 2020, at 11:09 AM, Alberto Gomez  wrote:
> 
> Thanks Owen.
> 
> It seems you already initiated the revert.
> 
> Best regards,
> 
> -Alberto
> 
> From: Owen Nichols 
> Sent: Saturday, April 11, 2020 8:04 PM
> To: dev@geode.apache.org 
> Subject: Re: Help with Concourse-CI redis integration test failing
> 
> Hi Alberto, it looks like Integration tests have been broken since PR#4937 
> was merged at the end of the day Friday.  However this PR had passed all its 
> PR checks (as of 5 hours before it was merged) so I suspect this was just an 
> unexpected interaction with the PR#4912 that came in just 3 hours before it).
> 
> In general there is an expectation in the Geode community that when a commit 
> breaks the pipeline, it should be reverted first, then fixed.
> 
> Alberto, since I don’t think you’re a committer, how about you initiate the 
> revert and I’ll approve it?  GitHub makes this pretty easy: go to 
> https://github.com/apache/geode/pull/4937 and click Revert near the 
> bottom-right of the page then click Create pull request.
> 
>> On Apr 11, 2020, at 10:01 AM, Alberto Gomez  wrote:
>> 
>> Hi,
>> 
>> I am seeing the following integration test case fail in the Concourse-CI 
>> when executed on my pull request (https://github.com/apache/geode/pull/4928):
>> 
>> org.apache.geode.redis.SetsIntegrationTest > 
>> testSRem_should_notRemoveMembersOfSetNotSpecified FAILED
>> 
>> Interestingly, if I run the test locally it passes. Also, the line of code 
>> where the failure occurs does not point to a valid line in the code I see:
>> 
>> at 
>> org.apache.geode.redis.SetsIntegrationTest.testSRem_should_notRemoveMembersOfSetNotSpecified(SetsIntegrationTest.java:784
>> 
>> Any idea on what could be happening?
>> 
>> Thanks in advance,
>> 
>> -Alberto G.
> 

Re: Help with Concourse-CI redis integration test failing

[Alberto Gomez]

I clicked on the Create Pull Request button but got the following error:

Pull request creation failed. Validation failed: You can't perform that action 
at this time.

-Alberto


From: Owen Nichols 
Sent: Saturday, April 11, 2020 8:35 PM
To: dev@geode.apache.org 
Subject: Re: Help with Concourse-CI redis integration test failing

Looks like GitHub creates a branch for the revert as soon as someone clicks 
Revert (I didn’t mean to, I just wanted to see what the next screen looks like, 
I didn’t click Create Pull Request).

If I create the pull request, then we need to find a 3rd person to approve it, 
so hoping you can still do it...

Alberto, will it let you click "Create pull request" from here:
https://github.com/apache/geode/compare/revert-4937-feature_parity_redis_srem_command

> On Apr 11, 2020, at 11:09 AM, Alberto Gomez  wrote:
>
> Thanks Owen.
>
> It seems you already initiated the revert.
>
> Best regards,
>
> -Alberto
> 
> From: Owen Nichols 
> Sent: Saturday, April 11, 2020 8:04 PM
> To: dev@geode.apache.org 
> Subject: Re: Help with Concourse-CI redis integration test failing
>
> Hi Alberto, it looks like Integration tests have been broken since PR#4937 
> was merged at the end of the day Friday.  However this PR had passed all its 
> PR checks (as of 5 hours before it was merged) so I suspect this was just an 
> unexpected interaction with the PR#4912 that came in just 3 hours before it).
>
> In general there is an expectation in the Geode community that when a commit 
> breaks the pipeline, it should be reverted first, then fixed.
>
> Alberto, since I don’t think you’re a committer, how about you initiate the 
> revert and I’ll approve it?  GitHub makes this pretty easy: go to 
> https://github.com/apache/geode/pull/4937 and click Revert near the 
> bottom-right of the page then click Create pull request.
>
>> On Apr 11, 2020, at 10:01 AM, Alberto Gomez  wrote:
>>
>> Hi,
>>
>> I am seeing the following integration test case fail in the Concourse-CI 
>> when executed on my pull request (https://github.com/apache/geode/pull/4928):
>>
>> org.apache.geode.redis.SetsIntegrationTest > 
>> testSRem_should_notRemoveMembersOfSetNotSpecified FAILED
>>
>> Interestingly, if I run the test locally it passes. Also, the line of code 
>> where the failure occurs does not point to a valid line in the code I see:
>>
>> at 
>> org.apache.geode.redis.SetsIntegrationTest.testSRem_should_notRemoveMembersOfSetNotSpecified(SetsIntegrationTest.java:784
>>
>> Any idea on what could be happening?
>>
>> Thanks in advance,
>>
>> -Alberto G.
>

Re: Help with Concourse-CI redis integration test failing

[Alberto Gomez]

Owen, I managed to create the revert pull request original pull request page.

https://github.com/apache/geode/pull/4947
[https://avatars3.githubusercontent.com/u/47359?s=400&v=4]
Revert "GEODE-7978: Improve tests for Redis Module SREM Command" by albertogpz 
· Pull Request #4947 · apache/geode
Reverts #4937
github.com


-Alberto


From: Alberto Gomez 
Sent: Saturday, April 11, 2020 8:49 PM
To: dev@geode.apache.org 
Subject: Re: Help with Concourse-CI redis integration test failing

I clicked on the Create Pull Request button but got the following error:

Pull request creation failed. Validation failed: You can't perform that action 
at this time.

-Alberto


From: Owen Nichols 
Sent: Saturday, April 11, 2020 8:35 PM
To: dev@geode.apache.org 
Subject: Re: Help with Concourse-CI redis integration test failing

Looks like GitHub creates a branch for the revert as soon as someone clicks 
Revert (I didn’t mean to, I just wanted to see what the next screen looks like, 
I didn’t click Create Pull Request).

If I create the pull request, then we need to find a 3rd person to approve it, 
so hoping you can still do it...

Alberto, will it let you click "Create pull request" from here:
https://github.com/apache/geode/compare/revert-4937-feature_parity_redis_srem_command

> On Apr 11, 2020, at 11:09 AM, Alberto Gomez  wrote:
>
> Thanks Owen.
>
> It seems you already initiated the revert.
>
> Best regards,
>
> -Alberto
> 
> From: Owen Nichols 
> Sent: Saturday, April 11, 2020 8:04 PM
> To: dev@geode.apache.org 
> Subject: Re: Help with Concourse-CI redis integration test failing
>
> Hi Alberto, it looks like Integration tests have been broken since PR#4937 
> was merged at the end of the day Friday.  However this PR had passed all its 
> PR checks (as of 5 hours before it was merged) so I suspect this was just an 
> unexpected interaction with the PR#4912 that came in just 3 hours before it).
>
> In general there is an expectation in the Geode community that when a commit 
> breaks the pipeline, it should be reverted first, then fixed.
>
> Alberto, since I don’t think you’re a committer, how about you initiate the 
> revert and I’ll approve it?  GitHub makes this pretty easy: go to 
> https://github.com/apache/geode/pull/4937 and click Revert near the 
> bottom-right of the page then click Create pull request.
>
>> On Apr 11, 2020, at 10:01 AM, Alberto Gomez  wrote:
>>
>> Hi,
>>
>> I am seeing the following integration test case fail in the Concourse-CI 
>> when executed on my pull request (https://github.com/apache/geode/pull/4928):
>>
>> org.apache.geode.redis.SetsIntegrationTest > 
>> testSRem_should_notRemoveMembersOfSetNotSpecified FAILED
>>
>> Interestingly, if I run the test locally it passes. Also, the line of code 
>> where the failure occurs does not point to a valid line in the code I see:
>>
>> at 
>> org.apache.geode.redis.SetsIntegrationTest.testSRem_should_notRemoveMembersOfSetNotSpecified(SetsIntegrationTest.java:784
>>
>> Any idea on what could be happening?
>>
>> Thanks in advance,
>>
>> -Alberto G.
>

Re: Help with Concourse-CI redis integration test failing

[Owen Nichols]

Merged the revert PR to develop.

John or Jens, please have another try (create a new branch from latest develop 
then revert the revert commit)

Alberto, and anyone else with a PR affected by this, just push an empty commit 
or another commit to your PR (or merge or rebase latest develop, if you prefer) 
to get the PR checks to run against the latest develop, then you should be good 
to go. 

> On Apr 11, 2020, at 11:52 AM, Alberto Gomez  wrote:
> 
> Owen, I managed to create the revert pull request original pull request page.
> 
> https://github.com/apache/geode/pull/4947
> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]
> Revert "GEODE-7978: Improve tests for Redis Module SREM Command" by 
> albertogpz · Pull Request #4947 · 
> apache/geode
> Reverts #4937
> github.com
> 
> 
> -Alberto
> 
> 
> From: Alberto Gomez 
> Sent: Saturday, April 11, 2020 8:49 PM
> To: dev@geode.apache.org 
> Subject: Re: Help with Concourse-CI redis integration test failing
> 
> I clicked on the Create Pull Request button but got the following error:
> 
> Pull request creation failed. Validation failed: You can't perform that 
> action at this time.
> 
> -Alberto
> 
> 
> From: Owen Nichols 
> Sent: Saturday, April 11, 2020 8:35 PM
> To: dev@geode.apache.org 
> Subject: Re: Help with Concourse-CI redis integration test failing
> 
> Looks like GitHub creates a branch for the revert as soon as someone clicks 
> Revert (I didn’t mean to, I just wanted to see what the next screen looks 
> like, I didn’t click Create Pull Request).
> 
> If I create the pull request, then we need to find a 3rd person to approve 
> it, so hoping you can still do it...
> 
> Alberto, will it let you click "Create pull request" from here:
> https://github.com/apache/geode/compare/revert-4937-feature_parity_redis_srem_command
> 
>> On Apr 11, 2020, at 11:09 AM, Alberto Gomez  wrote:
>> 
>> Thanks Owen.
>> 
>> It seems you already initiated the revert.
>> 
>> Best regards,
>> 
>> -Alberto
>> 
>> From: Owen Nichols 
>> Sent: Saturday, April 11, 2020 8:04 PM
>> To: dev@geode.apache.org 
>> Subject: Re: Help with Concourse-CI redis integration test failing
>> 
>> Hi Alberto, it looks like Integration tests have been broken since PR#4937 
>> was merged at the end of the day Friday.  However this PR had passed all its 
>> PR checks (as of 5 hours before it was merged) so I suspect this was just an 
>> unexpected interaction with the PR#4912 that came in just 3 hours before it).
>> 
>> In general there is an expectation in the Geode community that when a commit 
>> breaks the pipeline, it should be reverted first, then fixed.
>> 
>> Alberto, since I don’t think you’re a committer, how about you initiate the 
>> revert and I’ll approve it?  GitHub makes this pretty easy: go to 
>> https://github.com/apache/geode/pull/4937 and click Revert near the 
>> bottom-right of the page then click Create pull request.
>> 
>>> On Apr 11, 2020, at 10:01 AM, Alberto Gomez  wrote:
>>> 
>>> Hi,
>>> 
>>> I am seeing the following integration test case fail in the Concourse-CI 
>>> when executed on my pull request 
>>> (https://github.com/apache/geode/pull/4928):
>>> 
>>> org.apache.geode.redis.SetsIntegrationTest > 
>>> testSRem_should_notRemoveMembersOfSetNotSpecified FAILED
>>> 
>>> Interestingly, if I run the test locally it passes. Also, the line of code 
>>> where the failure occurs does not point to a valid line in the code I see:
>>> 
>>> at 
>>> org.apache.geode.redis.SetsIntegrationTest.testSRem_should_notRemoveMembersOfSetNotSpecified(SetsIntegrationTest.java:784
>>> 
>>> Any idea on what could be happening?
>>> 
>>> Thanks in advance,
>>> 
>>> -Alberto G.
>> 
>