Indxes and hints

[Roi Apelker]

Hi,

I have a few questions regarding indexes and hints, if someone could confirm 
the below it would be great:

- I have a situation where I use 3 field values in a select (something like 
select where A>1, B>1, C=true)
- A and B are fields on the key, and C is a field on the value.
- A and B are indexes
- I am looking for the most efficient way to execute the query above, in the 
situation where there is overflow to eviction files, meaning some of the data 
has already been evicted to a file, which slows down the select considerably 
(this is not persistence, but overflow).


1. Is it true to say, that the query as it is will load all the data values 
from the file, since the field C is part of the value, which is already 
persisted to file?
2. If I add a hint on A and B, will it mean that there will be a "2 phase 
search", first the select on A and B, and then, only on the results, on the 
field C? (this way, not all records will be loaded from file, only those that 
suit the A and B condition)
3. Is it possible to define an index on a value field? (i.e. not from the key) 
- will it work exactly like defining one form the key or are three any 
limitations? (again, I am looking to overcome the situation, where as it seems, 
the records are loaded unnecessarily from disk)

Thank you,

Roi


-Original Message-
From: Roi Apelker 
Sent: Thursday, August 24, 2017 7:03 PM
To: dev@geode.apache.org
Subject: eviction files

Hi,

I am looking into the internals of the eviction process,

Can anyone point me to the most important classes, the main mechanism "wheels" 
etc.?

Thanks,

Roi

-Original Message-
From: Roi Apelker
Sent: Wednesday, August 16, 2017 8:38 PM
To: dev@geode.apache.org
Subject: RE: continuous query internal mechanism questions

It seems like the code in the native client (in the version I have, which may 
be old) send the message to all servers:

CqResultsPtr CqQueryImpl::executeWithInitialResults(uint32_t timeout) {
  ...

  TcrMessage msg(TcrMessage::EXECUTECQ_WITH_IR_MSG_TYPE, m_cqName, 
m_queryString, CqState::RUNNING, isDurable(), m_tccdm);
  TcrMessage reply(true, m_tccdm);
  ChunkedQueryResponse* resultCollector = (new ChunkedQueryResponse(reply));
  reply.setChunkedResultHandler(static_cast(resultCollector));
  reply.setTimeout(timeout);

  GfErrType err = GF_NOERR;
  err = m_tccdm->sendSyncRequest(msg, reply); ..

And sendSyncRequest:
...

for (std::vector::iterator ep = m_endpoints.begin(); ep != 
m_endpoints.end(); ++ep) {
if ((*ep)->connected()) {
  (*ep)->setDM(this);
  opErr = sendRequestToEP(request, reply, *ep);//this will go to 
ThinClientDistributionManager

...


Can this be causing the issue?



-Roi





-Original Message-
From: Jason Huynh [mailto:jasonhu...@apache.org]
Sent: Tuesday, August 15, 2017 9:25 PM
To: dev@geode.apache.org
Subject: Re: continuous query internal mechanism questions

I am not quite sure how native client registers cqs. From my understanding:
with the java api, I believe there is only one message (ExecuteCQ message) that 
is executed on the server side and then replicated to the other nodes through 
the profile (OperationMessage).

It seems the extra ExecuteCQ message failing and then closing the cq might be 
putting the system in a weird state...

On Tue, Aug 15, 2017 at 7:56 AM Roi Apelker  wrote:

> Hi,
>
> I have been examining the continuous query registration mechanism for 
> quite some time This is related to an issue that I have, where 
> sometimes a node crashes (1 node out of 2), and the other one does not 
> send CQ events. The CQ is registered on a partitioned region which 
> resides on these 2 nodes.
>
> I noticed the following behavior, and I wonder if anyone can comment 
> regarding it, if it is justified or not and what is the reason:
>
> 1. When the software using the client (native client) registers for 
> the CQ, a CQ command (ExecuteCQ61) is received on both servers.
>  -- is this normal behaviour? Does the client actually send this 
> command to both servers?
>
> 2. When this command is received by a server, and the CQ is 
> registered, another registration message is sent to the other node via 
> an OperationMessage (REGISTER_CQ)
>  -- it seems that regularly, the server can handle this situation as 
> the second registration identifies the previous one and does not 
> affect it. but the question, why do we need this 2nd registration, if 
> there is a command sent to each server?
>
> 3. For some reason, sometimes there is a failure to complete the first 
> registration (executed by ExecuteCQ61) and then this failure causes a 
> closure to the CQ, which is accompanied with a close request to the 
> other node.
>  -- I assume by now, since 2 registrations and one closure have 
> occurred on node 2, the CQ is still active and the client receives 
> notifications.
>
> 4. Sometimes, 1 out of 5, once node 1 crashes, I get a cleanup 
> operation, caused by the crash (via MemberCrashedEvent

Re: [jira] [Created] (GEODE-3524) Geode native client compilation with Intel compiler is failing

[Jacob Barrett]

It looks like the actual failure is in libxml2. I would suggest downloading 
that source and compiling with Intel. If it works then it’s going to be a 
simple fix to get the CMake files touched up to compile it. If it doesn’t work 
then it may take some patching of the libxml2 makefiles.



> On Aug 27, 2017, at 6:33 AM, Daniel Farcovich (JIRA)  wrote:
> 
> Daniel Farcovich created GEODE-3524:
> ---
> 
> Summary: Geode native client compilation with Intel compiler is 
> failing
> Key: GEODE-3524
> URL: https://issues.apache.org/jira/browse/GEODE-3524
> Project: Geode
>  Issue Type: Bug
>  Components: native client
>Reporter: Daniel Farcovich
> Fix For: 1.3.0
> 
> 
> When setting environment variable CXX and CC to intel compiler 15.0.7.235, 
> compilation of c files is failing with the following warnings and errors:
> 
> [exec]   CCLD xmllint
> [exec] icc: command line warning #10006: ignoring unknown option 
> '-Wno-format-extra-args'
> [exec] icc: command line warning #10006: ignoring unknown option 
> '-Wcast-align'
> [exec] icc: command line warning #10006: ignoring unknown option 
> '-Waggregate-return'
> [exec] icc: command line warning #10006: ignoring unknown option 
> '-Wnested-externs'
> [exec] icc: command line warning #10006: ignoring unknown option 
> '-Wredundant-decls'
> [exec] ld: warning: libimf.so, needed by ./.libs/libxml2.so, not found 
> (try using -rpath or -rpath-link)
> [exec] ld: warning: libsvml.so, needed by ./.libs/libxml2.so, not found 
> (try using -rpath or -rpath-link)
> [exec] ld: warning: libirng.so, needed by ./.libs/libxml2.so, not found 
> (try using -rpath or -rpath-link)
> [exec] ld: warning: libintlc.so.5, needed by ./.libs/libxml2.so, not 
> found (try using -rpath or -rpath-link)
> [exec] ld: .libs/xmllint: hidden symbol `__intel_cpu_features_init_x' in 
> /opt/intel/15.0.7.235/composer_xe_2015.7.235/compiler/lib/intel64/libirc.a(cpu_feature_disp.o)
>  is referenced by DSO
> [exec] ld: final link failed: Bad value
> [exec] make[5]: *** [xmllint] Error 1
> [exec] make[4]: *** [all-recursive] Error 1
> [exec] make[3]: *** [all] Error 2
> [exec] make[2]: *** 
> [dependencies/libxml2/libxml2-extern-prefix/src/libxml2-extern-stamp/libxml2-extern-build]
>  Error 2
> [exec] make[1]: *** 
> [dependencies/libxml2/CMakeFiles/libxml2-extern.dir/all] Error 2
> [exec] make: *** [all] Error 2
> 
> 
> [~gregory.vort...@amdocs.com]
> 
> 
> 
> --
> This message was sent by Atlassian JIRA
> (v6.4.14#64029)

RE: Stored procedures on Apache Geode.

[marios390]

Hi John,

I am confused a little bit about the different ways of setting up geode cluster 
and deploying it in CF.
In short:

  *   If we choose gemfire we can easily provision it on CF via a tile, whereas 
if we go with Geode we have to setup cluster externally (i.e aws) correct ?
  *   I really like using annotations to configure embedded components (i.e 
locator, servers etc) and am wondering weather or not I can use those to deploy 
app  directly on CF without setting up a cluster externally.

Thanks a lot
Really appreciate it.

Marios


From: John Blum [via Apache Geode (Incubating) Developers Forum] 
[ml+s70738n24960...@n6.nabble.com]
Sent: Thursday, August 03, 2017 12:56 AM
To: Marios Sofocleous/IT/CREDITSAFE
Subject: Re: Stored procedures on Apache Geode.

Hi Marios-

Any application config annotated with @PeerCacheApplication /
@CacheServerApplication, or CacheFactoryBean directly in *JavaConfig*, or
 in XML, is a Geode (GemFire) Server, a peer member of the
distributed system (cluster) defined by a Locator.

For instance, I can...

gfsh>start locator --name=*LocatorX* --log-level=config
gfsh>start server --name=*ServerA* --disable-default-server
--log-level=config
gfsh>start server --name=*ServerB* --disable-default-server
--log-level=config

Then run the configuration-example,
AnnotationConfiguredGeodeServerApplication class from my IDE, modified as
so..

@SpringBootApplication
@CacheServerApplication(name = "AnnotationConfiguredGeodeServerApplication",
 port = AnnotationConfiguredGeodeServerApplication.GEODE_CACHE_SERVER_PORT,
*locators
= **"localhost[10334]"*)
@EnableCacheServers(servers = { @EnableCacheServer(port = 12480),
@EnableCacheServer(port = 40404) })
//@EnableLocator
//@EnableManager
...
@Import(EchoServerApplicationConfiguration.class)
public class AnnotationConfiguredGeodeServerApplication {
  ...
}

And then...

gfsh>list members
   Name| Id
-- |
--
*LocatorX*   |
10.99.199.5(LocatorX:57545:locator):1024
*ServerA*|
10.99.199.5(ServerA:57572):1025
*ServerB*|
10.99.199.5(ServerB:57579):1026
*AnnotationConfiguredGeodeServerApplication* |
10.99.199.5(AnnotationConfiguredGeodeServerApplication:57594):1027

Alternatively, I can...


@SpringBootApplication
@CacheServerApplication(name = "AnnotationConfiguredGeodeServerApplication",
  port = AnnotationConfiguredGeodeServerApplication.GEODE_CACHE_SERVER_PORT)
@EnableCacheServers(servers = { @EnableCacheServer(port = 12480),
@EnableCacheServer(port = 40404) })
@EnableLocator
@EnableManager
//@EnableSsl(components = { EnableSsl.Component.SERVER },
// keystore =
"/Users/jblum/pivdev/springonePlatform-2016/configuration-example/etc/geode/security/trusted.keystore",
// keystorePassword = "s3cr3t",
// keystoreType = "JKS",
// truststore =
"/Users/jblum/pivdev/springonePlatform-2016/configuration-example/etc/geode/security/trusted.keystore",
// truststorePassword = "s3cr3t")
@Import(EchoServerApplicationConfiguration.class)
public class AnnotationConfiguredGeodeServerApplication {
  ...
}


That is comment our SSL config, and run the *Spring* class as is.

NOTE: you can run with SSL, but then you must config all servers and
clients connecting to this Locator to use SSL.  This just simplifies
running the example with minimal fuss (such as when starting
Locators/Servers from *Gfsh*, where configuring SSL is much more difficult
to do successfully).

This example is running both an embedded *Locator* (on port 10334, by
default) and has enabled the *Manager* service (on port 1099, by default)
for JMX clients to connect (like *Gfsh*).  The *CacheServers* are
irrelevant for this demonstration.

Now, *without* even being connected, I can start several servers in *Gfsh*
and connect to this application (NOTE: the application must be running),
which again, is a *Spring Boot(ed)* GemFire peer cache member of the DS
(cluster that it forms with the *Locator* endpoint, namely...
localhost[10334]).

NOTE: the @EnableLocator annotation does let you change the embedded
*Locator's* port if you like.  Additionally, the @EnableManager annotation
allows you to change the port of the embedded *Manager* as well.

Ok, my application (AnnotationConfiguredGeodeServerApplication) is running
and I am not (yet) connected in *Gfsh*...

gfsh>list members
Command 'list members' was found but is not currently available (type
'help' then ENTER to learn about this command)

Then, I start some server (in a disconnected state)...

gfsh>start server --name=ServerA --disable-default-server
--log-level=config *--locators=localhost[10334]*
gfsh>start server --name=ServerB --disable-default-server
--log-level=config *--locators=localhost[10334]*

Now, I connect...

gfsh>connect
Connecting to Locator at [host=local

Build failed in Jenkins: Geode-nightly #936

[Apache Jenkins Server]

See 

--
[...truncated 212.36 KB...]
---
* What went wrong:
Execution failed for task ':geode-assembly:distributedTest'.
> There were failing tests. See the report at: 
> file://

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.
==

3: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':geode-assembly:integrationTest'.
> There were failing tests. See the report at: 
> file://

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.
==

4: Task failed with an exception.
---
* Where:
Script ' 
line: 116

* What went wrong:
Execution failed for task ':extensions/geode-modules-assembly:uploadArchives'.
> Could not find which method repositories() to invoke from this list:
public org.gradle.api.artifacts.dsl.RepositoryHandler 
org.gradle.api.tasks.Upload#repositories(groovy.lang.Closure)
public org.gradle.api.artifacts.dsl.RepositoryHandler 
org.gradle.api.tasks.Upload#repositories(org.gradle.api.Action)

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.
==

5: Task failed with an exception.
---
* Where:
Script ' 
line: 116

* What went wrong:
Execution failed for task ':geode-common:uploadArchives'.
> Could not find which method repositories() to invoke from this list:
public org.gradle.api.artifacts.dsl.RepositoryHandler 
org.gradle.api.tasks.Upload#repositories(groovy.lang.Closure)
public org.gradle.api.artifacts.dsl.RepositoryHandler 
org.gradle.api.tasks.Upload#repositories(org.gradle.api.Action)

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.
==

6: Task failed with an exception.
---
* Where:
Script ' 
line: 116

* What went wrong:
Execution failed for task ':geode-core:uploadArchives'.
> Could not find which method repositories() to invoke from this list:
public org.gradle.api.artifacts.dsl.RepositoryHandler 
org.gradle.api.tasks.Upload#repositories(groovy.lang.Closure)
public org.gradle.api.artifacts.dsl.RepositoryHandler 
org.gradle.api.tasks.Upload#repositories(org.gradle.api.Action)

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.
==

7: Task failed with an exception.
---
* Where:
Script ' 
line: 116

* What went wrong:
Execution failed for task ':geode-cq:uploadArchives'.
> Could not find which method repositories() to invoke from this list:
public org.gradle.api.artifacts.dsl.RepositoryHandler 
org.gradle.api.tasks.Upload#repositories(groovy.lang.Closure)
public org.gradle.api.artifacts.dsl.RepositoryHandler 
org.gradle.api.tasks.Upload#repositories(org.gradle.api.Action)

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.
==

8: Task failed with an exception.
---
* Where:
Script ' 
line: 116

* What went wrong:
Execution failed for task ':geode-json:uploadArchives'.
> Could not find which method repositories() to invoke from this list:
public org.gradle.api.artifacts.dsl.RepositoryHandler 
org.gradle.api.tasks.Upload#repositories(groovy.lang.Closure)
public org.gradle.api.artifacts.dsl.RepositoryHandler 
org.gradle.api.tasks.Upload#repositories(org.gradle.api.Action)

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.
==

9: Task failed with an exception.
---
* Where:
Script ' 
line: 116

* What went wrong:
Execution failed for task ':geode-junit:

[Spring CI] Spring Data GemFire > Nightly-ApacheGeode > #660 was SUCCESSFUL (with 2027 tests)

[Spring CI]

---
Spring Data GemFire > Nightly-ApacheGeode > #660 was successful.
---
Scheduled
2029 tests in total.

https://build.spring.io/browse/SGF-NAG-660/





--
This message is automatically generated by Atlassian Bamboo