[
https://issues.apache.org/jira/browse/GEODE-8730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17243444#comment-17243444
]
Bill Burcham edited comment on GEODE-8730 at 12/3/20, 7:02 PM:
---------------------------------------------------------------
>From the IDE I ran the docker Gradle task in geode-assembly to create a fresh
>Geode Docker image.
Then from
/Users/bburcham/Projects/geode/geode-assembly/src/acceptanceTest/resources/org/apache/geode/client/sni
I ran "docker-compose up" and from the Docker app dashboard I opened a shell
into the running ("geode") container.
Once in, I "apt-get update" and "apt-get install net-tools".
{noformat}
# netstat -lp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
PID/Program name
tcp 0 0 127.0.0.11:37221 0.0.0.0:* LISTEN
-
udp 0 0 127.0.0.11:42103 0.0.0.0:*
-
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node PID/Program name
Path
{noformat}
and then ran the gfsh startup script: "gfsh run
--file=/geode/scripts/geode-starter-2.gfsh". For reference that file contains:
{noformat}
start locator --name=locator-maeve --connect=false --redirect-output
--hostname-for-clients=locator-maeve
--properties-file=/geode/config/gemfire.properties
--security-properties-file=/geode/config/gfsecurity.properties
--J=-Dgemfire.ssl-keystore=/geode/config/locator-maeve-keystore.jks
start server --name=server-dolores --group=group-dolores
--hostname-for-clients=server-dolores --locators=geode[10334]
--properties-file=/geode/config/gemfire.properties
--security-properties-file=/geode/config/gfsecurity.properties
--J=-Dgemfire.ssl-keystore=/geode/config/server-dolores-keystore.jks
start server --name=server-clementine --group=group-clementine
--hostname-for-clients=server-clementine --server-port=40405
--locators=geode[10334] --properties-file=/geode/config/gemfire.properties
--security-properties-file=/geode/config/gfsecurity.properties
--J=-Dgemfire.ssl-keystore=/geode/config/server-clementine-keystore.jks
connect --locator=geode[10334] --use-ssl=true
--security-properties-file=/geode/config/gfsecurity.properties
create region --name=region-dolores --group=group-dolores --type=REPLICATE
create region --name=region-clementine --group=group-clementine --type=REPLICATE
{noformat}
{noformat}
# netstat -lp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
PID/Program name
tcp 0 0 geode:46867 0.0.0.0:* LISTEN
256/java
tcp 0 0 geode:43540 0.0.0.0:* LISTEN
515/java
tcp 0 0 0.0.0.0:40404 0.0.0.0:* LISTEN
419/java
tcp 0 0 0.0.0.0:40405 0.0.0.0:* LISTEN
515/java
tcp 0 0 0.0.0.0:40053 0.0.0.0:* LISTEN
515/java
tcp 0 0 0.0.0.0:46649 0.0.0.0:* LISTEN
256/java
tcp 0 0 geode:57053 0.0.0.0:* LISTEN
256/java
tcp 0 0 geode:55518 0.0.0.0:* LISTEN
515/java
tcp 0 0 geode:55486 0.0.0.0:* LISTEN
419/java
tcp 0 0 0.0.0.0:7070 0.0.0.0:* LISTEN
256/java
tcp 0 0 0.0.0.0:10334 0.0.0.0:* LISTEN
256/java
tcp 0 0 0.0.0.0:33953 0.0.0.0:* LISTEN
419/java
tcp 0 0 127.0.0.11:37221 0.0.0.0:* LISTEN
-
tcp 0 0 geode:48715 0.0.0.0:* LISTEN
419/java
tcp 0 0 0.0.0.0:1099 0.0.0.0:* LISTEN
256/java
udp 0 0 geode:41000 0.0.0.0:*
256/java
udp 0 0 geode:41001 0.0.0.0:*
419/java
udp 0 0 geode:41002 0.0.0.0:*
515/java
udp 0 0 127.0.0.11:42103 0.0.0.0:*
-
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node PID/Program name
Path
unix 2 [ ACC ] STREAM LISTENING 155159 256/java
/tmp/.java_pid256.tmp
unix 2 [ ACC ] STREAM LISTENING 158787 419/java
/tmp/.java_pid419.tmp
unix 2 [ ACC ] STREAM LISTENING 159071 515/java
/tmp/.java_pid515.tmp
{noformat}
Grouping these by PID: locator first, then cache servers:
{noformat}
tcp 0 0 0.0.0.0:10334 0.0.0.0:* LISTEN
256/java for locator clients
tcp 0 0 0.0.0.0:1099 0.0.0.0:* LISTEN
256/java for gfsh
tcp 0 0 0.0.0.0:7070 0.0.0.0:* LISTEN
256/java for browser (pulse)
tcp 0 0 0.0.0.0:46649 0.0.0.0:* LISTEN
256/java
tcp 0 0 geode:46867 0.0.0.0:* LISTEN
256/java
tcp 0 0 geode:57053 0.0.0.0:* LISTEN
256/java
udp 0 0 geode:41000 0.0.0.0:*
256/java for membership
tcp 0 0 0.0.0.0:40404 0.0.0.0:* LISTEN
419/java for client's cache
tcp 0 0 0.0.0.0:33953 *** 0.0.0.0:* LISTEN
419/java
tcp 0 0 geode:55486 0.0.0.0:* LISTEN
419/java for health
tcp 0 0 geode:48715 0.0.0.0:* LISTEN
419/java for peer's cache
udp 0 0 geode:41001 0.0.0.0:*
419/java for membership
tcp 0 0 0.0.0.0:40405 0.0.0.0:* LISTEN
515/java for client's cache
tcp 0 0 0.0.0.0:40053 *** 0.0.0.0:* LISTEN
515/java
tcp 0 0 geode:43540 0.0.0.0:* LISTEN
515/java for peer's cache
tcp 0 0 geode:55518 0.0.0.0:* LISTEN
515/java for health
udp 0 0 geode:41002 0.0.0.0:*
515/java for membership
{noformat}
I've highlighted with "***" the two bindings that are odd. These are ephemeral
ports but are not within the default (configured) port range 41000-61000. I
expect these are different each time we run and are the cause of this bug.
I searched the logs for those ports and didn't find them. I wonder what those
bindings are? A cache server binds these TCP ports:
* client's cache (40404, 40405 above)
* peer's cache ostensibly in port range (41000-61000)
* health monitoring also ostensibly in port range (41000-61000)
Of the three unknown TCP port bindings per cache server in the netstat output
above we only have categories for two (peer's cache, health monitoring.) What's
that third category?
In summary we have these two unexplained bindings (one per cache server) and we
have the one unexplained TCP binding before Geode even starts (see first
netstat above.)
was (Author: bburcham):
>From the IDE I ran the docker Gradle task in geode-assembly to create a fresh
>Geode Docker image.
Then from
/Users/bburcham/Projects/geode/geode-assembly/src/acceptanceTest/resources/org/apache/geode/client/sni
I ran "docker-compose up" and from the Docker app dashboard I opened a shell
into the running ("geode") container.
Once in, I "apt-get update" and "apt-get install net-tools".
{noformat}
# netstat -lp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
PID/Program name
tcp 0 0 127.0.0.11:37221 0.0.0.0:* LISTEN
-
udp 0 0 127.0.0.11:42103 0.0.0.0:*
-
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node PID/Program name
Path
{noformat}
and then ran the gfsh startup script: "gfsh run
--file=/geode/scripts/geode-starter-2.gfsh". For reference that file contains:
{noformat}
start locator --name=locator-maeve --connect=false --redirect-output
--hostname-for-clients=locator-maeve
--properties-file=/geode/config/gemfire.properties
--security-properties-file=/geode/config/gfsecurity.properties
--J=-Dgemfire.ssl-keystore=/geode/config/locator-maeve-keystore.jks
start server --name=server-dolores --group=group-dolores
--hostname-for-clients=server-dolores --locators=geode[10334]
--properties-file=/geode/config/gemfire.properties
--security-properties-file=/geode/config/gfsecurity.properties
--J=-Dgemfire.ssl-keystore=/geode/config/server-dolores-keystore.jks
start server --name=server-clementine --group=group-clementine
--hostname-for-clients=server-clementine --server-port=40405
--locators=geode[10334] --properties-file=/geode/config/gemfire.properties
--security-properties-file=/geode/config/gfsecurity.properties
--J=-Dgemfire.ssl-keystore=/geode/config/server-clementine-keystore.jks
connect --locator=geode[10334] --use-ssl=true
--security-properties-file=/geode/config/gfsecurity.properties
create region --name=region-dolores --group=group-dolores --type=REPLICATE
create region --name=region-clementine --group=group-clementine --type=REPLICATE
{noformat}
{noformat}
# netstat -lp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
PID/Program name
tcp 0 0 geode:46867 0.0.0.0:* LISTEN
256/java
tcp 0 0 geode:43540 0.0.0.0:* LISTEN
515/java
tcp 0 0 0.0.0.0:40404 0.0.0.0:* LISTEN
419/java
tcp 0 0 0.0.0.0:40405 0.0.0.0:* LISTEN
515/java
tcp 0 0 0.0.0.0:40053 0.0.0.0:* LISTEN
515/java
tcp 0 0 0.0.0.0:46649 0.0.0.0:* LISTEN
256/java
tcp 0 0 geode:57053 0.0.0.0:* LISTEN
256/java
tcp 0 0 geode:55518 0.0.0.0:* LISTEN
515/java
tcp 0 0 geode:55486 0.0.0.0:* LISTEN
419/java
tcp 0 0 0.0.0.0:7070 0.0.0.0:* LISTEN
256/java
tcp 0 0 0.0.0.0:10334 0.0.0.0:* LISTEN
256/java
tcp 0 0 0.0.0.0:33953 0.0.0.0:* LISTEN
419/java
tcp 0 0 127.0.0.11:37221 0.0.0.0:* LISTEN
-
tcp 0 0 geode:48715 0.0.0.0:* LISTEN
419/java
tcp 0 0 0.0.0.0:1099 0.0.0.0:* LISTEN
256/java
udp 0 0 geode:41000 0.0.0.0:*
256/java
udp 0 0 geode:41001 0.0.0.0:*
419/java
udp 0 0 geode:41002 0.0.0.0:*
515/java
udp 0 0 127.0.0.11:42103 0.0.0.0:*
-
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node PID/Program name
Path
unix 2 [ ACC ] STREAM LISTENING 155159 256/java
/tmp/.java_pid256.tmp
unix 2 [ ACC ] STREAM LISTENING 158787 419/java
/tmp/.java_pid419.tmp
unix 2 [ ACC ] STREAM LISTENING 159071 515/java
/tmp/.java_pid515.tmp
{noformat}
Grouping these by PID: locator first, then cache servers:
{noformat}
tcp 0 0 0.0.0.0:10334 0.0.0.0:* LISTEN
256/java for locator clients
tcp 0 0 0.0.0.0:1099 0.0.0.0:* LISTEN
256/java for gfsh
tcp 0 0 0.0.0.0:7070 0.0.0.0:* LISTEN
256/java for browser (pulse)
tcp 0 0 0.0.0.0:46649 0.0.0.0:* LISTEN
256/java
tcp 0 0 geode:46867 0.0.0.0:* LISTEN
256/java
tcp 0 0 geode:57053 0.0.0.0:* LISTEN
256/java
udp 0 0 geode:41000 0.0.0.0:*
256/java for membership
tcp 0 0 0.0.0.0:40404 0.0.0.0:* LISTEN
419/java for client's cache
tcp 0 0 0.0.0.0:33953 *** 0.0.0.0:* LISTEN
419/java
tcp 0 0 geode:55486 0.0.0.0:* LISTEN
419/java for health
tcp 0 0 geode:48715 0.0.0.0:* LISTEN
419/java for peer's cache
udp 0 0 geode:41001 0.0.0.0:*
419/java for membership
tcp 0 0 0.0.0.0:40405 0.0.0.0:* LISTEN
515/java for client's cache
tcp 0 0 0.0.0.0:40053 *** 0.0.0.0:* LISTEN
515/java
tcp 0 0 geode:43540 0.0.0.0:* LISTEN
515/java for peer's cache
tcp 0 0 geode:55518 0.0.0.0:* LISTEN
515/java for health
udp 0 0 geode:41002 0.0.0.0:*
515/java for membership
{noformat}
I've highlighted with "***" the two bindings that are odd. These are ephemeral
ports but are not within the default (configured) port range 41000-61000. I
expect these are different each time we run and are the cause of this bug.
I searched the logs for those ports and didn't find them. I wonder what those
bindings are? A cache server binds these TCP ports:
* client's cache (40404, 40405 above)
* peer's cache ostensibly in port range (41000-61000)
* health monitoring also ostensibly in port range (41000-61000)
Of the three unknown TCP port bindings per cache server in the netstat output
above we only have categories for two (peer's cache, health monitoring.) What's
that third category?
> CI failure: DualServerSNIAcceptanceTest fails to start server because port is
> in use
> ------------------------------------------------------------------------------------
>
> Key: GEODE-8730
> URL: https://issues.apache.org/jira/browse/GEODE-8730
> Project: Geode
> Issue Type: Bug
> Components: membership
> Reporter: Darrel Schneider
> Assignee: Bill Burcham
> Priority: Major
>
> The run is here:
> [https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/AcceptanceTestOpenJDK8/builds/587]
> {noformat}
> org.apache.geode.client.sni.DualServerSNIAcceptanceTest > classMethod FAILED
> com.palantir.docker.compose.execution.DockerExecutionException:
> 'docker-compose exec -T geode gfsh run
> --file=/geode/scripts/geode-starter-2.gfsh' returned exit code 1
> The output was:
> 1. Executing - start locator --name=locator-maeve --connect=false
> --redirect-output --hostname-for-clients=locator-maeve
> --properties-file=/geode/config/gemfire.properties
> --security-properties-file=********
> --J=-Dgemfire.ssl-keystore=/geode/config/locator-maeve-keystore.jks
> ...........................
> Locator in /locator-maeve on geode[10334] as locator-maeve is currently
> online.
> Process ID: 47
> Uptime: 16 seconds
> Geode Version: 1.14.0-build.0
> Java Version: 11.0.9.1
> Log File: /locator-maeve/locator-maeve.log
> JVM Arguments: -DgemfirePropertyFile=/geode/config/gemfire.properties
> -DgemfireSecurityPropertyFile=/geode/config/gfsecurity.properties
> -Dgemfire.enable-cluster-configuration=true
> -Dgemfire.load-cluster-configuration-from-dir=false
> -Dgemfire.ssl-keystore=/geode/config/locator-maeve-keystore.jks
> -Dgemfire.launcher.registerSignalHandlers=true -Djava.awt.headless=true
> -Dsun.rmi.dgc.server.gcInterval=9223372036854775806
> -Dgemfire.OSProcess.DISABLE_REDIRECTION_CONFIGURATION=true
> Class-Path:
> /geode/lib/geode-core-1.14.0-build.0.jar:/geode/lib/geode-dependencies.jar
> 2. Executing - start server --name=server-dolores --group=group-dolores
> --hostname-for-clients=server-dolores --locators=geode[10334]
> --properties-file=/geode/config/gemfire.properties
> --security-properties-file=********
> --J=-Dgemfire.ssl-keystore=/geode/config/server-dolores-keystore.jks
> .......
> Server in /server-dolores on geode[40404] as server-dolores is currently
> online.
> Process ID: 199
> Uptime: 5 seconds
> Geode Version: 1.14.0-build.0
> Java Version: 11.0.9.1
> Log File: /server-dolores/server-dolores.log
> JVM Arguments: -DgemfirePropertyFile=/geode/config/gemfire.properties
> -DgemfireSecurityPropertyFile=/geode/config/gfsecurity.properties
> -Dgemfire.start-dev-rest-api=false -Dgemfire.locators=geode[10334]
> -Dgemfire.use-cluster-configuration=true -Dgemfire.groups=group-dolores
> -Dgemfire.ssl-keystore=/geode/config/server-dolores-keystore.jks
> -Dgemfire.launcher.registerSignalHandlers=true -Djava.awt.headless=true
> -Dsun.rmi.dgc.server.gcInterval=9223372036854775806
> Class-Path:
> /geode/lib/geode-core-1.14.0-build.0.jar:/geode/lib/geode-dependencies.jar
> 3. Executing - start server --name=server-clementine
> --group=group-clementine --hostname-for-clients=server-clementine
> --server-port=40405 --locators=geode[10334]
> --properties-file=/geode/config/gemfire.properties
> --security-properties-file=********
> --J=-Dgemfire.ssl-keystore=/geode/config/server-clementine-keystore.jks
> ......The Cache Server process terminated unexpectedly with exit status
> 1. Please refer to the log file in /server-clementine for full details.
> Exception in thread "main" java.lang.RuntimeException: An IO error
> occurred while starting a Server in /server-clementine on geode[40405]:
> Network is unreachable; port (40405) is not available on localhost.
> at
> org.apache.geode.distributed.ServerLauncher.start(ServerLauncher.java:852)
> at
> org.apache.geode.distributed.ServerLauncher.run(ServerLauncher.java:737)
> at
> org.apache.geode.distributed.ServerLauncher.main(ServerLauncher.java:256)
> Caused by: java.net.BindException: Network is unreachable; port (40405)
> is not available on localhost.
> at
> org.apache.geode.distributed.AbstractLauncher.assertPortAvailable(AbstractLauncher.java:142)
> at
> org.apache.geode.distributed.ServerLauncher.start(ServerLauncher.java:794)
> ... 2 more
> ************************* Execution Summary ***********************
> Script file: /geode/scripts/geode-starter-2.gfsh
> Command-1 : start locator --name=locator-maeve --connect=false
> --redirect-output --hostname-for-clients=locator-maeve
> --properties-file=/geode/config/gemfire.properties
> --security-properties-file=/geode/config/gfsecurity.properties
> --J=-Dgemfire.ssl-keystore=/geode/config/locator-maeve-keystore.jks
> Status : PASSED
> Command-2 : start server --name=server-dolores --group=group-dolores
> --hostname-for-clients=server-dolores --locators=geode[10334]
> --properties-file=/geode/config/gemfire.properties
> --security-properties-file=/geode/config/gfsecurity.properties
> --J=-Dgemfire.ssl-keystore=/geode/config/server-dolores-keystore.jks
> Status : PASSED
> Command-3 : start server --name=server-clementine
> --group=group-clementine --hostname-for-clients=server-clementine
> --server-port=40405 --locators=geode[10334]
> --properties-file=/geode/config/gemfire.properties
> --security-properties-file=/geode/config/gfsecurity.properties
> --J=-Dgemfire.ssl-keystore=/geode/config/server-clementine-keystore.jks
> Status : FAILED
> at
> com.palantir.docker.compose.execution.Command.lambda$throwingOnError$12(Command.java:60)
> at
> com.palantir.docker.compose.execution.Command.execute(Command.java:50)
> at
> com.palantir.docker.compose.execution.DefaultDockerCompose.exec(DefaultDockerCompose.java:122)
> at
> com.palantir.docker.compose.execution.DelegatingDockerCompose.exec(DelegatingDockerCompose.java:86)
> at
> com.palantir.docker.compose.execution.RetryingDockerCompose.exec(RetryingDockerCompose.java:22)
> at
> com.palantir.docker.compose.DockerComposeRule.exec(DockerComposeRule.java:171)
> at
> org.apache.geode.client.sni.DualServerSNIAcceptanceTest.beforeClass(DualServerSNIAcceptanceTest.java:77)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
