This is an automated email from the ASF dual-hosted git repository.
ctubbsii pushed a commit to branch 2.1
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/2.1 by this push:
new 9e146f95a4 Backport and improve the cluster scripts to 2.1 (#5174)
9e146f95a4 is described below
commit 9e146f95a455e3d760fd50bc468373ffc5b3b021
Author: Christopher Tubbs <ctubb...@apache.org>
AuthorDate: Sat Dec 21 20:58:22 2024 -0500
Backport and improve the cluster scripts to 2.1 (#5174)
This backports the improvements to the accumulo-cluster and
accumulo-service scripts to 2.1 to improve script maintenance and
functionality across the branches. It also adds additional improvements
to merge up to the newer branches for 3.1.0 and 4.0.0.
Notable differences with the main branch:
* This version includes deprecated commands that are replaced by newer
versions: start-non-tservers, start-servers, stop-servers,
start-tservers, and stop-tservers
* This version includes the ability to control starting a
compaction-coordinator, which was removed in the main branch for 4.0
* This version does not allow specifying tserver groups (unsupported by
its cluster.yaml version) and uses a hard-coded value of "default"
* As before these changes, this version checks for old deprecated config
files present in conf/
* As before these changes, this version parses the cluster.yaml that was
supported for the 2.1 and 3.1 branches
* This version is modified to translate the variables emitted from its
version of the ClusterConfigParser to the newer names used in the main
branch
* This version does not require compactor groups to be configured (since
external compactions are optional)
* This version uses `-a <host>` instead of `-o
general.process.bind.address=<host>` for the bind address
* This version uses `-q` and `-g` for the group parameter
This includes relevant changes from #4966 to standardize the service
names, #5066 to reduce ssh connections to hosts, #5114 to support
starting/stopping groups of servers by type, #5126 to support resource
group overrides, #5116 to refactor and simplify the script argument
parsing using getopt, and the bugfixes and improvements from #5167
Additional improvements:
* Give the temporary file a better name, in case it doesn't get
cleaned up, it will be easier to figure out what created it
* Fix bug with invalid input that stopped processing options
after detecting one that was invalid, and failed to honor the
remaining good options, leading to serious issues affecting
a larger set of servers than intended; this was fixed by checking
the exit code from getopt
* Also fixes an issue where `stop-here` would communicate with the
accumulo manager and gracefully stop all tservers.
* Make sure we have the correct getopt tool (and not the older getopt)
* Verify the number of non-option parameters is exactly 1 (the command
to execute)
* Quote the variable containing the accumulo command since it could
contain spaces
* Remove --all (hasn't been included in any release yet, so that's fine)
When I was updating the usage, I realized I couldn't write up a
description of this that demonstrated that it made sense to exist;
it can only be used when none of the other service selection options
are used, but if none of the other service selection options are used,
then that's already the behavior, so the flag does nothing other than
introduce an error when it's misused; if we add it to be required,
then we introduce an error when no options are used, but that's the
most readable and intuitive case, so that would be weird; overall, it
seemed best to remove the flag; this simplifies the implementation
slightly, where we assume all service types, until that assumption is
corrected by the use of any other service type flag
* add more debugging output for original arguments vs. the parsed ones
* This fixes #5196 by moving the parsing of the command into the
`parse_args` method after getopt and strictly verifying that only one
non-option argument exists (also relaxes the constraint that it has to
be the first argument to the script; now it can be anywhere)
* add a quote function that uses getopt to do the quoting; as explained
in the inline comments, it produces more readable quoted output
* standardize function definition syntax (keeps both the optional
keyword "function" and the parens that become optional when the
keyword is used)
* Fix some grammar/spelling of script messages (fixes the typo of
"stoping tablet servers", because "stop" command requires a second "p"
when putting "ing" on the end)
* parse args before doing anything else in the script, since parsing
args doesn't depend on anything else, so if there's a problem parsing,
that should be noticed before doing anything else
* Add colors and improve messages
* Fix description and behavior of using server options multiple times
(it might be better to restrict being able to call it multiple times)
---------
Co-authored-by: Daniel Roberts <ddani...@gmail.com>
---
assemble/bin/accumulo-cluster | 1140 ++++++++++----------
assemble/bin/accumulo-service | 112 +-
assemble/conf/accumulo-env.sh | 32 +-
assemble/conf/accumulo.properties | 5 +
.../core/conf/cluster/ClusterConfigParser.java | 7 +-
5 files changed, 674 insertions(+), 622 deletions(-)
diff --git a/assemble/bin/accumulo-cluster b/assemble/bin/accumulo-cluster
index 130f7adf46..ab39fcec62 100755
--- a/assemble/bin/accumulo-cluster
+++ b/assemble/bin/accumulo-cluster
@@ -18,648 +18,649 @@
# under the License.
#
-function print_usage {
+#
+# Environment variables that can be set to influence the behavior
+# of this script
+#
+# ACCUMULO_LOCALHOST_ADDRESSES - set to a space delimited string of localhost
names
+# and addresses to override the default lookups
+#
+
+function print_usage() {
cat <<EOF
-Usage: accumulo-cluster <command> (<argument> ...) [<option> ...]
-
-Options:
- --dry-run Prints information and commands, but does not
execute them
-
-Commands:
- create-config Creates cluster config
- restart Restarts the Accumulo cluster
- start Starts Accumulo cluster
- stop Stops Accumulo cluster
- kill Kills Accumulo cluster
- start-non-tservers Deprecated. Starts all services except tservers
- start-servers [--all|--tservers|--no-tservers|--sservers
[group]|--compactors [queue]]
- Starts various server types, can optionally
specify a queue or group
- stop-servers [--all|--tservers| --no-tservers|--sservers
[group]|--compactors [queue]]
- Starts various server types, can optionally
specify a queue or group
- start-tservers Deprecated. Starts all tservers on cluster
- stop-tservers Deprecated. Stops all tservers on cluster
- start-here Starts all services on this node
- stop-here Stops all services on this node
+$(cyan Usage): $(green accumulo-cluster) <$(yellow command)> [<$(yellow
option)> ...]
+
+$(cyan Options):
+ $(cyan General options):
+ $(green --dry-run) Prints information and commands, but
does not execute them
+ $(green --local) Operates on matching local services only
(no SSH)
+
+ $(cyan 'Service type selection options (if none are used, all service types
are selected)'):
+ $(green --manager) Selects the manager service (oversees
cluster operations)
+ $(green --gc) Selects the gc service (cleans up unused
files)
+ $(green --monitor) Selects the monitor web service (shows
cluster information)
+ $(green --compaction-coordinator) Selects the compaction-coordinator
service (external compactions)
+ $(green --tservers) Selects the tablet server services
(read/write operations on tablets)
+ $(green --sservers)[$(yellow '=group')] Selects the scan server
services (read-only eventually consistent scans)
+ $(green --compactors)[$(yellow '=group')] Selects the compactor
services (external compactions)
+ $(green --no-tservers) $(yellow Deprecated). Selects service
types except tservers (for backwards compatibility)
+ $(yellow NOTE): some server types support an optional $(yellow group)
name. If it is not provided or is empty, then
+ all groups are considered. Declare multiple groups as a single
space-separated parameter. Later
+ options overwrite earlier ones, so $(yellow '--sservers="g1 g2"
--sserver') operates on all sserver groups.
+
+$(cyan Commands):
+ $(green create-config) Creates cluster config (ignores service
selection options)
+ $(green start) Starts Accumulo cluster services
+ $(green stop) Stops Accumulo cluster services
+ $(green restart) Restarts Accumulo cluster services
+ $(green kill) Kills Accumulo cluster services
+
+ $(cyan Deprecated commands):
+ $(green start-non-tservers) $(yellow Deprecated). Alias for "start
--no-tservers"
+ $(green start-servers) $(yellow Deprecated). Alias for "start"
+ $(green stop-servers) $(yellow Deprecated). Alias for "stop"
+ $(green start-tservers) $(yellow Deprecated). Alias for "start
--tservers"
+ $(green stop-tservers) $(yellow Deprecated). Alias for "stop
--tservers"
+ $(green start-here) $(yellow Deprecated). Alias for "start
--local"
+ $(green stop-here) $(yellow Deprecated). Alias for "stop
--local"
+
+$(cyan Examples):
+ $(purple 'accumulo-cluster start') $(blue '#
start all servers')
+ $(purple 'accumulo-cluster start --dry-run') $(blue '#
print debug information and commands to be executed')
+ $(purple 'accumulo-cluster start --local') $(blue '#
start all local services')
+ $(purple 'accumulo-cluster start --local --manager') $(blue '#
start local manager services')
+ $(purple 'accumulo-cluster start --tservers') $(blue '#
start all tservers')
+ $(purple 'accumulo-cluster start --sservers=group1') $(blue '#
start all group1 sservers')
+ $(purple 'accumulo-cluster start --sservers="group1 group2"') $(blue '#
start all group1 and group2 sservers')
+ $(purple 'accumulo-cluster start --local --manager --tservers') $(blue '#
Start the local manager and local tservers')
+
EOF
}
-function invalid_args {
- echo -e "Invalid arguments: $1\n"
+function checkTerminalSupportsColor() {
+ local c
+ # get the number of colors supported
+ c=$(tput colors 2>/dev/null) || c=-1
+ # if STDOUT is a terminal and the number of colors is at least 8
+ [[ -t 1 && $c -ge 8 ]]
+}
+
+function decolorize() {
+ # this only decolorizes escape sequences that we've added
+ # namely, the color codes 0;31m through 0;37m and the 0m reset
+ # it also can't deal with arrays, so can only work on a single param
+ if [[ $# -ne 1 ]]; then
+ echo "$(red ERROR): Couldn't decolorize multiple items: $*"
+ fi
+ local myvar=$1
+ myvar="${myvar//[[:cntrl:]]\[0;3[1-7]m/}" # remove the color codes
+ myvar="${myvar//[[:cntrl:]]\[0m/}" # remove the reset code
+ echo "$myvar"
+}
+function colorize() {
+ local c=$1
+ shift
+ [[ $COLOR_ENABLED == 1 ]] && echo -e "\\e[0;${c}m${*}\\e[0m" || echo "$@"
+}
+function red() { colorize 31 "$@"; }
+function green() { colorize 32 "$@"; }
+function yellow() { colorize 33 "$@"; }
+function blue() { colorize 34 "$@"; }
+function purple() { colorize 35 "$@"; }
+function cyan() { colorize 36 "$@"; }
+function white() { colorize 37 "$@"; }
+
+function parse_args() {
+ local originalArgs=("$@")
+
+ DEBUG=0
+ ARG_LOCAL=0
+ ARG_ALL=1
+ ARG_MANAGER=0
+ ARG_GC=0
+ ARG_MONITOR=0
+ ARG_COORDINATOR=0
+ ARG_TSERVER=0
+ ARG_TSERVER_GROUP=""
+ ARG_SSERVER=0
+ ARG_SSERVER_GROUP=""
+ ARG_COMPACTOR=0
+ ARG_COMPACTOR_GROUP=""
+
+ # the correct version of getopt will always exit with exit code 4 when
provided the `-T` flag
+ getopt -T &>/dev/null
+ if (($? != 4)); then
+ echo "$(red ERROR): Missing $(yellow util-linux) (or equivalent) version
of $(green getopt). Unable to continue."
+ exit 2
+ fi
+
+ if ! PARSE_OUTPUT=$(getopt -o "" --long
"dry-run,local,manager,gc,monitor,compaction-coordinator,no-tservers,tservers,sservers::,compactors::"
-n 'accumulo-cluster' -- "$@"); then
+ print_usage
+ exit 1
+ fi
+
+ eval set -- "$PARSE_OUTPUT"
+
+ while true; do
+ case "$1" in
+ --dry-run)
+ DEBUG=1
+ debug "original args: $(quote "${originalArgs[@]}")"
+ debug "parsed args: $PARSE_OUTPUT"
+ shift 1
+ ;;
+ --local)
+ ARG_LOCAL=1
+ shift 1
+ ;;
+ --manager)
+ ARG_ALL=0
+ ARG_MANAGER=1
+ shift 1
+ ;;
+ --gc)
+ ARG_ALL=0
+ ARG_GC=1
+ shift 1
+ ;;
+ --monitor)
+ ARG_ALL=0
+ ARG_MONITOR=1
+ shift 1
+ ;;
+ --compaction-coordinator)
+ ARG_ALL=0
+ ARG_COORDINATOR=1
+ shift 1
+ ;;
+ --tservers)
+ ARG_ALL=0
+ ARG_TSERVER=1
+ shift 1
+ ;;
+ --sservers)
+ ARG_ALL=0
+ ARG_SSERVER=1
+ ARG_SSERVER_GROUP=$2
+ shift 2
+ ;;
+ --compactors)
+ ARG_ALL=0
+ ARG_COMPACTOR=1
+ ARG_COMPACTOR_GROUP=$2
+ shift 2
+ ;;
+ --no-tservers)
+ echo "$(yellow WARN): The $(yellow "--no-tservers") option is
deprecated. Please specify the servers you wish to manage instead"
+ ARG_ALL=0
+ ARG_MANAGER=1
+ ARG_GC=1
+ ARG_MONITOR=1
+ ARG_COORDINATOR=1
+ ARG_SSERVER=1
+ ARG_COMPACTOR=1
+ shift 1
+ ;;
+ --)
+ shift
+ break
+ ;;
+ *)
+ echo "$(red ERROR): Unhandled option: $(yellow "$1")"
+ print_usage
+ exit 1
+ ;;
+ esac
+ done
+
+ if [[ $# -eq 0 ]]; then
+ invalid_args "<command> cannot be empty"
+ elif [[ $# -ne 1 ]]; then
+ # use getopt to display the remaining non-option parameters quoted for
readability
+ local remaining
+ remaining="$(getopt -o "" -- "$@")"
+ invalid_args "Only one <$(yellow command)> is allowed, but found:$(yellow
"${remaining#*--}")"
+ fi
+ ARG_CMD=$1
+ debug "ARG_CMD=$ARG_CMD"
+ debug "ARG_ALL=$ARG_ALL"
+ debug "ARG_LOCAL=$ARG_LOCAL"
+ debug "ARG_MANAGER=$ARG_MANAGER"
+ debug "ARG_GC=$ARG_GC"
+ debug "ARG_MONITOR=$ARG_MONITOR"
+ debug "ARG_COORDINATOR=$ARG_COORDINATOR"
+ debug "ARG_TSERVER=$ARG_TSERVER"
+ debug "ARG_TSERVER_GROUP=$ARG_TSERVER_GROUP"
+ debug "ARG_SSERVER=$ARG_SSERVER"
+ debug "ARG_SSERVER_GROUP=$ARG_SSERVER_GROUP"
+ debug "ARG_COMPACTOR=$ARG_COMPACTOR"
+ debug "ARG_COMPACTOR_GROUP=$ARG_COMPACTOR_GROUP"
+
+}
+
+function invalid_args() {
+ echo "$(red ERROR): $(yellow invalid arguments): $*"
+ echo
print_usage 1>&2
exit 1
}
-function parse_fail {
- echo "Failed to parse ${conf}/cluster.yaml"
+function parse_fail() {
+ echo "$(red ERROR): Failed to parse $(yellow "$conf/cluster.yaml")"
exit 1
}
-isDebug() {
+function isDebug() {
[[ $DEBUG == 1 ]]
}
-debug() {
- isDebug && echo "${@@P}"
+# if debug is on, print and return true; otherwise, return false
+function debug() {
+ isDebug && echo "$(blue DEBUG): $(cyan "$*")"
}
-debugAndRun() {
- debug "$@"
- if ! isDebug; then
- # shellcheck disable=SC2294
- eval "${@@P}"
- fi
+function quote() {
+ # use getopt to quote, because it uses quotes, rather than escaping spaces,
and that's easier to
+ # read; POSIXLY_CORRECT makes it ignore unrecognized options, so it still
quotes things that start
+ # with '-', but it only does this after the first non-option argument, so we
provide an empty
+ # string to trick it, and then we remove it in the regex that removes " --
'' " from the front of
+ # the parsed output, which is now quoted
+ local quoted
+ quoted=$(POSIXLY_CORRECT=1 getopt -o "" -- '' "$@") &&
+ [[ $quoted =~ ^[[:space:]]*--[[:space:]]*\'\'[[:space:]]*(.*)$ ]] && echo
"${BASH_REMATCH[1]}" &&
+ return
+ echo "$(red ERROR): $(yellow internal script error): unable to quote:
$(yellow "$*")"
+ exit 1
}
-function parse_config {
-
- for file in slaves tservers monitor tracers gc managers masters; do
- if [[ -f ${conf}/${file} ]]; then
- echo "ERROR: A '${file}' file was found in ${conf}/"
- echo "Accumulo now uses cluster host configuration information from
'cluster.yaml' and requires that the '${file}' file not be present to reduce
confusion."
- exit 1
- fi
- done
-
- if [[ ! -f ${conf}/cluster.yaml ]]; then
- echo "ERROR: A 'cluster.yaml' file was not found at ${conf}/cluster.yaml"
- echo "Please make sure it exists and is configured with the host
information. Run 'accumulo-cluster create-config' to create an example
configuration."
- exit 1
- fi
+# call debug to print, or execute if debug is off
+function debugOrRun() {
+ debug "$(quote "$@")" || "$@"
+}
- trap 'rm -f "$CONFIG_FILE"' EXIT
- CONFIG_FILE=$(mktemp) || exit 1
- ${accumulo_cmd} org.apache.accumulo.core.conf.cluster.ClusterConfigParser
"${conf}"/cluster.yaml "$CONFIG_FILE" || parse_fail
- #shellcheck source=/dev/null
- . "$CONFIG_FILE"
- debug "Parsed config:" && cat "$CONFIG_FILE"
- rm -f "$CONFIG_FILE"
+function canRunOnHost() {
- if [[ -z $MANAGER_HOSTS ]]; then
- echo "ERROR: managers not found in ${conf}/cluster.yaml"
- exit 1
+ # always true when non-local
+ if [[ $ARG_LOCAL == 0 ]]; then
+ return 0
fi
- if [[ -z $TSERVER_HOSTS ]]; then
- echo "ERROR: tservers not found in ${conf}/cluster.yaml"
+ if [[ -z $1 ]]; then
+ echo "$(red ERROR): Host argument expected but missing"
exit 1
fi
- if [[ -n $COMPACTION_QUEUES && -z $COORDINATOR_HOSTS ]]; then
- echo "WARN: External compaction queues configured, but no coordinator
configured"
- fi
-
- for queue in $COMPACTION_QUEUES; do
- Q="COMPACTOR_HOSTS_${queue}"
- if [[ -z ${!Q} ]]; then
- echo "WARN: External compaction queue $queue configured, but no
compactors configured for it"
+ local found=0
+ local addr
+ for addr in "${LOCAL_HOST_ADDRESSES[@]}"; do
+ if [[ $1 == "$addr" ]]; then
+ found=1
+ break
fi
done
+ [[ $found == 1 ]]
+}
- for group in $SSERVER_GROUPS; do
- G="SSERVER_HOSTS_${group}"
- if [[ -z ${!G} ]]; then
- echo "WARN: ScanServer group $group configured, but no hosts configured
for it"
+function parse_config() {
+
+ for file in slaves tservers monitor tracers gc managers masters; do
+ if [[ -f $conf/$file ]]; then
+ echo "$(red ERROR): A $(yellow "$file") file was found in $(yellow
"$conf/")"
+ echo "$(red ERROR): Accumulo now uses cluster host configuration
information from $(yellow cluster.yaml)"
+ echo "$(red ERROR): and requires that the $(yellow "$file") file not be
present to reduce confusion."
+ exit 1
fi
done
- unset manager1
- manager1=$(echo "${MANAGER_HOSTS}" | cut -d" " -f1)
+ local manager1
+ local tservers_found
+ local group
+ local G
- if [[ -z $MONITOR_HOSTS ]]; then
- echo "WARN: monitors not found in ${conf}/cluster.yaml, using first
manager host $manager1"
- MONITOR_HOSTS=$manager1
+ if [[ ! -f $conf/cluster.yaml ]]; then
+ echo "$(red ERROR): File not found $(yellow "$conf/cluster.yaml")"
+ echo "$(red ERROR): Please make sure it exists and is configured with the
host information."
+ echo "$(red ERROR): Run $(yellow accumulo-cluster create-config) to create
an example configuration."
+ exit 1
fi
- if [[ -z $GC_HOSTS ]]; then
- echo "WARN: gc not found in ${conf}/cluster.yaml, using first manager host
$manager1"
- GC_HOSTS=$manager1
+ trap 'rm -f "$CONFIG_FILE"' EXIT
+ CONFIG_FILE=$(mktemp --tmpdir "ClusterConfigParser-XXXXXXXX.out") || exit 1
+ "$accumulo_cmd" org.apache.accumulo.core.conf.cluster.ClusterConfigParser
"$conf/cluster.yaml" "$CONFIG_FILE" || parse_fail
+ #shellcheck source=/dev/null
+ . "$CONFIG_FILE"
+ debug "Parsed config from $(white "$conf/cluster.yaml")"
+ local line
+ if isDebug; then
+ while read -r line; do
+ debug "$(white "$line")"
+ done <"$CONFIG_FILE"
fi
+ rm -f "$CONFIG_FILE"
- if [[ -z $NUM_TSERVERS ]]; then
- echo "INFO: ${NUM_TSERVERS} tservers will be started per host"
+ # this might not be possible, since the ClusterConfigParser would probably
fail instead
+ if [[ -z $MANAGER_HOSTS ]]; then
+ echo "$(red ERROR): $(yellow managers) not found in $(yellow
"$conf/cluster.yaml")"
+ exit 1
fi
+ # This version doesn't support configurable tserver groups, so
+ # use a default group to allow code reuse with newer branches
+ # Shellcheck thinks these aren't used, but they are referenced
+ # indirectly by group name.
+ TSERVER_GROUPS="default"
+ #shellcheck disable=SC2034
+ TSERVERS_PER_HOST_default=${NUM_TSERVERS:-1}
+ #shellcheck disable=SC2034
+ TSERVER_HOSTS_default=$TSERVER_HOSTS
+
+ # Rename variables from this version's config parser to ones that
+ # this script expects, to support code reuse with newer branches
+ COMPACTOR_GROUPS=$COMPACTION_QUEUES
+ local var_name
for group in $SSERVER_GROUPS; do
- var_name="NUM_SSERVERS_${group}"
+ var_name="NUM_SSERVERS_$group"
if [[ -n ${!var_name} ]]; then
- echo "INFO: ${!var_name} scan servers will be started per host for group
${group}"
+ declare "SSERVERS_PER_HOST_$group"="${!var_name}"
+ else
+ declare "SSERVERS_PER_HOST_$group"="${NUM_SSERVERS:-1}"
fi
done
-
- for queue in $COMPACTION_QUEUES; do
- var_name="NUM_COMPACTORS_${queue}"
+ for group in $COMPACTOR_GROUPS; do
+ var_name="NUM_COMPACTORS_$group"
if [[ -n ${!var_name} ]]; then
- echo "INFO: ${!var_name} compactors will be started per host for queue
${queue}"
- fi
- done
-}
-
-function control_service() {
- control_cmd="$1"
- host="$2"
- service="$3"
-
- local last_instance_id
- last_instance_id=1
- [[ $service == "tserver" ]] && last_instance_id=${NUM_TSERVERS:-1}
- [[ $service == "sserver" ]] && last_instance_id=${NUM_SSERVERS:-1}
- [[ $service == "compactor" ]] && last_instance_id=${NUM_COMPACTORS:-1}
-
- for ((inst_id = 1; inst_id <= last_instance_id; inst_id++)); do
- # Only increment the service name when more than one service is desired.
- ACCUMULO_SERVICE_INSTANCE=""
- [[ last_instance_id -gt 1 ]] && ACCUMULO_SERVICE_INSTANCE="${inst_id}"
- [[ $service =~ ^compactor|sserver$ ]] &&
ACCUMULO_SERVICE_INSTANCE="${ACCUMULO_SERVICE_INSTANCE}_${5}"
-
- if [[ $host == localhost || $host == "$(hostname -s)" || $host ==
"$(hostname -f)" || "$(hostname -I)" =~ $host ]]; then
- #
- # The server processes take arguments (e.g. -p, -o, -q [in the case of
the Compactor]). Always add the -a argument
- # using the value of $host
- #
- if [[ $# -gt 3 ]]; then
- debugAndRun ACCUMULO_SERVICE_INSTANCE="${ACCUMULO_SERVICE_INSTANCE}"
"${bin}/accumulo-service" "$service" "$control_cmd" "-a" "$host" "${@:4}"
- else
- debugAndRun ACCUMULO_SERVICE_INSTANCE="${ACCUMULO_SERVICE_INSTANCE}"
"${bin}/accumulo-service" "$service" "$control_cmd" "-a" "$host"
- fi
+ declare "COMPACTORS_PER_HOST_$group"="${!var_name}"
else
- if [[ $# -gt 3 ]]; then
- EXTRA_ARGS="${*:4}"
- debugAndRun "$SSH" "$host" "bash -c
'ACCUMULO_SERVICE_INSTANCE=${ACCUMULO_SERVICE_INSTANCE} ${bin}/accumulo-service
\"$service\" \"$control_cmd\" \"-a\" \"$host\" $EXTRA_ARGS '"
- else
- debugAndRun "$SSH" "$host" "bash -c
'ACCUMULO_SERVICE_INSTANCE=${ACCUMULO_SERVICE_INSTANCE} ${bin}/accumulo-service
\"$service\" \"$control_cmd\" \"-a\" \"$host\"'"
- fi
+ declare "COMPACTORS_PER_HOST_$group"="${NUM_COMPACTORS:-1}"
fi
done
-}
-function start_service() {
- control_service start "$@"
-}
+ # This version requires a compaction coordinator for compactors
+ if [[ -n $COMPACTOR_GROUPS && -z $COORDINATOR_HOSTS ]]; then
+ echo "$(yellow WARN): External compactor group(s) configured, but no
coordinator configured"
+ fi
-function start_compactors() {
- if [[ -z $COMPACTION_QUEUES ]]; then
- echo "No compactor queues configured..."
+ if [[ -z $COMPACTOR_GROUPS ]]; then
+ # no compactor groups are required for this version
+ debug "No compactor groups configured"
else
- echo -n "Starting compactor servers ..."
- queues=$COMPACTION_QUEUES
- if [[ -n $1 ]]; then
- queues="$1"
- echo "Only starting servers for group: ${queues}"
- fi
- for queue in $queues; do
- var_name="NUM_COMPACTORS_${queue}"
- [[ -n ${!var_name} ]] && NUM_COMPACTORS=${!var_name}
- Q="COMPACTOR_HOSTS_${queue}"
- if [[ -n ${!Q} ]]; then
- for compactor in ${!Q}; do
- start_service "$compactor" compactor "-q" "$queue"
- done
- else
- echo "${queue} is not a valid queue ...exiting"
+ for group in $COMPACTOR_GROUPS; do
+ G="COMPACTOR_HOSTS_$group"
+ if [[ -z ${!G} ]]; then
+ echo "$(yellow WARN): External compactor group $(yellow "$group")
configured, but no compactors configured for it"
fi
done
fi
-}
-function stop_compactors() {
- if [[ -z $COMPACTION_QUEUES ]]; then
- echo "No compactor queues configured..."
+ tservers_found="false"
+ if [[ -z $TSERVER_GROUPS ]]; then
+ echo "$(yellow WARN): No tablet server groups configured"
else
- echo "Stopping compactor servers ..."
- queues=$COMPACTION_QUEUES
- if [[ -n $1 ]]; then
- queues="$1"
- echo "Only stopping servers for group: ${queues}"
- fi
- for queue in $queues; do
- var_name="NUM_COMPACTORS_${queue}"
- [[ -n ${!var_name} ]] && NUM_COMPACTORS=${!var_name}
- Q="COMPACTOR_HOSTS_${queue}"
- if [[ -n ${!Q} ]]; then
- for compactor in ${!Q}; do
- stop_service "$compactor" compactor "-q" "$queue"
- done
+ for group in $TSERVER_GROUPS; do
+ G="TSERVER_HOSTS_$group"
+ if [[ -z ${!G} ]]; then
+ echo "$(yellow WARN): tablet server group $(yellow "$group")
configured, but no hosts configured for it"
else
- echo "${queue} is not a valid compaction queue ...exiting"
+ tservers_found="true"
fi
done
fi
-}
-function start_sservers() {
- if [[ -z $SSERVER_GROUPS ]]; then
- echo "No scan server groups configured..."
- else
- echo "Starting scan servers ..."
- groups=$SSERVER_GROUPS
- if [[ -n $1 ]]; then
- groups="$1"
- echo "Only starting servers for group: ${groups}"
- fi
- for group in $groups; do
- var_name="NUM_SSERVERS_${group}"
- [[ -n ${!var_name} ]] && NUM_SSERVERS=${!var_name}
- G="SSERVER_HOSTS_${group}"
- if [[ -n ${!G} ]]; then
- for sserver in ${!G}; do
- start_service "$sserver" sserver "-g" "$group"
- done
- else
- echo "${group} is not a valid resource group ...exiting"
- fi
- done
+ if [[ $tservers_found != "true" ]]; then
+ echo "$(red ERROR): There are no tablet servers configured, Accumulo
requires at least $(yellow 1) tablets server to host system tables"
+ exit 1
fi
-}
-function stop_sservers() {
- if [[ -z $SSERVER_GROUPS ]]; then
- echo "No scan server groups configured..."
- else
- echo "Stopping scan servers ..."
- groups=$SSERVER_GROUPS
- if [[ -n $1 ]]; then
- groups="$1"
- echo "Only stopping servers for group: ${groups}"
- fi
- for group in $groups; do
- var_name="NUM_SSERVERS_${group}"
- [[ -n ${!var_name} ]] && NUM_SSERVERS=${!var_name}
- G="SSERVER_HOSTS_${group}"
- if [[ -n ${!G} ]]; then
- for sserver in ${!G}; do
- stop_service "$sserver" sserver "-g" "$group"
- done
- else
- echo "${group} is not a valid resource group ...exiting"
+ if [[ -n $SSERVER_GROUPS ]]; then
+ for group in $SSERVER_GROUPS; do
+ G="SSERVER_HOSTS_$group"
+ if [[ -z ${!G} ]]; then
+ echo "$(yellow WARN): scan server group $(yellow "$group") configured,
but no hosts configured for it"
fi
done
fi
-}
-function start_tservers() {
- echo -n "Starting tablet servers ..."
- count=1
- for server in $TSERVER_HOSTS; do
- echo -n "."
- start_service "$server" tserver &
- if ((++count % 72 == 0)); then
- echo
- wait
- fi
- done
- echo " done"
-}
-
-function start_all() {
- unset DISPLAY
+ manager1=$(echo "$MANAGER_HOSTS" | cut -d" " -f1)
- if [[ $1 != "--no-tservers" ]]; then
- start_tservers
+ if [[ -z $MONITOR_HOSTS ]]; then
+ echo "$(yellow WARN): monitors not found in $(yellow
"$conf/cluster.yaml"), using first manager host $(green "$manager1")"
+ MONITOR_HOSTS=$manager1
fi
- for manager in $MANAGER_HOSTS; do
- start_service "$manager" manager
- done
+ if [[ -z $GC_HOSTS ]]; then
+ echo "$(yellow WARN): gc not found in $(yellow "$conf/cluster.yaml"),
using first manager host $(green "$manager1")"
+ GC_HOSTS=$manager1
+ fi
- for gc in $GC_HOSTS; do
- start_service "$gc" gc
- done
+}
- for monitor in $MONITOR_HOSTS; do
- start_service "$monitor" monitor
- done
+function execute_command() {
+ control_cmd=$1
+ host=$2
+ service=$3
+ group=$4
+ shift 4
- for group in $SSERVER_GROUPS; do
- var_name="NUM_SSERVERS_${group}"
- [[ -n ${!var_name} ]] && NUM_SSERVERS=${!var_name}
- G="SSERVER_HOSTS_${group}"
- for sserver in ${!G}; do
- start_service "$sserver" sserver "-g" "$group"
- done
- done
+ local S
+ local servers_per_host
- for coordinator in $COORDINATOR_HOSTS; do
- start_service "$coordinator" compaction-coordinator
- done
+ S="${service^^}S_PER_HOST_$group"
+ S="${S//-/_}" # replace dashes in service/group name with underscores
+ servers_per_host="${!S:-1}"
- for queue in $COMPACTION_QUEUES; do
- var_name="NUM_COMPACTORS_${queue}"
- [[ -n ${!var_name} ]] && NUM_COMPACTORS=${!var_name}
- Q="COMPACTOR_HOSTS_${queue}"
- for compactor in ${!Q}; do
- start_service "$compactor" compactor "-q" "$queue"
- done
- done
+ if [[ $ARG_LOCAL == 1 ]]; then
+ debugOrRun bash -c "ACCUMULO_CLUSTER_ARG=$servers_per_host
\"$bin/accumulo-service\" $service $control_cmd -a $host $*"
+ else
+ debugOrRun "${SSH[@]}" "$host" "bash -c
'ACCUMULO_CLUSTER_ARG=$servers_per_host \"$bin/accumulo-service\" $service
$control_cmd -a $host $*'"
+ fi
+}
+function get_localhost_addresses() {
+ local localaddresses
+ local localinterfaces
+ local x
+ if [[ -n $ACCUMULO_LOCALHOST_ADDRESSES ]]; then
+ read -r -a localaddresses <<<"$ACCUMULO_LOCALHOST_ADDRESSES"
+ else
+ read -r -a localinterfaces <<<"$(hostname -I)"
+ read -r -a localaddresses <<<"$(getent hosts 127.0.0.1 ::1
"${localinterfaces[@]}" | paste -sd' ')"
+ fi
+ for x in "${localaddresses[@]}"; do echo "$x"; done | sort -u
}
-function start_here() {
+function control_services() {
+ unset DISPLAY
+ local operation=$1
- local_hosts="$(hostname -a 2>/dev/null) $(hostname) localhost 127.0.0.1
$(hostname -I)"
+ if [[ $operation != "start" && $operation != "stop" && $operation != "kill"
]]; then
+ echo "$(red ERROR): Invalid operation: $(yellow "$operation")"
+ exit 1
+ fi
- for host in $local_hosts; do
- for tserver in $TSERVER_HOSTS; do
- if echo "$tserver" | grep -q "^${host}\$"; then
- start_service "$host" tserver
- break
+ local tserver_groups
+ local addr
+ local group
+ local tserver
+ local G
+ if [[ $ARG_ALL == 1 && $ARG_LOCAL == 0 && $operation == "stop" ]]; then
+ echo "Stopping Accumulo cluster..."
+ if ! isDebug; then
+ if ! "$accumulo_cmd" admin stopAll; then
+ echo "Invalid password or unable to connect to the manager"
+ echo "Initiating forced shutdown in 15 seconds (Ctrl-C to abort)"
+ sleep 10
+ echo "Initiating forced shutdown in 5 seconds (Ctrl-C to abort)"
+ else
+ echo "Accumulo shut down cleanly"
+ echo "Utilities and unresponsive servers will shut down in 5 seconds
(Ctrl-C to abort)"
fi
+ sleep 5
+ fi
+ elif [[ $ARG_LOCAL == 1 && $ARG_TSERVER == 1 && $operation == "stop" ]]; then
+ tserver_groups=$TSERVER_GROUPS
+ if [[ -n $ARG_TSERVER_GROUP ]]; then
+ tserver_groups=$ARG_TSERVER_GROUP
+ fi
+ for addr in "${LOCAL_HOST_ADDRESSES[@]}"; do
+ for group in $tserver_groups; do
+ G="TSERVER_HOSTS_$group"
+ for tserver in ${!G}; do
+ if echo "$tserver" | grep -q "$addr"; then
+ if ! isDebug; then
+ "$accumulo_cmd" admin stop "$addr"
+ else
+ debug "Stopping tservers on $addr via admin command"
+ fi
+ fi
+ done
+ done
done
- done
+ elif [[ $ARG_ALL == 1 && $operation == "kill" ]]; then
+ echo "Killing Accumulo cluster..."
+ fi
- for host in $local_hosts; do
+ local count
+ local hosts
+ if [[ $ARG_ALL == 1 || $ARG_TSERVER == 1 ]]; then
+ tserver_groups=$TSERVER_GROUPS
+ if [[ -n $ARG_TSERVER_GROUP ]]; then
+ tserver_groups=$ARG_TSERVER_GROUP
+ fi
+ for group in $tserver_groups; do
+ local msg
+ local msgNoColor
+ msg="Executing $(green "$ARG_CMD") on $(purple tablet servers) for group
$(yellow "$group") ..."
+ msgNoColor=$(decolorize "$msg")
+ count=${#msgNoColor}
+ ((count > 71)) && count=69 # only print up to 3 more dots if the line is
too long
+ echo -n "$msg"
+ hosts="TSERVER_HOSTS_$group"
+ for tserver in ${!hosts}; do
+ if canRunOnHost "$tserver"; then
+ echo -n "."
+ execute_command "$operation" "$tserver" tserver "$group"
+ if ((++count % 72 == 0)); then
+ echo
+ wait
+ fi
+ fi
+ done
+ done
+ echo "done"
+ fi
+
+ local manager
+ if [[ $ARG_ALL == 1 || $ARG_MANAGER == 1 ]]; then
for manager in $MANAGER_HOSTS; do
- if echo "$manager" | grep -q "^${host}\$"; then
- start_service "$host" manager
- break
+ if canRunOnHost "$manager"; then
+ echo "Executing $(green "$ARG_CMD") on $(purple managers)"
+ execute_command "$operation" "$manager" manager "default"
fi
done
- done
+ fi
- for host in $local_hosts; do
+ local gc
+ if [[ $ARG_ALL == 1 || $ARG_GC == 1 ]]; then
for gc in $GC_HOSTS; do
- if echo "$gc" | grep -q "^${host}\$"; then
- start_service "$host" gc
- break
+ if canRunOnHost "$gc"; then
+ echo "Executing $(green "$ARG_CMD") on $(purple garbage collectors)"
+ execute_command "$operation" "$gc" gc "default"
fi
done
- done
+ fi
- for host in $local_hosts; do
+ local monitor
+ if [[ $ARG_ALL == 1 || $ARG_MONITOR == 1 ]]; then
for monitor in $MONITOR_HOSTS; do
- if echo "$monitor" | grep -q "^${host}\$"; then
- start_service "$host" monitor
- break
+ if canRunOnHost "$monitor"; then
+ echo "Executing $(green "$ARG_CMD") on $(purple monitors)"
+ execute_command "$operation" "$monitor" monitor "default"
fi
done
- done
-
- for group in $SSERVER_GROUPS; do
- var_name="NUM_SSERVERS_${group}"
- [[ -n ${!var_name} ]] && NUM_SSERVERS=${!var_name}
- for host in $local_hosts; do
- G="SSERVER_HOSTS_${group}"
- for sserver in ${!G}; do
- if echo "$sserver" | grep -q "^${host}\$"; then
- start_service "$sserver" sserver "-g" "$group"
- fi
- done
- done
- done
+ fi
- for host in $local_hosts; do
+ local coordinator
+ if [[ $ARG_ALL == 1 || $ARG_COORDINATOR == 1 ]]; then
for coordinator in $COORDINATOR_HOSTS; do
- if echo "$coordinator" | grep -q "^${host}\$"; then
- start_service "$coordinator" compaction-coordinator
+ if canRunOnHost "$coordinator"; then
+ echo "Executing $(green "$ARG_CMD") on $(purple compaction
coordinators)"
+ execute_command "$operation" "$coordinator" compaction-coordinator
"default"
fi
done
- done
+ fi
- for queue in $COMPACTION_QUEUES; do
- for host in $local_hosts; do
- var_name="NUM_COMPACTORS_${queue}"
- [[ -n ${!var_name} ]] && NUM_COMPACTORS=${!var_name}
- Q="COMPACTOR_HOSTS_${queue}"
- for compactor in ${!Q}; do
- if echo "$compactor" | grep -q "^${host}\$"; then
- start_service "$compactor" compactor "-q" "$queue"
+ local sserver_groups
+ local sserver
+ if [[ $ARG_ALL == 1 || $ARG_SSERVER == 1 ]]; then
+ sserver_groups=$SSERVER_GROUPS
+ if [[ -n $ARG_SSERVER_GROUP ]]; then
+ sserver_groups=$ARG_SSERVER_GROUP
+ fi
+ for group in $sserver_groups; do
+ echo "Executing $(green "$ARG_CMD") on $(purple scan servers) for group
$(yellow "$group")"
+ hosts="SSERVER_HOSTS_$group"
+ for sserver in ${!hosts}; do
+ if canRunOnHost "$sserver"; then
+ execute_command "$operation" "$sserver" sserver "$group" "-g"
"$group"
fi
done
done
- done
-
-}
-
-function end_service() {
- control_service "$@"
-}
-
-function stop_service() {
- end_service "stop" "$@"
-}
-
-function kill_service() {
- end_service "kill" "$@"
-}
-
-function stop_tservers() {
-
- echo "Stopping unresponsive tablet servers (if any)..."
- for host in $TSERVER_HOSTS; do
- stop_service "$host" tserver &
- done
-
- sleep 10
-
- echo "Stopping unresponsive tablet servers hard (if any)..."
- for host in $TSERVER_HOSTS; do
- kill_service "$host" tserver &
- done
-
- echo "Cleaning tablet server entries from zookeeper"
- if ! isDebug; then
- ${accumulo_cmd} org.apache.accumulo.server.util.ZooZap -tservers
- fi
-}
-
-function kill_all() {
- echo "Killing Accumulo cluster..."
-
- for manager in $MANAGER_HOSTS; do
- kill_service "$manager" manager
- done
-
- for gc in $GC_HOSTS; do
- kill_service "$gc" gc
- done
-
- for monitor in $MONITOR_HOSTS; do
- kill_service "$monitor" monitor
- done
-
- for group in $SSERVER_GROUPS; do
- var_name="NUM_SSERVERS_${group}"
- [[ -n ${!var_name} ]] && NUM_SSERVERS=${!var_name}
- G="SSERVER_HOSTS_${group}"
- for sserver in ${!G}; do
- kill_service "$sserver" sserver "-g" "$group"
- done
- done
-
- for host in $TSERVER_HOSTS; do
- kill_service "$host" tserver
- done
-
- for coordinator in $COORDINATOR_HOSTS; do
- kill_service "$coordinator" compaction-coordinator
- done
-
- for queue in $COMPACTION_QUEUES; do
- var_name="NUM_COMPACTORS_${queue}"
- [[ -n ${!var_name} ]] && NUM_COMPACTORS=${!var_name}
- Q="COMPACTOR_HOSTS_${queue}"
- for compactor in ${!Q}; do
- kill_service "$compactor" compactor "-q" "$queue"
- done
- done
-
- echo "Cleaning all server entries in ZooKeeper"
- if ! isDebug; then
- ${accumulo_cmd} org.apache.accumulo.server.util.ZooZap -manager -tservers
-compaction-coordinators -compactors -sservers
fi
-}
-function stop_all() {
- echo "Stopping Accumulo cluster..."
- if ! isDebug; then
- if ! ${accumulo_cmd} admin stopAll; then
- echo "Invalid password or unable to connect to the manager"
- echo "Initiating forced shutdown in 15 seconds (Ctrl-C to abort)"
- sleep 10
- echo "Initiating forced shutdown in 5 seconds (Ctrl-C to abort)"
- else
- echo "Accumulo shut down cleanly"
- echo "Utilities and unresponsive servers will shut down in 5 seconds
(Ctrl-C to abort)"
+ local compactor_groups
+ local compactor
+ if [[ $ARG_ALL == 1 || $ARG_COMPACTOR == 1 ]]; then
+ compactor_groups=$COMPACTOR_GROUPS
+ if [[ -n $ARG_COMPACTOR_GROUP ]]; then
+ compactor_groups=$ARG_COMPACTOR_GROUP
fi
- sleep 5
- fi
-
- # Look for processes not killed by 'admin stopAll'
-
- for end_cmd in "stop" "kill"; do
-
- for manager in $MANAGER_HOSTS; do
- end_service $end_cmd "$manager" manager
- done
-
- for gc in $GC_HOSTS; do
- end_service $end_cmd "$gc" gc
- done
-
- for monitor in $MONITOR_HOSTS; do
- end_service $end_cmd "$monitor" monitor
- done
-
- for group in $SSERVER_GROUPS; do
- var_name="NUM_SSERVERS_${group}"
- [[ -n ${!var_name} ]] && NUM_SSERVERS=${!var_name}
- G="SSERVER_HOSTS_${group}"
- for sserver in ${!G}; do
- end_service $end_cmd "$sserver" sserver "-g" "$group"
- done
- done
-
- for coordinator in $COORDINATOR_HOSTS; do
- end_service $end_cmd "$coordinator" compaction-coordinator
- done
-
- for queue in $COMPACTION_QUEUES; do
- var_name="NUM_COMPACTORS_${queue}"
- [[ -n ${!var_name} ]] && NUM_COMPACTORS=${!var_name}
- Q="COMPACTOR_HOSTS_${queue}"
- for compactor in ${!Q}; do
- end_service $end_cmd "$compactor" compactor "-q" "$queue"
+ for group in $compactor_groups; do
+ echo "Executing $(green "$ARG_CMD") on $(purple compactors) for group
$(yellow "$group")"
+ hosts="COMPACTOR_HOSTS_$group"
+ for compactor in ${!hosts}; do
+ if canRunOnHost "$compactor"; then
+ execute_command "$operation" "$compactor" compactor "$group" "-q"
"$group"
+ fi
done
done
-
- done
-
- # stop tserver still running
- stop_tservers
-
- echo "Cleaning all server entries in ZooKeeper"
- if ! isDebug; then
- ${accumulo_cmd} org.apache.accumulo.server.util.ZooZap -manager -tservers
-compaction-coordinators -compactors -sservers
fi
-}
-function stop_here() {
- # Determine hostname without errors to user
- hosts_to_check=("$(hostname -a 2>/dev/null | head -1)" "$(hostname -f)")
-
- if echo "${TSERVER_HOSTS}" | grep -Eq 'localhost|127[.]0[.]0[.]1'; then
+ if [[ $ARG_LOCAL == 0 && $ARG_ALL == 1 && ($operation == "stop" ||
$operation == "kill") ]]; then
if ! isDebug; then
- ${accumulo_cmd} admin stop localhost
- else
- debug "Stopping tservers on localhost via admin command"
+ echo "Cleaning all server entries in ZooKeeper"
+ "$accumulo_cmd" org.apache.accumulo.server.util.ZooZap -manager
-tservers -compaction-coordinators -compactors -sservers
fi
- else
- for host in "${hosts_to_check[@]}"; do
- for tserver in $TSERVER_HOSTS; do
- if echo "$tserver" | grep -q "$host"; then
- if ! isDebug; then
- ${accumulo_cmd} admin stop "$host"
- else
- debug "Stopping tservers on $host via admin command"
- fi
- fi
- done
- done
fi
- for host in "${hosts_to_check[@]}"; do
- for end_cmd in "stop" "kill"; do
- for svc in tserver gc manager monitor compaction-coordinator; do
- end_service $end_cmd "$host" $svc
- done
- for group in $SSERVER_GROUPS; do
- var_name="NUM_SSERVERS_${group}"
- [[ -n ${!var_name} ]] && NUM_SSERVERS=${!var_name}
- G="SSERVER_HOSTS_${group}"
- for sserver in ${!G}; do
- end_service $end_cmd "$sserver" sserver "-g" "$group"
- done
- done
- for queue in $COMPACTION_QUEUES; do
- var_name="NUM_COMPACTORS_${queue}"
- [[ -n ${!var_name} ]] && NUM_COMPACTORS=${!var_name}
- Q="COMPACTOR_HOSTS_${queue}"
- for compactor in ${!Q}; do
- end_service $end_cmd "$host" compactor "-q" "$queue"
- done
- done
- done
- done
}
function main() {
- if [[ -z $1 ]]; then
- invalid_args "<command> cannot be empty"
- fi
+ checkTerminalSupportsColor && COLOR_ENABLED=1 || COLOR_ENABLED=0
+ parse_args "$@"
# Resolve base directory
+ local SOURCE
SOURCE="${BASH_SOURCE[0]}"
- while [ -h "${SOURCE}" ]; do
- bin="$(cd -P "$(dirname "${SOURCE}")" && pwd)"
- SOURCE="$(readlink "${SOURCE}")"
- [[ ${SOURCE} != /* ]] && SOURCE="${bin}/${SOURCE}"
+ while [[ -L $SOURCE ]]; do
+ bin="$(cd -P "$(dirname "$SOURCE")" && pwd)"
+ SOURCE="$(readlink "$SOURCE")"
+ [[ $SOURCE != /* ]] && SOURCE="$bin/$SOURCE"
done
- bin="$(cd -P "$(dirname "${SOURCE}")" && pwd)"
- basedir=$(cd -P "${bin}"/.. && pwd)
- conf="${ACCUMULO_CONF_DIR:-${basedir}/conf}"
-
- accumulo_cmd="${bin}/accumulo"
- SSH='ssh -qnf -o ConnectTimeout=2'
+ bin="$(cd -P "$(dirname "$SOURCE")" && pwd)"
+ basedir=$(cd -P "$bin/.." && pwd)
+ conf="${ACCUMULO_CONF_DIR:-$basedir/conf}"
- # Copy input arguments into new array
- # removing any options
- DEBUG=0
- i=0
- declare -a program_args
- for arg in "$@"; do
- if [[ $arg == "--dry-run" ]]; then
- DEBUG=1
- else
- program_args[i++]="$arg"
- fi
- done
+ accumulo_cmd="$bin/accumulo"
+ SSH=('ssh' '-qnf' '-o' 'ConnectTimeout=2')
- debug "debug: ${DEBUG} args: ${program_args[*]}"
+ mapfile -t LOCAL_HOST_ADDRESSES < <(get_localhost_addresses)
+ debug "LOCAL_HOST_ADDRESSES=${LOCAL_HOST_ADDRESSES[*]}"
- case "${program_args[0]}" in
+ case "$ARG_CMD" in
create-config)
if [[ -f "$conf"/cluster.yaml ]]; then
- echo "ERROR : ${conf}/cluster.yaml already exists, not overwriting"
+ echo "ERROR : $conf/cluster.yaml already exists, not overwriting"
exit 1
fi
cat <<EOF >"$conf"/cluster.yaml
@@ -698,103 +699,80 @@ tserver:
# compactors_per_host.
#
tservers_per_host: 1
-#sservers_per_host:
+#sservers_per_host:
# - default: 1
#compactors_per_host:
# - q1: 1
-# - q2: 1
+# - q2: 1
EOF
;;
restart)
parse_config
- stop_all
+ control_services stop
+ control_services kill
# Make sure the JVM has a chance to fully exit
sleep 1
- start_all
+ control_services start
;;
start)
parse_config
- start_all
+ control_services start
;;
stop)
parse_config
- stop_all
+ control_services stop
;;
kill)
parse_config
- kill_all
- ;;
- start-non-tservers)
- echo "$1 is deprecated. Please use \`start-servers --no-tservers\`
instead"
- parse_config
- start_all --no-tservers
- ;;
- start-tservers)
- echo "$1 is deprecated. Please use \`start-servers --tservers\` instead"
- parse_config
- start_tservers
+ control_services kill
;;
start-here)
parse_config
- start_here
+ ARG_ALL=1
+ ARG_LOCAL=1
+ control_services start
;;
- stop-tservers)
- echo "$1 is deprecated. Please use \`stop-servers --tservers\` instead"
+ stop-here)
parse_config
- stop_tservers
+ ARG_ALL=1
+ ARG_LOCAL=1
+ control_services stop
+ control_services kill
;;
- stop-here)
+ start-non-tservers)
+ echo "'$ARG_CMD' is deprecated. Please specify the servers you wish to
start instead"
parse_config
- stop_here
+ ARG_MANAGER=1
+ ARG_GC=1
+ ARG_MONITOR=1
+ ARG_COORDINATOR=1
+ ARG_SSERVER=1
+ ARG_COMPACTOR=1
+ control_services start
;;
start-servers)
+ echo "'$ARG_CMD' is deprecated. Please use 'start' instead"
parse_config
- subcommand="${program_args[1]}"
- case "$subcommand" in
- "--all" | "")
- start_all
- ;;
- "--tservers")
- start_tservers
- ;;
- "--no-tservers")
- start_all --no-tservers
- ;;
- "--sservers")
- start_sservers "${program_args[@]:2}"
- ;;
- "--compactors")
- start_compactors "${program_args[@]:2}"
- ;;
- *)
- invalid_args "'$subcommand' is an invalid <command>"
- ;;
- esac
+ control_services start
;;
stop-servers)
+ echo "'$ARG_CMD' is deprecated. Please use 'stop' instead"
parse_config
- subcommand="${program_args[1]}"
- case "$subcommand" in
- "--all" | "")
- stop_all
- ;;
- "--tservers")
- stop_tservers
- ;;
- "--sservers")
- stop_sservers "${program_args[@]:2}"
- ;;
- "--compactors")
- stop_compactors "${program_args[@]:2}"
- ;;
- *)
- invalid_args "'$subcommand' is an invalid <command>"
- ;;
- esac
+ control_services stop
+ ;;
+ start-tservers)
+ echo "'$ARG_CMD' is deprecated. Please use 'start --tservers' instead"
+ ARG_TSERVER=1
+ control_services start
+ ;;
+ stop-tservers)
+ echo "'$ARG_CMD' is deprecated. Please use 'stop --tservers' instead"
+ ARG_TSERVER=1
+ control_services stop
;;
*)
- invalid_args "${program_args[0]} is an invalid <command>"
+ invalid_args "'$ARG_CMD' is an invalid <command>"
;;
esac
}
diff --git a/assemble/bin/accumulo-service b/assemble/bin/accumulo-service
index 390728fddc..2bc754c212 100755
--- a/assemble/bin/accumulo-service
+++ b/assemble/bin/accumulo-service
@@ -26,7 +26,7 @@ Services:
gc Accumulo garbage collector
monitor Accumulo monitor
manager Accumulo manager
- master Deprecated. Accumulo master
+ master Deprecated. Use 'manager' instead
tserver Accumulo tserver
compaction-coordinator Accumulo compaction coordinator (experimental)
compactor Accumulo compactor (experimental)
@@ -60,31 +60,69 @@ function rotate_log() {
fi
}
+function get_group() {
+ # Find the group parameter if any
+ local group="default"
+ local group_param=""
+ local param
+ for param in "$@"; do
+ if [[ -n $group_param ]]; then
+ # grab the group if the previous arg was the group param
+ group="$param"
+ break
+ elif [[ $param == '-q' || $param == '-g' ]]; then
+ # found the group parameter, the next arg is the group
+ group_param=$param
+ fi
+ done
+ echo "$group"
+}
+
function start_service() {
local service_type=$1
local service_name=$2
shift 2
- local pid_file="${ACCUMULO_PID_DIR}/accumulo-${service_name}.pid"
- if [[ -f $pid_file ]]; then
- pid=$(cat "$pid_file")
- if kill -0 "$pid" 2>/dev/null; then
- echo "$HOST : ${service_name} already running (${pid})"
- exit 0
- fi
+ local build_service_name="false"
+ if [[ -n $service_name ]]; then
+ # if service_name is supplied, then we are only starting one instance
+ servers_per_host=1
+ else
+ build_service_name="true"
+ servers_per_host=${ACCUMULO_CLUSTER_ARG:-1}
fi
- echo "Starting $service_name on $HOST"
- if [[ ${service_type} == "manager" ]]; then
- "${bin}/accumulo" org.apache.accumulo.manager.state.SetGoalState NORMAL
- fi
- outfile="${ACCUMULO_LOG_DIR}/${service_name}_${HOST}.out"
- errfile="${ACCUMULO_LOG_DIR}/${service_name}_${HOST}.err"
- rotate_log "$outfile"
- rotate_log "$errfile"
+ for ((process_num = 1; process_num <= servers_per_host; process_num++)); do
+ if [[ ${build_service_name} == "true" ]]; then
+ service_name="${service_type}_${group}_${process_num}"
+ fi
+ # The ACCUMULO_SERVICE_INSTANCE variable is used in
+ # accumulo-env.sh to set parameters on the command
+ # line.
+ export ACCUMULO_SERVICE_INSTANCE="${service_name}"
- nohup "${bin}/accumulo" "$service_type" "$@" >"$outfile" 2>"$errfile"
</dev/null &
- echo "$!" >"${pid_file}"
+ local pid_file="${ACCUMULO_PID_DIR}/accumulo-${service_name}.pid"
+ if [[ -f $pid_file ]]; then
+ pid=$(cat "$pid_file")
+ if kill -0 "$pid" 2>/dev/null; then
+ echo "$HOST : ${service_name} already running (${pid})"
+ continue
+ fi
+ fi
+ echo "Starting $service_name on $HOST"
+
+ if [[ ${service_type} == "manager" ]]; then
+ "${bin}/accumulo" org.apache.accumulo.manager.state.SetGoalState NORMAL
+ fi
+ outfile="${ACCUMULO_LOG_DIR}/${service_name}_${HOST}.out"
+ errfile="${ACCUMULO_LOG_DIR}/${service_name}_${HOST}.err"
+ rotate_log "$outfile"
+ rotate_log "$errfile"
+
+ nohup "${bin}/accumulo" "$service_type" "$@" >"$outfile" 2>"$errfile"
</dev/null &
+ echo "$!" >"${pid_file}"
+
+ done
# Check the max open files limit and selectively warn
max_files_open=$(ulimit -n)
@@ -127,6 +165,18 @@ function stop_service() {
local pid_file="${ACCUMULO_PID_DIR}/accumulo-${process}.pid"
control_process "TERM" "$process" "$pid_file"
done
+ elif [[ -n $ACCUMULO_CLUSTER_ARG ]]; then
+
+ servers_per_host=${ACCUMULO_CLUSTER_ARG:-1}
+ group=$(get_group "$@")
+
+ for ((process_num = 1; process_num <= servers_per_host; process_num++)); do
+ service_name="${service_type}_${group}_${process_num}"
+ echo "Stopping service process: $service_name"
+ local pid_file="${ACCUMULO_PID_DIR}/accumulo-${service_name}.pid"
+ control_process "TERM" "$service_name" "$pid_file"
+ done
+
else
echo "Stopping service process: $service_name"
local pid_file="${ACCUMULO_PID_DIR}/accumulo-${service_name}.pid"
@@ -144,6 +194,18 @@ function kill_service() {
local pid_file="${ACCUMULO_PID_DIR}/accumulo-${process}.pid"
control_process "KILL" "$process" "$pid_file"
done
+ elif [[ -n $ACCUMULO_CLUSTER_ARG ]]; then
+
+ servers_per_host=${ACCUMULO_CLUSTER_ARG:-1}
+ group=$(get_group "$@")
+
+ for ((process_num = 1; process_num <= servers_per_host; process_num++)); do
+ service_name="${service_type}_${group}_${process_num}"
+ echo "Stopping service process: $service_name"
+ local pid_file="${ACCUMULO_PID_DIR}/accumulo-${service_name}.pid"
+ control_process "KILL" "$service_name" "$pid_file"
+ done
+
else
local pid_file="${ACCUMULO_PID_DIR}/accumulo-${service_name}.pid"
control_process "KILL" "$service_name" "$pid_file"
@@ -182,6 +244,9 @@ function main() {
export conf="${ACCUMULO_CONF_DIR:-${basedir}/conf}"
export lib="${basedir}/lib"
+ group=$(get_group "$@")
+ export ACCUMULO_RESOURCE_GROUP="$group"
+
if [[ -f "${conf}/accumulo-env.sh" ]]; then
#shellcheck source=../conf/accumulo-env.sh
source "${conf}/accumulo-env.sh"
@@ -200,7 +265,7 @@ function main() {
local service_type="$1"
local command_name="$2"
shift 2
- local service_name=$service_type
+ local service_name=""
local all_flag=false
if [[ $service_type == "master" ]]; then
@@ -209,7 +274,7 @@ function main() {
fi
# Check and see if accumulo-cluster is calling this script
- if [[ -z $ACCUMULO_SERVICE_INSTANCE ]]; then
+ if [[ -z $ACCUMULO_CLUSTER_ARG ]]; then
# The rest of the arguments are from a user
if [[ $1 == "--all" ]]; then
all_flag=true
@@ -219,9 +284,6 @@ function main() {
service_name="$1"
fi
fi
- # Use the special bash env var from accumulo-cluster
- else
- service_name=${service_type}${ACCUMULO_SERVICE_INSTANCE}
fi
case "$service_type" in
@@ -234,10 +296,10 @@ function main() {
start_service "$service_type" "$service_name" "$@"
;;
stop)
- stop_service "$service_type" "$service_name" $all_flag
+ stop_service "$service_type" "$service_name" $all_flag "$@"
;;
kill)
- kill_service "$service_type" "$service_name" $all_flag
+ kill_service "$service_type" "$service_name" $all_flag "$@"
;;
list)
list_processes "$service_type"
diff --git a/assemble/conf/accumulo-env.sh b/assemble/conf/accumulo-env.sh
index bcde6907be..dc1799c7fc 100644
--- a/assemble/conf/accumulo-env.sh
+++ b/assemble/conf/accumulo-env.sh
@@ -89,23 +89,33 @@ JAVA_OPTS=(
## JVM options set for individual applications
# cmd is set by calling script that sources this env file
#shellcheck disable=SC2154
-case "$cmd" in
- manager | master) JAVA_OPTS=('-Xmx512m' '-Xms512m' "${JAVA_OPTS[@]}") ;;
- monitor) JAVA_OPTS=('-Xmx256m' '-Xms256m' "${JAVA_OPTS[@]}") ;;
- gc) JAVA_OPTS=('-Xmx256m' '-Xms256m' "${JAVA_OPTS[@]}") ;;
- tserver) JAVA_OPTS=('-Xmx768m' '-Xms768m' "${JAVA_OPTS[@]}") ;;
- compaction-coordinator) JAVA_OPTS=('-Xmx512m' '-Xms512m' "${JAVA_OPTS[@]}")
;;
- compactor) JAVA_OPTS=('-Xmx256m' '-Xms256m' "${JAVA_OPTS[@]}") ;;
- sserver) JAVA_OPTS=('-Xmx512m' '-Xms512m' "${JAVA_OPTS[@]}") ;;
- *) JAVA_OPTS=('-Xmx256m' '-Xms64m' "${JAVA_OPTS[@]}") ;;
+case "${ACCUMULO_RESOURCE_GROUP:-default}" in
+ default)
+ # shellcheck disable=SC2154
+ # $cmd is exported in the accumulo script, but not the accumulo-service
script
+ case "$cmd" in
+ manager | master) JAVA_OPTS=('-Xmx512m' '-Xms512m' "${JAVA_OPTS[@]}") ;;
+ monitor) JAVA_OPTS=('-Xmx256m' '-Xms256m' "${JAVA_OPTS[@]}") ;;
+ gc) JAVA_OPTS=('-Xmx256m' '-Xms256m' "${JAVA_OPTS[@]}") ;;
+ tserver) JAVA_OPTS=('-Xmx768m' '-Xms768m' "${JAVA_OPTS[@]}") ;;
+ compaction-coordinator) JAVA_OPTS=('-Xmx512m' '-Xms512m'
"${JAVA_OPTS[@]}") ;;
+ compactor) JAVA_OPTS=('-Xmx256m' '-Xms256m' "${JAVA_OPTS[@]}") ;;
+ sserver) JAVA_OPTS=('-Xmx512m' '-Xms512m' "${JAVA_OPTS[@]}") ;;
+ *) JAVA_OPTS=('-Xmx256m' '-Xms64m' "${JAVA_OPTS[@]}") ;;
+ esac
+ ;;
+ *)
+ echo "ACCUMULO_RESOURCE_GROUP named $ACCUMULO_RESOURCE_GROUP is not
configured in accumulo-env.sh"
+ exit 1
+ ;;
esac
## JVM options set for logging. Review log4j2.properties file to see how they
are used.
JAVA_OPTS=("-Daccumulo.log.dir=${ACCUMULO_LOG_DIR}"
- "-Daccumulo.application=${cmd}${ACCUMULO_SERVICE_INSTANCE}_$(hostname)"
+ "-Daccumulo.application=${ACCUMULO_SERVICE_INSTANCE}_$(hostname)"
"-Daccumulo.metrics.service.instance=${ACCUMULO_SERVICE_INSTANCE}"
"-Dlog4j2.contextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector"
- "-Dotel.service.name=${cmd}${ACCUMULO_SERVICE_INSTANCE}"
+ "-Dotel.service.name=${ACCUMULO_SERVICE_INSTANCE}"
"${JAVA_OPTS[@]}"
)
diff --git a/assemble/conf/accumulo.properties
b/assemble/conf/accumulo.properties
index 8aea483153..6e6de34cc6 100644
--- a/assemble/conf/accumulo.properties
+++ b/assemble/conf/accumulo.properties
@@ -31,3 +31,8 @@ instance.secret=DEFAULT
## Set to false if 'accumulo-util build-native' fails
tserver.memory.maps.native.enabled=true
+
+## (optional) include additional property files for a resource group
+## based on the ACCUMULO_RESOURCE_GROUP env var set in accumulo-service
+#include=group-${env:ACCUMULO_RESOURCE_GROUP}.properties
+#includeOptional=group-${env:ACCUMULO_RESOURCE_GROUP}.properties
diff --git
a/core/src/main/java/org/apache/accumulo/core/conf/cluster/ClusterConfigParser.java
b/core/src/main/java/org/apache/accumulo/core/conf/cluster/ClusterConfigParser.java
index dcc3d469e2..ff71848d2f 100644
---
a/core/src/main/java/org/apache/accumulo/core/conf/cluster/ClusterConfigParser.java
+++
b/core/src/main/java/org/apache/accumulo/core/conf/cluster/ClusterConfigParser.java
@@ -84,14 +84,12 @@ public class ClusterConfigParser {
(parentKey == null || parentKey.equals("")) ? "" : parentKey +
addTheDot(parentKey);
if (value instanceof String) {
results.put(parent + key, (String) value);
- return;
} else if (value instanceof List) {
((List<?>) value).forEach(l -> {
if (l instanceof String) {
// remove the [] at the ends of toString()
String val = value.toString();
results.put(parent + key, val.substring(1, val.length() -
1).replace(", ", " "));
- return;
} else {
flatten(parent, key, l, results);
}
@@ -102,9 +100,8 @@ public class ClusterConfigParser {
map.forEach((k, v) -> flatten(parent + key, k, v, results));
} else if (value instanceof Number) {
results.put(parent + key, value.toString());
- return;
} else {
- throw new RuntimeException("Unhandled object type: " + value.getClass());
+ throw new IllegalStateException("Unhandled object type: " +
value.getClass());
}
}
@@ -121,7 +118,7 @@ public class ClusterConfigParser {
out.printf(PROPERTY_FORMAT, section.toUpperCase() + "_HOSTS",
config.get(section));
} else {
if (section.equals("manager") || section.equals("tserver")) {
- throw new RuntimeException("Required configuration section is
missing: " + section);
+ throw new IllegalStateException("Required configuration section is
missing: " + section);
}
System.err.println("WARN: " + section + " is missing");
}
