github-actions[bot] commented on PR #21788:
URL: https://github.com/apache/doris/pull/21788#issuecomment-1633697033
#### `sh-checker report`
To get the full details, please check in the
[job]("https://github.com/apache/doris/actions/runs/5540201251";) output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In tools/emr_storage_regression/emr_tools.sh line 1:
set -eo pipefail
^-- SC2148 (error): Tips depend on target shell and yours is unknown. Add a
shebang or a 'shell' directive.
In tools/emr_storage_regression/emr_tools.sh line 69:
shift
^---^ SC2317 (info): Command appears to be unreachable. Check usage
(or ignore if invoked indirectly).
In tools/emr_storage_regression/emr_tools.sh line 83:
echo "$PROFILE"
^------^ SC2250 (style): Prefer putting braces around variable
references even when not strictly required.
Did you mean:
echo "${PROFILE}"
In tools/emr_storage_regression/emr_tools.sh line 84:
echo "$CASE"
^---^ SC2250 (style): Prefer putting braces around variable references
even when not strictly required.
Did you mean:
echo "${CASE}"
In tools/emr_storage_regression/ping_test/ping_poc.sh line 14:
echo "$SERVICE"
^------^ SC2250 (style): Prefer putting braces around variable
references even when not strictly required.
Did you mean:
echo "${SERVICE}"
In tools/emr_storage_regression/stardard_set/gen_spark_create_sql.sh line 4:
sh gen_tbl/gen_ssb_create_sql.sh "${BUCKET}"/ssb/ssb100_parquet
ssb100_parquet_hdfs parquet >> create.sql
^-- SC2129 (style): Consider using { cmd1; cmd2; } >> file instead of
individual redirects.
In
tools/emr_storage_regression/stardard_set/gen_tbl/gen_clickbench_create_sql.sh
line 2:
if [ -z $1 ]; then
^-------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
^-- SC2086 (info): Double quote to prevent globbing and word
splitting.
Did you mean:
if [[ -z "$1" ]]; then
In
tools/emr_storage_regression/stardard_set/gen_tbl/gen_clickbench_create_sql.sh
line 9:
if [ -z $2 ]; then
^-------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
^-- SC2086 (info): Double quote to prevent globbing and word
splitting.
Did you mean:
if [[ -z "$2" ]]; then
In
tools/emr_storage_regression/stardard_set/gen_tbl/gen_clickbench_create_sql.sh
line 18:
USE '"${db}"'_parquet;
^-- SC2016 (info): Expressions don't expand in single quotes,
use double quotes for that.
In
tools/emr_storage_regression/stardard_set/gen_tbl/gen_clickbench_create_sql.sh
line 129:
USE '"${db}"'_orc;
^-- SC2016 (info): Expressions don't expand in single quotes,
use double quotes for that.
In
tools/emr_storage_regression/stardard_set/gen_tbl/gen_clickbench_create_sql.sh
line 241:
USE '"${db}"'_text;
^-- SC2016 (info): Expressions don't expand in single quotes,
use double quotes for that.
In
tools/emr_storage_regression/stardard_set/gen_tbl/gen_clickbench_create_sql.sh
line 353:
USE '"${db}"'_iceberg;
^-- SC2016 (info): Expressions don't expand in single quotes,
use double quotes for that.
In tools/emr_storage_regression/stardard_set/gen_tbl/gen_ssb_create_sql.sh
line 2:
if [ -z $1 ]; then
^-------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
^-- SC2086 (info): Double quote to prevent globbing and word
splitting.
Did you mean:
if [[ -z "$1" ]]; then
In tools/emr_storage_regression/stardard_set/gen_tbl/gen_ssb_create_sql.sh
line 9:
if [ -z $2 ]; then
^-------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
^-- SC2086 (info): Double quote to prevent globbing and word
splitting.
Did you mean:
if [[ -z "$2" ]]; then
In tools/emr_storage_regression/stardard_set/gen_tbl/gen_ssb_create_sql.sh
line 16:
if [ -z $3 ]; then
^-------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
^-- SC2086 (info): Double quote to prevent globbing and word
splitting.
Did you mean:
if [[ -z "$3" ]]; then
In tools/emr_storage_regression/stardard_set/gen_tbl/gen_ssb_create_sql.sh
line 24:
USE '"${db}"';
^-- SC2016 (info): Expressions don't expand in single quotes,
use double quotes for that.
In tools/emr_storage_regression/stardard_set/gen_tbl/gen_ssb_create_sql.sh
line 36:
LOCATION "'"${db_loc}"/customer'";
^-- SC2016 (info): Expressions don't expand
in single quotes, use double quotes for that.
In tools/emr_storage_regression/stardard_set/gen_tbl/gen_ssb_create_sql.sh
line 57:
LOCATION "'"${db_loc}"/dates'";
^-- SC2016 (info): Expressions don't expand in
single quotes, use double quotes for that.
In tools/emr_storage_regression/stardard_set/gen_tbl/gen_ssb_create_sql.sh
line 78:
LOCATION "'"${db_loc}"/lineorder'";
^-- SC2016 (info): Expressions don't expand
in single quotes, use double quotes for that.
In tools/emr_storage_regression/stardard_set/gen_tbl/gen_ssb_create_sql.sh
line 91:
LOCATION "'"${db_loc}"/part'";
^-- SC2016 (info): Expressions don't expand in
single quotes, use double quotes for that.
In tools/emr_storage_regression/stardard_set/gen_tbl/gen_ssb_create_sql.sh
line 102:
LOCATION "'"${db_loc}"/supplier'";
^-- SC2016 (info): Expressions don't expand
in single quotes, use double quotes for that.
In tools/emr_storage_regression/stardard_set/gen_tbl/gen_tpch_create_sql.sh
line 2:
if [ -z $1 ]; then
^-------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
^-- SC2086 (info): Double quote to prevent globbing and word
splitting.
Did you mean:
if [[ -z "$1" ]]; then
In tools/emr_storage_regression/stardard_set/gen_tbl/gen_tpch_create_sql.sh
line 9:
if [ -z $2 ]; then
^-------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
^-- SC2086 (info): Double quote to prevent globbing and word
splitting.
Did you mean:
if [[ -z "$2" ]]; then
In tools/emr_storage_regression/stardard_set/gen_tbl/gen_tpch_create_sql.sh
line 16:
if [ -z $3 ]; then
^-------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
^-- SC2086 (info): Double quote to prevent globbing and word
splitting.
Did you mean:
if [[ -z "$3" ]]; then
In tools/emr_storage_regression/stardard_set/gen_tbl/gen_tpch_create_sql.sh
line 24:
USE '"${db}"';
^-- SC2016 (info): Expressions don't expand in single quotes,
use double quotes for that.
In tools/emr_storage_regression/stardard_set/gen_tbl/gen_tpch_create_sql.sh
line 36:
LOCATION "'"${db_loc}"/customer'";
^-- SC2016 (info): Expressions don't expand
in single quotes, use double quotes for that.
In tools/emr_storage_regression/stardard_set/gen_tbl/gen_tpch_create_sql.sh
line 56:
LOCATION "'"${db_loc}"/lineitem'";
^-- SC2016 (info): Expressions don't expand
in single quotes, use double quotes for that.
In tools/emr_storage_regression/stardard_set/gen_tbl/gen_tpch_create_sql.sh
line 64:
LOCATION "'"${db_loc}"/nation'";
^-- SC2016 (info): Expressions don't expand in
single quotes, use double quotes for that.
In tools/emr_storage_regression/stardard_set/gen_tbl/gen_tpch_create_sql.sh
line 77:
LOCATION "'"${db_loc}"/orders'";
^-- SC2016 (info): Expressions don't expand in
single quotes, use double quotes for that.
In tools/emr_storage_regression/stardard_set/gen_tbl/gen_tpch_create_sql.sh
line 90:
LOCATION "'"${db_loc}"/part'";
^-- SC2016 (info): Expressions don't expand in
single quotes, use double quotes for that.
In tools/emr_storage_regression/stardard_set/gen_tbl/gen_tpch_create_sql.sh
line 99:
LOCATION "'"${db_loc}"/partsupp'";
^-- SC2016 (info): Expressions don't expand
in single quotes, use double quotes for that.
In tools/emr_storage_regression/stardard_set/gen_tbl/gen_tpch_create_sql.sh
line 106:
LOCATION "'"${db_loc}"/region'";
^-- SC2016 (info): Expressions don't expand in
single quotes, use double quotes for that.
In tools/emr_storage_regression/stardard_set/run_queries.sh line 9:
echo $DB
^-^ SC2086 (info): Double quote to prevent globbing and word splitting.
^-^ SC2250 (style): Prefer putting braces around variable references
even when not strictly required.
Did you mean:
echo "${DB}"
In tools/emr_storage_regression/stardard_set/run_queries.sh line 14:
touch $RESULT_FILE
^----------^ SC2086 (info): Double quote to prevent globbing and word
splitting.
^----------^ SC2250 (style): Prefer putting braces around variable
references even when not strictly required.
Did you mean:
touch "${RESULT_FILE}"
In tools/emr_storage_regression/stardard_set/run_queries.sh line 15:
truncate -s 0 $RESULT_FILE
^----------^ SC2086 (info): Double quote to prevent globbing
and word splitting.
^----------^ SC2250 (style): Prefer putting braces around
variable references even when not strictly required.
Did you mean:
truncate -s 0 "${RESULT_FILE}"
In tools/emr_storage_regression/stardard_set/run_queries.sh line 18:
echo -n "query${QUERY_NUM}," | tee -a $RESULT_FILE
^----------^ SC2086 (info): Double
quote to prevent globbing and word splitting.
^----------^ SC2250 (style):
Prefer putting braces around variable references even when not strictly
required.
Did you mean:
echo -n "query${QUERY_NUM}," | tee -a "${RESULT_FILE}"
In tools/emr_storage_regression/stardard_set/run_queries.sh line 19:
for i in $(seq 1 $TRIES); do
^----^ SC2248 (style): Prefer double quoting even when
variables don't contain special characters.
^----^ SC2250 (style): Prefer putting braces around
variable references even when not strictly required.
Did you mean:
for i in $(seq 1 "${TRIES}"); do
In tools/emr_storage_regression/stardard_set/run_queries.sh line 21:
echo -n "$RES" | tee -a $RESULT_FILE
^--^ SC2250 (style): Prefer putting braces around variable
references even when not strictly required.
^----------^ SC2086 (info): Double quote to
prevent globbing and word splitting.
^----------^ SC2250 (style): Prefer putting
braces around variable references even when not strictly required.
Did you mean:
echo -n "${RES}" | tee -a "${RESULT_FILE}"
In tools/emr_storage_regression/stardard_set/run_queries.sh line 22:
[[ "$i" != "$TRIES" ]] && echo -n "," | tee -a $RESULT_FILE
^-- SC2250 (style): Prefer putting braces around variable
references even when not strictly required.
^----^ SC2250 (style): Prefer putting braces around
variable references even when not strictly required.
^----------^ SC2086
(info): Double quote to prevent globbing and word splitting.
^----------^ SC2250
(style): Prefer putting braces around variable references even when not
strictly required.
Did you mean:
[[ "${i}" != "${TRIES}" ]] && echo -n "," | tee -a "${RESULT_FILE}"
In tools/emr_storage_regression/stardard_set/run_queries.sh line 24:
echo "" | tee -a $RESULT_FILE
^----------^ SC2086 (info): Double quote to prevent
globbing and word splitting.
^----------^ SC2250 (style): Prefer putting braces
around variable references even when not strictly required.
Did you mean:
echo "" | tee -a "${RESULT_FILE}"
In tools/emr_storage_regression/stardard_set/run_queries.sh line 27:
done < $5
^-- SC2086 (info): Double quote to prevent globbing and word
splitting.
Did you mean:
done < "$5"
In tools/emr_storage_regression/stardard_set/run_standard_set.sh line 4:
if [ -z "$1" ]; then
^---------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
Did you mean:
if [[ -z "$1" ]]; then
In tools/emr_storage_regression/stardard_set/run_standard_set.sh line 10:
if [ -z "$2" ]; then
^---------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
Did you mean:
if [[ -z "$2" ]]; then
In tools/emr_storage_regression/stardard_set/run_standard_set.sh line 19:
if [ -z "$3" ]; then
^---------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
Did you mean:
if [[ -z "$3" ]]; then
In tools/emr_storage_regression/stardard_set/run_standard_set.sh line 25:
if [ "${case}" = 'ssb' ] ; then
^-------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in
Bash/Ksh.
Did you mean:
if [[ "${case}" = 'ssb' ]] ; then
In tools/emr_storage_regression/stardard_set/run_standard_set.sh line 27:
if [ "${format}" = 'parquet' ]; then
^-------------------------^ SC2292 (style): Prefer [[ ]] over [ ] for
tests in Bash/Ksh.
Did you mean:
if [[ "${format}" = 'parquet' ]]; then
In tools/emr_storage_regression/stardard_set/run_standard_set.sh line 29:
elif [ "${format}" = 'orc' ]; then
^---------------------^ SC2292 (style): Prefer [[ ]] over [ ] for
tests in Bash/Ksh.
Did you mean:
elif [[ "${format}" = 'orc' ]]; then
In tools/emr_storage_regression/stardard_set/run_standard_set.sh line 34:
elif [ "${case}" = 'tpch' ]; then
^--------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests
in Bash/Ksh.
Did you mean:
elif [[ "${case}" = 'tpch' ]]; then
In tools/emr_storage_regression/stardard_set/run_standard_set.sh line 36:
if [ "${format}" = 'parquet' ]; then
^-------------------------^ SC2292 (style): Prefer [[ ]] over [ ] for
tests in Bash/Ksh.
Did you mean:
if [[ "${format}" = 'parquet' ]]; then
In tools/emr_storage_regression/stardard_set/run_standard_set.sh line 38:
elif [ "${format}" = 'orc' ]; then
^---------------------^ SC2292 (style): Prefer [[ ]] over [ ] for
tests in Bash/Ksh.
Did you mean:
elif [[ "${format}" = 'orc' ]]; then
In tools/emr_storage_regression/stardard_set/run_standard_set.sh line 43:
elif [ "${case}" = 'clickbench' ]; then
^--------------------------^ SC2292 (style): Prefer [[ ]] over [ ] for
tests in Bash/Ksh.
Did you mean:
elif [[ "${case}" = 'clickbench' ]]; then
In tools/emr_storage_regression/stardard_set/run_standard_set.sh line 45:
if [ "${format}" = 'parquet' ]; then
^-------------------------^ SC2292 (style): Prefer [[ ]] over [ ] for
tests in Bash/Ksh.
Did you mean:
if [[ "${format}" = 'parquet' ]]; then
In tools/emr_storage_regression/stardard_set/run_standard_set.sh line 47:
elif [ "${format}" = 'orc' ]; then
^---------------------^ SC2292 (style): Prefer [[ ]] over [ ] for
tests in Bash/Ksh.
Did you mean:
elif [[ "${format}" = 'orc' ]]; then
In tools/emr_storage_regression/stardard_set/run_standard_set.sh line 52:
elif [ "${case}" = 'ssb_flat' ]; then
^------------------------^ SC2292 (style): Prefer [[ ]] over [ ] for
tests in Bash/Ksh.
Did you mean:
elif [[ "${case}" = 'ssb_flat' ]]; then
In tools/emr_storage_regression/stardard_set/run_standard_set.sh line 54:
if [ "${format}" = 'parquet' ]; then
^-------------------------^ SC2292 (style): Prefer [[ ]] over [ ] for
tests in Bash/Ksh.
Did you mean:
if [[ "${format}" = 'parquet' ]]; then
In tools/emr_storage_regression/stardard_set/run_standard_set.sh line 56:
elif [ "${format}" = 'orc' ]; then
^---------------------^ SC2292 (style): Prefer [[ ]] over [ ] for
tests in Bash/Ksh.
Did you mean:
elif [[ "${format}" = 'orc' ]]; then
For more information:
https://www.shellcheck.net/wiki/SC2148 -- Tips depend on target shell and
y...
https://www.shellcheck.net/wiki/SC2016 -- Expressions don't expand in
singl...
https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing
...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' returned error 1 finding the following formatting issues:
----------
--- tools/emr_storage_regression/emr_tools.sh.orig
+++ tools/emr_storage_regression/emr_tools.sh
@@ -84,28 +84,28 @@
echo "$CASE"
if [[ ${CASE} == 'ping' ]]; then
- sh ping_poc.sh "${ENDPOINT}" "${REGION}" "${SERVER}" "${AK}" "${SK}"
+ sh ping_poc.sh "${ENDPOINT}" "${REGION}" "${SERVER}" "${AK}" "${SK}"
elif [[ ${CASE} == 'test_set' ]]; then
- if [[ ${SERVER} == 'tx' ]]; then
- BUCKET=cosn://datalake-bench-cos-1308700295
- elif [[ ${SERVER} == 'ali' ]]; then
- BUCKET=oss://benchmark-oss
- fi
- if sh gen_spark_create_sql.sh "${BUCKET}"; then
- echo "Fail to generate spark obj table for test set"
- exit 1
- fi
- if sh gen_spark_create_sql.sh hdfs:///benchmark-hdfs; then
- echo "Fail to generate spark hdfs table for test set, import hdfs data
first"
- exit 1
- fi
- sh run_standard_set.sh hms_hdfs parquet
- sh run_standard_set.sh iceberg_hms parquet
- if [[ ${SERVER} == 'tx' ]]; then
- sh run_standard_set.sh hms_cos
- sh run_standard_set.sh iceberg_hms_cos parquet
- elif [[ ${SERVER} == 'ali' ]]; then
- sh run_standard_set.sh hms_oss
- sh run_standard_set.sh iceberg_hms_oss parquet
- fi
+ if [[ ${SERVER} == 'tx' ]]; then
+ BUCKET=cosn://datalake-bench-cos-1308700295
+ elif [[ ${SERVER} == 'ali' ]]; then
+ BUCKET=oss://benchmark-oss
+ fi
+ if sh gen_spark_create_sql.sh "${BUCKET}"; then
+ echo "Fail to generate spark obj table for test set"
+ exit 1
+ fi
+ if sh gen_spark_create_sql.sh hdfs:///benchmark-hdfs; then
+ echo "Fail to generate spark hdfs table for test set, import hdfs
data first"
+ exit 1
+ fi
+ sh run_standard_set.sh hms_hdfs parquet
+ sh run_standard_set.sh iceberg_hms parquet
+ if [[ ${SERVER} == 'tx' ]]; then
+ sh run_standard_set.sh hms_cos
+ sh run_standard_set.sh iceberg_hms_cos parquet
+ elif [[ ${SERVER} == 'ali' ]]; then
+ sh run_standard_set.sh hms_oss
+ sh run_standard_set.sh iceberg_hms_oss parquet
+ fi
fi
--- tools/emr_storage_regression/ping_test/ping_poc.sh.orig
+++ tools/emr_storage_regression/ping_test/ping_poc.sh
@@ -8,11 +8,9 @@
HMS_META_URI=$6
HMS_WAREHOUSE=$7
-
-
if [[ -z ${HMS_WAREHOUSE} ]]; then
- echo "$SERVICE"
-fi
+ echo "$SERVICE"
+fi
run_spark_create_sql() {
echo "${SERVICE}"
@@ -28,7 +26,7 @@
--conf spark.sql.catalog.hms.type=hive \
--conf spark.sql.defaultCatalog=hms \
--conf spark.sql.catalog.hms.warehouse="${HMS_WAREHOUSE}" \
- -f create_spark_ping > spark_create.log
+ -f create_spark_ping >spark_create.log
elif [[ ${SERVICE} == 'tx' ]]; then
spark-sql --jars
/usr/local/service/iceberg/iceberg-spark-runtime-3.2_2.12-0.13.1.jar \
--conf
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
\
@@ -37,7 +35,7 @@
--conf
spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.local.type=hadoop \
--conf spark.sql.catalog.local.warehouse=/usr/hive/warehouse \
- -f create_spark_ping > spark_create.log
+ -f create_spark_ping >spark_create.log
elif [[ ${SERVICE} == 'hw' ]]; then
echo "${SERVICE}"
fi
@@ -50,17 +48,16 @@
}
run_hive_create_sql() {
- hive -f create_hive_ping.sql 2> hive_create.log
+ hive -f create_hive_ping.sql 2>hive_create.log
}
run_spark_create_sql
run_hive_create_sq
## Step 2: make ping data
-spark-sql -f data_for_spark.sql > spark_data.log
-hive -f data_for_hive.sql > hive_data.log
+spark-sql -f data_for_spark.sql >spark_data.log
+hive -f data_for_hive.sql >hive_data.log
-
FE_HOST=172.16.1.163
USER=root
FE_QUERY_PORT=9035
@@ -77,33 +74,33 @@
[[ "${i}" != "${TRIES}" ]] && echo -n ","
done
QUERY_NUM=$((QUERY_NUM + 1))
- done < "${sql_file}"
+ done <"${sql_file}"
}
## Step 3: create external catalog in doris
case "${SERVICE}" in
- ali)
- sed -e 's#ENDPOINT#'"${ENDPOINT}"'#g' -e
's#META_URI#'"${HMS_META_URI}"'#g' -e 's#AK_INPUT#'"${AK}"'#g' -e
's#SK_INPUT#'"${SK}"'#g' create_catalog_aliyun.sql > emr_catalog.sql
- ## support dlf and jindo in beijing
- # DLF_ENDPOINT=datalake-vpc.cn-beijing.aliyuncs.com
- # sed -e 's#DLF_ENDPOINT#'"${DLF_ENDPOINT}"'#g'
create_catalog_aliyun.sql > emr_catalog.sql
- JINDO_ENDPOINT=cn-beijing.oss-dls.aliyuncs.com
- sed -e 's#JINDO_ENDPOINT#'"${JINDO_ENDPOINT}"'#g'
create_catalog_aliyun.sql > emr_catalog.sql
- ;;
- tx)
- sed -e 's#ENDPOINT#'"${ENDPOINT}"'#g' -e
's#META_URI#'"${HMS_META_URI}"'#g' -e 's#AK_INPUT#'"${AK}"'#g' -e
's#SK_INPUT#'"${SK}"'#g' create_catalog_tx.sql > emr_catalog.sql
- ;;
- aws)
- sed -e 's#ENDPOINT#'"${ENDPOINT}"'#g' -e
's#META_URI#'"${HMS_META_URI}"'#g' -e 's#AK_INPUT#'"${AK}"'#g' -e
's#SK_INPUT#'"${SK}"'#g' create_catalog_aws.sql > emr_catalog.sql
- ;;
- hw)
- sed -e 's#ENDPOINT#'"${ENDPOINT}"'#g' -e
's#META_URI#'"${HMS_META_URI}"'#g' -e 's#AK_INPUT#'"${AK}"'#g' -e
's#SK_INPUT#'"${SK}"'#g' create_catalog_hw.sql > emr_catalog.sql
- ;;
- *)
- echo "Internal error"
- exit 1
- ;;
- esac
+ali)
+ sed -e 's#ENDPOINT#'"${ENDPOINT}"'#g' -e
's#META_URI#'"${HMS_META_URI}"'#g' -e 's#AK_INPUT#'"${AK}"'#g' -e
's#SK_INPUT#'"${SK}"'#g' create_catalog_aliyun.sql >emr_catalog.sql
+ ## support dlf and jindo in beijing
+ # DLF_ENDPOINT=datalake-vpc.cn-beijing.aliyuncs.com
+ # sed -e 's#DLF_ENDPOINT#'"${DLF_ENDPOINT}"'#g'
create_catalog_aliyun.sql > emr_catalog.sql
+ JINDO_ENDPOINT=cn-beijing.oss-dls.aliyuncs.com
+ sed -e 's#JINDO_ENDPOINT#'"${JINDO_ENDPOINT}"'#g'
create_catalog_aliyun.sql >emr_catalog.sql
+ ;;
+tx)
+ sed -e 's#ENDPOINT#'"${ENDPOINT}"'#g' -e
's#META_URI#'"${HMS_META_URI}"'#g' -e 's#AK_INPUT#'"${AK}"'#g' -e
's#SK_INPUT#'"${SK}"'#g' create_catalog_tx.sql >emr_catalog.sql
+ ;;
+aws)
+ sed -e 's#ENDPOINT#'"${ENDPOINT}"'#g' -e
's#META_URI#'"${HMS_META_URI}"'#g' -e 's#AK_INPUT#'"${AK}"'#g' -e
's#SK_INPUT#'"${SK}"'#g' create_catalog_aws.sql >emr_catalog.sql
+ ;;
+hw)
+ sed -e 's#ENDPOINT#'"${ENDPOINT}"'#g' -e
's#META_URI#'"${HMS_META_URI}"'#g' -e 's#AK_INPUT#'"${AK}"'#g' -e
's#SK_INPUT#'"${SK}"'#g' create_catalog_hw.sql >emr_catalog.sql
+ ;;
+*)
+ echo "Internal error"
+ exit 1
+ ;;
+esac
run_query emr_catalog.sql
## Step 4: query ping
--- tools/emr_storage_regression/stardard_set/gen_spark_create_sql.sh.orig
+++ tools/emr_storage_regression/stardard_set/gen_spark_create_sql.sh
@@ -1,16 +1,15 @@
#!/bin/bash
BUCKET=$1
-sh gen_tbl/gen_ssb_create_sql.sh "${BUCKET}"/ssb/ssb100_orc
ssb100_orc_hdfs orc > create.sql
-sh gen_tbl/gen_ssb_create_sql.sh "${BUCKET}"/ssb/ssb100_parquet
ssb100_parquet_hdfs parquet >> create.sql
-sh gen_tbl/gen_ssb_create_sql.sh "${BUCKET}"/ssb/ssb100_text
ssb100_text_hdfs text >> create.sql
+sh gen_tbl/gen_ssb_create_sql.sh "${BUCKET}"/ssb/ssb100_orc ssb100_orc_hdfs
orc >create.sql
+sh gen_tbl/gen_ssb_create_sql.sh "${BUCKET}"/ssb/ssb100_parquet
ssb100_parquet_hdfs parquet >>create.sql
+sh gen_tbl/gen_ssb_create_sql.sh "${BUCKET}"/ssb/ssb100_text
ssb100_text_hdfs text >>create.sql
# tpch
-sh gen_tbl/gen_tpch_create_sql.sh "${BUCKET}"/tpch/tpch100_orc
tpch100_orc_hdfs orc >> create.sql
-sh gen_tbl/gen_tpch_create_sql.sh "${BUCKET}"/tpch/tpch100_parquet
tpch100_parquet_hdfs parquet >> create.sql
-sh gen_tbl/gen_tpch_create_sql.sh "${BUCKET}"/tpch/tpch100_text
tpch100_text_hdfs text >> create.sql
+sh gen_tbl/gen_tpch_create_sql.sh "${BUCKET}"/tpch/tpch100_orc
tpch100_orc_hdfs orc >>create.sql
+sh gen_tbl/gen_tpch_create_sql.sh "${BUCKET}"/tpch/tpch100_parquet
tpch100_parquet_hdfs parquet >>create.sql
+sh gen_tbl/gen_tpch_create_sql.sh "${BUCKET}"/tpch/tpch100_text
tpch100_text_hdfs text >>create.sql
# clickbench
-sh gen_tbl/gen_clickbench_create_sql.sh "${BUCKET}"/clickbench
clickbench_hdfs >> create.sql
+sh gen_tbl/gen_clickbench_create_sql.sh "${BUCKET}"/clickbench
clickbench_hdfs >>create.sql
# iceberg
# sh gen_tbl/gen_ssb_create_sql.sh oss://benchmark-oss/ssb/ssb100_iceberg
ssb100_iceberg iceberg >> create.sql
# sh gen_tbl/gen_tpch_create_sql.sh
oss://benchmark-oss/tpch/tpch100_iceberg tpch100_iceberg iceberg >> create.sql
# sh gen_tbl/gen_clickbench_create_sql.sh oss://benchmark-oss/clickbench
clickbench >> create.sql
-
---
tools/emr_storage_regression/stardard_set/gen_tbl/gen_clickbench_create_sql.sh.orig
+++
tools/emr_storage_regression/stardard_set/gen_tbl/gen_clickbench_create_sql.sh
@@ -2,7 +2,7 @@
if [ -z $1 ]; then
echo 'the first argument is database location'
exit
-else
+else
db_loc=$1
fi
---
tools/emr_storage_regression/stardard_set/gen_tbl/gen_ssb_create_sql.sh.orig
+++ tools/emr_storage_regression/stardard_set/gen_tbl/gen_ssb_create_sql.sh
@@ -2,7 +2,7 @@
if [ -z $1 ]; then
echo 'the first argument is database location'
exit
-else
+else
db_loc=$1
fi
---
tools/emr_storage_regression/stardard_set/gen_tbl/gen_tpch_create_sql.sh.orig
+++ tools/emr_storage_regression/stardard_set/gen_tbl/gen_tpch_create_sql.sh
@@ -2,7 +2,7 @@
if [ -z $1 ]; then
echo 'the first argument is database location'
exit
-else
+else
db_loc=$1
fi
--- tools/emr_storage_regression/stardard_set/run_queries.sh.orig
+++ tools/emr_storage_regression/stardard_set/run_queries.sh
@@ -17,11 +17,11 @@
while read -r query; do
echo -n "query${QUERY_NUM}," | tee -a $RESULT_FILE
for i in $(seq 1 $TRIES); do
- RES=$( mysql -vvv -h"${FE_HOST}" -u"${USER}" -P"${FE_QUERY_PORT}"
-D"${DB}" -e "${query}" | perl -nle 'print $1 if /((\d+\.\d+)+ sec)/' || :)
+ RES=$(mysql -vvv -h"${FE_HOST}" -u"${USER}" -P"${FE_QUERY_PORT}"
-D"${DB}" -e "${query}" | perl -nle 'print $1 if /((\d+\.\d+)+ sec)/' || :)
echo -n "$RES" | tee -a $RESULT_FILE
[[ "$i" != "$TRIES" ]] && echo -n "," | tee -a $RESULT_FILE
done
echo "" | tee -a $RESULT_FILE
QUERY_NUM=$((QUERY_NUM + 1))
-done < $5
+done <$5
--- tools/emr_storage_regression/stardard_set/run_standard_set.sh.orig
+++ tools/emr_storage_regression/stardard_set/run_standard_set.sh
@@ -4,12 +4,12 @@
if [ -z "$1" ]; then
echo 'need catalog name'
exit
-else
+else
catalog_name=$1
fi
if [ -z "$2" ]; then
format=parquet
-else
+else
format=$2
fi
FE_HOST=172.16.1.163
@@ -18,17 +18,17 @@
if [ -z "$3" ]; then
echo "run all test"
-else
+else
case=$3
fi
-if [ "${case}" = 'ssb' ] ; then
+if [ "${case}" = 'ssb' ]; then
# ssb
if [ "${format}" = 'parquet' ]; then
sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".ssb100_parquet queries/ssb_queries.sql
elif [ "${format}" = 'orc' ]; then
sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".ssb100_orc queries/ssb_queries.sql
- else
+ else
sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".ssb100_text queries/ssb_queries.sql
fi
elif [ "${case}" = 'tpch' ]; then
@@ -37,7 +37,7 @@
sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".tpch100_parquet queries/tpch_queries.sql
elif [ "${format}" = 'orc' ]; then
sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".tpch100_orc queries/tpch_queries.sql
- else
+ else
sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".tpch100_text queries/tpch_queries.sql
fi
elif [ "${case}" = 'clickbench' ]; then
@@ -46,7 +46,7 @@
sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".clickbench_parquet queries/clickbench_queries.sql
elif [ "${format}" = 'orc' ]; then
sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".clickbench_orc queries/clickbench_queries.sql
- else
+ else
sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".clickbench_text queries/clickbench_queries.sql
fi
elif [ "${case}" = 'ssb_flat' ]; then
@@ -55,7 +55,7 @@
sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".ssb100_parquet queries/ssb_flat_queries.sql
elif [ "${format}" = 'orc' ]; then
sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".ssb100_orc queries/ssb_flat_queries.sql
- else
+ else
sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".ssb100_text queries/ssb_flat_queries.sql
fi
fi
@@ -66,7 +66,7 @@
# sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".ssb100_parquet_hdfs queries/ssb_queries.sql
# elif [ "${format}" = 'orc' ]; then
# sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".ssb100_orc_hdfs queries/ssb_queries.sql
-# else
+# else
# sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".ssb100_text_hdfs queries/ssb_queries.sql
# fi
# elif [ "${case}" = 'tpch' ]; then
@@ -75,7 +75,7 @@
# sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".tpch100_parquet_hdfs queries/tpch_queries.sql
# elif [ "${format}" = 'orc' ]; then
# sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".tpch100_orc_hdfs queries/tpch_queries.sql
-# else
+# else
# sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".tpch100_text_hdfs queries/tpch_queries.sql
# fi
# elif [ "${case}" = 'clickbench' ]; then
@@ -84,7 +84,7 @@
# sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".clickbench_hdfs_parquet queries/clickbench_queries.sql
# elif [ "${format}" = 'orc' ]; then
# sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".clickbench_hdfs_orc queries/clickbench_queries.sql
-# else
+# else
# sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".clickbench_hdfs_text queries/clickbench_queries.sql
# fi
# elif [ "${case}" = 'ssb_flat' ]; then
@@ -93,11 +93,7 @@
# sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".ssb100_parquet_hdfs queries/ssb_flat_queries.sql
# elif [ "${format}" = 'orc' ]; then
# sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".ssb100_orc_hdfs queries/ssb_flat_queries.sql
-# else
+# else
# sh run_queries.sh "${FE_HOST}" "${USER}" "${PORT}"
"${catalog_name}".ssb100_text_hdfs queries/ssb_flat_queries.sql
# fi
# fi
-
-
-
-
----------
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
