Jeff-Shen-00 opened a new issue, #70315: URL: https://github.com/apache/doris/issues/70315
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description In main for loop, inner loop deals with one table, when 100 data file of table is finished, release parallel control token. The lock granularity is table level. There are 25 tables to load, that means, parallel larger than 25 has no effect to improve load efficiency. In fact, there are big tables, such as store_sales/catalog_sales, of which data files are loaded one by one, even though other small tables finished loading. So in order to improve load efficiency, change lock granularity from table level to data file level. Since there are 100 data files for each table, so parallel is not limited by 25, and can be enlarged greatly. ### Solution diff --git a/tools/tpcds-tools/bin/load-tpcds-data.sh b/tools/tpcds-tools/bin/load-tpcds-data.sh index 6197d0d6d18..8d489f34e96 100755 --- a/tools/tpcds-tools/bin/load-tpcds-data.sh +++ b/tools/tpcds-tools/bin/load-tpcds-data.sh @@ -171,22 +171,34 @@ for table_name in ${!table_columns[*]}; do # 对管道，读一行便少一行，每次只能读取一行 # 所有行读取完毕, 执行挂起, 直到管道再次有可读行 # 因此实现了进程数量控制 - read -r -u3 # 要批量执行的命令放在大括号内, 后台运行 - { - for file in "${TPCDS_DATA_DIR}/${table_name}"_{1..100}_*.dat; do - if ! [[ -f "${file}" ]]; then continue; fi - FILE_ID=$(echo "${file}" | awk -F'/' '{print $(NF)}' | awk -F'.' '{print $(1)}') + for file in "${TPCDS_DATA_DIR}/${table_name}"_{1..100}_*.dat; do + # 跳过不存在的文件（花括号展开可能产生不存在的路径） + if [[ ! -f "${file}" ]]; then + continue + fi + + # 获取令牌，阻塞直到有空闲槽位 + read -r -u3 + + { + # 立即保存循环变量值，避免后续循环迭代修改它们 + tname="$table_name" + f="$file" + + FILE_ID=$(echo "$f" | awk -F'/' '{print $(NF)}' | awk -F'.' '{print $(1)}') + columns="${table_columns[$tname]}" + if [[ -z ${TXN_ID} ]]; then ret=$(curl \ --location-trusted \ -u "${USER}":"${PASSWORD:-}" \ -H "Expect: 100-continue" \ -H "column_separator:|" \ - -H "columns: ${table_columns[${table_name}]}" \ - -T "${file}" \ - http://"${FE_HOST}":"${FE_HTTP_PORT:-8030}"/api/"${DB}"/"${table_name}"/_stream_load 2>/dev/null) + -H "columns: ${columns}" \ + -T "${f}" \ + http://"${FE_HOST}":"${FE_HTTP_PORT:-8030}"/api/"${DB}"/"${tname}"/_stream_load 2>/dev/null) else ret=$(curl \ --location-trusted \ @@ -194,20 +206,21 @@ for table_name in ${!table_columns[*]}; do -H "Expect: 100-continue" \ -H "label:${TXN_ID}_${FILE_ID}" \ -H "column_separator:|" \ - -H "columns: ${table_columns[${table_name}]}" \ - -T "${file}" \ - http://"${FE_HOST}":"${FE_HTTP_PORT:-8030}"/api/"${DB}"/"${table_name}"/_stream_load 2>/dev/null) + -H "columns: ${columns}" \ + -T "${f}" \ + http://"${FE_HOST}":"${FE_HTTP_PORT:-8030}"/api/"${DB}"/"${tname}"/_stream_load 2>/dev/null) fi if [[ $(echo "${ret}" | jq ".Status") == '"Success"' ]]; then - echo "----loaded ${file}" + echo "----loaded ${f}" else - echo -e "\033[31m----load ${file} FAIL...\n${ret}\033[0m" + echo -e "\033[31m----load ${f} FAIL...\n${ret}\033[0m" fi - done - # 归还令牌, 即进程结束后，再写入一行，使挂起的循环继续执行 - echo >&3 - } & + + # 归还令牌 + echo >&3 + } & + done done ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
