gyang94 commented on code in PR #2769:
URL: https://github.com/apache/fluss/pull/2769#discussion_r2901473203
##########
fluss-server/src/main/java/org/apache/fluss/server/coordinator/MetadataManager.java:
##########
@@ -692,13 +696,14 @@ public Set<String> getPartitions(TablePath tablePath) {
public void createPartition(
TablePath tablePath,
long tableId,
+ String remoteDataDir,
Review Comment:
I am thinking about a possible improvements here:
1.
Will it be better to put the remote dir selection logic inside into the
`MetadataManager.createPartition()` and `MetadataManager.createTable()`.
I notice that now there exist some code pattern that (a). select a
remoteDataDir value. (b) pass the value into createTable/createPartition
functions, when we need to createTable/partition otherwhere.
Put the step(a) inside those functions will make the code more focused and
void adding a remoteDataDir parameter in these functions.
2.
Is is good to add the 'remoteDataDir' field into the `TablePath` class? Make
the TablePath object carrys the remote dir info, and pass it through to
everywhere. Then we don't need to care about adding an independent
remoteDataDir parameter in functions.
The implementation now is good to work. Just put my thoughts here. What do
you think?
##########
fluss-server/src/main/java/org/apache/fluss/server/coordinator/remote/RemoteDirDynamicLoader.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.server.coordinator.remote;
+
+import org.apache.fluss.annotation.VisibleForTesting;
+import org.apache.fluss.config.ConfigOption;
+import org.apache.fluss.config.ConfigOptions;
+import org.apache.fluss.config.Configuration;
+import org.apache.fluss.config.cluster.ServerReconfigurable;
+import org.apache.fluss.exception.ConfigException;
+import org.apache.fluss.fs.FsPath;
+
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * Dynamic loader for remote data directories that supports runtime
reconfiguration.
+ *
+ * <p>This class manages the lifecycle of remote data directories and provides
a container for
+ * selecting remote data directories. It implements {@link
ServerReconfigurable} to support dynamic
+ * configuration updates at runtime without requiring a server restart.
+ *
+ * <p>The remote data directories are used for storing tiered storage data,
including:
+ *
+ * <ul>
+ * <li>KV snapshot data files for primary key tables
+ * <li>Remote log segments for log tiered storage
+ * </ul>
+ *
+ * <p>When creating a new table or partition, the coordinator server uses this
loader to select an
+ * appropriate remote data directory based on the configured selection
strategy (see {@link
+ * org.apache.fluss.config.ConfigOptions#REMOTE_DATA_DIRS_STRATEGY}).
+ */
+public class RemoteDirDynamicLoader implements ServerReconfigurable,
AutoCloseable {
+
+ private volatile RemoteDirContainer remoteDirContainer;
+ private Configuration currentConfiguration;
+
+ public RemoteDirDynamicLoader(Configuration configuration) {
+ this.currentConfiguration = configuration;
+ this.remoteDirContainer = new RemoteDirContainer(configuration);
+ }
+
+ /**
+ * Gets a container for managing and selecting remote data directories.
+ *
+ * <p>The container encapsulates the remote data directories and the
selector strategy used to
+ * choose directories.
+ *
+ * @return a container for remote data directories
+ */
+ public RemoteDirContainer getRemoteDataDirContainer() {
+ return remoteDirContainer;
+ }
+
+ @Override
+ public void validate(Configuration newConfig) throws ConfigException {
+ // Get and valid remote data dirs
+ List<String> remoteDataDirs =
+ newConfig
+ .getOptional(ConfigOptions.REMOTE_DATA_DIRS)
+ .orElseGet(() ->
currentConfiguration.get(ConfigOptions.REMOTE_DATA_DIRS));
+ for (int i = 0; i < remoteDataDirs.size(); i++) {
+ String remoteDataDir = remoteDataDirs.get(i);
Review Comment:
should we validate that the new remoteDataDirs contains all existing
remoteDataDirs? Just allow add new remote dirs, and deny remove any exsiting
remote dir.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
