[
https://issues.apache.org/jira/browse/HADOOP-17130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17158145#comment-17158145
]
Mukund Thakur commented on HADOOP-17130:
----------------------------------------
Initially we thought the ConcurrentModificationException might be because of
handling deprecations but I couldn't reproduce this using unit testing.
{code:java}
Map<String,String> result = new HashMap<String,String>();
Matcher m;
for(Map.Entry<Object,Object> item: getProps().entrySet()) {
if (item.getKey() instanceof String &&
item.getValue() instanceof String) {
m = p.matcher((String)item.getKey());
if(m.find()) { // match
result.put((String) item.getKey(),
substituteVars(getProps().getProperty((String) item.getKey())));
}
}
}{code}
But we still believe that this code subtituteVars() may be updating the props
concurrently and it would be a better if we do this substitution later after
fetching the list of matching properties.
CC [[email protected]]
> Configuration.getValByRegex() shouldn't be updating the results while
> fetching.
> -------------------------------------------------------------------------------
>
> Key: HADOOP-17130
> URL: https://issues.apache.org/jira/browse/HADOOP-17130
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
> Affects Versions: 3.1.3
> Reporter: Mukund Thakur
> Assignee: Mukund Thakur
> Priority: Major
>
> We have seen this stacktrace while using ABFS file system. After analysing
> the stack trace we can see that getValByRegex() is reading the properties and
> substituting the value in the same call. This may cause the
> ConcurrentModificationException.
> {code:java}
> Caused by: java.util.concurrent.ExecutionException:
> java.util.ConcurrentModificationException at
> java.util.concurrent.FutureTask.report(FutureTask.java:122) at
> java.util.concurrent.FutureTask.get(FutureTask.java:192) at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1877)
> ... 18 more Caused by: java.util.ConcurrentModificationException at
> java.util.Hashtable$Enumerator.next(Hashtable.java:1387) at
> org.apache.hadoop.conf.Configuration.getValByRegex(Configuration.java:3855)
> at
> org.apache.hadoop.fs.azurebfs.AbfsConfiguration.validateStorageAccountKeys(AbfsConfiguration.java:689)
> at
> org.apache.hadoop.fs.azurebfs.AbfsConfiguration.<init>(AbfsConfiguration.java:237)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:154)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:113)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3396) at
> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158) at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3456) at
> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3424) at
> org.apache.hadoop.fs.FileSystem.get(FileSystem.java:518) at
> org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
