[
https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jinglun updated HADOOP-15565:
-----------------------------
Description:
ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE.
It's children filesystems are cached in FileSystem.CACHE and shared by all the
ViewFileSystem instances. We could't simply close all the children filesystems
because it will break the semantic of FileSystem.newInstance().
We might add an inner cache to ViewFileSystem, let it cache all the children
filesystems. The children filesystems are not shared any more. When
ViewFileSystem is closed we close all the children filesystems in the inner
cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't be
too many FileSystem instances.
The FileSystem.CACHE caches the ViewFileSysem instance and the other
instances(the children filesystems) are cached in the inner cache.
was:
When we create a ViewFileSystem, all it's child filesystems will be cached by
FileSystem.CACHE. Unless we close these child filesystems, they will stay in
FileSystem.CACHE forever.
I think we should let FileSystem.CACHE cache ViewFileSystem only, and let
ViewFileSystem cache all it's child filesystems. So we can close ViewFileSystem
without leak and won't affect other ViewFileSystems.
I find this problem because i need to re-login my kerberos and renew
ViewFileSystem periodically. Because FileSystem.CACHE.Key is based on
UserGroupInformation, which changes everytime i re-login, I can't use the
cached child filesystems when i new a ViewFileSystem. And because
ViewFileSystem.close does nothing but remove itself from cache, i leak all it's
child filesystems in cache.
> ViewFileSystem.close doesn't close child filesystems and causes FileSystem
> objects leak.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-15565
> URL: https://issues.apache.org/jira/browse/HADOOP-15565
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Jinglun
> Assignee: Jinglun
> Priority: Major
> Attachments: HADOOP-15565.0001.patch, HADOOP-15565.0002.patch
>
>
> ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE.
> It's children filesystems are cached in FileSystem.CACHE and shared by all
> the ViewFileSystem instances. We could't simply close all the children
> filesystems because it will break the semantic of FileSystem.newInstance().
> We might add an inner cache to ViewFileSystem, let it cache all the children
> filesystems. The children filesystems are not shared any more. When
> ViewFileSystem is closed we close all the children filesystems in the inner
> cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't
> be too many FileSystem instances.
> The FileSystem.CACHE caches the ViewFileSysem instance and the other
> instances(the children filesystems) are cached in the inner cache.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
