[ 
https://issues.apache.org/jira/browse/SOLR-14660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17167449#comment-17167449
 ] 

Istvan Farkas commented on SOLR-14660:
--------------------------------------

I'm working on the HDFS package. As discussed with [~ichattopadhyaya], the 
first step of the plan is to move the HDFS code to a module that builds as a 
separate jar file and remove the direct HDFS references from the core. 
Loading the DirectoryFactory from a separate jar should already be possible. 
Loading it from a package requires the jira [~noble.paul] is working on.

I found the following reference points to hdfs:

- CheckHdfsIndex - standalone utility
- HdfsBackupRepository - used from solr.xml
- HdfsDirectoryFactory (from solrconfig.xml), also referenced by the 
HdfsBackupRepository, CheckHdfsIndex
- HdfsUpdateLog (invoked from UpdateHandler, // if (dirFactory instanceof 
HdfsDirectoryFactory)
- BlockDirectory also references hdfs directory

So far I moved the HDFS-related code and tests to a contrib module and created 
gradle build for it and also removed the references from core to HDFS.
The BlockDirectory reference was an instanceof check:

{code}
private long getFileModified(String name) throws IOException {
    if (in instanceof FSDirectory) {
      File directory = ((FSDirectory) in).getDirectory().toFile();
      File file = new File(directory, name);
      if (!file.exists()) {
        throw new FileNotFoundException("File [" + name + "] not found");
      }
      return file.lastModified();
    } else if (in instanceof HdfsDirectory) {
      return ((HdfsDirectory) in).fileModified(name);
    } else {
      throw new UnsupportedOperationException();
    }
  }
{code}

I added the getLastModified to the base class and the implementations to 
FSDirectory and HdfsDirectory.
Also the updatehandler had some direct references to the HdfsDirectory, I 
removed those.

At the unit test level, the tests which are supposed to test Hadoop-level 
operations like hadoop authentication are also referencing the HDFS code, it 
will need some additional work to clean up those.

I will work on stabilizing this as a separate jar and then sync up with 
[~ichattopadhyaya] and [~noble.paul] on making this a package.



> Migrating HDFS into a package
> -----------------------------
>
>                 Key: SOLR-14660
>                 URL: https://issues.apache.org/jira/browse/SOLR-14660
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Ishan Chattopadhyaya
>            Priority: Major
>
> Following up on the deprecation of HDFS (SOLR-14021), we need to work on 
> isolating it away from Solr core and making a package for this. This issue is 
> to track the efforts for that.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to