Hi Joe,
Yes I had made these changes for getting HDFS to work with Solr. Below are
config changes which I carried out:
-------------------------------- Changes in solr.in.cmd
--------------------------------
set SOLR_OPTS=%SOLR_OPTS% -Dsolr.directoryFactory=HdfsDirectoryFactory
set SOLR_OPTS=%SOLR_OPTS% -Dsolr.lock.type=hdfs
set SOLR_OPTS=%SOLR_OPTS% -Dsolr.hdfs.home=hdfs://
hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data
-------------------------------- Changes in solrconfig.xml
--------------------------------
<directoryFactory name="DirectoryFactory"
class="solr.HdfsDirectoryFactory">
<str name="solr.hdfs.home">hdfs://
hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data
</str>
<str name="solr.hdfs.confdir">/etc/hadoop/conf</str>
<bool name="solr.hdfs.blockcache.enabled">true</bool>
<int name="solr.hdfs.blockcache.slab.count">1</int>
<bool name="solr.hdfs.blockcache.direct.memory.allocation">true</bool>
<int name="solr.hdfs.blockcache.blocksperbank">16384</int>
<bool name="solr.hdfs.blockcache.read.enabled">true</bool>
<bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool>
<int name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">16</int>
<int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">192</int>
</directoryFactory>
Let me know if you have any comments?
Thanks!
On Mon, Aug 24, 2020 at 10:23 PM Joe Obernberger <
joseph.obernber...@gmail.com> wrote:
> Are you running with solr.lock.type=hdfs
> ?
>
> Have you defined your DirectoryFactory - something like:
>
> <directoryFactory name="DirectoryFactory"
> class="solr.HdfsDirectoryFactory">
> <bool name="solr.hdfs.blockcache.enabled">true</bool>
> <bool name="solr.hdfs.blockcache.global">true</bool>
> <int name="solr.hdfs.blockcache.slab.count">43</int>
> <bool
> name="solr.hdfs.blockcache.direct.memory.allocation">true</bool>
> <int name="solr.hdfs.blockcache.blocksperbank">16384</int>
> <bool name="solr.hdfs.blockcache.read.enabled">true</bool>
> <bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool>
> <int name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">128</int>
> <int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">1024</int>
> <str name="solr.hdfs.home">hdfs://nameservice1:8020/solr8.2.0</str>
> <str name="solr.hdfs.confdir">/etc/hadoop/conf.cloudera.hdfs1</str>
> </directoryFactory>
>
> -Joe
> On 8/20/2020 2:30 AM, Prashant Jyoti wrote:
>
> Hi Joe,
> These are the errors I am running into:
>
> org.apache.solr.common.SolrException: Error CREATEing SolrCore
> 'newcollsolr2_shard1_replica_n1': Unable to create core
> [newcollsolr2_shard1_replica_n1] Caused by: Illegal char <:> at index 4:
> hdfs://
> hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data/newcollsolr2/core_node3/data\
> <http://hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data/newcollsolr2/core_node3/data%5C>
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1256)
> at
> org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:93)
> at
> org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)
> at
> org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:352)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
> at
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:500)
> at
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.solr.common.SolrException: Unable to create core
> [newcollsolr2_shard1_replica_n1]
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1327)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1217)
> ... 47 more
> Caused by: org.apache.solr.common.SolrException: Illegal char <:> at index
> 4: hdfs://
> hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data/newcollsolr2/core_node3/data\
> <http://hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data/newcollsolr2/core_node3/data%5C>
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1072)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901)
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1306)
> ... 48 more
> Caused by: java.nio.file.InvalidPathException: Illegal char <:> at index
> 4: hdfs://
> hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data/newcollsolr2/core_node3/data\
> <http://hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data/newcollsolr2/core_node3/data%5C>
> at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
> at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
> at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
> at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
> at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
> at sun.nio.fs.AbstractPath.resolve(AbstractPath.java:53)
> at org.apache.solr.core.SolrCore.initUpdateLogDir(SolrCore.java:1380)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:958)
> ... 50 more
>
> Attaching it as a file too. Thanks!
>
> On Wed, Aug 19, 2020 at 9:37 PM Joe Obernberger <
> joseph.obernber...@gmail.com> wrote:
>
>> Your exception didn't come across - can you paste it in?
>>
>> -Joe
>>
>> On 8/19/2020 10:50 AM, Prashant Jyoti wrote:
>> > You're right Andrew. Even I read about that. But there's a use case for
>> > which we want to configure the said case.
>> >
>> > Are you also aware of what feature we are moving towards instead of
>> HDFS?
>> > Will you be able to help me with the error that I'm running into?
>> >
>> > Thanks in advance!
>> >
>> >
>> > On Wed, 19 Aug, 2020, 5:24 pm Andrew MacKay, <andrew.mac...@superna.net
>> >
>> > wrote:
>> >
>> >> I believe HDFS support is being deprecated in Solr. Not sure you want
>> to
>> >> continue configuration if support will disappear.
>> >>
>> >> On Wed, Aug 19, 2020 at 7:52 AM Prashant Jyoti <jtprash...@gmail.com>
>> >> wrote:
>> >>
>> >>> Hi all,
>> >>> Hope you are healthy and safe.
>> >>>
>> >>> Need some help with HDFS configuration.
>> >>>
>> >>> Could anybody of you share an example of the configuration with which
>> you
>> >>> are running Solr with HDFS in any of your production/dev environments?
>> >>> I am interested in the parts of SolrConfig.xml / Solr.in.cmd/sh which
>> you
>> >>> may have modified. Obviously with the security parts obfuscated.
>> >>>
>> >>> I am stuck at an error and unable to move ahead. Attaching the
>> exception
>> >>> log if anyone is interested to look at the error.
>> >>>
>> >>> Thanks!
>> >>>
>> >>> --
>> >>> Regards,
>> >>> Prashant.
>> >>>
>> >> --
>> >> CONFIDENTIALITY NOTICE: The information contained in this email is
>> >> privileged and confidential and intended only for the use of the
>> >> individual
>> >> or entity to whom it is addressed. If you receive this message in
>> error,
>> >> please notify the sender immediately at 613-729-1100 and destroy the
>> >> original message and all copies. Thank you.
>> >>
>> >
>>
>
>
> --
> Regards,
> Prashant.
>
>
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
> Virus-free.
> www.avg.com
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
> <#m_-6899573471279532520_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
>
--
Regards,
Prashant.
