Re: JNDI settings
In 4, with a separate servlet container, I used externally-defined, container-managed jndi data sources. However, I'm in the middle of an upgrade to 5 (before moving on to 6), and I had to add the DIH lib to each core's solrconfig.xml and add connection details to each core's DIH config xml. See this: https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler On Sep 12, 2016 11:29 AM, "Aristedes Maniatis" wrote: > I am using Solr 5.5 and wanting to add JNDI settings to Solr (for data > import). I'm new to Solr Cloud setup (previously I was running Solr running > as a custom bundled war) so I can't figure where to put the JNDI settings > with user/pass themselves. > > I don't want to add it to jetty.xml because that's part of the packaged > application which will be upgraded from time to time. > > Should it go into solr.xml inside the solr.home directory? If so, what's > the right syntax there? > > > Ari > > > -- > --> > Aristedes Maniatis > GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A >
Problem with Password Decryption in Data Import Handler
Hi Folks, (Using Solr 5.5.3.) As far as I know, the only place where encrypted password use is documented is in https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler, under the "Configuring the DIH Configuration File", in a comment in the sample XML file: Anyway, I can encrypt just fine: $ openssl enc -aes-128-cbc -a -salt -in stgps.txt enter aes-128-cbc encryption password: Verifying - enter aes-128-cbc encryption password: U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY= I can also decrypt just fine from the command line. However, if I use the encrypted password and encryptKeyFile in the config file, I end up with an error: "String length must be a multiple of four." https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4d9 How do I get this working? Thanks, Jamie
Re: Problem with Password Decryption in Data Import Handler
I tried it both ways yesterday--with a newline and without. On Thu, Oct 6, 2016 at 12:26 PM, Kevin Risden wrote: > I haven't tried this but is it possible there is a new line at the end in > the file? > > If you did something like echo "" > file.txt then there would be a new > line. Use echo -n "" > file.txt > > Also you should be able to check how many characters are in the file. > > Kevin Risden > > On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson > wrote: > > > Hi Folks, > > > > (Using Solr 5.5.3.) > > > > As far as I know, the only place where encrypted password use is > documented > > is in > > https://cwiki.apache.org/confluence/display/solr/ > > Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler, > > under the "Configuring the DIH Configuration File", in a comment in the > > sample XML file: > > > > > > > > Anyway, I can encrypt just fine: > > > > $ openssl enc -aes-128-cbc -a -salt -in stgps.txt > > enter aes-128-cbc encryption password: > > Verifying - enter aes-128-cbc encryption password: > > U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY= > > > > > > I can also decrypt just fine from the command line. > > > > However, if I use the encrypted password and encryptKeyFile in the config > > file, I end up with an error: "String length must be a multiple of four." > > > > https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4d9 > > > > How do I get this working? > > > > Thanks, > > Jamie > > >
Re: Problem with Password Decryption in Data Import Handler
It happens to be ten characters. On Thu, Oct 6, 2016 at 12:44 PM, Alexandre Rafalovitch wrote: > How long is the encryption key (file content)? Because the code I am > looking at seems to expect it to be at most 100 characters. > > Regards, >Alex. > > Newsletter and resources for Solr beginners and intermediates: > http://www.solr-start.com/ > > > On 6 October 2016 at 23:26, Kevin Risden wrote: > > I haven't tried this but is it possible there is a new line at the end in > > the file? > > > > If you did something like echo "" > file.txt then there would be a new > > line. Use echo -n "" > file.txt > > > > Also you should be able to check how many characters are in the file. > > > > Kevin Risden > > > > On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson > wrote: > > > >> Hi Folks, > >> > >> (Using Solr 5.5.3.) > >> > >> As far as I know, the only place where encrypted password use is > documented > >> is in > >> https://cwiki.apache.org/confluence/display/solr/ > >> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler, > >> under the "Configuring the DIH Configuration File", in a comment in the > >> sample XML file: > >> > >> > >> > >> Anyway, I can encrypt just fine: > >> > >> $ openssl enc -aes-128-cbc -a -salt -in stgps.txt > >> enter aes-128-cbc encryption password: > >> Verifying - enter aes-128-cbc encryption password: > >> U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY= > >> > >> > >> I can also decrypt just fine from the command line. > >> > >> However, if I use the encrypted password and encryptKeyFile in the > config > >> file, I end up with an error: "String length must be a multiple of > four." > >> > >> https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4d9 > >> > >> How do I get this working? > >> > >> Thanks, > >> Jamie > >> >
Re: Problem with Password Decryption in Data Import Handler
I'm at a brick wall. Here's the latest status:
Here are some sample commands that I'm using:
*Create the encryptKeyFile and encrypted password:*
encrypter_password='this_is_my_encrypter_password'
plain_db_pw='Oakton153'
cd /var/docker/solr_stage2/credentials/
echo -n "${encrypter_password}" > encpwd.txt
echo -n "${plain_db_pwd}" > plaindbpwd.txt
openssl enc -aes-128-cbc -a -salt -in plaindbpwd.txt -k
"${encrypter_password}"
rm plaindbpwd.txt
That generated this as the password, by the way:
U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o=
*Configure DIH configuration:*
...
By the way, /var/docker/solr_stage2/credentials/ is mapped to
/opt/solr/credentials/ in the docker container, so that's why the paths
*seem* different (but aren't, really).
*Authentication error when data import is run:*
Exception while processing: question document :
SolrInputDocument(fields:
[]):org.apache.solr.handler.dataimport.DataImportHandlerException:
Unable to execute query: select 'foo' as bar;Processing
Document # 1
at
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:69)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:323)
at
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:283)
at
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:52)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:244)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:461)
Caused by: java.sql.SQLInvalidAuthorizationSpecException: Could not
connect: Access denied for user 'root'@'ICZ2002912' (using password:
NO)
at
org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)
at
org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:71)
at org.mariadb.jdbc.Driver.connect(Driver.java:109)
at
org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:192)
at
org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:172)
at
org.apache.solr.handler.dataimport.JdbcDataSource.getConnection(JdbcDataSource.java:503)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:313)
... 12 more
Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could
not connect: Access denied for user 'root'@'ICZ2002912' (using
password: NO)
at
org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.authentication(AbstractConnectProtocol.java:524)
at
org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.handleConnectionPhases(AbstractConnectProtocol.java:472)
at
org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:374)
at
org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:763)
at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:469)
at org.mariadb.jdbc.Driver.connect(Driver.java:104)
... 16 more
On Thu, Oct 6, 2016 at 2:42 PM, Jamie Jackson wrote:
> It happens to be ten characters.
>
> On Thu, Oct 6, 2016 at 12:44 PM, Alexandre Rafalovitch > wrote:
>
>> How long is the encryption key (file content)? Because the code I am
>> looking at seems to expect it to be at most 100 characters.
>>
>> Regards,
>>Alex.
>>
>> Newsletter and resources for Solr beginners and intermediates:
>> http://www.solr-start.com/
>>
>>
>> On 6 October 2016 at 23:26, Kevin Risden
>> wrote:
>> > I haven't tried this but is it possible there is a new line at the end
>> in
>> > the file?
>> >
>> > If you did something like echo "" > file.txt then there would be a new
>> > line. Use echo -n "&quo
Re: Problem with Password Decryption in Data Import Handler
I should have mentioned that I verified connectivity with plain passwords:
>From the same machine that Solr's running on:
solr@000650cbdd5e:/opt/solr$ mysql -uroot -pOakton153 -h local.mysite.com
mysite -e "select 'foo' as bar;"
+-+
| bar |
+-+
| foo |
+-+
Also, if I add the plain-text password to the config, it connects fine:
So that is why I claim to have a problem with encryptKeyFile, specifically,
because I've eliminated general connectivity/authentication problems.
Thanks,
Jamie
On Wed, Nov 2, 2016 at 4:58 PM, Fuad Efendi wrote:
> In MySQL, this command will explicitly allow to connect from
> remote ICZ2002912 host, check MySQL documentation:
>
> GRANT ALL ON mysite.* TO 'root’@'ICZ2002912' IDENTIFIED BY ‘Oakton123’;
>
>
>
> On November 2, 2016 at 4:41:48 PM, Fuad Efendi (f...@efendi.ca) wrote:
>
> This is the root of the problem:
> "Access denied for user 'root'@'ICZ2002912' (using password: NO) “
>
>
> First of all, ensure that plain (non-encrypted) password settings work for
> you.
>
> Check that you can connect using MySQL client from ICZ2002912 to your
> MySQL & Co. instance
>
> I suspect you need to allow MySQL & Co. to accept connections
> from ICZ2002912. Plus, check DNS resolution, etc.
>
>
> Thanks,
>
>
> --
> Fuad Efendi
> (416) 993-2060
> http://www.tokenizer.ca
> Recommender Systems
>
>
> On November 2, 2016 at 2:37:08 PM, Jamie Jackson (jamieja...@gmail.com)
> wrote:
>
> I'm at a brick wall. Here's the latest status:
>
> Here are some sample commands that I'm using:
>
> *Create the encryptKeyFile and encrypted password:*
>
>
> encrypter_password='this_is_my_encrypter_password'
> plain_db_pw='Oakton153'
>
> cd /var/docker/solr_stage2/credentials/
> echo -n "${encrypter_password}" > encpwd.txt
> echo -n "${plain_db_pwd}" > plaindbpwd.txt
> openssl enc -aes-128-cbc -a -salt -in plaindbpwd.txt -k
> "${encrypter_password}"
>
> rm plaindbpwd.txt
>
> That generated this as the password, by the way:
>
> U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o=
>
> *Configure DIH configuration:*
>
>
>
> driver="org.mariadb.jdbc.Driver"
> url="jdbc:mysql://local.mysite.com:3306/mysite"
> user="root"
> password="U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o="
> encryptKeyFile="/opt/solr/credentials/encpwd.txt"
> />
> ...
>
>
> By the way, /var/docker/solr_stage2/credentials/ is mapped to
> /opt/solr/credentials/ in the docker container, so that's why the paths
> *seem* different (but aren't, really).
>
>
> *Authentication error when data import is run:*
>
> Exception while processing: question document :
> SolrInputDocument(fields:
> []):org.apache.solr.handler.dataimport.DataImportHandlerException:
> Unable to execute query: select 'foo' as bar; Processing
> Document # 1
> at org.apache.solr.handler.dataimport.DataImportHandlerException.
> wrapAndThrow(DataImportHandlerException.java:69)
> at org.apache.solr.handler.dataimport.JdbcDataSource$
> ResultSetIterator.(JdbcDataSource.java:323)
> at org.apache.solr.handler.dataimport.JdbcDataSource.
> getData(JdbcDataSource.java:283)
> at org.apache.solr.handler.dataimport.JdbcDataSource.
> getData(JdbcDataSource.java:52)
> at org.apache.solr.handler.dataimport.SqlEntityProcessor.
> initQuery(SqlEntityProcessor.java:59)
> at org.apache.solr.handler.dataimport.SqlEntityProcessor.
> nextRow(SqlEntityProcessor.java:73)
> at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(
> EntityProcessorWrapper.java:244)
> at org.apache.solr.handler.dataimport.DocBuilder.
> buildDocument(DocBuilder.java:475)
> at org.apache.solr.handler.dataimport.DocBuilder.
> buildDocument(DocBuilder.java:414)
> at org.apache.solr.handler.dataimport.DocBuilder.
> doFullDump(DocBuilder.java:329)
> at org.apache.solr.handler.dataimport.DocBuilder.execute(
> DocBuilder.java:232)
> at org.apache.solr.handler.dataimport.DataImporter.
> doFullImport(DataImporter.java:416)
> at org.apache.solr.handler.dataimport.DataImporter.
> runCmd(DataImporter.java:480)
> at org.apache.solr.handler.dataimport.DataImporter$1.run(
> DataImporter.java:461)
> Caused by: java.sql.SQLInvalidAuthorizationSpecException: Could not
> connect: Access denied for user 'root'@'ICZ2002912' (using password:
> NO)
> at org.mariadb.jdbc.internal.util.ExceptionMapper.get(
> ExceptionMapper.java:123)
> at org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(
> ExceptionMapper.java:71)
>
Re: Problem with Password Decryption in Data Import Handler
You were right, Fuad. There was a flaw in my script (inconsistent naming of
the `plain_db_pwd` variable.
Thanks for figuring that out.
For posterity, here's the fixed script:
encrypt_key=your_encryption_key
plain_db_pwd=your_db_password
cred_dir=/your/credentials/directory
cd "${cred_dir}
echo -n "${encrypt_key}" > encrypt.key
echo -n "${plain_db_pwd}" | openssl enc -aes-128-cbc -a -salt -k
"${encrypt_key}"
#==#
Then, in the DIH config:
encryptKeyFile="/your/credentials/directory/encrypt.key"
I have another, semi-related, issue that I'll bring up in another thread.
Thanks,
Jamie
On Wed, Nov 2, 2016 at 6:26 PM, Fuad Efendi wrote:
> Then I can only guess that in current configuration decrypted password is
> empty string.
>
> Try to manually replace some characters in encpwd.txt file to see if you
> get different errors; try to delete this file completely to see if you get
> different errors. Try to add new line in this file; try to change password
> in config file.
>
>
>
> On November 2, 2016 at 5:23:33 PM, Jamie Jackson (jamieja...@gmail.com)
> wrote:
>
> I should have mentioned that I verified connectivity with plain passwords:
>
> From the same machine that Solr's running on:
>
> solr@000650cbdd5e:/opt/solr$ mysql -uroot -pOakton153 -h local.mysite.com
> mysite -e "select 'foo' as bar;"
> +-+
> | bar |
> +-+
> | foo |
> +-+
>
> Also, if I add the plain-text password to the config, it connects fine:
>
> driver="org.mariadb.jdbc.Driver"
> url="jdbc:mysql://local.mysite.com:3306/mysite"
> user="root"
> password="Oakton153"
> />
>
>
> So that is why I claim to have a problem with encryptKeyFile, specifically,
> because I've eliminated general connectivity/authentication problems.
>
> Thanks,
> Jamie
>
> On Wed, Nov 2, 2016 at 4:58 PM, Fuad Efendi wrote:
>
> > In MySQL, this command will explicitly allow to connect from
> > remote ICZ2002912 host, check MySQL documentation:
> >
> > GRANT ALL ON mysite.* TO 'root’@'ICZ2002912' IDENTIFIED BY ‘Oakton123’;
> >
> >
> >
> > On November 2, 2016 at 4:41:48 PM, Fuad Efendi (f...@efendi.ca) wrote:
> >
> > This is the root of the problem:
> > "Access denied for user 'root'@'ICZ2002912' (using password: NO) “
> >
> >
> > First of all, ensure that plain (non-encrypted) password settings work
> for
> > you.
> >
> > Check that you can connect using MySQL client from ICZ2002912 to your
> > MySQL & Co. instance
> >
> > I suspect you need to allow MySQL & Co. to accept connections
> > from ICZ2002912. Plus, check DNS resolution, etc.
> >
> >
> > Thanks,
> >
> >
> > --
> > Fuad Efendi
> > (416) 993-2060
> > http://www.tokenizer.ca
> > Recommender Systems
> >
> >
> > On November 2, 2016 at 2:37:08 PM, Jamie Jackson (jamieja...@gmail.com)
> > wrote:
> >
> > I'm at a brick wall. Here's the latest status:
> >
> > Here are some sample commands that I'm using:
> >
> > *Create the encryptKeyFile and encrypted password:*
> >
> >
> > encrypter_password='this_is_my_encrypter_password'
> > plain_db_pw='Oakton153'
> >
> > cd /var/docker/solr_stage2/credentials/
> > echo -n "${encrypter_password}" > encpwd.txt
> > echo -n "${plain_db_pwd}" > plaindbpwd.txt
> > openssl enc -aes-128-cbc -a -salt -in plaindbpwd.txt -k
> > "${encrypter_password}"
> >
> > rm plaindbpwd.txt
> >
> > That generated this as the password, by the way:
> >
> > U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o=
> >
> > *Configure DIH configuration:*
> >
> >
> >
> > > driver="org.mariadb.jdbc.Driver"
> > url="jdbc:mysql://local.mysite.com:3306/mysite"
> > user="root"
> > password="U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o="
> > encryptKeyFile="/opt/solr/credentials/encpwd.txt"
> > />
> > ...
> >
> >
> > By the way, /var/docker/solr_stage2/credentials/ is mapped to
> > /opt/solr/credentials/ in the docker container, so that's why the paths
> > *seem* different (but aren't, really).
> >
> >
> > *Authe
Dates and DataImportHandler
Hi folks,
My DB server is on America/Chicago time. Solr (on Docker) is running on
UTC. Dates coming from my (MariaDB) data source seem to get translated
properly into the Solr index without me doing anything special.
However when doing delta imports using last_index_time (
http://wiki.apache.org/solr/DataImportHandlerDeltaQueryViaFullImport ), I
can't seem to get the date, which Solr provides, to be understood by the DB
as being UTC (and translated back, accordingly). In other words, the DB
thinks the Solr UTC date is local, so it thinks the date is ahead by six
hours.
'${dataimporter.request.clean}' != 'false'
or dt > '${dataimporter.last_index_time}'
I came up with this workaround, which seems to work:
'${dataimporter.request.clean}' != 'false'
/* ${user.timezone} is UTC, and the ${custom.dataimporter.datasource.tz}
property is set to America/Chicago */
or dt > CONVERT_TZ('${dataimporter.last_index_time}','${user.timezone}','${
custom.dataimporter.datasource.tz}')
However, isn't there a way for this translation to happen more naturally?
I thought maybe I could do something like this:
The above did set the property as expected (with a trailiing `+`), but
that didn't seem to help the DB understand/translate the date.
Thanks,
Jamie
Re: [bulk]: Dates and DataImportHandler
FWIW, I know mine worked, so maybe try:
I can't conceive of what the locale would possibly do when a dateFormat is
specified, so I omitted the attribute. (Maybe one can specify dateFormat
*or *locale--it seems like specifying both would cause a clash.) For what
it's worth, the format you're trying to write seems identical to the
default*, so I'm not sure what benefit you're getting by using that
propertyWriter.
*It's identical to *my* default, anyway. Maybe the default changes based on
one's system configuration, I don't know. This stuff isn't very well
documented.
On Tue, Sep 19, 2017 at 7:22 AM, Mannott, Birgit
wrote:
> Hi,
>
> I have a similar problem. I try to change the timezone for the
> last_index_time by setting
>
> type="SimplePropertiesWriter" locale="en_US" />
>
> in the section of my data-config.xml file.
>
> But when doing this I always get a NullPointerException on Delta Import:
>
> 2017-09-15 14:04:00.825 INFO (Thread-2938) [ x:mex_prd_dev1100-ap]
> o.a.s.h.d.DataImporter Starting Delta Import
> 2017-09-15 14:04:00.827 ERROR (Thread-2938) [ x:mex_prd_dev1100-ap]
> o.a.s.h.d.DataImporter Delta Import Failed
> org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to
> PropertyWriter implementation:SimplePropertiesWriter
> at org.apache.solr.handler.dataimport.DataImporter.
> createPropertyWriter(DataImporter.java:330)
> at org.apache.solr.handler.dataimport.DataImporter.
> doDeltaImport(DataImporter.java:439)
> at org.apache.solr.handler.dataimport.DataImporter.
> runCmd(DataImporter.java:476)
> at org.apache.solr.handler.dataimport.DataImporter.
> lambda$runAsync$0(DataImporter.java:457)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at java.text.SimpleDateFormat.(SimpleDateFormat.java:
> 598)
> at org.apache.solr.handler.dataimport.
> SimplePropertiesWriter.init(SimplePropertiesWriter.java:100)
> at org.apache.solr.handler.dataimport.DataImporter.
> createPropertyWriter(DataImporter.java:328)
> ... 4 more
>
> Has anyone an idea what is wrong or missing?
>
> Thanks,
> Birgit
>
>
>
> -Original Message-
> From: Jamie Jackson [mailto:jamieja...@gmail.com]
> Sent: Tuesday, September 19, 2017 3:42 AM
> To: solr-user@lucene.apache.org
> Subject: [bulk]: Dates and DataImportHandler
>
> Hi folks,
>
> My DB server is on America/Chicago time. Solr (on Docker) is running on
> UTC. Dates coming from my (MariaDB) data source seem to get translated
> properly into the Solr index without me doing anything special.
>
> However when doing delta imports using last_index_time (
> http://wiki.apache.org/solr/DataImportHandlerDeltaQueryViaFullImport ), I
> can't seem to get the date, which Solr provides, to be understood by the DB
> as being UTC (and translated back, accordingly). In other words, the DB
> thinks the Solr UTC date is local, so it thinks the date is ahead by six
> hours.
>
> '${dataimporter.request.clean}' != 'false'
>
> or dt > '${dataimporter.last_index_time}'
>
> I came up with this workaround, which seems to work:
>
> '${dataimporter.request.clean}' != 'false'
>
> /* ${user.timezone} is UTC, and the ${custom.dataimporter.datasource.tz}
> property is set to America/Chicago */
>
> or dt > CONVERT_TZ('${dataimporter.last_index_time}','${user.
> timezone}','${
> custom.dataimporter.datasource.tz}')
>
> However, isn't there a way for this translation to happen more naturally?
>
> I thought maybe I could do something like this:
>
>
> dateFormat="-MM-dd HH:mm:ssZ"
>
> type="SimplePropertiesWriter"
>
> />
>
> The above did set the property as expected (with a trailiing `+`), but
> that didn't seem to help the DB understand/translate the date.
>
> Thanks,
> Jamie
>
Re: [bulk]: Re: Dates and DataImportHandler
That's what I ended up doing too, for my MariaDB (MySQL derivative) DB:
CONVERT_TZ('${dataimporter.last_index_time}','${user.timezone}','${
custom.dataimporter.datasource.tz}')
user.timezone is Solr's time zone and custom.dataimporter.datasource.tz is
a property I set on startup.
The other option was to change Solr's timezone to match the DB, but I
thought that was even more of a hack.
On Thu, Sep 21, 2017 at 7:08 AM, Mannott, Birgit
wrote:
> As far as I understood, you can use the locale so that DIH saves the last
> index time for the given time zone and not for UTC. So if you set the
> locale according to the timezone of your DB you don't need to convert dates
> for comparison.
>
> But for me it's not working because every time I include settings for the
> propertyWriter (concrete data doesn't matter - I tried a lot), the null
> pointer exception appears. So I had to find a workaround and convert now
> the time saved by the DIH from UTC to the timezone of my DB when comparing
> for delta imports. But it's an ugly workaround I don't like.
>
> So, if anyone has an idea why this NPE occurs it would be great. Do I
> perhaps have to add something to solrconfig.xml?
>
> Thanks,
> Birgit
>
>
>
> -Original Message-
> From: Jamie Jackson [mailto:jamieja...@gmail.com]
> Sent: Tuesday, September 19, 2017 6:54 PM
> To: solr-user@lucene.apache.org
> Subject: [bulk]: Re: [bulk]: Dates and DataImportHandler
>
> FWIW, I know mine worked, so maybe try:
>
> type="SimplePropertiesWriter" />
>
> I can't conceive of what the locale would possibly do when a dateFormat is
> specified, so I omitted the attribute. (Maybe one can specify dateFormat
> *or *locale--it seems like specifying both would cause a clash.) For what
> it's worth, the format you're trying to write seems identical to the
> default*, so I'm not sure what benefit you're getting by using that
> propertyWriter.
>
> *It's identical to *my* default, anyway. Maybe the default changes based
> on one's system configuration, I don't know. This stuff isn't very well
> documented.
>
> On Tue, Sep 19, 2017 at 7:22 AM, Mannott, Birgit
> wrote:
>
> > Hi,
> >
> > I have a similar problem. I try to change the timezone for the
> > last_index_time by setting
> >
> > > type="SimplePropertiesWriter" locale="en_US" />
> >
> > in the section of my data-config.xml file.
> >
> > But when doing this I always get a NullPointerException on Delta Import:
> >
> > 2017-09-15 14:04:00.825 INFO (Thread-2938) [ x:mex_prd_dev1100-ap]
> > o.a.s.h.d.DataImporter Starting Delta Import
> > 2017-09-15 14:04:00.827 ERROR (Thread-2938) [ x:mex_prd_dev1100-ap]
> > o.a.s.h.d.DataImporter Delta Import Failed
> > org.apache.solr.handler.dataimport.DataImportHandlerException: Unable
> > to PropertyWriter implementation:SimplePropertiesWriter
> > at org.apache.solr.handler.dataimport.DataImporter.
> > createPropertyWriter(DataImporter.java:330)
> > at org.apache.solr.handler.dataimport.DataImporter.
> > doDeltaImport(DataImporter.java:439)
> > at org.apache.solr.handler.dataimport.DataImporter.
> > runCmd(DataImporter.java:476)
> > at org.apache.solr.handler.dataimport.DataImporter.
> > lambda$runAsync$0(DataImporter.java:457)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.NullPointerException
> > at java.text.SimpleDateFormat.(SimpleDateFormat.java:
> > 598)
> > at org.apache.solr.handler.dataimport.
> > SimplePropertiesWriter.init(SimplePropertiesWriter.java:100)
> > at org.apache.solr.handler.dataimport.DataImporter.
> > createPropertyWriter(DataImporter.java:328)
> > ... 4 more
> >
> > Has anyone an idea what is wrong or missing?
> >
> > Thanks,
> > Birgit
> >
> >
> >
> > -Original Message-
> > From: Jamie Jackson [mailto:jamieja...@gmail.com]
> > Sent: Tuesday, September 19, 2017 3:42 AM
> > To: solr-user@lucene.apache.org
> > Subject: [bulk]: Dates and DataImportHandler
> >
> > Hi folks,
> >
> > My DB server is on America/Chicago time. Solr (on Docker) is running
> > on UTC. Dates coming from my (MariaDB) data source seem to get
> > translated properly into the Solr index without me doing anything
> special.
> >
> > However when doing delta imports using last_index_time (
> > http://wiki.apache.org/solr/Data
Problem using user.timezone in DIH
Solr 6.6.0
I'm getting a weird issue and a misleading error when I try to use the
user.timezone property in a data import handler configuration which has
child entities:
Caused by: java.sql.SQLInvalidAuthorizationSpecException: Could not
connect: Access denied for user '{}'@'111.222.111.222' (using password: YES)
I was trying this in a query for a delta import:
CONVERT_TZ('${dataimporter.last_index_time}','${user.timezone}','${
custom.dataimporter.datasource.tz}')
It works great for cores with no child entities in the DIH config.
However, if I have any child entities, it dies. Here's a failing
configuration and its stack trace:
https://gist.github.com/jamiejackson/1ec544e6b87607098fbb1077a4eece30
I isolated it down to the use of user.timezone. Even if I simply add the
following in a SQL _comment_, it causes the problem. For example:
select /* ${user.timezone} */ column_a, column_b, ...
Bug?
Thanks,
Jamie
Re: How to restart solr in docker?
The usual procedure with containers is to restart or recreate the container. On Fri, Dec 15, 2017 at 11:55 AM, Buckler, Christine < christine.buck...@nordstrom.com> wrote: > What is the command for restarting solr on a docker image? I have modified > the solrconfig.xml to add the suggest plugin and now I want to update to > reflect this change. I was able to stop solr using “$ docker exec -it > my_solr stop all” but now I can’t figure out how to restart. > > Thanks, > Christine Buckler > >
Re: No space left on device - When I execute suggester component.
Have you checked disk consumption on the box yet?
On Dec 19, 2017 8:46 PM, "Fiz Newyorker" wrote:
> Hello Solr Experts,
>
> Good Evening !
>
> I have an issue , Whenever I run the Auto Suggest component ,
>
> http://lapp81.rno.com:8989/solr/Data/suggest?suggest=
> true&suggest.build=true&suggest.dictionary=fuzzySuggester&wt=json&
> suggest.q=Jerry
>
>
> I am getting the following Error .
>
> {"responseHeader":{"status":500,"QTime":341753},"error":{"msg":"No
> space left on device","trace":"java.io.IOException: No space left on
> device\n\tat sun.nio.ch.FileDispatcherImpl.write0(Native Method)\n\tat
> sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)\n\tat
> sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)\n\tat
> sun.nio.ch.IOUtil.write(IOUtil.java:65)\n\tat
> sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)\n\tat
> java.nio.channels.Channels.writeFullyImpl(Channels.java:78)\n\tat
> java.nio.channels.Channels.writeFully(Channels.java:101)\n\tat
> java.nio.channels.Channels.access$000(Channels.java:61)\n\tat
> java.nio.channels.Channels$1.write(Channels.java:174)\n\tat
> org.apache.lucene.store.FSDirectory$FSIndexOutput$1.
> write(FSDirectory.java:419)\n\tat
> java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)\n\tat
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)\
> n\tat
> java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)\n\tat
> org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(
> OutputStreamIndexOutput.java:53)\n\tat
> org.apache.lucene.util.OfflineSorter$ByteSequencesWriter.write(
> OfflineSorter.java:529)\n\tat
> org.apache.lucene.search.suggest.analyzing.AnalyzingSuggester.build(
> AnalyzingSuggester.java:480)\n\tat
> org.apache.lucene.search.suggest.Lookup.build(Lookup.java:190)\n\tat
> org.apache.solr.spelling.suggest.SolrSuggester.build(
> SolrSuggester.java:181)\n\tat
> org.apache.solr.handler.component.SuggestComponent.
> prepare(SuggestComponent.java:179)\n\tat
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(
> SearchHandler.java:269)\n\tat
> org.apache.solr.handler.RequestHandlerBase.handleRequest(
> RequestHandlerBase.java:173)\n\tat
> org.apache.solr.core.SolrCore.execute(SolrCore.java:2440)\n\tat
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)\n\tat
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)\n\tat
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:347)\n\tat
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:298)\n\tat
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1691)\n\tat
> org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:582)\n\tat
> org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)\n\tat
> org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:548)\n\tat
> org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:226)\n\tat
> org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1180)\n\tat
> org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:512)\n\tat
> org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)\n\tat
> org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1112)\n\tat
> org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)\n\tat
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
> ContextHandlerCollection.java:213)\n\tat
> org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:119)\n\tat
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:134)\n\tat
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(
> RewriteHandler.java:335)\n\tat
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:134)\n\tat
> org.eclipse.jetty.server.Server.handle(Server.java:534)\n\tat
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)\n\tat
> org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:251)\n\tat
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:273)\n\tat
> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)\n\tat
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
> SelectChannelEndPoint.java:93)\n\tat
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> executeProduceConsume(ExecuteProduceConsume.java:303)\n\tat
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceConsume(ExecuteProduceConsume.java:148)\n\tat
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(
> ExecuteProduceConsume.java:136)\n\tat
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:671)\n\tat
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
> QueuedThr
Scheduling Data Import Handler (DIH) Delta Imports
Hi Folks, I have DIH cores that are being indexed by my Lucee application. That works, but I'd like to make some improvements: - Make a standalone scheduler that's not part of a larger application. (FYI, I want to Dockerize the import-triggering service.) - Prevent import requests from stacking up. Some of my cores' delta imports run every 15 seconds, and they do so blindly/ignorantly. If there is contention, very occasionally, import jobs will run long and stack up, so I want to make the scheduler nicer/more intelligent. Maybe the service would check the import status to see if there's a job already running before requesting a new one. I can write such a thing myself, but does anybody have a Linux or cross-platform solution written already? Thanks, Jamie
Re: Scheduling Data Import Handler (DIH) Delta Imports
One last bump before I get crackin'... On Mon, Nov 21, 2016 at 11:54 AM, Jamie Jackson wrote: > Hi Folks, > > I have DIH cores that are being indexed by my Lucee application. That > works, but I'd like to make some improvements: > > >- Make a standalone scheduler that's not part of a larger application. >(FYI, I want to Dockerize the import-triggering service.) >- Prevent import requests from stacking up. Some of my cores' delta >imports run every 15 seconds, and they do so blindly/ignorantly. If there >is contention, very occasionally, import jobs will run long and stack up, >so I want to make the scheduler nicer/more intelligent. Maybe the service >would check the import status to see if there's a job already running >before requesting a new one. > > I can write such a thing myself, but does anybody have a Linux or > cross-platform solution written already? > > Thanks, > Jamie >
Solrj: Cannot retry request with a non-repeatable request entity.
Please help me get over this obstacle: Stack trace and (ColdFusion) snippet: https://gist.github.com/jamiejackson (This is an attempt to get Solrj to perform a deleteByQuery with ConcurrentUpdateSolrServer using basic authentication. I am a noob with the entire Solrj and HTTPClient stack. ) Please let me know if you need more info. FWIW, here's a message I got on the HttpClient mailing list: Jamie This question should have probably been addressed to Solrj community. >From the HC standpoint you have three options (1) make request entity repeatable (2) force authentication by doing a GET prior to doing PUT or POST (3) use 'expect-continue' handshake on the client side if the handshake is supported by the server side. Oleg You'll notice that I tried to set "expect-continue" (commented out in my snipppet), but that didn't seem to work. Thanks, Jamie
