It is kept in zookeeper within /configs/[collection_name], at least with my
SolrCloud 6.6.6.
bin/solr zk ls /configs/[your_collection_name]
Regards
Bernd
Am 08.09.20 um 21:40 schrieb yaswanth kumar:
> Can someone help me on how to persists the data that's updated in
> dataimport
Can someone help me on how to persists the data that's updated in
dataimport.properties file because it got a last index time so that my data
import depends on it for catching up the delta imports.
What I noticed is that every time when I restart solr this file is wiped
out and gettin
I figured it out. I use SQL Server, so this is my solution :
In TSQL, this can be converted to a UTC date time using :
CONVERT(datetimeoffset, '${dih.last_index_time}', 127)
Refs:
http://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html
http://msdn.microsoft.com/en-us/library/
Thank you for the response. This works if I invoke start.jar with java. In
my usecase however, I need to invoke start.jar directly (consoleless
service so that the user cannot close it accidentally). It doesnt pickup
user.timezone property when done this way. Is it possible to do this using
the tag
On 26 March 2014 02:44, Kiran J wrote:
>
> Hi
>
> Is it possible to set up the data import handler so that it keeps track of
> the last imported time in Zulu time and not local time ?
[...]
Start your JVM with the desired timezone, e.g.,
java -Duser.timezone=UTC -jar start.jar
Regards,
Gora
Hi
Is it possible to set up the data import handler so that it keeps track of
the last imported time in Zulu time and not local time ?
Its not very clear from the documentation how to do it or if it is even
possible to do it.
Ref:
http://wiki.apache.org/solr/DataImportHandler#Configuring_The_Pr
kody <
> > dileepajayak...@gmail.com> wrote:
> > Hi All,
> >
> > Can I please know what timestamp in the dataimport process is reordered
> as
> > the last_index_time in dataimport.properties?
> >
> > Is it the time that the last dataimport process started ?
> > OR
> > Is it the time that the last dataimport process finished?
> >
> >
> > Thanks,
> > Dileepa
> >
> >
>
>
ring updated documenets during the import.
>
> Ahmet
>
>
>
> On Sunday, January 26, 2014 9:10 PM, Dileepa Jayakody <
> dileepajayak...@gmail.com> wrote:
> Hi All,
>
> Can I please know what timestamp in the dataimport process is reordered as
> the last_index_tim
e:
> Hi All,
>
> Can I please know what timestamp in the dataimport process is reordered as
> the last_index_time in dataimport.properties?
>
> Is it the time that the last dataimport process started ?
> OR
> Is it the time that the last dataimport process finished?
>
>
> Thanks,
> Dileepa
>
>
reordered as
the last_index_time in dataimport.properties?
Is it the time that the last dataimport process started ?
OR
Is it the time that the last dataimport process finished?
Thanks,
Dileepa
Hi All,
Can I please know what timestamp in the dataimport process is reordered as
the last_index_time in dataimport.properties?
Is it the time that the last dataimport process started ?
OR
Is it the time that the last dataimport process finished?
Thanks,
Dileepa
It's last_index_time which is written after data import is finished
successfully. In case of an error, the file is unchanged and nothing is
updated.
-
Sumit Arora
--
View this message in context:
http://lucene.472066.n3.nabble.com/dataimport-properties-files-tp484p4111332.html
Sent from
Hi all,
When the information regarding last_import_time is written into
dataimport.properties file?
Is it at the start before actual indexing start or at the end?
If it is at the start than in cases where dataimport fails will the
dataimport.properties file will be rolled back to its last state
Hi,
Im using solr 4.4 cloud setup with external zookeeper and solr running in
tomcat.
For the initial indexing, i use csv files to load data to solr. Then we do
delta indexing from database table.
My zookeeper always thorws exception saying "could not read
dataimport.properties.
Where shou
- Is dataimport.properties ever written to the filesystem? (Trying to
determine if I have a permissions error because I don't see it anywhere
on disk).
- How do you manually edit dataimport.properties? My system is
periodically pulling in new data. If that process has issues, I want to
be
in Zookeeper, ie: no path and just
'dataimport.properties', unless it is in a subdir of your config, then
'/dataimport.properties'.
I have a deployment system template the properties file before it is
inserted into Zookeeper.
Tim
On 03/04/13 08:48 PM, Nathan Findley wrote:
- Is dataimport.properties ever written to the filesystem? (Trying to
determine if I have a permissions error because I don't see it anywhere
on disk).
- How do you manually edit dataimport.properties? My system is
periodically pulling in new data. If that process has issues, I want to
be
Someone with more zookeeper knowledge than I have can better answer this, but
there is code in place specifically for using DIH with SolrCloud to save the
"dataimport.properties" file in an appropriate place. The default path is:
/configs/{collection}/dataimport.properties I'm
Well, I saw, that when I ran the full/delta i,port process on the 2nd, 3rd
etc times I didn't so this exception any more. So I checked in my mysql
queries log what's going on in mysql when I was running delta import process
and I saw, that the queries got correct times on each delta-import
executio
ction/dataimport.properties)
zookeeper tries to access file in undefined path.
How can I tell zookeeper to use $home/solr/my_collection/conf/ folder for
creating data import.properties?
--
View this message in context:
http://lucene.472066.n3.nabble.com/dataimport-properties-not-created-updated-with-solr
When using SolrCloud, the dataimport.properties file goes to a different
location. See https://issues.apache.org/jira/browse/SOLR-3165 for more
information.
Also, while this feature works in 4.0.0, it is currently broken in
(not-released) 4.1 ("branch_4x") and the development Tr
Hi,
I have a problem with updating dataimport.properties - while running single
sold there is no problem at all. Everything works perfectly. But when I
switching to cloud configuration with 2 shards (like described in
http://wiki.apache.org/solr/SolrCloud ExampleA: Simple two shard cluster)
this
Hi,
We are working on a DIH for our project and we are persisting the
last_modified_date in the ZooKeeper directory. Our understanding is that the
properties are uploaded to ZooKeeper when the first SOLR node comes up. When
the SOLR nodes are restarted whatever is persisted in the properties is
lo
Hi everybody，I am a freshman here，hope to make friends with you
thank you so much travis, it is working so smoothly right now
-
Zeki ama calismiyor... Calissa yapar...
--
View this message in context:
http://lucene.472066.n3.nabble.com/SOLR-4-0-DataImport-frozen-or-fails-with-WARNING-Una
thank you so much travis, it is working so smoothly right now
-
Zeki ama calismiyor... Calissa yapar...
--
View this message in context:
http://lucene.472066.n3.nabble.com/SOLR-4-0-DataImport-frozen-or-fails-with-WARNING-Unable-to-read-dataimport-properties-tp4006077p4006506.html
Sent from
app=/solr path=/dataimport params={command=status}
> status=0 QTime=0
> Sep 07, 2012 4:08:21 PM
> org.apache.solr.handler.dataimport.SimplePropertiesWriter
> readIndexerProperties
> *WARNING: Unable to read: dataimport.properties*
> Sep 07, 2012 4:08:21 PM org.apache.solr.handler.dataimpor
21 PM
org.apache.solr.handler.dataimport.SimplePropertiesWriter
readIndexerProperties
*WARNING: Unable to read: dataimport.properties*
Sep 07, 2012 4:08:21 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
call
INFO: Creating a connection for entity user with URL:
jdbc:mysql://10.60.1.157:3396/poppen
Sep 07, 2012 4:08:
dataimport.properties :
interval=1
port=
server=localhost
doc.id=
params=/select?qt\=/dataimport&command\=delta-import&clean\=false&commit\=true
webapp=solr
syncEnabled=1
last_index_time=2012-06-27 13\:05\:18
doc.last_index_time=2012-06-27 13\:05\:18
syncCores=
how to set thes
i am not find where to set dataimport.properties in solr4.0 beta
who can help me ?
--
View this message in context:
http://lucene.472066.n3.nabble.com/in-solr4-0-where-to-set-dataimport-properties-tp4002766.html
Sent from the Solr - User mailing list archive at Nabble.com.
with the warning in the title of your message.
That is very likely because the user running DIH (typically the Jetty/
tomcat user) does not have permissions to read/write the
dataimport.properties file in your Solr conf/ directory
The relevant error in your log is the following one:
> *SEVER
Yeah, that happens. Glad you're past this issue thanks for closing it out.
Erick
On Thu, Jan 26, 2012 at 10:45 AM, Egonsith wrote:
> Erik,
>
> Thanks for the reply,
> i a bit embarres to say this is a clasiic example of a way to messy
> development enviroment and these erros were due to many dif
Erik,
Thanks for the reply,
i a bit embarres to say this is a clasiic example of a way to messy
development enviroment and these erros were due to many diffrent drivers and
xml files that were edited way to many times. i have cleaned my dev
enviromant and reinstalled tomcat and solr and am now g
r.dataimport.DataImporter
> loadDataConfig
> INFO: Data Configuration loaded successfully
> Jan 25, 2012 2:17:37 PM org.apache.solr.handler.dataimport.DataImporter
> doFullImport
> INFO: Starting Full Import
> Jan 25, 2012 2:17:37 PM org.apache.solr.handler.dataimport.SolrWriter
>
org.apache.solr.handler.dataimport.DataImporter
doFullImport
INFO: Starting Full Import
Jan 25, 2012 2:17:37 PM org.apache.solr.handler.dataimport.SolrWriter
readIndexerProperties
*WARNING: Unable to read: dataimport.properties*
Jan 25, 2012 2:17:37 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
call
INFO: Creating a connection for
You can add it yourself in admin-extra.html
Ephraim Ofir
-Original Message-
From: Nico Luna [mailto:nicolaslun...@gmail.com]
Sent: Friday, November 11, 2011 7:57 PM
To: solr-user@lucene.apache.org
Subject: How to read values from dataimport.properties in a production
environment
I&#
I'm trying to know the values stored in the dataimport.properties file in a
production environment using the solr admin feature, so I copied the same
behaviour as:
[Schema] and [Config] but changing the contentType property (from
contentType=text/xml to contentType=text):
http://localhost
are constantly updated tables, but my issue is not the sql
error but the fact that the delta import still reported that it had imported
successfully and still wrote out the last updated time to the
dataimport.properties file, so the next time it ran it missed a bunch of
documents that should have
are constantly updated tables, but my issue is not the sql
error but the fact that the delta import still reported that it had imported
successfully and still wrote out the last updated time to the
dataimport.properties file, so the next time it ran it missed a bunch of
documents that should have
Title pretty much says it all; I've configured the DIH in 3.1.0, and
it works great, except the delta-imports are always from the last time
a full-import happened, not a delta-import. After a delta-import,
dataimport.properties is completely untouched. The documentation
implies that the
java -Duser.timezone=UTC -jar start.jar ?
On Tue, Apr 26, 2011 at 9:54 AM, stockii wrote:
> Hello.
>
> How can i set the timezone oft java in my java properties ?
>
> my problem is, that in the dataimport-properties is a wrong timezone and i
> dont know how to set the correct timezone ... !?!? th
Hello.
How can i set the timezone oft java in my java properties ?
my problem is, that in the dataimport-properties is a wrong timezone and i
dont know how to set the correct timezone ... !?!? thx
-
--- System
One Server,
You have to explicitly call
http://:/solr/dataimport?command=delta-import to start
delta-import.
You can setup it as cronjob.
As per my knowledge dataimport.properties file contains parameter to be used
in DtataImport xml configuration file for example database related file
contains text as
Hi,
I am using solr 1.4 and was wondering if the dataimport.properties location
can be configured for DataImportHandler. Is there a config file or a
property that I can use to specify a custom location ?
Thanks,
Manali
I've finally found the problem causing the delta-import to fail and thought I
would post it here for future reference (if someone makes the same mistake I
did).
I had forgot to collect the "id" column in the deltaImportQuery. I should,
of course, have known this from the log entires about documen
Please note that Oracle ( or Oracle jdbc driver ) converts column
names to upper case eventhough you state them in lower case. If this
is the case then try to rewrite your query in the following form
select id as "id", name as "name" from table
On Thursday, June 24, 2010, warb wrote:
>
> Hello ag
Is there any chance that the "id" field is, indeed, missing for those
documents?
Does your schema require ID? I've also seen constraints added to a DB
that are not retro-active, so even if there is a constraint requiring ID
it's
still possible that some items in your DB don't have them.
A shot in
Hello again!
Upon further investigation it seems that something is amiss with
delta-import after all, the delta-import does not actually import anything
(I thought it did when I ran it previously but I am not sure that was the
case any longer.) It does complete successfully as seen from the front
Hi,
what I have experienced is that the primary key seems to be case
sensitive for the delta queries, at least for some jdcd drivers... see
http://lucene.472066.n3.nabble.com/Problem-with-DIH-delta-import-on-JDBC-tp763469p765262.html
... so make sure you specify it with the correct case (e.g.
Hello!
I am having some difficulties getting dataimport (DIH) to behave correctly
in Solr 1.4.0. Indexing itself works just as it is supposed to with both
full-import and delta-import adding modified or newly created records to the
index. The problem is however that the date and time of the last
On 4/20/2010 9:09 PM, caman wrote:
Shawn,
Is this your custom implementation?
"For a delta-import, minDid comes from
the maxDid value stored after the last successful import."
Are you updating the dataTable after the import was successful? How did you
handle this? I have similar scenario and y
On 4/18/2010 9:02 PM, Michael Tibben wrote:
I don't really understand how this will help. Can you elaborate ?
Do you mean that the last_index_time can be imported from somewhere
outside solr? But I need to be able to *set* what last_index_time is
stored in dataimport.properties, not get p
will work for my
use-case as well
thanks
From: Shawn Heisey-4 [via Lucene]
[mailto:ml-node+738653-1765413222-124...@n3.nabble.com]
Sent: Tuesday, April 20, 2010 4:35 PM
To: caman
Subject: Re: DIH dataimport.properties with
Michael,
The SolrEntityProcessor looks very
lta-import needs to be run. My query and
> deltaImportQuery are identical, though yours may not be.
>
> Good luck, no matter how you choose to approach this.
>
> Shawn
>
>
> On 4/18/2010 9:02 PM, Michael Tibben wrote:
>> I don't really understand how this will hel
somewhere
outside solr? But I need to be able to *set* what last_index_time is
stored in dataimport.properties, not get properties from somewhere else
On 18/04/10 10:02, Lance Norskog wrote:
The SolrEntityProcessor allows you to query a Solr instance and use
the results as DIH properties. You would have to create your own
regular query to do the delta-import instead of using the delta-import
feature.
I don't really understand how this will help. Can you elaborate ?
Do you mean that the last_index_time can be imported from somewhere
outside solr? But I need to be able to *set* what last_index_time is
stored in dataimport.properties, not get properties from somewhere else
On 18/04/
http://sematext.com/ :: Solr - Lucene - Nutch
Hadoop ecosystem search :: http://search-hadoop.com/
- Original Message
From: Michael Tibben
To: solr-user@lucene.apache.org
Sent: Thu, April 15, 2010 10:15:14 PM
Subject: DIH dataimport.properties with
Hi,
I am using the DIH to i
r@lucene.apache.org
>> Sent: Thu, April 15, 2010 10:15:14 PM
>> Subject: DIH dataimport.properties with
>>
>> Hi,
>
> I am using the DIH to import data from a mysql slave. However, the
>> slave sometimes runs behind the master. The delay is variable, most
:15:14 PM
> Subject: DIH dataimport.properties with
>
> Hi,
I am using the DIH to import data from a mysql slave. However, the
> slave sometimes runs behind the master. The delay is variable, most of the
> time
> it is in sync, but sometimes can run behind by a few minutes.
Th
Hi,
I am using the DIH to import data from a mysql slave. However, the slave
sometimes runs behind the master. The delay is variable, most of the
time it is in sync, but sometimes can run behind by a few minutes.
This is a problem, because DIH uses dataimport.properties to determine
the
On Sat, May 30, 2009 at 3:58 AM, wojtekpia wrote:
>
> I'd like to include a data version in my index, and it looks like
> dataimport.properties would be a nice place for it. Is there a way to add a
> custom name-value pair to that file?
>
No, I don't think so. We coul
Hi,
I'd like to include a data version in my index, and it looks like
dataimport.properties would be a nice place for it. Is there a way to add a
custom name-value pair to that file?
Thanks,
Wojtek
--
View this message in context:
http://www.nabble.com/Custom-Values-in-dataimport.prope
On Wed, May 20, 2009 at 11:44 PM, Wesley Small wrote:
> Is a place in a core's solrconfig, where one can set the directory/path
> where the dataimport.properties file is written to?
>
It is not configurable right now. Can you please open a jira issue for this?
--
Regards,
Shalin Shekhar Mangar.
Is a place in a core's solrconfig, where one can set the directory/path
where the dataimport.properties file is written to?
On 5/20/09 2:09 PM, "Giovanni De Stefano"
wrote:
> Doh,
>
> can you please rephrase?
>
> Giovanni
>
> On Wed, May 20, 2009 at 3:4
Doh,
can you please rephrase?
Giovanni
On Wed, May 20, 2009 at 3:47 PM, Wesley Small wrote:
> In Solr 1.3, is there a setting that allows one to modified the where the
> dataimport.properties file resides?
>
> In a production environment, the solrconfig directory needs to be
>
In Solr 1.3, is there a setting that allows one to modified the where the
dataimport.properties file resides?
In a production environment, the solrconfig directory needs to be read-only.
I have observed that the DIH process works regards, but a whooping errors is
put in the logs when the
that mean and it even kill my dataimport.properties (empty file), so
> nothing was write in this file then the delta-import, started to import
> from the very start I guess.
>
> Thanks a lot for your help,
> I wish you guys a lovely day,
>
>
>
> there is the error:
>
Hi,
Last night I've got an error during the importation and I don't get what
does that mean and it even kill my dataimport.properties (empty file), so
nothing was write in this file then the delta-import, started to import from
the very start I guess.
Thanks a lot for your help,
67 matches
Mail list logo
