Hello Everyone,
I'm trying to work out how, if at all possible, dynamicFields can be
imported from a dynamic data source through the DataImportHandler
configurations. Currently the DataImportHandler configuration file
requires me to name every single field I want to map in advance, but I do
not know the dynamicField set at this stage necessarily.
Here's my example schema.xml dynamic field definition:
<dynamicField name="*_sortable" type="alphaOnlySort" indexed="true"
stored="true"/>
My DataImportHandler import configuration file looks like this:
<dataSource name="Gateway1Source" type="HttpDataSource" baseUrl="
http://acproplatforms.internal/feeds.xml"; encoding="UTF-8"
connectionTimeout="15000" readTimeout="15000"/>
<document name="feeds">
<entity name="feed" processor="XPathEntityProcessor"
stream="true" forEach="/gateway/feedItem/" url="">
<field column="type" xpath="/gateway/feedItem/type"/>
...
</entity>
</document>
</dataConfig>
I have looked, very optimistically, at Script Transformers
(transformer="script:importDynamics"), specifically hoping the row in the
transformer function would hold the dynamic field content, but this was
silly thinking obviously, as they would already fall through had they made
it into here.
Has anyone managed to import into dynamic fields in advance of knowing
what they were going to be in the data source?
To give you an idea of why I want this, there's an application aggregating
web services from many sources, some of which contain patterns of fields I
know we'll want, and the nature of their data types, but which are added
to quite frequently. It seems aside from the field mappings here, the hard
work has been done in Solr to achieve this!
Kindest Regards,
Mark
From: Shawn Heisey <s...@elyograg.org>
To: solr-user@lucene.apache.org,
Date: 08/03/2012 14:58
Subject: Re: Understanding update handler statistics
On 3/8/2012 7:02 AM, stetogias wrote:
> Hi,
>
> Trying to understand the update handler statistics
> so I have this:
>
> commits : 2824
> autocommit maxDocs : 10000
> autocommit maxTime : 1000ms
> autocommits : 41
> optimizes : 822
> rollbacks : 0
> expungeDeletes : 0
> docsPending : 0
> adds : 0
> deletesById : 0
> deletesByQuery : 0
> errors : 0
> cumulative_adds : 17457
> cumulative_deletesById : 1959
> cumulative_deletesByQuery : 0
> cumulative_errors : 0
>
> my problem is with the cumulative part.
>
> If for instance I am doing a commit after each add and delete operation
then
> the sum of cumulative_adds plus
> cumulative_deletes plus cumulative_errors should much the commit number.
> is that right?
> And another question, these stats are since SOLR instance startup or
since
> update handler startup, these
> can differ as far as I understand...
>
> and from this part:
> docsPending : 0
> adds : 0
> deletesById : 0
> deletesByQuery : 0
> errors : 0
>
> I understand that if I had docsPending I should have adds(pending)
> deletes*(pending) but how could I have errors...
I'm fairly sure that adds and deletes refer to the number of documents
added or deleted. You can have many documents added and/or deleted for
each commit. I would not expect the sums to match, unless you are
adding or deleting only one document at a time and doing a commit after
every one. I hope you're not doing that, unless you're using trunk with
the near-realtime feature and doing soft commits, with which I have no
experience. Normally doing a commit after every document would be too
much of a load for good performance, unless there is a relatively long
time period between each add or delete.
Your question about errors - that probably tracks the number of times
that the update handler returned an error response, though I don't
really know. If I'm right, then that number, like commits, has little
to do with the number of documents.
Thanks,
Shawn
