Do you have a _version_ field in your schema? I actually just came back to
this thread with that thought and then saw your error - so that remains my
guess.
I'm going to improve the doc on the wiki around what needs to be defined
for SolrCloud - so far we have things in the example defaults, but its not
clear enough to users what needs to be there if they are using an old
schema or modifying the example.
<field name="_version_" type="long" indexed="true" stored="true"/>
If you do have a _version_ field, there is something to track down here for
sure.
On Wed, Feb 29, 2012 at 1:15 PM, Matthew Parker <
mpar...@apogeeintegration.com> wrote:
> Mark/Sami
>
> I ran the system with 3 zookeeper nodes, 2 solr cloud nodes, and left
> numShards set to its default value (i.e. 1)
>
> I looks like it finally sync'd with the other one after quite a while, but
> it's throwing lots of errors like the following:
>
> org.apache.solr.common.SolrException: missing _version_ on update from
> leader at
> org.apache.solr.update.processor.DistributtedUpdateProcessor.versionDelete(
> DistributedUpdateProcessor.java: 712)
> ....
> ....
> ....
>
> Is it normal to sync long after the documents were sent for indexing?
>
> I'll have to check and see whether the 4 solr node instance with 2 shards
> works after waiting for the system to sync.
>
> Regards,
>
> Matt
>
> On Wed, Feb 29, 2012 at 12:03 PM, Matthew Parker <
> mpar...@apogeeintegration.com> wrote:
>
> > I also took out my requestHandler and used the standard /update/extract
> > handler. Same result.
> >
> > On Wed, Feb 29, 2012 at 11:47 AM, Matthew Parker <
> > mpar...@apogeeintegration.com> wrote:
> >
> >> I tried running SOLR Cloud with the default number of shards (i.e. 1),
> >> and I get the same results.
> >>
> >> On Wed, Feb 29, 2012 at 10:46 AM, Matthew Parker <
> >> mpar...@apogeeintegration.com> wrote:
> >>
> >>> Mark,
> >>>
> >>> Nothing appears to be wrong in the logs. I wiped the indexes and
> >>> imported 37 files from SharePoint using Manifold. All 37 make it in,
> but
> >>> SOLR still has issues with the results being inconsistent.
> >>>
> >>> Let me run my setup by you, and see whether that is the issue?
> >>>
> >>> On one machine, I have three zookeeper instances, four solr instances,
> >>> and a data directory for solr and zookeeper config data.
> >>>
> >>> Step 1. I modified each zoo.xml configuration file to have:
> >>>
> >>> Zookeeper 1 - Create /zookeeper1/conf/zoo.cfg
> >>> ================
> >>> tickTime=2000
> >>> initLimit=10
> >>> syncLimit=5
> >>> dataDir=[DATA_DIRECTORY]/zk1_data
> >>> clientPort=2181
> >>> server.1=localhost:2888:3888
> >>> server.2=localhost:2889:3889
> >>> server.3=localhost:2890:3890
> >>>
> >>> Zookeeper 1 - Create /[DATA_DIRECTORY]/zk1_data/myid with the following
> >>> contents:
> >>> ==============================================================
> >>> 1
> >>>
> >>> Zookeep 2 - Create /zookeeper2/conf/zoo.cfg
> >>> ==============
> >>> tickTime=2000
> >>> initLimit=10
> >>> syncLimit=5
> >>> dataDir=[DATA_DIRECTORY]/zk2_data
> >>> clientPort=2182
> >>> server.1=localhost:2888:3888
> >>> server.2=localhost:2889:3889
> >>> server.3=localhost:2890:3890
> >>>
> >>> Zookeeper 2 - Create /[DATA_DIRECTORY]/zk2_data/myid with the following
> >>> contents:
> >>> ==============================================================
> >>> 2
> >>>
> >>> Zookeeper 3 - Create /zookeeper3/conf/zoo.cfg
> >>> ================
> >>> tickTime=2000
> >>> initLimit=10
> >>> syncLimit=5
> >>> dataDir=[DATA_DIRECTORY]/zk3_data
> >>> clientPort=2183
> >>> server.1=localhost:2888:3888
> >>> server.2=localhost:2889:3889
> >>> server.3=localhost:2890:3890
> >>>
> >>> Zookeeper 3 - Create /[DATA_DIRECTORY]/zk3_data/myid with the following
> >>> contents:
> >>> ====================================================
> >>> 3
> >>>
> >>> Step 2 - SOLR Build
> >>> ===============
> >>>
> >>> I pulled the latest SOLR trunk down. I built it with the following
> >>> commands:
> >>>
> >>> ant example dist
> >>>
> >>> I modified the solr.war files and added the solr cell and extraction
> >>> libraries to WEB-INF/lib. I couldn't get the extraction to work
> >>> any other way. Will zookeper pickup jar files stored with the rest of
> >>> the configuration files in Zookeeper?
> >>>
> >>> I copied the contents of the example directory to each of my SOLR
> >>> directories.
> >>>
> >>> Step 3 - Starting Zookeeper instances
> >>> ===========================
> >>>
> >>> I ran the following commands to start the zookeeper instances:
> >>>
> >>> start .\zookeeper1\bin\zkServer.cmd
> >>> start .\zookeeper2\bin\zkServer.cmd
> >>> start .\zookeeper3\bin\zkServer.cmd
> >>>
> >>> Step 4 - Start Main SOLR instance
> >>> ==========================
> >>> I ran the following command to start the main SOLR instance
> >>>
> >>> java -Djetty.port=8081 -Dhostport=8081
> >>> -Dbootstrap_configdir=[DATA_DIRECTORY]/solr/conf -Dnumshards=2
> >>> -Dzkhost=localhost:2181,localhost:2182,localhost:2183 -jar start.jar
> >>>
> >>> Starts up fine.
> >>>
> >>> Step 5 - Start the Remaining 3 SOLR Instances
> >>> ==================================
> >>> I ran the following commands to start the other 3 instances from their
> >>> home directories:
> >>>
> >>> java -Djetty.port=8082 -Dhostport=8082
> >>> -Dzkhost=localhost:2181,localhost:2182,localhost:2183 -jar start.jar
> >>>
> >>> java -Djetty.port=8083 -Dhostport=8083
> >>> -Dzkhost=localhost:2181,localhost:2182,localhost:2183 -jar start.jar
> >>>
> >>> java -Djetty.port=8084 -Dhostport=8084
> >>> -Dzkhost=localhost:2181,localhost:2182,localhost:2183 -jar start.jar
> >>>
> >>> All startup without issue.
> >>>
> >>> Step 6 - Modified solrconfig.xml to have a custom request handler
> >>> ===============================================
> >>>
> >>> <requestHandler name="/update/sharepoint" startup="lazy"
> >>> class="solr.extraction.ExtractingRequestHandler">
> >>> <lst name="defaults">
> >>> <str name="update.chain">sharepoint-pipeline</str>
> >>> <str name="fmap.content">text</str>
> >>> <str name="lowernames">true</str>
> >>> <str name="uprefix">ignored</str>
> >>> <str name="caputreAttr">true</str>
> >>> <str name="fmap.a">links</str>
> >>> <str name="fmap.div">ignored</str>
> >>> </lst>
> >>> </requestHandler>
> >>>
> >>> <updateRequestProcessorChain name="sharepoint-pipeline">
> >>> <processor class="solr.processor.SignatureUpdateProcessorFactory">
> >>> <bool name="enabled">true</bool>
> >>> <str name="signatureField">id</str>
> >>> <bool name="owerrightDupes">true</bool>
> >>> <str name="fields">url</str>
> >>> <str name="signatureClass">solr.processor.Lookup3Signature</str>
> >>> </processor>
> >>> <processor class="solr.LogUpdateProcessorFactory"/>
> >>> <processor class="solr.RunUpdateProcessorFactory"/>
> >>> </updateRequestProcessorChain>
> >>>
> >>>
> >>> Hopefully this will shed some light on why my configuration is having
> >>> issues.
> >>>
> >>> Thanks for your help.
> >>>
> >>> Matt
> >>>
> >>>
> >>>
> >>> On Tue, Feb 28, 2012 at 8:29 PM, Mark Miller <markrmil...@gmail.com
> >wrote:
> >>>
> >>>> Hmm...this is very strange - there is nothing interesting in any of
> the
> >>>> logs?
> >>>>
> >>>> In clusterstate.json, all of the shards have an active state?
> >>>>
> >>>>
> >>>> There are quite a few of us doing exactly this setup recently, so
> there
> >>>> must be something we are missing here...
> >>>>
> >>>> Any info you can offer might help.
> >>>>
> >>>> - Mark
> >>>>
> >>>> On Feb 28, 2012, at 1:00 PM, Matthew Parker wrote:
> >>>>
> >>>> > Mark,
> >>>> >
> >>>> > I got the codebase from the 2/26/2012, and I got the same
> inconsistent
> >>>> > results.
> >>>> >
> >>>> > I have solr running on four ports 8081-8084
> >>>> >
> >>>> > 8081 and 8082 are the leaders for shard 1, and shard 2, respectively
> >>>> >
> >>>> > 8083 - is assigned to shard 1
> >>>> > 8084 - is assigned to shard 2
> >>>> >
> >>>> > queries come in and sometime it seems the windows from 8081 and 8083
> >>>> move
> >>>> > responding to the query but there are no results.
> >>>> >
> >>>> > if the queries run on 8081/8082 or 8081/8084 then results come back
> >>>> ok.
> >>>> >
> >>>> > The query is nothing more than: q=*:*
> >>>> >
> >>>> > Regards,
> >>>> >
> >>>> > Matt
> >>>> >
> >>>> >
> >>>> > On Mon, Feb 27, 2012 at 9:26 PM, Matthew Parker <
> >>>> > mpar...@apogeeintegration.com> wrote:
> >>>> >
> >>>> >> I'll have to check on the commit situation. We have been pushing
> >>>> data from
> >>>> >> SharePoint the last week or so. Would that somehow block the
> >>>> documents
> >>>> >> moving between the solr instances?
> >>>> >>
> >>>> >> I'll try another version tomorrow. Thanks for the suggestions.
> >>>> >>
> >>>> >> On Mon, Feb 27, 2012 at 5:34 PM, Mark Miller <
> markrmil...@gmail.com
> >>>> >wrote:
> >>>> >>
> >>>> >>> Hmmm...all of that looks pretty normal...
> >>>> >>>
> >>>> >>> Did a commit somehow fail on the other machine? When you view the
> >>>> stats
> >>>> >>> for the update handler, are there a lot of pending adds for on of
> >>>> the
> >>>> >>> nodes? Do the commit counts match across nodes?
> >>>> >>>
> >>>> >>> You can also query an individual node with distrib=false to check
> >>>> that.
> >>>> >>>
> >>>> >>> If you build is a month old, I'd honestly recommend you try
> >>>> upgrading as
> >>>> >>> well.
> >>>> >>>
> >>>> >>> - Mark
> >>>> >>>
> >>>> >>> On Feb 27, 2012, at 3:34 PM, Matthew Parker wrote:
> >>>> >>>
> >>>> >>>> Here is most of the cluster state:
> >>>> >>>>
> >>>> >>>> Connected to Zookeeper
> >>>> >>>> localhost:2181, localhost: 2182, localhost:2183
> >>>> >>>>
> >>>> >>>> /(v=0 children=7) ""
> >>>> >>>> /CONFIGS(v=0, children=1)
> >>>> >>>> /CONFIGURATION(v=0 children=25)
> >>>> >>>> <<<<< all the configuration files, velocity info,
> xslt,
> >>>> etc.
> >>>> >>>>>>>>
> >>>> >>>> /NODE_STATES(v=0 children=4)
> >>>> >>>> MACHINE1:8083_SOLR (v=121)"[{"shard_id":"shard1",
> >>>> >>>>
> >>>> "state":"active","core":"","collection":"collection1","node_name:"..."
> >>>> >>>> MACHINE1:8082_SOLR (v=101)"[{"shard_id":"shard2",
> >>>> >>>>
> >>>> "state":"active","core":"","collection":"collection1","node_name:"..."
> >>>> >>>> MACHINE1:8081_SOLR (v=92)"[{"shard_id":"shard1",
> >>>> >>>>
> >>>> "state":"active","core":"","collection":"collection1","node_name:"..."
> >>>> >>>> MACHINE1:8084_SOLR (v=73)"[{"shard_id":"shard2",
> >>>> >>>>
> >>>> "state":"active","core":"","collection":"collection1","node_name:"..."
> >>>> >>>> /ZOOKEEPER (v-0 children=1)
> >>>> >>>> QUOTA(v=0)
> >>>> >>>>
> >>>> >>>>
> >>>> >>>
> >>>>
> /CLUSTERSTATE.JSON(V=272)"{"collection1":{"shard1":{MACHINE1:8081_solr_":{shard_id":"shard1","leader":"true","..."
> >>>> >>>> /LIVE_NODES (v=0 children=4)
> >>>> >>>> MACHINE1:8083_SOLR(ephemeral v=0)
> >>>> >>>> MACHINE1:8082_SOLR(ephemeral v=0)
> >>>> >>>> MACHINE1:8081_SOLR(ephemeral v=0)
> >>>> >>>> MACHINE1:8084_SOLR(ephemeral v=0)
> >>>> >>>> /COLLECTIONS (v=1 children=1)
> >>>> >>>> COLLECTION1(v=0 children=2)"{"configName":"configuration1"}"
> >>>> >>>> LEADER_ELECT(v=0 children=2)
> >>>> >>>> SHARD1(V=0 children=1)
> >>>> >>>> ELECTION(v=0 children=2)
> >>>> >>>>
> >>>> >>>> 87186203314552835-MACHINE1:8081_SOLR_-N_0000000096(ephemeral v=0)
> >>>> >>>>
> >>>> >>>> 87186203314552836-MACHINE1:8083_SOLR_-N_0000000084(ephemeral v=0)
> >>>> >>>> SHARD2(v=0 children=1)
> >>>> >>>> ELECTION(v=0 children=2)
> >>>> >>>>
> >>>> >>>> 231301391392833539-MACHINE1:8084_SOLR_-N_0000000085(ephemeral
> v=0)
> >>>> >>>>
> >>>> >>>> 159243797356740611-MACHINE1:8082_SOLR_-N_0000000084(ephemeral
> v=0)
> >>>> >>>> LEADERS (v=0 children=2)
> >>>> >>>> SHARD1 (ephemeral
> >>>> >>>> v=0)"{"core":"","node_name":"MACHINE1:8081_solr","base_url":"
> >>>> >>>> http://MACHINE1:8081/solr"}";
> >>>> >>>> SHARD2 (ephemeral
> >>>> >>>> v=0)"{"core":"","node_name":"MACHINE1:8082_solr","base_url":"
> >>>> >>>> http://MACHINE1:8082/solr"}";
> >>>> >>>> /OVERSEER_ELECT (v=0 children=2)
> >>>> >>>> ELECTION (v=0 children=4)
> >>>> >>>>
> >>>> 231301391392833539-MACHINE1:8084_SOLR_-N_0000000251(ephemeral
> >>>> >>> v=0)
> >>>> >>>>
> 87186203314552835-MACHINE1:8081_SOLR_-N_0000000248(ephemeral
> >>>> >>> v=0)
> >>>> >>>>
> >>>> 159243797356740611-MACHINE1:8082_SOLR_-N_0000000250(ephemeral
> >>>> >>> v=0)
> >>>> >>>>
> 87186203314552836-MACHINE1:8083_SOLR_-N_0000000249(ephemeral
> >>>> >>> v=0)
> >>>> >>>> LEADER (emphemeral
> >>>> >>>> v=0)"{"id":"87186203314552835-MACHINE1:8081_solr-n_000000248"}"
> >>>> >>>>
> >>>> >>>>
> >>>> >>>>
> >>>> >>>> On Mon, Feb 27, 2012 at 2:47 PM, Mark Miller <
> >>>> markrmil...@gmail.com>
> >>>> >>> wrote:
> >>>> >>>>
> >>>> >>>>>
> >>>> >>>>> On Feb 27, 2012, at 2:22 PM, Matthew Parker wrote:
> >>>> >>>>>
> >>>> >>>>>> Thanks for your reply Mark.
> >>>> >>>>>>
> >>>> >>>>>> I believe the build was towards the begining of the month. The
> >>>> >>>>>> solr.spec.version is 4.0.0.2012.01.10.38.09
> >>>> >>>>>>
> >>>> >>>>>> I cannot access the clusterstate.json contents. I clicked on
> it a
> >>>> >>> couple
> >>>> >>>>> of
> >>>> >>>>>> times, but nothing happens. Is that stored on disk somewhere?
> >>>> >>>>>
> >>>> >>>>> Are you using the new admin UI? That has recently been updated
> to
> >>>> work
> >>>> >>>>> better with cloud - it had some troubles not too long ago. If
> you
> >>>> are,
> >>>> >>> you
> >>>> >>>>> should trying using the old admin UI's zookeeper page - that
> >>>> should
> >>>> >>> show
> >>>> >>>>> the cluster state.
> >>>> >>>>>
> >>>> >>>>> That being said, there has been a lot of bug fixes over the past
> >>>> month
> >>>> >>> -
> >>>> >>>>> so you may just want to update to a recent version.
> >>>> >>>>>
> >>>> >>>>>>
> >>>> >>>>>> I configured a custom request handler to calculate an unique
> >>>> document
> >>>> >>> id
> >>>> >>>>>> based on the file's url.
> >>>> >>>>>>
> >>>> >>>>>> On Mon, Feb 27, 2012 at 1:13 PM, Mark Miller <
> >>>> markrmil...@gmail.com>
> >>>> >>>>> wrote:
> >>>> >>>>>>
> >>>> >>>>>>> Hey Matt - is your build recent?
> >>>> >>>>>>>
> >>>> >>>>>>> Can you visit the cloud/zookeeper page in the admin and send
> the
> >>>> >>>>> contents
> >>>> >>>>>>> of the clusterstate.json node?
> >>>> >>>>>>>
> >>>> >>>>>>> Are you using a custom index chain or anything out of the
> >>>> ordinary?
> >>>> >>>>>>>
> >>>> >>>>>>>
> >>>> >>>>>>> - Mark
> >>>> >>>>>>>
> >>>> >>>>>>> On Feb 27, 2012, at 12:26 PM, Matthew Parker wrote:
> >>>> >>>>>>>
> >>>> >>>>>>>> TWIMC:
> >>>> >>>>>>>>
> >>>> >>>>>>>> Environment
> >>>> >>>>>>>> =========
> >>>> >>>>>>>> Apache SOLR rev-1236154
> >>>> >>>>>>>> Apache Zookeeper 3.3.4
> >>>> >>>>>>>> Windows 7
> >>>> >>>>>>>> JDK 1.6.0_23.b05
> >>>> >>>>>>>>
> >>>> >>>>>>>> I have built a SOLR Cloud instance with 4 nodes using the
> >>>> embeded
> >>>> >>> Jetty
> >>>> >>>>>>>> servers.
> >>>> >>>>>>>>
> >>>> >>>>>>>> I created a 3 node zookeeper ensemble to manage the solr
> >>>> >>> configuration
> >>>> >>>>>>> data.
> >>>> >>>>>>>>
> >>>> >>>>>>>> All the instances run on one server so I've had to move ports
> >>>> around
> >>>> >>>>> for
> >>>> >>>>>>>> the various applications.
> >>>> >>>>>>>>
> >>>> >>>>>>>> I start the 3 zookeeper nodes.
> >>>> >>>>>>>>
> >>>> >>>>>>>> I started the first instance of solr cloud with the parameter
> >>>> to
> >>>> >>> have
> >>>> >>>>> two
> >>>> >>>>>>>> shards.
> >>>> >>>>>>>>
> >>>> >>>>>>>> The start the remaining 3 solr nodes.
> >>>> >>>>>>>>
> >>>> >>>>>>>> The system comes up fine. No errors thrown.
> >>>> >>>>>>>>
> >>>> >>>>>>>> I can view the solr cloud console and I can see the SOLR
> >>>> >>> configuration
> >>>> >>>>>>>> files managed by ZooKeeper.
> >>>> >>>>>>>>
> >>>> >>>>>>>> I published data into the SOLR Cloud instances from
> SharePoint
> >>>> using
> >>>> >>>>>>> Apache
> >>>> >>>>>>>> Manifold 0.4-incubating. Manifold is setup to publish the
> data
> >>>> into
> >>>> >>>>>>>> collection1, which is the only collection defined in the
> >>>> cluster.
> >>>> >>>>>>>>
> >>>> >>>>>>>> When I query the data from collection1 as per the solr wiki,
> >>>> the
> >>>> >>>>> results
> >>>> >>>>>>>> are inconsistent. Sometimes all the results are there, other
> >>>> times
> >>>> >>>>>>> nothing
> >>>> >>>>>>>> comes back at all.
> >>>> >>>>>>>>
> >>>> >>>>>>>> It seems to be having an issue auto replicating the data
> >>>> across the
> >>>> >>>>>>> cloud.
> >>>> >>>>>>>>
> >>>> >>>>>>>> Is there some specific setting I might have missed? Based
> upon
> >>>> what
> >>>> >>> I
> >>>> >>>>>>> read,
> >>>> >>>>>>>> I thought that SOLR cloud would take care of distributing and
> >>>> >>>>> replicating
> >>>> >>>>>>>> the data automatically. Do you have to tell it what shard to
> >>>> publish
> >>>> >>>>> the
> >>>> >>>>>>>> data into as well?
> >>>> >>>>>>>>
> >>>> >>>>>>>> Any help would be appreciated.
> >>>> >>>>>>>>
> >>>> >>>>>>>> Thanks,
> >>>> >>>>>>>>
> >>>> >>>>>>>> Matt
> >>>> >>>>>>>>
> >>>> >>>>>>>> ------------------------------
> >>>> >>>>>>>> This e-mail and any files transmitted with it may be
> >>>> proprietary.
> >>>> >>>>>>> Please note that any views or opinions presented in this
> e-mail
> >>>> are
> >>>> >>>>> solely
> >>>> >>>>>>> those of the author and do not necessarily represent those of
> >>>> Apogee
> >>>> >>>>>>> Integration.
> >>>> >>>>>>>
> >>>> >>>>>>> - Mark Miller
> >>>> >>>>>>> lucidimagination.com
> >>>> >>>>>>>
> >>>> >>>>>>>
> >>>> >>>>>>>
> >>>> >>>>>>>
> >>>> >>>>>>>
> >>>> >>>>>>>
> >>>> >>>>>>>
> >>>> >>>>>>>
> >>>> >>>>>>>
> >>>> >>>>>>>
> >>>> >>>>>>>
> >>>> >>>>>>>
> >>>> >>>>>>
> >>>> >>>>>> Matt
> >>>> >>>>>>
> >>>> >>>>>> ------------------------------
> >>>> >>>>>> This e-mail and any files transmitted with it may be
> proprietary.
> >>>> >>>>> Please note that any views or opinions presented in this e-mail
> >>>> are
> >>>> >>> solely
> >>>> >>>>> those of the author and do not necessarily represent those of
> >>>> Apogee
> >>>> >>>>> Integration.
> >>>> >>>>>
> >>>> >>>>> - Mark Miller
> >>>> >>>>> lucidimagination.com
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>
> >>>> >>>> ------------------------------
> >>>> >>>> This e-mail and any files transmitted with it may be proprietary.
> >>>> >>> Please note that any views or opinions presented in this e-mail
> are
> >>>> solely
> >>>> >>> those of the author and do not necessarily represent those of
> Apogee
> >>>> >>> Integration.
> >>>> >>>
> >>>> >>> - Mark Miller
> >>>> >>> lucidimagination.com
> >>>> >>>
> >>>> >>>
> >>>> >>>
> >>>> >>>
> >>>> >>>
> >>>> >>>
> >>>> >>>
> >>>> >>>
> >>>> >>>
> >>>> >>>
> >>>> >>>
> >>>> >>>
> >>>> >>
> >>>> >
> >>>> > ------------------------------
> >>>> > This e-mail and any files transmitted with it may be proprietary.
> >>>> Please note that any views or opinions presented in this e-mail are
> solely
> >>>> those of the author and do not necessarily represent those of Apogee
> >>>> Integration.
> >>>>
> >>>> - Mark Miller
> >>>> lucidimagination.com
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>
> >
>
> ------------------------------
> This e-mail and any files transmitted with it may be proprietary. Please
> note that any views or opinions presented in this e-mail are solely those
> of the author and do not necessarily represent those of Apogee Integration.
>
--
- Mark
http://www.lucidimagination.com
