Re: Challenges for Java hosting

[Bill Barker]

"Henri Gomez" <[EMAIL PROTECTED]> wrote in message 
news:[EMAIL PROTECTED]
2006/4/6, Remy Maucherat <[EMAIL PROTECTED]>:
>> Hi,
>>
>> This thread started (for whatever reason) on the private list as part of
>> an unrelated discussion. The point is to see what could be improved to
>> make Tomcat more suitable for shared hosting, which is a very nice goal,
>> but unfortunately with very serious issues.
>>
>> I don't see many improvements possibilities, as I consider the following
>> solutions and problems (where each user would at least need its own 
>> vhost):
>
>I see vhost as a client virtual server, where the 'client' is not a
>real / human end user but someone who ask you to host its applications
>with one or many webapps in it.
>
>> - Every virtual host gets its own appBase folder. Having its own folder
>> for JARs just won't work (or it means you were able to use the "shared"
>> folder successfully, which I doubt).
>
>What do you mean ? Did we have actually such own folder of vhosts jars
>with its own classloader ?  If not could it be done easily ?
>

In theory, it should be possible to inject a parentClassLoader into the 
StandardHost to do this.  I don't believe that StandardHost exposes this to 
JMX, so at the moment, you'd need to use introspection.  And, I've never 
actually tried to do this myself ;-).  Of course, the patch to enable doing 
it with JMX is trivial.

>Actually a client application get its classes from :
>
>- jars in common [lib, endorsed, i18n] and webapps lib directory
>
>- classes from common [ classes ] and webapps classes directory.
>
>Could we have instead a by vhost supplemental directory to get its own jar 
>?
>
>example :
>
>vhosta will grab in jars in /home/vhostsa/lib and common [lib,
>endorsed, i18n] and webapps lib directory
>
>vhostb will grab in jars in /home/vhostsb/lib and common [lib,
>endorsed, i18n] and webapps lib directory.
>
>In such case did the classloader for vhosta should be different from
>the one for vhostb.
>
>
>>If you use the
>> TOMCAT_HOME/TOMCAT_BASE stuff, each user can get its own "shared" folder.
>
>> - There are still tons of JVMs to manage and monitor, which may be a
>> problem.
>
>Why did there should be tons of JVM ? Couldn't we use a single JVM for
>each vhosts ? As many I prefer give more power and resource to a
>single JVM that spread thread and memory to many JVM where some could
>be at some time very less used than others.
>
>> - If the connector should be shared, with the servlet containers running
>> in separate processes, I don't see how to cross the process barrier
>> except by going back to square one (httpd in front, with AJP and many
>> JVMs/Tomcats, each with its own vhost).
>
>That's the current situation now, one Apache 2.x with multiples
>virtuals servers, each virtual server with its own JK/AJP worker to
>reach a specific Tomcat.
>
>> Some general problems for production are:
>
>> - No self tuning of the JVM.
>
>Of course but it's not better to have may JVM present in system with
>memory and thread/system resources use when for exemple users are only
>using one of the multiples JVM.
>
>Imagine for example users for differents country, said asia, europe and 
>USA.
>
>You setup one JVM / Application for ASIA users, one for Europeans
>users and the third for US users.
>
>During the day, you'll see activity on Asian JVM, then European JVM
>and then on US JVM, but even when only one is really used, the 2
>others are still in machine and consume memory and resources. Sad,
>this power could be better used for the really active JVM.
>
>> - No actual isolation, throttling capabilities, etc, provided by the JVM.
>
>JVM or OS ?
>
>> - Impossibility to control memory leaks (impossible to force discarding
>> classloaders and all associated class defs and instances, for example).
>
>What do you means ?
>
>> - Hard to do thread management (by that I mean, monitor and recover for
>> threads stuck in loops or deadlocked).
>
>Well it's also hard today with a 'dedicated' JVM, the only solution
>may be to restart the JVM ...
>
>> Any ideas ?
>
>Well if it could be done, I'm sure you're probably the best to imagine
>such solution.
>
>
>> I suppose native code could be used to improve the situation in some
>> areas (although I don't know how to do it ;) ).
>
>Native code ? Do you imagine some sort of sub-JVM launching ?
>
>> Rémy
>>
>> -
>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>> For additional commands, e-mail: [EMAIL PROTECTED]
>>
>> 




-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

DO NOT REPLY [Bug 31426] - OutOfMemoryError: unable to create new native thread - and tomcat "hangs"

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG·
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND·
INSERTED IN THE BUG DATABASE.

http://issues.apache.org/bugzilla/show_bug.cgi?id=31426





--- Additional Comments From [EMAIL PROTECTED]  2006-04-07 10:50 ---
We had the same issue (jdk1.5, tomcat 5.0.25) and it was solved by giving the vm
more PermGenSpace.

-- 
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the assignee for the bug, or are watching the assignee.

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: Challenges for Java hosting

[Tim Funk]

I was thinking that too. A big problem with JVM's is memory leaks. The easy 
solution is to restart tomcat. But that causes a period of downtime due to 
waiting for a restart.


Why not make mod_ajp "smarter" or create a tomcat launcher where some parent 
process (apache, or the launcher) listens for requests from the public and 
uses AJP to have tomcat serve the requests. [The downside is added latenency]


The parent process can spawn a new tomcat after some configurable time and 
kill the old one. By using the cluster code - the sessions can be synced 
before the old tomcat goes away.


This mechanism could also be used as a way to perform graceful restarts too 
where an web app upgrade is done and reloading webapps will cause memory leaks.


[Disclaimer: the above ramblings may not be based in reality]

-Tim

Henri Gomez wrote:



I suppose native code could be used to improve the situation in some
areas (although I don't know how to do it ;) ).



Native code ? Do you imagine some sort of sub-JVM launching ?



-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: Challenges for Java hosting

[Henri Gomez]

Very interesting stuff

My german is too bad but from what I could see it seems a good candidate.

Could this works be ported back into ASF Tomcat 5.5.x ? Or better
included in ASF Tomcat ?

>2006/4/7, Peter Rossbach <[EMAIL PROTECTED]>:
> Hey,
>
> Java/JSP and Tomcat for german hoster is a very bad story.  For two
> year we
> start a tomcat 5.0 based spezial tomcat distibution for hosting. The
> Centaurus Platform
> has show that effectiv hosting is possible. Problem is to find hoster
> that use that package.
> Look at http://centaurus.sourceforge.net/ and see that we have create
> a very cool tomcat bundle.
>
> Centaurus features:
>
> Used a patched tomcat 5.0.27
> full graphical and console installer package for LINUX/Windows
> You can have multiple installation profiles
> Very nice html admin/manager console for one instance
> Loading new Security Policy on demand without downtime
> every webapp can register there own policy part (admin can control
> this integration with rules)
> Integrate new host from templates without downtime
> Native Iintegration at os services with Java Service Wrapper
> Plugin concept for people to extends tomcat core functionality
> use of Mx4J Http adaptor with authorisation for remote control.
>
> At cvs head exist a not final tomcat 5.5 version.
>
> Regards
> Peter
> - Documentation only exist in german language.
>
>
>
> Am 06.04.2006 um 23:48 schrieb Leon Rosenberg:
>
> > On 4/6/06, Preston L. Bannister <[EMAIL PROTECTED]> wrote:
> >> Define "lightweight". :)
> >
> > only the basics you need for a webapp. no admin/manager, no
> > clustering, no gadgets.
> > To explain it:
> > Besides large portals with own server farms and millions of hits, I
> > often have small customers which get a dynamical website with some cms
> > etc. The problem I had and still have, is that typical hosting
> > providers (at least in germany) don't offer any support for java
> > webapps, best you can get is support for jsps only which sucks. This
> > is ugly, since the customer pays money for the webapp and asks me
> > afterwards, why should he rent a complete server to host it. Therefore
> > I started to rent servers myself, and re-rent it to the customer. The
> > customer gets the complete package, mail, backup, ftp/ssh access and
> > the webapp. To ensure this, each server has an apache running with two
> > jsp container instances on it, one for production, one for testing
> > versions. The customer pays less than he would pay for "professional"
> > hosting, and I can refinance the server with 3-4 customers. However,
> > having all test-webapps in one server, and needing to restart it from
> > time to time isn't really cool. I'd prefer to give each customer two
> > instances, which consumes low resources, maybe even multiple tomcat
> > instances in one jvm (is that possible?), and keep them independent
> > from each other. Therefore lightweight. And therefore pre-configured
> > :-)
> >
> > regards
> > Leon
> >
> >>
> >> If we are talking about a small number of users, with high average
> >> utilization, this might be a good solution.  In fact this is
> >> similar in
> >> resource usage to the virtual hosting (i.e. Xen) solutions.
> >>
> >> For more typical usage, the number of users is large, and the average
> >> utilization is low.  In this case one (very rarely used) JVM per
> >> user is
> >> somewhat expensive.
> >>
> >> Note you could reduce the expense with the same approach of using
> >> a fork()
> >> of a single image, expecting copy-on-write to radically reduce the
> >> real
> >> memory use (virtual memory use would be larger).
> >>
> >> Depends on what target you are trying to hit.  The hosting world
> >> (by numbers
> >> of users) is dominated by very low usage sites.  Is this a goal for
> >> Java/Tomcat hosting?  If you can beat PHP in CGI mode *both* in
> >> performance
> >> and in resource usage, then you have a pretty compelling
> >> solution.  If you
> >> are fatter or slower - this is going to disinterest a lot of hosting
> >> providers.
> >>
> >> Note that this notion is pretty much a non-starter if Linux does
> >> not do
> >> copy-on-write with fork().  This was a big deal back in the late
> >> 1980's (big
> >> Lisp apps forking "vi").  Don't know if this made it's way into
> >> Linux.  I'm
> >> pretty sure copy-on-write in fork() was in SunOS, but I don't know
> >> about
> >> Solaris.
> >>
> >>
> >> On 4/6/06, Leon Rosenberg <[EMAIL PROTECTED]> wrote:
> >>>
> >>> isn't it easier to give each user a pre-configured lightweight
> >>> but own
> >>> tomcat?
> >>>
> >>> leon
> >>>
> >>> On 4/6/06, Preston L. Bannister <[EMAIL PROTECTED]> wrote:
>  Well, that is one definition of "real applications".   There are
>  other
>  definitions.  :)
> 
> 
>  On 4/6/06, Tino Schwarze <[EMAIL PROTECTED]> wrote:
> >
> > On Thu, Apr 06, 2006 at 09:15:17AM -0700, Preston L. Bannister
> > wrote:
> >
> >> You have to consider how (or if) to

Re: Challenges for Java hosting

[Henri Gomez]

2006/4/7, Bill Barker <[EMAIL PROTECTED]>:
>
> "Henri Gomez" <[EMAIL PROTECTED]> wrote in message
> news:[EMAIL PROTECTED]
> 2006/4/6, Remy Maucherat <[EMAIL PROTECTED]>:
> >> Hi,
> >>
> >> This thread started (for whatever reason) on the private list as part of
> >> an unrelated discussion. The point is to see what could be improved to
> >> make Tomcat more suitable for shared hosting, which is a very nice goal,
> >> but unfortunately with very serious issues.
> >>
> >> I don't see many improvements possibilities, as I consider the following
> >> solutions and problems (where each user would at least need its own
> >> vhost):
> >
> >I see vhost as a client virtual server, where the 'client' is not a
> >real / human end user but someone who ask you to host its applications
> >with one or many webapps in it.
> >
> >> - Every virtual host gets its own appBase folder. Having its own folder
> >> for JARs just won't work (or it means you were able to use the "shared"
> >> folder successfully, which I doubt).
> >
> >What do you mean ? Did we have actually such own folder of vhosts jars
> >with its own classloader ?  If not could it be done easily ?
> >
>
> In theory, it should be possible to inject a parentClassLoader into the
> StandardHost to do this.  I don't believe that StandardHost exposes this to
> JMX, so at the moment, you'd need to use introspection.  And, I've never
> actually tried to do this myself ;-).  Of course, the patch to enable doing
> it with JMX is trivial.

Good to see it could be possible.

Next question will be could we have such functionality added in future
5.5.x versions ?

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: Challenges for Java hosting

[Peter Rossbach]

Hey Henri,

no problem. I have talk with Thorsten Kamann the other author and
we are happy to contribute the code. Then we can translate the  
complete docs

and finish the tomcat 5.5 integration.

Regards
Peter & Thorsten



Am 07.04.2006 um 13:16 schrieb Henri Gomez:


Very interesting stuff

My german is too bad but from what I could see it seems a good  
candidate.


Could this works be ported back into ASF Tomcat 5.5.x ? Or better
included in ASF Tomcat ?


2006/4/7, Peter Rossbach <[EMAIL PROTECTED]>:
Hey,

Java/JSP and Tomcat for german hoster is a very bad story.  For two
year we
start a tomcat 5.0 based spezial tomcat distibution for hosting. The
Centaurus Platform
has show that effectiv hosting is possible. Problem is to find hoster
that use that package.
Look at http://centaurus.sourceforge.net/ and see that we have create
a very cool tomcat bundle.

Centaurus features:

Used a patched tomcat 5.0.27
full graphical and console installer package for LINUX/Windows
You can have multiple installation profiles
Very nice html admin/manager console for one instance
Loading new Security Policy on demand without downtime
every webapp can register there own policy part (admin can  
control

this integration with rules)
Integrate new host from templates without downtime
Native Iintegration at os services with Java Service Wrapper
Plugin concept for people to extends tomcat core functionality
use of Mx4J Http adaptor with authorisation for remote control.

At cvs head exist a not final tomcat 5.5 version.

Regards
Peter
- Documentation only exist in german language.



Am 06.04.2006 um 23:48 schrieb Leon Rosenberg:


On 4/6/06, Preston L. Bannister <[EMAIL PROTECTED]> wrote:

Define "lightweight". :)


only the basics you need for a webapp. no admin/manager, no
clustering, no gadgets.
To explain it:
Besides large portals with own server farms and millions of hits, I
often have small customers which get a dynamical website with  
some cms

etc. The problem I had and still have, is that typical hosting
providers (at least in germany) don't offer any support for java
webapps, best you can get is support for jsps only which sucks. This
is ugly, since the customer pays money for the webapp and asks me
afterwards, why should he rent a complete server to host it.  
Therefore
I started to rent servers myself, and re-rent it to the customer.  
The

customer gets the complete package, mail, backup, ftp/ssh access and
the webapp. To ensure this, each server has an apache running  
with two

jsp container instances on it, one for production, one for testing
versions. The customer pays less than he would pay for  
"professional"

hosting, and I can refinance the server with 3-4 customers. However,
having all test-webapps in one server, and needing to restart it  
from

time to time isn't really cool. I'd prefer to give each customer two
instances, which consumes low resources, maybe even multiple tomcat
instances in one jvm (is that possible?), and keep them independent
from each other. Therefore lightweight. And therefore pre-configured
:-)

regards
Leon



If we are talking about a small number of users, with high average
utilization, this might be a good solution.  In fact this is
similar in
resource usage to the virtual hosting (i.e. Xen) solutions.

For more typical usage, the number of users is large, and the  
average

utilization is low.  In this case one (very rarely used) JVM per
user is
somewhat expensive.

Note you could reduce the expense with the same approach of using
a fork()
of a single image, expecting copy-on-write to radically reduce the
real
memory use (virtual memory use would be larger).

Depends on what target you are trying to hit.  The hosting world
(by numbers
of users) is dominated by very low usage sites.  Is this a goal for
Java/Tomcat hosting?  If you can beat PHP in CGI mode *both* in
performance
and in resource usage, then you have a pretty compelling
solution.  If you
are fatter or slower - this is going to disinterest a lot of  
hosting

providers.

Note that this notion is pretty much a non-starter if Linux does
not do
copy-on-write with fork().  This was a big deal back in the late
1980's (big
Lisp apps forking "vi").  Don't know if this made it's way into
Linux.  I'm
pretty sure copy-on-write in fork() was in SunOS, but I don't know
about
Solaris.


On 4/6/06, Leon Rosenberg <[EMAIL PROTECTED]> wrote:


isn't it easier to give each user a pre-configured lightweight
but own
tomcat?

leon

On 4/6/06, Preston L. Bannister <[EMAIL PROTECTED]> wrote:

Well, that is one definition of "real applications".   There are
other
definitions.  :)


On 4/6/06, Tino Schwarze <[EMAIL PROTECTED]> wrote:


On Thu, Apr 06, 2006 at 09:15:17AM -0700, Preston L. Bannister
wrote:


You have to consider how (or if) to allow for long-running

background

threads.  Successive requests for the same user will not use
the JVM
(whether this counts as an advantage or disadvantage is

debatable).  T

DO NOT REPLY [Bug 31426] - OutOfMemoryError: unable to create new native thread - and tomcat "hangs"

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG·
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND·
INSERTED IN THE BUG DATABASE.

http://issues.apache.org/bugzilla/show_bug.cgi?id=31426


[EMAIL PROTECTED] changed:

   What|Removed |Added

 Status|REOPENED|RESOLVED
 Resolution||WONTFIX




-- 
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the assignee for the bug, or are watching the assignee.

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

DO NOT REPLY [Bug 38768] - URL encoded unicode characters in GET request parameters cause the parameter not to present in the request object

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG·
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND·
INSERTED IN THE BUG DATABASE.

http://issues.apache.org/bugzilla/show_bug.cgi?id=38768


[EMAIL PROTECTED] changed:

   What|Removed |Added

 Status|ASSIGNED|RESOLVED
 Resolution||WONTFIX




--- Additional Comments From [EMAIL PROTECTED]  2006-04-07 13:10 ---
Your parameter is not encoded correctly, and I don't see any reason to add
special handling for this situation (of course, feel free to use a custom patch
in your own Tomcat installations).

-- 
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the assignee for the bug, or are watching the assignee.

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: Challenges for Java hosting

[Reinhard Moosauer]

Hi List,

good that you brought up this point. 
IMHO many people underestimated this subject in both  importance _and_ 
difficulty.
We know, shared java hosting can be done in 2 ways:
a) one JVM per host
b) many hosts in one JVM

As Remy noted, the java machine does not provide the necessary isolation 
for solution (b). Security Managers do their work for security, but not for 
stabilty. Solution (a) is expensive.
These fact facts have been standing for many years.

But now, the situation is slightly different!
In my opinion, we can use solution (a), if we utilize clustering.
The PHP folks say, simply make the server stateless and everything is easy. 
True, but tomcat does not work that way.

Now, with clustering, we could combine both. Consider the following:
1. Set up a couple of tomcat servers (at least 2). I call them 'node'
These can sit on a single server
2. Cluster and load-balance these nodes. They should been seen as a single
   tomcat server .
3. Now use this cluster a in solution (b) above.
4. Each node should be monitored and if a situation arises, that 
compromises the node's stability, restart that node.
(Memory nearly exhausted; endless looping threads; ...) 
5. Another node should have the complete state of all applications on the 
failing node, so no interrupt occurs.
6. precondition: n-1 nodes must be able to operate all vhosts

I see one only one enhancement, that must be done before:
The cluster _and_ the load-balancer must be vhost-aware. With this, you could 
achive a RAID-like situation.
It's an array of inexpensive tomcats :-)

just my two cents

R.



Am Donnerstag, 6. April 2006 16:12 schrieb Remy Maucherat:
> Hi,
>
> This thread started (for whatever reason) on the private list as part of
> an unrelated discussion. The point is to see what could be improved to
> make Tomcat more suitable for shared hosting, which is a very nice goal,
> but unfortunately with very serious issues.
>
> I don't see many improvements possibilities, as I consider the following
> solutions and problems (where each user would at least need its own vhost):
> - Every virtual host gets its own appBase folder. Having its own folder
> for JARs just won't work (or it means you were able to use the "shared"
> folder successfully, which I doubt). If you use the
> TOMCAT_HOME/TOMCAT_BASE stuff, each user can get its own "shared" folder.
> - There are still tons of JVMs to manage and monitor, which may be a
> problem.
> - If the connector should be shared, with the servlet containers running
> in separate processes, I don't see how to cross the process barrier
> except by going back to square one (httpd in front, with AJP and many
> JVMs/Tomcats, each with its own vhost).
>
> Some general problems for production are:
> - No self tuning of the JVM.
> - No actual isolation, throttling capabilities, etc, provided by the JVM.
> - Impossibility to control memory leaks (impossible to force discarding
> classloaders and all associated class defs and instances, for example).
> - Hard to do thread management (by that I mean, monitor and recover for
> threads stuck in loops or deadlocked).
>
> Any ideas ?
>
> I suppose native code could be used to improve the situation in some
> areas (although I don't know how to do it ;) ).
>
> Rémy
>
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]

-- 
Reinhard Moosauer
IT Beratung

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: Challenges for Java hosting

[Remy Maucherat]

Reinhard Moosauer wrote:

Now, with clustering, we could combine both. Consider the following:
1. Set up a couple of tomcat servers (at least 2). I call them 'node'
These can sit on a single server
2. Cluster and load-balance these nodes. They should been seen as a single
   tomcat server .
3. Now use this cluster a in solution (b) above.
4. Each node should be monitored and if a situation arises, that 
compromises the node's stability, restart that node.
(Memory nearly exhausted; endless looping threads; ...) 
5. Another node should have the complete state of all applications on the 
failing node, so no interrupt occurs.

6. precondition: n-1 nodes must be able to operate all vhosts


I had thought about this solution.

However, there are issues, as clustering is quite expensive (if there is 
more than one node, lol), and is incompatible with many applications. So 
for starters, clustering should be updated to add an option to not yell 
when a non-serializable attribute is added to a session, it should only 
operate with one node and then periodically (not too often, so that 
applications with do-it-yourself state will not break too often) start a 
second node, switch the traffic to it, and stop the first one. A similar 
process could also be done when redeploying things, etc. For regular 
hosting, this should provide good enough reliability (some state loss, 
but very decent availability).


I don't think it's actually doable to host Java webapps which break 
every 3 requests (going into infinite loops or deadlocking, etc), however.


Another downside is that the server needs to have CPU and memory 
headroom, since there will be a lot of full replication of sessions 
(during a node transition).


Note: Tim stated the same concept earlier too.

Rémy

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: Challenges for Java hosting

[Reinhard Moosauer]

Am Freitag, 7. April 2006 15:12 schrieb Remy Maucherat:
> Reinhard Moosauer wrote:
> > Now, with clustering, we could combine both. Consider the following:
> > 1. Set up a couple of tomcat servers (at least 2). I call them 'node'
> > These can sit on a single server
> > 2. Cluster and load-balance these nodes. They should been seen as a
> > single tomcat server .
> > 3. Now use this cluster a in solution (b) above.
> > 4. Each node should be monitored and if a situation arises, that
> > compromises the node's stability, restart that node.
> > (Memory nearly exhausted; endless looping threads; ...)
> > 5. Another node should have the complete state of all applications on the
> > failing node, so no interrupt occurs.
> > 6. precondition: n-1 nodes must be able to operate all vhosts
>
> I had thought about this solution.
>
> However, there are issues, as clustering is quite expensive (if there is
> more than one node, lol), and is incompatible with many applications. So
> for starters, clustering should be updated to add an option to not yell
> when a non-serializable attribute is added to a session, it should only
> operate with one node and then periodically (not too often, so that
> applications with do-it-yourself state will not break too often) start a
> second node, switch the traffic to it, and stop the first one. A similar
> process could also be done when redeploying things, etc. For regular
> hosting, this should provide good enough reliability (some state loss,
> but very decent availability).
>
> I don't think it's actually doable to host Java webapps which break
> every 3 requests (going into infinite loops or deadlocking, etc), however.
>

Ok. But you can kill the webapp with the amok-thread. So we will not have a 
break every 3 requests. (the thread can be tracked down to the failing 
webapp. Send a mail with the thread-stack-dump)
The ignore-not-serializable-sessions-objects thing would be very nice! 

> Another downside is that the server needs to have CPU and memory
> headroom, since there will be a lot of full replication of sessions
> (during a node transition).

Maybe it is still much better than the one-vhost-per-vm solution.
I think for hosters with thousands of vhosts, this would be quite interesting.

>
> Note: Tim stated the same concept earlier too.
>
> Rémy

What do you think about my suggestion to enhance the cluster?
(vhost<->node would no more be all-to-all, so you can optimize clusters with 
more than 2 nodes, which also reduces replication overhead)

>
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]

Reinhard

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Windows Itanium Support

[etheredgea]

I am currently porting our 32 bit application to a Windows 2k3 Itanium 2 
64 bit platform.  The tomcat windows installer fails  when attempting to 
install the service.  I would like to solve this problem.  I can't find 
the source for the windows installer... Is this available or is someone 
else doing this?  My 32 bit service code works fine on the itanium (with 
no modifications) so I don't think there needs to be a specialized 64 bit 
port of the installer or anything.

I have installed tomcat as a service on itanium by compiling procrun, 
renaming it to tomcat5.exe, copying it to the .zip installation, and 
running service.bat.  i would prefer, however, to get it running via the 
installation program.

Adam

Re: configurable AJP Buffer Size

[Jim Jagielski]

On Apr 5, 2006, at 1:04 PM, KARNATI, SRINIVASA R [AG/1000] wrote:

I would like to extend AJP buffer size (in  
org.apache.jk.common.MsgAjp) as a
configurable option. We really needed the buffer above 8 kb  
(because of

Kerberos integration.)

Here is the plan:

i) maxAJPBufferSize on connector definition will accept buffer size  
above 8

kb

For example:


if maxAJPBufferSize attribute is absent, then 8 kb is the default.

ii) add a new constructor to MsgAjp to take in the size parameter

I appreciate any comments/suggestions in this regard.



I would be -1 for AJP 1.3, but +1 as a functionality baseline
for 1.4. The issue is that the buffer size should ideally
be negotiated between the 2 ends.

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Tomcat connector for Lotus Domino

[Alexander V. Khokhlov]

Hi all!
I could not build "DSAPI plugin for Lotus Domino" tomcat_redirector.dll 
from JK-1.2.15 distribution for Win32 patform
I have Lotus Domino 6.5.4 and corresponding version of C API Toolkit
When I try to build project MSVC 6.0 show me following messages:

Build : warning : failed to (or don't know how to) build 
'D:\Work\jakarta-tomcat-connectors-1.2.15-src\jk\native\common\jk_env.c'
Compiling...
jk_dsapi_plugin.c
D:\Work\jakarta-tomcat-connectors-1.2.15-src\jk\native\domino\jk_dsapi_plugin.c(478)
 
: error C2065: 'SW_SHOWMAXIMIZED' : undeclared identifier
D:\Work\jakarta-tomcat-connectors-1.2.15-src\jk\native\domino\jk_dsapi_plugin.c(806)
 
: warning C4090: 'function' : different 'const' qualifiers
D:\Work\jakarta-tomcat-connectors-1.2.15-src\jk\native\domino\jk_dsapi_plugin.c(806)
 
: warning C4024: 'GetVariable' : different types for formal and actual 
parameter 5
D:\Work\jakarta-tomcat-connectors-1.2.15-src\jk\native\domino\jk_dsapi_plugin.c(807)
 
: warning C4090: 'function' : different 'const' qualifiers
D:\Work\jakarta-tomcat-connectors-1.2.15-src\jk\native\domino\jk_dsapi_plugin.c(807)
 
: warning C4024: 'GetVariable' : different types for formal and actual 
parameter 5
D:\Work\jakarta-tomcat-connectors-1.2.15-src\jk\native\domino\jk_dsapi_plugin.c(808)
 
: warning C4090: 'function' : different 'const' qualifiers
D:\Work\jakarta-tomcat-connectors-1.2.15-src\jk\native\domino\jk_dsapi_plugin.c(808)
 
: warning C4024: 'GetVariable' : different types for formal and actual 
parameter 5
D:\Work\jakarta-tomcat-connectors-1.2.15-src\jk\native\domino\jk_dsapi_plugin.c(809)
 
: warning C4090: 'function' : different 'const' qualifiers
D:\Work\jakarta-tomcat-connectors-1.2.15-src\jk\native\domino\jk_dsapi_plugin.c(809)
 
: warning C4024: 'GetVariable' : different types for formal and actual 
parameter 5
D:\Work\jakarta-tomcat-connectors-1.2.15-src\jk\native\domino\jk_dsapi_plugin.c(810)
 
: warning C4090: 'function' : different 'const' qualifiers
D:\Work\jakarta-tomcat-connectors-1.2.15-src\jk\native\domino\jk_dsapi_plugin.c(810)
 
: warning C4024: 'GetVariable' : different types for formal and actual 
parameter 5
D:\Work\jakarta-tomcat-connectors-1.2.15-src\jk\native\domino\jk_dsapi_plugin.c(811)
 
: warning C4090: 'function' : different 'const' qualifiers
D:\Work\jakarta-tomcat-connectors-1.2.15-src\jk\native\domino\jk_dsapi_plugin.c(811)
 
: warning C4024: 'GetVariable' : different types for formal and actual 
parameter 5
D:\Work\jakarta-tomcat-connectors-1.2.15-src\jk\native\domino\jk_dsapi_plugin.c(965)
 
: error C2198: 'jk_map_read_properties' : too few actual parameters
D:\Work\jakarta-tomcat-connectors-1.2.15-src\jk\native\domino\jk_dsapi_plugin.c(978)
 
: error C2198: 'jk_map_read_properties' : too few actual parameters
D:\Work\jakarta-tomcat-connectors-1.2.15-src\jk\native\domino\jk_dsapi_plugin.c(1014)
 
: error C2065: 'tqp' : undeclared identifier
D:\Work\jakarta-tomcat-connectors-1.2.15-src\jk\native\domino\jk_dsapi_plugin.c(1016)
 
: warning C4090: '=' : different 'const' qualifiers
jk_registry.c
fatal error C1083: Cannot open source file: 
'D:\Work\jakarta-tomcat-connectors-1.2.15-src\jk\native\common\jk_registry.c': 
No such file or directory
jk_env.c
fatal error C1083: Cannot open source file: 
'D:\Work\jakarta-tomcat-connectors-1.2.15-src\jk\native\common\jk_env.c': 
No such file or directory
Generating Code...
Error executing cl.exe.

tomcat_redirector.dll - 6 error(s), 14 warning(s)

Where i can find jk_registry.c and jk_env.c ?

Sincerely Yours,
Alexander Khokhlov

svn commit: r392352 - /tomcat/connectors/trunk/build.properties.default

[markt]

Author: markt
Date: Fri Apr  7 11:10:44 2006
New Revision: 392352

URL: http://svn.apache.org/viewcvs?rev=392352&view=rev
Log:
Remove references in build scripts to JavaService

Modified:
tomcat/connectors/trunk/build.properties.default

Modified: tomcat/connectors/trunk/build.properties.default
URL: 
http://svn.apache.org/viewcvs/tomcat/connectors/trunk/build.properties.default?rev=392352&r1=392351&r2=392352&view=diff
==
--- tomcat/connectors/trunk/build.properties.default (original)
+++ tomcat/connectors/trunk/build.properties.default Fri Apr  7 11:10:44 2006
@@ -158,11 +158,6 @@
 
commons-pool.loc=${mirror}/jakarta/commons/pool/binaries/commons-pool-${commons-pool.version}.tar.gz
 
 
-# - JavaService, version 1.2.0 or later -
-# do we really need this anymore?
-javaservice.home=${base.path}/javaservice
-javaservice.loc=http://www.alexandriasc.com/software/JavaService/JavaService-bin-1.2.0.zip
-
 # - Java Database Connectivity (JDBC) Optional Package, version 2.0 -  
  
 jdbc20ext.home=${base.path}/jdbc2_0-stdext  
 jdbc20ext.lib=${jdbc20ext.home} 



-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

svn commit: r392357 - /tomcat/connectors/trunk/coyote/src/java/org/apache/coyote/tomcat4/CoyoteRequest.java

[markt]

Author: markt
Date: Fri Apr  7 11:28:19 2006
New Revision: 392357

URL: http://svn.apache.org/viewcvs?rev=392357&view=rev
Log:
Fix NPE when no query string is specified

Modified:

tomcat/connectors/trunk/coyote/src/java/org/apache/coyote/tomcat4/CoyoteRequest.java

Modified: 
tomcat/connectors/trunk/coyote/src/java/org/apache/coyote/tomcat4/CoyoteRequest.java
URL: 
http://svn.apache.org/viewcvs/tomcat/connectors/trunk/coyote/src/java/org/apache/coyote/tomcat4/CoyoteRequest.java?rev=392357&r1=392356&r2=392357&view=diff
==
--- 
tomcat/connectors/trunk/coyote/src/java/org/apache/coyote/tomcat4/CoyoteRequest.java
 (original)
+++ 
tomcat/connectors/trunk/coyote/src/java/org/apache/coyote/tomcat4/CoyoteRequest.java
 Fri Apr  7 11:28:19 2006
@@ -1599,7 +1599,7 @@
  */
 public String getQueryString() {
 String queryString = coyoteRequest.queryString().toString();
-if (queryString.equals("")) {
+if (queryString == null || queryString.equals("")) {
 return (null);
 } else {
 return queryString;



-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

DO NOT REPLY [Bug 39247] New: - XSD for jspx and tagx

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG·
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND·
INSERTED IN THE BUG DATABASE.

http://issues.apache.org/bugzilla/show_bug.cgi?id=39247

   Summary: XSD for jspx and tagx
   Product: Tomcat 5
   Version: Unknown
  Platform: Other
OS/Version: other
Status: NEW
  Severity: normal
  Priority: P2
 Component: Servlet & JSP API
AssignedTo: tomcat-dev@jakarta.apache.org
ReportedBy: [EMAIL PROTECTED]


a) I was looking for an XSD schema for "XML view" JSP pages, and the only schema
I found was jspxml.xsd from 

http://svn.apache.org/viewcvs.cgi/tomcat/servletapi/servlet2.4-jsp2.0-tc5.x/jsr154/src/share/dtd/

Interestingly, there isn't a schema in the Sun's JSP 2.0 spec 
jsp-2_0-fr-spec.pdf.

http://www.google.ca/search?q=jsp-2_0-fr-spec.pdf+%2Bsite%3Asun.com

I removed a couple hitches before I started using jspxml.xsd for validation of
my *.jspx pages.  

i. The definition for the "directive.page/@pageEncoding" attribute value needs
to be fixed:

-
+

ii. Then there are spaces sprinkled liberally around the equal signs.  I wasn't
sure if that affected the validity of the schema itself.

iii. I couldn't find a definition for the "output" element.  It should be
allowed according to the JSP 2.0 spec:

  JSP.6.3.3 The jsp:output Element
  The jsp:output element can be used in JSP documents and in tag files in XML
  syntax.

b) There isn't an "attribute" element definition for JSP tag files in XML syntax
(JSP.8.6).  The spec doesn't enforce even a root element for tag files in XML 
view.

-- 
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the assignee for the bug, or are watching the assignee.

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: Challenges for Java hosting

[Paul Speed]

Reinhard Moosauer wrote:



Ok. But you can kill the webapp with the amok-thread. So we will not have a 
break every 3 requests. (the thread can be tracked down to the failing 
webapp. Send a mail with the thread-stack-dump)


You can't really kill a thread in Java.  We used to have a joke at a 
previous job that the only safe way to kill a thread in Java is 
System.exit(0).


And as I recall, while the javadocs for Thread.stop() detail some of the 
reasons, there is much more badness possible than even what it states.


-Paul

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Suggestion for new Session Manager

[Marc Riehm]

I would like to make a suggestion for a new Tomcat Session Manager.

I've been using Tomcat heavily for 5 years now at my work: 
http://www.maptuit.com. We have multiple Tomcat 
instances, but sessions are bound to individual instances. I've toyed with the 
idea of clustering, but I've shied away from it because I have concerns about 
race conditions in a frameset. Our tomcat apps make very heavy use of nested 
frames, and I'm concerned that in that environment, two requests may hit the 
server "simultaneously", and they may both modify the session. The result would 
be (I believe) a race condition, leaving the state of the session indeterminate 
after the two requests finish.

It might be nice to have a Session Manager with the following characteristics:
1) Centralized session storage;
2) Session locking for the duration of a request (thus avoiding the race 
condition described above);
3) Persistence.

One good way to implement such a Manager would be using an RDBMS. During the 
processing of a request, if the servlet accessed the Session 
(request.getSession()), the Manager would retrieve it from the RDBMS, using a 
write lock (e.g. "SELECT  FOR UPDATE" in MySQL). Upon completion of the 
request, the Manager would write the Session back to the database and COMMIT. 
This would be done in a transactional manner, so that a lock would be held on 
the record for the duration of the request.

Session timeout would have to be handled via periodic checks to the database, 
querying for sessions that had not been accessed in a while.

This would be a very clean, safe way to deal with the race condition scenario. 
Plus of course persistence comes for free.

The implementation would have to be done carefully, using JDBC connection 
pooling. The implementation described above implies that a JDBC connection 
would have to be held open for the duration of the request processing. This 
would have to be done very carefully. A try/finally block should be used, where 
the resources (JDBC connection + transaction + lock) would be released in the 
finally block.

The disadvantage here is performance. In my scenario above the two requests 
would be serialized (the second would block until the first was complete). But 
I do not think that that would be a big hit for many apps. Some extra latency 
loading the session would be incurred, but again for many apps I don't think 
that would be substantial. Also, one JDBC connection would have to be held for 
the duration of each session-sensitive request, so this might not be 
appropriate for applications with very large numbers of users.

I looked through the 5.5.16 code a bit. One issue that I see is that the 
Manager interface would not easily support a try/finally construct like that 
described above. The implementation would have to guarantee that the 
transaction was terminated and the JDBC connection freed up at the end of each 
request, and I'm not sure how to achieve that in Catalina 5.5.

Does anyone have any comments? If there was support for this idea, and if one 
of the tomcat developers was willing to point me in the right direction, I 
would certainly consider contributing the implementation.

Thanks, --Marc Riehm

Re: Suggestion for new Session Manager

[Filip Hanik - Dev Lists]

take a look at the PersistentManager

Marc Riehm wrote:

I would like to make a suggestion for a new Tomcat Session Manager.

I've been using Tomcat heavily for 5 years now at my work: 
http://www.maptuit.com. We have multiple Tomcat instances, but 
sessions are bound to individual instances. I've toyed with the idea of clustering, but I've 
shied away from it because I have concerns about race conditions in a frameset. Our tomcat apps 
make very heavy use of nested frames, and I'm concerned that in that environment, two requests 
may hit the server "simultaneously", and they may both modify the session. The result 
would be (I believe) a race condition, leaving the state of the session indeterminate after the 
two requests finish.

It might be nice to have a Session Manager with the following characteristics:
1) Centralized session storage;
2) Session locking for the duration of a request (thus avoiding the race 
condition described above);
3) Persistence.

One good way to implement such a Manager would be using an RDBMS. During the processing 
of a request, if the servlet accessed the Session (request.getSession()), the Manager 
would retrieve it from the RDBMS, using a write lock (e.g. "SELECT  FOR 
UPDATE" in MySQL). Upon completion of the request, the Manager would write the 
Session back to the database and COMMIT. This would be done in a transactional manner, so 
that a lock would be held on the record for the duration of the request.

Session timeout would have to be handled via periodic checks to the database, 
querying for sessions that had not been accessed in a while.

This would be a very clean, safe way to deal with the race condition scenario. 
Plus of course persistence comes for free.

The implementation would have to be done carefully, using JDBC connection 
pooling. The implementation described above implies that a JDBC connection 
would have to be held open for the duration of the request processing. This 
would have to be done very carefully. A try/finally block should be used, where 
the resources (JDBC connection + transaction + lock) would be released in the 
finally block.

The disadvantage here is performance. In my scenario above the two requests 
would be serialized (the second would block until the first was complete). But 
I do not think that that would be a big hit for many apps. Some extra latency 
loading the session would be incurred, but again for many apps I don't think 
that would be substantial. Also, one JDBC connection would have to be held for 
the duration of each session-sensitive request, so this might not be 
appropriate for applications with very large numbers of users.

I looked through the 5.5.16 code a bit. One issue that I see is that the 
Manager interface would not easily support a try/finally construct like that 
described above. The implementation would have to guarantee that the 
transaction was terminated and the JDBC connection freed up at the end of each 
request, and I'm not sure how to achieve that in Catalina 5.5.

Does anyone have any comments? If there was support for this idea, and if one 
of the tomcat developers was willing to point me in the right direction, I 
would certainly consider contributing the implementation.

Thanks, --Marc Riehm

  



-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

DO NOT REPLY [Bug 39250] New: - Tomcat 3.2.1 + JDK 1.4

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG·
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND·
INSERTED IN THE BUG DATABASE.

http://issues.apache.org/bugzilla/show_bug.cgi?id=39250

   Summary: Tomcat 3.2.1 + JDK 1.4
   Product: Tomcat 3
   Version: 3.2.1 Final
  Platform: Sun
OS/Version: SunOS
Status: NEW
  Severity: critical
  Priority: P3
 Component: Servlet
AssignedTo: tomcat-dev@jakarta.apache.org
ReportedBy: [EMAIL PROTECTED]


I am getting error messages on the console when a https page is invoked. The
error messages starts coming on the console immediately when the certificate
dialog appears.If we do not respond to the certificate dialog box, error
messages will continue to appear on the console and finally causing stack 
overflow.

Following are the error messages appearing on the console:
Ctx( ): 400 R( /) null
2006-04-07 04:54:40 - Ctx( ): IOException in: R( /) Connection closed by remote 
host
2006-04-07 04:54:40 - Ctx( ): IllegalStateException in: R( /) Current state =
FLUSHED, new state = CODING
2006-04-07 04:54:40 - Ctx( ): IllegalStateException in: R( /) Current state =
FLUSHED, new state = CODING
2006-04-07 04:54:40 - Ctx( ): IllegalStateException in: R( /) Current state =
FLUSHED, new state = CODING
.
.
.
2006-04-07 04:54:53 - Ctx( ): IllegalStateException in: R( /) Current state =
FLUSHED, new state = CODING
2006-04-07 04:54:53 - Ctx( ): Exception in: R( /) - java.lang.StackOverflowError
at sun.security.action.GetPropertyAction.run(GetPropertyAction.java:66)
at java.security.AccessController.doPrivileged(Native Method) at
java.io.PrintWriter.(PrintWriter.java:72)
at java.io.PrintWriter.(PrintWriter.java:57)
at 
org.apache.tomcat.context.ExceptionHandler.doService(DefaultCMSetter.java:268)
at org.apache.tomcat.core.Handler.service(Handler.java:286)
at org.apache.tomcat.core.ServletWrapper.service(ServletWrapper.java:372)
at org.apache.tomcat.core.ContextManager.handleError(ContextManager.java:1147)
at org.apache.tomcat.core.Handler.service(Handler.java:311)
at org.apache.tomcat.core.ServletWrapper.service(ServletWrapper.java:372)
at org.apache.tomcat.core.ContextManager.handleError(ContextManager.java:1147)

-- 
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the assignee for the bug, or are watching the assignee.

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

svn commit: r392461 - /tomcat/tc6.0.x/trunk/java/javax/annotation/Resource.java

[remm]

Author: remm
Date: Fri Apr  7 17:12:36 2006
New Revision: 392461

URL: http://svn.apache.org/viewcvs?rev=392461&view=rev
Log:
- Add default value, submitted by Bill Barker.

Modified:
tomcat/tc6.0.x/trunk/java/javax/annotation/Resource.java

Modified: tomcat/tc6.0.x/trunk/java/javax/annotation/Resource.java
URL: 
http://svn.apache.org/viewcvs/tomcat/tc6.0.x/trunk/java/javax/annotation/Resource.java?rev=392461&r1=392460&r2=392461&view=diff
==
--- tomcat/tc6.0.x/trunk/java/javax/annotation/Resource.java (original)
+++ tomcat/tc6.0.x/trunk/java/javax/annotation/Resource.java Fri Apr  7 
17:12:36 2006
@@ -37,5 +37,5 @@
 public AuthenticationType authenticationType() default 
AuthenticationType.CONTAINER;
 public boolean shareable() default true;
 public String description() default "";
-public String mappedName();
+public String mappedName() default "";
 }



-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: Challenges for Java hosting

[Henri Gomez]

And what about a JVM launching others JVMs ?

>From an admin point of vue, only one job/process to start / stop and
monitor, so more simple.

>From a developper point of vue the differents JVM could have the
isolation level required.


2006/4/7, Paul Speed <[EMAIL PROTECTED]>:
>
>
> Reinhard Moosauer wrote:
>
> >
> > Ok. But you can kill the webapp with the amok-thread. So we will not have a
> > break every 3 requests. (the thread can be tracked down to the failing
> > webapp. Send a mail with the thread-stack-dump)
>
> You can't really kill a thread in Java.  We used to have a joke at a
> previous job that the only safe way to kill a thread in Java is
> System.exit(0).
>
> And as I recall, while the javadocs for Thread.stop() detail some of the
> reasons, there is much more badness possible than even what it states.
>
> -Paul
>
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]