Fairness of request processing in Glassfish

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Fairness of request processing in Glassfish

Jesper Söderlund
I wonder if somebody could enlighten me with regards to the fairness of processing requests in Glassfish / Grizzly under load.

Is there some some guarantee that one heavily loaded application will not hog all the resources from another using the same grizzly instance?

Is the only way to "partion" the resources to create separate HTTP listeners?

Will these threads configured for the HTTP-listener run through the servlet container and down into the EJB-container?

It would be great to be able to specify a weight for an application in the deployment descriptor and have the request processing scheduled accordingly.

/Jesper

Reply | Threaded
Open this post in threaded view
|

Re: Fairness of request processing in Glassfish

Jeanfrancois Arcand-2
Salut,

Jesper Söderlund wrote:
> I wonder if somebody could enlighten me with regards to the fairness of
> processing requests in Glassfish / Grizzly under load.

By default, same fairness for every requests are applied, meaning the
first requests that comes in is always the first executed (something
Tomcat doesn't do). This is quite important and one of the reason why we
have good score with specJ2004: all requests have the same chance to be
executed. This is the SEDA part of Grizzly that allow use to do that :-)

>
> Is there some some guarantee that one heavily loaded application will
> not hog all the resources from another using the same grizzly instance?

By default, if an application takes more CPU/resources than the other,
Grizzly doesn't have a mechanism to isolate the application by default.
I said by default because Grizzly have a extension (with both 1.0 and
1.5) that allow resource consumptions management [1]. This is something
you should take  alook.

>
> Is the only way to "partion" the resources to create separate HTTP
> listeners?

No, see [2] for a good example.

>
> Will these threads configured for the HTTP-listener run through the
> servlet container and down into the EJB-container?

Yes.

>
> It would be great to be able to specify a weight for an application in
> the deployment descriptor and have the request processing scheduled
> accordingly.

Look at [1][2] :-) This is exactly what you need. 4homemedia.com
combined [2] with Grizzly Comet for their application and they told me
their performance has improved by isolating applications.

Thanks!

-- Jeanfrancois

[1]
http://weblogs.java.net/blog/jfarcand/archive/2006/04/resource_consum.html
[2]
http://weblogs.java.net/blog/jfarcand/archive/2007/06/improving_ajax_1.html

>
> /Jesper
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Fairness of request processing in Glassfish

Jesper Söderlund
Hmm, yes those two articles certainly point the way.

Any idea when a RCM functionality would be a supported option in Glassfish?

Some comments on the policies:
- reservation - This means that we have to over-allocate resources which are then partitioned into separate "pools", this way to allocating a percentage to each app is rather cumbersome to manage (even if there was a snazzy GUI instead of the system properties) since the balance will need to be reconfigured with each new application deployed.

- ceiling - Unfortunatly I did not fully understand the semantics of the ceiling policy, but I venture a guess that it's a common pool and that an app is maximally allowed a certain percentage allocation from that  pool, else it will block for a while, this seems quite restrictive but is effective at blocking "run-away" applications so it could be useful to set the ceiling quite high to prevent DDoS type of attacks (or bugs with the similar effect)

I'd also like to see a pool with a weighted policy where each app is allocated a weight and the resources allocated according to that weight, that would be much more robust, in terms of administration at least, incidentally this is also one of the algorithm available in Solaris resource management, perhaps some lessons can be learned from that? The weight per application should be configurable in the deployment description as well as in the admin GUI to override the deployment descriptor value. Applications which don't have a value would be assigned a default value.

Though I understand the motivation for having a default policy which optimizes on performance as that is quite visible in benchmarks, etc, but I think it would better if an policy which would prioritize fairness between applications would be just as easily configured or maybe even the default and the performance policy the exception which is configured for benchmark tests.

cheers,
/Jesper



2007/11/21, Jeanfrancois Arcand <[hidden email]>:
Salut,

Jesper Söderlund wrote:
> I wonder if somebody could enlighten me with regards to the fairness of
> processing requests in Glassfish / Grizzly under load.

By default, same fairness for every requests are applied, meaning the
first requests that comes in is always the first executed (something
Tomcat doesn't do). This is quite important and one of the reason why we
have good score with specJ2004: all requests have the same chance to be
executed. This is the SEDA part of Grizzly that allow use to do that :-)

>
> Is there some some guarantee that one heavily loaded application will
> not hog all the resources from another using the same grizzly instance?

By default, if an application takes more CPU/resources than the other,
Grizzly doesn't have a mechanism to isolate the application by default.
I said by default because Grizzly have a extension (with both 1.0 and
1.5) that allow resource consumptions management [1]. This is something
you should take  alook.

>
> Is the only way to "partion" the resources to create separate HTTP
> listeners?

No, see [2] for a good example.

>
> Will these threads configured for the HTTP-listener run through the
> servlet container and down into the EJB-container?

Yes.

>
> It would be great to be able to specify a weight for an application in
> the deployment descriptor and have the request processing scheduled
> accordingly.

Look at [1][2] :-) This is exactly what you need. 4homemedia.com
combined [2] with Grizzly Comet for their application and they told me
their performance has improved by isolating applications.

Thanks!

-- Jeanfrancois

[1]
http://weblogs.java.net/blog/jfarcand/archive/2006/04/resource_consum.html
[2]
http://weblogs.java.net/blog/jfarcand/archive/2007/06/improving_ajax_1.html

>
> /Jesper
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Fairness of request processing in Glassfish

Jeanfrancois Arcand-2
Salut,

Jesper Söderlund wrote:
> Hmm, yes those two articles certainly point the way.
>
> Any idea when a RCM functionality would be a supported option in Glassfish?

It is a supported options in v2, but not enabled by default. The
configuration part needs to be improved (via admin-gui)..I agree :-)


>
> Some comments on the policies:
> - reservation - This means that we have to over-allocate resources which
> are then partitioned into separate "pools", this way to allocating a
> percentage to each app is rather cumbersome to manage (even if there was
> a snazzy GUI instead of the system properties) since the balance will
> need to be reconfigured with each new application deployed.

Indeed. Every request to a newly deployed application will be rejected
(503) if the other deployed applications have reserved 100% of the
thread pool. But you can always leave a small amount of the thread count
available (let's say 10%).

>
> - ceiling - Unfortunatly I did not fully understand the semantics of the
> ceiling policy, but I venture a guess that it's a common pool and that
> an app is maximally allowed a certain percentage allocation from that  
> pool, else it will block for a while, this seems quite restrictive but
> is effective at blocking "run-away" applications so it could be useful
> to set the ceiling quite high to prevent DDoS type of attacks (or bugs
> with the similar effect)

Right. The main difference between the two is with reservation, if App A
has 30% allocated threads, App B 50%, and 20% left for other
application, if App A needs more that 30%, it can always use the 20%
unallocated. With ceiling, if 30% of the threads are used, then requests
to App A will not be executed (but instead parked...similar to a
continuation).

>
> I'd also like to see a pool with a weighted policy where each app is
> allocated a weight and the resources allocated according to that weight,

If I understand you correctly, one thread pool but a dispatcher that
pick up threads from the pool using the weight?


> that would be much more robust, in terms of administration at least,
> incidentally this is also one of the algorithm available in Solaris
> resource management, perhaps some lessons can be learned from that?

Agree. The RCM stuff is probably really similar to zones as well.

The
> weight per application should be configurable in the deployment
> description as well as in the admin GUI to override the deployment
> descriptor value. Applications which don't have a value would be
> assigned a default value.
>

Agree. I've filled this but never has a chance to fix it:

https://glassfish.dev.java.net/issues/show_bug.cgi?id=3058

> Though I understand the motivation for having a default policy which
> optimizes on performance as that is quite visible in benchmarks, etc,
> but I think it would better if an policy which would prioritize fairness
> between applications would be just as easily configured or maybe even
> the default and the performance policy the exception which is configured
> for benchmark tests.

Agree. You should reply to the v3 discussion in GlassFish asking to have
rcm enabled by default and better supported. I will speak on my side as
well :-)

A+

-- Jeanfrancois

>
> cheers,
> /Jesper
>
>
>
> 2007/11/21, Jeanfrancois Arcand <[hidden email]
> <mailto:[hidden email]>>:
>
>     Salut,
>
>     Jesper Söderlund wrote:
>      > I wonder if somebody could enlighten me with regards to the
>     fairness of
>      > processing requests in Glassfish / Grizzly under load.
>
>     By default, same fairness for every requests are applied, meaning the
>     first requests that comes in is always the first executed (something
>     Tomcat doesn't do). This is quite important and one of the reason why we
>     have good score with specJ2004: all requests have the same chance to be
>     executed. This is the SEDA part of Grizzly that allow use to do that :-)
>
>      >
>      > Is there some some guarantee that one heavily loaded application will
>      > not hog all the resources from another using the same grizzly
>     instance?
>
>     By default, if an application takes more CPU/resources than the other,
>     Grizzly doesn't have a mechanism to isolate the application by default.
>     I said by default because Grizzly have a extension (with both 1.0 and
>     1.5) that allow resource consumptions management [1]. This is something
>     you should take  alook.
>
>      >
>      > Is the only way to "partion" the resources to create separate HTTP
>      > listeners?
>
>     No, see [2] for a good example.
>
>      >
>      > Will these threads configured for the HTTP-listener run through the
>      > servlet container and down into the EJB-container?
>
>     Yes.
>
>      >
>      > It would be great to be able to specify a weight for an
>     application in
>      > the deployment descriptor and have the request processing scheduled
>      > accordingly.
>
>     Look at [1][2] :-) This is exactly what you need. 4homemedia.com
>     <http://4homemedia.com>
>     combined [2] with Grizzly Comet for their application and they told me
>     their performance has improved by isolating applications.
>
>     Thanks!
>
>     -- Jeanfrancois
>
>     [1]
>     http://weblogs.java.net/blog/jfarcand/archive/2006/04/resource_consum.html
>     <http://weblogs.java.net/blog/jfarcand/archive/2006/04/resource_consum.html>
>     [2]
>     http://weblogs.java.net/blog/jfarcand/archive/2007/06/improving_ajax_1.html
>
>      >
>      > /Jesper
>      >
>
>     ---------------------------------------------------------------------
>     To unsubscribe, e-mail: [hidden email]
>     <mailto:[hidden email]>
>     For additional commands, e-mail: [hidden email]
>     <mailto:[hidden email]>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Fairness of request processing in Glassfish

Jesper Söderlund
Yes, one pool and allocation by weight.

Cheers,
/Jesper

2007/11/22, Jeanfrancois Arcand <[hidden email]>:
Salut,

Jesper Söderlund wrote:
> Hmm, yes those two articles certainly point the way.
>
> Any idea when a RCM functionality would be a supported option in Glassfish?

It is a supported options in v2, but not enabled by default. The
configuration part needs to be improved (via admin-gui)..I agree :-)


>
> Some comments on the policies:
> - reservation - This means that we have to over-allocate resources which
> are then partitioned into separate "pools", this way to allocating a
> percentage to each app is rather cumbersome to manage (even if there was
> a snazzy GUI instead of the system properties) since the balance will
> need to be reconfigured with each new application deployed.

Indeed. Every request to a newly deployed application will be rejected
(503) if the other deployed applications have reserved 100% of the
thread pool. But you can always leave a small amount of the thread count
available (let's say 10%).

>
> - ceiling - Unfortunatly I did not fully understand the semantics of the
> ceiling policy, but I venture a guess that it's a common pool and that
> an app is maximally allowed a certain percentage allocation from that
> pool, else it will block for a while, this seems quite restrictive but
> is effective at blocking "run-away" applications so it could be useful
> to set the ceiling quite high to prevent DDoS type of attacks (or bugs
> with the similar effect)

Right. The main difference between the two is with reservation, if App A
has 30% allocated threads, App B 50%, and 20% left for other
application, if App A needs more that 30%, it can always use the 20%
unallocated. With ceiling, if 30% of the threads are used, then requests
to App A will not be executed (but instead parked...similar to a
continuation).

>
> I'd also like to see a pool with a weighted policy where each app is
> allocated a weight and the resources allocated according to that weight,

If I understand you correctly, one thread pool but a dispatcher that
pick up threads from the pool using the weight?


> that would be much more robust, in terms of administration at least,
> incidentally this is also one of the algorithm available in Solaris
> resource management, perhaps some lessons can be learned from that?

Agree. The RCM stuff is probably really similar to zones as well.

The
> weight per application should be configurable in the deployment
> description as well as in the admin GUI to override the deployment
> descriptor value. Applications which don't have a value would be
> assigned a default value.
>

Agree. I've filled this but never has a chance to fix it:

https://glassfish.dev.java.net/issues/show_bug.cgi?id=3058

> Though I understand the motivation for having a default policy which
> optimizes on performance as that is quite visible in benchmarks, etc,
> but I think it would better if an policy which would prioritize fairness
> between applications would be just as easily configured or maybe even
> the default and the performance policy the exception which is configured
> for benchmark tests.

Agree. You should reply to the v3 discussion in GlassFish asking to have
rcm enabled by default and better supported. I will speak on my side as
well :-)

A+

-- Jeanfrancois

>
> cheers,
> /Jesper
>
>
>
> 2007/11/21, Jeanfrancois Arcand < [hidden email]
> <mailto:[hidden email]>>:
>
>     Salut,
>
>     Jesper Söderlund wrote:
>      > I wonder if somebody could enlighten me with regards to the
>     fairness of
>      > processing requests in Glassfish / Grizzly under load.
>
>     By default, same fairness for every requests are applied, meaning the
>     first requests that comes in is always the first executed (something
>     Tomcat doesn't do). This is quite important and one of the reason why we
>     have good score with specJ2004: all requests have the same chance to be
>     executed. This is the SEDA part of Grizzly that allow use to do that :-)
>
>      >
>      > Is there some some guarantee that one heavily loaded application will
>      > not hog all the resources from another using the same grizzly
>     instance?
>
>     By default, if an application takes more CPU/resources than the other,
>     Grizzly doesn't have a mechanism to isolate the application by default.
>     I said by default because Grizzly have a extension (with both 1.0 and
>     1.5) that allow resource consumptions management [1]. This is something
>     you should take  alook.
>
>      >
>      > Is the only way to "partion" the resources to create separate HTTP
>      > listeners?
>
>     No, see [2] for a good example.
>
>      >
>      > Will these threads configured for the HTTP-listener run through the
>      > servlet container and down into the EJB-container?
>
>     Yes.
>
>      >
>      > It would be great to be able to specify a weight for an
>     application in
>      > the deployment descriptor and have the request processing scheduled
>      > accordingly.
>
>     Look at [1][2] :-) This is exactly what you need. 4homemedia.com
>     <http://4homemedia.com >
>     combined [2] with Grizzly Comet for their application and they told me
>     their performance has improved by isolating applications.
>
>     Thanks!
>
>     -- Jeanfrancois
>
>     [1]
>     http://weblogs.java.net/blog/jfarcand/archive/2006/04/resource_consum.html
>     < http://weblogs.java.net/blog/jfarcand/archive/2006/04/resource_consum.html>
>     [2]
>     http://weblogs.java.net/blog/jfarcand/archive/2007/06/improving_ajax_1.html
>
>      >
>      > /Jesper
>      >
>
>     ---------------------------------------------------------------------
>     To unsubscribe, e-mail: [hidden email]
>     <mailto:[hidden email]>
>     For additional commands, e-mail: [hidden email]
>     <mailto:[hidden email]>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]