PipelineFullException: Queue is full

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

PipelineFullException: Queue is full

Alan Williamson
We are starting to get a lot of these errors ...

---------------------
com.sun.grizzly.Controller doSelect
SEVERE: doSelect exception
com.sun.grizzly.PipelineFullException: Queue is full
at com.sun.grizzly.DefaultPipeline.execute(DefaultPipeline.java:266)
at com.sun.grizzly.DefaultPipeline.execute(DefaultPipeline.java:37)
at com.sun.grizzly.Context.execute(Context.java:316)
---------------------


the server is under relative load, but it doesn't take long.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: PipelineFullException: Queue is full

Jeanfrancois Arcand-2
Hi Alan,

The way Grizzly works is when a request arrive, it accept it and then
put it inside a queue. Once thread is ready/free, it takes the request
from the queue and execute it. Since we can't predict how long the
request will takes to execute (some extension that uses JDBC might that
a long time), we need to make sure the queue is not growing infinitely
to avoid OOM, so we set the max to 4096. The exception you are getting
is because the were 4096 requests queued...to change the default value,
you can call:

SelectorThread.setQueueSizeInBytes(...);

An alternative is to create more Threads so the queue is cleared faster.

Thanks

-- Jeanfrancois





Alan Williamson wrote:

> We are starting to get a lot of these errors ...
>
> ---------------------
> com.sun.grizzly.Controller doSelect
> SEVERE: doSelect exception
> com.sun.grizzly.PipelineFullException: Queue is full
> at com.sun.grizzly.DefaultPipeline.execute(DefaultPipeline.java:266)
> at com.sun.grizzly.DefaultPipeline.execute(DefaultPipeline.java:37)
> at com.sun.grizzly.Context.execute(Context.java:316)
> ---------------------
>
>
> the server is under relative load, but it doesn't take long.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: PipelineFullException: Queue is full

Alan Williamson
thanks for that clarification Jeanfrancois, as it turns out, that
explanation fits with the fact that lighttpd (our frontend load
balancer) had run out of socket handles and was doing a lot of back
queuing and flooding our backends with connections.

Jeanfrancois Arcand wrote:

> Hi Alan,
>
> The way Grizzly works is when a request arrive, it accept it and then
> put it inside a queue. Once thread is ready/free, it takes the request
> from the queue and execute it. Since we can't predict how long the
> request will takes to execute (some extension that uses JDBC might that
> a long time), we need to make sure the queue is not growing infinitely
> to avoid OOM, so we set the max to 4096. The exception you are getting
> is because the were 4096 requests queued...to change the default value,
> you can call:
>
> SelectorThread.setQueueSizeInBytes(...);
>
> An alternative is to create more Threads so the queue is cleared faster.
>
> Thanks
>
> -- Jeanfrancois
>
>
>
>
>
> Alan Williamson wrote:
>> We are starting to get a lot of these errors ...
>>
>> ---------------------
>> com.sun.grizzly.Controller doSelect
>> SEVERE: doSelect exception
>> com.sun.grizzly.PipelineFullException: Queue is full
>> at com.sun.grizzly.DefaultPipeline.execute(DefaultPipeline.java:266)
>> at com.sun.grizzly.DefaultPipeline.execute(DefaultPipeline.java:37)
>> at com.sun.grizzly.Context.execute(Context.java:316)
>> ---------------------
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: PipelineFullException: Queue is full

Alan Williamson
Following on from that thread, can we get a handle on how busy things
are at the moment?  What are the differences between these calls?

        st.getCurrentConnectionNumber()
        st.getCurrentThreadCountStats()
        st.getCountThreadsStats()


Alan Williamson wrote:
> thanks for that clarification Jeanfrancois, as it turns out, that
> explanation fits with the fact that lighttpd (our frontend load
> balancer) had run out of socket handles and was doing a lot of back
> queuing and flooding our backends with connections.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: PipelineFullException: Queue is full

Jeanfrancois Arcand-2
Hi Alan,

Alan Williamson wrote:
> Following on from that thread, can we get a handle on how busy things
> are at the moment?  What are the differences between these calls?
>
>     st.getCurrentConnectionNumber()
>     st.getCurrentThreadCountStats()
>     st.getCountThreadsStats()

Yes, mainly, just do:

SelectorThread.enableMonitoring(). An example is here:

https://grizzly.dev.java.net/nonav/xref-test/com/sun/grizzly/http/SelectorThreadStats.html

Then you can get a lot of statistics by calling like the number of
current opened connections, the number of keep-alive connection, etc.
The objects to looks at are:

// Return information on current connections:
getKeepAliveStats();

// About current Thread state
getCountThreadsStats();
getCurrentThreadCountStats()

// About request/responses (like how many 200/500 etc.)
getGlobalRequestProcessor().getCount200() // How many sucessfull request
getGlobalRequestProcessor().getProcessingTime()

The documentation is so nice here I have to look myself at the code :-(

Some links that can helps:

[Request]
https://grizzly.dev.java.net/nonav/xref/com/sun/grizzly/tcp/RequestInfo.html

[Threads/Pipeline]
https://grizzly.dev.java.net/nonav/xref/com/sun/grizzly/http/PipelineStatistic.html

[Keep-Alive]
https://grizzly.dev.java.net/nonav/xref/com/sun/grizzly/http/KeepAliveStats.html

[Aggregation]
https://grizzly.dev.java.net/nonav/xref/com/sun/grizzly/tcp/RequestGroupInfo.html

Take a look :-)

A+

-- jeanfrancois


>
>
> Alan Williamson wrote:
>> thanks for that clarification Jeanfrancois, as it turns out, that
>> explanation fits with the fact that lighttpd (our frontend load
>> balancer) had run out of socket handles and was doing a lot of back
>> queuing and flooding our backends with connections.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]