"Broken Pipe" and "client is busy or timeout" errors

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

"Broken Pipe" and "client is busy or timeout" errors

wild
This post was updated on .
Hello,

I am facing two kind of excpetion actually with grizlly comet support. When the handler try to write in the response, I sometimes have "Broken Pipe" or "client is busy or timeout".

I am using a concurrent comet handler because I use comet in a streaming mode. I have just modified the DefaultConcurrentCometHandler in order to give a limite a message send by one thread. The comet handler keep a reference on the httpresponse from the comet servlet when the user register.

I am using grizzly 1.0.39 in glassfish v2.

I have read some pieces of code and actually I see that the second exception is due to a timeout on a selector.select() that took too much time. Just for my understanding, did a call to the select method really send the data to the remote client ?

So, in order to try to fix my issues I have to know if they are due to grizzly internal logic (or my logic around grizzly) or due to netwrok settings that are not properly set.

Does anyone can help me ?

Thanks.

William.


Stack Trace :

ClientAbortException:  java.io.IOException: Broken pipe
        at org.apache.coyote.tomcat5.OutputBuffer.doFlush(OutputBuffer.java:386)
        at org.apache.coyote.tomcat5.OutputBuffer.flush(OutputBuffer.java:352)
        at org.apache.coyote.tomcat5.CoyoteOutputStream.flush(CoyoteOutputStream.java:176)
        at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:278)
        at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)
        at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
        ......
        at com.sun.enterprise.web.connector.grizzly.comet.DefaultNotificationHandler.notify(DefaultNotificationHandler.java:149)
        at com.sun.enterprise.web.connector.grizzly.comet.CometContext.notify(CometContext.java:553)
        ......
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcher.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
        at sun.nio.ch.IOUtil.write(IOUtil.java:71)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
        at com.sun.enterprise.web.connector.grizzly.OutputWriter.flushChannel(OutputWriter.java:92)
        at com.sun.enterprise.web.connector.grizzly.OutputWriter.flushChannel(OutputWriter.java:65)
        at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flushChannel(SocketChannelOutputBuffer.java:170)
        at com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.flushChannel(AsynchronousOutputBuffer.java:81)
        at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flushBuffer(SocketChannelOutputBuffer.java:203)
        at com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.flushBuffer(AsynchronousOutputBuffer.java:113)
Reply | Threaded
Open this post in threaded view
|

Re: "Broken Pipe" and "client is busy or timeout" errors

oleksiys
Administrator
Hi William,

pls. try to increase the keep-alive timeout-in-seconds attr [1] in the
GF domain.xml file, make it much bigger and check if you still have
exception coming.

> I am using a concurrent comet handler because I use comet in a streaming
> mode. I have just modified the DefaultConcurrentCometHandler in order to
> give a limite a message send by one thread. The comet handler keep a
> reference on the httpresponse from the comet servlet when the user register.
>
> I am using grizzly 1.0.39 in glassfish v2.
>
> I have read some pieces of code and actually I see that the second exception
> is due to a timeout on a selector.select() that took too much time. Just for
> my understanding, did a call to the select method really send the data to
> the remote client ?
channel.write() sends them, selector.select() just listens for
channel(s) I/O events and wakes up if any is available for processing.

> So, in order to try to fix my issues I have to know if they are due to
> grizzly internal logic (or my logic around grizzly) or due to netwrok
> settings that are not properly set.
Difficult to say, let's try to increase timeout and see.
If it doesn't work, can you pls. provide a simple testcase so we can
take a look at?

Thanks.

WBR,
Alexey.

[1]
<keep-alive max-connections="250" thread-count="1" timeout-in-seconds="30"/>


> Does anyone can help me ?
>
> Thanks.
>
> William.
>
>
> Stack Trace :
>
> ClientAbortException:  java.io.IOException: Broken pipe
>          at
> org.apache.coyote.tomcat5.OutputBuffer.doFlush(OutputBuffer.java:386)
>          at
> org.apache.coyote.tomcat5.OutputBuffer.flush(OutputBuffer.java:352)
>          at
> org.apache.coyote.tomcat5.CoyoteOutputStream.flush(CoyoteOutputStream.java:176)
>          at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:278)
>          at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)
>          at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
>          at
> com.big.comet.GrizzlyCometHandler.doWrite0(GrizzlyCometHandler.java:111)
>          at
> com.big.comet.GrizzlyCometHandler.doWrite(GrizzlyCometHandler.java:90)
>          at
> com.big.comet.GrizzlyCometHandler.writeEvent(GrizzlyCometHandler.java:99)
>          at
> com.big.comet.GrizzlyCometHandler.onEvent(GrizzlyCometHandler.java:129)
>          at
> com.big.comet.concurrent.GrizzlyConcurrentCometHandler.onEvent(GrizzlyConcurrentCometHandler.java:188)
>          at
> com.big.comet.concurrent.ConcurrentNotificationHandler.notify0(ConcurrentNotificationHandler.java:35)
>          at
> com.sun.enterprise.web.connector.grizzly.comet.DefaultNotificationHandler.notify(DefaultNotificationHandler.java:149)
>          at
> com.sun.enterprise.web.connector.grizzly.comet.CometContext.notify(CometContext.java:553)
>          at
> com.big.comet.GrizzlyMessageDispatcher$1.sendMessageToClient(GrizzlyMessageDispatcher.java:817)
>          at
> com.big.comet.GrizzlyMessageDispatcher$1.run(GrizzlyMessageDispatcher.java:806)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>          at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: Broken pipe
>          at sun.nio.ch.FileDispatcher.write0(Native Method)
>          at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>          at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
>          at sun.nio.ch.IOUtil.write(IOUtil.java:71)
>          at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>          at
> com.sun.enterprise.web.connector.grizzly.OutputWriter.flushChannel(OutputWriter.java:92)
>          at
> com.sun.enterprise.web.connector.grizzly.OutputWriter.flushChannel(OutputWriter.java:65)
>          at
> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flushChannel(SocketChannelOutputBuffer.java:170)
>          at
> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.flushChannel(AsynchronousOutputBuffer.java:81)
>          at
> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flushBuffer(SocketChannelOutputBuffer.java:203)
>          at
> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.flushBuffer(AsynchronousOutputBuffer.java:113)
>
>
> --
> View this message in context: http://grizzly.1045725.n5.nabble.com/Broken-Pipe-and-client-is-busy-or-timeout-errors-tp4268012p4268012.html
> Sent from the Grizzly - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: "Broken Pipe" and "client is busy or timeout" errors

wild
ok I will try to increase the timeout.... This change should fix the 2 errors ("Broken Pipe" or "client is busy or timeout") or only one ? Can you explain why it should help ?

if channel.write send the data, why do we have to call selector.select() ? After som check on client side, it seams that the connection is still alive but selector.select() is returning 0.

it is difficult to provide a test case because it happened sometimes, mainly when the load increase. Our use of comet is simple, we have implemented a comet handler inspired from ConcurrentCometHandler and we write in the httpResponse each time we have to write an event.

We implement our own thread pool in order to handler priority and use several comet context in the same comet engine.

thanks

william
Reply | Threaded
Open this post in threaded view
|

Re: "Broken Pipe" and "client is busy or timeout" errors

wild
In reply to this post by oleksiys
One more thing, when I display grizzly configuration in server.log file, it shows

socket-keep-alive-enabled: false

In order to make keep alive timeout active, should it be true ?
Reply | Threaded
Open this post in threaded view
|

Re: "Broken Pipe" and "client is busy or timeout" errors

oleksiys
Administrator
In reply to this post by wild
On 03/28/2011 02:45 PM, wild wrote:
> ok I will try to increase the timeout.... This change should fix the 2 errors
> ("Broken Pipe" or "client is busy or timeout") or only one ? Can you explain
> why it should help ?
Primarily "broken pipe", not sure where the 2nd exception comes from yet.

> if channel.write send the data, why do we have to call selector.select() ?
> After som check on client side, it seams that the connection is still alive
> but selector.select() is returning 0.
It depends which selector.select() you mean, the one in
OutputWriter.flushChannel()?
It's getting called when we can not flush the entire buffer, because the
underlying socket buffer is overloaded. (most probably client is not
able to read data as fast as server sends it). So we just wait until
client releases some space in the TCP buffer and let server write more data.

You can increase the write timeout as well by setting jvm-option
-Dcom.sun.enterprise.web.connector.grizzly.writeTimeout=XXX
in the domain.xml, the value is in milliseconds, by default it's equal
to 30000 (30 seconds).

> it is difficult to provide a test case because it happened sometimes, mainly
> when the load increase. Our use of comet is simple, we have implemented a
> comet handler inspired from ConcurrentCometHandler and we write in the
> httpResponse each time we have to write an event.
>
> We implement our own thread pool in order to handler priority and use
> several comet context in the same comet engine.
I see. Let's check what timeouts increasing will give us.

Thanks.

WBR,
Alexey.

> thanks
>
> william
>
> --
> View this message in context: http://grizzly.1045725.n5.nabble.com/Broken-Pipe-and-client-is-busy-or-timeout-errors-tp4268012p4268394.html
> Sent from the Grizzly - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: "Broken Pipe" and "client is busy or timeout" errors

oleksiys
Administrator
In reply to this post by wild
One more thing, when I display grizzly configuration in server.log file, it
> shows
>
> socket-keep-alive-enabled: false
>
> In order to make keep alive timeout active, should it be true ?
>
It's a bit different, this one is socket keep-alive and we're trying to
increase *HTTP* keep-live timeout :)

WBR,
Alexey.

> --
> View this message in context: http://grizzly.1045725.n5.nabble.com/Broken-Pipe-and-client-is-busy-or-timeout-errors-tp4268012p4268416.html
> Sent from the Grizzly - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: "Broken Pipe" and "client is busy or timeout" errors

wild
Changing HTTP keep alive does not change anything....

I have seen there is a bug GRIZZLY-959 about keepalive when acceptor thread is greater than 1...

I have patched my glassfish v2 with grizzly 1.0.39 by changing the prefix classpath. I took the file from grizzly repo (http://download.java.net/maven/2/com/sun/grizzly/grizzly-framework-http/1.0.39/).

I will try to increase socket buffer. Do you think it can help  ?


         
Reply | Threaded
Open this post in threaded view
|

Re: "Broken Pipe" and "client is busy or timeout" errors

oleksiys
Administrator

> Changing HTTP keep alive does not change anything....
So you still see broken pipe exception, right?

> I have seen there is a bug GRIZZLY-959 about keepalive when acceptor thread
> is greater than 1...
>
> I have patched my glassfish v2 with grizzly 1.0.39 by changing the prefix
> classpath. I took the file from grizzly repo
> (http://download.java.net/maven/2/com/sun/grizzly/grizzly-framework-http/1.0.39/).
Right, it's what you told in the 1st post, so it shouldn't be the case.

> I will try to increase socket buffer. Do you think it can help  ?
 From what you're reporting, it looks like bottleneck occurs when server
writes response to client, which should mean client is not able to
process server response as fast as server sends it.
If that's the case - you can try to increase server write timeout i
mentioned in prev. post.

The increasing of socket buffer may help to delay the problem, but not
sure if it would resolve it. Depends on your concrete usecase.

WBR,
Alexey.

>
>
>
> --
> View this message in context: http://grizzly.1045725.n5.nabble.com/Broken-Pipe-and-client-is-busy-or-timeout-errors-tp4268012p4268940.html
> Sent from the Grizzly - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: "Broken Pipe" and "client is busy or timeout" errors

wild
Yes I still have broken pipe exceptions... I have read that 5 minutes is the maximum keep alive timeout so i didn't increase it more than 5 minutes.

For the "client too busy" exception :

My problem is that we send a lot of "message" to the client connection, and one after another.

The sequence is

while(stillMessageToSend or max message limit not reached) {
    response.write(message);
    response.flush();
}

So If a increase the server timeout, i will keep a thread to many times and enqueue too many messages. If the client is too slow to read messages, then i think the only solution is ti reduce the number of messages send.

I will try to increase it but we need to send fast.....