http keep-alive

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

http keep-alive

Peter Speck
Hi,

I have a problem using Grizzly 1.6.0: the server becomes very slow at  
responding after a few seconds of load. I launch the server using:

SelectorThread st = new SelectorThread();
st.setPort(80);
st.setAdapter(new ProxyAdapter());
st.setCompression("off");
st.setDisplayConfiguration(true);
st.setFileCacheIsEnabled(false);
st.setLargeFileCacheEnabled(false);
st.setBufferResponse(false);
st.setKaTimeout(20);
st.setMaxThreads(50);
st.setBufferSize(32768);
st.setMaxKeepAliveRequests(8196);
st.initEndpoint();
st.startEndpoint();

(the server is an http proxy which forwards requests to other http  
servers).

I made a thread dump when the server didn't respond.  4 threads are  
in com.sun.grizzly.tcp.Response.finish, and the rest are like this:

"httpHttpWorkerThread-80-49" daemon prio=10 tid=0x0052c750  
nid=0x18bd600 in Object.wait() [0xf2b46000..0xf2b46ab0]
     at java.lang.Object.wait(Native Method)
     - waiting on <0x24cdadc0> (a java.util.Stack)
     at com.sun.grizzly.util.SelectorFactory.getSelector
(SelectorFactory.java:84)
     - locked <0x24cdadc0> (a java.util.Stack)
     at com.sun.grizzly.util.ByteBufferInputStream.doClearRead
(ByteBufferInputStream.java:306)
     at com.sun.grizzly.util.ByteBufferInputStream.doRead
(ByteBufferInputStream.java:259)
     at com.sun.grizzly.util.ByteBufferInputStream.read
(ByteBufferInputStream.java:185)
     at com.sun.grizzly.tcp.http11.InternalInputBuffer.fill
(InternalInputBuffer.java:708)
     at  
com.sun.grizzly.tcp.http11.InternalInputBuffer.parseRequestLine
(InternalInputBuffer.java:382)
     at com.sun.grizzly.http.DefaultProcessorTask.parseRequest
(DefaultProcessorTask.java:645)
     at com.sun.grizzly.http.DefaultProcessorTask.doProcess
(DefaultProcessorTask.java:527)
     at com.sun.grizzly.http.DefaultProcessorTask.process
(DefaultProcessorTask.java:773)
     at com.sun.grizzly.http.DefaultProtocolFilter.execute
(DefaultProtocolFilter.java:133)
     at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter
(DefaultProtocolChain.java:89)
     at com.sun.grizzly.DefaultProtocolChain.execute
(DefaultProtocolChain.java:67)
     at com.sun.grizzly.http.SelectorThread$2.execute
(SelectorThread.java:676)
     at com.sun.grizzly.Context.call(Context.java:266)
     at com.sun.grizzly.util.WorkerThreadImpl.run
(WorkerThreadImpl.java:179)

Is this busy-waiting for keep-alive, or is it something else?



The main grizzly thread:
"GriProxy-T-80:127.0.0.1:4000" prio=5 tid=0x0050d830 nid=0x1877a00  
runnable [0xf0d0a000..0xf0d0aab0]
     at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
     at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:118)
     at sun.nio.ch.KQueueSelectorImpl.doSelect
(KQueueSelectorImpl.java:69)
     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
     - locked <0x25e331d0> (a sun.nio.ch.Util$1)
     - locked <0x25e331e0> (a java.util.Collections$UnmodifiableSet)
     - locked <0x25e33188> (a sun.nio.ch.KQueueSelectorImpl)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
     at com.sun.grizzly.TCPSelectorHandler.select
(TCPSelectorHandler.java:344)
     at com.sun.grizzly.Controller.doSelect(Controller.java:280)
     at com.sun.grizzly.Controller.start(Controller.java:788)
     - locked <0x24e18320> (a java.lang.Object)
     at com.sun.grizzly.http.SelectorThread.startListener
(SelectorThread.java:1080)
     - locked <0x24de28b8> (a [Ljava.lang.Object;)
     at com.sun.grizzly.http.SelectorThread.startEndpoint
(SelectorThread.java:1052)
     at dk.vitality.util.socketProxy.server.grizzly.GriProxy.main
(GriProxy.java:96)
     at dk.vitality.util.socketProxy.server.grizzly.GriProxy$1.run
(GriProxy.java:61)
     at java.lang.Thread.run(Thread.java:613)


The adapter is:

class ProxyAdapter implements Adapter
{
    public void service(Request request, Response response) throws  
Exception
    {
       request.action(ActionCode.ACTION_REQ_HOST_ADDR_ATTRIBUTE, null);
       new RequestHandler(GriProxy.this, request, response);
    }

    public void afterService(Request request, Response response)  
throws Exception
    {
       request.recycle();
       response.recycle();
    }

    public void fireAdapterEvent(String string, Object object)
    {
    }
}


----
    - Peter Speck


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: http keep-alive

Oleksiy Stashok
Hello Peter,

which OS are you using?

WBR,
Alexey.

Peter Speck wrote:

> Hi,
>
> I have a problem using Grizzly 1.6.0: the server becomes very slow at
> responding after a few seconds of load. I launch the server using:
>
> SelectorThread st = new SelectorThread();
> st.setPort(80);
> st.setAdapter(new ProxyAdapter());
> st.setCompression("off");
> st.setDisplayConfiguration(true);
> st.setFileCacheIsEnabled(false);
> st.setLargeFileCacheEnabled(false);
> st.setBufferResponse(false);
> st.setKaTimeout(20);
> st.setMaxThreads(50);
> st.setBufferSize(32768);
> st.setMaxKeepAliveRequests(8196);
> st.initEndpoint();
> st.startEndpoint();
>
> (the server is an http proxy which forwards requests to other http
> servers).
>
> I made a thread dump when the server didn't respond.  4 threads are in
> com.sun.grizzly.tcp.Response.finish, and the rest are like this:
>
> "httpHttpWorkerThread-80-49" daemon prio=10 tid=0x0052c750
> nid=0x18bd600 in Object.wait() [0xf2b46000..0xf2b46ab0]
>     at java.lang.Object.wait(Native Method)
>     - waiting on <0x24cdadc0> (a java.util.Stack)
>     at
> com.sun.grizzly.util.SelectorFactory.getSelector(SelectorFactory.java:84)
>     - locked <0x24cdadc0> (a java.util.Stack)
>     at
> com.sun.grizzly.util.ByteBufferInputStream.doClearRead(ByteBufferInputStream.java:306)
>
>     at
> com.sun.grizzly.util.ByteBufferInputStream.doRead(ByteBufferInputStream.java:259)
>
>     at
> com.sun.grizzly.util.ByteBufferInputStream.read(ByteBufferInputStream.java:185)
>
>     at
> com.sun.grizzly.tcp.http11.InternalInputBuffer.fill(InternalInputBuffer.java:708)
>
>     at
> com.sun.grizzly.tcp.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:382)
>
>     at
> com.sun.grizzly.http.DefaultProcessorTask.parseRequest(DefaultProcessorTask.java:645)
>
>     at
> com.sun.grizzly.http.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:527)
>
>     at
> com.sun.grizzly.http.DefaultProcessorTask.process(DefaultProcessorTask.java:773)
>
>     at
> com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:133)
>
>     at
> com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:89)
>
>     at
> com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:67)
>
>     at
> com.sun.grizzly.http.SelectorThread$2.execute(SelectorThread.java:676)
>     at com.sun.grizzly.Context.call(Context.java:266)
>     at
> com.sun.grizzly.util.WorkerThreadImpl.run(WorkerThreadImpl.java:179)
>
> Is this busy-waiting for keep-alive, or is it something else?
>
>
>
> The main grizzly thread:
> "GriProxy-T-80:127.0.0.1:4000" prio=5 tid=0x0050d830 nid=0x1877a00
> runnable [0xf0d0a000..0xf0d0aab0]
>     at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>     at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:118)
>     at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:69)
>     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     - locked <0x25e331d0> (a sun.nio.ch.Util$1)
>     - locked <0x25e331e0> (a java.util.Collections$UnmodifiableSet)
>     - locked <0x25e33188> (a sun.nio.ch.KQueueSelectorImpl)
>     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     at
> com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:344)
>     at com.sun.grizzly.Controller.doSelect(Controller.java:280)
>     at com.sun.grizzly.Controller.start(Controller.java:788)
>     - locked <0x24e18320> (a java.lang.Object)
>     at
> com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1080)
>
>     - locked <0x24de28b8> (a [Ljava.lang.Object;)
>     at
> com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1052)
>
>     at
> dk.vitality.util.socketProxy.server.grizzly.GriProxy.main(GriProxy.java:96)
>
>     at
> dk.vitality.util.socketProxy.server.grizzly.GriProxy$1.run(GriProxy.java:61)
>
>     at java.lang.Thread.run(Thread.java:613)
>
>
> The adapter is:
>
> class ProxyAdapter implements Adapter
> {
>    public void service(Request request, Response response) throws
> Exception
>    {
>       request.action(ActionCode.ACTION_REQ_HOST_ADDR_ATTRIBUTE, null);
>       new RequestHandler(GriProxy.this, request, response);
>    }
>
>    public void afterService(Request request, Response response) throws
> Exception
>    {
>       request.recycle();
>       response.recycle();
>    }
>
>    public void fireAdapterEvent(String string, Object object)
>    {
>    }
> }
>
>
> ----
>    - Peter Speck
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: http keep-alive

Peter Speck

On 03/09/2007, at 10:11, Oleksiy Stashok wrote:

> which OS are you using?

Mac OS X 10.4.10 on PowerPC.

java version "1.5.0_06"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-112)
Java HotSpot(TM) Client VM (build 1.5.0_06-64, mixed mode, sharing)


----
    - Peter Speck


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: http keep-alive

Oleksiy Stashok
Hello Peter,
please try latest Grizzly 1.6.0 sources, I've fixed one possible
problem, hope it will help.

WBR,
Alexey.

Peter Speck wrote:

>
> On 03/09/2007, at 10:11, Oleksiy Stashok wrote:
>
>> which OS are you using?
>
> Mac OS X 10.4.10 on PowerPC.
>
> java version "1.5.0_06"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-112)
> Java HotSpot(TM) Client VM (build 1.5.0_06-64, mixed mode, sharing)
>
>
> ----
>    - Peter Speck
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: http keep-alive

Jeanfrancois Arcand-2
In reply to this post by Peter Speck
Hi Peter,

I think is the continuousExecution feature I've ported from Grizzly 1.0
that is causing problem. I did test it (and ran several perf benchmark)
but it seems there is still a bug . Mainly, there is no longer bytes
availables and the ProtocolChain needs to abort it execution, which is
clear it is not working right now.

Fix coming...

-- Jeanfrancois

Peter Speck wrote:

> Hi,
>
> I have a problem using Grizzly 1.6.0: the server becomes very slow at
> responding after a few seconds of load. I launch the server using:
>
> SelectorThread st = new SelectorThread();
> st.setPort(80);
> st.setAdapter(new ProxyAdapter());
> st.setCompression("off");
> st.setDisplayConfiguration(true);
> st.setFileCacheIsEnabled(false);
> st.setLargeFileCacheEnabled(false);
> st.setBufferResponse(false);
> st.setKaTimeout(20);
> st.setMaxThreads(50);
> st.setBufferSize(32768);
> st.setMaxKeepAliveRequests(8196);
> st.initEndpoint();
> st.startEndpoint();
>
> (the server is an http proxy which forwards requests to other http
> servers).
>
> I made a thread dump when the server didn't respond.  4 threads are in
> com.sun.grizzly.tcp.Response.finish, and the rest are like this:
>
> "httpHttpWorkerThread-80-49" daemon prio=10 tid=0x0052c750 nid=0x18bd600
> in Object.wait() [0xf2b46000..0xf2b46ab0]
>     at java.lang.Object.wait(Native Method)
>     - waiting on <0x24cdadc0> (a java.util.Stack)
>     at
> com.sun.grizzly.util.SelectorFactory.getSelector(SelectorFactory.java:84)
>     - locked <0x24cdadc0> (a java.util.Stack)
>     at
> com.sun.grizzly.util.ByteBufferInputStream.doClearRead(ByteBufferInputStream.java:306)
>
>     at
> com.sun.grizzly.util.ByteBufferInputStream.doRead(ByteBufferInputStream.java:259)
>
>     at
> com.sun.grizzly.util.ByteBufferInputStream.read(ByteBufferInputStream.java:185)
>
>     at
> com.sun.grizzly.tcp.http11.InternalInputBuffer.fill(InternalInputBuffer.java:708)
>
>     at
> com.sun.grizzly.tcp.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:382)
>
>     at
> com.sun.grizzly.http.DefaultProcessorTask.parseRequest(DefaultProcessorTask.java:645)
>
>     at
> com.sun.grizzly.http.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:527)
>
>     at
> com.sun.grizzly.http.DefaultProcessorTask.process(DefaultProcessorTask.java:773)
>
>     at
> com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:133)
>
>     at
> com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:89)
>
>     at
> com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:67)
>     at
> com.sun.grizzly.http.SelectorThread$2.execute(SelectorThread.java:676)
>     at com.sun.grizzly.Context.call(Context.java:266)
>     at com.sun.grizzly.util.WorkerThreadImpl.run(WorkerThreadImpl.java:179)
>
> Is this busy-waiting for keep-alive, or is it something else?
>
>
>
> The main grizzly thread:
> "GriProxy-T-80:127.0.0.1:4000" prio=5 tid=0x0050d830 nid=0x1877a00
> runnable [0xf0d0a000..0xf0d0aab0]
>     at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>     at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:118)
>     at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:69)
>     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     - locked <0x25e331d0> (a sun.nio.ch.Util$1)
>     - locked <0x25e331e0> (a java.util.Collections$UnmodifiableSet)
>     - locked <0x25e33188> (a sun.nio.ch.KQueueSelectorImpl)
>     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     at
> com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:344)
>     at com.sun.grizzly.Controller.doSelect(Controller.java:280)
>     at com.sun.grizzly.Controller.start(Controller.java:788)
>     - locked <0x24e18320> (a java.lang.Object)
>     at
> com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1080)
>     - locked <0x24de28b8> (a [Ljava.lang.Object;)
>     at
> com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1052)
>     at
> dk.vitality.util.socketProxy.server.grizzly.GriProxy.main(GriProxy.java:96)
>     at
> dk.vitality.util.socketProxy.server.grizzly.GriProxy$1.run(GriProxy.java:61)
>
>     at java.lang.Thread.run(Thread.java:613)
>
>
> The adapter is:
>
> class ProxyAdapter implements Adapter
> {
>    public void service(Request request, Response response) throws Exception
>    {
>       request.action(ActionCode.ACTION_REQ_HOST_ADDR_ATTRIBUTE, null);
>       new RequestHandler(GriProxy.this, request, response);
>    }
>
>    public void afterService(Request request, Response response) throws
> Exception
>    {
>       request.recycle();
>       response.recycle();
>    }
>
>    public void fireAdapterEvent(String string, Object object)
>    {
>    }
> }
>
>
> ----
>    - Peter Speck
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: http keep-alive

Peter Speck
In reply to this post by Oleksiy Stashok
On 03/09/2007, at 17:23, Oleksiy Stashok wrote:

> please try latest Grizzly 1.6.0 sources, I've fixed one possible  
> problem, hope it will help.


I have now tried the fix when the server was under load, but the  
problem remains.

After a few minutes, I have lots of threads waiting in  
sun.nio.ch.KQueueArrayWrapper.kevent0, the server feels very sluggish  
and then runs out of memory (java.lang.OutOfMemoryError).   There is  
approx. 170 keep-alive connections at this time of the day.

When I test the server using Apache Bench, the problem doesn't occur.

----
    - Peter Speck


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: http keep-alive

Jeanfrancois Arcand-2
Hi Peter,

sorry forgot to reply. I've fixed it on the trunk and ported the fix to
the 1.6.0 branch as well. Can you try it and let us know?

Thanks

-- Jeanfrancois

Peter Speck wrote:

> On 03/09/2007, at 17:23, Oleksiy Stashok wrote:
>
>> please try latest Grizzly 1.6.0 sources, I've fixed one possible
>> problem, hope it will help.
>
>
> I have now tried the fix when the server was under load, but the problem
> remains.
>
> After a few minutes, I have lots of threads waiting in
> sun.nio.ch.KQueueArrayWrapper.kevent0, the server feels very sluggish
> and then runs out of memory (java.lang.OutOfMemoryError).   There is
> approx. 170 keep-alive connections at this time of the day.
>
> When I test the server using Apache Bench, the problem doesn't occur.
>
> ----
>    - Peter Speck
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: http keep-alive

Peter Speck
On 05/09/2007, at 16:58, Jeanfrancois Arcand wrote:

> sorry forgot to reply. I've fixed it on the trunk and ported the  
> fix to the 1.6.0 branch as well. Can you try it and let us know?

I've checked out tip-of-trunk and tested it, but I still get tons of  
threads waiting in sun.nio.ch.KQueueArrayWrapper.kevent0
Is there any kind of debug log I could enable to pinpoint it?

----
    - Peter Speck


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: http keep-alive

Jeanfrancois Arcand-2


Peter Speck wrote:
> On 05/09/2007, at 16:58, Jeanfrancois Arcand wrote:
>
>> sorry forgot to reply. I've fixed it on the trunk and ported the fix
>> to the 1.6.0 branch as well. Can you try it and let us know?
>
> I've checked out tip-of-trunk and tested it, but I still get tons of
> threads waiting in sun.nio.ch.KQueueArrayWrapper.kevent0
> Is there any kind of debug log I could enable to pinpoint it?

This is the same stack trace as before, right? DO you have an idea of
what the http request looks like? I've tested with telnet and browser
and wasn't able to reproduce the problem after my changes.

Thanks

-- Jeanfrancois

>
> ----
>    - Peter Speck
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: http keep-alive

Peter Speck

On 06/09/2007, at 19:00, Jeanfrancois Arcand wrote:

> This is the same stack trace as before, right?

yes.

> DO you have an idea of what the http request looks like?

nope. It is a busy server with tons of different kinds of requests.  
I've tried to hit it with simple requests using Apache Bench, but it  
did not fail in that test.

----
    - Peter Speck


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]