Grizzly slowly leaking with comet support on version 1.0.39

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

Grizzly slowly leaking with comet support on version 1.0.39

wild
Hello,

I am observing a memory leak in grizzly comet support (1.0.39). After a few days of use, I have seen the memory growing slowly by looking gc logs.... Multiple full gc did not clear the memory. Memory groes of 200Mo a day avg.

I have performed an heam dump with jmap (3go) and analyze it with a tool. It seems that the collection named 'bannedKeys" contains in SelectorThread is always growing. I think I can provide you a report bit not the entire heap dump (too large I think).

Did anyone observe the same behavior ?

Thanks.

William.
Reply | Threaded
Open this post in threaded view
|

Re: Grizzly slowly leaking with comet support on version 1.0.39

oleksiys
Administrator
Hi William,

> I am observing a memory leak in grizzly comet support (1.0.39). After a few
> days of use, I have seen the memory growing slowly by looking gc logs....
> Multiple full gc did not clear the memory. Memory groes of 200Mo a day avg.
>
> I have performed an heam dump with jmap (3go) and analyze it with a tool. It
> seems that the collection named 'bannedKeys" contains in SelectorThread is
> always growing. I think I can provide you a report bit not the entire heap
> dump (too large I think).
>
> Did anyone observe the same behavior ?
Can you pls. file an issue on issuetracker? [1]

Also wanted to ask for more details...
Are you using HTTP pipelining together with comet? Are you reading off
all the input before registering a comet handler?

We can definitely fix the memory leak and start using WeakHashMap for
the bannedKeys collection, but just wanted to make sure we're not
missing anything else.

Thanks.

WBR,
Alexey.

[1] http://java.net/jira/browse/GRIZZLY
Reply | Threaded
Open this post in threaded view
|

Re: Grizzly slowly leaking with comet support on version 1.0.39

wild
I'm not sure I'm using an http pipeline. how can I make sure of that ?

The sequence of comet registration  (code in comet servlet) :

1. create handler and attach the http response
2. send a ack in http response outuput stream (write + flush)
3. add the comet handler in comet context by calling addCometHandler
4. response.flushBuffer(); => not sure why this code is here (legacy), I can try to remove it

Comet registration is made by a post or a get, so I think all the data have been read.... How can I ensure that all the data have been read in a servlet code ?

Tell me I am doing something wrong. Othrewise I will fill an issue in your tracker.

Regards,

William.
Reply | Threaded
Open this post in threaded view
|

Re: Grizzly slowly leaking with comet support on version 1.0.39

oleksiys
Administrator

> I'm not sure I'm using an http pipeline. how can I make sure of that?
It would be our next step after we make sure we've read entire request
body (if it's acceptable by your usecase).

> The sequence of comet registration  (code in comet servlet) :
>
> 1. create handler and attach the http response
> 2. send a ack in http response outuput stream (write + flush)
> 3. add the comet handler in comet context by calling addCometHandler
> 4. response.flushBuffer(); =>  not sure why this code is here (legacy), I can
> try to remove it
>
Hmm, let's remove flushBuffer, or at least swap 3. and 4.

> Comet registration is made by a post or a get, so I think all the data have
> been read.... How can I ensure that all the data have been read in a servlet
> code ?
IMO InputStream.read(...) has to return -1.

> Tell me I am doing something wrong. Othrewise I will fill an issue in your
> tracker.
Let's try to make sure we've read entire request body before doing the
addCometHandler(...), again if it's acceptable for you.

Thanks.

WBR,
Alexey.

Reply | Threaded
Open this post in threaded view
|

Re: Grizzly slowly leaking with comet support on version 1.0.39

wild
ok, I'll try sawp 3 and 4 and remove response.flushBuffer()

I'll ge back to you with the results of the test.

Thanks,

William.
Reply | Threaded
Open this post in threaded view
|

Re: Grizzly slowly leaking with comet support on version 1.0.39

wild
Swaping the 2 statements did not change anything.

So what should I do in order to make sure the body has been fully read ?

In fact, the service method of the servlet is overridden and the sequence he describe in the previous post is put in the service method of HttpServlet. Should I add the folowing code ?

byte[] buffer = new byte[1024];
while(request.getInputStream().read(buffer) != -1) {}

I still don't understand why I should read the body because nothing is supposed to be in it.

Regards,

William.


Reply | Threaded
Open this post in threaded view
|

Re: Grizzly slowly leaking with comet support on version 1.0.39

oleksiys
Administrator

> So what should I do in order to make sure the body has been fully read ?
>
> In fact, the service method of the servlet is overridden and the sequence he
> describe in the previous post is put in the service method of HttpServlet.
> Should I add the folowing code ?
>
> byte[] buffer = new byte[1024];
> while(request.getInputStream().read(buffer) != -1) {}
>
> I still don't understand why I should read the body because nothing is
> supposed to be in it.
If there is nothing to read the code above shouldn't have any effect.
But you mentioned that comet handler is being registered for HTTP posts
too, so let's just check that.

What are you using on client side? Is it a web browser?

Thanks.

Alexey.

Reply | Threaded
Open this post in threaded view
|

Re: Grizzly slowly leaking with comet support on version 1.0.39

wild
It can be a post yes, I can try that.

No it is not a web browser, it just open an http connection in java and keep it open in order to receive messages in a streaming mode.
Reply | Threaded
Open this post in threaded view
|

Re: Grizzly slowly leaking with comet support on version 1.0.39

wild
I've just made an heap dump and the problem is not the same as the previous one.

In my actuel heap dump, there is a huge amount of DefaultProcessorTask in a list named processorTaks in SelectorThread.

It is weird because the SelectorThread of the Administration Console is holding all the DefaultProcessorTask.

Are they share between all the SelectorThread even if they are not listening on the same port ?

If I use comet, did the task will return to the pool after calling addCometHandler ?

Regards,

William.
Reply | Threaded
Open this post in threaded view
|

Re: Grizzly slowly leaking with comet support on version 1.0.39

oleksiys
Administrator
On 04/12/2011 03:38 PM, wild wrote:

> I've just made an heap dump and the problem is not the same as the previous
> one.
>
> In my actuel heap dump, there is a huge amount of DefaultProcessorTask in a
> list named processorTaks in SelectorThread.
>
> It is weird because the SelectorThread of the Administration Console is
> holding all the DefaultProcessorTask.
>
> Are they share between all the SelectorThread even if they are not listening
> on the same port ?
No.

> If I use comet, did the task will return to the pool after calling
> addCometHandler ?
Yes, it should, but to the appropriate listener's pool, not to admin
console's one.
Can you pls. create a simple testcase for it? I'll check locally.

Thanks.

WBR,
Alexey.

> Regards,
>
> William.
>
> --
> View this message in context: http://grizzly.1045725.n5.nabble.com/Grizzly-slowly-leaking-with-comet-support-on-version-1-0-39-tp4295847p4298204.html
> Sent from the Grizzly - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Grizzly slowly leaking with comet support on version 1.0.39

wild
I think the heap analysis tool misslead me....

The admin console's selector thread contains by default a cache of processor task that has not been used, so its collection of processor task is not empty and appear to be the bigest objet of the heap. Each DefaultProcessorTask reference a the same selector thread

By using OQL query, I find out that only the comet's selector thread has an empty collection of processorTask but a collection of readTask that contains 6000 elements.

Processor Task are kept in an object called DefaultAsyncHandler. For some of this processors it seems that the buffer has not been flushed and DefaultProcessorTask keep a reference to the buffer that is not empty which keep the memory very high => buffer has not been flushed, the content remains in memory.

It is very difficult to give you a test case the code is complex and it happens on high load. We had a lot of error at the end of our test and when we have too many exception during write operation in the socket.

If an error occur during a write, then we close the connection by calling  cometContext().resumeCometHandler(this), mybe it is the cause of the problem.

Are we allowed to call resumeCometHandler method ?
How is it possible that buffer has not been flushed ?
Will the buffer be flush when the task will be reused ?

Thanks

William
Reply | Threaded
Open this post in threaded view
|

Re: Grizzly slowly leaking with comet support on version 1.0.39

wild
Hi,

Please can you tell me if it is a bug or just a cache that grows and will be reuse ?

It is very strange to have so many data in memory, event output buffer are kept in memory and seems not be cleared.

I will do a test with glassfish V3.1. Do you think it will have the same behavior ?

Thanks,

William.
Reply | Threaded
Open this post in threaded view
|

Re: Grizzly slowly leaking with comet support on version 1.0.39

oleksiys
Administrator
Hi William,

> Please can you tell me if it is a bug or just a cache that grows and will be
> reuse ?
it's a object cache, but it shouldn't grow infinitely.

> It is very strange to have so many data in memory, event output buffer are
> kept in memory and seems not be cleared.
Can you pls. make a screenshot of the memory dump so we understand which
buffer ref. you mean?
As I told it might be great to have some testcase. Important part is
sequence you call addHandler, remove/resumeHandler, try/catch/finally etc...

> I will do a test with glassfish V3.1. Do you think it will have the same
> behavior ?
No, comet impl. is different there.

Thanks.

WBR,
Alexey.

Reply | Threaded
Open this post in threaded view
|

Re: Grizzly slowly leaking with comet support on version 1.0.39

wild
Hello,

I made a implementation of comet with all the requirements and memory usage is still very high. I am using glassfish 3.1 last edition (so grizzly 1.9.x).

A list of processor tasks is growing in SelectorThread class. A jmap confirm that and full GC does not clean memory.

When I change an option on the tcp protocol used for comet, then the memory goes down (probably because all the threads are destroyed and the memory is cleaned up).

So the problem is located on the comet listener which create too much Processor Tasks (I think). When I start the server, the memory used is around 15Mo, then after a load test of 5800 users with comet streaming the memory after the test and a full GC is 750Mo and then after another load test with 5800 the memory is 1300Mo.

Even if the processor tasks is not supposed to grow, is there a way to delete this cache or clear it ? Is it sure that processor tasks are recycled with comet support ?

Thanks.

William.
Reply | Threaded
Open this post in threaded view
|

Re: Grizzly slowly leaking with comet support on version 1.0.39

oleksiys
Administrator
Hi William,

can i ask you to file an issue?
Will take a look at this asap.
Just want to make sure, which exactly Glassfish release are you testing
on, is it 3.1 release?

Thanks.

WBR,
Alexey.

On 06/29/2011 11:07 AM, wild wrote:

> Hello,
>
> I made a implementation of comet with all the requirements and memory usage
> is still very high. I am using glassfish 3.1 last edition (so grizzly
> 1.9.x).
>
> A list of processor tasks is growing in SelectorThread class. A jmap confirm
> that and full GC does not clean memory.
>
> When I change an option on the tcp protocol used for comet, then the memory
> goes down (probably because all the threads are destroyed and the memory is
> cleaned up).
>
> So the problem is located on the comet listener which create too much
> Processor Tasks (I think). When I start the server, the memory used is
> around 15Mo, then after a load test of 5800 users with comet streaming the
> memory after the test and a full GC is 750Mo and then after another load
> test with 5800 the memory is 1300Mo.
>
> Even if the processor tasks is not supposed to grow, is there a way to
> delete this cache or clear it ? Is it sure that processor tasks are recycled
> with comet support ?
>
> Thanks.
>
> William.
>
> --
> View this message in context: http://grizzly.1045725.n5.nabble.com/Grizzly-slowly-leaking-with-comet-support-on-version-1-0-39-tp4295847p4534526.html
> Sent from the Grizzly - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Grizzly slowly leaking with comet support on version 1.0.39

wild
I am using GlassFish Server Open Source Edition 3.1 (build 43).

I have filled a JIRA : GRIZZLY-1034

Do you want the memory analysis report ? I can export some pdf or printscrean. I am using MAT but I can't export a report.

Thanks.
Reply | Threaded
Open this post in threaded view
|

Re: Grizzly slowly leaking with comet support on version 1.0.39

Ryan Lubke-2
On 6/30/11 9:00 AM, wild wrote:
> I am using GlassFish Server Open Source Edition 3.1 (build 43).
>
> I have filled a JIRA : GRIZZLY-1034
>
> Do you want the memory analysis report ? I can export some pdf or
> printscrean. I am using MAT but I can't export a report.

Screenshots of the relevant data would be good.

Thanks!

> Thanks.
>
> --
> View this message in context: http://grizzly.1045725.n5.nabble.com/Grizzly-slowly-leaking-with-comet-support-on-version-1-0-39-tp4295847p4539333.html
> Sent from the Grizzly - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Grizzly slowly leaking with comet support on version 1.0.39

wild
Here it is :

Memory usage of EmbeddedHttp :

Meory usage of a ProcessorTask :

I will add it to the JIRA.
Reply | Threaded
Open this post in threaded view
|

Re: Grizzly slowly leaking with comet support on version 1.0.39

wild
Is it possible  for you to provide a patch for glassfish 3.1 ?

I think changing a the processorTasks cache to a bounded LinkedBlockingQueue will be enough. The fact that processorTasks list is filled with a lot of object means (for me) that all the processor tasks has been released.

Because I use comet, I will have one processor task active per user. So it grows a lot when I have a lot of user. What I don,t understand is why the size of queue increase betzeen stress tests. Is it possible that under high load, the poll method did not return an element event if there are element in it ?

For me, is the size of the processsor task list is high, I means that all the connection has been terminated correctly (tell me if I am wrong). Is there any way to see in jmx the size of this cache ?

Thanks

William.
Reply | Threaded
Open this post in threaded view
|

Re: Grizzly slowly leaking with comet support on version 1.0.39

oleksiys
Administrator
Hi William,

I'll try to reproduce and fix it asap.

Thanks.

WBR,
Alexey.

On 07/02/2011 07:34 PM, wild wrote:

> Is it possible  for you to provide a patch for glassfish 3.1 ?
>
> I think changing a the processorTasks cache to a bounded LinkedBlockingQueue
> will be enough. The fact that processorTasks list is filled with a lot of
> object means (for me) that all the processor tasks has been released.
>
> Because I use comet, I will have one processor task active per user. So it
> grows a lot when I have a lot of user. What I don,t understand is why the
> size of queue increase betzeen stress tests. Is it possible that under high
> load, the poll method did not return an element event if there are element
> in it ?
>
> For me, is the size of the processsor task list is high, I means that all
> the connection has been terminated correctly (tell me if I am wrong). Is
> there any way to see in jmx the size of this cache ?
>
> Thanks
>
> William.
>
>
> --
> View this message in context: http://grizzly.1045725.n5.nabble.com/Grizzly-slowly-leaking-with-comet-support-on-version-1-0-39-tp4295847p4545326.html
> Sent from the Grizzly - Users mailing list archive at Nabble.com.