NPE's at low concurrency using AsyncHttpClient

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

NPE's at low concurrency using AsyncHttpClient

Daniel Feist
Hi Alexey,

We're geting a weird issue when using AHC/Grizzly for outbound HTTP
POST operations.  When testing using POST and giving AHC an
InputStream to send, we get a significant number of NPE's occuring (1
to 5% of requests), even at just 10 client threads.  I havn't been
able to reproduce this locally yet, but it happens consistently on our
test enviroment.

I'm not sure how to get to the bottom of this (am in the middle of
trying) but given your much deeper knowledge of the codebase maybe you
have some insights/ideas. With a single client thread there are no
issues so it look like it may be some kind of race condition.  Also
passing a String to AHC or turn off chunking such that
com.ning.http.client.generators.InputStreamBodyGenerator doesn't get
used, there are no NPE's either.

Dan

-----

This is the error seen:

WARN  2015-02-13 09:06:38,336
[[http_module_proxy].proxyConfig.worker.10]
org.glassfish.grizzly.filterchain.DefaultFilterChain: GRIZZLY0013:
Exception during FilterChain execution
java.lang.NullPointerException
    at com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientEventFilter.onHttpContentEncoded(GrizzlyAsyncHttpProvider.java:1243)
~[?:?]
    at org.glassfish.grizzly.http.HttpCodecFilter.encodeHttpPacket(HttpCodecFilter.java:1463)
~[?:?]
    at org.glassfish.grizzly.http.HttpClientFilter.encodeHttpPacket(HttpClientFilter.java:283)
~[?:?]
Reply | Threaded
Open this post in threaded view
|

Re: NPE's at low concurrency using AsyncHttpClient

oleksiys
Administrator
Hi Dan,

which AHC version are you using?

WBR,
Alexey.

On 13.02.15 07:35, Daniel Feist wrote:

> Hi Alexey,
>
> We're geting a weird issue when using AHC/Grizzly for outbound HTTP
> POST operations.  When testing using POST and giving AHC an
> InputStream to send, we get a significant number of NPE's occuring (1
> to 5% of requests), even at just 10 client threads.  I havn't been
> able to reproduce this locally yet, but it happens consistently on our
> test enviroment.
>
> I'm not sure how to get to the bottom of this (am in the middle of
> trying) but given your much deeper knowledge of the codebase maybe you
> have some insights/ideas. With a single client thread there are no
> issues so it look like it may be some kind of race condition.  Also
> passing a String to AHC or turn off chunking such that
> com.ning.http.client.generators.InputStreamBodyGenerator doesn't get
> used, there are no NPE's either.
>
> Dan
>
> -----
>
> This is the error seen:
>
> WARN  2015-02-13 09:06:38,336
> [[http_module_proxy].proxyConfig.worker.10]
> org.glassfish.grizzly.filterchain.DefaultFilterChain: GRIZZLY0013:
> Exception during FilterChain execution
> java.lang.NullPointerException
>      at com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientEventFilter.onHttpContentEncoded(GrizzlyAsyncHttpProvider.java:1243)
> ~[?:?]
>      at org.glassfish.grizzly.http.HttpCodecFilter.encodeHttpPacket(HttpCodecFilter.java:1463)
> ~[?:?]
>      at org.glassfish.grizzly.http.HttpClientFilter.encodeHttpPacket(HttpClientFilter.java:283)
> ~[?:?]

Reply | Threaded
Open this post in threaded view
|

Re: NPE's at low concurrency using AsyncHttpClient

Daniel Feist
Hi,

1.8.x. We were originally using 1.8.14, but testing with 1.8.15 (which
has some changes to InputStreamBodyGenerator) made no difference.

Dan

On Fri, Feb 13, 2015 at 5:37 PM, Oleksiy Stashok
<[hidden email]> wrote:

> Hi Dan,
>
> which AHC version are you using?
>
> WBR,
> Alexey.
>
>
> On 13.02.15 07:35, Daniel Feist wrote:
>>
>> Hi Alexey,
>>
>> We're geting a weird issue when using AHC/Grizzly for outbound HTTP
>> POST operations.  When testing using POST and giving AHC an
>> InputStream to send, we get a significant number of NPE's occuring (1
>> to 5% of requests), even at just 10 client threads.  I havn't been
>> able to reproduce this locally yet, but it happens consistently on our
>> test enviroment.
>>
>> I'm not sure how to get to the bottom of this (am in the middle of
>> trying) but given your much deeper knowledge of the codebase maybe you
>> have some insights/ideas. With a single client thread there are no
>> issues so it look like it may be some kind of race condition.  Also
>> passing a String to AHC or turn off chunking such that
>> com.ning.http.client.generators.InputStreamBodyGenerator doesn't get
>> used, there are no NPE's either.
>>
>> Dan
>>
>> -----
>>
>> This is the error seen:
>>
>> WARN  2015-02-13 09:06:38,336
>> [[http_module_proxy].proxyConfig.worker.10]
>> org.glassfish.grizzly.filterchain.DefaultFilterChain: GRIZZLY0013:
>> Exception during FilterChain execution
>> java.lang.NullPointerException
>>      at
>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientEventFilter.onHttpContentEncoded(GrizzlyAsyncHttpProvider.java:1243)
>> ~[?:?]
>>      at
>> org.glassfish.grizzly.http.HttpCodecFilter.encodeHttpPacket(HttpCodecFilter.java:1463)
>> ~[?:?]
>>      at
>> org.glassfish.grizzly.http.HttpClientFilter.encodeHttpPacket(HttpClientFilter.java:283)
>> ~[?:?]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: NPE's at low concurrency using AsyncHttpClient

Daniel Feist
Hi,

After investigating further this is what I found.

It occured to me that the problem may not be with AHC/Grizzly but
rather provoked by the back-end service I was hitting.  So I switched
out the back-end implementation to see what happened:

Tomcat 6 hitting static resource (file in webapps/ROOT)
--------------------------------------------------------------------------
This is what causes the NPE's.  I realise it's probably a bad idea to
be testing with a POST request against a static resource, there was an
error in the test url we were using, but even so I wouldn't expect
NPE's client side.  The http body size is irrelevant and this only
occurs when InputStreamBodyGenerator is used. As i mentioned before
failures rate is 2-5% of requests.

Tomcat/Grizzly/Netty Echo Service
-----------------------------------------------
Using a echo service that reads in request stream from post and then
echo's response the NPE's go away but there is still always a singe
failed request.

Exception stack is:
1. Locally closed (java.io.IOException)
2. java.io.IOException: Locally closed (java.util.concurrent.ExecutionException)
 org.glassfish.grizzly.impl.SafeFutureImpl$Sync:363 (null)
3. java.util.concurrent.ExecutionException: java.io.IOException:
Locally closed (java.io.IOException)
 org.mule.module.http.internal.request.grizzly.GrizzlyHttpClient:274 (null)

In what cases does "locally closed" IOException occur in grizzly?  It
seems strange that it's consistently seen exactly once even when we
test with millions of request.

Note: When we tested with a Jetty echo implementation we saw no errors
at all, but with tomcat, grizzly, netty exactly one, weird!

I'll keep looking into this, but any pointers would be great!

thanks,

Dan


On Fri, Feb 13, 2015 at 6:15 PM, Daniel Feist <[hidden email]> wrote:

> Hi,
>
> 1.8.x. We were originally using 1.8.14, but testing with 1.8.15 (which
> has some changes to InputStreamBodyGenerator) made no difference.
>
> Dan
>
> On Fri, Feb 13, 2015 at 5:37 PM, Oleksiy Stashok
> <[hidden email]> wrote:
>> Hi Dan,
>>
>> which AHC version are you using?
>>
>> WBR,
>> Alexey.
>>
>>
>> On 13.02.15 07:35, Daniel Feist wrote:
>>>
>>> Hi Alexey,
>>>
>>> We're geting a weird issue when using AHC/Grizzly for outbound HTTP
>>> POST operations.  When testing using POST and giving AHC an
>>> InputStream to send, we get a significant number of NPE's occuring (1
>>> to 5% of requests), even at just 10 client threads.  I havn't been
>>> able to reproduce this locally yet, but it happens consistently on our
>>> test enviroment.
>>>
>>> I'm not sure how to get to the bottom of this (am in the middle of
>>> trying) but given your much deeper knowledge of the codebase maybe you
>>> have some insights/ideas. With a single client thread there are no
>>> issues so it look like it may be some kind of race condition.  Also
>>> passing a String to AHC or turn off chunking such that
>>> com.ning.http.client.generators.InputStreamBodyGenerator doesn't get
>>> used, there are no NPE's either.
>>>
>>> Dan
>>>
>>> -----
>>>
>>> This is the error seen:
>>>
>>> WARN  2015-02-13 09:06:38,336
>>> [[http_module_proxy].proxyConfig.worker.10]
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain: GRIZZLY0013:
>>> Exception during FilterChain execution
>>> java.lang.NullPointerException
>>>      at
>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientEventFilter.onHttpContentEncoded(GrizzlyAsyncHttpProvider.java:1243)
>>> ~[?:?]
>>>      at
>>> org.glassfish.grizzly.http.HttpCodecFilter.encodeHttpPacket(HttpCodecFilter.java:1463)
>>> ~[?:?]
>>>      at
>>> org.glassfish.grizzly.http.HttpClientFilter.encodeHttpPacket(HttpClientFilter.java:283)
>>> ~[?:?]
>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: NPE's at low concurrency using AsyncHttpClient

oleksiys
Administrator
Hi Dan,

you say it happens at low concurrency, so it shouldn't be difficult to
reproduce the failure.
Can you pls. share the testcase, it would be much easier for me to
figure out what's wrong.

Thank you.

WBR,
Alexey.

On 16.02.15 03:45, Daniel Feist wrote:

> Hi,
>
> After investigating further this is what I found.
>
> It occured to me that the problem may not be with AHC/Grizzly but
> rather provoked by the back-end service I was hitting.  So I switched
> out the back-end implementation to see what happened:
>
> Tomcat 6 hitting static resource (file in webapps/ROOT)
> --------------------------------------------------------------------------
> This is what causes the NPE's.  I realise it's probably a bad idea to
> be testing with a POST request against a static resource, there was an
> error in the test url we were using, but even so I wouldn't expect
> NPE's client side.  The http body size is irrelevant and this only
> occurs when InputStreamBodyGenerator is used. As i mentioned before
> failures rate is 2-5% of requests.
>
> Tomcat/Grizzly/Netty Echo Service
> -----------------------------------------------
> Using a echo service that reads in request stream from post and then
> echo's response the NPE's go away but there is still always a singe
> failed request.
>
> Exception stack is:
> 1. Locally closed (java.io.IOException)
> 2. java.io.IOException: Locally closed (java.util.concurrent.ExecutionException)
>   org.glassfish.grizzly.impl.SafeFutureImpl$Sync:363 (null)
> 3. java.util.concurrent.ExecutionException: java.io.IOException:
> Locally closed (java.io.IOException)
>   org.mule.module.http.internal.request.grizzly.GrizzlyHttpClient:274 (null)
>
> In what cases does "locally closed" IOException occur in grizzly?  It
> seems strange that it's consistently seen exactly once even when we
> test with millions of request.
>
> Note: When we tested with a Jetty echo implementation we saw no errors
> at all, but with tomcat, grizzly, netty exactly one, weird!
>
> I'll keep looking into this, but any pointers would be great!
>
> thanks,
>
> Dan
>
>
> On Fri, Feb 13, 2015 at 6:15 PM, Daniel Feist <[hidden email]> wrote:
>> Hi,
>>
>> 1.8.x. We were originally using 1.8.14, but testing with 1.8.15 (which
>> has some changes to InputStreamBodyGenerator) made no difference.
>>
>> Dan
>>
>> On Fri, Feb 13, 2015 at 5:37 PM, Oleksiy Stashok
>> <[hidden email]> wrote:
>>> Hi Dan,
>>>
>>> which AHC version are you using?
>>>
>>> WBR,
>>> Alexey.
>>>
>>>
>>> On 13.02.15 07:35, Daniel Feist wrote:
>>>> Hi Alexey,
>>>>
>>>> We're geting a weird issue when using AHC/Grizzly for outbound HTTP
>>>> POST operations.  When testing using POST and giving AHC an
>>>> InputStream to send, we get a significant number of NPE's occuring (1
>>>> to 5% of requests), even at just 10 client threads.  I havn't been
>>>> able to reproduce this locally yet, but it happens consistently on our
>>>> test enviroment.
>>>>
>>>> I'm not sure how to get to the bottom of this (am in the middle of
>>>> trying) but given your much deeper knowledge of the codebase maybe you
>>>> have some insights/ideas. With a single client thread there are no
>>>> issues so it look like it may be some kind of race condition.  Also
>>>> passing a String to AHC or turn off chunking such that
>>>> com.ning.http.client.generators.InputStreamBodyGenerator doesn't get
>>>> used, there are no NPE's either.
>>>>
>>>> Dan
>>>>
>>>> -----
>>>>
>>>> This is the error seen:
>>>>
>>>> WARN  2015-02-13 09:06:38,336
>>>> [[http_module_proxy].proxyConfig.worker.10]
>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain: GRIZZLY0013:
>>>> Exception during FilterChain execution
>>>> java.lang.NullPointerException
>>>>       at
>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientEventFilter.onHttpContentEncoded(GrizzlyAsyncHttpProvider.java:1243)
>>>> ~[?:?]
>>>>       at
>>>> org.glassfish.grizzly.http.HttpCodecFilter.encodeHttpPacket(HttpCodecFilter.java:1463)
>>>> ~[?:?]
>>>>       at
>>>> org.glassfish.grizzly.http.HttpClientFilter.encodeHttpPacket(HttpClientFilter.java:283)
>>>> ~[?:?]
>>>

Reply | Threaded
Open this post in threaded view
|

Re: NPE's at low concurrency using AsyncHttpClient

Daniel Feist
I wasn't able to reproduce this in a test case yet, even though I tried.

It appears this issues is caused by the combination of:

i) Server no reading full http body
ii) Potentially also timing

Given it only happens when posting to a tomcat (also jetty) static resource.

Dan

On Tue, Feb 17, 2015 at 4:50 AM, Oleksiy Stashok
<[hidden email]> wrote:

> Hi Dan,
>
> you say it happens at low concurrency, so it shouldn't be difficult to
> reproduce the failure.
> Can you pls. share the testcase, it would be much easier for me to figure
> out what's wrong.
>
> Thank you.
>
> WBR,
> Alexey.
>
>
> On 16.02.15 03:45, Daniel Feist wrote:
>>
>> Hi,
>>
>> After investigating further this is what I found.
>>
>> It occured to me that the problem may not be with AHC/Grizzly but
>> rather provoked by the back-end service I was hitting.  So I switched
>> out the back-end implementation to see what happened:
>>
>> Tomcat 6 hitting static resource (file in webapps/ROOT)
>> --------------------------------------------------------------------------
>> This is what causes the NPE's.  I realise it's probably a bad idea to
>> be testing with a POST request against a static resource, there was an
>> error in the test url we were using, but even so I wouldn't expect
>> NPE's client side.  The http body size is irrelevant and this only
>> occurs when InputStreamBodyGenerator is used. As i mentioned before
>> failures rate is 2-5% of requests.
>>
>> Tomcat/Grizzly/Netty Echo Service
>> -----------------------------------------------
>> Using a echo service that reads in request stream from post and then
>> echo's response the NPE's go away but there is still always a singe
>> failed request.
>>
>> Exception stack is:
>> 1. Locally closed (java.io.IOException)
>> 2. java.io.IOException: Locally closed
>> (java.util.concurrent.ExecutionException)
>>   org.glassfish.grizzly.impl.SafeFutureImpl$Sync:363 (null)
>> 3. java.util.concurrent.ExecutionException: java.io.IOException:
>> Locally closed (java.io.IOException)
>>   org.mule.module.http.internal.request.grizzly.GrizzlyHttpClient:274
>> (null)
>>
>> In what cases does "locally closed" IOException occur in grizzly?  It
>> seems strange that it's consistently seen exactly once even when we
>> test with millions of request.
>>
>> Note: When we tested with a Jetty echo implementation we saw no errors
>> at all, but with tomcat, grizzly, netty exactly one, weird!
>>
>> I'll keep looking into this, but any pointers would be great!
>>
>> thanks,
>>
>> Dan
>>
>>
>> On Fri, Feb 13, 2015 at 6:15 PM, Daniel Feist <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> 1.8.x. We were originally using 1.8.14, but testing with 1.8.15 (which
>>> has some changes to InputStreamBodyGenerator) made no difference.
>>>
>>> Dan
>>>
>>> On Fri, Feb 13, 2015 at 5:37 PM, Oleksiy Stashok
>>> <[hidden email]> wrote:
>>>>
>>>> Hi Dan,
>>>>
>>>> which AHC version are you using?
>>>>
>>>> WBR,
>>>> Alexey.
>>>>
>>>>
>>>> On 13.02.15 07:35, Daniel Feist wrote:
>>>>>
>>>>> Hi Alexey,
>>>>>
>>>>> We're geting a weird issue when using AHC/Grizzly for outbound HTTP
>>>>> POST operations.  When testing using POST and giving AHC an
>>>>> InputStream to send, we get a significant number of NPE's occuring (1
>>>>> to 5% of requests), even at just 10 client threads.  I havn't been
>>>>> able to reproduce this locally yet, but it happens consistently on our
>>>>> test enviroment.
>>>>>
>>>>> I'm not sure how to get to the bottom of this (am in the middle of
>>>>> trying) but given your much deeper knowledge of the codebase maybe you
>>>>> have some insights/ideas. With a single client thread there are no
>>>>> issues so it look like it may be some kind of race condition.  Also
>>>>> passing a String to AHC or turn off chunking such that
>>>>> com.ning.http.client.generators.InputStreamBodyGenerator doesn't get
>>>>> used, there are no NPE's either.
>>>>>
>>>>> Dan
>>>>>
>>>>> -----
>>>>>
>>>>> This is the error seen:
>>>>>
>>>>> WARN  2015-02-13 09:06:38,336
>>>>> [[http_module_proxy].proxyConfig.worker.10]
>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain: GRIZZLY0013:
>>>>> Exception during FilterChain execution
>>>>> java.lang.NullPointerException
>>>>>       at
>>>>>
>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientEventFilter.onHttpContentEncoded(GrizzlyAsyncHttpProvider.java:1243)
>>>>> ~[?:?]
>>>>>       at
>>>>>
>>>>> org.glassfish.grizzly.http.HttpCodecFilter.encodeHttpPacket(HttpCodecFilter.java:1463)
>>>>> ~[?:?]
>>>>>       at
>>>>>
>>>>> org.glassfish.grizzly.http.HttpClientFilter.encodeHttpPacket(HttpClientFilter.java:283)
>>>>> ~[?:?]
>>>>
>>>>
>
Reply | Threaded
Open this post in threaded view
|

Re: NPE's at low concurrency using AsyncHttpClient

oleksiys
Administrator
What happens if you increase concurrency and start to send POSTs at high
rate? Does it help to reproduce the problem?

WBR,
Alexey.

On 23.02.15 05:06, Daniel Feist wrote:

> I wasn't able to reproduce this in a test case yet, even though I tried.
>
> It appears this issues is caused by the combination of:
>
> i) Server no reading full http body
> ii) Potentially also timing
>
> Given it only happens when posting to a tomcat (also jetty) static resource.
>
> Dan
>
> On Tue, Feb 17, 2015 at 4:50 AM, Oleksiy Stashok
> <[hidden email]> wrote:
>> Hi Dan,
>>
>> you say it happens at low concurrency, so it shouldn't be difficult to
>> reproduce the failure.
>> Can you pls. share the testcase, it would be much easier for me to figure
>> out what's wrong.
>>
>> Thank you.
>>
>> WBR,
>> Alexey.
>>
>>
>> On 16.02.15 03:45, Daniel Feist wrote:
>>> Hi,
>>>
>>> After investigating further this is what I found.
>>>
>>> It occured to me that the problem may not be with AHC/Grizzly but
>>> rather provoked by the back-end service I was hitting.  So I switched
>>> out the back-end implementation to see what happened:
>>>
>>> Tomcat 6 hitting static resource (file in webapps/ROOT)
>>> --------------------------------------------------------------------------
>>> This is what causes the NPE's.  I realise it's probably a bad idea to
>>> be testing with a POST request against a static resource, there was an
>>> error in the test url we were using, but even so I wouldn't expect
>>> NPE's client side.  The http body size is irrelevant and this only
>>> occurs when InputStreamBodyGenerator is used. As i mentioned before
>>> failures rate is 2-5% of requests.
>>>
>>> Tomcat/Grizzly/Netty Echo Service
>>> -----------------------------------------------
>>> Using a echo service that reads in request stream from post and then
>>> echo's response the NPE's go away but there is still always a singe
>>> failed request.
>>>
>>> Exception stack is:
>>> 1. Locally closed (java.io.IOException)
>>> 2. java.io.IOException: Locally closed
>>> (java.util.concurrent.ExecutionException)
>>>    org.glassfish.grizzly.impl.SafeFutureImpl$Sync:363 (null)
>>> 3. java.util.concurrent.ExecutionException: java.io.IOException:
>>> Locally closed (java.io.IOException)
>>>    org.mule.module.http.internal.request.grizzly.GrizzlyHttpClient:274
>>> (null)
>>>
>>> In what cases does "locally closed" IOException occur in grizzly?  It
>>> seems strange that it's consistently seen exactly once even when we
>>> test with millions of request.
>>>
>>> Note: When we tested with a Jetty echo implementation we saw no errors
>>> at all, but with tomcat, grizzly, netty exactly one, weird!
>>>
>>> I'll keep looking into this, but any pointers would be great!
>>>
>>> thanks,
>>>
>>> Dan
>>>
>>>
>>> On Fri, Feb 13, 2015 at 6:15 PM, Daniel Feist <[hidden email]> wrote:
>>>> Hi,
>>>>
>>>> 1.8.x. We were originally using 1.8.14, but testing with 1.8.15 (which
>>>> has some changes to InputStreamBodyGenerator) made no difference.
>>>>
>>>> Dan
>>>>
>>>> On Fri, Feb 13, 2015 at 5:37 PM, Oleksiy Stashok
>>>> <[hidden email]> wrote:
>>>>> Hi Dan,
>>>>>
>>>>> which AHC version are you using?
>>>>>
>>>>> WBR,
>>>>> Alexey.
>>>>>
>>>>>
>>>>> On 13.02.15 07:35, Daniel Feist wrote:
>>>>>> Hi Alexey,
>>>>>>
>>>>>> We're geting a weird issue when using AHC/Grizzly for outbound HTTP
>>>>>> POST operations.  When testing using POST and giving AHC an
>>>>>> InputStream to send, we get a significant number of NPE's occuring (1
>>>>>> to 5% of requests), even at just 10 client threads.  I havn't been
>>>>>> able to reproduce this locally yet, but it happens consistently on our
>>>>>> test enviroment.
>>>>>>
>>>>>> I'm not sure how to get to the bottom of this (am in the middle of
>>>>>> trying) but given your much deeper knowledge of the codebase maybe you
>>>>>> have some insights/ideas. With a single client thread there are no
>>>>>> issues so it look like it may be some kind of race condition.  Also
>>>>>> passing a String to AHC or turn off chunking such that
>>>>>> com.ning.http.client.generators.InputStreamBodyGenerator doesn't get
>>>>>> used, there are no NPE's either.
>>>>>>
>>>>>> Dan
>>>>>>
>>>>>> -----
>>>>>>
>>>>>> This is the error seen:
>>>>>>
>>>>>> WARN  2015-02-13 09:06:38,336
>>>>>> [[http_module_proxy].proxyConfig.worker.10]
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain: GRIZZLY0013:
>>>>>> Exception during FilterChain execution
>>>>>> java.lang.NullPointerException
>>>>>>        at
>>>>>>
>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientEventFilter.onHttpContentEncoded(GrizzlyAsyncHttpProvider.java:1243)
>>>>>> ~[?:?]
>>>>>>        at
>>>>>>
>>>>>> org.glassfish.grizzly.http.HttpCodecFilter.encodeHttpPacket(HttpCodecFilter.java:1463)
>>>>>> ~[?:?]
>>>>>>        at
>>>>>>
>>>>>> org.glassfish.grizzly.http.HttpClientFilter.encodeHttpPacket(HttpClientFilter.java:283)
>>>>>> ~[?:?]
>>>>>

Reply | Threaded
Open this post in threaded view
|

Re: NPE's at low concurrency using AsyncHttpClient

Daniel Feist
Hi,

Sorry, I never did get to reproduce this and file an issue, it's still on the TODO list, just low down given it only affects POST's to static resource url's, not something you'd typically do.

BTW, cool to see you working on http2, i'll give you a shout when I find some time to take it for a test-drive.

Dan

On Mon, Feb 23, 2015 at 6:51 PM, Oleksiy Stashok <[hidden email]> wrote:
What happens if you increase concurrency and start to send POSTs at high rate? Does it help to reproduce the problem?

WBR,
Alexey.


On 23.02.15 05:06, Daniel Feist wrote:
I wasn't able to reproduce this in a test case yet, even though I tried.

It appears this issues is caused by the combination of:

i) Server no reading full http body
ii) Potentially also timing

Given it only happens when posting to a tomcat (also jetty) static resource.

Dan

On Tue, Feb 17, 2015 at 4:50 AM, Oleksiy Stashok
<[hidden email]> wrote:
Hi Dan,

you say it happens at low concurrency, so it shouldn't be difficult to
reproduce the failure.
Can you pls. share the testcase, it would be much easier for me to figure
out what's wrong.

Thank you.

WBR,
Alexey.


On 16.02.15 03:45, Daniel Feist wrote:
Hi,

After investigating further this is what I found.

It occured to me that the problem may not be with AHC/Grizzly but
rather provoked by the back-end service I was hitting.  So I switched
out the back-end implementation to see what happened:

Tomcat 6 hitting static resource (file in webapps/ROOT)
--------------------------------------------------------------------------
This is what causes the NPE's.  I realise it's probably a bad idea to
be testing with a POST request against a static resource, there was an
error in the test url we were using, but even so I wouldn't expect
NPE's client side.  The http body size is irrelevant and this only
occurs when InputStreamBodyGenerator is used. As i mentioned before
failures rate is 2-5% of requests.

Tomcat/Grizzly/Netty Echo Service
-----------------------------------------------
Using a echo service that reads in request stream from post and then
echo's response the NPE's go away but there is still always a singe
failed request.

Exception stack is:
1. Locally closed (java.io.IOException)
2. java.io.IOException: Locally closed
(java.util.concurrent.ExecutionException)
   org.glassfish.grizzly.impl.SafeFutureImpl$Sync:363 (null)
3. java.util.concurrent.ExecutionException: java.io.IOException:
Locally closed (java.io.IOException)
   org.mule.module.http.internal.request.grizzly.GrizzlyHttpClient:274
(null)

In what cases does "locally closed" IOException occur in grizzly?  It
seems strange that it's consistently seen exactly once even when we
test with millions of request.

Note: When we tested with a Jetty echo implementation we saw no errors
at all, but with tomcat, grizzly, netty exactly one, weird!

I'll keep looking into this, but any pointers would be great!

thanks,

Dan


On Fri, Feb 13, 2015 at 6:15 PM, Daniel Feist <[hidden email]> wrote:
Hi,

1.8.x. We were originally using 1.8.14, but testing with 1.8.15 (which
has some changes to InputStreamBodyGenerator) made no difference.

Dan

On Fri, Feb 13, 2015 at 5:37 PM, Oleksiy Stashok
<[hidden email]> wrote:
Hi Dan,

which AHC version are you using?

WBR,
Alexey.


On 13.02.15 07:35, Daniel Feist wrote:
Hi Alexey,

We're geting a weird issue when using AHC/Grizzly for outbound HTTP
POST operations.  When testing using POST and giving AHC an
InputStream to send, we get a significant number of NPE's occuring (1
to 5% of requests), even at just 10 client threads.  I havn't been
able to reproduce this locally yet, but it happens consistently on our
test enviroment.

I'm not sure how to get to the bottom of this (am in the middle of
trying) but given your much deeper knowledge of the codebase maybe you
have some insights/ideas. With a single client thread there are no
issues so it look like it may be some kind of race condition.  Also
passing a String to AHC or turn off chunking such that
com.ning.http.client.generators.InputStreamBodyGenerator doesn't get
used, there are no NPE's either.

Dan

-----

This is the error seen:

WARN  2015-02-13 09:06:38,336
[[http_module_proxy].proxyConfig.worker.10]
org.glassfish.grizzly.filterchain.DefaultFilterChain: GRIZZLY0013:
Exception during FilterChain execution
java.lang.NullPointerException
       at

com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientEventFilter.onHttpContentEncoded(GrizzlyAsyncHttpProvider.java:1243)
~[?:?]
       at

org.glassfish.grizzly.http.HttpCodecFilter.encodeHttpPacket(HttpCodecFilter.java:1463)
~[?:?]
       at

org.glassfish.grizzly.http.HttpClientFilter.encodeHttpPacket(HttpClientFilter.java:283)
~[?:?]



Reply | Threaded
Open this post in threaded view
|

Re: NPE's at low concurrency using AsyncHttpClient

oleksiys
Administrator
Hi Dan,

great! Let me know if you have more information on the issue. Just to make sure, you see the problem when you send POST to a StaticHttpHandler?

Thanks.

WBR,
Alexey.

On 23.03.15 09:41, Daniel Feist wrote:
Hi,

Sorry, I never did get to reproduce this and file an issue, it's still on the TODO list, just low down given it only affects POST's to static resource url's, not something you'd typically do.

BTW, cool to see you working on http2, i'll give you a shout when I find some time to take it for a test-drive.

Dan

On Mon, Feb 23, 2015 at 6:51 PM, Oleksiy Stashok <[hidden email]> wrote:
What happens if you increase concurrency and start to send POSTs at high rate? Does it help to reproduce the problem?

WBR,
Alexey.


On 23.02.15 05:06, Daniel Feist wrote:
I wasn't able to reproduce this in a test case yet, even though I tried.

It appears this issues is caused by the combination of:

i) Server no reading full http body
ii) Potentially also timing

Given it only happens when posting to a tomcat (also jetty) static resource.

Dan

On Tue, Feb 17, 2015 at 4:50 AM, Oleksiy Stashok
<[hidden email]> wrote:
Hi Dan,

you say it happens at low concurrency, so it shouldn't be difficult to
reproduce the failure.
Can you pls. share the testcase, it would be much easier for me to figure
out what's wrong.

Thank you.

WBR,
Alexey.


On 16.02.15 03:45, Daniel Feist wrote:
Hi,

After investigating further this is what I found.

It occured to me that the problem may not be with AHC/Grizzly but
rather provoked by the back-end service I was hitting.  So I switched
out the back-end implementation to see what happened:

Tomcat 6 hitting static resource (file in webapps/ROOT)
--------------------------------------------------------------------------
This is what causes the NPE's.  I realise it's probably a bad idea to
be testing with a POST request against a static resource, there was an
error in the test url we were using, but even so I wouldn't expect
NPE's client side.  The http body size is irrelevant and this only
occurs when InputStreamBodyGenerator is used. As i mentioned before
failures rate is 2-5% of requests.

Tomcat/Grizzly/Netty Echo Service
-----------------------------------------------
Using a echo service that reads in request stream from post and then
echo's response the NPE's go away but there is still always a singe
failed request.

Exception stack is:
1. Locally closed (java.io.IOException)
2. java.io.IOException: Locally closed
(java.util.concurrent.ExecutionException)
   org.glassfish.grizzly.impl.SafeFutureImpl$Sync:363 (null)
3. java.util.concurrent.ExecutionException: java.io.IOException:
Locally closed (java.io.IOException)
   org.mule.module.http.internal.request.grizzly.GrizzlyHttpClient:274
(null)

In what cases does "locally closed" IOException occur in grizzly?  It
seems strange that it's consistently seen exactly once even when we
test with millions of request.

Note: When we tested with a Jetty echo implementation we saw no errors
at all, but with tomcat, grizzly, netty exactly one, weird!

I'll keep looking into this, but any pointers would be great!

thanks,

Dan


On Fri, Feb 13, 2015 at 6:15 PM, Daniel Feist <[hidden email]> wrote:
Hi,

1.8.x. We were originally using 1.8.14, but testing with 1.8.15 (which
has some changes to InputStreamBodyGenerator) made no difference.

Dan

On Fri, Feb 13, 2015 at 5:37 PM, Oleksiy Stashok
<[hidden email]> wrote:
Hi Dan,

which AHC version are you using?

WBR,
Alexey.


On 13.02.15 07:35, Daniel Feist wrote:
Hi Alexey,

We're geting a weird issue when using AHC/Grizzly for outbound HTTP
POST operations.  When testing using POST and giving AHC an
InputStream to send, we get a significant number of NPE's occuring (1
to 5% of requests), even at just 10 client threads.  I havn't been
able to reproduce this locally yet, but it happens consistently on our
test enviroment.

I'm not sure how to get to the bottom of this (am in the middle of
trying) but given your much deeper knowledge of the codebase maybe you
have some insights/ideas. With a single client thread there are no
issues so it look like it may be some kind of race condition.  Also
passing a String to AHC or turn off chunking such that
com.ning.http.client.generators.InputStreamBodyGenerator doesn't get
used, there are no NPE's either.

Dan

-----

This is the error seen:

WARN  2015-02-13 09:06:38,336
[[http_module_proxy].proxyConfig.worker.10]
org.glassfish.grizzly.filterchain.DefaultFilterChain: GRIZZLY0013:
Exception during FilterChain execution
java.lang.NullPointerException
       at

com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientEventFilter.onHttpContentEncoded(GrizzlyAsyncHttpProvider.java:1243)
~[?:?]
       at

org.glassfish.grizzly.http.HttpCodecFilter.encodeHttpPacket(HttpCodecFilter.java:1463)
~[?:?]
       at

org.glassfish.grizzly.http.HttpClientFilter.encodeHttpPacket(HttpClientFilter.java:283)
~[?:?]