Weird issue with HTTP download sample.

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Weird issue with HTTP download sample.

Daniel Feist
Hi,

I've found what I think is something wierd going on with the http
download example included in the source tree.  If you run it with a
payload larger than chunk size of 1KB then and compare the source file
(read by Server), and the target file (saved to disk by the Client)
they are not the same.  The file recieved has some content but then is
nearly all 0's.  I've tried this with a number of different payload
sizes and see rougly the same.   I'm still looking into the root
cause, I'll let you know what I find, but it would be interesting to
see if anyone else has seen this before or has any thoughts on where
to look.

Is this the recommended approach for sending chunked responses with
Grizzly? Or rather should org.glassfish.grizzly.http.io.OutputBuffer
be used somehow?

thanks,
Dan
Reply | Threaded
Open this post in threaded view
|

Re: Weird issue with HTTP download sample.

Daniel Feist
The plot thickens ... seems this only happens with default
HeapMemoryManager, with Pooled/ByteBuffer managers things look good.

Is there and inherrent incompatability between ThreadLocal caching of
buffer used by PooledByteBuffer and the write()->onComplete()->write()
chain used for streaming response chunks?

Wondering if there is an alternative way to do response chunking, or
if default MemoryManager should be changed in order for this typical
use case to be supported correctly by Grizzly out of the box...

Dan

On Tue, Jan 6, 2015 at 10:47 PM, Daniel Feist <[hidden email]> wrote:

> Hi,
>
> I've found what I think is something wierd going on with the http
> download example included in the source tree.  If you run it with a
> payload larger than chunk size of 1KB then and compare the source file
> (read by Server), and the target file (saved to disk by the Client)
> they are not the same.  The file recieved has some content but then is
> nearly all 0's.  I've tried this with a number of different payload
> sizes and see rougly the same.   I'm still looking into the root
> cause, I'll let you know what I find, but it would be interesting to
> see if anyone else has seen this before or has any thoughts on where
> to look.
>
> Is this the recommended approach for sending chunked responses with
> Grizzly? Or rather should org.glassfish.grizzly.http.io.OutputBuffer
> be used somehow?
>
> thanks,
> Dan
Reply | Threaded
Open this post in threaded view
|

Re: Weird issue with HTTP download sample.

oleksiys
Administrator
In reply to this post by Daniel Feist
Hi Dan,

On 06.01.15 17:47, Daniel Feist wrote:

> Hi,
>
> I've found what I think is something wierd going on with the http
> download example included in the source tree.  If you run it with a
> payload larger than chunk size of 1KB then and compare the source file
> (read by Server), and the target file (saved to disk by the Client)
> they are not the same.  The file recieved has some content but then is
> nearly all 0's.  I've tried this with a number of different payload
> sizes and see rougly the same.   I'm still looking into the root
> cause, I'll let you know what I find, but it would be interesting to
> see if anyone else has seen this before or has any thoughts on where
> to look.
which exactly sample did you try?

> Is this the recommended approach for sending chunked responses with
> Grizzly? Or rather should org.glassfish.grizzly.http.io.OutputBuffer
> be used somehow?
If you use HttpServer API along with Response.getOutputStream() - it
should take care of the chunking. If the response is smaller than output
buffer size - it'll be sent using Content-Length, if the response is
larger or outputStream.flush() is explicitly called - the chinking
encoding will be used.

WBR,
Alexey.

>
> thanks,
> Dan

Reply | Threaded
Open this post in threaded view
|

Re: Weird issue with HTTP download sample.

Daniel Feist
Hi,

This is using org.glassfish.grizzly.samples.http.download.Server along
with the Client in same maven module of course.  I tested with the
latest version in 2.3.x branch.  See my second email for more info on
what I think is going wrong..

We are currently not using the HttpServer API (no real reason, just
didn't really need it), and thats why we are using the approach in
this example that leverages a CompletionHandler.  Does whats behind
Response.getOutputStream() use the same approach, or an alternative?

thanks,
Dan

On Tue, Jan 6, 2015 at 11:14 PM, Oleksiy Stashok
<[hidden email]> wrote:

> Hi Dan,
>
> On 06.01.15 17:47, Daniel Feist wrote:
>>
>> Hi,
>>
>> I've found what I think is something wierd going on with the http
>> download example included in the source tree.  If you run it with a
>> payload larger than chunk size of 1KB then and compare the source file
>> (read by Server), and the target file (saved to disk by the Client)
>> they are not the same.  The file recieved has some content but then is
>> nearly all 0's.  I've tried this with a number of different payload
>> sizes and see rougly the same.   I'm still looking into the root
>> cause, I'll let you know what I find, but it would be interesting to
>> see if anyone else has seen this before or has any thoughts on where
>> to look.
>
> which exactly sample did you try?
>
>> Is this the recommended approach for sending chunked responses with
>> Grizzly? Or rather should org.glassfish.grizzly.http.io.OutputBuffer
>> be used somehow?
>
> If you use HttpServer API along with Response.getOutputStream() - it should
> take care of the chunking. If the response is smaller than output buffer
> size - it'll be sent using Content-Length, if the response is larger or
> outputStream.flush() is explicitly called - the chinking encoding will be
> used.
>
> WBR,
> Alexey.
>
>>
>> thanks,
>> Dan
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Weird issue with HTTP download sample.

oleksiys
Administrator
Hi,


On 07.01.15 04:17, Daniel Feist wrote:
> This is using org.glassfish.grizzly.samples.http.download.Server along
> with the Client in same maven module of course.  I tested with the
> latest version in 2.3.x branch.  See my second email for more info on
> what I think is going wrong..
Fixed, nice catch!

> We are currently not using the HttpServer API (no real reason, just
> didn't really need it), and thats why we are using the approach in
> this example that leverages a CompletionHandler.  Does whats behind
> Response.getOutputStream() use the same approach, or an alternative?
Well, HttpServer API is similar to Servlet API, it might be easier to
use it, but there is nothing wrong with your approach.
And yes, HttpServer API uses the FilterChain similar to http-sample and
the one you use.

Thanks.

WBR,
Alexey.

>
> thanks,
> Dan
>
> On Tue, Jan 6, 2015 at 11:14 PM, Oleksiy Stashok
> <[hidden email]> wrote:
>> Hi Dan,
>>
>> On 06.01.15 17:47, Daniel Feist wrote:
>>> Hi,
>>>
>>> I've found what I think is something wierd going on with the http
>>> download example included in the source tree.  If you run it with a
>>> payload larger than chunk size of 1KB then and compare the source file
>>> (read by Server), and the target file (saved to disk by the Client)
>>> they are not the same.  The file recieved has some content but then is
>>> nearly all 0's.  I've tried this with a number of different payload
>>> sizes and see rougly the same.   I'm still looking into the root
>>> cause, I'll let you know what I find, but it would be interesting to
>>> see if anyone else has seen this before or has any thoughts on where
>>> to look.
>> which exactly sample did you try?
>>
>>> Is this the recommended approach for sending chunked responses with
>>> Grizzly? Or rather should org.glassfish.grizzly.http.io.OutputBuffer
>>> be used somehow?
>> If you use HttpServer API along with Response.getOutputStream() - it should
>> take care of the chunking. If the response is smaller than output buffer
>> size - it'll be sent using Content-Length, if the response is larger or
>> outputStream.flush() is explicitly called - the chinking encoding will be
>> used.
>>
>> WBR,
>> Alexey.
>>
>>> thanks,
>>> Dan
>>

Reply | Threaded
Open this post in threaded view
|

Re: Weird issue with HTTP download sample.

Daniel Feist
Do you have the JIRA # and/or commit reference so I can take a look
and see what the fix was?

Thanks!

On Wed, Jan 7, 2015 at 6:18 PM, Oleksiy Stashok
<[hidden email]> wrote:

> Hi,
>
>
> On 07.01.15 04:17, Daniel Feist wrote:
>>
>> This is using org.glassfish.grizzly.samples.http.download.Server along
>> with the Client in same maven module of course.  I tested with the
>> latest version in 2.3.x branch.  See my second email for more info on
>> what I think is going wrong..
>
> Fixed, nice catch!
>
>> We are currently not using the HttpServer API (no real reason, just
>> didn't really need it), and thats why we are using the approach in
>> this example that leverages a CompletionHandler.  Does whats behind
>> Response.getOutputStream() use the same approach, or an alternative?
>
> Well, HttpServer API is similar to Servlet API, it might be easier to use
> it, but there is nothing wrong with your approach.
> And yes, HttpServer API uses the FilterChain similar to http-sample and the
> one you use.
>
> Thanks.
>
> WBR,
> Alexey.
>
>
>>
>> thanks,
>> Dan
>>
>> On Tue, Jan 6, 2015 at 11:14 PM, Oleksiy Stashok
>> <[hidden email]> wrote:
>>>
>>> Hi Dan,
>>>
>>> On 06.01.15 17:47, Daniel Feist wrote:
>>>>
>>>> Hi,
>>>>
>>>> I've found what I think is something wierd going on with the http
>>>> download example included in the source tree.  If you run it with a
>>>> payload larger than chunk size of 1KB then and compare the source file
>>>> (read by Server), and the target file (saved to disk by the Client)
>>>> they are not the same.  The file recieved has some content but then is
>>>> nearly all 0's.  I've tried this with a number of different payload
>>>> sizes and see rougly the same.   I'm still looking into the root
>>>> cause, I'll let you know what I find, but it would be interesting to
>>>> see if anyone else has seen this before or has any thoughts on where
>>>> to look.
>>>
>>> which exactly sample did you try?
>>>
>>>> Is this the recommended approach for sending chunked responses with
>>>> Grizzly? Or rather should org.glassfish.grizzly.http.io.OutputBuffer
>>>> be used somehow?
>>>
>>> If you use HttpServer API along with Response.getOutputStream() - it
>>> should
>>> take care of the chunking. If the response is smaller than output buffer
>>> size - it'll be sent using Content-Length, if the response is larger or
>>> outputStream.flush() is explicitly called - the chinking encoding will be
>>> used.
>>>
>>> WBR,
>>> Alexey.
>>>
>>>> thanks,
>>>> Dan
>>>
>>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Weird issue with HTTP download sample.

oleksiys
Administrator
@@ -275,8 +275,8 @@ public class WebServerFilter extends BaseFilter {
              final Buffer buffer = memoryManager.allocate(1024);
             
              // prepare byte[] for InputStream.read(...)
-            final byte[] bufferByteArray = buffer.toByteBuffer().array();
-            final int offset = buffer.toByteBuffer().arrayOffset();
+            final byte[] bufferByteArray = buffer.array();
+            final int offset = buffer.arrayOffset();
              final int length = buffer.remaining();

The problem with the original code was that we didn't consider buffer.toByteBuffer().position();
Anyway the new code is simpler and takes care of that underneath.

Please note, that this sample won't work with memory managers, that allocate direct buffers, which don't have byte[] underneath.

WBR,
Alexey.

On 07.01.15 14:25, Daniel Feist wrote:

> Do you have the JIRA # and/or commit reference so I can take a look
> and see what the fix was?
>
> Thanks!
>
> On Wed, Jan 7, 2015 at 6:18 PM, Oleksiy Stashok
> <[hidden email]> wrote:
>> Hi,
>>
>>
>> On 07.01.15 04:17, Daniel Feist wrote:
>>> This is using org.glassfish.grizzly.samples.http.download.Server along
>>> with the Client in same maven module of course.  I tested with the
>>> latest version in 2.3.x branch.  See my second email for more info on
>>> what I think is going wrong..
>> Fixed, nice catch!
>>
>>> We are currently not using the HttpServer API (no real reason, just
>>> didn't really need it), and thats why we are using the approach in
>>> this example that leverages a CompletionHandler.  Does whats behind
>>> Response.getOutputStream() use the same approach, or an alternative?
>> Well, HttpServer API is similar to Servlet API, it might be easier to use
>> it, but there is nothing wrong with your approach.
>> And yes, HttpServer API uses the FilterChain similar to http-sample and the
>> one you use.
>>
>> Thanks.
>>
>> WBR,
>> Alexey.
>>
>>
>>> thanks,
>>> Dan
>>>
>>> On Tue, Jan 6, 2015 at 11:14 PM, Oleksiy Stashok
>>> <[hidden email]> wrote:
>>>> Hi Dan,
>>>>
>>>> On 06.01.15 17:47, Daniel Feist wrote:
>>>>> Hi,
>>>>>
>>>>> I've found what I think is something wierd going on with the http
>>>>> download example included in the source tree.  If you run it with a
>>>>> payload larger than chunk size of 1KB then and compare the source file
>>>>> (read by Server), and the target file (saved to disk by the Client)
>>>>> they are not the same.  The file recieved has some content but then is
>>>>> nearly all 0's.  I've tried this with a number of different payload
>>>>> sizes and see rougly the same.   I'm still looking into the root
>>>>> cause, I'll let you know what I find, but it would be interesting to
>>>>> see if anyone else has seen this before or has any thoughts on where
>>>>> to look.
>>>> which exactly sample did you try?
>>>>
>>>>> Is this the recommended approach for sending chunked responses with
>>>>> Grizzly? Or rather should org.glassfish.grizzly.http.io.OutputBuffer
>>>>> be used somehow?
>>>> If you use HttpServer API along with Response.getOutputStream() - it
>>>> should
>>>> take care of the chunking. If the response is smaller than output buffer
>>>> size - it'll be sent using Content-Length, if the response is larger or
>>>> outputStream.flush() is explicitly called - the chinking encoding will be
>>>> used.
>>>>
>>>> WBR,
>>>> Alexey.
>>>>
>>>>> thanks,
>>>>> Dan
>>>>

Reply | Threaded
Open this post in threaded view
|

Re: Weird issue with HTTP download sample.

Daniel Feist
Interesting.  What I don't understand though is why changing from
HeapMemoryManager->PooledMemoryManager (both using heap) fixed this
issue for me, without requiring any changes to WebServerFilter.

Dan

On Wed, Jan 7, 2015 at 7:31 PM, Oleksiy Stashok
<[hidden email]> wrote:

> @@ -275,8 +275,8 @@ public class WebServerFilter extends BaseFilter {
>              final Buffer buffer = memoryManager.allocate(1024);
>                           // prepare byte[] for InputStream.read(...)
> -            final byte[] bufferByteArray = buffer.toByteBuffer().array();
> -            final int offset = buffer.toByteBuffer().arrayOffset();
> +            final byte[] bufferByteArray = buffer.array();
> +            final int offset = buffer.arrayOffset();
>              final int length = buffer.remaining();
>
> The problem with the original code was that we didn't consider
> buffer.toByteBuffer().position();
> Anyway the new code is simpler and takes care of that underneath.
>
> Please note, that this sample won't work with memory managers, that allocate
> direct buffers, which don't have byte[] underneath.
>
> WBR,
> Alexey.
>
>
> On 07.01.15 14:25, Daniel Feist wrote:
>>
>> Do you have the JIRA # and/or commit reference so I can take a look
>> and see what the fix was?
>>
>> Thanks!
>>
>> On Wed, Jan 7, 2015 at 6:18 PM, Oleksiy Stashok
>> <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>>
>>> On 07.01.15 04:17, Daniel Feist wrote:
>>>>
>>>> This is using org.glassfish.grizzly.samples.http.download.Server along
>>>> with the Client in same maven module of course.  I tested with the
>>>> latest version in 2.3.x branch.  See my second email for more info on
>>>> what I think is going wrong..
>>>
>>> Fixed, nice catch!
>>>
>>>> We are currently not using the HttpServer API (no real reason, just
>>>> didn't really need it), and thats why we are using the approach in
>>>> this example that leverages a CompletionHandler.  Does whats behind
>>>> Response.getOutputStream() use the same approach, or an alternative?
>>>
>>> Well, HttpServer API is similar to Servlet API, it might be easier to use
>>> it, but there is nothing wrong with your approach.
>>> And yes, HttpServer API uses the FilterChain similar to http-sample and
>>> the
>>> one you use.
>>>
>>> Thanks.
>>>
>>> WBR,
>>> Alexey.
>>>
>>>
>>>> thanks,
>>>> Dan
>>>>
>>>> On Tue, Jan 6, 2015 at 11:14 PM, Oleksiy Stashok
>>>> <[hidden email]> wrote:
>>>>>
>>>>> Hi Dan,
>>>>>
>>>>> On 06.01.15 17:47, Daniel Feist wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I've found what I think is something wierd going on with the http
>>>>>> download example included in the source tree.  If you run it with a
>>>>>> payload larger than chunk size of 1KB then and compare the source file
>>>>>> (read by Server), and the target file (saved to disk by the Client)
>>>>>> they are not the same.  The file recieved has some content but then is
>>>>>> nearly all 0's.  I've tried this with a number of different payload
>>>>>> sizes and see rougly the same.   I'm still looking into the root
>>>>>> cause, I'll let you know what I find, but it would be interesting to
>>>>>> see if anyone else has seen this before or has any thoughts on where
>>>>>> to look.
>>>>>
>>>>> which exactly sample did you try?
>>>>>
>>>>>> Is this the recommended approach for sending chunked responses with
>>>>>> Grizzly? Or rather should org.glassfish.grizzly.http.io.OutputBuffer
>>>>>> be used somehow?
>>>>>
>>>>> If you use HttpServer API along with Response.getOutputStream() - it
>>>>> should
>>>>> take care of the chunking. If the response is smaller than output
>>>>> buffer
>>>>> size - it'll be sent using Content-Length, if the response is larger or
>>>>> outputStream.flush() is explicitly called - the chinking encoding will
>>>>> be
>>>>> used.
>>>>>
>>>>> WBR,
>>>>> Alexey.
>>>>>
>>>>>> thanks,
>>>>>> Dan
>>>>>
>>>>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Weird issue with HTTP download sample.

oleksiys
Administrator
On 07.01.15 20:40, Daniel Feist wrote:
> Interesting.  What I don't understand though is why changing from
> HeapMemoryManager->PooledMemoryManager (both using heap) fixed this
> issue for me, without requiring any changes to WebServerFilter.
Most probably because
ByteBuffer byteBuffer = buffer.toByteBuffer();

returned by PooledMemoryManager had position == 0, where
HeapMemoryManager wasn't 0.
So we read data from the file not to the Buffer's memory, but some
adjacent chunk.

WBR,
Alexey.

>
> Dan
>
> On Wed, Jan 7, 2015 at 7:31 PM, Oleksiy Stashok
> <[hidden email]> wrote:
>> @@ -275,8 +275,8 @@ public class WebServerFilter extends BaseFilter {
>>               final Buffer buffer = memoryManager.allocate(1024);
>>                            // prepare byte[] for InputStream.read(...)
>> -            final byte[] bufferByteArray = buffer.toByteBuffer().array();
>> -            final int offset = buffer.toByteBuffer().arrayOffset();
>> +            final byte[] bufferByteArray = buffer.array();
>> +            final int offset = buffer.arrayOffset();
>>               final int length = buffer.remaining();
>>
>> The problem with the original code was that we didn't consider
>> buffer.toByteBuffer().position();
>> Anyway the new code is simpler and takes care of that underneath.
>>
>> Please note, that this sample won't work with memory managers, that allocate
>> direct buffers, which don't have byte[] underneath.
>>
>> WBR,
>> Alexey.
>>
>>
>> On 07.01.15 14:25, Daniel Feist wrote:
>>> Do you have the JIRA # and/or commit reference so I can take a look
>>> and see what the fix was?
>>>
>>> Thanks!
>>>
>>> On Wed, Jan 7, 2015 at 6:18 PM, Oleksiy Stashok
>>> <[hidden email]> wrote:
>>>> Hi,
>>>>
>>>>
>>>> On 07.01.15 04:17, Daniel Feist wrote:
>>>>> This is using org.glassfish.grizzly.samples.http.download.Server along
>>>>> with the Client in same maven module of course.  I tested with the
>>>>> latest version in 2.3.x branch.  See my second email for more info on
>>>>> what I think is going wrong..
>>>> Fixed, nice catch!
>>>>
>>>>> We are currently not using the HttpServer API (no real reason, just
>>>>> didn't really need it), and thats why we are using the approach in
>>>>> this example that leverages a CompletionHandler.  Does whats behind
>>>>> Response.getOutputStream() use the same approach, or an alternative?
>>>> Well, HttpServer API is similar to Servlet API, it might be easier to use
>>>> it, but there is nothing wrong with your approach.
>>>> And yes, HttpServer API uses the FilterChain similar to http-sample and
>>>> the
>>>> one you use.
>>>>
>>>> Thanks.
>>>>
>>>> WBR,
>>>> Alexey.
>>>>
>>>>
>>>>> thanks,
>>>>> Dan
>>>>>
>>>>> On Tue, Jan 6, 2015 at 11:14 PM, Oleksiy Stashok
>>>>> <[hidden email]> wrote:
>>>>>> Hi Dan,
>>>>>>
>>>>>> On 06.01.15 17:47, Daniel Feist wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I've found what I think is something wierd going on with the http
>>>>>>> download example included in the source tree.  If you run it with a
>>>>>>> payload larger than chunk size of 1KB then and compare the source file
>>>>>>> (read by Server), and the target file (saved to disk by the Client)
>>>>>>> they are not the same.  The file recieved has some content but then is
>>>>>>> nearly all 0's.  I've tried this with a number of different payload
>>>>>>> sizes and see rougly the same.   I'm still looking into the root
>>>>>>> cause, I'll let you know what I find, but it would be interesting to
>>>>>>> see if anyone else has seen this before or has any thoughts on where
>>>>>>> to look.
>>>>>> which exactly sample did you try?
>>>>>>
>>>>>>> Is this the recommended approach for sending chunked responses with
>>>>>>> Grizzly? Or rather should org.glassfish.grizzly.http.io.OutputBuffer
>>>>>>> be used somehow?
>>>>>> If you use HttpServer API along with Response.getOutputStream() - it
>>>>>> should
>>>>>> take care of the chunking. If the response is smaller than output
>>>>>> buffer
>>>>>> size - it'll be sent using Content-Length, if the response is larger or
>>>>>> outputStream.flush() is explicitly called - the chinking encoding will
>>>>>> be
>>>>>> used.
>>>>>>
>>>>>> WBR,
>>>>>> Alexey.
>>>>>>
>>>>>>> thanks,
>>>>>>> Dan
>>>>>>