100MB uploads

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

100MB uploads

jbrisbin
Looks like Grizzly isn't letting me do 100MB uploads to my REST service. How do I increase the limits to allow this size of an upload? I tried setting the maxPostSize to 100MB and it something is still causing me to get an NPE in my handler code because the inputBuffer in the request is NULL (as are most of the properties on the Request...kinda weird).


Thanks!

Jon Brisbin
http//jbrisbin.com


Reply | Threaded
Open this post in threaded view
|

Re: 100MB uploads

oleksiys
Administrator
Hi Jon,

using this sample [1] I was able to upload much bigger files. Can you pls. check if it works for you?
If you can modify the given sample to reproduce NPE - it would also help.

Thanks.

Alexey.

[1] http://java.net/projects/grizzly/sources/git/content/samples/http-server-samples/src/main/java/org/glassfish/grizzly/samples/httpserver/nonblockinghandler/UploadHttpHandlerSample.java

On 07/22/2011 01:46 AM, Jon Brisbin wrote:
Looks like Grizzly isn't letting me do 100MB uploads to my REST service. How do I increase the limits to allow this size of an upload? I tried setting the maxPostSize to 100MB and it something is still causing me to get an NPE in my handler code because the inputBuffer in the request is NULL (as are most of the properties on the Request...kinda weird).


Thanks!

Jon Brisbin
http//jbrisbin.com



Reply | Threaded
Open this post in threaded view
|

Re: 100MB uploads

jbrisbin
Sorry for the noise here...the problem was mine not Grizzly's in this case...still looking into the head problem though

Sent from my iPhone

On Jul 22, 2011, at 4:05 AM, Oleksiy Stashok <[hidden email]> wrote:

Hi Jon,

using this sample [1] I was able to upload much bigger files. Can you pls. check if it works for you?
If you can modify the given sample to reproduce NPE - it would also help.

Thanks.

Alexey.

[1] http://java.net/projects/grizzly/sources/git/content/samples/http-server-samples/src/main/java/org/glassfish/grizzly/samples/httpserver/nonblockinghandler/UploadHttpHandlerSample.java

On 07/22/2011 01:46 AM, Jon Brisbin wrote:
Looks like Grizzly isn't letting me do 100MB uploads to my REST service. How do I increase the limits to allow this size of an upload? I tried setting the maxPostSize to 100MB and it something is still causing me to get an NPE in my handler code because the inputBuffer in the request is NULL (as are most of the properties on the Request...kinda weird).


Thanks!

Jon Brisbin
http//jbrisbin.com



Reply | Threaded
Open this post in threaded view
|

Re: 100MB uploads

jbrisbin
In reply to this post by oleksiys
It seems that, on the 100MB uploads, Apache Bench will report errors if I go above a certain level of simultaneous requests (it varies depending on the load on my development machine). I don't see them all the time and I'm not even sure how to trigger them other than to just throw an insane number of simultaneous requests at it. I don't always see tracebacks, though, so it could be apache bench. I'm not seeing that on Node.js, though, so that's why I think it has something to do with Grizzly at a really low level.

I keep getting closer to creating a small test project for some of this stuff. When I get a breather I try and take time to do that... :)


Thanks for the great, patient help all the same!

Jon Brisbin
http//jbrisbin.com



From: "Oleksiy Stashok" <[hidden email]>
To: [hidden email]
Sent: Friday, July 22, 2011 4:05:13 AM
Subject: Re: 100MB uploads

Hi Jon,

using this sample [1] I was able to upload much bigger files. Can you pls. check if it works for you?
If you can modify the given sample to reproduce NPE - it would also help.

Thanks.

Alexey.

[1] http://java.net/projects/grizzly/sources/git/content/samples/http-server-samples/src/main/java/org/glassfish/grizzly/samples/httpserver/nonblockinghandler/UploadHttpHandlerSample.java

On 07/22/2011 01:46 AM, Jon Brisbin wrote:
Looks like Grizzly isn't letting me do 100MB uploads to my REST service. How do I increase the limits to allow this size of an upload? I tried setting the maxPostSize to 100MB and it something is still causing me to get an NPE in my handler code because the inputBuffer in the request is NULL (as are most of the properties on the Request...kinda weird).


Thanks!

Jon Brisbin
http//jbrisbin.com




Reply | Threaded
Open this post in threaded view
|

Re: 100MB uploads

jbrisbin
In reply to this post by oleksiys
I did happen to notice this in your upload sample:

} finally {
  // restore ByteBuffer old position, cause we don't know how it
  // correlates with parent Buffer position (depends on Buffer implementation)
  byteBuffer.position(oldPos);
  // Skip number of written bytes
  in.skip(written);
}

Why is this needed? Should I be doing this as well?


Thanks!

Jon Brisbin
http//jbrisbin.com



From: "Oleksiy Stashok" <[hidden email]>
To: [hidden email]
Sent: Friday, July 22, 2011 4:05:13 AM
Subject: Re: 100MB uploads

Hi Jon,

using this sample [1] I was able to upload much bigger files. Can you pls. check if it works for you?
If you can modify the given sample to reproduce NPE - it would also help.

Thanks.

Alexey.

[1] http://java.net/projects/grizzly/sources/git/content/samples/http-server-samples/src/main/java/org/glassfish/grizzly/samples/httpserver/nonblockinghandler/UploadHttpHandlerSample.java

On 07/22/2011 01:46 AM, Jon Brisbin wrote:
Looks like Grizzly isn't letting me do 100MB uploads to my REST service. How do I increase the limits to allow this size of an upload? I tried setting the maxPostSize to 100MB and it something is still causing me to get an NPE in my handler code because the inputBuffer in the request is NULL (as are most of the properties on the Request...kinda weird).


Thanks!

Jon Brisbin
http//jbrisbin.com




Reply | Threaded
Open this post in threaded view
|

Re: 100MB uploads

oleksiys
Administrator
On 07/22/2011 10:03 PM, Jon Brisbin wrote:
I did happen to notice this in your upload sample:

} finally {
  // restore ByteBuffer old position, cause we don't know how it
  // correlates with parent Buffer position (depends on Buffer implementation)
  byteBuffer.position(oldPos);
  // Skip number of written bytes
  in.skip(written);
}

Why is this needed? Should I be doing this as well?
Depends how you work with NIOInputStream. In my sample I work directly with NIOInputStream's Buffer, so to make sure I don't break it or leave NIOInputStream in inconsistent state, I restore the original Buffer position and call skip() with number of bytes I've already processed.

WBR,
Alexey.
 


Thanks!

Jon Brisbin
http//jbrisbin.com



From: "Oleksiy Stashok" [hidden email]
To: [hidden email]
Sent: Friday, July 22, 2011 4:05:13 AM
Subject: Re: 100MB uploads

Hi Jon,

using this sample [1] I was able to upload much bigger files. Can you pls. check if it works for you?
If you can modify the given sample to reproduce NPE - it would also help.

Thanks.

Alexey.

[1] http://java.net/projects/grizzly/sources/git/content/samples/http-server-samples/src/main/java/org/glassfish/grizzly/samples/httpserver/nonblockinghandler/UploadHttpHandlerSample.java

On 07/22/2011 01:46 AM, Jon Brisbin wrote:
Looks like Grizzly isn't letting me do 100MB uploads to my REST service. How do I increase the limits to allow this size of an upload? I tried setting the maxPostSize to 100MB and it something is still causing me to get an NPE in my handler code because the inputBuffer in the request is NULL (as are most of the properties on the Request...kinda weird).


Thanks!

Jon Brisbin
http//jbrisbin.com





Reply | Threaded
Open this post in threaded view
|

Re: 100MB uploads

jbrisbin
Ok, cool. I think I can avoid a buffer copy by doing this as well (I'm using in.read(byte[]) now)...


Thanks!

Jon Brisbin
http//jbrisbin.com



From: "Oleksiy Stashok" <[hidden email]>
To: [hidden email]
Sent: Friday, July 22, 2011 3:30:37 PM
Subject: Re: 100MB uploads

On 07/22/2011 10:03 PM, Jon Brisbin wrote:
I did happen to notice this in your upload sample:

} finally {
  // restore ByteBuffer old position, cause we don't know how it
  // correlates with parent Buffer position (depends on Buffer implementation)
  byteBuffer.position(oldPos);
  // Skip number of written bytes
  in.skip(written);
}

Why is this needed? Should I be doing this as well?
Depends how you work with NIOInputStream. In my sample I work directly with NIOInputStream's Buffer, so to make sure I don't break it or leave NIOInputStream in inconsistent state, I restore the original Buffer position and call skip() with number of bytes I've already processed.

WBR,
Alexey.
 


Thanks!

Jon Brisbin
http//jbrisbin.com



From: "Oleksiy Stashok" [hidden email]
To: [hidden email]
Sent: Friday, July 22, 2011 4:05:13 AM
Subject: Re: 100MB uploads

Hi Jon,

using this sample [1] I was able to upload much bigger files. Can you pls. check if it works for you?
If you can modify the given sample to reproduce NPE - it would also help.

Thanks.

Alexey.

[1] http://java.net/projects/grizzly/sources/git/content/samples/http-server-samples/src/main/java/org/glassfish/grizzly/samples/httpserver/nonblockinghandler/UploadHttpHandlerSample.java

On 07/22/2011 01:46 AM, Jon Brisbin wrote:
Looks like Grizzly isn't letting me do 100MB uploads to my REST service. How do I increase the limits to allow this size of an upload? I tried setting the maxPostSize to 100MB and it something is still causing me to get an NPE in my handler code because the inputBuffer in the request is NULL (as are most of the properties on the Request...kinda weird).


Thanks!

Jon Brisbin
http//jbrisbin.com






Reply | Threaded
Open this post in threaded view
|

Re: 100MB uploads

jbrisbin
In reply to this post by oleksiys
So what's the absolute fastest way to stream an upload to disk? (Assuming you're competing with node.js and want to keep it even)

Sent from my iPhone

On Jul 22, 2011, at 3:30 PM, Oleksiy Stashok <[hidden email]> wrote:

On 07/22/2011 10:03 PM, Jon Brisbin wrote:
I did happen to notice this in your upload sample:

} finally {
  // restore ByteBuffer old position, cause we don't know how it
  // correlates with parent Buffer position (depends on Buffer implementation)
  byteBuffer.position(oldPos);
  // Skip number of written bytes
  in.skip(written);
}

Why is this needed? Should I be doing this as well?
Depends how you work with NIOInputStream. In my sample I work directly with NIOInputStream's Buffer, so to make sure I don't break it or leave NIOInputStream in inconsistent state, I restore the original Buffer position and call skip() with number of bytes I've already processed.

WBR,
Alexey.
 


Thanks!

Jon Brisbin
http//jbrisbin.com



From: "Oleksiy Stashok" [hidden email]>
To: [hidden email]
Sent: Friday, July 22, 2011 4:05:13 AM
Subject: Re: 100MB uploads

Hi Jon,

using this sample [1] I was able to upload much bigger files. Can you pls. check if it works for you?
If you can modify the given sample to reproduce NPE - it would also help.

Thanks.

Alexey.

[1] http://java.net/projects/grizzly/sources/git/content/samples/http-server-samples/src/main/java/org/glassfish/grizzly/samples/httpserver/nonblockinghandler/UploadHttpHandlerSample.java

On 07/22/2011 01:46 AM, Jon Brisbin wrote:
Looks like Grizzly isn't letting me do 100MB uploads to my REST service. How do I increase the limits to allow this size of an upload? I tried setting the maxPostSize to 100MB and it something is still causing me to get an NPE in my handler code because the inputBuffer in the request is NULL (as are most of the properties on the Request...kinda weird).


Thanks!

Jon Brisbin
http//jbrisbin.com





Reply | Threaded
Open this post in threaded view
|

Re: 100MB uploads

oleksiys
Administrator
IMO it would make sense (at least to try) to use SameThreadIOStrategy and Direct ByteBuffers. For sure it would make sense only if your HttpHandler is not blocking and you don't use SSL or compression.

WBR,
Alexey.


On 07/23/2011 01:36 AM, Jon Brisbin wrote:
So what's the absolute fastest way to stream an upload to disk? (Assuming you're competing with node.js and want to keep it even)

Sent from my iPhone

On Jul 22, 2011, at 3:30 PM, Oleksiy Stashok <[hidden email]> wrote:

On 07/22/2011 10:03 PM, Jon Brisbin wrote:
I did happen to notice this in your upload sample:

} finally {
  // restore ByteBuffer old position, cause we don't know how it
  // correlates with parent Buffer position (depends on Buffer implementation)
  byteBuffer.position(oldPos);
  // Skip number of written bytes
  in.skip(written);
}

Why is this needed? Should I be doing this as well?
Depends how you work with NIOInputStream. In my sample I work directly with NIOInputStream's Buffer, so to make sure I don't break it or leave NIOInputStream in inconsistent state, I restore the original Buffer position and call skip() with number of bytes I've already processed.

WBR,
Alexey.
 


Thanks!

Jon Brisbin
http//jbrisbin.com



From: "Oleksiy Stashok" [hidden email][hidden email]>
To: [hidden email]
Sent: Friday, July 22, 2011 4:05:13 AM
Subject: Re: 100MB uploads

Hi Jon,

using this sample [1] I was able to upload much bigger files. Can you pls. check if it works for you?
If you can modify the given sample to reproduce NPE - it would also help.

Thanks.

Alexey.

[1] http://java.net/projects/grizzly/sources/git/content/samples/http-server-samples/src/main/java/org/glassfish/grizzly/samples/httpserver/nonblockinghandler/UploadHttpHandlerSample.java

On 07/22/2011 01:46 AM, Jon Brisbin wrote:
Looks like Grizzly isn't letting me do 100MB uploads to my REST service. How do I increase the limits to allow this size of an upload? I tried setting the maxPostSize to 100MB and it something is still causing me to get an NPE in my handler code because the inputBuffer in the request is NULL (as are most of the properties on the Request...kinda weird).


Thanks!

Jon Brisbin
http//jbrisbin.com