any way to make this faster/more scalable?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

any way to make this faster/more scalable?

jbrisbin
Can I change the readBufferSize based on Content-Length? This is really the question: is there any way (besides eliminating the buffer copy) to make this faster or more scalable?

https://github.com/jbrisbin/node-vs-java-shootout/blob/master/src/main/groovy/shootout/UploadHandler.groovy

It does fine now up to >100MB. But trying to upload a 1GB file is killing my CPU (GC maybe?). Just wondered if there aren't some micro-optimizations I can make to shave a few microseconds off here or there.


Thanks!

Jon Brisbin
http//jbrisbin.com


Reply | Threaded
Open this post in threaded view
|

Re: any way to make this faster/more scalable?

Ryan Lubke-2
On 7/26/11 11:02 AM, Jon Brisbin wrote:
Can I change the readBufferSize based on Content-Length? This is really the question: is there any way (besides eliminating the buffer copy) to make this faster or more scalable?

https://github.com/jbrisbin/node-vs-java-shootout/blob/master/src/main/groovy/shootout/UploadHandler.groovy

It does fine now up to >100MB. But trying to upload a 1GB file is killing my CPU (GC maybe?). Just wondered if there aren't some micro-optimizations I can make to shave a few microseconds off here or there.

Have you tried decreasing the number of file writes you're performing (i.e., increase the value for the number of bytes that must be received before invoking the ReadHandler callbacks)?



Thanks!

Jon Brisbin
http//jbrisbin.com



Reply | Threaded
Open this post in threaded view
|

Re: any way to make this faster/more scalable?

jbrisbin
I did try that, but it didn't seem to make any difference w.r.t. run times.

I've actually decreased the value some as that makes it more scalable. I've uploaded a 1GB file so far and that only takes a couple seconds, even though memory usage goes through the roof.

Should the readBufferSize match the size I give notifyAvailable() (e.g. should I try to keep them in sync, or can they be different)?

Thanks!

Jon Brisbin
http//jbrisbin.com



From: "Ryan Lubke" <[hidden email]>
To: [hidden email]
Sent: Tuesday, July 26, 2011 4:20:31 PM
Subject: Re: any way to make this faster/more scalable?

On 7/26/11 11:02 AM, Jon Brisbin wrote:
Can I change the readBufferSize based on Content-Length? This is really the question: is there any way (besides eliminating the buffer copy) to make this faster or more scalable?

https://github.com/jbrisbin/node-vs-java-shootout/blob/master/src/main/groovy/shootout/UploadHandler.groovy

It does fine now up to >100MB. But trying to upload a 1GB file is killing my CPU (GC maybe?). Just wondered if there aren't some micro-optimizations I can make to shave a few microseconds off here or there.

Have you tried decreasing the number of file writes you're performing (i.e., increase the value for the number of bytes that must be received before invoking the ReadHandler callbacks)?



Thanks!

Jon Brisbin
http//jbrisbin.com




Reply | Threaded
Open this post in threaded view
|

Re: any way to make this faster/more scalable?

oleksiys
Administrator
On 07/27/2011 12:09 AM, Jon Brisbin wrote:
I did try that, but it didn't seem to make any difference w.r.t. run times.

I've actually decreased the value some as that makes it more scalable. I've uploaded a 1GB file so far and that only takes a couple seconds, even though memory usage goes through the roof.

Should the readBufferSize match the size I give notifyAvailable() (e.g. should I try to keep them in sync, or can they be different)?
I don't think these values should be somehow correlated.

BTW, I see you run perf. tests on localhost, it would also make sense to try to split client and server to exclude undesirable side-effects and make test results more reliable.

Thanks.

WBR,
Alexey.



Thanks!

Jon Brisbin
http//jbrisbin.com



From: "Ryan Lubke" [hidden email]
To: [hidden email]
Sent: Tuesday, July 26, 2011 4:20:31 PM
Subject: Re: any way to make this faster/more scalable?

On 7/26/11 11:02 AM, Jon Brisbin wrote:
Can I change the readBufferSize based on Content-Length? This is really the question: is there any way (besides eliminating the buffer copy) to make this faster or more scalable?

https://github.com/jbrisbin/node-vs-java-shootout/blob/master/src/main/groovy/shootout/UploadHandler.groovy

It does fine now up to >100MB. But trying to upload a 1GB file is killing my CPU (GC maybe?). Just wondered if there aren't some micro-optimizations I can make to shave a few microseconds off here or there.

Have you tried decreasing the number of file writes you're performing (i.e., increase the value for the number of bytes that must be received before invoking the ReadHandler callbacks)?



Thanks!

Jon Brisbin
http//jbrisbin.com