File upload loadtesting

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

File upload loadtesting

gagansnt
I am developing Grizzly based Fileserver which can handle large number of simultaneous file uploads. I have used the grizzly multipart-form upload sample code. In my testcase, I am uploading 30 files each of size 2GB. I am using 5 clients, so each is uploading 6 files.

Following image is taken from the ksar report of my server. It shows that user cpu is has not gone beyond 10-15% and most of the time System cpu and I/O wait is happening. Can somebody please elaborate what might be happening inside the code.

CPU utilization


Following image shows the network traffic happening through eth0 card of my server.
Interface traffic
Reply | Threaded
Open this post in threaded view
|

Re: File upload loadtesting

Ryan Lubke-2
On 5/10/11 12:05 AM, gagansnt wrote:
> I am developing Grizzly based Fileserver which can handle large number of
> simultaneous file uploads. I have used the grizzly multipart-form upload
> sample code. In my testcase, I am uploading 30 files each of size 2GB. I am
> using 5 clients, so each is uploading 6 files.
>
> Following image is taken from the ksar report of my server. It shows that
> user cpu is has not gone beyond 10-15% and most of the time System cpu and
> I/O wait is happening. Can somebody please elaborate what might be happening
> inside the code.

Can you share how you have the Grizzly runtime configured?

> CPU utilization
> http://grizzly.1045725.n5.nabble.com/file/n4383816/ksar.png
>
> Following image shows the network traffic happening through eth0 card of my
> server.
> Interface traffic
> http://grizzly.1045725.n5.nabble.com/file/n4383816/interface_traffic.png
>
> --
> View this message in context: http://grizzly.1045725.n5.nabble.com/File-upload-loadtesting-tp4383816p4383816.html
> Sent from the Grizzly - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: File upload loadtesting

gagansnt
I am running it as independent process. Using the HttpServer.createSimpleServer() following the code snippet of server main class -

HttpServer server = HttpServer.createSimpleServer();  
server.getServerConfiguration().addHttpHandler(new FormHttpHandler(),"/uploadFileForm");  
    server.getServerConfiguration().addHttpHandler(new UploaderHttpHandler(), "/uploadFile");  
       
       
    server.getServerConfiguration().setJmxEnabled(true);  
    logger.info("Application started");  
 
    try {  
         
      server.start();        
      System.out.println("Press any key to stop the server...");  
 
      System.in.read();  
    }
Reply | Threaded
Open this post in threaded view
|

Re: File upload loadtesting

gagansnt
Hi Alexey,

What do you say about the CPU graphs, I think you know much more about the internals of how Grizzly should behave. Is the graphs inline with your understanding.

Can you please help me understand the behavior.

Thanks in advance
Gagan
Reply | Threaded
Open this post in threaded view
|

Re: File upload loadtesting

Ryan Lubke-2
On 5/12/11 5:00 AM, gagansnt wrote:
Hi Alexey, 

What do you say about the CPU graphs, I think you know much more about the
internals of how Grizzly should behave. Is the graphs inline with your
understanding. 

During load testing, we never saw the CPU utilization (total) go above 20% (this was with 8 selectors and 120 silmulated clients).  The iowait, isn't network related, but disk related - per the man page: Percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.  So it seems your disk maybe can't keep up with the write requests. 

Can you please help me understand the behavior. 

Thanks in advance
Gagan

--
View this message in context: http://grizzly.1045725.n5.nabble.com/File-upload-loadtesting-tp4383816p4390041.html
Sent from the Grizzly - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: File upload loadtesting

oleksiys
Administrator
In reply to this post by gagansnt
Hi Gagan,

as Ryan suggested it might be related to the file I/O.
Just out of curiosity can you exclude all the file writings and just
swallow the uploading data? And take snapshot of that?

Thanks.

WBR,
Alexey.

On 05/12/2011 02:00 PM, gagansnt wrote:

> Hi Alexey,
>
> What do you say about the CPU graphs, I think you know much more about the
> internals of how Grizzly should behave. Is the graphs inline with your
> understanding.
>
> Can you please help me understand the behavior.
>
> Thanks in advance
> Gagan
>
> --
> View this message in context: http://grizzly.1045725.n5.nabble.com/File-upload-loadtesting-tp4383816p4390041.html
> Sent from the Grizzly - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: File upload loadtesting

gagansnt
Hi Alexey,

Thanks for replying. You and Ryan are right and infact after Ryan's response we were looking at various options available for speedy write. Trying different options like - RAID-0 with 2 disks ,3-disks and also trying to change the block size. So far we are getting consideration improvements. Will post our results soon.

Thanks,
Gagan