Grizzly in Hadoop

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Grizzly in Hadoop

Devaraj Das
Hi,

We are considering using Grizzly (1.5) in Hadoop (an open source framework
that has the MapReduce and Distributed File System implementations). The
main reason for using it is to optimize a framework phase called "shuffle".
In this phase we move lots of data across the network.

We are currently using HTTP for moving the data (actually files) and we use
Jetty5. Now we are thinking of moving to Grizzly (to have NIO and all its
niceness). But initial experiments with our benchmark showed that with
Grizzly the performance of the shuffle phase is nearly the same as we have
with Jetty5. This is not what we initially expected and hence would like to
get feedback on where we might be going wrong.

Hadoop is designed to run on large clusters of 100s of nodes (currently it
can run stable/reliably in a 1K node cluster). From the Grizzly point of
view, what needs to be known is that each node has a HTTP server. Both
Jetty5 and Grizzly provides the ability to have multiple handlers to service
the incoming requests.

There are 2 clients on each node, and each client has a configurable number
of fetcher threads. The fetcher code is written using the
java.net.URLConnection API.
Every node has both the server and the clients. The threads basically hit
the HTTP server asking for specific files. They all do this at once (with
some randomness in the order of hosts, maybe).

The benchmark that I tested with is a sort for ~5TB of data with a cluster
of 500 nodes. On the hardware side, I used a cluster that has 4 dualcore
processors in each machine. The machines are partitioned into racks with a
gigabit ethernet within the rack and 100Mbps across the racks. There are
roughly 78000 independent files spread across these 500 nodes each of size
~60KB that the client pulls (and again we have two such clients per node).
So you can imagine you have a massive all-all communication happening. The
configuration for the server and client is as follows:
 Grizzly configuration for port 9999
         maxThreads: 100
         minThreads: 1
         ByteBuffer size: 8192
         useDirectByteBuffer: false
         useByteBufferView: false
         maxHttpHeaderSize: 8192
         maxKeepAliveRequests: 256
         keepAliveTimeoutInSeconds: 10
         Static File Cache enabled: true
         Stream Algorithm :
com.sun.grizzly.http.algorithms.NoParsingAlgorithm
         Pipeline : com.sun.grizzly.http.LinkedListPipeline
         Round Robin Selector Algorithm enabled: false
         Round Robin Selector pool size: 0
         recycleTasks: true
         Asynchronous Request Processing enabled: false
I also tried some configs with multiple selectorReadThreads but didn't make
much difference.

The client has 30 fetcher threads and the way it is designed is that only
one fetch from any given host would happen at any point of time. So if a
server host, h1, has 'n' files that we should pull, we do that one at a time
(as opposed to multiple threads hitting that server to fetch multiple files
in parallel).

Also, we don't use the features of HTTP1.1 persistent connections or
pipelining. We fetch exactly one file and close the connection to the
server.

With the above setup, the performance I see is not different from what I see
with Jetty5. I see a lot of Read timeouts on the client side (and we have a
backoff (for the server host) on the client implementation whenever we fail
to fetch a file). I also saw some exceptions of the form on the server:

Jun 11, 2007 5:04:51 PM com.sun.grizzly.Controller doSelect
SEVERE: doSelect exception
java.io.IOException: Operation not permitted
        at sun.nio.ch.EPollArrayWrapper.epollCtl(Native Method)
        at
sun.nio.ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.java:202)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:183)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at
com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:277)
        at com.sun.grizzly.Controller.doSelect(Controller.java:218)
        at com.sun.grizzly.Controller.start(Controller.java:451)
        at
com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1158)
        at
com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1121)
        at com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1099)

Are we missing something in the configuration or something else?

Thanks for the help.

Regards,
Devaraj.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Grizzly in Hadoop

Devaraj Das
Forgot to mention one detail .. the file serving happens through the service
method in an Adapter that we implemented. It happens this way since we parse
a few arguments in the http request url to figure out which file to serve,
etc.

-----Original Message-----
From: Devaraj Das [mailto:[hidden email]]
Sent: Tuesday, June 12, 2007 11:57 AM
To: '[hidden email]'
Subject: Grizzly in Hadoop

Hi,

We are considering using Grizzly (1.5) in Hadoop (an open source framework
that has the MapReduce and Distributed File System implementations). The
main reason for using it is to optimize a framework phase called "shuffle".
In this phase we move lots of data across the network.

We are currently using HTTP for moving the data (actually files) and we use
Jetty5. Now we are thinking of moving to Grizzly (to have NIO and all its
niceness). But initial experiments with our benchmark showed that with
Grizzly the performance of the shuffle phase is nearly the same as we have
with Jetty5. This is not what we initially expected and hence would like to
get feedback on where we might be going wrong.

Hadoop is designed to run on large clusters of 100s of nodes (currently it
can run stable/reliably in a 1K node cluster). From the Grizzly point of
view, what needs to be known is that each node has a HTTP server. Both
Jetty5 and Grizzly provides the ability to have multiple handlers to service
the incoming requests.

There are 2 clients on each node, and each client has a configurable number
of fetcher threads. The fetcher code is written using the
java.net.URLConnection API.
Every node has both the server and the clients. The threads basically hit
the HTTP server asking for specific files. They all do this at once (with
some randomness in the order of hosts, maybe).

The benchmark that I tested with is a sort for ~5TB of data with a cluster
of 500 nodes. On the hardware side, I used a cluster that has 4 dualcore
processors in each machine. The machines are partitioned into racks with a
gigabit ethernet within the rack and 100Mbps across the racks. There are
roughly 78000 independent files spread across these 500 nodes each of size
~60KB that the client pulls (and again we have two such clients per node).
So you can imagine you have a massive all-all communication happening. The
configuration for the server and client is as follows:
 Grizzly configuration for port 9999
         maxThreads: 100
         minThreads: 1
         ByteBuffer size: 8192
         useDirectByteBuffer: false
         useByteBufferView: false
         maxHttpHeaderSize: 8192
         maxKeepAliveRequests: 256
         keepAliveTimeoutInSeconds: 10
         Static File Cache enabled: true
         Stream Algorithm :
com.sun.grizzly.http.algorithms.NoParsingAlgorithm
         Pipeline : com.sun.grizzly.http.LinkedListPipeline
         Round Robin Selector Algorithm enabled: false
         Round Robin Selector pool size: 0
         recycleTasks: true
         Asynchronous Request Processing enabled: false I also tried some
configs with multiple selectorReadThreads but didn't make much difference.

The client has 30 fetcher threads and the way it is designed is that only
one fetch from any given host would happen at any point of time. So if a
server host, h1, has 'n' files that we should pull, we do that one at a time
(as opposed to multiple threads hitting that server to fetch multiple files
in parallel).

Also, we don't use the features of HTTP1.1 persistent connections or
pipelining. We fetch exactly one file and close the connection to the
server.

With the above setup, the performance I see is not different from what I see
with Jetty5. I see a lot of Read timeouts on the client side (and we have a
backoff (for the server host) on the client implementation whenever we fail
to fetch a file). I also saw some exceptions of the form on the server:

Jun 11, 2007 5:04:51 PM com.sun.grizzly.Controller doSelect
SEVERE: doSelect exception
java.io.IOException: Operation not permitted
        at sun.nio.ch.EPollArrayWrapper.epollCtl(Native Method)
        at
sun.nio.ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.java:202)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:183)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at
com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:277)
        at com.sun.grizzly.Controller.doSelect(Controller.java:218)
        at com.sun.grizzly.Controller.start(Controller.java:451)
        at
com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1158)
        at
com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1121)
        at com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1099)

Are we missing something in the configuration or something else?

Thanks for the help.

Regards,
Devaraj.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Grizzly in Hadoop

Devaraj Das
In reply to this post by Devaraj Das
I am using jdk1.6.0 and I think I am hitting the bug
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6481709 ("epoll based
Selector throws java.io.IOException: Operation not permitted during load")
and that's why I am seeing the NIO exception as documented in my earlier
mail .. So maybe it can be treated as not being a Grizzly problem but could
it have effect on the network performance? Does this exception mean that we
will fail to fetch from the server that just threw the exception (and
thereby affect the network performance since our client implementation will
backoff)?

-----Original Message-----
From: Devaraj Das [mailto:[hidden email]]
Sent: Tuesday, June 12, 2007 12:02 PM
To: '[hidden email]'
Subject: RE: Grizzly in Hadoop

Forgot to mention one detail .. the file serving happens through the service
method in an Adapter that we implemented. It happens this way since we parse
a few arguments in the http request url to figure out which file to serve,
etc.

-----Original Message-----
From: Devaraj Das [mailto:[hidden email]]
Sent: Tuesday, June 12, 2007 11:57 AM
To: '[hidden email]'
Subject: Grizzly in Hadoop

Hi,

We are considering using Grizzly (1.5) in Hadoop (an open source framework
that has the MapReduce and Distributed File System implementations). The
main reason for using it is to optimize a framework phase called "shuffle".
In this phase we move lots of data across the network.

We are currently using HTTP for moving the data (actually files) and we use
Jetty5. Now we are thinking of moving to Grizzly (to have NIO and all its
niceness). But initial experiments with our benchmark showed that with
Grizzly the performance of the shuffle phase is nearly the same as we have
with Jetty5. This is not what we initially expected and hence would like to
get feedback on where we might be going wrong.

Hadoop is designed to run on large clusters of 100s of nodes (currently it
can run stable/reliably in a 1K node cluster). From the Grizzly point of
view, what needs to be known is that each node has a HTTP server. Both
Jetty5 and Grizzly provides the ability to have multiple handlers to service
the incoming requests.

There are 2 clients on each node, and each client has a configurable number
of fetcher threads. The fetcher code is written using the
java.net.URLConnection API.
Every node has both the server and the clients. The threads basically hit
the HTTP server asking for specific files. They all do this at once (with
some randomness in the order of hosts, maybe).

The benchmark that I tested with is a sort for ~5TB of data with a cluster
of 500 nodes. On the hardware side, I used a cluster that has 4 dualcore
processors in each machine. The machines are partitioned into racks with a
gigabit ethernet within the rack and 100Mbps across the racks. There are
roughly 78000 independent files spread across these 500 nodes each of size
~60KB that the client pulls (and again we have two such clients per node).
So you can imagine you have a massive all-all communication happening. The
configuration for the server and client is as follows:
 Grizzly configuration for port 9999
         maxThreads: 100
         minThreads: 1
         ByteBuffer size: 8192
         useDirectByteBuffer: false
         useByteBufferView: false
         maxHttpHeaderSize: 8192
         maxKeepAliveRequests: 256
         keepAliveTimeoutInSeconds: 10
         Static File Cache enabled: true
         Stream Algorithm :
com.sun.grizzly.http.algorithms.NoParsingAlgorithm
         Pipeline : com.sun.grizzly.http.LinkedListPipeline
         Round Robin Selector Algorithm enabled: false
         Round Robin Selector pool size: 0
         recycleTasks: true
         Asynchronous Request Processing enabled: false I also tried some
configs with multiple selectorReadThreads but didn't make much difference.

The client has 30 fetcher threads and the way it is designed is that only
one fetch from any given host would happen at any point of time. So if a
server host, h1, has 'n' files that we should pull, we do that one at a time
(as opposed to multiple threads hitting that server to fetch multiple files
in parallel).

Also, we don't use the features of HTTP1.1 persistent connections or
pipelining. We fetch exactly one file and close the connection to the
server.

With the above setup, the performance I see is not different from what I see
with Jetty5. I see a lot of Read timeouts on the client side (and we have a
backoff (for the server host) on the client implementation whenever we fail
to fetch a file). I also saw some exceptions of the form on the server:

Jun 11, 2007 5:04:51 PM com.sun.grizzly.Controller doSelect
SEVERE: doSelect exception
java.io.IOException: Operation not permitted
        at sun.nio.ch.EPollArrayWrapper.epollCtl(Native Method)
        at
sun.nio.ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.java:202)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:183)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at
com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:277)
        at com.sun.grizzly.Controller.doSelect(Controller.java:218)
        at com.sun.grizzly.Controller.start(Controller.java:451)
        at
com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1158)
        at
com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1121)
        at com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1099)

Are we missing something in the configuration or something else?

Thanks for the help.

Regards,
Devaraj.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Grizzly in Hadoop

Jeanfrancois Arcand-2
In reply to this post by Devaraj Das
Hi,

Devaraj Das wrote:
> Hi,
>
> We are considering using Grizzly (1.5) in Hadoop (an open source framework
> that has the MapReduce and Distributed File System implementations). The
> main reason for using it is to optimize a framework phase called "shuffle".
> In this phase we move lots of data across the network.

Cool :-) I know Hadoop as your Web 2.0 stack is also considering using
it :-)

>
> We are currently using HTTP for moving the data (actually files) and we use
> Jetty5. Now we are thinking of moving to Grizzly (to have NIO and all its
> niceness). But initial experiments with our benchmark showed that with
> Grizzly the performance of the shuffle phase is nearly the same as we have
> with Jetty5. This is not what we initially expected and hence would like to
> get feedback on where we might be going wrong.
>
> Hadoop is designed to run on large clusters of 100s of nodes (currently it
> can run stable/reliably in a 1K node cluster). From the Grizzly point of
> view, what needs to be known is that each node has a HTTP server. Both
> Jetty5 and Grizzly provides the ability to have multiple handlers to service
> the incoming requests.
>
> There are 2 clients on each node, and each client has a configurable number
> of fetcher threads. The fetcher code is written using the
> java.net.URLConnection API.
> Every node has both the server and the clients. The threads basically hit
> the HTTP server asking for specific files. They all do this at once (with
> some randomness in the order of hosts, maybe).
>
> The benchmark that I tested with is a sort for ~5TB of data with a cluster
> of 500 nodes. On the hardware side, I used a cluster that has 4 dualcore
> processors in each machine. The machines are partitioned into racks with a
> gigabit ethernet within the rack and 100Mbps across the racks. There are
> roughly 78000 independent files spread across these 500 nodes each of size
> ~60KB that the client pulls (and again we have two such clients per node).
> So you can imagine you have a massive all-all communication happening. The
> configuration for the server and client is as follows:
>  Grizzly configuration for port 9999
>          maxThreads: 100
>          minThreads: 1
>          ByteBuffer size: 8192
>          useDirectByteBuffer: false
>          useByteBufferView: false
>          maxHttpHeaderSize: 8192
>          maxKeepAliveRequests: 256
>          keepAliveTimeoutInSeconds: 10
>          Static File Cache enabled: true
>          Stream Algorithm :
> com.sun.grizzly.http.algorithms.NoParsingAlgorithm
>          Pipeline : com.sun.grizzly.http.LinkedListPipeline
>          Round Robin Selector Algorithm enabled: false
>          Round Robin Selector pool size: 0
>          recycleTasks: true
>          Asynchronous Request Processing enabled: false
> I also tried some configs with multiple selectorReadThreads but didn't make
> much difference.

keepAliveTimeoutInSeconds: 10 seems for me a little low...what is the
reason to have such a low number? Closing faster idle connection?


>
> The client has 30 fetcher threads and the way it is designed is that only
> one fetch from any given host would happen at any point of time. So if a
> server host, h1, has 'n' files that we should pull, we do that one at a time
> (as opposed to multiple threads hitting that server to fetch multiple files
> in parallel).
>
> Also, we don't use the features of HTTP1.1 persistent connections or
> pipelining. We fetch exactly one file and close the connection to the
> server.

That's answer my previous question :-) I would recommend setting the
keepAliveTimeoutInSeconds=0 then as I'm sure the performance will
improve (no call to the keep-alive subsystem).

>
> With the above setup, the performance I see is not different from what I see
> with Jetty5.

Can it be the benchmark itself that it not able to load more?


I see a lot of Read timeouts on the client side (and we have a
> backoff (for the server host) on the client implementation whenever we fail
> to fetch a file). I also saw some exceptions of the form on the server:

You seem to hit the epoll problem on Linux. I know there is a way to
avoid using epoll (a property). I will ping the NIO team and let you know.

Also, what exactly your Adapter implementation is doing? Can you share
the code? If I can have access to your setup, I would like to see if
using Grizzly 1.0.15 makes a difference (just to make sure we don't have
a bug in 1.5....as far as I can tell, I.5 is as fast as 1.0 on my
benchmark).

Thanks,

--Jeanfrancois


>
> Jun 11, 2007 5:04:51 PM com.sun.grizzly.Controller doSelect
> SEVERE: doSelect exception
> java.io.IOException: Operation not permitted
>         at sun.nio.ch.EPollArrayWrapper.epollCtl(Native Method)
>         at
> sun.nio.ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.java:202)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:183)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>         at
> com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:277)
>         at com.sun.grizzly.Controller.doSelect(Controller.java:218)
>         at com.sun.grizzly.Controller.start(Controller.java:451)
>         at
> com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1158)
>         at
> com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1121)
>         at com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1099)
>
> Are we missing something in the configuration or something else?
>
> Thanks for the help.
>
> Regards,
> Devaraj.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Grizzly in Hadoop

Jeanfrancois Arcand-2
In reply to this post by Devaraj Das


Devaraj Das wrote:
> I am using jdk1.6.0 and I think I am hitting the bug
> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6481709 ("epoll based
> Selector throws java.io.IOException: Operation not permitted during load")

Yes you are hitting that bug. This has been fixed in ur1 (confirmed by
the NIO lead just now :-))

> and that's why I am seeing the NIO exception as documented in my earlier
> mail .. So maybe it can be treated as not being a Grizzly problem but could
> it have effect on the network performance?

For sure it has :-)

Does this exception mean that we
> will fail to fetch from the server that just threw the exception (and
> thereby affect the network performance since our client implementation will
> backoff)?

Some connection will be closed. Can you try with 6.0 ur1

Thanks!

-- Jeanfrancois


>
> -----Original Message-----
> From: Devaraj Das [mailto:[hidden email]]
> Sent: Tuesday, June 12, 2007 12:02 PM
> To: '[hidden email]'
> Subject: RE: Grizzly in Hadoop
>
> Forgot to mention one detail .. the file serving happens through the service
> method in an Adapter that we implemented. It happens this way since we parse
> a few arguments in the http request url to figure out which file to serve,
> etc.
>
> -----Original Message-----
> From: Devaraj Das [mailto:[hidden email]]
> Sent: Tuesday, June 12, 2007 11:57 AM
> To: '[hidden email]'
> Subject: Grizzly in Hadoop
>
> Hi,
>
> We are considering using Grizzly (1.5) in Hadoop (an open source framework
> that has the MapReduce and Distributed File System implementations). The
> main reason for using it is to optimize a framework phase called "shuffle".
> In this phase we move lots of data across the network.
>
> We are currently using HTTP for moving the data (actually files) and we use
> Jetty5. Now we are thinking of moving to Grizzly (to have NIO and all its
> niceness). But initial experiments with our benchmark showed that with
> Grizzly the performance of the shuffle phase is nearly the same as we have
> with Jetty5. This is not what we initially expected and hence would like to
> get feedback on where we might be going wrong.
>
> Hadoop is designed to run on large clusters of 100s of nodes (currently it
> can run stable/reliably in a 1K node cluster). From the Grizzly point of
> view, what needs to be known is that each node has a HTTP server. Both
> Jetty5 and Grizzly provides the ability to have multiple handlers to service
> the incoming requests.
>
> There are 2 clients on each node, and each client has a configurable number
> of fetcher threads. The fetcher code is written using the
> java.net.URLConnection API.
> Every node has both the server and the clients. The threads basically hit
> the HTTP server asking for specific files. They all do this at once (with
> some randomness in the order of hosts, maybe).
>
> The benchmark that I tested with is a sort for ~5TB of data with a cluster
> of 500 nodes. On the hardware side, I used a cluster that has 4 dualcore
> processors in each machine. The machines are partitioned into racks with a
> gigabit ethernet within the rack and 100Mbps across the racks. There are
> roughly 78000 independent files spread across these 500 nodes each of size
> ~60KB that the client pulls (and again we have two such clients per node).
> So you can imagine you have a massive all-all communication happening. The
> configuration for the server and client is as follows:
>  Grizzly configuration for port 9999
>          maxThreads: 100
>          minThreads: 1
>          ByteBuffer size: 8192
>          useDirectByteBuffer: false
>          useByteBufferView: false
>          maxHttpHeaderSize: 8192
>          maxKeepAliveRequests: 256
>          keepAliveTimeoutInSeconds: 10
>          Static File Cache enabled: true
>          Stream Algorithm :
> com.sun.grizzly.http.algorithms.NoParsingAlgorithm
>          Pipeline : com.sun.grizzly.http.LinkedListPipeline
>          Round Robin Selector Algorithm enabled: false
>          Round Robin Selector pool size: 0
>          recycleTasks: true
>          Asynchronous Request Processing enabled: false I also tried some
> configs with multiple selectorReadThreads but didn't make much difference.
>
> The client has 30 fetcher threads and the way it is designed is that only
> one fetch from any given host would happen at any point of time. So if a
> server host, h1, has 'n' files that we should pull, we do that one at a time
> (as opposed to multiple threads hitting that server to fetch multiple files
> in parallel).
>
> Also, we don't use the features of HTTP1.1 persistent connections or
> pipelining. We fetch exactly one file and close the connection to the
> server.
>
> With the above setup, the performance I see is not different from what I see
> with Jetty5. I see a lot of Read timeouts on the client side (and we have a
> backoff (for the server host) on the client implementation whenever we fail
> to fetch a file). I also saw some exceptions of the form on the server:
>
> Jun 11, 2007 5:04:51 PM com.sun.grizzly.Controller doSelect
> SEVERE: doSelect exception
> java.io.IOException: Operation not permitted
>         at sun.nio.ch.EPollArrayWrapper.epollCtl(Native Method)
>         at
> sun.nio.ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.java:202)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:183)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>         at
> com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:277)
>         at com.sun.grizzly.Controller.doSelect(Controller.java:218)
>         at com.sun.grizzly.Controller.start(Controller.java:451)
>         at
> com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1158)
>         at
> com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1121)
>         at com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1099)
>
> Are we missing something in the configuration or something else?
>
> Thanks for the help.
>
> Regards,
> Devaraj.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Grizzly in Hadoop

Devaraj Das
Thanks for the confirmation! After moving to JDK-1.6u1, the problem seems to
have disappeared.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]]
Sent: Tuesday, June 12, 2007 6:27 PM
To: [hidden email]
Subject: Re: Grizzly in Hadoop



Devaraj Das wrote:
> I am using jdk1.6.0 and I think I am hitting the bug
> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6481709 ("epoll
> based Selector throws java.io.IOException: Operation not permitted
> during load")

Yes you are hitting that bug. This has been fixed in ur1 (confirmed by the
NIO lead just now :-))

> and that's why I am seeing the NIO exception as documented in my
> earlier mail .. So maybe it can be treated as not being a Grizzly
> problem but could it have effect on the network performance?

For sure it has :-)

Does this exception mean that we
> will fail to fetch from the server that just threw the exception (and
> thereby affect the network performance since our client implementation
> will backoff)?

Some connection will be closed. Can you try with 6.0 ur1

Thanks!

-- Jeanfrancois


>
> -----Original Message-----
> From: Devaraj Das [mailto:[hidden email]]
> Sent: Tuesday, June 12, 2007 12:02 PM
> To: '[hidden email]'
> Subject: RE: Grizzly in Hadoop
>
> Forgot to mention one detail .. the file serving happens through the
> service method in an Adapter that we implemented. It happens this way
> since we parse a few arguments in the http request url to figure out
> which file to serve, etc.
>
> -----Original Message-----
> From: Devaraj Das [mailto:[hidden email]]
> Sent: Tuesday, June 12, 2007 11:57 AM
> To: '[hidden email]'
> Subject: Grizzly in Hadoop
>
> Hi,
>
> We are considering using Grizzly (1.5) in Hadoop (an open source
> framework that has the MapReduce and Distributed File System
> implementations). The main reason for using it is to optimize a framework
phase called "shuffle".
> In this phase we move lots of data across the network.
>
> We are currently using HTTP for moving the data (actually files) and
> we use Jetty5. Now we are thinking of moving to Grizzly (to have NIO
> and all its niceness). But initial experiments with our benchmark
> showed that with Grizzly the performance of the shuffle phase is
> nearly the same as we have with Jetty5. This is not what we initially
> expected and hence would like to get feedback on where we might be going
wrong.

>
> Hadoop is designed to run on large clusters of 100s of nodes
> (currently it can run stable/reliably in a 1K node cluster). From the
> Grizzly point of view, what needs to be known is that each node has a
> HTTP server. Both
> Jetty5 and Grizzly provides the ability to have multiple handlers to
> service the incoming requests.
>
> There are 2 clients on each node, and each client has a configurable
> number of fetcher threads. The fetcher code is written using the
> java.net.URLConnection API.
> Every node has both the server and the clients. The threads basically
> hit the HTTP server asking for specific files. They all do this at
> once (with some randomness in the order of hosts, maybe).
>
> The benchmark that I tested with is a sort for ~5TB of data with a
> cluster of 500 nodes. On the hardware side, I used a cluster that has
> 4 dualcore processors in each machine. The machines are partitioned
> into racks with a gigabit ethernet within the rack and 100Mbps across
> the racks. There are roughly 78000 independent files spread across
> these 500 nodes each of size ~60KB that the client pulls (and again we
have two such clients per node).

> So you can imagine you have a massive all-all communication happening.
> The configuration for the server and client is as follows:
>  Grizzly configuration for port 9999
>          maxThreads: 100
>          minThreads: 1
>          ByteBuffer size: 8192
>          useDirectByteBuffer: false
>          useByteBufferView: false
>          maxHttpHeaderSize: 8192
>          maxKeepAliveRequests: 256
>          keepAliveTimeoutInSeconds: 10
>          Static File Cache enabled: true
>          Stream Algorithm :
> com.sun.grizzly.http.algorithms.NoParsingAlgorithm
>          Pipeline : com.sun.grizzly.http.LinkedListPipeline
>          Round Robin Selector Algorithm enabled: false
>          Round Robin Selector pool size: 0
>          recycleTasks: true
>          Asynchronous Request Processing enabled: false I also tried
> some configs with multiple selectorReadThreads but didn't make much
difference.

>
> The client has 30 fetcher threads and the way it is designed is that
> only one fetch from any given host would happen at any point of time.
> So if a server host, h1, has 'n' files that we should pull, we do that
> one at a time (as opposed to multiple threads hitting that server to
> fetch multiple files in parallel).
>
> Also, we don't use the features of HTTP1.1 persistent connections or
> pipelining. We fetch exactly one file and close the connection to the
> server.
>
> With the above setup, the performance I see is not different from what
> I see with Jetty5. I see a lot of Read timeouts on the client side
> (and we have a backoff (for the server host) on the client
> implementation whenever we fail to fetch a file). I also saw some
exceptions of the form on the server:
>
> Jun 11, 2007 5:04:51 PM com.sun.grizzly.Controller doSelect
> SEVERE: doSelect exception
> java.io.IOException: Operation not permitted
>         at sun.nio.ch.EPollArrayWrapper.epollCtl(Native Method)
>         at
>
sun.nio.ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.java:202)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:183)
>         at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>         at
> com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:277)
>         at com.sun.grizzly.Controller.doSelect(Controller.java:218)
>         at com.sun.grizzly.Controller.start(Controller.java:451)
>         at
>
com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1158)
>         at
>
com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1121)

>         at
> com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1099)
>
> Are we missing something in the configuration or something else?
>
> Thanks for the help.
>
> Regards,
> Devaraj.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Grizzly in Hadoop

Devaraj Das
In reply to this post by Jeanfrancois Arcand-2
Hi Jeanfrancois,
I will get back to you in detail in some time from now. But for now, I just
want to update you that setting the keepalivetimeout to 0 seemed to have
improved performance. Thanks for that tip. Now, do you think, if we start
using the persistent connections (transfer a batch of files at a time)
feature, it would help us significantly?
Also, will using multiple selectorReadThreads help the performance (since we
have multiple CPUs on our machines)?
I will send the source code of the adpater shortly...
Thanks,
Devaraj.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]]
Sent: Tuesday, June 12, 2007 6:25 PM
To: [hidden email]; Scott Oaks
Subject: Re: Grizzly in Hadoop

Hi,

Devaraj Das wrote:
> Hi,
>
> We are considering using Grizzly (1.5) in Hadoop (an open source
> framework that has the MapReduce and Distributed File System
> implementations). The main reason for using it is to optimize a framework
phase called "shuffle".
> In this phase we move lots of data across the network.

Cool :-) I know Hadoop as your Web 2.0 stack is also considering using it
:-)

>
> We are currently using HTTP for moving the data (actually files) and
> we use Jetty5. Now we are thinking of moving to Grizzly (to have NIO
> and all its niceness). But initial experiments with our benchmark
> showed that with Grizzly the performance of the shuffle phase is
> nearly the same as we have with Jetty5. This is not what we initially
> expected and hence would like to get feedback on where we might be going
wrong.

>
> Hadoop is designed to run on large clusters of 100s of nodes
> (currently it can run stable/reliably in a 1K node cluster). From the
> Grizzly point of view, what needs to be known is that each node has a
> HTTP server. Both
> Jetty5 and Grizzly provides the ability to have multiple handlers to
> service the incoming requests.
>
> There are 2 clients on each node, and each client has a configurable
> number of fetcher threads. The fetcher code is written using the
> java.net.URLConnection API.
> Every node has both the server and the clients. The threads basically
> hit the HTTP server asking for specific files. They all do this at
> once (with some randomness in the order of hosts, maybe).
>
> The benchmark that I tested with is a sort for ~5TB of data with a
> cluster of 500 nodes. On the hardware side, I used a cluster that has
> 4 dualcore processors in each machine. The machines are partitioned
> into racks with a gigabit ethernet within the rack and 100Mbps across
> the racks. There are roughly 78000 independent files spread across
> these 500 nodes each of size ~60KB that the client pulls (and again we
have two such clients per node).

> So you can imagine you have a massive all-all communication happening.
> The configuration for the server and client is as follows:
>  Grizzly configuration for port 9999
>          maxThreads: 100
>          minThreads: 1
>          ByteBuffer size: 8192
>          useDirectByteBuffer: false
>          useByteBufferView: false
>          maxHttpHeaderSize: 8192
>          maxKeepAliveRequests: 256
>          keepAliveTimeoutInSeconds: 10
>          Static File Cache enabled: true
>          Stream Algorithm :
> com.sun.grizzly.http.algorithms.NoParsingAlgorithm
>          Pipeline : com.sun.grizzly.http.LinkedListPipeline
>          Round Robin Selector Algorithm enabled: false
>          Round Robin Selector pool size: 0
>          recycleTasks: true
>          Asynchronous Request Processing enabled: false I also tried
> some configs with multiple selectorReadThreads but didn't make much
> difference.

keepAliveTimeoutInSeconds: 10 seems for me a little low...what is the reason
to have such a low number? Closing faster idle connection?


>
> The client has 30 fetcher threads and the way it is designed is that only
> one fetch from any given host would happen at any point of time. So if a
> server host, h1, has 'n' files that we should pull, we do that one at a
time
> (as opposed to multiple threads hitting that server to fetch multiple
files
> in parallel).
>
> Also, we don't use the features of HTTP1.1 persistent connections or
> pipelining. We fetch exactly one file and close the connection to the
> server.

That's answer my previous question :-) I would recommend setting the
keepAliveTimeoutInSeconds=0 then as I'm sure the performance will
improve (no call to the keep-alive subsystem).

>
> With the above setup, the performance I see is not different from what I
see
> with Jetty5.

Can it be the benchmark itself that it not able to load more?


I see a lot of Read timeouts on the client side (and we have a
> backoff (for the server host) on the client implementation whenever we
fail
> to fetch a file). I also saw some exceptions of the form on the server:

You seem to hit the epoll problem on Linux. I know there is a way to
avoid using epoll (a property). I will ping the NIO team and let you know.

Also, what exactly your Adapter implementation is doing? Can you share
the code? If I can have access to your setup, I would like to see if
using Grizzly 1.0.15 makes a difference (just to make sure we don't have
a bug in 1.5....as far as I can tell, I.5 is as fast as 1.0 on my
benchmark).

Thanks,

--Jeanfrancois


>
> Jun 11, 2007 5:04:51 PM com.sun.grizzly.Controller doSelect
> SEVERE: doSelect exception
> java.io.IOException: Operation not permitted
>         at sun.nio.ch.EPollArrayWrapper.epollCtl(Native Method)
>         at
>
sun.nio.ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.java:202)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:183)
>         at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>         at
> com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:277)
>         at com.sun.grizzly.Controller.doSelect(Controller.java:218)
>         at com.sun.grizzly.Controller.start(Controller.java:451)
>         at
>
com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1158)
>         at
>
com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1121)
>         at
com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1099)

>
> Are we missing something in the configuration or something else?
>
> Thanks for the help.
>
> Regards,
> Devaraj.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Grizzly in Hadoop

Devaraj Das

Here is the core of the "service(Request, Response)" adapter method

        response.setHeader(MAP_OUTPUT_LENGTH, Long.toString(<file_length>));

        int len = mapOutputIn.read(<from_a_file>);
        while (len > 0) {
          try {
            chunk.recycle();
            chunk.append(buffer, 0, len);
            response.doWrite(chunk);
          } catch (IOException ie) {
            isInputException = false;
            throw ie;
          }
          totalRead += len;
          if (totalRead == partLength) break;
          len = mapOutputIn.read(<from_a_file>);
        }

By the way, we don't set content-length and expect the server to do chunked
encoding (in some cases we could be serving very huge files in the order of
4-5 Gigs), and content-length header expects things to be 'int' (at least in
jetty). So we work around this problem by defining our own custom http
header that is a "long".

Here is the core of the client code

      URLConnection connection = path.openConnection();
      if (timeout > 0) {
        connection.setConnectTimeout(timeout);
        connection.setReadTimeout(timeout);
      }
      InputStream input = connection.getInputStream();
      //get the content length from a custom http header
      long length =
Long.parseLong(connection.getHeaderField(MAP_OUTPUT_LENGTH));

      OutputStream output = fileSys.create(localFilename);

          int len = input.read(buffer);
          while (len > 0) {
            totalBytes += len;
            output.write(buffer, 0 , len);
            if (currentThread.isInterrupted()) {
              throw new InterruptedException();
            }
            if (totalBytes == length) break;
            len = input.read(buffer);
          }
        } finally {
          output.close();
        }
      } finally {
        input.close();
      }

I am seeing exceptions of the form on the server (on quite a few nodes).
This seems to be contributing to the performance degradation now. Any clue
on this? Thanks much for the help.

2007-06-12 14:25:28,968 WARN org.apache.hadoop.mapred.TaskTracker:
getMapOutput(task_0001_m_077205_1,971) failed :
java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcher.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
        at sun.nio.ch.IOUtil.write(IOUtil.java:75)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
        at
com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:88)
        at
com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:57)
        at
com.sun.grizzly.http.SocketChannelOutputBuffer.flushChannel(SocketChannelOut
putBuffer.java:138)
        at
com.sun.grizzly.http.SocketChannelOutputBuffer.realWriteBytes(SocketChannelO
utputBuffer.java:125)
        at com.sun.grizzly.util.buf.ByteChunk.append(ByteChunk.java:331)
        at
com.sun.grizzly.tcp.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doW
rite(InternalOutputBuffer.java:856)
        at
com.sun.grizzly.tcp.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutput
Filter.java:136)
        at
com.sun.grizzly.tcp.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer
.java:614)
        at com.sun.grizzly.tcp.Response.doWrite(Response.java:587)
        at
org.apache.hadoop.mapred.TaskTracker$MapOutput.service(TaskTracker.java:1944
)
        at
com.sun.grizzly.http.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask
.java:597)
        at
com.sun.grizzly.http.DefaultProcessorTask.doProcess(DefaultProcessorTask.jav
a:528)
        at
com.sun.grizzly.http.DefaultProcessorTask.process(DefaultProcessorTask.java:
772)
        at
com.sun.grizzly.http.SelectorThread$3.execute(SelectorThread.java:745)
        at
com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolCh
ain.java:74)

-----Original Message-----
From: Devaraj Das [mailto:[hidden email]]
Sent: Tuesday, June 12, 2007 7:45 PM
To: [hidden email]; 'Scott Oaks'
Cc: 'Owen O'Malley'; 'Sameer Paranjpye'
Subject: RE: Grizzly in Hadoop

Hi Jeanfrancois,
I will get back to you in detail in some time from now. But for now, I just
want to update you that setting the keepalivetimeout to 0 seemed to have
improved performance. Thanks for that tip. Now, do you think, if we start
using the persistent connections (transfer a batch of files at a time)
feature, it would help us significantly?
Also, will using multiple selectorReadThreads help the performance (since we
have multiple CPUs on our machines)?
I will send the source code of the adpater shortly...
Thanks,
Devaraj.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]]
Sent: Tuesday, June 12, 2007 6:25 PM
To: [hidden email]; Scott Oaks
Subject: Re: Grizzly in Hadoop

Hi,

Devaraj Das wrote:
> Hi,
>
> We are considering using Grizzly (1.5) in Hadoop (an open source
> framework that has the MapReduce and Distributed File System
> implementations). The main reason for using it is to optimize a
> framework
phase called "shuffle".
> In this phase we move lots of data across the network.

Cool :-) I know Hadoop as your Web 2.0 stack is also considering using it
:-)

>
> We are currently using HTTP for moving the data (actually files) and
> we use Jetty5. Now we are thinking of moving to Grizzly (to have NIO
> and all its niceness). But initial experiments with our benchmark
> showed that with Grizzly the performance of the shuffle phase is
> nearly the same as we have with Jetty5. This is not what we initially
> expected and hence would like to get feedback on where we might be
> going
wrong.

>
> Hadoop is designed to run on large clusters of 100s of nodes
> (currently it can run stable/reliably in a 1K node cluster). From the
> Grizzly point of view, what needs to be known is that each node has a
> HTTP server. Both
> Jetty5 and Grizzly provides the ability to have multiple handlers to
> service the incoming requests.
>
> There are 2 clients on each node, and each client has a configurable
> number of fetcher threads. The fetcher code is written using the
> java.net.URLConnection API.
> Every node has both the server and the clients. The threads basically
> hit the HTTP server asking for specific files. They all do this at
> once (with some randomness in the order of hosts, maybe).
>
> The benchmark that I tested with is a sort for ~5TB of data with a
> cluster of 500 nodes. On the hardware side, I used a cluster that has
> 4 dualcore processors in each machine. The machines are partitioned
> into racks with a gigabit ethernet within the rack and 100Mbps across
> the racks. There are roughly 78000 independent files spread across
> these 500 nodes each of size ~60KB that the client pulls (and again we
have two such clients per node).

> So you can imagine you have a massive all-all communication happening.
> The configuration for the server and client is as follows:
>  Grizzly configuration for port 9999
>          maxThreads: 100
>          minThreads: 1
>          ByteBuffer size: 8192
>          useDirectByteBuffer: false
>          useByteBufferView: false
>          maxHttpHeaderSize: 8192
>          maxKeepAliveRequests: 256
>          keepAliveTimeoutInSeconds: 10
>          Static File Cache enabled: true
>          Stream Algorithm :
> com.sun.grizzly.http.algorithms.NoParsingAlgorithm
>          Pipeline : com.sun.grizzly.http.LinkedListPipeline
>          Round Robin Selector Algorithm enabled: false
>          Round Robin Selector pool size: 0
>          recycleTasks: true
>          Asynchronous Request Processing enabled: false I also tried
> some configs with multiple selectorReadThreads but didn't make much
> difference.

keepAliveTimeoutInSeconds: 10 seems for me a little low...what is the reason
to have such a low number? Closing faster idle connection?


>
> The client has 30 fetcher threads and the way it is designed is that
> only one fetch from any given host would happen at any point of time.
> So if a server host, h1, has 'n' files that we should pull, we do that
> one at a
time
> (as opposed to multiple threads hitting that server to fetch multiple
files
> in parallel).
>
> Also, we don't use the features of HTTP1.1 persistent connections or
> pipelining. We fetch exactly one file and close the connection to the
> server.

That's answer my previous question :-) I would recommend setting the
keepAliveTimeoutInSeconds=0 then as I'm sure the performance will improve
(no call to the keep-alive subsystem).

>
> With the above setup, the performance I see is not different from what
> I
see
> with Jetty5.

Can it be the benchmark itself that it not able to load more?


I see a lot of Read timeouts on the client side (and we have a
> backoff (for the server host) on the client implementation whenever we
fail
> to fetch a file). I also saw some exceptions of the form on the server:

You seem to hit the epoll problem on Linux. I know there is a way to avoid
using epoll (a property). I will ping the NIO team and let you know.

Also, what exactly your Adapter implementation is doing? Can you share the
code? If I can have access to your setup, I would like to see if using
Grizzly 1.0.15 makes a difference (just to make sure we don't have a bug in
1.5....as far as I can tell, I.5 is as fast as 1.0 on my benchmark).

Thanks,

--Jeanfrancois


>
> Jun 11, 2007 5:04:51 PM com.sun.grizzly.Controller doSelect
> SEVERE: doSelect exception
> java.io.IOException: Operation not permitted
>         at sun.nio.ch.EPollArrayWrapper.epollCtl(Native Method)
>         at
>
sun.nio.ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.java:202)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:183)
>         at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>         at
> com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:277)
>         at com.sun.grizzly.Controller.doSelect(Controller.java:218)
>         at com.sun.grizzly.Controller.start(Controller.java:451)
>         at
>
com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1158)
>         at
>
com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1121)
>         at
com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1099)

>
> Are we missing something in the configuration or something else?
>
> Thanks for the help.
>
> Regards,
> Devaraj.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Grizzly in Hadoop

Jeanfrancois Arcand-2
In reply to this post by Devaraj Das
Hi Devaraj,

Devaraj Das wrote:
> Hi Jeanfrancois,
> I will get back to you in detail in some time from now. But for now, I just
> want to update you that setting the keepalivetimeout to 0 seemed to have
> improved performance. Thanks for that tip. Now, do you think, if we start
> using the persistent connections (transfer a batch of files at a time)
> feature, it would help us significantly?

Yes it would as you will save the network overhead of opening a connection.

> Also, will using multiple selectorReadThreads help the performance (since we
> have multiple CPUs on our machines)?

Right now it is not supported with the http module (Just wakes up I
didn't port it yet). Hence that's why you aren't seeing any performance
difference.

Taking about your benchmark, It would be interesting to understand how
you configure Jetty. Your connections seem to be data intensive as
opposed to connection intensive. NIO is good at handling large amount of
connections without spending idle threads but if the threads are not
idle anyway (as in this case), the place to make this faster is only to
reduce the number of bcopy/memcopy calls. I'm just thinking loud here :-)


> I will send the source code of the adpater shortly...
> Thanks,

Thanks!

-- Jeanfrancois

> Devaraj.
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]]
> Sent: Tuesday, June 12, 2007 6:25 PM
> To: [hidden email]; Scott Oaks
> Subject: Re: Grizzly in Hadoop
>
> Hi,
>
> Devaraj Das wrote:
>> Hi,
>>
>> We are considering using Grizzly (1.5) in Hadoop (an open source
>> framework that has the MapReduce and Distributed File System
>> implementations). The main reason for using it is to optimize a framework
> phase called "shuffle".
>> In this phase we move lots of data across the network.
>
> Cool :-) I know Hadoop as your Web 2.0 stack is also considering using it
> :-)
>
>> We are currently using HTTP for moving the data (actually files) and
>> we use Jetty5. Now we are thinking of moving to Grizzly (to have NIO
>> and all its niceness). But initial experiments with our benchmark
>> showed that with Grizzly the performance of the shuffle phase is
>> nearly the same as we have with Jetty5. This is not what we initially
>> expected and hence would like to get feedback on where we might be going
> wrong.
>> Hadoop is designed to run on large clusters of 100s of nodes
>> (currently it can run stable/reliably in a 1K node cluster). From the
>> Grizzly point of view, what needs to be known is that each node has a
>> HTTP server. Both
>> Jetty5 and Grizzly provides the ability to have multiple handlers to
>> service the incoming requests.
>>
>> There are 2 clients on each node, and each client has a configurable
>> number of fetcher threads. The fetcher code is written using the
>> java.net.URLConnection API.
>> Every node has both the server and the clients. The threads basically
>> hit the HTTP server asking for specific files. They all do this at
>> once (with some randomness in the order of hosts, maybe).
>>
>> The benchmark that I tested with is a sort for ~5TB of data with a
>> cluster of 500 nodes. On the hardware side, I used a cluster that has
>> 4 dualcore processors in each machine. The machines are partitioned
>> into racks with a gigabit ethernet within the rack and 100Mbps across
>> the racks. There are roughly 78000 independent files spread across
>> these 500 nodes each of size ~60KB that the client pulls (and again we
> have two such clients per node).
>> So you can imagine you have a massive all-all communication happening.
>> The configuration for the server and client is as follows:
>>  Grizzly configuration for port 9999
>>          maxThreads: 100
>>          minThreads: 1
>>          ByteBuffer size: 8192
>>          useDirectByteBuffer: false
>>          useByteBufferView: false
>>          maxHttpHeaderSize: 8192
>>          maxKeepAliveRequests: 256
>>          keepAliveTimeoutInSeconds: 10
>>          Static File Cache enabled: true
>>          Stream Algorithm :
>> com.sun.grizzly.http.algorithms.NoParsingAlgorithm
>>          Pipeline : com.sun.grizzly.http.LinkedListPipeline
>>          Round Robin Selector Algorithm enabled: false
>>          Round Robin Selector pool size: 0
>>          recycleTasks: true
>>          Asynchronous Request Processing enabled: false I also tried
>> some configs with multiple selectorReadThreads but didn't make much
>> difference.
>
> keepAliveTimeoutInSeconds: 10 seems for me a little low...what is the reason
> to have such a low number? Closing faster idle connection?
>
>
>> The client has 30 fetcher threads and the way it is designed is that only
>> one fetch from any given host would happen at any point of time. So if a
>> server host, h1, has 'n' files that we should pull, we do that one at a
> time
>> (as opposed to multiple threads hitting that server to fetch multiple
> files
>> in parallel).
>>
>> Also, we don't use the features of HTTP1.1 persistent connections or
>> pipelining. We fetch exactly one file and close the connection to the
>> server.
>
> That's answer my previous question :-) I would recommend setting the
> keepAliveTimeoutInSeconds=0 then as I'm sure the performance will
> improve (no call to the keep-alive subsystem).
>
>> With the above setup, the performance I see is not different from what I
> see
>> with Jetty5.
>
> Can it be the benchmark itself that it not able to load more?
>
>
> I see a lot of Read timeouts on the client side (and we have a
>> backoff (for the server host) on the client implementation whenever we
> fail
>> to fetch a file). I also saw some exceptions of the form on the server:
>
> You seem to hit the epoll problem on Linux. I know there is a way to
> avoid using epoll (a property). I will ping the NIO team and let you know.
>
> Also, what exactly your Adapter implementation is doing? Can you share
> the code? If I can have access to your setup, I would like to see if
> using Grizzly 1.0.15 makes a difference (just to make sure we don't have
> a bug in 1.5....as far as I can tell, I.5 is as fast as 1.0 on my
> benchmark).
>
> Thanks,
>
> --Jeanfrancois
>
>
>> Jun 11, 2007 5:04:51 PM com.sun.grizzly.Controller doSelect
>> SEVERE: doSelect exception
>> java.io.IOException: Operation not permitted
>>         at sun.nio.ch.EPollArrayWrapper.epollCtl(Native Method)
>>         at
>>
> sun.nio.ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.java:202)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:183)
>>         at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>         at
>> com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:277)
>>         at com.sun.grizzly.Controller.doSelect(Controller.java:218)
>>         at com.sun.grizzly.Controller.start(Controller.java:451)
>>         at
>>
> com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1158)
>>         at
>>
> com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1121)
>>         at
> com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1099)
>> Are we missing something in the configuration or something else?
>>
>> Thanks for the help.
>>
>> Regards,
>> Devaraj.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Grizzly in Hadoop

Jeanfrancois Arcand-2
Hi,

Jeanfrancois Arcand wrote:

> Hi Devaraj,
>
> Devaraj Das wrote:
>> Hi Jeanfrancois,
>> I will get back to you in detail in some time from now. But for now, I
>> just
>> want to update you that setting the keepalivetimeout to 0 seemed to have
>> improved performance. Thanks for that tip. Now, do you think, if we start
>> using the persistent connections (transfer a batch of files at a time)
>> feature, it would help us significantly?
>
> Yes it would as you will save the network overhead of opening a connection.
>
>> Also, will using multiple selectorReadThreads help the performance
>> (since we
>> have multiple CPUs on our machines)?
>
> Right now it is not supported with the http module (Just wakes up I
> didn't port it yet). Hence that's why you aren't seeing any performance
> difference.

I've filled:

https://grizzly.dev.java.net/issues/show_bug.cgi?id=4

to track the issue.

-- Jeanfrancois

>
> Taking about your benchmark, It would be interesting to understand how
> you configure Jetty. Your connections seem to be data intensive as
> opposed to connection intensive. NIO is good at handling large amount of
> connections without spending idle threads but if the threads are not
> idle anyway (as in this case), the place to make this faster is only to
> reduce the number of bcopy/memcopy calls. I'm just thinking loud here :-)
>
>
>> I will send the source code of the adpater shortly...
>> Thanks,
>
> Thanks!
>
> -- Jeanfrancois
>
>> Devaraj.
>>
>> -----Original Message-----
>> From: [hidden email] [mailto:[hidden email]]
>> Sent: Tuesday, June 12, 2007 6:25 PM
>> To: [hidden email]; Scott Oaks
>> Subject: Re: Grizzly in Hadoop
>>
>> Hi,
>>
>> Devaraj Das wrote:
>>> Hi,
>>>
>>> We are considering using Grizzly (1.5) in Hadoop (an open source
>>> framework that has the MapReduce and Distributed File System
>>> implementations). The main reason for using it is to optimize a
>>> framework
>> phase called "shuffle".
>>> In this phase we move lots of data across the network.
>>
>> Cool :-) I know Hadoop as your Web 2.0 stack is also considering using it
>> :-)
>>
>>> We are currently using HTTP for moving the data (actually files) and
>>> we use Jetty5. Now we are thinking of moving to Grizzly (to have NIO
>>> and all its niceness). But initial experiments with our benchmark
>>> showed that with Grizzly the performance of the shuffle phase is
>>> nearly the same as we have with Jetty5. This is not what we initially
>>> expected and hence would like to get feedback on where we might be going
>> wrong.
>>> Hadoop is designed to run on large clusters of 100s of nodes
>>> (currently it can run stable/reliably in a 1K node cluster). From the
>>> Grizzly point of view, what needs to be known is that each node has a
>>> HTTP server. Both
>>> Jetty5 and Grizzly provides the ability to have multiple handlers to
>>> service the incoming requests.
>>>
>>> There are 2 clients on each node, and each client has a configurable
>>> number of fetcher threads. The fetcher code is written using the
>>> java.net.URLConnection API.
>>> Every node has both the server and the clients. The threads basically
>>> hit the HTTP server asking for specific files. They all do this at
>>> once (with some randomness in the order of hosts, maybe).
>>>
>>> The benchmark that I tested with is a sort for ~5TB of data with a
>>> cluster of 500 nodes. On the hardware side, I used a cluster that has
>>> 4 dualcore processors in each machine. The machines are partitioned
>>> into racks with a gigabit ethernet within the rack and 100Mbps across
>>> the racks. There are roughly 78000 independent files spread across
>>> these 500 nodes each of size ~60KB that the client pulls (and again we
>> have two such clients per node).
>>> So you can imagine you have a massive all-all communication
>>> happening. The configuration for the server and client is as follows:
>>>  Grizzly configuration for port 9999
>>>          maxThreads: 100
>>>          minThreads: 1
>>>          ByteBuffer size: 8192
>>>          useDirectByteBuffer: false
>>>          useByteBufferView: false
>>>          maxHttpHeaderSize: 8192
>>>          maxKeepAliveRequests: 256
>>>          keepAliveTimeoutInSeconds: 10
>>>          Static File Cache enabled: true
>>>          Stream Algorithm :
>>> com.sun.grizzly.http.algorithms.NoParsingAlgorithm
>>>          Pipeline : com.sun.grizzly.http.LinkedListPipeline
>>>          Round Robin Selector Algorithm enabled: false
>>>          Round Robin Selector pool size: 0
>>>          recycleTasks: true
>>>          Asynchronous Request Processing enabled: false I also tried
>>> some configs with multiple selectorReadThreads but didn't make much
>>> difference.
>>
>> keepAliveTimeoutInSeconds: 10 seems for me a little low...what is the
>> reason
>> to have such a low number? Closing faster idle connection?
>>
>>
>>> The client has 30 fetcher threads and the way it is designed is that
>>> only
>>> one fetch from any given host would happen at any point of time. So if a
>>> server host, h1, has 'n' files that we should pull, we do that one at a
>> time
>>> (as opposed to multiple threads hitting that server to fetch multiple
>> files
>>> in parallel).
>>>
>>> Also, we don't use the features of HTTP1.1 persistent connections or
>>> pipelining. We fetch exactly one file and close the connection to the
>>> server.
>>
>> That's answer my previous question :-) I would recommend setting the
>> keepAliveTimeoutInSeconds=0 then as I'm sure the performance will
>> improve (no call to the keep-alive subsystem).
>>
>>> With the above setup, the performance I see is not different from what I
>> see
>>> with Jetty5.
>>
>> Can it be the benchmark itself that it not able to load more?
>>
>>
>> I see a lot of Read timeouts on the client side (and we have a
>>> backoff (for the server host) on the client implementation whenever we
>> fail
>>> to fetch a file). I also saw some exceptions of the form on the server:
>>
>> You seem to hit the epoll problem on Linux. I know there is a way to
>> avoid using epoll (a property). I will ping the NIO team and let you
>> know.
>>
>> Also, what exactly your Adapter implementation is doing? Can you share
>> the code? If I can have access to your setup, I would like to see if
>> using Grizzly 1.0.15 makes a difference (just to make sure we don't
>> have a bug in 1.5....as far as I can tell, I.5 is as fast as 1.0 on my
>> benchmark).
>>
>> Thanks,
>>
>> --Jeanfrancois
>>
>>
>>> Jun 11, 2007 5:04:51 PM com.sun.grizzly.Controller doSelect
>>> SEVERE: doSelect exception
>>> java.io.IOException: Operation not permitted
>>>         at sun.nio.ch.EPollArrayWrapper.epollCtl(Native Method)
>>>         at
>>>
>> sun.nio.ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.java:202)
>>
>>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:183)
>>>         at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>         at
>>> com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:277)
>>>         at com.sun.grizzly.Controller.doSelect(Controller.java:218)
>>>         at com.sun.grizzly.Controller.start(Controller.java:451)
>>>         at
>>>
>> com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1158)
>>
>>>         at
>>>
>> com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1121)
>>
>>>         at
>> com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1099)
>>> Are we missing something in the configuration or something else?
>>>
>>> Thanks for the help.
>>>
>>> Regards,
>>> Devaraj.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Grizzly in Hadoop

Jeanfrancois Arcand-2
In reply to this post by Devaraj Das
Salut,

Devaraj Das wrote:

> Here is the core of the "service(Request, Response)" adapter method
>
>         response.setHeader(MAP_OUTPUT_LENGTH, Long.toString(<file_length>));
>
>         int len = mapOutputIn.read(<from_a_file>);
>         while (len > 0) {
>           try {
>             chunk.recycle();
>             chunk.append(buffer, 0, len);
>             response.doWrite(chunk);
>           } catch (IOException ie) {
>             isInputException = false;
>             throw ie;
>           }
>           totalRead += len;
>           if (totalRead == partLength) break;
>           len = mapOutputIn.read(<from_a_file>);
>         }
>
> By the way, we don't set content-length and expect the server to do chunked
> encoding (in some cases we could be serving very huge files in the order of
> 4-5 Gigs), and content-length header expects things to be 'int' (at least in
> jetty). So we work around this problem by defining our own custom http
> header that is a "long".

I can add an API for you if you want to use long (easy to do). Just file
an issue and I will make it for 1.5.2 :-) This is not the first time
this requirement are asked.


>
> Here is the core of the client code
>
>       URLConnection connection = path.openConnection();
>       if (timeout > 0) {
>         connection.setConnectTimeout(timeout);
>         connection.setReadTimeout(timeout);
>       }
>       InputStream input = connection.getInputStream();
>       //get the content length from a custom http header
>       long length =
> Long.parseLong(connection.getHeaderField(MAP_OUTPUT_LENGTH));
>
>       OutputStream output = fileSys.create(localFilename);
>
>           int len = input.read(buffer);
>           while (len > 0) {
>             totalBytes += len;
>             output.write(buffer, 0 , len);
>             if (currentThread.isInterrupted()) {
>               throw new InterruptedException();
>             }
>             if (totalBytes == length) break;
>             len = input.read(buffer);
>           }
>         } finally {
>           output.close();
>         }
>       } finally {
>         input.close();
>       }
>
> I am seeing exceptions of the form on the server (on quite a few nodes).
> This seems to be contributing to the performance degradation now. Any clue
> on this? Thanks much for the help.
>
> 2007-06-12 14:25:28,968 WARN org.apache.hadoop.mapred.TaskTracker:
> getMapOutput(task_0001_m_077205_1,971) failed :
> java.io.IOException: Broken pipe
>         at sun.nio.ch.FileDispatcher.write0(Native Method)
>         at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>         at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
>         at sun.nio.ch.IOUtil.write(IOUtil.java:75)
>         at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>         at
> com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:88)
>         at
> com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:57)
>         at
> com.sun.grizzly.http.SocketChannelOutputBuffer.flushChannel(SocketChannelOut
> putBuffer.java:138)
>         at
> com.sun.grizzly.http.SocketChannelOutputBuffer.realWriteBytes(SocketChannelO
> utputBuffer.java:125)
>         at com.sun.grizzly.util.buf.ByteChunk.append(ByteChunk.java:331)
>         at
> com.sun.grizzly.tcp.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doW
> rite(InternalOutputBuffer.java:856)
>         at
> com.sun.grizzly.tcp.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutput
> Filter.java:136)
>         at
> com.sun.grizzly.tcp.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer
> .java:614)
>         at com.sun.grizzly.tcp.Response.doWrite(Response.java:587)
>         at
> org.apache.hadoop.mapred.TaskTracker$MapOutput.service(TaskTracker.java:1944
> )
>         at
> com.sun.grizzly.http.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask
> .java:597)
>         at
> com.sun.grizzly.http.DefaultProcessorTask.doProcess(DefaultProcessorTask.jav
> a:528)
>         at
> com.sun.grizzly.http.DefaultProcessorTask.process(DefaultProcessorTask.java:
> 772)
>         at
> com.sun.grizzly.http.SelectorThread$3.execute(SelectorThread.java:745)
>         at
> com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolCh
> ain.java:74)

This exception just means the client is closing the connection before
the server has a chance to write the response. Are you seeing an
exception on the client side?

Thanks!

-- Jeanfrancois




>
> -----Original Message-----
> From: Devaraj Das [mailto:[hidden email]]
> Sent: Tuesday, June 12, 2007 7:45 PM
> To: [hidden email]; 'Scott Oaks'
> Cc: 'Owen O'Malley'; 'Sameer Paranjpye'
> Subject: RE: Grizzly in Hadoop
>
> Hi Jeanfrancois,
> I will get back to you in detail in some time from now. But for now, I just
> want to update you that setting the keepalivetimeout to 0 seemed to have
> improved performance. Thanks for that tip. Now, do you think, if we start
> using the persistent connections (transfer a batch of files at a time)
> feature, it would help us significantly?
> Also, will using multiple selectorReadThreads help the performance (since we
> have multiple CPUs on our machines)?
> I will send the source code of the adpater shortly...
> Thanks,
> Devaraj.
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]]
> Sent: Tuesday, June 12, 2007 6:25 PM
> To: [hidden email]; Scott Oaks
> Subject: Re: Grizzly in Hadoop
>
> Hi,
>
> Devaraj Das wrote:
>> Hi,
>>
>> We are considering using Grizzly (1.5) in Hadoop (an open source
>> framework that has the MapReduce and Distributed File System
>> implementations). The main reason for using it is to optimize a
>> framework
> phase called "shuffle".
>> In this phase we move lots of data across the network.
>
> Cool :-) I know Hadoop as your Web 2.0 stack is also considering using it
> :-)
>
>> We are currently using HTTP for moving the data (actually files) and
>> we use Jetty5. Now we are thinking of moving to Grizzly (to have NIO
>> and all its niceness). But initial experiments with our benchmark
>> showed that with Grizzly the performance of the shuffle phase is
>> nearly the same as we have with Jetty5. This is not what we initially
>> expected and hence would like to get feedback on where we might be
>> going
> wrong.
>> Hadoop is designed to run on large clusters of 100s of nodes
>> (currently it can run stable/reliably in a 1K node cluster). From the
>> Grizzly point of view, what needs to be known is that each node has a
>> HTTP server. Both
>> Jetty5 and Grizzly provides the ability to have multiple handlers to
>> service the incoming requests.
>>
>> There are 2 clients on each node, and each client has a configurable
>> number of fetcher threads. The fetcher code is written using the
>> java.net.URLConnection API.
>> Every node has both the server and the clients. The threads basically
>> hit the HTTP server asking for specific files. They all do this at
>> once (with some randomness in the order of hosts, maybe).
>>
>> The benchmark that I tested with is a sort for ~5TB of data with a
>> cluster of 500 nodes. On the hardware side, I used a cluster that has
>> 4 dualcore processors in each machine. The machines are partitioned
>> into racks with a gigabit ethernet within the rack and 100Mbps across
>> the racks. There are roughly 78000 independent files spread across
>> these 500 nodes each of size ~60KB that the client pulls (and again we
> have two such clients per node).
>> So you can imagine you have a massive all-all communication happening.
>> The configuration for the server and client is as follows:
>>  Grizzly configuration for port 9999
>>          maxThreads: 100
>>          minThreads: 1
>>          ByteBuffer size: 8192
>>          useDirectByteBuffer: false
>>          useByteBufferView: false
>>          maxHttpHeaderSize: 8192
>>          maxKeepAliveRequests: 256
>>          keepAliveTimeoutInSeconds: 10
>>          Static File Cache enabled: true
>>          Stream Algorithm :
>> com.sun.grizzly.http.algorithms.NoParsingAlgorithm
>>          Pipeline : com.sun.grizzly.http.LinkedListPipeline
>>          Round Robin Selector Algorithm enabled: false
>>          Round Robin Selector pool size: 0
>>          recycleTasks: true
>>          Asynchronous Request Processing enabled: false I also tried
>> some configs with multiple selectorReadThreads but didn't make much
>> difference.
>
> keepAliveTimeoutInSeconds: 10 seems for me a little low...what is the reason
> to have such a low number? Closing faster idle connection?
>
>
>> The client has 30 fetcher threads and the way it is designed is that
>> only one fetch from any given host would happen at any point of time.
>> So if a server host, h1, has 'n' files that we should pull, we do that
>> one at a
> time
>> (as opposed to multiple threads hitting that server to fetch multiple
> files
>> in parallel).
>>
>> Also, we don't use the features of HTTP1.1 persistent connections or
>> pipelining. We fetch exactly one file and close the connection to the
>> server.
>
> That's answer my previous question :-) I would recommend setting the
> keepAliveTimeoutInSeconds=0 then as I'm sure the performance will improve
> (no call to the keep-alive subsystem).
>
>> With the above setup, the performance I see is not different from what
>> I
> see
>> with Jetty5.
>
> Can it be the benchmark itself that it not able to load more?
>
>
> I see a lot of Read timeouts on the client side (and we have a
>> backoff (for the server host) on the client implementation whenever we
> fail
>> to fetch a file). I also saw some exceptions of the form on the server:
>
> You seem to hit the epoll problem on Linux. I know there is a way to avoid
> using epoll (a property). I will ping the NIO team and let you know.
>
> Also, what exactly your Adapter implementation is doing? Can you share the
> code? If I can have access to your setup, I would like to see if using
> Grizzly 1.0.15 makes a difference (just to make sure we don't have a bug in
> 1.5....as far as I can tell, I.5 is as fast as 1.0 on my benchmark).
>
> Thanks,
>
> --Jeanfrancois
>
>
>> Jun 11, 2007 5:04:51 PM com.sun.grizzly.Controller doSelect
>> SEVERE: doSelect exception
>> java.io.IOException: Operation not permitted
>>         at sun.nio.ch.EPollArrayWrapper.epollCtl(Native Method)
>>         at
>>
> sun.nio.ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.java:202)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:183)
>>         at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>         at
>> com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:277)
>>         at com.sun.grizzly.Controller.doSelect(Controller.java:218)
>>         at com.sun.grizzly.Controller.start(Controller.java:451)
>>         at
>>
> com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1158)
>>         at
>>
> com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1121)
>>         at
> com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1099)
>> Are we missing something in the configuration or something else?
>>
>> Thanks for the help.
>>
>> Regards,
>> Devaraj.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Grizzly in Hadoop

Devaraj Das
In reply to this post by Jeanfrancois Arcand-2
Hi Jeanfrancois,
Thanks for filing the bug! I would request you to fix this ASAP since it's
critical for us.
Regards,
Devaraj.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]]
Sent: Tuesday, June 12, 2007 10:14 PM
To: [hidden email]
Cc: 'Owen O'Malley'; 'Sameer Paranjpye'
Subject: Re: Grizzly in Hadoop

Hi,

Jeanfrancois Arcand wrote:

> Hi Devaraj,
>
> Devaraj Das wrote:
>> Hi Jeanfrancois,
>> I will get back to you in detail in some time from now. But for now,
>> I just want to update you that setting the keepalivetimeout to 0
>> seemed to have improved performance. Thanks for that tip. Now, do you
>> think, if we start using the persistent connections (transfer a batch
>> of files at a time) feature, it would help us significantly?
>
> Yes it would as you will save the network overhead of opening a
connection.
>
>> Also, will using multiple selectorReadThreads help the performance
>> (since we have multiple CPUs on our machines)?
>
> Right now it is not supported with the http module (Just wakes up I
> didn't port it yet). Hence that's why you aren't seeing any
> performance difference.

I've filled:

https://grizzly.dev.java.net/issues/show_bug.cgi?id=4

to track the issue.

-- Jeanfrancois

>
> Taking about your benchmark, It would be interesting to understand how
> you configure Jetty. Your connections seem to be data intensive as
> opposed to connection intensive. NIO is good at handling large amount
> of connections without spending idle threads but if the threads are
> not idle anyway (as in this case), the place to make this faster is
> only to reduce the number of bcopy/memcopy calls. I'm just thinking
> loud here :-)
>
>
>> I will send the source code of the adpater shortly...
>> Thanks,
>
> Thanks!
>
> -- Jeanfrancois
>
>> Devaraj.
>>
>> -----Original Message-----
>> From: [hidden email]
>> [mailto:[hidden email]]
>> Sent: Tuesday, June 12, 2007 6:25 PM
>> To: [hidden email]; Scott Oaks
>> Subject: Re: Grizzly in Hadoop
>>
>> Hi,
>>
>> Devaraj Das wrote:
>>> Hi,
>>>
>>> We are considering using Grizzly (1.5) in Hadoop (an open source
>>> framework that has the MapReduce and Distributed File System
>>> implementations). The main reason for using it is to optimize a
>>> framework
>> phase called "shuffle".
>>> In this phase we move lots of data across the network.
>>
>> Cool :-) I know Hadoop as your Web 2.0 stack is also considering
>> using it
>> :-)
>>
>>> We are currently using HTTP for moving the data (actually files) and
>>> we use Jetty5. Now we are thinking of moving to Grizzly (to have NIO
>>> and all its niceness). But initial experiments with our benchmark
>>> showed that with Grizzly the performance of the shuffle phase is
>>> nearly the same as we have with Jetty5. This is not what we
>>> initially expected and hence would like to get feedback on where we
>>> might be going
>> wrong.
>>> Hadoop is designed to run on large clusters of 100s of nodes
>>> (currently it can run stable/reliably in a 1K node cluster). From
>>> the Grizzly point of view, what needs to be known is that each node
>>> has a HTTP server. Both
>>> Jetty5 and Grizzly provides the ability to have multiple handlers to
>>> service the incoming requests.
>>>
>>> There are 2 clients on each node, and each client has a configurable
>>> number of fetcher threads. The fetcher code is written using the
>>> java.net.URLConnection API.
>>> Every node has both the server and the clients. The threads
>>> basically hit the HTTP server asking for specific files. They all do
>>> this at once (with some randomness in the order of hosts, maybe).
>>>
>>> The benchmark that I tested with is a sort for ~5TB of data with a
>>> cluster of 500 nodes. On the hardware side, I used a cluster that
>>> has
>>> 4 dualcore processors in each machine. The machines are partitioned
>>> into racks with a gigabit ethernet within the rack and 100Mbps
>>> across the racks. There are roughly 78000 independent files spread
>>> across these 500 nodes each of size ~60KB that the client pulls (and
>>> again we
>> have two such clients per node).
>>> So you can imagine you have a massive all-all communication
>>> happening. The configuration for the server and client is as follows:
>>>  Grizzly configuration for port 9999
>>>          maxThreads: 100
>>>          minThreads: 1
>>>          ByteBuffer size: 8192
>>>          useDirectByteBuffer: false
>>>          useByteBufferView: false
>>>          maxHttpHeaderSize: 8192
>>>          maxKeepAliveRequests: 256
>>>          keepAliveTimeoutInSeconds: 10
>>>          Static File Cache enabled: true
>>>          Stream Algorithm :
>>> com.sun.grizzly.http.algorithms.NoParsingAlgorithm
>>>          Pipeline : com.sun.grizzly.http.LinkedListPipeline
>>>          Round Robin Selector Algorithm enabled: false
>>>          Round Robin Selector pool size: 0
>>>          recycleTasks: true
>>>          Asynchronous Request Processing enabled: false I also tried
>>> some configs with multiple selectorReadThreads but didn't make much
>>> difference.
>>
>> keepAliveTimeoutInSeconds: 10 seems for me a little low...what is the
>> reason to have such a low number? Closing faster idle connection?
>>
>>
>>> The client has 30 fetcher threads and the way it is designed is that
>>> only
>>> one fetch from any given host would happen at any point of time. So if a
>>> server host, h1, has 'n' files that we should pull, we do that one at a
>> time
>>> (as opposed to multiple threads hitting that server to fetch multiple
>> files
>>> in parallel).
>>>
>>> Also, we don't use the features of HTTP1.1 persistent connections or
>>> pipelining. We fetch exactly one file and close the connection to the
>>> server.
>>
>> That's answer my previous question :-) I would recommend setting the
>> keepAliveTimeoutInSeconds=0 then as I'm sure the performance will
>> improve (no call to the keep-alive subsystem).
>>
>>> With the above setup, the performance I see is not different from what I
>> see
>>> with Jetty5.
>>
>> Can it be the benchmark itself that it not able to load more?
>>
>>
>> I see a lot of Read timeouts on the client side (and we have a
>>> backoff (for the server host) on the client implementation whenever we
>> fail
>>> to fetch a file). I also saw some exceptions of the form on the server:
>>
>> You seem to hit the epoll problem on Linux. I know there is a way to
>> avoid using epoll (a property). I will ping the NIO team and let you
>> know.
>>
>> Also, what exactly your Adapter implementation is doing? Can you share
>> the code? If I can have access to your setup, I would like to see if
>> using Grizzly 1.0.15 makes a difference (just to make sure we don't
>> have a bug in 1.5....as far as I can tell, I.5 is as fast as 1.0 on my
>> benchmark).
>>
>> Thanks,
>>
>> --Jeanfrancois
>>
>>
>>> Jun 11, 2007 5:04:51 PM com.sun.grizzly.Controller doSelect
>>> SEVERE: doSelect exception
>>> java.io.IOException: Operation not permitted
>>>         at sun.nio.ch.EPollArrayWrapper.epollCtl(Native Method)
>>>         at
>>>
>>
sun.nio.ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.java:202)

>>
>>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:183)
>>>         at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>         at
>>> com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:277)
>>>         at com.sun.grizzly.Controller.doSelect(Controller.java:218)
>>>         at com.sun.grizzly.Controller.start(Controller.java:451)
>>>         at
>>>
>>
com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1158)
>>
>>>         at
>>>
>>
com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1121)

>>
>>>         at
>> com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1099)
>>> Are we missing something in the configuration or something else?
>>>
>>> Thanks for the help.
>>>
>>> Regards,
>>> Devaraj.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Grizzly in Hadoop

Jeanfrancois Arcand-2
Hi Devaraj,

Devaraj Das wrote:
> Hi Jeanfrancois,
> Thanks for filing the bug! I would request you to fix this ASAP since it's
> critical for us.

Sure. We should have something this week if all goes well. Now if you
can't wait, you can always fallback to Grizzly 1.0. Not ideal, but a
possible workaround :-)

Thanks!

-- Jeanfrancois

> Regards,
> Devaraj.
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]]
> Sent: Tuesday, June 12, 2007 10:14 PM
> To: [hidden email]
> Cc: 'Owen O'Malley'; 'Sameer Paranjpye'
> Subject: Re: Grizzly in Hadoop
>
> Hi,
>
> Jeanfrancois Arcand wrote:
>> Hi Devaraj,
>>
>> Devaraj Das wrote:
>>> Hi Jeanfrancois,
>>> I will get back to you in detail in some time from now. But for now,
>>> I just want to update you that setting the keepalivetimeout to 0
>>> seemed to have improved performance. Thanks for that tip. Now, do you
>>> think, if we start using the persistent connections (transfer a batch
>>> of files at a time) feature, it would help us significantly?
>> Yes it would as you will save the network overhead of opening a
> connection.
>>> Also, will using multiple selectorReadThreads help the performance
>>> (since we have multiple CPUs on our machines)?
>> Right now it is not supported with the http module (Just wakes up I
>> didn't port it yet). Hence that's why you aren't seeing any
>> performance difference.
>
> I've filled:
>
> https://grizzly.dev.java.net/issues/show_bug.cgi?id=4
>
> to track the issue.
>
> -- Jeanfrancois
>
>> Taking about your benchmark, It would be interesting to understand how
>> you configure Jetty. Your connections seem to be data intensive as
>> opposed to connection intensive. NIO is good at handling large amount
>> of connections without spending idle threads but if the threads are
>> not idle anyway (as in this case), the place to make this faster is
>> only to reduce the number of bcopy/memcopy calls. I'm just thinking
>> loud here :-)
>>
>>
>>> I will send the source code of the adpater shortly...
>>> Thanks,
>> Thanks!
>>
>> -- Jeanfrancois
>>
>>> Devaraj.
>>>
>>> -----Original Message-----
>>> From: [hidden email]
>>> [mailto:[hidden email]]
>>> Sent: Tuesday, June 12, 2007 6:25 PM
>>> To: [hidden email]; Scott Oaks
>>> Subject: Re: Grizzly in Hadoop
>>>
>>> Hi,
>>>
>>> Devaraj Das wrote:
>>>> Hi,
>>>>
>>>> We are considering using Grizzly (1.5) in Hadoop (an open source
>>>> framework that has the MapReduce and Distributed File System
>>>> implementations). The main reason for using it is to optimize a
>>>> framework
>>> phase called "shuffle".
>>>> In this phase we move lots of data across the network.
>>> Cool :-) I know Hadoop as your Web 2.0 stack is also considering
>>> using it
>>> :-)
>>>
>>>> We are currently using HTTP for moving the data (actually files) and
>>>> we use Jetty5. Now we are thinking of moving to Grizzly (to have NIO
>>>> and all its niceness). But initial experiments with our benchmark
>>>> showed that with Grizzly the performance of the shuffle phase is
>>>> nearly the same as we have with Jetty5. This is not what we
>>>> initially expected and hence would like to get feedback on where we
>>>> might be going
>>> wrong.
>>>> Hadoop is designed to run on large clusters of 100s of nodes
>>>> (currently it can run stable/reliably in a 1K node cluster). From
>>>> the Grizzly point of view, what needs to be known is that each node
>>>> has a HTTP server. Both
>>>> Jetty5 and Grizzly provides the ability to have multiple handlers to
>>>> service the incoming requests.
>>>>
>>>> There are 2 clients on each node, and each client has a configurable
>>>> number of fetcher threads. The fetcher code is written using the
>>>> java.net.URLConnection API.
>>>> Every node has both the server and the clients. The threads
>>>> basically hit the HTTP server asking for specific files. They all do
>>>> this at once (with some randomness in the order of hosts, maybe).
>>>>
>>>> The benchmark that I tested with is a sort for ~5TB of data with a
>>>> cluster of 500 nodes. On the hardware side, I used a cluster that
>>>> has
>>>> 4 dualcore processors in each machine. The machines are partitioned
>>>> into racks with a gigabit ethernet within the rack and 100Mbps
>>>> across the racks. There are roughly 78000 independent files spread
>>>> across these 500 nodes each of size ~60KB that the client pulls (and
>>>> again we
>>> have two such clients per node).
>>>> So you can imagine you have a massive all-all communication
>>>> happening. The configuration for the server and client is as follows:
>>>>  Grizzly configuration for port 9999
>>>>          maxThreads: 100
>>>>          minThreads: 1
>>>>          ByteBuffer size: 8192
>>>>          useDirectByteBuffer: false
>>>>          useByteBufferView: false
>>>>          maxHttpHeaderSize: 8192
>>>>          maxKeepAliveRequests: 256
>>>>          keepAliveTimeoutInSeconds: 10
>>>>          Static File Cache enabled: true
>>>>          Stream Algorithm :
>>>> com.sun.grizzly.http.algorithms.NoParsingAlgorithm
>>>>          Pipeline : com.sun.grizzly.http.LinkedListPipeline
>>>>          Round Robin Selector Algorithm enabled: false
>>>>          Round Robin Selector pool size: 0
>>>>          recycleTasks: true
>>>>          Asynchronous Request Processing enabled: false I also tried
>>>> some configs with multiple selectorReadThreads but didn't make much
>>>> difference.
>>> keepAliveTimeoutInSeconds: 10 seems for me a little low...what is the
>>> reason to have such a low number? Closing faster idle connection?
>>>
>>>
>>>> The client has 30 fetcher threads and the way it is designed is that
>>>> only
>>>> one fetch from any given host would happen at any point of time. So if a
>>>> server host, h1, has 'n' files that we should pull, we do that one at a
>>> time
>>>> (as opposed to multiple threads hitting that server to fetch multiple
>>> files
>>>> in parallel).
>>>>
>>>> Also, we don't use the features of HTTP1.1 persistent connections or
>>>> pipelining. We fetch exactly one file and close the connection to the
>>>> server.
>>> That's answer my previous question :-) I would recommend setting the
>>> keepAliveTimeoutInSeconds=0 then as I'm sure the performance will
>>> improve (no call to the keep-alive subsystem).
>>>
>>>> With the above setup, the performance I see is not different from what I
>>> see
>>>> with Jetty5.
>>> Can it be the benchmark itself that it not able to load more?
>>>
>>>
>>> I see a lot of Read timeouts on the client side (and we have a
>>>> backoff (for the server host) on the client implementation whenever we
>>> fail
>>>> to fetch a file). I also saw some exceptions of the form on the server:
>>> You seem to hit the epoll problem on Linux. I know there is a way to
>>> avoid using epoll (a property). I will ping the NIO team and let you
>>> know.
>>>
>>> Also, what exactly your Adapter implementation is doing? Can you share
>>> the code? If I can have access to your setup, I would like to see if
>>> using Grizzly 1.0.15 makes a difference (just to make sure we don't
>>> have a bug in 1.5....as far as I can tell, I.5 is as fast as 1.0 on my
>>> benchmark).
>>>
>>> Thanks,
>>>
>>> --Jeanfrancois
>>>
>>>
>>>> Jun 11, 2007 5:04:51 PM com.sun.grizzly.Controller doSelect
>>>> SEVERE: doSelect exception
>>>> java.io.IOException: Operation not permitted
>>>>         at sun.nio.ch.EPollArrayWrapper.epollCtl(Native Method)
>>>>         at
>>>>
> sun.nio.ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.java:202)
>
>>>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:183)
>>>>         at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>         at
>>>> com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:277)
>>>>         at com.sun.grizzly.Controller.doSelect(Controller.java:218)
>>>>         at com.sun.grizzly.Controller.start(Controller.java:451)
>>>>         at
>>>>
> com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1158)
>>>>         at
>>>>
> com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1121)
>>>>         at
>>> com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1099)
>>>> Are we missing something in the configuration or something else?
>>>>
>>>> Thanks for the help.
>>>>
>>>> Regards,
>>>> Devaraj.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Grizzly in Hadoop

Devaraj Das
In reply to this post by Jeanfrancois Arcand-2
Regarding the jetty configuration, the major thing done there is to set the
the max/min threads and the numbers are *exactly* the same as I configure
for grizzly. We didn't tweak anything else in jetty5. Regarding your point
about reducing the memcpy/bcopy calls, I think it's worth investigating the
use of DirectByteBuffers and configure grizzly to use that. What do you
think?

-----Original Message-----
From: [hidden email] [mailto:[hidden email]]
Sent: Tuesday, June 12, 2007 10:01 PM
To: [hidden email]
Cc: 'Owen O'Malley'; 'Sameer Paranjpye'
Subject: Re: Grizzly in Hadoop

Hi Devaraj,

Devaraj Das wrote:
> Hi Jeanfrancois,
> I will get back to you in detail in some time from now. But for now, I
> just want to update you that setting the keepalivetimeout to 0 seemed
> to have improved performance. Thanks for that tip. Now, do you think,
> if we start using the persistent connections (transfer a batch of
> files at a time) feature, it would help us significantly?

Yes it would as you will save the network overhead of opening a connection.

> Also, will using multiple selectorReadThreads help the performance
> (since we have multiple CPUs on our machines)?

Right now it is not supported with the http module (Just wakes up I didn't
port it yet). Hence that's why you aren't seeing any performance difference.

Taking about your benchmark, It would be interesting to understand how you
configure Jetty. Your connections seem to be data intensive as opposed to
connection intensive. NIO is good at handling large amount of connections
without spending idle threads but if the threads are not idle anyway (as in
this case), the place to make this faster is only to reduce the number of
bcopy/memcopy calls. I'm just thinking loud here :-)


> I will send the source code of the adpater shortly...
> Thanks,

Thanks!

-- Jeanfrancois

> Devaraj.
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]]
> Sent: Tuesday, June 12, 2007 6:25 PM
> To: [hidden email]; Scott Oaks
> Subject: Re: Grizzly in Hadoop
>
> Hi,
>
> Devaraj Das wrote:
>> Hi,
>>
>> We are considering using Grizzly (1.5) in Hadoop (an open source
>> framework that has the MapReduce and Distributed File System
>> implementations). The main reason for using it is to optimize a
>> framework
> phase called "shuffle".
>> In this phase we move lots of data across the network.
>
> Cool :-) I know Hadoop as your Web 2.0 stack is also considering using
> it
> :-)
>
>> We are currently using HTTP for moving the data (actually files) and
>> we use Jetty5. Now we are thinking of moving to Grizzly (to have NIO
>> and all its niceness). But initial experiments with our benchmark
>> showed that with Grizzly the performance of the shuffle phase is
>> nearly the same as we have with Jetty5. This is not what we initially
>> expected and hence would like to get feedback on where we might be
>> going
> wrong.
>> Hadoop is designed to run on large clusters of 100s of nodes
>> (currently it can run stable/reliably in a 1K node cluster). From the
>> Grizzly point of view, what needs to be known is that each node has a
>> HTTP server. Both
>> Jetty5 and Grizzly provides the ability to have multiple handlers to
>> service the incoming requests.
>>
>> There are 2 clients on each node, and each client has a configurable
>> number of fetcher threads. The fetcher code is written using the
>> java.net.URLConnection API.
>> Every node has both the server and the clients. The threads basically
>> hit the HTTP server asking for specific files. They all do this at
>> once (with some randomness in the order of hosts, maybe).
>>
>> The benchmark that I tested with is a sort for ~5TB of data with a
>> cluster of 500 nodes. On the hardware side, I used a cluster that has
>> 4 dualcore processors in each machine. The machines are partitioned
>> into racks with a gigabit ethernet within the rack and 100Mbps across
>> the racks. There are roughly 78000 independent files spread across
>> these 500 nodes each of size ~60KB that the client pulls (and again
>> we
> have two such clients per node).
>> So you can imagine you have a massive all-all communication happening.
>> The configuration for the server and client is as follows:
>>  Grizzly configuration for port 9999
>>          maxThreads: 100
>>          minThreads: 1
>>          ByteBuffer size: 8192
>>          useDirectByteBuffer: false
>>          useByteBufferView: false
>>          maxHttpHeaderSize: 8192
>>          maxKeepAliveRequests: 256
>>          keepAliveTimeoutInSeconds: 10
>>          Static File Cache enabled: true
>>          Stream Algorithm :
>> com.sun.grizzly.http.algorithms.NoParsingAlgorithm
>>          Pipeline : com.sun.grizzly.http.LinkedListPipeline
>>          Round Robin Selector Algorithm enabled: false
>>          Round Robin Selector pool size: 0
>>          recycleTasks: true
>>          Asynchronous Request Processing enabled: false I also tried
>> some configs with multiple selectorReadThreads but didn't make much
>> difference.
>
> keepAliveTimeoutInSeconds: 10 seems for me a little low...what is the
> reason to have such a low number? Closing faster idle connection?
>
>
>> The client has 30 fetcher threads and the way it is designed is that
>> only one fetch from any given host would happen at any point of time.
>> So if a server host, h1, has 'n' files that we should pull, we do
>> that one at a
> time
>> (as opposed to multiple threads hitting that server to fetch multiple
> files
>> in parallel).
>>
>> Also, we don't use the features of HTTP1.1 persistent connections or
>> pipelining. We fetch exactly one file and close the connection to the
>> server.
>
> That's answer my previous question :-) I would recommend setting the
> keepAliveTimeoutInSeconds=0 then as I'm sure the performance will
> improve (no call to the keep-alive subsystem).
>
>> With the above setup, the performance I see is not different from
>> what I
> see
>> with Jetty5.
>
> Can it be the benchmark itself that it not able to load more?
>
>
> I see a lot of Read timeouts on the client side (and we have a
>> backoff (for the server host) on the client implementation whenever
>> we
> fail
>> to fetch a file). I also saw some exceptions of the form on the server:
>
> You seem to hit the epoll problem on Linux. I know there is a way to
> avoid using epoll (a property). I will ping the NIO team and let you know.
>
> Also, what exactly your Adapter implementation is doing? Can you share
> the code? If I can have access to your setup, I would like to see if
> using Grizzly 1.0.15 makes a difference (just to make sure we don't
> have a bug in 1.5....as far as I can tell, I.5 is as fast as 1.0 on my
> benchmark).
>
> Thanks,
>
> --Jeanfrancois
>
>
>> Jun 11, 2007 5:04:51 PM com.sun.grizzly.Controller doSelect
>> SEVERE: doSelect exception
>> java.io.IOException: Operation not permitted
>>         at sun.nio.ch.EPollArrayWrapper.epollCtl(Native Method)
>>         at
>>
> sun.nio.ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.jav
> a:202)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:183)
>>         at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>         at
>> com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:277)
>>         at com.sun.grizzly.Controller.doSelect(Controller.java:218)
>>         at com.sun.grizzly.Controller.start(Controller.java:451)
>>         at
>>
> com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:
> 1158)
>>         at
>>
> com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:
> 1121)
>>         at
> com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1099)
>> Are we missing something in the configuration or something else?
>>
>> Thanks for the help.
>>
>> Regards,
>> Devaraj.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Grizzly in Hadoop

Devaraj Das
In reply to this post by Jeanfrancois Arcand-2
By the way, one thing that is unclear to me is how the does the pipeline
threads interact with the read selector threads. So, for example, if I do
SelectorThread.setMaxThreads(100) &
SelectorThread.setSelectorReadThreadsCount(10),
1) would it mean that the all the 10 selector threads dumps data into that
one pipeline (with 100 threads)? Is that configurable (I guess so)?
2) For a multiple CPU machine (like a 4 CPU one), do I really require
multiple selector read threads if all I want to read is HTTP requests
(simple GET requests with some headers).
Assume the system is heavily loaded (esp. from the data IO point of view;
the server writes a lot, few kilobytes to many megabytes, of http response
bytes to the clients) and has to support ~1000 clients concurrently.

Also, a related question is since the data that the server writes to each
client could vary in kilobytes to megabytes range, do you think that we
should look at a different strategy for handling clients (depending on how
long it would take to serve a client assuming that we have a rough idea of
the serving time). I am coming from the angle that if let's say we have 100
threads (in the pool) serving 100 clients, and if for some reason we are
taking a long time to process those client requests (or writing responses),
then although we will accept requests from other clients, we can't service
them and those unfortunate clients could potentially be starved for a long
time.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]]
Sent: Tuesday, June 12, 2007 11:49 PM
To: [hidden email]
Cc: 'Owen O'Malley'; 'Sameer Paranjpye'; 'Tahir Hashmi'
Subject: Re: Grizzly in Hadoop

Hi Devaraj,

Devaraj Das wrote:
> Hi Jeanfrancois,
> Thanks for filing the bug! I would request you to fix this ASAP since
> it's critical for us.

Sure. We should have something this week if all goes well. Now if you can't
wait, you can always fallback to Grizzly 1.0. Not ideal, but a possible
workaround :-)

Thanks!

-- Jeanfrancois

> Regards,
> Devaraj.
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]]
> Sent: Tuesday, June 12, 2007 10:14 PM
> To: [hidden email]
> Cc: 'Owen O'Malley'; 'Sameer Paranjpye'
> Subject: Re: Grizzly in Hadoop
>
> Hi,
>
> Jeanfrancois Arcand wrote:
>> Hi Devaraj,
>>
>> Devaraj Das wrote:
>>> Hi Jeanfrancois,
>>> I will get back to you in detail in some time from now. But for now,
>>> I just want to update you that setting the keepalivetimeout to 0
>>> seemed to have improved performance. Thanks for that tip. Now, do
>>> you think, if we start using the persistent connections (transfer a
>>> batch of files at a time) feature, it would help us significantly?
>> Yes it would as you will save the network overhead of opening a
> connection.
>>> Also, will using multiple selectorReadThreads help the performance
>>> (since we have multiple CPUs on our machines)?
>> Right now it is not supported with the http module (Just wakes up I
>> didn't port it yet). Hence that's why you aren't seeing any
>> performance difference.
>
> I've filled:
>
> https://grizzly.dev.java.net/issues/show_bug.cgi?id=4
>
> to track the issue.
>
> -- Jeanfrancois
>
>> Taking about your benchmark, It would be interesting to understand
>> how you configure Jetty. Your connections seem to be data intensive
>> as opposed to connection intensive. NIO is good at handling large
>> amount of connections without spending idle threads but if the
>> threads are not idle anyway (as in this case), the place to make this
>> faster is only to reduce the number of bcopy/memcopy calls. I'm just
>> thinking loud here :-)
>>
>>
>>> I will send the source code of the adpater shortly...
>>> Thanks,
>> Thanks!
>>
>> -- Jeanfrancois
>>
>>> Devaraj.
>>>
>>> -----Original Message-----
>>> From: [hidden email]
>>> [mailto:[hidden email]]
>>> Sent: Tuesday, June 12, 2007 6:25 PM
>>> To: [hidden email]; Scott Oaks
>>> Subject: Re: Grizzly in Hadoop
>>>
>>> Hi,
>>>
>>> Devaraj Das wrote:
>>>> Hi,
>>>>
>>>> We are considering using Grizzly (1.5) in Hadoop (an open source
>>>> framework that has the MapReduce and Distributed File System
>>>> implementations). The main reason for using it is to optimize a
>>>> framework
>>> phase called "shuffle".
>>>> In this phase we move lots of data across the network.
>>> Cool :-) I know Hadoop as your Web 2.0 stack is also considering
>>> using it
>>> :-)
>>>
>>>> We are currently using HTTP for moving the data (actually files)
>>>> and we use Jetty5. Now we are thinking of moving to Grizzly (to
>>>> have NIO and all its niceness). But initial experiments with our
>>>> benchmark showed that with Grizzly the performance of the shuffle
>>>> phase is nearly the same as we have with Jetty5. This is not what
>>>> we initially expected and hence would like to get feedback on where
>>>> we might be going
>>> wrong.
>>>> Hadoop is designed to run on large clusters of 100s of nodes
>>>> (currently it can run stable/reliably in a 1K node cluster). From
>>>> the Grizzly point of view, what needs to be known is that each node
>>>> has a HTTP server. Both
>>>> Jetty5 and Grizzly provides the ability to have multiple handlers
>>>> to service the incoming requests.
>>>>
>>>> There are 2 clients on each node, and each client has a
>>>> configurable number of fetcher threads. The fetcher code is written
>>>> using the java.net.URLConnection API.
>>>> Every node has both the server and the clients. The threads
>>>> basically hit the HTTP server asking for specific files. They all
>>>> do this at once (with some randomness in the order of hosts, maybe).
>>>>
>>>> The benchmark that I tested with is a sort for ~5TB of data with a
>>>> cluster of 500 nodes. On the hardware side, I used a cluster that
>>>> has
>>>> 4 dualcore processors in each machine. The machines are partitioned
>>>> into racks with a gigabit ethernet within the rack and 100Mbps
>>>> across the racks. There are roughly 78000 independent files spread
>>>> across these 500 nodes each of size ~60KB that the client pulls
>>>> (and again we
>>> have two such clients per node).
>>>> So you can imagine you have a massive all-all communication
>>>> happening. The configuration for the server and client is as follows:
>>>>  Grizzly configuration for port 9999
>>>>          maxThreads: 100
>>>>          minThreads: 1
>>>>          ByteBuffer size: 8192
>>>>          useDirectByteBuffer: false
>>>>          useByteBufferView: false
>>>>          maxHttpHeaderSize: 8192
>>>>          maxKeepAliveRequests: 256
>>>>          keepAliveTimeoutInSeconds: 10
>>>>          Static File Cache enabled: true
>>>>          Stream Algorithm :
>>>> com.sun.grizzly.http.algorithms.NoParsingAlgorithm
>>>>          Pipeline : com.sun.grizzly.http.LinkedListPipeline
>>>>          Round Robin Selector Algorithm enabled: false
>>>>          Round Robin Selector pool size: 0
>>>>          recycleTasks: true
>>>>          Asynchronous Request Processing enabled: false I also
>>>> tried some configs with multiple selectorReadThreads but didn't
>>>> make much difference.
>>> keepAliveTimeoutInSeconds: 10 seems for me a little low...what is
>>> the reason to have such a low number? Closing faster idle connection?
>>>
>>>
>>>> The client has 30 fetcher threads and the way it is designed is
>>>> that only one fetch from any given host would happen at any point
>>>> of time. So if a server host, h1, has 'n' files that we should
>>>> pull, we do that one at a
>>> time
>>>> (as opposed to multiple threads hitting that server to fetch
>>>> multiple
>>> files
>>>> in parallel).
>>>>
>>>> Also, we don't use the features of HTTP1.1 persistent connections
>>>> or pipelining. We fetch exactly one file and close the connection
>>>> to the server.
>>> That's answer my previous question :-) I would recommend setting the
>>> keepAliveTimeoutInSeconds=0 then as I'm sure the performance will
>>> improve (no call to the keep-alive subsystem).
>>>
>>>> With the above setup, the performance I see is not different from
>>>> what I
>>> see
>>>> with Jetty5.
>>> Can it be the benchmark itself that it not able to load more?
>>>
>>>
>>> I see a lot of Read timeouts on the client side (and we have a
>>>> backoff (for the server host) on the client implementation whenever
>>>> we
>>> fail
>>>> to fetch a file). I also saw some exceptions of the form on the server:
>>> You seem to hit the epoll problem on Linux. I know there is a way to
>>> avoid using epoll (a property). I will ping the NIO team and let you
>>> know.
>>>
>>> Also, what exactly your Adapter implementation is doing? Can you
>>> share the code? If I can have access to your setup, I would like to
>>> see if using Grizzly 1.0.15 makes a difference (just to make sure we
>>> don't have a bug in 1.5....as far as I can tell, I.5 is as fast as
>>> 1.0 on my benchmark).
>>>
>>> Thanks,
>>>
>>> --Jeanfrancois
>>>
>>>
>>>> Jun 11, 2007 5:04:51 PM com.sun.grizzly.Controller doSelect
>>>> SEVERE: doSelect exception
>>>> java.io.IOException: Operation not permitted
>>>>         at sun.nio.ch.EPollArrayWrapper.epollCtl(Native Method)
>>>>         at
>>>>
> sun.nio.ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.jav
> a:202)
>
>>>>         at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:183)
>>>>         at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>         at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

>>>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>         at
>>>> com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:277)
>>>>         at com.sun.grizzly.Controller.doSelect(Controller.java:218)
>>>>         at com.sun.grizzly.Controller.start(Controller.java:451)
>>>>         at
>>>>
> com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:
> 1158)
>>>>         at
>>>>
> com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:
> 1121)
>>>>         at
>>> com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1099)
>>>> Are we missing something in the configuration or something else?
>>>>
>>>> Thanks for the help.
>>>>
>>>> Regards,
>>>> Devaraj.
>>>>
>>>> -------------------------------------------------------------------
>>>> -- To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>> --------------------------------------------------------------------
>>> - To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>>
>>> --------------------------------------------------------------------
>>> - To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Grizzly in Hadoop

Oleksiy Stashok
Hello Deveraj,

> By the way, one thing that is unclear to me is how the does the pipeline
> threads interact with the read selector threads. So, for example, if I do
> SelectorThread.setMaxThreads(100) &
> SelectorThread.setSelectorReadThreadsCount(10),
> 1) would it mean that the all the 10 selector threads dumps data into that
> one pipeline (with 100 threads)? Is that configurable (I guess so)?
>  
Selector threads will share the pipeline threads.

> 2) For a multiple CPU machine (like a 4 CPU one), do I really require
> multiple selector read threads if all I want to read is HTTP requests
> (simple GET requests with some headers).
> Assume the system is heavily loaded (esp. from the data IO point of view;
> the server writes a lot, few kilobytes to many megabytes, of http response
> bytes to the clients) and has to support ~1000 clients concurrently.
>  
IMHO Selector read threads will help you, as basically they are
responsible for handling more accept/read events -> more concurrent users.

> Also, a related question is since the data that the server writes to each
> client could vary in kilobytes to megabytes range, do you think that we
> should look at a different strategy for handling clients (depending on how
> long it would take to serve a client assuming that we have a rough idea of
> the serving time). I am coming from the angle that if let's say we have 100
> threads (in the pool) serving 100 clients, and if for some reason we are
> taking a long time to process those client requests (or writing responses),
> then although we will accept requests from other clients, we can't service
> them and those unfortunate clients could potentially be starved for a long
> time.
>  
Here we possibly come (again :) ) to "Async writes", which are not
clearly implemented in Grizzly, and IMHO, this is the place for improvement.
But what I would think first.... If it really takes long time for
response - may be it's the sign to think about better application
scalability ;)

Thanks.

WBR,
Alexey.

> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]]
> Sent: Tuesday, June 12, 2007 11:49 PM
> To: [hidden email]
> Cc: 'Owen O'Malley'; 'Sameer Paranjpye'; 'Tahir Hashmi'
> Subject: Re: Grizzly in Hadoop
>
> Hi Devaraj,
>
> Devaraj Das wrote:
>  
>> Hi Jeanfrancois,
>> Thanks for filing the bug! I would request you to fix this ASAP since
>> it's critical for us.
>>    
>
> Sure. We should have something this week if all goes well. Now if you can't
> wait, you can always fallback to Grizzly 1.0. Not ideal, but a possible
> workaround :-)
>
> Thanks!
>
> -- Jeanfrancois
>
>  
>> Regards,
>> Devaraj.
>>
>> -----Original Message-----
>> From: [hidden email] [mailto:[hidden email]]
>> Sent: Tuesday, June 12, 2007 10:14 PM
>> To: [hidden email]
>> Cc: 'Owen O'Malley'; 'Sameer Paranjpye'
>> Subject: Re: Grizzly in Hadoop
>>
>> Hi,
>>
>> Jeanfrancois Arcand wrote:
>>    
>>> Hi Devaraj,
>>>
>>> Devaraj Das wrote:
>>>      
>>>> Hi Jeanfrancois,
>>>> I will get back to you in detail in some time from now. But for now,
>>>> I just want to update you that setting the keepalivetimeout to 0
>>>> seemed to have improved performance. Thanks for that tip. Now, do
>>>> you think, if we start using the persistent connections (transfer a
>>>> batch of files at a time) feature, it would help us significantly?
>>>>        
>>> Yes it would as you will save the network overhead of opening a
>>>      
>> connection.
>>    
>>>> Also, will using multiple selectorReadThreads help the performance
>>>> (since we have multiple CPUs on our machines)?
>>>>        
>>> Right now it is not supported with the http module (Just wakes up I
>>> didn't port it yet). Hence that's why you aren't seeing any
>>> performance difference.
>>>      
>> I've filled:
>>
>> https://grizzly.dev.java.net/issues/show_bug.cgi?id=4
>>
>> to track the issue.
>>
>> -- Jeanfrancois
>>
>>    
>>> Taking about your benchmark, It would be interesting to understand
>>> how you configure Jetty. Your connections seem to be data intensive
>>> as opposed to connection intensive. NIO is good at handling large
>>> amount of connections without spending idle threads but if the
>>> threads are not idle anyway (as in this case), the place to make this
>>> faster is only to reduce the number of bcopy/memcopy calls. I'm just
>>> thinking loud here :-)
>>>
>>>
>>>      
>>>> I will send the source code of the adpater shortly...
>>>> Thanks,
>>>>        
>>> Thanks!
>>>
>>> -- Jeanfrancois
>>>
>>>      
>>>> Devaraj.
>>>>
>>>> -----Original Message-----
>>>> From: [hidden email]
>>>> [mailto:[hidden email]]
>>>> Sent: Tuesday, June 12, 2007 6:25 PM
>>>> To: [hidden email]; Scott Oaks
>>>> Subject: Re: Grizzly in Hadoop
>>>>
>>>> Hi,
>>>>
>>>> Devaraj Das wrote:
>>>>        
>>>>> Hi,
>>>>>
>>>>> We are considering using Grizzly (1.5) in Hadoop (an open source
>>>>> framework that has the MapReduce and Distributed File System
>>>>> implementations). The main reason for using it is to optimize a
>>>>> framework
>>>>>          
>>>> phase called "shuffle".
>>>>        
>>>>> In this phase we move lots of data across the network.
>>>>>          
>>>> Cool :-) I know Hadoop as your Web 2.0 stack is also considering
>>>> using it
>>>> :-)
>>>>
>>>>        
>>>>> We are currently using HTTP for moving the data (actually files)
>>>>> and we use Jetty5. Now we are thinking of moving to Grizzly (to
>>>>> have NIO and all its niceness). But initial experiments with our
>>>>> benchmark showed that with Grizzly the performance of the shuffle
>>>>> phase is nearly the same as we have with Jetty5. This is not what
>>>>> we initially expected and hence would like to get feedback on where
>>>>> we might be going
>>>>>          
>>>> wrong.
>>>>        
>>>>> Hadoop is designed to run on large clusters of 100s of nodes
>>>>> (currently it can run stable/reliably in a 1K node cluster). From
>>>>> the Grizzly point of view, what needs to be known is that each node
>>>>> has a HTTP server. Both
>>>>> Jetty5 and Grizzly provides the ability to have multiple handlers
>>>>> to service the incoming requests.
>>>>>
>>>>> There are 2 clients on each node, and each client has a
>>>>> configurable number of fetcher threads. The fetcher code is written
>>>>> using the java.net.URLConnection API.
>>>>> Every node has both the server and the clients. The threads
>>>>> basically hit the HTTP server asking for specific files. They all
>>>>> do this at once (with some randomness in the order of hosts, maybe).
>>>>>
>>>>> The benchmark that I tested with is a sort for ~5TB of data with a
>>>>> cluster of 500 nodes. On the hardware side, I used a cluster that
>>>>> has
>>>>> 4 dualcore processors in each machine. The machines are partitioned
>>>>> into racks with a gigabit ethernet within the rack and 100Mbps
>>>>> across the racks. There are roughly 78000 independent files spread
>>>>> across these 500 nodes each of size ~60KB that the client pulls
>>>>> (and again we
>>>>>          
>>>> have two such clients per node).
>>>>        
>>>>> So you can imagine you have a massive all-all communication
>>>>> happening. The configuration for the server and client is as follows:
>>>>>  Grizzly configuration for port 9999
>>>>>          maxThreads: 100
>>>>>          minThreads: 1
>>>>>          ByteBuffer size: 8192
>>>>>          useDirectByteBuffer: false
>>>>>          useByteBufferView: false
>>>>>          maxHttpHeaderSize: 8192
>>>>>          maxKeepAliveRequests: 256
>>>>>          keepAliveTimeoutInSeconds: 10
>>>>>          Static File Cache enabled: true
>>>>>          Stream Algorithm :
>>>>> com.sun.grizzly.http.algorithms.NoParsingAlgorithm
>>>>>          Pipeline : com.sun.grizzly.http.LinkedListPipeline
>>>>>          Round Robin Selector Algorithm enabled: false
>>>>>          Round Robin Selector pool size: 0
>>>>>          recycleTasks: true
>>>>>          Asynchronous Request Processing enabled: false I also
>>>>> tried some configs with multiple selectorReadThreads but didn't
>>>>> make much difference.
>>>>>          
>>>> keepAliveTimeoutInSeconds: 10 seems for me a little low...what is
>>>> the reason to have such a low number? Closing faster idle connection?
>>>>
>>>>
>>>>        
>>>>> The client has 30 fetcher threads and the way it is designed is
>>>>> that only one fetch from any given host would happen at any point
>>>>> of time. So if a server host, h1, has 'n' files that we should
>>>>> pull, we do that one at a
>>>>>          
>>>> time
>>>>        
>>>>> (as opposed to multiple threads hitting that server to fetch
>>>>> multiple
>>>>>          
>>>> files
>>>>        
>>>>> in parallel).
>>>>>
>>>>> Also, we don't use the features of HTTP1.1 persistent connections
>>>>> or pipelining. We fetch exactly one file and close the connection
>>>>> to the server.
>>>>>          
>>>> That's answer my previous question :-) I would recommend setting the
>>>> keepAliveTimeoutInSeconds=0 then as I'm sure the performance will
>>>> improve (no call to the keep-alive subsystem).
>>>>
>>>>        
>>>>> With the above setup, the performance I see is not different from
>>>>> what I
>>>>>          
>>>> see
>>>>        
>>>>> with Jetty5.
>>>>>          
>>>> Can it be the benchmark itself that it not able to load more?
>>>>
>>>>
>>>> I see a lot of Read timeouts on the client side (and we have a
>>>>        
>>>>> backoff (for the server host) on the client implementation whenever
>>>>> we
>>>>>          
>>>> fail
>>>>        
>>>>> to fetch a file). I also saw some exceptions of the form on the server:
>>>>>          
>>>> You seem to hit the epoll problem on Linux. I know there is a way to
>>>> avoid using epoll (a property). I will ping the NIO team and let you
>>>> know.
>>>>
>>>> Also, what exactly your Adapter implementation is doing? Can you
>>>> share the code? If I can have access to your setup, I would like to
>>>> see if using Grizzly 1.0.15 makes a difference (just to make sure we
>>>> don't have a bug in 1.5....as far as I can tell, I.5 is as fast as
>>>> 1.0 on my benchmark).
>>>>
>>>> Thanks,
>>>>
>>>> --Jeanfrancois
>>>>
>>>>
>>>>        
>>>>> Jun 11, 2007 5:04:51 PM com.sun.grizzly.Controller doSelect
>>>>> SEVERE: doSelect exception
>>>>> java.io.IOException: Operation not permitted
>>>>>         at sun.nio.ch.EPollArrayWrapper.epollCtl(Native Method)
>>>>>         at
>>>>>
>>>>>          
>> sun.nio.ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.jav
>> a:202)
>>
>>    
>>>>>         at
>>>>>          
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:183)
>  
>>>>>         at
>>>>>          
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>        
>>>>>         at
>>>>>          
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  
>>>>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>         at
>>>>> com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:277)
>>>>>         at com.sun.grizzly.Controller.doSelect(Controller.java:218)
>>>>>         at com.sun.grizzly.Controller.start(Controller.java:451)
>>>>>         at
>>>>>
>>>>>          
>> com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:
>> 1158)
>>    
>>>>>         at
>>>>>
>>>>>          
>> com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:
>> 1121)
>>    
>>>>>         at
>>>>>          
>>>> com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1099)
>>>>        
>>>>> Are we missing something in the configuration or something else?
>>>>>
>>>>> Thanks for the help.
>>>>>
>>>>> Regards,
>>>>> Devaraj.
>>>>>
>>>>> -------------------------------------------------------------------
>>>>> -- To unsubscribe, e-mail: [hidden email]
>>>>> For additional commands, e-mail: [hidden email]
>>>>>
>>>>>          
>>>> --------------------------------------------------------------------
>>>> - To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>>
>>>> --------------------------------------------------------------------
>>>> - To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>>        
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>>      
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>    
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>  

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]