Async Writes & recommendations

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Async Writes & recommendations

D. J. Hagberg (Sun)
Hi all --

Jeanfrancois and I had exchanged a couple emails but he thought some of
my questions would be better addressed by the larger group of users on
this mailing list.  So here goes for my first one:

I have all the data input processing working for my application, using
Controller and a ProtocolChain with SSLReadFilter and a custom Filter
for my application's binary message protocol.  All is well with this --
I have added partial-read support in my Filter and am happily
dispatching messages in my WorkerThreadImpl's.

The question is about how to deal with write's.  My app is essentially a
message queue, with mostly short binary messages (usu between 40-400
bytes).  Right now, I'm writing messages back out using SSLOutputWriter
in a synchronous fashion inside Callable's that are running in the
WorkerThreadImpl's.

It looks like the Grizzly framework and NIO in general supports
asynchronous writes, but there do not seem to be many examples of that,
confirmed by JFA.  It looks like most of the http response output is all
done synchronously, which makes sense due to inherent limitations in
following the javax.servlet specification.

It seems to me that tying up precious WorkerThread resources waiting for
synchronous write's to complete (where one is not bound by  the
javax.servlet specification) is a bit wasteful and limiting in
scalability, especially if I have a client on the other end of a slow
connection, but I don't have any hard numbers to back me up on this.

In my application, all the events and messages can be handled
asynchronously, so I can build up a queue of outgoing Message objects
for each connection.

One possibility is to *always* have an object waiting for OP_WRITE
events managed by the Controller and SelectorThread and to have that
object basically "spin" waiting for messages or byte chunks to be
written.  This seems inefficient at a gut level to me, especially if
waiting for messages/byte chunks involves any synchronization.

The other possibility would seem to be to *only* register an OP_WRITE
interest when there are messages in the outbound queue that need to be
written, triggered whenever a message is added to the outbound queue.
But in this case, there is an expense involved with waking up the
selector thread and registering/updating selection keys, etc.

Has anyone plowed into this area or am I breaking new ground here?

Thanks for any advice,

                        -=- D. J.

BTW, this is all for the back-end server scalability for
http://www.sun.com/sharedshell.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Async Writes & recommendations

Robert Greig
On 13/06/07, D. J. Hagberg (Sun) <[hidden email]> wrote:

> It seems to me that tying up precious WorkerThread resources waiting for
> synchronous write's to complete (where one is not bound by  the
> javax.servlet specification) is a bit wasteful and limiting in
> scalability, especially if I have a client on the other end of a slow
> connection, but I don't have any hard numbers to back me up on this.

I work on the Apache Qpid project, which is an implementation of the
AMQP protocol (for message oriented middleware). We don't use Grizzly
yet but I have spent a lot of time looking at this area for Qpid.

What is your primary goal? Is it to maximise the number of connections
that can be serviced concurrently or to maximise the data throughput
to clients? Also, is it important for clients to be able to read data
as they are sending it or is it more like http where they send then
receive?

With Qpid we were trying to maximise the throughput, while servicing a
reasonable number of connections (e.g. 2000 connections not 20,000).
It was also extremely important that we could write responses to
client at the same time that we read from them.

Are you in control of the client code as well as the server? One thing
we had to consider was memory usage - i.e. ensuring that the queues
did not get too big either due to badly written or just slow clients.

> The other possibility would seem to be to *only* register an OP_WRITE
> interest when there are messages in the outbound queue that need to be
> written, triggered whenever a message is added to the outbound queue.
> But in this case, there is an expense involved with waking up the
> selector thread and registering/updating selection keys, etc.

In our design, we have several I/O threads, half of which are
responsible for reading and half for writing. Connections are assigned
a thread on a round robin basis.

There is certainly an expense involved with waking up the thread.
However, OP_WRITE interest only needs to be registered when the kernel
buffer is full, and this is obviously partly dependent on how quickly
your clients are processing data.

The first design we had, threads were responsible for both reads and
writes but we wanted to be able to read and write from a socket at the
same time. We found it significantly reduced the memory usage of our
app since the build-up in the queues was far lower.

Sorry for providing more questions than answers but I think this is
such a "delicate" area.

RG

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Async Writes & recommendations

D. J. Hagberg (Sun)
Thanks for the questions, Robert -- all good I think.

Robert Greig wrote:

> On 13/06/07, D. J. Hagberg (Sun) <[hidden email]> wrote:
>> It seems to me that tying up precious WorkerThread resources waiting for
>> synchronous write's to complete (where one is not bound by  the
>> javax.servlet specification) is a bit wasteful and limiting in
>> scalability, especially if I have a client on the other end of a slow
>> connection, but I don't have any hard numbers to back me up on this.
>
> I work on the Apache Qpid project, which is an implementation of the
> AMQP protocol (for message oriented middleware). We don't use Grizzly
> yet but I have spent a lot of time looking at this area for Qpid.

It sounds like the protocols and goals may be similar.  Basically the
Shared Shell server acts as a rendesvous point for messaging among
clients interested in a particular "topic" (in this case a VT100-type
shell session on a server).

> What is your primary goal? Is it to maximise the number of connections
> that can be serviced concurrently or to maximise the data throughput
> to clients? Also, is it important for clients to be able to read data
> as they are sending it or is it more like http where they send then
> receive?

The goals are:
1. To keep all the content un-snoopable and ensure no man-in-the-middle
attacks.  So we're using SSL over TCP.
2. To scale to a "reasonable" number of concurrent connections
per-server, say 2000-4000.
3. To minimize latency in the round-trip times -- being a terminal style
client application, seeing keystroke results in sub-second time is expected.

> Are you in control of the client code as well as the server? One thing
> we had to consider was memory usage - i.e. ensuring that the queues
> did not get too big either due to badly written or just slow clients.

Yes, we control both clients and servers -- the lower-level parts of the
code are shared between client and server, and our protocol has a
"revision" number at the start of every message.  This rewrite is going
to bump the message revision number anyway, creating an incompatible
change.  Clients will auto-update with Java Web Start so it's not a big
deal to keep the two in sync.

The only constraint is that our client code still needs to support JDK
1.4.2.  Our Server code, which is what I am primarily concerned with
right now, will be using 1.5 or 1.6.

>> The other possibility would seem to be to *only* register an OP_WRITE
>> interest when there are messages in the outbound queue that need to be
>> written, triggered whenever a message is added to the outbound queue.
>> But in this case, there is an expense involved with waking up the
>> selector thread and registering/updating selection keys, etc.
>
> In our design, we have several I/O threads, half of which are
> responsible for reading and half for writing. Connections are assigned
> a thread on a round robin basis.

It looks like the Grizzly code has a single Selector thread for the
whole instance, then delegates to WorkerThread's to do the actual
reading and writing.  I'm pretty sure I want to split read and write
work for the exact reasons you mention (slow clients, possibly better
throughput).

> There is certainly an expense involved with waking up the thread.
> However, OP_WRITE interest only needs to be registered when the kernel
> buffer is full, and this is obviously partly dependent on how quickly
> your clients are processing data.

Makes sense.

> The first design we had, threads were responsible for both reads and
> writes but we wanted to be able to read and write from a socket at the
> same time. We found it significantly reduced the memory usage of our
> app since the build-up in the queues was far lower.

Again, I think we are on similar tracks here -- our clients can read &
write at the same time and expect to process every message
asynchronously.  The "old" design that I am chucking out here actually
had a TON of threads to handle this with old-school blocking I/O and
wait/notify on message and work queues.

> Sorry for providing more questions than answers but I think this is
> such a "delicate" area.

Agreed, and it makes a significant difference on the nature of the
protocol.  Plus, it sounds like both our applications have a behavior
that is very much divergent from HTTP's typical request/response/close
behavior.  Interesting, though, that AJAX is moving (abusing?) HTTP to
more of an asynch. messaging protocol...

Thanks for the discussion.  I'll take a look at the ideas in Qpid/AMQP
when I get a chance.

                        -=- D. J.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Async Writes & recommendations

Jeanfrancois Arcand-2
Hi,

Jumping :-)

D. J. Hagberg (Sun) wrote:

> Thanks for the questions, Robert -- all good I think.
>
> Robert Greig wrote:
>> On 13/06/07, D. J. Hagberg (Sun) <[hidden email]> wrote:
>>> It seems to me that tying up precious WorkerThread resources waiting for
>>> synchronous write's to complete (where one is not bound by  the
>>> javax.servlet specification) is a bit wasteful and limiting in
>>> scalability, especially if I have a client on the other end of a slow
>>> connection, but I don't have any hard numbers to back me up on this.
>>
>> I work on the Apache Qpid project, which is an implementation of the
>> AMQP protocol (for message oriented middleware). We don't use Grizzly
>> yet but I have spent a lot of time looking at this area for Qpid.
>
> It sounds like the protocols and goals may be similar.  Basically the
> Shared Shell server acts as a rendesvous point for messaging among
> clients interested in a particular "topic" (in this case a VT100-type
> shell session on a server).
>
>> What is your primary goal? Is it to maximise the number of connections
>> that can be serviced concurrently or to maximise the data throughput
>> to clients? Also, is it important for clients to be able to read data
>> as they are sending it or is it more like http where they send then
>> receive?
>
> The goals are:
> 1. To keep all the content un-snoopable and ensure no man-in-the-middle
> attacks.  So we're using SSL over TCP.
> 2. To scale to a "reasonable" number of concurrent connections
> per-server, say 2000-4000.
> 3. To minimize latency in the round-trip times -- being a terminal style
> client application, seeing keystroke results in sub-second time is
> expected.
>
>> Are you in control of the client code as well as the server? One thing
>> we had to consider was memory usage - i.e. ensuring that the queues
>> did not get too big either due to badly written or just slow clients.
>
> Yes, we control both clients and servers -- the lower-level parts of the
> code are shared between client and server, and our protocol has a
> "revision" number at the start of every message.  This rewrite is going
> to bump the message revision number anyway, creating an incompatible
> change.  Clients will auto-update with Java Web Start so it's not a big
> deal to keep the two in sync.
>
> The only constraint is that our client code still needs to support JDK
> 1.4.2.  Our Server code, which is what I am primarily concerned with
> right now, will be using 1.5 or 1.6.
>
>>> The other possibility would seem to be to *only* register an OP_WRITE
>>> interest when there are messages in the outbound queue that need to be
>>> written, triggered whenever a message is added to the outbound queue.
>>> But in this case, there is an expense involved with waking up the
>>> selector thread and registering/updating selection keys, etc.
>>
>> In our design, we have several I/O threads, half of which are
>> responsible for reading and half for writing. Connections are assigned
>> a thread on a round robin basis.
>
> It looks like the Grizzly code has a single Selector thread for the
> whole instance, then delegates to WorkerThread's to do the actual
> reading and writing.  I'm pretty sure I want to split read and write
> work for the exact reasons you mention (slow clients, possibly better
> throughput).

I would think implementing a new SelectorHandler (or two) can do the
job. One for OP_READ and OP_ACCEPT, and once dedicated to OP_WRITE. The
SelectorHandler that handle the OP_WRITE would have it own Pipeline
(Thread pool) and also its own ProtocolChain, which would only contains
one ProtocolFilter (an SSLWriteFilter). I don't think having two
Selector here will significantly impact performance (this is what I did
for supporting Comet (http long polling) in Grizzly.

The current Comet implementation currently support async read/write via
a second Selector. I suspect I can generalize the implementation so it
can be used outside of Comet. The current OutputWriter main purpose is
to mimic synchronous write using temporary Selector. But as part of the
sailfin project [1], the SIP container will need to do real async write
and using the current OutputWriter is not a solution. I'm currently
working with the Ericsson team (who are on that list as well) to replace
their current NIO implementation with Grizzly. One of their request is
to have async write. Hopefully we can come with a solution that apply to
all :-)


>
>> There is certainly an expense involved with waking up the thread.
>> However, OP_WRITE interest only needs to be registered when the kernel
>> buffer is full, and this is obviously partly dependent on how quickly
>> your clients are processing data.
>
> Makes sense.
>
>> The first design we had, threads were responsible for both reads and
>> writes but we wanted to be able to read and write from a socket at the
>> same time. We found it significantly reduced the memory usage of our
>> app since the build-up in the queues was far lower.
>
> Again, I think we are on similar tracks here -- our clients can read &
> write at the same time and expect to process every message
> asynchronously.  The "old" design that I am chucking out here actually
> had a TON of threads to handle this with old-school blocking I/O and
> wait/notify on message and work queues.
>
>> Sorry for providing more questions than answers but I think this is
>> such a "delicate" area.
>
> Agreed, and it makes a significant difference on the nature of the
> protocol.  Plus, it sounds like both our applications have a behavior
> that is very much divergent from HTTP's typical request/response/close
> behavior.  Interesting, though, that AJAX is moving (abusing?) HTTP to
> more of an asynch. messaging protocol...
>
> Thanks for the discussion.  I'll take a look at the ideas in Qpid/AMQP
> when I get a chance.

Robert, I still need to look at your code, but are you using a queue
with a dedicated thread pool for OP_WRITE or a second Selector (or
something completely different?)

Thanks

-- Jeanfrancois

[1] https://sailfin.dev.java.net/
>
>             -=- D. J.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Async Writes & recommendations

Robert Greig
On 13/06/07, Jeanfrancois Arcand <[hidden email]> wrote:

> Robert, I still need to look at your code, but are you using a queue
> with a dedicated thread pool for OP_WRITE or a second Selector (or
> something completely different?)

For each reader thread, we have a second thread doing writes, and a
second Selector used for OP_WRITE. A given thread (and selector) is
responsible for multiple sessions. There is a separate queue of
sessions that have data to flush (from which the writer thread pulls
off sessions).

IIRC we have logically something like this:

forever
{
while (getSessionWithDataToFlush from queue of sessions)
{
    write out data from session up to a "fairness" maximum of say 512k
    if kernel buffer fills up, register interest in OP_WRITE
    if kernel buffer hasn't filled up but we have still got data to
write out, put the session back on the queue
}
selector.select, and add any sessions to the queue of sessions which
can write out at least one byte
}

RG

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]