Grizzly stops working after UDP 5 packets

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Grizzly stops working after UDP 5 packets

Radim Kolar
in my dns server i have protocol filter chain set as follows:
ReadFilter()
DNSQueryFilter()
AuthoritativeDNSFilter()
ResponseMakerFilter()
UDPWriteFilter()

they are stateless, one class instance is used for serving all requests.
Problem is that Grizzly stops working after 5 in/out packets, most likely
he runs of available worker threads.

what i am doing wrong? Need i add some filter after UDPWriteFilter and
call some terminate function so grizzly can recycle worker thread?

UDPWriteFilter registers key in postExecute for OP_WRITE, is this right?
because my work flow read -> process -> write (terminate work, but prepare
to read next packet again, not write more data). so after my job, key should
be registered in select set only for OP_READ.

btw: my dns server is 3x faster than bind 9.3.3

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Grizzly stops working after UDP 5 packets

Jeanfrancois Arcand-2
Hi,

Radim Kolar SF.NET wrote:

> in my dns server i have protocol filter chain set as follows:
> ReadFilter()
> DNSQueryFilter()
> AuthoritativeDNSFilter()
> ResponseMakerFilter()
> UDPWriteFilter()
>
> they are stateless, one class instance is used for serving all requests.
> Problem is that Grizzly stops working after 5 in/out packets, most likely
> he runs of available worker threads.
>
> what i am doing wrong? Need i add some filter after UDPWriteFilter and
> call some terminate function so grizzly can recycle worker thread?

Which JDK are you using and on which platform? Can you do a jstack
<PID>? Most likely it block in UDPWriteFilter line 78 (for probably 60
seconds). You might want to either reduce the timeout on the
OutputWriter or use the new async Write queue :-)

Thanks

-- Jeanfrancois

>
> UDPWriteFilter registers key in postExecute for OP_WRITE, is this right?
> because my work flow read -> process -> write (terminate work, but prepare
> to read next packet again, not write more data). so after my job, key should
> be registered in select set only for OP_READ.
>
> btw: my dns server is 3x faster than bind 9.3.3
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Grizzly stops working after UDP 5 packets

Radim Kolar
> Which JDK are you using and on which platform? Can you do a jstack <PID>?
> Most likely it block in UDPWriteFilter line 78 (for probably 60 seconds).
> You might want to either reduce the timeout on the OutputWriter or use the
> new async Write queue :-)
no, all worker threads are blocked at waitForTask():292 and main thread
is blocked in select()

server socket is not closed according to netstat -a

i think problem is that UDP support not very suitable for writing servers.

1. udpwritefilter should query UDPSelectorHandler(boolean isClient) and in
server mode avoid closing channel on errors

2. because all Contexts share same SelectionKey() there must be
races between worker threads playing with registering key selectorHandler.register. UDPWriter registers OP_WRITE and Reader OP_READ.

BTW is there reason why UDPWriter registers in postExecute OP_WRITE?
this may be useful only if UDPWriter is at top of protocol chain, but writer
needs to get data to write from somewhere so its unlikely to be first in chain.

I will write my own version of UDPWriter (not closing socket and not registering OP_WRITE) and test it.

This is most likely cause of deadlock in my app because one worker deregisters
OP_READ from server channel and all packets starts to be ignored.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Grizzly stops working after UDP 5 packets

Oleksiy Stashok
Hello Radim,

if you will be interested, please try to use AsyncRead/Write queues.
Think, for UDP usecase, they could be very useful.
For sure their implementation was recently done, so can not be sure 100%
it works without issues.

Thanks.

WBR,
Alexey.

Radim Kolar SF.NET wrote:

>> Which JDK are you using and on which platform? Can you do a jstack <PID>?
>> Most likely it block in UDPWriteFilter line 78 (for probably 60 seconds).
>> You might want to either reduce the timeout on the OutputWriter or use the
>> new async Write queue :-)
>>    
> no, all worker threads are blocked at waitForTask():292 and main thread
> is blocked in select()
>
> server socket is not closed according to netstat -a
>
> i think problem is that UDP support not very suitable for writing servers.
>
> 1. udpwritefilter should query UDPSelectorHandler(boolean isClient) and in
> server mode avoid closing channel on errors
>
> 2. because all Contexts share same SelectionKey() there must be
> races between worker threads playing with registering key selectorHandler.register. UDPWriter registers OP_WRITE and Reader OP_READ.
>
> BTW is there reason why UDPWriter registers in postExecute OP_WRITE?
> this may be useful only if UDPWriter is at top of protocol chain, but writer
> needs to get data to write from somewhere so its unlikely to be first in chain.
>
> I will write my own version of UDPWriter (not closing socket and not registering OP_WRITE) and test it.
>
> This is most likely cause of deadlock in my app because one worker deregisters
> OP_READ from server channel and all packets starts to be ignored.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>  

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Grizzly stops working after UDP 5 packets

Jeanfrancois Arcand-2
In reply to this post by Radim Kolar
Radim,

Radim Kolar SF.NET wrote:

>> Which JDK are you using and on which platform? Can you do a jstack <PID>?
>> Most likely it block in UDPWriteFilter line 78 (for probably 60 seconds).
>> You might want to either reduce the timeout on the OutputWriter or use the
>> new async Write queue :-)
> no, all worker threads are blocked at waitForTask():292 and main thread
> is blocked in select()
>
> server socket is not closed according to netstat -a
>
> i think problem is that UDP support not very suitable for writing servers.
>
> 1. udpwritefilter should query UDPSelectorHandler(boolean isClient) and in
> server mode avoid closing channel on errors
>
> 2. because all Contexts share same SelectionKey() there must be
> races between worker threads playing with registering key selectorHandler.register. UDPWriter registers OP_WRITE and Reader OP_READ.

Hum...I agree :-)

>
> BTW is there reason why UDPWriter registers in postExecute OP_WRITE?
> this may be useful only if UDPWriter is at top of protocol chain, but writer
> needs to get data to write from somewhere so its unlikely to be first in chain.
>
> I will write my own version of UDPWriter (not closing socket and not registering OP_WRITE) and test it.
>
> This is most likely cause of deadlock in my app because one worker deregisters
> OP_READ from server channel and all packets starts to be ignored.

OK I guess the Write filter needs to be re-worked :-) Let me take a look
today.

A+

-- Jeanfrancois


>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

reworked UPDWriter

Radim Kolar
> OK I guess the Write filter needs to be re-worked :-) Let me take a look
> today.
I reworked writer as follows:

UDPWriteFilter f4 = new UDPWriteFilter() {
        @Override
        public boolean postExecute(Context ctx) throws IOException {
                ctx.setKeyRegistrationState(Context.KeyRegistrationState.REGISTER);
                ((WorkerThread)Thread.currentThread()).getByteBuffer().clear();

                return true;
}

and my app starts working nicely without known race conditions.

Can grizzly framework increase number of buffered arriving UDP packets so
no packets gets lost on traffic spikes?

I suspect that Grizzly can serve more than 1 packet at once using different
threads, right? I have to check app and framework for more possible race
conditions.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: reworked UPDWriter

Jeanfrancois Arcand-2


Radim Kolar SF.NET wrote:

>> OK I guess the Write filter needs to be re-worked :-) Let me take a look
>> today.
> I reworked writer as follows:
>
> UDPWriteFilter f4 = new UDPWriteFilter() {
> @Override
>         public boolean postExecute(Context ctx) throws IOException {
> ctx.setKeyRegistrationState(Context.KeyRegistrationState.REGISTER);
> ((WorkerThread)Thread.currentThread()).getByteBuffer().clear();
>
>        return true;
> }
>
> and my app starts working nicely without known race conditions.

Your change looks good. I will incorporate it.


>
> Can grizzly framework increase number of buffered arriving UDP packets so
> no packets gets lost on traffic spikes?
>
> I suspect that Grizzly can serve more than 1 packet at once using different
> threads, right? I have to check app and framework for more possible race
> conditions.

Can you try it by applying the following:

UDPSelectorHandler ush = new UDPSelectorHandler(){

     /**
      * Handle OP_READ.
      * @param ctx <code>Context</code>
      * @param key <code>SelectionKey</code>
      * @return false if handled by a <code>CallbackHandler</code>,
otherwise true
      */
     public boolean onReadInterest(final SelectionKey key,final Context ctx)
     throws IOException{
         // Keep the packet coming
         //key.interestOps(key.interestOps() & (~SelectionKey.OP_READ));

         Object attach = key.attachment();

         if (asyncQueueReader.isAsyncQueueReaderEnabledFor(key)) {
             final Context context = pollContext(ctx, key);
             context.setCurrentOpType(Context.OpType.OP_READ);
             invokeAsyncQueueReader(context);
             return false;
         } else if (attach instanceof CallbackHandler){
             final Context context = pollContext(ctx, key);
             context.setCurrentOpType(Context.OpType.OP_READ);
             invokeCallbackHandler((CallbackHandler) attach, context);
             return false;
         } else {
             return true;
         }
     }

}

and let us know if the performance improved? What I did is commented out
the key.interestOps(key.interestOps() & (~SelectionKey.OP_READ)), which
means as soon as OP_READ are available, the Selector will wakes up. I
suspect this is what you need.

Thanks!

-- Jeanfrancois


>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]