fragmented data. possible solution?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

fragmented data. possible solution?

Carlos Alexandre Queiroz
Hi there,

Please, be nice with me as it is my first post on this list and I am
not an expert on grizzly -:).

I wrote a very simple client/server app based on the examples posted
on grizzly website. the app works fine when the client does not send
too many messages to the server. What I mean about too much is, I have
5, 10 threads on the client side with a infinite loop send messages
with no stop. I only have a Thread.yield() method in the end of the
loop. Using this approach I start to get errors on the server as the
message does not arrive entirely. I am sending a java object wrapped
into a bytebuffer, when I try to convert the message to the java
object I got the error. However, if I change the yield method to the
sleep method with times around 100, 200, or more, the messages arrive
on the server side nicely with no error. This behaviour lead me to
think that the server is not being able to process too many messages,
even though I've defined the pipeline max threads to 20, 30, etc, that
is  higher than the threads set to the client.

Both apps (client and server) are running on the same machine. But, I
did some tests running on separate machines and I got the same
behaviour.  I've posted the code below, please let me know what could
be changed to fix this behaviour.

Another question:
My client app opens the connection on the startup, then I start to
send messages. However, I've notice that after some time the
connection closes automatically, maybe some timeout. So, I have to
open the connection again.
Is there some way to redefine this timeout?


thanks in advance,


Below there are some snippets of my code:

Server side, defining the controller, the pipeline and the filters:

controller = new Controller();
tcpHandler = new TCPSelectorHandler();
tcpHandler.setPort(localDetector.getPort());
Pipeline pipeline = new DefaultPipeline();
pipeline.setMaxThreads(threads * 10);
pipeline.setMinThreads(5);
controller.setPipeline(pipeline);
tcpHandler.setProtocolChainInstanceHandler(new
DefaultProtocolChainInstanceHandler()
{
    final ProtocolChain protocolChain = new DefaultProtocolChain();
    public ProtocolChain poll()
    {
         protocolChain.addFilter(new ReadFilter());
         protocolChain.addFilter(new MessageFilter());
         return protocolChain;
     }

     public boolean offer(ProtocolChain instance)
     {
         return true;
     }
});
controller.addSelectorHandler(tcpHandler);
controller.start();


Message Filter code, handles the message:

 final WorkerThread workerThread = ((WorkerThread) Thread.currentThread());
 ByteBuffer buffer = workerThread.getByteBuffer();
 buffer.flip();
 byte[] data = new byte[buffer.remaining()];
 int position = buffer.position();
 buffer.get(data);
 buffer.position(position);
 ByteArrayInputStream bais = new ByteArrayInputStream(data);
 ObjectInputStream ois = new ObjectInputStream(bais);
 final SDiACMessage message = (SDiACMessage) ois.readObject();


Client side, sending the message:

 buf.flip();
 if (!executor.connectorHandler.isConnected())
  {
        executor.connectorHandler.close();
        executor.connectorHandler = (TCPConnectorHandler) cController
                       .acquireConnectorHandler(Controller.Protocol.TCP);
        executor.connectorHandler.connect(new
InetSocketAddress(executor.hostname,
                   executor.port));
  }
  long size = executor.connectorHandler.write(buf, true);

--
thanks,

Carlos Alexandre Queiroz

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: fragmented data. possible solution?

Luiz Hamilton R L Soares
Hi,
 
I think your problem is here(Message filter):

byte[] data = new byte[buffer.remaining()];
int position = buffer.position();
buffer.get(data);
buffer.position(position);
ByteArrayInputStream bais = new ByteArrayInputStream(data);
 
If you didn't received all your data, you are going to have some problems here. You can also 1,2,3,4.... objects when you do that:byte[] data = new byte[buffer.remaining()];
 
Hope it helps,
 
Luiz Soares


Carlos Alexandre Queiroz <[hidden email]> escreveu:
Hi there,

Please, be nice with me as it is my first post on this list and I am
not an expert on grizzly -:).

I wrote a very simple client/server app based on the examples posted
on grizzly website. the app works fine when the client does not send
too many messages to the server. What I mean about too much is, I have
5, 10 threads on the client side with a infinite loop send messages
with no stop. I only have a Thread.yield() method in the end of the
loop. Using this approach I start to get errors on the server as the
message does not arrive entirely. I am sending a java object wrapped
into a bytebuffer, when I try to convert the message to the java
object I got the error. However, if I change the yield method to the
sleep method with times around 100, 200, or more, the messages arrive
on the server side nicely with no error. This behaviour lead me to
think that the server is not being able to process too many messages,
even though I've defined the pipeline max threads to 20, 30, etc, that
is higher than the threads set to the client.

Both apps (client and server) are running on the same machine. But, I
did some tests running on separate machines and I got the same
behaviour. I've posted the code below, please let me know what could
be changed to fix this behaviour.

Another question:
My client app opens the connection on the startup, then I start to
send messages. However, I've notice that after some time the
connection closes automatically, maybe some timeout. So, I have to
open the connection again.
Is there some way to redefine this timeout?


thanks in advance,


Below there are some snippets of my code:

Server side, defining the controller, the pipeline and the filters:

controller = new Controller();
tcpHandler = new TCPSelectorHandler();
tcpHandler.setPort(localDetector.getPort());
Pipeline pipeline = new DefaultPipeline();
pipeline.setMaxThreads(threads * 10);
pipeline.setMinThreads(5);
controller.setPipeline(pipeline);
tcpHandler.setProtocolChainInstanceHandler(new
DefaultProtocolChainInstanceHandler()
{
final ProtocolChain protocolChain = new DefaultProtocolChain();
public ProtocolChain poll()
{
protocolChain.addFilter(new ReadFilter());
protocolChain.addFilter(new MessageFilter());
return protocolChain;
}

public boolean offer(ProtocolChain instance)
{
return true;
}
});
controller.addSelectorHandler(tcpHandler);
controller.start();


Message Filter code, handles the message:

final WorkerThread workerThread = ((WorkerThread) Thread.currentThread());
ByteBuffer buffer = workerThread.getByteBuffer();
buffer.flip();
byte[] data = new byte[buffer.remaining()];
int position = buffer.position();
buffer.get(data);
buffer.position(position);
ByteArrayInputStream bais = new ByteArrayInputStream(data);
ObjectInputStream ois = new ObjectInputStream(bais);
final SDiACMessage message = (SDiACMessage) ois.readObject();


Client side, sending the message:

buf.flip();
if (!executor.connectorHandler.isConnected())
{
executor.connectorHandler.close();
executor.connectorHandler = (TCPConnectorHandler) cController
.acquireConnectorHandler(Controller.Protocol.TCP);
executor.connectorHandler.connect(new
InetSocketAddress(executor.hostname,
executor.port));
}
long size = executor.connectorHandler.write(buf, true);

--
thanks,

Carlos Alexandre Queiroz

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



Abra sua conta no Yahoo! Mail, o único sem limite de espaço para armazenamento!
Reply | Threaded
Open this post in threaded view
|

Re: fragmented data. possible solution?

Carlos Alexandre Queiroz
Hi Luiz,
Thanks for reply.

Sorry, but I did not understand what you mean about "you can also
1,2,3,4.. objects". Do you mean count the objects?

The problem is I don't know the size of the message, so I cannot
verify that. Maybe what I can do is to verify the end of the message
adding some special character. I am not sure if it works because I am
serialising a Java object, so this adding could mess up with the
object deserialisation. I'll give a try.

thanks in advance,



On 30/03/2008, Luiz Hamilton R L Soares <[hidden email]> wrote:

> Hi,
>
> I think your problem is here(Message filter):
>
> byte[] data = new byte[buffer.remaining()];
>  int position = buffer.position();
>  buffer.get(data);
>  buffer.position(position);
> ByteArrayInputStream bais = new ByteArrayInputStream(data);
>
> If you didn't received all your data, you are going to have some problems
> here. You can also 1,2,3,4.... objects when you do that:byte[] data = new
> byte[buffer.remaining()];
>
> Hope it helps,
>
> Luiz Soares
>
>
>
> Carlos Alexandre Queiroz <[hidden email]> escreveu:
> Hi there,
>
> Please, be nice with me as it is my first post on this list and I am
> not an expert on grizzly -:).
>
> I wrote a very simple client/server app based on the examples posted
> on grizzly website. the app works fine when the client does not send
> too many messages to the server. What I mean about too much is, I have
> 5, 10 threads on the client side with a infinite loop send messages
> with no stop. I only have a Thread.yield() method in the end of the
> loop. Using this approach I start to get errors on the server as the
> message does not arrive entirely. I am sending a java object wrapped
> into a bytebuffer, when I try to convert the message to the java
> object I got the error. However, if I change the yield method to the
> sleep method with times around 100, 200, or more, the messages arrive
> on the server side nicely with no error. This behaviour lead me to
> think that the server is not being able to process too many messages,
> even though I've defined the pipeline max threads to 20, 30, etc, that
> is higher than the threads set to the client.
>
> Both apps (client and server) are running on the same machine. But, I
> did some tests running on separate machines and I got the same
> behaviour. I've posted the code below, please let me know what could
> be changed to fix this behaviour.
>
> Another question:
> My client app opens the connection on the startup, then I start to
> send messages. However, I've notice that after some time the
> connection closes automatically, maybe some timeout. So, I have to
> open the connection again.
> Is there some way to redefine this timeout?
>
>
> thanks in advance,
>
>
> Below there are some snippets of my code:
>
> Server side, defining the controller, the pipeline and the filters:
>
> controller = new Controller();
> tcpHandler = new TCPSelectorHandler();
> tcpHandler.setPort(localDetector.getPort());
> Pipeline pipeline = new DefaultPipeline();
> pipeline.setMaxThreads(threads * 10);
> pipeline.setMinThreads(5);
> controller.setPipeline(pipeline);
> tcpHandler.setProtocolChainInstanceHandler(new
> DefaultProtocolChainInstanceHandler()
> {
> final ProtocolChain protocolChain = new DefaultProtocolChain();
> public ProtocolChain poll()
> {
> protocolChain.addFilter(new ReadFilter());
> protocolChain.addFilter(new MessageFilter());
> return protocolChain;
> }
>
> public boolean offer(ProtocolChain instance)
> {
> return true;
> }
> });
> controller.addSelectorHandler(tcpHandler);
> controller.start();
>
>
> Message Filter code, handles the message:
>
> final WorkerThread workerThread = ((WorkerThread) Thread.currentThread());
> ByteBuffer buffer = workerThread.getByteBuffer();
> buffer.flip();
> byte[] data = new byte[buffer.remaining()];
> int position = buffer.position();
> buffer.get(data);
> buffer.position(position);
> ByteArrayInputStream bais = new ByteArrayInputStream(data);
> ObjectInputStream ois = new ObjectInputStream(bais);
> final SDiACMessage message = (SDiACMessage) ois.readObject();
>
>
> Client side, sending the message:
>
> buf.flip();
> if (!executor.connectorHandler.isConnected())
> {
> executor.connectorHandler.close();
> executor.connectorHandler = (TCPConnectorHandler) cController
> .acquireConnectorHandler(Controller.Protocol.TCP);
> executor.connectorHandler.connect(new
> InetSocketAddress(executor.hostname,
> executor.port));
> }
> long size = executor.connectorHandler.write(buf, true);
>
> --
> thanks,
>
> Carlos Alexandre Queiroz
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
>
>
>  ________________________________
> Abra sua conta no Yahoo! Mail, o único sem limite de espaço para
> armazenamento!
>
>


--
thanks,

Carlos Alexandre Queiroz

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: fragmented data. possible solution?

Sebastien Dionne
Carlos, please post your solution.

I posted a message similar the that last week, but I wasn't using glizzly but plain NIO.  I used a buffer the read the input, and if the pattern "End of Query" is not found, I copy the buffer into a larger buffer and loop until a found the EOQ.

I would prefer having a specified the Objects that I can received, and be notified when I received one, and the rest is consider junk.. something like that could simplier the code the read.


2008/3/29, Carlos Alexandre Queiroz <[hidden email]>:
Hi Luiz,
Thanks for reply.

Sorry, but I did not understand what you mean about "you can also
1,2,3,4.. objects". Do you mean count the objects?

The problem is I don't know the size of the message, so I cannot
verify that. Maybe what I can do is to verify the end of the message
adding some special character. I am not sure if it works because I am
serialising a Java object, so this adding could mess up with the
object deserialisation. I'll give a try.

thanks in advance,




On 30/03/2008, Luiz Hamilton R L Soares <[hidden email]> wrote:
> Hi,
>
> I think your problem is here(Message filter):
>
> byte[] data = new byte[buffer.remaining()];
>  int position = buffer.position();
>  buffer.get(data);
>  buffer.position(position);
> ByteArrayInputStream bais = new ByteArrayInputStream(data);
>
> If you didn't received all your data, you are going to have some problems
> here. You can also 1,2,3,4.... objects when you do that:byte[] data = new
> byte[buffer.remaining()];
>
> Hope it helps,
>
> Luiz Soares
>
>
>
> Carlos Alexandre Queiroz <[hidden email]> escreveu:
> Hi there,
>
> Please, be nice with me as it is my first post on this list and I am
> not an expert on grizzly -:).
>
> I wrote a very simple client/server app based on the examples posted
> on grizzly website. the app works fine when the client does not send
> too many messages to the server. What I mean about too much is, I have
> 5, 10 threads on the client side with a infinite loop send messages
> with no stop. I only have a Thread.yield() method in the end of the
> loop. Using this approach I start to get errors on the server as the
> message does not arrive entirely. I am sending a java object wrapped
> into a bytebuffer, when I try to convert the message to the java
> object I got the error. However, if I change the yield method to the
> sleep method with times around 100, 200, or more, the messages arrive
> on the server side nicely with no error. This behaviour lead me to
> think that the server is not being able to process too many messages,
> even though I've defined the pipeline max threads to 20, 30, etc, that
> is higher than the threads set to the client.
>
> Both apps (client and server) are running on the same machine. But, I
> did some tests running on separate machines and I got the same
> behaviour. I've posted the code below, please let me know what could
> be changed to fix this behaviour.
>
> Another question:
> My client app opens the connection on the startup, then I start to
> send messages. However, I've notice that after some time the
> connection closes automatically, maybe some timeout. So, I have to
> open the connection again.
> Is there some way to redefine this timeout?
>
>
> thanks in advance,
>
>
> Below there are some snippets of my code:
>
> Server side, defining the controller, the pipeline and the filters:
>
> controller = new Controller();
> tcpHandler = new TCPSelectorHandler();
> tcpHandler.setPort(localDetector.getPort());
> Pipeline pipeline = new DefaultPipeline();
> pipeline.setMaxThreads(threads * 10);
> pipeline.setMinThreads(5);
> controller.setPipeline(pipeline);
> tcpHandler.setProtocolChainInstanceHandler(new
> DefaultProtocolChainInstanceHandler()
> {
> final ProtocolChain protocolChain = new DefaultProtocolChain();
> public ProtocolChain poll()
> {
> protocolChain.addFilter(new ReadFilter());
> protocolChain.addFilter(new MessageFilter());
> return protocolChain;
> }
>
> public boolean offer(ProtocolChain instance)
> {
> return true;
> }
> });
> controller.addSelectorHandler(tcpHandler);
> controller.start();
>
>
> Message Filter code, handles the message:
>
> final WorkerThread workerThread = ((WorkerThread) Thread.currentThread());
> ByteBuffer buffer = workerThread.getByteBuffer();
> buffer.flip();
> byte[] data = new byte[buffer.remaining()];
> int position = buffer.position();
> buffer.get(data);
> buffer.position(position);
> ByteArrayInputStream bais = new ByteArrayInputStream(data);
> ObjectInputStream ois = new ObjectInputStream(bais);
> final SDiACMessage message = (SDiACMessage) ois.readObject();
>
>
> Client side, sending the message:
>
> buf.flip();
> if (!executor.connectorHandler.isConnected())
> {
> executor.connectorHandler.close();
> executor.connectorHandler = (TCPConnectorHandler) cController
> .acquireConnectorHandler(Controller.Protocol.TCP);
> executor.connectorHandler.connect(new
> InetSocketAddress(executor.hostname,
> executor.port));
> }
> long size = executor.connectorHandler.write(buf, true);
>
> --
> thanks,
>
> Carlos Alexandre Queiroz
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
>
>
>  ________________________________
> Abra sua conta no Yahoo! Mail, o único sem limite de espaço para
> armazenamento!
>
>


--
thanks,

Carlos Alexandre Queiroz

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: fragmented data. possible solution?

Oleksiy Stashok
In reply to this post by Carlos Alexandre Queiroz
Hello Carlos,

think Luiz meant, that you don't have mechanism of recognition the end  
of message.
That's why in your situation it is possible to read either half of  
message at one time - and you will not be able to deserialize it; or  
you can read several messages (1,2,3,4 objects) at once, it will mean  
you need to parse them some specific way again.
Here are 2 simple ways, how you can send messages to be able to split  
them on receiver side:
(1) The simplest solution you can try - is to send message length  
before sending the message. This will mean, that you need to buffer  
your message before sending it to know its length.
(2) You can use some <END OF MESSAGE> symbol(s) and split incoming  
bytes to messages using that symbol(s).

If you're sending small messages - it's better to use (1), if messages  
are becoming bigger (2), IMHO, is preferable.

Thanks.

WBR,
Alexey.

On Mar 30, 2008, at 0:18 , Carlos Alexandre Queiroz wrote:

> Hi Luiz,
> Thanks for reply.
>
> Sorry, but I did not understand what you mean about "you can also
> 1,2,3,4.. objects". Do you mean count the objects?
>
> The problem is I don't know the size of the message, so I cannot
> verify that. Maybe what I can do is to verify the end of the message
> adding some special character. I am not sure if it works because I am
> serialising a Java object, so this adding could mess up with the
> object deserialisation. I'll give a try.
>
> thanks in advance,
>
>
>
> On 30/03/2008, Luiz Hamilton R L Soares  
> <[hidden email]> wrote:
>> Hi,
>>
>> I think your problem is here(Message filter):
>>
>> byte[] data = new byte[buffer.remaining()];
>> int position = buffer.position();
>> buffer.get(data);
>> buffer.position(position);
>> ByteArrayInputStream bais = new ByteArrayInputStream(data);
>>
>> If you didn't received all your data, you are going to have some  
>> problems
>> here. You can also 1,2,3,4.... objects when you do that:byte[] data  
>> = new
>> byte[buffer.remaining()];
>>
>> Hope it helps,
>>
>> Luiz Soares
>>
>>
>>
>> Carlos Alexandre Queiroz <[hidden email]> escreveu:
>> Hi there,
>>
>> Please, be nice with me as it is my first post on this list and I am
>> not an expert on grizzly -:).
>>
>> I wrote a very simple client/server app based on the examples posted
>> on grizzly website. the app works fine when the client does not send
>> too many messages to the server. What I mean about too much is, I  
>> have
>> 5, 10 threads on the client side with a infinite loop send messages
>> with no stop. I only have a Thread.yield() method in the end of the
>> loop. Using this approach I start to get errors on the server as the
>> message does not arrive entirely. I am sending a java object wrapped
>> into a bytebuffer, when I try to convert the message to the java
>> object I got the error. However, if I change the yield method to the
>> sleep method with times around 100, 200, or more, the messages arrive
>> on the server side nicely with no error. This behaviour lead me to
>> think that the server is not being able to process too many messages,
>> even though I've defined the pipeline max threads to 20, 30, etc,  
>> that
>> is higher than the threads set to the client.
>>
>> Both apps (client and server) are running on the same machine. But, I
>> did some tests running on separate machines and I got the same
>> behaviour. I've posted the code below, please let me know what could
>> be changed to fix this behaviour.
>>
>> Another question:
>> My client app opens the connection on the startup, then I start to
>> send messages. However, I've notice that after some time the
>> connection closes automatically, maybe some timeout. So, I have to
>> open the connection again.
>> Is there some way to redefine this timeout?
>>
>>
>> thanks in advance,
>>
>>
>> Below there are some snippets of my code:
>>
>> Server side, defining the controller, the pipeline and the filters:
>>
>> controller = new Controller();
>> tcpHandler = new TCPSelectorHandler();
>> tcpHandler.setPort(localDetector.getPort());
>> Pipeline pipeline = new DefaultPipeline();
>> pipeline.setMaxThreads(threads * 10);
>> pipeline.setMinThreads(5);
>> controller.setPipeline(pipeline);
>> tcpHandler.setProtocolChainInstanceHandler(new
>> DefaultProtocolChainInstanceHandler()
>> {
>> final ProtocolChain protocolChain = new DefaultProtocolChain();
>> public ProtocolChain poll()
>> {
>> protocolChain.addFilter(new ReadFilter());
>> protocolChain.addFilter(new MessageFilter());
>> return protocolChain;
>> }
>>
>> public boolean offer(ProtocolChain instance)
>> {
>> return true;
>> }
>> });
>> controller.addSelectorHandler(tcpHandler);
>> controller.start();
>>
>>
>> Message Filter code, handles the message:
>>
>> final WorkerThread workerThread = ((WorkerThread)  
>> Thread.currentThread());
>> ByteBuffer buffer = workerThread.getByteBuffer();
>> buffer.flip();
>> byte[] data = new byte[buffer.remaining()];
>> int position = buffer.position();
>> buffer.get(data);
>> buffer.position(position);
>> ByteArrayInputStream bais = new ByteArrayInputStream(data);
>> ObjectInputStream ois = new ObjectInputStream(bais);
>> final SDiACMessage message = (SDiACMessage) ois.readObject();
>>
>>
>> Client side, sending the message:
>>
>> buf.flip();
>> if (!executor.connectorHandler.isConnected())
>> {
>> executor.connectorHandler.close();
>> executor.connectorHandler = (TCPConnectorHandler) cController
>> .acquireConnectorHandler(Controller.Protocol.TCP);
>> executor.connectorHandler.connect(new
>> InetSocketAddress(executor.hostname,
>> executor.port));
>> }
>> long size = executor.connectorHandler.write(buf, true);
>>
>> --
>> thanks,
>>
>> Carlos Alexandre Queiroz
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>>
>>
>> ________________________________
>> Abra sua conta no Yahoo! Mail, o único sem limite de espaço para
>> armazenamento!
>>
>>
>
>
> --
> thanks,
>
> Carlos Alexandre Queiroz
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]