who can help me?how a server base grizzly receive 2MB data from client

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

who can help me?how a server base grizzly receive 2MB data from client

windshome
I write a server base grizzly,I write a client which send data to server and receive response from server. I send 50 bytes data and 2k data to server ok,but when I send 20K and 2MB data,server no response.
I view the code of grizzly,find the ByteBuffer of a workthread capity is 8192,then I modify it to 81920,then 20k data can receive by server.

If my Server set the init size of ByteBuffer is 81920,I think it would use to much memory,who can tell me some method,will dynamic just the buffer,can receive some bytes data ,or some MB data?



My Server's protocol parser code :


        final ProtocolFilter parserProtocolFilter = new ParserProtocolFilter() {
            public ProtocolParser newProtocolParser() {
                return new ProtocolParser() {
                    private boolean isExpectingMoreData = false;
                    private ByteBuffer byteBuffer;
                    private Request message;
                   
                    public boolean hasMoreBytesToParse() {
                        return false;
                    }
                   
                    public boolean isExpectingMoreData() {
                        return isExpectingMoreData;
                    }
                   
                    public Object getNextMessage() {
                        return message;
                    }

                    public boolean hasNextMessage() {
                   
                   
                        ByteBuffer dup = byteBuffer.duplicate();
                        System.out.println( "byteBuffer.position2()=" + byteBuffer.position() );
                        if (byteBuffer.position() == 0){
                            System.out.println("byteBuffer.position() == 0");
                        isExpectingMoreData = true;
                            return false;
                        }
                        dup.flip();
                       

                        if( dup.remaining()< 4 ){
                            isExpectingMoreData = true;
                            return false;                        
                        }

                        byte[] bs = new byte[4];
                        dup.get(bs);
                        int len= (bs[0] << 24) + (bs[1] << 16) + (bs[2] << 8) + (bs[3] << 0);
                       
                        if( dup.remaining() < len  ){
                            isExpectingMoreData = true;
                            return false;                        
                        }
                        byte[] data = new byte[len];
                        dup.get( data );
                       
                        try {
                                                        message = new ByteRequest(data);
                                                } catch (Exception e) {
                                                        e.printStackTrace();
                                                        message=null;
                                                        return false;
                                                }
                        return true;
                       
                    }

                    public void startBuffer(ByteBuffer bb) {
                        byteBuffer = bb;
                    }

                    public boolean releaseBuffer() {
                        byteBuffer = null;
                        message = null;
                        return false;
                    }
                   
                };
            }
        };      






Reply | Threaded
Open this post in threaded view
|

Re: who can help me?how a server base grizzly receive 2MB data from client

Jeanfrancois Arcand-2
Hi,

sorry for the delay...

windshome wrote:
> I write a server base grizzly,I write a client which send data to server and
> receive response from server. I send 50 bytes data and 2k data to server
> ok,but when I send 20K and 2MB data,server no response.
> I view the code of grizzly,find the ByteBuffer of a workthread capity is
> 8192,then I modify it to 81920,then 20k data can receive by server.
>
> If my Server set the init size of ByteBuffer is 81920,I think it would use
> to much memory,who can tell me some method,will dynamic just the buffer,can
> receive some bytes data ,or some MB data?

What are you doing with the bytes? Are you saving the bytes somewhere on
disk/db (freeing your memory) or you must keep them in memory?

Thanks

-- Jeanfrancois


>
>
>
> My Server's protocol parser code :
>
>
>         final ProtocolFilter parserProtocolFilter = new
> ParserProtocolFilter() {
>             public ProtocolParser newProtocolParser() {
>                 return new ProtocolParser() {
>                     private boolean isExpectingMoreData = false;
>                     private ByteBuffer byteBuffer;
>                     private Request message;
>                    
>                     public boolean hasMoreBytesToParse() {
>                         return false;
>                     }
>                    
>                     public boolean isExpectingMoreData() {
>                         return isExpectingMoreData;
>                     }
>                    
>                     public Object getNextMessage() {
>                         return message;
>                     }
>
>                     public boolean hasNextMessage() {
>                    
>                    
>                         ByteBuffer dup = byteBuffer.duplicate();
>                         System.out.println( "byteBuffer.position2()=" +
> byteBuffer.position() );
>                         if (byteBuffer.position() == 0){
>                             System.out.println("byteBuffer.position() ==
> 0");
>                         isExpectingMoreData = true;
>                             return false;
>                         }
>                         dup.flip();
>                        
>
>                         if( dup.remaining()< 4 ){
>                             isExpectingMoreData = true;
>                             return false;                        
>                         }
>
>                         byte[] bs = new byte[4];
>                         dup.get(bs);
>                         int len= (bs[0] << 24) + (bs[1] << 16) + (bs[2] <<
> 8) + (bs[3] << 0);
>                        
>                         if( dup.remaining() < len  ){
>                             isExpectingMoreData = true;
>                             return false;                        
>                         }
>                         byte[] data = new byte[len];
>                         dup.get( data );
>                        
>                         try {
> message = new ByteRequest(data);
> } catch (Exception e) {
> e.printStackTrace();
> message=null;
> return false;
> }
>                         return true;
>                        
>                     }
>
>                     public void startBuffer(ByteBuffer bb) {
>                         byteBuffer = bb;
>                     }
>
>                     public boolean releaseBuffer() {
>                         byteBuffer = null;
>                         message = null;
>                         return false;
>                     }
>                    
>                 };
>             }
>         };      
>
>
>
>
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: who can help me?how a server base grizzly receive 2MB data from client

windshome

I process the data to make a digital signature, and return signed data to client.

Jeanfrancois Arcand-2 wrote
Hi,

sorry for the delay...

windshome wrote:
> I write a server base grizzly,I write a client which send data to server and
> receive response from server. I send 50 bytes data and 2k data to server
> ok,but when I send 20K and 2MB data,server no response.
> I view the code of grizzly,find the ByteBuffer of a workthread capity is
> 8192,then I modify it to 81920,then 20k data can receive by server.
>
> If my Server set the init size of ByteBuffer is 81920,I think it would use
> to much memory,who can tell me some method,will dynamic just the buffer,can
> receive some bytes data ,or some MB data?

What are you doing with the bytes? Are you saving the bytes somewhere on
disk/db (freeing your memory) or you must keep them in memory?

Thanks

-- Jeanfrancois


>
>
>
> My Server's protocol parser code :
>
>
>         final ProtocolFilter parserProtocolFilter = new
> ParserProtocolFilter() {
>             public ProtocolParser newProtocolParser() {
>                 return new ProtocolParser() {
>                     private boolean isExpectingMoreData = false;
>                     private ByteBuffer byteBuffer;
>                     private Request message;
>                    
>                     public boolean hasMoreBytesToParse() {
>                         return false;
>                     }
>                    
>                     public boolean isExpectingMoreData() {
>                         return isExpectingMoreData;
>                     }
>                    
>                     public Object getNextMessage() {
>                         return message;
>                     }
>
>                     public boolean hasNextMessage() {
>                    
>                    
>                         ByteBuffer dup = byteBuffer.duplicate();
>                         System.out.println( "byteBuffer.position2()=" +
> byteBuffer.position() );
>                         if (byteBuffer.position() == 0){
>                             System.out.println("byteBuffer.position() ==
> 0");
>                         isExpectingMoreData = true;
>                             return false;
>                         }
>                         dup.flip();
>                        
>
>                         if( dup.remaining()< 4 ){
>                             isExpectingMoreData = true;
>                             return false;                        
>                         }
>
>                         byte[] bs = new byte[4];
>                         dup.get(bs);
>                         int len= (bs[0] << 24) + (bs[1] << 16) + (bs[2] <<
> 8) + (bs[3] << 0);
>                        
>                         if( dup.remaining() < len  ){
>                             isExpectingMoreData = true;
>                             return false;                        
>                         }
>                         byte[] data = new byte[len];
>                         dup.get( data );
>                        
>                         try {
> message = new ByteRequest(data);
> } catch (Exception e) {
> e.printStackTrace();
> message=null;
> return false;
> }
>                         return true;
>                        
>                     }
>
>                     public void startBuffer(ByteBuffer bb) {
>                         byteBuffer = bb;
>                     }
>
>                     public boolean releaseBuffer() {
>                         byteBuffer = null;
>                         message = null;
>                         return false;
>                     }
>                    
>                 };
>             }
>         };      
>
>
>
>
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@grizzly.dev.java.net
For additional commands, e-mail: users-help@grizzly.dev.java.net
Reply | Threaded
Open this post in threaded view
|

Re: who can help me?how a server base grizzly receive 2MB data from client

Jeanfrancois Arcand-2
Salut,

windshome wrote:
>
> I process the data to make a digital signature, and return signed data to
> client.

Interesting :-) The number of ByteBuffer created is equal to the number
of active Threads by default in Grizzly. Right now our default thread
pool doesn't purge inactive threads, so it may  or may not be a problem
if you need a lot of threads. You might want to replace the default
thread pool with a one from java.util.concurrent.* that can purge
inactive thread and their associated byte buffer.

Are you able to determine the size of your expected traffic? If your VM
is properly tuned (I will let Charlie gives some hints in case you are
interested) it shouldn't be a problem, assuming you don't need 1000
threads :-)

A+

-- Jeanfrancois


>
>
> Jeanfrancois Arcand-2 wrote:
>> Hi,
>>
>> sorry for the delay...
>>
>> windshome wrote:
>>> I write a server base grizzly,I write a client which send data to server
>>> and
>>> receive response from server. I send 50 bytes data and 2k data to server
>>> ok,but when I send 20K and 2MB data,server no response.
>>> I view the code of grizzly,find the ByteBuffer of a workthread capity is
>>> 8192,then I modify it to 81920,then 20k data can receive by server.
>>>
>>> If my Server set the init size of ByteBuffer is 81920,I think it would
>>> use
>>> to much memory,who can tell me some method,will dynamic just the
>>> buffer,can
>>> receive some bytes data ,or some MB data?
>> What are you doing with the bytes? Are you saving the bytes somewhere on
>> disk/db (freeing your memory) or you must keep them in memory?
>>
>> Thanks
>>
>> -- Jeanfrancois
>>
>>
>>>
>>>
>>> My Server's protocol parser code :
>>>
>>>
>>>         final ProtocolFilter parserProtocolFilter = new
>>> ParserProtocolFilter() {
>>>             public ProtocolParser newProtocolParser() {
>>>                 return new ProtocolParser() {
>>>                     private boolean isExpectingMoreData = false;
>>>                     private ByteBuffer byteBuffer;
>>>                     private Request message;
>>>                    
>>>                     public boolean hasMoreBytesToParse() {
>>>                         return false;
>>>                     }
>>>                    
>>>                     public boolean isExpectingMoreData() {
>>>                         return isExpectingMoreData;
>>>                     }
>>>                    
>>>                     public Object getNextMessage() {
>>>                         return message;
>>>                     }
>>>
>>>                     public boolean hasNextMessage() {
>>>                    
>>>                    
>>>                         ByteBuffer dup = byteBuffer.duplicate();
>>>                         System.out.println( "byteBuffer.position2()=" +
>>> byteBuffer.position() );
>>>                         if (byteBuffer.position() == 0){
>>>                             System.out.println("byteBuffer.position() ==
>>> 0");
>>>                         isExpectingMoreData = true;
>>>                             return false;
>>>                         }
>>>                         dup.flip();
>>>                        
>>>
>>>                         if( dup.remaining()< 4 ){
>>>                             isExpectingMoreData = true;
>>>                             return false;                        
>>>                         }
>>>
>>>                         byte[] bs = new byte[4];
>>>                         dup.get(bs);
>>>                         int len= (bs[0] << 24) + (bs[1] << 16) + (bs[2]
>>> <<
>>> 8) + (bs[3] << 0);
>>>                        
>>>                         if( dup.remaining() < len  ){
>>>                             isExpectingMoreData = true;
>>>                             return false;                        
>>>                         }
>>>                         byte[] data = new byte[len];
>>>                         dup.get( data );
>>>                        
>>>                         try {
>>> message = new ByteRequest(data);
>>> } catch (Exception e) {
>>> e.printStackTrace();
>>> message=null;
>>> return false;
>>> }
>>>                         return true;
>>>                        
>>>                     }
>>>
>>>                     public void startBuffer(ByteBuffer bb) {
>>>                         byteBuffer = bb;
>>>                     }
>>>
>>>                     public boolean releaseBuffer() {
>>>                         byteBuffer = null;
>>>                         message = null;
>>>                         return false;
>>>                     }
>>>                    
>>>                 };
>>>             }
>>>         };      
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: who can help me?how a server base grizzly receive 2MB data from client

windshome

Oh,I think you are not understand me, my problem is when the bytebuffer is full,but all my data hasn't read
complete(my data size is 2MB), how can i read all data?
Jeanfrancois Arcand-2 wrote
Salut,

windshome wrote:
>
> I process the data to make a digital signature, and return signed data to
> client.

Interesting :-) The number of ByteBuffer created is equal to the number
of active Threads by default in Grizzly. Right now our default thread
pool doesn't purge inactive threads, so it may  or may not be a problem
if you need a lot of threads. You might want to replace the default
thread pool with a one from java.util.concurrent.* that can purge
inactive thread and their associated byte buffer.

Are you able to determine the size of your expected traffic? If your VM
is properly tuned (I will let Charlie gives some hints in case you are
interested) it shouldn't be a problem, assuming you don't need 1000
threads :-)

A+

-- Jeanfrancois


>
>
> Jeanfrancois Arcand-2 wrote:
>> Hi,
>>
>> sorry for the delay...
>>
>> windshome wrote:
>>> I write a server base grizzly,I write a client which send data to server
>>> and
>>> receive response from server. I send 50 bytes data and 2k data to server
>>> ok,but when I send 20K and 2MB data,server no response.
>>> I view the code of grizzly,find the ByteBuffer of a workthread capity is
>>> 8192,then I modify it to 81920,then 20k data can receive by server.
>>>
>>> If my Server set the init size of ByteBuffer is 81920,I think it would
>>> use
>>> to much memory,who can tell me some method,will dynamic just the
>>> buffer,can
>>> receive some bytes data ,or some MB data?
>> What are you doing with the bytes? Are you saving the bytes somewhere on
>> disk/db (freeing your memory) or you must keep them in memory?
>>
>> Thanks
>>
>> -- Jeanfrancois
>>
>>
>>>
>>>
>>> My Server's protocol parser code :
>>>
>>>
>>>         final ProtocolFilter parserProtocolFilter = new
>>> ParserProtocolFilter() {
>>>             public ProtocolParser newProtocolParser() {
>>>                 return new ProtocolParser() {
>>>                     private boolean isExpectingMoreData = false;
>>>                     private ByteBuffer byteBuffer;
>>>                     private Request message;
>>>                    
>>>                     public boolean hasMoreBytesToParse() {
>>>                         return false;
>>>                     }
>>>                    
>>>                     public boolean isExpectingMoreData() {
>>>                         return isExpectingMoreData;
>>>                     }
>>>                    
>>>                     public Object getNextMessage() {
>>>                         return message;
>>>                     }
>>>
>>>                     public boolean hasNextMessage() {
>>>                    
>>>                    
>>>                         ByteBuffer dup = byteBuffer.duplicate();
>>>                         System.out.println( "byteBuffer.position2()=" +
>>> byteBuffer.position() );
>>>                         if (byteBuffer.position() == 0){
>>>                             System.out.println("byteBuffer.position() ==
>>> 0");
>>>                         isExpectingMoreData = true;
>>>                             return false;
>>>                         }
>>>                         dup.flip();
>>>                        
>>>
>>>                         if( dup.remaining()< 4 ){
>>>                             isExpectingMoreData = true;
>>>                             return false;                        
>>>                         }
>>>
>>>                         byte[] bs = new byte[4];
>>>                         dup.get(bs);
>>>                         int len= (bs[0] << 24) + (bs[1] << 16) + (bs[2]
>>> <<
>>> 8) + (bs[3] << 0);
>>>                        
>>>                         if( dup.remaining() < len  ){
>>>                             isExpectingMoreData = true;
>>>                             return false;                        
>>>                         }
>>>                         byte[] data = new byte[len];
>>>                         dup.get( data );
>>>                        
>>>                         try {
>>> message = new ByteRequest(data);
>>> } catch (Exception e) {
>>> e.printStackTrace();
>>> message=null;
>>> return false;
>>> }
>>>                         return true;
>>>                        
>>>                     }
>>>
>>>                     public void startBuffer(ByteBuffer bb) {
>>>                         byteBuffer = bb;
>>>                     }
>>>
>>>                     public boolean releaseBuffer() {
>>>                         byteBuffer = null;
>>>                         message = null;
>>>                         return false;
>>>                     }
>>>                    
>>>                 };
>>>             }
>>>         };      
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@grizzly.dev.java.net
>> For additional commands, e-mail: users-help@grizzly.dev.java.net
>>
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@grizzly.dev.java.net
For additional commands, e-mail: users-help@grizzly.dev.java.net
Reply | Threaded
Open this post in threaded view
|

Re: who can help me?how a server base grizzly receive 2MB data from client

Harsha Godugu
windshome wrote:
Oh,I think you are not understand me, my problem is when the bytebuffer is
full,but all my data hasn't read 
complete(my data size is 2MB), how can i read all data?
Hi:   Please note that the bytebuffer size that is used for i/o here, in Grizzly, will have a constant threshold. That means, say, if the application can send /receive data (on both server/client side), the size of such a buffer during the process could vary from zero to the threshold.  This threshold has got a limit. It can be 8K, 16K or 256k to the max... based on the system limits and application needs.  It can not be 8Mb or some gig..  The idea is,  the bytebuffer that we use in grizzly (for i/o) is really a BUFFER, meaning a temporary place to read /write to/from a channel. That means, the underlying application needs to store the flushed  bits somewhere else.  So, as soon as you read say 8K bytes and then if expect more to read, we need store the read bytes somewhere else (in some other object/file/shared memory..etc) and then wait to read the rest of the data until your protocol handler tells you that there are no next messages to parse! 

hths
-Harsha
Jeanfrancois Arcand-2 wrote:
  
Salut,

windshome wrote:
    
I process the data to make a digital signature, and return signed data to
client.
      
Interesting :-) The number of ByteBuffer created is equal to the number 
of active Threads by default in Grizzly. Right now our default thread 
pool doesn't purge inactive threads, so it may  or may not be a problem 
if you need a lot of threads. You might want to replace the default 
thread pool with a one from java.util.concurrent.* that can purge 
inactive thread and their associated byte buffer.

Are you able to determine the size of your expected traffic? If your VM 
is properly tuned (I will let Charlie gives some hints in case you are 
interested) it shouldn't be a problem, assuming you don't need 1000 
threads :-)

A+

-- Jeanfrancois


    
Jeanfrancois Arcand-2 wrote:
      
Hi,

sorry for the delay...

windshome wrote:
        
I write a server base grizzly,I write a client which send data to
server
and
receive response from server. I send 50 bytes data and 2k data to
server
ok,but when I send 20K and 2MB data,server no response.
I view the code of grizzly,find the ByteBuffer of a workthread capity
is
8192,then I modify it to 81920,then 20k data can receive by server.

If my Server set the init size of ByteBuffer is 81920,I think it would
use
to much memory,who can tell me some method,will dynamic just the
buffer,can
receive some bytes data ,or some MB data?
          
What are you doing with the bytes? Are you saving the bytes somewhere on 
disk/db (freeing your memory) or you must keep them in memory?

Thanks

-- Jeanfrancois


        
My Server's protocol parser code :


        final ProtocolFilter parserProtocolFilter = new
ParserProtocolFilter() {
            public ProtocolParser newProtocolParser() {
                return new ProtocolParser() {
                    private boolean isExpectingMoreData = false;
                    private ByteBuffer byteBuffer;
                    private Request message;
                    
                    public boolean hasMoreBytesToParse() {
                        return false;
                    }
                    
                    public boolean isExpectingMoreData() {
                        return isExpectingMoreData;
                    }
                    
                    public Object getNextMessage() {
                        return message;
                    }

                    public boolean hasNextMessage() {
                    	
                    	
                        ByteBuffer dup = byteBuffer.duplicate();
                        System.out.println( "byteBuffer.position2()=" +
byteBuffer.position() );
                        if (byteBuffer.position() == 0){
                            System.out.println("byteBuffer.position()
==
0");
                        	isExpectingMoreData = true;
                            return false;
                        }
                        dup.flip();
                        

                        if( dup.remaining()< 4 ){
                            isExpectingMoreData = true;
                            return false;                        	
                        }

                        byte[] bs = new byte[4];
                        dup.get(bs);
                        int len= (bs[0] << 24) + (bs[1] << 16) + (bs[2]
<<
8) + (bs[3] << 0);
                       
                        if( dup.remaining() < len  ){
                            isExpectingMoreData = true;
                            return false;                        	
                        }
                        byte[] data = new byte[len];
                        dup.get( data );
                        
                        try {
							message = new ByteRequest(data);
						} catch (Exception e) {
							e.printStackTrace();
							message=null;
							return false;
						}
                        return true;
                        
                    }

                    public void startBuffer(ByteBuffer bb) {
                        byteBuffer = bb;
                    }

                    public boolean releaseBuffer() {
                        byteBuffer = null;
                        message = null;
                        return false;
                    }
                    
                };
            }
        };       







          
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



        
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



    

  

Reply | Threaded
Open this post in threaded view
|

Re: who can help me?how a server base grizzly receive 2MB data from client

windshome

when a message receive,we read it from a socket channel,but if we think all data isn't read completely, we return false ,and we will regist a read event in socket selector, we will read from the socketchannel use the buffer in old workthread.


in my ProtocolParser I will not clear the buffer before read all data completely,so when buffer's postion equals buffer's limit, read from channel will return size 0.

if I save data in my context and clear the buffer, return false after clear buffer,it will correct?


Harsha Godugu wrote
windshome wrote:

>Oh,I think you are not understand me, my problem is when the bytebuffer is
>full,but all my data hasn't read
>complete(my data size is 2MB), how can i read all data?
>
Hi:   Please note that the bytebuffer size that is used for i/o here, in
Grizzly, will have a constant threshold. That means, say, if the
application can send /receive data (on both server/client side), the
size of such a buffer during the process could vary from zero to the
threshold.  This threshold has got a limit. It can be 8K, 16K or 256k to
the max... based on the system limits and application needs.  It can not
be 8Mb or some gig..  The idea is,  the bytebuffer that we use in
grizzly (for i/o) is really a BUFFER, meaning a temporary place to read
/write to/from a channel. That means, the underlying application needs
to store the flushed  bits somewhere else.  So, as soon as you read say
8K bytes and then if expect more to read, we need store the read bytes
somewhere else (in some other object/file/shared memory..etc) and then
wait to read the rest of the data until your protocol handler tells you
that there are no next messages to parse!

hths
-Harsha

>Jeanfrancois Arcand-2 wrote:
>  
>
>>Salut,
>>
>>windshome wrote:
>>    
>>
>>>I process the data to make a digital signature, and return signed data to
>>>client.
>>>      
>>>
>>Interesting :-) The number of ByteBuffer created is equal to the number
>>of active Threads by default in Grizzly. Right now our default thread
>>pool doesn't purge inactive threads, so it may  or may not be a problem
>>if you need a lot of threads. You might want to replace the default
>>thread pool with a one from java.util.concurrent.* that can purge
>>inactive thread and their associated byte buffer.
>>
>>Are you able to determine the size of your expected traffic? If your VM
>>is properly tuned (I will let Charlie gives some hints in case you are
>>interested) it shouldn't be a problem, assuming you don't need 1000
>>threads :-)
>>
>>A+
>>
>>-- Jeanfrancois
>>
>>
>>    
>>
>>>Jeanfrancois Arcand-2 wrote:
>>>      
>>>
>>>>Hi,
>>>>
>>>>sorry for the delay...
>>>>
>>>>windshome wrote:
>>>>        
>>>>
>>>>>I write a server base grizzly,I write a client which send data to
>>>>>server
>>>>>and
>>>>>receive response from server. I send 50 bytes data and 2k data to
>>>>>server
>>>>>ok,but when I send 20K and 2MB data,server no response.
>>>>>I view the code of grizzly,find the ByteBuffer of a workthread capity
>>>>>is
>>>>>8192,then I modify it to 81920,then 20k data can receive by server.
>>>>>
>>>>>If my Server set the init size of ByteBuffer is 81920,I think it would
>>>>>use
>>>>>to much memory,who can tell me some method,will dynamic just the
>>>>>buffer,can
>>>>>receive some bytes data ,or some MB data?
>>>>>          
>>>>>
>>>>What are you doing with the bytes? Are you saving the bytes somewhere on
>>>>disk/db (freeing your memory) or you must keep them in memory?
>>>>
>>>>Thanks
>>>>
>>>>-- Jeanfrancois
>>>>
>>>>
>>>>        
>>>>
>>>>>My Server's protocol parser code :
>>>>>
>>>>>
>>>>>        final ProtocolFilter parserProtocolFilter = new
>>>>>ParserProtocolFilter() {
>>>>>            public ProtocolParser newProtocolParser() {
>>>>>                return new ProtocolParser() {
>>>>>                    private boolean isExpectingMoreData = false;
>>>>>                    private ByteBuffer byteBuffer;
>>>>>                    private Request message;
>>>>>                    
>>>>>                    public boolean hasMoreBytesToParse() {
>>>>>                        return false;
>>>>>                    }
>>>>>                    
>>>>>                    public boolean isExpectingMoreData() {
>>>>>                        return isExpectingMoreData;
>>>>>                    }
>>>>>                    
>>>>>                    public Object getNextMessage() {
>>>>>                        return message;
>>>>>                    }
>>>>>
>>>>>                    public boolean hasNextMessage() {
>>>>>                    
>>>>>                    
>>>>>                        ByteBuffer dup = byteBuffer.duplicate();
>>>>>                        System.out.println( "byteBuffer.position2()=" +
>>>>>byteBuffer.position() );
>>>>>                        if (byteBuffer.position() == 0){
>>>>>                            System.out.println("byteBuffer.position()
>>>>>==
>>>>>0");
>>>>>                         isExpectingMoreData = true;
>>>>>                            return false;
>>>>>                        }
>>>>>                        dup.flip();
>>>>>                        
>>>>>
>>>>>                        if( dup.remaining()< 4 ){
>>>>>                            isExpectingMoreData = true;
>>>>>                            return false;                        
>>>>>                        }
>>>>>
>>>>>                        byte[] bs = new byte[4];
>>>>>                        dup.get(bs);
>>>>>                        int len= (bs[0] << 24) + (bs[1] << 16) + (bs[2]
>>>>><<
>>>>>8) + (bs[3] << 0);
>>>>>                      
>>>>>                        if( dup.remaining() < len  ){
>>>>>                            isExpectingMoreData = true;
>>>>>                            return false;                        
>>>>>                        }
>>>>>                        byte[] data = new byte[len];
>>>>>                        dup.get( data );
>>>>>                        
>>>>>                        try {
>>>>> message = new ByteRequest(data);
>>>>> } catch (Exception e) {
>>>>> e.printStackTrace();
>>>>> message=null;
>>>>> return false;
>>>>> }
>>>>>                        return true;
>>>>>                        
>>>>>                    }
>>>>>
>>>>>                    public void startBuffer(ByteBuffer bb) {
>>>>>                        byteBuffer = bb;
>>>>>                    }
>>>>>
>>>>>                    public boolean releaseBuffer() {
>>>>>                        byteBuffer = null;
>>>>>                        message = null;
>>>>>                        return false;
>>>>>                    }
>>>>>                    
>>>>>                };
>>>>>            }
>>>>>        };      
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>---------------------------------------------------------------------
>>>>To unsubscribe, e-mail: users-unsubscribe@grizzly.dev.java.net
>>>>For additional commands, e-mail: users-help@grizzly.dev.java.net
>>>>
>>>>
>>>>
>>>>        
>>>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: users-unsubscribe@grizzly.dev.java.net
>>For additional commands, e-mail: users-help@grizzly.dev.java.net
>>
>>
>>
>>    
>>
>
>  
>
Reply | Threaded
Open this post in threaded view
|

Re: who can help me?how a server base grizzly receive 2MB data from client

Peter Murray-7
In reply to this post by windshome
I was going to say my but I am glad I did not since my assumptions  
were wrong now that I see
all the experts have answered.

Is your client a Java POJO? Would you mind sharing it?

I have yet to make any real progress on getting a POJO client to talk  
to the Server side.
( Currently I only get time on the train to work on it :( )

I do not know if it matters for your problem, but I do not see any  
flush in your code.
Does Comet do this automatically? The other examples seem to  
explicitly call it.


Peter


On Jan 29, 2008, at 10:41 PM, windshome wrote:

>
> I write a server base grizzly,I write a client which send data to  
> server and
> receive response from server. I send 50 bytes data and 2k data to  
> server
> ok,but when I send 20K and 2MB data,server no response.
> I view the code of grizzly,find the ByteBuffer of a workthread  
> capity is
> 8192,then I modify it to 81920,then 20k data can receive by server.
>
> If my Server set the init size of ByteBuffer is 81920,I think it  
> would use
> to much memory,who can tell me some method,will dynamic just the  
> buffer,can
> receive some bytes data ,or some MB data?
>
>
>
> My Server's protocol parser code :
>
>
>        final ProtocolFilter parserProtocolFilter = new
> ParserProtocolFilter() {
>            public ProtocolParser newProtocolParser() {
>                return new ProtocolParser() {
>                    private boolean isExpectingMoreData = false;
>                    private ByteBuffer byteBuffer;
>                    private Request message;
>
>                    public boolean hasMoreBytesToParse() {
>                        return false;
>                    }
>
>                    public boolean isExpectingMoreData() {
>                        return isExpectingMoreData;
>                    }
>
>                    public Object getNextMessage() {
>                        return message;
>                    }
>
>                    public boolean hasNextMessage() {
>                    
>                    
>                        ByteBuffer dup = byteBuffer.duplicate();
>                        System.out.println( "byteBuffer.position2()=" +
> byteBuffer.position() );
>                        if (byteBuffer.position() == 0){
>                            System.out.println("byteBuffer.position()  
> ==
> 0");
>                         isExpectingMoreData = true;
>                            return false;
>                        }
>                        dup.flip();
>
>
>                        if( dup.remaining()< 4 ){
>                            isExpectingMoreData = true;
>                            return false;                        
>                        }
>
>                        byte[] bs = new byte[4];
>                        dup.get(bs);
>                        int len= (bs[0] << 24) + (bs[1] << 16) +  
> (bs[2] <<
> 8) + (bs[3] << 0);
>
>                        if( dup.remaining() < len  ){
>                            isExpectingMoreData = true;
>                            return false;                        
>                        }
>                        byte[] data = new byte[len];
>                        dup.get( data );
>
>                        try {
> message = new ByteRequest(data);
> } catch (Exception e) {
> e.printStackTrace();
> message=null;
> return false;
> }
>                        return true;
>
>                    }
>
>                    public void startBuffer(ByteBuffer bb) {
>                        byteBuffer = bb;
>                    }
>
>                    public boolean releaseBuffer() {
>                        byteBuffer = null;
>                        message = null;
>                        return false;
>                    }
>
>                };
>            }
>        };
>
>
>
>
>
>
>
> --
> View this message in context: http://www.nabble.com/who-can-help-me-how-a-server-base-grizzly-receive-2MB-data-from-client-tp15174927p15174927.html
> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

Peter Murray
Manager,
Relationship Trading

[hidden email]
Office: +1 646-268-9916
Cell   : +1 914-414-3727
AIM   : retepmurray


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: who can help me?how a server base grizzly receive 2MB data from client

Oleksiy Stashok
In reply to this post by windshome
Hello,

windshome wrote:

> when a message receive,we read it from a socket channel,but if we think all
> data isn't read completely, we return false ,and we will regist a read event
> in socket selector, we will read from the socketchannel use the buffer in
> old workthread.
>
>
> in my ProtocolParser I will not clear the buffer before read all data
> completely,so when buffer's postion equals buffer's limit, read from channel
> will return size 0.
>
> if I save data in my context and clear the buffer, return false after clear
> buffer,it will correct?
>  
I would your last idea it's correct.
Just not sure if it's good idea to have all the message read at once. As
you told it's possible for you to have 2mb-20mb messages (may be I'm
wrong), think it's not too good to have such big buffers allocated. It's
also bad from point, that some DOS attacks can easily make your server
to come to OutOfMemory situation.
IMHO, it could make sense to think about implementation, where there
will be no need for such big buffers, but have some constant size buffer
(let's say 8-16K), where current read data will be put, processed. Then
this buffer could be released and reused for next read.

WBR,
Alexey.

>
>
> Harsha Godugu wrote:
>  
>> windshome wrote:
>>
>>    
>>> Oh,I think you are not understand me, my problem is when the bytebuffer is
>>> full,but all my data hasn't read
>>> complete(my data size is 2MB), how can i read all data?
>>>
>>>      
>> Hi:   Please note that the bytebuffer size that is used for i/o here, in
>> Grizzly, will have a constant threshold. That means, say, if the
>> application can send /receive data (on both server/client side), the
>> size of such a buffer during the process could vary from zero to the
>> threshold.  This threshold has got a limit. It can be 8K, 16K or 256k to
>> the max... based on the system limits and application needs.  It can not
>> be 8Mb or some gig..  The idea is,  the bytebuffer that we use in
>> grizzly (for i/o) is really a BUFFER, meaning a temporary place to read
>> /write to/from a channel. That means, the underlying application needs
>> to store the flushed  bits somewhere else.  So, as soon as you read say
>> 8K bytes and then if expect more to read, we need store the read bytes
>> somewhere else (in some other object/file/shared memory..etc) and then
>> wait to read the rest of the data until your protocol handler tells you
>> that there are no next messages to parse!
>>
>> hths
>> -Harsha
>>
>>    
>>> Jeanfrancois Arcand-2 wrote:
>>>  
>>>
>>>      
>>>> Salut,
>>>>
>>>> windshome wrote:
>>>>    
>>>>
>>>>        
>>>>> I process the data to make a digital signature, and return signed data
>>>>>          
> to
>  
>>>>> client.
>>>>>      
>>>>>
>>>>>          
>>>> Interesting :-) The number of ByteBuffer created is equal to the number
>>>> of active Threads by default in Grizzly. Right now our default thread
>>>> pool doesn't purge inactive threads, so it may  or may not be a problem
>>>> if you need a lot of threads. You might want to replace the default
>>>> thread pool with a one from java.util.concurrent.* that can purge
>>>> inactive thread and their associated byte buffer.
>>>>
>>>> Are you able to determine the size of your expected traffic? If your VM
>>>> is properly tuned (I will let Charlie gives some hints in case you are
>>>> interested) it shouldn't be a problem, assuming you don't need 1000
>>>> threads :-)
>>>>
>>>> A+
>>>>
>>>> -- Jeanfrancois
>>>>
>>>>
>>>>    
>>>>
>>>>        
>>>>> Jeanfrancois Arcand-2 wrote:
>>>>>      
>>>>>
>>>>>          
>>>>>> Hi,
>>>>>>
>>>>>> sorry for the delay...
>>>>>>
>>>>>> windshome wrote:
>>>>>>        
>>>>>>
>>>>>>            
>>>>>>> I write a server base grizzly,I write a client which send data to
>>>>>>> server
>>>>>>> and
>>>>>>> receive response from server. I send 50 bytes data and 2k data to
>>>>>>> server
>>>>>>> ok,but when I send 20K and 2MB data,server no response.
>>>>>>> I view the code of grizzly,find the ByteBuffer of a workthread capity
>>>>>>> is
>>>>>>> 8192,then I modify it to 81920,then 20k data can receive by server.
>>>>>>>
>>>>>>> If my Server set the init size of ByteBuffer is 81920,I think it would
>>>>>>> use
>>>>>>> to much memory,who can tell me some method,will dynamic just the
>>>>>>> buffer,can
>>>>>>> receive some bytes data ,or some MB data?
>>>>>>>          
>>>>>>>
>>>>>>>              
>>>>>> What are you doing with the bytes? Are you saving the bytes somewhere
>>>>>>            
> on
>  
>>>>>> disk/db (freeing your memory) or you must keep them in memory?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> -- Jeanfrancois
>>>>>>
>>>>>>
>>>>>>        
>>>>>>
>>>>>>            
>>>>>>> My Server's protocol parser code :
>>>>>>>
>>>>>>>
>>>>>>>        final ProtocolFilter parserProtocolFilter = new
>>>>>>> ParserProtocolFilter() {
>>>>>>>            public ProtocolParser newProtocolParser() {
>>>>>>>                return new ProtocolParser() {
>>>>>>>                    private boolean isExpectingMoreData = false;
>>>>>>>                    private ByteBuffer byteBuffer;
>>>>>>>                    private Request message;
>>>>>>>                    
>>>>>>>                    public boolean hasMoreBytesToParse() {
>>>>>>>                        return false;
>>>>>>>                    }
>>>>>>>                    
>>>>>>>                    public boolean isExpectingMoreData() {
>>>>>>>                        return isExpectingMoreData;
>>>>>>>                    }
>>>>>>>                    
>>>>>>>                    public Object getNextMessage() {
>>>>>>>                        return message;
>>>>>>>                    }
>>>>>>>
>>>>>>>                    public boolean hasNextMessage() {
>>>>>>>                    
>>>>>>>                    
>>>>>>>                        ByteBuffer dup = byteBuffer.duplicate();
>>>>>>>                        System.out.println( "byteBuffer.position2()="
>>>>>>> +
>>>>>>> byteBuffer.position() );
>>>>>>>                        if (byteBuffer.position() == 0){
>>>>>>>                            System.out.println("byteBuffer.position()
>>>>>>> ==
>>>>>>> 0");
>>>>>>>                         isExpectingMoreData = true;
>>>>>>>                            return false;
>>>>>>>                        }
>>>>>>>                        dup.flip();
>>>>>>>                        
>>>>>>>
>>>>>>>                        if( dup.remaining()< 4 ){
>>>>>>>                            isExpectingMoreData = true;
>>>>>>>                            return false;                        
>>>>>>>                        }
>>>>>>>
>>>>>>>                        byte[] bs = new byte[4];
>>>>>>>                        dup.get(bs);
>>>>>>>                        int len= (bs[0] << 24) + (bs[1] << 16) +
>>>>>>> (bs[2]
>>>>>>> <<
>>>>>>> 8) + (bs[3] << 0);
>>>>>>>                      
>>>>>>>                        if( dup.remaining() < len  ){
>>>>>>>                            isExpectingMoreData = true;
>>>>>>>                            return false;                        
>>>>>>>                        }
>>>>>>>                        byte[] data = new byte[len];
>>>>>>>                        dup.get( data );
>>>>>>>                        
>>>>>>>                        try {
>>>>>>> message = new ByteRequest(data);
>>>>>>> } catch (Exception e) {
>>>>>>> e.printStackTrace();
>>>>>>> message=null;
>>>>>>> return false;
>>>>>>> }
>>>>>>>                        return true;
>>>>>>>                        
>>>>>>>                    }
>>>>>>>
>>>>>>>                    public void startBuffer(ByteBuffer bb) {
>>>>>>>                        byteBuffer = bb;
>>>>>>>                    }
>>>>>>>
>>>>>>>                    public boolean releaseBuffer() {
>>>>>>>                        byteBuffer = null;
>>>>>>>                        message = null;
>>>>>>>                        return false;
>>>>>>>                    }
>>>>>>>                    
>>>>>>>                };
>>>>>>>            }
>>>>>>>        };      
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>          
>>>>>>>
>>>>>>>              
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: [hidden email]
>>>>>> For additional commands, e-mail: [hidden email]
>>>>>>
>>>>>>
>>>>>>
>>>>>>        
>>>>>>
>>>>>>            
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>>
>>>>
>>>>    
>>>>
>>>>        
>>>  
>>>
>>>      
>>
>>    
>
>  

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: who can help me?how a server base grizzly receive 2MB data from client

windshome
In reply to this post by Peter Murray-7

My problem is when my client send a large request, must I store this data slience to my context?

Peter Murray-7 wrote
I was going to say my but I am glad I did not since my assumptions  
were wrong now that I see
all the experts have answered.

Is your client a Java POJO? Would you mind sharing it?

I have yet to make any real progress on getting a POJO client to talk  
to the Server side.
( Currently I only get time on the train to work on it :( )

I do not know if it matters for your problem, but I do not see any  
flush in your code.
Does Comet do this automatically? The other examples seem to  
explicitly call it.


Peter


On Jan 29, 2008, at 10:41 PM, windshome wrote:

>
> I write a server base grizzly,I write a client which send data to  
> server and
> receive response from server. I send 50 bytes data and 2k data to  
> server
> ok,but when I send 20K and 2MB data,server no response.
> I view the code of grizzly,find the ByteBuffer of a workthread  
> capity is
> 8192,then I modify it to 81920,then 20k data can receive by server.
>
> If my Server set the init size of ByteBuffer is 81920,I think it  
> would use
> to much memory,who can tell me some method,will dynamic just the  
> buffer,can
> receive some bytes data ,or some MB data?
>
>
>
> My Server's protocol parser code :
>
>
>        final ProtocolFilter parserProtocolFilter = new
> ParserProtocolFilter() {
>            public ProtocolParser newProtocolParser() {
>                return new ProtocolParser() {
>                    private boolean isExpectingMoreData = false;
>                    private ByteBuffer byteBuffer;
>                    private Request message;
>
>                    public boolean hasMoreBytesToParse() {
>                        return false;
>                    }
>
>                    public boolean isExpectingMoreData() {
>                        return isExpectingMoreData;
>                    }
>
>                    public Object getNextMessage() {
>                        return message;
>                    }
>
>                    public boolean hasNextMessage() {
>                    
>                    
>                        ByteBuffer dup = byteBuffer.duplicate();
>                        System.out.println( "byteBuffer.position2()=" +
> byteBuffer.position() );
>                        if (byteBuffer.position() == 0){
>                            System.out.println("byteBuffer.position()  
> ==
> 0");
>                         isExpectingMoreData = true;
>                            return false;
>                        }
>                        dup.flip();
>
>
>                        if( dup.remaining()< 4 ){
>                            isExpectingMoreData = true;
>                            return false;                        
>                        }
>
>                        byte[] bs = new byte[4];
>                        dup.get(bs);
>                        int len= (bs[0] << 24) + (bs[1] << 16) +  
> (bs[2] <<
> 8) + (bs[3] << 0);
>
>                        if( dup.remaining() < len  ){
>                            isExpectingMoreData = true;
>                            return false;                        
>                        }
>                        byte[] data = new byte[len];
>                        dup.get( data );
>
>                        try {
> message = new ByteRequest(data);
> } catch (Exception e) {
> e.printStackTrace();
> message=null;
> return false;
> }
>                        return true;
>
>                    }
>
>                    public void startBuffer(ByteBuffer bb) {
>                        byteBuffer = bb;
>                    }
>
>                    public boolean releaseBuffer() {
>                        byteBuffer = null;
>                        message = null;
>                        return false;
>                    }
>
>                };
>            }
>        };
>
>
>
>
>
>
>
> --
> View this message in context: http://www.nabble.com/who-can-help-me-how-a-server-base-grizzly-receive-2MB-data-from-client-tp15174927p15174927.html
> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@grizzly.dev.java.net
> For additional commands, e-mail: users-help@grizzly.dev.java.net
>

Peter Murray
Manager,
Relationship Trading

peter.murray@fxall.com
Office: +1 646-268-9916
Cell   : +1 914-414-3727
AIM   : retepmurray


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@grizzly.dev.java.net
For additional commands, e-mail: users-help@grizzly.dev.java.net
Reply | Threaded
Open this post in threaded view
|

Re: who can help me?how a server base grizzly receive 2MB data from client

Harsha Godugu
In reply to this post by windshome
windshome wrote:
when a message receive,we read it from a socket channel,but if we think all
data isn't read completely, we return false ,and we will regist a read event
in socket selector, we will read from the socketchannel use the buffer in
old workthread.
  
correct.

in my ProtocolParser I will not clear the buffer before read all data
completely,so when buffer's postion equals buffer's limit, read from channel
will return size 0.
  
aha... here comes the question, what do we do when you are expecting more data and there is no-room (in the ByteBuffer, BB) to fill.  This can happen all the time when the application handles huge number of bytes.. like 2+Mb data.   Every protocol is different. I note that, your application is having a custom protocol. So, this custom protocol should have a way to handle the case of chunking / fragmentation.  It should tell the server/client how many bytes are coming still.. and when it's done sending all.
It's in the protocol. 

Say for example,  this case, Grizzly's buffer size is 8K. Now, we are going to handle say, 2Mb from one end to the other (client/server).  Once you read say the first 8K bytes, you need to process them , or store them, do-whatever, to empty the underlying buffer for the next read. This is where the parser comes into picture. Ask the following question.. what would be the  optimal size of the buffer, to parse one complete message. Based on the optimal size (during design), you can set the BB size in Grizzly accordingly, before starting the Controller.  The goal is, as soon as we read "n number of bytes" equivalent to constitute one single message, we give those read bytes to a parser for further action. In some application,  that action could be masking the read bytes or alter every eleventh byte ..etc. etc.. 
Say, for example, some extreme case, your optimal size of BB is 1Mb, then, that would be a bad design. Why? When doing i/o you want to do it as fast as as you can considering network bandwidth etc etc..  the application will run into OOMs  and will bailout very fast.  Too small size will make the BB full occasionally and the parser can not parse completely because, it still needs that one last byte which is expected be red yet.  During those times, you might consider recreating a new BB (similar to realloc  in C) for those the last bits. 

So, in a nut shell, it all depends on the Protocol, Parser and ByteBuffer size.

Good luck :-)

-Harsha

if I save data in my context and clear the buffer, return false after clear
buffer,it will correct? 
  
Harsha Godugu wrote:
  
windshome wrote:

    
Oh,I think you are not understand me, my problem is when the bytebuffer is
full,but all my data hasn't read 
complete(my data size is 2MB), how can i read all data?

      
Hi:   Please note that the bytebuffer size that is used for i/o here, in 
Grizzly, will have a constant threshold. That means, say, if the 
application can send /receive data (on both server/client side), the 
size of such a buffer during the process could vary from zero to the 
threshold.  This threshold has got a limit. It can be 8K, 16K or 256k to 
the max... based on the system limits and application needs.  It can not 
be 8Mb or some gig..  The idea is,  the bytebuffer that we use in 
grizzly (for i/o) is really a BUFFER, meaning a temporary place to read 
/write to/from a channel. That means, the underlying application needs 
to store the flushed  bits somewhere else.  So, as soon as you read say 
8K bytes and then if expect more to read, we need store the read bytes 
somewhere else (in some other object/file/shared memory..etc) and then 
wait to read the rest of the data until your protocol handler tells you 
that there are no next messages to parse! 

hths
-Harsha

    
Jeanfrancois Arcand-2 wrote:
 

      
Salut,

windshome wrote:
   

        
I process the data to make a digital signature, and return signed data
          
to
  
client.
     

          
Interesting :-) The number of ByteBuffer created is equal to the number 
of active Threads by default in Grizzly. Right now our default thread 
pool doesn't purge inactive threads, so it may  or may not be a problem 
if you need a lot of threads. You might want to replace the default 
thread pool with a one from java.util.concurrent.* that can purge 
inactive thread and their associated byte buffer.

Are you able to determine the size of your expected traffic? If your VM 
is properly tuned (I will let Charlie gives some hints in case you are 
interested) it shouldn't be a problem, assuming you don't need 1000 
threads :-)

A+

-- Jeanfrancois


   

        
Jeanfrancois Arcand-2 wrote:
     

          
Hi,

sorry for the delay...

windshome wrote:
       

            
I write a server base grizzly,I write a client which send data to
server
and
receive response from server. I send 50 bytes data and 2k data to
server
ok,but when I send 20K and 2MB data,server no response.
I view the code of grizzly,find the ByteBuffer of a workthread capity
is
8192,then I modify it to 81920,then 20k data can receive by server.

If my Server set the init size of ByteBuffer is 81920,I think it would
use
to much memory,who can tell me some method,will dynamic just the
buffer,can
receive some bytes data ,or some MB data?
         

              
What are you doing with the bytes? Are you saving the bytes somewhere
            
on 
  
disk/db (freeing your memory) or you must keep them in memory?

Thanks

-- Jeanfrancois


       

            
My Server's protocol parser code :


       final ProtocolFilter parserProtocolFilter = new
ParserProtocolFilter() {
           public ProtocolParser newProtocolParser() {
               return new ProtocolParser() {
                   private boolean isExpectingMoreData = false;
                   private ByteBuffer byteBuffer;
                   private Request message;
                   
                   public boolean hasMoreBytesToParse() {
                       return false;
                   }
                   
                   public boolean isExpectingMoreData() {
                       return isExpectingMoreData;
                   }
                   
                   public Object getNextMessage() {
                       return message;
                   }

                   public boolean hasNextMessage() {
                   	
                   	
                       ByteBuffer dup = byteBuffer.duplicate();
                       System.out.println( "byteBuffer.position2()="
+
byteBuffer.position() );
                       if (byteBuffer.position() == 0){
                           System.out.println("byteBuffer.position()
==
0");
                       	isExpectingMoreData = true;
                           return false;
                       }
                       dup.flip();
                       

                       if( dup.remaining()< 4 ){
                           isExpectingMoreData = true;
                           return false;                        	
                       }

                       byte[] bs = new byte[4];
                       dup.get(bs);
                       int len= (bs[0] << 24) + (bs[1] << 16) +
(bs[2]
<<
8) + (bs[3] << 0);
                      
                       if( dup.remaining() < len  ){
                           isExpectingMoreData = true;
                           return false;                        	
                       }
                       byte[] data = new byte[len];
                       dup.get( data );
                       
                       try {
							message = new ByteRequest(data);
						} catch (Exception e) {
							e.printStackTrace();
							message=null;
							return false;
						}
                       return true;
                       
                   }

                   public void startBuffer(ByteBuffer bb) {
                       byteBuffer = bb;
                   }

                   public boolean releaseBuffer() {
                       byteBuffer = null;
                       message = null;
                       return false;
                   }
                   
               };
           }
       };       







         

              
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



       

            
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



   

        
 

      

    

  

Reply | Threaded
Open this post in threaded view
|

Re: who can help me?how a server base grizzly receive 2MB data from client

windshome
Thanks, My problem have been resolved.

I re-create bytebuffer in WorkThreadImpl when the position of buffer==capity of buffer.

When I send 3M or bigger request data, OOM occured, should I change the 4M real bytebuffer to largger?

And my idea is extend bytebuffer, but I have not found the method.So ,I reallocate a heap view and put the old buffer data in it,is there some method to directory buffer,avoid re allocate?

Another problem is I haven't found where can store my infomation of session. My Server and client protocol  is :

1.Client open a socket
2.Client send hello requset
3.server receive hello,generate a 16 byte radom ,send random as challenge response to client
4.client receive challenge response, send real business requset, add the server challenge in request,and make a dagital signature of all request data,and send it all in same socket
5.server receive real request,verify digital signature,and check challenge of session

all this step must in a same socket,when socket close,we must re-generate a new challenge.


for this communication protocol, I must store the challenge in some session( a session is base a socket, not http session ),but I cannot find somewhere store it.Contex whill change in 2 times of protocol chain's execute,and workthread will change.


in apache Mina,I find session, can i use something use as session?


Harsha Godugu wrote
windshome wrote:
> when a message receive,we read it from a socket channel,but if we think all
> data isn't read completely, we return false ,and we will regist a read event
> in socket selector, we will read from the socketchannel use the buffer in
> old workthread.
>  
correct.
>
> in my ProtocolParser I will not clear the buffer before read all data
> completely,so when buffer's postion equals buffer's limit, read from channel
> will return size 0.
>  
aha... here comes the question, what do we do when you are expecting
more data and there is no-room (in the ByteBuffer, BB) to fill.  This
can happen all the time when the application handles huge number of
bytes.. like 2+Mb data.   Every protocol is different. I note that, your
application is having a custom protocol. So, this custom protocol should
have a way to handle the case of chunking / fragmentation.  It should
tell the server/client how many bytes are coming still.. and when it's
done sending all.
It's in the protocol.

Say for example,  this case, Grizzly's buffer size is 8K. Now, we are
going to handle say, 2Mb from one end to the other (client/server).  
Once you read say the first 8K bytes, you need to process them , or
store them, do-whatever, to empty the underlying buffer for the next
read. This is where the parser comes into picture. Ask the following
question.. what would be the  optimal size of the buffer, to parse one
complete message. Based on the *optimal size* (during design), you can
set the BB size in Grizzly accordingly, before starting the Controller.  
The goal is, as soon as we read "n number of bytes" equivalent to
constitute one single message, we give those read bytes to a parser for
further action. In some application,  that action could be masking the
read bytes or alter every eleventh byte ..etc. etc..
Say, for example, some extreme case, your optimal size of BB is 1Mb,
then, that would be a bad design. Why? When doing i/o you want to do it
as fast as as you can considering network bandwidth etc etc..  the
application will run into OOMs  and will bailout very fast.  Too small
size will make the BB full occasionally and the parser can not parse
completely because, it still needs that one last byte which is expected
be red yet.  During those times, you might consider recreating a new BB
(similar to realloc  in C) for those the last bits.

So, in a nut shell, it all depends on the Protocol, Parser and
ByteBuffer size.

Good luck :-)

-Harsha

> if I save data in my context and clear the buffer, return false after clear
> buffer,it will correct?
>  
> Harsha Godugu wrote:
>  
>> windshome wrote:
>>
>>    
>>> Oh,I think you are not understand me, my problem is when the bytebuffer is
>>> full,but all my data hasn't read
>>> complete(my data size is 2MB), how can i read all data?
>>>
>>>      
>> Hi:   Please note that the bytebuffer size that is used for i/o here, in
>> Grizzly, will have a constant threshold. That means, say, if the
>> application can send /receive data (on both server/client side), the
>> size of such a buffer during the process could vary from zero to the
>> threshold.  This threshold has got a limit. It can be 8K, 16K or 256k to
>> the max... based on the system limits and application needs.  It can not
>> be 8Mb or some gig..  The idea is,  the bytebuffer that we use in
>> grizzly (for i/o) is really a BUFFER, meaning a temporary place to read
>> /write to/from a channel. That means, the underlying application needs
>> to store the flushed  bits somewhere else.  So, as soon as you read say
>> 8K bytes and then if expect more to read, we need store the read bytes
>> somewhere else (in some other object/file/shared memory..etc) and then
>> wait to read the rest of the data until your protocol handler tells you
>> that there are no next messages to parse!
>>
>> hths
>> -Harsha
>>
>>    
>>> Jeanfrancois Arcand-2 wrote:
>>>  
>>>
>>>      
>>>> Salut,
>>>>
>>>> windshome wrote:
>>>>    
>>>>
>>>>        
>>>>> I process the data to make a digital signature, and return signed data
>>>>>          
> to
>  
>>>>> client.
>>>>>      
>>>>>
>>>>>          
>>>> Interesting :-) The number of ByteBuffer created is equal to the number
>>>> of active Threads by default in Grizzly. Right now our default thread
>>>> pool doesn't purge inactive threads, so it may  or may not be a problem
>>>> if you need a lot of threads. You might want to replace the default
>>>> thread pool with a one from java.util.concurrent.* that can purge
>>>> inactive thread and their associated byte buffer.
>>>>
>>>> Are you able to determine the size of your expected traffic? If your VM
>>>> is properly tuned (I will let Charlie gives some hints in case you are
>>>> interested) it shouldn't be a problem, assuming you don't need 1000
>>>> threads :-)
>>>>
>>>> A+
>>>>
>>>> -- Jeanfrancois
>>>>
>>>>
>>>>    
>>>>
>>>>        
>>>>> Jeanfrancois Arcand-2 wrote:
>>>>>      
>>>>>
>>>>>          
>>>>>> Hi,
>>>>>>
>>>>>> sorry for the delay...
>>>>>>
>>>>>> windshome wrote:
>>>>>>        
>>>>>>
>>>>>>            
>>>>>>> I write a server base grizzly,I write a client which send data to
>>>>>>> server
>>>>>>> and
>>>>>>> receive response from server. I send 50 bytes data and 2k data to
>>>>>>> server
>>>>>>> ok,but when I send 20K and 2MB data,server no response.
>>>>>>> I view the code of grizzly,find the ByteBuffer of a workthread capity
>>>>>>> is
>>>>>>> 8192,then I modify it to 81920,then 20k data can receive by server.
>>>>>>>
>>>>>>> If my Server set the init size of ByteBuffer is 81920,I think it would
>>>>>>> use
>>>>>>> to much memory,who can tell me some method,will dynamic just the
>>>>>>> buffer,can
>>>>>>> receive some bytes data ,or some MB data?
>>>>>>>          
>>>>>>>
>>>>>>>              
>>>>>> What are you doing with the bytes? Are you saving the bytes somewhere
>>>>>>            
> on
>  
>>>>>> disk/db (freeing your memory) or you must keep them in memory?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> -- Jeanfrancois
>>>>>>
>>>>>>
>>>>>>        
>>>>>>
>>>>>>            
>>>>>>> My Server's protocol parser code :
>>>>>>>
>>>>>>>
>>>>>>>        final ProtocolFilter parserProtocolFilter = new
>>>>>>> ParserProtocolFilter() {
>>>>>>>            public ProtocolParser newProtocolParser() {
>>>>>>>                return new ProtocolParser() {
>>>>>>>                    private boolean isExpectingMoreData = false;
>>>>>>>                    private ByteBuffer byteBuffer;
>>>>>>>                    private Request message;
>>>>>>>                    
>>>>>>>                    public boolean hasMoreBytesToParse() {
>>>>>>>                        return false;
>>>>>>>                    }
>>>>>>>                    
>>>>>>>                    public boolean isExpectingMoreData() {
>>>>>>>                        return isExpectingMoreData;
>>>>>>>                    }
>>>>>>>                    
>>>>>>>                    public Object getNextMessage() {
>>>>>>>                        return message;
>>>>>>>                    }
>>>>>>>
>>>>>>>                    public boolean hasNextMessage() {
>>>>>>>                    
>>>>>>>                    
>>>>>>>                        ByteBuffer dup = byteBuffer.duplicate();
>>>>>>>                        System.out.println( "byteBuffer.position2()="
>>>>>>> +
>>>>>>> byteBuffer.position() );
>>>>>>>                        if (byteBuffer.position() == 0){
>>>>>>>                            System.out.println("byteBuffer.position()
>>>>>>> ==
>>>>>>> 0");
>>>>>>>                         isExpectingMoreData = true;
>>>>>>>                            return false;
>>>>>>>                        }
>>>>>>>                        dup.flip();
>>>>>>>                        
>>>>>>>
>>>>>>>                        if( dup.remaining()< 4 ){
>>>>>>>                            isExpectingMoreData = true;
>>>>>>>                            return false;                        
>>>>>>>                        }
>>>>>>>
>>>>>>>                        byte[] bs = new byte[4];
>>>>>>>                        dup.get(bs);
>>>>>>>                        int len= (bs[0] << 24) + (bs[1] << 16) +
>>>>>>> (bs[2]
>>>>>>> <<
>>>>>>> 8) + (bs[3] << 0);
>>>>>>>                      
>>>>>>>                        if( dup.remaining() < len  ){
>>>>>>>                            isExpectingMoreData = true;
>>>>>>>                            return false;                        
>>>>>>>                        }
>>>>>>>                        byte[] data = new byte[len];
>>>>>>>                        dup.get( data );
>>>>>>>                        
>>>>>>>                        try {
>>>>>>> message = new ByteRequest(data);
>>>>>>> } catch (Exception e) {
>>>>>>> e.printStackTrace();
>>>>>>> message=null;
>>>>>>> return false;
>>>>>>> }
>>>>>>>                        return true;
>>>>>>>                        
>>>>>>>                    }
>>>>>>>
>>>>>>>                    public void startBuffer(ByteBuffer bb) {
>>>>>>>                        byteBuffer = bb;
>>>>>>>                    }
>>>>>>>
>>>>>>>                    public boolean releaseBuffer() {
>>>>>>>                        byteBuffer = null;
>>>>>>>                        message = null;
>>>>>>>                        return false;
>>>>>>>                    }
>>>>>>>                    
>>>>>>>                };
>>>>>>>            }
>>>>>>>        };      
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>          
>>>>>>>
>>>>>>>              
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe@grizzly.dev.java.net
>>>>>> For additional commands, e-mail: users-help@grizzly.dev.java.net
>>>>>>
>>>>>>
>>>>>>
>>>>>>        
>>>>>>
>>>>>>            
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@grizzly.dev.java.net
>>>> For additional commands, e-mail: users-help@grizzly.dev.java.net
>>>>
>>>>
>>>>
>>>>    
>>>>
>>>>        
>>>  
>>>
>>>      
>>
>>    
>
>  
Reply | Threaded
Open this post in threaded view
|

Re: who can help me?how a server base grizzly receive 2MB data from client

Harsha Godugu
windshome wrote:
Thanks, My problem have been resolved.

  
Good.  I've a but here...
I re-create bytebuffer in WorkThreadImpl when the position of buffer==capity
of buffer.

When I send 3M or bigger request data, OOM occured, should I change the 4M
real bytebuffer to largger?
  
Take a step back and think about the application design.  Grizzly does not have a facility yet, to grow/shrink the underlying bytebuffer on-demand. May be in future we might have one. 
To come to think of 4Mb... it is way too much (unless your app. needs very high end memory and processing requirements..). I still do not understand the need for such a huge size for doing i/o.  So, I don't have a suggestion there. Your app. should consider sending data and processing data in parts or chunks.  This is nothing to do with Grizzly.

And my idea is extend bytebuffer, but I have not found the method.So ,I
reallocate a heap view and put the old buffer data in it,is there some
method to directory buffer,avoid re allocate?

  
Ask the following question. What's the minimum size of the data, that can suffice to process data (in the parser.)  Then use the BB.compact() or BB.position(resetPosition) for next read. The idea is to discard the PARSED /processed bytes and to re-use the underlying BB , from the current unfilled location.  This is very tricky.  With this, you will rarely need re-allocation. You can also, use multiple bye buffers or a pool of byte buffers. Again this is out of Grizzly.  When you use multiple BBs, make sure, you release the BB back to the pool and no leaks of memory.
Another problem is I haven't found where can store my infomation of session.
My Server and client protocol  is :

1.Client open a socket
2.Client send hello requset
3.server receive hello,generate a 16 byte radom ,send random as challenge
response to client
4.client receive challenge response, send real business requset, add the
server challenge in request,and make a dagital signature of all request
data,and send it all in same socket
5.server receive real request,verify digital signature,and check challenge
of session 

all this step must in a same socket,when socket close,we must re-generate a
new challenge.
  
Create a(your own) Session object with whatever your application needs and store it outside Grizzly. I think, you might want to use onConnect() or onAccept() callback to create a session and store the session.  At the begining of selection cycle, we get a Context object, which can be used to store the sessiob object. The same context gets passed to the down-calls frow which we can access session object. But, context is a temporary object that last through one selection cycle. So, there should be a map with keys for each session during client connection request (and accept  made on the serverside).

hths,
-Harsha


for this communication protocol, I must store the challenge in some session(
a session is base a socket, not http session ),but I cannot find somewhere
store it.Contex whill change in 2 times of protocol chain's execute,and
workthread will change.


in apache Mina,I find session, can i use something use as session?



Harsha Godugu wrote:
  
windshome wrote:
    
when a message receive,we read it from a socket channel,but if we think
all
data isn't read completely, we return false ,and we will regist a read
event
in socket selector, we will read from the socketchannel use the buffer in
old workthread.
  
      
correct.
    
in my ProtocolParser I will not clear the buffer before read all data
completely,so when buffer's postion equals buffer's limit, read from
channel
will return size 0.
  
      
aha... here comes the question, what do we do when you are expecting 
more data and there is no-room (in the ByteBuffer, BB) to fill.  This 
can happen all the time when the application handles huge number of 
bytes.. like 2+Mb data.   Every protocol is different. I note that, your 
application is having a custom protocol. So, this custom protocol should 
have a way to handle the case of chunking / fragmentation.  It should 
tell the server/client how many bytes are coming still.. and when it's 
done sending all.
It's in the protocol. 

Say for example,  this case, Grizzly's buffer size is 8K. Now, we are 
going to handle say, 2Mb from one end to the other (client/server).  
Once you read say the first 8K bytes, you need to process them , or 
store them, do-whatever, to empty the underlying buffer for the next 
read. This is where the parser comes into picture. Ask the following 
question.. what would be the  optimal size of the buffer, to parse one 
complete message. Based on the *optimal size* (during design), you can 
set the BB size in Grizzly accordingly, before starting the Controller.  
The goal is, as soon as we read "n number of bytes" equivalent to 
constitute one single message, we give those read bytes to a parser for 
further action. In some application,  that action could be masking the 
read bytes or alter every eleventh byte ..etc. etc.. 
Say, for example, some extreme case, your optimal size of BB is 1Mb, 
then, that would be a bad design. Why? When doing i/o you want to do it 
as fast as as you can considering network bandwidth etc etc..  the 
application will run into OOMs  and will bailout very fast.  Too small 
size will make the BB full occasionally and the parser can not parse 
completely because, it still needs that one last byte which is expected 
be red yet.  During those times, you might consider recreating a new BB 
(similar to realloc  in C) for those the last bits. 

So, in a nut shell, it all depends on the Protocol, Parser and 
ByteBuffer size.

Good luck :-)

-Harsha

    
if I save data in my context and clear the buffer, return false after
clear
buffer,it will correct? 
  
Harsha Godugu wrote:
  
      
windshome wrote:

    
        
Oh,I think you are not understand me, my problem is when the bytebuffer
is
full,but all my data hasn't read 
complete(my data size is 2MB), how can i read all data?

      
          
Hi:   Please note that the bytebuffer size that is used for i/o here, in 
Grizzly, will have a constant threshold. That means, say, if the 
application can send /receive data (on both server/client side), the 
size of such a buffer during the process could vary from zero to the 
threshold.  This threshold has got a limit. It can be 8K, 16K or 256k to 
the max... based on the system limits and application needs.  It can not 
be 8Mb or some gig..  The idea is,  the bytebuffer that we use in 
grizzly (for i/o) is really a BUFFER, meaning a temporary place to read 
/write to/from a channel. That means, the underlying application needs 
to store the flushed  bits somewhere else.  So, as soon as you read say 
8K bytes and then if expect more to read, we need store the read bytes 
somewhere else (in some other object/file/shared memory..etc) and then 
wait to read the rest of the data until your protocol handler tells you 
that there are no next messages to parse! 

hths
-Harsha

    
        
Jeanfrancois Arcand-2 wrote:
 

      
          
Salut,

windshome wrote:
   

        
            
I process the data to make a digital signature, and return signed
data
          
              
to
  
      
client.
     

          
              
Interesting :-) The number of ByteBuffer created is equal to the
number 
of active Threads by default in Grizzly. Right now our default thread 
pool doesn't purge inactive threads, so it may  or may not be a
problem 
if you need a lot of threads. You might want to replace the default 
thread pool with a one from java.util.concurrent.* that can purge 
inactive thread and their associated byte buffer.

Are you able to determine the size of your expected traffic? If your
VM 
is properly tuned (I will let Charlie gives some hints in case you are 
interested) it shouldn't be a problem, assuming you don't need 1000 
threads :-)

A+

-- Jeanfrancois


   

        
            
Jeanfrancois Arcand-2 wrote:
     

          
              
Hi,

sorry for the delay...

windshome wrote:
       

            
                
I write a server base grizzly,I write a client which send data to
server
and
receive response from server. I send 50 bytes data and 2k data to
server
ok,but when I send 20K and 2MB data,server no response.
I view the code of grizzly,find the ByteBuffer of a workthread
capity
is
8192,then I modify it to 81920,then 20k data can receive by server.

If my Server set the init size of ByteBuffer is 81920,I think it
would
use
to much memory,who can tell me some method,will dynamic just the
buffer,can
receive some bytes data ,or some MB data?
         

              
                  
What are you doing with the bytes? Are you saving the bytes
somewhere
            
                
on 
  
      
disk/db (freeing your memory) or you must keep them in memory?

Thanks

-- Jeanfrancois


       

            
                
My Server's protocol parser code :


       final ProtocolFilter parserProtocolFilter = new
ParserProtocolFilter() {
           public ProtocolParser newProtocolParser() {
               return new ProtocolParser() {
                   private boolean isExpectingMoreData = false;
                   private ByteBuffer byteBuffer;
                   private Request message;
                   
                   public boolean hasMoreBytesToParse() {
                       return false;
                   }
                   
                   public boolean isExpectingMoreData() {
                       return isExpectingMoreData;
                   }
                   
                   public Object getNextMessage() {
                       return message;
                   }

                   public boolean hasNextMessage() {
                   	
                   	
                       ByteBuffer dup = byteBuffer.duplicate();
                       System.out.println(
"byteBuffer.position2()="
+
byteBuffer.position() );
                       if (byteBuffer.position() == 0){
                          
System.out.println("byteBuffer.position()
==
0");
                       	isExpectingMoreData = true;
                           return false;
                       }
                       dup.flip();
                       

                       if( dup.remaining()< 4 ){
                           isExpectingMoreData = true;
                           return false;                        	
                       }

                       byte[] bs = new byte[4];
                       dup.get(bs);
                       int len= (bs[0] << 24) + (bs[1] << 16) +
(bs[2]
<<
8) + (bs[3] << 0);
                      
                       if( dup.remaining() < len  ){
                           isExpectingMoreData = true;
                           return false;                        	
                       }
                       byte[] data = new byte[len];
                       dup.get( data );
                       
                       try {
							message = new ByteRequest(data);
						} catch (Exception e) {
							e.printStackTrace();
							message=null;
							return false;
						}
                       return true;
                       
                   }

                   public void startBuffer(ByteBuffer bb) {
                       byteBuffer = bb;
                   }

                   public boolean releaseBuffer() {
                       byteBuffer = null;
                       message = null;
                       return false;
                   }
                   
               };
           }
       };       







         

              
                  
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



       

            
                
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



   

        
            
 

      
          
    
        
  
      

    

  

Reply | Threaded
Open this post in threaded view
|

Re: who can help me?how a server base grizzly receive 2MB data from client

Jeanfrancois Arcand-2
In reply to this post by windshome

Hi,

windshome wrote:
> Thanks, My problem have been resolved.
>
> I re-create bytebuffer in WorkThreadImpl when the position of buffer==capity
> of buffer.
>
> When I send 3M or bigger request data, OOM occured, should I change the 4M
> real bytebuffer to largger?

Well, growing indefinitely is dangerous IMO. There is no way you can
stream the bytes or store them outside the BB. You really have to load
all the bytes in memory before doing something with them?

>
> And my idea is extend bytebuffer, but I have not found the method.So ,I
> reallocate a heap view and put the old buffer data in it,is there some
> method to directory buffer,avoid re allocate?

No they are not. If you need to do that, your should slice a large heap
ByteBuffer (so bytes are moved inside, not copied). Take a look at:

https://grizzly.dev.java.net/nonav/xref/com/sun/grizzly/util/ByteBufferFactory.html


>
> Another problem is I haven't found where can store my infomation of session.
> My Server and client protocol  is :
>
> 1.Client open a socket
> 2.Client send hello requset
> 3.server receive hello,generate a 16 byte radom ,send random as challenge
> response to client
> 4.client receive challenge response, send real business requset, add the
> server challenge in request,and make a dagital signature of all request
> data,and send it all in same socket
> 5.server receive real request,verify digital signature,and check challenge
> of session
>
> all this step must in a same socket,when socket close,we must re-generate a
> new challenge.
>
>
> for this communication protocol, I must store the challenge in some session(
> a session is base a socket, not http session ),but I cannot find somewhere
> store it.Contex whill change in 2 times of protocol chain's execute,and
> workthread will change.
>
>
> in apache Mina,I find session, can i use something use as session?

Use the Context setAttribute/getAttribute (same as IoSession) and set
the appropriate AttributeHolder:

> 06     /**
> 207      * Return <code>AttributeHolder</code>, which corresponds to the
> 208      * given <code>AttributeScope</code>
> 209      *
> 210      * @param scope - <code>AttributeScope</code>
> 211      * @return - <code>AttributeHolder</code> instance, which contains
> 212      *           <code>AttributeScope</code> attributes
> 213      */
> 214     public AttributeHolder getAttributeHolderByScope(AttributeScope scope){
> 215         AttributeHolder holder = null;
> 216         switch(scope) {
> 217             case REQUEST:
> 218                 holder = this;
> 219                 break;
> 220             case SELECTOR:
> 221                 holder = selectorHandler;
> 222                 break;
> 223             case CONTROLLER:
> 224                 holder = controller;
> 225                 break;
> 226         }
> 227        
> 228         return holder;
> 229     }

See:

https://grizzly.dev.java.net/nonav/xref/com/sun/grizzly/Context.html

https://grizzly.dev.java.net/nonav/xref/com/sun/grizzly/TCPSelectorHandler.html

Mainly, based on the scope you what to store element, you can store it
at the Context or Selector on Controller level.


Thanks

-- Jeanfrancois


>
>
>
> Harsha Godugu wrote:
>> windshome wrote:
>>> when a message receive,we read it from a socket channel,but if we think
>>> all
>>> data isn't read completely, we return false ,and we will regist a read
>>> event
>>> in socket selector, we will read from the socketchannel use the buffer in
>>> old workthread.
>>>  
>> correct.
>>> in my ProtocolParser I will not clear the buffer before read all data
>>> completely,so when buffer's postion equals buffer's limit, read from
>>> channel
>>> will return size 0.
>>>  
>> aha... here comes the question, what do we do when you are expecting
>> more data and there is no-room (in the ByteBuffer, BB) to fill.  This
>> can happen all the time when the application handles huge number of
>> bytes.. like 2+Mb data.   Every protocol is different. I note that, your
>> application is having a custom protocol. So, this custom protocol should
>> have a way to handle the case of chunking / fragmentation.  It should
>> tell the server/client how many bytes are coming still.. and when it's
>> done sending all.
>> It's in the protocol.
>>
>> Say for example,  this case, Grizzly's buffer size is 8K. Now, we are
>> going to handle say, 2Mb from one end to the other (client/server).  
>> Once you read say the first 8K bytes, you need to process them , or
>> store them, do-whatever, to empty the underlying buffer for the next
>> read. This is where the parser comes into picture. Ask the following
>> question.. what would be the  optimal size of the buffer, to parse one
>> complete message. Based on the *optimal size* (during design), you can
>> set the BB size in Grizzly accordingly, before starting the Controller.  
>> The goal is, as soon as we read "n number of bytes" equivalent to
>> constitute one single message, we give those read bytes to a parser for
>> further action. In some application,  that action could be masking the
>> read bytes or alter every eleventh byte ..etc. etc..
>> Say, for example, some extreme case, your optimal size of BB is 1Mb,
>> then, that would be a bad design. Why? When doing i/o you want to do it
>> as fast as as you can considering network bandwidth etc etc..  the
>> application will run into OOMs  and will bailout very fast.  Too small
>> size will make the BB full occasionally and the parser can not parse
>> completely because, it still needs that one last byte which is expected
>> be red yet.  During those times, you might consider recreating a new BB
>> (similar to realloc  in C) for those the last bits.
>>
>> So, in a nut shell, it all depends on the Protocol, Parser and
>> ByteBuffer size.
>>
>> Good luck :-)
>>
>> -Harsha
>>
>>> if I save data in my context and clear the buffer, return false after
>>> clear
>>> buffer,it will correct?
>>>  
>>> Harsha Godugu wrote:
>>>  
>>>> windshome wrote:
>>>>
>>>>    
>>>>> Oh,I think you are not understand me, my problem is when the bytebuffer
>>>>> is
>>>>> full,but all my data hasn't read
>>>>> complete(my data size is 2MB), how can i read all data?
>>>>>
>>>>>      
>>>> Hi:   Please note that the bytebuffer size that is used for i/o here, in
>>>> Grizzly, will have a constant threshold. That means, say, if the
>>>> application can send /receive data (on both server/client side), the
>>>> size of such a buffer during the process could vary from zero to the
>>>> threshold.  This threshold has got a limit. It can be 8K, 16K or 256k to
>>>> the max... based on the system limits and application needs.  It can not
>>>> be 8Mb or some gig..  The idea is,  the bytebuffer that we use in
>>>> grizzly (for i/o) is really a BUFFER, meaning a temporary place to read
>>>> /write to/from a channel. That means, the underlying application needs
>>>> to store the flushed  bits somewhere else.  So, as soon as you read say
>>>> 8K bytes and then if expect more to read, we need store the read bytes
>>>> somewhere else (in some other object/file/shared memory..etc) and then
>>>> wait to read the rest of the data until your protocol handler tells you
>>>> that there are no next messages to parse!
>>>>
>>>> hths
>>>> -Harsha
>>>>
>>>>    
>>>>> Jeanfrancois Arcand-2 wrote:
>>>>>  
>>>>>
>>>>>      
>>>>>> Salut,
>>>>>>
>>>>>> windshome wrote:
>>>>>>    
>>>>>>
>>>>>>        
>>>>>>> I process the data to make a digital signature, and return signed
>>>>>>> data
>>>>>>>          
>>> to
>>>  
>>>>>>> client.
>>>>>>>      
>>>>>>>
>>>>>>>          
>>>>>> Interesting :-) The number of ByteBuffer created is equal to the
>>>>>> number
>>>>>> of active Threads by default in Grizzly. Right now our default thread
>>>>>> pool doesn't purge inactive threads, so it may  or may not be a
>>>>>> problem
>>>>>> if you need a lot of threads. You might want to replace the default
>>>>>> thread pool with a one from java.util.concurrent.* that can purge
>>>>>> inactive thread and their associated byte buffer.
>>>>>>
>>>>>> Are you able to determine the size of your expected traffic? If your
>>>>>> VM
>>>>>> is properly tuned (I will let Charlie gives some hints in case you are
>>>>>> interested) it shouldn't be a problem, assuming you don't need 1000
>>>>>> threads :-)
>>>>>>
>>>>>> A+
>>>>>>
>>>>>> -- Jeanfrancois
>>>>>>
>>>>>>
>>>>>>    
>>>>>>
>>>>>>        
>>>>>>> Jeanfrancois Arcand-2 wrote:
>>>>>>>      
>>>>>>>
>>>>>>>          
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> sorry for the delay...
>>>>>>>>
>>>>>>>> windshome wrote:
>>>>>>>>        
>>>>>>>>
>>>>>>>>            
>>>>>>>>> I write a server base grizzly,I write a client which send data to
>>>>>>>>> server
>>>>>>>>> and
>>>>>>>>> receive response from server. I send 50 bytes data and 2k data to
>>>>>>>>> server
>>>>>>>>> ok,but when I send 20K and 2MB data,server no response.
>>>>>>>>> I view the code of grizzly,find the ByteBuffer of a workthread
>>>>>>>>> capity
>>>>>>>>> is
>>>>>>>>> 8192,then I modify it to 81920,then 20k data can receive by server.
>>>>>>>>>
>>>>>>>>> If my Server set the init size of ByteBuffer is 81920,I think it
>>>>>>>>> would
>>>>>>>>> use
>>>>>>>>> to much memory,who can tell me some method,will dynamic just the
>>>>>>>>> buffer,can
>>>>>>>>> receive some bytes data ,or some MB data?
>>>>>>>>>          
>>>>>>>>>
>>>>>>>>>              
>>>>>>>> What are you doing with the bytes? Are you saving the bytes
>>>>>>>> somewhere
>>>>>>>>            
>>> on
>>>  
>>>>>>>> disk/db (freeing your memory) or you must keep them in memory?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> -- Jeanfrancois
>>>>>>>>
>>>>>>>>
>>>>>>>>        
>>>>>>>>
>>>>>>>>            
>>>>>>>>> My Server's protocol parser code :
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>        final ProtocolFilter parserProtocolFilter = new
>>>>>>>>> ParserProtocolFilter() {
>>>>>>>>>            public ProtocolParser newProtocolParser() {
>>>>>>>>>                return new ProtocolParser() {
>>>>>>>>>                    private boolean isExpectingMoreData = false;
>>>>>>>>>                    private ByteBuffer byteBuffer;
>>>>>>>>>                    private Request message;
>>>>>>>>>                    
>>>>>>>>>                    public boolean hasMoreBytesToParse() {
>>>>>>>>>                        return false;
>>>>>>>>>                    }
>>>>>>>>>                    
>>>>>>>>>                    public boolean isExpectingMoreData() {
>>>>>>>>>                        return isExpectingMoreData;
>>>>>>>>>                    }
>>>>>>>>>                    
>>>>>>>>>                    public Object getNextMessage() {
>>>>>>>>>                        return message;
>>>>>>>>>                    }
>>>>>>>>>
>>>>>>>>>                    public boolean hasNextMessage() {
>>>>>>>>>                    
>>>>>>>>>                    
>>>>>>>>>                        ByteBuffer dup = byteBuffer.duplicate();
>>>>>>>>>                        System.out.println(
>>>>>>>>> "byteBuffer.position2()="
>>>>>>>>> +
>>>>>>>>> byteBuffer.position() );
>>>>>>>>>                        if (byteBuffer.position() == 0){
>>>>>>>>>                          
>>>>>>>>> System.out.println("byteBuffer.position()
>>>>>>>>> ==
>>>>>>>>> 0");
>>>>>>>>>                         isExpectingMoreData = true;
>>>>>>>>>                            return false;
>>>>>>>>>                        }
>>>>>>>>>                        dup.flip();
>>>>>>>>>                        
>>>>>>>>>
>>>>>>>>>                        if( dup.remaining()< 4 ){
>>>>>>>>>                            isExpectingMoreData = true;
>>>>>>>>>                            return false;                        
>>>>>>>>>                        }
>>>>>>>>>
>>>>>>>>>                        byte[] bs = new byte[4];
>>>>>>>>>                        dup.get(bs);
>>>>>>>>>                        int len= (bs[0] << 24) + (bs[1] << 16) +
>>>>>>>>> (bs[2]
>>>>>>>>> <<
>>>>>>>>> 8) + (bs[3] << 0);
>>>>>>>>>                      
>>>>>>>>>                        if( dup.remaining() < len  ){
>>>>>>>>>                            isExpectingMoreData = true;
>>>>>>>>>                            return false;                        
>>>>>>>>>                        }
>>>>>>>>>                        byte[] data = new byte[len];
>>>>>>>>>                        dup.get( data );
>>>>>>>>>                        
>>>>>>>>>                        try {
>>>>>>>>> message = new ByteRequest(data);
>>>>>>>>> } catch (Exception e) {
>>>>>>>>> e.printStackTrace();
>>>>>>>>> message=null;
>>>>>>>>> return false;
>>>>>>>>> }
>>>>>>>>>                        return true;
>>>>>>>>>                        
>>>>>>>>>                    }
>>>>>>>>>
>>>>>>>>>                    public void startBuffer(ByteBuffer bb) {
>>>>>>>>>                        byteBuffer = bb;
>>>>>>>>>                    }
>>>>>>>>>
>>>>>>>>>                    public boolean releaseBuffer() {
>>>>>>>>>                        byteBuffer = null;
>>>>>>>>>                        message = null;
>>>>>>>>>                        return false;
>>>>>>>>>                    }
>>>>>>>>>                    
>>>>>>>>>                };
>>>>>>>>>            }
>>>>>>>>>        };      
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>          
>>>>>>>>>
>>>>>>>>>              
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: [hidden email]
>>>>>>>> For additional commands, e-mail: [hidden email]
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>        
>>>>>>>>
>>>>>>>>            
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: [hidden email]
>>>>>> For additional commands, e-mail: [hidden email]
>>>>>>
>>>>>>
>>>>>>
>>>>>>    
>>>>>>
>>>>>>        
>>>>>  
>>>>>
>>>>>      
>>>>    
>>>  
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Pojo client - a work in progress

Peter Murray-7

I have attached a small client which talks to the "counter" example.
It works (renamed to counter2 for the moment) to the extent that it  
increments the counter and get the "success" message back.

Is my next step to set up a
     CometHandler<HttpServletResponse>
class and have it listen for events from Comet as is done in the  
servlet?

Thanks...


Peter


PS when I have this working as a client for counter I would like to  
add it back to the example in SVN. Everyone in the world is not using  
AJAX :)



public class CounterClient
{

     /**
      * @param args the command line arguments
      */
     public static void main(String[] args)
     {
         try
         {

             URL url = new URL("http://localhost:8080/counter2/long_polling 
");
             URLConnection conn = url.openConnection();
             conn.setDoOutput(true);

             BufferedWriter out =
                     new BufferedWriter(new  
OutputStreamWriter(conn.getOutputStream()));
             out.write("/counter2/long_polling\r\n");
             out.flush();
             out.close();
             BufferedReader in =
                     new BufferedReader(new  
InputStreamReader(conn.getInputStream()));

             String response;
             while ((response = in.readLine()) != null)
             {
                 System.out.println(response);
                 Thread.sleep(1);
             }
         //in.close();
         }
         catch (MalformedURLException ex)
         {
             System.out.println("Exception: " + ex.toString());
         }
         catch (IOException ex)
         {
             System.out.println("Exception: " + ex.toString());
         }
         catch (InterruptedException ex)
         {
             System.out.println("Exception: " + ex.toString());
         }
     }
}


Peter Murray
Manager,
Relationship Trading

[hidden email]
Office: +1 646-268-9916
Cell   : +1 914-414-3727
AIM   : retepmurray


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Pojo client - a work in progress

Jeanfrancois Arcand-2


Peter Murray wrote:
>
> I have attached a small client which talks to the "counter" example.
> It works (renamed to counter2 for the moment) to the extent that it
> increments the counter and get the "success" message back.

This is great.

>
> Is my next step to set up a
>     CometHandler<HttpServletResponse>
> class and have it listen for events from Comet as is done in the servlet?

Yes. But to be sure, are you using Grizzly Comet, Grizzlet or GlassFish v2?

Thanks. Let me know once you want to add you client stuff inside the
workspace.

-- Jeanfrancois


>
> Thanks...
>
>
> Peter
>
>
> PS when I have this working as a client for counter I would like to add
> it back to the example in SVN. Everyone in the world is not using AJAX :)
>
>
>
> public class CounterClient
> {
>
>     /**
>      * @param args the command line arguments
>      */
>     public static void main(String[] args)
>     {
>         try
>         {
>
>             URL url = new
> URL("http://localhost:8080/counter2/long_polling");
>             URLConnection conn = url.openConnection();
>             conn.setDoOutput(true);
>
>             BufferedWriter out =
>                     new BufferedWriter(new
> OutputStreamWriter(conn.getOutputStream()));
>             out.write("/counter2/long_polling\r\n");
>             out.flush();
>             out.close();
>             BufferedReader in =
>                     new BufferedReader(new
> InputStreamReader(conn.getInputStream()));
>
>             String response;
>             while ((response = in.readLine()) != null)
>             {
>                 System.out.println(response);
>                 Thread.sleep(1);
>             }
>         //in.close();
>         }
>         catch (MalformedURLException ex)
>         {
>             System.out.println("Exception: " + ex.toString());
>         }
>         catch (IOException ex)
>         {
>             System.out.println("Exception: " + ex.toString());
>         }
>         catch (InterruptedException ex)
>         {
>             System.out.println("Exception: " + ex.toString());
>         }
>     }
> }
>
>
> Peter Murray
> Manager,
> Relationship Trading
>
> [hidden email]
> Office: +1 646-268-9916
> Cell   : +1 914-414-3727
> AIM   : retepmurray
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]