Grizzly Kernel and Selector Waiting Threads

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Grizzly Kernel and Selector Waiting Threads

Steve Curtis

Hi, I’m working on a project that is currently using SimpleFramework java web server.  I suspect we are running out of worker threads but unfortunately the api does not give me a mechanism to get a hold of the thread executors.


With this in mind I am looking at using Grizzly as an alternative, since on the face on it I should be able to get this information.


I’m creating the server as follows:


private HttpServer server;

private NetworkListener listener;

public static final String localhost = "localhost";

 

    public void runServer(int port) throws IOException

    {

        logger.info("starting grizzly framework server on port {}", port);

 

        server = new HttpServer();

 

        listener = new NetworkListener("grizzly", localhost, port);

        listener.setSecure(false);

        listener.getTransport().setIOStrategy(WorkerThreadIOStrategy.getInstance());

 

        server.addListener(listener);

 

        final ServerConfiguration serverConfiguration = server.getServerConfiguration();

        serverConfiguration.addHttpHandler(new HttpHandler()

                                           {

                                               public void service(Request request, Response response) throws Exception

                                               {

                                                   final SimpleDateFormat format = new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss zzz", Locale.UK);

                                                   final String date = format.format(new Date(System.currentTimeMillis()));

                                                   response.setContentType("text/plain");

                                                   response.setContentLength(date.length());

                                                   response.getWriter().write(date);

                                               }

                                           }

        );

 

        server.start();

 

        logger.info("bootstrap of grizzly framework server complete, running on http://{}:{}", localhost, port);

    }

 

I was trying to print off details of the kernel and thread queue sizes but the queue object is null:


logger.info("kernel thread pool queue {}", listener.getTransport().getKernelThreadPoolConfig().getQueue());

logger.info("worker thread pool queue {}", listener.getTransport().getWorkerThreadPoolConfig().getQueue());

 

I think the number of pending incoming requests and queued workers should be the size of each of these Queue objects if they not null?


Note maybe there is a better way of getting this sort of information?


Thanks

Reply | Threaded
Open this post in threaded view
|

Re: Grizzly Kernel and Selector Waiting Threads

Ryan Lubke-2
Hi Steve,

Please see inline...

November 10, 2015 at 09:53

Hi, I’m working on a project that is currently using SimpleFramework java web server.  I suspect we are running out of worker threads but unfortunately the api does not give me a mechanism to get a hold of the thread executors.


With this in mind I am looking at using Grizzly as an alternative, since on the face on it I should be able to get this information.


I’m creating the server as follows:


private HttpServer server;

private NetworkListener listener;

public static final String localhost = "localhost";

 

    public void runServer(int port) throws IOException

    {

        logger.info("starting grizzly framework server on port {}", port);

 

        server = new HttpServer();

 

        listener = new NetworkListener("grizzly", localhost, port);

        listener.setSecure(false);

        listener.getTransport().setIOStrategy(WorkerThreadIOStrategy.getInstance());

 

        server.addListener(listener);

 

        final ServerConfiguration serverConfiguration = server.getServerConfiguration();

        serverConfiguration.addHttpHandler(new HttpHandler()

                                           {

                                               public void service(Request request, Response response) throws Exception

                                               {

                                                   final SimpleDateFormat format = new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss zzz", Locale.UK);

                                                   final String date = format.format(new Date(System.currentTimeMillis()));

                                                   response.setContentType("text/plain");

                                                   response.setContentLength(date.length());

                                                   response.getWriter().write(date);

                                               }

                                           }

        );

 

        server.start();

 

        logger.info("bootstrap of grizzly framework server complete, running on <a class="moz-txt-link-freetext" href="http://">http://{}:{}", localhost, port);

    }

 

I was trying to print off details of the kernel and thread queue sizes but the queue object is null:


logger.info("kernel thread pool queue {}", listener.getTransport().getKernelThreadPoolConfig().getQueue());

logger.info("worker thread pool queue {}", listener.getTransport().getWorkerThreadPoolConfig().getQueue());


getQueue() will return null unless you explicitly set your own Queue implementation.  If it's null, when the thread pool is initialized,
Grizzly will choose its own implementation, but this won't be reflected in the configuration object.

 

I think the number of pending incoming requests and queued workers should be the size of each of these Queue objects if they not null?


Not necessarily.  You can have a large number of queued tasks, but a small fixed number of workers.


Note maybe there is a better way of getting this sort of information?


Yes, you can look at providing a ThreadPoolProbe [1] implementation.    This will allow you to track thread allocation/deallocation/queue/dequeue events within the pool.  The probe implementation can be registered by calling getInitialMonitoringConfig() on the worker thread pool configuration before starting the server.  Additionally, you may want to review the thread pool documentation [2].  There are performance considerations when configuring the min/max values of the pool size.

[1] https://grizzly.java.net/docs/2.3/apidocs/org/glassfish/grizzly/threadpool/ThreadPoolProbe.html
[2] https://grizzly.java.net/coreconfig.html#/Thread_Pool_Configuration


Thanks


Reply | Threaded
Open this post in threaded view
|

Re: Grizzly Kernel and Selector Waiting Threads

Steve Curtis
Thanks Ryan, that's very useful - although as well as being notified about those key events from the probe (eg when max threads allocated for exampe) I'd really like to try and get some metrics around how heavily loaded the system is since I suspect that is what is happening on my application today.

Trying to boil it down to a couple of key questions:

[1] I can get the core and max size of the worker pool from listener.getTransport().getWorkerThreadPoolConfig() but is there a mechanism to get the current pool size?
[2] I managed to get the number of waiting worker requests by using the following code, is this the best way to get this info?

   ((GrizzlyExecutorService)listener.getTransport().getWorkerThreadPool())
       .getConfiguration().getQueue().size();

[3] Is there a mechanism to get the selector thread details in a similar manner to the worker above?
[4] Quite a fundamental question but what is the difference between kernel thread and selector/worker threads and should I be considering sizing of these as well?

Thanks in advance,

Steve

On 10 November 2015 at 16:37, Ryan Lubke <[hidden email]> wrote:
Hi Steve,

Please see inline...

November 10, 2015 at 09:53

Hi, I’m working on a project that is currently using SimpleFramework java web server.  I suspect we are running out of worker threads but unfortunately the api does not give me a mechanism to get a hold of the thread executors.


With this in mind I am looking at using Grizzly as an alternative, since on the face on it I should be able to get this information.


I’m creating the server as follows:


private HttpServer server;

private NetworkListener listener;

public static final String localhost = "localhost";

 

    public void runServer(int port) throws IOException

    {

        logger.info("starting grizzly framework server on port {}", port);

 

        server = new HttpServer();

 

        listener = new NetworkListener("grizzly", localhost, port);

        listener.setSecure(false);

        listener.getTransport().setIOStrategy(WorkerThreadIOStrategy.getInstance());

 

        server.addListener(listener);

 

        final ServerConfiguration serverConfiguration = server.getServerConfiguration();

        serverConfiguration.addHttpHandler(new HttpHandler()

                                           {

                                               public void service(Request request, Response response) throws Exception

                                               {

                                                   final SimpleDateFormat format = new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss zzz", Locale.UK);

                                                   final String date = format.format(new Date(System.currentTimeMillis()));

                                                   response.setContentType("text/plain");

                                                   response.setContentLength(date.length());

                                                   response.getWriter().write(date);

                                               }

                                           }

        );

 

        server.start();

 

        logger.info("bootstrap of grizzly framework server complete, running on http://{}:{}", localhost, port);

    }

 

I was trying to print off details of the kernel and thread queue sizes but the queue object is null:


logger.info("kernel thread pool queue {}", listener.getTransport().getKernelThreadPoolConfig().getQueue());

logger.info("worker thread pool queue {}", listener.getTransport().getWorkerThreadPoolConfig().getQueue());


getQueue() will return null unless you explicitly set your own Queue implementation.  If it's null, when the thread pool is initialized,
Grizzly will choose its own implementation, but this won't be reflected in the configuration object.

 

I think the number of pending incoming requests and queued workers should be the size of each of these Queue objects if they not null?


Not necessarily.  You can have a large number of queued tasks, but a small fixed number of workers.


Note maybe there is a better way of getting this sort of information?


Yes, you can look at providing a ThreadPoolProbe [1] implementation.    This will allow you to track thread allocation/deallocation/queue/dequeue events within the pool.  The probe implementation can be registered by calling getInitialMonitoringConfig() on the worker thread pool configuration before starting the server.  Additionally, you may want to review the thread pool documentation [2].  There are performance considerations when configuring the min/max values of the pool size.

[1] https://grizzly.java.net/docs/2.3/apidocs/org/glassfish/grizzly/threadpool/ThreadPoolProbe.html
[2] https://grizzly.java.net/coreconfig.html#/Thread_Pool_Configuration


Thanks



Reply | Threaded
Open this post in threaded view
|

Re: Grizzly Kernel and Selector Waiting Threads

Ryan Lubke-2
Steve,

Sorry for the delay.  Please see inline.

November 11, 2015 at 10:58
Thanks Ryan, that's very useful - although as well as being notified about those key events from the probe (eg when max threads allocated for exampe) I'd really like to try and get some metrics around how heavily loaded the system is since I suspect that is what is happening on my application today.

Trying to boil it down to a couple of key questions:

[1] I can get the core and max size of the worker pool from listener.getTransport().getWorkerThreadPoolConfig() but is there a mechanism to get the current pool size?
You could certainly do this with the probe.  Have an AtomicInteger that you're able to query that will be incremented/decremented as thread allocation/deallocation events occur.  Please see our JMX ThreadPoolProbe [1] as a possible starting point.
[2] I managed to get the number of waiting worker requests by using the following code, is this the best way to get this info?

   ((GrizzlyExecutorService)listener.getTransport().getWorkerThreadPool())
       .getConfiguration().getQueue().size();
Yep.

[3] Is there a mechanism to get the selector thread details in a similar manner to the worker above?
Yes.  You can register a probe with the kernel thread pool configuration (ThreadPoolConfig.getInitialMonitoringConfig()).
[4] Quite a fundamental question but what is the difference between kernel thread and selector/worker threads and should I be considering sizing of these as well?
The threads within the kernel pool are the selector threads. 
Typically the kernel pool will be sized for you automatically based on the number of available processors.  In general, this should be sufficient. 

Thanks in advance,

Steve


[1] https://github.com/GrizzlyNIO/grizzly-mirror/blob/38419dd9a1bcd793d7619cae5c3e23e11f54adfd/modules/monitoring/grizzly/src/main/java/org/glassfish/grizzly/threadpool/jmx/ThreadPool.java#L214
November 10, 2015 at 10:37
Hi Steve,

Please see inline...

November 10, 2015 at 09:53

Hi, I’m working on a project that is currently using SimpleFramework java web server.  I suspect we are running out of worker threads but unfortunately the api does not give me a mechanism to get a hold of the thread executors.


With this in mind I am looking at using Grizzly as an alternative, since on the face on it I should be able to get this information.


I’m creating the server as follows:


private HttpServer server;

private NetworkListener listener;

public static final String localhost = "localhost";

 

    public void runServer(int port) throws IOException

    {

        logger.info("starting grizzly framework server on port {}", port);

 

        server = new HttpServer();

 

        listener = new NetworkListener("grizzly", localhost, port);

        listener.setSecure(false);

        listener.getTransport().setIOStrategy(WorkerThreadIOStrategy.getInstance());

 

        server.addListener(listener);

 

        final ServerConfiguration serverConfiguration = server.getServerConfiguration();

        serverConfiguration.addHttpHandler(new HttpHandler()

                                           {

                                               public void service(Request request, Response response) throws Exception

                                               {

                                                   final SimpleDateFormat format = new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss zzz", Locale.UK);

                                                   final String date = format.format(new Date(System.currentTimeMillis()));

                                                   response.setContentType("text/plain");

                                                   response.setContentLength(date.length());

                                                   response.getWriter().write(date);

                                               }

                                           }

        );

 

        server.start();

 

        logger.info("bootstrap of grizzly framework server complete, running on <a moz-do-not-send="true" href="http://" class="moz-txt-link-freetext">http://{}:{}", localhost, port);

    }

 

I was trying to print off details of the kernel and thread queue sizes but the queue object is null:


logger.info("kernel thread pool queue {}", listener.getTransport().getKernelThreadPoolConfig().getQueue());

logger.info("worker thread pool queue {}", listener.getTransport().getWorkerThreadPoolConfig().getQueue());


getQueue() will return null unless you explicitly set your own Queue implementation.  If it's null, when the thread pool is initialized,
Grizzly will choose its own implementation, but this won't be reflected in the configuration object.

 

I think the number of pending incoming requests and queued workers should be the size of each of these Queue objects if they not null?


Not necessarily.  You can have a large number of queued tasks, but a small fixed number of workers.


Note maybe there is a better way of getting this sort of information?


Yes, you can look at providing a ThreadPoolProbe [1] implementation.    This will allow you to track thread allocation/deallocation/queue/dequeue events within the pool.  The probe implementation can be registered by calling getInitialMonitoringConfig() on the worker thread pool configuration before starting the server.  Additionally, you may want to review the thread pool documentation [2].  There are performance considerations when configuring the min/max values of the pool size.

[1] https://grizzly.java.net/docs/2.3/apidocs/org/glassfish/grizzly/threadpool/ThreadPoolProbe.html
[2] https://grizzly.java.net/coreconfig.html#/Thread_Pool_Configuration


Thanks


November 10, 2015 at 09:53

Hi, I’m working on a project that is currently using SimpleFramework java web server.  I suspect we are running out of worker threads but unfortunately the api does not give me a mechanism to get a hold of the thread executors.


With this in mind I am looking at using Grizzly as an alternative, since on the face on it I should be able to get this information.


I’m creating the server as follows:


private HttpServer server;

private NetworkListener listener;

public static final String localhost = "localhost";

 

    public void runServer(int port) throws IOException

    {

        logger.info("starting grizzly framework server on port {}", port);

 

        server = new HttpServer();

 

        listener = new NetworkListener("grizzly", localhost, port);

        listener.setSecure(false);

        listener.getTransport().setIOStrategy(WorkerThreadIOStrategy.getInstance());

 

        server.addListener(listener);

 

        final ServerConfiguration serverConfiguration = server.getServerConfiguration();

        serverConfiguration.addHttpHandler(new HttpHandler()

                                           {

                                               public void service(Request request, Response response) throws Exception

                                               {

                                                   final SimpleDateFormat format = new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss zzz", Locale.UK);

                                                   final String date = format.format(new Date(System.currentTimeMillis()));

                                                   response.setContentType("text/plain");

                                                   response.setContentLength(date.length());

                                                   response.getWriter().write(date);

                                               }

                                           }

        );

 

        server.start();

 

        logger.info("bootstrap of grizzly framework server complete, running on <a class="moz-txt-link-freetext" href="http://">http://{}:{}", localhost, port);

    }

 

I was trying to print off details of the kernel and thread queue sizes but the queue object is null:


logger.info("kernel thread pool queue {}", listener.getTransport().getKernelThreadPoolConfig().getQueue());

logger.info("worker thread pool queue {}", listener.getTransport().getWorkerThreadPoolConfig().getQueue());

 

I think the number of pending incoming requests and queued workers should be the size of each of these Queue objects if they not null?


Note maybe there is a better way of getting this sort of information?


Thanks