Difference Between Netty3 and Netty4

Instruction

Netty 4 has been released three years, compared to Netty 3, Netty 4 in the memory model and the thread model has been modified. Now Netty 4 is already very mature, and for the Square company, Netty 4 also has a significant feature: the HTTP/2 protocol of the native support. Square expects its mobile devices to use the HTTP/2 protocol and is switching the back-end RPC framework to gRPC: an RPC/2 protocol-based RPC framework. Therefore, Tracon as a proxy service, must support HTTP/2 protocol.

Analysis

Single Thread Channel

Unlike Netty 3, Netty 4’s inbound (data input) and outbound (data out) events are all in the same thread. This is the time to write the processor, you can remove the thread-safe code. However, this change also makes the upgrade process to meet the conditions of competition caused by the problem.

In Netty 3, the operation for the pipeline is thread-safe, but in Netty 4, all operations are placed as events in the event loop asynchronously. As a proxy service, Tracon will have an independent inbound channel to interact with the upstream server, and a separate outbound channel to interact with the downstream server. In order to improve performance, the connection to the downstream server is cached, so that these events may occur concurrently when an event in the event loop triggers a write operation. This is fine for Netty 3, and every write will be completed before returning; but for Netty 4, these operations go into an event loop, which can lead to out-of-order messages.

Therefore, in the block test, and occasionally encounter the data sent to the order is not arrived, resulting in test failure.

When upgrading from Netty 3 to Netty 4, it is important to note that events are dispatched asynchronously if they are triggered outside the event loop.

codes as follows:

1
2
3
4
5
6
7
8
9
@Override
public void userEventTriggered(ChannelHandlerContext ctx,     Object evt) throws
    if (evt.equals(SslHandshakeCompletionEvent.SUCCESS)) {
        Principal peerPrincipal = engine.getSession().getPeerPrincipal();
     // 身份验证
     // ...
    }
    super.userEventTriggered(ctx, evt);
}

Memory Leak

Since direct memory is used by NIO, it is necessary to monitor direct memory for Netty network libraries, which can be done by using JMX beanjava.nio: type = BufferPool, name = direct.

Netty 4 introduces a thread-local Recyler based on thread-local variables to reclaim object pools. By default, a collector can hold up to 262k objects, for ByteBuf, less than 64k of the default shared cache object. In other words, each collector can hold up to 17G of direct memory.

Normally, the NIO cache is sufficient to cope with an instantaneous amount of data. However, if you have a very slow reading of the back-end, will greatly increase the memory usage. In addition, when the cache memory in the NIO read and write by other threads, the allocation of the memory of the thread will not be able to reclaim the memory.

The Netty project has also made some corrections to the problem of limiting the growth of objects that can not be recovered by the reclaimer, which causes memory exhaustion:

  • Allows you to set the maximum number of WeakOrderQueue instances per thread;

  • The memory allocation / sharing ratio is introduced in the reclaimer;

  • Porting SendBufferPooled from Netty 3.10, used when using a non-shared ByteBuf allocator (ByteBufAllocator);

From the experience of upgrading Netty 4, it is recommended that all developers configure the Recycler based on available memory and threads. The maximum number of objects that the Reclaimer can hold can be set with the -Dio.netty.recycler.maxCapacity parameter, and the maximum memory limit can be set with the -Dio.netty.threadLocalDirectBufferSize parameter. If you want to completely close the collector, you can set-Dio.netty.recycler.maxCapacity 0

Use Global Exception Process, codes as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
 class LoggingExceptionHandler implements Thread.UncaughtExceptionHandler {
    private static final Logger logger = Logger.getLogger(LoggingExceptionHandler.class);
    /** 注册成默认处理器 */
    static void registerAsDefault() {
        Thread.setDefaultUncaughtExceptionHandler(new LoggingExceptionHandler());
    }
    @Override
    public void uncaughtException(Thread t, Throwable e) {
        if (e instanceof Exception) {
            logger.error("Uncaught exception killed thread named '" + t.getName() + "'.", e);
        } else {
            logger.fatal("Uncaught error killed thread named '" + t.getName() + "'." + " Exiting now.", e);
            System.exit(1);
        }
    }
}

HTTP decoder refactor

Netty 4 refactored the HTTP decoders, and specifically improved support for chunked data. HTTP message body is split into HttpContent object, if the HTTP data through the block transmission, there will be more than one HttpContent order to arrive, when the data block transmission end, there will be a LastHttpContent object. It should be noted that, LastHttpContent inherited from HttpContent, must not be handled in the following manner:

1
2
3
4
5
6
If (msg instanceof HttpContent) {
   ...
}}
If (msg instanceof LastHttpContent) {
   ... / / The last block will be repeated, the previous if already contains the LastHttpContent
}}

For LastHttpContent there is a need to note that when receiving this object, HTTP message body may have been transmitted over, at this time LastHttpContent just as HTTP transmission end (similar to EOF).

Ref

Square-Netty-3-Netty-4