Ping/Pong exception in Depth websocket

Hello guys,

when I am opening an onDepthEvent websocket on many pairs(all btc pairs), I start getting this error

java.net.SocketTimeoutException: sent ping but didn't receive pong within 20000ms (after 0 successful ping/pongs)

When checking binance official websocket api which says:
The websocket server will send a ping frame every 3 minutes. If the websocket server does not receive a pong frame back from the connection within a 10 minute period, the connection will be disconnected. Unsolicited pong frames are allowed.

I though the Ping/Pong should be sent every 3 min?
When checking BinanceApiServiceGenerator the httpClient is created:
Dispatcher dispatcher = new Dispatcher(); dispatcher.setMaxRequestsPerHost(500); dispatcher.setMaxRequests(500); sharedClient = new OkHttpClient.Builder() .dispatcher(dispatcher) .pingInterval(20, TimeUnit.SECONDS) .build();
Doesnt that mean its trying to ping and receive a pong from server within 20 seconds?
Maybe thats why the error is happening?

Thanks for any tips!

By the way, the websocket is receiving data, just the onFailure() gets triggered (I think after the first unsuccessful ping/pong)

@tomastoth

@tomastoth look at lysine-dev/okhttp#2239
This issue states that we cannot use okhttp at the moment to support binance ping/pong fully. Client only does ping from its side which is fine, but binance is also sending ping to client and client should respond with pong back. Which I assume is not possible atm with okttp.

As a workaround for websockets, you may try this implementation:
https://github.com/eugene-kamenev/binance-better-websocket/blob/master/src/main/java/com/binance/BinanceWebSocketListener.java
Which is based on netty & async http client and can be integrated without pain.
You can easily implement ping from your side as well. It is only required if your socket connection keep silence for some timeframe which you can add to listener and implement reconnect yourself.
Also this listener implementation uses jackson correctly 😋

Hope this helps.

@eugene-kamenev

That is not quite true @eugene-kamenev, see https://github.com/square/okhttp/blob/master/okhttp/src/main/java/okhttp3/internal/ws/RealWebSocket.java. It supports writing a pong frame to the server, it is just not exposed in the API (which makes sense imho, as it is a protocol frame, and does not make sense to expose it in the application layer).

What do you mean by the listener implementation not using Jackson correctly? Is it because ObjectMapper is not being cached? I will put a fix for that.

@tomastoth

@tomastoth, are you able to share a working example where I will be able to quickly reproduce the problem? The pingInterval is to check if the connection is still alive on Binance's side.

The memory issues should be fixed by dd121ac, please update your repo.

@joaopsilva

@joaopsilva You are correct. Looks like okhttp should support what is required.. But only after I migrated to netty for websockets my app started using much less heap and websocket streaming never looses data. I am running app for months now and didnt experienced anything weird like before. Dont know what could cause this.. But I assume issue sits at okhttp side..
@joaopsilva Yes, jackson mapper should be cached (singleton).

@joaopsilva

@joaopsilva this is the full stack trace
java.net.SocketTimeoutException: sent ping but didn't receive pong within 20000ms (after 0 successful ping/pongs) at okhttp3.internal.ws.RealWebSocket.writePingFrame(RealWebSocket.java:546) at okhttp3.internal.ws.RealWebSocket$PingRunnable.run(RealWebSocket.java:530) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset$$$capture(FutureTask.java:308) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
For example today, it kept running without a problem and after 600 ping/pongs i received a SocketTimeoutException. It restarted and now I keep getting this exception again. Its like the Binance doesnt accept depth socket request from me anymore or something

@tomastoth

@tomastoth look at square/okhttp#2239
This issue states that we cannot use okhttp at the moment to support binance ping/pong fully. Client only does ping from its side which is fine, but binance is also sending ping to client and client should respond with pong back. Which I assume is not possible atm with okttp.

As a workaround for websockets, you may try this implementation:
https://github.com/eugene-kamenev/binance-better-websocket/blob/master/src/main/java/com/binance/BinanceWebSocketListener.java
Which is based on netty & async http client and can be integrated without pain.
You can easily implement ping from your side as well. It is only required if your socket connection keep silence for some timeframe which you can add to listener and implement reconnect yourself.
Also this listener implementation uses jackson correctly 😋

Hope this helps.

I am kinda new to okhttp and asyncwebsockets, can you please integrate your new type of websocket into the api and share it with us others so we can use it? I find it kinda hard implementing the websocket from your code. Thanks

@tomastoth

@tomastoth Have you checked at tests folder? https://github.com/eugene-kamenev/binance-better-websocket/blob/master/src/test/groovy/com/binance/WebSocketTestSpec.groovy
I will try to find spare time this weekend to provide PR.

@tomastoth

@tomastoth Have you checked at tests folder? https://github.com/eugene-kamenev/binance-better-websocket/blob/master/src/test/groovy/com/binance/WebSocketTestSpec.groovy
I will try to find spare time this weekend to provide PR.

Okay, I have implemented it the same way. I am opening 123 different threads each one has a socket open like yours and I am getting this
java.util.concurrent.ExecutionException: java.net.ConnectException: handshake timed out
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:201)
at com.tomastoth.new_orderbook_saver.SaveToDb.startWebsocket(SaveToDb.java:74)
at com.tomastoth.new_orderbook_saver.SaveToDb.run(SaveToDb.java:50)
Caused by: java.net.ConnectException: handshake timed out
at org.asynchttpclient.netty.channel.NettyConnectListener.onFailure(NettyConnectListener.java:179)
at org.asynchttpclient.netty.channel.NettyConnectListener$1.onFailure(NettyConnectListener.java:151)
at org.asynchttpclient.netty.SimpleFutureListener.operationComplete(SimpleFutureListener.java:26)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121)
at io.netty.handler.ssl.SslHandler$6.run(SslHandler.java:1778)
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLException: handshake timed out

Maybe its bad to have so many threads?

@tomastoth

@tomastoth You should not use many threads and connections.. It is enough to have single EventLoopGroup with multiple threads and asynchttpclient instance, and it should serve all your connections.. You should apply multi-pair subscription through url (like in tests I provided). Then you can serve many connections throught single event loop group.
Remember that you should not block any incoming event, as I can see from your stacktrace you are trying to communicate with db from event handler.. This is also wrong, you should collect all incoming events into something like LinkedBlockingQueue, then call queue.drainTo at another thread which will perfom your communication with db and other things you need. Otherwise you are blocking event handler which is a bad practice..

@tomastoth

@tomastoth You should not use many threads and connections.. It is enough to have single EventLoopGroup with multiple threads and asynchttpclient instance, and it should serve all your connections.. You should apply multi-pair subscription through url (like in tests I provided). Then you can serve many connections throught single event loop group.
Remember that you should not block any incoming event, as I can see from your stacktrace you are trying to communicate with db from event handler.. This is also wrong, you should collect all incoming events into something like LinkedBlockingQueue, then call queue.drainTo at another thread which will perfom your communication with db and other things you need. Otherwise you are blocking event handler which is a bad practice..

I havent done async programming before so its kinda new for me. So you think that is the reason I am getting the exception?

has this issue been fixed? I get a couple of these error these days: Error in Websocket sent ping but didn't receive pong within 20000ms (after 13 successful ping/pongs)

@joaopsilva

Getting same issue recently.
sent ping but didn't receive pong within 20000ms (after 43 successful ping/pongs)
@joaopsilva Please tell us the reason and solution if there any. Thanks.

@eugene-kamenev

Hello folks, well, I have recently tried the latest JAVA API and get a ping/pong error on the very first ping request. If I ignore it, it seems to continue ok.

I then tried @eugene-kamenev solution, but I keep getting the following appearing in the logs about 10 minutes in to running and then the webservice is obviously closed (and the onclose method was called and the reason was closed abnormally):

2020-06-13 15:32:22.160  INFO 72814 --- [ntLoopGroup-2-2] c.b.a.c.i.BinanceApiWebSocketListener    : WebSocket NettyWebSocket{channel=[id: 0x152b7a49, L:/<IP ADDRESS>:59018 - R:stream.binance.com/18.182.78.165:9443]} received ping, sending pong back..
2020-06-13 15:33:19.963  INFO 72814 --- [ntLoopGroup-2-2] c.b.a.c.i.BinanceApiWebSocketListener    : WebSocket NettyWebSocket{channel=[id: 0x152b7a49, L:/<IP ADDRESS>:59018 - R:stream.binance.com/18.182.78.165:9443]} was closed. Reason was: Connection was closed abnormally (that is, with no close frame being received).

Has something changed on Binance recently? I wasn't receiving these ping/pong errors prior?

If I run the latest version of the Binance API code it seems it still uses OKHTTP and as mentioned I get ping/pong errors when connecting to Binance?

EDIT:

If I delete .pingInterval(...) from the sharedhttp client in BinanceApiServiceGenerator.java (from the latest repository pull), it doesn't die after the set interval period, whether its 20s, 30s or other (after experimenting). It seems Binance DOES NOT send a pong request to the client ping (any more?)

```

static {
Dispatcher dispatcher = new Dispatcher();
dispatcher.setMaxRequestsPerHost(500);
dispatcher.setMaxRequests(500);
sharedClient = new OkHttpClient.Builder()
.dispatcher(dispatcher)
//.pingInterval(30, TimeUnit.SECONDS)
.build();
}

Lastly, in the onFailure event...

new BinanceApiCallback() {

			@Override
			public void onFailure(Throwable cause) {
				LOG.error("onFailure: " + cause.getMessage());
				
				cause.printStackTrace();
				
				if (cause instanceof SocketTimeoutException) {
					webSocket = bWSClient.onDepthEvent(cryptoListForChosenCurrency, onDepthEvents);
				}
	        }

I can re-initialise the websocket, but leaving pingInterval set to 20, this code is executed at the ~20second mark 100% of the time. I guess this prevents the 24 hour reconnect :) but every 20 seconds, not so good.

I ended up solving this heatbeat problem using the NETTY provided example code for websockets.

See: https://github.com/netty/netty/blob/4.1/example/src/main/java/io/netty/example/http/websocketx/client/WebSocketClientHandler.java except I added the following few lines of code for a Server side "ping" to the client.

`else if (frame instanceof PingWebSocketFrame) {
System.out.println("WebSocket Client received ping");

    	ch.writeAndFlush(new PongWebSocketFrame(frame.content().retain()));
    	// ch.writeAndFlush(new PongWebSocketFrame());
    	
    }`

I noticed without the "flush", it still disconnected from the websocket near/around the 10 minute mark. It now works without issue.

Navigation Menu

Repository navigation

Ping/Pong exception in Depth websocket #202

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Ping/Pong exception in Depth websocket #202

Description

Activity

tomastoth commented on Feb 19, 2019

eugene-kamenev commented on Mar 3, 2019

joaopsilva commented on Mar 9, 2019

joaopsilva commented on Mar 9, 2019

eugene-kamenev commented on Mar 10, 2019

tomastoth commented on Mar 11, 2019

tomastoth commented on Mar 11, 2019

eugene-kamenev commented on Mar 12, 2019

tomastoth commented on Mar 12, 2019

eugene-kamenev commented on Mar 12, 2019

tomastoth commented on Mar 12, 2019

sf202020sf commented on Apr 10, 2020

diffstorm commented on May 27, 2020

cdfleischmann commented on Jun 13, 2020

cdfleischmann commented on Jun 16, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions