Web socket scripts stop at random times

punit

I am running two websocket scripts which subscribe to different et of token (Each subscribe about 40 scripts on average in nse_fo)

Problem is that the scripts ends at random time without any visible exception or errors though the token is valid. Only pattern I could see is that the hearbeat is not received for TS just before the script completion.

for example below, the HB was fine till 11:40, next hb is expected at 11:41:11, but its not received and the script fails 15 seconds later (completes without error, but expectation is not to end)

{'ak': 'ok', 'msg': 'heartbeat', 'task': 'hb', 'ts': '2021-03-23 11:38:11.052232'}
{'ak': 'ok', 'msg': 'heartbeat', 'task': 'hb', 'ts': '2021-03-23 11:39:11.060757'}
{'ak': 'ok', 'msg': 'heartbeat', 'task': 'hb', 'ts': '2021-03-23 11:40:11.119478'}

-- Job completed successfully at 2021/03/23 11:41:26 (GMT+5.5).
-- End of log.

I am not sure if this is due to heartbeat fail and retry, I do not have debug logs.

@rajanprabu @rjbanna , I could see you have posted the debug logs for ws scripts. Could you help me with the ws snippet to capture the exceptions.

BR,
Ashok

admin

Hi @Ashok we have reported this issue to the team we are working on this.We will update you on this.

jiwan1986

@Ashok You can use the logging module in Python for debug purposes. I have attached snippet for your reference.

import logging

logging.basicConfig(filename = 'app.log', filemode = 'w', format = '%(asctime)s - %(message)s', level = logging.DEBUG)

logging.info('Test message')

punit

@rjbanna , Yes I am using this module .. infact I have small function to create the log object and return to the function based on the script name... like below...

import logging
import os.path
from pathlib import Path

def getLogger(name,pathName=False):

    fileName    =  pathName + name + '.log'

    if not os.path.isfile(fileName):
        Path(fileName).touch()

    log_format = '%(asctime)s.%(msecs)03d - %(name)8s - %(message)s'
	
    logging.basicConfig(level       = logging.DEBUG,
                        format      = log_format,
                        filename    = fileName,
						datefmt     ='%Y-%m-%d %H:%M:%S')
						
    return logging.getLogger(name)

and using the returned logger to log the exceptions / errors.

However I cannot catch the exceptions where the failed heartbeat is.. like the ones shown by @rajanprabu

punit

@admin , the problem resurfaced today with much more impact, due to which script could not close the SL due to the non availability of ticks resulting in few thousands of loss booking.

here is what happened..

{'ak': 'ok', 'msg': 'heartbeat', 'task': 'hb', 'ts': '2021-03-25 13:20:43.049293'}
{'ak': 'ok', 'msg': 'heartbeat', 'task': 'hb', 'ts': '2021-03-25 13:21:43.108353'}
{'ak': 'ok', 'msg': 'heartbeat', 'task': 'hb', 'ts': '2021-03-25 13:22:43.175920'}
{'ak': 'ok', 'msg': 'heartbeat', 'task': 'hb', 'ts': '2021-03-25 13:24:06.394106'}
{'ak': 'ok', 'msg': 'heartbeat', 'task': 'hb', 'ts': '2021-03-25 13:24:43.777559'}

# Job completed successfully at 2021/03/25 13:43:46 (GMT+5.5).

After the heartbeat @ 12:43, tickets stooped to come, hearbeat not failed and script was waiting for the ticks..
the failed hb happened only at 13:43 and scripts got complete and new instance was started as per my set-up.
During this 20 minutes, market trend changed and the positions were blind..

I do not know if these are fixed in latest version, but this un-reliability will dearly cost traders and in turn AB/Smartapi

@rjbanna @rajanprabu , Did you ever faced these issues ?

BR
Ashok

jiwan1986

@Ashok No need to create object for logger. You can use my code. I'm using that code and I'm able to see all the exceptions including heartbeat failed.

jiwan1986

@Ashok I haven't encountered this issue yesterday, though sometimes heartbeat failed. I was receiving the ticks even though the heartbeats failed message kept coming. Socket was reconnected but heartbeat function was not aware about it. There is issue with heartbeat function which should be resolved as soon as possible. This is why it is important to place SL order right after the first order is placed. Because you don't know when something gets broken in future. @admin please look into this.

admin

Hi @rjbanna We have already reported the issue to the team we are trying to solve it asap.

Surya 1

@rjbanna @Ashok

Yes I faced it twice yesterday.. But whoever this happened I normally stopped the script and restarted it. I think angel team is working on it.

jiwan1986

@rajanprabu Yes, I do the same but my algo make use of some past calculations and live candles. So if I start it again, I would need to calculate all the values manually which is cumbersome process. Angel team is working on it.

Surya 1

@rjbanna

Agreed.. its pretty hard. Hope this gets resolved soon.

admin

@rajanprabu @rjbanna @Ashok we have tried to handle all this issues in the new version of python sdk.We will be testing it and reverting you back soon.

punit

@rjbanna, Still am not very sure on it... If its not too much trouble, shall we connect privately ?

Surya 1

@admin

I have tested the new SDK.. so far it passed all my stress tests.. I will continue my testing and I will keep it posted. Thanks to the team. Very good job.

jiwan1986

@admin I'm also testing the new SDK, no issues till now. Thanks for quick issue resolution.

punit

@admin @rjbanna Thanks. Moved to new SDK 1.2.3 now... Thanks for support

Surya 1

@Ashok

Pease note new SDK needs changes to the script. You need to add ws.websocket_connection() to on_connect

from smartapi import WebSocket 
FEED_TOKEN= "your feed token"
CLIENT_CODE="your client Id"
token="channel you want the information of" #"nse_cm|2885&nse_cm|1594&nse_cm|11536"
task="task" #"mw"|"sfi"|"dp"
ss = WebSocket(FEED_TOKEN, CLIENT_CODE)

def on_tick(ws, tick):
    print("Ticks: {}".format(tick))

def on_connect(ws, response):
    ws.websocket_connection() 
    ws.send_request(token,task) 
    
def on_close(ws, code, reason):
    ws.stop()

# Assign the callbacks.
ss.on_ticks = on_tick
ss.on_connect = on_connect
ss.on_close = on_close

ss.connect( )

punit

@rajanprabu Noted Rajan.. bhavesh informed me about that...

jiwan1986

@admin Today also I got heartbeat failed message.

punit

@rjbanna Same here.