Where does screeni-py Fetches its data from?

[sachinkar]

Hello. I just stumbled upon this interesting tool. Ran a few test queries and saw a line that says "Getting Stock Codes From NSE..." which got me wondering how does screeni-py get its data? Do you host a separate private server/DB where it sends all its queries and gets the data from or do you use NSE's API or something? I haven't gone through the source code yet so just figured I'd leave a question here in Q&A section.

[pranjal-joshi]

Hi @sachinkar,
The Fetcher class uses the CSV file from the old NSE website to get the group of stocks by their market cap. This only provides the Name of the stocks and not the OHLC data.
Please view the following source code to get more idea

Screeni-py/src/classes/Fetcher.py

Lines 37 to 68 in 6f9bb0f

	def fetchCodes(self, tickerOption,proxyServer=None):
	listStockCodes = []
	if tickerOption == 12:
	return list(nse.get_stock_codes(cached=False))[1:]
	tickerMapping = {
	1: "https://www1.nseindia.com/content/indices/ind_nifty50list.csv",
	2: "https://www1.nseindia.com/content/indices/ind_niftynext50list.csv",
	3: "https://www1.nseindia.com/content/indices/ind_nifty100list.csv",
	4: "https://www1.nseindia.com/content/indices/ind_nifty200list.csv",
	5: "https://www1.nseindia.com/content/indices/ind_nifty500list.csv",
	6: "https://www1.nseindia.com/content/indices/ind_niftysmallcap50list.csv",
	7: "https://www1.nseindia.com/content/indices/ind_niftysmallcap100list.csv",
	8: "https://www1.nseindia.com/content/indices/ind_niftysmallcap250list.csv",
	9: "https://www1.nseindia.com/content/indices/ind_niftymidcap50list.csv",
	10: "https://www1.nseindia.com/content/indices/ind_niftymidcap100list.csv",
	11: "https://www1.nseindia.com/content/indices/ind_niftymidcap150list.csv"}
	url = tickerMapping.get(tickerOption)
	try:
	if proxyServer:
	res = requests.get(url,proxies={'https':proxyServer})
	else:
	res = requests.get(url)
	cr = csv.reader(res.text.strip().split('\n'))
	next(cr) # skipping first line
	for row in cr:
	listStockCodes.append(row[2])
	except Exception as error:
	print(error)
	return listStockCodes

After getting the stock names, screenipy uses Yahoo finance (yfinance) API to get OHLC data as per the screening configuration during the market hours.

Screeni-py/src/classes/Fetcher.py

Lines 111 to 118 in 6f9bb0f

	data = yf.download(
	tickers=stockCode+".NS",
	period=period,
	interval=duration,
	proxy=proxyServer,
	progress=False,
	timeout=10
	)

In After-Market hours, the GitHub Actions automatically fetch & cache the closing data from the same API and the screenipy will be using this cache instead of re-fetching through yfinance because the market is not ticking any live data and cache is several times faster than pulling the data from the API.

[sachinkar]

Hello @pranjal-joshi, Thanks for your reply. I now have a better understanding of the tool.

As you mentioned, the Fetcher class uses the CSV file from the old NSE website to get the group of stocks by their market cap, I was wondering why not use the new website to do it, as we all know the old site's life if now numbered and soon to be deprecated and removed. I've complied the link source of the new site for you to use, here it is:

https://archives.nseindia.com/content/indices/ind_nifty50list.csv
https://archives.nseindia.com/content/indices/ind_niftynext50list.csv
https://archives.nseindia.com/content/indices/ind_nifty100list.csv
https://archives.nseindia.com/content/indices/ind_nifty200list.csv
https://archives.nseindia.com/content/indices/ind_nifty200list.csv
https://archives.nseindia.com/content/indices/ind_nifty500list.csv
https://archives.nseindia.com/content/indices/ind_niftysmallcap50list.csv
https://archives.nseindia.com/content/indices/ind_niftysmallcap100list.csv
https://archives.nseindia.com/content/indices/ind_niftysmallcap250list.csv
https://archives.nseindia.com/content/indices/ind_niftymidcap50list.csv
https://archives.nseindia.com/content/indices/ind_niftymidcap100list.csv
https://archives.nseindia.com/content/indices/ind_niftymidcap150list.csv

Furthermore, I also see that currently, screenipy is NSE only. How about adding BSE screening as well? We can call all the relevant data from BSE site as well for all the indices, as we can see on BSE's Bhav Copy page here: https://www.bseindia.com/markets/MarketInfo/BhavCopy.aspx
Direct link would be something like this: https://www.bseindia.com/download/BhavCopy/Equity/EQ_ISINCODE_100522.zip /
https://www.bseindia.com/download/BhavCopy/Equity/EQ100522_CSV.ZIP and soo on.

As per the Index data, we can fetch it from the Historical Data page here: https://www.bseindia.com/Indices/IndexArchiveData.html

Direct link would be something like this: https://api.bseindia.com/BseIndiaAPI/api/ProduceCSVForDate/w?strIndex=SENSEX&dtFromDate=10/05/2022&dtToDate=10/05/2022

I haven't done proper research on these yet, but I am sure we can figure out faster and better ways if we put in our time.

[pranjal-joshi]

Great!
We can update the CSV URLs in the new version. Please go through the Contribution guidelines given on the repo and feel free to make a contribution if possible.
It was a personal preference to consider NSE only as usually there is good volume and liquidity is observed on NSE as compared to BSE. Also, the Free account of TradingView supports live charts from NSE but 15 minute delayed charts from BSE.
There are currently no facilities for Index & F&O trading in the current version of Screeni-py (v1.35) but looking forward for a feature development in collaboration if possible.

[sachinkar]

Yes, I'd surely contribute. As I am new to python I am eager to learn. And since I am an active trader I think I can come in with a lot of new feature suggestions in the coming days as I use and test the tool more. Will keep in touch.

[pranjal-joshi]

CSV Links have been updated in b98f629