Quantcast
Channel: Recent Questions - Stack Overflow
Viewing all articles
Browse latest Browse all 18095

How to scrape through Single page Application websites in python using bs4

$
0
0

I am scraping players name through the NBA website. The player's name webpage is designed using a single page application. The Players are distributed across several pages in alphabetical order. I am unable to extract the names of all the players.Here is the link: https://in.global.nba.com/playerindex/

from selenium import webdriverfrom bs4 import BeautifulSoupclass make():    def __init__(self):        self.first=""        self.last=""driver= webdriver.PhantomJS(executable_path=r'E:\Downloads\Compressed\phantomjs-2.1.1-windows\bin\phantomjs.exe')driver.get('https://in.global.nba.com/playerindex/')html_doc = driver.page_sourcesoup = BeautifulSoup(html_doc,'lxml')names = []layer = soup.find_all("a",class_="player-name ng-isolate-scope")for a in layer:    span = a.find("span",class_="ng-binding")    thing = make()    thing.first = span.text    spans = a.find("span",class_="ng-binding").find_next_sibling()    thing.last = spans.text    names.append(thing)

Viewing all articles
Browse latest Browse all 18095

Trending Articles