Aug-03-2019, 04:00 PM
I managed to fix the code. here is the new one:
import requests from bs4 import BeautifulSoup def spider(max_pages): page = 1 while page <= max_pages: main_page = 'https://wikipedia.org/wiki/' search = input("Enter your search: ") page_to_search = main_page + str(search) source_code = requests.get(page_to_search) plain_text = source_code.text soup = BeautifulSoup(plain_text, features="html.parser") for menu in soup.findAll('span', class_='toctext'): print(menu.text) spider(1)