Python Forum
Scraping problems. Pls help with a correct request query.
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Scraping problems. Pls help with a correct request query.
#6
Sorry, I forgot to post
this is a module that allows you to use a variable indent with html, and adds line feeds where appropriate.
Makes html much easier to read
here's the script:
from bs4 import BeautifulSoup
import requests
import pathlib


class PrettifyPage:
    def __init__(self):
        pass

    def prettify(self, soup, indent):
        pretty_soup = str()
        previous_indent = 0
        for line in soup.prettify().split("\n"):
            current_indent = str(line).find("<")
            if current_indent == -1 or current_indent > previous_indent + 2:
                current_indent = previous_indent + 1
            previous_indent = current_indent
            pretty_soup += self.write_new_line(line, current_indent, indent)
        return pretty_soup

    def write_new_line(self, line, current_indent, desired_indent):
        new_line = ""
        spaces_to_add = (current_indent * desired_indent) - current_indent
        if spaces_to_add > 0:
            for i in range(spaces_to_add):
                new_line += " "		
        new_line += str(line) + "\n"
        return new_line

if __name__ == '__main__':
    pp = PrettifyPage()
Reply


Messages In This Thread
RE: Scraping problems. Pls help with a correct request query. - by Larz60+ - Oct-01-2019, 07:17 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  POST request with form data issue web scraping hoff1022 1 2,726 Aug-14-2020, 10:25 AM
Last Post: kashcode
  Scraping a dynamic data-table in python through AJAX request filozofo 1 3,923 Aug-14-2020, 10:13 AM
Last Post: kashcode
  The correct POST request abhie_lp 5 3,045 Jun-05-2020, 07:27 AM
Last Post: buran
  Scraping problems with Python requests. gtlhbkkj 1 1,903 Jan-22-2020, 11:00 AM
Last Post: gtlhbkkj
  Scraping problems. Pls help with a correct request query. gtlhbkkj 0 1,532 Oct-09-2019, 12:00 PM
Last Post: gtlhbkkj
  web scraping to csv formatting problems bluethundr 4 2,829 Jul-04-2019, 02:00 AM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020