Python Obstacles | Jeet-Kune-Do | BS4 (Tags > MariaDB) [URL/Local HTML]

[BrandonKastning]

Python Obstacles | Jeet-Kune-Do | BS4 (Tags > MariaDB) [URL/Local HTML]

I am working on solving my Thread: "Securing King James Bible (KJV) + King James Bible 1611 (KJV1611) in MariaDB/MySQL"

Source Blogs/Tutorials: https://itsmycode.com/python-urllib-erro...forbidden/ (Helped Resolve 403 Forbidden & Taught me how to implement User-Agent Spoofing within my code).

I am looking to achieve the following and this is my progress so far!

A.1) URL > MariaDB using PyMySQL and BeautifulSoup4 (Single File = Single Row Parsed) [Currently Working]

Database Name: HolyBible
Table Name: Jeet-Kune-Do-A

Database Schematics as Follows:

Column #1: h1_text | Datatype: TEXT | Null
Column #2: p_text | Datatype: TEXT | Null

Working Python 3.9 Code is as Follows:

# 06_Jeet-Kune-Do.A
# 02/07/2022 
# Python 3.9.9
# Objective: Parse a Single HTML File using Beautiful Soup 4 and PyMySQL and Insert to MariaDB
# Target: kingsjamesbibleonline.org
# Securing God's Word in MariaDB for Sharpen Your Sword Ministries 
# Source 1: https://itsmycode.com/python-urllib-error-httperror-http-error-403-forbidden/ | User-Agent Spoofing
# python-forum.io Thread URL: https://python-forum.io/thread-36325-post-153148.html

import urllib.request
import pymysql
import pymysql.cursors
from bs4 import BeautifulSoup

# User-Agent Spoofing - urllib Request
from urllib.request import Request, urlopen
req = Request('https://www.kingjamesbibleonline.org/Matthew-24-24/', headers={'User-Agent': 'Mozilla/5.0'})
html = urlopen(req).read()
#html = urlopen(req,timeout=10).read()

# Assign a Python Variable to Beautiful Soup 4's HTML Parser
htmlParse = BeautifulSoup(html, 'html.parser')

# Select HTML Element for Data Parsing #1
htmlParse.find_all("h1")[0].get_text()

# Select HTML Element for Data Parsing #2
htmlParse.find_all("p")[0].get_text()

# Connection to MariaDB 10.5.x with a Database selected using PyMySQL
connection = pymysql.connect(host='localhost',
                 user='username',
                 password='password',
                 db='HolyBible',
                 charset='utf8mb4',
                 cursorclass=pymysql.cursors.DictCursor)

# Assign a Variable to BeautifulSoup4 Parsing using soup.find_all("") function 
# which is telling BeautifulSoup4 to find all <h1> & <p> tags
# and store tags signified as zero [0]  and then 
# strips the tags using soup.find_all("h1").get_text() & soup.find_all("p").get_text() 
# leaving us only the text to pass to MariaDB for storage
h1_text = htmlParse.find_all("h1")[0].get_text()
p_text = htmlParse.find_all("p")[0].get_text()

try: 
    with connection.cursor() as cursor: 
            sql = "INSERT INTO `Jeet-Kune-Do-A` (`h1_text`,`p_text`) VALUES (%s, %s)" 
            cursor.execute(sql, (h1_text,p_text)) 

    connection.commit() 
finally:
    connection.close()

# Alert Brandon | Payload Delivered Successfully!
print ("Python Script Executed && Payload Delievered Successfully!")

Output after Running 06_Jeet-Kune-Do.A.py:

root@FireDragon:/home/brandon/Python/06_Jeet-Kune-Do/Part.A.Single.HTML.File.2.MariaDB# python3.9 06_Jeet-Kune-Do.A.py 
Python Script Executed && Payload Delievered Successfully!
root@FireDragon:/home/brandon/Python/06_Jeet-Kune-Do/Part.A.Single.HTML.File.2.MariaDB#

Screenshots:


html link creator





A.2) Multiple URL's w/ Loops using a Linklist.txt > MariaDB using PyMySQL and BeautifulSoup4 (Multiple URL's = Multiple Rows Parsed) [Not Sure How to Modify A.1 to create a working A.2]

B.1) Local HTML File (From WGET Download) > MariaDB using PyMySQL and BeautifulSoup4 (Single HTML Local File = Single Row Parsed) [Solved: 02/07/2022 @ Approx. 20:23 ]

Source Tutorials/Blogs: https://gist.github.com/ScribbleGhost/e5...7fdb6ec9ea

Database Name: HolyBible
Table Name: Jeet-Kune-Do-B

Database Schematics as Follows:

Column #1: h1_text | Datatype: TEXT | Null
Column #2: p_text | Datatype: TEXT | Null

Working Python 3.9 Code is as Follows:

# 06_Jeet-Kune-Do.B.1
# 02/07/2022 
# Python 3.9.9
# Objective: Parse a Single HTML File using Beautiful Soup 4 and PyMySQL and Insert to MariaDB
# Target: kingsjamesbibleonline.org
# Securing God's Word in MariaDB for Sharpen Your Sword Ministries 
# Source 1: https://gist.github.com/ScribbleGhost/e5b6a808681004d207a3ee7fdb6ec9ea | Local HTML File Parsing
# python-forum.io Thread URL: https://python-forum.io/thread-36325-post-153148.html

import pymysql
import pymysql.cursors
from bs4 import BeautifulSoup


#soup = BeautifulSoup(open("index.html", encoding="utf8"), "html.parser")
soup = BeautifulSoup(open("Matthew-24-21/index.html", encoding="utf8"), "html.parser")

#print(soup.find_all("div", class_="someclass"))
#print(soup.find_all("h1").get_text())

print(soup.find("h1").get_text())
print(soup.find("p").get_text())

# Connection to MariaDB 10.5.x with a Database selected using PyMySQL
connection = pymysql.connect(host='localhost',
                 user='username',
                 password='password',
                 db='HolyBible',
                 charset='utf8mb4',
                 cursorclass=pymysql.cursors.DictCursor)

# Assign a Variable to BeautifulSoup4 Parsing using soup.find_all("") function 
# which is telling BeautifulSoup4 to find all <h1> & <p> tags
# and store tags signified as zero [0]  and then 
# strips the tags using soup.find_all("h1").get_text() & soup.find_all("p").get_text() 
# leaving us only the text to pass to MariaDB for storage

#h1_text = htmlParse.find_all("h1")[0].get_text()
#p_text = htmlParse.find_all("p")[0].get_text()

h1_text = soup.find("h1").get_text()
p_text = soup.find("p").get_text()

try: 
    with connection.cursor() as cursor: 
            sql = "INSERT INTO `Jeet-Kune-Do-B` (`h1_text`,`p_text`) VALUES (%s, %s)" 
            cursor.execute(sql, (h1_text,p_text)) 

    connection.commit() 
finally: 
    connection.close()

# Alert Brandon | Payload Delivered Successfully!
print ("Python Script Executed && Payload Delievered Successfully!")

Output after Running 06_Jeet-Kune-Do.B.1.py:

brandon@FireDragon:~/Desktop/Exodus.Python.Forum.io.Obstacles/06.Jeet-Kune-Do.B.1$ python3.9 06_Jeet-Kune-Do.B.1.py
Matthew 24:21
“For then shall be great tribulation, such as was not since the beginning of the world to this time, no, nor ever shall be.”
King James Version (KJV)
Python Script Executed && Payload Delievered Successfully!
brandon@FireDragon:~/Desktop/Exodus.Python.Forum.io.Obstacles/06.Jeet-Kune-Do.B.1$

Screenshots:








B.2) Local HTML File(s) in a Folder (Batch from WGET Download) > MariaDB using PyMySQL and BeautifulSoup4 (Multiple HTML Local Files = Multiple Rows Parsed) [Not Sure How to Modify A.2 to create a working B.2]

Thank you everyone for this forum! I will edit this if/when I make progress on my own! Appreciate everyone!

Best Regards,

Brandon!