line in html:
Output:
<b>Host Software</b> S. Crocker
I want Host software and author separated
i get the title with x.find('b'),
I am tired and this is not poping out of my weary brain
what about author?
whats the next tag after </b> ?
Here's two sets of table entries:
Output:
<tr valign="top">
<td valign="top">
<script type="text/javascript">
doMainDocLink('RFC0001');
</script><noscript>0001</noscript>
</td>
<td>
<b>Host Software</b> S. Crocker
[ April 1969 ]
(TXT = 21088)
(Status: UNKNOWN)
(Stream: Legacy)
(DOI: 10.17487/RFC0001)
</td>
</tr>
<tr valign="top">
<td valign="top">
<script type="text/javascript">
doMainDocLink('RFC0002');
</script><noscript>0002</noscript>
</td>
<td>
<b>Host software</b> B. Duvall
[ April 1969 ]
(TXT = 17145)
(Status: UNKNOWN)
(Stream: Legacy)
(DOI: 10.17487/RFC0002)
</td>
</tr>
The text after the <b> tag varies in length and content
The page is located here:
https://www.rfc-editor.org/rfc-index.html
im actually not sure how to do that other than string splitting after getting that
td
But that is assuming the structure is always
Host Software X. XXXXXX
from bs4 import BeautifulSoup
html = '''
<tr valign="top">
<td valign="top">
<script type="text/javascript">
doMainDocLink('RFC0001');
</script><noscript>0001</noscript>
</td>
<td>
<b>Host Software</b> S. Crocker
[ April 1969 ]
(TXT = 21088)
(Status: UNKNOWN)
(Stream: Legacy)
(DOI: 10.17487/RFC0001)
</td>
</tr>
<tr valign="top">
<td valign="top">
<script type="text/javascript">
doMainDocLink('RFC0002');
</script><noscript>0002</noscript>
</td>
<td>
<b>Host software</b> B. Duvall
[ April 1969 ]
(TXT = 17145)
(Status: UNKNOWN)
(Stream: Legacy)
(DOI: 10.17487/RFC0002)
</td>
</tr>
'''
soup = BeautifulSoup(html, 'html.parser')
tds = soup.find_all('td')
td = tds[1]
print(td.text.split()[:2])
print(td.text.split()[2:4])Output:
['Host', 'Software']
['S.', 'Crocker']
I thought so, even if there is another way, that will work fine so long as i consider
that there may not be any author (presented)
I think that splitting on whitespaces is not enough, there are both longer titles and multiple authors. I tried dirty way with extracting <b> and splitting rest on "[" on your url.
from bs4 import BeautifulSoup as bs
import requests
url = "https://www.rfc-editor.org/rfc-index.html"
soup = bs(requests.get(url).text, 'html.parser')
for btag in soup.select("td b")[1:]:
title = btag.text
author = btag.parent.text[len(title)+1:].partition("[")[0].strip()
print("Title: {}\nAuthor: {}\n".format(title, author))gives
Output:
Title: Augmented BNF for Syntax Specifications: ABNF
Author: D. Crocker, P. Overell
Title: Host Software
Author: S. Crocker
Title: Host software
Author: B. Duvall
Title: Documentation conventions
Author: S.D. Crocker
...
...
Title: ARPA Network Functional Specifications
Author: G. Deloche
Title: Host Software
Author: G. Deloche
I like it, that works well
Now here's the funny part.
I did a little more looking around the web site, and voila, there it was, a text file with everything I was looking for.
Not a loss at all, though, because I learned just a little more.
Now if I can keep that up until I retire at 90, it will be good!