Hi Everyone,
First of all thanks for the assistance. I am a noob i python and need assistance with some website parsing.
The website requires a login which i managed to do and managed to print the page i needed (one of many).
the issue i have is that i am trying to parse the page and search for two specific things.
My code is
i need to pull the name of the member and the user-id
it can be seen in this line
how it is clear.
thanks
First of all thanks for the assistance. I am a noob i python and need assistance with some website parsing.
The website requires a login which i managed to do and managed to print the page i needed (one of many).
the issue i have is that i am trying to parse the page and search for two specific things.
My code is
from requests import Session
from bs4 import BeautifulSoup as bs
with Session() as s:
site = s.get("https://connectedinvestors.com/login")
bs_content = bs(site.content, "html.parser")
token = bs_content.find("input", {"name":"infusion_id"})["value"]
login_data = {"email":"myemailaddress","password":"mypassword", "infusion_id":token}
s.post("https://connectedinvestors.com/login",login_data)
home_page = s.get("https://connectedinvestors.com/friends-list")
print(home_page.content)when i do the print i get entire page data which looks like this (just a sample)Output:class="row" style="">\r\n <div class="col-sm-4">\r\n <div itemscope itemtype="http://schema.org/Organization" class="investorcard clearfix" id="454517">\n <div class="cardarea">\n <div class="banner small">\n <span itemprop="department">Single-family <span class="hidden-md hidden-sm hidden-xs">Fix & Flip</span></span>\n <span class="pull-right">Founder & Managing Partner</span>\n </div>\n <div class="cardarea-figure-container row level-2">\n <div class="card-link-container friends">\n <a itemprop="url" href="/member/jonathan-kessous/friends" class="card-link">\n <i title="This member is directly connected with 10261 real estate professionals" class="fa fa-user icon"> 10261</i>\n </a>\n </div>\n <div class="cardarea-figure-col1 col-sm-12 col-lg-5">\n <a itemprop="url" href="/member/jonathan-kessous">\n <figure class="circle profile bordered bordered-level-2">\n <img itemprop="image" src="/uploads/user/454517/img_59cd6354c15a8.jpg" alt="Jonathan Kessous"/>\n </figure>\n </a>\n </div>\n <div class="cardarea-figure-col2 col-sm-12 col-lg-7">\n <h4 class="card-name" itemprop="founder">\n <a itemprop="url" href="/member/jonathan-kessous">\n Jonathan Kessous </a>\n </h4>\n <h5 itemprop="legalName" class="card-company">\n Global Investments Group LLC </h5>\n <address class="small no-margin" itemscope itemtype="http://schema.org/Address">\n <span itemprop="locality">CLEVELAND</span>, <span itemprop="region">OH</span>\n <br>\n </address>\n\n\n </div>\n </div>\n <div class="row clear">\n <div class="col-lg-12">\n <div class="cardbottom">\n <div class="col-sm-12">\n <div class="pad-medium vertical-pad-only">\n \n \n \n \n </div>\n </div>\n </div>\n </div>\n </div>\n </div>\n</div>\n </div>\r\n <div class="col-sm-8">\r\n <section class="sub fade-light">\r\n <form role="form" action="/friends-list" method="post">\r\n <div class="row">\r\n <div class="form-group col-sm-4">\r\n <label for="zipcode" class="col-sm-12 control-label">Locations of interest</label>\r\n <div class="col-sm-12">\r\n \n<input type="text" name="zipcode" id="zipcode" value="" class="col-sm-12 form-control" placeholder="Enter Zip Code or City,State" /> </div>\r\n </div>\r\n <div class="form-group col-sm-4">\r\n <label for="strategy" class="col-sm-12 control-label">Strategy</label>\r\n <div class="col-sm-12">\r\n \n<select name="strategy" id="strategy" class="col-sm-12 form-control" maxlength="10" 0="All" 1="Wholesale" 4="Fix & Flip" 5="Buy & Hold" 2="Develop" 3="Lend" 6="Learn">\n <option value="0" label="All">All</option>\n <option value="1" label="Wholesale">Wholesale</option>\n <option value="4" label="Fix & Flip">Fix & Flip</option>\n <option value="5" label="Buy & Hold">Buy & Hold</option>\n <option value="2" label="Develop">Develop</option>\n <option value="3" label="Lend">Lend</option>\n <option value="6" label="Learn">Learn</option>\n</select> </div>\r\n </div>\r\n <div class="form-group col-sm-4">\r\n <label for="strategy" class="col-sm-12 control-label">Focus</label>\r\n <div class="col-sm-12">\r\n \n<select name="focus" id="focus" class="col-sm-12 form-control" maxlength="10">\n <option value="0" label="All">All</option>\n <option value="1" label="Single-family">Single-family</option>\n <option value="2" label="Multi-family">Multi-family</option>\n <option value="3" label="Commercial">Commercial</option>\n <option value="4" label="Land">Land</option>\n <option value="5" label="Notes">Notes</option>\n <option value="6" label="Tax Liens">Tax Liens</option>\n</select> </div>\r\n </div>\r\n </div>\r\n <div class="row">\r\n <div class="form-group col-sm-4">\r\n <label for="strategy" class="col-sm-12 control-label">Licenses</label>\r\n <div class="col-sm-12">\r\n \n<select name="licenses" id="licenses" class="col-sm-12 form-control" maxlength="10">\n <option value="" label="All">All</option>\n <option value="1" label="Real Estate Agent">Real Estate Agent</option>\n <option value="2" label="Real Estate Broker">Real Estate Broker</option>\n <option value="3" label="Real Estate Mortgage">Real Estate Mortgage</option>\n <option value="4" label="Attorney">Attorney</option>\n <option value="5" label="CCIM">CCIM</option>\n <option value="6" label="CPA">CPA</option>\n <option value="7" label="CFA">CFA</option>\n <option value="22" label="Contractor">Contractor</option>\n <option value="23" label="Appraiser">Appraiser</option>\n <option value="24" label="Financial Advisor">Financial Advisor</option>\n <option value="25" label="Lender">Lender</option>\n</select> </div>\r\n </div>\r\n <div class="form-group col-sm-4">\r\n <label for="rank" class="col-sm-12 control-label"> </label>\r\n <div class="col-sm-12">\r\n <input type="submit" class="btn btn-brightgreen" value="Search">\r\n <input type="reset" class="btn btn-warning" value="Reset">\r\n </div>\r\n </div>\r\n </div>\r\n </form>\r\n </section>\r\n </div>\r\n </div>\r\n</section>\r\n </div>\n </div>\n <br/>\n <div class="row">\n <div class="col-sm-12">\n <section class="parent tab-content list">\n <header>\n <h2>Jonathan Kessous\'s Friends</h2>\n </header>\n <div id="loadingList"><img src="/_/img/ajax-loader.gif"> Loading Results...</div>\n <div id="friends" class="row" style="display: none;">\n <div class="pull-left user-block">\n <div itemscope itemtype="http://schema.org/Organization" class="investorcard clearfix" id="15401">\n <div class="cardarea">\n <div class="banner small">\n <span itemprop="department">Single-family <span class="hidden-md hidden-sm hidden-xs">Wholesale</span></span>\n <span class="pull-right">CEO</span>\n </div>\n <div class="cardarea-figure-container row level-2">\n <div class="card-link-container settings">\n <div class="settings-container">\n <a href="javascript:;" class="card-link">\n <i class="fa fa-gear"></i>\n </a>\n <ul class="dropdown-menu">\n <li>\n <a data-toggle="modal" role="menuitem" data-target="#smallModal" href="/view/flag/rudy-acosta/15401" id="MU15401" rel="nofollow">\n <i class="fa fa-flag"></i>\n Report\n </a>\n </li>\n <li>\n <a class="no-margin" data-click="removeFriend" href="/friend-delete/rudy-acosta">\n <i class="fa fa-user icon icon-bright"></i>\n Un-Friend\n </a>\n </li>\n <li>\n <a class="no-margin" href="/message-list/friend/15401">\n <i class="fa fa-envelope icon icon-bright"></i>\n View all messages\n </a>\n </li>\n </ul>\n </div>\n </div>\n <div class="card-link-container friends">\n <a itemprop="url" href="/member/rudy-acosta/friends" class="card-link">\n <i title="This member is directly connected with 373 real estate professionals" class="fa fa-user icon"> 373</i>\n </a>\n </div>\n <div class="cardarea-figure-col1 col-sm-12 col-lg-5">\n <a itemprop="url" href="/member/rudy-acosta">\n <figure class="circle profile bordered bordered-level-2">\n <img itemprop="image" src="/uploads/user/15401/img_544bf971f23de.jpg" alt="Rudy Acosta"/>\n </figure>\n </a>\n </div>\n <div class="cardarea-figure-col2 col-sm-12 col-lg-7">\n <h4 class="card-name" itemprop="founder">\n <a itemprop="url" href="/member/rudy-acosta">\n Rudy Acosta </a>\n </h4>\n <h5 itemprop="legalName" class="card-company">\n RSA Investor LLC </h5>\n <address class="small no-margin" itemscope itemtype="http://schema.org/Address">\n <span itemprop="locality">BOLINGBROOK</span>, <span itemprop="region">IL</span>\n <br>\n </address>\n\n\n </div>\n </div>\n <div class="row clear">\n <div class="col-lg-12">\n <div class="cardbottom">\n <div class="col-sm-12">\n <hr class="margin-mild"/>\n </div>\n <div class="col-sm-12">\n <div class="pad-medium vertical-pad-only">\n \n \n \n \n <a class="btn-popover pull-right message-block btn btn-chrome btn-sm" data-target="#smallModal" role="menuitem" data-toggle="modal" href="/user/15401/message">\n <i class="fa fa-envelope"></i>\n Message\n </a>\n </div>\n </div>\n </div>\n </div>\n </div>\n </div>\n</div>\n </div>\n <div class="pull-left user-block">\n <div itemscope itemtype="http://schema.org/Organization" class="investorcard clearfix" id="542099">\n <div class="cardarea">\n <div class="banner small">\n <span itemprop="department">Multi-family <span class="hidden-md hidden-sm hidden-xs">Buy & Hold</span></span>\n <span itemprop="department" class="pull-right">Agent</span>\n </div>\n <div class="cardarea-figure-container row level-1">\n <div class="card-link-container settings">\n <div class="settings-container">\n <a href="javascript:;" class="card-link">\n <i class="fa fa-gear"></i>\n </a>\n <ul class="dropdown-menu">\n <li>\n <a data-toggle="modal" role="menuitem" data-target="#smallModal" href="/view/flag/henry-angeli/542099" id="MU542099" rel="nofollow">\n <i class="fa fa-flag"></i>\n Report\n </a>\n </li>\n <li>\n <a class="no-margin" data-click="removeFriend" href="/friend-delete/henry-angeli">\n <i class="fa fa-user icon icon-bright"></i>\n Un-Friend\n </a>\n </li>\n <li>\n <a class="no-margin" href="/message-list/friend/542099">\n <i class="fa fa-envelope icon icon-bright"></i>\n View all messages\n </a>\n </li>\n </ul>\n </div>\n </div>\n <div class="card-link-container friends">\n <a itemprop="url" href="/member/henry-angeli/friends" class="card-link">\n <i title="This member is directly connected with 94 real estate professionals" class="fa fa-user icon"> 94</i>\n </a>\n </div>\n <div class="cardarea-figure-col1 col-sm-12 col-lg-5">\n <a itemprop="url" href="/member/henry-angeli">\n <figure class="circle profile bordered bordered-level-1">\n <img itemprop="image" src="/uploads/user/542099/img_5da5bd1188e8e.jpg" alt="Henry Angeli"/>\n </figure>\n </a>\n </div>\n <div class="cardarea-figure-col2 col-sm-12 col-lg-7">\n <h4 class="card-name" itemprop="founder">\n <a itemprop="url" href="/member/henry-angeli">\n Henry Angeli </a>\n </h4>\n <h5 itemprop="legalName" class="card-company">\n Henry Buys Homes LLC </h5>\n <address class="small no-margin" itemscope itemtype="http://schema.org/Address">\n <span itemprop="locality">JACKSONVILLE</span>, <span itemprop="region">FL</span>\n <br>\n </address>\n\n\n </div>\n </div>\n <div class="row clear">\n <div class="col-lg-12">\n <div class="cardbottom">\n <div class="col-sm-12">\n <hr class="margin-mild"/>\n </div>\n <div class="col-sm-12">\n <div class="pad-medium vertical-pad-only">\n <a target="_blank" href="http://www.facebook.com/henrybuyshomes/"><img src="/_/img/facebook.png" alt="fb"/></a>\n \n <a target="_blank" href="http://www.linkedin.com/in/henrybuyshomes/"><img src="/_/img/linkedin.png" alt="linkdedin"/></a>\n \n <a target="_blank" href="http://www.twitter.com/Henrybuyshomes"><img src="/_/img/twitter.png" alt="twitter"/></a>\n \n <a target="_blank" href="http://plus.google.com/108773975694528171981"><img src="/_/img/youtube.png" alt="youtube"/></a>\n \n <a class="btn-popover pull-right message-block btn btn-chrome btn-sm" data-target="#smallModal" role="menuitem" data-toggle="modal" href="/user/542099/message">\n <i class="fa fa-envelope"></i>\n Message\n </a>\n </div>\n </div>\n </div>\n </div>\n </div>\n </div>\n</div>\n </div>\n <div class="pull-left user-block">\n <div itemscope itemtype="http://schema.org/Organization" class="investorcard clearfix" id="417469">\n <div class="cardarea">\n <div class="banner small">\n <span itemprop="department">Single-family <span class="hidden-md hidden-sm hidden-xs">Fix & Flip</span></span>\n <span itemprop="department" class="pull-right">Contractor</span>\n </div>\n <div class="cardarea-figure-container row level-1">\n <div class="card-link-container settings">\n <div class="settings-container">\n <a href="javascript:;" class="card-link">\n <i class="fa fa-gear"></i>\n </a>\n <ul class="dropdown-menu">\n <li>\n <a data-toggle="modal" role="menuitem" data-target="#smallModal" href="/view/flag/eric-horton/417469" id="MU417469" rel="nofollow">\n <i class="fa fa-flag"></i>\n Report\n </a>\n </li>\n <li>\n <a class="no-margin" data-click="removeFriend" href="/friend-delete/eric-horton">\n <i class="fa fa-user icon icon-bright"></i>\n Un-Friend\n </a>\n </li>\n <li>\n <a class="no-margin" href="/message-list/friend/417469">\n <i class="fa fa-envelope icon icon-bright"></i>\n View all messages\n </a>\n </li>\n </ul>\n </div>\n </div>\n <div class="card-link-container friends">\n <a itemprop="url" href="/member/eric-horton/friends" class="card-link">\n <i title="This member is directly connected with 138 real estate professionals" class="fa fa-user icon"> 138</i>\n </a>\n </div>\n <div class="cardarea-figure-col1 col-sm-12 col-lg-5">\n <a itemprop="url" href="/member/eric-horton">\n <figure class="circle profile bordered bordered-level-1">\n <img itemprop="image" src="/uploads/user/417469/img_589c7e1b7b5e7.jpg" alt="Eric Horton"/>\n </figure>\n </a>\n </div>\n <div class="cardarea-figure-col2 col-sm-12 col-lg-7">\n <h4 class="card-name" itemprop="founder">\n <a itemprop="url" href="/member/eric-horton">\n Eric Horton </a>\n </h4>\n <h5 itemprop="legalName" class="card-company">\n E S HORTON, INC. </h5>\n <address class="small no-margin" itemscope itemtype="http://schema.org/Address">\n <span itemprop="locality">OCEANSIDE</span>, <span itemprop="region">CA</span>\n <br>\n </address>\n\n\n </div>\n </div>\n <div class="row clear">\n <div class="col-lg-12">\n <div class="cardbottom">\n <div class="col-sm-12">\n <hr class="margin-mild"/>\n </div>\n <div class="col-sm-12">\n <div class="pad-medium vertical-pad-only">\n \n \n \n \n <a class="btn-popover pull-right message-block btn btn-chrome btn-sm" data-target="#smallModal" role="menuitem" data-toggle="modal" href="/user/417469/message">\n <i class="fa fa-envelope"></i>\n Message\n </a>\n </div>\n </div>\n </div>\n </div>\n </div>\n </div>\n</div>\n </div>\n <div class="pull-left user-block">\n <div itemscope itemtype="http://schema.org/Organization" class="investorcard clearfix" id="376076">\n <div class="cardarea">\n <div class="banner small">\n <span itemprop="department">Single-family <span class="hidden-md hidden-sm hidden-xs">Wholesale</span></span>\n </div>\n <div class="cardarea-figure-container row level-2">\n <div class="card-link-container settings">\n <div class="settings-container">\n <a href="javascript:;" class="card-link">\n <i class="fa fa-gear"></i>\n </a>\n <ul class="dropdown-menu">\n <li>\n <a data-toggle="modal" role="menuitem" data-target="#smallModal" href="/view/flag/dorian-ramos/376076" id="MU376076" rel="nofollow">\n <i class="fa fa-flag"></i>\n Report\n </a>\n </li>\n <li>\n <a class="no-margin" data-click="removeFriend" href="/friend-delete/dorian-ramos">\n <i class="fa fa-user icon icon-bright"></i>\n Un-Friend\n </a>\n </li>\n <li>\n <a class="no-margin" href="/message-list/friend/376076">\n <i class="fa fa-envelope icon icon-bright"></i>\n View all messages\n </a>\n </li>\n </ul>\n </div>\n </div>\n <div class="card-link-container friends">\n <a itemprop="url" href="/member/dorian-ramos/friends" class="card-link">\n <i title="This member is directly connected with 2453 real estate professionals" class="fa fa-user icon"> 2453</i>\n </a>\n </div>\n <div class="cardarea-figure-col1 col-sm-12 col-lg-5">\n <a itemprop="url" href="/member/dorian-ramos">\n <figure class="circle profile bordered bordered-level-2">\n <img itemprop="image" src="/uploads/user/376076/img_5802bfd3a9302.jpg" alt="Dorian Ramos"/>\n </figure>\n </a>\n </div>\n <div class="cardarea-figure-col2 col-sm-12 col-lg-7">\n <h4 class="card-name" itemprop="founder">\n <a itemprop="url" href="/member/dorian-ramos">\n Dorian Ramos </a>\n </h4>\n <h5 itemprop="legalName" class="card-company">\n Merdorian LLC </h5>\n <address class="small no-margin" itemscope itemtype="http://schema.org/Address">\n <span itemprop="locality">MONROE</span>, <span itemprop="region">NY</span>\n <br>\n </address>\n\n\n </div>\n </div>\n <div class="row clear">\n <div class="col-lg-12">\n <div class="cardbottom">\n <div class="col-sm-12">\n <hr class="margin-mild"/>\n </div>\n <div class="col-sm-12">\n <div class="pad-medium vertical-pad-only">\n <a target="_blank" href="http://www.facebook.com/doriangramos"><img src="/_/img/facebook.png" alt="fb"/></a>\n \n <a target="_blank" href="http://www.linkedin.com/in/MerDorianMarketing"><img src="/_/img/linkedin.png" alt="linkdedin"/></a>\n \n <a target="_blank" href="http://www.twitter.com/rock72pd2010"><img src="/_/img/twitter.png" alt="twitter"/></a>\n \n <a target="_blank" href="http://plus.google.com/merdorianmarketing"><img src="/_/img/youtube.png" alt="youtube"/></a>\n \n <a class="btn-popover pull-right message-block btn btn-chrome btn-sm" data-target="#smallModal" role="menuitem" data-toggle="modal" href="/user/376076/message">\n <i class="fa fa-envelope"></i>\n Message\n </a>\n </div>\n </div>\n </div>\n </div>\n </div>\n </div>\n</div>\n </div>\n <div class="pull-left user-block">\n <div itemscope itemtype="http://schema.org/Organization" class="investorcard clearfix" id="4568">\n <div class="cardarea">\n <div class="banner small">\n <span itemprop="department">Single-family <span class="hidden-md hidden-sm hidden-xs">Wholesale</span></span>\n </div>\n <div class="cardarea-figure-container row level-2">\n <div class="card-link-container settings">\n i need to pull the name of the member and the user-id
it can be seen in this line
<a itemprop="url" href="/member/rudy-acosta">\n <figure class="circle profile bordered bordered-level-2">\n <img itemprop="image" src="/uploads/user/15401/img_544bf971f23de.jpg" alt="Rudy Acosta"/>\n how it is clear.
thanks
Reply