|
Search Results
|
| Post |
Author |
Forum |
Replies |
Views |
Posted
[asc]
|
| |
|
Thread: Logging into a website with requests
Post: RE: Logging into a website with requests
| I found an easier way to do it. For anyone interested, they have an RSS feed that generates a unique url for each account and does not require a login. You can navigate your saved searches to find the... |
|
HiImNew |
Web Scraping & Web Development |
3 |
6,348 |
Jul-06-2018, 06:00 AM |
| |
|
Thread: Logging into a website with requests
Post: RE: A crazy project involving webpage interaction
| Edit: Wrong thread, mb.
I tried logging in manually and then copying over my requests headers like this:
headers = {
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,im... |
|
HiImNew |
Web Scraping & Web Development |
3 |
6,348 |
Jul-05-2018, 09:37 AM |
| |
|
Thread: Logging into a website with requests
Post: Logging into a website with requests
| I'm trying to log into globenewswire.com with requests on my reader account. After I log in I want to go to a page on my account that requires my login to access. I'm using this code to basically pars... |
|
HiImNew |
Web Scraping & Web Development |
3 |
6,348 |
Jul-04-2018, 04:59 AM |
| |
|
Thread: Need help Multiprocessing with BeautifulSoup
Post: RE: Need help Multiprocessing with BeautifulSoup
| Figured out a solution. For anyone interested:
import math
from multiprocessing import Pool
from multiprocessing import cpu_count
# results_list = list of requests.get(<url>).content items (list... |
|
HiImNew |
Web Scraping & Web Development |
4 |
5,999 |
Jun-07-2018, 05:31 AM |
| |
|
Thread: Need help Multiprocessing with BeautifulSoup
Post: RE: Need help Multiprocessing with BeautifulSoup
| I fixed it, put back the in. Sorry that was an attempt from a while ago (few hours) and had to recreate it form memory. |
|
HiImNew |
Web Scraping & Web Development |
4 |
5,999 |
Jun-06-2018, 08:23 AM |
| |
|
Thread: Need help Multiprocessing with BeautifulSoup
Post: Need help Multiprocessing with BeautifulSoup
| I am trying to look through html for certain tags and (if a certain tag is found, have python notify me as quickly as possible) This is my code so far:
import ast
import bs4 as bs
doc = open('C:/Users... |
|
HiImNew |
Web Scraping & Web Development |
4 |
5,999 |
Jun-06-2018, 04:00 AM |
| |
|
Thread: Web Scraping efficiency improvement
Post: Web Scraping efficiency improvement
| Is there any way to speed up a web-scraper by having multiple computers contribute to processing a list of urls? Like computer A takes urls 1 - 500 and computer B takes urls 501 - 1000, etc. I am look... |
|
HiImNew |
General Coding Help |
0 |
2,387 |
Jun-01-2018, 08:52 PM |
| |
|
Thread: Web Scraping with a Bot
Post: RE: Web Scraping with a Bot
| Selenium doesnt work either. I was thinking of using my actual browser and automating that, but theyve banned my ip cause they know its connected to a bot. They havent really banned it but loading a p... |
|
HiImNew |
Web Scraping & Web Development |
8 |
8,507 |
May-07-2018, 05:46 PM |
| |
|
Thread: Web Scraping with a Bot
Post: RE: Web Scraping with a Bot
| I tried this exactly in IDE:
>>> import requests
>>> proxies = {
'http': 'http://159.89.191.201:3128',
'https': 'http://159.89.191.201:3128',
}
>>> s = requests.Sess... |
|
HiImNew |
Web Scraping & Web Development |
8 |
8,507 |
May-06-2018, 07:09 PM |
| |
|
Thread: Web Scraping with a Bot
Post: RE: Web Scraping with a Bot
| I am using the modules: beautifulSoup, urllib, and requests. do does have user agents? I thought that was something only modules that mimicked browers carried, like Selenium. |
|
HiImNew |
Web Scraping & Web Development |
8 |
8,507 |
May-06-2018, 03:55 PM |
| |
|
Thread: Web Scraping with a Bot
Post: Web Scraping with a Bot
| I wanted to set up a daily web-scraper for this site. The scraper does about 100 requests/hour every day, however the site seems to be blocking all bots. This is the url: 'https://www.nvsos.gov/' and ... |
|
HiImNew |
Web Scraping & Web Development |
8 |
8,507 |
May-06-2018, 06:35 AM |
| |
|
Thread: TypeError: must be str, not ResultSet
Post: RE: TypeError: must be str, not ResultSet
| (Feb-15-2018, 06:35 AM)Larz60+ Wrote: Use this url: https://www.businesswire.com/portal/site...PAGE/news/
It would really be helpful if you could show the error verbatim.
They contain some very use... |
|
HiImNew |
Web Scraping & Web Development |
3 |
8,099 |
Feb-15-2018, 06:44 AM |
| |
|
Thread: TypeError: must be str, not ResultSet
Post: TypeError: must be str, not ResultSet
| I have tried freaking everything and for some reason google searches aren't providing much.
This is code to parse multiple pages of the same url, but for some reason I keep on getting this error.
... |
|
HiImNew |
Web Scraping & Web Development |
3 |
8,099 |
Feb-15-2018, 06:21 AM |
| |
|
Thread: Pygsheets Keeps inserting a blank row, Need Help
Post: Pygsheets Keeps inserting a blank row, Need Help
| I apoligize in advance for the overly complicated code, this is just a segment of a larger part.
while True:
import bs4 as bs
import urllib.request
import re
import pygsheets
import httplib2
h... |
|
HiImNew |
General Coding Help |
1 |
3,210 |
Jan-26-2018, 02:07 AM |
| |
|
Thread: Need Help Opening A New Tab in Selenium
Post: RE: Need Help Opening A New Tab in Selenium
| >>> driver.execute_script("window.open('');")This actually did switch focus to the new window it created. Thank you again metulburr! :)
That one line was all I needed.
Edit: I realized what... |
|
HiImNew |
Web Scraping & Web Development |
2 |
39,371 |
Jan-15-2018, 05:53 AM |
| |
|
Thread: Need Help Opening A New Tab in Selenium
Post: Need Help Opening A New Tab in Selenium
| It seems simple enough, and I've tried a few different ways but none of them have worked so far. To Open a new tab in Chrome (which works the same way when done manually by hand in Selenium's Browser)... |
|
HiImNew |
Web Scraping & Web Development |
2 |
39,371 |
Jan-14-2018, 08:05 PM |
| |
|
Thread: Unable to load cookies with Pickle
Post: RE: Unable to load cookies with Pickle
| (Jan-04-2018, 03:05 AM)snippsat Wrote: You seems to do some do some strange stuff like running all in a while loop for interactive interpreter(>>>).
The while loop is not needed as Selenium... |
|
HiImNew |
Web Scraping & Web Development |
5 |
11,830 |
Jan-04-2018, 03:37 AM |
| |
|
Thread: Unable to load cookies with Pickle
Post: RE: Unable to load cookies with Pickle
| THANK YOU DADDY! IT WORKED!!! |
|
HiImNew |
Web Scraping & Web Development |
5 |
11,830 |
Jan-04-2018, 02:44 AM |
| |
|
Thread: Unable to load cookies with Pickle
Post: Unable to load cookies with Pickle
| Full Code:
>>> while True:
from selenium import webdriver
driver = webdriver.Chrome()
import pickle
cookies = pickle.load(open("cookies.pkl", "rb"))
while True:
for cookie in cookies:
... |
|
HiImNew |
Web Scraping & Web Development |
5 |
11,830 |
Jan-04-2018, 01:30 AM |
| |
|
Thread: Need Help with Error #503
Post: RE: Need Help with Error #503
| Completely worked, thank you dude! |
|
HiImNew |
Web Scraping & Web Development |
5 |
11,262 |
Dec-29-2017, 05:59 AM |
