|
Search Results
|
| Post |
Author |
Forum |
Replies |
Views |
Posted
[asc]
|
| |
|
Thread: Need Help Typing Text into Tough Form [xpath / selenium]
Post: Need Help Typing Text into Tough Form [xpath / sel...
| I'm struggling **huh** I have no idea how to proceed. I just need to send text to these 2 fields.
The code is the exact same for both fields and its dynamic - it will switch between [1] and [2] de... |
|
digitalmatic7 |
General Coding Help |
0 |
1,746 |
Jun-05-2019, 06:46 AM |
| |
|
Thread: Run Python Script From My VPS?
Post: Run Python Script From My VPS?
| I have a VPS with CentOS7 / nginx / plesk installed on it. I was wondering how I can use it to run a Python script? Do I have to go through the whole process of installing python onto the server, simi... |
|
digitalmatic7 |
General Coding Help |
1 |
2,857 |
Feb-20-2019, 09:58 PM |
| |
|
Thread: Need To Scrape Some Links
Post: RE: Need To Scrape Some Links
| (Oct-09-2018, 12:30 AM)micseydel Wrote: For future reference, providing that in output tags instead of linking off-site would be appreciated. I'm in between tasks at work at the moment but I usually... |
|
digitalmatic7 |
Web Scraping & Web Development |
2 |
2,645 |
Oct-09-2018, 02:33 AM |
| |
|
Thread: Need To Scrape Some Links
Post: Need To Scrape Some Links
| All links I need to scrape contain "www/delivery"
Example snippets of how it can look in the source code https://pastebin.com/3Lgyb3Cd
As you see there's so much variation, I find it very hard to gr... |
|
digitalmatic7 |
Web Scraping & Web Development |
2 |
2,645 |
Oct-09-2018, 12:18 AM |
| |
|
Thread: Web Scraping with a Bot
Post: RE: Web Scraping with a Bot
| If I remember correctly Distil has very advanced javascript bot detection. You'll need to be a very good coder to bypass it. You might want to try macros on your normal browser and slow down your requ... |
|
digitalmatic7 |
Web Scraping & Web Development |
8 |
8,506 |
May-07-2018, 03:14 AM |
| |
|
Thread: BS4 - How Can I Scrape These Links?
Post: BS4 - How Can I Scrape These Links?
| [Image: https://image.prntscr.com/image/QJCy04MT...nLQRSA.png]
I need to scrape all the URLS on the page but as you can see the href has no class or ID to hook into.
Is it possible to scrape all th... |
|
digitalmatic7 |
Web Scraping & Web Development |
1 |
2,327 |
May-07-2018, 02:30 AM |
| |
|
Thread: Break Up Tuple List Into A Dataframe?
Post: RE: Break Up Tuple List Into A Dataframe?
| Thanks for bringing that issue to light mlieqo
I get it working using this code:
[t if type(t) == tuple else (t,) for t in results] |
|
digitalmatic7 |
General Coding Help |
2 |
4,340 |
Apr-26-2018, 05:29 PM |
| |
|
Thread: Break Up Tuple List Into A Dataframe?
Post: Break Up Tuple List Into A Dataframe?
| It's my first time working with a tuple list, and I've run into a very strange issue.
df = pd.DataFrame(tuple_list)Most items in the tuple list have 3 elements (url, number, sample data). Some items... |
|
digitalmatic7 |
General Coding Help |
2 |
4,340 |
Apr-26-2018, 08:31 AM |
| |
|
Thread: Passing data between functions, got it working, but need clarification please
Post: RE: Passing data between functions, got it working...
| Thanks! I'm feeling more confident I understand passing arguments now, but I tried to apply it to a new script and I'm getting an error.
In pycharm it says:
Parameter 'fruit' unfilledWhen I run the ... |
|
digitalmatic7 |
Data Science |
4 |
15,972 |
Apr-18-2018, 09:30 AM |
| |
|
Thread: Passing data between functions, got it working, but need clarification please
Post: RE: Passing data between functions, got it working...
| Quote:test1 is to be executed and the results will be the argument to test2
hmmm.. I read that part 10x and it finally started clicking a bit, but it still seems kinda weird
test2(df) is pulling the... |
|
digitalmatic7 |
Data Science |
4 |
15,972 |
Apr-18-2018, 07:25 AM |
| |
|
Thread: Passing data between functions, got it working, but need clarification please
Post: Passing data between functions, got it working, bu...
| I made a basic script that passes a dataframe between 2 functions.
I understand how it works, except for the meaning of the last line.
test2(test1())Does this mean that all the code from test1 is ... |
|
digitalmatic7 |
Data Science |
4 |
15,972 |
Apr-18-2018, 06:52 AM |
| |
|
Thread: Problem With Simple Multiprocessing Script
Post: RE: Problem With Simple Multiprocessing Script
| Phew.. I think I'm 95% done with this script.. just hit another issue though **coffee**
I'm taking URLs I want processed, adding them to a dataframe with Pandas, and trying to pass that through map ... |
|
digitalmatic7 |
Web Scraping & Web Development |
11 |
9,227 |
Apr-16-2018, 07:18 PM |
| |
|
Thread: Import Requests Does not Work
Post: RE: Import Requests Does not Work
| I think with Windows you can't install lxml through pip
You need to download the whl file from the website and run it through CMD |
|
digitalmatic7 |
General Coding Help |
12 |
25,353 |
Apr-16-2018, 08:07 AM |
| |
|
Thread: Problem With Simple Multiprocessing Script
Post: RE: Problem With Simple Multiprocessing Script
| I looked over some __init__ tutorials, but I still don't really understand what that does. How would I setup Counter class within the scraper function?
I was playing around with multiprocessing mana... |
|
digitalmatic7 |
Web Scraping & Web Development |
11 |
9,227 |
Apr-13-2018, 06:59 AM |
| |
|
Thread: Problem With Simple Multiprocessing Script
Post: RE: Problem With Simple Multiprocessing Script
| I've run into issues getting a counter to work inside the scraper function.
I just need a very basic counter that increments for each URL (iteration) that is processed. I tried using a global variab... |
|
digitalmatic7 |
Web Scraping & Web Development |
11 |
9,227 |
Apr-12-2018, 07:51 AM |
| |
|
Thread: Problem With Simple Multiprocessing Script
Post: RE: Problem With Simple Multiprocessing Script
| (Apr-10-2018, 06:55 PM)nilamo Wrote: (Apr-10-2018, 06:40 PM)digitalmatic7 Wrote: p.map(scraper, list1)map will call the function, scraper, once for each item in the iterable, list1. It might be a... |
|
digitalmatic7 |
Web Scraping & Web Development |
11 |
9,227 |
Apr-10-2018, 08:52 PM |
| |
|
Thread: Problem With Simple Multiprocessing Script
Post: RE: Problem With Simple Multiprocessing Script
| Hmmmm. I got it working.. but.. now I'm so confused..
How does it manage to pull "url" from list1? I was surprised this script actually works lol.
from multiprocessing import Lock, Pool
from time im... |
|
digitalmatic7 |
Web Scraping & Web Development |
11 |
9,227 |
Apr-10-2018, 06:40 PM |
| |
|
Thread: Problem With Simple Multiprocessing Script
Post: Problem With Simple Multiprocessing Script
| Is it possible to multiprocess this script without breaking it up into functions?
I'm trying to keep it as barebones and simple as possible.
# EXTREMELY SIMPLE SCRAPING SCRIPT
from time import s... |
|
digitalmatic7 |
Web Scraping & Web Development |
11 |
9,227 |
Apr-09-2018, 06:38 PM |
| |
|
Thread: Can I browse the net using python?
Post: RE: Can I browse the net using python?
| You can use Selenium with Chromedriver and a UK proxy.
If you just want to browse the Internet and appear to be from the UK, you can use your regular browser with a proxy. Selenium is more for autom... |
|
digitalmatic7 |
Web Scraping & Web Development |
3 |
2,989 |
Apr-06-2018, 04:38 AM |
| |
|
Thread: Request Get Timeout Issue
Post: Request Get Timeout Issue
| I've been cycling through a list of 1000 URLs and scraping the source.. everything works great, but every once in a while I hit a problematic URL that keeps timing out over and over.
HTTPConnectionP... |
|
digitalmatic7 |
Web Scraping & Web Development |
1 |
7,613 |
Apr-03-2018, 10:54 PM |
