Mar-26-2025, 03:47 PM
Hi, I’m working on a web scraping project using Python and I'm encountering an issue when using a Chrome Extension to interact with the page. Here’s the scenario:
I’m scraping a website and using the Chrome Extension to interact with the page. Once the extension is done performing its task, it clicks on the page and redirects to a second page. This part works fine.
However, the first page requires the cursor to be moved while the extension is interacting with the page. To solve this, I’m using the human_cursor (or any other web cursor library), and it works sometimes. But here’s the problem:
Sometimes, when the cursor is moving and the extension finishes its task, the page redirects while the cursor is still moving. This results in losing control of the page, and I can't extract data from the second page.
Here are the things I’ve tried to solve the issue:
Only moving the cursor when the page is finished loading: This didn’t solve the issue, as it doesn’t detect and stop the cursor fast enough. I also don’t have control over when the extension is done.
Using the system cursor instead of the web cursor: This solves the problem but limits my ability to use multi-threading, which I need for this project. I need to make the web cursor work for this.
Using Action Chains for testing: I’ve also tested using Action Chains to move the cursor, but I encounter the same issue.
I have limited control over the extension, and I can’t integrate its functionality directly into Python due to CORS and other limitations.
As a beginner in Python, I’m not sure how to tackle this problem efficiently. Could anyone point me in the right direction or suggest a better approach?
Thanks in advance!
I’m scraping a website and using the Chrome Extension to interact with the page. Once the extension is done performing its task, it clicks on the page and redirects to a second page. This part works fine.
However, the first page requires the cursor to be moved while the extension is interacting with the page. To solve this, I’m using the human_cursor (or any other web cursor library), and it works sometimes. But here’s the problem:
Sometimes, when the cursor is moving and the extension finishes its task, the page redirects while the cursor is still moving. This results in losing control of the page, and I can't extract data from the second page.
Here are the things I’ve tried to solve the issue:
Only moving the cursor when the page is finished loading: This didn’t solve the issue, as it doesn’t detect and stop the cursor fast enough. I also don’t have control over when the extension is done.
Using the system cursor instead of the web cursor: This solves the problem but limits my ability to use multi-threading, which I need for this project. I need to make the web cursor work for this.
Using Action Chains for testing: I’ve also tested using Action Chains to move the cursor, but I encounter the same issue.
I have limited control over the extension, and I can’t integrate its functionality directly into Python due to CORS and other limitations.
As a beginner in Python, I’m not sure how to tackle this problem efficiently. Could anyone point me in the right direction or suggest a better approach?
Thanks in advance!