Python Forum
Need to run 100+ Chrome’s with Selenium but can’t get past ~15 without breaking
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Need to run 100+ Chrome’s with Selenium but can’t get past ~15 without breaking
#1
I have a script with selenium where I need to run about 100 different chromes and 15 tabs per chrome. I can easily run 10 chromes with 15 tabs using an AWS instance with 16 VCPUs and 32GB of RAM, no problem. But when I upgrade my instance to 128 VCPUs and 512GB of RAM somehow I can still only run about 10 or 15 chromes before it just completely breaks. I am carrying out different automated processes on each Chrome.

At this point, I am considering just having 10 different instances running 10 chromes, but would like some way to centrally control them. It would be much more convenient to just run the 100 chromes on a single instance.

Please help with your experience or thoughts.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  ChromeDriver breaking Python script genericusername12414 1 214 Mar-14-2024, 09:39 AM
Last Post: snippsat
  Why can't I copy and past only ONE specific tab? NewWorldRonin 8 692 Jan-12-2024, 06:31 PM
Last Post: deanhystad
  Openpyxl module breaking my code EnderSM 5 985 May-26-2023, 07:26 PM
Last Post: snippsat
  breaking out of nested loops Skaperen 3 1,173 Jul-18-2022, 12:59 AM
Last Post: Skaperen
  How to use += after breaking for loop? Frankduc 2 1,443 Dec-31-2021, 01:23 PM
Last Post: Frankduc
  How to fix while loop breaking off raphy11574 3 2,124 Oct-12-2021, 12:56 PM
Last Post: deanhystad
  breaking a large program into functions, not acting as expected Zane217 9 2,931 Sep-18-2021, 12:37 AM
Last Post: Zane217
Exclamation Help in breaking bytes into bits through masking and shifting kamui123 9 4,435 Jan-11-2021, 07:42 AM
Last Post: kamui123
  Selenium Chrome error Leo_Red 2 1,820 Nov-05-2020, 02:40 PM
Last Post: Leo_Red
  Request blocking in Chrome Incognito mode pyseeker 0 2,239 Nov-04-2020, 08:51 PM
Last Post: pyseeker

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020