Python Forum
PRAW and PyQt: Immense slowdown when retrieving Reddit posts and adding them to GUI
Thread Rating:
  • 3 Vote(s) - 3.67 Average
  • 1
  • 2
  • 3
  • 4
  • 5
PRAW and PyQt: Immense slowdown when retrieving Reddit posts and adding them to GUI
#1
I'm trying to make a program using PRAW and PyQt that gathers Reddit posts from subscribed subreddits and displays them in a GUI window. The code I currently have seems to work, but each time the program tries to gather posts, the program slows down so much, it is unusable. The program manages retrieving posts by running a function every 5 seconds using a QTimer, as shown in the following code:

class App:
#...
    def check(self):
        self.model.checkPosts()
    def __init__(self):
        self.model = Model(self)
        self.app = QApplication([])
        self.app.setStyle("Fusion")
        self.timer = QTimer()
        self.timer.setInterval(5000)
        self.timer.timeout.connect(self.check)
#...


class Model:
#...
	def checkPosts(self):
		
		subs = self.reddit.user.subreddits()

		try:
			
			for sub in subs:

				posts = sub.hot(limit=1)

				for post in posts:

					if not(post.title in self.postList):
						NewPostController(self.app, self).addPost(post)					

		except NotFound as error:
			print("Subreddit not found")

		except Redirect as error:
			print("Subreddit not found")

		except InvalidToken as error:
			self.auth(self.user,self.password)
#...

class NewPostController:
#...
	def addPost(self, post):
		self.model.postList.append(post.title)
		link = LinkWidget(post=post)
		self.app.layout.addWidget(link)

class LinkWidget(QLabel):

	def __init__(self, parent=None, post=None):
		QLabel.__init__(self,parent=parent);
		self.setFrameShape(QFrame.Shape.StyledPanel)
		self.setFrameShadow(QFrame.Shadow.Raised)
		self.setMaximumWidth(self.window().frameGeometry().width())
		self.setAutoFillBackground(True)
		palette = QPalette()
		palette.setColor(QPalette.Background, Qt.white)
		self.setPalette(palette)
		self.setText('<a href="' + post.url + '" style="text-decoration:none;">'+post.title+"</a>")
		self.setOpenExternalLinks(True)
		self.setWordWrap(True)
		font = QFont("Arial", 17, QFont.Bold)
		self.setFont(font)
I've tried using the threading module to execute the Model.checkThreads() in a separate thread to prevent the slowdown, but it didn't do anything. Is there any way I can prevent the slowdown from happening? Is my code inefficient?
Reply
#2
What exactly is slow? If you're scraping a website, they might throttle you if they detect it.

We can't figure out what part is slow just looking at the code, so you need to do the experiments to figure which exact line(s) are problematic and then we might be of more help.
Reply
#3
(Dec-28-2018, 05:49 PM)micseydel Wrote: What exactly is slow? If you're scraping a website, they might throttle you if they detect it.

We can't figure out what part is slow just looking at the code, so you need to do the experiments to figure which exact line(s) are problematic and then we might be of more help.

Thanks for the suggestion. It turns out the checkMethods function kept fetching the user's subscribed subreddits every time it was run, and it was costing a lot of time.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  KeyError while retrieving ESPN data john317ab 2 803 Nov-29-2023, 09:07 PM
Last Post: john317ab
  .get() not retrieving value? Sedos101 2 558 Aug-25-2023, 11:48 AM
Last Post: deanhystad
  [Solved] Retrieving a pdf from sqlite3 BigMan 4 2,303 Mar-12-2022, 01:56 PM
Last Post: deanhystad
  Retrieving a column from a data set using a function Bayle 6 2,327 Oct-06-2021, 08:52 PM
Last Post: Bayle
  Reddit bot thread Gabriel777 0 26,573 Sep-20-2021, 02:57 AM
Last Post: Gabriel777
  Retrieving Cookies whois1230 2 2,168 Nov-21-2020, 12:01 PM
Last Post: snippsat
  Problem: Retrieving Form data PythonDev 3 3,073 Oct-16-2020, 02:09 AM
Last Post: PythonDev
  Retrieving dictionary keys within with another dictionay bazcurtis 8 2,806 Oct-29-2019, 10:06 PM
Last Post: bazcurtis
  Retrieving items from JSON bazcurtis 12 5,039 Oct-27-2019, 05:18 PM
Last Post: bazcurtis
  Adding markers to Folium map only adding last element. tantony 0 2,119 Oct-16-2019, 03:28 PM
Last Post: tantony

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020