Python Forum
How to reallocating memory space in multithreading program in python ?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to reallocating memory space in multithreading program in python ?
#1
#from multiprocessing.pool import ThreadPool
#from multiprocessing.dummy import Pool as ThreadPool
from multiprocessing import Pool
from multiprocessing.dummy import Pool as ThreadPool 
import tweepy
import random
import json
import simplejson
import re
from tweepy import OAuthHandler
from tweepy.auth import OAuthHandler
import gc
import sys

import time
from datetime import datetime
import threading
import psycopg2

# Consumer keys and access tokens, used for OAuth

consumer_key = "xxxxxxxx"
consumer_secret = "xxxxxxxx"
access_token = "xxxxxxxxxx"
access_token_secret = "xxxxxxx"


# OAuth process, using the keys and tokens
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)


# Creation of the actual interface, using authentication
try:
    api = tweepy.API(auth, wait_on_rate_limit=True, wait_on_rate_limit_notify=True, retry_count=10, retry_delay=60)
    #auth = api.VerifyCredentials()
except Exception as e:
    print("Exception is occured",e)
count =0 

def twitterjson(string):

            results = tweepy.Cursor(api.user_timeline, screen_name=sm_handle, count=100).items()
            for tweet in results:
                with open(sm_handle+'.json', 'w') as outfile:
                    outfile.write(json.dumps(tweet._json))
                    print (tweet)
     return 1;
            
if __name__ == "__main__":
  while True:
    try:
        # Establishing a PSQL connection
        conn = psycopg2.connect("dbname='sample' user='postgres' host='localhost' port='5432'")
        print("opened database successfully")

        # create a psycopg2 cursor that can execute queries
        cursor = conn.cursor()

        # twitter_users_query = "select twitter_user_id, twitter_screen_name from twitter_users;"
        twitter_users_query = "select usernames from test;"
        cursor.execute(twitter_users_query)
        #pool = ThreadPool(10) 

        rows = cursor.fetchall()
        names=[]
        for row in rows:
            sm_handle = row[0]
            names.append(sm_handle)
            print("sm_handles: "+sm_handle)
            threads = []

           # for i in rows:
        for name in names:
            print ("name:"+name)
     
    except Exception as e:
        print("unable to connect to the database : ", e)
    try:   
        pool = ThreadPool(100) 
        gc.collect()
        #pool.terminate()
        results=pool.map(twitterjson, names)
        pool.terminate(100)
        print ("Completed"+results)
        pool.terminate()
        pool.close() 
        pool.join()
        print("This is sleeping for 1 minute..............")
        #time.sleep(60)
        print('Done')
        
        #pool.terminate()
        #pool.close() 
        #pool.join()
        #pool.terminate()

    except Exception as e:
         print("Exception is occured",e)
         pool.terminate();
Reply
#2
Could you be more explicit when you say "reallocating memory space"?
If it ain't broke, I just haven't gotten to it yet.
OS: Windows 10, openSuse 42.3, freeBSD 11, Raspian "Stretch"
Python 3.6.5, IDE: PyCharm 2018 Community Edition
Reply
#3
How to free the memory utilzation(RAM) in completed threads in above Example
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Python rule about the space character surrounding the equal sign ineuw 10 1,519 Sep-21-2023, 09:17 AM
Last Post: ineuw
  Python Tkinter Simple Multithreading Question AaronCatolico1 5 1,475 Dec-14-2022, 11:35 PM
Last Post: deanhystad
  python list out top 10 Memory utilisation mg24 0 785 Nov-20-2022, 10:51 PM
Last Post: mg24
  python Multithreading on single file mg24 3 1,672 Nov-05-2022, 01:33 PM
Last Post: snippsat
Bug Embedded Python Memory Leaks Alexei 1 1,003 Sep-16-2022, 01:15 PM
Last Post: Alexei
  Removing Space between variable and string in Python coder_sw99 6 6,125 Aug-23-2022, 01:15 PM
Last Post: louries
  multithreading Hanyx 4 1,282 Jul-29-2022, 07:28 AM
Last Post: Larz60+
Question Problems with variables in multithreading Wombaz 2 1,286 Mar-08-2022, 03:32 PM
Last Post: Wombaz
  python memory rd_rakesh 1 1,627 Jul-14-2021, 11:15 AM
Last Post: jefsummers
  Multithreading question amadeok 0 1,748 Oct-17-2020, 12:54 PM
Last Post: amadeok

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020