Multiprocessing OSError 'too many open files'

[DreamingInsanity]

I am using multiprocessing to do some link processing.
The code I am using to create the processes is the same as shown in one of my earlier posts:

1 2 3 4 5 6 7 8 9 10

while (len(processes) < len(list(urls))): #checks for current processes alive being less than all the links needing processing     if (len(processes) - len([p for p in processes if not p.is_alive()]) < OPTIONS['max_proc']):         p = Process(target = Links.process_link, args=(urls[index], OPTIONS)) #create a new process         processes.append(p) #add it to array         p.start()           index += 1               for p in processes:     p.join() #dont continue main script until processes have finished

The problem is I get this error:

Error:
OSError: [Errno 24] Too many open files

The first thing I tried was, using contextlib's closing function, closing the requests I make to web page like this:

1	with closing(urlopen(Request(url, headers={'User-Agent': 'Mozilla/3.0'}), context=CONTEXT)) as response:

That didn't.
I do open a file and write to it many time, so I used the same closing function to see if that would help:

1	with closing(gzip.GzipFile(DIR_PATH+'/links.data.gz', 'a')) as lnk: #close file after we have finished with it

Again, that didn't work.
That means it is something to do with the amount of processes being opened. There are a few answers online, but non worked. The closest one to my exact problem said add p.terminate() under p.join(), so it would close the process after it's done.
However, the error occurs before it even makes it to the for loop. And since it's an OSError, the isn't really much (any) useful information in the stack trace.

What is the problem here?

[micseydel]

I expect you to know by now, it's incredibly difficult for us to help with code when we're provided only 10 lines of who knows how much. What you should do is make a copy of your code and simplify it until there's nothing that can be removed. Any user-input should be hard-coded to reproduce the problem, and any line of code that can be removed without preventing the problem being reproduced should be removed.

Once you've gotten to that point, you'll have code that we can actually look at and start trying to figure out what's wrong. The fewer lines of code you're able to reduce to, the more likely you will be to get a satisfactory response.

[ibreeden]

A Python error message usually also tells on which line the error occurs. Can you share that information?

[DreamingInsanity]

I expect you to know by now, it's incredibly difficult for us to help with code when we're provided only 10 lines of who knows how much. What you should do is make a copy of your code and simplify it until there's nothing that can be removed. Any user-input should be hard-coded to reproduce the problem, and any line of code that can be removed without preventing the problem being reproduced should be removed.

Once you've gotten to that point, you'll have code that we can actually look at and start trying to figure out what's wrong. The fewer lines of code you're able to reduce to, the more likely you will be to get a satisfactory response.

That's fair enough. I was trying to keep the code to minimum.
I've removed unnecessary code, and added a few hardcoded values.

Here's the code - I believe the libraries you need are requests, urllib and psaw.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103

class Links():     def start_sorting_links():         processes = []         index = 0           api = PushshiftAPI()         SUBMISSIONS = api.search_submissions(subreddit='wellthatsucks', filter=['url', 'over_18'], limit=500)           urls = [subs.d_['url'] for subs in SUBMISSIONS] #get all the urls and put in array           while (len(processes) < len(list(urls))): #checks for current processes alive being less than all the links needing processing             if (len(processes) - len([p for p in processes if not p.is_alive()]) < OPTIONS['max_proc']):                 p = Process(target = Links.process_link, args=(urls[index], )) #create a new process                 processes.append(p) #add it to array                 p.start()                   index += 1                               for p in processes:             p.join() #dont continue main script until processes have finished       def process_link(url):         with closing(gzip.GzipFile('/links.data.gz', 'a')) as lnk: #close file after we have finished with it             if('/imgur.com/' in url): #imgur plain                 tree = Links.parse_link(url)                 Links.Imgur.imgur_plain(tree, lnk)             elif('/i.imgur.com/' in url): #imgur direct                 Links.Imgur.imgur_direct(url, lnk)             elif('/gfycat.com/' in url): #gfycat plain                 tree = Links.parse_link(url)                 Links.Gfycat.gfycat_plain(tree, lnk)             elif('/thumbs.gfycat.com/' in url): #gfycat direct                 Links.Gfycat.gfycat_direct(url, lnk)             elif('/i.redd.it/' in url): #reddit image / direct                 Links.Reddit.i_reddit(url, lnk)             elif('/v.redd.it' in url): #reddit video / plain                 Links.Reddit.v_reddit(url, lnk)             elif('/giphy.com/' in url): #giphy plain                 tree = Links.parse_link(url)                 Links.Giphy.giphy_plain(tree, lnk)             elif('/media.giphy.com/' in url): #giphy partial direct                 Links.Giphy.giphy_part_direct(url, lnk)             elif('/i.giphy.com/' in url): #giphy full direct                 Links.Giphy.giphy_full_direct(url, lnk)             else:                 pass #clearly not anything we want here!             lnk.flush() #flush the buffer               class Imgur():         def imgur_plain(tree, file):             l = tree.xpath('/html/head/link[12]')             try:                 direct_link = [i.attrib['href'] for i in l][0]             except Exception:                 return             n = tree.xpath('/html/body/div[7]/p[2]')             nsfw = [i.text for i in n]             file.write('{0}\n'.format(str(direct_link)).encode())         def imgur_direct(url, file): #direct links don't need processing             file.write('{0}\n'.format(str(url)).encode())                   class Gfycat():         def gfycat_plain(tree, file):             l = tree.xpath('/html/head/meta[51]')             direct_link = [i.attrib['content'] for i in l][0]             #almost impossible to find out whether gif is nsfw or not on gfycat             file.write('{0}\n'.format(str(direct_link)).encode()) #just gonna risk it         def gfycat_direct(url, file): #direct links don't need processing             file.write('{0}\n'.format(str(url)).encode()) #just gonna risk it       class Reddit(): ####NSFW         def i_reddit(url, file): #direct links don't need processing             file.write('{0}\n'.format(str(url)).encode())           def v_reddit(url, file): #v.reddit is video, and will direct you back to normal reddit page so i need to get direct link             full_url_jsonified = str(requests.get(url).url) + '.json'#v.reddit redirects to normal reddit, and i need that link, so i get it. add json to the end, so i get json data about video             try:                 req = urlopen(full_url_jsonified, context=CONTEXT)             except HTTPError:                     return                            data = json.load(req) #load the request into a json format             direct_link = data[0]['data']['children'][0]['data']['secure_media']['reddit_video']['fallback_url'] #it uses the fallback url which is just a direct url to the video                  file.write('{0}\n'.format(str(direct_link)).encode())       class Giphy():         def giphy_plain(tree, file):             l = tree.xpath('/html/head/meta[19]')             direct_link = [i.attrib['content'] for i in l][0].replace('media', 'i', 1) #replace first instance of 'media' with 'i' and that will get you direct link             #very few giphy gifs have a pg rating / nsfw tag             file.write('{0}\n'.format(str(direct_link)).encode())           def giphy_part_direct(url, file):             suffix = url.split(".")[-1]             direct_link = url.replace('media', 'i', 1) #changing one of the 'media's makes it a direct link             file.write('{0}\n'.format(str(direct_link)).encode())           def giphy_full_direct(url, lnk):             file.write('{0}\n'.format(str(url)).encode())       def parse_link(url):         htmlparser = etree.HTMLParser() #create parser         with closing(urlopen(Request(url, headers={'User-Agent': 'Mozilla/3.0'}), context=CONTEXT)) as response: #get the html of the page - the headers make it seem like Mozilla is acessing the page otherwise some sites can detect if it is python trying to access and block the connection             return etree.parse(response, htmlparser) #create an element tree out of it

---

A Python error message usually also tells on which line the error occurs. Can you share that information?

I'm actually struggling to reproduce the error.
I've run it twice, and the error hasn't occurred. I'm wondering if I was getting the error because there were too many processes left over from other times I have run the program.

Multiprocessing makes a huge spread difference in this part of the code, but if it's going to be finicky like this, I might not even continue with it.