Python Forum
os.walk does not see files that are in the folder
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
os.walk does not see files that are in the folder
#1
Hello!

I used the following code some time ago for a project involving parsing tweets. Now, I want to extract more features associated with those tweets and re-running it on the same files I parsed before; however, receive an unexpected error. The file 1.json mentioned in the error is in the directory (along with hundreds of other json's). What am I doing wrong?

import os
import json
import pandas as pd
import numpy as np
from collections import defaultdict

elements_keys = ['created_at', 'text', 'lang', 'geo', 'location', 'quote_count', 'reply_count', 'retweet_count', 'favorite_count', 'in_reply_to_screen_name', 'screen_name', 'description', 'verified', 'followers_count', 'friends_count', 'listed_count', 'favourites_count', 'statuses_count']
elements = defaultdict(list)

for dirs, subdirs, files in os.walk('/Users/user/Desktop/'):
    for file in files:
        if file.endswith('.json'):
            with open(file, 'r') as input_file: # print (tweet.keys())
                for line in input_file:
                    try:
                        tweet = json.loads(line)
                        items = [(key, tweet[key]) for key in elements_keys] # should raise error if any key is missing
                        for key, value in items:
                            elements[key].append(value)
                    except:
                        continue

df=pd.DataFrame({'created_at': pd.Index(elements['created_at']),
                 'text': pd.Index(elements['text']),
                 'lang': pd.Index(elements['lang']),
                 'geo': pd.Index(elements['geo']),
                 'location': pd.Index(elements['location']),
                 'quote_count': pd.Index(elements['quote_count']),
                 'reply_count': pd.Index(elements['reply_count']),
                 'retweet_count': pd.Index(elements['retweet_count']),
                 'favorite_count': pd.Index(elements['favorite_count']),
                 'in_reply_to_screen_name': pd.Index(elements['in_reply_to_screen_name']),
                 'screen_name': pd.Index(elements['screen_name']),
                 'verified': pd.Index(elements['verified']),
                 'followers_count': pd.Index(elements['followers_count']),
                 'friends_count': pd.Index(elements['friends_count']),
                 'listed_count': pd.Index(elements['listed_count']),
                 'favourites_count': pd.Index(elements['favourites_count']),
                 'statuses_count': pd.Index(elements['statuses_count'])})

df.to_csv('df.csv')

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-21-fba7b7321509> in <module>()
     11     for file in files:
     12         if file.endswith('.json'):
---> 13             with open(file, 'r') as input_file: # print (tweet.keys())
     14                 for line in input_file:
     15                     try:

FileNotFoundError: [Errno 2] No such file or directory: '1.json'

Oh I think I just figured it out:

for dirs, subdirs, files in os.walk('/Path/*.json'):
It's been a while I used Python. Sorry :)
Reply
#2
https://docs.python.org/3/library/os.html#os.walk Wrote:To get a full path (which begins with top) to a file or directory in dirpath, do os.path.join(dirpath, name).
I think that's what you're looking for.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
Smile Python & MSGraph - Learning to Walk ITMan020324 2 354 Feb-04-2024, 04:37 PM
Last Post: ITMan020324
  Compare folder A and subfolder B and display files that are in folder A but not in su Melcu54 3 466 Jan-05-2024, 05:16 PM
Last Post: Pedroski55
  Rename files in a folder named using windows explorer hitoxman 3 693 Aug-02-2023, 04:08 PM
Last Post: deanhystad
  Rename all files in a folder hitoxman 9 1,384 Jun-30-2023, 12:19 AM
Last Post: Pedroski55
  How to loop through all excel files and sheets in folder jadelola 1 4,331 Dec-01-2022, 06:12 PM
Last Post: deanhystad
  python gzip all files from a folder mg24 3 3,812 Oct-28-2022, 03:59 PM
Last Post: mg24
  delete all files and subdirectory from a main folder mg24 7 1,527 Oct-28-2022, 07:55 AM
Last Post: ibreeden
  Merge all json files in folder after filtering deneme2 10 2,253 Sep-18-2022, 10:32 AM
Last Post: deneme2
  Compare filename with folder name and copy matching files into a particular folder shantanu97 2 4,390 Dec-18-2021, 09:32 PM
Last Post: Larz60+
  EasySNMP Walk/BulkWalk pylance 3 2,031 Nov-29-2021, 12:00 PM
Last Post: pylance

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020