+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Data Science (https://python-forum.io/forum-44.html)
+--- Thread: Automating the code using os.walk (/thread-2359.html)
RE: Automating the code using os.walk - kiton - Apr-13-2017
As a follow-up for the this topic: Following your helpful suggestions I carefully checked all the downloaded files and their contents. (Miraculously) it appeared that the structure of JSONs in the 2012 batch was very different from that in 2013, 2014, and 2016. Therefore, I discarded 2012 data and successfully proceeded late one. Also, as advised I broke up the 450GB monthly pieces into smaller chunks to facilitate processing. Thank you for guidance!