Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
mixed types
#1
I have mixed data types when trying to load in several CSV files into jupyter notebook. How can I fix this? Thanks for any help

Code
import os
import glob
import pandas as pd
os.chdir("M:\My Folder\Trial")
extension = 'csv'
all_filenames = [i for i in glob.glob('*.{}'.format(extension))]

combined_csv = pd.concat([pd.read_csv(f) for f in all_filenames ])
Here is my error
Error:
C:\Users\xxxxxx\AppData\Local\Continuum\anaconda3\lib\site-packages\IPython\core\interactiveshell.py:3249: DtypeWarning: Columns (2,8) have mixed types. Specify dtype option on import or set low_memory=False. if (await self.run_code(code, result, async_=asy)):
Reply
#2
this looks suspiciously like you are running out of memory.
Reply
#3
Looks like this may of solved it:

combined_csv = pd.concat([pd.read_csv(f, sep=',', error_bad_lines=False, index_col=False, dtype='unicode') for f in all_filenames ])
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  List of mixed data types to bytes medatib531 1 2,304 Mar-16-2020, 11:53 AM
Last Post: scidam
  Reading mixed ASCII/binary serial input N8UR 1 3,684 Jun-11-2019, 02:49 PM
Last Post: N8UR
  mixed decimal point characters Skaperen 1 2,158 Mar-10-2019, 05:23 PM
Last Post: stullis
  Mixed string,Integer input variable issue maderdash 2 2,714 Nov-06-2018, 09:46 AM
Last Post: snippsat
  Multivariate mixed effects models primeprover 1 2,563 Aug-19-2018, 01:14 PM
Last Post: ketanm

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020