Python Forum

Full Version: Concatenate multiple csv files
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Hello python gurus,

I am trying to concatenate multiple data frames but when I run the application it gives me this error:
KeyError: 'Level Date must be same as name (None)'
I am following this tutorial: Python for finance

The tutorial's way of joining the csv files generates a combined csv file with all the stocks but gives a new column for each stock (500). Ideally I just want one column called "Tickers" not 500. So I am trying melt instead of outer join but not getting it to work.


Full code is here: https://codeshare.io/amDv7o

def compile_data():
    with open("sp500tickers.pickle", "rb") as f:
        tickers = Cpickle.load(f)

    main_df = pd.DataFrame()

    for count, ticker in enumerate(tickers):
        df = pd.read_csv('stock_dfs/{}.csv'.format(ticker))
        df.reset_index('Date', inplace=True)

        df.rename(columns={'Adj Close': ticker}, inplace=True)
        df.drop(['Open', 'High', 'Low', 'Close', 'Volume'], 1, inplace=True)

        if main_df.empty:
            main_df = df
        else:
            main_df = main_df.melt(id_vars=['Date'], var_name='Ticker', value_name='Closed')

        if count % 10 == 0:
            print(count)
    print(main_df.head())

    main_df.to_csv('sp500_joined_closes.csv')

compile_data()
always show full error traceback, verbatim, and include which imports are being used.
I don't use pandas (or rarely pickle) much, but this line looks suspect:
df.reset_index('Date', inplace=True)