Python Forum

Hi all, and thank you for your help in advance.

I am experiencing a strange issue with Pandas. I have a program that reads an entire CSV into a dataframe, changes the contents of the dataframe and writes the whole CSV back to disk. Unfortunately when the CSV or dataframe reaches 365 records, it either reads the first 365 records or write the first 365 records.

Has anyone else had an issue with this?

INSTALLED VERSIONS
------------------
commit : None
python : 3.8.10.final.0
python-bits : 64
OS : Linux
OS-release : 5.13.0-1022-aws
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8



pandas : 0.25.3
numpy : 1.17.4
pytz : 2021.3
dateutil : 2.8.1
pip : 20.0.2
setuptools : 52.0.0
Cython : None
pytest : 4.6.9
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.0.2
lxml.etree : 4.5.0
html5lib : 1.0.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.3
IPython : None
pandas_datareader: None
bs4 : 4.9.3
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : 4.5.0
matplotlib : 3.1.2
numexpr : 2.7.1
odfpy : None
openpyxl : 3.0.3
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : None
scipy : 1.3.3
sqlalchemy : None
tables : 3.6.1
xarray : None
xlrd : 2.0.1
xlwt : 1.3.0
xlsxwriter : 3.0.2

Quote:Unfortunately when the CSV or dataframe reaches 365 records, it either reads the first 365 records or write the first 365 records

I assume you mean when the dataframe is larger than 365 records. Correct?


I think the problem is elsewhere. 365 rows (records?) is pretty small for pandas.

This program creates a dataframe with 1000 rows, each row having 1000 columns. It writes just fine to a csv file, and when I read it back in, the new dataframe equals the old.

import pandas as pd

data = {str(i):list(range(1000)) for i in range(1000)}
df = pd.DataFrame(data)
df.to_csv("test.csv", index=False)

df2 = pd.read_csv("test.csv")
print(df2.equals(df))

Output:
True

What have you done to try and debug this problem? 365 is a really interesting number. I can think of all kinds of code I could write that would stop working at 365, but the fault would be mine, not pandas. Can you post the code that is writing your csv file?

Hi Dean.

Thank you so much for your help. I have created a loads of records and found out that the issue has arisen from having two programs accessing the CSV.

Thanks again.