Read and save file in chucksize

[zinho]

I need read a txt file using chuncksize, after save into xlsx.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

import os, sys import pandas as pd import xlwings as xw     caminho = os.path.abspath(os.path.dirname(sys.argv[0])) txt = caminho + '\\Fat2.txt' xls = caminho + '\\cobfat.xlsx' existe_txt = os.path.exists(txt) existe_xls = os.path.exists(xls)     def check_path(x):     # Função para verificar se há arquivo no caminho e com o nome correto.     if x != True:         st = 'O arquivo não foi encontrado.\nVerifique se o nome está no local e com o nome correto.'     else:         st = 'ok'     return st     big_file = 5000   if check_path(existe_txt):     # Lista para guardados os dados do arquivo     lst_xl = []       # Leitura do arquivo usando chuck size     for df in pd.read_csv(txt, sep="|", header=None, encoding='ISO-8859-1', chunksize=big_file):         lst_xl.append(df)         print('Gravando ...', df.shape)       # Gravar dados no Excel     df_xl = pd.concat(lst_xl, axis=0)     writer = pd.ExcelWriter('output.xlsx')     df_xl.to_excel(writer,'Plan1')     writer.save()     # I get error "error memory limit exceeded" after try save this in file.

[zinho]

Hi, any help

Cross-post
I don't know how to write after read this 80k lines from file into excel via pandas
Thank you

[zinho]

Hi guys.
I solved using this line

1	xw.Range('A2').options(index=False, header=False).value = df_x

[DeaD_EyE]

This part is useless:

1 2	existe_txt = os.path.exists(txt) existe_xls = os.path.exists(xls)

The cause is, even if the files both are existing,
they could disappear after the check. This is not
a protection against Exception like FileNotFoundError
or PermissionError.

Python has for everything Exceptions.

1 2 3 4 5 6 7 8 9 10 11 12 13 14

def main():     try:         data = pd.read_csv(txt, sep="|", header=None, encoding='ISO-8859-1', chunksize=big_file)         writer = pd.ExcelWriter('output.xlsx')     except (FileNotFoundError, PermissionError) as err:         print(err)         return     for df in data:         ...       ...         writer.save()

You could catch more general Exception.
If you get for example a PermissionError and want to look up the inheritance:

1	PermissionError.mro()

Output:
[PermissionError, OSError, Exception, BaseException, object]

This Exception could be catched by: PermissionError, OSError, Exception