Python Forum
Read and save file in chucksize
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Read and save file in chucksize
#1
I need read a txt file using chuncksize, after save into xlsx.

import os, sys
import pandas as pd
import xlwings as xw


caminho = os.path.abspath(os.path.dirname(sys.argv[0]))
txt = caminho + '\\Fat2.txt'
xls = caminho + '\\cobfat.xlsx'
existe_txt = os.path.exists(txt)
existe_xls = os.path.exists(xls)


def check_path(x):
    # Função para verificar se há arquivo no caminho e com o nome correto. 
    if x != True:
        st = 'O arquivo não foi encontrado.\nVerifique se o nome está no local e com o nome correto.'
    else:
        st = 'ok'
    return st


big_file = 5000

if check_path(existe_txt):
    # Lista para guardados os dados do arquivo
    lst_xl = []

    # Leitura do arquivo usando chuck size
    for df in pd.read_csv(txt, sep="|", header=None, encoding='ISO-8859-1', chunksize=big_file):
        lst_xl.append(df)
        print('Gravando ...', df.shape)

    # Gravar dados no Excel
    df_xl = pd.concat(lst_xl, axis=0)
    writer = pd.ExcelWriter('output.xlsx')
    df_xl.to_excel(writer,'Plan1')
    writer.save()
    # I get error "error memory limit exceeded" after try save this in file.
Reply
#2
Hi, any help

Cross-post
I don't know how to write after read this 80k lines from file into excel via pandas
Thank you
Reply
#3
Hi guys.
I solved using this line
xw.Range('A2').options(index=False, header=False).value = df_x
Reply
#4
This part is useless:
existe_txt = os.path.exists(txt)
existe_xls = os.path.exists(xls)
The cause is, even if the files both are existing,
they could disappear after the check. This is not
a protection against Exception like FileNotFoundError
or PermissionError.

Python has for everything Exceptions.

def main():
    try:
        data = pd.read_csv(txt, sep="|", header=None, encoding='ISO-8859-1', chunksize=big_file)
        writer = pd.ExcelWriter('output.xlsx')
    except (FileNotFoundError, PermissionError) as err:
        print(err)
        return
    for df in data:
        ...

    ...


    writer.save()
You could catch more general Exception.
If you get for example a PermissionError and want to look up the inheritance:
PermissionError.mro()
Output:
[PermissionError, OSError, Exception, BaseException, object]
This Exception could be catched by: PermissionError, OSError, Exception
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Open/save file on Android frohr 0 281 Jan-24-2024, 06:28 PM
Last Post: frohr
  Recommended way to read/create PDF file? Winfried 3 2,786 Nov-26-2023, 07:51 AM
Last Post: Pedroski55
  python Read each xlsx file and write it into csv with pipe delimiter mg24 4 1,314 Nov-09-2023, 10:56 AM
Last Post: mg24
  how to save to multiple locations during save cubangt 1 509 Oct-23-2023, 10:16 PM
Last Post: deanhystad
  save values permanently in python (perhaps not in a text file)? flash77 8 1,121 Jul-07-2023, 05:44 PM
Last Post: flash77
  read file txt on my pc to telegram bot api Tupa 0 1,052 Jul-06-2023, 01:52 AM
Last Post: Tupa
  parse/read from file seperated by dots giovanne 5 1,044 Jun-26-2023, 12:26 PM
Last Post: DeaD_EyE
  Formatting a date time string read from a csv file DosAtPython 5 1,162 Jun-19-2023, 02:12 PM
Last Post: DosAtPython
  How do I read and write a binary file in Python? blackears 6 6,020 Jun-06-2023, 06:37 PM
Last Post: rajeshgk
  Read csv file with inconsistent delimiter gracenz 2 1,149 Mar-27-2023, 08:59 PM
Last Post: deanhystad

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020