Python Forum
merge two csv files into output.csv using Subprocess
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
merge two csv files into output.csv using Subprocess
#2
Use Python instead of depending on OS infrastructure. This program could not run on Linux/Mac for example.
Here an example function to merge files:
from pathlib import Path


def merge_files(input_files: list[str | Path], output_file: str | Path) -> None:
    # user of code could supply a list with paths as str
    # or the paths as Path objetcs
    # but later Path objects are required, so all objects
    # are converted here
    input_files = [Path(file) for file in input_files]
    output_file = Path(output_file)
    
    # opening file in binary mode, which prevents encoding errors
    # if all input_files are encoded with the same encoding
    # Path objects do have the open method, which supports context managers
    with output_file.open("wb") as fd_out:
        # iterating over all input files
        for input_file in input_files:
            # same here, opening the input file in binary mode
            with input_file.open("rb") as fd_in:
                # https://realpython.com/lessons/assignment-expressions/
                # read 4KiB chunks
                while chunk := fd_in.read(4 * 1024 ** 1):
                    fd_out.write(chunk)
                
                # if the last line of input_file has no
                # lineseperator at the end, adding one to it
                # keep in mind, that everything is in binary mode
                
                # os.linesep is OS depended
                if not chunk.endswith(os.linesep.encode("ascii")):
                    fd_out.write(os.linesep.encode("ascii"))
Pedroski55 likes this post
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply


Messages In This Thread
RE: merge two csv files into output.csv using Subprocess - by DeaD_EyE - Dec-07-2022, 09:53 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  merge all xlsb files into csv mg24 0 364 Nov-13-2023, 08:25 AM
Last Post: mg24
  Merge all json files in folder after filtering deneme2 10 2,418 Sep-18-2022, 10:32 AM
Last Post: deneme2
  Merge htm files with shutil library (TypeError: 'module' object is not callable) Melcu54 5 1,658 Aug-28-2022, 07:11 AM
Last Post: Melcu54
  Increment text files output and limit contains Kaminsky 1 3,242 Jan-30-2021, 06:58 PM
Last Post: bowlofred
  How to merge all the files in a directory in to one file sutra 3 2,683 Dec-10-2020, 12:09 AM
Last Post: sutra
  opening files and output of parsing leodavinci1990 4 2,579 Oct-12-2020, 06:52 AM
Last Post: bowlofred
  Merge JSON Files Ugo 4 4,669 Aug-20-2020, 06:25 AM
Last Post: ndc85430
  How to get program output from subprocess.Popen? glestwid 1 2,392 Aug-19-2020, 05:44 AM
Last Post: buran
  How to read multiple csv files and merge data rajeshE 0 1,966 Mar-28-2020, 04:01 PM
Last Post: rajeshE
  error merge text files ledgreve 3 2,725 Nov-18-2019, 12:41 PM
Last Post: DeaD_EyE

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020