Sorting and Merging text-files [SOLVED]

[AlphaInc]

Hello everybody,

With a batch-script I export mutliple text-files to a specific folder where I want to merge them into one.
Therefore I used the type-command in windows cmd which worked fine. But after comparing the files I noticed that the order is wrong. I found out the CMD sorts them differently (file_1,file_10,file_11,file_2,file_3,file_33) instead of how I'm doing it (file_1,file_2,file_3,file_10,file_11,file_33).

Now I would like to use a Python Script do merge them together in the way I would sort them (file_1,file_2,file_3,file_10,file_11,file_33).

This is what I have so far but I don't know how to go on:

#!/usr/bin/env python3

import os
import re

folder_path = "../Outputs/"

for root, dirs, files in os.walk(folder_path, topdown = False):
    for name in files:
        if name.endswith(".txt"):
            file_name = os.path.join(root, name)

Edit: All my files start with string "file_" and an ongoing number. It differs how many files there will be after I exported them so I can't set it up manually.

[Gribouillis]

Why are you calling os.walk() if you want to merge the files only in one folder?

[DeaD_EyE]

#!/usr/bin/env python3

from pathlib import Path


def sort_by_int(path):
    # Path has the stem attribute, which is
    # the filename without the last extension
    # to sort the paths by integer, you
    # need to get the integer part of the str
    # and convert it to an integer
    # the _ is the character where you can split
    # maxsplit=1 does only split once,
    # so you get two elements back
    # if the _ is missing, split will raise an Exception
    return int(path.stem.split("_", maxsplit=1)[1])


# Use the high level Path object
outputs = Path.home() / "Outputs"
# print(outputs)
# Path: /home/username/Outputs

# use glob for easier search
# rglob is to search recursive
# glob and rglob replicates the shell-syntax
# the wildcard is one * and a ? stands for one character

search = "file_*.txt"
# sorted takes a key argument, which is used to define how it's sorted
# sort_by_int just returns an int and the sorted function
# is using this number to sort
sorted_outputs = sorted(outputs.glob(search), key=sort_by_int)
# the result is a list
# sorted consumes the iterable object and returns a list
# with the sorted elements

# now using this sorted list with Path objects
for path in sorted_outputs:
    # glob does not differe between files, directories or other
    # elements
    # so you need to check, if path is a file
    if path.is_file():
        # if it's a file, then open it
        # the Path onject do have the method open
        # it supports like the open function a context manager
        with path.open() as fd:
            # iterating this file line by line
            # where the line end is not stripped away
            for line in fd:
                # print the line, but tell print not to add an additional
                # line end, because the line has already a line end
                print(line, end="")
                # you can use the stdout to redirect the output
                # in your shell to a file for example or netcat
                # or gzip etc...

Relevant documentation:

• https://docs.python.org/3/library/pathlib.html
  
• https://docs.python.org/3/library/functions.html#sorted
  
• https://docs.python.org/3/library/glob.html (pathlib uses this also)
  
• https://docs.python.org/3/library/functions.html#print
  
• https://docs.python.org/3/library/stdtyp...#str.split
  

Using the program (I named it randomly searchp.py):

Output:
[andre@andre-Fujitsu-i5 ~]$ python3 searchp.py | gzip > output.txt.gz
[andre@andre-Fujitsu-i5 ~]$ zcat output.txt.gz 
666
123
000

I created the Outputs directory in my home directory and put 3 files in, where each file had only one line with a newline character at the end.

[deanhystad]

If you have control of the file names the easiest solution is to generate file names that can be sorted properly: file_000, file_001, file_010. file_100

[AlphaInc]

#!/usr/bin/env python3

from pathlib import Path


def sort_by_int(path):
    # Path has the stem attribute, which is
    # the filename without the last extension
    # to sort the paths by integer, you
    # need to get the integer part of the str
    # and convert it to an integer
    # the _ is the character where you can split
    # maxsplit=1 does only split once,
    # so you get two elements back
    # if the _ is missing, split will raise an Exception
    return int(path.stem.split("_", maxsplit=1)[1])


# Use the high level Path object
outputs = Path.home() / "Outputs"
# print(outputs)
# Path: /home/username/Outputs

# use glob for easier search
# rglob is to search recursive
# glob and rglob replicates the shell-syntax
# the wildcard is one * and a ? stands for one character

search = "file_*.txt"
# sorted takes a key argument, which is used to define how it's sorted
# sort_by_int just returns an int and the sorted function
# is using this number to sort
sorted_outputs = sorted(outputs.glob(search), key=sort_by_int)
# the result is a list
# sorted consumes the iterable object and returns a list
# with the sorted elements

# now using this sorted list with Path objects
for path in sorted_outputs:
    # glob does not differe between files, directories or other
    # elements
    # so you need to check, if path is a file
    if path.is_file():
        # if it's a file, then open it
        # the Path onject do have the method open
        # it supports like the open function a context manager
        with path.open() as fd:
            # iterating this file line by line
            # where the line end is not stripped away
            for line in fd:
                # print the line, but tell print not to add an additional
                # line end, because the line has already a line end
                print(line, end="")
                # you can use the stdout to redirect the output
                # in your shell to a file for example or netcat
                # or gzip etc...

Relevant documentation:

• https://docs.python.org/3/library/pathlib.html
  
• https://docs.python.org/3/library/functions.html#sorted
  
• https://docs.python.org/3/library/glob.html (pathlib uses this also)
  
• https://docs.python.org/3/library/functions.html#print
  
• https://docs.python.org/3/library/stdtyp...#str.split
  

Using the program (I named it randomly searchp.py):

Output:
[andre@andre-Fujitsu-i5 ~]$ python3 searchp.py | gzip > output.txt.gz
[andre@andre-Fujitsu-i5 ~]$ zcat output.txt.gz 
666
123
000

I created the Outputs directory in my home directory and put 3 files in, where each file had only one line with a newline character at the end.

Thanks for your reply. So when I start the script on my windows machine there is no output. Do I need to do some tweaking of your script ?

[snippsat]

So when I start the script on my windows machine there is no output. Do I need to do some tweaking of your script ?

On Windows the home path will be

C:\Users\<username>\Outputs

You can give path to where you have the .txt files,if not want to make this Outputs folder.
Eg.

outputs = Path(r'G:\div_code\answer')

Test and i write code to save lines,as redirect as show bye DeaD_EyE may not work for you as eg gzip,zcat is not a part of Windows.

from pathlib import Path

def sort_by_int(path):
    # Path has the stem attribute, which is
    # the filename without the last extension
    # to sort the paths by integer, you
    # need to get the integer part of the str
    # and convert it to an integer
    # the _ is the character where you can split
    # maxsplit=1 does only split once,
    # so you get two elements back
    # if the _ is missing, split will raise an Exception
    return int(path.stem.split("_", maxsplit=1)[1])

# Use the high level Path object
#outputs = Path.home() / "Outputs"
outputs = Path(r'G:\div_code\answer')
#print(outputs)
# Path: /home/username/Outputs

# use glob for easier search
# rglob is to search recursive
# glob and rglob replicates the shell-syntax
# the wildcard is one * and a ? stands for one character

search = "file_*.txt"
# sorted takes a key argument, which is used to define how it's sorted
# sort_by_int just returns an int and the sorted function
# is using this number to sort
sorted_outputs = sorted(outputs.glob(search), key=sort_by_int)
#print(sorted_outputs)
# the result is a list
# sorted consumes the iterable object and returns a list
# with the sorted elements

# now using this sorted list with Path objects
lines = []
for path in sorted_outputs:
    # glob does not differe between files, directories or other
    # elements
    # so you need to check, if path is a file
    if path.is_file():
        print(path)
        # if it's a file, then open it
        # the Path onject do have the method open
        # it supports like the open function a context manager
        with path.open() as fd:
            # iterating this file line by line
            # where the line end is not stripped away
            for line in fd:
                # print the line, but tell print not to add an additional
                # line end, because the line has already a line end
                print(line)
                lines.append(line.strip())
                #zf.writestr(str(path), f)
                # you can use the stdout to redirect the output
                # in your shell to a file for example or netcat
                # or gzip etc...

with open('lines.txt', 'w') as f:
    f.write('\n'.join(lines))

Output:
G:\div_code\answer\file_1.txt
line1
G:\div_code\answer\file_3.txt
line2
G:\div_code\answer\file_10.txt
line3
G:\div_code\answer\file_33.txt
line4

lines.txt:

Output:
line1
line2
line3
line4

[AlphaInc]

So when I start the script on my windows machine there is no output. Do I need to do some tweaking of your script ?

On Windows the home path will be

C:\Users\<username>\Outputs

You can give path to where you have the .txt files,if not want to make this Outputs folder.
Eg.

outputs = Path(r'G:\div_code\answer')

Test and i write code to save lines,as redirect as show bye DeaD_EyE may not work for you as eg gzip,zcat is not a part of Windows.

from pathlib import Path

def sort_by_int(path):
    # Path has the stem attribute, which is
    # the filename without the last extension
    # to sort the paths by integer, you
    # need to get the integer part of the str
    # and convert it to an integer
    # the _ is the character where you can split
    # maxsplit=1 does only split once,
    # so you get two elements back
    # if the _ is missing, split will raise an Exception
    return int(path.stem.split("_", maxsplit=1)[1])

# Use the high level Path object
#outputs = Path.home() / "Outputs"
outputs = Path(r'G:\div_code\answer')
#print(outputs)
# Path: /home/username/Outputs

# use glob for easier search
# rglob is to search recursive
# glob and rglob replicates the shell-syntax
# the wildcard is one * and a ? stands for one character

search = "file_*.txt"
# sorted takes a key argument, which is used to define how it's sorted
# sort_by_int just returns an int and the sorted function
# is using this number to sort
sorted_outputs = sorted(outputs.glob(search), key=sort_by_int)
#print(sorted_outputs)
# the result is a list
# sorted consumes the iterable object and returns a list
# with the sorted elements

# now using this sorted list with Path objects
lines = []
for path in sorted_outputs:
    # glob does not differe between files, directories or other
    # elements
    # so you need to check, if path is a file
    if path.is_file():
        print(path)
        # if it's a file, then open it
        # the Path onject do have the method open
        # it supports like the open function a context manager
        with path.open() as fd:
            # iterating this file line by line
            # where the line end is not stripped away
            for line in fd:
                # print the line, but tell print not to add an additional
                # line end, because the line has already a line end
                print(line)
                lines.append(line.strip())
                #zf.writestr(str(path), f)
                # you can use the stdout to redirect the output
                # in your shell to a file for example or netcat
                # or gzip etc...

with open('lines.txt', 'w') as f:
    f.write('\n'.join(lines))

Output:
G:\div_code\answer\file_1.txt
line1
G:\div_code\answer\file_3.txt
line2
G:\div_code\answer\file_10.txt
line3
G:\div_code\answer\file_33.txt
line4

lines.txt:

Output:
line1
line2
line3
line4

Thanks also for your help. I can see that the Script process all my files but it does not create an output-file

Edit: My Bad, just looked after the wrong name. I found it. Thank you.

[AlphaInc]

On Windows the home path will be

C:\Users\<username>\Outputs

You can give path to where you have the .txt files,if not want to make this Outputs folder.
Eg.

outputs = Path(r'G:\div_code\answer')

Test and i write code to save lines,as redirect as show bye DeaD_EyE may not work for you as eg gzip,zcat is not a part of Windows.

from pathlib import Path

def sort_by_int(path):
    # Path has the stem attribute, which is
    # the filename without the last extension
    # to sort the paths by integer, you
    # need to get the integer part of the str
    # and convert it to an integer
    # the _ is the character where you can split
    # maxsplit=1 does only split once,
    # so you get two elements back
    # if the _ is missing, split will raise an Exception
    return int(path.stem.split("_", maxsplit=1)[1])

# Use the high level Path object
#outputs = Path.home() / "Outputs"
outputs = Path(r'G:\div_code\answer')
#print(outputs)
# Path: /home/username/Outputs

# use glob for easier search
# rglob is to search recursive
# glob and rglob replicates the shell-syntax
# the wildcard is one * and a ? stands for one character

search = "file_*.txt"
# sorted takes a key argument, which is used to define how it's sorted
# sort_by_int just returns an int and the sorted function
# is using this number to sort
sorted_outputs = sorted(outputs.glob(search), key=sort_by_int)
#print(sorted_outputs)
# the result is a list
# sorted consumes the iterable object and returns a list
# with the sorted elements

# now using this sorted list with Path objects
lines = []
for path in sorted_outputs:
    # glob does not differe between files, directories or other
    # elements
    # so you need to check, if path is a file
    if path.is_file():
        print(path)
        # if it's a file, then open it
        # the Path onject do have the method open
        # it supports like the open function a context manager
        with path.open() as fd:
            # iterating this file line by line
            # where the line end is not stripped away
            for line in fd:
                # print the line, but tell print not to add an additional
                # line end, because the line has already a line end
                print(line)
                lines.append(line.strip())
                #zf.writestr(str(path), f)
                # you can use the stdout to redirect the output
                # in your shell to a file for example or netcat
                # or gzip etc...

with open('lines.txt', 'w') as f:
    f.write('\n'.join(lines))

Output:
G:\div_code\answer\file_1.txt
line1
G:\div_code\answer\file_3.txt
line2
G:\div_code\answer\file_10.txt
line3
G:\div_code\answer\file_33.txt
line4

lines.txt:

Output:
line1
line2
line3
line4

Thanks also for your help. I can see that the Script process all my files but it does not create an output-file

Edit: My Bad, just looked after the wrong name. I found it. Thank you.

Sorry once again but I get an error:

Traceback (most recent call last):
  File "FileProcessing_11.py", line 30, in <module>
    sorted_outputs = sorted(outputs.glob(search), key=sort_by_int)
  File "FileProcessing_11.py", line 13, in sort_by_int
    return int(path.stem.split("_", maxsplit=1)[1])
IndexError: list index out of range

Could it be because the files got spaces in it?

[AlphaInc]

Thanks also for your help. I can see that the Script process all my files but it does not create an output-file

Edit: My Bad, just looked after the wrong name. I found it. Thank you.

Sorry once again but I get an error:

Traceback (most recent call last):
  File "FileProcessing_11.py", line 30, in <module>
    sorted_outputs = sorted(outputs.glob(search), key=sort_by_int)
  File "FileProcessing_11.py", line 13, in sort_by_int
    return int(path.stem.split("_", maxsplit=1)[1])
IndexError: list index out of range

Could it be because the files got spaces in it?

[snippsat]

Could it be because the files got spaces in it?

I couple of tips how you troubleshoot this.

def sort_by_int(path):
    print(path)
    print(path.stem)
    # Path has the stem attribute, which is

Bye adding this you see what happen before error.

Test.

>>> f = Path(r'G:\div_code\answer\file_33.txt')
>>> f.stem
'file_33'
>>> f.stem.split('_', maxsplit=1)
['file', '33']
>>> f.stem.split('_', maxsplit=1)[1]
'33'

Make your error.

>>> f = Path(r'G:\div_code\answer\file33.txt')
>>> f.stem
'file33'
>>> f.stem.split('_', maxsplit=1)
['file33']
>>> f.stem.split('_', maxsplit=1)[1]
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
IndexError: list index out of range

In your first post all files you show file_1,file_2,file_3,file_10...ect all had a _,then it should work.
With those print() or add repr()(see all like eg space) you will se all files input before the error.

print(repr(path))
print(repr(path.stem))