Text parsing

[Arik]

Hello!

I parse emails with receipts to collect a database using Python.
Idea is to concatenate line starting with "&" or lowercase alphabet letter with previous line.

That part of the code is:

# Concatenate lines starting with '&' or starting with lowercase to the end of the previous line
    combined_lines = []
    for line in soup.stripped_strings:
    if (line.startswith('&') or line[0].islower()) and combined_lines:
        combined_lines[-1] += ' ' + line.lstrip()
    else:
        combined_lines.append(line)

The "&" is working but lowercase is not.
What is my mistake?

deanhystad write Mar-11-2024, 05:01 AM:
Please post all code, output and errors (it it's entirety) between their respective tags. Refer to BBCode help topic on how to post. Use the "Preview Post" button to make sure the code is presented as you expect before hitting the "Post Reply/Thread" button.

[deanhystad]

The posted code doesn't work for & or lower. Maybe it is a cut/paste errror and you meant this:

combined_lines = []
for line in soup.stripped_strings:
    if (line.startswith('&') or line[0].islower()) and combined_lines:
        combined_lines[-1] += ' ' + line.lstrip()
    else:
        combined_lines.append(line)

[Arik]

import pandas as pd
from imap_tools import MailBox
from bs4 import BeautifulSoup
import re

# Connect to the specified mail server and login using the provided credentials
with MailBox(server).login(login, password, folder) as mailbox:
    # Fetch emails
    emails = []
    for msg in mailbox.fetch():
        # Message body
        message_body = msg.text or msg.html

        # Parse the HTML content using BeautifulSoup
        soup = BeautifulSoup(message_body, 'html.parser')

        # Concatenate lines starting with '&' to the end of the previous line
        combined_lines = []
        for line in soup.stripped_strings:
            if (line.startswith('&') or line[0].islower()) and combined_lines:
                combined_lines[-1] += ' ' + line.lstrip()
            else:
                combined_lines.append(line)

        combined_text = '\n'.join(combined_lines)

        # Replace tabs with line feeds
        combined_text = combined_text.replace('\t', '\n')
        
        # Remove blank rows
        combined_text = '\n'.join([line.strip() for line in combined_text.splitlines() if line.strip()])

        # Remove leading spaces and tabs in every row
        combined_text = '\n'.join([line.lstrip() for line in combined_text.splitlines()])

        # Remember the first row
        first_row = combined_text.split('\n', 1)[0]

        # Find the text between "Goods" and "Payment"
        match = re.search(r'Goods(.*?)Payment', combined_text, re.DOTALL)
        if match:
            combined_text = match.group(1).strip()

        # Take only the first 12 letters for "order"
        order_value = first_row[:12]

        # Collect the email data 
        email_data = {
            'subject': msg.subject,
            'from': msg.from_,
            'to': msg.to,
            'date': msg.date,
            'body': combined_text,
            'order': order_value
        }
        emails.append(email_data)

    # Convert the emails to a pandas DataFrame
    df = pd.DataFrame(emails)

[Arik]

There are no errors.
Output is correct if the line starts with "&", but in case of the line starts with lowercase letter it is not concatenated with the previous line.

[deanhystad]

Do you have any lines that start with a lowercase letter? If the first character of line is whitespace (space or tab), your test will fail. Maybe you should call strip() before you check the first letter.

[Gribouillis]

Output is correct if the line starts with "&", but in case of the line starts with lowercase letter it is not concatenated with the previous line.

Can you post an example block of lines where it doesn't work?