PDFminer outputs unreadable text during conversion from PDF to TXT

[Gromila131]

Need to convert text from PDF to TXT. Trying to do it via pdfminer. Tried other libraries. But all unsuccessfully. How can I solve this problem

1 2 3 4 5 6

import pdfminer.high_level   with open ('2.pdf', 'rb') as file:      file1 = open (r'2.txt', 'a+')      pdfminer.high_level.extract_text_to_fp(file, file1)      file1.close()

[Pedroski55]

This is a tricky one! Please let me know if you figure it out!

I thought maybe the table lines confused the text reader, so I got each row from the table, but that did not help!

I thought ftfy might be the answer, but the text comes out the same:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

import pymupdf from pathlib import Path import ftfy   path2pdf = '/home/pedro/Downloads/a.pdf' path2text = '/home/pedro/Downloads/a.pdf.txt'   doc = pymupdf.open(path2pdf) for page in doc:     tabs = page.find_tables()   # there is only 1 table tab = tabs[0] data = [] for line in tab.extract(encoding=pymupdf.TEXT_ENCODING_CYRILLIC):  # print cell text for each row     print(f'This line has {len(line)} cells')     print(line)     data.append(line)   for d in data:     for i in range(len(d)):         if d[i] == '':             d[i] = 'empty'         elif d[i] == None:             d[i] = 'empty'   row_strings = [''.join(s) for s in data] text = ''.join(row_strings) page_text = ftfy.fix_text(text)

I also tried an online OCR but that came back with gibberish too!

Since the pdf displays correctly, the correct information must be in there!

[Gromila131]

This is a tricky one! Please let me know if you figure it out!

I thought maybe the table lines confused the text reader, so I got each row from the table, but that did not help!

I thought ftfy might be the answer, but the text comes out the same:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

import pymupdf from pathlib import Path import ftfy   path2pdf = '/home/pedro/Downloads/a.pdf' path2text = '/home/pedro/Downloads/a.pdf.txt'   doc = pymupdf.open(path2pdf) for page in doc:     tabs = page.find_tables()   # there is only 1 table tab = tabs[0] data = [] for line in tab.extract(encoding=pymupdf.TEXT_ENCODING_CYRILLIC):  # print cell text for each row     print(f'This line has {len(line)} cells')     print(line)     data.append(line)   for d in data:     for i in range(len(d)):         if d[i] == '':             d[i] = 'empty'         elif d[i] == None:             d[i] = 'empty'   row_strings = [''.join(s) for s in data] text = ''.join(row_strings) page_text = ftfy.fix_text(text)

I also tried an online OCR but that came back with gibberish too!

Since the pdf displays correctly, the correct information must be in there!

Now I'm converting to the correct format via online services. PDF--->DOCX---->TXT
In this case, the text comes out line by line and is readable.
But I can't make such a converter in Python

[Pedroski55]

I searched a lot, but I can't find an answer.

The PDF has 2 embedded fonts. They are both types of Times Roman, which is a common font. The PDF was made on MacOS.

Maybe if you Python it on an apple computer, you will get the correct output.

I tried saving as binary, then opening but that did not work:

1 2 3 4 5 6 7

with pymupdf.open(path2pdf) as doc:  # open document     text = chr(12).join([page.get_text() for page in doc])     # write as a binary file to support non-ASCII characters     pathlib.Path(path2pdf + ".txt").write_bytes(text.encode())   with open(path2text, encoding="utf-8") as f:     text = f.read()

Like I said, the PDF displays correctly, so the information must be in there! How to extract it??

[Gromila131]

This is a tricky one! Please let me know if you figure it out!

I thought maybe the table lines confused the text reader, so I got each row from the table, but that did not help!

I thought ftfy might be the answer, but the text comes out the same:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

import pymupdf from pathlib import Path import ftfy   path2pdf = '/home/pedro/Downloads/a.pdf' path2text = '/home/pedro/Downloads/a.pdf.txt'   doc = pymupdf.open(path2pdf) for page in doc:     tabs = page.find_tables()   # there is only 1 table tab = tabs[0] data = [] for line in tab.extract(encoding=pymupdf.TEXT_ENCODING_CYRILLIC):  # print cell text for each row     print(f'This line has {len(line)} cells')     print(line)     data.append(line)   for d in data:     for i in range(len(d)):         if d[i] == '':             d[i] = 'empty'         elif d[i] == None:             d[i] = 'empty'   row_strings = [''.join(s) for s in data] text = ''.join(row_strings) page_text = ftfy.fix_text(text)

I also tried an online OCR but that came back with gibberish too!

Since the pdf displays correctly, the correct information must be in there!

Now I'm converting to the correct format via online services. PDF--->DOCX---->TXT
In this case, the text comes out line by line and is readable.
But I can't make such a converter in Python

I didn't tell you why I'm doing this.

I'm currently creating a Python program that will analyze apartment sales. In my country, developers post a project declaration on the website every month, which indicates the number of apartments sold (it's in PDF format).

I need to take the information from there and put it into an Excel table. Maybe I can take it directly from the PDF file? But now I can only extract it from TXT. Here's my code

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138

import openpyxl #работа эксель from openpyxl import Workbook, load_workbook import numpy as np import pandas as pd import openpyxl as xl import re from openpyxl.styles import Alignment, Font import sys     def main():     pd_txt = 'obj59216-pd30-000303 (2).txt'       # Название ЖК     with open(pd_txt) as text_file:         for num, line in enumerate(text_file, 1):             if '10.6.1' in line:                 name_of_the_building = int(num + 1)                 print("Название ЖК:")                 f = open(pd_txt)                 lines = f.readlines()                 NAME_OF_THE_BUILDING = (lines[name_of_the_building])                 print(lines[name_of_the_building])                 f.close()       # Срок сдачи     with open(pd_txt) as text_file:         for num, line in enumerate(text_file, 1):             if '17.1 (5)' in line:                 deadline = int(num+8)                 print("Срок сдачи:")                 f = open(pd_txt)                 lines = f.readlines()                 DEADLINE = (lines[deadline])                 print(lines[deadline])                 f.close()       # Квартир всего     with open(pd_txt) as text_file:         for num, line in enumerate(text_file, 1):             if 'Количество жилых помещений:' in line:                 total_apartments = int(num)                 print("Квартир всего:")                 f = open(pd_txt)                 lines = f.readlines()                 TOTAL_APARTMENTS = (lines[total_apartments])                 print(lines[total_apartments])                 f.close()       # Квартир продано     with open(pd_txt) as text_file:         for num, line in enumerate(text_file, 1):             if '19.7.1.1.1.1' in line:                 sold_apartments=int(num+1)                 print("Квартир продано:")                 f = open(pd_txt)                 lines = f.readlines()                 SOLD_APARTMENTS = (lines[sold_apartments])                 print(lines[sold_apartments])                 f.close()       # Продано М2     with open(pd_txt) as text_file:         for num, line in enumerate(text_file, 1):             if '19.7.2.1.1.1' in line:                 sold_meters=int(num)                 print("Продано М2:")                 f = open(pd_txt)                 lines = f.readlines()                 just = lines[sold_meters]                 sold_meters_value = just.split(": ")[1]                 SOLD_METERS = (sold_meters_value[:-3])                 print(sold_meters_value[:-3])                 f.close()         # Заработано ₽     with open(pd_txt) as text_file:         for num, line in enumerate(text_file, 1):             if '19.7.3.1.1.1' in line:                 money_received=int(num+1)                 print("Заработано ₽:")                 f = open(pd_txt)                 lines = f.readlines()                 print(lines [money_received][:-5])                 MONEY_RECEIVED = (lines[money_received][:-5] )                 f.close()         # Дата загрузки ПД     with open(pd_txt) as text_file:         for num, line in enumerate(text_file, 1):             if 'ПРОЕКТНАЯ ДЕКЛАРАЦИЯ' in line:                 project_declaration_date=int(num)                 print("Дата загрузки ПД")                 f = open(pd_txt)                 lines = f.readlines()                 PROJECT_DECLARATION_DATE = (lines[project_declaration_date][-8:])                 print (lines[project_declaration_date][-8:])                 f.close()       # Номер ПД     with open(pd_txt) as text_file:         for num, line in enumerate(text_file, 1):             if 'ПРОЕКТНАЯ ДЕКЛАРАЦИЯ' in line:                 project_declaration_num=int(num)                 print("Номер ПД")                 f = open(pd_txt)                 lines = f.readlines()                 PROJECT_DECLARATION_NUM = (lines[project_declaration_num][2:11])                 print (lines[project_declaration_num][2:11])                 f.close()       """     gp = 7.2024     print(PDD)     print(PDD==gp)     """       PDN = (str(PROJECT_DECLARATION_NUM))       # Existing Excel file     existing_file = 'excel1.xlsx'     # New data to append     new_data = [[PROJECT_DECLARATION_NUM, PROJECT_DECLARATION_DATE, NAME_OF_THE_BUILDING, DEADLINE, TOTAL_APARTMENTS, SOLD_APARTMENTS,                  SOLD_METERS, MONEY_RECEIVED]]     # Load existing workbook     wb = load_workbook(existing_file)     # Select the active sheet     ws = wb[PDN]     # Append new data     for row in new_data:         ws.append(row)     # Save the workbook     wb.save(existing_file)     main()

Maybe someone will find it useful

[Pedroski55]

I noticed in some Chinese pdfs, that, internally, the pdf used strange combinations of Chinese characters to represent a letter or letters. Something is going on internally to map combinations of characters to other characters.

I don't know enough about the internal structure of PDFs to figure that out!

Have you managed to successfully get the Russian text from the PDF using Python?

[Pedroski55]

You can get the data in PDF tables to Pandas like this, but I still have that weird stuff:

1 2 3 4 5 6 7 8 9 10

import pandas as pd import fitz # aka pymupdf   path2pdf = '/home/pedro/Downloads/a.pdf'   # get all tables in the PDF doc = fitz.open(path2pdf) for page in doc:     tabs = page.find_tables() df = pd.DataFrame(tabs[0].extract())

You can save the df as Excel easily.

Please let me know if you find out how to get the proper text!