matplotlib chart text font misery with 5x8 dot matrix

[Knabbler]

I can't get matplotlib to create my plots the way it needs to. This problem of font representation doesn't exist in Pillow.
After 40 hours of searching, reading, trying with the help of Google and ChatGPT, I give up and turn to you for help.

System description:
I have various sensors that periodically transmit data via radio to a server running Node-Red. Node-Red has various tasks, the only important thing here is that it writes various measurement data into an SQLite database.
There is an information display in the living room that shows, among other things, various diagrams. It consists of a 7.5 inch e-paper display from Waveshare with a resolution of 800x480 pixels and a color depth of 1 bit (black and white). It is controlled by an ESP32 microcontroller. It wakes up every quarter of an hour, gets the pixel data (48 Kbytes) from the server via web socket, pushes it onto the display and goes back to sleep.
A Python program runs on the server, which extracts and transmits the pixel data for the display from a bitmap file upon request from the ESP32.
A Python program also runs on the server, which, triggered by Node-Red, fetches data from the database, creates various diagrams, arranges them on a canvas of 800x480 pixels and then saves this canvas as a bitmap file with 1 bit color depth.
All of these components are currently working together.

The problem lies in the presentation of the matplotlib diagrams. Because the display has such a low resolution, the text to be displayed must be correspondingly small and still be reasonably readable. The choosen font is a dot matrix structure of 8x5 pixels (5x8_lcd_hd44780u_a02.ttf). And this way, text characters must be max 8 pixel high and 5 pixel wide.

I use Python 3.12.1, matplotlib 3.8.4 and pillow 10.3.0 on a Windows 10 PC.
The font is installed and included in the fontlist-v330.json file.

I'm attaching an example code to play with and the resulting bitmap. The way the text looks in pillow figures is how it should look in the matplotlib charts. Also attached is a bitmap from my actual regular code under development, just to get a rough idea.

import matplotlib.pyplot as plt
from matplotlib import font_manager as fm, rcParams
from PIL import Image, ImageDraw, ImageFont

fig, ax = plt.subplots(figsize=(2.64, 2.32), dpi=100)
custom_font = "5x8 LCD HD44780U A02"

ax.set_title('Test-Txt', fontname = custom_font, size = 8)
ax.set_xlabel('xLabel', fontname = custom_font, size = 8)
ax.set_ylabel('yLabel', fontname = custom_font, size = 8)
plt.text(.1, .5, 'TXT pos .2.5', fontname = custom_font, size = 8)

plt.tick_params(axis='both', which='both', labelsize=8)
for tick in ax.get_xticklabels():
    tick.set_fontname(custom_font)

for tick in ax.get_yticklabels():
    tick.set_fontname(custom_font)
    
plt.savefig('tst.png')

chart = Image.open('tst.png')
canvas = Image.new("1", (800, 480), "white")
draw = ImageDraw.Draw(canvas)

text = 'ImageFontTest'
font_name = "5x8.ttf"
font_size = 8
font = ImageFont.truetype(font_name, font_size)

canvas.paste(chart, (50, 50))
draw.text((98, 123), 'PilDrawText_098/123', font=font, fill="black")

canvas.save("result.bmp")