Python Forum
Byte string catenation inefficient in 3.7?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Byte string catenation inefficient in 3.7?
#1
I'm noticing something odd with byte string catenation versus text string.
Im building a bmp file. The core code is (ignoring row padding and headers etc)
for Y in range(0, XY[1]): # (BMPs are L to R from the bottom L row)
        for X in range(0, XY[0]):
            # square rainbow as oposed to real data
            x = floor((255 * X)/XY[0])
            y = floor((255 * Y)/XY[1])
            (r,g,b) = [x, y, 128]  #Colour(data[x ,y])
            pixels += struct.pack('<BBB',b,g,r)
and Ive run the code in Python 2.7 (char strings) and in Python 3.7 (byte strings)
This was tested on a 1024 x 1024 pixel image, and the elapsed time was logged every 100 rows.

Overall, Python 2.7 runs in 2.4 seconds, Python 3.7 runs in over 8 minutes (i7-8700 CPU @ 3.2GHz, 16 GB RAM)

3.7 is orders of magnitude slower, and there is a major slow down when the string gets around the 1 MB mark as illustrated by these graphs of elapsed time every 100 rows (approx 300k of data added to the string):
Graphs of elapsed time bytes vs text

Ive tried googling this and havent found much, which surprised me - Id have thought this would have been a major issue. Apart from the obvious slowness of the byte strings, the pattern is very striking. What happens at the 1MB mark? Obviously a change of algorithm of some sort.

Does anyone have any advice? I'm wondering about converting the struct.pack() output to char string (cf Python 2.7) , catenate that and then convert back to byte string when it is complete. Or would that conversion back be super slow?
Reply


Messages In This Thread
Byte string catenation inefficient in 3.7? - by RMJFlack - Aug-16-2019, 07:32 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  pyreadstat write_sav inefficient mikisDeWitte 2 2,762 Jun-21-2021, 09:49 AM
Last Post: mikisDeWitte
  'utf-8' codec can't decode byte 0xe2 in position 122031: invalid continuation byte tienttt 12 11,590 Sep-18-2020, 10:10 PM
Last Post: tienttt
  'utf-8' codec can't decode byte 0xda in position 184: invalid continuation byte karkas 8 31,717 Feb-08-2020, 06:58 PM
Last Post: karkas
  First Byte of a string is missing while receiving data over TCP Socket shahrukh1987 3 4,259 Nov-20-2019, 10:34 AM
Last Post: shahrukh1987
  HELP: String of Zero's and One's to binary byte schwasskin 1 3,878 May-19-2019, 07:31 AM
Last Post: heiner55
  4 byte hex byte swap from binary file medievil 7 22,121 May-08-2018, 08:16 AM
Last Post: killerrex
  get the content of the byte as string ricardons 5 3,697 Mar-02-2018, 02:41 PM
Last Post: ricardons
  byte string Skaperen 5 3,878 Feb-04-2018, 08:58 AM
Last Post: Gribouillis
  byte string in python2 Skaperen 4 4,353 Nov-23-2017, 03:13 AM
Last Post: Skaperen
  Does Python 3.x have a built-in byte string compare function? Raptor88 2 16,425 Feb-18-2017, 10:44 AM
Last Post: Raptor88

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020