Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Byte string catenation inefficient in 3.7?
#1
I'm noticing something odd with byte string catenation versus text string.
Im building a bmp file. The core code is (ignoring row padding and headers etc)
for Y in range(0, XY[1]): # (BMPs are L to R from the bottom L row)
        for X in range(0, XY[0]):
            # square rainbow as oposed to real data
            x = floor((255 * X)/XY[0])
            y = floor((255 * Y)/XY[1])
            (r,g,b) = [x, y, 128]  #Colour(data[x ,y])
            pixels += struct.pack('<BBB',b,g,r)
and Ive run the code in Python 2.7 (char strings) and in Python 3.7 (byte strings)
This was tested on a 1024 x 1024 pixel image, and the elapsed time was logged every 100 rows.

Overall, Python 2.7 runs in 2.4 seconds, Python 3.7 runs in over 8 minutes (i7-8700 CPU @ 3.2GHz, 16 GB RAM)

3.7 is orders of magnitude slower, and there is a major slow down when the string gets around the 1 MB mark as illustrated by these graphs of elapsed time every 100 rows (approx 300k of data added to the string):
Graphs of elapsed time bytes vs text

Ive tried googling this and havent found much, which surprised me - Id have thought this would have been a major issue. Apart from the obvious slowness of the byte strings, the pattern is very striking. What happens at the 1MB mark? Obviously a change of algorithm of some sort.

Does anyone have any advice? I'm wondering about converting the struct.pack() output to char string (cf Python 2.7) , catenate that and then convert back to byte string when it is complete. Or would that conversion back be super slow?
Quote

Top Page

Messages In This Thread
Byte string catenation inefficient in 3.7? - by RMJFlack - Aug-16-2019, 07:32 AM

Possibly Related Threads...
Thread Author Replies Views Last Post
  'utf-8' codec can't decode byte 0xda in position 184: invalid continuation byte karkas 7 287 Sep-12-2019, 11:19 PM
Last Post: newbieAuggie2019
  HELP: String of Zero's and One's to binary byte schwasskin 1 1,164 May-19-2019, 07:31 AM
Last Post: heiner55
  4 byte hex byte swap from binary file medievil 7 6,190 May-08-2018, 08:16 AM
Last Post: killerrex
  get the content of the byte as string ricardons 5 859 Mar-02-2018, 02:41 PM
Last Post: ricardons
  byte string Skaperen 5 1,049 Feb-04-2018, 08:58 AM
Last Post: Gribouillis
  byte string in python2 Skaperen 4 1,466 Nov-23-2017, 03:13 AM
Last Post: Skaperen
  Does Python 3.x have a built-in byte string compare function? Raptor88 2 4,393 Feb-18-2017, 10:44 AM
Last Post: Raptor88

Forum Jump:


Users browsing this thread: 1 Guest(s)