Breaking Change for Python 3.7 - 3.12

[DeaD_EyE]

Breaking Change for Python 3.7 - 3.12

Since Python 3.10.7 a breaking change were introduced.

Problem

A Denial Of Service (DoS) issue was identified in CPython because we use binary bignum’s for our int implementation. A huge integer will always consume a near-quadratic amount of CPU time in conversion to or from a base 10 (decimal) string with a large number of digits. No efficient algorithm exists to do otherwise.

It is quite common for Python code implementing network protocols and data serialization to do int(untrusted_string_or_bytes_value) on input to get a numeric value, without having limited the input length or to do log("processing thing id %s", unknowingly_huge_integer) or any similar concept to convert an int to a string without first checking its magnitude. (http, json, xmlrpc, logging, loading large values into integer via linear-time conversions such as hexadecimal stored in yaml, or anything computing larger values based on user controlled inputs… which then wind up attempting to output as decimal later on). All of these can suffer a CPU consuming DoS in the face of untrusted data.

Everyone auditing all existing code for this, adding length guards, and maintaining that practice everywhere is not feasible nor is it what we deem the vast majority of our users want to do.

This issue has been reported to the Python Security Response Team multiple times by a few different people since early 2020, most recently a few weeks ago while I was in the middle of polishing up the PR so it’d be ready before 3.11.0rc2.
Mitigation

After discussion on the Python Security Response Team mailing list the conclusion was that we needed to limit the size of integer to string conversions for non-linear time conversions (anything not a power-of-2 base) by default. And offer the ability to configure or disable this limit.

The Python Steering Council is aware of this change and accepts it as necessary.

In most cases, you won't be affected.
The limit can be changed with the function sys.set_int_max_str_digits.

To clarify, this won't influence integer calculations. It limits the conversion from str to int or int to str.


To test the limit, do following:

import math


math.factorial(1560)

Error:
ValueError: Exceeds the limit (4300) for integer string conversion

PS: I forgot to share the video, where I have seen this: https://www.youtube.com/watch?v=eTucYT2LpNU

[Gribouillis]

I can't test this because I only have Python 3.10.4 but having read the linked discussion can you tell us whether the following works?

import math
from gmpy2 import mpz
x = math.factorial(1560)
print(mpz(x))

[Larz60+]

Works fine Using Linux-mint 20.2, python 3.10.6:

>>> import math
>>> from gmpy2 import mpz
>>> x = math.factorial(1560)
>>> print(mpz(x))
>>>

Output:
5897352625981184879227999439774256554713087289390486637518412580047469765506026570480850215716402625588982839611185040927090679940867230577850081748987102007545252609912630188378708424762837599509873952159765186354955772604038437661971282856199517434021758698812111470710905240599516020746610920813756999054138124035015882513894292868464805119538642075401418967294460257612595336039248382369550744137090029260256115684057909699775920850169859097429668379901892636934825292864019043418544126489462752873723476379975844213582468600713607194034634304522043422561043069966464811320320166569570675444190783388817080306390538333701704729274720562476043131554278655376657380528535105624607301549409249763381156557228538984271085472718587162943644687745281302036757500114787932750319457553068399725077351602518938253463883298791204440078193499890319102660176831117245395188129115154449683019546318351332976644645837189101927967808012568751572941769184413176137629635875253068519135767473608908545013364377153840162603277824317499107747779020295177378417151297976108680853516793116193033014503118587354667553577103089166225187392727111881646282507162413815636447738356974339900251048900625003746721217466641868944695398385111319804915460012475111073309970966122148256120190820532577319706253491896722929922034355786208876129430098341195939224926595581644140768845795322948325643446303828669022002198082862351041569849003331170563023991430730975498774239814184968255645482148585496740219471524561050620955881584091762307563911272099484913137489442405753040467666199593884271212951994130939171548165921793848749368336052317069257618428371687270381154483255168334611013240353100248413253714130209383445666326969622789266039979925912111695122514498216274647212931538728288443791043308060367176628290713589640285626287007669168938283293648524526536480583963773973655247235822575056103210170607848590368617986266408251817073856131737169296701643388997736005997844507081931180329879972615630220004509665804702611283076325923506673551455546073975227613488780313013662926292751475783568207894848487368715952634495321375140684751539615279634840576659448045882965742647482443898644142855589149711966292994768560305975273309945505290430318377953455499051285847546204560183931396015935400176104128687644797341197716442947424685552081672508167187603062371512386748672551976920525327833756927493459252154686590158325818757983996188580301840298590811195227009126790391015051151761615251596796309949427919584837430650957913793499308833805913970071155250336299114434000505248231433811715507720126327814449661571174648483214206682827469612662794657575970739255486664982584999881309953745236432458757256425183818994565626938721068663935103260466426362451188796468718190704726645979757932324488452891325906869999885474932166089265781609521118144228225653964347427130916748580594562628291302702320166841262812661616016844201460264361912832199799832146957000035840278019433800929699158330992782947832785948560428015682062229707440815966626741568073868530298420246682599224079635431760288105703221066962185242662484584988319023527171695029794489481988246651618605170673484376929557988358223125607181854967842777356199939440761845529759598174989027880726249885539387062866267659442265305399270655100760036707888161088436667389405500958022958811169568027327255917360664461544288206093838174701013544539044481340959014350762769356513656509830650998618479762105788061225809087162661456740509172000203156179347728774921133503093012545226359285319071763947070300970984017983694805265930944889533551785510805145713806560224019211592719143744866895362525287532538274270936495070678237736049201616608281396322639787873447288016542000310662602234764318441387789125495314465266624957414305518562267250134230821146880194878871797154696505141040689012319560293683218212945225997352770515688575242759398851340670689545229331702626915028264658616641635266989716930560000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
>>>

[Gribouillis]

Works fine Using Linux-mint 20.2, python 3.10.6:

But the problem does not exist before 3.10.7 so it is not useful with 3.10.6.

[Larz60+]

But the problem does not exist before 3.10.7 so it is not useful with 3.10.6.

Sorry, It's early in the A.M here (5:21), and just got up. I use pyenv, which I just upgraded this morning, but it doesn't have python 3.10.7 available yet.

[snippsat]

I only have Python 3.10.4 but having read the linked discussion can you tell us whether the following works?

tom@tom-VirtualBox:~$ python -V
Python 3.10.7

tom@tom-VirtualBox:~$ rich gm.py
import math
from gmpy2 import mpz

x = math.factorial(1560)
print(mpz(x))

Output:
tom@tom-VirtualBox:~$ python gm.py
5897352625981184879227999439774256554713087289390486637518412580047469765506026570480850215716402625588982839611185040927090679940867230577850081748987102007545252609912630188378708424762837599509873952159765186354955772604038437661971282856199517434021758698812111470710905240599516020746610920813756999054138124035015882513894292868464805119538642075401418967294460257612595336039248382369550744137090029260256115684057909699775920850169859097429668379901892636934825292864019043418544126489462752873723476379975844213582468600713607194034634304522043422561043069966464811320320166569570675444190783388817080306390538333701704729274720562476043131554278655376657380528535105624607301549409249763381156557228538984271085472718587162943644687745281302036757500114787932750319457553068399725077351602518938253463883298791204440078193499890319102660176831117245395188129115154449683019546318351332976644645837189101927967808012568751572941769184413176137629635875253068519135767473608908545013364377153840162603277824317499107747779020295177378417151297976108680853516793116193033014503118587354667553577103089166225187392727111881646282507162413815636447738356974339900251048900625003746721217466641868944695398385111319804915460012475111073309970966122148256120190820532577319706253491896722929922034355786208876129430098341195939224926595581644140768845795322948325643446303828669022002198082862351041569849003331170563023991430730975498774239814184968255645482148585496740219471524561050620955881584091762307563911272099484913137489442405753040467666199593884271212951994130939171548165921793848749368336052317069257618428371687270381154483255168334611013240353100248413253714130209383445666326969622789266039979925912111695122514498216274647212931538728288443791043308060367176628290713589640285626287007669168938283293648524526536480583963773973655247235822575056103210170607848590368617986266408251817073856131737169296701643388997736005997844507081931180329879972615630220004509665804702611283076325923506673551455546073975227613488780313013662926292751475783568207894848487368715952634495321375140684751539615279634840576659448045882965742647482443898644142855589149711966292994768560305975273309945505290430318377953455499051285847546204560183931396015935400176104128687644797341197716442947424685552081672508167187603062371512386748672551976920525327833756927493459252154686590158325818757983996188580301840298590811195227009126790391015051151761615251596796309949427919584837430650957913793499308833805913970071155250336299114434000505248231433811715507720126327814449661571174648483214206682827469612662794657575970739255486664982584999881309953745236432458757256425183818994565626938721068663935103260466426362451188796468718190704726645979757932324488452891325906869999885474932166089265781609521118144228225653964347427130916748580594562628291302702320166841262812661616016844201460264361912832199799832146957000035840278019433800929699158330992782947832785948560428015682062229707440815966626741568073868530298420246682599224079635431760288105703221066962185242662484584988319023527171695029794489481988246651618605170673484376929557988358223125607181854967842777356199939440761845529759598174989027880726249885539387062866267659442265305399270655100760036707888161088436667389405500958022958811169568027327255917360664461544288206093838174701013544539044481340959014350762769356513656509830650998618479762105788061225809087162661456740509172000203156179347728774921133503093012545226359285319071763947070300970984017983694805265930944889533551785510805145713806560224019211592719143744866895362525287532538274270936495070678237736049201616608281396322639787873447288016542000310662602234764318441387789125495314465266624957414305518562267250134230821146880194878871797154696505141040689012319560293683218212945225997352770515688575242759398851340670689545229331702626915028264658616641635266989716930560000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

Without gmpy mpz.

tom@tom-VirtualBox:~$ rich gm1.py 
import math
# from gmpy2 import mpz

x = math.factorial(1560)
print(x)

Error:
tom@tom-VirtualBox:~$ python gm1.py[
Traceback (most recent call last):
  File "/home/tom/gm1.py", line 5, in <module>
    print(x)
ValueError: Exceeds the limit (4300) for integer string conversion

[snippsat]

I use pyenv, which I just upgraded this morning, but it doesn't have python 3.10.7 available yet.

pyenv dos have 3.10.7 as i use pyenv in post over.

pyenv update

[Gribouillis]

This looks like a regression in Python to me. They'd better fix their suboptimal algorithm for base conversions.

[Larz60+]

pyenv dos have 3.10.7 as i use pyenv in post over.

needed coffee. properly installed now.

[stevendaprano]

There is a long (nearly 200 posts) thread discussing this breaking change on Discuss.

Alas, the moderators have shut the discussion down. I feel, perhaps a bit unfairly, that the security team were rationalising and defending their decision rather than being open to constructive criticism.

We're now nearly two weeks since the threat has been publicly disclosed, and it has been widely discussed in public. Have there been any exploits for it yet? I feel that the security team has exaggerated the threat, and failed to distinguish between high threat data leaks and privilege escalation, versus low threat DOS/performance attacks.