Stack trace shows different exception type than print

[micseydel]

Python's JSON module supports deserializing Javascript infinities (although the official JSON spec doesn't). Specifically, it supports "Infinity" and "-Infinity" in particular. I need to be able to deserialize "+Infinity" though. Here is my attempt at achieving this

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

import json   class PositiveInfinityJSONDecoder(json.JSONDecoder):     def __init__(self, *args, **kwargs):         super().__init__(*args, **kwargs)         self.original_scan_once = self.scan_once         self.scan_once = self._scan_once       def _scan_once(self, string, idx):         print("Custom _scan_once entered; trying default first...")         try:             return self.original_scan_once(string, idx)         except json.decoder.JSONDecodeError as e:             print("Default failed, trying custom logic...")             try:                 nextchar = string[idx]             except IndexError:                 raise StopIteration(idx) from e             if nextchar == '+' and string[idx:idx + 9] == '+Infinity':                 return self.parse_constant('Infinity'), idx + 9             raise e         except Exception as e:             print(f"Got an unexpected exception from default (type: {type(e)}) - {e}")             raise e   print(json.loads('+Infinity', cls=PositiveInfinityJSONDecoder))

The result of this is that (apparently) a StopIteration exception is thrown, although its traceback looks like a JSONDecodeError

Output:
$ python3 confusion.py 
Custom _scan_once entered; trying default first...
Got an unexpected exception from default (type: <class 'StopIteration'>) - 0
Traceback (most recent call last):
  File "confusion.py", line 28, in <module>
    print(json.loads('+Infinity', cls=PositiveInfinityJSONDecoder))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 361, in loads
    return cls(**kw).decode(s)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I'm baffled because the stack trace clearly shows a JSONDecodeError being raised, but at runtime when I try to catch it, it appears to have a different type. In case the class is doing something funky, I also tried catching a ValueError (its superclass) but that didn't work either.

What does work is catching a StopIteration exception, but that doesn't seem right. I'm afraid that there's something weird going on here and if I catch the wrong exception, there will be some surprise down the line. Does anyone know what's going on here?

[sheeba_weeba]

Henlo Fren,

My naem Sheba and I luvs da slithery codes please.

The JSONDecodeError in your codes is from JSONDecoder.raw_decode which calls self.scan_once.

Here is a good codes please:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

import json   class PositiveInfinityJSONDecoder(json.JSONDecoder):     def __init__(self, *args, **kwargs):         super().__init__(*args, **kwargs)         self.original_scan_once = self.scan_once         self.scan_once = self._scan_once       def _scan_once(self, string, idx):         print("Custom _scan_once entered; trying default first...")         try:             return self.original_scan_once(string, idx)         except StopIteration as e:  # scan_once should throw a StopIteration             print("Default failed, trying custom logic...")             try:                 nextchar = string[idx]             except IndexError:                 raise StopIteration(idx) from e             if nextchar == '+' and string[idx:idx + 9] == '+Infinity':                 return self.parse_constant('Infinity'), idx + 9             else:                 raise StopIteration(idx)         except Exception as e:             print(f"Got an unexpected exception from default (type: {type(e)}) - {e}")             raise e   print(json.loads('Infinity', cls=PositiveInfinityJSONDecoder)) print() print(json.loads('+Infinity', cls=PositiveInfinityJSONDecoder)) print() print(json.loads('+BadData', cls=PositiveInfinityJSONDecoder)) print()

Output:
$ python3 confusion.py 
Custom _scan_once entered; trying default first...
inf

Custom _scan_once entered; trying default first...
Default failed, trying custom logic...
inf

Custom _scan_once entered; trying default first...
Default failed, trying custom logic...
Traceback (most recent call last):
  File "confusion.py", line 31, in <module>
    print(json.loads('+BadData', cls=PositiveInfinityJSONDecoder))
  File "/usr/lib/python3.6/json/__init__.py", line 367, in loads
    return cls(**kw).decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Sincerely,
Sheba Weeba Possum Woo

[micseydel]

Thanks for the reply but I feel like I must be missing something. I already mentioned that a StopIteration error appears to work, but that's the problem. As you can see in the link you provided, raw_decode should be throwing a JSONDecodeError, not a StopIteration error, and you can see that when I print a stacktrace the type of the error appears different than if I print the type of the error object.

[sheeba_weeba]

There are two exceptions being thrown:

1. json.loads('+Infinity') is called.
   
2. ...
   
3. raw_decode('+Infinity') is called.
   
4. raw_decode calls PositiveInfinityJSONDecoder._scan_once.
   
5. PositiveInfinityJSONDecoder._scan_once calls self.original_scan_once (a.k.a. JSONDecoder().scan_once).
   
6. self.original_scan_once raises a StopIteration as expected.
   
7. PositiveInfinityJSONDecoder._scan_once prints the StopIteration exception and reraises it.
   
8. raw_decode excepts StopIteration and raises JSONDecodeError.
   
9. JSONDecodeError is not caught, so it prints the stack trace for JSONDecodeError.
   

[scidam]

There are two version of make_scanner in json module, c_make_scanner and py_make_scanner. We need to force Python to use Python version (py_make_scanner).
The following isn't to be an elegant solution, but it straightforward and clear:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74

import json from json.decoder import PosInf, _CONSTANTS from json.scanner import NUMBER_RE   _CONSTANTS.update({'+Infinity': PosInf})  # Update known constants   def py_make_scanner(context):  # there are c-version, we need to force python to use version only     parse_object = context.parse_object     parse_array = context.parse_array     parse_string = context.parse_string     match_number = NUMBER_RE.match     strict = context.strict     parse_float = context.parse_float     parse_int = context.parse_int     parse_constant = context.parse_constant     object_hook = context.object_hook     object_pairs_hook = context.object_pairs_hook     memo = context.memo       def _scan_once(string, idx):         try:             nextchar = string[idx]         except IndexError:             raise StopIteration(idx) from None           if nextchar == '"':             return parse_string(string, idx + 1, strict)         elif nextchar == '{':             return parse_object((string, idx + 1), strict,                 _scan_once, object_hook, object_pairs_hook, memo)         elif nextchar == '[':             return parse_array((string, idx + 1), _scan_once)         elif nextchar == 'n' and string[idx:idx + 4] == 'null':             return None, idx + 4         elif nextchar == 't' and string[idx:idx + 4] == 'true':             return True, idx + 4         elif nextchar == 'f' and string[idx:idx + 5] == 'false':             return False, idx + 5           m = match_number(string, idx)         if m is not None:             integer, frac, exp = m.groups()             if frac or exp:                 res = parse_float(integer + (frac or '') + (exp or ''))             else:                 res = parse_int(integer)             return res, m.end()         elif nextchar == 'N' and string[idx:idx + 3] == 'NaN':             return parse_constant('NaN'), idx + 3         elif nextchar == 'I' and string[idx:idx + 8] == 'Infinity':             return parse_constant('Infinity'), idx + 8         elif nextchar == '-' and string[idx:idx + 9] == '-Infinity':             return parse_constant('-Infinity'), idx + 9         elif nextchar == '+' and string[idx:idx + 9] == '+Infinity':  #  These lines were added;             return parse_constant('+Infinity'), idx + 9         else:             raise StopIteration(idx)       def scan_once(string, idx):         try:             return _scan_once(string, idx)         finally:             memo.clear()       return scan_once      class PositiveInfinityJSONDecoder(json.JSONDecoder):     def __init__(self, *args, **kwargs):         super().__init__(*args, **kwargs)         self.scan_once = py_make_scanner(self)  # Force to use python (not c) version of py_make_scnner      print(json.loads('+Infinity', cls=PositiveInfinityJSONDecoder))

[micseydel]

Thanks scidamn! I had meant to loop back around to that and forgot. There's one small improvement to what you had that I will be incorporating - if line 55 just uses "Infinity" instead of "+Infinity" then you can drop the _CONSTANTS update.