In this series looking at features introduced by every version of Python 3, we continue our look at the new Python 3.11 release, taking a look at new language features around exceptions and error handling.
This is the 25th of the 35 articles that currently make up the “Python 3 Releases” series.
After looking at the performance improvements in Python 3.11 in the last article, in this one I’m going to run through the first set of what I feel are the main new features in the release, some changes to exceptions and their handling.
As an aside, you’ll probably have noticed these articles on 3.11 are smaller than some of my previously monolithic beasts in this series — I’ve come to the conclusion that it’s easier for me and you both if I don’t try to cram so much into them, and hopefully keep them more accessible.
Here’s a headline summary of what I’ll be discussing in this article:
except
.Exception grouping, in the words of PEP 654, allows programs to raise and handle multiple unrelated exceptions simultaneously. It defines a new exception type ExceptionGroup
, which represents a group of exceptions, and a new except*
syntax for handling these groups.
In this section I try to provide an overview of the important aspects, but there are some intricate subtleties that I can’t cover in full detail here, particularly around differences between raising and reraising exceptions withint exception group handlers, so do check out the PEP if you need a higher level of detail. If this section gets a little too mentally taxing, just skip to the next one!
Before we go into the technical details, it’s fair to ask what’s the motivation for such a change. The PEP has a fairly extensive Rationale section, but to summarise the specific examples it lists are:
The exception groups are themselves exceptions, and there are two:
ExceptionGroup
Exception
, and it itself is a subclass of Exception
.BaseExceptionGroup
BaseException
, and it itself is a subclass of BaseException
.Since these themselves are Exception
and BaseException
subclasses, existing except
and other mechanisms work on them. Their constructors take a message string, much like Exception
, but also a list of other exceptions to include in the group.
>>> from traceback import print_exception
>>>
>>> try:
... raise ExceptionGroup("This is my exception group", [
... IndexError(5),
... KeyError("five"),
... RuntimeError("cinq")
... ])
... except Exception as exc:
... print_exception(exc)
...
+ Exception Group Traceback (most recent call last):
| File "<stdin>", line 2, in <module>
| ExceptionGroup: This is my exception group (3 sub-exceptions)
+-+---------------- 1 ----------------
| IndexError: 5
+---------------- 2 ----------------
| KeyError: 'five'
+---------------- 3 ----------------
| RuntimeError: cinq
+------------------------------------
The consequence of these groups being themselves exceptions is that they can be nested — there’s nothing to stop you raising an ExceptionGroup
which itself contains one or more ExceptionGroup
instances as its contained exceptions. This means you can think of these groups as less of a container, and potentially more as a tree — although I don’t know in practice how often this sort of nesting is likely to be useful.
This is quite flexible, but it might make writing generic code to deal with these exceptions quite challenging — fortunately I suspect that most people will just need some fairly simple behaviours to deal with them.
There are two basic ways to handle these exceptions, which I’ll discuss below.
ExceptionGroup
¶The easiest way to handle these exception groups is to specifically catch them with something like this:
try:
some_function()
except ExceptionGroup as exc_group:
...
At this point, there’s probably one of a couple of things you want to do. One would be to simply log the exception and move on — you can do this without any special effort, as we saw in the interaction session above the traceback for one of these groups includes the details for each contained exception.
The other likely option would be to look for certain types of exception which can be specially handled, and extract them from the group. There are a couple of methods on the group which can be used for this purpose. Both of these methods take a filter function, which expects to be passed an Exception
or BaseException
instance as the sole argument, and should return a bool
.
subgroup()
True
. Any exception groups which are empty after filtering are elided, and if the group as a whole contains no exceptions then None
is returned instead of an empty group.split()
subgroup()
, except it returns a 2-tuple where the first value is the same as for subgroup()
, and the second value is the negation — i.e. a group of all the exceptions not included in the first return value.For completeness there is also an exceptions
attribute which is a tuple
containing all of the exceptions within the group, which can be used to iterate through them manually, should that be required.
As a potentially contrived example, let’s imagine code which wishes to make a TCP connection to any one of the set of IP addresses returned for a domain, but only wants to log if any of the errors were ConnectionRefusedError
, and introduce a delay in this case to avoid triggering security warnings.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
|
Incidentally, socket.create_connection()
is one of only two examples in the library which I can find which has been updated to raise an ExceptionGroup
, although only if the argument all_errors=True
is passed. The other one I’ve come across is in asyncio.TaskGroup
, which uses an ExceptionGroup
(or BaseExceptionGroup
) to share exceptions raised by tasks within the group.
except*
Syntax¶Whilst the approach of catching ExceptionGroup
explicitly like this is simple enough, it’s also a bit of a pain in some ways. What you’ll probably want to handle are the individual exceptions within the group, and using this approach you’d need to iterate over them yourself and do your own exception dispatch. The subgroup()
and split()
methods can be used for this purpose, along with the exceptions
attribute, but a new except*
syntax provides a more convenient approach.
The new except*
syntax allows subsets of exception types to be extracted from the exception group, as with the subgroup()
method, and handled within the a nearly normal except
block. It’s important to note with this syntax, however, that the extracted exception will still be an ExceptionGroup
(or BaseExceptionGroup
) instance — Python doesn’t iterate over the grouped exceptions individually for you.
You can see a simple example of it in action here:
>>> from traceback import print_exception
>>>
>>> try:
... raise ExceptionGroup("Message", [
... IndexError(1),
... KeyError("two"),
... ValueError(3),
... KeyError("four"),
... IndexError(5)
... ])
... except* ValueError as exc_group:
... print("ValueError:")
... print_exception(exc_group)
... except* (KeyError, IndexError) as exc_group:
... print("IndexError and ValueError:")
... print_exception(exc_group)
...
ValueError:
+ Exception Group Traceback (most recent call last):
| File "<stdin>", line 2, in <module>
| ExceptionGroup: Message (1 sub-exception)
+-+---------------- 1 ----------------
| ValueError: 3
+------------------------------------
IndexError and ValueError:
+ Exception Group Traceback (most recent call last):
| File "<stdin>", line 2, in <module>
| ExceptionGroup: Message (4 sub-exceptions)
+-+---------------- 1 ----------------
| IndexError: 1
+---------------- 2 ----------------
| KeyError: 'two'
+---------------- 3 ----------------
| KeyError: 'four'
+---------------- 4 ----------------
| IndexError: 5
+------------------------------------
The listed exception type(s) are extracted from the group as if using subgroup()
and the resultant filtered group is set as the exception argument, exc_group
in the examples above. In this example I’m just printing them, but you could equally well use the exceptions
attribute to iterate across them and perform some action on each one individually.
As with normal try
…except
blocks, each clause is processed in turn, and each exception is only handled by the first matching block, even if the specification of a later block overlaps with it. Also, any exceptions remaining in the group which weren’t handled by any block will be put into a new group and propagate upwards.
Another aspect that’s important to realise is that these blocks will also handle regular exceptions which aren’t contained within a group — the PEP refers to these as naked exceptions. Since the code expects groups, however, these are wrapped in a group so the code handling them can be consistent without the need for additional special cases.
>>> try:
... raise ValueError("hello, world")
... except* ValueError as exc_group:
... print_exception(exc_group)
...
| ExceptionGroup: (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "<stdin>", line 2, in <module>
| ValueError: hello, world
+------------------------------------
To further keep the clauses consistent in their coding style, this new except*
syntax cannot be combined with the original syntax — each try
…except
block must be all of one or the other. The excerpt below demonstrates the SyntaxError
that occurs if you do mix them.
>>> try:
... pass
... except ValueError:
... pass
... except* IndexError as e:
File "<stdin>", line 5
except* IndexError as e:
^^^^^^^
SyntaxError: cannot have both 'except' and 'except*' on the same 'try'
In addition to this limitation, bare catch-all except*
blocks, with no exception type specified, are not allowed because the meaning would be quite confusing. Also, you can’t specify ExceptionGroup
(or BaseExceptionGroup
) as the exception type within an except*
block, only an except
block — this is because the meaning would be rather ambiguous, so it’s been made invalid to avoid confusion.
Finally, the flow control statements break
, continue
and return
cause a SyntaxError
if they occur within an except*
block and would cause flow to exit the block, despite the fact that they’re all valid in a normal except
block. This decision was taken because the exceptions within a group are assumed to be independent of each other, so the handling of one of them shouldn’t be allowed to affect the handling of the others. However, if one except*
block was allowed to change the flow of the code, then this principle would be broken.
>>> for i in range(1):
... try:
... some_function()
... except* ValueError:
... break
...
File "<stdin>", line 5
SyntaxError: 'break', 'continue' and 'return' cannot appear in an except* block
These statements do appear to be legal if they are internal to the block itself, however, and do not affect the flow through other exception handling blocks. This does mean it is still possible to write code that skips handling of other exceptions, as long as their type is handled in the same clause.
>>> try:
... raise ExceptionGroup("Test", [
... ValueError(1),
... ValueError(2)
... ])
... except* ValueError as exc_group:
... for exc in exc_group.exceptions:
... print(exc)
... break
...
1
One more aspect that is of interest is the tracebacks which are printed for each exception within the group.
A normal exception has a single traceback object, stored as the __traceback__
attribute of the exception. This is a series of execution stack frames which lead from the point at which the exception was raised to where it was caught.
For exception groups the situation is similar, except that __traceback__
of the group represents the path from where the group was raised to where it was caught, and the __traceback__
of each exception within the group represents the path from where that exception was raised to where it was caught before being included in the group. When the group is printed, the tracebacks of each grouped exception is also displayed.
To illustrate this, consider the code below which raises an ExceptionGroup
based on several exceptions caught whilst processing a contrived operation.
exceptiongroups.py | |
---|---|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
|
When executed, the ExceptionGroup
propagates to the outermost scope and is printed, producing the output shown below.
+ Exception Group Traceback (most recent call last):
| File "/tmp/exceptiongroups.py", line 28, in <module>
| main()
| File "/tmp/exceptiongroups.py", line 25, in main
| print(double_and_mean("1.2, 3.4, ∞, 0"))
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/tmp/exceptiongroups.py", line 21, in double_and_mean
| raise ExceptionGroup("Failed to calculate mean", exceptions)
| ExceptionGroup: Failed to calculate mean (2 sub-exceptions)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/tmp/exceptiongroups.py", line 13, in double_and_mean
| values.append(convert(arg))
| ^^^^^^^^^^^^
| File "/tmp/exceptiongroups.py", line 4, in convert
| ret = float(arg)
| ^^^^^^^^^^
| ValueError: could not convert string to float: '∞'
+---------------- 2 ----------------
| Traceback (most recent call last):
| File "/tmp/exceptiongroups.py", line 16, in double_and_mean
| ret = statistics.geometric_mean(values)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/Users/andy/.pyenv/versions/3.11.1/lib/python3.11/statistics.py", line 489, in geometric_mean
| raise StatisticsError('geometric mean requires a non-empty dataset '
| statistics.StatisticsError: geometric mean requires a non-empty dataset containing positive numbers
+------------------------------------
You can see that the initial traceback for the group as a whole leads to line 21, where the group was raised. The tracebacks for the ValueError
and StatisticsError
contained within it, however, lead to line 4 and line 16 respectively, which are the lines which triggered them.
This looks like it has the potential to be a very useful feature, although perhaps only applicable in a fairly restricted set of cases. There are also a few challenges to adoption that I can see, however, and I wanted to finish up talking about a few things to consider.
Firstly, developers using libraries or functions updated to take advantage of this feature are going to have to include some careful notes in their documentation to remind developers they may want to switch to using except*
. The feature has been implemented fairly carefully, however, so existing exception handling stands a good chance of doing some sort of a decent job — in particular, except Exception
should still do something useful. The differentiation between ExceptionGroup
and BaseExceptionGroup
should make sure that except Exception
clauses don’t end up catching the few exceptions which aren’t derived from it (e.g. GeneratorExit
and KeyboardInterrupt
). Overall this doesn’t seem like a major issue, but worth checking the release notes for libraries you use over the next year or two if they provide the sort of functionality that might feasibly be updated to use this grouping. Also be aware this isn’t a guarantee, however, since they might have a dependency which is updated to use grouping, but not mention it in their own release notes at all.
Secondly, as with any new Python feature this is going to make libraries depend on newer Python versions. Unlike some others, however, it’s going to be rather fiddly to take advantage of this in an optional manner — in other words, allow those using Python 3.11 to benefit from exception grouping, but still leave the code able to run unmodified on earlier Python versions. To do so would mean a difference in external interface depending on what Python version you’re using, and that could make life tricky for clients of you library as they upgrade. This might mean that only standard library modules, implicitly tied to the Python release cycle, may use this feature for awhile.
Thirdly, and perhaps most fundamentally, this feature has some tricky complexities that might elude developers for awhile until they become accustomed to it. Without wanting to get too bogged down in details, you can get some tricky subtleties if you mess around with nesting exception groups within each other, and with combinations of bare raise
statements along with explicitly re-raising exceptions. If you don’t believe me, take a look at the code below and, without running it, see if you can predict the structure of the exception group that will propagate to the outermost scope.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 |
|
If you think it’s easy, run it and see if you were right. If you get that right first time with only a few moments thought, you’re a much better Python programmer than I!
If you’re still a bit puzzled, however, the diagram below may (or may not!) help you understand the grouping of exceptions that results — the sharp rectangles all represent ExceptionGroup
instances, the rounded rectangles represent naked exceptions, and they’re all laballed with the line of code in the example above which is in the most recent entry in their traceback.
Working out why the code above leads to this grouping is a great way to test your understanding of this feature. As a hint, the part I find most confusing is working out where the top-level ExceptionGroup
comes from, since there doesn’t appear to be anything which would create this. To work this out, compare the difference between line 36 and line 39, and read the section Raising Exceptions in an except*
Block in the PEP.
If you read all the way through the previous section, you may be happy to hear that the other two features covered in this article have considerably less scope for detail! The first of these is the ability to add notes to exceptions.
Very simply, exceptions now have a list of string notes that can be added using a new add_note()
method, which is provided by BaseException
. These are stored in a list
under the __notes__
attribute, and they are also included whenever the exceptions are printed. As an aside, the __notes__
attribute is only created on the first call to add_note()
, so don’t rely on it in exception handling code. You can see this all in action in the simple example below.
>>> try:
... e = Exception("Description")
... e.add_note("First note")
... e.add_note("Second note")
... raise e
... except Exception as exc:
... print("Notes:", repr(exc.__notes__))
... exc.add_note("Third note")
... raise
...
Notes: ['First note', 'Second note']
Traceback (most recent call last):
File "<stdin>", line 5, in <module>
Exception: Description
First note
Second note
Third note
There are a few uses for this, but I’d imagine one of the primary ones is when exceptions are caught and then rethrown — you often want to add some contextual information, but you don’t want to lose the original exception information in the process. Exception chaining, added way back in Python 3.0, goes some way to achieving this but the ability to append notes is also a simple and convenient way to add more information.
PEP 678 recommends that chaining is still the preferred option for cases of re-raising exceptions, but notes should be used in cases where the original exception object should be preserved, or in the case of exception groups, as described above. Speaking of exception groups, the notes for each exception within the group are also printed, which might be useful at times — see the snippet below.
>>> e1 = Exception("one")
>>> e1.add_note("This is one")
>>> e2 = RuntimeError("two")
>>> e2.add_note("This is two")
>>> raise ExceptionGroup("This is the group", (e1, e2))
+ Exception Group Traceback (most recent call last):
| File "<stdin>", line 1, in <module>
| ExceptionGroup: This is the group (2 sub-exceptions)
+-+---------------- 1 ----------------
| Exception: one
| This is one
+---------------- 2 ----------------
| RuntimeError: two
| This is two
+------------------------------------
The last error-handling related change is an improvement to the accuracy of error reporting, defined by PEP 657, building on the better error reporting that we saw in Python 3.10. Instead of just showing the line numbers for an entry in the traceback, the specific expression within the line can also now be highlighted.
To see the difference, let’s see the exception raised by the following code:
foo.py | |
---|---|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
|
In Python 3.10 you can see that the traceback only indicates the line numbers:
$ python3.10 foo.py
Traceback (most recent call last):
File "/tmp/foo.py", line 31, in <module>
sys.exit(main())
File "/tmp/foo.py", line 26, in main
if not set_up_logging(CONFIG):
File "/tmp/foo.py", line 13, in set_up_logging
if not config_dict["logging"]["logging"]:
KeyError: 'logging'
In Python 3.11, however, the specific expressions within the line are highlighted:
$ python3.11 foo.py
Traceback (most recent call last):
File "/tmp/foo.py", line 31, in <module>
sys.exit(main())
^^^^^^
File "/tmp/foo.py", line 26, in main
if not set_up_logging(CONFIG):
^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/foo.py", line 13, in set_up_logging
if not config_dict["logging"]["logging"]:
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
KeyError: 'logging'
The final entry in the traceback particularly highlights the utility of this, because it indicates which of the keys is the missing one, which is ambiguous in the Python 3.10 version.
To provide this information required some updates to the bytecode to store the start and end columns of the source code which produced each bytecode instrucion. Because this increases the storage required for these instructions, both in memory and on disk in .pyc
files, only a single byte is added for each offset. This means that only offsets from 0-255 characters can be stored, with the column offsets being zero-based. Any expressions extended beyond this limit lose their offset information.
Below is an example script which disassembles the aptly named inefficient_sum()
function and shows the line and column information for each bytecode produced. For convenience, the comment at the top shows the column offsets (reach each 2-digit value downwards) — remember that they’re zero-based, so the #
at the start is column zero.
offset.py | |
---|---|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
|
The dis.get_instructions()
function produces an iterator of dis.Instruction
instances. In Python 3.11 these have an added position
attribute of a new dis.Positions
class which provides start and end lines and columns, as used in the code shown above.
The output produced should be as follows:
$ python offset.py
RESUME(0).....................: line=5-5 col=0-0
LOAD_FAST(values).............: line=6-6 col=11-17
POP_JUMP_FORWARD_IF_TRUE(10)..: line=6-7 col=4-25
LOAD_FAST(sum_so_far).........: line=7-7 col=15-25
RETURN_VALUE(None)............: line=7-7 col=8-25
LOAD_FAST(values).............: line=8-8 col=19-25
UNPACK_EX(1)..................: line=8-8 col=4-16
STORE_FAST(first).............: line=8-8 col=4-9
STORE_FAST(rest)..............: line=8-8 col=12-16
LOAD_GLOBAL(inefficient_sum)..: line=9-9 col=11-26
LOAD_FAST(rest)...............: line=10-10 col=8-12
LOAD_FAST(sum_so_far).........: line=11-11 col=8-18
LOAD_FAST(first)..............: line=11-11 col=21-26
BINARY_OP(0)..................: line=11-11 col=8-26
PRECALL(2)....................: line=9-12 col=11-5
CALL(2).......................: line=9-12 col=11-5
RETURN_VALUE(None)............: line=9-12 col=4-5
As a final note, if you don’t want the overhead of these offsets added to your code you can either set the environment variable PYTHONNODEBUGRANGES=1
or you can pass the command-line option -X no_debug_ranges
to the interpreter. This won’t break any code, but the column offsets will come out as None
if queried, and the tracebacks won’t be decorated with the highlighted expressions on each line.
$ python -X no_debug_ranges offset.py
RESUME(0).....................: line=5-5 col=None-None
LOAD_FAST(values).............: line=6-6 col=None-None
POP_JUMP_FORWARD_IF_TRUE(10)..: line=6-6 col=None-None
⋮
Some useful changes here, although it’s early days yet so I must admit I’m a little unsure where these will lie on the utility scale that runs from “couldn’t live without it” to “I forgot it was there”. Exception grouping seems useful for some cases, but I suspect those cases will be a little limited, and the mechanics of it seem a little complicated for some developers. That said, for the cases where it’s useful, it’s going to be a lot better than people rolling their own bespoke mechanism for doing the same thing in each library.
Adding notes to exceptions is a much simpler mechanism, and it strikes me this is likely to apply to more situations — although exception chaining could be used in some of these, the notes are likely easier for everyone and I suspect they might catch on well.
Finally, the expression highlighting in errors basically comes for free for developers, if you ignore the slight code size overhead, and I think this probably shows the greatest potential to improve life for all Python developers.
All in all, though, nothing to dislike here, in my view.