# Hacking the CPython interpreter (Whython part 1)¶

This blog post is inspired by the talk CPython Meanderings by James Powell.

The CPython interpreter is not a hugely complicated software project, which means it's not too hard to add on to as a learning project. Thanks to the strong philosophy against premature optimisation, little interpreter behaviour is obscured.

Python is also largely very well-documented and has a friendly core development community which are great places to go for help. However, you can get very far with a little file searching and debugging.

If you know a little about the internals of Python already, particularly the bytecode compilation model it uses, and can follow a little C, you'll have no problem with this. I won't go into detail about Python bytecode, but here are some recommended reading links:

So what are we going to do with CPython? Well, since this is not really a practical approach to extending the language, we might as well go with something silly. So have you heard of Code Golf? It's a sport where you aim to write a program to complete a particular task, but with the code as short as possible. Python is not typically hugely competitive as languages go, so I thought I'd add something to it to make it better. When golfing, best practices truly go out the window, such as correct error handling. Python, however, is not as forgiving with errors as (say) JavaScript or a Shell language, so let's "fix" that: I'm going to add a short-circuiting inline exception catching operator.

Let's define the operator's behaviour a bit more specifically:

• we'll call it ?, since the ? character is not used anywhere in vanilla Python syntax
• it will have looser precedence than or
• it will still always print the exception when it occurs to make debugging easier

Since this feature is actually a terrible idea, I'm going to call it Exception Mishandling.

By the way, this will form a project called Whython (because, why would I do this?), so I'll be naming things using a Py_Y prefix.

A final warning: this code will walk you through almost exactly the steps I took, including all the trouble-shooting that came with it, because I hope to show you that you could have worked all of this out yourself with some understanding of C-isms and educated guesswork. I will skip over the typos and other obvious mistakes I make though.

## Initial setup¶

First, get yourself a copy of the Python source code. For these purposes I'll be using Python 3.9.5 taken directly from the 3.9 branch on GitHub.

 1 2 3 4 $git clone [email protected]:python/cpython.git$ cd cpython $git fetch 3.9$ git checkout 3.9 

Now, let's create a branch for our new feature:

 1 $git checkout -b exception_mishandling  And finally, create an initial build from here. I'm passing the --with-pydebug flag to configure to give us some helpful debugging information. (Again, this isn't something I magically knew - you can find a list of the options with ./configure --help and then search for debugging-related ones)  1 2 3 4 $ ./configure --with-pydebug $make$ ./python --version Python 3.9.5+ 

## Parsing¶

The first step in the Python interpreter pipeline is parsing. Let's add our ? operator to the parser.

The Python parser, since version 3.9, uses a PEG grammar which generates a parser. The instructions for developing it are on the devguide.

First, let's define the question mark token in Grammar/tokens:

  1 2 3 4 5 6 7 8 9 10 --- a/Grammar/Tokens +++ b/Grammar/Tokens @@ -53,6 +53,7 @@ ATEQUAL '@=' RARROW '->' ELLIPSIS '...' COLONEQUAL ':=' +Y_EXC_MISHANDLING '?' OP AWAIT 

Now we'll add the grammar rule for a ? expression. I don't fully understand much of this, to be honest, but it's largely copied from some of the existing boolean and binary operators' definitions. It also references a _Py_ExcMishandle action, which I think will create the AST node.

  1 2 3 4 5 6 7 8 9 10 11 12 --- a/Grammar/python.gram +++ b/Grammar/python.gram @@ -380,6 +380,9 @@ lambda_param_maybe_default[NameDefaultPair*]: | a=lambda_param c=default? &':' { _PyPegen_name_default_pair(p, a, c, NULL) } lambda_param[arg_ty]: a=NAME { _Py_arg(a->v.Name.id, NULL, NULL, EXTRA) } +y_exc_mishandle[expr_ty] (memo): + | a=disjunction '?' b=y_exc_mishandle { _Py_Y_ExcMishandle(a, b, EXTRA) } + | disjunction disjunction[expr_ty] (memo): | a=conjunction b=('or' c=conjunction { c })+ { _Py_BoolOp( Or, 

But it needs to be used somewhere else in the grammar, because otherwise it would be unreachable. Since we're adding it above disjunction (an or expression), we'll replace all other uses of disjunction with y_exc_mishandle:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 --- a/Grammar/python.gram +++ b/Grammar/python.gram @@ -327,8 +327,8 @@ expressions[expr_ty]: | a=expression ',' { _Py_Tuple(CHECK(_PyPegen_singleton_seq(p, a)), Load, EXTRA) } | expression expression[expr_ty] (memo): - | a=disjunction 'if' b=disjunction 'else' c=expression { _Py_IfExp(b, a, c, EXTRA) } - | disjunction + | a=y_exc_mishandle 'if' b=y_exc_mishandle 'else' c=expression { _Py_IfExp(b, a, c, EXTRA) } + | y_exc_mishandle | lambdef lambdef[expr_ty]: @@ -522,9 +525,9 @@ kvpair[KeyValuePair*]: a=expression ':' b=expression { _PyPegen_key_value_pair(p for_if_clauses[asdl_seq*]: | for_if_clause+ for_if_clause[comprehension_ty]: - | ASYNC 'for' a=star_targets 'in' ~ b=disjunction c=('if' z=disjunction { z })* { + | ASYNC 'for' a=star_targets 'in' ~ b=y_exc_mishandle c=('if' z=y_exc_mishandle { z })* { CHECK_VERSION(6, "Async comprehensions are", _Py_comprehension(a, b, c, 1, p->arena)) } - | 'for' a=star_targets 'in' ~ b=disjunction c=('if' z=disjunction { z })* { + | 'for' a=star_targets 'in' ~ b=y_exc_mishandle c=('if' z=y_exc_mishandle { z })* { _Py_comprehension(a, b, c, 0, p->arena) } | invalid_for_target 

We also need to define the new Y_ExcMishandle AST node type in Python.asdl:

  1 2 3 4 5 6 7 8 9 10 --- a/Parser/Python.asdl +++ b/Parser/Python.asdl @@ -53,6 +53,7 @@ module Python -- BoolOp() can use left & right? expr = BoolOp(boolop op, expr* values) + | Y_ExcMishandle(expr try_expr, expr except_expr) | NamedExpr(expr target, expr value) | BinOp(expr left, operator op, expr right) | UnaryOp(unaryop op, expr operand) 

This AST item is the action that gets called in the generated parser, I think.

Finally, we need to regenerate the parser code, as described in the devguide:

 1 2 3 4 $./configure --with-pydebug # create Makefile$ make regen-token $make regen-ast$ make regen-pegen 

### Testing it out¶

 1 2 $./configure --with-pydebug # not sure if this was actually necessary to do again$ make 

And just like that, it actually compiled, with few warnings and no errors! I did scan through the compiler output, though, to see how bad GCC thought my code so far was, and I found only one noticeable problem, repeated a few times:

 1 2 3 Python/ast.c:223:5: warning: enumeration value ‘Y_ExcMishandle_kind’ not handled in switch [-Wswitch] Python/compile.c:4993:5: warning: enumeration value ‘Y_ExcMishandle_kind’ not handled in switch [-Wswitch] Python/symtable.c:1535:5: warning: enumeration value ‘Y_ExcMishandle_kind’ not handled in switch [-Wswitch] 

Some issues in compile.c were to be expected, because we haven't implemented any support beyond parsing yet. I didn't know what symtable.c was until now, but it turns out it's an intermediate step in compilation that works out variable scoping.

However, an issue in ast.c seemed more pressing. It was in the validate_expr function, which seemed to be responsible for checking semantic issues in the syntax that couldn't be identified by the parser.

Just for fun, I decided to try our newly built Python anyway: - python --version worked - python alone worked and gave me a REPL with most functionality working fine

I knew for a fact simply trying to run some code with a ? operator in it would crash the interpreter. Instead I decided to try, first of all, using the ast module to see if this expression could be correctly parsed:

 1 2 3 4 5 $python -m ast << EOF 1 ? 2 EOF python: Parser/pegen/pegen.c:1161: _PyPegen_run_parser: Assertion PyAST_Validate(res)' failed. Abort (core dumped)  Yikes. I got greedy - I guess the AST validation was necessary. ### AST validation¶ Let's find the definition of the function that errored:  1 2 3 4 $ grep -r PyAST_Validate -*- snip -*- Python/ast.c:PyAST_Validate(mod_ty mod) -*- snip -*- 

and analysing the functions it calls, we see that validate_expr was almost certainly the culprit: at the bottom of the function is a handler for if the switch-case doesn't match anything:

 1 2 PyErr_SetString(PyExc_SystemError, "unexpected expression"); return 0; 

So, drawing heavy inspiration from the nearby expression handling code for binary operations, I came up with this:

  1 2 3 4 5 6 7 8 9 10 11 12 --- a/Python/ast.c +++ b/Python/ast.c @@ -227,6 +227,9 @@ validate_expr(expr_ty exp, expr_context_ty ctx) return 0; } return validate_exprs(exp->v.BoolOp.values, Load, 0); + case Y_ExcMishandle_kind: + return validate_expr(exp->v.Y_ExcMishandle.try_body, Load) && + validate_expr(exp->v.Y_ExcMishandle.except_body, Load); case BinOp_kind: return validate_expr(exp->v.BinOp.left, Load) && validate_expr(exp->v.BinOp.right, Load); 

And parsing it works!

  1 2 3 4 5 6 7 8 9 10 11 $make$ ./python -m ast << EOF 1 ? 2 EOF Module( body=[ Expr( value=Y_ExcMishandle( try_body=Constant(value=1), except_body=Constant(value=2)))], type_ignores=[]) 

## Compiling¶

If we take the next step in the Python interpreter's pipline, and try to compile some code that uses our operator, we'll see an unexpected result:

 1 2 >>> compile("1 ? 2", "string", "eval") at 0x7fe4b6af0110, file "string", line 1> 

It seemed to compile just fine! Weird, huh? Now, if we run it, we're surely not going to get a success...

 1 2 3 >>> eval(_) python: Python/ceval.c:2055: _PyEval_EvalFrameDefault: Assertion EMPTY()' failed. Abort (core dumped) 

And so the rabbit hole goes deeper. I want to see why that compiled successfully, and what bytecode it produced:

 1 2 3 4 5 >>> from dis import dis >>> compile("1 ? 2", "string", "eval") at 0x7fe4b6af0110, file "string", line 1> >>> dis(_) 1 0 RETURN_VALUE 

I think I know why it crashed now: it tried to return the top of the stack, but nothing had been pushed on to it. I'm also guessing that the reason the code is so short is because the compiler didn't know how to handle our new AST node, but it added the RETURN_VALUE because all code must end in it, or at least it does by default.

Interestingly, if we try to compile it in exec mode instead of eval mode, it does crash while compiling:

 1 2 3 >>> code = compile("1 ? 2", "string", "exec") python: Python/compile.c:5507: stackdepth: Assertion depth >= 0' failed. Abort (core dumped) 

Anyway, let's get on to dealing with compilation.

### Symbol table¶

First I want to have a look at the symtable.c we found earlier. The switch statement our compiler was complaining about is here:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 switch (e->kind) { case NamedExpr_kind: if(!symtable_handle_namedexpr(st, e)) VISIT_QUIT(st, 0); break; case BoolOp_kind: VISIT_SEQ(st, expr, e->v.BoolOp.values); break; case BinOp_kind: VISIT(st, expr, e->v.BinOp.left); VISIT(st, expr, e->v.BinOp.right); break; case UnaryOp_kind: VISIT(st, expr, e->v.UnaryOp.operand); break; -*- snip -*- } 

As before, we can pretty much copy from BinOp's implementation without too much bother:

  1 2 3 4 5 6 7 8 9 10 11 12 13 --- a/Python/symtable.c +++ b/Python/symtable.c @@ -1540,6 +1540,10 @@ symtable_visit_expr(struct symtable *st, expr_ty e) case BoolOp_kind: VISIT_SEQ(st, expr, e->v.BoolOp.values); break; + case Y_ExcMishandle_kind: + VISIT(st, expr, e->v.Y_ExcMishandle.try_body); + VISIT(st, expr, e->v.Y_ExcMishandle.except_body); + break; case BinOp_kind: VISIT(st, expr, e->v.BinOp.left); VISIT(st, expr, e->v.BinOp.right); 

and as far as I can tell, that's all we need to do here.

## Understanding try-except¶

### Control-flow graphs¶

The devguide page for the CPython compiler informs us of the two major remaining steps: transformation of AST into a Control Flow Graph, and emission of bytecode based on that. The control flow graph is very close to the final bytecode, but does not have jump instructions "resolved", so-to-speak; they are described as labels only. The CFG is implemented using "basic blocks", which are just "chunks" of bytecode, but the jump instructions point to other basic blocks instead of using offset numbers. When the compiler emits an instruction, it does so by appending it to the active basic block, which is changed using compiler_use_next_block.

Much of this information, by the way, I got only by inference while reading the source code in compile.c. It's surprisingly simple to follow.

### Exception handling and frame blocks¶

Let's disassemble a try-except statement in the simplest case, from which we can take inspiration:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 >>> dis("try: a\nexcept: b") 1 0 SETUP_FINALLY 8 (to 10) 2 LOAD_NAME 0 (a) 4 POP_TOP 6 POP_BLOCK 8 JUMP_FORWARD 16 (to 26) 2 >> 10 POP_TOP 12 POP_TOP 14 POP_TOP 16 LOAD_NAME 1 (b) 18 POP_TOP 20 POP_EXCEPT 22 JUMP_FORWARD 2 (to 26) 24 RERAISE >> 26 LOAD_CONST 0 (None) 28 RETURN_VALUE 

Each Python stack frame has a stack of blocks, which represent loops and try-except syntactic blocks. They are used to keep track of where to jump if an exception occurs, and where to jump with the break and continue statements (and the corresponding BREAK and CONTINUE opcodes).

Again, I learnt this by examining CPython's code (ceval.c, the main interpreter loop), so don't feel like I just magically found this out.

Here's a simplified and annotated bytecode template that we need to write for an exception mishandling expression like a ? b, and that also explains the blocks in more detail:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 Line Byte Opcode Argument # Push frame block telling interpreter where to jump in case an exception occurs 1 0 SETUP_FINALLY 8 (to 8) # Evaluate contents of try block (in this case just the expression a) 2 LOAD_NAME 0 (a) # If we reached this point, the try block was evaluated without error. We need to remove the frame block # so that future code doesn't keep the exception handler 4 POP_BLOCK # Skip the except handler block because no error occurred. 6 JUMP_FORWARD 12 (to 18) >> # This is where we get if an error occurs. Here the interpreter will push a special frame block, # "EXCEPT_HANDLER", which is just used to record the fact we're inside an except block so Python can # reset its internal state afterwards. # The interpreter will also push 3 things: exception type, exception object, and traceback object. We # don't care about any of these, so pop them all. 8 POP_TOP # type 10 POP_TOP # exc 12 POP_TOP # tb # Evaluate contents of except block (in this case just the expression b) 14 LOAD_NAME 1 (b) # Tell Python we've finished in the except block, and to reset its internal state (clear sys.exc_info). 16 POP_EXCEPT >> # Here is where we'll jump to if no error occurred. # End of the expression, so return the relevant value 18 RETURN_VALUE 

At the points in the bytecode where the interpreter at runtime will push a new frame block, we need to inform the compiler

## Back to compiling¶

All that remains is to write the code to compile it. Here's the function template we'll need:

 1 2 3 4 static int compiler_exception_mishandling(struct compiler *c, expr_ty e) { return 1; } 

The function returns either 0 or 1 to indicate whether compilation succeeded.

As a sanity check, we'll assert that the expression is the right kind. I'm only adding this because all the other compiler functions have it.

 1 assert(e->kind == Y_ExcMishandle_kind); 

Now, to save myself the effort of writing the meat of the function from scratch, I'm going to copy the set of functions called by compiler_try_except by getting a trace of all the functions called with a debugger.

 1 2 $make$ gdb ./python 

I'll start the interpreter first, so we don't break on a whole load of things we don't care about:

 1 2 3 4 (gdb) run Python 3.9.5+ [...] Type "help", "copyright", "credits" or "license" for more information. >>> ^C 

(get back to GDB by pressing Ctrl+C)

How I'll break on the first line of compiler_try_except (the line number may be different for you depending on what version you start from and what modifications you've made already), and type in a try-except statement that roughly corresponds to the desired behaviour of exception mishandling:

  1 2 3 4 5 6 7 8 9 10 (gdb) break Python/compile.c:3080 Breakpoint 1 at 0x55555569632d: file Python/compile.c, line 3080. (gdb) continue Continuing. >>> try: 1 ... except: 1 ... Breakpoint 1, compiler_try_except (c=0x7fffffffd570, s=0x555555c4a960) at Python/compile.c:3080 

My plan from here is to just keep stepping through the compiler function using next and copy all the functions that are called into compiler_exception_mishandling, tweaking them as necessary:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 3080 body = compiler_new_block(c); (gdb) next 3081 except = compiler_new_block(c); (gdb) 3082 orelse = compiler_new_block(c); (gdb) 3083 end = compiler_new_block(c); (gdb) 3084 if (body == NULL || except == NULL || orelse == NULL || end == NULL) (gdb) 3086 ADDOP_JREL(c, SETUP_FINALLY, except); (gdb) 3087 compiler_use_next_block(c, body); (gdb) 3088 if (!compiler_push_fblock(c, TRY_EXCEPT, body, NULL, NULL)) (gdb) 3090 VISIT_SEQ(c, stmt, s->v.Try.body); (gdb) 3091 ADDOP(c, POP_BLOCK); (gdb) 3092 compiler_pop_fblock(c, TRY_EXCEPT, body); (gdb) 3093 ADDOP_JREL(c, JUMP_FORWARD, orelse); (gdb) 3094 n = asdl_seq_LEN(s->v.Try.handlers); (gdb) 3095 compiler_use_next_block(c, except); (gdb) 3097 if (!compiler_push_fblock(c, EXCEPTION_HANDLER, NULL, NULL, NULL)) (gdb) 3100 excepthandler_ty handler = (excepthandler_ty)asdl_seq_GET( (gdb) 3102 if (!handler->v.ExceptHandler.type && i < n-1) (gdb) 3104 SET_LOC(c, handler); (gdb) 3105 except = compiler_new_block(c); (gdb) 3106 if (except == NULL) (gdb) 3108 if (handler->v.ExceptHandler.type) { (gdb) 3113 ADDOP(c, POP_TOP); (gdb) 3114 if (handler->v.ExceptHandler.name) { (gdb) 3167 cleanup_body = compiler_new_block(c); (gdb) 3168 if (!cleanup_body) (gdb) 3171 ADDOP(c, POP_TOP); (gdb) 3172 ADDOP(c, POP_TOP); (gdb) 3173 compiler_use_next_block(c, cleanup_body); (gdb) 3174 if (!compiler_push_fblock(c, HANDLER_CLEANUP, cleanup_body, NULL, NULL)) (gdb) 3176 VISIT_SEQ(c, stmt, handler->v.ExceptHandler.body); (gdb) 3177 compiler_pop_fblock(c, HANDLER_CLEANUP, cleanup_body); (gdb) 3178 ADDOP(c, POP_EXCEPT); (gdb) 3179 ADDOP_JREL(c, JUMP_FORWARD, end); (gdb) 3181 compiler_use_next_block(c, except); (gdb) 3099 for (i = 0; i < n; i++) { (gdb) 3183 compiler_pop_fblock(c, EXCEPTION_HANDLER, NULL); (gdb) 3184 ADDOP(c, RERAISE); (gdb) 3185 compiler_use_next_block(c, orelse); (gdb) 3186 VISIT_SEQ(c, stmt, s->v.Try.orelse); (gdb) 3187 compiler_use_next_block(c, end); (gdb) 3188 return 1 

Here's that cleaned up a little (I removed the unexecuted branches, except the error handling ones, whose bodies I added (they're all just return 0;)), and with some of the variables adapted to use the AST types we defined):

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 static int compiler_exception_mishandling(struct compiler *c, expr_ty e) { basicblock *body, *except, *orelse, *cleanup_body, *end; assert(e->kind == Y_ExcMishandle_kind); body = compiler_new_block(c); except = compiler_new_block(c); orelse = compiler_new_block(c); end = compiler_new_block(c); if (body == NULL || except == NULL || orelse == NULL || end == NULL) return 0; ADDOP_JREL(c, SETUP_FINALLY, except); compiler_use_next_block(c, body); if (!compiler_push_fblock(c, TRY_EXCEPT, body, NULL, NULL)) return 0; VISIT(c, expr, e->v.Y_ExcMishandle.try_body); ADDOP(c, POP_BLOCK); compiler_pop_fblock(c, TRY_EXCEPT, body); ADDOP_JREL(c, JUMP_FORWARD, orelse); compiler_use_next_block(c, except); if (!compiler_push_fblock(c, EXCEPTION_HANDLER, NULL, NULL, NULL)) return 0; except = compiler_new_block(c); if (except == NULL) return 0; ADDOP(c, POP_TOP); cleanup_body = compiler_new_block(c); if (!cleanup_body) return 0; ADDOP(c, POP_TOP); ADDOP(c, POP_TOP); compiler_use_next_block(c, cleanup_body); if (!compiler_push_fblock(c, HANDLER_CLEANUP, cleanup_body, NULL, NULL)) return 0; VISIT(c, expr, e->v.Y_ExcMishandle.except_body); compiler_pop_fblock(c, HANDLER_CLEANUP, cleanup_body); ADDOP(c, POP_EXCEPT); ADDOP_JREL(c, JUMP_FORWARD, end); compiler_use_next_block(c, except); compiler_pop_fblock(c, EXCEPTION_HANDLER, NULL); ADDOP(c, RERAISE); compiler_use_next_block(c, orelse); compiler_use_next_block(c, end); return 1; } 

Finally, let's add a handler for expressions of this kind to the compiler function for arbitrary expressions, in the switch-case pointed out to us by gcc earlier:

  1 2 3 4 5 6 7 8 9 10 11 --- a/Python/compile.c +++ b/Python/compile.c @@ -4996,6 +5037,8 @@ compiler_visit_expr1(struct compiler *c, expr_ty e) ADDOP(c, DUP_TOP); VISIT(c, expr, e->v.NamedExpr.target); break; + case Y_ExcMishandle_kind: + return compiler_exception_mishandling(c, expr); case BoolOp_kind: return compiler_boolop(c, e); case BinOp_kind: 

### Trying it out¶

Let's try out this handywork!

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 $./configure --with-pydebug$ make \$ ./python >>> import dis >>> dis.dis("1 ? 2") 1 0 SETUP_FINALLY 6 (to 8) 2 LOAD_CONST 0 (1) 4 POP_BLOCK 6 RETURN_VALUE >> 8 POP_TOP 10 POP_TOP 12 POP_TOP 14 LOAD_CONST 1 (2) 16 POP_EXCEPT 18 RETURN_VALUE 20 RERAISE 22 RETURN_VALUE 

Looks not bad! And the compiler didn't even crash! However, there are a few things wrong: - why are there extra RETURN_VALUE instructions? - why is there a RERAISE instruction?

After scouring the CPython codebase for instances of RETURN_VALUE, and a bit of Google-fu, I found that they came from the peephole optimiser, a system which makes simple optimisations to the bytecode after its main compile step. One of the things it does is replace unconditional JUMP instructions that point to RETURN_VALUE instructions, with RETURN_VALUE instructions directly. We can see the code without them if we compile the expression in a single-expression context (as is used in the Python REPL) which prints the expression instead of returning it:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 >>> dis.dis(compile("1 ? 2", "", "single")) 1 0 SETUP_FINALLY 6 (to 8) 2 LOAD_CONST 0 (1) 4 POP_BLOCK 6 JUMP_FORWARD 14 (to 22) >> 8 POP_TOP 10 POP_TOP 12 POP_TOP 14 LOAD_CONST 1 (2) 16 POP_EXCEPT 18 JUMP_FORWARD 2 (to 22) 20 RERAISE >> 22 PRINT_EXPR 24 LOAD_CONST 2 (None) 26 RETURN_VALUE 

Now as for the RERAISE instruction, it looks like it will always be skipped, so it must be a weird side-effect of copying the code for a try-except statement. I'll come to removing that later.

Let's see what happens if we actually try to use this 1 ? 2 expression:

 1 2 >>> 1 ? 2 1 

Wow! Just like that, it worked! Let's try the case in which an error occurs, by dividing by zero:

 1 2 3 >>> 1/0 ? 2 python: Objects/object.c:411: PyObject_Repr: Assertion !_PyErr_Occurred(tstate)' failed. Abort (core dumped) 

That's irritating - and worse, all it tells us is that a PyErr_Occurred, which is vague and tells us nothing about the root cause.

I spent a long time floundering here: adding all sorts of debugging instrumentation to my copy of the interpreter to work out what exactly was going wrong, but I got nowhere.

The final insight I needed to solve the problem came from comparing the disassemblies of these two functions:

 1 2 3 4 5 6 def f(): try: return a except: return b def g(): return a ? b 

These should be identical in behaviour, but let's look at their bytecode:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 >>> dis.dis(f) 2 0 SETUP_FINALLY 6 (to 8) 2 LOAD_GLOBAL 0 (a) 4 POP_BLOCK 6 RETURN_VALUE 3 >> 8 POP_TOP 10 POP_TOP 12 POP_TOP 14 LOAD_GLOBAL 1 (b) 16 ROT_FOUR 18 POP_EXCEPT 20 RETURN_VALUE 22 RERAISE 24 LOAD_CONST 0 (None) 26 RETURN_VALUE >>> dis.dis(g) 2 0 SETUP_FINALLY 6 (to 8) 2 LOAD_GLOBAL 0 (a) 4 POP_BLOCK 6 RETURN_VALUE >> 8 POP_TOP 10 POP_TOP 12 POP_TOP 14 LOAD_GLOBAL 1 (b) 16 POP_EXCEPT 18 RETURN_VALUE 20 RERAISE 22 RETURN_VALUE 

Besides the dangling LOAD_CONST in the first, the substantial change is the addition of a ROT_FOUR in the working one. It turns out this is necessary.

The fix, then, was simple: insert a ROT_FOUR instruction after the handler expression is visited:

  1 2 3 4 5 6 7 8 9 10 --- a/Python/compile.c +++ b/Python/compile.c @@ -3214,6 +3214,7 @@ compiler_exception_mishandling(struct compiler *c, expr_ty e) { if (!compiler_push_fblock(c, HANDLER_CLEANUP, cleanup_body, NULL, NULL)) return 0; VISIT(c, expr, e->v.Y_ExcMishandle.except_body); + ADDOP(c, ROT_FOUR); compiler_pop_fblock(c, HANDLER_CLEANUP, cleanup_body); ADDOP(c, POP_EXCEPT); ADDOP_JREL(c, JUMP_FORWARD, end); 

## Next steps¶

Where do I go from here?

### Documentation and tests¶

This feature has no unit tests and is undocumented. Were this an official addition to Python (God help us), it would need those.

### Printing error messages¶

For debugging purposes, it's probably useful to print the error tracebacks even though they're ignored.

### Ignoring SystemExit and KeyboardInterrupt¶

It might be worth letting these two non-errors pass through silently, because they (particularly KeyboardInterrupt) might cause some confusing and unreliable behaviour.

First published: 2021-09-19
Last updated: 2021-09-19