The -O flag is documented as removing assert instructions as though you never wrote them. But that's not all it does. It also sets a constant, __debug__, which is normally True, to False. The value of __debug__ is known at compile time, so Python can use it to completely discard conditional blocks predicated on __debug__, just as it does with assert statements, when running with the -O flag. And in fact, this is exactly what Python does!
The upshot is that you can write debug code that is stripped by Python's -O flag without abusing assert. This is easily demonstrated using Python's bytecode dissambler, dis.
from dis import dis
def debug_func():
if __debug__:
print "debugging"
return
def noop_func():
return
print "debug_func:"
dis(debug_func)
print "noop_func:"
dis(noop_func)
Save this as debugtest.py, then execute it with python -O debugtest.py. You'll see that the disassembly of the two functions is identical aside from the line number offsets. It's just as if we never even wrote the if statement and its subordinate print statement! Literally zero performance impact to production code.
debug_func:
4 0 LOAD_CONST 0 (None)
3 RETURN_VALUE
noop_func:
2 0 LOAD_CONST 0 (None)
3 RETURN_VALUE
What's more, when we run this script without the -O flag, Python optimizes away the test. That is, Python knows __debug__ is true at compile time, and so it just compiles the code inside the if statement as if it weren't inside an if statement! Here's what the disassembly of debug_func looks like when __debug__ is True (i.e., no -O flag is used):
debug_func:
3 0 LOAD_CONST 1 ('debugging')
3 PRINT_ITEM
4 PRINT_NEWLINE
4 5 LOAD_CONST 0 (None)
8 RETURN_VALUE
By comparison, here's what it would look like if we were using some other conditional (say, a global variable called DEBUG). You'll see that this is much more complicated, and if you time it, you'll find that executing the test at runtime actually adds significant overhead.
debug_func:
2 0 LOAD_GLOBAL 0 (DEBUG)
3 JUMP_IF_FALSE 9 (to 15)
6 POP_TOP
3 7 LOAD_CONST 1 ('debugging')
10 PRINT_ITEM
11 PRINT_NEWLINE
12 JUMP_FORWARD 1 (to 16)
>> 15 POP_TOP
>> 16 LOAD_CONST 0 (None)
19 RETURN_VALUE
So basically, Python will not only strip debugging code if it's conditionalized by testing __debug__, it will also slightly improve the performance of your debug code when running in debug mode compared to testing a runtime flag. And best of all, it does this magic using the same command line flag, -O, that strips assert statements! (For completeness, I should mention here that the PYTHONOPTIMIZE environment variable serves the same function as -O.)
But wait, there's more! If you use an else clause with your if __debug__ statement, Python is smart enough to strip whichever clause doesn't apply and "inline" the clause that does!
def get_run_mode():
if __debug__:
return "debug"
else:
return "production"
dis(get_run_mode) running without -O:
3 0 LOAD_CONST 1 ('debug')
3 RETURN_VALUE
dis(get_run_mode) running with -O:
5 0 LOAD_CONST 1 ('production')
3 RETURN_VALUE
Once again, for comparison, here's how the bytecode looks when the function is written to force runtime evaluation of the condition, by using a global variable DEBUG instead of __debug__:
2 0 LOAD_GLOBAL 0 (DEBUG)
3 JUMP_IF_FALSE 5 (to 11)
6 POP_TOP
3 7 LOAD_CONST 1 ('debug')
10 RETURN_VALUE
>> 11 POP_TOP
5 12 LOAD_CONST 2 ('production')
15 RETURN_VALUE
16 LOAD_CONST 0 (None)
19 RETURN_VALUE
So, is Python smart enough to optimize if not __debug__ in the same way? Sadly, no:
def not_debug_test():
if not __debug__:
print "production"
Bytecode:
>>> dis(not_debug_test)
2 0 LOAD_GLOBAL 0 (__debug__)
3 JUMP_IF_TRUE 9 (to 15)
6 POP_TOP
3 7 LOAD_CONST 1 ('production')
10 PRINT_ITEM
11 PRINT_NEWLINE
12 JUMP_FORWARD 1 (to 16)
>> 15 POP_TOP
>> 16 LOAD_CONST 0 (None)
19 RETURN_VALUE
So if you want to write code that's run only in production, don't use if not __debug__. Write it like this instead:
if __debug__:
pass
else:
print "production"
This is ugly, but arguably, it should be: you generally shouldn't write code that is only run in production, because it doesn't get tested.
What about conditional expressions, such as x = "yes" if __debug__ else "no"? Sadly, Python does not optimize these. Similarly, __debug__ and x and __debug__ or x are not optimized, though they could be.
So what did we learn?
- Use if __debug__ to write debug code (along with else if desired).
- Don't make up your own flag for this, as it will prevent Python from being clever.
- Don't use if not __debug__ because this will also prevent Python from being clever.
- Prefer if statements to using __debug__ in logical expressions.
- Use assert to assert invariants, not to perform stupid Python tricks like I presented last June.
- Use the -O command line flag (or PYTHONOPTIMIZE) to tell Python when it's running in production. If you don't, you may be executing debugging code you don't want, with the potential performance degradation that implies.
By the way, in Python 3.x, True and False are also constants whose value is known at compile-time, and Python optimizes if True and if False similarly. In Python 2.x, the values of True and False can be changed at run time (seriously, try it if you don't believe me!), so this optimization isn't possible. None can't be changed in Python 2.x, but is only a true compile-time constant in Python 3.x, with the upshot that code under if None is also subject to being stripped out in Python 3.x but not in Python 2.x.
Hi..
ReplyDeleteI have a question for you about one of your answered question in stack overflow. I am unable to find any way to contact you.. hence commenting here. It is a closed thread about a year back http://stackoverflow.com/questions/10853412/can-pexpect-be-told-to-ignore-a-pattern-or-signal
Can you please help me regarding this. Thanks
You can reach me via engyrus /AT/ gmail.com, or just post your question here for that matter, possibly I'll make a blog entry out of it. :-)
Delete