Friday, November 8, 2013

SPT #2: One global to rule them all

"Stupid Python Tricks" explores ways to abuse Python features for fun and profit. Stupid Python Tricks may well be diametrically opposed to best practices. Don your peril-sensitive sunglasses before proceeding.

You know how when you have a lot of globals and it's a pain to modify them, because you have to declare the globals you want to modify in each function using the global statement? Yeah, me neither. After all, globals are a poor programming practice! And I never engage in those...

However, for educational porpoises, or at least didactic dolphins, here's a short hack that takes advantage of the fact that Python globals can be accessed, but not modified, without declaring them global. Basically, what we do is replace the built-in globals() function with an object that lets you read and set globals via attribute access. Instead of doing this:

def my_func(a, b, c):
    global ans
    ans = a + b + c

You can instead do:

import globals

def my_func(a, b, c):
    globals.ans = a + b + c

This works fine because you aren't re-binding any global names; instead, you are merely mutating an existing object. Or so Python believes... bwahahaha! While we didn't save any lines of code in this short example, imagine if we had dozens of functions that used globals. We'd save literally dozens of global statements! And we could dine on spaghetti code for weeks.

Calling this object, i.e. globals(), continues to return the modules's globals dictionary, just like before, thanks to a __call__() method on the class.

Without further ado, then, here's

from inspect import currentframe
from sys import modules

class Globals(object):

    def __getattribute__(self, key, currentframe=currentframe):
            return currentframe().f_back.f_globals[key]
        except KeyError:
            pass # if we raise NameError here we get 2 err msgs
        raise NameError("global name '%s' is not defined" % key)

    def __setattr__(self, key, value, currentframe=currentframe):
        currentframe().f_back.f_globals[key] = value

    def __call__(self, currentframe=currentframe):
        return currentframe().f_back.f_globals

globals = Globals()
modules[__name__] = globals   # so other modules will use it
import globals                # make sure that worked

# simple test
globals.answer = 42
assert globals.answer is answer

This short bit of code demonstrates more than one dirty, dirty hack. inspect.currentframe() is used to allow us to manipulate the globals of the module containing the function it's called from, rather than the globals of its own module. We assign into sys.modules to replace the globals module object with our Globals class instance so that you only need import globals, not from globals import globals. Since it's a functional replacement for the built-in globals() function, we could stick it into the __builtins__ namespace so other modules would get it even without importing it, but even I have my limits!

Monday, March 18, 2013

IDTKAP #4: __debug__ and -O

Wayyyyy back in June 2012, I posted the first Stupid Python Trick, showing a way to abuse the assert statement to write debug code that is completely stripped when you run Python using the -O flag to enable optimization.

The -O flag is documented as removing assert instructions as though you never wrote them. But that's not all it does. It also sets a constant, __debug__, which is normally True, to False. The value of __debug__ is known at compile time, so Python can use it to completely discard conditional blocks predicated on __debug__, just as it does with assert statements, when running with the -O flag. And in fact, this is exactly what Python does!

The upshot is that you can write debug code that is stripped by Python's -O flag without abusing assert. This is easily demonstrated using Python's bytecode dissambler, dis.

from dis import dis

def debug_func():
    if __debug__:
       print "debugging"

def noop_func():

print "debug_func:"
print "noop_func:"

Save this as, then execute it with python -O You'll see that the disassembly of the two functions is identical aside from the line number offsets. It's just as if we never even wrote the if statement and its subordinate print statement! Literally zero performance impact to production code.

  4           0 LOAD_CONST               0 (None)
              3 RETURN_VALUE

  2           0 LOAD_CONST               0 (None)
              3 RETURN_VALUE

What's more, when we run this script without the -O flag, Python optimizes away the test. That is, Python knows __debug__ is true at compile time, and so it just compiles the code inside the if statement as if it weren't inside an if statement! Here's what the disassembly of debug_func looks like when __debug__ is True (i.e., no -O flag is used):

 3           0 LOAD_CONST               1 ('debugging')
             3 PRINT_ITEM
             4 PRINT_NEWLINE

 4           5 LOAD_CONST               0 (None)
             8 RETURN_VALUE

By comparison, here's what it would look like if we were using some other conditional (say, a global variable called DEBUG). You'll see that this is much more complicated, and if you time it, you'll find that executing the test at runtime actually adds significant overhead.

  2           0 LOAD_GLOBAL              0 (DEBUG)
              3 JUMP_IF_FALSE            9 (to 15)
              6 POP_TOP

  3           7 LOAD_CONST               1 ('debugging')
             10 PRINT_ITEM
             11 PRINT_NEWLINE
             12 JUMP_FORWARD             1 (to 16)
        >>   15 POP_TOP
        >>   16 LOAD_CONST               0 (None)
             19 RETURN_VALUE

So basically, Python will not only strip debugging code if it's conditionalized by testing __debug__, it will also slightly improve the performance of your debug code when running in debug mode compared to testing a runtime flag. And best of all, it does this magic using the same command line flag, -O, that strips assert statements!

But wait, there's more! If you use an else clause with your if __debug__ statement, Python is smart enough to strip whichever clause doesn't apply and "inline" the clause that does!

def get_run_mode():
    if __debug__:
        return "debug"
        return "production"

dis(get_run_mode) running without -O:
  3           0 LOAD_CONST               1 ('debug')
              3 RETURN_VALUE

dis(get_run_mode) running with -O:
  5           0 LOAD_CONST               1 ('production')
              3 RETURN_VALUE

Once again, for comparison, here's how the bytecode looks when the function is written to force runtime evaluation of the condition, by using a global variable DEBUG instead of __debug__:

 2           0 LOAD_GLOBAL              0 (DEBUG)
             3 JUMP_IF_FALSE            5 (to 11)
             6 POP_TOP

 3           7 LOAD_CONST               1 ('debug')
            10 RETURN_VALUE
       >>   11 POP_TOP

 5          12 LOAD_CONST               2 ('production')
            15 RETURN_VALUE
            16 LOAD_CONST               0 (None)
            19 RETURN_VALUE

So, is Python smart enough to optimize if not __debug__ in the same way? Sadly, no:

def not_debug_test():
    if not __debug__:
        print "production"


>>> dis(not_debug_test)
  2           0 LOAD_GLOBAL              0 (__debug__)
              3 JUMP_IF_TRUE             9 (to 15)
              6 POP_TOP

  3           7 LOAD_CONST               1 ('production')
             10 PRINT_ITEM
             11 PRINT_NEWLINE
             12 JUMP_FORWARD             1 (to 16)
        >>   15 POP_TOP
        >>   16 LOAD_CONST               0 (None)
             19 RETURN_VALUE

So if you want to write code that's run only in production, don't use if not __debug__. Write it like this instead:

    if __debug__:
         print "production"

What about conditional expressions, such as x = "yes" if __debug__ else "no"? Sadly, Python does not optimize these. Similarly, __debug__ and x and __debug__ or x are not optimized, though they could be.

So what did we learn?
  1. Use if __debug__ to write debug code (along with else if desired).
  2. Don't make up your own flag for this, as it will prevent Python from being clever.
  3. Don't  use if not __debug__ because this will also prevent Python from being clever.
  4. Prefer if statements to using __debug__ in logical expressions.
  5. Use assert to assert invariants, not to perform stupid Python tricks like I presented last June.
  6. Use the -O command line flag to tell Python when it's running in production. If you don't, you may be executing debugging code you don't want, with the potential performance degradation that implies.
Thanks to user Reddit user "brucifer" who posted this informative comment, and to user "Rainfly_X" who brought it to my attention.

By the way, in Python 3.x, True and False are also constants whose value is known at compile-time, and Python optimizes if True and if False similarly. In Python 2.x, the values of True and False can be changed at run time (seriously, try it if you don't believe me!), so this optimization isn't possible. None can't be changed in Python 2.x, but is only a true compile-time constant in Python 3.x, with the upshot that code under if None is also subject to being stripped out in Python 3.x but not in Python 2.x.

Sunday, December 9, 2012

Better Python APIs

Oz Katz has a good entry over at his blog, Eventual Consistency, about making better APIs for your Python libraries.

Wednesday, December 5, 2012

PP #1: The always-true condition

Python Pitfalls are quirks of the Python programming language that may trip up those new to the language or to programming. First in yet another series.

My experience with Python is that my code usually does what I expect it to—in fact, more so than with any other programming language I've used. But that doesn't mean it's impossible to write code that doesn't do what you expect! Far from it. I see code like this on Stack Overflow pretty much every day:

while True:
    command = raw_input("command> ").lower().strip()
    if command == "stop" or "exit":
    elif command == "help":
        print "quit, exit: exit the program"
        print "help:       display this help message" 
    elif command:
        print "Unrecognized command."

Can you guess the question? It's some variation of "No matter what command I enter, it always exits."

And the reason is always the same: a Boolean expression using or that is always truthy. (See PF #1: Truth and Consequences.) In this case, it's this line:

    if command == "quit" or "exit":

The programmer who wrote that line intended it to match either "quit" or "exit." Instead, it matches any command. The reason for this becomes obvious if we look at how Python evaluates the condition.

As described in PF #2: And and/or or, the value of a Boolean or operation is truthy if either operand is truthy. The operands of or here are:
  • command == "quit"
  • "exit"
The latter operand, being a string with nonzero length, is truthy in Python, and therefore the result of the or is always truthy. Therefore, this test matches everything and the suite under the if is always executed!

This error is particularly pernicious because it reads exactly how you would phrase it in English, and expresses the programmer's intent clearly. It's just that the meaning of or is more specific in Python than in English.

One way to write this test so that it works as expected is to explicitly tell Python to test command against both alternatives:

    if command == "quit" or command == "exit":

Of course, there's a shorter and clearer way to write this in Python:

    if command in ("quit", "exit"):

If you will be testing for a lot of alternatives and performance is important, use a set defined outside the loop. Testing for membership in an existing set is a lot faster than testing for membership in a tuple that's created new each time the test is executed.

exit_synonyms = set(["quit", "halt", "stop", 
                     "end", "whoa", "exit"])

while True:
    command = raw_input("command> ").lower().strip()
    if command in exit_synonyms:
    # and so on as above

Tuesday, December 4, 2012

PF #3: Non-short-circuiting Boolean operations

Third entry in the Python Foundations series, thoroughly exploring the basic building blocks of Python programs.

Wow, it's been months, hasn't it? Our last Python Foundations entry talked about how the Boolean operators and and or work in Python, including their short-circuting behavior. But what if you don't want short-circuiting? What if you want both operands of and or or to be fully evaluated even when the first operand's value is sufficient to determine the expression's truth value?

Here, we can take advantage of the fact that Booleans are a special kind of integer in Python (see IDKTAP #2: Booleans as fancy integers), and perform math on them.

If you create a table of the results of adding the various combinations of 0 and 1, which are the integer values of False and True in Python:

+  | 0  1
0  | 0  1
1  | 1  2

It bears a striking resemblance to the results of the Boolean or operator: the result is truthy (see PF #1: Truth and Consequences) if either of the operands is truthy.

Similarly, the multiplication operator is a dead ringer for Boolean and:

*  | 0  1
0  | 0  0
1  | 0  1

Zero times anything is zero, just as False and anything is False.

If you have studied digital electronics, you may recognize this as Boolean arithmetic: arithmetic using only the values 0 and 1, where and is in fact equivalent to multiplication and or is equivalent to addition. The result of addition is "clamped" to 1 (that is, the highest result an addition can be is 1), and the only valid operations are addition and multiplication. It may seem like a toy arithmetic, but it's the foundation of everything a computer does.

We can use this equivalence of logical and arithmetic operations to implement non-short-circuiting ("long-circuiting"?) and and or in Python. To support truthy and falsy values, which may not actually be integers, we convert the operands to proper Booleans first using the bool() constructor.

bool(x) + bool(y) + bool(z)    # x or y or z
bool(x) * bool(y) * bool(z)    # x and y and z

Here, x, y, and z are always fully evaluated. If one of them is, say, a function call, the function is always called. In the equivalent expression using Boolean operators, remember, the expression will short-circuit as soon as it can, and may not evaluate all arguments. We explored how to exploit this behavior for fun and profit in PF #2.

Conveniently, multiplication has precedence over (is performed before) addition, just as and has precedence over or, so an expression like the following works exactly as you would expect:

bool(a) * bool(b) + bool(x) * bool(y)    # a and b or x and y

In fact, this is an easy way to remember which Boolean operator has precedence: because and is equivalent to multiplication, it has precedence over or, which is equivalent to addition. This is a rule I had trouble remembering when I started programming.

So far, what we've discussed isn't really specific to Python; you can do the same non-short-circuiting Boolean math in virtually any language with only minor syntax changes. We can, however, use Python's any() and all() functions to help make our non-short-circuiting Boolean operators a little more readable. Both functions operate on a sequence.
  • any() - Returns True if any element is truthy (equivalent to Boolean or)
  • all() - Returns True if all elemens are truthy (equivalent to Boolean and)
Technically, any()and all() do in fact short-circuit: they will stop inspecting elements in the sequence as soon as they see a value that definitively determines the result. That is, all() returns False when it sees the first falsy element, because a single such element is enough to know that not all elements are truthy. And any() returns True when it sees the first truthy element, because that's enough to know that there is at least one such.

However, the short-circuiting behavior is apparent only with generators, which "lazily" calculates a sequence one element at a time. (There will surely be a Python Foundations entry on generators in the future!) If you pass a sequence (i.e., a list or a tuple) to any() or all(), all the elements of the sequence will be fully evaluated when the sequence is constructed, before any() or all() is even involved. The fact that any() and all() short-circuit is then merely a performance optimization. Useful, in other words, but not exploitable for side effects as with and and or.

And so we can use any() and all() as non-short-circuiting or and and, respectively. x, y, and z below may be expressions of arbitrary complexity, and will be evaluated fully.

any([x, y])      # x or y
all([x, y])      # x and y
any([x, y, z])   # x or y or z (any number of arguments)
all((x, y, z))   # you may use a tuple instead of a list

As for the last example, I personally prefer using the square brackets and passing a list since it stands out more and makes it clearer what is happening, but constructing a tuple is faster, a fact which may be useful in some circumstances.

any() and all() can be nested to implement compound non-short-circuiting Boolean operations:

any([all([a, b]), all([x, y]])    # a and b or x and y

However, this gets difficult to read pretty quickly (unless, perhaps, you have experience with Lisp, where all operations are prefix).. Fortunately, any() and all() always return True or False (not the deciding element from the sequence, which may be merely truthy or falsy), so you can easily combine their results with + and *.

all([a, b]) + all([x, y])         # a and b or x and y, as above

We're almost finished here; this has become rather more in-depth than I intended! But while were talking about any() and all(), I should mention that it is something of a shame that neither returns the value that made the decision (i.e., the first truthy value for any() or the first falsy value for all()). This keeps them from being used to actually find the first truthy or falsy value in a sequence, which is especially a pity when you're using them with a generator because that value, having been consumed within any()/all(), is forever lost.

You can write your own versions of these functions that do return the deciding value, but because they are implemented in Python rather than in C, they will be slower than the built-ins. Still, here are such implementations.

def all(iterable):
    for item in iterable: 
        if not item: return item
    return True

def any(iterable):
    for item in iterable: 
        if item: return item
    return False

Wednesday, July 11, 2012

IDKTAP #3: Backticks

Did you know that quoting an expression using backticks is an alias for repr() in Python 2.x? That is, it prints the "representation" of the result of the expression, which is generally what you'd have to paste into your Python source file to produce that same object.

print `"Think, it ain't illegal yet!"`

They took this out in Python 3.

Thursday, July 5, 2012

IDKTAP #2: Booleans as fancy integers

I Didn't Know That About Python uncovers surprising tidbits about our favorite programming language. 

In Python, a Boolean (type bool) is a subtype of integers. True is equal to 1 and False is equal to 0.

True  == 1   # True
False == 0   # True

The special methods called __str__ and __repr__ define how an object is converted to a string and how it is represented in an interactive context, respectively. So bool is basically implemented like this:

class bool(int):
    def __str__(self):
        return "True" if self else "False"
    __repr__ = __str__

True, False = bool(1), bool(0)

(There are some other details I'm glossing over, such as the fact that True and False are singletons and only ever have one instance. That is, bool(1) is always the same object. The example above does not have this behavior. For a fun exercise, try adding it.)

The surprising thing is that in Python 2, True and False are simply built-in identifiers assigned to two specific bool singletons. This means that they can be reassigned:

True = 3

This will very likely cause quite odd behavior in your programs, which is why Python 3 reserves True and False as actual language keywords and doesn't let you reassign them. Still, issubclass(bool, int) remains True.