Showing posts with label Stupid Python Tricks. Show all posts
Showing posts with label Stupid Python Tricks. Show all posts

Thursday, December 16, 2021

SPT #5: The endless print

 The blog roars back to life; with a hiss and a clank, suddenly a new Stupid Python Trick appears!

This one's a quickie. Python`s print() function takes an end argument that lets you change what gets printed after the rest of things in that call have been printed. By default, this is a newline. It's common to set this to an empty string to suppress the newline:

print("There is no newline", end="")

But you can do the same by just including what you want to print as the end argument.

print(end="There is no newline")

By my count, that saves us four whole keystrokes: two quotes, a comma, and a space! Are we feeling stupid yet?

Tuesday, March 3, 2015

SPT #4: There is no try

"Stupid Python Tricks" explores sexy ways to whip Python features until they beg for mercy. Stupid Python Tricks may well be fifty different shades of gray. Reader discretion is advised.

I've recently been wishing that Python`s set type had a method like add() but which returned a Boolean indicating whether the item being added was already in the set. We could add this behavior to the add() method without fear of breaking much code, since most uses will ignore the return value, but I'd rather keep add() as fast as possible. So let's call this new method added() and have it return True if the item needed to be added, and False otherwise. You can derive a new class from set, so let's go ahead and do that:

class Set(set):
   def added(self, item):
       result = item not in self   # True if item needs to be added
       self.add(item)
       return result

Note our self.added() here is not conditional in any way; it doesn't need to be. set.add() is idempotent: adding the same item multiple times doesn't hurt anything, and it's actually faster to do the add() even if it's not necessary (since that stays in the fast C implementation of the set type) than to try to avoid the unnecessary add() with an if statement.

Our new method is convenient for deduplicating lists while retaining the order of their items:

pies = ["apple", "banana cream", "apple", "boysenberry",
        "apple", "pumpkin", "banana cream"]
seen =  Set()  # keeps track of pies we have already seen
pies[:] = (pie for pie in pies if seen.added(pie))
print(pies)

Result: ["apple", "banana cream", "boysenberry", "pumpkin"]

Be right back; I'm hungry for pie now.

OK. Our added() method works fine. There's nothing wrong with it. But doesn't it seem a little... inelegant... to have to store the result of the set membership test in a local variable, add the new item, and finally return the value we previously squirreled away? Why can't we simply return the result, and then do the add? Because the return ends the function's execution? Don't be silly; we won't let that stop us!

class Set(set):
   def added(self, item):
       try:     return item not in self
       finally: self.add(item)

Not only is this an unconventional use of try, it's also a wee bit slower than our earlier version. And that's why we call it "Stupid Python Tricks."

Thursday, February 19, 2015

SPT #3: The constant singleton

"Stupid Python Tricks" explores ways to abuse Python features for the hell of it. Stupid Python Tricks may well be perpendicular to maintainability and sanity. Abandon all hope, ye who enter here.

Classes can be convenient namespaces for corralling related objects. You don't even need to instantiate such a class; you can just use the class itself, since attribute access works the same for classes and instances. (If you need methods in the namespaces, it is better to instantiate the class, since otherwise you need to decorate each method with @classmethod. But for data, it works great, and saves the step of instantiation.)

One category of related objects that can be useful to put in a namespace are constants. You can even override __setattr__ to slap the programmer's hand should he (or she) try to change those tempting attributes-what-should-be-constants. But disappointment lurks ahead... declaring __setattr__ on a class won't actually override the behavior of setting the attribute on the class! See, for example, this:

class Const(object):
    answer = 42
    def __setattr__(self, name, value):
        raise TypeError("variables may vary, but constants never do")

The intent here is to prevent the class attribute answer (or any other attribute, for that matter) from being set or changed. But it doesn't work:

    Const.answer = 13    # no problem!

It works only if we instantiate the class:

    const = Const()
    const.answer = 42    # TypeError

One way to solve this is to just go ahead and instantiate the class right after we define it. We can even give the instance the same name as the class, a cheap way of making it a little more difficult to make additional instances (not that it would matter too much if someone did, aside from a little bit of memory overhead, and not like it's difficult to get the class's type even if it's not in a convenient namespace):

     Const = Const()

But now suppose we want to write a base class that provides the behavior of denying attribute changes, so that we can write subclasses that hold our constant definitions. Now we've gotta instantiate each of them because the desired behavior isn't fully contained in the class itself. And if we forget, the constants aren't constant anymore! Ugh. What we need to do is somehow override the attribute-setting behavior of the class itself. How?

Well, we know that to override attribute-setting behavior on an instance, we define __setattr__ on its class. We also know that every Python class is itself an object, which implies that a class is an instance of some other class. So what we want to do is define __setattr__ on the class's class. 

A class's class is called its metaclass. Just as every ordinary instance is created from a class, so every class is an instance created from a metaclass. In Python, the default metaclass is a class called type, so to write our own metaclass, we must inherit from type.

To sum up, we can override the attribute-setting behavior of our Const base class by writing a __getattr__ method on a metaclass, then using that metaclass to define our base class. Metaclasses are inherited, so the behavior of our base class will carry over to any classes derived from it. Which is exactly what we want!

While we're at it, we can also define what happens if someone tries to instantiate the class, by overriding the __call__ method. The __call__ method of type is what calls __new__ and then __init__ to construct the class. Since we're writing our own metaclass, we can make it do something different.

What should trying to instantiate a class that's just being used as a namespace do? Maybe raise TypeError? Or, since someone who tries to instantiate the class is probably trying to get access to its attributes, just return the class itself (i.e., make the class a singleton, i.e., make the class its own instance, i.e., make instantiating the class a no-op)? This being Stupid Python Tricks, let's do that! (Also, it's less typing than raising a proper exception, although of course writing a sentence explaining that ruins the economy.)

    class ConstMeta(type):
        def __setattr__(cls, name, value):
            raise TypeError("variables may vary, but constants never do")
        def __call__(cls):
            return cls

Now we can define a base class. (Python 2 and Python 3 have slightly different syntax for specifying the metaclass.)

    # Python 2 version
    class Const(object):
        __metaclass__ = ConstMeta

    # Python 3 version
    class Const(metaclass=ConstMeta):
         pass

Aaand now we can use that base class to define namespaces in which the attributes can't be modified:

    class nums(Const):
        ANSWER = 42
        FUN = 69
        UNLUCKY = 13
        PI = 3.1415927  # close enough

Now let's try changing all circles to hexagons:

     nums.PI = 3   # nope!

You might next reasonably ask: how does the class get defined in the first place if its metaclass disallows putting attributes on it? And the answer is that defining an attribute in the body of a class definition doesn't call __setattr__! Instead, the attributes are placed in a dictionary, and when the class definition is complete, that dictionary becomes the class's __dict__ attribute, where all its attributes are stored. Try it:

     print (nums.__dict__.items())

Now you might think that knowing about __dict__ will let you modify these read-only attributes anyway, even though we've written a metaclass to prevent it. Right?

    nums.__dict__["FUN"] -= 1     # Right????

But as it happens, the __dict__ attribute of a class is not actually a Python dict object, but rather a proxy object that does not allow item assignment. Foiled! You might also think you could just use object.__setattr__ ... but that doesn't work either! Foiled again!

All right already... type.__setattr__ does work:

    # Deep Thought had an off-by-one error
    type.__setattr__(nums, "ANSWER", 41)

Of course, you could also just go with the nuclear option:

    del ConstMeta.__setattr__    # kaboom

Which goes to show you... even when you think you've put up a wall around something in Python, you really haven't. Ever. That's why Python's unofficial motto is "I'll show you mine if you show me yours." Whoops, I mean, "We're all consenting adults here."

And now you know... the rest of the story.

Friday, November 8, 2013

SPT #2: One global to rule them all

"Stupid Python Tricks" explores ways to abuse Python features for fun and profit. Stupid Python Tricks may well be diametrically opposed to best practices. Don your peril-sensitive sunglasses before proceeding.

You know how when you have a lot of globals and it's a pain to modify them, because you have to declare the globals you want to modify in each function using the global statement? Yeah, me neither. After all, globals are a poor programming practice! And I never engage in those...

However, for educational porpoises, or at least didactic dolphins, here's a short hack that takes advantage of the fact that Python globals can be accessed, but not modified, without declaring them global. Basically, what we do is replace the built-in globals() function with an object that lets you read and set globals via attribute access. Instead of doing this:

def my_func(a, b, c):
    global ans
    ans = a + b + c

You can instead do:

import globals

def my_func(a, b, c):
    globals.ans = a + b + c

This works fine because you aren't re-binding any global names; instead, you are merely mutating an existing object. Or so Python believes... bwahahaha! While we didn't save any lines of code in this short example, imagine if we had dozens of functions that used globals. We'd save literally dozens of global statements! And we could dine on spaghetti code for weeks.

Calling this object, i.e. globals(), continues to return the modules's globals dictionary, just like before, thanks to a __call__() method on the class.

Without further ado, then, here's globals.py:

from inspect import currentframe
from sys import modules

class Globals(object):


    def __getattribute__(self, key, currentframe=currentframe):
        try:
            return currentframe().f_back.f_globals[key]
        except KeyError:
            pass # if we raise NameError here we get 2 err msgs
        raise NameError("global name '%s' is not defined" % key)
 

    def __setattr__(self, key, value, currentframe=currentframe):
        currentframe().f_back.f_globals[key] = value
 

    def __call__(self, currentframe=currentframe):
        return currentframe().f_back.f_globals

globals = Globals()
modules[__name__] = globals   # so other modules will use it
import globals                # make sure that worked


# simple test
globals.answer = 42
assert globals.answer is answer


This short bit of code demonstrates more than one dirty, dirty hack. inspect.currentframe() is used to allow us to manipulate the globals of the module containing the function it's called from, rather than the globals of its own module. We assign into sys.modules to replace the globals module object with our Globals class instance so that you only need import globals, not from globals import globals. Since it's a functional replacement for the built-in globals() function, we could stick it into the __builtins__ namespace so other modules would get it even without importing it, but even I have my limits!

Thursday, June 21, 2012

SPT #1: Automatically stripping debug code

"Stupid Python Tricks" explores ways to abuse Python features for fun and profit. This is the first in a series. Stupid Python Tricks are generally not best practices and may well be worst practices. Read at your own risk.

The standard Python interpreter ("CPython") has a command line option -O that triggers "optimization." Currently, the only real optimization performed is to ignore assert statements. And they're not simply ignored at runtime; they are never even compiled into the byte code executed by the Python virtual machine.

The value of this optimization is that you can sprinkle asserts liberally throughout your code to make sure it fails fast when something goes wrong, making it easier to debug. Yet, because all those statements are stripped out when Python is run in optimized mode, there's no performance penalty when the code is put into production.

Wouldn't it be great if you could strip all your debug code just as easily? Many of us write functions like this to let us easily turn off our debug messages:

def debug_print(*args):    
    if DEBUG:
        for arg in args: print arg,
        print

This can be optimized a bit to minimize overhead of the debug statements by checking the DEBUG flag only once and defining a do-nothing function when the flag isn't set:

def debug_print(*args):    
    for arg in args: print arg,
    print
if not DEBUG: debug_print = lambda *args: None

But there are still all those function calls to the dummy function being executed when running in non-debug mode. Plus, of course, you still need to define the DEBUG variable. So running in production requires both that you change that variable and put -O on the command line, doubling your chances of getting it wrong.

How can we abuse Python's optimization to actually strip out the calls to our debug_print function? Simple: by writing it as an assertion. To avoid raising an AssertionError, of course, debug_print must always return True.

def debug_print(*args):
    for arg in args: print arg,
    print
    return True

assert debug_print("Checkpoint 1")

Now we just need to run our script with -O and all those debug_print calls will be stripped automatically like we never even wrote them.

If you're using Python 3 (or Python 2.6 or 2.7 with from __future__ import print_function), print is already a function. Seems like a waste to define a new debug_print in that case. But we need the result to be True and print() returns None, which evaluates as False in a Boolean context. Well, you can just write one of the following, any of which is guaranteed to be True (the first only for functions that return None or another falsey value, the other two always) and prevent assert from sounding an alarm.

assert not print("Checkpoint 1")
assert print("Checkpoint 1") or True
assert [print("Checkpoint 1")]

Of these three, the last is what elevates this trick to the height of stupidity. Exploiting the fact that Python considers any non-empty container True, we simply make a list containing the return value from the function we called. The resulting code looks more like an unfamiliar bit of Python syntax than a dirty hack, but a dirty hack it is.

By the way, this Stupid Python Trick obviously also works for calls to loggers or any other function you want to call in debug mode.

Why this is a bad idea: It's very specific to current CPython behavior, which could change in the future, and may not have the desired effects with other Python implementations like IronPython and Jython. (Although it probably wouldn't hurt anything except possibly performance.) Furthermore, it's not really asserting anything about the program (the truth value is guaranteed to be True after all), but rather using the assert statement for its secondary effects, damaging Python's generally excellent readability.