Python Recipes are simple bits of my code you can use freely in your own programs. First in a series. Yes, I'm gonna have a lot of different series of posts on this blog.
Here's a file-like class that allows access to lines in the file using Python indexing or slicing notation without reading the whole file into memory. It's sort of a list/file hybrid since it does have methods for incrementally reading from the file, too.
class linefile(object):
def __init__(self, path):
self.file = open(path)
self.offsets = []
def __enter__(self):
return self
def __exit__(self, *e):
self.file.close()
def readline(self):
self.offsets.append(int(self.file.tell()))
text = self.file.readline()
if not text:
self.offsets.pop()
return text
def readlines(self):
for text in self:
pass
return self
def next(self):
return self.readline()
def __iter__(self):
text = self.readline()
while text:
yield text
text = self.readline()
def __getitem__(self, index):
offset = self.offsets[index]
tell = self.file.tell()
if type(index) is slice:
if slice.step == 1: # step is 1, this is fastest
self.file.seek(offset[0])
text = [self.file.readline() for line in offset]
else:
text = []
for o in offset:
self.file.seek(o)
text.append(self.file.readline())
else:
self.file.seek(offset)
text = self.file.readline()
self.file.seek(tell)
return text
def __len__(self):
return len(self.offsets)
Unlike a regular file, a linefile is read-only and text-only. Internally, the instance stores the offsets of each line that has been read. The lines themselves are retrieved from the file on demand; the instance does not store any lines. Positive offsets index from the beginning of the file and negative offsets index from the last line read from the file, which may not be the end of the file. (Index -1, then, is the line before the current one.) You can use the readlines() method to read all the lines and thereby note the offset of each line of the file; this will allow the entire file to be accessed by index or slice as though it were a list. The linefile object is iterable and is also a context manager (i.e, it supports the with statement).
Tuesday, June 26, 2012
Sunday, June 24, 2012
The world's stupidest Mac backup software
So it appears that I'm running the world's stupidest backup software on my Mac.
Last night, my "media" hard disk (with all my photos and music) failed. No problem, I have a backup. I look on the backup drive and boom, there's no backup.
Apparently this software was trying to keep the backup in sync with the original disk when I deleted files. The original disk was gone, so it made the backup match... by deleting it.
Yeah, that was real helpful. Thanks, NTI Shadow.
All that stuff can probably (probably) be recovered, since nothing else has been written to the backup drive. And I'm pretty sure the original disk can also be recovered, for a price. Still, what a pain.
Last night, my "media" hard disk (with all my photos and music) failed. No problem, I have a backup. I look on the backup drive and boom, there's no backup.
Apparently this software was trying to keep the backup in sync with the original disk when I deleted files. The original disk was gone, so it made the backup match... by deleting it.
Yeah, that was real helpful. Thanks, NTI Shadow.
All that stuff can probably (probably) be recovered, since nothing else has been written to the backup drive. And I'm pretty sure the original disk can also be recovered, for a price. Still, what a pain.
Friday, June 22, 2012
PF #1: Truth and Consequences
Intended for those new to the language but not to programming, "Python Foundations" surveys the parts of Python that you need to
know about to take full advantage of Python's power. First in a series.
Many programming languages have somewhat loose concepts of truth and falsity. Objects can have a truth value even if they're not Booleans. In C, truth is canonically represented as -1, but any non-zero value is considered true in a Boolean context such as an if statement.
Python takes this a step further, considering empty containers (including strings) to be false. The constant None is also considered false. Other objects are generally considered true, although this can be overridden (we'll discuss how in a moment). This property is useful for making code like this more readable:
name = raw_input("What is your name? ")
while not name: # instead of while name == ""
name = raw_input("Seriously, what's your name? ")
Since truth values are a little flexible, Python programmers have adopted the terms truthy and falsy (or falsey) to refer to an object's implicit truth value when used in a Boolean context. (I hasten to add that I don't believe these terms were coined in the Python community.)
In other words, the list [1, 2, 3] is not literally equal to the constant True, but it is truthy because if you tested it with an if statement, that if statement's body would be executed.
Instances of classes are generally truthy unless they are derived from a class that has some other built-in behavior (for example, a list, which, remember, is truthy when it contains any items). Functions, classes, iterators/generators, and modules are also truthy.
You can override the implicit truth value of your own classes by defining either a __len__() or __nonzero__()* special method. If your class has a __len__() method, it is probably a container, and Python will treat its instances like one: false when its length is zero and true when its length is nonzero. The __nonzero__() method is more explicit and can indicate the instance's truth value even for non-container classes. If a class has both of these methods, __nonzero__() takes precedence.
Here is a list subclass that is always truthy, even when empty:
class truthylist(list):
def __nonzero__(self):
return True # always
When would you want to use such a list? Well, consider a situation in which you are reading records from a database or elements from an XML file and will return a list of them. If an error occurs, you have two choices: raise an exception or return an error code. In some scenarios, it's even convenient to just return an empty list on error, since then you can use the same code path to iterate over it whether there was an error or not. But then you don't know why you got the empty list: was it because there was an error, or because there was no data of the type you requested? The truthy list gives us a solution.
def getrecords(key):
try:
result = ... # get the records here
else:
return result if result else truthylist()
except Exception:
return []
Now, when we call this function, we can just check to see if the result is truthy. If it is, we successfully retrieved the records (even if there are none). At the same time, we retain the ability to iterate over the records without regard for what happened, if that's what we want to do.
records = getrecords("DNA")
for record in records: print record
if records = []: print "No records found",
if not records: print "due to error",
print
Admittedly, this verges on a Stupid Python Trick. The "empty container is falsy" convention is so engrained, other Python programmers will find truthylist more than a little odd.
In the next installment of Python Foundations, we'll look at Python's logical operators and how implicit truth values interact with them.
*In Python 3, the __nonzero__() special method was renamed __bool__() to better match the other type-coercion methods such as __str__() and __int__().
Many programming languages have somewhat loose concepts of truth and falsity. Objects can have a truth value even if they're not Booleans. In C, truth is canonically represented as -1, but any non-zero value is considered true in a Boolean context such as an if statement.
Python takes this a step further, considering empty containers (including strings) to be false. The constant None is also considered false. Other objects are generally considered true, although this can be overridden (we'll discuss how in a moment). This property is useful for making code like this more readable:
name = raw_input("What is your name? ")
while not name: # instead of while name == ""
name = raw_input("Seriously, what's your name? ")
Since truth values are a little flexible, Python programmers have adopted the terms truthy and falsy (or falsey) to refer to an object's implicit truth value when used in a Boolean context. (I hasten to add that I don't believe these terms were coined in the Python community.)
In other words, the list [1, 2, 3] is not literally equal to the constant True, but it is truthy because if you tested it with an if statement, that if statement's body would be executed.
Instances of classes are generally truthy unless they are derived from a class that has some other built-in behavior (for example, a list, which, remember, is truthy when it contains any items). Functions, classes, iterators/generators, and modules are also truthy.
You can override the implicit truth value of your own classes by defining either a __len__() or __nonzero__()* special method. If your class has a __len__() method, it is probably a container, and Python will treat its instances like one: false when its length is zero and true when its length is nonzero. The __nonzero__() method is more explicit and can indicate the instance's truth value even for non-container classes. If a class has both of these methods, __nonzero__() takes precedence.
Here is a list subclass that is always truthy, even when empty:
class truthylist(list):
def __nonzero__(self):
return True # always
When would you want to use such a list? Well, consider a situation in which you are reading records from a database or elements from an XML file and will return a list of them. If an error occurs, you have two choices: raise an exception or return an error code. In some scenarios, it's even convenient to just return an empty list on error, since then you can use the same code path to iterate over it whether there was an error or not. But then you don't know why you got the empty list: was it because there was an error, or because there was no data of the type you requested? The truthy list gives us a solution.
def getrecords(key):
try:
result = ... # get the records here
else:
return result if result else truthylist()
except Exception:
return []
Now, when we call this function, we can just check to see if the result is truthy. If it is, we successfully retrieved the records (even if there are none). At the same time, we retain the ability to iterate over the records without regard for what happened, if that's what we want to do.
records = getrecords("DNA")
for record in records: print record
if records = []: print "No records found",
if not records: print "due to error",
Admittedly, this verges on a Stupid Python Trick. The "empty container is falsy" convention is so engrained, other Python programmers will find truthylist more than a little odd.
In the next installment of Python Foundations, we'll look at Python's logical operators and how implicit truth values interact with them.
*In Python 3, the __nonzero__() special method was renamed __bool__() to better match the other type-coercion methods such as __str__() and __int__().
Thursday, June 21, 2012
Tweetery @Engyrus
You can find me on Twitter @Engyrus. I'll post there each time I post something here.
SPT #1: Automatically stripping debug code
"Stupid Python Tricks" explores ways to abuse Python features for fun and profit. This is the first in a series. Stupid Python Tricks are generally not best practices and may well be worst practices. Read at your own risk.
The standard Python interpreter ("CPython") has a command line option -O that triggers "optimization." Currently, the only real optimization performed is to ignore assert statements. And they're not simply ignored at runtime; they are never even compiled into the byte code executed by the Python virtual machine.
The value of this optimization is that you can sprinkle asserts liberally throughout your code to make sure it fails fast when something goes wrong, making it easier to debug. Yet, because all those statements are stripped out when Python is run in optimized mode, there's no performance penalty when the code is put into production.
Wouldn't it be great if you could strip all your debug code just as easily? Many of us write functions like this to let us easily turn off our debug messages:
def debug_print(*args):
if DEBUG:
for arg in args: print arg,
print
This can be optimized a bit to minimize overhead of the debug statements by checking the DEBUG flag only once and defining a do-nothing function when the flag isn't set:
def debug_print(*args):
for arg in args: print arg,
print
if not DEBUG: debug_print = lambda *args: None
But there are still all those function calls to the dummy function being executed when running in non-debug mode. Plus, of course, you still need to define the DEBUG variable. So running in production requires both that you change that variable and put -O on the command line, doubling your chances of getting it wrong.
How can we abuse Python's optimization to actually strip out the calls to our debug_print function? Simple: by writing it as an assertion. To avoid raising an AssertionError, of course, debug_print must always return True.
def debug_print(*args):
for arg in args: print arg,
print
return True
assert debug_print("Checkpoint 1")
Now we just need to run our script with -O and all those debug_print calls will be stripped automatically like we never even wrote them.
If you're using Python 3 (or Python 2.6 or 2.7 with from __future__ import print_function), print is already a function. Seems like a waste to define a new debug_print in that case. But we need the result to be True and print() returns None, which evaluates as False in a Boolean context. Well, you can just write one of the following, any of which is guaranteed to be True (the first only for functions that return None or another falsey value, the other two always) and prevent assert from sounding an alarm.
assert not print("Checkpoint 1")
assert print("Checkpoint 1") or True
assert [print("Checkpoint 1")]
Of these three, the last is what elevates this trick to the height of stupidity. Exploiting the fact that Python considers any non-empty container True, we simply make a list containing the return value from the function we called. The resulting code looks more like an unfamiliar bit of Python syntax than a dirty hack, but a dirty hack it is.
By the way, this Stupid Python Trick obviously also works for calls to loggers or any other function you want to call in debug mode.
Why this is a bad idea: It's very specific to current CPython behavior, which could change in the future, and may not have the desired effects with other Python implementations like IronPython and Jython. (Although it probably wouldn't hurt anything except possibly performance.) Furthermore, it's not really asserting anything about the program (the truth value is guaranteed to be True after all), but rather using the assert statement for its secondary effects, damaging Python's generally excellent readability.
The standard Python interpreter ("CPython") has a command line option -O that triggers "optimization." Currently, the only real optimization performed is to ignore assert statements. And they're not simply ignored at runtime; they are never even compiled into the byte code executed by the Python virtual machine.
The value of this optimization is that you can sprinkle asserts liberally throughout your code to make sure it fails fast when something goes wrong, making it easier to debug. Yet, because all those statements are stripped out when Python is run in optimized mode, there's no performance penalty when the code is put into production.
Wouldn't it be great if you could strip all your debug code just as easily? Many of us write functions like this to let us easily turn off our debug messages:
def debug_print(*args):
if DEBUG:
for arg in args: print arg,
This can be optimized a bit to minimize overhead of the debug statements by checking the DEBUG flag only once and defining a do-nothing function when the flag isn't set:
def debug_print(*args):
for arg in args: print arg,
if not DEBUG: debug_print = lambda *args: None
But there are still all those function calls to the dummy function being executed when running in non-debug mode. Plus, of course, you still need to define the DEBUG variable. So running in production requires both that you change that variable and put -O on the command line, doubling your chances of getting it wrong.
How can we abuse Python's optimization to actually strip out the calls to our debug_print function? Simple: by writing it as an assertion. To avoid raising an AssertionError, of course, debug_print must always return True.
def debug_print(*args):
for arg in args: print arg,
return True
assert debug_print("Checkpoint 1")
Now we just need to run our script with -O and all those debug_print calls will be stripped automatically like we never even wrote them.
If you're using Python 3 (or Python 2.6 or 2.7 with from __future__ import print_function), print is already a function. Seems like a waste to define a new debug_print in that case. But we need the result to be True and print() returns None, which evaluates as False in a Boolean context. Well, you can just write one of the following, any of which is guaranteed to be True (the first only for functions that return None or another falsey value, the other two always) and prevent assert from sounding an alarm.
assert not print("Checkpoint 1")
assert print("Checkpoint 1") or True
assert [print("Checkpoint 1")]
Of these three, the last is what elevates this trick to the height of stupidity. Exploiting the fact that Python considers any non-empty container True, we simply make a list containing the return value from the function we called. The resulting code looks more like an unfamiliar bit of Python syntax than a dirty hack, but a dirty hack it is.
By the way, this Stupid Python Trick obviously also works for calls to loggers or any other function you want to call in debug mode.
Why this is a bad idea: It's very specific to current CPython behavior, which could change in the future, and may not have the desired effects with other Python implementations like IronPython and Jython. (Although it probably wouldn't hurt anything except possibly performance.) Furthermore, it's not really asserting anything about the program (the truth value is guaranteed to be True after all), but rather using the assert statement for its secondary effects, damaging Python's generally excellent readability.
IDKTAP #1: Omitting "print" in the Python shell
"I Didn't Know That About Python" covers surprising little things I've learned about Python. Nothing earthshattering. First in a series.
Most Python hackers have spent a lot of time exploring in the Python interactive shell, and therefore know that Python obligingly prints the results of expressions for you when they return a value:
>>> 2+2
4
You might be forgiven for assuming Python is just printing the result of the last expression it evaluated. But in fact, it's printing the result of any and all expressions that you don't do anything else with.
>>> 2+2; 3+3
4
6
>>> for x in xrange(5): x
...
0
1
2
3
4
This behavior, though subtly different from what you might expect, lets you omit print much of the time in Python interactive mode, saving you precious fractions of a second each time. Because life is too short to spend it typing print!
Thanks to Joel Cornett for pointing this out in a comment on a StackOverflow answer.
Most Python hackers have spent a lot of time exploring in the Python interactive shell, and therefore know that Python obligingly prints the results of expressions for you when they return a value:
>>> 2+2
4
You might be forgiven for assuming Python is just printing the result of the last expression it evaluated. But in fact, it's printing the result of any and all expressions that you don't do anything else with.
>>> 2+2; 3+3
4
6
>>> for x in xrange(5): x
...
0
1
2
3
4
This behavior, though subtly different from what you might expect, lets you omit print much of the time in Python interactive mode, saving you precious fractions of a second each time. Because life is too short to spend it typing print!
Thanks to Joel Cornett for pointing this out in a comment on a StackOverflow answer.
Wednesday, June 20, 2012
Welcome to Engyrus
Engyrus is my one-man consulting sideline specializing in the Python programming language. I created this blog to share Python tips and tricks and information about the Python projects I'm working on, among other things. There'll probably be some Web stuff and some utilities made with AutoHotKey too.
I'm Jerry Kindall, technical writer by trade. At my day job, I use Python to build and validate our documentation. Python has put the fun back in programming for me. Coding in Python reminds me a lot of my Apple II hacking days, but with longer variable names and fewer GOTOs.
About the name: "Engyrus" is a disused scientific name for the genus Python (you know, snakes) dating back to the 1830s. As a collector of words, I really found this one lovely, so I appropriated it. (Much to my pleasure, the domain was available, too!)
I'm Jerry Kindall, technical writer by trade. At my day job, I use Python to build and validate our documentation. Python has put the fun back in programming for me. Coding in Python reminds me a lot of my Apple II hacking days, but with longer variable names and fewer GOTOs.
About the name: "Engyrus" is a disused scientific name for the genus Python (you know, snakes) dating back to the 1830s. As a collector of words, I really found this one lovely, so I appropriated it. (Much to my pleasure, the domain was available, too!)
Subscribe to:
Posts (Atom)