Learning Python (2013)

Part VII. Exceptions and Tools

Chapter 34. Exception Coding Details

In the prior chapter we took a quick look at exception-related statements in action. Here, we’re going to dig a bit deeper—this chapter provides a more formal introduction to exception processing syntax in Python. Specifically, we’ll explore the details behind the try, raise, assert, andwith statements. As we’ll see, although these statements are mostly straightforward, they offer powerful tools for dealing with exceptional conditions in Python code.

NOTE

One procedural note up front: The exception story has changed in major ways in recent years. As of Python 2.5, the finally clause can appear in the same try statement as except and else clauses (previously, they could not be combined). Also, as of Python 3.0 and 2.6, the new with context manager statement has become official, and user-defined exceptions must now be coded as class instances, which should inherit from a built-in exception superclass. Moreover, 3.X sports slightly modified syntax for the raise statement and except clauses, some of which is available in 2.6 and 2.7.

I will focus on the state of exceptions in recent Python 2.X and 3.X releases in this edition, but because you are still very likely to see the original techniques in code for some time to come, along the way I’ll point out how things have evolved in this domain.

The try/except/else Statement

Now that we’ve seen the basics, it’s time for the details. In the following discussion, I’ll first present try/except/else and try/finally as separate statements, because in versions of Python prior to 2.5 they serve distinct roles and cannot be combined, and still are at least logically distinct today. Per the preceding note, in Python 2.5 and later except and finally can be mixed in a single try statement; we’ll see the implications of that merging after we’ve explored the two original forms in isolation.

Syntactically, the try is a compound, multipart statement. It starts with a try header line, followed by a block of (usually) indented statements; then one or more except clauses that identify exceptions to be caught and blocks to process them; and an optional else clause and block at the end. You associate the words try, except, and else by indenting them to the same level (i.e., lining them up vertically). For reference, here’s the general and most complete format in Python 3.X:

try:

    statements              # Run this main action first

except name1:

    statements              # Run if name1 is raised during try block

except (name2, name3):

    statements              # Run if any of these exceptions occur

except name4 as var:

    statements              # Run if name4 is raised, assign instance raised to var

except:

    statements              # Run for all other exceptions raised

else:

    statements              # Run if no exception was raised during try block

Semantically, the block under the try header in this statement represents the main action of the statement—the code you’re trying to run and wrap in error processing logic. The except clauses define handlers for exceptions raised during the try block, and the else clause (if coded) provides a handler to be run if no exceptions occur. The var entry here has to do with a feature of raise statements and exception classes, which we will discuss in full later in this chapter.

How try Statements Work

Operationally, here’s how try statements are run. When a try statement is entered, Python marks the current program context so it can return to it if an exception occurs. The statements nested under the try header are run first. What happens next depends on whether exceptions are raised while the try block’s statements are running, and whether they match those that the try is watching for:

§  If an exception occurs while the try block’s statements are running, and the exception matches one that the statement names, Python jumps back to the try and runs the statements under the first except clause that matches the raised exception, after assigning the raised exception object to the variable named after the as keyword in the clause (if present). After the except block runs, control then resumes below the entire try statement (unless the except block itself raises another exception, in which case the process is started anew from this point in the code).

§  If an exception occurs while the try block’s statements are running, but the exception does not match one that the statement names, the exception is propagated up to the next most recently entered try statement that matches the exception; if no such matching try statement can be found and the search reaches the top level of the process, Python kills the program and prints a default error message.

§  If an exception does not occur while the try block’s statements are running, Python runs the statements under the else line (if present), and control then resumes below the entire try statement.

In other words, except clauses catch any matching exceptions that happen while the try block is running, and the else clause runs only if no exceptions happen while the try block runs. Exceptions raised are matched to exceptions named in except clauses by superclass relationships we’ll explore in the next chapter, and the empty except clause (with no exception name) matches all (or all other) exceptions.

The except clauses are focused exception handlers—they catch exceptions that occur only within the statements in the associated try block. However, as the try block’s statements can call functions coded elsewhere in a program, the source of an exception may be outside the trystatement itself.

In fact, a try block might invoke arbitrarily large amounts of program code—including code that may have try statements of its own, which will be searched first when exceptions occur. That is, try statements can nest at runtime, a topic I’ll have more to say about in Chapter 36.

try Statement Clauses

When you write a try statement, a variety of clauses can appear after the try header. Table 34-1 summarizes all the possible forms—you must use at least one. We’ve already met some of these: as you know, except clauses catch exceptions, finally clauses run on the way out, and elseclauses run if no exceptions are encountered.

Formally, there may be any number of except clauses, but you can code else only if there is at least one except, and there can be only one else and one finally. Through Python 2.4, the finally clause must appear alone (without else or except); the try/finally is really a different statement. As of Python 2.5, however, a finally can appear in the same statement as except and else (more on the ordering rules later in this chapter when we meet the unified try statement).

Table 34-1. try statement clause forms

Clause form

Interpretation

except:

Catch all (or all other) exception types.

except name:

Catch a specific exception only.

except name as value:

Catch the listed exception and assign its instance.

except (name1name2):

Catch any of the listed exceptions.

except (name1name2) as value:

Catch any listed exception and assign its instance.

else:

Run if no exceptions are raised in the try block.

finally:

Always perform this block on exit.

We’ll explore the entries with the extra as value part in more detail when we meet the raise statement later in this chapter. They provide access to the objects that are raised as exceptions.

Catching any and all exceptions

The first and fourth entries in Table 34-1 are new here:

§  except clauses that list no exception name (except:) catch all exceptions not previously listed in the try statement.

§  except clauses that list a set of exceptions in parentheses (except (e1, e2, e3):) catch any of the listed exceptions.

Because Python looks for a match within a given try by inspecting the except clauses from top to bottom, the parenthesized version has the same effect as listing each exception in its own except clause, but you have to code the statement body associated with each only once. Here’s an example of multiple except clauses at work, which demonstrates just how specific your handlers can be:

try:

    action()

except NameError:

    ...

except IndexError:

    ...

except KeyError:

    ...

except (AttributeError, TypeError, SyntaxError):

    ...

else:

    ...

In this example, if an exception is raised while the call to the action function is running, Python returns to the try and searches for the first except that names the exception raised. It inspects the except clauses from top to bottom and left to right, and runs the statements under the first one that matches. If none match, the exception is propagated past this try. Note that the else runs only when no exception occurs in action—it does not run when an exception without a matching except is raised.

Catching all: The empty except and Exception

If you really want a general “catchall” clause, an empty except does the trick:

try:

    action()

except NameError:

    ...                   # Handle NameError

except IndexError:

    ...                   # Handle IndexError

except:

    ...                   # Handle all other exceptions

else:

    ...                   # Handle the no-exception case

The empty except clause is a sort of wildcard feature—because it catches everything, it allows your handlers to be as general or specific as you like. In some scenarios, this form may be more convenient than listing all possible exceptions in a try. For example, the following catches everything without listing anything:

try:

    action()

except:

    ...                   # Catch all possible exceptions

Empty excepts also raise some design issues, though. Although convenient, they may catch unexpected system exceptions unrelated to your code, and they may inadvertently intercept exceptions meant for another handler. For example, even system exit calls and Ctrl-C key combinations in Python trigger exceptions, and you usually want these to pass. Even worse, the empty except may also catch genuine programming mistakes for which you probably want to see an error message. We’ll revisit this as a gotcha at the end of this part of the book. For now, I’ll just say, “use with care.”

Python 3.X more strongly supports an alternative that solves one of these problems—catching an exception named Exception has almost the same effect as an empty except, but ignores exceptions related to system exits:

try:

    action()

except Exception:

    ...                   # Catch all possible exceptions, except exits

We’ll explore how this form works its voodoo formally in the next chapter when we study exception classes. In short, it works because exceptions match if they are a subclass of one named in an except clause, and Exception is a superclass of all the exceptions you should generally catch this way. This form has most of the same convenience of the empty except, without the risk of catching exit events. Though better, it also has some of the same dangers—especially with regard to masking programming errors.

NOTE

Version skew note: See also the raise statement ahead for more on the as portion of except clauses in try. Syntactically, Python 3.X requires the except E as V: handler clause form listed in Table 34-1 and used in this book, rather than the older except E, V: form. The latter form is still available (but not recommended) in Python 2.6 and 2.7: if used, it’s converted to the former.

The change was made to eliminate confusion regarding the dual role of commas in the older form. In this form, two alternate exceptions are properly coded as except (E1, E2):. Because 3.X supports the as form only, commas in a handler clause are always taken to mean a tuple, regardless of whether parentheses are used or not, and the values are interpreted as alternative exceptions to be caught.

As we’ll see ahead, though, this option does not modify the scoping rules in 2.X: even with the new as syntax, the variable V is still available after the except block in 2.X. In 3.X, V is not available later, and is in fact forcibly deleted.

The try else Clause

The purpose of the else clause is not always immediately obvious to Python newcomers. Without it, though, there is no direct way to tell (without setting and checking Boolean flags) whether the flow of control has proceeded past a try statement because no exception was raised, or because an exception occurred and was handled. Either way, we wind up after the try:

try:

    ...run code...

except IndexError:

    ...handle exception...

# Did we get here because the try failed or not?

Much like the way else clauses in loops make the exit cause more apparent, the else clause provides syntax in a try that makes what has happened obvious and unambiguous:

try:

    ...run code...

except IndexError:

    ...handle exception...

else:

    ...no exception occurred...

You can almost emulate an else clause by moving its code into the try block:

try:

    ...run code...

    ...no exception occurred...

except IndexError:

    ...handle exception...

This can lead to incorrect exception classifications, though. If the “no exception occurred” action triggers an IndexError, it will register as a failure of the try block and erroneously trigger the exception handler below the try (subtle, but true!). By using an explicit else clause instead, you make the logic more obvious and guarantee that except handlers will run only for real failures in the code you’re wrapping in a try, not for failures in the else no-exception case’s action.

Example: Default Behavior

Because the control flow through a program is easier to capture in Python than in English, let’s run some examples that further illustrate exception basics in the context of larger code samples in files.

I’ve mentioned that exceptions not caught by try statements percolate up to the top level of the Python process and run Python’s default exception-handling logic (i.e., Python terminates the running program and prints a standard error message). To illustrate, running the following module file, bad.py, generates a divide-by-zero exception:

def gobad(x, y):

    return x / y

def gosouth(x):

    print(gobad(x, 0))

gosouth(1)

Because the program ignores the exception it triggers, Python kills the program and prints a message:

% python bad.py

Traceback (most recent call last):

  File "bad.py", line 7, in <module>

    gosouth(1)

  File "bad.py", line 5, in gosouth

    print(gobad(x, 0))

  File "bad.py", line 2, in gobad

    return x / y

ZeroDivisionError: division by zero

I ran this in a shell window with Python 3.X. The message consists of a stack trace (“Traceback”) and the name of and details about the exception that was raised. The stack trace lists all lines active when the exception occurred, from oldest to newest. Note that because we’re not working at the interactive prompt, in this case the file and line number information is more useful. For example, here we can see that the bad divide happens at the last entry in the trace—line 2 of the file bad.py, a return statement.[68]

Because Python detects and reports all errors at runtime by raising exceptions, exceptions are intimately bound up with the ideas of error handling and debugging in general. If you’ve worked through this book’s examples, you’ve undoubtedly seen an exception or two along the way—even typos usually generate a SyntaxError or other exception when a file is imported or executed (that’s when the compiler is run). By default, you get a useful error display like the one just shown, which helps you track down the problem.

Often, this standard error message is all you need to resolve problems in your code. For more heavy-duty debugging jobs, you can catch exceptions with try statements, or use one of the debugging tools that I introduced in Chapter 3 and will summarize again in Chapter 36, such as the pdbstandard library module.

Example: Catching Built-in Exceptions

Python’s default exception handling is often exactly what you want—especially for code in a top-level script file, an error often should terminate your program immediately. For many programs, there is no need to be more specific about errors in your code.

Sometimes, though, you’ll want to catch errors and recover from them instead. If you don’t want your program terminated when Python raises an exception, simply catch it by wrapping the program logic in a try. This is an important capability for programs such as network servers, which must keep running persistently. For example, the following code, in the file kaboom.py, catches and recovers from the TypeError Python raises immediately when you try to concatenate a list and a string (remember, the + operator expects the same sequence type on both sides):

def kaboom(x, y):

    print(x + y)               # Trigger TypeError

try:

    kaboom([0, 1, 2], 'spam')

except TypeError:              # Catch and recover here

    print('Hello world!')

print('resuming here')         # Continue here if exception or not

When the exception occurs in the function kaboom, control jumps to the try statement’s except clause, which prints a message. Since an exception is “dead” after it’s been caught like this, the program continues executing below the try rather than being terminated by Python. In effect, the code processes and clears the error, and your script recovers:

% python kaboom.py

Hello world!

resuming here

Keep in mind that once you’ve caught an error, control resumes at the place where you caught it (i.e., after the try); there is no direct way to go back to the place where the exception occurred (here, in the function kaboom). In a sense, this makes exceptions more like simple jumps than function calls—there is no way to return to the code that triggered the error.


[68] As mentioned in the prior chapter, the text of error messages and stack traces tends to vary slightly over time and shells. Don’t be alarmed if your error messages don’t exactly match mine. When I ran this example in Python 3.3’s IDLE GUI, for instance, its error message text showed filenames with full absolute directory paths.

The try/finally Statement

The other flavor of the try statement is a specialization that has to do with finalization (a.k.a. termination) actions. If a finally clause is included in a try, Python will always run its block of statements “on the way out” of the try statement, whether an exception occurred while the tryblock was running or not. Its general form is:

try:

    statements                 # Run this action first

finally:

    statements                 # Always run this code on the way out

With this variant, Python begins by running the statement block associated with the try header line as usual. What happens next depends on whether an exception occurs during the try block:

§  If an exception does not occur while the try block is running, Python continues on to run the finally block, and then continues execution past the try statement.

§  If an exception does occur during the try block’s run, Python still comes back and runs the finally block, but it then propagates the exception up to a previously entered try or the top-level default handler; the program does not resume execution below the finally clause’s trystatement. That is, the finally block is run even if an exception is raised, but unlike an except, the finally does not terminate the exception—it continues being raised after the finally block runs.

The try/finally form is useful when you want to be completely sure that an action will happen after some code runs, regardless of the exception behavior of the program. In practice, it allows you to specify cleanup actions that always must occur, such as file closes and server disconnects where required.

Note that the finally clause cannot be used in the same try statement as except and else in Python 2.4 and earlier, so the try/finally is best thought of as a distinct statement form if you are using an older release. In Python 2.5, and later, however, finally can appear in the same statement as except and else, so today there is really a single try statement with many optional clauses (more about this shortly). Whichever version you use, though, the finally clause still serves the same purpose—to specify “cleanup” actions that must always be run, regardless of any exceptions.

NOTE

As we’ll also see later in this chapter, as of Python 2.6 and 3.0, the new with statement and its context managers provide an object-based way to do similar work for exit actions. Unlike finally, this new statement also supports entry actions, but it is limited in scope to objects that implement the context manager protocol it leverages.

Example: Coding Termination Actions with try/finally

We saw some simple try/finally examples in the prior chapter. Here’s a more realistic example that illustrates a typical role for this statement:

class MyError(Exception): pass

def stuff(file):

    raise MyError()

file = open('data', 'w')     # Open an output file (this can fail too)

try:

    stuff(file)              # Raises exception

finally:

    file.close()             # Always close file to flush output buffers

print('not reached')         # Continue here only if no exception

When the function in this code raises its exception, the control flow jumps back and runs the finally block to close the file. The exception is then propagated on to either another try or the default top-level handler, which prints the standard error message and shuts down the program. Hence, the statement after this try is never reached. If the function here did not raise an exception, the program would still execute the finally block to close the file, but it would then continue below the entire try statement.

In this specific case, we’ve wrapped a call to a file-processing function in a try with a finally clause to make sure that the file is always closed, and thus finalized, whether the function triggers an exception or not. This way, later code can be sure that the file’s output buffer’s content has been flushed from memory to disk. A similar code structure can guarantee that server connections are closed, and so on.

As we learned in Chapter 9, file objects are automatically closed on garbage collection in standard Python (CPython); this is especially useful for temporary files that we don’t assign to variables. However, it’s not always easy to predict when garbage collection will occur, especially in larger programs or alternative Python implementations with differing garbage collection policies (e.g., Jython, PyPy). The try statement makes file closes more explicit and predictable and pertains to a specific block of code. It ensures that the file will be closed on block exit, regardless of whether an exception occurs or not.

This particular example’s function isn’t all that useful (it just raises an exception), but wrapping calls in try/finally statements is a good way to ensure that your closing-time termination activities always run. Again, Python always runs the code in your finally blocks, regardless of whether an exception happens in the try block.[69]

Notice how the user-defined exception here is again defined with a class—as we’ll see more formally in the next chapter, exceptions today must all be class instances in 2.6, 3.0, and later releases in both lines.


[69] Unless Python crashes completely, of course. It does a good job of avoiding this, though, by checking all possible errors as a program runs. When a program does crash hard, it is usually due to a bug in linked-in C extension code, outside of Python’s scope.

Unified try/except/finally

In all versions of Python prior to release 2.5 (for its first 15 years of life, more or less), the try statement came in two flavors and was really two separate statements—we could either use a finally to ensure that cleanup code was always run, or write except blocks to catch and recover from specific exceptions and optionally specify an else clause to be run if no exceptions occurred.

That is, the finally clause could not be mixed with except and else. This was partly because of implementation issues, and partly because the meaning of mixing the two seemed obscure—catching and recovering from exceptions seemed a disjoint concept from performing cleanup actions.

In Python 2.5 and later, though, the two statements have merged. Today, we can mix finally, except, and else clauses in the same statement—in part because of similar utility in the Java language. That is, we can now write a statement of this form:

try:                               # Merged form

    main-action

except Exception1:

    handler1

except Exception2:                 # Catch exceptions

    handler2

...

else:                              # No-exception handler

    else-block

finally:                           # The finally encloses all else

    finally-block

The code in this statement’s main-action block is executed first, as usual. If that code raises an exception, all the except blocks are tested, one after another, looking for a match to the exception raised. If the exception raised is Exception1, the handler1 block is executed; if it’sException2, handler2 is run, and so on. If no exception is raised, the else-block is executed.

No matter what’s happened previously, the finally-block is executed once the main action block is complete and any raised exceptions have been handled. In fact, the code in the finally-block will be run even if there is an error in an exception handler or the else-block and a new exception is raised.

As always, the finally clause does not end the exception—if an exception is active when the finally-block is executed, it continues to be propagated after the finally-block runs, and control jumps somewhere else in the program (to another try, or to the default top-level handler). If no exception is active when the finally is run, control resumes after the entire try statement.

The net effect is that the finally is always run, regardless of whether:

§  An exception occurred in the main action and was handled.

§  An exception occurred in the main action and was not handled.

§  No exceptions occurred in the main action.

§  A new exception was triggered in one of the handlers.

Again, the finally serves to specify cleanup actions that must always occur on the way out of the try, regardless of what exceptions have been raised or handled.

Unified try Statement Syntax

When combined like this, the try statement must have either an except or a finally, and the order of its parts must be like this:

try -> except -> else -> finally

where the else and finally are optional, and there may be zero or more excepts, but there must be at least one except if an else appears. Really, the try statement consists of two parts: excepts with an optional else, and/or the finally.

In fact, it’s more accurate to describe the merged statement’s syntactic form this way (square brackets mean optional and star means zero-or-more here):

try:                               # Format 1

    statements

except [type [as value]]:          # [type [, value]] in Python 2.X

    statements

[except [type [as value]]:

    statements]*

[else:

    statements]

[finally:

    statements]

try:                               # Format 2

    statements

finally:

    statements

Because of these rules, the else can appear only if there is at least one except, and it’s always possible to mix except and finally, regardless of whether an else appears or not. It’s also possible to mix finally and else, but only if an except appears too (though the except can omit an exception name to catch everything and run a raise statement, described later, to reraise the current exception). If you violate any of these ordering rules, Python will raise a syntax error exception before your code runs.

Combining finally and except by Nesting

Prior to Python 2.5, it is actually possible to combine finally and except clauses in a try by syntactically nesting a try/except in the try block of a try/finally statement. We’ll explore this technique more fully in Chapter 36, but the basics may help clarify the meaning of a combined try—the following has the same effect as the new merged form shown at the start of this section:

try:                               # Nested equivalent to merged form

    try:

        main-action

    except Exception1:

        handler1

    except Exception2:

        handler2

    ...

    else:

        no-error

finally:

    cleanup

Again, the finally block is always run on the way out, regardless of what happened in the main action and regardless of any exception handlers run in the nested try (trace through the four cases listed previously to see how this works the same). Since an else always requires an except, this nested form even sports the same mixing constraints of the unified statement form outlined in the preceding section.

However, this nested equivalent seems more obscure to some, and requires more code than the new merged form—though just one four-character line plus extra indentation. Mixing finally into the same statement makes your code arguably easier to write and read, and is a generally preferred technique today.

Unified try Example

Here’s a demonstration of the merged try statement form at work. The following file, mergedexc.py, codes four common scenarios, with print statements that describe the meaning of each:

# File mergedexc.py (Python 3.X + 2.X)

sep = '-' * 45 + '\n'

print(sep + 'EXCEPTION RAISED AND CAUGHT')

try:

    x = 'spam'[99]

except IndexError:

    print('except run')

finally:

    print('finally run')

print('after run')

print(sep + 'NO EXCEPTION RAISED')

try:

    x = 'spam'[3]

except IndexError:

    print('except run')

finally:

    print('finally run')

print('after run')

print(sep + 'NO EXCEPTION RAISED, WITH ELSE')

try:

    x = 'spam'[3]

except IndexError:

    print('except run')

else:

    print('else run')

finally:

    print('finally run')

print('after run')

print(sep + 'EXCEPTION RAISED BUT NOT CAUGHT')

try:

    x = 1 / 0

except IndexError:

    print('except run')

finally:

    print('finally run')

print('after run')

When this code is run, the following output is produced in Python 3.3; in 2.X, its behavior and output are the same because the print calls each print a single item, though the error message text varies slightly. Trace through the code to see how exception handling produces the output of each of the four tests here:

c:\code> py −3 mergedexc.py

---------------------------------------------

EXCEPTION RAISED AND CAUGHT

except run

finally run

after run

---------------------------------------------

NO EXCEPTION RAISED

finally run

after run

---------------------------------------------

NO EXCEPTION RAISED, WITH ELSE

else run

finally run

after run

---------------------------------------------

EXCEPTION RAISED BUT NOT CAUGHT

finally run

Traceback (most recent call last):

  File "mergedexc.py", line 39, in <module>

    x = 1 / 0

ZeroDivisionError: division by zero

This example uses built-in operations in the main action to trigger exceptions (or not), and it relies on the fact that Python always checks for errors as code is running. The next section shows how to raise exceptions manually instead.

The raise Statement

To trigger exceptions explicitly, you can code raise statements. Their general form is simple—a raise statement consists of the word raise, optionally followed by the class to be raised or an instance of it:

raise instance               # Raise instance of class

raise class                  # Make and raise instance of class: makes an instance

raise                        # Reraise the most recent exception

As mentioned earlier, exceptions are always instances of classes in Python 2.6, 3.0, and later. Hence, the first raise form here is the most common—we provide an instance directly, either created before the raise or within the raise statement itself. If we pass a class instead, Python calls the class with no constructor arguments, to create an instance to be raised; this form is equivalent to adding parentheses after the class reference. The last form reraises the most recently raised exception; it’s commonly used in exception handlers to propagate exceptions that have been caught.

NOTE

Version skew note: Python 3.X no longer supports the raise ExcArgs form that is still available in Python 2.X. In 3.X, use the raise Exc(Args) instance-creation call form described in this book instead. The equivalent comma form in 2.X is legacy syntax provided for compatibility with the now-defunct string-based exceptions model, and it’s deprecated in 2.X. If used, it is converted to the 3.X call form.

As in earlier releases, a raise Exc form is also allowed to name a class—it is converted to raise Exc() in both versions, calling the class constructor with no arguments. Besides its defunct comma syntax, Python 2.X’s raise also allowed for either string or class exceptions, but the former is removed in 2.6, deprecated in 2.5, and not covered here except for a brief mention in the next chapter. Use classes for new exceptions today.

Raising Exceptions

To make this clearer, let’s look at some examples. With built-in exceptions, the following two forms are equivalent—both raise an instance of the exception class named, but the first creates the instance implicitly:

raise IndexError             # Class (instance created)

raise IndexError()           # Instance (created in statement)

We can also create the instance ahead of time—because the raise statement accepts any kind of object reference, the following two examples raise IndexError just like the prior two:

exc = IndexError()           # Create instance ahead of time

raise exc

excs = [IndexError, TypeError]

raise excs[0]

When an exception is raised, Python sends the raised instance along with the exception. If a try includes an except name as X: clause, the variable X will be assigned the instance provided in the raise:

try:

    ...

except IndexError as X:      # X assigned the raised instance object

    ...

The as is optional in a try handler (if it’s omitted, the instance is simply not assigned to a name), but including it allows the handler to access both data in the instance and methods in the exception class.

This model works the same for user-defined exceptions we code with classes—the following, for example, passes to the exception class constructor arguments that become available in the handler through the assigned instance:

class MyExc(Exception): pass

...

raise MyExc('spam')          # Exception class with constructor args

...

try:

    ...

except MyExc as X:           # Instance attributes available in handler

    print(X.args)

Because this encroaches on the next chapter’s topic, though, I’ll defer further details until then.

Regardless of how you name them, exceptions are always identified by class instance objects, and at most one is active at any given time. Once caught by an except clause anywhere in the program, an exception dies (i.e., won’t propagate to another try), unless it’s reraised by anotherraise statement or error.

Scopes and try except Variables

We’ll study exception objects in more detail in the next chapter. Now that we’ve seen the as variable in action, though, we can finally clarify the related version-specific scope issue summarized in Chapter 17. In Python 2.X, the exception reference variable name in an except clause is notlocalized to the clause itself, and is available after the associated block runs:

c:\code> py −2

>>> try:

...     1 / 0

... except Exception as X:               # 2.X does not localize X either way

...     print X

...

integer division or modulo by zero

>>> X

ZeroDivisionError('integer division or modulo by zero',)

This is true in 2.X whether we use the 3.X-style as or the earlier comma syntax:

>>> try:

...     1 / 0

... except Exception, X:

...     print X

...

integer division or modulo by zero

>>> X

ZeroDivisionError('integer division or modulo by zero',)

By contrast, Python 3.X localizes the exception reference name to the except block—the variable is not available after the block exits, much like a temporary loop variable in 3.X comprehension expressions (3.X also doesn’t accept 2.X’s except comma syntax, as noted earlier):

c:\code> py −3

>>> try:

...     1 / 0

... except Exception, X:

SyntaxError: invalid syntax

>>> try:

...     1 / 0

... except Exception as X:               # 3.X localizes 'as' names to except block

...     print(X)

...

division by zero

>>> X

NameError: name 'X' is not defined

Unlike compression loop variables, though, this variable is removed after the except block exits in 3.X. It does so because it would otherwise retain a reference to the runtime call stack, which would defer garbage collection and thus retain excess memory space. This removal occurs, though, even if you’re using the name elsewhere, and is more extreme policy than that used for comprehensions:

>>> X = 99

>>> try:

...     1 / 0

... except Exception as X:               # 3.X localizes _and_ removes on exit!

...     print(X)

...

division by zero

>>> X

NameError: name 'X' is not defined

>>> X = 99

>>> {X for X in 'spam'}                  # 2.X/3.X localizes only: not removed

{'s', 'a', 'p', 'm'}

>>> X

99

Because of this, you should generally use unique variable names in your try statement’s except clauses, even if they are localized by scope. If you do need to reference the exception instance after the try statement, simply assign it to another name that won’t be automatically removed:

>>> try:

...     1 / 0

... except Exception as X:               # Python removes this reference

...     print(X)

...     Saveit = X                       # Assign exc to retain exc if needed

...

division by zero

>>> X

NameError: name 'X' is not defined

>>> Saveit

ZeroDivisionError('division by zero',)

Propagating Exceptions with raise

The raise statement is a bit more feature-rich than we’ve seen thus far. For example, a raise that does not include an exception name or extra data value simply reraises the current exception. This form is typically used if you need to catch and handle an exception but don’t want the exception to die in your code:

>>> try:

...     raise IndexError('spam')         # Exceptions remember arguments

... except IndexError:

...     print('propagating')

...     raise                            # Reraise most recent exception

...

propagating

Traceback (most recent call last):

  File "<stdin>", line 2, in <module>

IndexError: spam

Running a raise this way reraises the exception and propagates it to a higher handler (or the default handler at the top, which stops the program with a standard error message). Notice how the argument we passed to the exception class shows up in the error messages; you’ll learn why this happens in the next chapter.

Python 3.X Exception Chaining: raise from

Exceptions can sometimes be triggered in response to other exceptions—both deliberately and by new program errors. To support full disclosure in such cases, Python 3.X (but not 2.X) also allows raise statements to have an optional from clause:

raise newexception from otherexception

When the from is used in an explicit raise request, the expression following from specifies another exception class or instance to attach to the __cause__ attribute of the new exception being raised. If the raised exception is not caught, Python prints both exceptions as part of the standard error message:

>>> try:

...     1 / 0

... except Exception as E:

...     raise TypeError('Bad') from E              # Explicitly chained exceptions

...

Traceback (most recent call last):

  File "<stdin>", line 2, in <module>

ZeroDivisionError: division by zero

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "<stdin>", line 4, in <module>

TypeError: Bad

When an exception is raised implicitly by a program error inside an exception handler, a similar procedure is followed automatically: the previous exception is attached to the new exception’s __context__ attribute and is again displayed in the standard error message if the exception goes uncaught:

>>> try:

...     1 / 0

... except:

...     badname                                    # Implicitly chained exceptions

...

Traceback (most recent call last):

  File "<stdin>", line 2, in <module>

ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "<stdin>", line 4, in <module>

NameError: name 'badname' is not defined

In both cases, because the original exception objects thus attached to new exception objects may themselves have attached causes, the causality chain can be arbitrary long, and is displayed in full in error messages. That is, error messages might give more than two exceptions. The net effect in both explicit and implicit contexts is to allow programmers to know all exceptions involved, when one exception triggers another:

>>> try:

...     try:

...         raise IndexError()

...     except Exception as E:

...         raise TypeError() from E

... except Exception as E:

...     raise SyntaxError() from E

...

Traceback (most recent call last):

  File "<stdin>", line 3, in <module>

IndexError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "<stdin>", line 5, in <module>

TypeError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "<stdin>", line 7, in <module>

SyntaxError: None

Code like the following would similarly display three exceptions, though implicitly triggered here:

try:

    try:

        1 / 0

    except:

        badname

except:

    open('nonesuch')

Like the unified try, chained exceptions are similar to utility in other languages (including Java and C#) though it’s not clear which languages were borrowers. In Python, it’s a still somewhat obscure extension, so we’ll defer to Python’s manuals for more details. In fact, Python 3.3 adds a way to stop exceptions from chaining, per the following note.

NOTE

Python 3.3 chained exception suppression: raise from None. Python 3.3 introduces a new syntax form—using None as the exception name in the raise from statement:

raise newexception from None

This allows the display of the chained exception context described in the preceding section to be disabled. This makes for less cluttered error messages in applications that convert between exception types while processing exception chains.

The assert Statement

As a somewhat special case for debugging purposes, Python includes the assert statement. It is mostly just syntactic shorthand for a common raise usage pattern, and an assert can be thought of as a conditional raise statement. A statement of the form:

assert test, data              # The data part is optional

works like the following code:

if __debug__:

    if not test:

        raise AssertionError(data)

In other words, if the test evaluates to false, Python raises an exception: the data item (if it’s provided) is used as the exception’s constructor argument. Like all exceptions, the AssertionError exception will kill your program if it’s not caught with a try, in which case the data item shows up as part of the standard error message.

As an added feature, assert statements may be removed from a compiled program’s byte code if the -O Python command-line flag is used, thereby optimizing the program. AssertionError is a built-in exception, and the __debug__ flag is a built-in name that is automatically set toTrue unless the -O flag is used. Use a command line like python –O main.py to run in optimized mode and disable (and hence skip) asserts.

Example: Trapping Constraints (but Not Errors!)

Assertions are typically used to verify program conditions during development. When displayed, their error message text automatically includes source code line information and the value listed in the assert statement. Consider the file asserter.py:

def f(x):

    assert x < 0, 'x must be negative'

    return x ** 2

% python

>>> import asserter

>>> asserter.f(1)

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File ".\asserter.py", line 2, in f

    assert x < 0, 'x must be negative'

AssertionError: x must be negative

It’s important to keep in mind that assert is mostly intended for trapping user-defined constraints, not for catching genuine programming errors. Because Python traps programming errors itself, there is usually no need to code assert to catch things like out-of-bounds indexes, type mismatches, and zero divides:

def reciprocal(x):

    assert x != 0              # A generally useless assert!

    return 1 / x               # Python checks for zero automatically

Such assert use cases are usually superfluous—because Python raises exceptions on errors automatically, you might as well let it do the job for you. As a rule, you don’t need to do error checking explicitly in your own code.

Of course, there are exceptions for most rules—as suggested earlier in the book, if a function has to perform long-running or unrecoverable actions before it reaches the place where an exception will be triggered, you still might want to test for errors. Even in this case, though, be careful not to make your tests overly specific or restrictive, or you will limit your code’s utility.

For another example of common assert usage, see the abstract superclass example in Chapter 29; there, we used assert to make calls to undefined methods fail with a message. It’s a rare but useful tool.

with/as Context Managers

Python 2.6 and 3.0 introduced a new exception-related statement—the with, and its optional as clause. This statement is designed to work with context manager objects, which support a new method-based protocol, similar in spirit to the way that iteration tools work with methods of the iteration protocol. This feature is also available as an option in 2.5, but must be enabled there with an import of this form:

from __future__ import with_statement

The with statement is also similar to a “using” statement in the C# language. Although a somewhat optional and advanced tools-oriented topic (and once a candidate for the next part of the book), context managers are lightweight and useful enough to group with the rest of the exception toolset here.

In short, the with/as statement is designed to be an alternative to a common try/finally usage idiom; like that statement, with is in large part intended for specifying termination-time or “cleanup” activities that must run regardless of whether an exception occurs during a processing step.

Unlike try/finally, the with statement is based upon an object protocol for specifying actions to be run around a block of code. This makes with less general, qualifies it as redundant in termination roles, and requires coding classes for objects that do not support its protocol. On the other hand, with also handles entry actions, can reduce code size, and allows code contexts to be managed with full OOP.

Python enhances some built-in tools with context managers, such as files that automatically close themselves and thread locks that automatically lock and unlock, but programmers can code context managers of their own with classes, too. Let’s take a brief look at the statement and its implicit protocol.

Basic Usage

The basic format of the with statement looks like this, with an optional part in square brackets here:

with expression [as variable]:

    with-block

The expression here is assumed to return an object that supports the context management protocol (more on this protocol in a moment). This object may also return a value that will be assigned to the name variable if the optional as clause is present.

Note that the variable is not necessarily assigned the result of the expression; the result of the expression is the object that supports the context protocol, and the variable may be assigned something else intended to be used inside the statement. The object returned by theexpression may then run startup code before the with-block is started, as well as termination code after the block is done, regardless of whether the block raised an exception or not.

Some built-in Python objects have been augmented to support the context management protocol, and so can be used with the with statement. For example, file objects (covered in Chapter 9) have a context manager that automatically closes the file after the with block regardless of whether an exception is raised, and regardless of if or when the version of Python running the code may close automatically:

with open(r'C:\misc\data') as myfile:

    for line in myfile:

        print(line)

        ...more code here...

Here, the call to open returns a simple file object that is assigned to the name myfile. We can use myfile with the usual file tools—in this case, the file iterator reads line by line in the for loop.

However, this object also supports the context management protocol used by the with statement. After this with statement has run, the context management machinery guarantees that the file object referenced by myfile is automatically closed, even if the for loop raised an exception while processing the file.

Although file objects may be automatically closed on garbage collection, it’s not always straightforward to know when that will occur, especially when using alternative Python implementations. The with statement in this role is an alternative that allows us to be sure that the close will occur after execution of a specific block of code.

As we saw earlier, we can achieve a similar effect with the more general and explicit try/finally statement, but it requires three more lines of administrative code in this case (four instead of just one):

myfile = open(r'C:\misc\data')

try:

    for line in myfile:

        print(line)

        ...more code here...

finally:

    myfile.close()

We won’t cover Python’s multithreading modules in this book (for more on that topic, see follow-up application-level texts such as Programming Python) but the lock and condition synchronization objects they define may also be used with the with statement, because they support the context management protocol—in this case adding both entry and exit actions around a block:

lock = threading.Lock()                        # After: import threading

with lock:

    # critical section of code

    ...access shared resources...

Here, the context management machinery guarantees that the lock is automatically acquired before the block is executed and released once the block is complete, regardless of exception outcomes.

As introduced in Chapter 5, the decimal module also uses context managers to simplify saving and restoring the current decimal context, which specifies the precision and rounding characteristics for calculations:

with decimal.localcontext() as ctx:            # After: import decimal

    ctx.prec = 2

    x = decimal.Decimal('1.00') / decimal.Decimal('3.00')

After this statement runs, the current thread’s context manager state is automatically restored to what it was before the statement began. To do the same with a try/finally, we would need to save the context before and restore it manually after the nested block.

The Context Management Protocol

Although some built-in types come with context managers, we can also write new ones of our own. To implement context managers, classes use special methods that fall into the operator overloading category to tap into the with statement. The interface expected of objects used in withstatements is somewhat complex, and most programmers only need to know how to use existing context managers. For tool builders who might want to write new application-specific context managers, though, let’s take a quick look at what’s involved.

Here’s how the with statement actually works:

1.    The expression is evaluated, resulting in an object known as a context manager that must have __enter__ and __exit__ methods.

2.    The context manager’s __enter__ method is called. The value it returns is assigned to the variable in the as clause if present, or simply discarded otherwise.

3.    The code in the nested with block is executed.

4.    If the with block raises an exception, the __exit__(typevaluetraceback) method is called with the exception details. These are the same three values returned by sys.exc_info, described in the Python manuals and later in this part of the book. If this method returns a false value, the exception is reraised; otherwise, the exception is terminated. The exception should normally be reraised so that it is propagated outside the with statement.

5.    If the with block does not raise an exception, the __exit__ method is still called, but its typevalue, and traceback arguments are all passed in as None.

Let’s look at a quick demo of the protocol in action. The following, file withas.py, defines a context manager object that traces the entry and exit of the with block in any with statement it is used for:

class TraceBlock:

    def message(self, arg):

        print('running ' + arg)

    def __enter__(self):

        print('starting with block')

        return self

    def __exit__(self, exc_type, exc_value, exc_tb):

        if exc_type is None:

            print('exited normally\n')

        else:

            print('raise an exception! ' + str(exc_type))

            return False    # Propagate

if __name__ == '__main__':

    with TraceBlock() as action:

        action.message('test 1')

        print('reached')

    with TraceBlock() as action:

        action.message('test 2')

        raise TypeError

        print('not reached')

Notice that this class’s __exit__ method returns False to propagate the exception; deleting the return statement would have the same effect, as the default None return value of functions is False by definition. Also notice that the __enter__ method returns self as the object to assign to the as variable; in other use cases, this might return a completely different object instead.

When run, the context manager traces the entry and exit of the with statement block with its __enter__ and __exit__ methods. Here’s the script in action being run under either Python 3.X or 2.X (as usual, mileage varies slightly in some 2.X displays, and this runs on 2.6, 2.7, and 2.5 if enabled):

c:\code> py −3 withas.py

starting with block

running test 1

reached

exited normally

starting with block

running test 2

raise an exception! <class 'TypeError'>

Traceback (most recent call last):

  File "withas.py", line 22, in <module>

    raise TypeError

TypeError

Context managers can also utilize OOP state information and inheritance, but are somewhat advanced devices for tool builders, so we’ll skip additional details here (see Python’s standard manuals for the full story—for example, there’s a new contextlib standard module that provides additional tools for coding context managers). For simpler purposes, the try/finally statement provides sufficient support for termination-time activities without coding classes.

Multiple Context Managers in 3.1, 2.7, and Later

Python 3.1 introduced a with extension that eventually appeared in Python 2.7 as well. In these and later Pythons, the with statement may also specify multiple (sometimes referred to as “nested”) context managers with new comma syntax. In the following, for example, both files’ exit actions are automatically run when the statement block exits, regardless of exception outcomes:

with open('data') as fin, open('res', 'w') as fout:

    for line in fin:

        if 'some key' in line:

            fout.write(line)

Any number of context manager items may be listed, and multiple items work the same as nested with statements. In Pythons that support this, the following code:

with A() as a, B() as b:

    ...statements...

is equivalent to the following, which also works in 3.0 and 2.6:

with A() as a:

    with B() as b:

        ...statements...

Python 3.1’s release notes have additional details, but here’s a quick look at the extension in action—to implement a parallel lines scan of two files, the following uses with to open two files at once and zip together their lines, without having to manually close when finished (assuming manual closes are required):

>>> with open('script1.py') as f1, open('script2.py') as f2:

...     for pair in zip(f1, f2):

...         print(pair)

...

('# A first Python script\n', 'import sys\n')

('import sys                  # Load a library module\n', 'print(sys.path)\n')

('print(sys.platform)\n', 'x = 2\n')

('print(2 ** 32)              # Raise 2 to a power\n', 'print(x ** 32)\n')

You might use this coding structure to do a line-by-line comparison of two text files, for example—replace the print with an if for a simple file comparison operation, and use enumerate for line numbers:

with open('script1.py') as f1, open('script2.py') as f2:

    for (linenum, (line1, line2)) in enumerate(zip(f1, f2)):

        if line1 != line2:

            print('%s\n%r\n%r' % (linenum, line1, line2))

Still, the preceding technique isn’t all that useful in CPython, because input file objects don’t require a buffer flush, and file objects are closed automatically when reclaimed if still open. In CPython, the files would be reclaimed immediately if the parallel scan were coded the following simpler way:

for pair in zip(open('script1.py'), open('script2.py')):   # Same effect, auto close

    print(pair)

On the other hand, alternative implementations such as PyPy and Jython may require more direct closure inside loops to avoid taxing system resources, due to differing garbage collectors. Even more usefully, the following automatically closes the output file on statement exit, to ensure that any buffered text is transferred to disk immediately:

>>> with open('script2.py') as fin, open('upper.py', 'w') as fout:

...     for line in fin:

...         fout.write(line.upper())

...

>>> print(open('upper.py').read())

IMPORT SYS

PRINT(SYS.PATH)

X = 2

PRINT(X ** 32)

In both cases, we can instead simply open files in individual statements and close after processing if needed, and in some scripts we probably should—there’s no point in using statements that catch an exception if it means your program is out of business anyhow!

fin  = open('script2.py')

fout = open('upper.py', 'w')

for line in fin:                        # Same effect as preceding code, auto close

    fout.write(line.upper())

However, in cases where programs must continue after exceptions, the with forms also implicitly catch exceptions, and thereby also avoid a try/finally in cases where close is required. The equivalent without with is more explicit, but requires noticeably more code:

fin  = open('script2.py')

fout = open('upper.py', 'w')

try:                                    # Same effect but explicit close on error

    for line in fin:

        fout.write(line.upper())

finally:

    fin.close()

    fout.close()

On the other hand, the try/finally is a single tool that applies to all finalization cases, whereas the with adds a second tool that can be more concise, but applies to only certain objects types, and doubles the required knowledge base of programmers. As usual, you’ll have to weigh the tradeoffs for yourself.

Chapter Summary

In this chapter, we took a more detailed look at exception processing by exploring the statements related to exceptions in Python: try to catch them, raise to trigger them, assert to raise them conditionally, and with to wrap code blocks in context managers that specify entry and exit actions.

Up to this point, exceptions probably seem like a fairly lightweight tool, and in fact, they are; the only substantially complex thing about them is how they are identified. The next chapter continues our exploration by describing how to implement exception objects of your own; as you’ll see, classes allow you to code new exceptions specific to your programs. Before we move ahead, though, let’s work through the following short quiz on the basics covered here.

Test Your Knowledge: Quiz

1.    What is the try statement for?

2.    What are the two common variations of the try statement?

3.    What is the raise statement for?

4.    What is the assert statement designed to do, and what other statement is it like?

5.    What is the with/as statement designed to do, and what other statement is it like?

Test Your Knowledge: Answers

1.    The try statement catches and recovers from exceptions—it specifies a block of code to run, and one or more handlers for exceptions that may be raised during the block’s execution.

2.    The two common variations on the try statement are try/except/else (for catching exceptions) and try/finally (for specifying cleanup actions that must occur whether an exception is raised or not). Through Python 2.4, these were separate statements that could be combined by syntactic nesting; in 2.5 and later, except and finally blocks may be mixed in the same statement, so the two statement forms are merged. In the merged form, the finally is still run on the way out of the try, regardless of what exceptions may have been raised or handled. In fact, the merged form is equivalent to nesting a try/except/else in a try/finally, and the two still have logically distinct roles.

3.    The raise statement raises (triggers) an exception. Python raises built-in exceptions on errors internally, but your scripts can trigger built-in or user-defined exceptions with raise, too.

4.    The assert statement raises an AssertionError exception if a condition is false. It works like a conditional raise statement wrapped up in an if statement, and can be disabled with a –O switch.

5.    The with/as statement is designed to automate startup and termination activities that must occur around a block of code. It is roughly like a try/finally statement in that its exit actions run whether an exception occurred or not, but it allows a richer object-based protocol for specifying entry and exit actions, and may reduce code size. Still, it’s not quite as general, as it applies only to objects that support its protocol; try handles many more use cases.