Python Exception Class Hierarchy

Python Exception Handling – MemoryError

Continuing along through our in-depth Python Exception Handling series, today we’ll dig into Python’s MemoryError. As with all programming languages, Python includes a fallback exception for when the interpreter completely runs out of memory and must abort current execution. In these (hopefully rare) instances, Python raises a MemoryError, giving the script a chance to catch itself and break out of the current memory draught and recover. However, since Python uses the C language’s malloc() function for its memory management architecture, it is not guaranteed that all processes will be able to recover — in some cases, a MemoryError will result in an unrecoverable crash.

In today’s article we’ll examine the MemoryError in more detail, starting with where it sits in the larger Python Exception Class Hierarchy. We’ll also examine a simple code sample that illustrates how large memory allocations can occur, how the behavior of using massive objects differs depending on the particular computer architecture and Python version in use, and how MemoryErrors may be raised and handled. Let’s get into it!

The Technical Rundown

All Python exceptions inherit from the BaseException class, or extend from an inherited class therein. The full exception hierarchy of this error is:

Full Code Sample

Below is the full code sample we’ll be using in this article. It can be copied and pasted if you’d like to play with the code yourself and see how everything works.

When Should You Use It?

In most situations, a MemoryError indicates a major flaw in the current application. For example, an application that accepts files or user data input could run into MemoryErrors if the application has insufficient sanity checks in place. There are tons of scenarios where memory limits can be problematic, but for our code illustration we’ll just stick with a simple allocation in local memory using strings and arrays.

The most important factor in whether your own applications are likely to experience MemoryErrors is actually the computer architecture the executing system is running on. Or, even more specifically, the architecture your version of Python is using. If you’re using a 32-bit Python then the maximum memory allocation given to the Python process is exceptionally low. The specific maximum memory allocation limit varies and depends on your system, but it’s usually around 2 GB and certainly no more than 4 GB.

On the other hand, 64-bit Python versions are more or less limited only by your available system memory. In practical terms, a 64-bit Python interpreter is unlikely to experience memory issues, or if it does, the issue is a much bigger deal since it’s likely impacting the rest of the system anyway.

To test this stuff out we’ll be using the psutil to retrieve information about the active process, and specifically, the psutil.virtual_memory() method, which provides current memory usage stats when invoked. This information is printed within the print_memory_usage() function:

We’ll start by using the Python 3.6.4 32-bit version and appending MEGA_STR strings (which contain one million characters each) onto the end of an array until the process catches a MemoryError:

After we run out of memory and break out of the loop we output the memory usage of the array, along with overall memory usage stats. The result of running this function is the following output:

This shows our base memory usage at the top, and the array size we created at the bottom. As expected, after about 15 seconds of execution on my system we experienced a MemoryError. The alloc_max_str() function test creates a large string instead of an array, but we should see similar results:

Sure enough, executing alloc_max_str() results in a raised MemoryError after a relatively short execution period:

As mentioned, there is a huge difference between 32- and 64-bit versions of Python. If we swap to Python 3.6.4 64-bit and execute the same code no MemoryError has been thrown after 5+ minutes of iteration. As discussed, this is because 64-bit Python isn’t artificially limited, but can more or less use most of the available system memory!

Airbrake’s robust error monitoring software provides real-time error monitoring and automatic exception reporting for all your development projects. Airbrake’s state of the art web dashboard ensures you receive round-the-clock status updates on your application’s health and error rates. No matter what you’re working on, Airbrake easily integrates with all the most popular languages and frameworks. Plus, Airbrake makes it easy to customize exception parameters, while giving you complete control of the active error filter system, so you only gather the errors that matter most.

Check out Airbrake’s error monitoring software today and see for yourself why so many of the world’s best engineering teams use Airbrake to revolutionize their exception handling practices!