Dark theme

Geography Programming Courses

Using a profiler

One type of bug we haven't really discussed is runtime bugs which are associated with the environment outside the code; for example, the computer's processing power. Code runs with more or less efficiency, depending on how it is written, but very inefficient (or efficient but demanding) code can exceed the computing capacity it is running on. This is often in terms of the processing power available, or, more usually, the memory. When computers run out of memory on chips (RAM) they start writing to so-called Virtual Memory, which is usually the harddrive of the machine. Harddrives run very slowly in comparison with most over things on a computer, so you can see a dramatic slow down in a program. Occasionally you can exceed available resources altogether and either the program and/or the machine will crash. Whether machines crash or not, we certainly don't want code running longer than needs be: indeed, some code may take so long to run, it isn't practically worth running (a flood model that takes two days to predict floods based on current rainfall is useless for emergency preparation).

We can do some back of the envelope calculations to determine whether a program is likely to run or not. For example, if we say that a function takes a millisecond (0.001s), which would be fastish in Python, and we've got to run the function in an nested set of loops that loop 100*100 times, we can estimate the program will take ~10s to run. If this is, instead, processing 10000*10000 data points, it will take 100,000s or ~28 hours. One way to analyse algorithms is to guess at which parts are most intensive (usually loop structures and network reads) and estimate the order of magnitude of time required based on this.

We can equally make some estimates of memory needs. Python makes this slightly harder, by not having rules about how much memory is used by different variable types, but if we assume ~4 bytes (32 bits, i.e. single ones or zeros) for an integer number, we can see that a 100*100 2D array of ints will take up 40,000 bytes or ~39KiB (1KiB of memory == 1024 bytes &ndash we'll come back to what this is in a second*). However, 10000*10000 ints would require 400,000,000 bytes or ~0.3GiB. 100,000*100,000 ints would require ~37GiB, which is more than most machines can handle. The caveat to this is that Python changes the allocation based on the operating system and hardware. While you can find out the size allocation to something for your system using, for example:
import sys
sys.getsizeof("A")
sys.getsizeof(2)
sys.getsizeof(2.0)
you need to be aware this might not be true on someone else's machine (A nice discussion, including various samples, can be found on Stackoverflow). If Python runs out of memory, a MemoryError will be raised (the program will break and messages printed to the command prompt).

*Note that although memory is often quoted in kilobytes (Kb), megabytes (Mb), and gigabytes (Gb), almost always what is actually meant is kibibytes (KiB), mebibytes (MiB), and gibibytes (GiB). Because memory is binary, we count it in binary, and so rather than increasing by 1000s, we increase by 1024s. "kilo", "mega", and "giga" refer to decimal sequences increasing by 1000, but are often informally used for "kibi", "mebi", and "gibi" by manufacturers because people are more familiar with the terms. As programmers, the difference between a GiB and a Gb is quite large (73,741,824 bytes more in a GiB), so it is good to know the difference. There's more on this on Wikipedia.

So, this estimation is helpful, but a bit hit-and-miss in Python. What can we do to analyse a program we already have running, but which is running much slower than we expect? What we need is a "profiler".