They use Euler's formula to determine that, were it possible, it would produce 7 'faces' (regions on the paper, including the area outside the graph). The smallest region we can make (by number of edges involved) is a triangle (or other 3 sided region). A 2 edged region would mean both edges connect the same pair of vertices and a 1 edged region would be an edge starting and ending on the same vertex, neither of which happen in K5. So to make 7 faces we need at least 21 edges. Except really we need half that since each edge will be part of the boundary of exactly 2 faces (one on each side). So we need 10.5 edges, minimum, to construct the 7 faces in our hypothetical planar drawing, but K5 only has 10 edges. So our hypothetical planar embedding cannot exist.