Let G be a group. The permutability graph of subgroups of G, denoted by Γ(G), is a graph with all the proper subgroups of G as its vertices and two distinct vertices in Γ(G) are adjacent if and only if the corresponding subgroups permute in G. In this paper, we classify the finite groups whose permutability graphs of subgroups are planar. In addition, we classify the finite groups whose permutability graphs of subgroups are one of outerplanar, path, cycle, unicyclic, claw-free or C4-free. Also, we investigate the planarity of permutability graphs of subgroups of infinite groups.