The Fermat problem is a crucial topological issue corresponding to fractal networks. In this paper, we discuss the average Fermat distance (AFD) of the Vicsek polygon network and analyze structural properties. We construct the Vicsek polygon network based on Vicsek fractal in an iterative way. Given the structure of network, we present an elaborate analysis of the Fermat point under various situations. The special network structure allows a way to calculate the AFD based on average geodesic distance (AGD). Moreover, we introduce the Vicsek polygon fractal and calculate its AGD and AFD. Its relationship with the network enables us to deduce the above two indices of the network directly. The results show that both in network and fractal, the ratio of AFD and AGD tends to 3/2, which demonstrates that both of them can serve as indicators of small-world property of complex networks. In fact, in Vicsek polygon network, the AFD grows linearly with network order, implying that our evolving network does not possess the small-world property.