Sexual selection, including mate choice and intrasexual competition, is responsible for the evolution of some of the most elaborated and sexually dimorphic traits in animals. Although there is clear sexual dimorphism in the shape of human faces, it is not clear whether this is similarly due to mate choice, or whether mate choice affects only part of the facial shape difference between men and women. Here we explore these questions by investigating patterns of both facial shape and facial preference across a diverse set of human populations. We find evidence that human populations vary substantially and unexpectedly in both the range and pattern of facial sexually dimorphic traits. In particular, European and South American populations display larger levels of facial sexual dimorphism than African populations. Neither cross-cultural differences in facial shape variation, differences in body height between sexes, nor differing preferences for facial sex-typicality across countries, explain the observed patterns of facial dimorphism. Altogether, the association between morphological sex-typicality and attractiveness is moderate for women and weak (or absent) for men. Analysis that distinguishes between allometric and non-allometric components reveals that non-allometric sex-typicality is preferred in women’s faces but not in faces of men. This might be due to different regimes of ongoing sexual selection acting on men, such as stronger intersexual selection for body height and more intense intrasexual physical competition, compared with women.