In my doctoral dissertation research at the University of Maryland (advised by Gerald Borgia), I tested the cognitive performance hypothesis that males with better cognitive performance have higher reproductive success. Male display and female mate choice are highly complex processes in many species of animals and likely requires the need for complex and high-functioning brains. By presenting male satin bowerbirds (Ptilonorhynchus violaceus) problem solving tests, I have been able to show that males who are better problem solvers have higher reproductive success. To develop these problem solving tests, I took advantage of male aversion to red objects on their bowers and created an obstacle to removing the offending red things. You can read the publication that resulted from this work here: [pdf].
I also found that individuals who were on average better at a variety of cognitive tasks, including problem solving ability, were sexually preferred. Measures which integrate information from a large variety of cognitve tasks were strongly correlated with male mating success; in fact these were some of the best predictors of mating success in this species [pdf]. How do females choose these males with better cognitive ability? The answer to this question has turned out to be very complex, with individual display traits not being good indicators of cognitive ability. Rather, males who are able to most effectively produce the most complete and complex displays are more likely to have better overall cognitive ability [free-access link].
Inspired by this work, a Master’s student at the University of Denver I helped advise (Ross Minter) found that male stickleback fish with better inhibitory control are preferred by females [open access link]. Furthermore, male sticklebacks are better at inhibitory control than females, perhaps because of the increased demands of this cognitive ability in parenting, which males do alone, although these cognitive differences are largely explained by differences in neophobia [open access link].