It appears from the Gemara that there were different historical periods. At first, when the yavam and yevama had pure intentions, and performed yibum with the intention of fulfilling the mitzva, yibum was preferable. In later years, when people could no longer be relied upon to have the proper intent, halitza became the preferred option.
This most basic question is discussed on our daf, where we find a disagreement between Abba Shaul and the Hakhamim. Abba Shaul argues that a person who performs yibum because he finds the yevama beautiful, because he wants to be married to her or for some other reason (the Rivan suggests that this refers to monetary gain), it is as though he engaged in a forbidden sexual act. The Hakhamim point to the passage (Devarim 25:5) that commands the yavam to “come upon” her, and see no qualifications in the performance of the mitzva.
It is not clear whether Abba Shaul actually believes that a person who performs yibum with intentions that are not totally pure does not fulfill the mitzva. The Nimukei Yosef, for example, argues that Abba Shaul accepts the fact that the act of yibum will work, even if it is done with the wrong intentions, and proper intent is only a Rabbinic requirement. From the Ramban, however, it appears that according to Abba Shaul someone who performs yibum with the wrong intent will not be married to her, since the commandment was not properly fulfilled.
There is no clear conclusion in the Gemara with regard to this disagreement. Although it appears that our Gemara leans towards the position of the Hakhamim, other Gemarot appear to accept Abba Shaul’s position. During the Ge’onic period different communities followed different rulings, with Neharde’a following Abba Shaul and Sura accepting the position of the Hakhamim. In modern times rulings differ between Sefardic and Ashkenazic communities. Ashkenazim usually recommend halitza, while Sefardim encourage yibum.