Vegans who argue from ethical grounds usually use some version of Utilitarianism, which is compassion made universal and exclusive. Peter Singer, for example. David Hume had a reply: for ethics built around compassion, the question is, do we have ENOUGH compassion to motivate us to follow the ethical rule prescribed? He said "Sometimes we do, and sometimes we don't, and when we don't, why should we?"
Most philosophers have concluded that Utilitarianism is "too simple a theory". In my youth I tried following a version of it, motivated by my then religion, but found there is a high rate of burnout for aspiring saints.
In my understanding, we can avoid Hume's is-ought problem by building an ethic entirely of hypothetical oughts, of the form "If you want X then you ought to do Y", because Y is a necessary or efficient means of achieving X. You get a consequentialist ethic with an ultimate goal and a set of recommended derivative means to that end, and it becomes an objective question whether those means are effective, and whether some other set of means might be more effective. But the ultimate goal of the system remains a matter of choice. For example, I use the "social contract" approach. If you want to maintain peaceful and cooperative relations with your neighbors, don't kill, steal, lie, or break agreements, and more generally, follow the Golden Rule... or at least the Silver, Do not do unto others what you would not wish them to do to you.
In this approach, a "good person" is a desirable neighbor, desirable from the point of view of folks who seek to live in peace and raise families. IMHO that is the "default" ultimate goal of members of a social species, who survive by cooperating in groups. The great majority of people are going to value that goal highly, because it is the goal favored by natural selection.
So, "who is my neighbor?" All persons. What defines a person? IMHO it is the ability to learn language. I would also add to the "social contract" an insurance clause granting certain rights to FORMER persons, and an "adopted honorary person" clause, granting certain rights to animals and "pre-persons" who have been adopted by persons as members of their own family, taking responsibility for their upkeep, behavior, and training. The precise terms of the "social contract" will be culturally relative, subject to negotiation and change.
A compassionate person would clearly be a more desirable neighbor than a callous one. We have reason to require, as a matter of social custom and norms, a certain degree of compassion from our neighbors, as long as we are willing to practice the same degree of compassion ourselves.A modest degree would require we abstain from cruel sports, causing animals to suffer for our amusement. A greater degree would require humane treatment of farm animals. A still higher degree would require abstaining from hunting or farming the more intelligent of animals; eggs and fish still OK to eat, cetaceans, pigs, etc. off the menu. A still higher degree would require veganism. Higher still would be the Jains, who wear cloth over their mouth and nose to avoid accidentally inhaling gnats, and brush the paths in front of them with brooms to avoid stepping on ants. In diet, beyond veganism there is fruitarianism, where "fruit" is defined as "anything that eventually drops off the plant", including nuts, seeds, and "green fruit" like avocados, cucumbers, tomatoes. Someday there may be "ethical synthfoodists", who only eat lab-grown proteins, carbs, and fats, that were never part of any living thing. The degree of compassion required by social norms will be determined by cultural conversation.
There are a variety of arguments for reducing or eliminating meat in the diet, But I do not think the ethical argument is compelling by itself. Animals who will never learn to speak language do not count as "persons" unless we decide as a society to include them.