Why the Chosen would participate in a system that oppresses the majority of the Gifted?
In my Domains of the Chosen series, the Chosen are the potent, ageless rulers of a sprawling Empire that clawed its way to power after surviving a massive magical cataclysm. The Gifted are those who develop the ability to wield magic, and in the Domains they are considered too dangerous to be allowed to develop their talents freely. The Gifted can choose to become Vassals who are sundered from the most destructive aspects of their magic, or to fight for their right to join the ranks of the Chosen as Gladiators.
The answer, in short, is to view the Gifted as weapons of mass destruction. States with nuclear weapons frown on other states trying to develop weapons of mass destruction, but tend be accepting of those that already have them. This even holds up with enemies: Kim Jong Un is dangerously unhinged and could be a much greater and more lasting threat than Isis, but because seems to have nuclear weapons we must practice detente with him instead of regime change.
The long answer is that the Chosen see other magic-wielders as a threat. The Reckoning began because the powerful Gifted of old began a massive war for dominance. The war was of such impressive scope that new races were created (Armodons and Minotaurs are among these and the created races suffer greater racial stigma in the Domains, because they are the product of magic) and the nations of old were mostly destroyed or became puppet states of powerful Gifted. That war went on and on, ending only when the forces that were wielded spun out of control, resulting in massive storms of Chaotic magic that scoured life from the entire planet and tainted the landscape.
The Chosen represent the Gifted who survived because they set aside their differences (temporarily, for survival) and made a pact with the people with the only safe haven around, Krass. Krass needed the Chosen for extra protection, and to help feed and shelter the massive influx of refugees that made their way to the city. The covenant they made was to the benefit of both groups; people hated the Gifted because of The Reckoning, but they needed them to survive. The Chosen needed shelter and could not survive without people (someone needs to grow food, make clothes, etc).
But The Chosen are not a monumental group. They are old enemies who often trust each other less than than anyone else. Any new Gifted who reaches the status of Chosen, migt be an ally for an enemy faction. Thus they use the Great Games as a way to control who has a shot.
It is also worth noting that by the time any Gladiator has a chance to join the the Chosen they have a large amount of popular support from years of public performance in the Arena, which counteracts the lingering fear of the Gifted for most citizens.
Finally a key point is that the Gift is not hereditary. The Chosen do not have a greater chance of having children with the Gift than anyone else. Thus any Chosen with children has a large chance of having ungifted kids; if they love those kids then they have an automatic desire to protect them from other Gifted. If the Gift were hereditary I expect things would play out very differently, with magical-aristocractic families ruling over ungifted peasant slaves.
In the end it is all about power. We can see the lengths that people go to keep and amass power throughout history, frequently killing their own family members and engaging in horrifying atrocities. In the Domains of the Chosen, magic is power.