In other words, the algorithm won't be biased because there is a buck to be made by removing the bias.
But the efficient market hypothesis is weird. It's kind of like Schrodinger's Cat. If you can find some specific instance where it's wrong and publish it then the market corrects itself, so if you observe a bias then it ceases to exist. So now on one hand, we know that biases can exist because people have discovered them in the past; it's just that those biases have already been published and are being taken into account. We can speculate that more exist but we don't know what they are. On the other hand, maybe now we've finally found them all (or the remaining ones are negligible) and the efficient market hypothesis becomes true.
But notice that the thing to do if you believe the efficient market hypothesis is wrong in a particular case is clear. Prove it, because you'll be able to make a lot of money by making the world more just.
Which is why we need transparency. Because the market participants can't find the bias if they don't have the information, and then the efficient market hypothesis will certainly be false.
I made no claims about the EMH. My only claim is that if bias like what you describe exists, it's something that statisticians rather than politicians need to identify and fix, and that the statisticians in the position to fix it have perfect incentives to do so.
Politicians have neither the ability nor incentive.
Similarly, if someone were claiming that we need political oversight of UI/UX choices, and that American web pages need more red/white/blue to improve conversion rates, I'd suggest that this is a job for web designers rather than politicians. I'd also say that if you just want red/white/blue designs for patriotic purposes, you should openly advocate for that and not talk about conversion rate.
https://en.wikipedia.org/wiki/Efficient-market_hypothesis
In other words, the algorithm won't be biased because there is a buck to be made by removing the bias.
But the efficient market hypothesis is weird. It's kind of like Schrodinger's Cat. If you can find some specific instance where it's wrong and publish it then the market corrects itself, so if you observe a bias then it ceases to exist. So now on one hand, we know that biases can exist because people have discovered them in the past; it's just that those biases have already been published and are being taken into account. We can speculate that more exist but we don't know what they are. On the other hand, maybe now we've finally found them all (or the remaining ones are negligible) and the efficient market hypothesis becomes true.
But notice that the thing to do if you believe the efficient market hypothesis is wrong in a particular case is clear. Prove it, because you'll be able to make a lot of money by making the world more just.
Which is why we need transparency. Because the market participants can't find the bias if they don't have the information, and then the efficient market hypothesis will certainly be false.