No stress! My first reaction to this was OMG THIS IS AN INCREDIBLE RESOURCE!! We are 100% grateful for this sort of feedback! Here is a direct quote of what I said at 8:17am this morning when someone sent me the article and I scanned it:
this is amazing!!
[8:19 AM]The first obvious win is routing around providers that arent handling image inputs correctly. That should be straightforward
[8:19 AM]The effort param stuff...I thought we had addressed that, but will dig in. This is incredible feedback
[8:20 AM]We should hire this guy.
Yes, you hit it on the head. It's worth it, but you have to do your own evaluation and promotion.
Example I forgot to mention: `:nitro` ranking is not the fastest, I do a round robin sampling with representative payloads to find out the fastest providers and reorder my list on the fly.
I considered an automatic promotion path but decided against it I want to actually review the data myself first.
What's your model churn rate like? I was worried about customer experience by same day maybe same work getting a totally different model response (also caching is worse)
Yes, it is awfully inconsistent today. They run some tests (accuracy table buried on model page) but they are sparse and only capture a single moment in time. I would love to see OpenRouter take this more seriously.
reply