If 2 teams are well matched either could win on a given day.I do. If you finished second in your conference, how can you possibly say you're the best there is? It's already been proven that you're not.
When you have 2 conferences that have most of the top teams it's not surprising that the second place team in a conference could in fact be the best team(eg Georgia and Alabama 2021).
Only allowing 1 team per conference would put SEC and the B1G teams at a disadvantage in a 4 team playoff. Obviously when you go to 12 it would be a non starter. So I'm not sure exactly what you are advocating.