I agree with your first thought, however disagree with the last sentence. I personally find games against USC, UCLA, Washington and Oregon MUCH more interesting than games versus Rutgers, Maryland, Indiana, NW, etc..
Regarding people nervous that Minnesota could be on the outside looking in...