To me there are a few things to consider:
1. How long will you be out there? If you are just going to the game it's potentially one answer, if you are going to the game and maybe a little time around the stadium maybe a different answer, going to the game and days to explore "nearby" is yet a different answer.
2. Frequency of play. The Gophers play at Purdue every other year, at Rutgers once every 6 years minimum (next in 2025) and may never play at Fresno State again in my lifetime.
3. Frequency of visiting area. If you're going to be there for a while and can visit an adjoining area (Chicago, Philly/NYC, Vegas/etc.), is there one you've never been to, hate going to, etc.?
4. Anything drawing you to the area (Relatives, friends you haven't seen in a while, work location, etc.).
5. Friends. Do you have friends going to one? It's always fun going as a group.
I'm struggling with this year especially because I don't want to miss out on Fresno, but several friends are going to Rutgers and my wife wants to use that as an excuse to visit her cousin. Plus I don't have a lot of vacation this year, so I don't have the luxury of doing two (other than maybe a quick Purdue trip).
For me, the game experience is always at least interesting (usually fun, other than occasional blowouts). It's more the rest of the experience that makes the trip memorable. But I rarely just go to the game.