I don't leave early, but I understand why some do. And it happens in a lot of other places, too. I've witnessed it firsthand in Boston, New York, Phoenix, DC, Chicago, San Diego, San Francisco and a few I've probably forgot about.
A big part of the problem with college football is that games now are too damn long. With all the tv timeouts and the ridiculously long halftime (the NFL manages to do just fine with only 12 minutes), some people just get tired of being there. And before everyone blames the old people, I see way more 20-30 somethings taking off before it's over than the older crowd. This was especially true in my section yesterday.