I think it's understandable that fans stop supporting teams, but it does have an impact on the overall strength of the program. For instance, I understand that my devotion to Gopher football is probably a little illogical. I spend quite a bit of time and money following and supporting the program. I couldn't really make a super strong logical argument as to why I would support the program pretty much under any circumstances, but the fact is that I do. Now, the truth is that the U of MN has few fans like most of us (in comparison to a big time program). It is one of the criteria that makes for a big time program.
Now, as to how that can affect the program's wins and losses. Well, look at Oklahoma. They were able to get Bob Stoops to leave Florida and go to Oklahoma (something we had tried to do and failed at a couple years earlier). Part of the reason is because Oklahoma was a big time program despite not even being .500 for 4 seasons prior to his arrival. They were also able to pay Stoops a higher salary.
Part of the reason why we haven't been able to get our pick of the litter in terms of coaching candidates is because we don't have as big of a die hard fan base (who donates money and is a gigantic cash draw to the University).
Another example would be Gopher hockey.
The amount of borderline illogically devoted fans to a program in comparison to other programs in that sport plays a role on the overall success of the program.
I really think this is a chicken-egg argument. I don't think that Bob Stoops went to Oklahoma because of their fan base, per se. I think their fan base is bigger because it's a big-time program. In other words, I don't think it's their fan base that makes them a big-time program. It's their past and more recent success that makes them a big-time program, and the fans simply follow then in greater numbers.
Clearly the big-money boosters at big programs do make a difference. But I think that's a separate issue from "fan base."