It depends if you have weekly delivery from a distributor. If you do, then I wouldn't worry about it; offer everything that the distributor carries and for which you have mat corners. Why limit a customer's selection? Many are used to seeing a wide variety from other frame shops and may not like having such a narrow selection.
If you don't have weekly delivery and your supplier delivers via UPS, and you don't have enough volume to order often, then that is a different story.
If your are considering this because you don't have free timely delivery, then consider that there are close to 25 shades of whites and near whites and that these whites are the most difficult (to me) to select the correct one. If delivery is an issue and you have to limit your selection, you might be hard pressed to keep it under 75-100 and still satisfy most needs. And even then, what about the customer that wants something different (linens, suedes, leathers, etc)?
If you are going to offer only a limited selection, then your need to limit your self to a single type, and that would have to be rag only. I believe that you will be doing yourself and your customers a disservice by offering only 25.
Are there others out there that limit their mat selection this much?