It's funny the things that come to you when you are working on rules for a specifically NON-D&D game. The KOHE system, which I've written about on Google+. I'm building a spell point system for it because characters do not have levels. Seeing how flexible characters are, I began to think about D20 style class systems again. (Near the end of the post is a link to a system I used to generate classes for a retroclone.) Here's what comes from rumination on custom classes.
I've read it somewhere that the only currency to manage classes in AD&D and older versions of D&D was Experience Points. Building on that premise, Crabaugh wrote a class creation system that allowed the creation of various classes whereby more power equaled more experience points per level.
In the 2e DMG, a similar system was developed, but the rules said upfront that the system couldn't be used to build the standard classes and that the rules were mainly designed for GMs to use for NPCs.
Then came Skills & Options (aka 2.5e).
I am still not a fan of it. However, the system did allow for a simpler way to do customized classes and multiclass characters. For example, even the PHB mentioned Priests of Specific Mythoi with different spell choices and alternatives to Turn Undead. I always created Priests of Specific Mythoi for my campaigns mostly for the alternatives to Turn Undead. At the time, I didn't always like the choices, but they made sense for Clerics of a God of War, etc.
Still, Priests of Specific Mythoi are still not different classes, just variations of the Priest class. What I want in making custom classes, I want something more than variations of a theme and a mixture of two existing classes. Since new class concepts are very difficult to come by, especially in a set of tables, at least let there be several forms of magic, unique abilities that do not currently belong to the standard five classes (thinking 2e) and a way to adapt spells into spell-like abilities. I realize that this is still variations of a theme, but at least it can add a sense of multiple cultures reflecting multiple ways of doing things.
For example, in 2e, the Shaman class had Spirit Magic. Invoking spirits required a ritual, but the spirit allowed for certain abilities and spells. It's not a spellbook and the spells are a mixture of traditional Mage and Priest spells.
Another example, outside 2e, would be a set of abilities centered around ghosts and shadows. I'm thinking more B/X and OD&D here. In the Basic Rules, shadows were explicitly defined as "not" undead. Going back further to Strategic Review #3, Ghosts were defined as "not true undead", either. (Yeah, I know that changed later, but work with me here.) An ability to temporarily grapple incorporeal creatures would be unique in any class-building system I've seen. As stated in an earlier post, this is also a natural segue to the use of psionics.
Some time ago, I wrote a Class_Generator that allows me to correctly generate the standard classes for Basic Fantasy. So, I wondered if I could adapt it to "add-in" these ideas. So far, adding in abilities that are equivalent to Turn Undead is pretty straightforward. Adding in the aforementioned Spirit Magic was pretty easy as well. However, trying to add in a Verb/Noun magic system is more complicated and I'm not quite there yet.
We'll see how progress goes.