My problems with hexes is that even a guy who dumps exp in 27 hexes is going to have problems affecting fiends, chumps at zot5 etc.
Some comments on this:
1) Enslavement exists, so anywhere where hexable monsters exist you can at least use your hexable monsters to fight the non-hexable ones for you. Dracs in zot have laughably bad MR when you adjust for when they appear, and they're pretty good vs monsters. If you actually have 27 hexes you can nab orb guardians too.
2) If you ignore specifically zot:5 (which you will have other ways to deal with) then the entire remainder of a 3-rune game is very vulnerable to hexes. Things in spider/swamp/shoals all have bad MR, all the vaults humans have bad MR (even wardens only have 60), and the monsters you run into in depths and zot 1-4 are quite hexable. There are also a decent number of quite hexable things in both hells and pan as well as abyss, though you don't get a free rune there like you kind of do elsewhere. Slime and tomb are admittedly pretty bad if you have literally no offense except hexes, though if you're also stabbing you can sometimes sneak up on TRJ and oneshot it (and likely can manage tomb as well; you don't have to kill things!).
3) Even if we assume that training a skill to 27 in the first place is a good use of xp, hexes doesn't look particularly bad to me. The returns after ~14 or so are still better than you get from at least staves, poison, usually invocations, shields, transmutations, and (for offensive purposes) stealth. (It's harder to judge stealth fairly here; 27 stealth is actually quite good, but you need other skills also. Kind of like hexes!)
Now, that's not to say that hexes are perfectly fine as-is and they don't need improvement. But I don't think adjusting the return on skill investment is necessary; spellpower is arguably even more important for hexes than it is for conjurations (I would argue it is).
Some problems with hexes (I do not necessarily have solutions for these):
1) The binary nature of using them. The big problem here is there's no such thing as a partial success; you can't "half-enslave" a monster, or "half-confuse" a monster (well I guess you could implement things like that but I don't see how that would work very well). Eliminating the binary-failure half of this is something I've seen some suggestions for (like failure increases the success rate of future attempts), but that's not actually the half that I personally think is problematic. Reducing duration of successful hexes does little here, unless you do things like go all the way down to 2-3 turns or so, which then causes other problems. (Why would you ever bother to confuse things? Even enslavement is questionable at that duration, though EH would still work fine.)
2) Enslavement exists as a hex spell. This is a pretty big problem because it's level 4 and it's literally better than an instant kill. It's also available in two starting books, so the problems it causes are very obvious (as opposed to things like old poly spell, which was problematic but that was masked by it not being common and not in starting books). Making the spell rarer doesn't really solve anything, so probably the spell needs to either be adjusted or (preferably) eliminated. (It's already a wand effect.) But having enslavement at level 4 causes problems for all single-target hexes (they need to either be low level or somehow be better, and the latter is pretty much impossible) and for multi-target hexes too, since enslavement effectively disables more than just the monster on which you use it.
---
I don't think the suggestion in the OP is useful, since you're actually suggesting to make hexes
less useful on the monsters where you're complaining they're not useful now: consider a 20% hex success rate and apply that to the OP. There are currently in crawl quite a few situations where a 20% success rate on a hex is still worth using. That would turn into basically never if you make them more "reliable", since it goes down to near-0%.
---
not exactly related, but:
it would also be nice to have other viable stabbing weapons than just daggers.
How are you going to do this? You need to create effects
better than an instant kill, because the investment needed to dagger-stab the vast majority of things to death if you can do any stabbing at all is very minimal. Why would I ever bother confusing something when my other option is to kill it in one attack? You could do some sort of enslavement-stab, maybe, but that's the only effect in crawl that would ever be worth sacrificing an instant-kill for.