I spent a significant chunk of time this weekend paring down my short list of CMS's to research, and closely comparing the most popular CMS offerings. The biggest things I'm looking for are: data integrity, a pluggable API, advanced caching, package deployment, friendly URLs, content scheduling, content prototyping, multi-site support, and a unified content model. When I say "unified content model" I'm referring to a single, well defined data model that is extensible, and able to house most conceivable content types, without requiring new APIs to facilitate every new content structure. For example, if I add a new component to my CMS that exports content to NML (Natalie Markup Language), it should be able to interface with the content API and handle all content types, without requiring any special cases for a particular content type. New functionality, be it back end, front end, or in between, should aim at being global, applying to all content on all sites; otherwise, if you're engaging regularly in customization, then you're only enjoying half of the benefits of deploying a CMS.
I was surprised to see that not all major CMS's offer each of the fundamental features listed above. This may be owing to the fact that these systems were designed years ago, without the benefit of the hindsight that we have now, seeing the need for a unified content model. A good example of this issue is content caching. I'm only ankle deep in the process of high level comparisons between Mambo/Joomla!, Typo3, Zope/Plone, and Drupal, and it will be a while before I'm able to get down to low level inspections of their caching mechanisms, but from cursory examinations of other systems it looks as if advanced caching is often an afterthought, with progress and solutions being hindered by the absence of a unified content model. It's trite to capture a parsed HTML page and cache it for an hour, disregarding any updates to its constituents during that hour (that's what Squid is for). I'm hoping to find a CMS that manages the web of relationships between every element of the system that impacts whether or not cached content requires refreshing. An admin should be able to manually empty or refresh the content cache, but they should never be required to.
The CMS community is highly fragmented, and there is no accepted central resource on CMS eminence. At last count, this EC Mag article tallied 1879 CMS offerings. The problem with this proliferation is that a number of platforms remain poorly documented. Composing software and not documenting it, especially for a system on the scale of a CMS, is rather like wasting time playing hide and seek with your intellect.