"Which holds should I buy?" - in a flooded market, this is a common question.
There are plenty of sites and magazines reviewing climbing holds. From the perspective of a climber buying the holds, that's great. People can pick up a copy of Climbing mag or browse the web and figure out which holds are the most ergonomic. Or which holds have the best texture. Some sites cover how the holds were shipped. Can we agree that for a product which has an industry standard of being guaranteed not to break, it's kind of silly to rate how well the holds set and climb based on what ply of wax paper they were lovingly wrapped in?
It's hard to assert that texture and comfort aren't important to the big question above, but I have a hard time seeing those type of reviews as more than one-dimensional. The routesetter is going to look at an ergonomic hold with no directionality and be completely uninspired. If you can't create good movement, the hold might as well have the texture of broken glass.
That being said, my goal in reviewing holds covers the following topics:
Directionality - How does the hold perform in an angular, spatial sense? The biggest factors here are: 1) can you easily and quickly tweak the route by altering the angle of the hold, without compromising the grip style your movement depends on? 2) Can the hold be used to force body positions based on limited grip styles or foot placements? Directionality is possibly the most important concept in forcing good movement.
Aesthetics - An often overlooked aspect of setting. Holds are the building block for good aesthetics. Maybe without even knowing it, routesetters implement fundamental design principles - proximity, alignment, figure/ground, repetition, and contrast. Holds should facilitate pleasant route design elements.
Versatility - On the polar end from directionality we have versatility. How many different functions can the hold employ? Hold size, angle, number of grip styles, potential for foot placements are all factors; general moves like matching, hand/foot matching, toe-ins, etc all require different styles of hold.
Utility - Handles factors related to how much effort it takes to utilize the hold. For instance: Are the washers set evenly and perfectly perpendicular to the wall? How durable is the hold? (hardly a factor for most modern holds, but still relevant.) Is the hold well-constructed and equitable in texture and shape? Does the weight of the hold impact its usability? Does the size of the hold vs its usable space negatively affect available wall real estate? Does the hold sit flush to the wall? Is the bolt hole central enough to the hold to discourage spinning? Basically, a catchall for potential routesetting frustrations.
Climbing - Finally, how does the hold climb? Texture, ergonomics, fun factor. These are relevant elements to the topic but they are discussed at length elsewhere, so for my reviews the hold's "climbability" won't be the headliner.
In summary, I will be aiming to review holds as a routesetter, not a climber.
One more thing, while we're talking about buying holds. The blunt force pision of climbing holds into categories like "pinch," "crimp," etc. is a well-intentioned mistake. A pinch and a crimp/edge are the same thing at the proper angle. I'd say about 80% of holds marketed as "slopers" are really "pinches" - anything smaller than about six inches square and it's very unlikely you're going to prevent people from getting their thumb wrapped around the back of it.
There is clearly no industry uniform for this taxonomy, and it's frustrating to see just how little usability research has been done. The prospect of sorting through all the holds available and picking some based on what little criteria is available is pretty daunting. Telling people the actual (measured) size and weight of the hold is a good start, along with providing a comparison image including a hand actually using the hold. Don't be presumptuous about how the hold will be used. In fact, vague hold "difficulty" is more useful than a blanket category, in my opinion. While it might be up for discussion whether a hold is an "edge" or a "crimp," I think it will hold true across the board that it's tough to hold on to it on a steep wall.
Happy setting for now!