tompoland The reason to stay blinded until you’re done with your set of three tasting comparisons is to eliminate the possibility of unconscious bias. Your method of marking the cups, then having your wife hand you one and then the other without letting you know which one was which is sound. But there are always possibilities that something may tip you off as to which cup is which.
For example, let’s say that one of the two grinders produces espresso with more crema than the other. You do the first blind test, and then you see which grinder was associated with more crema. For the next comparison, if you see that one cup has more crema than the other, at some level you’ll already know which cup came from which grinder, which makes the comparison non-blinded at that point. This leads to reinforcement bias – whatever impression you had of the cup with more crema from the first comparison will color your evaluation of the second comparison.
This sort of accidental tip off doesn’t have to be consciously seen to cause an observational bias. Picking up on these things on a subconscious level can cause an observational bias as well.
Doing the identification after the first comparison can mess up the subsequent comparisons in another way. For variety, let’s suppose that in the first comparison you note that one espresso looks darker than the other, and then you find out which grinder produced the lighter cup. But let’s also suppose that the color difference had nothing to do with the grinder, and that was just happenstance. You don’t know this at this point, but the association with one grinder with lighter espresso is now there in your mind. If there’s another color difference in the second or third comparison, the observational bias now has the possibility of you assigning attributes to the espresso that actually doesn’t correlate with the grinder.
Keeping the identities of the cups secret to you until you’ve finished your set of three comparisons eliminates these problems. Again, this is standard practice for clinical trials I’ve worked on, so it’s not like I’m being fussy about this point just to be fussy. It’s really standard practice. If I was reviewing a study submitted for publication, and I saw that the observations were unblinded before they were all collected, that would be the first thing I or any other reviewer would pounce on.
To put it another way – you’ve done so much work to eliminate non-grinder factors in your testing plan, and have done this in a rigorous manner. It would be a shame to not take this last step to make the plan bulletproof.