Emc2 He has used the same coffee and measured the data multiple times. Not entirely sure if repeated measures ANOVA testing can be used, but it is the same coffee being tested in different conditions. Repeated measures ANOVA shows significant difference but post hoc testing shows a difference between only the Blind Shaker and Autocomb.

If you consider each extraction to be independent (despite the coffee being the same), I suppose you could use ANOVA.

If he used different coffees with the same set up & grind setting, then you would have a lot more scatter & noise, because different coffees hit different extractions at the same parameters and that is normal & even desirable. Using the same coffee is a more practical & feasible test of the specific effect of the tool.

Wouldn’t ANOVA testing favour the conditions that had means closest to the overall mean, and not necessarily pick up on a more consistent outlier?

Cuprajake Gotta chase that 0.0000015% of ey

It’s more like an increase of up to quarter quarter over the span of samples at a glance, but I agree with the notion that the increase in extraction doesn’t mean a lot unless liking scores are also included. Bear in mind these gizmos are usually marketed as specifically increasing extraction and/or making extractions more consistent…this doesn’t seem to be born out, even if not conclusively disproved.

Under-extracted espresso is the most common failure I experience in cafes…apart from the ones that fill a demitasse!

Samo Smrke is a coffee scientist working at Zurich in the coffee excellence team, of all the purported coffee science that gets articles in the press and sparks forum fads, I would have more faith in him and the rest of the team there (Wellenger & Yeretzian).

  • Emc2 replied to this.

        MWJB ANOVA is not robust against outliers as they will cause the estimate to deviate significantly and increase the chances of type-1 errors. Can’t see many outliers in this data.

        It would be useful to know what test was used. For such a small data, where the number of observations is more than the sample size, I suspect you need specialised statistical tests.

        Another issue is that even with similar extraction times within a group, the EY was variable. There are so many variables that affect EY, so this isn’t surprising. Which begs the question- how reliable is it to use EY as the dependent variable. In my view, a more practical and pragmatic approach for such small samples would be to use flavour to evaluate differences.

        • MWJB replied to this.

            Emc2 You have to pick one aspect to be the focus, extraction time was not the target, EY was as this is the claim most often made by the gizmo makers. The standard deviation of 0.32%EY is certainly small enough to consider the blind shaker condition ‘the same extraction’. Sure taste preference scores would be great, but then you would still want EY so that you could present like for like (or as close as can reasonably be expected) samples.

                MWJB Yes, extraction time was not the target, it was a variable that was kept constant. EY was the only dependent variable. My point is that the variability in EY cannot be due to variable extraction times (as Lance points out too). So if there is variability, there must be other variables at play affecting EY that you cannot control.

                How do you control for this variability? By taking multiple measurements. N=8 is not enough to make any statistical judgments. Especially if you are doing multiple comparisons, the chances of finding a difference purely due to chance increase exponentially.

                Lance claims that the Blind Shaker is superior overall as the mean difference in EY is 0.7%. This is based on the difference in mean values between different groups. I do not think this is a valid statement. For example, If you do a t-test comparing Blind Shaker vs Moonraker, you do not get a statistically significant difference. ( p is .10768)

                Post hoc testing shows a statistically significant difference only for Blind Shaker vs Autocomb.

                Also, what is the minimum difference in EY that can be detected by taste? Ultimately that is what matters.

                The inference you could make is that all methods are equal. But this completely ignores the probability of type 2 error- not finding a difference because the sample size was too small.

                The method that I think makes more sense, is to do a double blind study. Someone else plays with the parameters, another person brings it over the Lance and his job would be to just judge the flavours. Use a different coffee for each extraction and try out the different methods. Not entirely sure what the appropriate sample size would be- this will need power analysis

                • MWJB replied to this.

                    Emc2 Also, what is the minimum difference in EY that can be detected by taste? Ultimately that is what matters.

                    Not really, again, the devices are marketed on their effect on EY. Cups at the same EY (within reasonable expectation) will still differ in taste, +/-0.3%EY has been mooted as the detectable level, but I don’t personally think it is that clear cut. A coffee with 10 brews at the same parameters might have a stdev of 0.4EY and this be considered good/acceptable, but we’re brewing tiny amounts, in short times and, yes as you say there will always be some variability that we cannot control. Context and reasonable expectation is important in making sense of this.

                    EY is an objective measure of brew efficiency and useful for gauging objective consistency. It has been found to have a correlation to regions of interest where people have a preference for taste too, but it is not a direct indicator of taste/quality (as stated for the last 70yrs), hence why I think it important to also have taste scores - EY without taste is too vague to be meaningful, taste scores without an objective value makes troubleshooting & sharing more difficult. (You can’t fix a coffee you will never like by knowing EY, you might be missing out on a better result with one you do if you are consistently off in EY).

                    If you are going to carry out tests yourself for your channel, it would be informative if you did some EY measurements yourself (follow VST protocols whichever refractometer you buy).

                    My apologies, I didn’t notice at first look that Lance only did 8 measurements per condition (he does state this in the video), I agree a larger sample size would have been more robust (He stated that he pulled 100 shots…I wrongly assumed that would be enough to get at least 10 per condition).

                    Yes, double blind testing would be ideal, with sensory testing amongst a random sample who are unaware of what the test is…but now you are talking thousands (or tens of thousands) of pounds, teams of people & finding people in the street who drink espresso…good luck with that 😀

                    Look around your coffee gear, which device/accessory that you have, or even that you know of, has been tested to the degree that you suggest? It’s the lay of the land that we are reliant on sole agents & garden shed protocols to ‘test’ this stuff. ☺️

                    • Emc2 replied to this.

                        MWJB I agree that most of the gear we use does not have any valid experimental data to support their use. Such robust testing can only be done by full-time academics. But what I would like to see is an acknowledgment of the limitations of the study and to refrain from making absolute statements such as the one about a 0.7% increase in EY.

                        I review a fair few research articles- this is perhaps why my expectations are a bit high.

                          MWJB my mistake. I’ve corrected it in the post above

                            Meh. I took the study as more “ You can remove a lot of faff in your process without much detrimental effects if you do this.”

                            It’s not necessarily about what is best. What is good enough.

                              Personally taste tests would always be well personal to the tester other than broadly saying good/bad imho. He would never win going that route instead using repeatably recordable numbers.

                              However my simplistic view on the vid was overall how little there was between the various methods - so if i find a way to simplify further my workflow - or if I get some benefit in prepping my coffee its a win 😂

                              And I have ordered a cheap blind shaker from amazon just to try it out by using that rather direct into my portafilter.

                              I still need to try making a really under then over extracted shot just to see how it tastes for my taste buds… dont think I nailed that yet 👍

                              Loonster The point is that the data doesn’t show that! You can’t make any conclusions based on it. I guess the only way is to try it yourself. I’ve ordered a shaker as well.

                                Suddenly a rush on shakers 😂

                                Though not the weber - 65 quid and then 30 odd shipping - plus you would get hit with VAT etc

                                  I got one for 15 from amazon but its coming from china.. i did sort of look at aliexpress and got alot of we dont ship to the UK

                                  • Emc2 replied to this.

                                    MattH it’s often a problem with the settings. If you haven’t selected a country in your profile, you will get this message.

                                        Emc2 aha - thanks for that - I will try that out 👍

                                          Tried shaking the coffee in the grind cup this morning for about 20 seconds or so. Surprisingly, the espresso took longer to extract and tasted good. Not sure about the EY, but this method does seem to take away the need for deep WDT. ( I did puck rake though).

                                          it is getting more and more ridiculous. instead of having a simple, smooth, relatively foolproof workflow - people are trying to make pulling a shot a ritual instead. As if only them know and can enjoy a good cup