I am not sure that standardization would be useful for such a project. However, I am sure that public availability is a key parameter. Anybody will be able to get the tests, run them, and criticize them.

About performance vs. security, what you say is true, and we cannot avoid marketers who make unfair claims and customers who make bad decisions. The banchmark that we are developing will actually make things better, as performance claims made today are often unfounded. In addition, as security evaluators, we cannot include performance as a criterion, and we end up assessing a high-performance card with a much slower ultra-secure with the same criteria.

Finally, about measuring security, I am not a strong believer. When I see what we do at Trusted Labs in a security evaluation, I can’t really imagine a way to click on a button and do the same thing. But then, I am not sure that industry workers in the 70’s guessed that they would be replaced by robots, so I will wait for researchers to do their work.

About participation, the MESURE project (or at least its funding) will soon be over, so the project will start being a real open source project, and I hope that we will have many contributors (lexdabear, maybe?) enhancing the tests and making new ones.

Did you try to approach smart card controller manufacturers to participate in MESURE? Maybe it would have more relevance for them if we make it official in a standardization body (e.g. GP or JCF) .

You mentioned that the biggest effort is to build an active community, which not only improves the benchmark tool, but also has means to publish the results. My concern is that the results won’t be comparable due to different level of security, e.g. one product which is CC/MCI/FIPS certified will have a hard time competing against a loyalty card, and the message won’t be right.

At the beginning it was mentioned that the next step is to find a way how to measure security. It would be great if this project finds a way .. then you would have a meaningful benchmark about the performance. But I think a security benchmark will be something like the evaluation of dancing qualities .. the result will be as different as the judges are.

I appreciate your presentation about applet performance (I didn’t attend it neither, but looking at the slides it must haven been great). The guidelines hit the nail on the head and it will be definitely my commandments for applet programming. I wish I could have skipped the rest of the module, just to listen to yours. Well done.