Mesure and more

Trusted Labs is involved in the Mesure project, whose goal is to develop open benchmarks for Java Card. There are few partners to this project (CNAM, INRIA/POPS, and us), and in particular, no major manufacturer. Yet, the idea is here to start a community that will publish some results. Performance is a sensitive piece information about any product, and even more about products that are never publicly tested and compared.

One good news is that other people are thinking about Mesure as a way to publish benchmarks. In particular, the people who developed random number generation benchmark (from the universities of Bordeaux and Limoges) mentioned in their presentation that their work could be associated to the results of Mesure.

Publication time is getting closer, and I hope that some people will actually use the benchmarks, and that this will lead to interesting results. However, there will also be a very strong challenge for us: will we be able to build a community around Mesure, in order to improve it and make it live? The publicly funded project will end soon, so this will become a true open source project. We will need more contributors, and I hope that the research teams involved in smart card research, as well as the industrial R&D teams, will help us build this community.

Performance measurement and quality measurements are simply a first step here, and there are many more things that could be shared between teams. What we need here is a platform, shared by many actors, and which provide shared services to all of us. I hope that the news will be good next year, with a few results derived from Mesure, and a few additions to the project.

2 Comments

  • lexdabear wrote:

    I attended this presentation, but missed the last half hour or so. The quality was not very good and the only interesting part was the academic work how to obtain an accurate and reproducible result.

    Did you try to approach smart card controller manufacturers to participate in MESURE? Maybe it would have more relevance for them if we make it official in a standardization body (e.g. GP or JCF) .

    You mentioned that the biggest effort is to build an active community, which not only improves the benchmark tool, but also has means to publish the results. My concern is that the results won’t be comparable due to different level of security, e.g. one product which is CC/MCI/FIPS certified will have a hard time competing against a loyalty card, and the message won’t be right.

    At the beginning it was mentioned that the next step is to find a way how to measure security. It would be great if this project finds a way .. then you would have a meaningful benchmark about the performance. But I think a security benchmark will be something like the evaluation of dancing qualities .. the result will be as different as the judges are.

    I appreciate your presentation about applet performance (I didn’t attend it neither, but looking at the slides it must haven been great). The guidelines hit the nail on the head and it will be definitely my commandments for applet programming. I wish I could have skipped the rest of the module, just to listen to yours. Well done.

  • The MESURE project is actually funded by the French government. To make a long story short, the idea started as an industry-wide project, and ended up as a small 3-partner, mostly academic project.

    I am not sure that standardization would be useful for such a project. However, I am sure that public availability is a key parameter. Anybody will be able to get the tests, run them, and criticize them.

    About performance vs. security, what you say is true, and we cannot avoid marketers who make unfair claims and customers who make bad decisions. The banchmark that we are developing will actually make things better, as performance claims made today are often unfounded. In addition, as security evaluators, we cannot include performance as a criterion, and we end up assessing a high-performance card with a much slower ultra-secure with the same criteria.

    Finally, about measuring security, I am not a strong believer. When I see what we do at Trusted Labs in a security evaluation, I can’t really imagine a way to click on a button and do the same thing. But then, I am not sure that industry workers in the 70’s guessed that they would be replaced by robots, so I will wait for researchers to do their work.

    About participation, the MESURE project (or at least its funding) will soon be over, so the project will start being a real open source project, and I hope that we will have many contributors (lexdabear, maybe?) enhancing the tests and making new ones.

Leave a Reply

Your email is never shared.Required fields are marked *