19-23 September 2022
Prague, Czech Republic
Europe/Amsterdam timezone

Metrics Framework: Measuring the Success of a Recommendation System

20 Sep 2022, 19:00
1st floor (Vienna Andel Prague)

1st floor

Vienna Andel Prague

Poster Machine Learning/Artificial Intelligence Posters (presenters at poster)


Themis Zamani (GRNET)


A Recommender System (RS) is designed to suggest relevant content or products to users that might like or purchase. RS are growing more popular both commercially and in the research community, by offering personalized experiences to unique users. EOSC is also using a modern RS in EOSC Marketplace.
Measuring the success of a RS is a very important and laborious task. We introduce an independent metrics framework as a service to support the evaluation and adaptation of recommendation mechanisms. The evaluation is quantitatively being performed by processing information such as resources, user actions, ratings, and recommendations in order to measure the impact of the AI-enhanced services and user satisfaction as well as to incorporate this feedback and improve the services provided, via a user-friendly API and dashboard UI. The framework consists of 3 components. Preprocessor which tasks are: (i) data retrieval through a connector module that claims and transforms data from various sources, (ii) service-associated knowledge, (iii) dissociated and dummy data removal, (iv) relation tags dispatch to information that marks various associations in the data, i.e. registered or anonymous -related users and services, (iv) statistics information . RSmetrics responsible for processing the data, computing the designated evaluation metrics, and producing the necessary information in a homogenized manner. A web service presenting reports through a rich UI/dashboard and a rest API. This work is part of the Developments for the EOSC Core RS by WP5 of the EOSC Future Project.
The current version of the implementation features: (i) simple metrics and statistics (ii) complex ones, such as diversity, that indicates if services are recommended equally often; Novelty,; Hit Rate, and Click-Through Rate. The RS evaluation framework is constantly expanding with new features, metrics, and utilities, in order to lead to more robust, data adaptable, and good quality RS designs.

Topic Machine Learning/Artificial Intelligence

Primary authors

Themis Zamani (GRNET) Mr Kostas Kagkelidis (National Infrastructures for Research and Technology - GRNET. (Greece)) Mr Nikolaos Triantafyllis (National Infrastructures for Research and Technology (GRNET)) Kostas Koumantaros (GRNET)

Presentation Materials