Evaluating an Itinerary Recommendation Algorithm for Runners
Based on prior literature and our own experience, we realized that runners struggle to find new
and pleasant itinerary that match their distance and difficulty targets.
Therefore, they usually stick to the paths they know or might use other
runners’ traces shared by existing tools. Our goal is to propose pleasant roundtrip
tours for runners in unknown locations, to solve the cold start recommendation problem,
by mining map data without requiring the availability of running traces.
To do so the 1st task of our algorithm is to compute a pleasantness score.
A pleasant running tour is one composed of path segments that match the runners’ preferences.
Based on initial interviews, we specified the runners’ standard preferences to be the
proximity to nature, type of path, distance to high traffic roads and path steepness.
Our algorithm rates each segment of the map according to these 4 main characteristics.
Here is an example of how four segments would be rated by our algorithm.
With this scored graph, a starting point, a target distance and an elevation gain objective,
it tries to generate the most pleasant tour. 1st, from a trivial short tour, it grows by
selecting possible extensions, until the objectives are reached
and no further improvement is found Then it runs the post-processing step
to improve the shape of the tour by avoiding backtracking or unnecessary turns.
In order to evaluate the quality of the proposed tour, we designed an experiment with the main
objective of assessing the correlation between the score given by the algorithm
and the users’ ratings, at a segment level. Additionally, our evaluation should also validate
that the tool improves exploration for the users. We chose to conduct an asynchronous remote
experiment in a real environment as the usability evaluation of a recommender
system should consider the context of its use. We built an Android mobile app prototype
and generated 3 tours for each user, in which we identified segments to give feedback on.
We also implemented a basic visual navigation support.
In order to measure the said correlation, we asked the user, during the run, to rate on a five stars
scale the segments they have just completed. At the end of the tour, they would rate the
overall tour, rank the segments, and explain what they liked and disliked about the segments.
We recruited 24 participants in 3 different cities in France,
but only eleven of them were able to participate due to COVID confinement measures.
Participants installed our prototype on their smartphone, selected a starting
point far from their usual running places and completed the suggested running tours.
We collected their ratings and feedbacks for analysis.
31 itineraries were run, and 93 segments evaluated
The analysis of the quantitative data showed a moderate correlation of 0.46
between users rating and the computed score by our algorithm
Data also shows that the three most influential score characteristics values are consistent with
users’ perception. For instance, segments that users rated high because of nature
had a high computed nature score of 0.72. The analysis of qualitative data showed that
most users appreciated the concept and the quality of recommendations. It helped them discovering new
areas. However, they had concerns regarding safety and negative ambience of some places.
They also faced issues due to incorrect or incomplete map data labelling, where the
tours included paths that did not exist, crossed private properties or were temporarily closed.
A complementary approach of mining map data and combining
more reliable local knowledge would generate a personalized recommendation.
It furthermore illustrated that embedding flexibility within the itinerary would
be useful and allow dynamic adjustments in unplanned situations like construction work.
Finally, we acknowledge that our evaluation settings came with challenges and limitations.
In a real-world usability test as ours, it is difficult to eliminate the effect of confounding
variables. We also had fewer participants than originally planned which is a drawback for forming
reliable statistical data. Other issues also included unsatisfactory navigation.
Despite the challenges, we validated our algorithm
and outlined suggestions for future recommender systems for runners.