good morning i am mauro mezzini i’m going to present this work tracking museum visitors
through convolutional object detectors in collaboration with carla limongelli giuseppe
sansonetti and carlo de medio of roma tre university
so what we want to make is a system for profiling visitors in a museum so let's look what kind
of information we want to collect about visitors and how difficult is to collect this informations
first of all we want to collect and understand how much time the visitor spend in a museum
this information can be collected relatively easily because we can imagine for example
to put for example an electronic gate at the entrance of the museum an electronic gate
at the exit of the museum another information that we are interested on is which artworks
she watched that during the visit collecting this information is much more difficult then
the previous one because we cannot go after each visitor and record which artwork was
watched or not was not watched another information which follows directly from the previous one
is but once a we record what at work was watched we want to know how long time the visitor
spend in front of a particular artwork if the previous one was difficult this is even
more difficult to carry on if done manually by hand and finally we want to even record
the facial reactions so the movement the direction of the eyes the movement of the eyebrows the
movement of the of the mouth all these facial reactions can be interesting for for inferring
emotions or reaction or emotional reaction in front of the artwork cleary if the previous
points were very difficult to to record if done manually this is clearly very infeasible
if done manually so what we need is automatic system for tracking visitors in a museum there
exist already systems proposed for automatic tracking the visitor in a museum the first
system kind of system gives to the visitor a device an electronic devices device and
this device is carried by the visitor and is detected there are detectors around the
museum and but this system requires somewhat visitor collaboration the visitor should carry
on some device should be responsible for not switching off and something like this another
kind of system uses lidar in this case this system does not require visitor collaboration
at all but sometimes it is not very accurate at distinguish visitors one from another and
we may argue that a single lidar device is somewhat expensive because it cost at around
thousands dollars so we propose another system based on the recent progress on convolutional
neural networks and it leverages on the fact that the convolutional object detectors are
very very effective in recognising predefined objects so the idea is very very simple we
have a badge like the one like the one here the badge that is used usually in congresses
or conferences and is given for free then we ask visitor to wear the badge and go visit
the museum and the we have on the wall of the museum cameras that record video and the
object detector detects in the with the video the badge worn by the visitor the detection
of the badge means not only that to infer the presence or the absence of the badge inside
the video frame but also to draw the tightest bounding box around it and as i go farthest
the camera we observe that the dimension in pixels of the bounding box changes depending
on the distance based on a very simple trigonometric reasoning we we are able to very accurately
measure the distance of the badge from the camera first of all we need to know the angular
amplitude of a pixel psi and then knowing the dimension in pixels of the badge we can
calculate the real angle beta that the badge forms with the camera and then knowing the
real dimension of the badge which are 10 centimetres by 14 14 centimetres height and we can compute
the distance ell of the badge from the camera with the same reasoning we can compute the
angle alpha that the badge forms with the median line line of the video frame so we
have polar coordinates distance ell and alpha that allow us to very accurately determine
the position of the visitor inside the museum also knowing the frame rate of the video we
know the the time the visitor spend inside the museum the time the visitors spend in
front of a particular artwork and so on furthermore we we not only may detect the badge but also
we may detect the face the faces and then using these faces we may detect for example
any facial reactions for example the direction of the eyes or changes in the eyebrows or
in the mouth in order to infer any emotional reaction in front of any artwork the visitor
watched now i want to show you how the system works
by showing you a couple of videos that I shot in this first video we see that the system
recognises the object and the face the faces are copied in the lower left corner and we
we may use this face later for detecting any facial reaction or the form of the eyebrows
or the form of the mouth or the direction of the face we see that we may recognise different
faces inside the same frame of course and the in the upper left corner we have the distance
computed by the system and the angle alpha that the badge form with the median line of
the video so this is the end of my presentation thank you very much