Hi, my name is Jane E. Today I’m here to present our work on Dynamic Guidance for Decluttering Photographic Compositions.
It can be incredibly frustrating if you take a photo only to realize after the fact that there is some unwanted clutter in the background that distracts from the main focus of the image.
Notice here, in the left image, the subject blends into pattern behind it.
And in the right image, there are unwanted objects along the left border.
Reframing to clarify the subject and remove these distractors really helps improve these images and better directs the viewers’ eyes.
Our eyes are drawn to regions of high contrast, so light on dark, and dark on light,
thus contrast around the subject will help clarify and declutter the overall image — called subject-background separation,
whereas contrast in other regions especially the border of the image, will distract, causing the eye to be attracted away from the focal subject — called image-border flicker
Photographers recommend a number of methods to be able to more easily see the contrast in an image.
For example, they recommend squinting at the image to see a blurred and higher contrast version of the image,
or looking at the image in grayscale to focus on contrast without aspects of color.
On the right, the photographer draws a line around the subject.
Areas where contrast is maybe too low between the subject and background are shown as gaps.
Looking back at our earlier examples, here are the images with subjects outlined in this way.
You may notice from the right image, this outlining method doesn't do anything to highlight image border flicker.
Inspired by this idea of using outlines to highlight contrast and lack of contrast,
we hoped to recreate this outlining as an overlay directly in the camera,
also extending it to contrast along the image borders.
We wondered if this was an idea that novices could interpret in the camera,
so we started by testing a low-fi prototype.
We ran a Wizard-of-Oz study with 19 participants, where experimenters manually drew outlines on overlaid transparencies taped to the phone.
Here are two photos taken by our participants.
We instructed them each to frame a photo of a person interacting with an object of their choice.
Once they took their initial photo, experimenters outlined the objects in the photo on a transparency, shown below.
We then gave the phone back to the user displaying the photo with the transparency overlay, and asked them to take another version of the photo.
Whether they made any changes to the photo was up to them.
Here you see the updated photos that the participants took.
You can see that both participants adjusted the angle of the camera to better emphasize the focal interaction,
and in doing so removed a lot of the office clutter.
Our low-fi prototype showed promise as we saw many such cases where participants adjusted composition to remove clutter upon seeing the overlay.
However, notice that this interaction is different from having an interactive overlay as it happens as a review step.
In order to realize the interaction, we designed a number of possible implementations of the overlay.
Here you see a subset of these overlay designs applied to the video on the left.
I don’t have time to describe them in detail here, but inspired by existing work on image abstraction,
we incorporated elements of flattening the colors, edge detection, and darkening of the image.
Given these overlay options, we conducted an online survey with 29 participants to better understand which elements were helpful to participants for recognizing clutter
in order to determine the design of our final overlay.
While many liked color flattening for identifying subject-background separation, it was significantly less helpful for image border flicker,
whereas participants found color-coded edges helpful for addressing both decluttering principles.
Therefore, we decided to go with a color-coded edge-based design for our final overlay, which I’ll now describe.
Given an image,
our tool detects edges throughout the image.
However, we want to be able to focus on edges related to subject-background separation and image border flicker.
In particular, we need to identify a border around the subject.
We realized we could estimate the image subjects using object-based saliency maps.
So given this saliency map,
we segment the image into regions describing the subject, subject border, image border, and remaining background.
These segmentations are used to classify and color code the edges:
yellow designates edges within and around the subject,
cyan for edges around the image border,
and white for the remaining background edges.
We merge these to form two edge-based overlays.
The first shows all edges color-coded,
and the second only shows relevant edges around the subject and image borders.
We found in our survey that sometimes participants liked the extra context of the white lines, but other times it could be distracting.
Here is the resulting tool in action.
By default the app shows the edges around just the subject and image borders with a black background.
The user can adjust the slider at the top to bring in more or less of the image color.
They can toggle to bring in the remaining edges in the subject and background.
They can also turn off the visualization by holding a finger down anywhere on the screen or using the toggle at the bottom right.
Given this tool, we wanted to understand if it could help users capture more clear and decluttered photos,
and if users felt creative while using the tool.
Since we needed to run remote studies, we had participants select two different locations, with subjects of 3 different scales: small, medium, and large.
Thus, participants completed a total 6 photo tasks: 3 at each of 2 locations of their choosing.
Here are a few participant photos for each task to give you a sense of the differences in scale for these tasks.
As noted before, photographers sometimes suggest looking at a photo in grayscale to focus on overall contrast.
In fact, many mentioned that they currently use grayscale in their camera even for taking color photos.
Therefore we compare our tool against this method that photographers currently employ as our baseline.
We ran remote studies over Zoom with 18 participants to understand if the tool would help users declutter photos, and if users felt creative while using the tool.
Users experienced two different conditions of the tool.
One with a grayscale overlay,
and a second highlighting edges along the subject and image borders.
For each photo, we asked participants to self-assess them based on subject-background separation and image border flicker.
We actually did not see a significant improvement in overall quality in terms of these principles.
However, participants did believe that the tool was helpful for the task of capturing clear and decluttered images.
It also made them more confident in their ability to address the decluttering principles of subject-background separation and image border flicker.
Participants described feeling like they could take fewer photos,
because they could be more confident in each photo they took.
"Usually when I take photos, I take a ton at once, but didn’t do that here.
I didn’t need to because I was being so precise.
I noticed myself reconsidering the composition more:
e.g. should I have these things in the edges?"
Here you can see this participant refines the camera angle until the edges in the overlay look as she wants.
She is watching the hook on the top left, making sure it doesn’t end up in her shot.
"Potentially a distractor in the... in the photo,
and so I'm trying to angle my phone so that it doesn't catch that in there.
Um... yup."
In addition to encouraging confidence, we found that the tool further encouraged creativity through exploring the space in new ways.
"It caused me to experiment more…
didn't see it as a rule that I needed to minimize lines,
but the tool made it easy to move around and check by that metric, how 'good' it was."
Here you can see how an unexpected edge highlight encourages the participant to explore different backgrounds and compositions.
This participant notices an unwanted edge at the bottom of the plant from stove top,
so she tries out different camera angles to remove the stove,
but notices the dark panel in the background, which she doesn’t want.
She turns the camera back towards the white wall,
but notices the high contrast object at the top left entering her shot.
We hope that we’ve shown how this type of in-camera guidance can help a user to explore their options while taking a photo.
Thank you for listening and I’d be happy to take any questions.