Hi my name is Jane E, and I’m here to present my work, joint with Cynthia, Jianming, Radomír, and Jose at Adobe,
and my Stanford collaborators Ohad, Pat, and James.
For this project, we looked at designing interfaces for composition guidance at capture-time.
Composition is an important aspect of photography.
It describes how the physical elements in an image,
in particular, those of higher visual attention
are arranged relative to each other in the frame.
Good composition can help make an image more visually appealing to the viewer,
and draw their eyes to the important parts of the image.
One common way of describing composition, is using composition grids, like the rule of thirds.
The rule of thirds suggests that it can often be better to place a subject around the thirds lines
rather than centered, like the door in this image.
It suggests that this image is better composed than a similar image with the door near the center
as is approximated here,
or with the door along the right thirds line
as shown here.
The rule says that this is not as well-composed
because there should be space, or breathing room, in front of the man.
Here are a few other images with the rule of thirds grid overlaid.
For these images, these are the more relevant lines as they align with important objects or boundaries in the images.
As we saw in the first image, the left third line aligns with the door.
In the middle image, the left vertical line goes through the staircase,
and the horizontal line aligns with the edge of the ceiling.
Finally in the image on the right, this line aligns with the horizon.
However the rule of thirds is rather restrictive and can only describe a small set of compositional elements.
Notice in these images, there are many other strong lines that aren’t being described by the rule of thirds.
In addition to thirds, there are other proportions that are known to be visually pleasing.
The harmonic armature is a composition grid that encompasses a variety of these proportions such as halves and quarters, formed by intersecting diagonals.
This can describe a wider range of compositions.
Here, the quarter line is actually probably better than the third for describing the position of the door and person.
In this next image, this diagonal almost perfectly aligns with the very blurred fence in the foreground.
And the center lines and diagonals highlight the symmetry and vanishing point at the end of the hall.
In this last image, this center line also emphasizes the symmetry of the image.
And this one aligns with the bottom of the building.
These nicely frame the triangular shape of the bridge leading into the museum.
I would probably pick these as the most relevant lines for describing these compositions, but as we just saw, many lines are reasonable.
It can be overwhelming to find these alignments and interpret composition with these more complex grids.
Here are some images annotated by professional photographers to explain composition in blog posts and books.
Photographers will walk through different aspects of a photo as I just did, highlighting the relevant subsets of lines as they go.
This helps focus the attention of the readers.
We wanted to automate this process and show these highlighted lines as overlays directly in the camera.
But first we wanted to understand if such a tool would be useful and if so, how to best design it.
So we asked a few questions about how to design such an interface.
First, we wanted it to fit in the current workflow of photographers,
so we wondered — do people want any in-camera guidance?
And if so, what type of guidance is helpful at capture-time?
Second, we wondered if highlighted composition grids were a representation that novice users would be able to interpret without expert guidance.
To answer this first question, we conducted a survey amongst people who are interested in photography
to better understand how people work on their photography craft.
And to answer the second, we collected a dataset of composition annotations from workers on Mechanical Turk.
I’ll start by talking about the first question.
We conducted a survey with 127 participants, most of whom had some experience taking photography classes either in-person or through books/videos.
We asked them about their photographic process and their learning experiences,
including what was effective about the different learning methods and whether any of it could be helpful at capture-time.
Many participants expressed that they appreciated that in-person classes provided individualized and immediate feedback in the context of the current photo they were working on.
Participants expressed wanting feedback on refining the current image.
They also mentioned that they prefer to only make small framing adjustments while editing,
and would therefore also appreciate suggestions for larger changes in the camera.
These points suggest that feedback should be context-aware,
and should support both exploration of new perspectives and ideas as well as refinement of the current idea.
Finally they expressed concern that the interface might be too distracting and disruptive, and therefore would slow down their photographic process.
This was especially important in situations with time pressure, such as when capturing a fleeting moment or when traveling with a group.
Thus, the interface should be minimally intrusive.
The user should be able to disregard the guidance when needed,
such as in moments when it is important to just quickly capture a shot,
or when the guidance differs from their own creative intentions.
Thus, these became our 3 design goals.
We want to design guidance that is context-aware and able to adapt to the current image.
It should support exploration by helping photographers discover new ideas as well as refinement by helping photographers perfect an existing idea.
And finally, it should be minimally intrusive and not hinder the creative freedom of the photographer.
After answering the first question, we believed that in-camera composition guidance could indeed be helpful.
These design goals reinforced our interface idea of overlaying contextual composition feedback directly in the camera.
By helping users understand the current composition, it would allow them to adjust and explore in the moment.
Now that we know that we want to overlay contextual composition feedback,
there are many ways to represent composition.
We wondered if highlighting lines in a composition grid, like the harmonic armature, was a helpful representation for users to interpret composition.
And if so, how do we decide what lines are most relevant to highlight.
To answer this, we ran a mechanical turk study where we had workers annotate 10 images per task, after some basic training on what composition is.
They were asked to select 1-4 lines from the harmonic armature that best describe the composition of the physical elements in the image.
For the purposes of explanation, say we just had these 4 workers annotate this image.
We want to compute scores for each line in the composition grid based on its “popularity.”
For each line, its score is the % of worker annotations that include that line.
Here, 75% of worker annotations, 3 of the 4, include the center horizontal line,
while only 25% select the diagonal, just the one in the bottom left.
Here is the scored annotation for this image —
darker lines correspond to lower scores, brighter lines to higher scores.
We empirically set 40% as a score threshold to pick a set of lines to be “ground truth.”
For this image, We have these two lines remaining.
Overall the average number of lines per image before this filtering was 2.17,
and after applying the threshold, the average number of lines per ground truth annotation was 1.82,
meaning not too many lines were actually removed per image, suggesting that workers were relatively consistent.
Here are two other ground truth annotations.
Notice that workers are selecting lines that intersect with salient regions in the images.
In the left image, lines intersect with the water droplet,
the middle with the animal’s eye,
and on the right, along the boundaries in the landscape.
We also actually got feedback from the workers expressing that they found the task to be fun.
In summary, from our formative studies we learned that guidance should be context-aware,
and support exploration and refinement,
while being minimally intrusive.
Thus, providing composition guidance through a camera overlay seemed reasonable.
We also learned that novices do indeed find the task of annotating photo composition by highlighting lines in a composition grid to be useful and approachable.
They were also relatively consistent in their annotations, and highlighted lines through areas of visual saliency.
Thus, we wanted to design an algorithm that would adaptively select the most visually salient lines of the harmonic armature.
In order to do so, we needed to give each line a score based it proximity to visually salient regions in the image.
We start by computing a per pixel saliency map.
For each line of the 14 lines in the armature, we generate a gaussian mask along the line and multiply each one by the saliency map.
Summing up the per pixel product gives us the score for each line.
We denote the lines with the top 3 scores as our adaptive armature.
For each of these images, here is our saliency-based adaptive armature compared to the ground truth mechanical turk annotation.
Notice that the ground truth often has 2 selected lines.
We chose to highlight 3 in order to provide more ideas to the user as they used our tool.
Here is our capture-time tool again.
It overlays these relevant lines, or adaptive armatures, on the camera viewfinder.
Here, the saliency algorithm is detecting that the green object on the bench is salient,
and is selecting lines that intersect with it to represent the current composition.
We ran a study with 24 participants to understand if the tool would help users capture photos that they believed were more well-composed,
and if users felt creative while using the tool.
Users experienced two different conditions of the tool.
One with a static composition grid,
and a second dynamically highlighting our adaptive armatures.
Participants completed 6 photo tasks: 3 at each of 2 locations.
The tasks were designed to increase in scale and complexity and to correspond in style across the two locations.
We counterbalanced location and condition to avoid biases due to location preferences or learning effects.
In both the static and adaptive conditions,
we saw participants snapping elements in the image
to lines or points on the composition grid.
For example, this participant places the subject of the photo at different intersection points.
Here, the participant aligns the edges of the building to various vertical lines on the grid.
Along the center,
now the thirds,
and the other thirds line.
Notice how he pauses at the lines to refine each alignment and examine the composition.
Here are a couple of photos that participants captured using the static guidance,
and here are a few captured using the adaptive guidance.
We found that participants indeed believed that the photos they captured with the adaptive guidance were better composed.
However, when measuring creativity using the creativity support index, we did not see a significant difference.
Nonetheless, we heard a lot of qualitative support suggesting that users felt like the tool encouraged them to explore new creative ideas
and made them feel confident in their ability to compose photos.
Let’s watch some participants from the summative study in action, framing their shots.
Here we see how seeing different highlighted lines gave participants new ideas to explore.
I want the side of the bench to align with the diagonal that was highlighted.
I actually didn't see it fit into this corner until the line on the screen showed up.
So the highlighting of lines gave her this new idea to try.
And I'll try to line up the saliency lines with the window, ends of the window I guess?
So here she aligns with the bottom,
and here the tops of the windows.
And then I'll try doing the saliency line with the edge of the building?
The highlighting also provided confirmation to the participants, making them more confident in their photos.
Ok I see the lines that I want lighting up, so this looks good.
Oh yea, it's really nice because the lines go through the Bulbasaur.
Our tool shows that adaptive armatures can improve photographic composition,
allowing people to explore a variety of compositions while situated in the environment.
We introduced this capture-time tool for composing photos,
and are interested in designing capture-time interfaces for other aspects of photography.
We hope that others also consider the space of designing interfaces for guidance directly in the camera.
Thanks for listening.