Hello, I am Saquib!
I am presenting our paper on graphiti, a framework and a system to do graph analytics on images
and videos by sketching.
I am the co-founder of Tero Labs and Universal Machine.
And my co-authors are Faria Huq from Tero Labs and Syed Arefinul Haque from Northeastern
University.
Let’s first talk about a common process of graph analytics.
Data Scientists are always studying natural or simulated events
And these events are usually captured using different sensors, and video and imaging tools
Almost always, the raw data needs to go through filtering and parameterization.
Then we load the data in our favorite IDEs and construct, analyze, and visualize graphs
We also go back and forth between these steps to do any useful inference.
Now, let’s go back to the beginning of this process.
What do these natural events look like?
Let me show some scientific examples.
Many biologists study the proximity and activity networks of genetically modified ants.
Networks are formed based on the proximity between ants.
Physicists study the synchronous network of fireflies.
Measuring the collective pulsating of fireflies help us study dynamical systems and chaos
theory.
Astrophysicists may use graph analytics to study images like these, which represent galaxy
evolution over time.
By parameterizing the scene based on brightness and proximity, we can find networks of similar
galaxies.
If we focus on video and imaging, the data would need to pass through a computer vision
pipeline.
We also need to parameterize the scene and its properties to create the networks.
In addition, the analyst needs to work in the realm of abstract mental models,
and switch between the visual and abstract representations.
They also need to constantly compare and interpret the analytics outputs with actual events data.
What if we replaced such a messy process with fluid interactions and direct manipulation,
using pen-touch interfaces and sketching?
What if we also introduced the abstract mental models gradually, on top of the visual data?
graphiti is a direct manipulation and drawing tool for graph algebra, analytics, and visualization,
powered by image analysis and computer vision algorithms.
Every detected object in the video can be treated as a node in a graph
We can already see that some edges are formed between some ants when we load the video.
Generally, there are some interesting and useful ways to construct graphs based on visual
parameters in a scene.
We can form graphs by By thresholding the visual variables extracted
from every detected object By object category
By proximity between the nodes By connecting the sites that objects traverse
And by changing the time window for edge construction in videos.
Here, we demonstrate a graph construction using proximity of nodes and temporal accumulation
of edges.
Using a graph construction menu, we can adjust parameters like proximity radius and time
window to construct the desired graph between different ants.
Any node, edge, or graph at any frame can be used for analytics, using direct manipulation
techniques.
Using the function lasso brush, we can select any portion of the graph and apply graph analytics
functions on that portion.
For example, here we want to sort the ants by their degree in the network.
We can drag any detected node or graph into the function as an argument.
This allows us to construct functions visually.
We can evaluate the function, and then the same ants will continue to be labelled with
the output of the function even after that particular frame.
In graphiti, we have implemented graph algorithms to explore any graph in four layers of network
representations.
Simplicial complex and hypergraphs reveal higher level structures in a graph.
Overall, a user can explore the visual and abstract representations in progressive layers
of abstraction.
In graphiti, we can use direct manipulation and drawing to manipulate and analyze graphs.
For example, we can draw nodes and edges to create graphs, and combine or edit existing
graphs using direct manipulation.
We can also apply any graph or network analytics functions on these graphs
all elements in graphiti are reusable as other function's arguments.
In the paper, we describe five types of reusable elements in graphiti.
We also describe how only five reusable elements are adequate to cover a wide range of functions
typically used in graph analytics
Next, let’s look at how we can study fireflies and galaxy networks.
We can detect the fireflies by defining the proximity, color, and blob size thresholds.
Since the woman and the lamp are not part of the desired network, we can use a ignore
lasso to crop them out of the network.
Comparing two different frames show us that pre-synced networks are more sparse.
In this cosmology example, we use community detection to find clusters of similar galaxies.
We also use the shortest path function to find out a minimal distance between early
and late galaxies.
In the paper, we discuss the current design space of graph analytics tools, and why graphiti
falls in the unexplored region of algebraic and visual tools.
The paper also describes many other examples that clarify this framing.
More details can be found in the paper.
For comments and feedback, please reach out to us.
We’d love to hear from you!
Thank you!