good afternoon, guys. my name is joo. i'm a product designer
and i'm currently working at naver now, a video and livestream platform in south korea. i'm really
excited to have this opportunity to share one of my best projects with all of the figma fans today.
Since last year, i have created a program for figma and zeplin which in the long term
i aim for it to become the ai spell checker for design systems.
in this session i will explain the key motivations that inspired me to begin this project
i will also discuss my experience in developing the figma plugin along
with interesting feedbacks and reactions that i received as deploying this prototype last summer
the motivation of this project stems from my design system experience. design system
is a complete set of standards that are intended to manage this design at
scale using reusable ui components and ux patterns
now most of you here watching this livestream might be familiar with design systems, for example
google material design. as you know, google material design defines how to create google
themed product through a common set of design standards and front-end components and more
so these days design system is extremely common when it comes to creating new user
interface for large scale products and service to ensure their consistency and productivity.
however it still has major obstacles. one of which is the increasing complexity with the increasing
number of ui components and patterns including it in the same library and another important thing is
that it is super difficult to replace components one by one in the regularly designed files
because of those kind of the complexity and problems
even if designers shape the same screen just like this
this one used the library exactly but
but another one has legacy or detached instance just like this and some of them even has just
emoji or unicorn or whatever something like that so they look similar but at the same time
they are completely different in the design systems point of view right
in design system teams we work super hard to prevent those kind of human mistakes for example
we update guidance website frequently we also did design qa very often
and in my case we even published cartoon manuals to make my designers easy to
understand what is design system and what kind of things they should follow
but we all know that human resources were always limited so we should think differently and that
is the point that i started thinking about how to solve this problem using machine learning because
we all have been living with many automatic services using machine learning such as
face recognition and mouth detection for example so if we can integrate the ai
technology into our design work that a computer runs the number of design
design elements instead of the human spell checks and autonomously recommend the design
grammar to human if we find something that does not conform to the guideline
this will not only be convenient for design system user but it will also reduce tons of burden
on people like me who should manage this system after i point out this
i started to make it with tensorflow and figma just like this
and figma execute my program take your time and click next as you can see my program draw
many boxes and provide the checkpoint information let's deep dive into it
the sky blue boxes are ui objects found by tensorflow.js and the orange boxes are
design system information provided by figma my program compares these box two boxes
automatically find the mistakes and let designers know which part to revise
the name of this project is figma ml and as you can see i have been trying to create
an ai spell check for design systems via figma apis and tensorflow it's fully open source
so if anyone if you are interested in this plugin you can install it from github or figma community
it was a small project but it wasn't easy for me to make it work there was three major challenges
that i faced and that i had to serve
a much larger ui dataset and more optimized ml model and finally i had to find the best
way to make my designers fill this ai power seamlessly on their design process
since last year i have been trying to solve those problems one by one
for data i met a singapore based startup called mobbin they have over 50,000 high
quality mobile screenshots for product designers it's really amazing data so i contacted them and
presented my project just like this and got over 10,000 high quality ui screenshot data for free
moreover i received the free labeling support from korean startup called selectstar
we worked super hard together and annotated over 50,000 bounding boxes it was huge amount
of the data and i sincerely appreciate the contributions of those two companies
to my project we still have really great relationships
after i prepared my data set then my next step was creating tensorflow model
i downloaded some resources from google tensorflow hub and custom trained so that they
could detect the ui object and experimented with the performance of those models
after several experiments i chose a model that's suitable for my project for ui detection
the result is just like this here is an example of a music app and as you can see
these blue boxes are accurately highlighting ui objects right from the web browser in real time
if you want to find more technical detail go to the tensorflow
youtube channel and find the tensorflow.js show and tell series that i involved recently
and the last challenge was how to utilize those ai detections on figma
and how to provide useful information for designers based on my ai so i
decided to compare those detections to figma's design system information
i started with drawing two different types of boxes on the iframe area on figma plugin
in my assumption each blue boxes should match up with a yellow one if the design is not done
perfectly the boxes will not be aligned so i decided to create a plugin and logic to
point out this misalignment to designers so that they know which part they had to revise
so i added logic to select and compare boxes for example select by size
or overlapping areas
finally text similarity in labels
as a result as you can see on the screen i drew two different types of de-boxes on
the iframe on figma plugin and pointed out the misalignment just right there
which means now the tensorflow model that i trained with my new custom ui dataset
learns behind the web browser of figma has begun to provide the useful information for designers
the best way to illustrate this is to see in action together
the figma ml plugin. choose the screen on figma and
execute my plugin see what are there on the left and check the next button and
ai detected the ui and read designs know what are the expected to problems just like this
here
i also created the zeplin version for people who prefer to communicate through zeplin
similarly choose the model and then now the browser can recognize ui elements
and compare the detection with design library and provide the checkpoint information
through zeplin comment api so in summary i solved my challenges via a global partnership
transfer learning and fine tuning and figma plugins
after distributing this prototype i
shared my story via my blog medium and twitter and the reaction was so much fun
first of all anthony from figma designer advocate phrased my project as a very interesting case
we had a lot of conversations and improved my project even further
zeplin introduced my project to their newsletter and on their twitter channel as a super use case
i really appreciate it and we still have a good relationship
jason from google tensorflow team also
praised this project as very interesting and amazing journey
finally i presented this project as the deview21 conference which is one of the largest
developer conference in south korea it was an unforgettable memory to me
all of those responses were very grateful to me and i'm currently trying to take this project
to the next level let's get back in. number one the improved accuracy is super important
and i'm constantly striving to improve the detection and ai detection accuracy
all of my experiments are uploaded on github so if anyone wanted to see more detail
check out my project on github and please contribute your idea
and i'm also working on adding customizable features for detecting
users design systems instead of detecting general ui object
finally i'm trying to upgrade my program so that we can render styles and ui patterns
beyond just discovering uis it was huge project so we need some more time to
make it work but we are trying to make trying to improve our project
the lesson learned. i have gained a lot from this project
among them i wanted let you know which ones were the most meaningful for me
when i look back on my design experience
one of the worst things happening at my workplace was the mastery of inefficiencies
basically if something were incredibly inefficient i would eventually discover some of my own tricks
shortcuts and master these inefficiencies then i would forgot that the problem was
incredibly inefficient and just continue to use my shortcuts and follow the norm
design system has provided huge momentum to measure shift our design process from
tracking pixel to ui component based communication
it's huge but it is not perfect yet there are a lot of inefficiencies out there
my case is also a kind of a small example
previously i thought it was not my job to solve this kind of problems but through this project
journey i run and finally realized that figma is not just a ui tool but a solution that helps solve
the problems of our work design environment our work environment
so the only thing we need we need to do is not put our problems behind us
there are a lot of plugin projects out there and many plug-in projects are still waiting
for your contribution and your idea so do not spend the rest of your life following the norm
and take the chance to change the world and change our environment
thanks for watching and if you have any questions feel free to contact me on twitter or linkedin.