Hi everyone my name is Tarfah Alrashed. I'm a PhD student in computer science at MIT
In this presentation I will be talking about a tool
that we built to democratize access to web APIs
Back in 2017 The Economist published a story
where they described data as the oil of the digital era
this is not really surprising knowing how data can really
improve our lives. From researchers and students
to support their scientific research
to journalist with the emerge of data journalism
oil is a great metaphor because like oil
data also need to be extracted but from the web
accordingly, tons of efforts have been made in the
last couple of decades to make data open
for anyone and everyone to access, reuse and share
there are many data portals that allow access to
thousands of datasets in specific domains like
government data and there are generic search engines
that allow users discover datasets on the web
like Google Datasets Search
these data portals and search engines are great
but the web is full of interesting data that goes
beyond datasets that can be discovered and accessed
through these data portals and search engines
for each one of these websites there are unlimited
number of queries that can be run on it which
results at an unlimited number of datasets
these data are valuable but they're definitely not open
not for everyone at least
let's take NASA as an example
NASA is known for their efforts
to make their data open for the public
they created multiple data portals and they
work closely with Google Dataset Search to make
their datasets accessible through their search engine
but NASA has way more valuable data than
what we can find and access through
their data portal and search engines
so how can people access this valuable data?
In a survey we conducted with more than a 100 people
we wanted to understand people's experience with
retrieving web data. What sites data that they're
interested in, how do they access this data,
and the challenges that they face with that
Some people said that they get data manually
by copying and pasting the data this would
obviously only works with small-scale data
otherwise it is tedious and a waste of time
another way that they used to get data
is by scraping web pages
many scrapers do require programming
there are libraries and frameworks
for scraping that evolved over time
and there are a number of recent work leverages
programming by example (wrapper induction) and
programming by demonstration to learn not just from
the content but also from the steps a human demonstrator
takes to extract the content that they want
these tools don't work with all websites and they
continue to wrestle with errors inherited in such learned
approaches, which could arise from inadequate or
inaccurate examples as well as the frequent
changes to web content presentations
while programmers can and do
use these non-programmer tactics they also find
ever more sites offering APIs
and compared to scrapers APIs provide a more reliable
way to access and retrieve clean web data since they
do not rely on parsing complex web page structures
many APIs also offer advanced searches and
provide more extensive data that can be accessed by
scraping web pages
but they do require programming
let's quickly refresh on what an API actually is
API stands for application programming interface
which is a set of functions that allow programmers
to create applications that access data
offered by other web services
and although not all websites offer APIs,
web APIs are definitely increasing
In 2016 the ProgrammableWeb
which is an API directory had 13,000 APIs
now it has more than 22,000 APIs
and most of them are RESTful APIs
in addition from our survey 80% of participants
wanted data from websites that offer APIs
let's go back to our example of NASA
NASA does provide an amazing API that allows
programmers to write code to access their data
but none of the current online data portals and
search engines allow easy access to these
APIs data by everyone. So what are
the challenges of making these web APIs accessible?
from our survey we found
a number of obstacles with APIs
in order to access data through web APIs
people need to write code making them inaccessible
to non-programmers, and even for programmers
APIs can be hard to learn and use as there is no
standard way that they're designed and there's
a great variability in the quality of the documentation
in addition most modern web services return
structured data in JSON and XML formats
which can contain large and complex hierarchies and
are difficult for people to parse and paginate through
given that some web APIs offer more rich data
than what we can access through their sites
then what we really want is a query interface
that allows everyone to access and query these
APIs data and hides the complexity of these APIs
So we did. we built a query interface
that we called SNAPI
that allows people to query and retrieve
desired data form web APIs
in this demo we're showing how a user
using SNAPI can query and filter
YouTube Search API to retrieve its data
see this is great
building a query interface for a couple of APIs
could be manageable only by programmers of course
but the reality is there are thousands of web APIs
and these APIs are different
in addition both programmers
and non-programmers need their data
creating a query interface on top of every API
is just unreasonable
a solution to this problem would be to automate the
process of generating a query interface on top of every
website that has an API such that everyone
no matter what their technical background is
can query and retrieve the APIs data
well it is true that APIs are different but many
of them are assembled from a small palette
of possible design choices and the solution
is to describe those design choices in a
machine readable fashion so our query interface
SNAPI can read that description and use it to
generate both a suitable query interface
and an appropriate API call
so in the absence of a single standard API
it might be possible at least to create a standard way
of describing the many APIs that are in use
early efforts include WSDL and WADL both written XML
more recently API blueprint, RAML
and OpenAPI description have emerged
these API descriptions have been used primarily
in two ways. First to automatically generate API
connector code in a variety of languages
and second to automatically generate
documentation describing the API to developers
but while these improve the developer experience
they offer no value to non-programmers who still
face the barrier of writing code to access APIs
and parse the results
so we designed an API description that we called HAAPI
HAAPI extends the OpenAPI specification that
has emerged recently as part of the Swagger project
OpenAPI provides much information necessary
for SNAPI but is not sufficient
OpenAPI does not specify information that is
necessary for creating a UI to query the API
like human readable parameter names
and since it is assumed that the code
will be integrated into a larger system
that uses the API invocations
OpenAPI does not specify information
about how to authenticate to the API
or paginate to gather large number of results
which we support in our API description
when a web API is described with HAAPI
a query interface can automatically be
generated on top of that API
and although writing this description could be easier
than writing code for programmers at least
it does not really guide users and specifically
non-programmers to describe and access APIs
so we built another tool that we called WRAPI
that empowers a user to author
the HAAPI description for an API
simply by filling out a web form
WRAPI can guide even non-programmers
with little or no understanding of APIs
to unpack the necessary information
from the API documentation
these three components form ScrAPIr
a repository of HAAPI descriptions and UIs
for querying and authoring these descriptions
using scraper, any user can first use WRAPI
to author a HAAPI description of any API
once this is done, just once per API
any other user can use SNAPI to query this API
and download its response data
ScrAPIr thus empower users who want data
to standardize and access data query APIs
without demanding any cooperation
from the sites providing these APIs
in this demo I'll be showing how a user
can access YouTube Search API using ScrAPIr
WRAPI guides the user through a number
of steps to provide the necessary information
to access an API
to begin the user provides a YouTube Search API URL
which can be found in the API documentation
the user must then describes relevant query
and authentication parameters in the API
to assist users ScrAPIr gathers information
about common API parameters by searching
GitHub and the API Guru description repository
and display them in a table
this minimizes the effort of looking
for the parameters in the API documentation
ScrAPIr also provides a real-time feedback from
querying the API to help the user detect errors
in this case the user needs to provide
a value for the parameter
another message will show up asking the
user to sign up and get the API key value
then the user needs to enable that API
under their Google account
once the user provides all the required parameters
the response goes green indicating that the user
has successfully access the API and the API's
returned results will be displayed on the right pane
the user can then add and remove parameters and
they can hide unimportant parameters from other users
some APIs require authentication
and/or header parameters
in this example the authentication is the key that
was provided as one of the parameters here
so we can skip this section
from the response the user can then
choose the field that they want to retrieve by
searching for them in the right pane
the YouTube API returns five videos by default
to be able to get more results the user needs
to fill out the pagination section which is optional
the user only needs to provide the parameter name
and the type of pagination that the API supports
before publishing the API for others to use
the user can preview their integration
and once the user is done they can
connect that API to ScrAPIr
this API is now connected and other users
can access it and query its data
you might think that only programmers can use
ScrAPIr to access an API but as I mentioned
in the demo not only we help populate some of the
required information automatically for the user
but we also offer examples
and demos for each one of the steps to guide
those who are unfamiliar with APIs to understand and
find relevant information from the API documentation
one of our non-programmer participant was able to
add this YouTube Search API in just seven minutes
to evaluate our system we first needed to
evaluate how many of the web APIs can be
described by our API description HAAPI
HAAPI builds heavily on OpenAPI which
has already demonstrated broad uptake
in the developer community and makes only
limited extensions primarily to cover pagination,
authentication and presentation
we sampled the ProgrammableWeb
focusing on the search category
and found that 90% of search APIs
can be described by HAAPI
such that SNAPI can query them
of the 10% not describable by HAAPI
one used a different style of pagination and two
required passing parameters as part of the URL path
which ScrAPIr now supports
but at the time of submitting the paper it did not
and that improved our coverage of the sample to 97%
we then conducted a user study to mainly
answer two questions about our system
can first-time programmers access and query
web APIs using WRAPI faster than writing code
which is a traditional way of accessing APIs
and can first-time non-programmers
access APIs using WRAPI
we recruited 20 participants, 10 programmers
and 10 non-programmers for a two hour user study
programmer users went through two sessions
one by using WRAPI and the other one writing code
where we counterbalance the order of methods, half
started with WRAPI and the other half started with code
while non-programmers only used WRAPI
participants were assigned one of 3 APIs or tasks
and these APIs are Yelp, Reddit and Google Books
all 20 participants completed their task using WRAPI
and 9 of the 10 programmers completed writing code
for the programmers who finished both
WRAPI and coding tasks the average time they spent
to complete their tasks using WRAPI was 7 minutes
versus 27 minutes by writing code
a factor of 3.8 times faster
and the average time that non-programmers spent
to complete their tasks using WRAPI was 19 minutes
which is faster than programmers writing code
our tool was able also to guide non-programmers
to access those API successfully
this chart shows the average time programmers
spent using WRAPI and writing code
and the average time non-programmers spent
using WRAPI on the 3 different APIs/tasks
because we expected that the time spent on the
first session would be highly affected by the time spent
reading and understanding the API documentation
while it would already be known in the second session
we decided to measure the time spent on
documentation separate from the time spent
using WRAPI and writing code
we got some really nice qualitative results
some of the common threads were that
our programmer participants after trying
to access APIs by writing code really liked
how they can easily find and extract data
without programming and how it's much
more scalable than writing code
for our non-programmer participants
they really liked how ScrAPIr helped them understand
the general flavor of APIs and how it made it easier
for them to navigate through accessing these APIs
like any other system ScrAPIr has some limitations
An obvious limitation is that it only works
with websites that offer APIs
but as I mentioned earlier web APIs are increasing
also some APIs have limitations that we can't really
control like requiring users to pay to use them or
limiting the number of results user can get at a time
ScrAPIr allows you to add individual API endpoint
but we're currently working on allowing it
to support connecting multiple APIs
we're also currently working on building a library for
ScrAPIr that allows programmers to access APIs
added to ScrAPIr directly into their code
which is something that our programmer
participants asked for
thank you so much for listening
and feel free to try our tool