Unknown: Hi everyone. Today many web sites offer third party
access to their data through web API's by manually including URLs
with arbitrary endpoints parameters, authentication and
pagination, among other things makes API use challenging for
programmers and impossible for novices. In addition, each API
offers its own distinct data model properties and methods
that a user must learn even when the site's manage the same types
of data as many others. In this work, we built a client library
that allows users to interact with API's data by directly
manipulating standard local objects in their own programming
environment without invoking any API methods. In addition, we
simplify building applications over API's data without any
programming at all. We do so by designing a standardized machine
readable ontology that describes almost all web APIs as a
collection of standard typed objects. Consider the scenario
where a user who tends to search for restaurants on Yelp and
Foursquare to compare their ratings and reviews now wants to
build one application that would allow them to search for
restaurants on both sides at the same time. Now, this seems like
an impossible task for a non-programmer to do. But for a
programmer like this user, it seems like a doable task. This
user knows that both sites offer APIs that allow them to access
these sites' data. So what would it take for the user to use
these APIs and build such an application. They first go to
the API documentation, then they write code to make an HTTP
request to the business Search API endpoint, then they need to
specify the parameters, set up authentication, they get the
result, then they iterate through those pages to fetch
more results, than they get the result as long as the JSON
response. So they need to write more code to parse the results
to finally get the exact properties that they need. But
they realize that the reviews are not returned by this
endpoint, because Yelp requires you to make another HTTP request
to get the business reviews. So they write code to make that
HTTP request, and finally they get the reviews. Well, they're
not done yet. Because now they need to write more code to
connect these two endpoints where for every business, they
need to make an HTTP request to the reviews endpoint to get its
reviews. And now they get the exact data that they want. So
they take their Yelp data, and they plug it into an HTML
templating language that reads and displays an object's
properties. And now they have an application that reads Yelp
data. Well, they're not completely done with Yelp,
because now they need to write more code to make this
application dynamic. Meaning reads from an input and search
Yelp API and displays the result. And now they're done.
Now the user need to check the Foursquare API. And they're
thinking, Oh, we got this, foursquare is pretty much like
Yelp. So its API must be the same. Well, they're pretty much
wrong. They start with the Foursquare API by writing code
to make an HTTP request to the venue search endpoint, they are
shocked to realize that this endpoint does not return most of
the properties that they need. They go over the API
documentation. And then they realize that they need to make
another HTTP request to the venue endpoint to get the rest
of the properties, then make another HTTP request to get the
reviews. And finally, they write more code to connect these
different endpoints to get the data that they need. Wow, what a
tedious process for getting just 12 properties. Now, they take
their Foursquare data and replace the Yelp data in their
application with the Foursquare data, and whoops, the
application will not work. Well of course, it won't work because
Yelp's data has a completely different schema than
Foursquare's. So the user will need to write code to
consolidate these two different schemas, then write more code to
search these two sites at the same time. A system that we have
previously built called ScrAPIr helps minimize some of that
human labor for Yelp, and similarly for Foursquare.
ScrAPIr allows programmers and non-programmers to access
individual API endpoints without writing code. But ScrAPIr among
other things does not help connect these endpoints. And it
certainly does not integrate data from different APIs, which
leaves non-programmers unable to build such an application. An
investigation we conducted on web APIs revealed that these
APIs can easily be clustered based on the types of data that
they offer. And although these sites are semantically similar,
technically, they are not. And we saw that with Yelp and
Foursquare, their APIs work differently, and their data
models look completely different. Now Yelp and
Foursquare, believe it or not, are fairly simple API's. They
have a couple of types, and they only allow us to read their
data. To get a better understanding of how APIs work.
Let's quickly look at another set of similar sites which offer
write as well as read to their data. Let's take these three
video sites. These APIs offer an average of 14 types and an
average of 130 API endpoints. Looking at one of the types that
these sites offer, which is playlist, there are so many
things that can be done with the objects of this type, we can
read and updated a playlist's properties, we can create a new
playlist and remove an existing one, among other methods, and we
can also search these sites for playlists. To be able to perform
these simple tasks with these three APIs. We need to write
code that performs an average of nine HTTP requests, connect
these requests together, then consolidate the data models of
these different sites. Semantically, many of these
API's are object oriented. Many endpoints accept an ID
identifying a particular object, there are endpoints for creating
and deleting objects and for reading and updating some or all
of the properties of an object. However, the semantics of these
operations are not exposed by the inputs. Instead, they can be
found in the human readable documentation. Users must read
the documentation to learn which endpoints manipulate which of
the various site's objects. So can we expose the site's data to
users, as no different from the data a user would work with in
their own local environment? In particular, can we present the
data as objects of different types, each with specific
properties that a user can get and set, which can be created or
deleted, and possibly with methods that can be invoked on
objects of a givn type or on the site? Even more ambitiously, Can
we interact with the playlist's objects in the same way across
different sites? Well, given that these APIs are not
standardized,
it might be possible to reach our goals if we can standardize
the descriptions of these APIs. So we built Shapir, which
consists of a standardized API description that we called Wo
PI that maps the API's ob ect to standardized data ty
es and methods. Given a WoOPI de cription Shapir can auto
atically generate a client lib ary that presents the site'
data as objects in the app ication's local environme
t. Shapir integrates with Mavo, library that empowers non progr
mmers to create interactive app ications simply by authoring
TML. And finally Shapir p ovides a graphical interface tha
permits any user even a non-p ogrammer to describe any API u
ing WoOPI. To be able to desig WoOPI, we needed to deter
ine how general such a descripti n would need to be in order to
capture the diversity of cur ent web APIs. So we picked a
random sample of 40 web APIs rom the ProgrammableWeb, a cata
ogue of more than 24,000 APIs. e then analyzed these APIs to u
derstand what was common among them. We found the
ollowing commonalities. All of these APIs offer objects of
various types, all identifie by ID. They have one endpoint
er type that accepts an ID and returns the object's basi
properties, generally those t at are small data types. They
also provide other endpoints or accessing each large property
of an object such as a vide file, or an array of associated
objects, and these ar among other commonalities that
can be found in details in our paper. Based on these findings
we designed WoOPI a machin readable description that wo
ld permit automatic gener tion of a library that present
APIs data as a collection of ty ed objects in the local enviro
ment, such that reading or mo ifying the objects' prope
ties would automatically rigger the invocation of appr
priate API endpoints to erform the requested modifica
ion on the site. To achieve our first goal WoOPI built
n HAAPI description, from our previous project ScrAPIr, in
hich even non programmers can au hor. HAAPI provides API en
points' URL, parameters, schema f response data, pagination, au
hentication, and WoOPI map an API of any methods an
parameters as described by HAA I to a normalized API of type
objects with read, update, cre te, delete, among other me
hods. WoOPI also supports tandardization of objects over m
ltiple APIs. WoOPI describes how to map property names and obj
cts returned by the API nto Schema.org standard types
nd properties. Schema.org is a s te that offers a common set
of schemas for describing ob ects and their properties on t
e web, such as news artic es, discussion forums, and eve
ts, among other types. To veri y that WoOPI does support most
eb API's, we picked another ran om sample of 60 APIs from the Pr
grammableWeb, and found that 90% of these APIs can be fully desc
ibed by WoOPI such that a JavaS ript library which we call Shapi
JS can read WoOPI description an automatically generate object
and methods to query these APIs and manipulate their objects.
Now with ShapirJS, users can r ad and write properties of t
ese objects and invoke methods o them or on the site as if t
ey were local. And the library t kes care of making the n
cessary API calls to provide o modify the relevant data. And g
ving that WoOPI maps to obje ts to standard Schema.org t
pes. It permits application w itten using ShapirJS over one
site to work unchanged over any other site, providing sem
ntically equivalent data. To all w novices to build app
ications using ShapirJS with ut writing any JavaScript code
We integrate ShapirJS with avo, which is an HTML templ
ting language that reads and w ites objects properties. Using
Shapir with Mavo, HTML autho can build applications to searc
one site's API or multiple APIs offering the same types of data,
allowing our user to build his application that search
s both Yelp and Foursq are at the same time withou
writing any JavaScript code Users can also filter the data
given the set of param ters supported by these APIs in
addition, Shapir-Mavo su ports getting objects from th
se APIs by their ID. The sim le application reads a pla
list from Dailymotion and pla s its videos. To allow users to
create the WoOPI description we build ShapirUI, a graphical int
rface that lets any user eve non programmers to describe the
e APIs. In this demo, we are howing how a user can use Shap
rUI to describe part of the D ilymotion API, specif
cally its playlist of videos First, the user enter the UR
of the site or the API. Then S apirUI if permitted by the user
will generate a prelimin ry schema using some heuristi
s that maps API's endpoints an field on the right to the Schem
.org types and properties on he left. The user can then add a
d remove types and properties.
And they can also describe the search functionality to let
other users search the API for videos, or other types. And they
can also describe how to create a new objects and delete
existing one by connecting these methods to the appropriate API
endpoints. When the user is done, they can save the
description and ShapirUI will generate ShapirJS and Mavo code
snippets for the user to copy and use in their application. We
evaluated the effectiveness of our system through a series of
user study. For ShapirJS, we conducted a between subjects
user study comparing our library to the popular Swagger API
integration library. Participants who used ShapirJS
completed their task with an average time of 8 minutes and
five out of six participants who used Swagger Client completed
their task with an average time of 45 minutes. For the
Shapir-Mavo user study we recruited 16 participants 8
programmers and 8 non-programmers and the average
time that they both spent to build an application interacting
with multiple APIs was only four minutes. Thanks for listening
and feel free to use our tool