Hello and welcome.
I am starting a new series of videos
all about working with data.
Working with data in JavaScript, in particular, using
the p5.js JavaScript framework.
But, of course, this topic is much bigger
than any one library or framework or programming
language.
It's really about, there's this world out there
and there's lots of stuff.
You could get that stuff and use it in a project.
And why might you want to do that and what kinds of results
can you get with that?
And really, I want to focus on how to do that.
So what I have here is, kind of, a quick demonstration.
I would love to just sit here and show you
100 amazing and interesting projects made
with data on the web.
But I'm just going to show you one, which, I think,
relates to what the kinds of things
I'm hoping to demonstrate in these videos,
is this project called the Wind Map,
it's by Fernanda Viegas and Martin Wattenberg.
And you can see here, there is a map of the United States.
There are these white streaks flowing across the map.
And you could imagine, there's some underlying data, latitudes
and longitudes, wind speeds, wind direction.
And somehow, that data is being read
and then the data is being interpreted and manifested
through drawing.
So thicker white lines for higher wind speeds,
lines in a particular direction for a particular wind
direction.
And so, what are the building blocks of a project like this?
What are all the pieces?
How do you read the data?
How do you cycle through the data?
How do you draw with the data?
And what if the data comes from this thing
called an API, which requires you to log in and have,
like, secret key, and all of this horribly
scary, strange stuff.
I would like to make this seem like a very easy and welcoming,
friendly thing for you to be able to do.
That's my goal here.
So in this first video, what I want
to do is make a list of the kinds of ways
you might find data.
And I've got a few examples over here.
But I think just to get away from the computer for a second,
I'm going to move over here.
Hello again.
So let's think about this question.
So first of all, I've got to go back to the computer again.
Let's say you're interested in a particular kind of thing.
Like you like flowers.
Flowers, that's, like, your thing.
Oh, boy.
I really should have planned for this in advance.
But let's say flowers is your thing.
So you might google, "flowers."
And then you're going to get all this commercial stuff trying
to sell you flowers.
Well, I really just want to know how to grow flowers.
I'm really off topic.
I don't care.
How to grow beautiful flowers.
OK.
Whatever.
So I could go here be like, oh, look here's
this website about flower gardens
and I want to get some data about flowers.
I'm not really getting anywhere.
I could be a much better googler and I
could say, "flower database."
No, forget database.
"Flower data."
Ah!
Flower data sets.
Now I'm getting somewhere.
The point of what I'm doing is actually just
to be sort of random here.
This will happen to you.
You have an interest in a piece or kind
of data and googling around and you're finding things.
What are you going to find?
So first of all, I think there's two approaches to a project.
One is you actually have an interest
and you're looking for a particular kind of data.
But right now you might say to yourself, I'll do that later.
I'm just going to use some existing examples of data sets
that are out there that you, me, that I
am going to show you how to do.
But there is a reason for talking about this.
So you're going to come across and you're
going to see a lot of things.
So first of all, the data could come in a variety of formats.
Here's a list of some formats that you might see.
CSV.
What is that?
XML.
What is that?
JSON.
What is that?
You might just find a text file.
You might just find a PDF.
Or you might actually not find any of these formats,
but you see the data right there on the web page.
I mean, if that data is there.
How do you get access to that if it's not
in one of these quote unquote "formats."
And then there's this side discussion of,
what if you see this word "API"?
So what you're looking for, when it's working with data,
is you want to find somebody on the internet who says,
I have this data.
It's about flowers and I would like you to have it.
Please enjoy it.
And I prepared it for you with examples and documentation
to make it easy for you.
And a particular keyword, in the content
of what you're looking for, is you might be looking
for data of the format JSON.
The reason why you might be looking for this
is, JSON stands for a JavaScript Object Notation.
That might be convenient, right?
Because we, you, I have spent many a day
making videos about JavaScript Object Notation.
Because I could say, here's a flower.
And its name is sunflower.
And its color is like--
I don't want to use the-- it's fine.
Whatever.
You get the idea.
I could put a variable for its color.
I could have a variable--
I went off.
I was thinking about flowers and I lost my train of thought.
So this is JavaScript Object Notation.
It's how you write objects into your code.
It turns out that this data format
is exactly the same as this.
Now there's a little bit of nuance there
of some slight differences.
But if you find data that's been prepared
it's like somebody said, are you programming in JavaScript?
Would you like data made for you to use?
Here it is.
So this is an ideal scenario.
You're happy if you find data in this JavaScript Object
Notation.
And the next video I'm gonna make
is essentially all about what that is and the actual syntax.
Now another format of data that you
might find very intuitive and familiar to work with,
is this format known as CSV.
Now CSV is a particular kind of a larger format, which I really
mean tabular data.
So what does it mean for data to be in a table?
You have name, age, gender.
Dan, 42, male.
That's how old I am.
Let's just get that out there on the internet.
I'm youthful at heart though.
And then there's some other data there.
I could put my phone number in there
and I could say my phone number and make this video even more
embarrassing.
But you're familiar with this probably--
whoops, that's the wrong thing I was going to open,
like Google Docs, that's tabular data.
And in fact, you can have a Google Spreadsheet
and access that Google spreadsheet
in your JavaScript program.
And in fact, you can tie that to a Google form.
So this is a topic that I want to explore.
The reason why I'm mentioning CSV as a particular format,
is that what a raw format for tabular data
is to have all of the cells separated by commas.
So you can see an example of this.
Here is a table of comma separated values.
The first row being the header, "x, y, diameter, name."
The next row being some values.
And in fact, this is a p5 sketch.
That kind of loads that and using the data,
puts the things on the screen.
So at some point I'll demonstrate
that in more detail in a particular video dedicated
to just CSV.
There's lots of CSV data out there.
Here's one that I think is interesting to look at.
Right here.
This is Citi Bike data.
Citi Bike is a bike sharing system in New York City
and they have all of the trip data.
So the date, the latitude, the longitude,
the address, where it was picked up, the time.
It is a huge data set.
How would you load that data set, read that data set,
maybe draw lines of people riding
their bikes all over the map of New York City.
So this is the kind of thing that I
want to explore in these sets of videos.
So that's really what I've got here with CSV data and JSON
data.
Now XML is another standardized format for data.
These three up here are really standardized format.
What I mean by standardized format
is, you the human being want to look at data
in a nice presented way.
Like if you're interested in movies--
I don't know why I'm plugging some IMDB here,
but you could go to this website and be like,
oh, look at all this data here.
And I can see it and I can soak it in
and I can find the movie that I'm interested in and yay.
But this is not useful for a computer to read.
This is designed, visually, for you to look at with your eyes.
And you could make the same argument
that this Google Doc has actually
done that as well with the column
headings and colors and lines of grids, all that sort of stuff.
But the point of these data formats,
if I come back over here, is that these are all
designed for computers to read.
There's commas separating everything.
There's this programming syntax that
has kind of got curly brackets, and semicolons, and quotes,
and commas.
So it's the standardized format.
If you can find a standardized format, that's a good thing.
P5 has a particular set of functions
that allow you to read JSON data very easily,
read CSV data very easily, read XML data in a slightly
trickier way, but it works.
But you might also be in a scenario,
at some point, where you just found
the data not in a format that's meant for a computer to read.
Like there's just a text file with a lot of stuff in it.
Or there's just a web page and it's all there,
but it's like a mess and it's not
in a standard format or some PDF document.
So, at some point, I would like to make one video
to talk about how you might deal with some scenarios like that
if you find a particular piece of data online
but it's not handed to you in this nice, friendly JSON way.
And then, of course, there's this question of API.
Now if you have a fixed data set,
just like, this is the data.
It might come as a file.
But if there is a huge data set or a data set that's
changing over time, like Twitter is a data set that's
changing over time because people
are posting to it continuously.
All the articles that appeared in the New York Times
in 1950, that's not really changing over time.
I don't think.
Unless was could do time travel.
But anyway.
You could download that and have that.
So an API stands for Application Programming Interface.
Meaning, you're writing an application,
somebody else has an application,
that other person says, oh, I would love for your application
to talk to mine.
So I'm going to make an interface for those two things
to talk to each other.
And that interface might be sending
data in one of these formats back and forth.
So the tricky thing is if, just to use the New York
Times as an example, the New York Times has an API,
it's very nice of them.
They're like, hey, we have this newspaper and all this stuff
and why don't you read it like human being
or why doesn't your computer program read it.
And we'll make it easier by giving it to you in an API.
But you've got to log in, get an account, get a secret key,
figure out how to query the API, how
do you say I want the articles from this year.
Or that have this certain key word in them.
And so that's the kind of thing we're gonna do.
And just to run through a few examples of that.
For example, here if I'm interested in flowers.
This is me querying the New York Times Article search--
didn't try these examples to see if they worked.
Let me try a different one.
Let's try this one, I'm going to search for the word "computer."
This is very sad.
I have to go look at why the New York Times one didn't work.
But one of the APIs that I'm gonna
spend some time showing you is an API called WordNik.
Let me move this over here.
Where you can see here I've put in the word "apple,"
and I've click this button called Examples.
And WordNik has given me back a set of example sentences
that have the word "apple" in it.
This is Google Image Search, I passed it the word "apple"
and I'm getting some--
I like flowers better.
Let's go back to "flower."
And you can see I'm getting a whole bunch of flower images.
So this API, meaning Application Programming Interface,
Google has a server.
I have a program.
Those two programs are talking to each other
because Google has this API built.
But the point is, I can't just download all of Google's data
as a big CSV.
I've got a query that API.
So how does that work?
Instagram's another one.
Flower.
Whoops.
Let's just turn that off.
Google Sheets, we talked about.
OK, so that's kind of the basic--
I don't know how well this went.
It's too late now, because you've already
watched this video if you're watching it now.
But the next video that I'm gonna make
is only really going to be about what is the JSON format?
How does it work?
How do you pull the data format of JSON into your project
and do something with it.
So that's what I'm going to do in a few short minutes,
in the next video.
Thanks for watching and see you soon.
OK.
Wait, I need.