[Dr. Wilson] So let's now start talking about what is the deep web.
Because your web server can say a few things.
As well as saying "here's your page"
it can also say "I don't know what that page is."
So if I say...
Can we go to "nottingham.ac.uk/freeStuffForMax"?
It goes: I'm sorry I don't know what that is.
Page not found. We don't have "free stuff for max".
It can also say you're not allowed to go there.
So it can say if I want to go to
"nottingham.ac.uk/hr/increaseMySalary.html"
It will say "sorry you can't go there" that's a bit of the web you're not allowed to see and
that's really what the Deep Web is is bits of the web that google is not to
see or can't find or hasn't found and so the easiest way of looking at the Deep
Web is it's just a huge portion of the web which google doesn't have the
password to go and see your hasn't come across yet or is sitting unconnected to
the rest of web somewhere
there's four bits
of what makes the deep web or four nice examples, let's say, of what makes the
Deep Web different from the web
one is that stuff behind passwords so any place that you go we have to type in
the password is a place that Google can coax it doesn't have your password so it
it can't look at your bank information, it can't look at your profile page on amazon.co.uk
and in fact if you make your profile on facebook not public
you essentially pulled your profile on facebook down into the deep web rather
than having it on the surface bit of the web and but all the normal pages like
the main page of facebook.com is on the normal web the surface web that google
can see it just simply can't get X it doesn't have to probe the password that
you have
there's also lots of dynamic content so this is the second part content which
gets generated based on what you do or where you are is that
and so if you go on to amazon.co.uk and search for something Google hasn't
seen that search results because it hasn't typed in every single possible
word you could ever type into amazon search
so you see the products. It's found most of the products, so it's indexed all those
pages because it has links to on the front page of amazon but it hasn't seen
all the dynamic things that you create because you typed in a word that it had
thought typing in and so this is dynamic content that it can't access and so your
facebook feed is another type of dynamic content which is always changing and so
it's A) it can't get there because it doesn't have the password and B) the last time i looked
it was different to the time you just looked so hasn't seen it
there's also something called the robot exclusion standard which is a nice thing
that's most websites have a little file which is robots.txt on the server and
that says to google or other things are browsing automatically you can look at
those pages but please don't look at those pages and Google simply just
doesn't look at them because it was asked not to. So there's large portions that the
website designers say please don't index this but you can look at that
[Brady] Does that just rely on trust though
[Dr Wilson] Yes. Yeah, so you could write a spider that works like Google spider which just
ignores robots.txt but you'd be somewhat breaking the rules by doing that. Um...
[Brady] Is that a legal thing, or just...?
[Dr Wilson] I don't know is the fair answer
It could be... it could just be a politeness thing just like
most standards on the internet. And so then the four type of stuff
that's in the Deep Web is just stuff that nobody's ever linked to it, so it's never
been found as this google spiders have never followed a link to find it and
therefore it's just it's there but Google hasn't seen it so it's not that
it's hard to access is just so deep that nobody's found it
so my laptop has a web server on it HTTP server and so i can host web files on my
laptop and when I connected it to the Internet
it got a unique address and so people can type in that address and then see the
web pages that are on my laptop and so technically kind of anybody could do
that and then you just tell your friends
the unique kind of code for how to get there.
Not the web link because you didn't create web link maybe and your friends type in
the 1.128.193.274, go look at your laptop and then never tell
anybody else about it because your little secret web page for you and your
friends and so that's just a part of the web that Google never came across and it
won't come across it until either you tell Google about it or someone links to
it from another web page that Google's seen. And so all this is deep web stuff and
which is perfectly normal stuff that Google just doesn't have access to or
hasn't found and so that's the best way to look at the Deep Web is just all the
parts of the web that Google can't just walk up and see, or hasn't come and
visited yet. So how big is the Deep Web is an interesting question
Google's index for the normal web or surface web went past
1 trillion pages not too long ago that was a kind of milestone it went
past
so Google knows about one trillion pages but latest estimates of the Deep Web
that it's ninety-five percent whereas the five percent that google has seen is
there's one trillion pages so it's much much bigger 500 times bigger and that
sort of makes sense because ninety five percent of facebook is stuff you look at
after you've logged in with your password so and there's only a small
portion that Google can see which is before you log in and and there's whole
websites which you just need a password to access forums or academic literature
or just huge resources of content that google doesn't go and see
and so it's very very big it's just not very scary and not very unusual
it's just behind passwords. So this takes us then towards the dark web
because we now know what the internet is and the physical connections we got the
web is which is just software for transferring information and we know the
Deep Web is which is just the bits as the web we have seen so what is the dark
web
so the best way to look at the dark web is that it's just the web but built
differently so it's not surprising really and then the web was just some
software that someone built in this describes how people share information
and so you could rebuild that and put it so saying get that page you could have
built it to say send me that page
you know it's just asked a different question use a different code speaking
it's like a different language and so there are several dark webs on the
internet they sit on the internet and overlay the same internet like the
web does
they just talk different languages they have secret handshakes and and they
encrypt things in different ways and they store websites in different places
and you have to look at you have to go somewhere else to find out where it is
and so a good example of a dark web is tour which is the onion where they call
it then you go to . onion pages rather than . com pages
you know it's just different language different codes and they the way you
asked for a page is differently so ever
where you might have in to Chrome and say I'd like to see that page about
horses which is on the dark web and the dark web goes or would you can get page
about horses
that's not how we do things here so you kind of have to say the panda has orange
shoes and then suddenly gives you that page about horses that you wanted the
interest is just talking a different language is essentially transferring
pages of information like the web does
you just have to ask it the right way so does not mean that all the people who've
got websites out there would have to rebuild their website as an onion type
we're only if you want to be only on the onion whip and so you can be browsing
the interweb using Tor which is the onion browser and that we can look
perfectly well a normal web pages you can go to Google on it but it also knows
how to or where to find out where the onion physical computers are and then it
knows how to ask for it and and it's the other thing it does there was something
is it sends you on a secret strange and unusual route through the internet to
get there
so if you look at google.com on Tor but when i did this morning it was in German
because the last place that I came out on the strange the route was in the
computer somewhere in Germany and this is sort of why they call it the onion
web because it has many layers that you go through
so when it asks to view a web page it wraps up that request into several
layers of encryption and then send it to the first computer which unlocks the
first layer and says okay thanks now i know where to go for the second one goes
to a computer in Australia and says unwrap the next layer and then it goes
to a computer German accent and that one asks my google have come to see it and
then it kind of gets wrapped back up again comes all the way back to you and
which is why they call it the onion and web for the onion browser
it sounds like a kind of huge game of pass the parcel mixed in with one of
those old nineteen eighties films where somebody's trying to trace that
complicated really clear phone call or something
I'm gonna bounce just call through nine different relay stations throughout the
world and our two satellites this designed to be more anonymous and more
private and what you're doing so it's also a bit like a secret spy who's kind
of got into a building suddenly and taking off his coat and hat he's wearing
an apron and then walks into the kitchen and seems perfectly normal in there and
then as you get out the back of the kitchen he takes off the apron is in
overalls and then suddenly drops down into the sewer and he kind of goes along
a bit in there is a overall is looking like a sewer here and then pops up the
top takes off his overalls in a perfect suit ready to go for a meeting somewhere
to get a package and then has to do the reverse to get back in
so it protects you and anonymity by encrypting everything and lots of layers
sending in an unusual routes that you would normally expect and then avoids
being sort of seen on the
more servers and then on the way and by doing that
so yeah so basically that's why I into your analogy that come interact with
just the normal world or you can talk to the spies is that a good way of thinking
of it
yeah it's it can do either the secret stuff that's on Tor. I can do the normal
stuff that's on the normal web and so people have to say all the dark web
that's the place where bad things happen and that's not what it's designed for
the fact that it's anonymous means that bad things can happen on it more easily
but you can do good things on it when you have more anonymity so it's slightly
safer perhaps to go and do your online banking in a public cafe
if you encrypted first and there is a HTTPS which is secure HTTP and that's
one way of making more secure but then you can also wrap it and onion layers
and send often so suddenly you're sort of more safe in a public place and doing
something that you want to do more privately
so is it dangerous is a question of what use your anonymity for rather than that
it's specifically designed for dangerous things because people anonymous they can
do things which are less legal perhaps. I don't know there's some known flaw of
the onion web and there's ways of tracking what happens if you carefully
synchronize events that happened the different stages and you can also if you
view things like Flash over the Tor browser that breaks all the ideas of it
because it's this built-in different way so when you login to tourists and to say
don't use Flash because that will give away your location but if you do this
normal stuff that will be fine, and but there are people who use it to put on
pages so those things like the silkroad which is people knew about which is a
place where people could sell and buy drugs because and you were more
anonymous so that people could host the website more honest me and all the
people could go and visit in on
anonymously. So it's illegal activity that's been facilitated by anonymity
there's lots of positive things that could be facilitated by anonymity so
people who are in an oppressive areas can browse the web or anonymously
they can communicate with people more honestly and so on
so let's bring it all back together and talk about the difference in the Deep
Web the dark web the web and so on so and the Deep Web was just an area which
is fairly normal of the web or just behind passwords or does not found and
the dark web is a part of the deep work for us it's a different copy of the web
really which specifically tries to hide itself was talking secret code is
wearing camouflage and its hiding as much as it can be different and to be out of
the way away from google and so then just makes it private
so the physical internet of computers is kind of physical connections or
computers together that have IP addresses each one of those physical
computers can have a web server on it so that it can talk in software and share
information and pages and so this is the web
the Deep Web is just part of it which people can't get too easily and then the
dark web is like a copy of the web that talks in just the different ways
different languages
start with the basics of the weapon internet and because most people confuse
those as well as confusing the Deep Web in the dark where and how the different
and we're just going to try and figure all that out today