Welcome to office hours for week 4.
Before we get into your questions,
there's a little issue I'd like to mention with regard to the homeworks.
Some of you have been asking about how to delete cookies.
The way I had intended for you to do this for the logout portion of the homework--
deleting cookies is how you log out--
was for you to basically just set an empty cookie.
Unbeknownst to me--but now I've learned--webapp 2, the framework we're using
that's built into App Engine, also has a delete cookie method
that you can use to delete a cookie.
And all that does is it sets the cookie to blank,
just like you would do by hand if you were manipulating the header.
Both should work fine. Delete cookie is probably a little bit easier.
But not all frameworks have that sort of thing built in,
and I wasn't really looking for it.
If you read the docs and you find that sort of thing, good for you. It should work fine.
Okay. >>Okay. So let's-- What have you got for me?
Okay. First off, we've got some questions on authentication
and when we should roll our own authentication libraries,
when we should use ones that are provided,
and if we could go into more professional authentication libraries like OAuth, for example,
a little bit in depth. >>Okay.
Authentication is a tricky issue.
If you get it wrong, it can be a painful process.
I told you that story about how I got it wrong during Reddit, and it was a painful process.
So the decision whether to roll your own or use what's built in,
it's almost the same thing, because if you're going to use what's built in,
you need to understand how it works so you can decide
whether you're better off rolling your own.
The one that's built into webapp2, I looked into it in response to this question,
and what I found is that they basically have almost the exact same algorithm
that we discussed in the lecture--
having the password, creating a random salt,
hashing the whole thing together and storing that.
By default they use SHA1, and you can specify whatever hashing system you would like.
I mentioned in the lecture if you have the option, you should use bcrypt.
Somebody else on the forums mentioned scrypt.
I'm not super familiar with scrypt. It seems like the same concept but a little bit better.
The whole idea of bcrypt versus one of the SHA hashing systems
is SHA and MD5 and those guys were designed to be fast,
mostly for message authentication,
and bcrypt is designed deliberately to be slow
so that if somebody wants to hash all of your passwords, it's going to take a while.
Now, with regard to things like Google's built-in authentication based on Google users,
if that fits your app, use it, by all means.
It's kind of a neat thing, but if you want users to have passwords and custom behavior,
you're going to have to build your own thing or use another library.
But if just logging in with Google suffices, yeah, great.
The next part of the question was about OAuth.
OAuth is a protocol for authenticating with third party websites.
I wouldn't use it if you can avoid it.
There's a successor to OAuth called OAuth 2 which works a lot better.
If you try to implement OAuth, you will be grumpy for weeks.
I'm still getting over it.
OAuth 2 is much more straightforward,
and it's basically, if you want to integrate with--
I think Twitter and Google has an OAuth 2 integration. Facebook does as well.
You basically sign up for them, you get an API key, some credentials,
and there's a little kind of back and forth message exchange
that basically says, "I am this developer, and you agreed to give me access."
So it's not just random access, and you don't have to worry about passwords.
It's a little bit more complicated.
There's all these tokens and timeouts and stuff like that to deal with,
but there are OAuth 2 libraries in Python that work just fine.
So long story short, use what fits your app.
You'll find yourself rolling your own very often.
At Reddit and Hipmunk we rolled our own both times.
But if the Google login works for you, great.
And if you need OAuth 2, good luck. [chuckles]
The next question we had was about database constraints
and people had some issues with user names not being unique
in the Google Datastore and how to enforce that
and when you need to enforce that. >>Okay, sure.
So the database consistency issue, that's a great question,
and it's been lurking in the back of my mind
ever since I started teaching you guys about Datastore.
There's kind of 2 main ways to approach this problem.
The first would be using transactions.
I'm not super familiar with how transactions work in Datastore,
which is why I haven't spent a whole lot of time on it.
And I'm not going to spend a whole lot of time on it now,
but the general idea is most database systems, Datastore included
and also PostgreSQL and MySQL and all the SQL ones,
have this notion of a transaction where you basically say, "Begin transaction,"
and then you can do all sorts of operations.
You can do some selects, you can do updates, you can do inserts.
And then you commit that transaction.
And if any of those queries fail for whatever reason, the whole transaction fails.
And so you can turn a bunch of operations into 1 cohesive unit.
So that's 1 way of solving this problem.
And actually, most of the SQL databases allow you to have constraints on the tables as well.
So you can just say, "This field is supposed to be unique."
Now, Datastore doesn't allow those constraints, at least to my knowledge.
So the way I would solve it in Datastore,
not understanding the transactions super well, is to use Memcached.
Memcached is an external system.
We'll be talking about it in Unit 6,
largely from the point of view of caching
so you don't have to hit the database on every request
because Memcached is very fast.
It's basically a giant shared key value store.
Memcached can be used to basically behave as a shared locking system.
So for the user concurrency issue, the way I would do it probably
is let's say you have 2 people trying to register the same user name at the same time.
The first case is that user already exists.
Both of those queries from separate requests are going to say, "Does this user exist?"
and they'll both fail because the user exists.
If the user doesn't exist, the problem can arise that both queries hit the database
at the same time and it says, "Oh, great, the user doesn't exist,"
and then allows 2 registrations to happen at the same time,
and then that user name gets inserted twice.
The way you would work around that is both queries hit the database,
return to the app server as false, the user doesn't exist,
and then you would have them both write to Memcached.
Memcached has an operation called Add which basically says,
"Put this item in the cache if it doesn't already exist."
And if they both try to add that user name to the cache at the same time,
one of them will fail because it's an atomic operation.
So they'll both try to add it, one will fail,
and he'll return to the user, "User already exists," or, "I can't register this name right now,"
and the other one will go ahead with the registration process
and then, when the registration process is complete, delete that value from the cache
or it will expire on its own--that sort of thing.
In summary, I would use Memcached.
You basically need some shared state between all of your application servers,
and that's one of the tricky things when you're scaling,
because the more shared state you have, the harder it is to scale.
And Memcached is a good place to store this sort of thing.
So I hope that helps.
Okay. >>What's next? >>Okay. So we had a question about storing static files,
storing data files that you're serving through Lynx or whatever on your site
and where to put them and how to organize them. >>Okay, great.
So the easy answer is just look at the code that I've put up
for the homework solutions 2 and 3 at this point.
Both have a static directory.
And in App Engine you can basically define app.yaml.
You can give it a directory specifically for this purpose.
In my case, I've just called it Static.
And then you can put whatever file you want in there,
and your URLs will be whatever you define in your app.yaml.
So mine is /static/ and then whatever file.
When you're operating on a bigger scale,
often what you do is you have a web server dedicated to only serving static files.
So you'd probably still put all of your static files under 1 path in your URLs.
Very often it's /static.
And then you have a whole subsystem designed that's optimized for serving static files
and caching them very aggressively and doing all of that stuff.
But for our purposes right here, for little stuff,
just look at the example code I have in my app.yaml for defining the static directory.
And then I just put files in that directory in my project, and it should work just fine for you.
Okay. So, possibly embarrassing but probably not, we had a note
that Select * is not the only thing that's allowed in Datastore queries and GQL queries.
So could you talk a little about that and why you mentioned that? >>Sure.
I mentioned in the lecture for Unit 3 that in Datastore
you can only do queries that are of the nature of Select * from 1 table.
And it turns out that by the time that lecture aired, that is no longer a true statement.
Google released an update to App Engine to allow you to do more complex select queries.
So you can select 1 column or multiple columns or *--whatever you want.
So that's an improvement to App Engine.
We're teaching a little bit to a moving target, so no big deal there.
If you want to use those fancy select statements, knock yourself out.
You don't really need to use them in any of the homeworks I'm doing in this class.
But Google App Engine gets a little bit better every day,
and this is an example of it.
A little bit of unfortunate timing, but that's how it goes sometimes.
The final question we had was about infrastructure
and how you go about planning how to build your infrastructure
and grow your infrastructure when you're designing your app. >>Okay.
Designing your infrastructure ahead of time is important,
and this is actually mostly what we talk about in Unit 6.
You don't need to scale ahead of time.
That definitely would kind of fall under premature optimization--
making things too complex when all you're writing is a simple blog
or ASCII Chan or something like that.
But it's good to have in mind the approach you're going to take
so you don't shut any doors that you may need to go through later.
Generally, what I do is-- So I've been working on Amazon for years now.
Both Reddit and Hipmunk use Amazon service.
And the way Amazon service works is you just rent machines from them--
physical machines.
You never get to see these machines, but you have access to them.
So we'll have a couple machines for our database
and a couple machines for our app servers.
Always whenever I have a database, I always replicate it at least once.
Especially an Amazon system, machines can come and go.
But even if you're running your own machines, a machine can fail.
A hard drive will die, bad memory, power failure.
There's lots of reasons you'll lose a machine.
So I always replicate a database at least once--usually twice--
so that even if we lose one of our machines, if we lose our master database,
even if the site goes down for a little bit as a result--
maybe we're not very good at automatically switching over to our slaves--
we want to make sure we don't actually lose the data.
There's a couple classes of failures.
Users will forgive you for having a little bit of downtime;
they're not so friendly when you actually lose their data.
So no matter how small a system I'm working on,
I at least replicate the database once.
If that's not a can of worms you want to open
if you're really running a small thing for yourself,
I'd suggest at the minimum at least dumping your database every day,
maybe every couple of hours, and uploading that to a third party service.
We would use Amazon's S3. Google has data storage.
Dropbox, downloading, it doesn't matter.
Just make sure you store that data somewhere else
so that if that machine dies, you lose it, it gets stolen, whatever, your data is safe.
That's usually the most important thing.
And then of course as your system gets bigger,
you can start being a little bit more thoughtful about how you store huge amounts of data
in a reliable way.
One of the advantages of using Google's Datastore is they do all that for you.
If you're running your own machine, you have to do all of this yourself.
Amazon is kind of in the middle where you're still getting your own machines
but you have this really nice infrastructure.
In the Datastore, you don't even really deal with the notion of machines and replication.
Google just handles it.
And that's one of the reasons we're using it in this class,
so you don't have to think about that stuff right now.
Okay. I think that's all for this time.
Anything you want to add or any comments you want to make for the upcoming weeks?
Sure.
I haven't seen a whole lot of questions in the forums or in the office hours
about Unit 4 stuff, so that makes me think you all either figured it out or quit.
Good job.
It sounds like everybody is doing well.
Thanks again to everybody that's been helping out in the forums.
Sean and I do the best we can in terms of trying to find questions that are easy to grade,
but you guys always come up with creative solutions that are technically correct
but then fit exactly what we're asking.
So thanks for being patient with us. We try to work with you.
And if you have those answers you think are correct
and our grading scripts don't think it's so,
it's perfectly reasonable--and you guys have been doing so--to just let us know
and we'll adapt accordingly.
Best of luck in Unit 5. That one should be a little bit simpler and a lot of fun.
So we'll see you there. >>See you.