In this problem, we're going to build on some of the JavaScript lesson
we did in the previous one,
and we're going to handle identifiers, numbers, and strings.
Real quick, let's ask ourself, what are the differences between those?
So we have an identifier, and as we talked about it in lecture,
identifier is a name for something, usually a function or a variable.
We then have numbers, and we're going to be able to handle numbers such as
-1, 2., 3.5, etc.
Then we have strings.
This string contains the word, "Blah, an escaped quote--so the string contains this double quote,
Blah, question mark--I don't know if you heard my raised pitch at the end. Blah?--
and then another double quote, and it ends.
We indicate the end of the string with unescaped double quotes.
So we want to write regular expressions to match these things
because that's how we're going to write the functions that tell the lexer how to tokenize it.
So let's do the identifier first.
This one's pretty straight forward.
We need at least 1 letter, and it can't be an underscore,
but the casing doesn't matter.
After that, it can be as long as we want, and it can have as many underscores as we want
and then letters as well where the casing doesn't matter.
So that's our regular expression for the identifier.
For the number, it might be a little trickier.
Let's say we have this number--542.555.
We're going to write a regular expression to match this as well as all the other numbers,
but this has all the pieces that we need to consider in a regular expression.
The first thing is the minus sign.
The minus sign can only occur at the beginning, and it's also optional.
We then can match at least 1 but possibly more numbers up into a decimal point
if it exists, but it also can just end there such as the -1 we have here.
So the numbers [0 - 9], and it has to be 1 or more of them.
This goes there. This matches that part.
Now we're going to have a chunk that's optional.
If we go down this route, we have to have a period,
and we have to escape the period because it has meaning in a regular expression.
But after the period, we can only have numbers, and there can be 0 or more of them.
0 for the case in where you just end in a period, which is often denoted to mean
you want a floating point.
But in this case, we're going to treat everything as a floating point.
And then a star.
So that's a regular expression.
To go back, the dot goes here and these 5's go there.
Now we just want to do strings.
We have a regular expression.
The first thing and the last thing we have to match are going to be the double quotes,
and I use a single quote because I don't have to escape the double quotes in this,
and that makes it nice.
In between the double quotes, we can have 0 or more characters inside of them.
Those characters can be in 2 groups.
It's either escaped--like we have here--or it's just a character by itself--all the other letters.
So let's do the not escaped first.
That's going to be anything that's not a double quote or a backslash.
We have to escape the backslash if we want to match that,
so anything but those.
So it's going to be everything here, all of this, but not these quotes right here
because there's a backslash or a quote there.
If we get any one of those, we're going to match a different rule
where we have to have a backslash, but after that backslash, we can have any character,
and that's what the period means.
And I forgot to put my question mark. There we go.
That's the regular expression that we're going to use.
As you can see, as these regular expressions get more complicated, doing more complex things,
they get messy. Messy--very, very quickly.
Sometimes it's really conventient to write a regular expression like this.
Sometimes you might just write a bit of code to do the same thing.
But in this case, regular expressions are super powerful and super fun,
and that's what we're doing here.
Here I have my solution.
I'm going to have 3 types of tokens--
the identifier, the number, and the string, which we went over.
I have the identifier, which is exactly the regular expression we came up with.
All we do is identify that type.
For the number, when it matches, we're going to call float,
which converts the string value that is in the token when we get it
to an actual floating-point number.
And then for the string case, we're going to strip off the quotes and just get the contents
of the string itself--
that's what matters most--
ignore whitespace, have to deal with new lines and the errors as well.