(upbeat music)
Okay, and welcome to this month's GoingNative.
As always, I'm Steve Carroll, the Dev Manager
for the Microsoft C++ Team.
And with us today, we have...
Hi, my name is Ulzii, Ulzii Luvsanbat.
I'm the Engineering Manager for the Microsoft
C++ Compiler Front-end Team.
The team that's responsible for
lexing and parsing the C++ code.
Okay, great, so today we have a big announcement.
What is it?
So, today with Visual Studio 2017 15.7 release,
the Compiler Team's happy to announce
that the Rejuvenation Project
that we've been undertaking for the last six years,
we've had a big milestone in catching up
with the C++ Standard that we have been working on.
Right, it's the last major milestone on a long road
that we've been working towards.
So, we've talked about the Rejuvenation Project before
on the thing, I think we had Jim and Mark on
to talk about the details of that.
But can you give folks just a quick overview
of what we've been up to and why?
Yeah, so Rejuvenation Project essentially means,
we have a very old code base for the MSVC Compiler,
rightfully so, it dates back to 1982 or so,
without the comments aside.
And we never really had a chance to rewrite from scratch
in the last 36, 37 years.
And you still haven't. (laughs)
Yeah, and we still haven't.
But we always had to ship features based on this code base
and keep up with the C++ Standard, new evolving language,
as well as, demands and requests from internal teams
and customers around our Compiler and C++ features.
So, rejuvenation essentially means,
okay, let's take a look at this old code base that we have
and let's try to replace parts, if not the whole thing,
and modernize it in a way that we can accommodate
latest new C++ advanced features, as well as,
make it more efficient and make it bug-free going forward.
Yeah, I think the key thing that I really want
to drive home for people is the state of the art
of how to build and architect a compiler
has changed a lot over the last 30 years.
And so because we're still six years ago
using that architecture, there were all sorts of things
they required basically magic of very dedicated developers
to try to get them to work.
I think just like doing variadic templates
in the old code base was something like a miracle.
And we didn't want anymore miracles, we wanted to make it
a better place for us to get things done
and so that we could keep up.
Yeah, so, one clear example is
the compiler was written in a way that we couldn't do
Two-Phase Name Lookup, which unfortunately meant a lot
of the advanced C++ features depended on
this behavior of the compiler.
Because the code had been written that way, right?
Yes, so we said six years ago before Rejuvenation
and the actual thing that happened was
with Visual Studio 2015 R.T.W. release
we added a new fully balanced semantic tree
that replaced the old model.
For static analysis an AST tree
that was different from what the compiler
had internally generated code to the back-end.
And this is the thing that would've been like
if you use /analyze in the product.
You were hitting these code paths,
but if weren't hitting /analyze,
you probably didn't notice we did all that work.
Yeah, so that was a huge milestone in terms
of the Rejuvenation work.
There was little visible value to the customers outside
and the fact that many customers didn't even notice
that we did it, is the successful outcome, right?
But after that though, because we had
that foundation tree implemented inside the compiler,
we were able to focus more on creating
a syntax accurate tree, or AST, Abstract Syntax Tree.
Top down in order to support actually
enable Two-Phase Name Lookup
and all the other advanced C++ features.
And I think also, we completely rewrote the parser.
Exactly, so to do that we obviously could not
continue on with the old parser.
The old parser here is a YACC-generated parser.
While that was good enough to do a lot of things over
the many years, we needed to totally replace that
with a new recursive-decent parser in the compiler.
But the challenge was, we needed to do that in place.
So today, a lot of you don't know the two parsers
are actually running side-by-side
in order to support all these features.
I wanna really stress for people the fact that like,
we had to take that choice because of the fact
of how much code exists in the universe that requires all
of the quirks of our non-standardized version of C++.
So you really had to find a way that we could
chart through both keeping people's, customers' code
they also wrote in 1982 and expect it to keep working
every time they get a new version of Visual Studio
while still being able to add all these magical, new, fancy
features that are from C++11, 14, and 17.
It's like swimming in between.
Which brings us to permissive- I think, right?
Yes, so, as you just said, we have obligation
to many of our loyal customers and we thank you for that.
That we cannot be breaking them with new release
or under the notion of modernizing the Compiler.
Right.
So we had to be clever.
Deliberate. Different ways.
We had to find ways deliver value to customers,
new customers, as well as,
keep the legacy enterprise customers
or people who depend on the old behavior
without being burdened by these new features
or by the new compiler changes.
So in order to do that, one of the channels
that we created inside the compiler, or mode,
if you will say, is permissive-.
It's basically a way of saying, okay,
if you ever want a compiler to be more strict
and less permissive about non-standard features
that we have had in our compiler,
then here's a mode that you could turn on
and start writing standard C++ code
that's portable, that could be compiled by MSVC,
as well as, could be compiled by
Clang and gcc if you want to.
And so if you throw that,
you're getting mostly the new code
and if you're not throwing that,
you're sometimes still getting the new code
because our intention is of course to have only one.
Yes.
But yeah.
permissive- as I said, is one of the channels.
We've also introduced new channels for new features
based on the specific standards
of the standard feature of versions
and the standard version switches.
Those are still C++14, C++17.
So this way we could organize around for the new features
when they get implemented.
And in some ways matching the other compilers.
You get sort of it.
We're still catching up.
Mess of innovation, yeah. (chuckles)
But so that's great, alright.
So where does that bring us to?
Okay, so where are we today?
First of all, what should people expect to have working
starting in 15.7?
So, 15.7 is a good milestone.
It's an important milestone as we said before.
We expect our users, developers to throw the C++17 switch
and expect all the C++17 compiler language features
to be available under that switch.
Just to put the small asterisks on there.
In the libraries, I believe the character conversion things,
there are a few more to come in a future update.
As far as we know, those are not available
in actually anyone's library yet.
So that's good to know.
Yeah, which is why I'm saying compiler, the language.
So we'd be excited and it'd be a big announcement
if it were just C++17 being language complete module
of that particular library.
But there are also things from the old standards
that we're adding here, right?
Yeah, yeah, so the C++11 and C++98,
they have their features that we have been working on
and I've been shipping partial support in last few releases.
Those are Expression SFINAE and Two-Phase Name Lookup.
These are big behemoths.
We couldn't just do it in one release.
As I said earlier, it depends on a lot of the inner workings
of the compiler that we work on.
We're finally with this release saying
that we are caught up with these.
Of course we have a few bugs.
We are tracking those bugs, but we're happy to say
that we are done with these features.
Right, so let's talk about,
this is great, what's next though?
Like what's left to do?
What's left in our conformance backlog?
And then we'll get to what we want people to do for us next.
What's next?
What do you mean, what's next?
We're done! (both laughing)
Yeah, all the bugs are out, we're done!
It's wonderful, which means we'll be retiring
to a desert island, sipping out of
some sort of umbrella'ed drink.
It that correct?
So as we enable ourselves, as we modernize our Compiler,
we open ourselves into many more opportunities
and many more features.
Obviously, we are going to keep up with the C++ Standard.
We don't want to fall behind.
We want to keep up with them.
We wanna stay ahead of features that are released already.
Yeah, we got a lot of T.S. work to do.
Contribute to TS technical specifications, drive them.
An example is, we're already working on C++ Modules TS
in our compiler and we're going to continue working
through that and as well as, look into other TSs
that will be available in the next C++ Standard.
So let me play the role of the angry internet commenter.
We can do a little Q & A right now, okay?
So, what about the preprocessor?
What about the preprocessor?
(both laughing)
So yes, the preprocessor from our point of view,
it is also old and we are definitely working on re-writing
the preprocessor from the ground up as well.
And we want to follow the same model as we did
with the new parser.
Replace it in place without any impact
to the existing customers.
But I think also, like the other stuff that we did.
The preprocessor has enough weird quirks in it
that we'll have to be careful about
how people opt in and tell us that they need that.
So it may be one of these permissive- kinda situations.
Just to set expectations, I think, right?
When I say the same model, what I mean is exactly that.
We need to find a way that,
that we could carry both the new customers
and the old customers forward without hindering them
with any adoption blockers.
So the other question I get asked all the time
by people on the internet for instance, is like,
okay, but, you say that you've caught up on conformance,
but I still have such and such a bug.
So if people are seeing conformance gaps, like,
what should they do or what should they expect from us?
That's a good question.
I also used to be the test lead for the compiler Team
for I don't know, 10 years or so.
(both laughing)
So this is near to your heart?
Oh yeah, testing is a big deal to me and to our team.
Bugs, when we have bugs, we look at Dev-community bugs
for Compiler and those, we take them very, very seriously.
We want to fix them, we want to fix them all.
As much as we can.
So we have taken many positive actions.
Many different approaches in how we want
to validate the compiler in every list.
Just quickly, historically, we used to have these
built-in, in-house tests that have developed ourselves.
We also purchased third party tests
from Plum Hall, Perennial, McCluskey
and we ensure those are okay.
We built some big internal code bases
and we said this is good and we ship it
and it's been good enough-ish for the past releases.
But going forward, and last few releases,
we have focused more towards community code bases.
Open source stuff.
Yeah, open source libraries.
Projects that are open source.
The code bases that we've never considered
or never really built in-house.
We've only received bugs after we ship
and then we would fix it and it will go back and forth
three times until the project finally builds clean.
So we wanted to go after those.
And today we have two testing repositories
that we actively use on a daily basis.
We have vcpkg as you all know,
the build package manager.
That has what, six, 700 now?
I think we're at 900 now actually.
Wow, so 900 C++ libraries that target Visual Studio,
Build, and Windows, right?
Mm-hmm.
So those are built on daily basis.
At least on weekly basis with the development compiler.
So that we catch any regressions before
we actually release the compiler.
So certainly if you have open source stuff
you should add it to vcpkg so we can make sure
that we don't break it. Oh yes, definitely.
If you are in open source library owner,
a C++ open source library owner, and you want to ensure
that MSVC compiler in the future doesn't break you,
or vice versa, right?
The best way for you to do that would be
to subscribe yourself and add yourself to vcpkg.
And like for instance, we actually end up
making contributions when they are necessary changes
in the compiler and stuff like that.
Yes, so, just real quick.
Looking back, I would actually consider
and I would actually say that building open source libraries
in project from GitHub through vcpkg or
in our daily model. Custom one, yeah.
That has been the most valuable testing method
for compiler then our existing in-house test
that we have written.
Those are good baseline tests.
Yeah, they're years ahead.
It's just the language is evolving so fast.
There's new code just coming up left and right.
Keeping up with that, the only way to do that
is reaching out somehow reaching out the latest C++ code
and that to me, is vcpkg.
Got it. Okay.
Any other things where you want to tell people,
like any libraries that you want to call out
as we know we're working on them?
Oh yeah, yeah.
So we definitely want to go after some popular libraries
like Ranges, Range-v3, and the Ranges
for the standards one that's being considered.
The concepts based one.
As well as Boost Hana.
We've been working on last many releases.
We've identified where the problems are.
And with these Two-Phase Name Lookup expression fixes,
those are getting slashed.
We still have some stranglers that we have to fix though.
Not quite yet on these ones.
Yes and also Facebook Folly just came online.
That had never been built with MSVC.
And so okay, call to action.
So 15.7 at the moment that we are recording this
is in the Preview Channel.
Please by all means, download that.
If you have code, please build with it.
We are pretty confident that we have solid quality in that,
but definitely want to hear about it.
If we have any conformance gaps, let us know.
Add your package to vcpkg so we can make sure
that we keep it running as we go forwards.
Please, by all means, go to Dev community,
look for any bugs that you have.
And in particular, if you could up-vote ones
if somebody else's filed it, that will help us
do our prioritization work.
We're gonna fix everything.
We're gonna fix all the bugs,
but we are gonna do them in order.
Please help us use your up-votes to identify
the most important ones to go after first.
Yes.
Okay, and thank you very much.
Yeah, thank you.
(upbeat music)
(closing sound effects)