Welcome to Office Mechanics. On today’s show, if you've ever wondered how network
connectivity to Office 365 works, we’ll give you an insider's look at data center
coverage, we’ll demystify how you can optimize network performance by planning connectivity
to Office 365. And keep watching as we share top troubleshooting tips to help you and your
users get the best performance with Office 365.
I’m joined by networking expert Paul Andrew, welcome.
Great to be here. Why don’t we start by talking about some of the scale of the service.
Right, if you think about it, it’s not a trivial task to keep Office 365 up and running
with the constant level of networking coverage and performance
office 365 is a global service it’s built for customers all over the world.
And we’re currently available to 140 different markets and growing.
And each of those markets is covered by Microsoft data centers in that region.
So, for example, if you are a customer in the UK you'd be served out of the Europe region
which has data centers in Dublin and in Amsterdam.
Right, and so that answers the question right data center coverage but how about the network
itself?
Yeah Microsoft actually has one of the largest networks in the world even we compared to
the telecommunications companies and Internet companies and it's used for connections between
Microsoft data centers and also for more than 50 interconnections to the public internet
which are in 23 countries.
We actually have physical or logical routing agreements with over 1500 network operators
and ISP's. We also make use all content delivery networks and edge nodes to further improve
performance.
So that's a lot of connectivity that we have that we're managing but how do we actually
optimize that connectivity for customers using Office 365?
Right, so the important thing is routing packets over that network from a customer network.
We work with different ISP's and network operators to make sure that routing works really well.
We had our recent case where a customer in the UK was actually routing packets over the
Atlantic Ocean three times before connecting to our US data centers and so we were able
to work with the ISP to improve their routing and avoid that.
So it sounds like we’ve got a lot of infrastructure, lot of great reports but how do I think about
connecting to the service on the Internet as a user office 365?
We often think about connecting to the internet
as if it was connecting from your PC to the cloud
or connecting to the Wi-Fi access points in the hallway.
But of course that's not true.
There is a lot of infrastructure,
a lot of networking equipment out there.
So in reality
the internet is made up mostly of submarine cables
because as the Internet has grown
it's much easier to add submarine cables off the back of a ship
than it is to dig more trenches throughout the globe.
There are submarine cables around lot of different continents around
Africa, India and Asia Pacific
which is largely what makes up the internet.
So it sounds like a lot of things are outside of our control as admins for Office 365 but
I bet a lot of people are wondering what they can do as admins to really optimize performance
in Office 365.
Great, so there are a ton of things you can do as an admin. To start, when using cloud
services like office 365 your internet connection becomes much more important than before.
The first thing you want to do is a capacity planning exercise. You need to know about
how much new Internet traffic is going to be generated by using Office 365 and how that
compares with the available bandwidth that you have.
And we have tools for a lot of the office 365 workloads to calculate capacities so why
don’t you show us one of those.
Great! So let’s look at the tool for OneDrive for Business
Okay
So this is an Excel spreadsheet and on this first page you can see we’re setting up
some details about the environment that you have. We’re setting up a user profile for
lite users and user profile for standard users. And you have to plug in the average file size,
the average new files that the users generate per day, the average number of files they
edit, and the number files that are pre-existing for each of these profiles.
And we plug in a few default estimates so you can get started very quickly.
Yep, once you set those we go over to this client mix page, where you can set up for
each site that you have, in each location, specifying the time zone and the number, the
type of users that you there, how many Windows clients you have, Mac clients and mobile clients.
so if I go ahead and updates one of these numbers you can see that my calculations for
network bandwidth are updated. And this shows me an estimate of how much bandwidth I'll
need it also shows me the number of TCP connections and that's important for configuring your
perimeter network devices
And you can even look at things like traffic by time of day with this, right?
Yes so let's take a look as we scroll down, because we filled in those time zones for
those different offices its gonna estimate how much bandwidth users are using at different
times of the day so you can see how your network banwidth changes.
And these are just estimates. So something that you can you even, in terms of more testing
is really look at the actual bandwidth being consumed by users and monitor that traffic
as a baseline right?
Yeah, these estimates are definitely only as good as the numbers that you plug in at
the start. So we do recommend that you some networking equipment maybe routers that you
have or network packet trace tools to monitor actual bandwidth that is used by a pilot set
of users and then extrapolate that out to the full set of users that'll give you a better
estimate.
So we looked at capacity another thing that comes up a lot is latency. So how do we test
for latency in terms of our connections back into Office 365?
Right, so we’ll want to take a look at the network connectivity between users and Office
365 and let me show you a tool here for looking at the latency. We can take a look at the
latency using psping. Psping is a tool that measures the TCP connection to port 80. You
can see 80 there on the parameters that I'm using and it shows here that in my test I’m
and average of eight milliseconds between my client and Office 365. The next thing I
want to take a look at
Hold on what's the difference been psping and normal ping.exe before we
Oh well, normal ping is using the ICMP protocol
Okay
And we want to use the TCP protocol connect to port 80 instead because ICMP protocol is
often blocked by networking equipment it also may be routed all prioritized differently
over the network.
I see this is a great tool if you don’t have it in your tool belt already psping is
free from sysinternal, so it’s a great tool to measure
Alright, so let's take a look at one more psping command connecting to our proxy server
and so this will tell me how long it takes for my client machine to connect to my company's
perimeter network and you can see an average there of just five milliseconds. And you can
tell how long it takes from the perimeter network to Office 365 by subtraction, so we've
got another three milliseconds. These are obviously fantastic numbers that we’re getting
here.
When does it start getting troublesome cause we’ve got very fast connectivity here from
Redmond but what about people out in the world that might have 200 to 300 millisecond delay
times?
Right so if we're at 300 or 350 millisecond delays then you'd want to take a look at where
is that latency is it inside by network connecting to the perimeter or is it outside my network
where my ISP's a network operators are involved and then you can narrow in and take more of
a look at out what equipment you got on what's going on there.
Okay, so we’ve looked at things like capacity and also latency but I know another thing
is really throughput or how fast the network is actually flowing. Do we have a tool to
actually monitor that?
Yeah, absolutely so another common issue that we find with flow control is TCP window scaling
and so where your looking for issues, you want to take a look and see if you have this
issue. This particularly impacts large files being transferred over long distances.
I’ve got a ferry analogy, so if I think about a car it might have 1500 bytes the acknowledgement
is actually the ferry that comes back and with that I can get 40 cars on the ferry.
Remember that number forty it’s important. Then that goes out and maybe another ferry
comes back in that that acknowledgement can also take another 64k another 40 cars. That's
really tiny though because when I got scaling turned on, the ferry goes back out now scaling
turned on and I can actually do not 40 but 700 thousand of these cars because now I'm
getting a gigabyte per ack back for each trip that goes around. So these acknowledgements
that come back and forth I'm actually sending, instead of 64k, a gig per ack.
Right and this is really an artifact of the internet growing and the amount a bandwidth
that we have growing but the speed of light over fiber optic cable has not got any faster.
Right.
And so over these long distances we have a huge bandwidths we really have to have larger
buffers and have TP window scaling turned on.
So what tool can I use to tell whether or not scaling is turned on?
Great! So we can look at their very specifically using a packet trace tool like Microsoft Network
Monitor 3.4 that I have here. And here I have a capture of a TCP connection being initiated
and we've got the syn packet and the ack packet and if I zoom in here you can see that in
the ack packet from the TCP connection it says here literally scale factor not supported
and this is indicating that either the client and or the server that I'm connecting to or
more likely some piece and networking equipment in between is blocking TCP window scaling
from being able to be used.
Okay.
So what I'd like to see is, if I go over to this other packet trace that I have, you can
see here the same syn & ack packet but in this case is says negotiated scale factor.
It’s showing me that the TCP window scaling was negotiated and we can make use of those
much larger buffer sizes that are available
Right so this is a really great in terms of telling whether or not your network infrastructures
up today but I know even beyond latency capacity throughput there are other issues that we
can solve for things like DNS and how we’ve architected our network right?
Yeah, so Microsoft has a global network with many Internet egress points a customers also
often have networks between their offices all over the world with multiple internet
egress point as well. So we want to optimize the connection between the user's PC, wherever
they may be, to our server between those two networks.
Okay.
So if you think about an example where the user is in Singapore and they're connecting
to maybe our APAC data centers which are in Singapore and Hong Kong you want to connect
basically directly to those using the different internet connections that we have and that
the customer has.
Right, cause sometimes the DNS might be set up in say North American and if those DNS
calls go all the way back to North America then back in the Singapore then it’s going
to cause a bit more delay right?
Right. We do this geo DNS thing and if you look up our IP address for our servers in
North America we're going to give you servers that are in North America and that means that
poor user in Singapore is going to do a round trip all the way to North America and then
internally on a network to connect back to the mailboxes in Singapore and Hong Kong anyway
so we want to try and avoid that situation.
Okay, so if I’m in Office 365 admin and really want to check for some of this stuff,
what tools can I use there?
Great. So it's really easy to check for this problem. So here I am on my PC and I've actually
created a VPN into a Microsoft Office in Singapore to replicate that example that we talked about.
So if I do a DNS lookup in Singapore down we can take a look you can see it's connected
me to a Singapore DNS server represented by SIN and these IP addresses that it’s showing
me they're actually in Singapore and Hong Kong they are part of our Asia-Pacific data
centers for Office 365. So then what I want to do is now that I've checked my office in
Singapore, I’ll disconnect from my VPN and by disconnecting from the VPN my machine is
now really it's in Redmond Washington in the United States.
Okay.
So immediately I can do that DNS lookup again and you can see I'm connecting to a different
DNS server this is one that is in the United States and I've got different IP addresses
that have come back and this DNS servers giving me server addresses back in the United States.
If I saw the same IP addresses for the same servers then I'd know that I potentially have
this DNS problem where i'm looking up in the wrong place.
Great, so we’ve seen between all the things we can do with capacity latency throughput
even looking at how we can isolate these DNS issues there's a lot of stuff that we can
you as an admin. Are there any more tips you for troubleshooting?
There's lots more of these common issues that you can check for you should check out our
blog and please use the blog also to let us know if you have questions about network connectivity
for Office 365
Of course you can stay up to date with all the news about Office on the office blog and
also on the Mechanics Wednesdays on Office Mechanics. Thanks for watching and goodbye
for now.