ARINDAM: Hello, everyone,
Welcome to Setting Up for Success
with Exchange Online Learning Path.
This session is aimed to help you
to easily establish Exchange Hybrid Mailflow.
My name is Arindam,
and I'm an Exchange and Office 365 BETA Engineer
with Microsoft.
And I'll be hosting today's session for this topic.
So here is the quick agenda,
We'll discuss about Hybrid Mailflow,
various Hybrid Mail routing Scenarios,
and Connectors involved with Hybrid routing.
We'll also discuss about certificate based Security
and Message Attribution.
We will learn how the Hybrid Mailflow actually works
and where to look at if it doesn't work as expected.
So let's start with some basics of Hybrid Mailflow
and understand why it is important.
If you have mailboxes in both
Exchange On-Premises and Office 365,
you will want the seamless look and feel
of a single Exchange Organization.
And our Hybrid Configuration wizard provides you
with a streamlined process
to configure a hybrid deployment between
On-Premises and Exchange Online.
When you run the HCW to get that consistent experience,
it configures many things in the background
and in that process,
it also creates and configures
the Connectors for Hybrid Mailflow
between On-Premises and Exchange Online.
When emails are exchanged to these connectors,
these emails are validated,
and the successful validation make these messages
considered as an internal.
Hybrid Mailflow also helps to preserve
the organization headers
and provides Certificate based security.
When you configure Hybrid Mailflow,
it also offers you various mail routing possibilities.
Like, accepting inbound emails on Exchange Online first,
possibility to utilize EOP and ATP protection filters,
centralized mail routing to name a few.
Moving forward while planning Hybrid,
one of the most important considerations,
may be to plan your email routing.
So let's discuss some of the Routing Scenarios.
In Hybrid, our recommended way
to configure Mailflow is pointing MX to Office 365,
because this allows you to utilize
our EOP and ATP protections to its finest ability.
In this Hybrid Routing, you can route incoming messages
to Exchange Online by pointing MX record to office 365.
That inbound mail flow to On-Premises,
follows the the right arrow,
while outbound mail flow follows the green arrow.
Now, some organizations might have a business
or regulatory need to route all messages through
their On-Premises server.
For that requirement, you can point your domain's MX record
to your organization's email server, instead of Office 365.
So inbound email first delivered to
your on-prem environment following the red arrow,
and then it is outed to Exchange Online for mailboxes,
which are already migrated to cloud.
So in this scenario, the on-prim server
has the responsibility to filter the inbound messages.
I want to point out many of our UPN ATP filters
does not work properly with this setup.
And we recommend using enhanced filtering for connectors
to overcome this limitation.
I will discuss more about enhanced filtering
in one of the upcoming slides.
If there is a hard business or regulatory requirement to
route all messages through On-Premises
as an alternate to previous scenario,
customers can utilize something called
Centralized Mail Transport.
With Centralized Mail Transport enabled,
you can point your domains in MX record to Office 365,
Which will route the inbound messages to On-Premises first,
before delivering locally.
This follows the path depicted by the blue arrow
in this picture.
Outbound emails from office 365
will also go through On-Premises server
before sending out to internet,
which is shown with yellow lines here.
On-prem mail routing to and from internet
follows the green and red line.
And this set up provides optimal stuff spam filtering
and protects your organization because the UPN ATP
protections are fully utilized.
You can enable Centralized Mail Transport,
or CMT while running the hybrid configuration wizard
by selecting this,
"enable Centralized Mail Transport" checkbox.
Although, we always recommend that you point your
MX record to office 365,
we realized there are legitimate business reasons
to route your email somewhere else.
A common scenario resembles the diagram here.
In this example, the MX record points to
third part filter, the message goes there first,
it's processed and then sent to Exchange Online mailboxes.
However, with this setup, when the message comes
into Exchange Online protection,
UP beliefs that the third party filter
is the source of the message.
This isn't a limitation of Office 365,
it's simply how SMTP works.
And that creates issues with how UPN ATP filters,
causing unexpected filtering decisions.
To avoid such unexpected results,
We have a feature called Enhanced Filtering for Connectors.
Enhanced Filtering for Connectors allows UP to overlook
or skip your internal or trusted IP addresses to find the
last known external IP address of the message.
This previous IP would be the actual source IP
or IP of the message.
I have provided the link of the article here,
and encourage everyone to go to that link
and learn more about Enhanced Filtering for Connectors.
Now that we understood various mail routing scenarios,
let's understand different Connectors
that are involved in the routing.
So there are two sets of connectors involved here.
For Mailflow from Office 365 to exchange On-Premises,
there is an outbound connector from Office 365.
And on the On-Premises side,
the email would be accepted by
the Default Frontend Receive Connector.
In the other direction, for routing email from
On-Premises to Office 365,
a SIN Connector is created
called Outbound to Office 365 in the On-Premises side.
Also an inbound connector on Exchange Online side will
accept the email from On-Premises servers.
All these connectors are either created newly,
or existing connectors that are modified for a new run
Hybrid Configuration wizard.
We'll discuss about each of these connectors
in details in the following slides.
As mentioned earlier, the existing
Default Frontend Receive Connector
accepts email from Office 365.
And when you run HCW, this existing connector
is modified to make it enabled to
properly accept Hybrid Mailflow from cloud.
So if you go to the HCW log, you will see
Set-Receive Connector Command being
run under Hybrid wizard,
and a couple of parameters are modified.
The first one is TLS Domain Capabilities,
which would be modified to accept cloud services made.
And second, TLS Certificate name
will be set in the format of Issuer
and Subject name of the third party trusted certificate
that you have selected for Hybrid Mailflow.
HCW will also create a new send connector called
Outbound to office 365,
that sends email from on-premises, to Office 365.
I want to call out some of the important parameters.
First one is,
Address Space of the Connector that matches to
the target address of the Office 365 remote mailboxes,
which is domain.mail.onmicrosoft.com.
Next is Cloud Services Mail Enabled parameter,
which should be set to true or proper Hybrid Mailflow.
Also the TLS Certificate Name parameter,
which again will be in the same format of Issuer and Subject
name of the trusted certificate.
It's important that the certificate name
or subject alternative name on the certificate
matches with the accepted domain of the tenant
for proper message attribution.
On Office 365 side, a new inbound connector
of type on-premises will be created to accept
emails from on-devices servers.
Cloud Services Mail Enabled parameter will be set to true,
and TLS Sender Certificate name,
will match your accepted domain.
Also, an outbound connector
of type on-premises will be created
to send email to on-prem.
The HCW wizard will set the Smart Host .22
On-Premise server based on your input
while running the wizard.
Cloud Services Mail Enabled parameter would be set to true,
just like all other connectors created by HCW.
Now, once you finished running Hybrid Configuration wizard,
how do we know that all the settings
are properly configured as part of the process?
Assume that wizard worked as expected,
there are a few quick mail protests that you can do to
understand if Hybrid mailflow working correctly.
First, you can send it across premises email,
the sender's name should be resolved to the discipline
Also CrossTenant Auth Header should be set as internal.
You should also be able to see various organization headers,
like Message Directionality or X-Originating
or Processed-By-Journaling Agent,
is correctly set and preserved after
the message has traveled across devices.
Hybrid Mailflow also includes,
Certificate based Security and Message Attribution.
Message Attribution is the process of routing
the Inbound email to the correct Office 365 tenant.
to who is the message actually belongs to.
For On-Premises to Exchange Online Hybrid email,
this happens when the Certificate ame or Subject
alternative name configured on the
on-prem send connector matches with
the tenant inbound connector of type On-Premises.
And the properly run HCW ensure
these two matches correctly.
And this is how the magic happened in the background.
Here, I'm sharing the Protocol Log of
an email sent from On-Premises to Office 365,
using the Outbound Office 365 Send Connector.
The On-prem server saves the certificate to Office 365.
The certificate name or SAN of the certificate is then used
to look up a matching tenant inbound connector,
that are of type On-Premises across all tenants
within the EOP forest.
From all matched connectors, we next look for tenant
that has that TLS Sender Certificate Name domain,
configured as an accepted domain.
The most specific match will win.
If one is found the message sent to the tenant
and also attributed as originating from the tenant
and treated as internal.
Now let's discuss the opposite direction.
Email sent from Exchange Online to On-Premises.
Here also, we take the help of certificate matching
between the EXO Outbound Connector
with the On-Premises Receive Connector,
to achieve Certificate based security.
HCW again ensures these two matches correctly.
Here I'm sharing the protocol log of
Default Front and Receive Connector,
for an inbound email from Exchange Online to
On-Premises.
After the certificate been exchanged,
Exchange Online send XOORG SMTP Command extension,
with "MAIL FROM" Command.
On-Premises server checks if the Receive Connectors
TLS domain capabilities is set to
accept Cloud Services Mail.
It also checks if the XOORG domain mentioned with
"MAIL FROM" Command matches that
On-Premises accepted domain.
If everything matches, On-Premises server will
treat the connection as authenticated
and will promote cross premises headers to org headers.
"X-OriginatorOrg" will be set
to your verified domain in the EXO.
Now let's discuss field troubleshooting scenarios
and what to look and why to look
if Hybrid Mailflow is broken.
Our first scenario is Hybrid email send from
On-Premises to Exchange Online are treated as anonymous
or looping between on-prem and Office 365,
when centralized mail routings is enabled
This can happen when the Certificate based Authentication
fails between On-premises and Office 365.
And if you'll check SIN Connector protocol log from
On-premises, you might see
domain validation or CRL validation failed
because of Revocation Offline.
Revocation Offline error can happen,
if your firewall rules are preventing the On-Premises
server to reach Office 365 CRL distribution points.
So make sure your On-Premises server is able
to reach Office 365 URLs and IP addresses,
which also includes CRL distribution points.
In case you're using a proxy server,
make sure the machine account has permission
to use the proxy server to ease
the Office 365 Certificate revocation lists.
Our next problem is Inbound email from internet,
which is routed through on-prem
is treated as internal on Office 365.
And if you treat email from internet as internal,
it will bypass many Exchange Online protection filters.
If we have restricted groups that accept email
only from authenticated users
inbound email from internet would get
delivered to both groups as well.
Now, how, why this might have happened.
This usually happens when
there is an on-prem receive connector with,
"Externally Secured" option enabled.
Normally, if the email has entered to
an externally secured receive connector,
we treat that message as internal
and from authenticated source.
And the same information will be passed to Office 365
because of the Hybrid Mail Routing.
An evidence that you probably hit this scenario,
is when you find the Auth Mechanism header set to 10.
To resolve this issue,
make sure the On-Premises server
is not accepting inbound email from internet
through an externally secure receive connector.
Another problem we have seen is,
internal message sent from on-prem to EXO
being marked as "Anonymous",
and you have checked everything but found
no configuration issue at your end.
If this happens, probably you hit an issue
we call "attribute to wrong tenant".
If another tenant has inbound connector
with same value of TLS Sender Certificate name,
or send the IP address as yours,
the email could attribute to the other tenant,
instead of yours, causing wrong attribution.
If the other tenant is owned by you,
make sure both tenants have different
TLS Sender Certificate name configured.
But if you do not have control
over the wrongly configured tenant,
you need to open a case with us,
and we will contact that administrator of the other tenant
and solve the issue for you.
The last scenario we are going to discuss today is
SPF Failure when sending Outbound emails from On-Premises.
So here On-premises has two Send Connectors,
one just for Office 365
also known as Outbound to Office 365.
And the other one to send mail to internet.
But you might encounter SPF failures when sending to
domains, which are hosted in clouds.
Now this is another type of
"attribution to wrong tenant" problem.
If the two send connector from On-premises,
one going to internet
and the other going to Office 365,
has the same TLS certificate name,
which by the way is very common,
the message will be attributed to your tenant,
before sending to that recipient tenant,
following that red arrow.
So you need to include EOP on your SPF.
This is the reason why we always recommend to include
spf.protection.outlook.com on your domain's SPF record.
Although you might be sending email to internet
directly from your on-prem server.
Now here are some more troubleshooting tools
that you can use to solve
the Hybrid Mailflow issues.
Message tracking, will you insights
of how the message has traveled,
and if the message is attributed
to the correct connector or 10th.
You can use Get-Messagetrace
or Start-Historicalsearch to get the details.
From on-prim side, you can use Protocol Log
to see various SMTP commands,
and if there is any failure,
like we showed you during our discussion.
You can also use important message Headers like,
X-MS-Exchange-Organization AuthAs
MessageDirectionality or X-OriginatorOrg
to understand if things are working as expected.
So that's it for establishing Exchange Hybrid Mailflow.
To watch more in our series,
please check out aka.ms/MicrosoftIgnite2020/SuccesswithEXO.
And here are some other great sessions
that you might find helpful.
Thank you so much for your time and have a great day.
Goodbye.