In this video, we are going to explore some techniques for troubleshooting email problems
in PI Notifications 2016 R2 or later.
We will walk through a troubleshooting example where a user did not get their daily notification
email as expected.
We are using a simple architecture where:
PI Notifications Service 2018 SP2 is installed on PIClient01 and
The SMTP Server is installed on PISRV01
Let's say we have a daily alarm notification that is configured to send a notification
every day at 6:30 AM.
We can see here under the trigger pane that this notification is triggered by the '6:30
AM Alarm' event frame generation analysis.
Under Subscriptions > View/Edit Subscriptions, we can see that the notification is configured
to send an email to one subscriber - student01 at the start of the event frame generated
by the '6:30 AM Alarm' analysis.
From the analysis tab, we can check the logic of the event frame generation analysis.
Based on this logic, a zero second event frame should be generated when the analysis is triggered
at the specified schedule of 6:30 AM.
This type of analysis logic where the start trigger condition and the end trigger condition
both evaluate to true is often seen when the purpose of our notification is to be an alarm.
The start and end triggers are both set to true in this case for simplicity.
However, It could be any logic where both start and end trigger evaluate to true.
We can find the event frames generated for this element by right-clicking on the element
> Find > Event Frames.
These are all the event frames generated by all analyses under this element
When a notification is sent for an event frame, the notifications service will create an annotation
on the event frame with information such as the number of subscribers the notification
was sent to, the send status, send time, etc.
This annotation will have a description that will tell us whether a notification was sent
successfully or not.
We can check the annotation description for by right-clicking on today's event frame > select
properties > click on Annotations Annotation description is not a default column
so we will have to add it by clicking on this gear icon and selecting Description.
We can expand the column for more details We see that the send status is "Failed".
This means that the PI Notifications Service attempted to send an email but failed.
So how can we troubleshoot this?
Before starting the troubleshooting process, we first need to ask ourselves what changed
in our system recently?
The answer might lead to the root cause of the issue and save us a lot of time.
If you don't know what changed, then the first thing we need to do is check the message logs
around the time that the notification email should have been sent.
Check message logs: Let's check the logs from PICLIENT01 since
the PI Notifications Service is installed on that machine.
The PI Notifications Service logs are stored in the %ProgramData%\OSIsoft\PINotifications\Logs
directory The archive folder stores the logs for the
past seven days by default.
If the occurrence we are troubleshooting was in the past few days, we would want to check
the messages in those files as well.
However since our issue started today, we'll focus on this one.
By default the log level is informational.
If there are not messages in the log about the failure, we might need to increase the
debug level to "Trace" on the PI Notifications Service.(Steps on how to do that provided
an article linked in the description below).
We can see in the logs that the PI Notifications Service failed to send the email because it
could not reach the SMTP server at the time the issue occurred.
Let's send another test email to verify whether or not the issue is still happening
Send a test email: We can send a test email using the email delivery
channel plugin in PI System Explorer.
Go to Contacts > Tools > Delivery Channel Plugins
Right-click on email > select settings Here, we see that the primary SMTP server
is PISRV01 and we connect to it through port 25.
We don't have a backup SMTP server so if we fail to connect to the primary SMTP server,
we will not have another SMTP server to fallback on
Another important parameter is the send email.
While it would not cause the error that we've seen in the logs, the send would fail if it
were specified in incorrect format.
Click on Test…
We will send a test email to student01 Alright, looks like this fails the same way
This indicates that there might be something blocking the connection
Let's make sure that we can connect to PISRV01 through port 25 from PICLIENT01.
Verify that firewall port is open: From PICLIENT01, using Windows PowerShell,
we will run the following command to verify that TCP port 25 is open on PISRV01.
If this returns a blank output, that means the port is open.
Otherwise, an error will be thrown.
Running the command returns an error.
This indicates that we cannot connect to PISRV01 via port 25.
Perhaps an admin enabled the windows firewall on PISRV01 where the SMTP server is installed
and did not create a rule to allow connections from all essential applications.
At this point, it would be good to involve your IT department as most users don't have
access to SMTP servers.
I happen to know that this is a Firewall issue so let's go to PISRV01 and open the port.
Open port: On PISRV01, open Windows Firewall with Advanced
Security We can see that the domain firewall is on
Let's create a rule that allows inbound connections via port 25
If the port was open on the server and still couldn't connect via port 25, it could mean
that there is a network firewall blocking the connection.
In that case, please contact your IT department to further investigate.
In this case it is a Windows firewall so we will go ahead and create a rule for port 25.
Under Actions, click on "New Rule…"
We'll select the Port radio button > Next Under Specific local ports, we'll enter 25
> Next For the purpose of this video, I'll leave
the rest of the settings as default.
Consult with your IT department for specific recommendations at your company
I'll call this rule SMTP Server inbound port Let's go back to PICLIENT01 and verify that
the port is now open
Verify resolution: From PowerShell, I'll re-run the same command
to test the connection to PISRV01 through port 25
Alright!
It looks like it's successful now From PI System Explorer, let's test send another
email Alright!
That was also successful Finally, we can recalculate our analysis from
today and verify that we received a notification email
Backfill analysis: To backfill the analysis, right-click > Backfill/Recalculate
Since event frame generation analyses cannot be backfilled, the automatic selection will
be a recalculation.
This will permanently delete event frames in the time range selected and create new
ones Click on Start
Alright, now that it has completed, let's verify the annotation description
Looks good!
From Microsoft outlook logged in as student01, we can see that the notification generated
as a result of the recalculation has been received.
This concludes this troubleshooting video.
I hope you enjoyed it!