Troubleshooting SharePoint Alerts

A common theme running through the SharePoint newsgroups is the frustration of troubleshooting SharePoint Alerts such as when they stop working altogether or work intermittently. I am trying to organise a comprehensive step-by-step troubleshooting guide that should help solve the most common problems. Its a work in progress so if you know of anything that should be added then let me know by adding a comment.

Troubleshooting Flowchart 1) Difference between Confirmation Emails and Alert Emails. To make best use of the guide to troubleshooting why SharePoint Alert emails stop working its helpful to have an little understanding of what is going on under the covers. For our purposes Email Alerts can be split into two types

Confirmation Emails
When you first setup an alert a confirmation email is sent to the recipient of the alerts. This email is sent immediately from the Web Front End (WFE) serving the page at that time and running under the Identity that the IIS worker process (w3wp.exe) is using.

Email Alerts
The Windows SharePoint Services Timer service (owstimer.exe) is responsible for various jobs including sending out email alerts By default this runs every 5 minutes this means that even when you set Send e-mail immediately its not really immediately, there could be a short wait for your emails to be sent.

What does mean for Troubleshooting?

These two types of email are being sent at different times from different components of SharePoint and often using different Identities running on different servers. (Tip if you are brave you can check the TimerLock table to see on which server your various timer jobs will run) So just because you get the Confirmation Emails you cant assume that the Email Alerts will be work (and vice-versa) :

There may be a problem stopping the SharePoint timer job running that doesnt effect the confirmation emails. There may be a problem with the configuration of the alerts that only effects the alert emails Does your email infrastructure may not allow the emails to be relayed from the server the timer job runs on but does allow the WFE Does your anti-virus software stop emails being sent from the server/process (owstimer.exe) or even user identity that the timer service jobs run under but allows the confirmation emails

You get the idea! For troubeshooting purposes you should treat them quite separately.

2) Troubleshooting SharePoint email alerts Checking the Email Infrastructure

This part of the alert troubleshooting guide will talk you through the steps necessary to verify that emails can be successfully routed from your SharePoint server to the users mailbox. You should have first read the introduction so you know the difference between the confirmation Emails and Alert emails and understand why :

Even if you get one of those you may still need to verify how your email setup is configured. You need to check each WFE and your index server

You may also want to read this short SMTP primer if you are new to SMTP and related email technologies.

Sending test emails

First check the email settings in SharePoint central administration are correct (SharePoint Central Administration > Operations > Outgoing Emails Settings) note the Outbound SMTP server, From and Reply To addresses. Use these settings to send some test emails using a tool like the SMTP Test Tool or Telnet on your SharePoint servers to send some test emails. If you dont receive the emails you expect verify.

Does the SMTP server permit connections (relays) from your SharePoint servers? This post shows how you can configure Microsoft Exchange to accept connections from SharePoint

Is the destination SMTP server configured to deliver emails or relay them to another SMTP server and if so is it working? Is there anything that may block SMTP transmissions on port 25 (common to stop mass mailing worms/viruses). For example :o A firewall running on or in between the SharePoint server and the SMTP server o Antivirus software o Remember that if your security blocks SMTP transmissions based on process or account then test tools such as Telnet or SMTP Test tool will be a different process and will often be running under a different account to SharePoint and the timer service.

Does your SMTP server require a username/password or SSL? o In which case you will have to install something like Windows IIS SMTP server (locally on or another server), set SharePoint to deliver emails to this and then deliver emails directly or relay those emails onto your main SMTP server. This guide shows how you would set IIS SMTP server to relay messages from SharePoint to smtp.google.com as an example.

If you do get the emails you expect from these test tools

Verify that you havent got any anti-virus or other security in place that could allow these test messages through but still block the ones from SharePoint see Is there anything that may block SMTP transmissions above Go back to the flowchart and continue onto the next step

You are able to send test emails to some users, but not others
(Note this section does not apply to emails sent by SharePoint but only to test emails sent using a tool like SMTP Test tool or telnet see above) The good news is that you know in this case that its not a problem with SharePoint its time to bring in whomever is an expert on your email infrastructure but a few tips to get you started :

Is this users email is being delivered successfully from other sources, what about external email addresses such as hotmail / gmail? o This can help narrow down the problem between problems on the sending side an receiving side. Do emails to users on the internal domain get delivered and external email addresses fail (or vice versa) in which case you need to look in detail at your email infrastructure to determine where the problem lies. o If internal emails are getting through but external ones are not then use the SMTPDiag utility to verify DNS configuration on your SMTP server are not stopping external delivery.

As always the wildcards are SPAM filters and email auto processing rules always suspect these first! 3) Troubleshooting SharePoint Alerts Timer Jobs This part of the Troubleshooting SharePoint Email Alerts guide covers how to verify that the SharePoint Timer jobs that send these alerts are running correctly. If you are getting the Initial confirmation emails but not the Alerts emails then this may be the culprit. Its probably the least understood part of the troubleshooting process; reading through the newsgroups/blogosphere you will notice lots of posts explaining how something seemingly

random like disabling and enabling alerts fixed someones problem, who said Reboot? not another.

but

But hey, whilst its nice to understand why something worked sometimes we just need to get it working!

Check the Windows SharePoint Services Timer service (owstimer.exe) is running

If its not running try and start it. Check the Windows Event logs for problems any problems are almost certainly to do with the account its running under and permissions given.

Check the Timer Jobs Status in SharePoint Central Admnistration

SharePoint 3.0 Central Administration > Operations > Timer Job Satus Check for any columns where status != Succeeded and progress < 100% for clues. Check the last Started time is in the range you would expect (different jobs run on different schedules, check the Timer Job Definitions)

Use STSADM to verify properties

Note To add to STSADMs to paths use SET PATH=%PATH%;"c:Program FilesCommon
FilesMicrosoft Sharedweb server extensions12BIN" STSADM -o getproperty -url http://YourSiteURL -pn alerts-enabled You should see <Property Exists=Yes /> STSADM -o getproperty -url http://YourSiteURLpn job-immediate-alerts

You should see something like the default (but can be changed) <Property Exists=Yes Value=every 5 minutes between 0 and 59 />

Use STSADM to reset properties

Some users have reported that even though these properties are listed correctly above, simply resetting them solved the problem. Be aware, some but not all have reported that this can remove existing alerts.

stsadm o setproperty url http://YourSiteURL pn alerts-enabled pv False stsadm o setproperty url http://YourSiteURL pn alerts-enabled pv True

stsadm o setproperty -url http://YourSiteURL pn job-immediate-alerts pv Every 5 minutes between 0 and 59"

Reference for alerts-enabled and job-immediate-alerts properties

Use STSADM to re-register the alert templates

Again, some people have reported that this works with no agreement on the cause.

stsadm -o updatealerttemplates -url http://YourSiteURL -f "c:Program FilesCommon FilesMicrosoft Sharedweb server extensions12TEMPLATEXMLalerttemplates.xml" -LCID 1033

Where LCID is your languages Locale ID, 1033 is English US

Reference for updatealerttemplates operation

Other things to look at

See Steve Chen and harikumhs posts where they talk about checking through ULS jobs and the database tables that underpin SharePoint Alerts.

ImmedSubscriptions (Alerts for emails that are sent immediately when changes occur)

SchedSubscriptions (Daily or weekly scheduled alerts)

EventLog (Events for which only non-immediate alerts exist)

EventCache (Events for which users have requested alerts. WSS inserts events into this table as they occur)

TimerLock (Records the server that processes the timerjobs)

Of course if you get to this stage you may want to consider placing a call to Microsoft Product Support.

Further reading

KB942989 If you back up a SharePoint web application and then restore it t a new farm, some SharePoint timer jobs are not restored successfully now fixed in Post Service pack 1 hotfix 941422 Jan 31st 2008 and subsequent cumulative updates.

Steve Chen blogs.technet.com/steve_chen Alerts in SharePoint (Troubleshooting MOSS/WSS)

Harikumh Troubleshooting Alerts

4) Troubleshooting SharePoint Alerts User List and List Permissions

This part of the Troubleshooting SharePoint Email Alerts guide covers situations where you are getting emails for some users but not for others. If :

The users that do/do not get alerts can be separated by email domain Its intermittent i.e. users will sometimes get email alerts but sometimes not, for the same list/list item.

(Not to be confused with users who always get alerts for some lists/list items, but never for others) Then the culprit is probably the email infrastructure. As always the wild card is spam filtering and email rules so thoroughly discount these first!

Check the People and Groups list for a correct email address.
Check the site collections people and groups list for a correct email address. Dont assume this is correct and skip this step, yes I know its correct in AD but seriously, dont skip this! Also be aware that site collections (even in the same web application) are independent and have there own user lists you may have to check each one. Site Actions > Site Settings > People and Groups > All People. Find the users and check the email address is correct.

Check the list Permissions

The initial email alert confirmation is sent out regardless of the permissions on the list. Subesquent alert emails are security trimmed i.e. they are only sent out if the users has at least read permissions on the list an the list item that would have triggered the email.

Does the site inherit permissions from the site collection? Check List Settings >Permissions and Management >Permissions for this list o Does this list inherit permissions from its parent web site? Go back to List Settings > General Settings > Advanced Settings o Under Item-level permissions are users only allowed to Read their own items? Go to the list view. Click on an item and select Manage Permissions

Does this list item inherit permissions from its parent list? If not then does the user have permission on this list item?

To verify; check the user in question can open both the list and the item in question from SharePoint. You can also inspect Central Administration > Operations Diagnostic Logging for clues such as
"Alertsjob results for *** delivery: 10 prematches, 10 passed filtering, 8 of 10 passed security trimming..."

See Further Reading below.

Further Reading
http://patrickfeltz.spaces.live.com/blog/cns!41A4F8DC87858F8F!197.entry

5) Troubleshooting SharePoint Alerts Upgrade from 2003 to 2010

Problem
Consider the following scenario. You perform a database migration to upgrade Microsoft Windows SharePoint Services 2.0 to Microsoft Windows SharePoint Services 3.0. To do this, you deploy Windows SharePoint Services 3.0. Then, you add the content databases to the Web applications in the new Windows SharePoint Services 3.0 environment. However, after the migration, Windows SharePoint Services 3.0 does not send e-mail notifications when content changes in a migrated list or in a migrated document library. Users experience the following symptoms:

Users do not receive e-mail notifications for existing alerts for a list or for a document library that was migrated from Windows SharePoint Services 2.0. Users do not receive e-mail notifications for new alerts that they create for a list or for a document library that was migrated from Windows SharePoint Services 2.0. To receive e-mail notifications, the user must first delete the existing alert. Then, the user must create a new alert.

Cause
This issue occurs if the URL of the Windows SharePoint Services 2.0 server differs from the URL of the Windows SharePoint 3.0 server. For example, this issue occurs if the URL of the

Windows SharePoint Services 3.0 server is http://ServerNameVersion3, and the URL of the Windows SharePoint Services 2.0 server is http://ServerNameVersion2. The ImmedSubscription table in the content database has a Siteurl column. If the value in the Siteurl column does not match the URL of the Web application, Windows SharePoint Services does not send e-mail notifications when content in a list or in a document library changes.

Fixes and References

KB936759 E-mail notifications for alerts are not sent when content in a migrated list or in a migrated document library changes after you perform a database migration to upgrade to Windows SharePoint Services 3.0

6) Other troubleshooting: Some Technical details on Alerts: Once you have done all the previous investigations and tests but still having no glue or idea what may causing your alerts issue, you can dig a bit deeper into the SharePoint Mysteries: What's Under the Hood? Alerts are processed by the OWSTimer job. In central admin you may see just the Immediate alerts job, but this one processes both, the immediate and the scheduled alerts at intervals of every 5 minutes by default. When you create an alert a stored procedure in SQL Server will run to make the database changes to add the alert. Essentially, this inserts various values into the ImmedSubscriptions or SchedSubscriptions tables in the content database. On the SchedSubscriptions table we add some more columns to handle the timing of alert delivery. In these tables are stored some key values we need to flag each alert type for the further processing. Alerts will be security trimmed at send time, ensuring that unauthorized access to content is not possible. Security Trimming Unlike in WSS V2, security trimming will be a key component of the alerts infrastructure SharePoint will check both when the user creates an alert and when sending an alert that the user has permission to view the item. There are two steps to the process of alerts security trimming:

1. When list items are added, modified, or deleted: a. SharePoint records item metadata in the alerts event cache for the changed item. b. SharePoint does a lookup on the Permission table for the item and stores it in the alerts event cache as dbo.eventcache.acl. 2. When alert events match an immediate/daily/weekly subscription: a. Processing is segmented by site collection, so that cached user tokens are used most effectively. b. The users WSS token is retrieved from the User table. c. A request is made to the AD for each receiving users NT token. 3. An ACL check is done with the stored binary ACL blob, users WSS token and NT token for read access. Back to Top Logging Alert processing and notification sending will be logged. The following table describes the events logged, in both verbose (logs all events, for reference) and exception modes (exceptional events the admin needs to be aware of). Event Verbose Exception logging only X X X X

Alerts created Alert modified Alert deleted

Alert Filter/Formatting could not be X applied Alert status was modified end e-mail notification successfully SMTP server connection not available X X X

X X X

Recipient not found (mailbox error) X Filter incorrect, user alert and filter X deleted Begin matching in SQL Complete matching in SQL Begin XML matching X X X

Complete XML matching Begin ACL trimming End ACL trimming Back to Top

X X X

Backup and Restore Alerts will get backed up with site content. Restore will restore alerts for sites. Generating Alert E-mails Once the alert is created the system will check for a match every few minutes by looking for changes in the content database. When a change is found and it matches a subscription, an alert e-mail is generated. In the event that versioning is enabled and the user with the alert does not have access to the new version an alert will not be sent. The process that supports this on the server site is the timer service. Back to Top Immediate Alerts Each list basetype in WSS will have an alert template associated with it, which drives the UI behaviors, filtering, and e-mail content associated with a given alert. Alert templates will be stored in the property bag for the list if it has been customized. This allows for the case where the list is provisioned from a basetype, and then columns are added or removed from it. The mechanism to support multiple alert templates per list is to have separate toolbar buttons for the list with the appropriate alert template passed in as a parameter on the subnew.aspx page. SharePoint also supports multiple formats for the field to be rendered in email: 1. String: Including HTML. Text is truncated at 255 characters, which is set as an attribute in the template. 2. Image: Image is rendered in the message, and included in the message. 3. Link: The user is sent a hyperlink, for e-mail and HTTP links. We collect multiple edits made to a given document/item in a quantum (5 minutes), and represent them as one change. This means, for example, that five simultaneous edits of a document are represented as one change in the

immediate e-mail notification (instead of sending five separate e-mails). The e-mail will be formatted as a digest alert. Back to Top Scheduled Alerts There will be one daily digest sent per web. The sections in the mail are composed of each of the list types or items the user has created a daily alert for (individual sub-sections are defined in the formatting section for daily per alert template).

Multiple edits to the same item appear under one row in the section o changed information is aggregated with multiple asterisks in a given row if necessary. The Announcements digest section has the text "Alert created by 'UserName'" o when creating digests for the user, if content is due to an alert created by another user, this text should be emitted below the digest section for that alert. The links for items take users to item display forms. The links for e-mail addresses or hyperlinks boot a new e-mail message or Internet Explorer The links for lists take users to the default view of a list (Note that SharePoint does not remember if an alert was specific to a view). The list of items is sorted by the list-name alphabetically (item alerts also appear with a list header).

Database Tables To follow the trace of an alert process you can look into the content database tables on your SQL server. The tables that are most important when alerts are to be generated are the tables in the content database that start with the word event. In general terms, these tables are used as follows:
ImmedSubscriptions SchedSubscriptions EventCache records the immediate alerts settings records the scheduled alerts settings Change Log: records all events in SharePoint, so either the alert changes, stores events used in immediate alerts along with an id that is used in eventbatches. The eventtype is also stored in this table. This table contains events for which only non-immediate alerts exist and stores events used for digest alerts; Keeps processed alerts for last 7 days along with EventData which was present in EventCache before alert got processed

EventLog

EventBatches

Keeps track of the most recent event that has been processed for alerting, Keeps the last change log ID and time stamp from EventCache after which new processing should start when Immediate Alerts job runs records the timestamp when a scheduled alert has to be processed by the timerjob and used as a temp table during alert processing; Matches the change log ID with the Subscription ID (alert ID) from SchedSubscriptions table and permission result. Entry in this table gets flushed once scheduled alerts is sent for that subscription

EventSubsMatches

EventReceivers TimerLock

Used for custom event handlers records the server that processes the timerjobs

Back to Top How to troubleshoot alerts? 1. Enable Logging: First of all go to Central Admin page and screw up your Diagnostic (ULS) logging to verbose - All Errors. 2. Repro the Issue: Start repro your issue by creating an alert (immediate and/or scheduled) and note the timestamp you started as well as the timestamp you configured the scheduled alert to be fired. 3. Try to clear the cache (credits to Angelo Palma, who prosted this comment: Thank you!) First stop the windows SharePoint timer job then clear the cache as below "Go to C:\ Drive (make sure that all hidden folders are available) C:\ProgramData\Microsoft\SharePoint\Config\GUID\, sort by type and delete all files EXCEPT the cache.ini file. Once all files have been deleted edit the cache file by opening the file, change the value to 1 and save." Now start the windows sharepoint timer job Perform some actions like changes, deletions, uploads etc. and note the timestamp of changes and wait about 5-10 minutes like this: (above solution worked for US) 1. On you SharePoint site log in as a regular user. 2. Create several alerts both immediate and digest (on digest alerts set the

delivery time to be the top of the next hour). 3. Log in as a different user and modify, add, and delete some list items. 4. Start Outlook on the client machine and view the alerts. You will have to wait until the top of the next hour to see the digest alerts (scheduled) Back to Top Troubleshooting steps Check if you have received any alert notifications at all! Check now for specific alerts received or not received (Initial email, immediate alert email, scheduled alert email) SQL Queries Check on SQL Server if you can find your alert by using these queries for instance:
1: /* you must gather the ListID from the EventCache Table */ 2: select * from [Content_DB].[dbo].[Eventlog] (nolock) where ListID = 'xxx' 3: /* if not having the ListID in place */ 4: select top 10 * from [Content_DB].[dbo].[Eventlog](nolock) order by id desc 5: 6: select * from [Content_DB].[dbo].[ImmedSubscriptions] (nolock) where [UserEmail] = User.email@domain.com 7: 8: /* if not looking for a specific user-email */ 9: select * from [Content_DB].[dbo].[ImmedSubscriptions] (nolock) 10: 11: select * from [Content_DB].[dbo].[SchedSubscriptions] (nolock) where [UserEmail] = User.email@domain.com 12: 13: /* if not looking for a specific user-email */ 14: select * from [Content_DB].[dbo].[SchedSubscriptions] (nolock) 15: 16: /* check documents corresponding to the affected list */ 17: select * from [Content_DB].[dbo].[EventCache] (nolock)order by [TimeLastModified] 18: 19: select top 25 * from [Content_DB].[dbo].[EventSubsMatches] (nolock) order by id desc 20:

Queries against the "Config_DB":

1: select * from [Config_DB].[dbo].[objects] (nolock) where name like '%job-immediate-alerts%' 2: 3: select * from [Config_DB].[dbo].[objects] (nolock) where properties like '%SPContentDatabase%'

Note: System Account is a special account (application pool) for which no security trimming happens. Hence this account will not get digest alerts.

Back to Top Check the Grid view for: Can you find your alerts in ImmedScubscriptions / SchedScubscriptions table? Do you have a related record in EventCache due to the Timestamp (TimeLastModified) you noted from your repro? Do you see the record for scheduled alert in the EventSubsMatches table?

Check also your ULS logs (maybe with ULS viewer) for entries like that: (by default located at (C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\logs) [] Begin invoke timer job Immediate Alerts, id [] [] AlertsJob loaded 9 of 9 event data records [] [] AlertsJob loaded 5 of 5 subscription records [] [] Alertsjob results for immediate delivery: 9 prematches, 9 passed filtering, 5 of 9 passed security trimming, 0 [] [] Alertsjob results for scheduled delivery: 0 prematches, [] [] AlertsJob processed 0 daily notifications [] [] AlertsJob processed 0 weekly notifications [] [] []

Can you see any records indicating that your alerts are processed at all?

Check your Event log for suspicious errors related to the timeframe and probably containing phrases like exception, HResult errors, etc.. Check on your Server if the SharePoint services are running under the designated accounts. (Maybe you can just retype the credentials and restart the service and run "stsadm -o execadmsvcjobs" to trigger the immediate job processing)

Check if you are on an almost actual patch level as probably the cause of your issue already has been fixed in one of the last updates? So if all those steps are dont get you closer to the cause you are at least much more better prepared for the next step on calling Microsoft Support for assistance. With all these troubleshooting steps done before you can provide all these actions to the support engineer and this may speed up the resolution as the Support Guys then can go directly and already narrowed down to a particular area with very deep troubleshooting steps for you! So hopefully this post could help you somehow. Ill be on updating this post whenever some new or deeper insights can be published to get this topic a bit more structured in depth and usage.

Reference:

Refer below sites for troubleshooting: http://sharepointalert.info/troubleshooting-sharepoint-alerts/ http://blogs.technet.com/b/steve_chen/archive/2009/11/20/alerts-in-sharepoint-troubleshootingmoss-wss.aspx