Walkthrough : Capturing SharePoint Online specific events via the Office 365 Management API

Hi,

Although the Microsoft documentation on this topic is already quite good, I think that it might sound a bit rude to people who aren’t familiar with Azure AD Authentication and Office 365 APIs in general, especially because that API is probably one of the most complex ones.
Therefore, I decided to write this short walkthrough on how to capture some SharePoint Online events such as who shared a document with external users and get notified about that and possibly intervene.

Manual checking

First and foremost, you should know that there is a dedicated UI in Office 365 to look for numerous events. When launching the Security & Compliance option from the App Launcher and going to Search & Investigation ==> Audit Log Search, you get presented this interface:

While this is great, it’s more a reactive process and useful for reporting. However, what you can achieve with the API is to get notified when things happen and act upon them. That’s what we’re going to see in the next sections.

Create the Azure AD App

A prerequisite to use the Management APIs is to create an Azure AD Application and give it the required permissions to consume the API. For this, we only need to use the Client Credentials Flow (app permissions) as we don’t need to involve the end user to track these events.

Steps :

• Go to manage.windowsazure.com
• Select the Active Directory that corresponds to your Office 365 and click on Applications
• Create a new Application of type Web Application and/or WebAPI give it a name
• Fill-in the properties this way
  
  We don’t care about the sign-in URL as we will use the App only mode.
• Click on Configure, grab the ClientID that was automatically generated and create a new Key (the value will show up when clicking on the save button). In the section labelled Permissions to other applications, click on Add application and select Office 365 Management API, then assign all the permissions in the Application Permissions. You should end up like this:
  
  Now click save and grab the secret value. You’ll need the ClientID and secret value in the next steps.

Subscribe to a source

At the time of writing, one can subscribe to three different sources : SharePoint, Azure Active Directory and Exchange. It would be more efficient to have a more granular approach as for instance subscribe to the SharingInvitationCreated operation but it doesn’t seem possible so far. When you subscribe to a source, you can decide to use a webhook or not. If you want to be notified, you must use a webhook that will be called by Office 365 when an event occurs. So, let’s create one.

Steps to create the webhook:

• In Visual Studio, create a new project of type ASP.NET application, chose the default ASP.NET MVC template, chose Host into the Cloud
• Rename the controller to audit and make sure you comment out the [Authorize] attribute in front of your controller. Indeed, 365 will call the webhook anonymously so leaving this attribute in place would even prevent 365 from creating the subscription. When creating a subscription with a webhook, 365 makes a HTTP request on your webhook which must return 200, else the subscription isn’t created.
• Once done, you must implement the HTTP POST operation as this is how 365 will come back to the webhook. When an event occurs, 365 performs a HTTP POST and sends a notification packet in the body. Here is an example of the HTTP POST controller hook:

  public void Post([FromBody]dynamic value)
  {
    try
    {
      JArray notifications = JArray.Parse(value.ToString());
      foreach(var notification in notifications)
      {
        LogHelper.Log(notification["contentUri"].ToString());
      }
  
    }
    catch(Exception ex)
    {
      LogHelper.Log(ex.Message);
    }
  }
  

  In the above example, I’m logging the value of the retrieved contentUri which is a pointer to the actual event. I’m logging that into an Azure Queue. I’d recommend to avoid performing HTTP requests to the actual content from within the webhook for scalability and performance reasons. It is much easier to push these notifications to a queue or some similar mechanisms (service bus+subscribers) in order to process those queued items asynchronously.

• Publish your ASP.NET application to an Azure WebApp. Make sure you didn’t enable the Organizational Authentication. Your controller operations must be accessible anonymously. You can easily test that with a browser.

Now, it’s time to create the subscription itself. For that, we’ll need our ClientID and ClientSecret from the Azure AD Application.

• Create a new Console Application
• Install the ADAL nuget package.
• Start to get a token

  ClientCredential cred = new ClientCredential("{clientid}","{clientsecret}");
  AuthenticationContext ctx = new AuthenticationContext("https://login.windows.net/{tenantid}");
  AuthenticationResult res = ctx.AcquireToken("https://manage.office.com", cred);

  Depending on the ADAL version you use, you might not be able to work with synchronous operations, if it’s the case, just do the same as above with the asynchronous operations. From a Console Program, you might want to use Nito.AsyncEx Nuget package to work with an async main method.

Now, let’s create the subscription itself :

HttpWebRequest req = HttpWebRequest.Create(
  "https://manage.office.com/api/v1.0/{tenantid}/activity/feed/subscriptions/start?contentType=Audit.SharePoint") as HttpWebRequest;

req.Headers.Add("Authorization", "Bearer " + res.AccessToken);
req.ContentType = "application/json";
req.Method = "POST";
string hook =
   @"{'webhook' : {
       'address': 'https://yourwebhook.azurewebsites.net/api/audit',
       'authId': 'o365eyskensnotificationaad',
       'expiration': ''}
     }";
req.ContentLength = hook.Length;
using (var streamWriter = new StreamWriter(req.GetRequestStream()))
{
  streamWriter.Write(hook);
  streamWriter.Flush();
  streamWriter.Close();
}

The authId is an optional attribute which will be used as an HTTP Header by 365 when posting back to the webhook. Beware that you can share a webhook among multiple sources (SharePoint, Azure AD, Exchange) but of course, it will be called whenever an event (or a batch of events) occurs in these systems, which can generate a lot of requests. Your webhook should be highly available but this is something easy with Azure PaaS.
With the above code, we tell the system to register on SharePoint via the Audit.SharePoint content type. We reuse the Access Token obtained from our App and that’s it!. When performing this operation, 365 will call the webhook and expects a 200 in return. As of now, whenever an event occurs in SharePoint, it will be tracked by our webhook. The URL of our webhook targets /api/audit which is the name of our controller.

Testing our subscription

Now that we have a webhook that logs the contentUri into a queue, we can make some tests to see if this queue gets new entries when something happens in SharePoint Online. So, if you share a file with an external user, you should get an entry similar to this one after a few minutes (beware that a higher delay is expected for the first subscription):

https://manage.office.com/api/v1.0/tenantid/activity/feed/Audit/...$tenantidspo2016070515$...$04

This points to the actual content which is my case corresponds to this:

[{"CreationTime":"2016-07-05T19:20:13",
  "Id":"ec6de304-9e54-42f0-8078-08d3a4e7dcb8",
  "Operation":"SharingInvitationCreated",
  "OrganizationId":"{tenantid}",
  "RecordType":14,"UserKey":"i:0h.f|membership|xxxx@live.com",
  "UserType":0,
  "Version":1,
  "Workload":"SharePoint",
  "ClientIP":"xxx.zzz.114.1",
  "ObjectId":"https:\/\/eyskens.sharepoint.com\/Shared Documents\/mydoc.pdf",
  "UserId":"xxx@eyskens.onmicrosoft.com",
  "EventSource":"SharePoint",
  "ItemType":"File",
  "ListItemUniqueId":"2ecad62d-1d3b-4f2c-bad8-9fc4ec8e3f1a",
  "Site":"xxx-59e5-4062-a0fb-77d5b69da002",
  "UserAgent":"Mozilla\/5.0...",
  "WebId":"xxx-307f-420b-82a3-ff16886b9722",
  "EventData":"<ExpirationDate>10\/3\/2016 3:20:11 PM<\/ExpirationDate>",
  "TargetUserOrGroupType":"Guest",
  "TargetUserOrGroupName":"demo@dummydomain.com"
}]

I have obfuscated some data but you’ve got the point here : you can see who shared what and with who. The operation is SharingInvitationCreated so you can filter your actions based on the operation type. It would be nicer to be able to subscribe directly to this type to avoid receiving unwanted notifications but it doesn’t seem feasible so far.

If we analyze the pointer (above), we see that right after the TenantId, you have a string like this one spo2016070515 which is the source (SharePoint Online – spo), plus the date followed by some sort of category. Indeed, I noticed that FileAccessed operations is 13, FileDeleted 14 and Sharing actions 15 but this is not documented so it might be risky to filter notifications based on that. However, it would be nice to have some guidance here to skip notifications for which we’re not interested in.

In the enterprise wolrd, some products such as IBM QRadar could be used to do this job as they ship with a module that connects to the 365 Management API. However, if you can’t afford to buy such products (and consultancy that accompanies it…), extracting the events yourself and acting upon them isn’t rocket science.

Happy Coding!