Archive

Archive for the ‘Paul Mather’ Category

#ProjectOnline #PowerBI Report – Include #HTML formatting #PPM #PMOT #PowerQuery #OData #REST Part 3

January 16, 2018 Leave a comment
Paul Mather
I am a Project Server and SharePoint consultant but my main focus currently is around Project Server.
I have been working with Project Server for nearly five years since 2007 for a Microsoft Gold Certified Partner in the UK, I have also been awared with the Microsoft Community Contributor Award 2011.
I am also a certified Prince2 Practitioner.

This article has been cross posted from pwmather.wordpress.com (original article)

Following on from my 2nd post in this mini series on reporting including HTML formatting in Power BI, in this post we will look at a couple more options that will refresh in the Power BI App Service. If you missed the previous posts, the links are below:

Part 1: https://pwmather.wordpress.com/2018/01/03/projectonline-powerbi-report-include-html-formatting-ppm-pmot-powerquery-odata-rest-part-2/

Part 2: https://pwmather.wordpress.com/2018/01/03/projectonline-powerbi-report-include-html-formatting-ppm-pmot-powerquery-odata-rest-part-2/

The options we will look at in this post require a process to get the Project HTML data into a source that can be queried from Power BI with one call. Firstly I will demonstrate a simple PowerShell script that will get the data and write this to a SharePoint list on the PWA site. This is process is very similar to a Project Online snapshot solution starter script I published back in August 2016: https://pwmather.wordpress.com/2016/08/26/projectonline-data-capture-snapshot-capability-with-powershell-sharepoint-office365-ppm-bi/ Once you have a script running to capture the data on the defined scheduled you will see something similar to the screen shot seen below:

image

Here you can see my process has run twice, once back in August when I first wrote this script and just now when I ran it again. As this is based on sales demo data, you can see in the two expanded examples the data has not changed but in a real world usage I’d like to think the data would have changed / been updated! Having the data in one list enables a SharePoint OData call from Power BI, as I have included the ProjectId in the data on the list, this can easily be joined with the data from the main Project OData Reporting API. As this data is in a SharePoint list you might need to consider the user permissions / access to the list. If this was running on a schedule, either from a Windows Scheduled task if on-prem or maybe a scheduled Azure Function if you wanted to make use of Azure PaaS, set the schedule to run before the reports were due allowing time for this process to complete. I won’t cover the PowerShell script in detail here as I will create a dedicated post for that in a week or two, but I will highlight the changes required if you were to start with the OData snapshot example.

  • The first API called was updated in this example to change the select query to just return the ProjectId:

image

  • After the while statement, the script will start a foreach loop and set the ProjectId to a variable:

image

  • Then the REST URL is constructed and the ProjectId is passed in. The select query includes the Project Name, Project ID and the Multiline custom fields that I want to include. I then make the various REST calls in a try / catch block, firstly to get the data:

image

  • Then to write the data to the SharePoint list:

image

Once that runs successfully with an account that has full access to all projects and edit access the the SharePoint list, your target list will contain all of the projects along with the selected fields. As mentioned, I will post this full script in a week or two once I get a chance to tidy a few bits up in the code sample but hopefully the screenshots of the changes along with the snapshot example PowerShell script, there will be enough pointers to get started. Now the data is in a single source, it is very simple to use in Power BI.

In Power BI Desktop add a new OData feed, in the URL field enter the SharePoint list REST URL for the source list, for example the REST URL I used is: https://tenant.sharepoint.com/sites/PWA/_api/web/lists/getbytitle(‘ProjectMutliLineFields’)/Items    where ProjectMutliLineFields is the name of my SharePoint list. Edit the query to launch the Power BI query editor. In this example, my source SharePoint list contains duplicate projects but in my report I want to only see the latest. The steps below will transform the data so that the report only has the latest version for each project record. Rename the query to IDandDate then remove all columns except for the ProjectId and Created columns:

image

Now group by ProjectId and get the Max Created value, I called this column “Latest”:

image

That will give you a list of unique Project IDs using the latest record. Now add a 2nd OData feed and use the same SharePoint list REST URL as in the previous step. Remove columns that are not required, I removed all expect for Title, ProjectId, Created and the multiline fields. Then rename the columns to meaningful names if required:

image

This query will currently contain the duplicate project records based on my example list, next I will merge this query with the IDandDate query using the ProjectId column and the Created/Latest column:

image

Hold down the Ctrl key to select more than one column per table for the merge.

This will add the new column into the table:

image

Click the double arrow on the column heading to expand the column then select the aggregate radio button. On the dropdown menu next to Latest select Maximum:

image

This will show a date value for the latest records, where a null is displayed, there is a duplicate record with a later date:

image

Filter out the null records from the Max of Latest date column and that is it. For the purpose of this blog post, I also added a 3rd query to the Project OData API to show data from the two sources. Close and Apply the data then ensure the relationships are correct, I also set the IDandDate query to be hidden in the report view:

image

Then design your report as needed making use of the same HTML Viewer custom visual:

image

As you can see, this is just a simple example like the others just to highlight the HTML formatting being rendered in Power BI.

Another option without having to write and maintain any custom code or write the data to a SharePoint list does make use of a 3rd party tool that extracts the Project Online data into an Azure SQL database as the data changes in Project Online. This particular tool is developed by the Product Dev team I lead at CPS and is called DataStore. This product is part of our edison365 product suite but is available on its own. This isn’t sales pitch so I won’t go into details here but I just wanted to give another option as some people prefer no code solutions. There are also other software vendors that do similar products for Project Online but I’m not sure if they include the multiline project level fields with the HTML. So using this tool or similar (check they include the HTML fields), you can get all of your Project Online data into an Azure SQL database, as mentioned, the DataStore tool will also include the HTML data as displayed below in the example SQL query below:

image

Power BI can get data from the Azure SQL Server and this data will also refresh in the Power BI App Service.

Feel free to contact me if you have any queries or questions but hopefully that gives you some ideas on including the HTML formatting in your Project Online reports using Power BI!

Advertisements
Categories: Paul Mather, Work Tags:

#ProjectServer and #SharePoint 2013 / 2016 January 2018 Cumulative Update #PS2013 #SP2013 #PS2016 #SP2016 #MSProject

January 13, 2018 Leave a comment
Paul Mather
I am a Project Server and SharePoint consultant but my main focus currently is around Project Server.
I have been working with Project Server for nearly five years since 2007 for a Microsoft Gold Certified Partner in the UK, I have also been awared with the Microsoft Community Contributor Award 2011.
I am also a certified Prince2 Practitioner.

This article has been cross posted from pwmather.wordpress.com (original article)

The Office 2016 January 2018 updates and cumulative updates are now available, please see the links below:

http://bit.ly/2mtR6fv

Project 2016 January 2018 update:
http://bit.ly/2DcP4uu

SharePoint Server 2016 / Project Server 2016 January 2018 update: 
http://bit.ly/2mrb0HP & http://bit.ly/2DczJKk

The Office 2013 January 2018 updates and cumulative updates are now available, please see the links below:

http://bit.ly/2mtR6fv

Project Server 2013 January 2018 CU Server Roll up package:
http://bit.ly/2msRwCJ

Project Server 2013 January 2018 update:
http://bit.ly/2DcFvvO

Project 2013 January 2018 update:
http://bit.ly/2mtR9bb

Also worth noting, if you haven’t done so already, install Service Pack 1 http://bit.ly/1uorn2C first if installing the January 2018 CU for 2013.

As always, fully test these updates on a replica test environment before deploying to production.

Categories: Paul Mather, Work Tags:

#ProjectOnline #PowerBI Report – Include #HTML formatting #PPM #PMOT #PowerQuery #OData #REST Part 2

January 5, 2018 Leave a comment
Paul Mather
I am a Project Server and SharePoint consultant but my main focus currently is around Project Server.
I have been working with Project Server for nearly five years since 2007 for a Microsoft Gold Certified Partner in the UK, I have also been awared with the Microsoft Community Contributor Award 2011.
I am also a certified Prince2 Practitioner.

This article has been cross posted from pwmather.wordpress.com (original article)

Following on from my first post discussing including HTML formatting for Project Online Power BI Reports, in this post we will look at a summary of options to get the correct data into Power BI then walkthrough one of those options. In part 3, the final part, we will look at one of the other options to get the data.

For those of you that missed part 1, see the post here: https://pwmather.wordpress.com/2018/01/01/projectonline-powerbi-report-include-html-formatting-ppm-pmot-powerquery-odata-rest-part-1/

As per the first post, it is very simple to have the data rendered in Power BI to include the HTML formatting, the slightly more tricky part is to get the Project Online data into Power BI with the HTML included.

First, a bit of background on where in Project Online you can access the enterprise project multiline custom fields with the HTML included. As per the first post, you need to access the REST API ({PWAURL}/_api/ProjectServer) to get the data with the HTML included as the OData Reporting API ({PWAURL}/_api/ProjectData) has had the HTML removed. Using the REST API we can view the endpoints at the root:

SNAGHTML59e8e263

This is the REST API to programmatically interact with the Project Online data, you can create, read, update and delete data using this API depending on your access. For this reporting post we only need to read data, carryout the steps with an account that has access to all projects in the PWA instance like an Admin account.

The endpoint we need is /Projects:

SNAGHTML59ef8e46

This will detail all of the projects the logged on user has access to – it is recommended to carry out these steps with an account that has access to all projects in the PWA instance otherwise you might / will see errors in later steps. For each project detailed you will see a few key project level properties including things like Name, Description, Created Date, ID to name a few. It is also possible to navigate from there using the Project ID to get more details for that project. For example you can get the project tasks using the following URL: ProjectServer/Projects(‘{ProjectGUID}’)/Tasks or get the project team using this URL: ProjectServer/Projects(‘{ProjectGUID}’)/ProjectResources. To get the enterprise project level multiline custom fields we need to use the following URL: ProjectServer/Projects(‘{ProjectGUID}’)/IncludeCustomFields. Accessing this URL for the specified Project GUID (replace the placeholder with an actual project GUID) you will see more properties for that project including the multiline custom fields we need:

SNAGHTML59fcff6f

Notice the HTML in the custom field outlined in red in the image above. You would need to do this call for all projects but using the correct Project GUIDs. Also worth pointing out, in this API the custom fields are referenced using the internal names, for example Custom_x005f_4d0daaaba6ade21193f900155d153dd4 rather than the display names. You can use the custom fields endpoint to map the internal names to the display names: /ProjectServer/CustomFields.

So that covers the background on how you access the multiline custom field data that includes the HTML using the REST API, next we look at how to do this from Power BI. What makes it slightly more tricky than just using the normal OData Reporting API is that you have to make a call dynamically for each project GUID if you are using the REST API directly. In this series of posts we will look at calling this API dynamically straight from Power BI (covered later on in this post) but that has a limitation and also another method to get this data from one call / endpoint but that requires a bit of custom code / a 3rd party tool but does remove the limitation / issue. I will cover off the latter option in the 3rd blog post including a code sample / snippet.

Moving on to Power BI and getting this data dynamically and explaining the limitation. This process with follow the same approach I documented a while back to report on project site data using the the SharePoint list REST API: https://pwmather.wordpress.com/2016/01/05/want-to-query-cross-project-site-sharepoint-lists-in-projectonline-projectserver-powerbi-powerquery-bi-office365-excel-ppm/ As per the post above, this will require a custom function and a custom column to call the function. The limitation of this approach is that it works fine in the Power BI Desktop client but the data will not currently refresh in the Power BI App service. There might be workarounds to this limitation but that is beyond the scope of this blog post.

Firstly get a REST URL for one project that includes custom fields, for example I have used this: https://tenant.sharepoint.com/sites/pwa/_api/ProjectServer/Projects(‘ad641588-f34b-e511-89e3-00059a3c7a00‘)/IncludeCustomFields?$Select=Id,Name,Custom_x005f_4d0daaaba6ade21193f900155d153dd4 – replace the parts highlighted in yellow with details from your PWA instance. In this example I have included just one of my multiline custom fields but include as many multiline fields as required, just separate them using a comma. As mentioned before, use the /CustomFields endpoint to identify the correct custom fields to include in the select statement. You can see below, the example multiline field I have used is called “Status Summary”

SNAGHTML5cb3b4d5

Now add this URL as a data source in Power BI using the Get Data > OData feed option. That will open the Query Editor and show the record:

image

Update the Query Name to something like projectHTMLCFsFunction as this query will be turned into a function. In the Query Editor, on the View tab access the Advanced Editor and you will see your query:

SNAGHTML5cc54192

The full query will be similar to this:

let
    Source = OData.Feed("https://tenant.sharepoint.com/sites/pwa/_api/ProjectServer/Projects(‘ad641588-f34b-e511-89e3-00059a3c7a00’)/IncludeCustomFields?$Select=Id,Name,Custom_x005f_4d0daaaba6ade21193f900155d153dd4")
in
    Source

This needs to be modified to turn this into a parameterised function like below, parts highlighted in yellow are added / edited:

let loadHTMLCFs = (GUID as text) =>
    let
        Source = OData.Feed("https://tenant.sharepoint.com/sites/pwa/_api/ProjectServer/Projects(‘"&GUID&"‘)/IncludeCustomFields?$Select=Id,Name,Custom_x005f_4d0daaaba6ade21193f900155d153dd4")
    in
        Source
in  loadHTMLCFs

A screen shot below to show the completed query in the Query Editor as the formatting is clearer, bits added / edited are outlined in red:

SNAGHTML5cc4e299

Click Done in the Query Editor and you will see the following:

image

No need to do anything with the parameter or buttons. Now we need to add another data source in for the other data feeds required in the report, for the purpose of this blog I will just add in the minimum required and that is the default OData Reporting API /Projects endpoint to get the other project fields into the report. In the Query Editor on the Home tab click New Source > OData feed and add in the OData Reporting API URL: https://tenant.sharepoint.com/sites/pwa/_api/ProjectData then select the tables required. For this blog post I have just selected Projects. Using the Query Editor, remove unwanted columns, rename columns etc. You will need to keep at least ProjectId and ProjectType, they are required. For the purpose of the blog post I have just selected ProjectId, ProjectType, ProjectName and ProjectOwnerName. Using ProjectType, filter out ProjectType 7 as this is the Timesheet Project record. Keeping this in the dataset will cause errors later on.

Once you have edited the query as required a new custom column needs to be added to invoke the function created earlier. Click the Add Column tab then click Custom Column. Give the column a name such as GetProjectHTMLCFs then enter the following: projectHTMLCFsFunction([ProjectId]) as seen below:

image

projectHTMLCFsFunction is the name of the function we created earlier and we are passing in the ProjectId. When clicking OK, this might take a while depending on how many projects you have as this will invoke the function for each project and call the REST API, passing in the ProjectId for that row and bring back the records. Once completed you will see the records as below in the new custom column:

SNAGHTML5cdd9337

Now the column needs to be expanded, click the double arrow in the custom column heading and expand the multiline custom fields, in this example I just have one:

image

Click OK and the data will refresh / load then display the data for the multiline columns:

SNAGHTML5ce22300

Notice we have the HTML in the data! Rename the columns for the correct display names then when completed, click Close & Apply. The changes will now be applied to the Power BI Report and load the data. Add in the HTML Viewer custom visual as detailed in blog post 1 then add the data on the the report canvas as you would normally. Ensure that the multiline custom fields use the HTML Viewer custom visual:

image

An example with a normal table visual and the HTML Viewer visual:

image

That’s it, design your Project Status reports to now include the HTML formatting your users have added. Just remember though, this will only refresh in the Power BI Desktop client. It can be published to the Power BI App service but the data will be static and will not update, you would need to open the report in the Power BI Desktop client, refresh it then publish it back into the Power BI service.

Next up in part 3 we will look at a slightly different approach to get the data in Power BI that does enable the report / data to refresh in the Power BI App service.

Categories: Paul Mather, Work Tags:

#ProjectOnline #PowerBI Report – Include #HTML formatting #PPM #PMOT #PowerQuery #OData #REST Part 1

January 1, 2018 Leave a comment
Paul Mather
I am a Project Server and SharePoint consultant but my main focus currently is around Project Server.
I have been working with Project Server for nearly five years since 2007 for a Microsoft Gold Certified Partner in the UK, I have also been awared with the Microsoft Community Contributor Award 2011.
I am also a certified Prince2 Practitioner.

This article has been cross posted from pwmather.wordpress.com (original article)

My first post for 2018, Happy New Year to all! This post is the first of 2 or 3 posts covering HTML formatting in your Power BI reports from Project Online multiline project level custom fields as seen below – screenshot from mock up / demo data:

image

For those of you that are familiar with the Project Online Reporting API, Microsoft made a change back in May 2016 to remove the HTML from the OData API ({PWAURL}/_api/ProjectData): http://bit.ly/24hh9nb. This was due to requests from customers so that Excel / Power BI reports could contain cleansed data without having to remove the HTML from the strings yourself. As mentioned in the blog post above, the HTML strings for multiline project custom fields are still available from the REST API ({PWAURL}/_api/ProjectServer).

Back in November 2017 a new custom Power BI visual was released to render HTML: http://bit.ly/2CskInt, this now means that you can include the nicely formatted text from Project Online multiline project level custom fields in your Power BI reports. A couple of screen shots below show what your project custom field multiline data probably looks like today in your reports and what it could look like. Ignore the very basic dull looking report, this is purely just to demo the HTML rendering.

Without the HTML formatting from the OData API – it is just a block of text:

image

With the HTML formatting – it is nicely formatted and readable:

image

This matches the text on the Project Detail Page (PDP) in the Project Web App for that example demo project:

image

To be able to include the HTML formatting there are two parts:

  • Get the data that includes the HTML
  • Add the HTML Viewer custom visual to your Power BI Desktop client

The latter being very simple from the Power BI Desktop client by either clicking the ellipsis in the Visualizations pane:

image

Or using the button on the Home ribbon:

image

Then search for the HTML viewer and add it:

image

In the next 1 or 2 posts I will cover some different options for getting access to the data that includes the HTML.

Categories: Paul Mather, Work Tags:

#ProjectOnline time phased data rollup for #OData reporting #PPM #PMOT #BI #Excel #PowerBI

November 17, 2017 Leave a comment
Paul Mather
I am a Project Server and SharePoint consultant but my main focus currently is around Project Server.
I have been working with Project Server for nearly five years since 2007 for a Microsoft Gold Certified Partner in the UK, I have also been awared with the Microsoft Community Contributor Award 2011.
I am also a certified Prince2 Practitioner.

This article has been cross posted from pwmather.wordpress.com (original article)

Recently it was announced that it would be possible to rollup some of the data in the time phased feeds for Project Online, the support documentation can be found here: http://bit.ly/2yQUZQp

Currently today in Project Online, the time phased data is stored in the Reporting schema at the day level. For some organisations, this is too granular and they end up having to aggregate the data for reports to weekly / monthly etc. For those customers, having the data at the day level isn’t convenient as storage / performance improvements can be gained from having the data stored at source pre-aggregated. With this change, that will now be possible.

I believe this feature will start rolling out in the next week or two but let’s have a quick look at the options. From the PWA Settings menu you will see a new option under Enterprise Data for Reporting as seen below:

image

This shows the following page:

image

As this new feature has been rolled out to an existing PWA site, this defaults to Daily but new PWA sites created once this feature is rolled out to the tenant will have this setting set to Never.

Let’s look at the impact on the data using my simple project plan that has a task with a duration of 5 days:

image

Using the TaskTimephasedDataSet you can see the data below for Task 2:

image

As expected, there are 5 days displaying work. I will now change the setting to Weekly:

image

For this change to take effect I will need to publish all of my projects but for the purpose of this blog post I will just publish my test project. Refreshing my Excel data, you can see I have two rows as the task spans two weeks:

image

The hours are aggregated on the first day of the week as defined by the PWA site regional settings:

image

Now I will increase the task duration to 50 days to span a few months and set the reporting to monthly then publish my test project. Updated project:

image

Updated to Monthly:

image

Updated Excel report:

image

As you can see the hours are now aggregated on the first day of the month. You can also base this on the fiscal periods defined in PWA.

The feeds that are impacted by this change are:

  • AssignmentBaselineTimephasedDataSet
  • AssignmentTimephasedDataSet
  • TaskBaselineTimephasedDataSet
  • TaskTimephasedDataSet

Once available in your tenant, set the time phased data reporting setting as defined by your reporting requirements and publish all of the projects. I would recommend you did this on a non-production PWA instance first as you might need to update you reports, apps etc. that consume date from those four feeds. Also remember to set this up for new PWA instances created once this feature is live as they will be set to Never.

Keep an eye out for this feature reaching your tenant soon.

Categories: Paul Mather, Work Tags:

#ProjectServer and #SharePoint 2013 / 2016 November 2017 Cumulative Update #PS2013 #SP2013 #PS2016 #SP2016 #MSProject

November 16, 2017 Leave a comment
Paul Mather
I am a Project Server and SharePoint consultant but my main focus currently is around Project Server.
I have been working with Project Server for nearly five years since 2007 for a Microsoft Gold Certified Partner in the UK, I have also been awared with the Microsoft Community Contributor Award 2011.
I am also a certified Prince2 Practitioner.

This article has been cross posted from pwmather.wordpress.com (original article)

The Office 2016 November 2017 updates and cumulative updates are now available, please see the links below:

http://bit.ly/2mwJoEv

Project 2016 November 2017 update:
http://bit.ly/2z84ZZP

SharePoint Server 2016 / Project Server 2016 November 2017 update: 
http://bit.ly/2mwJps3 & http://bit.ly/2z6B8RP

The Office 2013 November 2017 updates and cumulative updates are now available, please see the links below:

http://bit.ly/2mwJoEv

Project Server 2013 November 2017 CU Server Roll up package:
http://bit.ly/2mwJpZ5

Project Server 2013 November 2017 update:
http://bit.ly/2z8527X

Project 2013 November 2017 update:
http://bit.ly/2mvIlox

Also worth noting, if you haven’t done so already, install Service Pack 1 http://bit.ly/1uorn2C first if installing the November 2017 CU for 2013.

As always, fully test these updates on a replica test environment before deploying to production.

Categories: Paul Mather, Work Tags:

#ProjectOnline #PPM #PowerBI Report Pack v2 #BI #Reporting #PowerQuery #DAX #Office365

October 31, 2017 Leave a comment
Paul Mather
I am a Project Server and SharePoint consultant but my main focus currently is around Project Server.
I have been working with Project Server for nearly five years since 2007 for a Microsoft Gold Certified Partner in the UK, I have also been awared with the Microsoft Community Contributor Award 2011.
I am also a certified Prince2 Practitioner.

This article has been cross posted from pwmather.wordpress.com (original article)

Back in January this year I published my first Power BI report pack for Project Online, the post can be found here: http://bit.ly/2ivjxct I have now published the second version of my report pack for Project Online. This version can be download from the link below:

http://bit.ly/2iiH1zX

This report pack consists of 8 reports for Project Online, these reports can be seen below:

Portfolio Report page:

image

Issues Report page:

image

Risks Report page:

image

Project Report page:

image

Resource Demand Report page:

image

Resource Report page:

image

Timesheet Summary Report page:

image

Timesheet Detail Report page:

image

These reports only use default intrinsic fields so it should work for all Project Online deployments.

Once downloaded, the report pack data sources will need to be updated to point to your target Project Online PWA instance. To do this you will need the Power BI desktop tool which is a free download here: http://bit.ly/2gRjCoQ

Open the downloaded PWMatherProjectOnlinePowerBIReportPackv2.pbit template file in Power BI Desktop and follow the steps below to point the data sources to your Project Online PWA instance:

 

  • In the parameter window that opens, enter the full Project Online PWA URL
  • Click Load
  • The data will now start to load and you will be prompted to connect
  • On the OData feed window, click Organizational account and click Sign in and enter credentials as required
  • Click Connect
  • On the Privacy levels window set the privacy as required
  • Click Save
  • The data will load – this may take a few minutes depending on the dataset size in Project Online
  • Access the Project Report page and select a project from the project filter
  • Save the report

This file can either be emailed around to colleagues with details on how to update the credentials to their own or what would be better is to publish the report to your Power BI workspace can create an organisational content pack that others can add to their Power BI workspace. If the Power BI organisational content pack is the chosen option, you might want to create a Dashboard first. See a previous blog post on this: http://bit.ly/2l2u5kS

Enjoy, I hope you like it Smile

Categories: Paul Mather, Work Tags:
%d bloggers like this: