SharePoint (and Project Server) Shenanigans

On a recent project I came across a strange problem with Excel Calculation Services.

On a SharePoint 2013 farm with multiple web applications, Excel services was working perfectly on a couple of the web applications but not on others. The trusted locations and process identity settings were set and checked but were not at fault, digging through the ULS logs was showing this.

10/01/2014 15:37:57.58  w3wp.exe (0x7130)        0x4220  Excel Services Application            Excel Calculation Services                ech0      Medium               ExcelService.PostProcessRequest: web method: OpenWorkbook, got exception Id=GenericFileOpenError; Microsoft.Office.Excel.Server.CalculationServer.FileOpenException: The workbook cannot be opened. —> Microsoft.Office.Excel.Server.Host.HostFileException —> Microsoft.SharePoint.SPException: Cannot complete this action.  Please try again. —> System.Runtime.InteropServices.COMException: Cannot complete this action.  Please try again.

    at Microsoft.SharePoint.Library.SPRequestInternalClass.GetFileAsStream(String bstrUrl, String bstrWebRelativeUrl, Boolean bHonorLevel, Byte iLevel, OpenBinaryFlags grfob, String bstrEtagNotMatch, Object punkSPFileMgr, Boolean bHonorCustomIrm, IrmProtectionParams fileIrmSettings, UInt32& pdwVirusCheckStatus, String& pVirusCheckMessage, String& pEtagNew, String& pContentTagNew, SPFileInfo& pFileProps)

    at Microsoft.SharePoint.Library.SPRequest.GetFileAsStream(String bstrUrl, String bstrWebRelativeUrl, Boolean bHonorLevel, Byte iLevel, OpenBinaryFlags grfob, String bstrEtagNotMatch, Object punkSPFileMgr, Boolean bHonorCustomIrm, IrmProtectionParams fileIrmSettings, UInt32& pdwVirusCheckStatus, String& pVirusCheckMessage, String& pEtagNew, String& pContentTagNew, SPFileInfo& pFileProps)     — End of inner exception stack trace —

    at Microsoft.SharePoint.SPGlobal.HandleComException(COMException comEx)

    at Microsoft.SharePoint.Library.SPRequest.GetFileAsStream(String bstrUrl, String bstrWebRelativeUrl, Boolean bHonorLevel, Byte iLevel, OpenBinaryFlags grfob, String bstrEtagNotMatch, Object punkSPFileMgr, Boolean bHonorCustomIrm, IrmProtectionParams fileIrmSettings, UInt32& pdwVirusCheckStatus, String& pVirusCheckMessage, String& pEtagNew, String& pContentTagNew, SPFileInfo& pFileProps)

    at Microsoft.SharePoint.SPFile.GetFileStream(SPWeb web, String fileUrl, Boolean honorLevel, SPFileLevel level, OpenBinaryFlags openOptions, String etagNotMatch, SPFileStreamManager spMgr, SPFileRightsManagementSettings rightsManagementSettings, Boolean throwOnVirusFound, SPVirusCheckStatus& virusCheckStatus, String& virusCheckMessage, String& etagNew, String& contentTagNew, SPFileInfo& fileprops)

    at Microsoft.SharePoint.SPFile.GetFileStream(OpenBinaryFlags openOptions, String etagNotMatch, String& etagNew, String& contentTagNew)

    at Microsoft.SharePoint.SPFile.OpenBinaryStream(SPOpenBinaryOptions openOptions, String etagNotMatch, String& etagNew)

    at Microsoft.SharePoint.SPFile.OpenBinaryStream(SPOpenBinaryOptions openOptions)

    at Microsoft.Office.Excel.Server.MossHost.MossHostFileLoader.FetchStreamInternal()     — End of inner exception stack trace —

    at Microsoft.Office.Excel.Server.MossHost.MossHostFileLoader.FetchStreamInternal()

    at Microsoft.Office.Excel.Server.MossHost.MossHostFileLoader.FetchStream(IIdentity currentIdentity)

    at Microsoft.Office.Excel.Server.CalculationServer.SharePointFileLoader.FetchStream()     — End of inner exception stack trace —

    at Microsoft.Office.Excel.Server.CalculationServer.SharePointFileLoader.FetchStream()

    at Microsoft.Office.Excel.Server.CalculationServer.BaseWorkbookManager.CacheStream(CachedFile cachedFile, FileLoader loader)

    at Microsoft.Office.Excel.Server.CalculationServer.BaseWorkbook.Microsoft.Office.Excel.Server.CalculationServer.ICachedFileProvider.SaveFile(CachedFile cachedFile)

    at Microsoft.Office.Excel.Server.CalculationServer.CachedFile.SaveFile(ICachedFileProvider fileProvider)

    at Microsoft.Office.Excel.Server.CalculationServer.LocalDiskCache.ReserveCachedFileAndMarkUsed(FileId fileId, ICachedFileProvider fileProvider, Boolean replaceExistingFile)

    at Microsoft.Office.Excel.Server.CalculationServer.BaseWorkbook.ReserveCachedFileAndMarkUsed(FileLoader fileLoader, SessionId previousSessionId)

    at Microsoft.Office.Excel.Server.CalculationServer.BaseWorkbook.<FileStreamJob>b__e(IssueLoadThreadArg arg)

    at Microsoft.Office.Excel.Server.CalculationServer.BaseWorkbook.ExecuteWorkbookJob(WorkbookJobDelegate workbookJob, Object param)     24e5bd9c-6708-8070-314e-c1443430d9e0 

Looking at the Excel Services file cache for a working request we were seeing this.

but for a non working request we were seeing this.

the only commonality seemed to be that the non-working web applications were using RBS for BLOB storage.

Digging through the ULS logs some I finally found the following error.

System.Data.SqlClient.SqlException (0x80131904): The EXECUTE permission was denied on the object ‘rbs_fn_get_blob_reference’

A quick google turned up some blogs about database permissions for service accounts, following this blog http://blog.sharepointsite.co.uk/2014/01/sp-2013-ssrs-failing-after-rbs-enabled.html

I added the db_rbs_admin database role to the Excel Services service account for each RBS enabled database and Excel services sprang into life.