For your reference, these are our previous seven articles:

1. Part 1, BDC Purpose and technical architecture
2. Part 2, Application Definition File (ADF) and its Development
3. Part 3, Developing an ADF to Connect to Web Services
4. Part 4, Consuming Business Data through Web Parts and SharePoint Lists
5. Part 5, Implementing Enterprise Search with Business Data
6. Part 6, Integrating User Profiles with Business Data
7. Part 7, Working with the BDC Object Model

With that, let’s dive into BDC actions and start by asking what is an action.

What is an action?

An action is simply an operation upon a selected item. The operation can retrieve information about the selected item, or it can launch a program or Web page with the selected item as a parameter. The operation is a predefined URL link in each BDC item that enables you to act on the item, such as editing, or mapping it using Live Search as we demonstrated in Part 4.
An action is composed of a URL target where the action is to occur, and the matching parameters in the URL QueryString format. When we associate an action with an entity, we can pass parameters to that action based on the contents of one or more fields. For example, in the Live Search example in Part 4, we associate the Live Search page’s URL with a customer and provide the customer’s address to parameterize the search.

From the above, major characteristics of actions are as follows:

1. URL-based, parameter-driven
2. Actions on any entity and appear everywhere an entity goes
3. User can take action in ECB menu
4. Custom actions means custom code

A perfect example of action is the default View Profile action, as shown in Figure 1.

Figure 1  View Profile action in a Business Data Actions Web Part

Through a View Profile action we can navigate to the profile page of an entity. Clicking the View Profile action in a Business Data Actions Web Part as shown in Figure 1 above, as well as in a Business Data column, or in a Business Data List Web Part, lets us view the details of an item on that item’s profile page. Figure 2 shows that a Profile page that displays employee details.

Figure 2  A Profile page that displays employee details

What is a Profile page?

A profile page is automatically generated for each entity that has a SpecificFinder method after BDC imports an ADF file, a profile page can display all the information from a record in the BDC.  As we’ve seen in Figure 2, a profile page can display all the fields in a record for a specific employee. The default profile page contains a BDC Item Web Part for that entity.
If we put the profile page in Edit mode, there are actually two Web parts: a Business Data Item Builder and a Business Data Item that connect together. The Item Builder parses the querystring (http://win7:10001/ssp/admin/Content/Employee.aspx?EmployeeID=1) and passes the EmployeeID value to the Business Data Item, which then executes the SpecificFinder method and returns a record of Employee data.

Figure 3  Profile page components

Default Profile Page and default action

As we have seen from above section, the default action (View Profile) along with the default Profile page is automatically created when the entity has a SpecificFinder method.  To provide an ECB menu user experience, we can also add actions in a Business Data List Web part without using the Business Data Actions Web Part.

Defining an Action

There are three ways to add an action: 1) defining it in the ADF, 2) adding it in the BDC admin page after the ADF has been imported into BDC, and 3) through the BDC Object Model. Here we’re focus the first two.

Defining it in the ADF

To define an Action for the specified BDC entity, we’ll first locate the <Actions> section within the <Entities> and <Entity> hierarchy.

Figure 4  Actions is nested within the Entities/Entity in ADF

And insert the <action> portion like this one:

<Actions>
<Action Name=”Update Employee” Position=”1″ IsOpenedInNewWindow=”false” Url=”pages/editemployee.aspx?EmployeeID={0}” ImageUrl=”">
<ActionParameters>
<ActionParameter Name=”EmployeeID” Index=”0″ />
</ActionParameters>
</Action>

Adding an action in the BDC admin page

In the SSP administration page (View Application –> View Entity), the action is defined below and the URL that takes EmployeeID as a parameter: http://dpm:10001/ssp/admin/Content/Employee.aspx?EmployeeID={0}

Figure 5  Add an action the SSP Admin page

Custom Actions

We also can create custom actions for entities in BDC. Among other things, custom actions allow users to navigate to any custom page to manipulate business data.
Here are some scenarios that custom actions can do:

1. Parameterize custom search
2. Update employee in HR system
3. Customize profile page for an entity
4. Integrate BDC search into enterprise report center

We’ll be creating custom actions in the last part the article.

Figure 6 Enable View Profile action in a Business Data List Web part

This enables the ECB-style action, we will specify a Title property and add the following in the ADF:
<Entity EstimatedInstanceCount=”0″ Name=”Employee”>
<Properties>
<Property Name=”Title” Type=”System.String”>LastName</Property>
</Properties>
</Entity>

It specifies which column in the Business Data List/Related List Web part displays the Actions menu. It also specifies the name of the TypeDescriptor that corresponds to the column you want to display as the default column in Business Data lists.

In the Business Data Column scenario, we can also display the action menu and link the column to the business data profile in a Business Data list.

Figure 7  View Profile action in Business Data Column

This is default behavior OOB for a Business Data column.  When adding the column, just make sure the following two are checked:

• Display the actions menu
• Link this column to the profile page to

Figure 7  Enable the actions menu and profile page in Business Data list

These profile pages are used and being indexed by BDC Search. The following search results display the profile pages that contain the search keyword (s).  Notice in Figure 8 how a search for Great Lakes will display the profile pages for both this customer and their orders.

Figure 8  BDC Search Results in Profile Pages

Enhancing the Profile Page

By default, the profile page is hosted on the Shared Services Provider site collection where the BDC administration site is located. The search result (that is, the URL) links back to the Shared Services Administration site collection address that is not scalable when you extend the Web application. In real world, we may not want to give end users access to the SSP Admin site for security reason. A better practice is to create our own profile pages and host them in any site that is not part of the SSP site collection.
In this example, we’ve created a dashboard page in Report Center to replace the default Profile Page. The Report Center provides enterprise reporting capabilities that are scalable and centralized.  We can direct users to custom profile pages by updating the View Profile action.

Figure 9  Replace the default Profile Page with Custom Report Center

As an alternative, we can also replace the default the URL in Action element in ADF with the path to the Profile.aspx page:

<Actions>
<Action Position=”1″ IsOpenedInNewWindow=”false” Url=”http://reportcenter/Pages/Profile.aspx?EmployeeID={0}” ImageUrl=”/_layouts/1033/images/viewprof.gif” Name=”View Profile”>
<ActionParameters>
<ActionParameter Index=”0″ Name=”EmployeeID” />
</ActionParameters>
</Action>
</Actions>

An action to call the Profile.aspx page for each Employee ID has been defined. This action will be the default action provides in the search result.

Now let’s create Profile page to display information about the Employee entity in Report Center. (The Report Center is a site that provides a central location for business-intelligence-related information.)

1. In the http://reportcenter site, create a new Blank Web Part Page called “Profile.aspx”.
2. Add the following two web parts: Query String (URL) Filter Web Part and Report Viewer Web Part.
3. Configure the Query String (URL) Filter Web Part to pickup parameter “EmployeeID” from the URL query string.
4. Configure the Report Viewer Web Part to display the employee profile report called “Employee Profile”.
5. Connect the Query String (URL) Filter Web Part parameter to the SQL Server Reporting Services Report Viewer Web Part parameter “EmployeeID”.

Figure 10  SQL Server Reporting Services Report Viewer

Note: Please make sure you use this Web Part and not the OOB Report Viewer Web Part.

Test that the page is picking up the URL string parameter “EmployeeID” correctly by typing the following URL: http://reportcenter/Pages/Profile.aspx?EmployeeID=108

Figure 11  Custom Profile Page in Report Center

Creating Custom Actions

Creating custom actions requires custom code. The approach we take is to treat the BDC source simply as a data reader, and to build our CRUD (create/read/update/delete) logic in custom code. The BDC object model provides the means to retrieve collections, list items and enumerators, and we can use GenericInvoker to perform CRUD, which has been extensively covered in Part 7. We get the Web Parts to display data, and then we can use custom actions to create and update records by using Web parts and InfoPath Forms.

Creating Web Parts

We have created and deployed two Web Parts using the Object Model as shown below, with one inserting a record and one updating an existing record. (Special thanks to Lightning Tools! I’ve used BDC Meta Man to help with creating the Web parts.)

Figure 12  The two Custom Web Parts can be used to add and update employee records

Add these two actions in the SSP Admin page:

Figure 13  Adding actions in the SSP Admin page

Now you can access the two actions from the content menu in the Employee List:

Figure 14  Custom Actions Add Employee and Update Employee in the context menu

Figure 15  Add Employee Web Part

Figure 16  Update Employee Web Part

Creating InfoPath Form

MSDN has a great article titled Using Business Data Catalog Actions to Pass Parameters to InfoPath 2007 Browser Forms that walks you through this in detail.  So I’ve created our InfoPath Form and added the action using the Form Template path:
http://dpm/_layouts/FormServer.aspx?XsnLocation=http://dpm/FormServerTemplates/EmployeeUpdate.xsn&DefaultItemOpen=1&EmployeeID={0}&FirstName={1}&LastName={2}

Figure 17  Adding Custom Action that uses InfoPath Form

Figure 18  Update Employee in InfoPath Form

Conclusion

In this final article, we have focused on actions that users can take on the business entities. We can define actions in ADF or add it “after the fact”. We have also created custom actions using the BDC object model by using custom Web parts and InfoPath forms to enable CRUD (create/read/update/delete) operation support. Once added, actions can be performed from within the list, which becomes relatively transparent to the user where the data comes from. [The statement that the BDC data is read-only is not true anymore as we have just demonstrated.] As you have learned from our 8-part series, BDC enables user interaction in business intelligence, enterprise content management, collaboration and social features, and enterprise search.

Looking Forward to SharePoint Server 2010

BDC has evolved to what is now called BCS (business connectivity services) in SharePoint 2010. Even though we can use BDC to enable connectivity to external systems, it was difficult to create solutions because of the lack of a designer and it was not so simple to create a solution that enabled users to make changes and write that data back to the external store. BCS now enables you to integrate LOB data into SharePoint and Office with full CRUD support, tighter integration with Office client applications, and better tools for modeling business entities. BCS is all about connecting to external data. We look forward to sharing with you the exciting new features in BCS and what they can do for you in the near future. Until then, aloha!

1. Create a datawarehouse
2. Create a data source
3. Create a Time dimension
4. Add Indicators (the pretty bits)
5. Create KPIs
6. Create a scorecard
7. Define how KPIs rollup
8. Add a report
9. Create a dashboard
10. Publish the dashboard

Sounds so simple doesn’t it? Well, I guess, if you have planned your KPIs properly as we discussed in the previous article, it really is.

1. Create a datawarehouse

Why can’t I just get it directly from my transactional system or my existing datawarehouse, you ask? Sure you can. But let’s be realistic for a second. A scorecard is made up of Actuals and Targets and it compares unrelated data. Do you see where I’m going with this? Let me speak plainly. The data is coming from different places so it will have different, keys and structures and so on. Plus somehow, it all must relate – even when it isn’t related. What I mean by that is that you need a single Time dimension and probably a single Business Hierarchy dimension. Oh and don’t forget Scenario. By all means, if all that information already resides in an existing datawarehouse go for it, but chances are it doesn’t. And we will be building a scorecard; which is all about aggregation. So what tool does aggregation the best? A cube baby!

2. Create a data source

In PerformancePoint, you can have datasources that are multidimensional, relational (ODBC) or tabular lists (Excel Services, Excel Spreadsheet, SharePoint List or SQL table).

Obviously, the most used option is Analysis Services – and so it should be. Multidimensional sources have all that handy stuff like aggregation, optimisation and encapsulated business logic. Obviously there are circumstances where other sources are necessary, such as displaying lists or real time data but generally speaking, it’s wise to take advantage of the speed of cubes wherever possible, as I so zealously advocated in the previous step.

3. Time. It’s all relative right?

Every scorecard, pretty much, needs a time dimension. One of the joys of PerformancePoint is that a lot of the hard work is baked in for you. Bless their cotton socks! You’ve got a Time dimension in your cube (and it contains an appropriately structured hierarchy right?) so PerformancePoint just needs to know what it’s called and, as at today, which member represents “Now”. I don’t mean you need a “Current” member, I mean literally, today is 1^st July so select the member for 1^st July. Oh yeah, and it’s a “Day” (as opposed to say, a month).

* ok yeah, this shows 2002 but that’s because my sample database doesn’t contain 2009 J So you’ll just have to imagine. You can do that right?

While you’re at it, tell me which level of the hierarchy refers to a year, which a month and so on.

PerformancePoint will work out “CurrentPeriod” from here on in. Handy!

4. Indicators. The Pretty Bits

Dashboards are ALL about the pretty. If you think otherwise, well, you’re kidding your-table-loving-self. Dashboards aren’t for you. They’re for marketing people, business analysts, general managers and business development managers et al… you know the types… they just want to see things at a glance and they’ll gush and kiss you inappropriately if it looks real purdy. Lucky you. I’m generalising sure – I’ve created dashboards for financey people and other more-data-centric types but the majority of the audience is not. If they were, they’d be using ProClarity or Excel to do analysis not a dashboard. I will say though, the finance people claim to be data-oriented but show them a pretty graph (which links out to some data) and they’re putty. FTW!

Indicators are your stoplights, and smiley faces (why? Why? I ask you!) and gauges. The heart and soul of the scorecard. It’s what people look at before they look at anything else. PerformancePoint comes with a tonne of them.

The screen shot above shows the categories you have to choose from – and each category contains umpteen indicators. Plus you can add your own custom ones. Nice.

5. Create KPIs – both Actuals and Targets

As discussed in the previous article in this series, planning your KPIs is critical. There are a few planning tips I would suggest:

• For your initial deployment, limit the number of KPIs to no more than 10
• Automate as much as is humanly possible; including Targets
• Don’t try to be too clever – measure things that are easily understood by the reader
• Don’t forget Targets under the weight of ETL’ing your Actuals – it ain’t a KPI without a Target and Targets are not as easy to design
• Make sure that the readers have agreed with each other on what the measurement or business rule is to be

So you have a list of KPIs, you’ve loaded the Actuals and Targets into your datasource and now we need to create them in Dashboard Designer. It’s as simple as hitting New KPI. The designer presumes you want an Actual and a Target and it will add your previously select set of indicators to the Target.

See how under Data Mappings, it says Fixed Values. That’s what it defaults to. A fixed value. Unlikely that you want to use a fixed value and luckily it’s as easily as clicking on it to map it to your data source. You then hit change source, select your cube and map it to a measure enter some MDX if you prefer – particularly handy if you were too lazy to get your Target values into your datawarehouse and you just want to calculate them on the fly.

You can also specify how you want the KPI to be displayed on the scorecard using the Number Format column eg with commas, to 2 decimal places etc or you can let PerformancePoint work it out based on the format of your measure in the cube.

Thresholds is where we define what’s good, what’s bad, what’s ok and what’s a disaster!

Calculation is where we define how our KPIs aggregate. Sum is the default.

We probably also want to group our KPIs into objectives such as Financial, People, Environment etc so we will need to create those as well. It’s the same process as creating a KPI.

When creating a KPI or objective, on the first page of the wizard, you’ll also notice a little check box “Grant Read to All Authenticated Users”. KPIs can be used by any number of scorecards or dashboards. That’s part of the beauty of them. Say you’re developing a dashboard for a particular department and the finance team already has a dashboard which is a global view, you can re-use their KPIs and just filter it to your particular team or department on your dashboard. If however, the data in the KPI is sensitive you may never want anyone else to see it so in that case, you would untick the option. Remember that PerformancePoint is intended for development by business analysts or other technically minded super-users so the idea is that everything can be shared and re-used as needed.

6. Create a scorecard

Now we have the core elements for our scorecard so we can build it. Like everything in PerformancePoint, right-click and Add New will do the trick. There are a few templates available to you, which just sets it up with a default layout and data source mapping. The options available are:

• Microsoft Analysis Services
• Standard
  • Blank
  • Fixed Values
    Handy for creating mockups for showing customers, managers etc
• Tabular
  • Excel 2007
  • Excel Services
  • SharePoint List
  • SQL Tabular

The option you pick really comes down to what type of data source you created way back in the beginning. Mostly, I find myself choosing Blank just because it’s easier to fiddle around with exactly what I want. A browser appears on the right hand side, which shows all the objects that are available to the scorecard and you can drag & drop them onto the scorecard. With KPIs, guides appear showing you where to drop the object in order to make it a parent, a child or a sibling. KPIs can be children of other KPIs or of Objectives or they can sit by themselves – it’s up to you.

You can also add dimensions to the scorecard (presuming they exist in your cube of course)!

7. Define how KPIs Rollup

Once our scorecard has been created, we need to decide how our KPIs aggregate. There are 3 types of rollup:

• Average weighted rollup
  Displays an indicator that reflects a calculated score, combined with the weighting that is assigned to each child.
• Indicator count Rollup
  Displays a count of the indicator images that are used in individual KPIs;, or child KPIs of each objective KPI eg
• Worst child rollup
  Displays an indicator that reflects the child KPI that has the worst score.

8. Create a Report

Dashboards are more than just scorecards. We can also add reports of varying types.

In most examples the report will need to already exist somewhere eg on our SSRS portal or in an Excel Services document library in SharePoint however Analytic Chart and Analytic Graph can be created wholly within PerformancePoint. A simple drag & drop interface is used to build these on-the-fly reports.

Note:  I will be discussing reports in much greater detail in the next article in the series.

9. Create a dashboard

We have reports, we have a scorecard so now we can build a dashboard. PerformancePoint gives us a few default layouts to start with.

Most often, I find the Header, 2 Columns is the one I go for so I can stick my Page filters in the top row, a scorecard on the left and a report on the right. Once the empty dashboard is created, the browser pane on the left will show you a list of objects that can be dragged & dropped onto the drop zones.

Drag the scorecard to the left column, the analytic report to the right zone and oh, now we need some Page Filters.

Time is the most obvious one, so start there. Time Intelligence will respect and understand your “CurrentPeriod” selections made earlier. You can add more filters (from other dimensions) using Member Selection or MDX query etc. These Filters will now appear in the object browser pane for you to drop onto the dashboard. In the example given, they’d go into the Header zone.

Of course it doesn’t end there, somehow the Page Filters have to be linked to the scorecard and the report in the other zones. Simply drag the filter onto the “Drop Fields to Create Links” drop zone within each object that you want to link.

Each object has an “endpoint”. A popup will appear that allows you to define how the filter and the object should be mapped to each other. It could be from a unique member or it could be a formula – this would be mapped to the endpoint for the object you dragged it to.

10. Publish the Dashboard

Yep that’s actually it. Dashboard done. From here it gets published to a SharePoint document library. You will be prompted to enter the SharePoint URL and select the appropriate library. What does that mean? Well, you need the library to exist, you need write access to it and your SharePoint installation needs the PerformancePoint web parts installed.

That dear friends, is the whole shebang!

	Part 1 – Getting to Know You – Intro to SharePoint BI
	Part 2 – Sell! Sell! Sell! Why Build a Dashboard Anyway?
	Part 3 – KPIs – Where do I Come From?
	Part 4 – Dissecting Dashboards
	Part 5 – We Really Need to Talk – Scorecards & Reports
	Part 6 – Beware the Bogeyman – Securing Dashboards

Siyonara amigos! Till next time.

Kristen Hodges, MCTS, MCITP
http://www.bi-tch.blogspot.com

http://b-iq.org

References

http://office.microsoft.com/en-us/performancepoint/HA102711891033.aspx

Or would it? It may be that we allow for 10% leeway before we say that our variance was bad. So in our example, we would actually have a green light, or maybe orange. We would only get a red light if our variance was greater than -$10,000. Let’s face it; we’re still making money, just not as much as we had hoped.

Did you notice that I didn’t say “Revenue” was the KPI? In fact, I said that the performance of Actual Revenue to a target is the KPI i.e. the variance. This is important. KPIs measure PER-FOR-MANCE.

So, a KPI measures how well we performed against a given target.

That makes sense right? So what’s the fuss about scorecards and KPIs and all that? Well… balanced scorecards are designed to show overall performance of the organisation. Not just easily quantifiable things like Profit Margin. And how do you compare, equate and aggregate information such as Employee Satisfaction, Customer Satisfaction, Operational Effectiveness, Environmental Sustainability and Profit Margin altogether?

 

Where do I Come From?
	Now we get down to the meat of it. And what this has to do with SharePoint. Someone, somewhere has to actually physically build our KPIs, store them, access them and use them.  What are My Options? PerformancePoint Monitoring & Analysing PerformancePoint delivers excellent reusability and sharing of KPIs, customisable indictors. Coming from the Office suite of tools, it is a UI* which doesn’t require IT involvement – it is aimed at BAs and super-users. Integrates with existing technologies, SQL and SharePoint. SharePoint PerformancePoint still uses SharePoint as it’s UI but here I’m talking about building KPIs themselves. As of Office 14, PerformancePoint M&A will be entirely bundled into MOSS. Which I guess means that SharePoint’s existing rudimentary KPI building functionality will disappear. Yes, it’s rudimentary and simple – it’s ok for really small stuff but if you’re from an IT department, my advice is, don’t waste your time. SSAS Get your MDX skills going because you’re gonna be writing a lot of MDX queries. That’s not a bad thing. I’m just saying that’s what you will be doing. Also, all information relating to a single KPI must be contained within that cube where the KPI sits. And you’re gonna have to create the interface using Excel or suchlike yourself. Numerous Other Software Vendors It all comes down to cost doesn’t it? And integration. And user familiarity with the UI*. I know I’m biased. I like PerformancePoint. It integrates nicely with technologies that I’m already using (SQL and SharePoint). And I like that it’s gonna be bundled with MOSS. So sue me (no, don’t really)… * user interface
Who Am I?
	Profit – What Else is There?   I could give you a massive list of KPIs, but I ain’t gonna. Check out the EPM Review. They have an EXHAUSTIVE list of common KPIs. It’ll knock your socks off… or provide just the impetus you’ve been needing to clean out your garage. Who am I to judge? Just remember this… you want to measure performance in all areas that are important to you. While there is some commonality across organisations, there is by no means uniformity. This means, the very first thing you’ve got to do, is consider your organisations values, mission and strategic plan. This will guide you on what’s important FOR YOUR ORGANISATION. Once you know what you want to achieve, then you need to identify what data you already have that can be used to quantify those areas. Then consider, what data you could capture but don’t already have. Finally, don’t forget to think through and identify what data you can’t capture. It may be just as important to know what’s missing. How Long is a Piece of String Defining targets is not as simple as it seems initially. Again, this is something you need to think through at the outset. Targets could be manually entered values, but based on what? It could be as simple as Revenue Last Year + 4%. But someone somewhere has to decide how many percent, and whether that base revenue number should include Revenue from discontinuing entities, or interest or a number of other factors. The fact is, you need to think this through separately for each and every KPI.
Getting on the Merry-Go-Round
	Common issues you will face when designing your KPIs…  Defining the KPI What’s the formula? Each segment of the organisation defines things in slightly different ways. In order to measure them effectively, a corporate view needs to be taken. This can be tricky. If you want the data used to derive the results to be meaningful then you either have to (a) ensure it’s 100% automated and requires no business unit level input or (b) find a way to get everyone to agree on a common definition. Good luck with that! Weighting the KPI Some KPIs are more important than others. Well, maybe not more important but certainly the impact on the organisation of poor performance, or conversely good performance, is more greatly felt. So we need to identify what matters. Again, getting everyone to agree can be challenging. But this is an important thought process and should be well considered before jumping in the deep end. It helps to look at your strategic plan and that may point you in an appropriate direction. Too Many KPIs I see this time and time again. How do you hit the bullseye when you’re aiming everywhere? It’s got to be achievable, not the KPI targets but the project itself. Getting a scorecard up and running is challenge, no doubt about it. But getting people to use it, now that’s the real challenge. So start small. I’ll say it again. Start SMALL. Did you miss it? START SMALL!! Get users comfortable with a few key areas before you throw a thousand or a hundred or even 50 KPIs at them. They just won’t read it. They won’t understand what they are looking at. My suggestion, seriously, start with 10 at the most. I know there are BAs and scorecard developers out there looking at me in stunned silence (well, reading at me in stunned silence… or perhaps not, perhaps you are yelling at the screen (if so, ask yourself why because I can’t actually hear you). Of course, you can revisit it later… just don’t start with information overload. That’s all I’m saying.
	Part 1 – Getting to Know You – Intro to SharePoint BI
	Part 2 – Sell! Sell! Sell! Why Build a Dashboard Anyway?
	Part 3 – KPIs – Where do I Come From?
	Part 4 – Dissecting Dashboards
	Part 5 – We Really Need to Talk – Scorecards & Reports
	Part 6 – Beware the Bogeyman – Securing Dashboards

Siyonara amigos! Till next time.

Kristen Hodges, MCTS, MCITP
http://www.bi-tch.blogspot.com

http://b-iq.org

References

http://www.epmreview.com/KPI-Library.html
http://nickbarclay.blogspot.com/search?updated-max=2008-01-09T15%3A44%3A00-05%3A00&max-results=20

In Part 1, I covered the purpose and technical architecture of the BDC.  A quick look at the BDC in action was also introduced.

In Part 2, I covered the Application Definition File (ADF) in depth and how to develop one to connect to a back-end database.

In Part 3, I continued the discussion of the ADF, but looked at how to develop an ADF to connect to Web Services.

In this Part 4, which is much less technical, the focus is two primary ways to consume BDC data through the UI.  This will include Business Data Web Parts and incorporating Business Data into SharePoint lists.

Business Data Web Parts

Business Data Web Parts are the most common way to surface BDC data to end-users.  Using them, users can filter and sort the data to quickly find the content needed.  Their flexibility and ease of use empowers users to perform many tasks that would otherwise require custom development.  In MOSS Enterprise edition, there are five Web Parts and we will cover each in this article:  Business Data List, Business Data Item, Business Data Related List, Business Data Actions, and Business Data Item Builder.  Each of these Web Parts will only provide a read-only view of the external data, so don’t plan on making any changes through them.  There are ways of updating the external data, but we will wait until Part 8 to get into this.

Business Data List Overview

The Business Data List is the most powerful and most commonly used Business Data Web Part.  At its heart, it is a Data View Web Part and is used to display a sheet of columns and rows (or items).  The items displayed are based on filter criteria that the user enters.  Here is a simple example where we find all customers located in the WA (Washington) Region.

Figure 1: Business Data List Web Part

These customers that you see are defined in a SQL Database, and the information is pulled in real time.  This is a powerful way to expose your line-of-business data within a centralized SharePoint interface.

Let’s quickly walk through how I got here.  As with any Web Part, you start off by editing a Web Page (Figure 2) and adding a Web Part into one of the Web Part zones (Figure 3).

Figure 2: Editing the Page from the Site Actions Menu

Figure 3: Adding a Web Part to a Zone

After clicking Add a Web Part, you will be presented with all the Web Parts that are available for that site collection.  In the Business Data category, you should see the five Business Data Web Parts.

Figure 4: Business Data Web Parts

After you select the Business Data List Web Part and click the Add button, you will be presented with this next screen, prompting you to now configure the Web Part.

Figure 5: Default display after adding a Business Data List Web Part to a zone

After you click the Open the tool pane link, you are presented with all the configuration properties for the Business Data List Web Part.  The required setting is that you must select the Type from the catalog as shown here.

Figure 6: Specifying the type

The Type refers to the entity in one of the ADFs that have been loaded.  The entity must also have a finder method defined.  You can either enter in the name directly and click the Check Types icon .  Or, you can select the Browse icon to see what is available.  (Security Note: Only those applications and types that you have Select in Clients and Execute permission on will be shown.)  In the image shown below, one of the four types can be selected.

Figure 7: Browsing applications and types stored in the catalog

In the first screen shot above (Figure 1), you saw a list of customers.  To get this, I selected the Customers type from SynergyData (the name of the BDC Application).  Initially, this will return all of the columns that are defined in the Finder method within the ADF as shown below.  (Note: I also exited Edit Mode to return to the regular view).

Figure 8: Business Data List Ready for Action

At this point, the Web Part is ready for filtering and sorting operations.  To filter, you first choose one of the Filter choices from the drop down list shown on the top left as shown here:

Figure 9

What is shown are the filter options that are defined by the FilterDescriptors as covered in Part 2.  Here is what is defined for this Customers entity in the ADF:

Figure 10

If the type of filter is a Wildcard, you will have several filter options that you can select in the second drop down list as shown here:

Figure 11

If you’ve specified Comparison, you only have one choice:

Figure 12

You can add additional criteria by clicking the Add button and adding another filter.  These are all processed as an “And” operation, meaning that only items that match all criteria are shown.  To sort, as with any SharePoint view, you simply click on any of the columns headings.

You can also filter by using the column headings.  In this example, I start off by searching for all customers in the SP region.  This initially returns six items.  I then filtered the display to only show those in Sao Paulo.  The advantage is that you can still perform some filter operations even though the column is not defined as a filter descriptor.  Here is the result:

Figure 13: Column Filtering on City

What if you wanted to just filter columns?  For example, perhaps you only need to show a handful of columns for each customer.  This is easy and is done by editing the default view for the Web Part.  To do this, you can go back into edit mode for the page and click the Edit View link as shown here:

Figure 14: Editing Business Data List View

This will then take you to a screen where you can adjust the view in a number of ways.  One of the choices is to select the columns as their position.  As you can see below, this works just like a regular view for a SharePoint list.

Figure 15: Selecting columns to display

From this same screen, you also have some other useful features.  For example, you can “hard code” a specific filter so that only certain items display.  For example, perhaps each region has its own SharePoint page, and on that page, you only want to display the customers in that region.  You could configure the settings as shown here and prevent the user from changing the filter:

Figure 16: Only show customers in the SP Region

Here is the result:

Figure 17: Customers in the SP Region (and user cannot change the filter)

While editing the view, you have many more options such as the default sort and additional filter options.  I’ll leave these to your own study.

Business Data List Advanced Settings

Let’s turn our attention to more advanced capabilities.  Since the Business Data List Web Part is a type of Data View Web Part, you have a number of settings such as customizing the XSL or Parameters.

The XSL (eXtensible Stylesheet Language) is an XML-based transformation language that converts XML that is returned from BDC into HTML that is displayed for the Web Part.  This XSL is generated automatically when you select a catalog type such as customers.  While you can edit this XSL directly inside a dialog box for the Web Part, it is much easier to copy and paste this into a file and edit within Visual Studio, SharePoint Designer or your favorite XSL editor.  Keep in mind that the XSL contains not only the HTML results, it also contains the HTML for the selection criteria, the Actions toolbar, and the column headers.

A simple XSL change would be to enable alternating line styles.  The result looks like this:

Figure 18: Modified XSL to highlight alternating items

To enable this, I simply inserted the <xsl:if> condition you see here highlighted in this code snippet:

Figure 19: XSL

When defining the XSL, you can either link to an XSL stylesheet (located within a document library) or define the XSL for this specific Web Part instance.  Linking the XSL would be best for identical Web Parts that you have in different locations.  Since the XSL is generated for the specific columns, a shared XSL file cannot be used different Business Data Lists with different types (entities).  Having a linked XSL sheet is also helpful to ensure that changes made to the Web Part will not erase your custom XSL code.

For those not too comfortable with XSL, you have some control over the Web Part using SharePoint Designer (SPD).  Like the changes to the view we saw above, SPD also allows you to define filtering and sorting behavior.  One advantage with it, however, is that you can define filters based on variables such as the query string.

To help explain, let’s adapt the Region example we just covered in Figure 17 above where we hard coded the filter for the SP Region.  In this case, let’s pull that Region from the query string instead which would give us a more flexible design.

This section assumes you have a basic understanding of editing pages in SPD.  For a brief overview, see my friend Ishai Sagi’s introduction here.  Once in SPD, open up the page for the web site where the Web Part lives.  We click on the Web Part and access the Common Data View Tasks by clicking on the tiny icon as highlighted here in red:

Figure 20: Customizing Web Part in SPD

Once you click the icon, it will expand the Common Data View Tasks dialog as shown here:

Figure 21: Common Data View Tasks Dialog

From this point, click the Parameters link.  Click the New Parameter button and fill in the contents as shown below.  This will create a new parameter named Region that will be pulled from the query string.

Figure 22: Adding a New Parameter

Now that the Region parameter is created, we can create the filter that is based on its value.  Return to the main Common Data View Tasks dialog and this time select Filter.  Create a new filter by as you see here.  Note: The [Region] value is the parameter that you just created.

Figure 23: Adding a New Filter Based on the Parameter

Save all of your work in SPD and refresh the page in the browser.  If you do nothing else, you’ll find that your Business Data View Web Part does not return any items.  This is because the Web Part is expecting a Region query string value.  Edit the URL in the address bar, and add the query string and value after your page.  For example:

http://WebApp/teamsite/default.aspx?Region=SP

Now run a new query using the Web Part.  You should find that all the item results are automatically filtered to only the SP region.  If you prefer, you can supply a default value for the parameter when you create it as shown in Figure 22 above.

Important Note: When editing the Web Part using SPD, it will change the current XSL and Parameter settings.  So, be careful if you will be customizing the XSL manually and also through SPD as it will overwrite your changes.

Business Data Item Web Part

Now that we have thoroughly covered the Business Data List Web Part, the remaining Web Parts are much easier to learn.  The next one on our list is the Business Data Item.  This Web Part is intended to just show you the column values for a single item that is returned from BDC.  It is often used in conjunction when you connect Web Parts as you’ll see shortly.  Before we go there, let’s first look at a simple example.

Here you see the contents of one customer:

Figure 24: Business Data Item Web Part

To configure the Business Data Item Web Part, first set the Type from the catalog picker.  Then specify the item by searching for it.  Here is how it the configuration looks:

Figure 25: Configuring the Business Data Item

While this is easy to do, it’s not too practical to hard code in a single item.  Instead, let’s see how you can connect this to our Business Data List Web Part.  This will allow you to select a customer from the list and then display the customer’s details.  To start, add a Business Data List and Business Data Item Web Part on the same page.  They can be in different zones but must be on the same page.  You must also make sure they are both set to the same Type (such as Customers).  From here, you need to establish a Web Part connection between them. You can do it as shown here:

Figure 26: Connecting Business Data List to Business Data Item

This will send the selected item from the Customers List (on the left) to the Customers Item (on the right).  Once these are connected, you get the display as shown below.  Notice that selecting The Cracker Box shows this customer in the Business Data Item Web Part on the right.

Figure 27: Result of Web Part Connection

Business Data Related List Web Part

The Business Data Related List Web Part is used when you want to pull related items using an association defined in the ADF.  For example, perhaps we want pull the sick days a certain employee has taken.  Or we want to list the orders for a selected customer.  Let’s see how this latter example would work.  To start, let’s add a Business Data List and Business Data Related List Web Part to our page.  As we have been doing, we configure the Business Data List to use the Customer Type in the catalog.  We then point the Business Data Related List to the Orders Type.  Here’s how the Business Data Related List looks when configuring it:

Figure 28: Configuring the Business Data Related List

Notice that you must specify the relationship on the Orders Type that you want to use.  This choice is provided to us since it was defined in the ADF as you can see here:

Figure 29: Association between Customers and Orders in the ADF

Now that the Web Parts are configured, we need to establish a connection between them.  Once that is done, we get the following result:

Figure 30: Result of Connecting Business Data List and Business Data Related List

Of course, if you prefer, you can have the selected customer drive both the Orders List and the single customer in the Business Data Item.  Here’s how it would then look:

Figure 31: Connecting Three Web Parts Together

Business Data Actions

Actions are a common way of displaying or editing a single item, and we will cover them in detail within Part 8.  To give you a basic overview, let’s look at a sample action that is defined in the ADF.  Here is one that is defined for our Customers entity:

Figure 32: Action Defined for Customer Entity in ADF

Notice that the action has a name (in the figure above it is View Profile) and contains one or more ActionParameter elements.  This column (or columns) is often the Identifier for the entity.  This means it is used to uniquely identify this item.  For our customer’s entity, the CustomerID will be provided as a query string variable to a page called dbo.Customers.aspx.  How this is processed is covered in the Business Data Item Builder Web Part a bit later.

Now that you understand the basics of actions, the Business Data Actions Web Part simply lists all of the actions defined for a single item.  Here is how the configuration for the Web Part looks:

Figure 33: Configuring the Business Data Actions Web Part

Notice that you configure it much like the Business Data Item Web Part.  The only difference is you can specify which actions to display and how should they be displayed (i.e. the Style).  The Style options are Bulleted List, List, and Tool bar.  Once you have it configured (and connected to a Business Data List Web Part), here is how the results look with a Bulleted List:

Figure 34: Connecting the Business Data List and Business Data Actions

When you click on the View Profile link, it takes you to the Url defined for this action which is http://spserver:22222/ssp/admin/Content/dbo.Customers.aspx?CustomerID=LAZYK.  Incidentally, you can also access these actions by clicking the View Profile context menu as shown here:

Figure 35: Accessing the Actions from the Business Data List

As you can see, this is a flexible way of sending a single item to any web page, whether this is in SharePoint or an external web site.  We’ll look at other possibilities with Actions in Part 8.

Business Data Item Builder

The Business Data Item Builder is a special, invisible Web Part that is responsible for extracting the value from a query string variable and delivering it to a Business Data Item Web Part.  By invisible, I mean it is only visible in edit mode; when in normal operation, it is hidden.

When adding this Web Part to a page, there is no configuration on it other than connecting it to a Business Data Item Web Part.  The way it works is by extracting a variable from the query string (for example CustomerID) and sending it to the Business Data Item Web Part.  It knows what variable to extract by reading the ActionParameter defined in the ADF.

A good way of understanding how the Business Data Item Builder Web Part works is by looking at one of the profile pages.  A profile page, which I will cover in Part 8, is a page that typically displays a single item in some way.  Profile pages can be created automatically when you import the ADF.  For an automatically created profile page, this is comprised of a Business Data Item Builder and a Business Data Item Web Part.

To demonstrate, when clicking the View Profile link shown in Figure 35 above takes us to the auto-generated profile page as shown below.  Notice you only see one Business Data Item Web Part.

Figure 36: Profile for a Single Customer

If we edit the page, however, you’ll notice the Business Data Item Builder.  This web part is automatically connected to the Business Data Item Web Part.

Figure 37: Editing the Profile Page for Customers

That is the only purpose for the Business Data Item Builder, so you will probably not find yourself using it all that often.

With that, this wraps up our discussion on the out of the box Web Parts for the Business Data Catalog.  Let’s turn our attention to the second (and much shorter) half of this article.

Incorporating Business Data within Lists

While the Business Data Web Parts are useful, sometimes you just need to embed external data right within a SharePoint list or library.  For example, perhaps you already have a contact list in SharePoint for all your customers, and you are maintaining this separately from your CRM system.  A better solution would be to add Business Data columns to your contact list and pull in some of this data from CRM.  This means you would only maintain the customer entry in CRM, and these changes are pulled into SharePoint.

Another advantage is that you can see information from multiple sources all merged together.  Let’s say that a list of the products your company sells is stored within a SharePoint picture library.  You can add a Business Data column to the library which can pull the current product name and inventory level from an external ERP database.  Now you can see the product’s picture, the product name, and its inventory in one consolidated view.

When you add a Business Data column to a list, SharePoint will pull the data from the external system and store a cached copy of it in the list.  This is quite different from Web Parts where you are always pulling the real-time value from the external system.  The advantages of caching the data in the list are faster performance and the flexibility of having the data stored right in SharePoint.  The drawbacks are the that the additional columns of data will increase the size of the list (and content database).  Also, for volatile data that changes often, the cached data will quickly get stale.  As you might guess, the common-sense rule of thumb is to only pull data that does not need real-time status.  However, as you will see, it is easy to refresh the cache.

Just like the Business Data Web Parts, you should consider the cached data in the list as read only, and SharePoint does not give you an interface to change it.

Now that you have a basic idea how this works, let’s take a look at the picture library example that pulls the current product name and inventory value from a database.  For this demonstration, I already have a Products entity defined within an ADF.  I have created a picture library that has three product pictures added.  The All Pictures view looks like this:

Figure 38: Initial View of the Pictures Library

From the Settings menu, let’s add a Business Data column. The column named will be called Product Name as you can see here:

Figure 39: Adding a Business Data column called Product Name

When you add a Business Data column, you need to specify the Type, just as you would with the Web Parts.  Below you can see I have selected the Products Type.  I have also added two columns as you can see here

Figure 40: Selecting the Columns from the Products Type

Notice that you can add as other columns if you like.  I have added the ProductName (the main column in the drop down list) and UnitsInStock which gives us the current inventory value.  After we add these two columns, here is how the All Pictures view now looks:

Figure 41: All Pictures View after Adding Business Data Column

This is nice, but you can see that it doesn’t automatically pick up the values from the database.  When you add a Business Data column to a list, you need to map up each item in the list to the matching item in the database.  This is a 1:1 mapping.  Here is how you could manually do this using the UI.  Go to the edit properties for an item, and you’ll see the Business Data column as you see here:

Figure 42: Matching up Product Name Item

In the text box, you can type in a unique product name and click the Check Names icon .  Or you can also click the Browse icon and search for it.  Here is how the browse works:

Figure 43: Finding Matching Row

In the figure above, we are editing the flow regulator picture, so we search for this product by name.  This just executes a search on the Finder method and returns all the products which have this name.  For this product, there’s only one with this name, so we select it.  We are then returned to the edit item page and we accept the changes.  We have now mapped this one product’s picture to its matching database row.  I won’t show you, but I also mapped the other two pictures as well.  When returning to the All Pictures View, you can see that the two columns’ values were pulled from the database and stored in the list:

Figure 44: All Pictures View Now Showing External Data

Remember these values are cached in the list, so when the inventory changes, this will not be automatically reflected in the list.  Without writing code, there is no automated way to refresh this.  It’s very easy for a user to refresh the list, and you do this by clicking the refresh icon listed next to the Product Name column heading.  After clicking refresh, you are greeted with this confirmation page:

Figure 45: Updating Cached Business Data in List

In this case, updating three items is minor—if you had thousands (or more) items, it could take a while!  Security Note: If a user only has read permissions to the list, they will still see the refresh icon but will not be able to refresh its data.

Another drawback to this that you might notice is that it can be quite tedious manually mapping up each item in the list to each row in the database.  You may be able to automate some of this through code as I discuss in this blog article.

At this point, you have seen how to add Business Data columns to a SharePoint list.  Now that these are cached within the list, the data stored behaves almost like regular columns.  You can filter, sort, group, all the usual things.  Remember, though, you cannot edit through the list and have the change stored back in the database.  In other words, the synchronization is not two way.  In fact, the columns that BDC adds to the list are read only.

Wrap Up

Great job on making it all the way to the end.  You didn’t just skip down here did you?    I know this was a long article, but hopefully you were able to go through it quickly, and it gives you a strong grasp on everything you need to know with Business Data Web Parts and incorporating Business Data within your SharePoint lists.

Coming up next in Part 5, we’ll take a close look at how the search engine can crawl BDC data so that your searches encompass your business data as well.  Until then…

Why Web Services

While relational databases have proliferated greatly over the last 20 years, there are some inherent difficulties with them.  For starters, they are often proprietary.  While structured query language (SQL) is an ANSI-standard language, vendors choose to develop their own flavor of SQL such as PL-SQL with Oracle or T-SQL with Microsoft.  In order to query from a database, you need a vendor-specific driver, making integration between systems difficult.  Furthermore, the network connection is often based on remote procedure calls (or RPCs) that won’t typically work over an HTTP connection.  This makes connecting to them over the Internet or through firewalls a challenge.

Web Services have matured in the last few years as the de-facto way of integrating systems together.  Service-oriented architecture (SOA), which is based on Web Service technology, offers us a very compelling approach to building and integrating systems today.  Its loosely-coupled nature and vendor-neutral design make it far more flexible and practical for the changing needs of organizations.  Exposing line of business data over Web Services gives you more control, more universal access, HTTP compatibility, and is in line with emerging technologies such as cloud computing.

Since a primary goal of SharePoint is to be the central interface for critical data assets, it’s likely that SharePoint will need to integrate with external systems.  In article two of this series, we covered how the ADF can describe an external database.  In addition to the advantages we just covered, there may be other reasons why Web Services might be the better choice.  For example, perhaps you already have a Web Services architecture in place that you want to leverage.  Or you have data that comes from a myriad of data sources such as databases, XML files, or services in the Internet cloud.  With Web Services, your content can originate in virtually any source.  By exposing these data sources through a Web Service, you provide a single way for SharePoint to consume this data, regardless where it is located.  This is depicted in Figure 1 here:

Figure 1: Exposing Web Services to MOSS

Developing Your Web Services

Now that we have covered why Web Services might be right technology, let’s dive into a sample Web Service that we can consume from BDC.  As you can see in Figure 1 above, the data can originate from any back-end data source.  In this first example, it will come from the AdventureWorks database from a Microsoft SQL Server.  The data we want will be returned using a simple stored procedure named GetEmployees.  Here is the code for the stored procedure:

create proc [dbo].[GetEmployees]
as
select Employee.EmployeeID,
        Contact.FirstName,
        Contact.LastName,
        Employee.Title,
        Employee.HireDate
from HumanResources.Employee
join Person.Contact
on Employee.ContactID = Contact.ContactID

As you can see, we are returning four columns.  We join from the Employee to the Contact table to retrieve all the columns we need.  We then call into this stored procedure from our C#-based Web Method.  This method is coded within an ASP.NET Web Service project as shown here:

[WebMethod]
public List<Employee> GetEmployees()
{
    Database db = DatabaseFactory.CreateDatabase("Connection String");
    IDataReader employeesReader;
    List<Employee> employees = new List<Employee>();

    DbCommand cmd = db.GetStoredProcCommand("GetEmployees");

    employeesReader = db.ExecuteReader(cmd);
    while (employeesReader.Read())
    {
        employees.Add(new Employee((int)employeesReader["EmployeeID"],
                      employeesReader["FirstName"].ToString(),
                      employeesReader["LastName"].ToString(),
                      (DateTime)employeesReader["HireDate"]));
    }
    employeesReader.Close();

    return employees;
}

Version 4.1 of the data access application block from the Enterprise Library is used to return the Employee data.  This is not required as any form of ADO.NET code will work, but this simplifies our data retrieval and is a good practice.  The data returned from the Web Method is a generic list based on the Employee class.  You can consider this Employee class to be our business object representing a single employee.  Here is the part of the class with the four columns and two constructors.

public class Employee
{
    private int _id;
    private string _first;
    private string _last;
    private DateTime _hireDate;

    public int ID
    {
        get { return _id; }
        set { _id = value; }
    }
    //Additional getter/setter for remaining columns go here 

    public Employee(int id, string first, string last, DateTime hireDate)
    {
        this._id = id;
        this._first = first;
        this._last = last;
        this._hireDate = hireDate;
    }

    public Employee()
    {
        this._id = 0;
        this._first = string.Empty;
        this._last = string.Empty;
        this._hireDate = DateTime.Parse("1/1/2000");
    }
}

One key advantage of using a generic list when writing the Web Method is that it serializes nicely into XML.  It also describes the complex types in the XML Schema (XSD) in a BDC-friendly way which will make the authoring of the ADF file much easier.  When Employees are returned from the Web Method, here is how one of them will be described in XML:

Figure 2: Employee element returned from Web Service

In the Web Service, we also define two additional methods, one for an IdEnumerator and one for a Specific Finder.  For brevity sake, I only show the interface for each.  They are coded similarly, and you can download the full code at the end of this article.

public List<int> GetEmployeeIDs () //IdEnumerator 
public Employee GetEmployeeByID(int EmployeeID) //Specific Finder

GetEmployeeIDs (the IdEnumerator) just returns the ID for each Employee.  The Specific Finder returns a single Employee object based on the supplied EmployeeID.

Creating the Application Definition File

Now that we have defined a simple Web Service exposing employees, let’s look at how this is represented in our ADF.  Since the basic structure of the ADF remains the same with a Web Service as compared to a database, we’ll only focus on the areas that pertain to calling Web Services.

In Figure 3 below is our LobSystem element which is the root of the ADF.  You’ll notice that the Type attribute indicates WebService.  Inside Properties, a namespace prefix is defined for this LOB.  In our case, we call this HR (for Human Resources).  We also set the URL to the Web Service.  When this ADF is loaded into MOSS, the Web Services Description Language (WSDL) for this URL is retrieved and an internal proxy class is generated that BDC uses.  This concept is very similar when adding a Web Reference to a Visual Studio project.

Figure 3: LobSystem Root Element in ADF

Moving down in the schema we find our LobSystemInstance which is where connection details are specified.  For a Web Service, there are several options such as how to authenticate and how many simultaneous connections are allowed.  In our case, our needs are simple, so we only need to list the name of our LobSystem which is HR.  (Note: since we didn’t specify how to authenticate, this will use RevertToSelf settings which means that the application pool identity will be the credentials used if the Web Service requires authentication.)

Figure 4: LobSystemInstances in ADF

Where it gets interesting is how we start to define our methods for each entity.  The secret to writing the Web Service-based ADF is to define the TypeDescriptor elements correctly.  Each TypeDescriptor points to an element node in the Web Service XML request or response, and the basic rule is that you need to nest your TypeDescriptors to match the hierarchy.  For example, here is our Finder method that will call the GetEmployees Web Method and return a list of Employees.

Figure 5: Return Parameter for GetEmployees Web Method

Let’s go through what we’re seeing here.  First off, the Name attribute for the Method must match the name of the Web Method.  In this example, it is GetEmployees, which we introduced above.  The TypeDescriptor elements that are highlighted reflect our hierarchy in the response.  The first one defines an array of Employee elements (denoted in part by the IsCollection=”true”).  The second one defines each Employee, and the final four define each column that is returned.  In a snippet of the actual XML response shown in Figure 6 below, notice the Name attribute for the TypeDescriptor matches the Element name.  This name match-up is not required in all cases but is recommended.

Figure 6: Actual XML Response from Web Service (GetEmployees Method)

Let’s look at one more example.  Here is our IdEnumerator which returns the IDs for all Employees.  Here is the method we define in the ADF:

Figure 7: ADF Entry for IdEnumerator

In this case, we return an array of Int32 (32-bit Integers).  Below, here is the first part of the the corresponding XML that is returned from the Web Service.  Notice how the two levels in the XML, ArrayOfInt and int, align (i.e. int is a child element of ArrayOfInt).

Figure 8: Actual XML Response from Web Service (GetEmployeeIDs Method)

Consuming Employees from UI

So now that we have our Web Service ready, and our ADF loaded and secured within our SSP (as covered in the Article #2), it’s time to see the what this effort yields us in the user interface.  Let’s first take a look at the Business Data List Web Part.  We will examine this Web Part in more detail in the next article, but for now, here is a basic view of the Employee entity from our catalog:

Figure 9: Employees showing in a Business Data List Web Part

Here is a recap of how this executes:  These records are pulled in real time through a proxy class created from the ADF, through the Web Service and into the stored procedure running inside SQL Server.  Since we have defined IdEnumerator and Specific Finder methods, we can also set up a Content Source and crawl this BDC data.  This allows us to issue search queries against the external system, and this will be covered in depth in Article 5.  For now, I have set this up with the Employee entity in the ADF, and this is the result when we search for Rob (someone’s first name):

Figure 10: Search Results for Employees Based on Name

And finally, let’s look at an auto-created profile page.  This is created by the BDC that allows us to view all the details for a single record.  These types of pages will be covered in more detail in Article 8.

Figure 11: Profile Page for a Single Employee

With that, you should have a basic idea in how the ADF needs to match the response from the Web Service.  If you’ll be developing these ADFs from scratch, I advise you to look at the WSDL provided by the Web Service to understand the Complex Types in the XSD.  I also recommend looking at the actual XML data that is coming back.  From here, and with some practice, you will be able to construct your ADFs.  The process for input parameters (for example, with FilterDescriptors) is the same.

Amazon Web Service

The second example we’ll look at is a bit more complex.  Within the MOSS SDK, a sample ADF file exists to call into Amazon to do a search for a product.  This is quite powerful and great demonstration of services in the cloud.  For this section, we will look at a modified form of the sample ADF, so even if you’ve seen this, I recommend you keep reading.  Let’s first show you how this looks from the user’s perspective.  I have added a Business Data List Web Part to a web page and have pointed it to the Amazon entity in the catalog.  Here is how it shows the results when searching for SharePoint books:

Figure 12: Keyword Search Against Amazon Web Service

As you can see, it has returned the first ten titles which match these keywords.  The book’s ISBN and Title are shown, and Title links to the actual book URL on Amazon’s web site.  This was done by pulling the raw data from Amazon through the BDC and displaying it with a custom XSLT Stylesheet.

To understand how this works, let’s first understand the XML that will be delivered to BDC.  Here is what a single book (an Item) looks like in the XML Response from the Web Service (Note: I have collapsed the ItemLinks element since it’s not used in this demonstration):

Figure 13: Actual XML Response from Amazon Web Service

As you can see, we have the ISBN returned in an element called ASIN.  We also have URL stored in the DetailPageURL element.  Lastly, details about the book are stored in a child element called ItemAttributes, and this is where we need to extract the book’s title.  Taking a look at the XSD defined in the WSDL provided by Amazon, we see the definition for each Item and ItemAttributes.  (Note: I have edited the file to only focus on these two complex types).

Figure 14: A Selected Portion of Amazon’s Web Service WSDL

Now that we know what we’re dealing with in terms of schema, let’s see how the ADF comes together.  Again, I have modified the sample Amazon ADF file.  Specifically, I have removed an unused entity, and added additional logic to expose the book’s title.  Here is our single entity, called Item.

Figure 15: Item Entity Defined in ADF

Keywords is the friendly name for our filter as you can see in the screenshot in Figure 12 above.  As part of the input parameters, we provide the value of this Keyword, a subscription ID (required by Amazon to call into the Web Service), and some default values for what we’re searching (Books in this example).  These won’t be shown for brevity sake.  The concept for these input parameters is the same as the returned output in that you must define hierarchical TypeDescriptor elements to match the XML request.  With that, let’s focus on the returned response.  The hierarchy here is pretty deep, so we’ll just focus on the inner-most section that begins with an array of items.  Here is how this looks:

Figure 16: TypeDescriptor Hierarchy for Returning Items

It might look imposing at first glance, but if you follow the arrows, you can see how this maps up with the actual XML response shown in Figure 13 above.  The tricky part with this example is that we want to return Title to be at the same level as ASIN (the ISBN) and DetailPageURL.  In the XML response, Title is nested in a child node called ItemAttributes.  This becomes a problem because we need to return a flat set of columns and rows just like in a sheet in Excel.  In our case, the desired columns are ISBN, DetailPageURL and Title.  In other words, hierarchies are not allowed.  So, we somehow need to “promote” Title up one level.

We solve this problem by first setting a localized name for ItemAttributes to be Title.  This just means we rename this column to Title.  To get the actual value for Title, we use what’s called ComplexFormatting.  ComplexFormatting allows us to generate the value, and we do this using the FormatString property.  The {0} (just as with the String.Format method) will generate a new string from the value of a child element called Title.

With this described ADF, we have now effectively mapped the fairly complex result from Amazon into a pretty simple display.  If we load the ADF as is and add a Business Data List Web Part that points to this Item entity, we get the following result:

Figure 17: Raw Output of our Item Entity

Here are our three columns, ISBN, DetailPageURL and Title.  Of course, we would like the Title to just be a hyperlink to the specified URL.  We can handle this with a small change to the Web Part’s XSLT.  This is what yields the display shown in Figure 12.

Recommendations

Now that you have a pretty good idea how to structure your own, let me leave you with some recommendations that should help you construct your Web Service-based ADFs.

• The BDC only supports Simple Object Access Protocol (SOAP)-based Web Services, not REST-ful (REpresentational State Transfer) ones.
• Due to the structure of the ADF files, you will not easily be able to consume existing Web Services unless they closely match the concepts of Finder, Specific Finder, and IdEnumerator.  You might find yourself having to write specific Web Services, which is why I provided a sample on how this can work.
• If your Web Services are developed using the approach used above, you should have some success working with ADF generation tools like BDC Design Studio, BDC Meta Man, and BDC Definition Editor.  If not, you will likely need to generate the ADFs by hand.  Even when some design tools, the effort is more involved as the utility cannot infer the Identifier or FilterDescriptors for entities.  In some cases, you might get part of the ADF described using a tool, and then you’ll need to go in and make some tweaks to it.
• If you find yourself needing to write a lot of Web Service wrappers around your existing databases, you might find a utility from the same company that makes BDC Meta Man helpful.  It’s called BDC Web Man, and you can find it here.
• Keep your returned Web Service XML structures simple.  As you can see, deep hierarchies are much harder to represent in the ADF.  You’ll probably find it much easier to adjust the Web Service output than trying to craft the ADF to work with a complex structure.  For example, while you can return and consume a DataSet in the ADF, the amount of nesting makes this a very complex task, and you’ll need to code it all by hand.
• As much as possible, review the WSDL file and actual XML results when structuring the ADF file.  This will help ensure you are representing your TypeDescriptor hierarchies correctly.
• If you are developing new Web Services using .NET technologies, you can use either classic Web Services (i.e. ASMX) or the newer Windows Communications Framework (WCF) style, provided you configure it to support SOAP.
• Since your Web Service WSDL will be downloaded and parsed when you load the ADF into your SSP, make sure the Web Service is physically accessible at that time.
• Authenticating to the Web Service through HTTP or SOAP headers is supported through the BDC.  See the following article from the MOSS SDK for details on how this works.
• When troubleshooting problems with your ADF, be sure to check both the Windows Application EventLog and SharePoint’s diagnostic log files (C:\Program Files\Common Files\microsoft shared\Web Server Extensions\12\LOGS).  These will provide you with much more detail on the nature of the problem.
• As I’ve suggested, you want to review the actual XML results from your Web Services.  These can be difficult to capture at times.  To help, you might find some value in using a network sniffer such as Network Monitor.  This allows you to capture the actual network packets allowing you to study the traffic between the SharePoint server and the Web Service server.  You can download Microsoft’s Network Monitor for free from here.

Download Sample Code

In the sample code available here, you will find the following code:

• Amazon ADF (Note: you will need your own Amazon Web Service (AWS) Account)
• Modified Amazon XSLT for the Business Data List Web Part
• The complete Employee Web Service (VS.NET 2008 project)
• The Employee ADF
• Employee Stored Procedures for AdventureWorks database

ADF Overview

Since we understand the BDC to be a middle man between the user interface and some back-end system, we need some way of mapping how the back end system is structured.  This mapping can then be consumed from the various interfaces such as Web Parts and searching that we covered in the last article.  Since BDC is designed to pull data from virtually any relational database or SOAP-based Web Service, you’d correctly guess that the ADF must be flexible and generic in nature.  As mentioned already, the ADF is an XML file with a specific schema and its concept can be likened to a logical database diagram.  By logical, I mean that it does not have any vendor-specific features such as data types.  Keep in mind that it must also be flexible enough to connect to Web Services as well, so it can’t be too database-centric as well.

XML makes good sense for the definition as it is a declarative way to define the back-end in a generic way.  By being declarative, and not done programmatically through code, it helps ensure that we have a loosely-coupled design.  This is very important as front-ends and back-ends are likely to change, and we want to make sure the maintenance effort spent on this middle layer is minimized.  As we’ll see later, having it XML based also makes it easier to automatically generate this definition.

In Figure 1 below, you can see a conceptual overview of how the ADF is used.  The browser makes a regular request where BDC data is needed (e.g. a BDC Web Part, Search, etc.).  The MOSS web front end receives the request, and as part of its processing, hands the BDC request off to the Object Model. The Object Model (also known as the API or Application Programming Interface) gets the definition for this ADF from its catalog.  With the details on the ADF known, the Object Model requests the data from the database or Web Service.  The response is returned and then formatted as appropriate for delivery back to the browser.

Figure 1: Data flow overview with BDC

To best understand the ADF, it helps to have an understanding of some of the terminology used.  This will relate directly to structures found in the XML definition.  Those with a database background (and also those developers with knowledge of ADO.NET) should pick these concepts up right away.

Line of Business.  Line of business (or LOB) is just a term that refers to a back-end system.  As you’ll see shortly, a single ADF is comprised of one or more LOB instances.

Entity.  An entity works like a table.  The entity term is also used when designing a logical database diagram, and the concept is the same.

Method.  A method is how we request data (or query) from an entity.  For a database, it is usually a SQL select statement or stored procedure.  For a Web Service, it would be a Web Method.  Each entity supports multiple methods.

Parameter.  Parameters are the inputs and outputs to a method.  Each parameter has a name and a .NET data type (e.g. System.Int32 or System.String).

Filter Descriptor.  Filter descriptors provide an easy way for users to filter records (rows) found in an entity.  It is mostly commonly used with the Business Data List Web Part.  For example, you might want to allow users to quickly find a company by its name.

Association.  Associations are how relationships are represented.  For example, an Employee entity might have an association with a VacationRequest entity.  For a database, these are normally used on child tables that have a foreign key.

Action. Actions are used with entities and allow you to link to external applications or custom SharePoint pages for additional processing.  These are commonly used in write-back situations, and we will look closer at these in the final article in this series.

The ADF Schema

Now that you have a pretty good idea of what the ADF does and some of the vocabulary, let’s go ahead and examine the XML Schema for an ADF.  (Note: this is not intended to be a detailed discussion; for a deeper look, please consult the MOSS SDK (Software Development Kit) here: http://msdn.microsoft.com/en-us/library/ms563661.aspx.)  One of the screen shots from this SDK is very helpful in understanding the high-level hierarchy of the ADF.  You can see it here:

Figure 2: ADF Schema Overview

LobSystem is the root element in the ADF file.  It contains attributes to the required namespaces, specifies whether it is a Web Service or Database, and sets the Name of the ADF.

LobSystemInstance is where you define your connection in the ADF.  Even though the XML schema looks like it supports multiple connections, you can only have a single connection defined in your ADF.  Through Property child elements, you define the connection string to this data source.  Those with ADO.NET experience will quickly recognize this example below as a Windows-integrated login to a SQL Server.

Figure 3: LobSystemInstance

The details of authentication is a complex subject, and we’ll just cover the basics here.  When connecting, you have four options when it comes to authentication.

1) PassThrough.  PassThrough can be used to authenticate the user in one of two ways, using Windows credentials or database credentials. With Windows (shown in Figure 3 above), the client’s Windows credentials are forwarded to the back end system.  This can be used for both databases and Web Services.  The problem with this, however, is that unless you’re using Kerberos (or your Web Service/database server is on the same physical machine as your SharePoint web front end), this will fail.  This is due to Windows NT Lan Manager’s (NTLM) inablility to delegate.  With database credentials, you use a database-specific username and password to log in.  This can only be used against a database server, not a Web Service.

2) RevertToSelf. This means the credentials of the Application pool is used to authenticate the user.  This will solve the the delegation problem from above, but all users will effectively use the same login on the back end.

3) WindowsCredentials/RDBCredentials.  These are used in conjunction with Single Sign On (SSO).  In this model, the user’s credentials (either a Windows login or a database-specific login) for the external system are pulled from the SSO database.

4) Credentials.  This is only used if you are connecting to a Web Service and cannot use Windows authentication.

For more details on authentication, I recommend you start your reading here:  http://msdn.microsoft.com/en-us/library/ms566523.aspx.

Within LobSystemInstance, each entity for this connection is defined.  The entity itself is not very interesting as it just defines a friendly name by which users will reference it.  Within each entity, you define an identifier and methods.  An identifier is the primary key, or the column(s) from the entity that can be used to uniquely identify each row.  These are very important as they are used in a number of ways.  Often, these are ID columns, such as CustomerID as shown here.

Figure 4: Identifier

Also part of each entity are the methods.  These are really the heart of the ADF as it defines the actual data that will be pulled.  It is where you specify a select statement (e.g. select * from table), a stored procedure, or a Web Method if it’s a Web Service.  Since there may be multiple ways to query data from an entity, you can have multiple methods.  Here is a simple example with returning a few columns from a table named Customers.

Figure 5: Method

Within a method, you may want to define one or more ways in which this method can be filtered by users.  This is used within the BDC Web Parts.  For example, for the list of customers above, imagine your company has hundreds of them.  How will your users easily find the one they are looking for?  Start by asking yourself what are the common ways they might want to lookup a customer—perhaps by Name, by Region, or by a Contact Name.  If this makes sense then you understand FilterDescriptors.  Here is a simple look at two FilterDescriptor entries that have been made.  The Name attribute you see here defines the friendly name that users will work with, so make sure this name makes sense to them.

Figure 6: FilterDescriptor

Now that you understand filtering, let’s go back to methods for a bit.  In the ADF, there are different types of methods, and the three primary types we work with are Finder, SpecificFinder and IdEnumerator.  Finder methods are used when you want to return one or more rows from an entity.  For example, find all customers who are in the North America region.  SpecificFinder is used to find a single row and is used in conjunction with the Identifier mentioned above. For example, find me the one Customer that has CustomerID = 42.  IdEnumerator is used to return the identifier value for each row.  In code terms, this would look something like Select CustomerID from Customers.  The BDC uses this when crawling for Enterprise Search integration.  You will find each method type defined within a MethodInstance.

Let’s move on to Parameters.  Parameters allow you to describe the exact input and output for methods.  Input parameters (Direction = “In”) are used with Filter Descriptors.  There are different types of output parameters, but the most common are Return parameters (Direction = “Return”).  These are used to describe each column that is returned from a method.  For each parameter, you define the friendly name and its .NET-matching data type.  In Figure 7 below, we see the two parameters that are defined for our two FilterDescriptors above.  These are input parameters as shown here:

Figure 7: Parameter

Last but not least, let’s cover Associations.  We use Associations to define the actual relationships that exist in the external system.  BDC can directly work with one-to-one and one-to-many types of relationships.  When you define an Assocation, you define the SourceEntity (i.e., the parent or primary) and the DestinationEntity (i.e., the child or foreign).  You also must point to a method in the child entity that returns all the child rows based on a single parent row.  This method will usually “join” these two tables together.  In SQL terms, it would be something like

Select * from Orders
join Customers on Orders.CustomerID = Customers.CustomerID
where Orders.CustomerID = 42

Here is an example on how an Association is defined in the ADF:

Figure 8: Association

Creating ADFs

In the small snippets above, you can probably see that the XML syntax won’t likely roll off your fingertips.  It takes time and practice to get good at writing them.  These can be created in any text or XML editor, including Visual Studio.  If you will be using Visual Studio, I recommend you import the BDCMetaData.xsd schema file so at least you get IntelliSense.  You can follow the guidance on how to set this up from SharePoint MVP Andrew Connell here.

Fortunately, there are tools out there that can reverse engineer the database (or Web Service) and generate most if not all of the ADF file for you. Even if you only have to write one ADF, these tools may be worth it.  The three main ones available to you are

• BDC Definition Editor
• MOSS BDC Design Studio
• BDCMetaMan

BDC Definition Editor is free and comes from Microsoft.  You can install it after you install the MOSS SDK.  Sadly, with this tool, you get what you pay for.  It doesn’t generate Finder methods, crashes quite a lot, and requires you to install SQL Server 2005 Express (even if you have a full SQL Server instance installed).  It’s certainly better than nothing but not recommended for regular use.

MOSS BDC Design Studio is a commercial product from Simego.  The latest version 2.0 has a nice wizard interface when setting up each entity.  It can be used to generate ADFs on common databases like SQL Server and Oracle as well as Web Services. It is reasonably priced at $200 for a single license.

BDCMetaMan from Lightning Tools is the top-of-the-line and de-facto standard for generating ADFs.  It is well written and fully featured.  It is also the most expensive at $1200 per license but well worth it if you use it enough.

Loading the ADF into MOSS

Once your ADF file has been generated, it needs to be loaded into your SharePoint environment.  This is done through your Shared Services Provider (SSP).  To get there from Central Administration, click on your SSP name in the Quick Launch menu on the left, then click Import application definition in the Business Data Catalog section.  Loading it is as simple as browsing to the file and clicking the Import button.  When you upload the ADF, it will validate the XML file. It will then create entries in your SSP Database for each LobSystemInstance and Entity.

Once the ADF is loaded, you might want to take a look at its structure.  Just click on View applications from the main SSP screen, and then navigate down through your named ADF and into each entity.  With the exception of custom actions, everything else is read only, so you can’t really make any changes.

Setting Permissions

When you’re done poking around, you’ll probably want to set permissions on your new definition.  Permissions apply to these levels in the ADF hierarchy: LobSystem (the whole ADF), Entity, Method and MethodInstance.  This means you can secure at a high level such as each ADF, or very granularly such as each Method within an Entity.  Unfortunately, the administration GUI only lets you set permissions down to the Entity level, so this is the lowest level that most people work with. You can also set permissions at a higher level (even the whole BDC) and copy these down to lower levels.  When setting permissions, you have four permissions to choose from.  Each is introduced here:

• Edit.  This permission allows you to delete or add new ADFs into BDC.  You would grant this to those who would be uploading new ADFs into your SSP.
• Execute.  This allows you to execute Method Instances.  If you apply this permission at the entity level, it applies to all methods and method instances for this entity.
• Select in Clients.  This defines whether you can see the entity when configuring BDC Web Parts or adding a BDC column to a list.
• Set Permissions.  This allows you to change permissions for other users at the various levels as described in this section.

That said, the common approach is to give users both the Execute and Select in Clients permission for all entities that they will be working with.  As standard with SharePoint, you can assign permissions to Users or Groups.

With that, we bring this Part Two article to a close.  By now, you should have a solid understanding of how BDC works and how to create, load and secure your ADF entities.  In the next session, we’ll focus more on creating ADFs for Web Service systems.  From there, we’ll then move over to the UI and how to leverage the value that these ADF entries give you.

We have three goals for this series:  1) Give you a strong understanding of what BDC is and how it works; 2) Ensure you recognize the value it brings organizations; 3) Provide you with the technical know-how to set it up and use its many features.

BDC Overview

As you already know, Microsoft Office SharePoint Server (MOSS) provides tremendous value as an intranet solution. One of the ways it excels is that it provides a single access point for the data assets in your organization. In many cases, these assets are stored in a Word doc, a PDF, an Excel workbook, etc.  In other words, it’s just a file. We call these sources unstructured data due to their flexibility in storing almost endless forms of information. It’s this flexibility that has made MS Office such a popular software suite. Users can work in a rich client application and then simply save the file, presumably into a document library.

This is where the power of MOSS as a document management system comes in. When coupled with an enterprise search engine, it’s even greater. Despite the unstructured nature of these files, users can go into their portal and by either a few navigation clicks or a quick search, they can find the answers they need. Having this centralized location for answers yields not only productive employees, but happy ones. It may seem that users have the best of both worlds: the flexibility of storing data the way they want, and the ability to publish it knowing that other users can find it.

But hang on, it’s not quite this easy. Sometimes, structure is necessary. Structure imposes rules and order. Do we really want to store all our customer orders in a single Excel sheet? Structure helps ensure we have valid, manageable, and most importantly, trusted data. For these structured data needs, IT professionals have relied on the tried and true relational database, and many vendors such as Oracle, IBM and Microsoft have made a bundle selling their flavor of SQL.

With these relational database management systems, information is locked in server-side databases. You usually work with the data through a separate application, perhaps a rich client or a web-based interface. These structured databases are the lifeblood for critical systems such as accounting, CRM (Customer Relationship Management), ERP (Enterprise Resource Planning), and a myriad of other line-of-business needs.

In fact, these solutions have become so popular and easy to build, they have sprung up all over the organization into what I like to call database sprawl. Typically, each of these systems is another data island, disconnected from the others. In many cases, information is duplicated across these systems, causing chaos where order was expected. From the user’s standpoint, they can be perceived as very inflexible and frustrating: Where is the data I need?  How do I log into that system?  Once I’m logged in, how do I access it?  By the time each question is answered, precious time is lost.  Or worse, we give up and make decisions based on no data at all.

This yields an interesting dichotomy in the way we store data. On one end, we have the unstructured nature of MS Office; on the other, we have the structure provided by the relational database. Interestingly, both can result in chaos. What you’ll be pleased to know is that MOSS can also bring some order to the database sprawl as well, and it does so in a very similar way.

With Business Data Catalog, part of MOSS Enterprise, we again get directed us back to the SharePoint interface to get access to our data. With BDC, this structured data can be exposed through Web Parts; it can be indexed and searched; it can be merged with data found in lists. With BDC connections to your back-end systems, we can find and consume both structured and unstructured data assets that exist in our organization. And we do this from one location, one interface.

Architectural Overview

A holistic view of what BDC can do and how it works is best depicted in Figure 1.

Figure 1: BDC Architecture

What you see along the top are the primary ways BDC can expose data from your back-end systems. Here is an overview of each of these from left to right:

1) BDC data can be surfaced using the many out-of-the-box Web Parts. This is the most common way and very easy for the users to search, filter and sort for information. You simply add one of the BDC Web Parts to any page, point it to one of the sources in the catalog and users are ready to go. All data is pulled in real-time using this approach.

2) SharePoint Lists allows you to merge data that lives in SharePoint along with data from a back-end system. For example, perhaps you have customer names stored within a list. From your CRM system, you would also like to see contact information for each customer. By adding a BDC field to your list, you can copy the corresponding data from your back-end system and store it within the list. Typically this matchup is done one-to-one. To ensure the copied data doesn’t get stale, users can refresh it on demand.

3) Search allows you to set up a content source and point it to an application in the BDC. SharePoint then goes through through BDC to crawl your structured data source and store the keywords in its index. Your search queries can now go across both structured and unstructured sources giving you a unified set of results.

4) User Profile Importer allows you to supplement attributes found in the User Profile store from external databases. This is important for many organizations as their Active Directory or LDAP directories are rarely populated with all user details. For example, how many IT administrators will actually fill in the birthday or skills attribute? Data like this is usually found in HR systems anyway.  Using BDC, these values can be pulled and matched up with the user, providing a complete set of accurate values.

5) With custom applications, you can avoid having to write another data layer for your new application. Instead of directly connecting to the system using ADO.NET or a Web Service, you can consume the data using the BDC Object Model. This gives you a more flexible, declarative style of talking to your back end.

With each of these approaches, you may have noticed that the concept is predominantly read-only in nature. This is similar to decision support systems.  That is, we usually use BDC to consume the data from these back end systems, but rarely write back to it. In the final article in this series, you’ll see how you can write to your back end system.

A Quick Look at BDC in Action

Let’s see how this works.  In this first example, we demonstrate the Business Data List Web Part (#1 from above).  This makes the finding and displaying of rows from a table very easy.  As you see here, we are finding all rows from the Customers table (perhaps from a CRM database) that contain the word “market” in the name.

Figure 2: Business Data List Web Part

In the second example, we see how users can search for both documents and BDC sources (#3 from above).  A search for Great Lakes has been issued, and we see both document and BDC Search Results listed together.  We’ll go into far more detail on these features and others as we get into other articles.

Figure 3: BDC Search Results

How Does BDC Work

Now that you have a better idea of what it can do, you’re probably wondering how does it work? As you can see, the BDC is a middle man between your interface and the back end. To provide this universal interface, it uses a database-like XML syntax that is called an Application Definition File or ADF. This ADF defines the connection, the entities (for a database, these are usually tables, views or stored procedures), how to filter, what associations (relationships) exist and what user-friendly names should be used. These names can even be localized in different languages. This ADF is intended to be a generic interface describing the dictionary of information available. Once these ADF files are stored within the catalog, the first four options above are available without having to write any code. Of course, security is an important consideration when we talk about exposing sensitive data, so we’ll see how the BDC addresses this concern in the next article.

For the actual back end, as you can see in Figure 1 above, this can be any ADO.NET accessible database with an OLEDB or ODBC driver.  This covers most every database from MS Access and SQL Server to Oracle and IBM DB2. The back-end can also be a specially-written SOAP-based Web service.  The BDC is configured within a Shared Services Provider (SSP) which allows you to take advantage of BDC technology within any Web Applications within your SharePoint Farm.

Series Outline

With that, that nearly brings us to a close for this first article. But before we let you go, here is an outline on how we are structuring this series.  To get the full value, you should go through these in order, but we’ll try to keep them discrete so that you can jump around and focus on the ones that are most important to you. Note: these may slightly change as we go through each of these.

Part 1
In this first part, as you have now read, the focus is be on the concepts, the value it can bring organizations, the architecture, and an overview of each article in the series.

Part 2
This article will introduce the Application Definition File. By the end, you’ll understand the role it plays, how it works, and have a high-level grasp of its schema.  We’ll also introduce tools that allow you to generate these XML files.  Securing ADFs will also be covered in this article.

Part 3
This article will cover how the Web Services can be used instead of a direct connection to a database.  It will look at the kind of Web methods you’ll need to write if you want to expose your systems to BDC this way.

Part 4
This will be a two-part article.  In first half, we’ll look at the out of the box Web Parts that BDC provides.  These is the most common way that this structured data is presented to users, so we’ll give it a solid overview.  The second half will cover how you can cache BDC data into SharePoint lists.  This will allow you to marry SharePoint data and BDC together into one consolidated view.

Part 5
This article will cover everything you need to know to to configure Enterprise Search to crawl BDC data.  We’ll cover content sources, scopes, and how to configure MOSS for both full and incremental crawls of your structured data.

Part 6
As mentioned above, BDC can supplement properties contained within User Profiles. This allows you to pull critical data elements from external ERP or HR systems and map them to out-of-the-box or custom User Profile properties.  This article will cover everything you need to know to get this working for you.

Part 7
This article will apply mostly to the developers out there and will cover the BDC Object Model.  You’ll see how custom Web Parts, application pages or other custom-built SharePoint applications can consume BDC database without your having to write another data layer.

Part 8
In the final article, we will cover Custom Actions how these can be used to safely write changes to the back-end database or Web Service.  While not used often, this may prove to be a practical way for users to maintain some of the data in these external systems.

Generally, when you start talking about scorecards, people picture the balanced scorecard. The balanced scorecard has 4 perspectives (or objectives), according to the Kaplan methodology which seems to be generally recognised as the most common balanced scorecard. These are: Financial, Operational, People and Customer. The KPIs (Key Performance Indicators) which make up each perspective are specific to your organisation and might be based on your organisation’s 5-year strategy plan or a core set of performance measures or a combination of both. For example:

However this is not the only type of scorecard. How about a sustainability scorecard which shows these perspectives:

Or these:

Now let’s look at an entirely strategy-based approach to a balanced scorecard:

How about a scorecard for use by call centre staff:

As you can see there is no limit to the ways you can use a scorecard except your imagination. But that’s all very well and good, how do you the humble technician explain it to the decision makers? There really are two audiences – those making the IT strategic decisions and business managers, which in my experience means Marketing Managers and Finance Managers. Somehow you need both parties to come to the party at the same time. Which can be a challenge… but that’s a problem you’ll have to nut out for your own organisation. How to manage them? That I can help with!

Ok so first, let’s address the business managers.

Now let’s address the second part of the equation, those holding the IT purse-strings. There are a few wins from their perspective (dodgy scorecard pun intended) too:

• Leverage existing investments in SharePoint and SQL
• Dashboards tend to create warm-fuzzy feelings in high-level managers because they look pretty
• Reduce printing of flat reports
• Reduce storage of large spreadsheets and Access databases
• Reduce emailing spreadsheets around the organisation
• One version of the truth!

But you know what else, if you can get the business managers on board first, the 2^nd part will be a lot easier!

The fact is, hysterical pronouncements about the death of balanced scorecards are somewhat exaggerated. In fact, I’d say a complete crock! If I can see at a glance, the performance of my organisation or my area or my whatever! I am a happy chappy. That’s just common sense. As to the rest, well, I pity the fool. As some old dude with a lot of bling once said.

	Part 1 – Getting to Know You – Intro to SharePoint BI
	Part 2 – Sell! Sell! Sell! Why Build a Dashboard Anyway?
	Part 3 – KPIs – Where do I Come From?
	Part 4 – Dissecting Dashboards
	Part 5 – We Really Need to Talk – Scorecards & Reports
	Part 6 – Beware the Bogeyman – Securing Dashboards

Siyonara amigos! Till next time.

Kristen Hodges, MCTS, MCITP
http://www.bi-tch.blogspot.com

http://b-iq.org

References

http://www.isixsigma.com/library/content/c081013a.asp
http://www.bscol.com/pdf/Kaplan-Nagel-Mar04DM.pdf

Business Intelligence (or BI as the cool kids are calling it) has been buzz word du jour for quite some time now. Gartner research lists it as the Number 1 technology priority for 2008 for CIO’s the 3rd year running. Clearly someone important cares! As SharePoint professionals, how can we deliver on these priorities? And what does SharePoint have to do with BI anyway?

What Is Business Intelligence?

As they say, let’s start at the very beginning. Business Intelligence (as if I need to explain it to you!) is a generic term used to describe the technology and processes whose primary purpose is to monitor, analyse and plan thereby driving Business Process Improvement, which is, shock horror(!!), according to the same Gartner research, the Number 1 business priority… who’da thunk it!

Microsoft’s Monitor, Analyze, Plan paradigm is mentioned alot when we talk about BI. It is these three actions which should be driving technology needs and the output of them that should be driving business decisions. That makes it kinda important. It’s all very well to give the Finance Director a report showing how many sales were made this month but unless you start comparing them to last month or last year AND forecasting next year meaningfully then what’s the point? It’s all about informed decision-making and planning.

What’s SharePoint Got to do With it?

When it comes to SharePoint there is, quite rightfully, a lot of focus on portals, collaboration and Content Management. And yet SharePoint is actually plays a big part in the BI space.

With SharePoint we can extend Excel 2007 by using Excel Services for more scalable, manageable spreadsheets. With Report Centre, we can create dashboards from disparate sources, we can talk to Reporting Services, Excel and our LOB systems. With the Business Data Catalog (BDC) we can connect to LOB systems and access data from, for example, SAP and make it available in SharePoint. And that’s before we start considering complementary products such as PerformancePoint.

Performance Who?

So you might notice that I snuck in a little mention of PerformancePoint there. Maybe you’ve heard of it, maybe you haven’t. I’m going to assume the latter because it’s a newish technology. You may have heard of it before under its previous moniker, Business Scorecard Manager or BSM. Still no? Think KPIs, scorecards and dashboards. Now add planning & forecasting. Oh, and don’t forget ProClarity for slice & dice analysis, data mining and yet more KPIs. It’s a bit of a behemoth sure but the idea is that PerformancePoint is the enterprise-level tool for developing everything required for the BI lifecycle – that is, to monitor, analyse and plan. And guess what PerformancePoint uses as it’s delivery tool? You guessed it, SharePoint! There are specific webparts designed just for delivering PerformancePoint. In fact, for users it’s the only way to interact with the monitoring (ie dashboards, scorecards and KPIs) aspects of PerformancePoint. So now we know what PerformancePoint is, why am I banging on about it so much?

What Am I Talking About?

This is the first of a series of 6 articles which is going to focus on SharePoint BI. In particular, building dashboards for SharePoint. And while there are many ways of building dashboards, as I will cover in detail in the part 3, PerformancePoint is a key technology in this armoury with the greatest scope for enterprise-level development. No one method is ‘better’ than the other, they are all Microsoft technologies after all, but each does serve a slightly different purpose. Consider the difference between using Excel, Access and SQL Server to create a database. Sure they’re all do-able but you aren’t gonna create an enterprise application using Excel as a back-end now are you? The same holds true with PerformancePoint and SharePoint. That old saying “the right tool for the right job” is just as relevant in technology as in anything else.

Dashboard, KPIs, Scorecards. Whatever!

BI does have its own set of jargon, doesn’t it! So what does it all mean? Think of it this way… dashboards contain scorecards, reports and any other objects you can think of (applications, links, files etc). Scorecards contain KPIs.

KPIs are “Key Performance Indicators” or measures of how well (or poorly) targets were met.
When talking about KPIs, you can always assume that we have at least an Actual Value and a Target Value. We may also have forecast values.

Scorecards are made up of many KPIs. These KPIs may be grouped. And different weightings applied based on the impact or importance of a KPI. KPI groups and KPIs may be rolled up to show overall performance. Scorecards often include indicators or little graphical flags which show how close to target our KPIs are.

Dashboards are a single page contained in a Document Library. A dashboard may contain multiple scorecards, as well as related reports and other files.

What Are We Going to Cover?

As I mentioned previously, in part 3 we’re going to discuss the various tools available to us in building KPIs, the benefits and risks of each. In part 4, we will learn how to build a dashboard and in part 5, how to get scorecards to communicate with reports and other scorecards for that matter. Finally, dashboard in hand, we will learn how to make sure nobody’s sticky-beaking where they shouldn’t be in part 6. But to get things moving, our next article will answer the question at the heart of it all… why build a dashboard anyway?

I look forward to the next 5 articles, your comments, thoughts and questions.

Cheers,

Kristen Hodges, MCTS, MCITP
http://www.bi-tch.blogspot.com
http://b-iq.org

References

http://www.gartner.com/it/page.jsp?id=587309
http://office.microsoft.com/en-us/sharepointtechnology/FX101758691033.aspx
http://www.microsoft.com/sharepoint/capabilities/bi/overview.mspx