Commerce Server Staging (CSS) and Catalogs

A few things you didn’t know about CSS and how catalogs move from one environment to another. The process of using the CSS is pretty simple just use the MMC console for CSS and setup your project but there are some undocumented and some scenarios that will cause you some issues that you need to be aware.

So will I sing praise unto thy name for ever. that my Commerce Server site may daily perform

Someone had asked as to why his relationship would not move form one environment to another and I didn’t dive too deep into it and I couldn’t help him. As it turns out that my current client run into something very similar. So did some work and found out the reason.

Here is the scenario, have two catalogs in catalog one create a relationship add items form catalog two. Now setup a CSS job that will move your items from one server to another but choose catalog one. The job should be expression based. The relationship will not move by default. If you look there are three values that can be set for an expressional export None, Immediate Children and All Levels. If you select either Immediate Children or All Levels only then your relationship will be moved to the target environment.

The reason is simple you setup a relationship between two different catalogs but you select only one to move so you have to be explicit on what the export should do.

Another issue I have noticed is that when running CSS to move products is overwriting hierarchies and relationships. When you use the Catalog Manager this feature is available.

But this option is missing when using CSS. It’s true that this option is not available through the UI but you can modify this using the project XML. When you create projects using CSS the project properties are created in the registry and xml file if using the data project. The project files are stored in “C:\Program Files\Microsoft Commerce Server 2007\Staging\Projects” if Commerce Server is installed using defaults. In the xml file there is an ImportOption that is set to Yes=True by default but you change this behavior.

Now an over all look on the Import option for over writing category hierarchy and relationships. The product is using two catalog features but have only one option what if you wanted to overwrite your relationships but not your category hierarchy or the other way around, well your out of luck. Here is another scenario where things will get a bit tricky what if you only wanted to export your price property? In this case you would setup your CSS and set the property import option to overwrite relationship and category hierarchy to true. The exported XML file will only have one property the price. When you do the import the primary parent category will be removed if you had setup primary category with your product. So why does this happen? The reason is simple the primary parent category is setup as a property with the product when the export happens it exports all the parent categories and when an import happens the primary parent is removed because we have parent categories and setup to over write relationships.

Hacking Commerce Server Staging Part III

As promised here is the code for the previous post on hacking the CSS engine.

Download Code…

Hacking Commerce Server Staging Part II

In the previous post I showed you how to create a custom UI for the Commerce Server Staging (CSS). In this post I will show you how you can create a custom handler that will fire and perform some action based on the custom UI.

Custom Handler

As noted in previous post CSS uses managed code for data import\exports. CSS engine works on using a factory object to create handlers depending on the project type name. The project type name is specified in the “BusinessDataTypes.xml” file located in “C:\Program Files\Microsoft Commerce Server 2007\Staging\Bin\” folder if Commerce Server was installed with defaults. In my sample I have a custom name called “Max's Test”. When the CSS engine identifies an export or import it reads the “BusinessDataTypes.xml” file then look at the project xml file and finds the type identified. So our project xml file should have “Max's Test” as the BusinessDataTypeName attribute. When the match is made the CSS engine will load the “HandlerAssemblyQualifiedName” attribute value. In the sample “CSSTestUI.CustomHandler” is the handler. The CustomHandler class inherits from IBusinessDataStagingHandler having two methods for implementation Import\Export.

Let’s take a look at a simple implementation:

In the sample above instead of doing real work I am just writing and reading the xml project data file. Please note I am not writing xml so if you open it in a xml browser you will generate errors.

Theda theda that’s all folks

Once you create your own custom handler just make sure to sing the assembly and GAC it. Then you need to stop\start the CSS Service before the project works. To debug your code you would attach to the MMC console for the UI and the CSS Service for the handler.

Good luck creating cool custom CSS projects.

Once I have access to ftp I will upload the project files in these blog posts. The project was created VS2008 and tested on windows 2008\Commerce Server 2007. There should be no issues with Commerce Server 2009 if you do find issues please send me an email so I can post it for others.

Hacking Commerce Server Staging

Way back when I was a program manger on the Commerce Server Product Group one of my features was Commerce Server Staging (CSS). While going over CSS with the dev team and PMs I wanted to make CSS extensible so customers could create their own CSS projects. But as is with everything else this decision was met with “We don’t have time and resources” and it was dropped.

The dev manager thought it was good idea and stated that he would keep the APIs public that way in the next version of Commerce Server they could revisit the subject. Here we are with the new version and no extensibility. Now as you know the CS product has gone through much turn over and that dev manager is with a different team with MS.

In this post I am going to show you how you could hack CSS and build your own custom projects.

CSS UI Basics

CSS UI is only visible using the MMC console. CSS is a windows service but there are APIs that talk to the service. The API support COM as well as .NET (COM wrappers) that allow you to pretty much do everything that you can using MMC.  Having COM means you can write VBScript code for automation (note the Java Script is not supported due to Java script not supporting parameter passing values by ref).There is also a command line tool and once again it almost does everything MMC can.

But this section is about the UI so lets talk about that. The Business Data UIs of CSS is written in .NET. So there is a whole framework that handles this for you. Which means the MMC loads .NET controls when you open a project or create one using the wizard. How does CSS know which tabs to load on the project properties? CSS when installed creates a folder in the Root of Commerce Sever “%COMMERCE_SERVER_ROOT%” called staging. In the staging folder there a sub folder called bin. In the bin folder there is an xml file BusinessDataTypes.xml that holds the types of business data that CSS support.

If you look into the content of the BusinessDataTypes.xml file (see table below) notice that you have Handlers and UIProgIDs. The UIProgIDs are used to load controls onto project tabs. The Handlers are used to export and import data when the project is executed.

So can you create your own UI? Sure why not. I will show you a quick sample which I will outline in this post.

Create CSS Custom UI

1. The first thing you want to do is create a project in Visual Studio and select Class Library.

2. Step two add references by implementing the IBusinessDataUserInterface which exists in the “C:\Program Files\Microsoft Commerce Server 2007\Assemblies\CSStagingUserInterfaces.dll” assembly if commerce server was installed in it’s default location. Please note that this is also GACed. The next reference is Microsoft.CommerceServer.Staging.BusinessDataStagingFramework.dll also located in the same directory and GACed.

3. Now that you have the two references you will need to add a Control to your project and have it inherit IBusinessDataUserInterface and implement all the interface. Your class should look like the image below:
    
   Notice the class attribute make sure that you have the same with your own unique Guid.

4. Before we can actually write code to our UI we need to add registration code:

   
   

   #region Component Registration [ComRegisterFunction] private static void ComRegister(Type t) {     string name = @"CLSID\" + t.GUID.ToString("B");     using (RegistryKey key = Registry.ClassesRoot.OpenSubKey(name, true))     {         key.CreateSubKey("Control").Close();         using (RegistryKey key2 = key.CreateSubKey("MiscStatus"))         {             key2.SetValue("", "131457");         }         using (RegistryKey key3 = key.CreateSubKey("TypeLib"))         {             key3.SetValue("", Marshal.GetTypeLibGuidForAssembly(t.Assembly).ToString("B"));         }         using (RegistryKey key4 = key.CreateSubKey("Version"))         {             Version version = t.Assembly.GetName().Version;             string str2 = string.Format("{0}.{1}", version.Major, version.Minor);             if (str2 == "0.0")             {                 str2 = "1.0";             }             key4.SetValue("", str2);         }     } } [ComUnregisterFunction] private static void ComUnregister(Type t) {     string subkey = @"CLSID\" + t.GUID.ToString("B");     Registry.ClassesRoot.DeleteSubKeyTree(subkey); } #endregion Component Registration

5. Ok now we are ready to populate our interface implementations. For the ModuleName replace the text to with your own custom name. The wizardMode variable is used when creating projects using the wizard. When creating a project using the wizard mode you will not see your custom control as the UI for the process is hard coded to only display the four Commerce Sever Business Types. But you can view your control using non wizard mode. The modeReadOnly is used to distinguish between Operator and Administrator. Remember an Operator can not modify project properties.
   Private variables:

   
   

   internal const string ModuleName = "Max's Test"; private BusinessDataCollection businessDataCollection; private bool modeReadOnly; private bool wizardMode; private BusinessData businessData;

   
   

   Flush Method:

   
   

   public void Flush() {     for (int i = this.businessDataCollection.Count - 1; i >= 0; i--)     {         BusinessData businessData = this.businessDataCollection[i];         if (!(businessData.BusinessDataTypeName != ModuleName))         {             this.businessDataCollection.Remove(businessData);         }     } }

   InitializeSate Method:

   public void InitializeSate(string siteName, Microsoft.CommerceServer.Staging.BusinessDataCollection collection, bool modeReadOnly) {     if (siteName == null)     {         throw new ArgumentNullException("siteName");     }     if (collection == null)     {         throw new ArgumentNullException("collection");     }     this.modeReadOnly = modeReadOnly;     this.businessDataCollection = collection;     this.businessData = GetFirstBusinessData(collection, ModuleName);     if (this.businessData == null)     {         this.businessData = new BusinessData();         this.businessData.BusinessDataTypeName = ModuleName;         this.businessData.BusinessDataId = Guid.NewGuid();         BusinessDataOption opt = new BusinessDataOption("checkBox1", checkBox1.Checked.ToString());         this.businessData.ExportOptions.Add(opt);         BusinessDataOption opt2 = new BusinessDataOption("textBox1", textBox1.Text);         this.businessData.ExportOptions.Add(opt2);         this.businessDataCollection.Add(this.businessData);     }     else     {         string opt = Get(this.businessData.ExportOptions, "checkBox1");         string opt2 = Get(this.businessData.ExportOptions, "textBox1");         checkBox1.Checked = Convert.ToBoolean(opt);         textBox1.Text = opt2;     } }

   
   SaveState:
   

   public void SaveState() {     if (!this.modeReadOnly)     {         // put code for read only     }     // We will assume administrator     BusinessDataOption opt = GetOption(this.businessData.ExportOptions, "checkBox1");     opt.Value = checkBox1.Checked.ToString();     BusinessDataOption opt2 = GetOption(this.businessData.ExportOptions, "textBox1");     opt2.Value = textBox1.Text; }

   
   WizardMode:
   

   public bool WizardMode {     set     {         this.wizardMode = value;     } }

   
   Helper Methods:
   

   internal static BusinessData GetFirstBusinessData(BusinessDataCollection collection, string type) {     foreach (BusinessData data in collection)     {         if (data.BusinessDataTypeName == type)         {             return data;         }     }     return null; } internal static BusinessDataOption GetOption(BusinessDataOptionCollection options, string optionName) {     foreach (BusinessDataOption option in options)     {         if (option.Name == optionName)         {             return option;         }     }     return null; } internal static string Get(BusinessDataOptionCollection options, string optionName) {     if (options == null)     {         throw new ArgumentNullException("options");     }     if (optionName == null)     {         throw new ArgumentNullException("optionName");     }     foreach (BusinessDataOption option in options)     {         if (option.Name == null)         {             throw new ArgumentNullException("option");         }         if (option.Name.Trim().ToUpper(CultureInfo.InvariantCulture) == optionName.ToUpper(CultureInfo.InvariantCulture))         {             if (option.Value == null)             {                 throw new ArgumentNullException(optionName);             }             return option.Value.Trim().ToUpper(CultureInfo.InvariantCulture);         }     }     return null; }

    

6. Ok now that we have our code lets add a new control as we referenced a checkbox and textbox.
   

7. Compile your project and make sure to sign the assembly as will add it to the GAC.

8. Modify the BusinessDataTypes.xml file in “C:\Program Files\Microsoft Commerce Server 2007\Staging\Bin” to reflect the GACed assembly.
   

9. GAC the assembly and stop and restart the CSS service.

10. Now open CSS MMC and create a project not using the wizard.
    
    Notice that we have our control in the project properties. I have not figured out how to publish the custom control name to the tab, but all works. Add some values in your custom control tab and other project properties such as site name and destination. Select OK and save your project. Now examine the xml file that is created for your project.
     

As you modify the project properties the settings will be saved in the project xml file. Notice that the options are ExportOption name\value. You can also have ImportOptions. In the SaveState method I added code to save the control options in ExportOptions collection you can if you have values that need to be saved for import in the ImportOptions collection.
Ok that’s it for now in the next post I will show you how to create your own handler that will get fired for your custom control.

Catalog Performance Tip

You may notice over time that your site is slow specifically the Catalog calls. In this post I hope to address some performance improvements by doing adding a maintenance task.

Catalog System

I am not going to tell you what the Catalog System is  you can always find info in the help files but I will explain some of the internals that slows down over time. Looking at how Commerce Server retrieve data is a bit interesting. The Catalog System has two databases one Your_SiteName_ProductCatalog and another MCSC_CatalogScratch.

All of the Catalogs, Categories, Products and their relationship is held in Your_SiteName_ProductCatalog database. The MCSC_CatalogScratch is a persisted temporary storage. Some Catalog APIs when called internally need to store their data into temp tables and then do a select of that. Any time you perform a search or use paganation among other calls the temporary location is used.

MSCS_CatalogScratch Database

So what is this database? The thought behind this database was the right idea but used incorrectly. If you have written advanced stored procedures you have at one time or another used temp tables. The creation of temp tables are a performance hit. So what the Commerce Server product did was to create a temp database and create it’s temp table in that database. So a table is created when a query is made and the next request that comes a long will do a quick search of system tables to see if a table already exists if so then it will use it else it will create a new table. Over time you have have thousands of tables but that’s not the only bad part. In some Catalog System stored procedures (which I have done sql traces) I have realized that they used hints like KEEPFIXED PLAN. Since multiple queries of different type use these tables it’s possible that you get a plan that is not very optimized and slow down the execution of the stored procedure.

You are a diseases and I am the cure

So how do you fix this. Well you really can’t touch the Commerce Server stored procedures but you can delete the temp tables to help performance. You should ideally have a job\task that periodically goes and deletes these tables. The VBScript that cleans out these tables is located in “C:\Program Files\Microsoft Commerce Server 2007\Tools\CatalogCleanup.vbs” if Commerce Server was installed using defaults. But you don’t have to use the VBScript just use the stored procedure that it calls.

EXEC dbo.ctlg_DropPersistentTablesInScratchDB 0

EXEC dbo.ctlg_DropPersistentTablesInScratchDB 1

There are two values you can pass it zero and one. I can’t remember exactly but if you use the zero value it will only delete tables that are not currently being used. If you pass one then all tables are deleted regardless if they are being used or not. I have done this with some customers who have had over several thousand tables and the stored procedure returned errors not to worry just keep running the stored procedure until you have not errors.

No comment!!!

I have been asked as to why the comment section of my blog is turned off. It seems that a porn site is trying to up their rating so they posted comment with links. It was becoming a headache to manage so I turned them off. It’s not that I don’t want you to comment I just don’t want to promote porn :). If you have a comment or question please send them to maximum2000 AT hotmail DOT com.

So should I or shouldn't...

I have always been asked the question of moving to CS2009 or staying with CS2007. Currently there are two versions CS2009 and CS2009\R2 which can make the choice even more difficult. Before I go through the benefits of the different version let me just come right out and say it.

I have never been a big fan of CS2009, the reason is that CS2009 was supposed to be a solution accelerator and now it has been made into a product. I believe that it is far from a complete product :(. If you go over the Microsoft site and read the benefits of CS2009 Microsoft is going to give a glass of fruit punch and once you drink, you are going to see rainbows everywhere you look. The only benefit I can find with CS2009 is that you get an extended life support.

SharePoint Integration

What about the SharePoint integration? Sorry, you could have done that with CS200x it’s a matter of modifying the web.config to make the integration happen. Well how about the WebParts? You don’t need SharePoint to have WebParts integration but you do get the source code in the new template site of CS2009 which forces you to use SharePoint. I have looked at the WebParts and they are as complicated to understand as the previous StarterSite :(. It’s a fine line between performance and architecture. Sometime you just need to put a nail in a board, no need to get fancy just get a hammer and drive the nail in.

New API

What about the new API isn’t it cool having a unified theory err… API? I have worked with the new API and I am sorry to say it’s much harder to work with and longer lines of code to write. The only benefit I can see is with R2 where you can have multiple tier architecture. But remember that will come with a cost so even if you are on one server for presentation and Business Logic you will have interprocessing which is expensive and degrade performance. Now some folks at the Product Group have been bold enough to tell me that the new CS2009 API are faster than the old. I find this very hard to believe. The reason is that the new API is built on top of the old API they are not going directly to the CS stored procedures so how in the world can the new API be faster.

New Feature of CS2009

Doesn’t CS2009 give you any new features? Of course it does. Customer can leave feedback, cool huh. Not really, the mechanism of leaving feedback is only if you have SharePoint integrated because that feedback\comment data is stored in SharePoint you can’t use it on a none SharePoint site.

Break Down

So what does CS2009 give you:

• SharePoint Integration
• Unified API
• Product Feedback\Comment\Rating
• CS2009\R2 multi tier architecture
  • Remember CS2009\R2 is 64 bit only
• Has been said that CS2009 has bug fixes that CS2007 does not (I have not found a list of these bugs :()
  • WTF, are you kidding me
• Longer Product Life Support

Please correct me if I am wrong on any of these points or missing something crucial.

Bottom Line

If you have CS2007 then stay on it no need unless you come to end of life of the product then upgrade. The upgrade will be simple just install CS2009 and your old API will work. None of the old API has changed.

If your business need to have multi tier architecture then go for CS2009\R2 when it’s RTMed.

Real Time Inventory? Hmm?

Does Commerce Server support real time inventory system? That’s the question of the day and I hope to shed some light on this subject.

Commerce Server Inventory System

The Inventory is a bit different than the other Systems in Commerce Server. In order to use the Inventory System you have to have the Catalog System but the other way is not true. You can have a Catalog System with no Inventory. Even thought there is a relationship between Inventory and Catalog they are two different systems. At least that’s what I used to think.

If you pay attention to the documentation of Commerce Server it’s not very clear about how closely integrated the two systems are. When caching of Catalog System is turned on then the Product object will cache the Inventory data. But what if I want to call the Inventory API so you can only get the Inventory data. What happens then? Well true believers you get the data from cache! Say what? Yes you get the data from the cache. What if I call the Refresh method? Nope, sorry that only refreshes the Inventory object not make a trip back to the database.

Commerce Server’s Real Time Inventory

So how in the world can Commerce Server be real time? The UpdateInventory and CheckInventory pipeline components are real time as they go to the database and call the stored procedures directly. In fact the documentation only notes that these components are real time.

So how do I get the inventory from the database?

Sadly,  you can’t. You have either wait for the data to fall of the cache or create your own stored procedure to fetch the data. A bit disappointing that you have to write custom call to check the inventory levels :(. By the way I looked at how difficult it would be to change this and it’s just an override boolean internally with the Inventory object. The Inventory code is hard coded to check the cache first. Say what!

When the gun goes bang bang bang, who's gonna know who's the one?

Got Marketing!

When the first request is made to a Commerce Server site Commerce Server loads it's modules and populate it's caches based on the configuration of web.config. One of these modules and Cache that can be modified is the Marketing data.

Marketing Runtime

Commerce Server loads it's Marketing data (Ads and discounts) in one shot by calling mktg_spRuntimeLoadDiscounts stored procedure. You can change this behavior by having your own stored procedure called. Why would you want to do that maybe  you want additional columns to be returned or have a staging environment and you want all discounts that are active and not active to be evaluated. Just make sure that you not omit any columns and records that the original stored procedure returns.

Change the default behavior of loading Marketing data