Finally, I put a new version of the web service. In this version, the new functionalities are the following ones:

• on the ITQWebApi class: new methods in order to see if the SEC connection is down or not, and also the Oracle connection too: IsSECDown, IsOracleDown. New methods permitting to manage the automatic reconnection to Oracle or to the SEC: IsAutomaticSEC, SetAutomaticSEC, IsAutomaticOracle, SetAutomaticOracle. By setting in a automatin toc mode, depending on the timeout that you can configure in the web.config file, if the SEC - or Oracle - is down, it tries to connect again.
• new class : ITQWebEvent, that permits to notify the creation/modification/deletion of instruments, or the creation/modification/deletion of deals.

Concerning the detection of the Oracle connection, I'm just using the fact that I need a specific thread on which all connections to the API should be done. Since, between different calls, this thread has nothing to do other than waiting for calls, on timeout, it calls the CSRApi::HandleEvent, and makes other treatment like detecting if the Oracle connection is down or not. For detecting if the Oracle is down - it means, to detect if the static Oracle OCI cursor used by the CSRSqlQuery class is valid or invalid -, I didn't find another way than making a simple query on database, for instance "select null from dual". When the Oracle connection is down, I try to reinitialize the connection to Oracle by calling the O_connect method, available on v5.3.2 in the Low Level toolkit header g_oci.h. This method, not documented, accepts as parameter the user returned by CSRApi::GetDBAccess. It deletes the static OCI cursor and creates a new one. Of course, it could be that some other Sophis objects keep a copy of the preceding cursor, but, for instance, after a reconnection, I didn't get errors when calling GetCSRInstrument or getting CSRTransaction objects that were not already loaded in memory, and using CSRSqlQuery methods on other queries run perfectly.

Concerning the ITQWebEvent, the more difficult was trying to get all the tags needed to send the right events, and their associated type. First, in order to retrieve them, I developed a method DumpEvent, that dumps all the events, received in the Sophis log file - including child events of events -. Second, when getting errors, I used the assembly code with the associated .pdb files in order to see which tests were done. For instance, for each test on tag, the assembly sequence is the following one:

0E5EA7A1  push        edx 
0E5EA7A2  push        61616372h
0E5EA7A7  lea           ecx,[esp+14h]
0E5EA7AB  call          sophis::misc::CSREvent::Get (0E5E89B0h)
0E5EA7B0  mov         edi,eax
0E5EA7B2  test          edi,edi

Otherwise, I added logs on the managed C++ side too, by using the log4net assembly.

Using massively complex and exotic structured financial products takes a lot of time to calculate. The solution is to distribute the calculations on a grid. One of the most problem is then the synchronization of the market data between the client (Sophis Risque or Value) and the server on which calculations occur. I will give there a sample on how to distribute easily Sophis calculations, for all pricing models, on a grid.

The purpose of this article is not really to show how to get the best path to calculate several instruments in a minimum time, neither to show how to collect statistical pricing data for calculating the best path. It will not give the solution on how to push the market data on the grid too. The purpose of this article is simply to demonstrate the use of the CSRAsynchronousTreatement class provided by the Sophis toolkit, and how to load the Sophis API in the grid in order to avoid access violation due to the monothreading of the API. Such a solution is already given as sample for the webservice, but in fact, it is the same design pattern.

I developed myself a gridcomputing system, for 5 years, using the DCOM technology. It is quite a little solution, but this solution is free (I distribute it as freeware and will upload it as next resource on this website), and permits at least to divide the pricing by 4, if using a bi-processor dual core for instance, since the calculations will take 4 processors on the computer instead of 1 as Sophis Risque do it, and when the distribution is "well-done".

The solution is composed of 3 dll's, sources are given in order to be compiled with the right version of the Sophis API... All the stuff was recently compiled with VS2005, using the Sophis toolkit Risque 5.3.1.1 and tested on this environment, under MS Vista:

•  2 dll's for the server side - gridcomputing system -: a plugin for the grid, SchedulerSophis.dll, and a wrapper, SchedulerSphBridge.dll, to access dynamically the Sophis API pricing stuff always on the same thread. SchedulerSphBridge.dll has to be put in the Sophis Risque installation folder. It loads all Sophis dll's as a normal Sophis API do it. The executable that loads this dll is SchDispatcher.exe, so a SchDispatcher.ini file can be configured in order to configue the Sophis API too.
• 1 dll for the client side, SchedulerSphPlugin.dll, a Sophis toolkit one, that is loaded in Sophis Risque, enabling the distribution from the client side. It provides two menus - scenarii -, in the Sophis user menu, in order to configure the access to the grid, and a package pricing model, "WM gridcomputed".

The configuration of the access to the gridcomputing system is done through a Sophis scenario. All data of the configuration are stored in a .ini configuration file. In the WM server field, the computer that is used as grid manager should be input. This gridcomputing can configure several grids, it means, several groups of computers. If calculation have to be done on a specific computer, the computer name should be input, otherwise, in case of a gridname, the gridname should be preceded by the '@' character.

The grid can be configured as follow, using the console of the gridcomputing system:

In our sample, the calculations were distributed on a pool threading, using 4 processes on the same computer. The Sophis API have to be loaded at first. This task is executed on all computers and all processes of the grid at first as follow. A trigger can be configured in order to load the Sophis API automatically at the gridcomputing initialization.

Calculations can be then executed, using a Sophis package with a specific model, "WM Gridcomputed". This model overrides the RecomputeAll method in order to call an instance of the CSRAsynchronousTreatment.

Once executed, all the results of the instrument calculation are pushed to the client. The client retrieves them and makes the aggregation. The market data that is used is the one of the package configuration dialog (the CSRInstrumentDialog is in fact a CSRMarketData). Preferences and data of the market data are pushed to the server/grid. Concerning the market data of the package, in fact, only the pricing date has to be pushed. The grid can be configured as follow:

I will make a .msi and push the grid solution on the website on next week-end.

When implementing new portfolio columns, using the CSRPortfolioColumn class, in most cases, memory cache is needed in order to display data: each time that the user clicks on the folio list, changing the selection or the size of the window will generate a call to the GetPortfolioCell, Get PositionCell or GetUnderlyingCell methods. Then, of course, making queries directly on the database in these methods shall be avoid. Instead, a memory cache shall be filled at the beginning and queries on the cache should be made.

The purpose of this article is then to describe how to implement such a cache. It should be generic: instead of implementing a cache for each column, with a hard coding in C++ of the query, a generic class will be designed. Of course, the main problem of such an implementation is the refresh of the data, through the SEC or Real Time. A solution will be described.

The specifications:

• no code to generate, queries to fill the cache should be described outside the C++ code. Just having to configure the query, the C++ code should be able to discover the fields and the field types returned by the query,
• a cache manager should be able to give the list of available caches, able to load, reset, update one of the cache,
• a cache should be not necessary linked to a SQL query. It could be a XML file, a flat file or any implementation, then, a factory of implementations should be welcome,
• it could be thread safe: we can imagine that the load/refresh of the data could be made on a second thread
• in order to implement specific search, several indexes on the same data could be defined, outside the C++ code
• refresh of the data on specific SEC /DRT tag, without C++ coding, all the configuration should be made outside
• code should be called from C#, or able to be interfaced with managed C++, in order to be able to build web services easily, C++ interfaces -abstract classes - should be defined

Concerning the SQL cache implementation, in order to discover file types dynamically, in a thread safe mode, I didn't use the CSRSqlQuery class. This class does not provide interfaces to discover dynamically the fields, thus, the use of this class will need to describe the return of the query for each query, that's not really nice. Secondly, as for all SQL Oracle queries provided by  the Sophis toolkit, it is not thread safe. Instead, I'm using OleDB, ATL and an own implementation of CDynamicAccessor.

Concerning the cache manager, all configurations are done in the itq_memorycache, itq_memorycache_indexes, itq_memorycache_index_fields, itq_memorycache_events and itq_memorycache_event_tags. Several queries could be defined. At the moment, only one implementation, corresponding to the SQL cache, is done, so only one cache type (0 corresponds to SQL) should be used.

The itq_memorycache table gives the list of the configured caches. The description field will give the query to load all items, the itemdescription field will give the query to load one item. The primary key is used to configure the item query, so, when itemdescription is configured -the itemdescription field is optional-, the primary key should be defined too.

The itq_memorycache_indexes and itq_memorycache_index_fields tables describe the available indexes on the data of the cache. You can define indexes by column names or column ids. When using column ids, queries will be faster. To describe an index, you just have to give it a name, and the list of the available fields, in the right order, that you want to use.

The itq_memorycache_events and itq_memorycache_event_fields tables permit to configure the SEC/Real Time update. Different actions could be defined on RT tags: loading all items, loading one item, deleting all items, deleting one item, updating one item. Taking a look at the samples in the .sql provided file will help you to configure other tags. Of course, if you don't want to refresh data one SEC/RT, you just have to set no tags/events for one cache definition...

Tuning of the strings: one class, ITQString, was defined in order to optimize the use for the memory, the memory fragmentation and the performance. There is, of course, no pointer on internal data in this class. All the tuning between the OleDB description fields and the memory cache types is defined in the ITQSQLCache::Initialize method. There is no bug on the length of the optimizations of the strings. In order to see why it was configured like that, you just have to run the following query on your database: select sum(length(libelle))/count(*) from titres...

As result, a small class helping me to define user columns very easily, ITQInstrumentColumn and their different implementations, show how to use these memory caches. Of course, an interface permitting the definition of the user column using a specific configured cache will be nice...

All intefaces are defined in IITQMCInclude.h file. It is not a big job to make a wrapper in order to use it in a webservice: all the stuff takes me only one week to define and implements all the code in C++.

Concerning the work that is currently in the pipe, but not currently done:

• loading items, refresh and index packing on other threads, adding another type of load - actually 2 types: at the launching of Sophis Risque, and on demand -
• adding thread locks when accessing to the data
• dialogs, in order to list all the caches, and enable the load/refresh by the end user, according to the rights of this user
• interactive user column configuration that will use the memory cache

Concerning the SEC/RT update, it runs perfectly. For instance, in our sample, when changing the name of the share from AXA to AXA BELGIUM, it will update the memory cache too, by generating a SQL query on database (could be nice to schedule it on other threads too), and refresh all screens that were registered as memory cache clients - using the IITQClient interface -, in our case, for instance, the portfolio view.