ITQuants blog

By Philippe Bonneau on 10/11/2016 10:24 PM

I guess that everybody who uses Fusion Capital Sophis or Fusion Sophis already got the classic "Internal error on MFC message" dialog box that announciates that a crash, access violation or whatever, occured.

In most cases, unfortunately, classic logs are not sufficient to investigate why the problem occurs. The goal of this post is to propose a method that permits to dump the callstack when such an event occurs.

By Philippe Bonneau on 6/3/2016 2:13 PM

Before the v7, in order to serialize specific indicators calculated by the internal quant libraries, virtual methods like CSRInstrument::GetCalculationData and CSRInstrument::SetCalculationData were used. These methods are not deprecated and a specific implementation of the class CSRComputationResults should be used. The goal of this post is to explain how to declare such a class and to give a complete sample ready to use.

By Philippe Bonneau on 4/29/2016 11:35 AM

In my previous post, I've described how to update third parties using the API, as it was done by several Misys clients. This time, continuing on this subject, I will explain how to update third parties, using XML files. I remembered that I've implemented this solution at least one time by a client, in a web service. Third parties were updated using the Sophis xml definition, that you can find in the party.xsd file stored in the schema folder. The web service implementation is already described more in details in this post : "How to develop a IIS webservice using the API". Unfortunately, the C++ code is not anymore maintained in the v7: C++ header files that were present in the Low Level Toolkit folders are not delivered anymore. A workaround exists, this is the subject of this post of how to replace this code.

By Philippe Bonneau on 4/28/2016 6:11 PM

For clients on which the Sophis Risque or Sophis Value was already installed, during the migration, it is quite easy to detect which modifications have to be done when comparing the code. But this step is only one of the different tasks that have to be done. For example, almost nothing was changed on the interface CSRThirdParty (eq CSMThirdParty in C#) concerning the third parties (entities, depositaries, counterparties, brokers...). But unlucky, when trying to use some piece of code that permits to update the external reference, the domiciliation, the broker or whatever which field, you will got an exception in v7, indicating that the method is not implemented. This is a well known problem on each client migration.

By Philippe Bonneau on 3/8/2016 6:20 PM

Testing recently the calculations of some instruments (Convertible bonds) on the calculation servers, I've discovered that very old instruments which are not anymore in position were sent to calculation servers. By the way, these instruments are not valid too, and trying to calculate them today will generate some check error messages. These instruments were stored in position in some folios several years ago (trade date far away from the well-known REPORTING.DATE_DEB date), but are still loaded in memory and sent to calculation servers. The problem does not appear when calculating the folio locally, only when calculated by calculation servers, and the whole calculation of the folio is then rejected. The goal of this post is to propose a workaround that permits to calculate the folios which contain such instruments.

By Philippe Bonneau on 2/2/2016 1:49 PM

Some days ago, I talked about the possibility to use the Sophis.Core.Data, Sophis.Event.Core and Sophis.Event.Risque assemblies, in order to listen to specific events for toolkit columns performance optimizations. I've tried to use the same mechanism in a batch, using the following code :

class Program
{
    static void ProcessEvent(Sophis.Event.IEvent myEvent, ref bool bProcess)
    {
    }
    static void Main(string[] args)
    {
        CSMApi _api = new CSMApi();
        try
        {
            _api.Initialise();
        }
        catch (Exception e)
        {
            return;
        }
        Sophis.Event.SophisEventHandler _handler = new Sophis.Event.SophisEventHandler(ProcessEvent);
        Sophis.Event.SophisEventManager.Instance.AddHandler(_handler, Sophis.Event.Thread.MainProcess, Sophis.Event.Layer.Model);
        bool _stop = false;
        while (!_stop)
        {
            Sophis.Event.SophisEventManager.Instance.Dispatch();
            System.Threading.Thread.Sleep(500);
        }

Unfortunately, this code does not work as is. Putting a break point in the ProcessEvents method shows that it is never called, even if messages are received and processed. The goal of this post is to explain how it works and what it is missing.

By Philippe Bonneau on 1/25/2016 4:26 PM

One client asked me to add some very visible message in Fusion Invest that permits to warn when the pricing date is not the current date. A good sample of such a message is the native toolbar that is displayed by the application when the simulation mode is selected. The expected result should be the following one:

By the way, the core code of Fusion Invest is still written in MFC/C++. Adding such a toolbar is possible by written some C++ code that uses the MFC classes. The goal of this post is to give the code that permits to activate such a functionality.

By Philippe Bonneau on 12/30/2015 6:37 PM

When overloading Sophis dialogs with the toolkit, you need the Win32 resource IDD of the dialog in order to get the resource template somewhere in the Sophis dll's (in most cases in the SophisRiskEn.dll), and customize it by adding new controls. In previous versions of Sophis, there was an option that permitted to display this IDD, by right double-clicking on the Sophis logo in the About box, and, after, by right clicking in the dialog.

Right-clicking it in the v7 About box does not enable it anymore, even if this option is still usefull. Looking at the code of the dialog, I've found finally the way to display it. The goal of this post is to share it.

By Philippe Bonneau on 12/11/2015 11:34 AM

In most cases, when portfolio columns are toolkitted, they use an internal memory cache to avoid SQL queries on the database when the portfolio lines are displayed (on scrolling or what ever). In order to refresh this memory cache, the Sophis API provides a method, CSRPortfolioColumn::GetRefreshVersion, which is incremented on some actions, for instance when a deal is inserted. Most toolkit implémentations use only this flag to refresh the cache. Unfortunately, testing only the version flag could generate some freeze, when a lot of a deals are inserted and SEC events are received. The goal of this post is to propose a solution in v7 to optmize the refresh of the cache.

By Philippe Bonneau on 3/12/2015 5:30 PM

In the last version provided by Misys concerning their product Fusion Invest and Fusion Banking Sophis (v7.1.3.x), new functionalities were added to the SRM module (for Sophis Risk Management, ex-VAR module), that permits to calculate asynchronously data on calculation servers, to retrieve the results and store them in a Reference Price server, and finally to display them on the client screens. 

I think that it is one of the most powerful enhancement done by Misys on this version. For clients having problems when using toolkit columns, this permits to increase the performance when, for example, aggregation is done on a very large perimeter, with lot of dependencies between columns.

The goal of this post is to explain the architecture of this component and how to use it.

By Philippe Bonneau on 3/12/2015 12:28 AM

Since one of my conversation between Alexis de Bernis, me and a Misys consultant was forwarded to several clients by Misys, the best thing I can do is perhaps to publish it. The topic was analyzed and resolved by Alexis, who I thank again for his judicious remark. The description concerns the way we have to use the CSMExtraction, and most of the class of the C# API provided by Misys for their product Fusion Capital Sophis.

By Philippe Bonneau on 2/5/2015 5:15 PM

Working on the migration from Sophis Value v4 to Fusion Invest v7 (same product by Misys, only the version is different), just by changing the access to the database, from DEV to UAT one, I got a message that I never had before:

Well, well, I thought, it should be due to the fact that Oracle scripts were not launched on the database, and there should be a mismatch between the dll's used (DEV) and the database (UAT). Unfortunately, the dump was the same one, recently loaded and updated by the same SQL script on both servers.

By Philippe Bonneau on 12/7/2014 8:01 PM

The root cause of needing this information is that we want to display user portfolio columns on which we need to calculate aggregation. I've already seen some client implementation on which the aggregation is calculated when the callback CSRPortfolioColumn::GetPositionCell is called, when detecting any change on the GetRefreshVersion of the same class.

The advantage is that this code is called only if the column is displayed, but unfortunately, that's not the right way to do it and for this reason:

 - on GetPositionCell, we don't know at which level we need to calculate the aggregation and the parsing of all folios is then needed. When many folios are already loaded, it decreases dramatically the performances and the software freezes in most cases.

- on the CSRGlobalFunctions, Misys provides a method, StartPortfolioAddition, which is called each time that the core engine needs to make an aggregation. For instance, changing the currency of a folio will call this method, with the folioId corresponding to the root folio on which the aggregation needs to be recalculated. 

In the last case, the method does not give the list of columns on which we need to aggregate. To detect it, Misys provides unfortunately no official and supported method.

The goal of this post is to give the way to get it, in all Sophis versions.

By Philippe Bonneau on 11/30/2014 9:37 PM

I've already implemented several times user columns at the clients of Misys. Each time, taking a look at the developments already done by the client or specific deliverables made by Misys, I encounter the same problems of performance, which are:

- in most cases,  no memory cache is implemented, calculations are done on the fly, with, last but not least, direct access to the database, sometimes hidden by the complexity of the Sophis API,

- if a memory cache is implemented, the refresh is not necessary very well optimized. In most cases, no SEC listener is developped and the refresh is done according to the version given by CSRPortfolioColumn::GetRefreshVersion.

- calculations of aggregations are done on CSRPortfolioColumn::GetPositionCell (or CSMPortfolioColumn.GetPositionCell in C#) instead of being implemented in the CSRGlobalFunctions::StartPortfolioAddition or EndPortfolioCalculations, with no indication on the level on which the aggregation should start. It means that for each refresh needed, the algorithm implemented will parse the whole portfolio hierarchy each time. If all folios are loaded, the parsing can freeze the whole GUI.

The goal of this post is then to give the rules for implementing user columns as it should.

By Philippe Bonneau on 11/30/2014 9:25 PM

This week, I just wondered why the Sophis R&D didn't permit to override the CSRGlobalFunctions class in C# before the 6.3 version. If you take a look at the DotNet toolkit, you will see that a CSMGlobalFunctions is defined, but no static method Register exist as for the other classes. This point was several times scrolled up to Sophis in the toolkit forums, and I know some clients who ask Sophis to provide a toolkit permitting their own implementation. Unfortunately, they don't have the code and the interface provided in C# does not implement all virtual of CSRGlobalFunctions, for instance StartPortfolioAddition which is a callback on which we can implement the agregation of results for user columns that we want to store in a cache.

For remember, the overriding of the CSRGlobalFunctions class permits to be called when the folios are calculated (StartPortfolioCalculation, EndPortfolioCalculation).

The goal of this post is to describe how to implement a C++ implementation of the CSRGlobalFunctions that permits to implement callback in C#.

By Philippe Bonneau on 10/29/2014 11:47 PM

Working on a migration to the last version of Fusion Invest (7.1.x), and even if most Sophis developers warn me that this class will be deprecated in future versions of Sophis Risque/Value, I've discovered how deep changes were done on it.

Trying to migrate my performance tool ITQLogger in order to test the new grid cache server, I've got a lot of modifications to do and most cases are now obsolete..

For remember, for those who don't know this class, it was previously mostly used by developers in this case: 

 - changes are done on database, without using the API,

 -  other programs connected to this database have to be notified, to refresh the data.

For example, the last development I've made using a Sophis native 'AJTI' event was done when I had to insert directly in the TRS_BASKET and TRS_BASKET_ADJUSTMENT Oracle tables records, and notify other GUI clients that the instrument was modified. Such development was done due to the well-known poor performance on Sophis basket swaps, making it impossible to use the native CSRInstrument::Save method when a lot of adjustments are done (like on portfolio swaps).

The goal of this post will discuss which workarounds have to be done.
By Philippe Bonneau on 10/10/2014 8:07 AM

Using the Sophis toolkit, it is possible to display a list when typing in a edit control. For example, in the default native transaction dialog, some references of instruments appear once the first characters are filled. The class that permits to reproduce this behaviour is CSRInstrumentCode which inherits from CSRGenericAutoCompletion. According to the comments provided in the sphcode.h header, there should be a sample in a SphSrc/AutoCompletion folder... Unfortunately, I haven't found such a sample, and looking for questions/answers on the Sophis support forum, I only have found one question without answer. This post will show how to implement some code in order to get the expected result, it means giving a list of results when typing some characters as follow:

By Philippe Bonneau on 9/30/2014 2:15 PM

I've got a question today on how to get the values displayed in the Inflation Curve Numerical Results dialog, as described below. This dialog can be displayed by clicking on the menu Data\Inflation and then by double-clicking on one index, and then on the toolbar button which appears on the right of the RIC one.

The goal of this post is to describe how to get the values and the ones retrieved from the quotes too.

By Philippe Bonneau on 9/29/2014 2:05 PM

Using the Sophis toolkit, it is easy to save user data by filling the parameter columnName in the CSRElement constructor and the ones of its derivated classes. When overriding the CSRInstrumentDialog class, the data will be then stored in a field of the Sophis native Oracle table TITRES, on which you have to add the expected field. Prefixing the field to remember that the field was created by the client is the common rule used. For other user data on which an entire Oracle table needs to be created, it is safe to override the CSRFitDialog::Save method too. Now, if you have to store other data even when having no GUI open, when using batches for example, the best is to add an implementation of the CSRInstrumentAction class, and to save data on the callback NotifyCreated and NotifyModified. All these things are basic concepts explained in the Sophis documentation. There are some cases on which we need to save data after the Oracle commit. This post will explain how to implement it.

By Philippe Bonneau on 9/15/2014 5:07 PM

I was asked recently if I knew how to change the color of the titlebar on Sophis Risque in order to catch the user attention. At first, I've made a wrong answer: "Easy, you just need to paint on the WM_NCPAINT as Sophis Risque does it when the simulation mode is selected (and as I've developed it on the v4)" !! ...

Unfortunately, this mode does not work anymore as it was developed for Window 95/2000.... A lot of articles can be found on the Net (like codeproject or codeguru), but no one gives the code to run it with the themes that were developed on XP, or on Vista/7 when the Desktop Window Manager service is started. For example, when the Windows Desktop Manager is disabled, as best effort, and using the MS SDK functions ::DrawCaption and ::DrawFrameControl, the dialog will look like this one:

Note that the minimize, maximize and close buttons do not have the new style as expected.

The goal of this post is to describe how to do it.

By Philippe Bonneau on 7/29/2014 5:04 PM

One client, a fund asset manager, asked me recently if I didn't have any tool that permits to make non-regression tests or performance tests when migrating from a version of Risque or Value to another one.

The goal of this post is to provide such a tool, when doing non-regression performance tests. This is based on the fact that the SEC server linked to the Oracle server is the bottleneck of the Sophis architecture.

When each Sophis Risque/Value client receives a notification, in most cases on versions <5.2 and in some cases after on Sophis internal messages, it does a query on the Oracle server. On user events, it depends how it was developed.

By Philippe Bonneau on 7/16/2014 11:28 PM

I've decided to restore some posts that were published on the previous it-quants.com website. These posts are old and published between 2007 and 2008, but are still interesting. One of the most interesting post described how to build a IIS webservice in order to export some functions that can be used by Talend, which is now a well-known French ETL tool.

By Philippe Bonneau on 7/10/2014 8:47 AM

Some functions of the Sophis toolkit API are exported and stored in this dll. It contains at least the classes that permit to add contextual menus to the portfolio view, overloading the CFD deal input dialog, the portfolio headers (which correspond to the dialog in which the main sensitivities of the folio are displayed) and payment methods.

By the way, this dll implements some functionalities for the .Net toolkit, it means that it is linked to the MS C# .Net framework, loading other assemblies when it is loaded by dlls or executables linked to SphPortfolioGUI.lib.

I have the case where most of the dialog overloadings and Sophis toolkit calls are stored in one toolkit dll. Unfortunately, this dll exports a lot of functionalities that were used by night batches. It means that the night batches load the dll twice:

  • one using the static link, when MS starts the process
  • the second one, by the Sophis API, since it is declared as a toolkit/plugin dll and loaded dynamically when the CSRApi::Initialise method is called.

I wanted to add contextual menus to the porfolioview. It causes no problem when the toolkit dll is loaded by Sophis Risque, but does not work when night batches are launched, since it tries to load assemblies without initialilzing the .Net security as SphRisque does it (creating specific AppDomain for example). I got an access violation as below:

I will explain in this post how to resolve this problem.

By Philippe Bonneau on 7/8/2014 11:07 AM

In one of my previous posts that can be found there: http://www.it-quants.com/Blogs/tabid/83/EntryId/39/Sophis-Risque-how-to-manage-limit-memory-allocation.aspx, I've detailed a method that works fine on the following OS: Windows XP and Windows 8, with or without Citrix, using the MS CreateJobObject method. 

Unfortunately, Microsoft has changed the behaviour of the jobs on Windows 7, and limit the creation to one job on this OS. Citrix is using already one job to control the limitations that can be driven through their configuration manager. For example, we will consider the case where the memory consumption is configured per Sophis user. The goal of this post is to explain the solution that could be retained, without exporting the rights to the Citrix configuration and let keeping Risque/Value manage the memory consumption as before.

By Philippe Bonneau on 5/29/2014 10:31 PM

In some circumtances, it is interesting to get the list of columns which are currently used in the different portfolio configurations. Getting such a list permits to see which toolkit columns are really used. Missing columns can be removed from the toolkit dlls.

The configuration of portfolio columns are stored in a BLOB field of the Oracle table USER_SETTINGS. We just need to use an XPATH query to get the list.
By Philippe Bonneau on 5/21/2014 11:06 PM

For remember, on SQL databases, triggers permit to change the behaviour of insertions, modifications or deletions made by a program, by adding SQL notification callbacks. This permits to make complex check on field values, or to copy some record in audit tables for example.

On Oracle, a trigger can be implemented after or before the action. On before triggers, values can be changed by the notification callback. On after, we are sure that any modification done by the program or on before triggers is taken into account in the :new record.

 Sometimes, complex check should be made on several tables, during a transactional modification for example. In such a case, a BEFORE COMMIT trigger is mainly required. Even if it is more recommanded to do such a check in the client program, on third party databases, there is no choice, additional checks can be done only on the database server. Unfortunately, Oracle does not permit such a trigger implementation directly, a workaround has to be developped. This is the purpose of this post.

By Philippe Bonneau on 5/15/2014 4:39 PM

Using Sophis Risque 5.3.7 and equivalent Value version or less, negotiation date of the deals done by the trader are stored into database using the local time zone of the machine on which Sophis Risque is launched. Unfortunately, on multisites/multi countries architecture, the timezone is not stored into the database, raising an issue on the last requirements for compliance and regulatory checks. The goal of this post is to explain how to add the required timezone.

By Philippe Bonneau on 5/2/2014 2:05 PM

There could be several ways to do it. For example, I've already seen some which were very slow, creating temporary tables and indexes, and making join between the records by comparing the max value... Of course, the goal of this post is to show the fatest way to do it.

By Philippe Bonneau on 3/27/2014 5:29 PM

On Sophis Risque 5.3.7 and Value equivalent version, some data needed for the pricing or that could be needed for traceability for regulatory convenience are not audited. Triggers and supplementary tables need to be defined in order to duplicate them, and to offer to the internal audit service the possibillity to know which user has modified the data.

By Philippe Bonneau on 3/24/2014 3:29 PM

Auditing some performance at one client, I needed to make some search on the database in order to see how the database is growing. This is a part of the results of this study and could be considered as a beginning to reference some other useful queries. It is using in most cases the audit tables and not the logs. Of course, the use of the Sophis logs could enhance the results made by these queries too.

By Philippe Bonneau on 3/14/2014 6:10 PM

On Sophis Risque and Value, it is possible to add specific user rights, by adding tabs on user/group configuration. Tabs and fields are added using the Oracle table USER_RIGHT_TABLE. On previous versions of Sophis, such rights were defined using C++ toolkit and the obsolete class CSRUserRights.

By Philippe Bonneau on 2/20/2014 6:39 PM

The Sophis API provides a method, CSRSqlQuery::MultipleWrite, which enables the developer to insert several records in one command, using arrays. Unfortunately, the method doesn't work fine if an error occurs during an insertion. In this case, next calls to this method will return unpredictable results, like not data found:

 This error occurs in fact only when the optional parameter that corresponds to the Oracle cursor is not filled. In this case, the global one which is used remains in uncertain state after the error. This error occurs at least on 5.3.4, 5.3.7 versions of Sophis Risque.

By Philippe Bonneau on 2/6/2014 6:59 PM

Even on x64 OS environment, it is not interesting to let the process using all the available memory. In some cases, like on CITRIX servers, the fact that one user consumes a lot of quantity will impact all users too. The goal of this post is to analyze all the possibilities that gives the MS SDK, in order to capture either bad memory allocation or to limit the process to a certain quantity of RAM.

By Philippe Bonneau on 12/6/2013 1:14 PM

When trying to use LoadGeneralElement with a CSREditList declared in the transaction dialog, without mapping it to an Oracle table, it does not work. The method will return true when getting some value of one cell, but the output address won't be filled, even if GetGeneralLineCount returns the right number of lines.

Some piece of code has to be added, using the internal callbacks.

By Philippe Bonneau on 10/2/2013 5:53 PM

In most cases, when installed on the local hard disk, there is no problem to launch Sophis Risque.

Launching Sophis Risque from a network shared folder is quite a little bit harder, due to the fact that Sophis Risque loads 3 .Net assemblies which have to be registered on the GAC.

By Philippe Bonneau on 9/19/2013 4:58 PM
When taking a look at some database fields which contain absolute and relative dates, it appears that both formats are stored in a numeric value. In order to know the real value that is stored, some conversion has to be made, using specific limits. In other words, it means that the Sophis Oracle num_to_date method has not be used alone.
By Philippe Bonneau on 7/30/2013 8:39 AM
After debugging some code, in order to know why the methods CSRInstrument::GetInstance and gApplicationContext->GetCSRInstrument returned null in some cases, I discovered that the method CSRInstrument::MultiSave was used badly. In this case, the instrument was duplicated using the Clone method, setting the instrument code using the SetCode to 0, and using reserveCode in relationship with GetNewCode. The insertion into database was done using the CSRInstrument::MultiSave with a pmModification flag.
By Philippe Bonneau on 3/5/2013 12:16 PM

Sometimes, we need to save informations ourselves in some Oracle table without using the mapping between some CSREdit control and some field in the Oracle TITRES table. Sadly, on Sophis Risque, and despite several requests I made, there is no specific callback to implement for the Oracle rollback or commit. Thus, the only way to implement the save of additional informations, and recommended by Sophis, is the following one:

By Philippe Bonneau on 11/30/2012 2:37 PM

On the server side, several types of server, agreed in most cases by the Corporate architect, are used:

 - IIS + .Net
 - Websphere + Java

 - ....

 Depending of the type of webservers, the implementation in Sophis Risque, in C++, can be done easily or not. This post will describe the development done to access a secured webservice deployed on a Websphere webservice.

By Philippe Bonneau on 11/8/2012 1:22 PM

Using the CSRCustomMenu class gives the opportunity for the developer to control at least the behaviour of the items and the type of the internal variable used for storing the information. The subject of this entry is to detail the virtual methods which have to be overloaded in order to control the type of the variable.

By Philippe Bonneau on 10/10/2012 2:29 PM

Imagine that the instrument dialog was overloaded by toolkit, and that the sicovam is required to make some SQL query when opening the insturment dialog. If there is no test on the fact that the instrument is displayed in audit mode, this will certainly fail since the sicovam given by the CSRInstrument will correspond to the "new" sicovam and not the one used in current mode.

By Philippe Bonneau on 9/28/2012 11:33 AM

In most cases, and on versions of Sophis Risque before the 6.0, it is not possible to access to the native fields using the toolkit Sopshis method GetElementByRelativeId, beacause such fields are not natively a CSRElement.

The goal of this post is to show how to use the Low Level Toolkit, in order to update the display, and the internal structure on a specific case (Folio creation/modification).

By Philippe Bonneau on 7/12/2012 2:43 PM

Needed and usefull for instance on unitary tests when creating new instruments in order to get new references and not to violate primary keys

Search blog