Email This Author Comment On Article About The Author More Articles By This Author

Introduction

A trading organization relies on commodity market data. A commodity trade is only as good as the Market Data it is based on. Incorrect or delayed commodity market prices/information lead to a lack of understanding of market movements and situations, resulting in un-informed, and potentially loss-making, trading decisions.

We have seen multiple instances of many key business processes (Pricing and Modeling, Risk reporting, Market Analysis, Settlements Reporting etc) coming to a standstill, or having to rely on manufactured data to be able to continue, due to delayed or incorrect data. In the worst cases, the processes continue unaware of the fact that data is either missing or incorrect, resulting in significant issues.

As a result, we believe it is imperative that Market Data systems in an energy trading organization have the following characteristics:

1. Available -- Systems are always available with appropriate failover mechanisms

2. Timely -- Systems always provide market data at the time expected with appropriate backup mechanisms

3. Complete -- Systems must recognize missing data upfront to be able to proactively invoke the failover mechanisms, and notify providers and consumers

4. Correct -- Systems must recognize instances of invalid data to be able to proactively notify providers and consumers for corrective action, or take corrective action where possible

Let's first look briefly at different typical environmental aspects of a market data system, and issues that prevent effective market data dissemination.

Different environmental aspects of a market data system setup

1. Sourcing -- We have seen three distinct approaches for sourcing Market Data:
   1. 3^rd party Market Data Software Providers -- Typically via custom products/databases installed at the client site. Data is distributed within the client organization via custom mechanisms. Some popular 3rd party software providers include MorningStar.Inc, GlobalView, Sungard etc.

   2. Independent Market Data Providers -- This is achieved through reports via email, FTP site and website access. These include financial data providers like Argus Media, TFS, Reuters etc.

   3. Public/Licensed sites -- Users directly download data from public/licensed sites for consumption. This includes exchanges, ISOs such as EEX, ICE etc

2. Licensing -- Many market data providers charge consumers for use of their data. This can take two forms:
   1. Direct -- License fees for obtain data directly from independent market data providers in different forms

   2. Indirect -- License fees for obtaining data via a 3rd party software provider. Sometimes, publicly available information also needs to be licensed by the provider to obtain it via a 3rd party software provider

3. Prevalent technical architecture -- The different technical architectures that are typically used for providing market data across an organization comprise of the following:
   1. Distributed -- The data is distributed via standardized communication channels and formats across the organization

   2. Point-to-point -- Direct interfaces are built from the market data software, or source, to the consumer applications

These aspects have a bearing on the success of market data distribution and usage. We have seen some typical issues as a result of these environmental aspects.

Typical Issues of a market data system setup

We have seen the following typical issues.

1. Data Sourcing issues
   1. Delayed data -- Issues with file formats, data formats, delayed communication, software failure etc can prevent 3^rd party software providers from communicating data expected or received from the independent market data providers.

   2. Incomplete data -- Sometimes the independent market data providers might fail to publish certain data sets. Additionally, 3^rd party software providers might pass through the data sets with gaps to the end consumers

   3. Incorrect data -- Everyone makes mistakes. Data providers are no exceptions. Often the independent data providers publish corrections, but the 3^rd party providers might again pass through these incorrect data sets to the end consumers

2. Licensing
   1. Access Restrictions - Users might access the data without licensing, and this can be a viewed as a breach of the data licensing terms and can have legal implications. On the other hand, users might have valid need for the data, so there needs to be a mechanism to enable licensing and legal access to the required data

3. System issues
   1. Bus Failure -- The underlying Bus fails due to log file size increases, credentials etc impacting data communication

   2. Server/Component failure -- The underlying server or database might encounter issues (space, memory etc) that can cause components from functioning, or data from being added or extracted

4. Operational issues -- In addition to the above, lack of a proper operational infrastructure compounds the problem due to slow response times from lack of advance notification of, or information about, a problem

To address these issues, we propose the following process and architecture based on our knowledge of successful implementation and continuation of market data systems.

Process and Architecture Recommendations for a Market Data system

Process -- We believe a standardized request to response process must be implemented with a centralized market data management team coordinating all process. This market data management team must socialize and implement the following processes:

1. Standardized data request process -- Standardized templates must be used by the users to request for the needed data, and the requests sent to the market data team

2. Standard Licensing process -- The market data team must (with the license management sub-organization, if necessary), perform the relevant contract discussions with the relevant market data providers, using the data requests

3. Access Management -- Only authorized users must be able to access licensed data and provisions need to enabled for this in the system and processes

4. Audit Process -- Regular Audits must be run to determine usage (users/systems) that consume different data feeds. This process can also rely on catalogs to keep track of data being provided to or requested by the organization, and the consumers thereof

5. Pro-active issue identification and notification -- While issues cannot be eliminated, provisions need to be built so any delayed, incomplete or incorrect data delivery can be identified and the relevant stakeholders (sources, consumers, and coordination team(s)) can be aware at early stages

6. Back up mechanisms -- Since issues can occur, backup processes need to be built to ensure delivery of critical data sets so business processes can continue unhindered

Architecture -- To support these processes, the following architectural approaches have proven successful:

1. Primary System -- We propose building a primary distributed architecture using standardized communication formats. By defining a standardized communication format, market data can be published to a centralized location/bus where authorized users are able to consume it. This carries the advantages of ensuring seamless communication and shielding from changes on the source side. Consumers can, if needed, covert to proprietary formats for consumption. With advances in technology, we see that standardization of formats is possible, and also that bus architectures (while they can be viewed as single points of failure) provide a mechanism

2. Secondary System -- Customized applications, working on a point-to-point mode, must be built to be used for ad-hoc data sourcing. Data can be sourced either from the primary system via available APIs, or directly from the sources, where possible. These have the advantage of being built over time, based on criticality, and can address potential issues resulting from a centralized mechanism. In certain scenarios, it is possible for specific consumers to choose a point-to-point mechanism as the regular mechanism

3. Monitoring module -- A monitoring module doing all, or parts of, the following must be built:
   1. Health Check -- Via custom tools or product add-ons, this is to obtain the ability to check for availability of systems, and required data, at specified times. The health check can be scheduled to run at specific times of the day, or ad-hoc, to do the necessary checks and produce status reports. This can enable issues to be made known to relevant parties at the onset instead of later

   2. Correctness Checks -- Via custom tools or product add-ons, obtain the ability to validate data to ensure it lies within acceptable ranges. This is a more elaborate and complex check compared to the health check, but can enable identification of data correctness issues upfront. This can be optionally built based on the organizations ability to identify and respond to incorrect data in downstream systems instead of at the source

   3. Health Reporting -- Details of the above can be consolidated into reports, and used to automatically notify IT, Business and market data providers in case of issues, to take further action

   4. Automatic fail-over -- The secondary systems can be setup to be triggered by the monitoring module in case the Health or Correctness checks. This can also be viewed as an optional step depending on approach to Level2 or Level3 support (which can invoke the fail-over processes), and response times of responsible parties (market data providers or system owners) to address data delays/incompleteness/incorrectness or system unavailability

4. Security Module
   1. Access Management -- Setup user data license information via a UI or via uploads, to build a mapping of users to data sets they are licensed to

   2. Single access point - Once the access management mapping is setup, this layer must always be invoked when users trying accessing data. While this does not prevent uses from sharing data via email etc, it will prevent unauthorized access from the source, and also serve as means of auditing who is accessing what data

Summary

We have seen these steps help improve SLAs, identify issues upfront, increase productivity, and resulting in smoother business processes. For e.g, at a European Energy Trading company, implementation of some aspects resulted in the following:

1. Automated health check reports reduced the manual data validation effort by 1 hour everyday

2. Automated notification reduced liaison time between business, support, and development by approximately 1 hour on an average, per incident

3. Access management has helped identify pockets of unauthorized usage

4. Development of backup systems has reduced lead times for data provision, in the absence of primary system, by 2-3 hours (including time for checks, bringing primary system online etc)

We have seen advantages in following the architecture recommended above, and believe it can be adopted (with the relevant customization and in stages) successfully for a robust market data delivery process.