Liddle Thoughts

When I am dealing with customers and Partners it never ceases to amaze me how innovative many of  the solutions and products, that have GigaSpaces embedded, are. I thought it would be well worth a quick review of some of these:

ActivePivot: ActivePivot™ by Quartet is a real time object oriented OLAP engine that provides information dashboards from front office to back office including trade blotters, value-at-risk drill down analysis, inventory positions, cash flow and security inventories, credit risk, online risk and hedging analytics in real time. You can view a case study of a combined ActivePivot / GigaSpaces at a major French bank here.

An example of using active Pivot with GigaSpaces could be that market data feeds are written into a space using any of GigaSpaces’ APIs . ActivePivot is able to connect to the core in-memory data and register for  events. When a tick is updated in-memory, GigaSpaces sends a notification to ActivePivot which in turn re-aggregates the relevant cube branch. This means that the aggregated cube is always updated and reflects the latest market state. In addition ActivePivot queries GigaSpaces for the raw data when Excel sends a drill-down request, which enables ActivePivot to deliver a very quick response time.

GigaSystemsBuilder: This uses the model driven development (MDD) capabilities of JeeWiz to generate Grid and Cloud-based systems under Gigasystems technologies.  Comprehensive PoC’s take less than a week and using JeeWiz to generate 90% plus of the final system is an efficient and low cost to utilise these emerging technologies. You can see a screencast of this in action here.

Excel Federated Grid:  This builds on our core Excel integration to enable organisations who use Excel in a trading scenario to scale their use of Excel to prevent bottenecks, to parallelize processing, or to automate algorithmic trading that uses Excel. You can read more about this in a prior blog post. To learn more about the GigSpaces/Microsoft Excel Integration and its applicability to finance have a read of this Microsoft written article.

Contact Centre: Nortel’s Contact Center application portfolio provides a single interface for blending inbound/outbound voice, e-mail, web chat, and IM customer interactions, businesses can collaborate with their customers through the device and method of choice extending a superior customer experience. 

Orbyte Solutions: Orbyte Solutions provide a Trading Application portfolio that works in-house or on public clouds such as Amazon EC2 and is powered by GigaSpaces Cloud Tools.  The applications include Foreign Exchange Trading, Spread-betting and encompass charting and widget features using their own innovative rich clients. You can see my prior blog posts on this here and here.

Hyper Rig:  Hyper Rig is a technical framework for Risk Management that can be added to your existing set up, complement it and enhance it. As such, the architecture has been designed to include features such as orchestration, exceptions management and audit, messaging, integrated security, automatic failover, monitoring, load-balancing, virtual computation-grid and data quality management. 

SmartStream: GigaSpaces XAP is embedded within SmartStream’s next generation Transaction Lifecycle Management technology suite. Using GigaSpaces’s proven technology, SmartStream can deliver on demand and predictable scalability to support clients’ high-performance processing requirements. You can read more about the integration from SmartStream’s site.

Online Gaming: I recently blogged about an online gaming application that is available on Facebook but whose engine is a commercial engine. It is the first totally enabled online Cloud gaming network and is powered by GigaSpaces Cloud Tools.

These are just a selection of application that use /embed GigaSpaces that I have touched. There are many  thousands more that are used in all different parts of the world, either on premise or on the cloud.

Tags: ActivePivot, Cloud, Cloud Computing, Contact Centre, Excel, GigaSpaces, HyperRig, Microsoft, Orbyte, SmartStream TLM, XAP

If you are a Mac user and want to take advantage of some of the API tools for Salesforce.com then have a look at:

Maildrop - Great utility that allows you to post emails to Accounts or Support cases from Entourage or the default Mac Mail App. You can download it here.

SFDCFuse - This Mounts your SalesForce Doc Volume as a repository. You can download it here. (you also need MacFuse pre-installed)

Quicksilver Plugin - If you use QuickSilver then enough said. You can download it here.

SF3 - This enables a two way sync between your Address Book and Calendar with Salesforce. You can download it here.

Trapdoor - Store your SalesForce logins securely in keychains. Download it here.

Firefox Search Plugin: This plug-in adds the ability to run a salesforce.com advanced search, or a supportforce solution search in your salesforce.com account, right from your Firefox search bar.

AppBar: Written by 3 salesforce employees the app bar add shortcuts to Firefox as a plug-in to interact with salesforce.

Mass Update Anything - Mass Update Anything, as the name implies, updates fields in bulk, based on a SOQL query. In addition, you can choose to delete fields in bulk as well. It’s a great little tool that does one job very well, and will save you from CSV hell.

Tags: Apple, MacFuse, Maildrop, Quicksilver, Salesforce, Salesforce API, SF3, SFDCFuse, Trapdoor

One of the really strong points of the GigaSpaces technology is the strength of the client interoperability. As well as being fully interoperable with .Net and C++ (and if you are interested in these technologies I would suggest you check out the C++ article here and the .Net articles here and here) GigaSpaces has integration with Microsoft Excel.

This enables organisations who use Excel in a trading scenario to scale their use of Excel to prevent bottenecks, to parallelize processing, or to automate algorithmic trading that uses Excel.

When using Excel for Algorithmic trading  or for headless calculations a data feed can be input into the in-memory GigaSpaces data space which triggers phases of calculations in distinct unattended Excel spreadsheets. This in turn generates values to a data cache that can be picked up as events and displayed within an attended Excel sheet.

Visually we can view this as below:

The diagram shows:

·      An External Data Feed Handler 

·      A number of Excel Compute Support Processing Units (PUs) which are managed by the GigaSpaces Service Grid. Each PU contains worker components of two types:

1.     workers that select and execute Excel compute tasks.

2.     workers that select and execute manager tasks.

·       An attended Excel sheet that has an add-in that enables it to listen to the GigaSpaces data cache and display value changes are a result of data changes in the cache. Both User-Defined Function (UDF) and Real-Time Data (RTD) add-in approaches can be implemented.

The way this work is as follows:

1.     The External Data Feed Handler reads data from a file and uses it to create an initial data value and a task to manage the overall computation stages. The data value is written to the cache and the manager task is injected using a imple task submission API. A Compute Fabric worker executes the injected manager task.

2.     Execution of the manager task picks up the initial data value and spawns a set of Excel compute tasks. These compute tasks execute a number of parallel calculations using the first sheet. The manager task waits all spawned tasks to complete.

3.     The manager task collates the results from spawned tasks and uses them as input to a calculation on a second sheet. The manager task spawns another Excel compute task to perform this second calculation.

4.     The manager task converts the result of the second calculation into a result data value that it writes back to the GigaSpaces data cache.

5.     UDF/RTD add-in to the interactive Excel sheet sees the data change event in the GigaSpaces data cache, obtains the data values and updates the sheet.

Key to accomplish this is an Excel Compute Support Processing Unit,  which is managed by the GigaSpaces Service Grid. The diagram below illustrates the components of this processing unit.

Inside each processing unit (PU) instance a FederatedWorkerFactory communicates with other FederatedWorkerFactory instances in the grid to form a federated fabric that hosts a number of open Excel workbook instances., The fabric can be instructed to host a number of instances of the same Excel workbook to provide redundancy and failover. The fabric ensures that the Excel workbook instances are distributed as evenly as possible between the PUs available to host them.

When the FederatedWorkerFactory receives an ownership request for a given workbook it creates a worker to pick up and execute ExcelComputeTasks that are tagged with the name of the workbook., ExcelComputeTasks delegate calculation to a resident Excel Compute Manager, If additional PUs are added or removed from the fabric then the fabric automatically and dynamically rebalances the Excel workbook instances to maintain an optimum balance., Additional workbooks can be added (or removed) from the fabric dynamically whilst the system is running and the number of instances of already hosted workbooks can be raised or lowered.

The Excel Compute Manager within the PU manages a configurable pool of “headless” Excel processes. Each Excel process is lightweight and can manage one or more open workbooks.

The fabric ensures that the optimum number of Excel processes and open workbooks are  running on each GigaSpaces Service Grid node. There is a significant relative performance overhead to loading workbooks in Excel. Workbooks are therefore opened lazily on first request, but then remain open, ensuring that repeat calculation requests to the same workbook are optimized for performance.

If a given PU manages more than one open workbook, it will perform calculations for each workbook concurrently, whilst ensuring that each Excel instance in the pool is managed in a thread-safe manager. This enables good scalability of concurrent Excel compute both within a single node, and across the nodes available in the grid. The configurable Excel pool size determines the upper bound of the number of concurrent workbook calculations that can be performed by a single PU.

If an Excel instance is damaged, dies, or is killed, the fabric ensures that it is removed from the pool of available Excel instances and, if necessary a new instance is spawned. Any in-flight calculation request is automatically retried once the workbook has been re-opened with another Excel instance.

The Excel PU, tasks and fabric specific to this solution build upon the GigaSpaces Excel integration and were developed by a partner to give a solution oriented approach to working with Excel in a Grid. If you would like to know more about this then please feel free to contact me.

Tags: Algorithmic Trading, Excel, GigaSpaces, Headless Excel, Microsoft, Scaling Excel

The latest cloud enabled application using the GigaSpaces EC2 cloud tools went live on Facebook yesterday. This is a Facebook application built by Advanced Gaming Labs. The application is the first completely multi-player global Casino Application that runs exclusively on the Cloud and was built on the cloud from the ground up using GigaSpaces.





If you would like to play the application you can do this on Facebook here. 


There are lots of firsts for this application and you’ll be hearing lots more about in the forthcoming weeks as other news about the application and other launches using the engine become available.

Tags: Amazon, Blackjack, Cloud, Cloud Computing, EC2, Gambliing, Gaming, GigaSpaces

In my prior post I posted the slides from the  London Amazon Web Services event. The video of the Orbyte presentation about their GigaSpaces built Trading solution is now available and can be viewed below or from this link.

Cédric Roll - ORbyte Solutions - AWS Start-up Event - 28-04-2009 from Skills Matter on Vimeo.

Tags: Amazon, Cloud Computing, EC2, GigaSpaces, Orbyte, Trading in the Cloud

I will be speaking at the Online gaming high scalability special interest group in London on 9th July. Registration for this event is now open, but be quick because there is already only 44 tickets left.

My session is entitled “The space base gaming advantage” and will explore, technically, how GigaSpaces Space Based Architecture brings technical benefits that give a specific advantage to gaming application. We will delve into what these advantages are, what they are an alternative to, and how they are architected.

There are lots of interesting sessions and it should be a really interesting day to understand topics such as cloud security, transaction processing in the cloud, scaling in the cloud, and some real world case studies of gambling and betting applications that have been deployed in the cloud.

Tags: Cloud Computing, Event, London, online gaming, scalability, Space based Architecture

I’m currently at the Amazon Web Service Startup event in the UK. It’s been a good day and there have been a lot of good presentations. GigaSpaces had a cloud client, Orbyte, referenced in a prior post presenting their “Trading in the Clouds” solution. You can see the presentation that they presented below.
 

Tags: Amazon, Cloud Computing, EC2, GigaSpaces, Orbyte, Trading

Ross Clark posted a highly astute article in today’s Daily Express entitled ‘Labour has made Britain a land fit for freeloaders’. How sad that this is so true.

“It isn’t work that pays in Britain any more: it is whining and cheating.”

At the top of this list are the ministers and government themselves. Gordon Brown is so scared of a ruling coming into play for MP’s to open their expenses to show us what they have been spending *our* money on he wants to change the rules so that MP’s are effectively paid a bonus for turning up for work. You couldn’t make it up……

Tags: Politics

One of the GigaSpaces customers I have been working with recently is called Orbyte Solutions. The team behind Orbyte have many years experience developing FX Trading and Spread betting platforms for large banks and financial organisations. The Retail CFD, FX and SpreadBetting is a highly competitive market which operates on tight spreads and low commissions. Clients are sophisticated and expect product innovation, advanced trading features and above all being able to trade quickly in any conditions.

The key drivers for these businesses to remain competitive can be summarised as:

- Consistent, reliable and efficient service delivery under any market conditions

- Product innovation and time to market

- Cost effectiveness

Historically the platforms which support these businesses have been based on a tier architecture (at best 3-tier at worst 2-tier). They are often heavily reliant on relational databases. Sometimes they implement some level of data caching and messaging. Orbyte observed that most companies in this sector have developed these caching and messaging components internally. What might have given them a technical edge 10 years ago have now become a burden with a high cost of ownership. Using GigaSpaces Orbyte have built solutions that allow Retail CFD, FX and SpreadBetting businesses can get more “bang for their buck” by moving away from the traditional tier-based model and internally developed “plumbing” components (messaging, caching, and deployment).

As GigaSpaces makes it easy to build systems that work as easily on the Cloud as they do internally Orbyte have been able put a demo of their system on the Amazon EC2 Cloud.

In terms of EC2 instances - this is currently running on

**Until 5PM GMT 23rd April the demo system will be live on the EC2 Cloud for you to have a sneak peak**.

You can use the connection details below.

Tags: Amazon, Cloud Computing, EC2, FX Exchange, GigaSpaces, Spread Betting, Trading, Use Case

One of the interesting things about working with any Middleware solution (and of course GigaSpaces is unique as the Middleware is virtualised) is the interesting projects that come up with and the way in which the technology is applied. One of the projects that is coming to fruition is one in which GigaSpaces is being used to develop a 52 country data hub for a mobile Sales solution.


Phones and handheld devices today are much more powerful than fully fledged computers were 10 years ago, but when interacting with data and services, similar challenges exist to traditional web applications, namely performance, latency and scale.


The users of the system’s in question are  Sales Representatives in different Countries who use the handheld devices to place orders, schedule deliveries etc of Retail stock. The hand-held devices will also contain accounts, sales and other pertinent information from a back-end  CRM system. The handhelds will operate disconnected from any central system but may connect at intervals during the day over GPRS or Wireless links. An initial data download at start-of-day will provide the information required for business conducted that day. During the day uploads of changes (deltas) might occur, and finally at end of day a full reconciliation and synchronization process will update both the Siebel database and handhelds.


The solution copes with data conflicts, where both Representative’s data and central CRM data have been modified,  by applying specific business rules at a record or field level.


The solution also provides buffering/queueing mechanisms to manage the flows into and out of the CRM system in order to accommodate the volume/bandwidth restrictions imposed by the package, whilst still allowing thousands of representatives to initiate synchronization operations at the same time.


Information extraction from the CRM is implemented using the CRM’s data API and delivery to the CRM is mediated through the CRM Web Services interface.


A mobile solution from this particular CRM vendor already exists which the company in question actually already used, but it did not suit their needs as it was a relatively thick client and it was silo’d over multiple CRM implementations. It was also not loosely coupled.
M/br/>
The new system had to provide isolation of the handheld software from the CRM. Why ? Because the solution needs to be abstracted from the actual Siebel schema implementation to allow for future change, which included the possible substitution of the CRM package with an alternative CRM solution. The solution also needed to allow scalability to many thousands of Sales Representatives without overloading the existing Siebel or back-end systems.


The high level architecture for this system is outlined below:
















GigaSpaces uniquely provided:


- A Modular, task-based  approach to services
- Flexible interfaces for  upload and polling
- Pluggable business  logic for  synchronisation
- Standards-based Web  Services provide  handheld connectivity
- Ability for WAN level sychronisation
- In-Memory Cloud for low latency and fast performance


Given the current penchant for Cloud this type of solution could become much more pervasive for organisations in the future.

Tags: GigaSpaces, Global Cloud, latency, Mobile Data, performance