Mark-IT






     

 

back to Home

 


 

 X-Tables - a White Paper V1.2, 02 May 2001

1. What is x-tables

x-tables is an exciting new product for providing ultra-fast cross-tabulations, statistics and 'export' reports from large SQL/ORACLE type databases. It has been developed by bringing together two technologies -

- Mature set of data-tabulation systems as developed within the Market Research world over the last 30 years. These tabulators are similar to pivot-tables within Excel but with a simpler interface and many more facilities

- Conventional applications and systems that rely on SQL, ORACLE (as well as Access) for managing their data-bases

x-tables has 4 primary goals
- To provide users with a very user-friendly interface for obtaining their reports - one that requires no knowledge of the underlying data-base or variable names. And not restricted to just the key/index'd variables.
- Is able to "understand" any given data-base structure - directly - without input from the d/b programmers
- To provide super-fast processing speeds - in excess of 2 million records/rows per minute on a medium speed PC
- To provide the results in a wide range of formats, print, graph, csv, pdf etc

Why x-tables

All users of conventional databases need to produce reports, which aggregate the data to show summary results in tabular form. They need to do this very quickly - even on large databases. Most of these users have no formal programming knowledge and need something far easier than SQL to specify their request. They also need full flexibility of handling the output - very similar to Excel's features

The Advertising/Market research world has had the benefit of interactive tabulators for over 10 years - based on the very formal structure of "questionnaire data". Many clients are already using this software and it is estimated that well over 100,000 marketing and research executives at the world's largest companies are using these tools. However these users are restricted to the Marketing/Research divisions - x-tables now brings this technology to a much wider audience.

How does it work

There are two steps in the data-preparation phase

1. In order to provide ultra fast speeds and not to adversely affect the performance of the live database, x-tables pulls off a snapshot of the full database in text/ASCII format. It does this automatically - no programming is required and the system extracts from SQL table-definitions, the schema. It understands all the variables or columns (i.e. how they are linked) and what type they are. If required, the user can restrict the number of variables. This extract file is in an industry standard ASCII form supported by XML meta-data that describes the data

2. This extract file is processed and compressed to produce the final analysis file to be resident on a network or Internet. It uses a concept of transposition where each variable/column becomes a mini-data-file in its own right - this can be compressed - often to less than 10% of the full database plus indices.

The user then accesses this data via a Windows based proprietary tabulator - using one of a set of family of programs. It will also work via a regular Internet Browser

What products are in the x-tables family

There are 3 products within the x-tables family

1. Lite for reading Excel/DBF files. This is a very low price (less than $50) demonstration program that provides basic functionality and is designed to read directly any simple and flat XLS or DBF file

2. Standard - for processing SQL/Oracle tables. A medium priced ($30,000) converter that will read linked databases to produce the analysis file. Tabulators cost $800 but dropping to less than $50 for site and network licences

3. Professional - as #2 above, but with a price tag of $80,000. This includes a sub-system to generate additional variables and validate the data using a proprietary programming language - ideally suited for this type of data



Development team

The primary design and development of x-tables system is under the direction of Mark Katz (see http://www.mark-it.co.uk). Mark brings with him over 30 years in the data-analytics world - including over 5 years developing using web-based systems. During that time, he developed a full range of internet-based tabulation/report delivery systems (http://www.e-tabs.com) and was responsible building an advanced e-marketplace Exchange system for the financial sector - http://www.interactiveintelligence.co.uk/

Since x-tables takes advantage of other packages already in the marketplace, a minimal amount of development was required to provide links between the various sub-system. On-going enhancements are carried out by a team of highly skilled programmers - plus a support team.



More reading

Tim Macer services at http://www.macer.co.uk/home.html - an independent web-site dedicated to the software available for the market research world. Although most of it relates to the data-collection (paper, phone and web-based), it contains good references to the most popular cross-tab and report packages.

The main players

Perhaps the main cross-tab packages is SAS http://www.sas.com/products/base/ which already provides much of the functionality of x-tables, however it is a big, complex and expensive piece of software - and not as fast as x-tables.

Data-warehouse software and OLAP technology shares common objectives with x-tables. However it represents a major investment of software, hardware, people and time.

triple-s - the (xml) model for interchange of data encapsulating both the data and meta-data (descriptions of that data). See more information about this evolving open standard at http://www.triple-s.org/

top