Blog Post

Data moves.  A lot.

Intra-company.  Inter-company.  For analysis.  For distribution and sale.  For teams working from home.  For partners, vendors, regulators.  To the cloud.  From the cloud.

Downloading, moving, and uploading data is slow.  Firms address inefficient data movement in two ways:

• By centralizing data in the cloud and providing pointers with access rather than moving the data. For use cases in which data is not needed in other locations, this is a good solution.
• By working to improve bandwidth, limit packet loss, reduce distances – all to speed data transmission. Yet even when transmission is optimized, time to extract and import data is a constraint.

Exponam Origins

Unsatisfied with current solutions to speed data sharing and distribution, we sought a new approach.  Rather than addressing the issue of time to move a single bit of data, we focused on the data itself.  What could we do to decrease the size of the package, and make it more efficient for extract and import?

Today, data is transferred in different formats – csv, json, xml – but these formats all suffer the same fundamental flaws of being large and inefficient for extract and import.  These formats have been used for generations – from a time when data sets were much smaller.  From a time when a few million rows was a tremendous amount of data.

Exponam .BIG

At Exponam, we have created a new data file type – tailor made for today’s data sets.  Our data file, a .BIG file, is highly compressed and is optimized for efficient extract and import.  It can be used to transport, share, and explore hundreds of millions of rows quickly and easily.  In tests, a .BIG file is blazingly fast when transferring ultra-large data sets – transferring in minutes, data that took hours or days in other formats.

 

Accessing Data

Once files have been distributed and shared, we need an easy way to access and explore data. With the Exponam Explorer, a user can instantly open, filter, sort, and find data from within a .BIG file.

It is easy to explore data of any size – hundreds of rows or millions of rows – making this a great solution when data won’t fit in an Excel spreadsheet.

Users no longer need to spend hours migrating data to databases and writing queries – data is available instantly in a spreadsheet.  Alternately, users can query .BIG files directly via JDBC – with the query performance of a database.

 

Data Security

Quickly transferring and accessing data aren’t the only data problems.

The world has a major problem with data security.  We are constantly learning about yet another company which experienced a major data breach.  Not only is data stolen from databases, it is stolen from data files extracted from databases.  Extracted data files are not secure.  And they are everywhere.

So we made .BIG files fully secure – exceeding today’s security demands around information.  And they stay secure.  You never decrypt/decompress .BIG like with other compression options.  Exponam .BIG files are secure at rest, in process, and in transit.

Data files sitting in email, on laptops, and in the cloud are a security risk as long as they exist.  A file which was downloaded or shared one day can be compromised years later.  Exponam .BIG files can be generated with specified durations for file access – from hours to days or years.  Even more, .BIG files can be dynamically controlled – enabling user specific entitlements and access rights.

.BIG files are tamper-proof and their provenance is guaranteed.  The publisher is certifiable and both the file properties and data are unalterable.

 

About Us

At Exponam, we evangelize “Empowerment through Data.”  We have created the Exponam Explorer^TM, Exponam Builder^TM, and Exponam .BIG^TM file format to enable Secure Sharing and Exploring of Data.  Learn more.  Visit us at www.exponam.com.

Download This Article.

Exponam founder, Roger Dunn, walks us through a brief history of computing and data files. He examines the prevalence and deficiencies of CSV files and the reasons for creating the Exponam data solution.