Data Flow in InSights

<< Click to Display Table of Contents >>

Navigation:  Concepts >

Data Flow in InSights

It is important to understand how Data flows through the InSights Into Data Application - as it is what the application is all about.

 

Data enters the application from either a database SQL query or a CSV/Excel file import.  Currently these are the two ways that data can be brought into the application. (we have plans on being able to import from other sources in future releases).  Once there, it is stored in InMemory tables, which in turn are used to power all the visualizations and analytics.  

 

So - when you run a query (or import a file) - the data is loaded into an in-memory table.  You can have multiple in-memory tables there at any time.  And as the program is a 64 bit application - it can hold a large amount of data in memory if it needs to (Approximately 16 Exobytes - in case you were wondering, but as most computers companies use max out at 64 or 128 GB or RAM, you shouldn't have to worry about this limit :) ).  Note that each in-memory datatable can hold just over 16 Million rows.

 

Once all the data is in the program (the retrieval of the data from either DB or file is the slowest part), then you are ready to start working with it.  Once the data is loaded into the application, it does not matter where it came from.  InSights Into Data treats any memory table exactly the same.  This means that all the other program features, such as dashboards, reports, etc. will all work with them.  

 

The following diagram shows the rough data flow into and around the <%PRODNAME% application.  

 

InsightsInMemoryTableOverview

 

 

As the various program features reference the in-memory tables, the names of these tables become very important.  When you have Reports, Dashboards, etc. dealing with the data, they used the name of the In Memory Table.  So when you define a Query, part of that definition is to assign the Result Table Name.  As noted, this name is very important - as it is how every other feature interacts with the data.  You have to be careful to not have two Queries with a Result Table Name.  This could end up confusing the Reports, Dashboards, etc.   To add one more twist to the naming issue - if you have an existing In Memory table with the same name as the SQL Query or File Import Result Table Name that is about to be run, the new In Memory table will have a number appended to its name until a unique name is found.  This means that if the Dashboards or Reports need a Result Table Name with a certain name - it will be pointing to the wrong table...

 

 

The following screen capture shows where the user can enter the Result Table Name:

Query_TableNames