Your Data Isn’t That Big

I’m going to tell you an anecdote. At IFARHU, we manage about 1.5 TB of data regarding student payments. With this data, it is possible to develop information that allows us, within a context, to generate knowledge and make decisions or simply understand the execution. This is a procedure known as “Knowledge Generation”.

IFARHU, like any other technologically mature organization, has an untapped gold mine. We have tens of millions of records that are not used for the generation of knowledge or for decision making, something that is commonly done through the implementation of Business Intelligence (BI) in some aspect.

BI is a very broad term. BI is “an umbrella of terms that include applications, infrastructures, tools and best practices that allow access and analysis of information to improve and optimize decisions and performance” ^fn-1. These BI systems must include 4 components

1. An information warehouse (a Data warehouse (DW)), with its source information.
2. Business Analytics (the why).
3. Business Metrics Management (Business Performance Metrics (BPM)).
4. User Interface, such as information boards (Dashboards).

It is important to understand that every Business Intelligence system is based on the solution of a reason, the “why.”

#.Big Data

Big Data is a term, rather than a technology as it is erroneously known. Authors describe it as the appearance of the following terms (known as the Vs of Big Data):

• Volume: Due to the large amount of data.
• Variety: Due to the different way in which these data arise and their variable form.
• Speed: Due to the exponential way in which this data grows and the (fast) speed at which it needs to be accessed.
• Veracity: By having complete and correct data.
• Variability: Due to the variation of the data over time.
• Visualization: Due to the need to “digest” the data.
• Value: For being the core reason for Big Data, for the generation of value, of “knowledge”.

Now, how big is Big Data?

#.The mistake: thinking big

As engineers, we tend to rely on data… I mean, the mass. Our first decision was to implement a small Hadoop cluster on small, old machines, taking advantage of one of the advantages of Hadoop to run on almost any hardware.

It is important to understand that Hadoop, more than a system, is an ecosystem that is composed of several systems internally (HDFS, YARN, etc.).

After the implementation of the cluster, it was necessary to make a kind of Data warehouse with Hadoop and its respective ETL procedures (Extract Transform Load). To do this:- Sqoop was implemented for extraction from Relational Databases.

• Implemented Hive as Data warehouse and to use HiveQL instead of MapReduce.
• ETL scripts were implemented using Hive and Python (which can be executed with Hadoop).

After all this and having the clean data to be processed, it was time to test the solution.

To our surprise, the result was not the best. While we are talking about tens of millions of rows on simple hardware, the result of a simple clustering could take several minutes. We even tried using a columnar system (Parquet). The result improved, but perhaps it was too much.

This makes us think at this moment, is our data really big?

#.Everything must begin with understanding the “why”

Do we really understand the problem we are going to solve? The answer in our case and that of many others is “no.”

These problems are not engineering problems, in most cases, but organizational problems or situations. These problems can even be a question like “What are the products that sell the most on the weekends?” or better yet: “What products are most likely to sell next month?“.

In the case of IFARHU, after some analysis, we noticed that it was necessary to understand the population and understand the complete demographics: gender, age, location, etc.

Being a country of around 4 million inhabitants, if we could save 70kbs for each of the inhabitants (taking into consideration that not all of them have IFARHU benefits, even) we could only need 286.72 Gb.

Just doing this small calculation makes us think that really our data was not that big. Even a relational database would do the job.

#.Conclusion and future work

Big Data knowledge and projects must always generate value for the organization and this must be the north. It is important to understand the problem first, and then proceed to see the possible solution.

With the definition of the problem and its understanding, it is possible to diagram possible solutions through technology.

In our case, we will be testing various parallel alternatives, such as using a small server in MongoDB and Spark as a data processing layer. We will use Hadoop when we feel it is totally necessary.