Points to remember before computing large data problem

Got the seminar from @shanmuganandhg

1. What is the size of data?
2. What is your ram size?
3. Will the data fit into your ram?
4. So, If you need to run the analysis for the streaming data,everytime will you do the computation for the whole amount of data(sequential logic) or else
storing the computed state into memory(combinational logic) and processing from it for the streaming data?

After took the decision from the fourth question

5.Think about How to be implemented For example, In python(consider main memory,low data size) or postgres(table,large data size)


If you want to do sequential logic you can also think about colocation of the data(grouping the data).This will make the job simpler.


About Navaneethan

mixed feelings...
This entry was posted in General_Tech, Prob_system. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s