Introduction –
Data Engineering is the aspect of data science where we deal with data collection and analysis. This profile provides you the opportunity to create an infrastructure where you can handle huge amounts of data in an optimized fashion. The transfer of data and query execution should all be optimized to handle the huge amount of data with reduced IO. This also takes into account the encryption/decryption and compression. We have various tools available in market to make this possible. We will go through all one by one. I will include NoSQL DBs and BigData and the Engineering required to make data available for the analysts and Data Scientist.
What we need to be expert in these are –
- System Architecture
- Programming
- Database Design and Configuration
- Data Models
- Relational and Dimensional Models
- Data Flow
From cars to E commerce, we need data for operation of each business. So, lets learn variety of tools and technologies to implement our data pipelines.