Incremental/Delta-Load-in-Hive
Delta load in hive is a major problem faced by industries and only few approaches were there to perform this in hive. One of those approch will give more optimal result with no performance issues. You will find two files inside this repository (click on the link given at the bottom)- incremental.sh - A Shell script that contains relevant command to automate the process. part.hql - A hive query file that contains commands to perform dynamic partitioning. Steps to perform incremental append- Create a partitioned table in hive. Example: CREATE EXTERNAL TABLE base_table ( ID int, Name string, Contact_No bigint, Time_stamp string ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION '/user/hive/base_table'; Load the data from your database initially using any tool (like sqoop) in that particular directory('/user/hive/base_table'). Give permission to user/hive directory using following