Recovering Data Without Reprocessing
Through Apache Spark
Objective
To create an algorithm that recovers data from the same point instead of re-processing from scratch.
What’s the need?
Because the earlier process did not allow data retention for intermediate results.
Solution
A multi-step checkpoint algorithm with data recovery options.
Discuss Your Problem With Our Experts!
Connect With ExpertsImplementation Approach
01
Data recovery algorithm
recovers data from its
last executed step
recovers data from its
last executed step
02
One database table to
keep jobs checkpoint log
for each checkpoint
keep jobs checkpoint log
for each checkpoint
03
Storing the current checkpointed data frame in another table & storing table menu name in the log table for each checkpoint. Record the sequence for each checkpoint & task
04
Checking for checkpoints
already recorded in the log
table when the job starts
already recorded in the log
table when the job starts
05
Extracting the last checkpoint
stage when the job fails if the
last point was found to be true
stage when the job fails if the
last point was found to be true
06
Loading the checkpointed data frame
instead of executing the job till the
next stage starts which is not checkpointed
Discuss Your Idea!
Request a call back