Objective

To create an algorithm that recovers data from the same point instead of re-processing from scratch.

overview-img
what's-the-need-img

What’s the need?

Because the earlier process did not allow data retention for intermediate results.

Solution

A multi-step checkpoint algorithm with data recovery options.

solution-img

Discuss Your Problem With Our Experts!

Connect With Experts

Implementation Approach

01
Data recovery algorithm
recovers data from its
last executed step
02
One database table to
keep jobs checkpoint log
for each checkpoint
03
Storing the current checkpointed data frame in another table & storing table menu name in the log table for each checkpoint. Record the sequence for each checkpoint & task
04
Checking for checkpoints
already recorded in the log
table when the job starts
05
Extracting the last checkpoint
stage when the job fails if the
last point was found to be true
06
Loading the checkpointed data frame instead of executing the job till the next stage starts which is not checkpointed

Discuss Your Idea!

statue of liberty
USA
+1 (646) 203-1075
india gate
India
Noida | Indore | Pune | Ahmedabad
Request a call back