Project Name
ML Training Job Duration Was a Scheduling Unknown, A joblib Meta-Model Fixed It at 90% Prediction Accuracy
A mid-to-large enterprise running high volumes of ML training jobs had no mechanism to predict job duration from metadata before submission. Resources were allocated by estimation, short high-priority jobs queued behind long unpredictable runs, and cluster capacity was over-provisioned to absorb uncertainty. Applying its AI-First approach, Ksolves built a regression meta-model trained on historical execution telemetry, serialised with joblib, and deployed as a pre-submission prediction API delivering 90% accuracy.
- No Visibility Into Training Job Duration Before Submission: Duration was unknown until execution completed, making all pre-submission resource planning estimation rather than data-driven scheduling.
- Cluster Resource Allocation Based on Intuition: Without duration predictions, pre-allocation relied on worst-case estimates, causing systematic over-provisioning when jobs ran short and under-provisioning when they ran long.
- High-Priority Short Jobs Queued Behind Long Runs: Queues were ordered by arrival time, not predicted duration or priority. Short high-priority jobs sat idle behind multi-hour runs with no promotion mechanism.
- Capacity Planning Built on Worst-Case Assumptions: Capacity planning defaulted to worst-case provisioning, increasing infrastructure cost without a commensurate improvement in scheduling performance.
- No Feedback Loop From Historical Execution Data: Rich historical execution data existed but was not used to inform future scheduling. Each job submission was evaluated in isolation, ignoring the pattern information in the record.
- SLA Commitments Difficult Without Duration Estimates: Duration uncertainty propagated directly into schedule confidence, making model delivery SLAs inherently imprecise and creating downstream planning uncertainty.
Ksolves trained a regression meta-model on historical job execution telemetry to predict training job duration from metadata features before submission. Serialised with joblib and deployed as a prediction API. The governing principle: turn job duration from an operational unknown into a reliable scheduling input.
- Historical Job Telemetry Collection and Feature Engineering: Telemetry pipeline captures metadata profile and actual runtime for every completed training job. Feature engineering layer normalises, encodes, and weights raw metadata by correlation with execution time variation into structured feature vectors.
- Regression Meta-Model Training: Regression model trained on labelled execution telemetry, learning non-linear relationships between metadata feature combinations (dataset dimensions, feature count, model type, hardware configuration) and actual job runtimes.
- joblib Serialisation for Production Deployment: Trained meta-model serialised with joblib, producing a compact, version-controlled artefact deployable with minimal infrastructure overhead, loaded at sub-millisecond latency, and updated on retraining cycles without disrupting the prediction service.
- Pre-Submission Duration Prediction API: Serialised meta-model exposed as a prediction endpoint returning predicted duration with confidence interval - queried by ML Engineers before submission and by scheduling systems for priority-aware job sequencing.
- Continuous Model Improvement: Meta-model periodically retrained as new job completions add labelled examples, improving accuracy as workload characteristics and dataset distributions evolve.
Technology Stack
| Category | Technology |
|---|---|
| AI / ML | Regression Meta-Model (joblib) |
| MLOps | Historical Job Telemetry Collection Layer |
| Processing | Metadata Feature Engineering Pipeline |
| Platform | Pre-Submission Duration Prediction API |
| Architecture | Continuous Model Retraining on New Telemetry |
| Methodology | Metadata-to-Runtime Correlation Analysis |
- 90% Prediction Accuracy - Duration From Unknown to Operational Input: Meta-model delivers 90% prediction accuracy from metadata features alone, making duration a reliable scheduling input before the job consumes any cluster resource.
- Evidence-Based Resource Allocation Replacing Over-Provisioning: Duration predictions with confidence intervals enable right-sized allocation per job - reducing waste for predictably short jobs while ensuring capacity for long-duration runs.
- Priority-Aware Job Sequencing From Pre-Submission Intelligence: Scheduling systems and ML Engineers can now sequence jobs by business priority and predicted duration, clearing short high-priority runs ahead of long low-priority ones without post-submission intervention.
- Credible SLA Commitments on Model Delivery: 90% accurate duration predictions provide the quantified estimates needed for credible model delivery SLAs, with confidence intervals communicating schedule risk explicitly.
- Historical Execution Knowledge Applied to Every Future Scheduling Decision: The meta-model encodes the organisation's complete execution history into a reusable prediction capability that improves with every completed job that extends the training dataset.
“Before this, every job submission was a guess on how long it would take. Now we query the model with the metadata before we submit, and we get a duration estimate we can actually plan around. The 90% accuracy means we have stopped over-provisioning our cluster just to absorb uncertainty.”
-ML Engineering Lead or Data Science Manager.
An MLOps platform where training job duration was unknown until completion, forcing cluster over-provisioning and imprecise SLA commitments, was transformed through Ksolves AI/ML consulting services. A regression meta-model serialised with joblib now delivers 90% accurate duration predictions from job metadata before submission. Resources allocated by evidence. Jobs sequenced by priority and duration. SLA commitments backed by quantified estimates. Every completed job improves future predictions. The meta-model turns accumulated execution history into a permanent scheduling intelligence layer.
Still Submitting Training Jobs Without Knowing How Long They Will Take?
