Project Name
From Expensive LLMs to Lean AI: How Ksolves Cuts Model Inference Costs by 50% with SLMs and AI FinOps
A North American supply-chain analytics provider processes bills of lading, commercial invoices, customs declarations and carrier rate sheets for shippers and freight forwarders, running on Apache Spark and Apache Airflow with high daily document volume. The company had rebuilt its extraction, classification, and data-quality pipelines on Generative AI and planned several new AI-powered products. But LLM inference costs were rising in step with document volume, eroding margins precisely when the product roadmap depended on using more AI, not less. Leadership paused all new AI initiatives until costs could be explained and controlled.
- Rapidly Rising Inference Spend: AI inference costs grew month-over-month, outpacing revenue growth and steadily eroding gross margin with no clear visibility into drivers.
- One Premium Model for Every Task: Every call, from simple document classification to complex multi-page extraction, went to the same frontier model at top-tier rates, regardless of actual task complexity.
- No Cost Attribution: AI spend arrived as a single monthly invoice with no link to teams, pipelines, customers or use cases, making it impossible to identify where optimization should start.
- Repeated Inference for Near-Duplicates: A large share of documents were recurring carrier templates and resubmitted invoices, yet each triggered a fresh full-price inference call.
- Latency Affecting Customer Commitments: Slow response times and provider rate limits stretched overnight batch windows, putting next-morning deliveries at risk during peak season.
- Data Residency Pressure: New customer contracts required commercial and customs data to stay in the client's cloud environment, which third-party APIs could not guarantee.
Ksolves applied one guiding principle: every inference request should go to the smallest model that meets its accuracy requirement, and every dollar should be traceable to a business owner.
Each new model first ran in shadow mode alongside the existing frontier model, and traffic was shifted over one task at a time only after it matched or beat the frontier model on benchmark checks, with an instant fallback if quality ever dropped.
- AI Cost Governance with Budget Guardrails: Deployed LiteLLM as the single entry point for all pipeline traffic, enforcing per-team budgets, rate limits, and spend alerts.
- Fine-Tuned Small Language Models with Multi-LoRA Serving: Built task-specific LoRA adapters on Llama 3.1 8B using labelled historical documents, all served from one vLLM deployment with confidence-based LLM model routing to the frontier model for complex cases.
- Unit-Cost Attribution: Every call is tagged in Langfuse by team, pipeline, customer, and task, and combined with OpenCost GPU allocation data into a shared cost-per-document dashboard that engineering and finance review regularly.
- Semantic Caching: A Redis vector cache recognises near-duplicate documents and returns previously validated results instead of running inference again.
- Self-Hosted LLM Inference: The SLM pool runs on Amazon EKS with autoscaling GPU nodes managed by Karpenter, removing dependence on provider rate limits.
- In-VPC Processing by Default: SLMs run inside the client's private cloud so most documents never leave the environment, with sensitive data redacted before any fallback calls reach external APIs.
Technology Stack
| Category | Technology |
|---|---|
| AI/ML | Llama 3.1 8B with LoRA on vLLM |
| Integration | LiteLLM AI Gateway |
| Processing | Apache Spark and Apache Airflow |
| Database | Redis (Vector Cache) |
| Platform | Langfuse and OpenCost |
| Infrastructure | Amazon EKS with Karpenter |
- Inference Spend Roughly Halved: Total AI inference costs, including GPU infrastructure and fallback usage, cut by about half.
- Stable Cost per Document: Cost per inference optimization. Cost per document dropped significantly and remained stable as document volume continued to grow.
- Faster Batch Processing: Batch windows shortened substantially and next-morning delivery commitments are consistently met.
- Accuracy Maintained and Improved: Domain-tuned adapters matched or exceeded the previous extraction accuracy on the benchmark dataset.
- Full Cost Visibility: Every dollar of AI spend is mapped to a team, pipeline, and customer.
A supply-chain analytics company whose LLM bill was growing faster than revenue with no cost visibility was transformed through Ksolves’ AI FinOps and small language models optimization services. A governed, right-sized inference platform routes most traffic to fine-tuned small language models running inside the client’s own cloud, with cost attribution to every team and pipeline. Inference costs were roughly halved while accuracy and processing speed improved. The in-VPC design supports customer data-residency requirements. With costs under control, the client is launching its planned AI products on the same platform, each with a cost target set from day one.
Is your LLM Bill Growing Faster Than Revenue?
