Project Name
Real-Time Faceted Search Addition to a MongoDB Collection With Millions of Records Without a Hardware Upgrade
Our client is a large-enterprise EdTech platform headquartered in North America, serving universities, corporate training teams, and certification bodies. A core feature of their platform is an instructor-facing question bank, a collection of millions of content items that educators can search, filter, and assemble into assessments and courses.
As the question bank scaled, the platform’s MongoDB-backed search implementation, originally built on aggregation pipelines, could no longer keep pace with the performance and functionality expectations of instructors who needed fast, filterable discovery across a deeply tagged content library. The problem was not the database. It was the search approach sitting on top of it.
Millions of records, full collection scans for every facet count, case-sensitive matching that returned different results for the same search term, and an infrastructure bill that grew with content volume rather than with business value.
- Full Collection Scans for Every Facet Count: Every filter request (subject, difficulty, content type, or tag) triggered a MongoDB aggregation pipeline that scanned the entire collection. With millions of records, these scans became increasingly slow, resource-intensive, and difficult to scale.
- Unreliable Case-Insensitive Matching: MongoDB's default text search returned different results for searches like "Biology" and "biology." The lack of consistent case normalization caused instructors to miss relevant content and reduced confidence in search results.
- No True Faceted Navigation: The existing aggregation-based approach supported only basic filtering. Dynamic multi-facet filtering with real-time count updates across all filter options resulted in unacceptable query latency.
- Rising Infrastructure Costs: As the question bank expanded, the MongoDB cluster had to be scaled vertically to handle aggregation workloads. Costs increased with data volume, while search performance saw little improvement.
- Slow Multi-Field Keyword Search: Searches across multiple fields required separate queries and application-level result merging, leading to slower response times and inconsistent relevance.
- No Relevance Ranking: Search results were returned in collection order instead of relevance order, making it harder for instructors to quickly find the most useful content.
Ksolves, an AI-first Big Data consulting services company, implemented MongoDB Atlas Search on the existing question bank collection. We layered a Lucene-backed full-text search and faceting capability directly onto MongoDB without requiring a migration to a separate search service.
- MongoDB Atlas Search Integration: Ksolves implemented MongoDB Atlas Search on the existing question bank, adding Lucene-powered full-text search and faceting without migrating to a separate search platform. The existing data model, write path, and application logic remained unchanged.
- Atlas Search Index with Custom Field Mappings: Created a search index with field-level mappings for titles, tags, subjects, and difficulty, enabling accurate faceting, field-specific relevance boosting, and efficient queries without full collection scans.
- StringFacet for Real-Time Facet Counts: Implemented Atlas Search StringFacet to generate dynamic counts for subject, difficulty, content type, and tags alongside search results in a single query, eliminating expensive aggregation pipelines.
- Case-Normalized Search with Custom Analyzers: Configured custom analyzers with lowercase token filters, ensuring searches like "Biology," "biology," and "BIOLOGY" always return consistent results.
- Multi-Field Relevance Scoring: Replaced application-side result merging with Atlas Search compound queries that search across multiple fields simultaneously and rank results by relevance.
- Autocomplete for Faster Discovery: Enabled Atlas Search autocomplete on titles and tags using edge n-gram tokenization, allowing instructors to find content as they type and improving the overall search experience.
Technology Stack
| Category | Technology |
|---|---|
| Architecture | MongoDB Atlas Search |
| Processing | StringFacet |
| Platform | Custom Analyzers |
| Database | MongoDB |
| Integration | Atlas Search Compound Query |
Measurable outcomes that improved search performance, scalability, and the overall content discovery experience.
- Facet Count Latency Reduced from Seconds to Milliseconds: StringFacet computes all facet counts in a single Atlas Search query, reducing response times from seconds to milliseconds and eliminating full collection scans.
- Consistent Case-Insensitive Search: Custom analyzers with lowercase token filters ensure searches like "Biology" and "biology" return identical results across all indexed fields.
- Multi-Facet Filtering Without Extra Infrastructure: Atlas Search supports dynamic multi-facet filtering and real-time count updates in a single query without requiring additional infrastructure or vertical cluster scaling.
- Reduced Infrastructure Scaling Pressure: By offloading search and faceting to Atlas Search, cluster scaling is driven by write workloads instead of search traffic, helping control long-term infrastructure costs.
As the question bank scaled to millions of records, MongoDB aggregation pipelines could no longer deliver the speed, relevance, or scalability required for an effective content discovery experience. By implementing MongoDB Atlas Search, Ksolves enabled fast full-text search, real-time faceted navigation, consistent case-insensitive matching, and relevance-based ranking without changing the existing data model or application architecture. The result is a scalable, cost-efficient search foundation that can be replicated across any MongoDB-backed platform managing large and growing datasets.
Is Your MongoDB Collection Serving Slow Faceted Search through Aggregation Pipelines that Scan Millions of Records?
