Project Name
Embedded Analytics Platform Using AWS QuickSight, Redshift, and Serverless ETL
The client is a B2B SaaS EdTech company serving universities, colleges, and professional training institutions across North America, Europe, and Asia-Pacific. With over 500 institutional clients and more than 2 million active learners on the platform, they manage high volumes of structured and behavioral data generated across courses, assessments, and live sessions.
Despite capturing rich engagement data, including video drop-offs, quiz attempts, and session behaviors, the platform had no way to surface these insights inside its own application. Instructors were being routed to a separate third-party BI tool, which meant broken workflows, additional logins, and an analytics experience so disconnected from daily use that most instructors simply ignored it.
The organization partnered with Ksolves, an AI-First Company, to design and deliver a fully embedded, role-aware analytics platform built on AWS QuickSight and a serverless ETL pipeline that would bring insights directly into the product interface without a single additional login. The result was a 90% improvement in dashboard load time, full ETL automation, and a 3x increase in analytics feature adoption within 60 days of launch.
- Instructor Blindness Killing Analytics Adoption: Instructors had no visibility into learner engagement without leaving the platform entirely. The friction of switching to a separate BI tool meant most never checked engagement data at all, making the analytics investment effectively invisible to its intended users.
- Behavioral Data Too Complex for Direct Analytics: MongoDB stored deeply nested engagement records, including video interactions, quiz attempts, and session events that could not be queried directly for analytics without significant transformation. Insights sat locked in raw, inaccessible form.
- Multi-Tenant Compliance Risk: The platform served hundreds of institutions simultaneously. The architecture required strict per-instructor data isolation, and any misconfiguration in the analytics layer would represent a compliance failure and a trust breach across the entire client base.
- No Automated Data Pipeline: Data refreshes were manual, inconsistent, and time-consuming. There was no fault-tolerant orchestration framework, meaning stale data regularly reached dashboards, and engineers spent hours each week on pipeline management instead of platform development.
- Orchestrated Serverless ETL: AWS Step Functions were deployed to orchestrate the full ETL pipeline, initiating AWS DMS tasks for both full-load and CDC (Change Data Capture) transfers from PostgreSQL, triggering Lambda functions built with PyMongo and Node.js to extract and flatten deeply nested MongoDB documents, and managing downstream Redshift view creation and cleanup. The result was a fully automated, fault-tolerant pipeline running on a daily schedule with zero manual intervention.
- Data Normalization from MongoDB: AWS Lambda functions transformed multi-level nested structures, including video interactions, quiz attempts, and embedded sub-documents into analytics-ready relational tables, including engagement_events, session_view_summary, and poll_response_aggregate, making previously inaccessible behavioral data fully queryable.
- Embedded QuickSight Dashboards: Dashboards were built in Amazon QuickSight with SPICE for in-memory rendering. When users authenticated through the Spring Boot application, they were programmatically registered with QuickSight and served secure, token-based embed URLs via the generateEmbedUrlForRegisteredUser API. Dashboards rendered inside the application through an iframe, so instructors saw analytics as a native part of the product with no awareness of QuickSight and no separate login required.
- Row-Level Security and Multi-Tenant Isolation: QuickSight RLS policies filtered every data view by user role and content identifier, ensuring instructors could only access data relevant to their own classes. QuickSight Namespaces and IAM Policies enforced full data isolation across all institutional tenants.
Technology Stack
| Layer | Technology |
|---|---|
| Data Sources | PostgreSQL (AWS RDS), MongoDB Atlas |
| Orchestration | AWS Step Functions |
| ETL & Transformation | AWS Lambda (PyMongo, Node.js), AWS DMS (CDC + Full Load) |
| Data Warehouse | Amazon Redshift |
| BI & Visualisation | Amazon QuickSight (SPICE in-memory engine) |
| Embedding Method | generateEmbedUrlForRegisteredUser API, iframe |
| Backend Framework | Spring Boot |
| Security | IAM Policies, QuickSight RLS, Namespace Isolation |
- 90%+ Improvement in Dashboard Load Time: QuickSight SPICE dashboards render in under 1 second, a 90%+ improvement over the 8 to 12 second average of the previous external BI tool.
- 100% ETL Automation: Every pipeline run is fully automated via AWS Step Functions, eliminating an estimated 15+ hours per week of manual data operations effort.
- 3x Increase in Analytics Adoption: Removing the separate BI login increased analytics feature engagement by an estimated 3x within 60 days of launch.
- 2M+ Learner Records Unified: All structured (PostgreSQL) and behavioral (MongoDB) data consolidated in a single Redshift warehouse, enabling cross-source analytics for the first time.
- Zero Data-Access Incidents Post-Launch: RLS policies enforce role-based data isolation across all instructor and admin roles with no security breaches recorded.
- 10x Scalability Achieved: Serverless architecture scaled from 500 to 5,000+ concurrent users with zero infrastructure changes.
“Before this project, our instructors had to leave the platform entirely to check engagement data, and most of them never did. Ksolves built something we did not think was possible: analytics that feel native to the product, with no extra logins and no training needed. The impact on instructor adoption was immediate.”
— VP of Product, Global EdTech Platform (name withheld by request)
The engagement transformed the client’s analytics capability from an ignored external tool into an embedded, role-aware experience that instructors actually use. As an AI-First Company, we helped the team move from disconnected third-party dashboards, manual pipeline management, and near-zero adoption to a serverless ETL architecture, sub-second dashboard performance, and analytics built directly into the product interface.
As the platform expands into new markets and introduces AI-driven personalization, this RLS-secured, serverless foundation is built to scale alongside it without re-engineering the data layer.
For EdTech and SaaS product teams looking to deliver deeply integrated, secure analytics without a standalone BI dependency, Ksolves Big Data Consulting services deliver the architecture and execution that modern platforms demand.
Connect with our experts today or reach us at sales@ksolves.com.
Ready to Embed Analytics Directly Into Your Product?
