Project Name
How Ksolves Engineered a Scalable, Embedded Analytics Platform for a Global EdTech Company
A leading EdTech company with a global presence wanted to improve the learning experience using data-driven insights. Their platform helps universities and institutions deliver, manage, and analyze educational content for students, teachers, and administrators. As their user base grew, it became important to provide built-in, easy-to-use analytics within the platform.
The engineering team needed to give instructors and admins access to useful insights directly within the platform without using third-party tools or making them log in separately. To do this, they had to solve several technical challenges:
- Different types of data sources: Structured data was stored in PostgreSQL (RDS), while user activity and other complex data were stored in MongoDB, often in deeply nested formats.
- Complex data preparation: Data from MongoDB had to be cleaned, flattened, and enriched before it could be used for analytics.
- Embedded, login-free UX: Dashboards had to be fully embedded into the app so users could view them without leaving the platform or logging into QuickSight.
- Fully automated ET: The system had to refresh data daily automatically and reliably, with no manual effort, using a fault-tolerant process.
- Disparate Data Sources: Structured data resided in PostgreSQL (RDS), while deeply nested behavioral data lived in MongoDB Atlas.
Ksolves resolves the challenges by delivering the following solution:
- Orchestrated Serverless ETL: AWS Step Functions orchestrated the entire ETL control flow, seamlessly coordinating multiple data sources and processes. It initiated DMS tasks to handle both full and CDC (Change Data Capture) loads from PostgreSQL. In parallel, it triggered Lambda functions built with PyMongo and Node.js to extract, reshape, and flatten deeply nested MongoDB documents. Finally, Step Functions managed the downstream processing by invoking Lambda-based SQL execution to create and clean up views in Amazon Redshift, ensuring a smooth, automated pipeline from source to warehouse.
- Data Normalization: The MongoDB data featured multi-level nesting, arrays, and embedded sub-documents, such as video interactions and quiz attempts. AWS Lambda functions extracted and transformed this complex structure into domain-driven relational tables, including engagement_events, session_view_summary, and poll_response_aggregate.
- Embedded QuickSight Dashboards: Dashboards were built using Amazon QuickSight with SPICE for fast, in-memory rendering. When users authenticated through the Spring Boot application, they were programmatically registered with QuickSight if not already provisioned. Secure, token-based embed URLs were then generated using generateEmbedUrlForRegisteredUser, enabling seamless dashboard integration within the application via iframe, eliminating the need for users to directly interact with or even be aware of QuickSight.
- Row-Level Security (RLS): (RLS) policies in Amazon QuickSight were implemented to filter data views based on user roles and associated content identifiers. This ensured that instructors could only access data relevant to their own content and classes. To support a multi-tenant architecture, QuickSight namespaces and IAM policies were used to isolate and secure data across different user groups.
- User Experience: Delivered a seamless analytics experience, embedded directly within the application—no external logins or training required.
- Insight Accessibility: Empowered content creators to effortlessly monitor engagement, drop-offs, and interaction trends.
- Automation & Reliability: Achieved a fully automated, resilient ETL pipeline running daily via AWS Step Functions with zero manual effort.
- Data Agility: Consolidated disparate SQL and NoSQL sources into a centralized, analytics-ready Redshift warehouse.
- Performance: QuickSight dashboards backed by SPICE deliver sub-second load times with daily refreshed views.
- Security & Governance: Enforced Row-Level Security (RLS) to ensure compliant, role-based access to contextual data.
- Scalability: Serverless architecture supports effortless scaling from hundreds to thousands of concurrent users.
- Engineering Velocity: Leveraged event-driven Lambdas and low-code QuickSight embedding to accelerate time-to-insight.
By adopting a cloud-native, event-driven, and BI-embedded architecture, the engineering team turned fragmented datasets into seamless, actionable insights delivered directly within the product interface. No context switching, no login barriers, and no operational burden. This approach highlights how serverless ETL pipelines, secure dashboard embedding, and QuickSight’s row-level security (RLS) can enable product teams to deliver deeply integrated analytics with the scalability, security, and agility essential for modern SaaS applications.
Looking to Build Embedded Analytics That Scale?
