Scalable Serverless Data Platform with Audio Integration
A cloud-native solution for processing massive CSV datasets with advanced audio playback and filtering features — built for speed, scalability, and zero-database overhead.
Project Overview
This project is a modern data processing platform designed to handle large-scale CSV files stored in AWS S3. It enables organizations to:
• Process and merge multiple CSV files dynamically
• Integrate and play corresponding audio files per record
• Filter data in real time by date, name, phone number, or ID
• Manage access through secure, role-based authentication
All of this is achieved using a fully serverless AWS backend and a modern, lightweight frontend architecture.
Complex CSV Processing without Databases
The system is built to process thousands of records per day, with each row in a CSV containing:
• Metadata: phone number, ID, timestamps, etc.
• S3 path to an associated audio file
• References to recording dates and tags
Instead of using traditional databases, the platform reads directly from S3, merges files dynamically, and delivers a unified data experience.
Advanced Filtering and Search
To maintain performance across large datasets, the platform uses:
Date-Based Prefix Filtering
Filters S3 objects using YYYYMMDD format prefixes (e.g., 20250225), allowing quick lookup of files by date.
Real-Time Search
Users can search across all files for specific names, phone numbers, or time ranges — with consistently fast results even across large volumes of data.
Cloud-Native Backend Architecture
Built entirely on AWS using a serverless design for scalability and cost-efficiency:
Python Lambda Functions
Handles all data processing logic, including CSV parsing, merging, and audio URL generation. Lambda memory and time limits are optimized for high throughput.
API Gateway
Provides REST endpoints for frontend-to-backend communication with rate limiting, error handling, and structured responses.
AWS Cognito Authentication
Implements secure authentication with multi-role support:
• Admins: full access
• Playback users: stream audio, view data
• View-only users: read access
Integrated Audio Playback
Each CSV row includes a reference to a stored audio file. Audio is playable directly from the platform using a built-in player with:
• Instant streaming from S3
• Inline playback controls in each data row
• Zero context switching — data and media in the same interface
Frontend & Deployment
React + Vite + Tailwind CSS
A modern frontend built with:
• Fast loading using Vite
• Responsive UI with Tailwind CSS
• Real-time interactivity and seamless UX
AWS S3 + CloudFront
Deployed as a static site on AWS S3 with global distribution via CloudFront:
• High performance worldwide • TLS/SSL encryption by default • Scales instantly to handle traffic spikes
Data Visualization & UX
The platform uses custom data tables designed for usability at scale:
• Pagination, sorting, and virtual scrolling
• Real-time searching across thousands of rows
• Seamless integration of audio with each record
Role-Based Access Control
The permission model supports multiple user types with precise control:
Full Access Users
Can view all data, download records, and stream audio
Playback Users
Can view and stream audio but cannot download
View-Only Users
Can view the data without any interaction capabilities
This structure ensures security, privacy, and operational efficiency across teams.
Performance & Cost Optimization
• Resolved early bottlenecks by optimizing Lambda memory and execution strategy
• Replaced traditional databases with direct S3-based architecture
• Reduced infrastructure costs while improving scalability
Key Innovations
• Fully serverless backend architecture
• Prefix-based S3 filtering for time-based data queries
• Real-time search without traditional indexing
• Seamless integration of media into tabular data views
• Multi-role access control using AWS Cognito
• High-performance static frontend deployment
Business Impact
This platform enables organizations to manage large volumes of structured and unstructured data (audio + metadata) without relying on costly infrastructure. By combining performance, scalability, and usability, the system unlocks new possibilities in operational data handling and reporting.