Real Estate Operations Platform

The client owns a real estate business in the United States. They had an old system built with ASP1 and running on a Windows virtual server. They only had access to the virtual server, and they had no idea where the database was or how the application was running.

In their day-to-day lives, they faced so many issues. The app would sometimes freeze all of a sudden, and when it did, there was no way to do anything but restart it. Most of the filtering was not working, and when it did, it took a long time. A few pages were taking 60+ seconds, and some were timing out due to database timeouts.

As the app was very old, the UI was poor and hard to navigate. For all those reasons, the client was receiving regular complaints from their customer.

The client wanted everything to be deployed on AWS. They also wanted to apply maximum security to the application and infrastructure.

They wanted all the old records from the older database to remain exactly as they were. There were more than 50 million records in the database, so migrating the data without any changes or loss was an important requirement.

Another requirement was that everything needed to load very fast. The client was uploading new data via .xlsx files, and every day they uploaded a large number of new records. For the data upload, I had to make sure the data was in the correct format before adding it to the database.

Although the client wanted everything deployed on AWS, there was also a cost constraint. They were open to spending a fair amount on it, but not too much.

There was also a deadline for developing the application. Because the older application was almost unusable, they wanted to start using the new application as soon as possible.

The existing system was also difficult to understand. The client had access only to the virtual server and lacked proper documentation for the application or database.

The first challenge was to back up the database. For me, this was the hardest part of the project because I didn't have any access to the database. But I had admin access to the virtual machine. So I did some intensive research on how to back up the database with admin access to the VM. After doing some research, I found a solution that was a bit risky. So yeah, the solution worked, and I was able to get the backup of the database.

I didn't have the source code of the previous app, which was an ASP1 application, so I had to implement all the features by investigating and exploring the old platform. The client was very helpful throughout the process. They were one of the most helpful clients I have ever worked with.

The client wanted everything on AWS. I considered using PostgreSQL for the project, but I was a bit confused about whether to use AWS Aurora or AWS RDS. Aurora's configuration did not meet the project's requirements. I needed the database to deliver consistent performance, and I didn't want to risk relying on scaling when the application needed to load everything very quickly. So I chose AWS RDS.

For the front-end framework, I chose React without a second thought. The client had a deadline, and React is very easy to use. So I used React and Vite for the front end. The UI framework was Shadcn.

For the back-end, I chose NestJS. As the application needed to be very reliable and I had a time limit, NestJS was the easiest choice. With NestJS, I didn't use Express, which is the default choice. I chose Fastify because it is lightweight and better suited to handling the application's high request volume.

For deployment, I used AWS Elastic Beanstalk. Elastic Beanstalk made it easier to deploy the Node.js application and handle horizontal and vertical scaling. I used load balancing with two instances for the back end.

The client's decision to use AWS was primarily to ensure the app's security. So security was one of the most important things for the project. For authentication, I used AWS Cognito. I also used Cognito-based roles and permissions to handle authorization. For the front end, I used AWS CloudFront. For rate limiting, I used AWS ElastiCache for Redis, which is a managed Redis service.

Role-based access control was also a really important feature of the project. There were more than 5 user types with different roles. Depending on the subscription, different users received different data from each API. There was significant variation in the data each user could access, so the APIs had to handle these differences carefully.

At peak time, I scaled the application to handle more than 30K requests per minute. I optimized all the queries to achieve peak performance. I used a trial-and-error method for this. I wrote different versions of each query and measured how long each took. Based on that, I kept the most performing query. I indexed the database according to its needs. I used different indexing for different searches and filters. I used B-Tree indexes for the fields that needed them and GIN indexes where they were more suitable, including for full-text search using tsvector. I also cached as many queries as possible by determining their expiry times. Because of better queries, indexing, and caching, most queries now take less than 200-250ms, whereas previous queries took more than 60 seconds.

For file uploads in .xlsx and .csv formats, I had to do a lot of parsing, and even then the providers were providing the data in the wrong format. So I had to implement strict parsing rules and provide instant feedback to the provider when the format was incorrect while they were uploading data files. Also, to ensure there is no duplicate data in the app, I had to implement a check and notify both the admin and the provider that they were uploading duplicate data. Though finding duplicate data in 50M+ records is a hard job, I optimized the implementation to ensure it runs quickly.

We are also storing those uploaded files in AWS S3. Previously, the client had to store those files manually as a backup. But now all those files are stored in S3, and the client can download those files anytime they want.

The new system has been in production for more than 1 year and has been running smoothly.

The biggest improvement was the performance. In the previous application, some pages took more than 60 seconds to load, and others timed out or didn't work properly. In the new system, those pages now load in around 200–250ms.

The new system also successfully handles more than 50 million records while supporting more than 30K requests per minute during peak times.

The new data upload system also added strict validation and duplicate detection, which helped prevent incorrect or duplicate data from entering the system. The data providers are now uploading around 5K records per day, sometimes a bit less.

The client was also very happy with the new design and the performance improvements. The new system replaced an application that was almost unusable and has been running reliably in production for more than a year.