It started with a simple yet ambitious question: How can we make digital advertising more meaningful?
As an engineer, I’ve always been intrigued by the intersection of technology and human connection. Leading the creation of a scalable, modular framework for personalized recommendations became more than just a project.
It was an opportunity to architect a software that not only adapted to user needs but also scaled to create a broader, lasting impact for businesses and audiences alike.
This framework was a transformative effort that redefined social media advertising. It processed an enormous scale of transactions per second, seamlessly integrated with external services, and ensured data freshness through asynchronous streaming.
The system was built to support real-time, high-performance personalized product recommendations for users across a rapidly evolving digital landscape.
At the company where I led this initiative, scalability wasn’t just a goal—it was the foundation of everything we created. The result was a system that elevated user engagement, optimized ad effectiveness, and aligned with business goals in an ever-changing environment.
Back in 2017, as a Lead Software Development Engineer, I was tasked with leading the tech design and implementation of a modular framework capable of delivering personalized recommendations in near real-time on all popular social media networks.
The challenge was immense: designing a system to process a high throughput of transactions per second with low latency and high reliability. The core of the framework was its ability to source and serve personalized recommendations at scale, using ML for real-time optimization and asynchronous data aggregation to deliver relevant, up-to-date advertisements.
The system was designed to source, aggregate, and serve streaming data from multiple external sources, enhancing both the accuracy and freshness of recommendations.
The key to building such a high-performance system was its distributed, multi-tier, and service-oriented architecture.
Leveraging cloud services and serverless technologies, we created a framework that could seamlessly scale in response to fluctuating demand. Each layer of the system was designed to maintain peak performance, regardless of the growing volume of data or the increasing complexity of requests.
The combination of Real-Time Recommendation Generation and Streaming Data Aggregation, along with a configurable Recommender Registry, was essential. Real-time recommendation generation used a modular architecture for others to build custom recommenders, while the streaming data layer aggregated and processed vast amounts of data to asynchronously generate highly available services in the same recommender framework.
Amazon is well-known for personalization and recommendations, which help customers discover items they might otherwise not have found. (Source: Amazon Science)
Facebook Ad Auction (Source: Medium)
These components worked in tandem, delivering fast, reliable recommendations that were relevant and timely.
Creating a system of this complexity wasn’t without its challenges. One of the most difficult aspects was integrating multiple external services while ensuring efficiency and performance. Every component had to work seamlessly together, from the APIs to the data schema, all while adhering to strict technical constraints.
Another challenge was ensuring the system could operate efficiently while handling real-time streaming data.
We implemented asynchronous processing mechanisms to decouple data aggregation from recommendation generation, allowing both processes to scale independently and operate in parallel.
This approach not only improved responsiveness but also increased the overall throughput of the system.
This project required close collaboration with various teams—engineers, product managers, data scientists, and external partners. Together, we aligned on solutions that balanced innovation with operational excellence.
Engineers focused on the technical details, while product managers ensured the system aligned with the broader business goals, and data scientists fine-tuned the machine learning ad-bidding models. The result was a highly effective, scalable system that could adapt to evolving business needs and user expectations.
In 2017, digital ad spend per internet user in the US was $319.40. From 2017 to 2024, this number increased threefold. Digital ad spending per person is expected to continue rising. In 2025, this is set to surpass $1,000 per person and rise further to $1,081 in 2026 and $1,146 in 2027. (Source: Statista/Oberlo)
The business value of this modular framework was immediately clear. By improving the personalization of advertisements, the system led to higher user engagement and significantly boosted conversion rates.
Marketing teams saw increased operational efficiency as the system handled much of the technical complexity, allowing them to focus more on strategy and less on troubleshooting and integration. The framework auto-scaled to a countless number of campaigns and strategies, as opposed to the ones limited by the number of personnel in the marketing team.
Beyond the metrics, the real success of the system was its ability to create meaningful user experiences. It wasn’t just about revenue—it was about making users feel understood and valued. Personalizing the ad experience helped build trust and foster deeper connections between brands and their audiences. This reinforced key lessons that have shaped my career and approach to engineering.
First, scalability is paramount. A great system must be designed to scale not just for today’s needs but for tomorrow’s opportunities as well. Building with scalability in mind ensures that systems can adapt to the complexities of an ever-evolving business and technology landscape.
Second, user-centric design is crucial. While meeting technical requirements is important, it’s equally essential that the system enhances the user experience, making users feel understood and valued.
Lastly, collaboration plays a central role. No system is created in isolation; great solutions emerge from working together and leveraging diverse viewpoints and expertise.
Ensuring that every decision aligns with technical and business objectives ultimately leads to impactful results.
I hold a B.Tech & M.Tech in Computer Science from IIT BHU, where my thesis on context-based collaborative filtering shaped my approach to personalized recommendation systems. Reflecting on our work, I’m proud of how the system not only addressed immediate challenges but also set the stage for future innovations in digital advertising, demonstrating the transformative power of scalable, modular design.
As I continue to build systems, I’m reminded of why I chose engineering in the first place: to create meaningful solutions that solve real-world problems and leave a lasting impact. What drives me isn’t technology for technology’s sake—it’s the power to build systems that bring people together, spark inspiration, and create progress.
Engineering is my avenue to influence the future, and there’s so much more to come.
About the Author
 |
Harshit Sharan is a Software Engineer with expertise in distributed systems, cloud technologies, and machine learning. At Amazon, he leads impactful projects, including modular frameworks for personalized recommendations, driving innovation and scalability. |
For more insights or to collaborate, feel free to connect with me onLinkedIn or reach out via email at hsincredible@gmail.com.
