Rue du Commerce - Catalogs
In order to make Rue du Commerce a leader again, we revamped the catalogs uploads processing to manage tens of millions of products every day. Moving from a night batch process to a real-time update.
Rue du Commerce - Catalogs Uploads Processing
We managed the projects in several phases.
First, we analyzed the global steps to process the catalogs uploads. We encapsulated the complexity in a “black box” and started to build around a brand new process.
Second, we started to empty the black box feature by feature.
We built what we called the turbine, putting cache mechanisms with Redis, MD5 signature, messaging technologies…
Third, we cut catalogs in small packets, ordering the packets taking into account the business priority (what do we sell right now…). The turbine was highly distributed to launch workers on demand, taking into account the priority level.
At the end, we benefited from a real-time system to uploads packets from a minimum of one offer from one merchant when we needed it.