Products // open source
Smartfeed
The product-feed engine for ecommerce and marketplaces.
An ecommerce home, category or search screen is really several systems at once: recommendations, promoted items, best-sellers and local stock. Getting them into one fast, consistent scroll is fiddly backend work: fetch them together, drop duplicates, re-rank, cache. Smartfeed is the engine that does all of it, and you shape the feed in a config file, so a new source or a new blend ships from config, with no code release.
What it solves
A feed comes from many places
Take a category page. The personalized row comes from one service, the sponsored slots from another, best-sellers from a third, and local stock from a fourth. Each is a separate backend, with its own speed and its own way of handing back pages.
So the feed has to call all of them, and call them together so the page stays fast. It drops duplicates that show up from more than one source, re-ranks the result, and caches it so scrolling does not hammer slow backends. And it has to stay consistent while someone keeps scrolling.
Historically each product surface wrote that glue by hand, then rewrote it for the next one.
How it works
A feed is a tree
You describe the feed as a small tree of nodes in config. Every node is one of three kinds, and they nest as deep as you need.
A SubFeed is a leaf: it wraps one of your own fetch functions, the point where Smartfeed calls your code. A Wrapper is a pipeline node: it adds optional dedup, re-rank and cache, each switched on alone. A Mixer is a combiner: it fetches its children at once and merges them by a rule you pick.
The app builds the feed once, then asks for one page at a time. A request plans each source's share, fetches the sources together, removes repeats, re-ranks, caches, and returns one page plus a cursor: a token you pass back to get the next page. Every item leaves stamped with its position and its source, so you can answer why it showed up where it did.
Stack
What you get
Reshape the feed in config
Adding a source, changing a blend from 40/60 to 30/70, pinning a sponsored slot, setting up an A/B test: each is a config edit. No code change, no redeploy. That is where the time goes back to the product team.
Sources are fetched together, so a page waits about as long as the single slowest source, well under the sum of them all. Caching keeps scrolling cheap: the first page in a session does the heavy work once, and later pages slice from storage.
One limit, worth naming up front: the most valuable features ride on Redis, a fast in-memory store. Without it a feed still works, but it runs uncached, and an item shown earlier can reappear later.
The blend
Five ways to blend sources
Mixers are the main lever a product team tunes, and they do not all interleave. Two lay down solid blocks; three weave items together.
Percentage splits the page by fixed shares, say 40/60, a solid block per source. Append concatenates several sources with an even split, then trims to one page. Positional pins one source to set slots and lets another fill the rest, so sponsored items land at guaranteed positions.
Gradient drifts the ratio as the user scrolls, from trending toward personalized picks further down. Distribute takes turns across all sources, so no two neighbors are the same kind: the marketplace rule of no two items from one seller in a row.
Where it breaks
Keep in mind
Graceful failure is opt-in, per source. By default one failing source or a failed re-rank fails the whole page, so you set a flag on the ones you want to degrade softly.
Distribute costs more backend traffic than it shows: it asks every source for a full page, then keeps only a page's worth. Size for it.
On cache loss the feed rebuilds from page one; it does not resume mid-scroll. And we claim no latency or throughput numbers here, so measure speed and cost against your own sources and traffic.
Related