About this series. We are building a recommendation engine from scratch on Google's own store data — the public GA4 export of the Google Merchandise Store — and publishing what comes out, including what does not work. Part 3 showed that a behavioural model returns an empty list for a product nobody has viewed, and proposed letting new products borrow the recommendations of products nearest them by meaning. This is the test of that proposal.
It did not work very well. The idea recovered roughly an eighth of the distance between the fallback most shops use today and the recommendations a product earns from its own history — and the interesting part is why, because the obvious explanation turns out to be wrong.
The honest problem with evaluating a cold-start fix is that cold products are, by definition, the ones nobody interacts with. In our test window only 34 sessions began with a product that had no history at all — far too few to separate a good idea from a bad one.
So we ran a simulated cold start. Take products that do have history, hide their own co-view data, serve them their nearest neighbour's recommendations instead, and score on the same held-out sessions. That gives 2,453 sessions of evidence, a clean upper bound (what the product's own history achieves) and a clean lower bound (the category fallback). It is the standard way to get statistical power out of a cold-start question, and it is slightly generous to the method, since these products are better connected than a genuinely new line would be.
| What the shopper was shown | Opened one of the ten | Share of the gap closed |
|---|---|---|
| The product's own history | 51.1% | — the ceiling |
| Inherited from its nearest neighbour | 35.6% | 12% |
| The category's top sellers | 33.4% | — the floor |
Inheritance beat the category fallback by 2.2 percentage points. On 2,453 sessions that is outside chance, but it is small: it closes about an eighth of the 17.7-point gap between the floor and the ceiling.
The obvious diagnosis is that our similarity measure was too crude — shared words in the product name plus category, rather than a language model's embeddings. We expected to write that conclusion. Then we looked at the pairs it actually chose:
| Product | Nearest neighbour it borrowed from |
|---|---|
| Google Crew Socks | YouTube Crew Socks |
| Google Camp Mug Ivory | Google Camp Mug Gray |
| Google Zip Hoodie F/C | Google Youth FC Zip Hoodie |
| YouTube Icon Tee Grey | YouTube Icon Tee Charcoal |
| Google Clear Pen 4-Pack | Google Cloud Pen |
| Google Land & Sea Cotton Cap | Google Land & Sea Unisex Tee |
Those are the pairs a merchandiser would choose by hand. If the matching is already picking the near-identical product, a better similarity measure has very little room left to improve on it — which means embeddings would probably not rescue this either, and the assumption underneath the idea is what failed.
A co-view list is not a list of similar products. It is a record of the paths people take through your shop — which pages follow which, driven by navigation, merchandising, search results and whatever was on the home page that week.
Two near-identical products often sit on different paths. The grey tee is reached from the tees listing; the charcoal one from a promotion; a mug from the Drinkware page and its twin from a campus collection. They look alike to a human and to an embedding model, and they are browsed in different company. That is the gap the method could not close.
Similarity is Jaccard overlap on product-name tokens, after dropping words carried by more than 30% of the catalogue (“google” is not a distinguishing feature here), with ties broken by category match and then price proximity. A product may only borrow from a lender with at least 20 sessions of history:
neighbour AS (
SELECT item_a AS item_key, item_b AS lender
FROM similarity s
JOIN item_n ON item_n.item_key = s.item_b
WHERE item_n.n >= 20 -- the lender must have something to lend
QUALIFY ROW_NUMBER() OVER (
PARTITION BY item_a
ORDER BY jaccard DESC, same_cat DESC, price_gap ASC) = 1
)
Two limits worth stating. We used text similarity rather than embeddings because no Vertex AI connection exists on this warehouse yet; given that the matching already produces near-identical pairs, we do not expect embeddings to change the conclusion, but we have not proven that. And 3.4% of test sessions had no eligible neighbour at all, which a genuinely new product in a thin category would hit more often.
bigquery-public-data.ga4_obfuscated_sample_ecommerce — the obfuscated GA4
export of the Google Merchandise Store, published by Google through the Cloud Public
Datasets Program. Trained on data to 10 January 2021, tested on sessions from 11 to 31
January; crawler traffic excluded throughout, for the reasons in
part 1.
Related: The lifetime-value forecast you can actually bid on, part 3 of our customer-lifetime-value series — another case where the measurement contradicted the advice we had just published. All articles.
Discover practical, scalable solutions tailored to your business priorities.