Don't Trust Newegg's New AI-Generated Review Summaries

Whether we're looking to buy some headphones on Amazon, or pick a place for dinner on Google Maps, we turn to the reviews for guidance. But you can't blindly trust online reviews: Some are written by bots, others by people paid to write a good review.

You always need to take a moment to consider whether the review you're reading is honest and informed, which is just one more reason to not rely on Newegg's new AI review summaries.

Newegg's AI summaries are misleading

As reported by Digital Trends, Newegg is now using ChatGPT to summarize customer reviews on its various computer parts and tech products. When you scroll down to the Reviews section of a product page, you'll now see a new box, called "What Reviewers Are Saying About the Product." If you click the small (?) next to the title, you see what's powering this new section:

"The following compilation of pros, cons, and summary is the result of an AI algorithm that analyzed and generated content based on customer reviews. Efforts have been made to ensure accuracy but individual experiences, opinions, and interpretations may vary and influence the generated content. The AI-generated content is provided as-is, without any warranties or guarantees of any kind, either expressed or implied."

True to this statement, you'll see a list of pros, cons, as well as a short review of the product based on an aggregate of customer reviews, called Summary^AI. In theory, this feature might seem like a good idea. After all, the star rating we see at the top of every product page is an average of all the user reviews, so maybe this is just the next step: a summary of what users liked about a product, and what they didn't.

The problem is, these summaries often suck, ranging from merely unhelpful to downright confusing. Take a look at this PS5 bundle, where the review summary reads:

"The PS5 bundle is highly praised for its fast shipping, brand new condition, and great performance. Customers appreciate the quick delivery and that the product arrived as advertised. Some customers found the price to be high, but overall, the reviews are positive and recommend the product."

Yeah, my favorite thing about my PS5 is how fast it shipped, too.

It seems the AI can't differentiate between reviews discussing Newegg's shipping and the actual product itself. Unless you're looking to find out whether Newegg has a habit of sending broken PS5s to customers' doors, but beyond that, you're probably more interested in specific information about the console.

Digital Trends also highlighted a contradictory AI review summary for the MSI Katana laptop, which praises the machine for having an efficient cooling system, while in the same review calling out the loud fans and "hot running temperatures." Perhaps the AI saw a review from one user who thought the machine had a great cooling system and included it, then pulled reviews from other users complaining the cooling system was, in fact, not so efficient. But it's hard to tell at a glance, and certainly confusing--which defeats the purpose of perusing a review summary in the first place.

The AI can sometimes generate a helpful summary. When the reviews are generally consistent, it can pull useful data points and report on them accordingly. The Summary^AI review for this Gigabyte monitor, for example, seems to weight the general pros and cons of the reviews well enough. The quality of the summaries tends to vary widely.

AI isn't ready for reviews

Despite some solid summaries in the bunch, it seems the tech just isn't good enough yet to make this a useful feature. One person's complaint could end up in the summary when it was really just an outlier issue, while the AI could miss a real issue that many people are experiencing. There's also no way to tell from the general summary how many users agree with the data points the AI is presenting. Did everybody love the monitor's color accuracy? Was it only one person who found a dead pixel on their screen? How many people really praised the PS5 bundle for its fast shipping?

These are things that you pick up when reading multiple reviews yourself. You can get a full picture of what each reviewer encountered, and weigh multiple reviews against one another, giving you a general idea of how many people had similar experiences. Digital Trends notes that Newegg is only testing these summaries on a few products to start, but I've found them on many, so it seems the testing is rolling out quickly.

This isn't Newegg's first foray into AI-generated content. As Digital Trends investigated back in March, the company tried using ChatGPT to help users build PCs. It didn't go well: The builder would outright ignore your requests, returning results for unbalanced PCs that didn't fit your needs.

AI is certainly much further along than most of us expected it to be. But it doesn't seem smart enough to accurately tell you what people thought about the stuff they bought.