How We Review Products

• 4 min read

Our Methodology
#

Every SetupScore review is built on multi-source intelligence. We aggregate and cross-reference expert reviews, YouTube video analysis, Reddit community discussions, and Amazon customer data, building a structured consensus for every product from 20-50 independent sources.

Across our catalog, we’ve processed 1,000+ sources from 170+ publications and extracted 2,500+ individual claims, each one cross-referenced to identify patterns, consensus, and disagreements.

The Pipeline
#

Every product goes through six analytical stages:

1. Source Discovery
#

We discover relevant sources across four platforms:

  • Expert review sites like RTINGS, PCMag, Tom’s Hardware, The Verge, and 160+ more
  • YouTube video reviews and comparisons, fully transcribed for analysis
  • Reddit community discussions and subreddit threads from real owners

Each product typically yields 20-50 independent sources covering expert, video, and community perspectives.

2. Content Extraction
#

Each discovered source goes through structured extraction:

  • Web articles parsed and cleaned for analysis
  • YouTube videos fully transcribed
  • Reddit threads extracted with discussion context
  • Ratings, pros, cons, and specific product claims identified per source

3. Source Analysis
#

Every source is independently analyzed:

  • Sentiment analysis measuring overall satisfaction and tone
  • Bias detection flagging sponsored content, affiliate-driven language, or manufacturer relationships
  • Rating normalization converting PCMag’s 4/5, RTINGS’ 7.8/10, and Reddit upvote sentiment to a universal 10-point scale

4. Cross-Source Consensus
#

Rather than averaging scores, we build a structured consensus map:

  • Confirmed strengths, positive attributes backed by multiple independent sources
  • Recurring issues raised across different platforms and reviewer types
  • Source disagreements where expert opinion diverges from community experience, surfaced explicitly so you can evaluate both sides
  • Use-case recommendations with “Best for” and “Not recommended for” derived from aggregated evidence across source types

5. Amazon Enrichment
#

Each review is enriched with live Amazon data across six international markets (US, UK, DE, FR, ES, IT):

  • Current pricing and availability
  • Customer review insights
  • Product imagery
  • Geo-targeted buy links

6. Aspect Discovery & Scoring
#

Once the overall product data is assembled, we run a second pass focused on per-dimension analysis. An LLM reads through all extracted claims and discovers the aspects that reviewers actually talk about for each product. No predetermined checklist. Headphones surface Sound Quality, Noise Cancellation, Comfort. Monitors surface Display Quality, Color Accuracy, Ergonomics. The taxonomy comes from the reviews themselves.

Every claim is then classified into its relevant aspect, scored by agreement weight (how many independent sources back the same observation), and assembled into the Performance Breakdown you see on each review page.

Scoring
#

Every product receives a SetupScore (0-100), computed algorithmically from:

  • Cross-source consensus strength
  • Normalized ratings from expert and community sources
  • Grade-inflation compression (tech publications tend to rate generously, we account for that)
  • Outlier dampening to limit the impact of one extreme rating
  • Credibility-weighted averaging, where established sources carry more weight
  • Source-count confidence adjustment, so a single glowing review can’t inflate a score

Scores are algorithmic. No editorial adjustments, no pay-for-placement.

Performance Breakdown
#

The overall score tells you the headline. But you probably want more than a headline.

Every product also gets aspect scores: per-dimension ratings for things like Sound Quality, Build Quality, Typing Experience, or Color Accuracy. These aren’t predetermined categories. They’re discovered from what reviewers actually discuss for each specific product, so the breakdown always reflects what matters for that product.

Aspect scores are agreement-weighted. When 8 independent sources all praise noise cancellation, that carries more weight than a single mention. Same for weaknesses. Five different reviewers flagging a creaky headband? That’s a pattern, not one person’s bad luck.

This means aspect scores can diverge significantly from the overall score. A headphone rated 87 overall can score 100 on Noise Cancellation and 61 on Portability. That’s not a bug. That’s the point: showing you exactly where a product shines and where it falls short, regardless of the headline number.

For aspects with limited data, scores blend toward the overall product score rather than swinging to extremes. Small samples shouldn’t produce wild numbers.

Editorial Standards
#

  • No paid reviews or rankings
  • No manufacturer influence on scores
  • Negative findings included when the data supports them
  • Every review shows source count and analysis date
  • Full affiliate disclosure on every page with buy links

Source Transparency
#

Every SetupScore review includes:

  • Total number of sources analyzed
  • Source types used (expert, video, community)
  • Analysis date
  • Individual source citations

Questions?
#

Questions about our methodology? Email: [email protected]

Last updated: March 28, 2026