Skip to the content
Agent Tools

We Ran 48 Real Broken Product CSVs Through Our Catalog Checker: It Caught 6, and Where It Fell Short

Our checker found 7 of 7 planted errors, then caught only 6 of 48 real broken files. Both numbers are true, and the gap between them is the point of this post.

By OpenKrill. Published 2026-10-31. Based on our own checks of reports collected 2026-10-21.

How we tested

On 2026-10-21 we collected 307 public reports of failed product imports from Shopify, WooCommerce and Google Merchant Center help threads, GitHub issues and Stack Overflow. 48 of them came with the broken file itself (44 unique files: 35 full files and 9 pasted fragments). The other 259 described only the error.

Before scoring, we wrote down which of our rules counts as catching each reported error class, and whether that error can be seen in the file's own bytes at all. Then we ran every saved file through the same checks the live tool uses. These are our own measurements.

The planted test

We built one audit CSV with 7 known errors. All 7 were found in 1 call, each with the row, the rule, the fix and a link to the Shopify doc. Before a fix, only 5 of 7 were found.

  • Blocking: row 4 duplicate variant, row 5 invalid status, row 6 bad weight.
  • Warnings: row 3 bad barcode check digit, row 5 compare-at price not higher, row 6 number in scientific notation, row 6 image URL not https.

The row 3 finding names the right check digit, 2. Shopify's own product template produced 0 findings, so the checker does not cry wolf on a clean file.

The real files

  • All file cases: 6 of 48 caught (12.5%).
  • Where the error shows in the file: 6 of 43 (14.0%).
  • Shopify files: 6 of 20 (30.0%).
  • WooCommerce files: 0 of 28 (0.0%).
  • Where we have a rule for the file's error class: 6 of 11 (54.5%), 4 of them with the right advice.
  • Shopify files with at least one blocking finding: 12 of 20 (60.0%).

Of the 259 reports that described only an error, we have a rule for the reported class in 92 (35.5%). Only 136 describe an error a file checker could see. The other 123 came from a crawl, store settings, an upload check or a remote image.

Where it fell short

  • It does not read a WooCommerce product CSV at all. Every WooCommerce file was refused as "not a product CSV", so 0 of 28 means "format not supported", not 28 misses.
  • 19 of the 48 file cases were importer bugs, not bad data. No file checker can catch those from the file alone.
  • On 6 of the 12 Shopify files it flagged, the blocking finding was a different problem from the one the poster had.
  • 2 of the 6 catches gave the wrong advice.
  • We aimed for about 70 real files and found 44. We did not pad the set with files whose thread reported no error.

What to do before your next import

This applies if you import product CSVs into Shopify. If your file is for WooCommerce, our tool will not read it.

  1. Run the CSV through a checker before you import.
  2. For each finding, open the Shopify doc it points to and compare the row against the column rule.
  3. Fix blocking findings first, then warnings.
  4. If the result is clean, read it as "no known error found", not "safe to import".

If you want the row, the rule, the fix and the Shopify doc link for each finding so you can do that comparison yourself, Catalog QA Desk returns all four.

A planted test shows a checker works. A set of real broken files shows how often it helps: here, 6 times in 48.

Frequently asked questions

What does our tool return for a product CSV in one call?

For each finding it returns the row, the rule that was broken, the fix and a link to the Shopify doc. Findings are split into blocking findings and warnings.

Does it check WooCommerce product files?

No. Every one of the 28 WooCommerce files was refused as not a product CSV, so those results reflect an unsupported format.

Why can a clean result still fail on import?

Because 19 of the 48 file cases were importer bugs, and 123 of the 259 error-only reports came from things outside a file, such as store settings or a remote image. A file check cannot see those.

Sources