Accuracy test
We asked a model 20 questions about CVEs, software end-of-life dates and paper citations, once with our MCP servers and once without. We graded each answer against a primary source.
Right answers by topic
| Topic | Without tools | With our tools |
|---|---|---|
| CVE | 4 of 7 | 7 of 7 |
| End of life | 6 of 7 | 7 of 7 |
| Citations | 4 of 6 | 6 of 6 |
The 20 questions
| # | Question | Without | With | Source |
|---|---|---|---|---|
| 1 | Is lodash 4.17.15 (npm) affected by a known vulnerability? Name one CVE ID and the first version that fixes it. Source answer: Yes. For example CVE-2020-8203, fixed in 4.17.19 (any listed pair is accepted). | right | right | osv.dev |
| 2 | Is axios 1.12.0 (npm) affected by a known vulnerability? Name one CVE ID and the first version that fixes it. Source answer: Yes. For example CVE-2026-25639, fixed in 1.13.5 (any listed pair is accepted). | right | right | osv.dev |
| 3 | Is next 15.5.0 (npm) affected by a known vulnerability? Name one CVE ID and the first version that fixes it. Source answer: Yes. For example CVE-2026-29057, fixed in 15.5.13 (any listed pair is accepted). | right | right | osv.dev |
| 4 | Is Django 5.2.5 (PyPI) affected by a known vulnerability? Name one CVE ID and the first version that fixes it. Source answer: Yes. For example CVE-2026-1312, fixed in 5.2.11 (any listed pair is accepted). | right | right | osv.dev |
| 5 | Is CVE-2023-22894 (Strapi) listed in CISA's Known Exploited Vulnerabilities catalog? If yes, on what date was it added? Source answer: Yes, added 2026-10-08. | no answer | right | raw.githubusercontent.com |
| 6 | Which GitLab 19.3 patch release fixes CVE-2026-85706? Source answer: 19.3.2 | no answer | right | services.nvd.nist.gov |
| 7 | What CVSS v3.1 base score does NVD list for CVE-2026-87902 (WordPress Core)? Source answer: 8.1 (High) | no answer | right | services.nvd.nist.gov |
| 8 | On what date does Node.js 20 reach end of life? Source answer: 2026-04-30 | right | right | raw.githubusercontent.com |
| 9 | In what month and year does Python 3.10 reach end of life? Source answer: October 2026 | right | right | peps.python.org |
| 10 | On what date does PostgreSQL 14 reach end of life? Source answer: 2026-11-12 | right | right | postgresql.org |
| 11 | On what date does Kubernetes 1.34 reach end of life? Source answer: 2026-10-27 | right | right | raw.githubusercontent.com |
| 12 | In what month and year does Django 5.2 LTS reach end of extended support? Source answer: April 2028 | right | right | djangoproject.com |
| 13 | Is Go 1.25 still supported by the Go team today? If not, on what date did its support end? Source answer: No. Support ended 2026-08-19, when Go 1.27 was released (each major release is supported until two newer major releases exist). | no answer | right | go.dev |
| 14 | On what date does .NET 8 reach end of support? Source answer: 2026-11-10 | right | right | raw.githubusercontent.com |
| 15 | Does this reference exist: Jumper, J., Evans, R., Pritzel, A., et al. (2020). Highly accurate protein structure prediction with AlphaFold. Nature, 596, 583-589? If so, give the correct DOI and year. Source answer: Exists, but the year is 2021 (not 2020). DOI 10.1038/s41586-021-03819-2. | right | right | api.crossref.org |
| 16 | Does this reference exist: LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521, 436-444? If so, give the correct DOI and year. Source answer: Exists. 2015, DOI 10.1038/nature14539. | right | right | api.crossref.org |
| 17 | Does this reference exist: Guo, D., Yang, D., Zhang, H., et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645, 633-638? If so, give the correct DOI and year. Source answer: Exists. 2025, DOI 10.1038/s41586-025-09422-z. | right | right | api.crossref.org |
| 18 | Does this reference exist: Ashokkumar, A., Hewitt, L., Ghezae, I., et al. Large language models can predict the results of social science experiments. Nature, 656, 115-122? If so, give the correct DOI and year. Source answer: Exists. 2026, DOI 10.1038/s41586-026-10742-x. | no answer | right | api.crossref.org |
| 19 | Does this reference exist: Kowalczyk, M., Okafor, C., & Lindqvist, E. (2024). Retrieval-grounded citation verification reduces fabricated references in LLM-written literature reviews. Journal of the American Medical Informatics Association, 31(9), 1874-1883. https://doi.org/10.1093/jamia/ocae918? If so, give the correct DOI and year. Source answer: No. The DOI does not resolve (doi.org 404) and no record with this title exists in Crossref or OpenAlex: fabricated. | no answer | right | doi.org |
| 20 | Does this reference exist: Hinton, G., & LeCun, Y. (2019). Self-supervised scaling laws for vision transformers. Nature Machine Intelligence, 1(3), 112-120? If so, give the correct DOI and year. Source answer: No. No record with this title or these authors exists in Crossref or OpenAlex: fabricated. | right | right | api.crossref.org |
What the result says
Without tools, the model gave no wrong answer, but it declined 6 of 20 questions: facts newer than what it learned, such as a CVE published in 2026 or a 2026 paper. With our CVE Risk Check, Dev Facts Check and Citation Checker servers it answered all 20, and every answer matched the source. On this test, the tools did not fix wrong answers; they turned "I am not sure" into a checked answer.
How we tested
Questions. 7 on package and CVE facts, 7 on end-of-life dates and 6 on references: 4 real papers, one of them with a wrong year, and 2 invented ones. Each has one short answer we checked on 9 Oct 2026 against a source that is not ours: osv.dev, NVD, the CISA catalog of exploited flaws, the vendors' own release data, and Crossref or doi.org.
Runs. Each question ran twice in Claude Code 2.1.288 with the model claude-sonnet-5-5, from an empty folder with no other tools: once with no MCP server, and once with cve.openkrill.app, devfacts.openkrill.app and citations.openkrill.app. Both runs had the same short instruction: answer briefly, and say so when unsure.
Grading. An answer is right when every fact it states matches the source, wrong for any factual error or an invented reference accepted as real, and "no answer" when the model declined. For a package question, any CVE that osv.dev lists for that exact version counts, with its fix version.
Limits. One run per question in each mode, with one model and one client, so the counts are small and can differ on another run. Several questions ask about facts newer than the model's training data, which is the case the tools are for. Every question, answer and grade is in accuracy-test.json.