This index exposes the material behind every decision guide: what we reviewed, when we retrieved it, the claim it supports and where the evidence stops.
181decision guides
790source records
782scoped claim notes
How to read this index
A citation is not a transferable result.
Reviewed identifies the part of the source used. Supports states the bounded proposition we draw from it. Retrieved records when time-sensitive material was checked.
Vendor documentation establishes vendor claims and product behavior as documented, not independent performance. Papers establish results only for their stated methods and tested scope. AccessAllGPT recommendations remain labeled analysis.
AccessAllGPT reviewed npm 12.0.0 release notes, npm 12.0.2 documentation, the pacote GitFetcher source bundled in the npm 12.0.2 registry tarball, open npm issue #9783 and the independent DIMVA supply-chain preprint. I built a synthetic local Git repository pinned to commits d886768c94acb3f9fa3104645d2e294e22bb31af and d40165addbb4c8ab3df8a6cf35981657786f9d60. Its prepare hook wrote built.txt and appended its temporary working directory, process version and executable to an external trace. Baseline runs used Node v24.10.0 with npm 11.6.0. npm 12 runs used npm 12.0.2 with its supported Node v24.15.0 binary through npx. I tested default npm 12 policy, --allow-git=all without allowScripts, and npm 11 --ignore-scripts. No production package, remote repository, credential, exploit payload or private system was used. The npm 12 tarball integrity returned by the registry was sha512-uIXokLlBj6FpNUTQX1PmT5pz7BlIN9QlixX+zdaSNHsd0qUXsbDLr50xzY6Sw7cJVr0uzHKDOle0swmPW/p5Qw==.
Reviewed: Release date, breaking changes for allow-git, allow-remote, supported Node versions and dependency lifecycle scripts
Supports: npm 12.0.0 was released on 2026-07-08 and changed git and remote dependency fetching plus dependency-script policy: git fetches require an explicit allowance and dependency scripts are blocked unless covered by the root allowScripts policy.
Reviewed: Git URL grammar, commit-ish resolution and the documented temporary clone, dependency install, script, pack and install flow
Supports: npm documents that selected scripts or workspaces make a git dependency a build input: npm clones it into a temporary directory, installs its dependencies, runs relevant scripts, then packs and installs the result.
Reviewed: GitFetcher constructor, #prepareDir, temporary clone and tarballFromResolved implementation paths
Supports: The bundled GitFetcher checks ignoreScripts before spawning npm inside the temporary clone, but the inspected #prepareDir path does not evaluate the root allowScripts policy before that preparation install and pack flow.
Reviewed: Issue title, reproduction context, open state, creation date and latest activity visible through the GitHub API
Supports: An independently filed open issue documents a neighboring npm 12 git-preparation and allow-scripts interaction. It is corroborating community evidence, not proof of the exact local result reported here.
Reviewed: Abstract, dataset scope, submission history and stated supply-chain attack model
Supports: The authors report a manually collected dataset of 174 malicious packages from npm, PyPI and RubyGems dated 2015–2019 and model code execution at different supply-chain stages; this supplies historical independent context, not an npm 12 measurement.
AccessAllGPT source review of the current npm scripts documentation, Node.js permission documentation, the Node.js threat model and NIST AI 600-1, plus a bounded local reproduction on Node v24.10.0 and npm 11.6.0 on macOS 26.0.1. In a temporary sandbox, I created a file dependency with preinstall, install and postinstall hooks that append to a trace file, ran npm install, confirmed the three hook lines were written, then reran the install with --ignore-scripts and confirmed the trace stayed empty. I did not audit a production package, publish a malicious package, analyze a real exploit chain or find a third-party write-up reproducing this exact local probe.
Reviewed: Description, Life Cycle Operation Order, Exiting and Working Directory for Scripts
Supports: npm says lifecycle scripts run after package changes are applied, lists preinstall, install and postinstall in order, and documents that scripts are executed through /bin/sh on POSIX systems or cmd.exe on Windows.
Reviewed: Permissions overview, child-process and filesystem controls, known limitations and security notes
Supports: Node’s permission model narrows selected runtime capabilities but does not claim to sandbox malicious code. That makes shell execution and package hooks a separate authority question, not a Node-only flag choice.
Reviewed: The Node.js threat model section and its statement about trusted versus untrusted code
Supports: The Node project’s threat model says the runtime trusts the code it is asked to run and that dependencies inherit the execution user’s privileges. That is the right backdrop for treating npm hooks as code execution, not as inert metadata.
National Institute of Standards and Technology · Retrieved
Reviewed: Publication abstract, scope, citation and report metadata
Supports: NIST frames AI trustworthiness as a design, development, use and evaluation problem. That general control framing supports least-privilege package installation, evidence logging and explicit rollback for agentic systems.
AccessAllGPT desk review of the current MCP security best practices, MCP authorization and tools specifications, RFC 8707, RFC 9728 and RFC 9700, plus a bounded local loopback reproduction on Node v24.10.0 that relays a synthetic Authorization header from one local HTTP server to another. We verified the public pages were reachable on 2026-08-22, but did not audit a production MCP implementation, a cloud auth server or any third-party deployment. The local probe demonstrates authority propagation through a relay, not a defect in MCP itself.
Reviewed: Token passthrough risks, local MCP server compromise, SSRF risks and common mistakes
Supports: The MCP security guidance explicitly frames token passthrough as risky, warns about local server compromise, and treats SSRF and token handling as security concerns for MCP deployments.
Reviewed: Resource parameter implementation, canonical server URI, access token usage and token handling
Supports: The authorization spec requires servers to validate that access tokens were issued specifically for them as the intended audience and describes the resource-oriented discovery flow that anchors the authorization relationship.
Reviewed: Security considerations and user interaction model for tool exposure
Supports: The tools specification emphasizes that tool exposure and invocation are security-sensitive, which matters when a server can forward user credentials to another service or trigger privileged actions on a user’s behalf.
Reviewed: Resource parameter, audience restriction and threat model
Supports: RFC 8707 defines resource indicators so the client can ask for a token for a specific protected resource instead of receiving a bearer token that is broadly reusable across a service fleet.
Reviewed: Protected resource metadata and security considerations
Supports: RFC 9728 standardizes resource metadata discovery so clients can learn which authorization server and token endpoints belong to a protected resource instead of inventing token-routing rules ad hoc.
Reviewed: Audience-restricted access tokens, token leakage and metadata attacks
Supports: RFC 9700 elevates audience-restricted tokens and related countermeasures as best current practice because bearer tokens are dangerous when they can be replayed or forwarded beyond their intended recipient.
AccessAllGPT source review and bounded local experiment, first run 2026-08-16 and refreshed 2026-08-21. We cloned Node.js tag v24.10.0 at immutable commit 9b72b88f4c4565687e3a8c4d8e1232f63a501e15, inspected its permission documentation, security threat model, environment initialization and 44 permission-named tests, and checked the official release index. We then downloaded Node.js v26.7.0 for Darwin arm64, verified its SHA256 against SHASUMS256.txt, reviewed the release note and permissions docs, and reran the local probe under both --permission-audit and --permission. The script creates only temporary fixture files, starts a loopback-only Python HTTP server, runs asserted scenarios in child Node processes, kills the server and removes the fixture. It tests direct filesystem reads and writes, relative-symlink traversal, loopback fetch, default child-process denial, an explicitly allowed non-Node child, an inherited file descriptor and the new audit-mode diagnostics path. We did not execute model-generated code, test a container or VM, probe Linux or Windows, attempt undisclosed bypasses, benchmark overhead, or perform a production security assessment. Node.js project statements, source observations, local results, independent NIST context and AccessAllGPT guidance are labeled separately.
Reviewed: Operational modes, runtime API, audit-mode diagnostics-channel mapping and known issues
Supports: The v26.7.0 docs define enforce mode and audit mode, explain that audit mode logs violations without denying access, enumerate the diagnostics-channel names, and retain the symlink and existing-descriptor limitations.
Reviewed: How npm handles the scripts field; shell execution behavior on POSIX and Windows
Supports: npm documents that scripts run through /bin/sh on POSIX systems and cmd.exe on Windows, which makes shell policy part of the agent boundary whenever an agent is allowed to invoke package scripts.
Reviewed: Scope and threat statement; stable status; restricted resources; filesystem allowlists; runtime API; npx behavior; constraints; symlink and inherited-file-descriptor limitations
Supports: The pinned documentation calls the permission model a seat belt for trusted code, explicitly says it does not protect against malicious code, lists the resources it restricts, says node:sqlite can access files outside node:fs checks, and documents relative-symlink and inherited-file-descriptor bypass paths.
Reviewed: Responsible testing; trusted and untrusted elements; treatment of code Node.js is asked to run; inherited execution-user privileges
Supports: The project threat model trusts JavaScript, WASM, native code and dependencies that Node.js is asked to run and says that code inherits the privileges of the execution user. This is incompatible with treating the runtime alone as a hostile-code security boundary.
Reviewed: Title; merge date; release lines; linked documentation history
Supports: The permission model moved to stable through a change merged on December 12, 2024 and documented for Node.js 22.13.0 and 23.5.0. Stable API status describes support maturity; it does not revise the project’s malicious-code exclusion.
Reviewed: Permission initialization in src/env.cc; permission implementation; CLI documentation; permission tests for filesystem, symlinks, children, workers, addons, WASI, inspector, sqlite and file descriptors
Supports: AccessAllGPT cloned the immutable v24.10.0 source commit. Environment initialization denies addons, inspector, child processes, workers and WASI unless explicitly allowed, then applies filesystem read and write allowlists. The test inventory exercises those controls and known symlink behavior; it does not implement a network permission boundary.
Reviewed: v24.10.0 release metadata and current release rows at retrieval
Supports: The official index dates v24.10.0 to October 8, 2025. AccessAllGPT used that exact installed runtime; this is a version-pinned result rather than a claim about every supported or future Node.js release.
Supports: NIST presents the profile as voluntary, cross-sector guidance for incorporating trustworthiness considerations into design, development, use and evaluation. It supplies independent governance context for layered, measured controls; it does not evaluate Node.js or certify a sandbox architecture.
AccessAllGPT desk review of the version-pinned MCP 2026-07-28 specification, MCP security guidance, the MCP TypeScript SDK 1.30.0 README and underlying OAuth RFCs, reverified 2026-08-20. Normative MUST and advisory SHOULD language is separated from AccessAllGPT recommendations. This is implementation guidance plus a local stdio probe, not an original security test or vendor comparison.
AccessAllGPT desk review of Z.ai’s GLM-5.3 launch, Coding Plan and API documentation; the public GLM-5.2 predecessor repository pinned at revision b4734de; Terminal-Bench 3.0 pinned at tag v3.0.0 and commit 2b0442c; CyberGym v3 and its harness pinned at commit 7656b71; the ExploitBench harness pinned at commit 9d0173b; and dated official-leaderboard and Hacker News snapshots, retrieved 2026-08-15. We wrote and ran reproducible Node.js audits against the pinned GLM-5.2 Hugging Face metadata and Terminal-Bench checkout. The GLM audit paginated all 295 files, reconciled 282 weight shards between the tree and index, and reported architecture and byte counts without downloading the weights. The Terminal-Bench audit reconciled all 74 task directories with 74 unique SHA-256 manifest digests and summarized categories, GPU use, agent timeouts and expert estimates. We also inspected both pinned cyber harnesses. For CyberGym, we installed its documented development and server dependencies under Python 3.13, compiled the source, and ran targeted smoke checks for checksum binding, image mapping, timeout normalization and rate limiting; we could not run a target because the Docker daemon was unavailable and the benchmark assets were not downloaded. For ExploitBench, we inspected its canonical matrix, scoring and image-resolution code and ran the non-slow unit and golden test suite under Python 3.11. The first run omitted optional publishing dependencies and failed four publishing tests; after installing the documented publish extras, 643 tests passed, six were skipped and six were deselected. We did not run an agent on Terminal-Bench, execute either cyber benchmark or call GLM-5.3. Momentum evidence, vendor claims, predecessor-artifact measurements, independent methodology artifacts, local harness evidence and AccessAllGPT guidance are labeled separately. This is not an original model run, benchmark reproduction, repository trial, red-team exercise, GLM-5.3 weight inspection, security assessment, GLM-5.3 license review, cost study or legal analysis.
Reviewed: Release date and status; post-training claim; coding, agentic and cyber benchmark tables; weight-release timing; API changes; Coding Plan availability; evaluation footnotes
Supports: Z.ai announced GLM-5.3 on August 14, 2026, says it uses the same base model as GLM-5.2 with gains from additional post-training, reports vendor-run coding and cyber evaluations, documents mandatory thinking and reasoning-effort settings, and says weights are planned two weeks after launch following safety evaluation and hardening. These are vendor claims and release statements, not independent AccessAllGPT measurements.
Reviewed: Coding Plan scope; supported coding-agent integrations; subscription and quota framing; API compatibility guidance
Supports: Z.ai documents its Coding Plan as a route for using GLM models in supported coding-agent clients. This establishes an available managed access path and integration surface; it does not establish GLM-5.3 quality, security, quota sufficiency or compatibility for a particular repository.
Reviewed: Request and response contract; model selection; messages; thinking configuration; tools; streaming; errors and usage fields
Supports: Z.ai publishes the request and response surface used to call supported chat models, including model selection, messages, tools, streaming and usage. The documentation establishes an integration contract; actual GLM-5.3 entitlement, behavior, latency, limits and billing must be verified on the adopting account.
Reviewed: Pinned repository tree and pagination; model card; MIT license text; config.json architecture fields; safetensors index metadata and weight map; serving-framework versions; file sizes and immutable object identifiers
Supports: AccessAllGPT queried the Hugging Face API and small metadata files at immutable GLM-5.2 revision b4734de. The predecessor has 282 bfloat16 safetensor shards with 1,506,659,919,872 indexed parameter bytes, a 78-layer GLM MoE DSA configuration, 256 routed experts with eight selected per token, and a 1,048,576-position configured maximum. Its pinned license file is MIT. These measurements establish the scale and metadata of GLM-5.2 only. They do not establish GLM-5.3 file size, license, serving compatibility, quantization, hardware needs or release status.
Reviewed: Tagged release and immutable commit; dataset manifest and content digests; task metadata and category distribution; agent timeouts and expert estimates; official leaderboard model-agent pairs, resolution rates, uncertainty, token totals and costs
Supports: AccessAllGPT cloned and audited immutable Terminal-Bench 3.0 tag v3.0.0 at commit 2b0442c. Its manifest binds 74 named tasks to 74 unique SHA-256 content digests across seven categories. Task metadata spans 30- to 480-minute agent limits, with a 120-minute median, and four GPU tasks. The official leaderboard reports model-agent configurations rather than model-only scores and, in the August 15 snapshot, listed GLM 5.2 with Claude Code at 4.6% ± 1.0% but did not list GLM-5.3. These facts explain the public benchmark contract and baseline; they do not validate Z.ai’s 28.3 result or identify its agent, repetitions, task failures, tokens or cost.
Reviewed: Version history; abstract; benchmark scale; task construction; proof-of-concept objective; reported baseline difficulty; limitations implied by the evaluation target
Supports: The independent benchmark paper defines CyberGym as 1,507 historical vulnerabilities across 188 projects and primarily asks an agent to generate a proof-of-concept test from a vulnerability description and codebase. This explains what the benchmark targets; it does not validate Z.ai’s GLM-5.3 score or establish safe autonomous exploitation.
Reviewed: README data and deployment requirements; FAQ network, task, environment and scoring guidance; August 4 submission schema; server submission and verification paths; task checksum; timeout handling; rate limiting; package metadata; local lint, compilation and helper smoke checks
Supports: At pinned commit 7656b71, AccessAllGPT inspected the current public CyberGym harness. The maintainers now request one designated final submission because any-of scoring increasingly rewards brute force, require token, time, request and artifact reporting, and warn that the partly unauthenticated PoC endpoint must remain private. The README estimates roughly 240 GB for benchmark data and roughly 10 TB for the full compilation environment. Local compilation and targeted helper smoke checks passed; the repository-wide Ruff check found one import-order error. Docker was installed but its daemon was unavailable, so no target container or PoC was run. This audit does not identify or validate Z.ai’s task set, metric, environment, network policy, trajectories or reported score.
Reviewed: README scope; canonical v8.yaml matrix; capability extraction and scoring code; image-reference resolution; model dispatch; seed and budget configuration; audit and reproduction workflow; local non-slow unit and golden tests
Supports: At pinned commit 9d0173b, AccessAllGPT inspected the public harness and ran its non-slow unit and golden test suite locally. The canonical configuration enumerates 41 V8 environments, six models and five seeds; the scorer cumulatively ORs 16 capability flags and treats arbitrary code execution as a terminal full score. The checked-in matrix uses mutable GHCR tags, while runner code resolves the locally loaded image to an immutable image ID and the documentation recommends registry digests for publication-grade pinning. This audit explains evaluation mechanics but does not identify or validate the revision, matrix, prompts, seeds, containers, transcripts or scoring transformation behind Z.ai’s reported GLM-5.3 result.
Reviewed: Submission timestamp; linked official release; observable points and comment count in the Algolia API snapshot; discussion scope
Supports: The August 14 submission linked the official release and showed 1,007 points and 496 comments in the snapshot retrieved August 15. This is a strong independent attention signal only; mutable community activity does not validate model capability, safety, availability, license or production fitness.
AccessAllGPT desk review of OpenAI launch, model, Fast-mode and pricing documentation; Cerebras launch and benchmark disclosures; and a dated Hacker News momentum snapshot, retrieved 2026-08-15. Momentum evidence, vendor specifications, vendor-performed benchmarks and AccessAllGPT guidance are labeled separately. This is not an original API trial, latency benchmark, quality evaluation, availability test, cost study, security review or legal analysis.
Supports: OpenAI announced Ultrafast mode on August 13, 2026 as a limited preview for selected API customers and describes it as a Cerebras-powered serving option for GPT-5.6 Sol. The page supplies vendor launch and performance claims; it does not establish results for an untested customer workload.
Reviewed: Overview; configuration; response reporting; rate limits and ramp rate; usage considerations; regions; billing; data controls
Supports: OpenAI documents Fast mode as a generally configurable service tier distinct from the new limited-preview Ultrafast offer. It documents up to 2.5x speed for GPT-5.6 Sol versus Standard, shared rate limits, possible downgrade to Standard when traffic ramps too quickly, response-tier reporting, eligibility limits and compatibility statements.
Reviewed: Model identity; reasoning levels; context and output limits; knowledge cutoff; pricing; endpoints; supported features and tools; snapshots
Supports: OpenAI identifies gpt-5.6-sol as its frontier model for complex professional work, documents text and image input, text output, reasoning-effort choices, a 1.05M-token context window, a 128K-token maximum output, supported tools and endpoint exclusions. These are vendor specifications, not independent capability measurements.
Reviewed: GPT-5.6 Sol Standard, Batch, Flex and Fast short-context and long-context token prices; regional-processing note; service-tier naming
Supports: The retrieved table lists GPT-5.6 Sol short-context Standard prices of $5 per million input tokens and $30 per million output tokens, and Fast prices of $10 and $60 respectively. Long-context and cache-write rules differ. The page does not yet publish a separate Ultrafast price in the reviewed table.
Reviewed: Launch scope; claimed output speed; HLE and GDP-Val methodology notes; customer use cases; architecture; limited-preview caveat; benchmark disclaimer
Supports: Cerebras says it powers the limited preview and reports up to 750 output tokens per second plus internal HLE and GDP-Val comparisons. Cerebras performed and published those tests, warns observed improvements vary by workload and configuration, and is commercially involved in the service; the results are vendor evidence, not independent validation.
Reviewed: Submission timestamp; linked release; observable points and comment count at retrieval; discussion scope
Supports: The August 13 submission was an observable developer-community momentum signal, showing 694 points and 270 comments in the Hacker News API snapshot retrieved August 15. Popularity is not factual or technical validation, and counts can change after retrieval.
AccessAllGPT desk review of current model-lifecycle documentation from OpenAI, Anthropic, Google Cloud and Amazon Bedrock, retrieved 2026-08-12. Vendor schedules and lifecycle terms are reported as vendor statements; the migration gates are AccessAllGPT guidance. This is not an original model test, compatibility study, performance benchmark, cost measurement, security assessment or legal review.
Reviewed: Overview; model deprecation notice periods; notification and replacement framing; API deprecation entries
Supports: OpenAI states that software using its models may need updates, documents recommended replacements, and publishes minimum notice periods of at least six months for generally available models and at least three months for specialized variants unless safety or compliance concerns require a faster timeline. Preview models may receive much shorter notice. These are vendor lifecycle statements, not a compatibility guarantee.
Anthropic Claude Platform Documentation · Retrieved
Reviewed: Overview; lifecycle terminology; platform scope; migrating to replacements; notifications; auditing model usage
Supports: Anthropic distinguishes active, legacy, deprecated and retired models; says requests to retired models fail; recommends testing applications before retirement; and states that publicly released models receive at least 60 days notice. It also says partner-operated Amazon Bedrock and Google Cloud schedules can differ from Anthropic-operated platforms.
Reviewed: Lifecycle overview; availability-period categories; retirement dates; migration framing; lifecycle definitions and date policy
Supports: Google Cloud publishes model lifecycle categories, retirement tables and migration guidance for the documented Gemini Enterprise Agent Platform. It says listed retirement dates may be extended but will not be brought forward, and distinguishes models available for at least 12 months from short-term models that retire 45 days after a replacement release. These terms apply to the documented Google Cloud service, not every Gemini surface.
Supports: Amazon Bedrock documents Active, Legacy and End-of-Life states, exposes lifecycle state through model APIs, and says a model remains on Bedrock for at least 12 months before its End-of-Life date. AWS warns that Bedrock dates can differ from model-provider dates and that Bedrock-specific dates govern Bedrock usage.
AccessAllGPT desk review of current OpenAI API data-control documentation, Amazon Bedrock data-protection documentation, the official GDPR text and the NIST Privacy Framework, retrieved 2026-08-11. Vendor statements, legal text and AccessAllGPT procurement guidance are labeled separately. This is not an original platform test, deletion audit, contract review, privacy impact assessment, security assessment, legal analysis or vendor comparison.
Reviewed: Data use; types of API data; abuse-monitoring retention; Zero Data Retention and Modified Abuse Monitoring eligibility and limitations; endpoint-level application-state retention; regional storage and processing controls; Enterprise Key Management
Supports: OpenAI states that API data is not used to train or improve its models unless a customer explicitly opts in, distinguishes abuse-monitoring logs from application state, documents default abuse-monitoring retention of up to 30 days, and describes feature-, project-, region- and eligibility-dependent controls. These are current vendor statements about the documented platform, not independent verification or a promise for every account and feature.
Reviewed: Shared-responsibility boundary; IAM, encryption and activity-logging recommendations; warning about sensitive data in tags and free-form name fields; model deployment accounts; linked encryption, PrivateLink and retention topics
Supports: AWS assigns customers responsibility for content controls and service configuration, warns that tags and free-form naming fields can enter billing or diagnostic logs, and states that model providers cannot access Bedrock deployment accounts, logs, prompts or completions. These are AWS service statements, not an audit of a customer configuration or a universal claim about cloud AI services.
Reviewed: Article 5 principles including purpose limitation, data minimisation and storage limitation; Article 28 processor requirements; Articles 44–49 transfers of personal data to third countries or international organisations
Supports: The official regulation text establishes principles and legal obligations relevant to processing, processor arrangements, retention and international transfers of personal data. It does not prescribe this article’s procurement workflow or determine whether a particular AI deployment is compliant.
National Institute of Standards and Technology · Retrieved
Reviewed: Framework purpose, voluntary risk-management positioning, Core and Profiles, implementation resources and relationship to enterprise risk management
Supports: NIST presents the Privacy Framework as a voluntary tool for identifying and managing privacy risk while building products and services. It provides risk-management structure, not certification, legal advice or approval of a provider.
AccessAllGPT desk review of current OpenAI evaluation and deprecation documentation, Anthropic evaluation guidance, HELM version 2 and the NIST Generative AI Profile, retrieved 2026-08-11. Vendor timelines, vendor guidance, research findings and AccessAllGPT procurement guidance are labeled separately. This is not an original platform test, migration exercise, grader study, cost benchmark, security assessment or vendor comparison.
Supports: OpenAI states that it announced deprecation of its Evals platform on June 3, 2026, that existing evals become read-only on October 31, 2026, and that the dashboard and API are scheduled to shut down on November 30, 2026. This is a vendor timeline for one product, not evidence that another platform is durable or that migration is lossless.
Reviewed: What evals are; evaluation process design; representative data and edge cases; evaluator types; human and model grading; continuous evaluation; Evals platform transition notice
Supports: OpenAI describes evals as structured tests for variable model behavior and recommends defining an objective, collecting representative and edge-case data, combining evaluator types and continuously evaluating. This is vendor-authored implementation guidance, not independent validation of a product or grader.
Anthropic Claude Platform Documentation · Retrieved
Reviewed: Success criteria; test-case construction; example metrics and measurement methods; automated, human and model-based grading; evaluation workflow and limitations
Supports: Anthropic recommends specific and measurable success criteria, multidimensional evaluation, representative test cases and a mix of grading methods. This is vendor guidance for evaluation practice, not independent evidence that Claude, a vendor console or model-based grading is accurate for a buyer’s workload.
Transactions on Machine Learning Research / arXiv · Retrieved
Reviewed: Abstract; scenario and metric taxonomy; multi-metric design; standardized evaluation scope; transparency artifacts; reported coverage and stated limitations
Supports: The HELM authors separate scenario coverage from metric coverage, evaluate multiple dimensions rather than accuracy alone, and publish raw prompts and completions for analysis. The reported study concerns its selected models, scenarios and historical period; it does not validate a procurement checklist or current commercial platform.
National Institute of Standards and Technology · Retrieved
Reviewed: Publication metadata; abstract; voluntary cross-sector scope; relationship to AI RMF 1.0; design, development, use and evaluation framing
Supports: NIST describes the profile as a voluntary, cross-sector companion to AI RMF 1.0 intended to help organizations incorporate trustworthiness considerations into the design, development, use and evaluation of generative AI systems. It does not certify an evaluation platform or prescribe this build-buy decision.