A protocol for cross-scale agreement analysis

Abstract

Agreement between anticholinergic scales is interpretable only when medication identity, source version, context, and missingness are specified. This article proposes a reproducible protocol for cross-scale comparison using original source observations. It does not report a newly computed agreement coefficient or reclassify any medicine. The proposed methods freeze source versions, establish compatible identity and route/formulation matches, distinguish coverage from value agreement, and report the exact denominator for each comparison. Descriptive cross-tabulations precede any statistic, and ordinal methods are considered only when their assumptions fit the categories being compared. Missing entries remain missing and are never converted to zero. The expected result is a transparent account of where sources overlap, differ, or cannot be compared, rather than a consensus rating. Clinical validity remains a separate question: high agreement would not prove correctness, and disagreement would not by itself identify a superior source or an appropriate treatment decision.

Introduction

Two scales can appear to disagree because they classify the same medicine differently, or because they cover different medicines. They can also appear to agree after unmatched or incomplete records have been silently excluded. A meaningful analysis needs to distinguish these situations before calculating a summary statistic.

The objective is to outline a protocol that preserves the original evidence and makes the comparison reproducible. The article is a methods proposal, not a report of a completed cross-scale study. No synthetic UnifiedACB category is created from the planned comparison.

Methods

The protocol would identify each source version and freeze the dataset and mapping layer before analysis. A source family name alone may be insufficient if there are revised tables, corrections, or separate public representations. The input record should retain literal labels, locators, qualifications, and public-reuse scope.

The unit of analysis would be defined explicitly. A literal source row, a normalized ingredient concept, and a route-specific product are different units. The primary comparison should include only records judged compatible under prespecified identity and context rules. Unresolved or incompatible matches should be reported with reasons rather than forced into the matrix.

Coverage would be described before agreement. For each source pair, report the set represented in both sources, represented only in the first, represented only in the second, and unresolved under the matching rules. A published zero belongs among observed values; an absent entry does not. The denominator for value agreement is therefore not automatically the full medication directory.

The initial descriptive output would cross-tabulate the original categories for comparable observed records. This allows readers to see the pattern of disagreement rather than only a single coefficient. Source-native values should remain unchanged. Any display recoding must be explicitly defined and should not imply that categories from different methods have identical clinical meaning.

Choice of a statistic would follow the measurement question. Exact agreement can be described when matching categories is meaningful. An ordinal agreement statistic may be considered only if the ordering and any weighting scheme are justified. Correlation answers a different question from exact agreement and should not be relabeled. No particular coefficient or weighting is mandated here without inspecting the scales’ definitions.

Sensitivity analyses would be prespecified for alternative defensible identity rules, route restrictions, source versions, or coverage subsets. They should test a concrete uncertainty rather than select the most favorable result. The report should retain all planned analyses, including results that weaken an apparent conclusion.

The protocol would also define quality checks: duplicate observations, unexpected values, conflicting within-source records, and source anomalies. Such issues should be resolved or preserved explicitly before analysis. A numerical result produced from invalid matches is not made reliable by a sophisticated statistical method.

Results

No new cross-tabulation, agreement percentage, correlation, or coefficient is reported in this article. The proposed protocol yields a set of defined outputs: source coverage, compatible comparison sets, original-category tables, justified agreement summaries, and documented sensitivity analyses.

The public dataset already preserves source-version identifiers and missingness distinctions needed for such a protocol. UnifiedACB public evidence dataset, v3.0 RC12 final That structural readiness does not mean the proposed analysis has been completed. It also does not establish that all pairs of scales use clinically interchangeable categories or that every medication can enter a common denominator.

Discussion

Coverage can change the apparent interpretation substantially. If a source lists only a subset of medicines, an analysis restricted to the shared set may look more consistent than the experience of reviewing a complete clinical regimen. Both the shared-set result and the coverage difference should therefore be reported. Neither should substitute for the other.

Missing-as-zero is particularly damaging because it invents observations. It can create false agreement when both sources omit a medicine or false disagreement when one lists a value and the other has no entry. The resulting matrix no longer represents the publications it claims to compare. Preserving missingness is a scientific requirement, not merely a display preference.

High agreement also needs restraint. Sources can share development inputs or inherit classifications, so similar results do not necessarily provide independent confirmation. A comparison should discuss the methods and provenance behind agreement rather than treat consistency as proof of clinical validity. Conversely, disagreement can reveal different scope or evidence rules without making either source useless.

The clinical question remains separate from the classification analysis. A scale that agrees closely with another may not be the most relevant method for a particular population or outcome. A study of agreement does not compare treatment effectiveness and cannot identify which medicine should be changed for a patient.

An honest report should let another analyst reproduce the denominator. This requires the frozen input versions, matching rules, exclusions, missing states, and transformations. A table of coefficients without those details is difficult to interpret and may conceal the decisions that most strongly influenced the result.

Make excluded pairs visible

The comparison report should include a reasoned inventory of records that could not enter the primary matrix. A route mismatch, an unresolved name, and an absent source row are different exclusions with different implications. Reporting only the final shared denominator can conceal these differences. The exclusion inventory would allow a reviewer to determine whether apparent agreement is supported across the intended medication set or depends on a narrow group of easily matched records. It would also identify concrete mapping questions for later work without inventing values for the unresolved pairs.

Limitations

The article does not select a final statistic, perform a sample-size calculation, or analyze the current public rows for agreement. Those choices require the exact scale pair, category definitions, and research objective. No numerical concordance result should be inferred from the existence of this protocol.

Source heterogeneity, incomplete coverage, unresolved products, and nonindependent development histories may limit interpretation even after a careful analysis. A methodologically sound agreement study would still not establish a universal gold standard, an individual clinical risk estimate, or permission to average ordinal scores into a new public value.

Conclusion

Compare only what is genuinely comparable, show the coverage denominator, and preserve every source’s original categories and missing states. Agreement statistics can summarize a defined comparison, but they cannot repair invalid identity matching or replace clinical validation. The protocol’s purpose is transparency, not forced consensus.

References

1. Vennard O, Stewart C, Tolia M, Soiza RL, Myint PK. Anticholinergic Medication Burden Scales: A Systematic Review. J Am Geriatr Soc. 2026;74(6):1771-1784. Source
2. Chew ML, et al. Anticholinergic activity of 107 medications commonly used by older adults. J Am Geriatr Soc. 2008;56:1333-1341. Source
3. Carnahan RM, et al. The Anticholinergic Drug Scale as a measure of drug-related anticholinergic burden: associations with serum anticholinergic activity. J Clin Pharmacol. 2006;46:1481-1486. Source
4. Rudolph JL, et al. The anticholinergic risk scale and anticholinergic adverse effects in older persons. Arch Intern Med. 2008;168:508-513. Source
5. Lisibach A, Benelli V, Ceppi MG, et al. Quality of anticholinergic burden scales and their impact on clinical outcomes: a systematic review. Eur J Clin Pharmacol. 2021;77(2):147-162. Source
6. UnifiedACB. Public evidence dataset. v3.0 RC12 final. Read-only source observations, source registry, and preserved anomaly records. Source

Disclosure

Funding: None. Conflict of interest: None declared for the UnifiedACB educational project.

Educational evidence summary. This content does not provide patient-specific medical advice and does not replace clinical judgment, medication reconciliation, or prescribing guidance.