Compatibility audit: current behavior probes

This first milestone for #235 records 47 limited probes, not complete original JMESPath or JMESPath Community conformance. It preserves current behavior, including known differences. A passing test means jpx still matches the recorded observations; it does not mean every spec expectation is met.

The machine-readable source of every row is cases.json. It records the exact input, expression, original/Community expectation, source, shared-parser outcome, default/strict evaluation, and focused error diagnostics. The test also checks CLI exit status (0 for values, 1 for errors), stderr, and JSON output. CLI null results produce no stdout; the engine test checks the actual null value independently. Error labels in this audit are classifications for these probes, not a new runtime error API. In particular, jpx currently reports an undefined lexical variable using an unknown-function diagnostic.

Reference baseline

• Original: published specification, plus the existing original compliance fixtures at 53abcc37901891cf4308fcd910eab287416c4609. Rows marked with fixture coordinates are checked against the vendored input, expression, and expectation; those files are not modified. Other rows are locally authored probes based on the spec. Function expectations assume the original standard function set, without custom extension registrations.
• Community: pinned specification repository at bb4a12c5cad990cae3a83491834c447aadc3ceba. Row sources link to that commit, including accepted JEPs. The pinned snapshot was inspected locally; reproduction does not require that checkout or network access.
• Community published specification is a moving publication link, not the reproducibility pin. The pinned consolidated GRAMMAR.ABNF does not yet contain ternary syntax; the ternary expectation here comes specifically from accepted Community JEP-21. This audit does not silently equate proposals, the consolidated grammar, and the published site.

Original JEP-18 is the lexical-scope proposal, tracked by accepted Community JEP-11a. It is not part of the original published grammar used as this baseline. Community JEP-18 instead adds group_by. Community JEP-16 adds arithmetic, JEP-17 adds root $, JEP-19 clarifies pipe/null evaluation, and JEP-21 adds ternary conditionals. Parenthesized grouping is separate from group_by. See the pinned lexical-scope cross-reference and grouping proposal for the numbering distinction.

Grammar, function restrictions, and build features

The shared-parser column is tested with both a runtime containing extensions and Runtime::strict(). Function registration does not select a different parser. The default/strict columns are each tested through JpxEngine and the CLI (--strict for the latter); CLI cases use isolated child configuration. Strict policy removes extension functions and rejects let AST nodes after parsing. It does not guarantee a separate original-only grammar: the bare let identifier failure in both modes is one observable consequence.

jpx-core enables let-expr by default. Normal CLI dependency resolution therefore enables it even though the CLI's own default feature list is empty. The test asserts actual runtime let support and never skips let rows based on cfg!(feature = "let-expr") in the CLI crate. Adding --features let-expr explicitly should produce the same matrix. --no-default-features -p jpx alone does not disable a dependency's default features. Embedders building core without its default features must opt into let-expr; that separate build is not represented by the default/strict columns. The core feature-off lexer check below is only a build-gate probe, not a second conformance matrix.

Reproduce

From the repository root, with the locked Cargo dependencies cached:

cargo test --offline -p jpx --test compatibility_audit
cargo test --offline -p jpx --features let-expr --test compatibility_audit
cargo test --offline -p jpx-core --no-default-features --lib lexer::tests::ensures_eq_invalid_without_let_expr -- --exact
cargo test --offline -p jpx-core --test compliance
python3 scripts/generate_compatibility_audit.py --check

The audit test asserts parser acceptance separately from evaluation, checks values as JSON, and requires specific error substrings instead of accepting any failure. It does not execute another JMESPath implementation as an oracle. The original compliance suite is a separate, broader baseline; these selected CLI/engine probes do not replace it. No Community compliance suite is vendored or executed in this milestone.

After a reviewed fixture change, regenerate the document with:

python3 scripts/generate_compatibility_audit.py

The generator never updates observations from a running binary. --check fails on documentation drift and tells the caller how to regenerate it.

Matrix

Every row is a probe. Inputs are in the fixture. syntax means a grammar rejection; unknown-function means function-call grammar was accepted but the function was unavailable. strict-policy means a parsed let expression was rejected by strict evaluation policy. Value cells show JSON, so null and the string "null" are different results.

Grammar

Probe	Expression	Original expectation	Community expectation	Shared parser	jpx default	jpx strict
ternary-true	true ? 1 : 2	error: syntax source	1 source	reject	error: syntax	error: syntax
arithmetic-add	6 + 2	error: syntax source	8 source	reject	error: syntax	error: syntax
arithmetic-subtract	6 - 2	error: syntax source	4 source	reject	error: syntax	error: syntax
arithmetic-multiply	6 * 2	error: syntax source	12 source	reject	error: syntax	error: syntax
arithmetic-divide	6 / 2	error: syntax source	3 source	reject	error: syntax	error: syntax
arithmetic-modulo	6 % 2	error: syntax source	0 source	reject	error: syntax	error: syntax
arithmetic-integer-divide	6 // 2	error: syntax source	3 source	reject	error: syntax	error: syntax
root-document	$	error: syntax source	{"a":1} source	reject	error: syntax	error: syntax
root-after-pipe	a | $.b	error: syntax source	2 source	reject	error: syntax	error: syntax
parentheses	(false || true) && false	false source	false source	accept	false	false

• ternary-true: Community JEP-21. One branch-selection probe; no claim about precedence or short-circuit conformance.
• arithmetic-add: Community JEP-16; ASCII binary operator probe only.
• arithmetic-subtract: Community JEP-16; ASCII binary operator probe only.
• arithmetic-multiply: Community JEP-16; ASCII binary operator probe only.
• arithmetic-divide: Community JEP-16; ASCII binary operator probe only.
• arithmetic-modulo: Community JEP-16; ASCII binary operator probe only.
• arithmetic-integer-divide: Community JEP-16; ASCII binary operator probe only.
• root-document: Community JEP-17 root reference is distinct from lexical $name variables.
• root-after-pipe: Root should remain the original document after a pipe.
• parentheses: Parenthesized expressions are distinct from the group_by function.

Function

Probe	Expression	Original expectation	Community expectation	Shared parser	jpx default	jpx strict
group-by-expref	group_by(@, &k)	error: unknown-function source	{"a":[{"k":"a"},{"k":"a"}],"b":[{"k":"b"}]} source	accept	{"a":[{"k":"a"},{"k":"a"}],"b":[{"k":"b"}]}	error: unknown-function
group-by-null-key	group_by(@, &k)	error: unknown-function source	{"a":[{"k":"a"}]} source	accept	{"a":[{"k":"a"}],"null":[{"k":null},{}]}	error: unknown-function
group-by-boolean-key	group_by(@, &k)	error: unknown-function source	error: invalid-type source	accept	{"true":[{"k":true}]}	error: unknown-function
group-by-string-argument	group_by(@, 'k')	error: unknown-function source	error: invalid-type source	accept	{"a":[{"k":"a"}]}	error: unknown-function

• group-by-expref: Community JEP-18 is a function addition, not new grammar.
• group-by-null-key: Community discards null/missing keys; jpx retains a "null" group.
• group-by-boolean-key: Community requires string/null keys; jpx stringifies the boolean.
• group-by-string-argument: The string field-name argument is a jpx extension to the Community expref signature.

Let/build

Probe	Expression	Original expectation	Community expectation	Shared parser	jpx default	jpx strict
let-binding	let $x = 1 in $x	error: syntax source	1 source	accept	1	error: strict-policy
let-shadow-restore	let $x = 1 in [let $x = 2 in $x, $x]	error: syntax source	[2,1] source	accept	[2,1]	error: strict-policy
let-stops-projection	let $x = foo[*] in $x[0]	error: syntax source	[0,1] source	accept	[0,1]	error: strict-policy
let-wildcard	let $x = @ in $x[*]	error: syntax source	[1,2] source	accept	[1,2]	error: strict-policy
let-unbound	let $x = 1 in $missing	error: syntax source	error: undefined-variable source	accept	error: unknown-function	error: strict-policy

• let-binding: Original JEP-18 proposal / Community JEP-11a; absent from the original published grammar. Core let-expr is enabled transitively in CLI builds.
• let-shadow-restore: Nested lexical scope and restoration probe.
• let-stops-projection: A bound projection is a value; indexing must not resume that projection.
• let-wildcard: Wildcard access on a variable is measured, not inferred from older guidance.
• let-unbound: Community requires an evaluation-time undefined-variable error. jpx currently formats it as an unknown-function error.

Literal/identifier

Probe	Expression	Original expectation	Community expectation	Shared parser	jpx default	jpx strict
identifier-let	let	7 source	7 source	reject	error: syntax	error: syntax
identifier-in	in	8 source	8 source	accept	8	8
quoted-let	"let"	7 source	7 source	accept	7	7
identifier-booleans-null	[true, false, null]	["field-true","field-false","field-null"] source	["field-true","field-false","field-null"] source	accept	["field-true","field-false","field-null"]	["field-true","field-false","field-null"]
quoted-identifier	"true"	"field-true" source	"field-true" source	accept	"field-true"	"field-true"
raw-string	'true'	"true" source	"true" source	accept	"true"	"true"
json-string	"true"	"true" source	"true" source	accept	"true"	"true"
json-boolean	true	true source	true source	accept	true	true
json-null	null	null source	null source	accept	null	null
malformed-json-literal	{"a":}	error: syntax source	error: syntax source	reject	error: syntax	error: syntax

• identifier-let: Identifiers refer to fields; literals do not. Let/in are specified as contextual keywords.
• identifier-in: Identifiers refer to fields; literals do not. Let/in are specified as contextual keywords.
• quoted-let: Identifiers refer to fields; literals do not. Let/in are specified as contextual keywords.
• identifier-booleans-null: Collision probe: the input deliberately contains fields named true, false and null.
• quoted-identifier: Collision probe: the input deliberately contains fields named true, false and null.
• raw-string: Collision probe: the input deliberately contains fields named true, false and null.
• json-string: Collision probe: the input deliberately contains fields named true, false and null.
• json-boolean: Collision probe: the input deliberately contains fields named true, false and null.
• json-null: Collision probe: the input deliberately contains fields named true, false and null.
• malformed-json-literal: Backtick JSON must be well formed.

Pipe/projection

Probe	Expression	Original expectation	Community expectation	Shared parser	jpx default	jpx strict
projection-index	foo[*][0]	[0,2,4] source	[0,2,4] source	accept	[0,2,4]	[0,2,4]
pipe-stops-projection	foo[*] | [0]	[0,1] source	[0,1] source	accept	[0,1]	[0,1]
projection-omits-null	[*].k	[1] source	[1] source	accept	[1]	[1]
pipe-null-multiselect	null | [@]	null source	[null] source	accept	null	null
dot-null-function	null.type(@)	"null" source	null source	accept	"null"	"null"
pipe-null-function	null | type(@)	"null" source	"null" source	accept	"null"	"null"

• projection-index: Indexing continues the projection.
• pipe-stops-projection: A pipe stops the projection.
• projection-omits-null: Projection omits null results; compare map below.
• pipe-null-multiselect: Original multiselect-list null rule returns null; Community JEP-19 explicitly specifies [null] for this pipe.
• dot-null-function: Original subexpression pseudocode evaluates the RHS; Community JEP-19 adds a null short-circuit. This probes the clarification, not every null context.
• pipe-null-function: A function on the right of a pipe still evaluates on null.

Original reference

Probe	Expression	Original expectation	Community expectation	Shared parser	jpx default	jpx strict
original-parentheses	(Number || True) && False	false source	false source	accept	false	false
original-pipe	foo[].bar[] | [0][0]	{"baz":"one"} source	{"baz":"one"} source	accept	{"baz":"one"}	{"baz":"one"}

• original-parentheses: Original compliance grouping/precedence reference. Original fixture boolean.json, group 2, case 18 (zero-based).
• original-pipe: Original compliance nested projection/pipe reference. Original fixture pipe.json, group 2, case 0 (zero-based).

Standard function

Probe	Expression	Original expectation	Community expectation	Shared parser	jpx default	jpx strict
length-unicode	length('✓foo')	4 source	4 source	accept	4	4
length-type-error	length(false)	error: invalid-type source	error: invalid-type source	accept	error: invalid-type	error: invalid-type
join-element-type	join(',', ["a", 0])	error: invalid-type source	error: invalid-type source	accept	error: invalid-type	error: invalid-type
not-null-arity	not_null()	error: invalid-arity source	error: invalid-arity source	accept	error: invalid-arity	error: invalid-arity
sort-by-expref	sort_by(people, &age)[].name	[3,"a","c","b","d"] source	[3,"a","c","b","d"] source	accept	[3,"a","c","b","d"]	[3,"a","c","b","d"]
map-retains-null	map(&foo.bar, array)	["yes1","yes2",null] source	["yes1","yes2",null] source	accept	["yes1","yes2",null]	["yes1","yes2",null]
length-arity	length()	error: invalid-arity source	error: invalid-arity source	accept	error: invalid-arity	error: invalid-arity
sort-by-string-rejected	sort_by(@, 'k')	error: invalid-type source	error: invalid-type source	accept	error: invalid-type	error: invalid-type
contains-nonstring-search	contains('abc', 1)	false source	false source	accept	false	false
unknown-function	audit_missing(@)	error: unknown-function source	error: unknown-function source	accept	error: unknown-function	error: unknown-function

• length-unicode: length(string|array|object): code points, not UTF-8 bytes. Original fixture functions.json, group 0, case 41 (zero-based).
• length-type-error: length rejects boolean input and names valid types. Original fixture functions.json, group 0, case 48 (zero-based).
• join-element-type: join(string, array[string]) rejects a numeric array member. Original fixture functions.json, group 0, case 89 (zero-based).
• not-null-arity: not_null(any, ...) requires at least one argument. Original fixture functions.json, group 0, case 140 (zero-based).
• sort-by-expref: sort_by(array, expression->number|string). Original fixture functions.json, group 2, case 3 (zero-based).
• map-retains-null: map(expression, array) retains null results, unlike a projection. Original fixture functions.json, group 5, case 0 (zero-based).
• length-arity: Exactly one argument required.
• sort-by-string-rejected: The standard function requires &k, even though jpx group_by also accepts a string.
• contains-nonstring-search: contains(array|string, any): a non-string search in a string returns false.
• unknown-function: Function-call grammar accepts unknown names; evaluation rejects the unavailable function.

Differences recorded by these probes

This list is derived from the fixture expectations, not a support score. It includes differences in rejection phase/class: strict-policy and original syntax errors both reject let expressions, but at different stages.

• Default versus original: group-by-expref, group-by-null-key, group-by-boolean-key, group-by-string-argument, let-binding, let-shadow-restore, let-stops-projection, let-wildcard, let-unbound, identifier-let.
• Default versus community: ternary-true, arithmetic-add, arithmetic-subtract, arithmetic-multiply, arithmetic-divide, arithmetic-modulo, arithmetic-integer-divide, root-document, root-after-pipe, group-by-null-key, group-by-boolean-key, group-by-string-argument, let-unbound, identifier-let, pipe-null-multiselect, dot-null-function.
• Strict versus original: let-binding, let-shadow-restore, let-stops-projection, let-wildcard, let-unbound, identifier-let.
• Strict versus community: ternary-true, arithmetic-add, arithmetic-subtract, arithmetic-multiply, arithmetic-divide, arithmetic-modulo, arithmetic-integer-divide, root-document, root-after-pipe, group-by-expref, group-by-null-key, group-by-boolean-key, group-by-string-argument, let-binding, let-shadow-restore, let-stops-projection, let-wildcard, let-unbound, identifier-let, pipe-null-multiselect, dot-null-function.

Remaining work for #235

• Design a canonical capability registry that separates grammar, evaluation semantics, function availability/signatures, strict policy, and build features. Decide versioned original/Community identifiers and a single source of truth before connecting consumers. This fixture is not that API.
• Audit the exhaustive grammar and operator precedence/associativity, arithmetic types/errors, Unicode operators, ternary truthiness and lazy branches, root/current/lexical scope, contextual keywords, escaping, identifier collisions, slices, and every projection/null context.
• Pin and integrate the Community compliance corpus, resolve discrepancies between accepted JEPs and consolidated/published specs, and expand original compliance coverage across strict/default and core feature configurations.
• Audit all standard and extension function signatures, error classes, evaluation order, aliases and overrides; verify CLI, engine, MCP and Python parity and configuration paths. Representative probes do not establish this.
• Decide and implement any behavior changes separately, then update reviewed expectations. No new syntax, modes, registry APIs, engine_info fields, MCP metadata, dependencies or runtime changes are introduced here.