fix: add JSON-aware comparison to Python comparator fallback

[mashraf-222]

Summary

• Fixed JSON comparison bug in Python fallback comparator
• Prevents false negatives when comparing semantically identical JSON
• Matches Java Comparator behavior for cross-language consistency

Problem

The compare_invocations_directly() function in comparator.py was using simple string comparison for JSON results:

elif orig_result != cand_result:  # Line 311

This caused false negatives when JSON was semantically identical but formatted differently:

Example False Negative:

• Original: {"a":1,"b":2}
• Candidate: { "b": 2, "a": 1 }

These are semantically identical but fail string comparison due to:

1. Different whitespace
2. Different key ordering

The Java Comparator handles this correctly by parsing JSON and doing deep equality, but the Python fallback did not.

Solution

Added JSON-aware comparison to match Java Comparator behavior:

New Helper Function: _compare_json_values()

def _compare_json_values(json1: str | None, json2: str | None) -> bool:
    # 1. Handle None values
    if json1 is None and json2 is None:
        return True
    if json1 is None or json2 is None:
        return False
    
    # 2. Fast path: exact string match
    if json1 == json2:
        return True
    
    # 3. Parse JSON and compare objects
    try:
        obj1 = json.loads(json1)
        obj2 = json.loads(json2)
        return obj1 == obj2
    except (json.JSONDecodeError, TypeError):
        # 4. Fallback to string comparison if parsing fails
        return json1 == json2

Updated Comparison Logic:

elif not _compare_json_values(orig_result, cand_result):
    # Results differ (using JSON-aware comparison)

What This Fixes

✅ Whitespace differences - {"a":1} == { "a": 1 }
✅ Key ordering - {"a":1,"b":2} == {"b":2,"a":1}
✅ Nested objects - Deep equality on nested JSON structures
✅ Numeric types - 42 == 42.0 (Python's JSON behavior)
✅ Invalid JSON - Gracefully falls back to string comparison

❌ Array ordering - [1,2,3] != [3,2,1] (correct behavior - order matters)

Tests Added

Updated Existing Test:

• test_whitespace_in_json - Now expects True (was incorrectly expecting False)

New Test Class: TestJsonComparison (8 tests)

1. test_json_key_ordering_difference

   • {"a":1,"b":2,"c":3} == {"c":3,"a":1,"b":2}

2. test_json_whitespace_and_ordering_combined

   • Tests combined whitespace and key order differences

3. test_json_nested_object_comparison

   • {"outer":{"inner":{"value":123}}} with whitespace variations

4. test_json_array_comparison_order_matters

   • Ensures [1,2,3] != [3,2,1] (order matters)

5. test_json_invalid_json_falls_back_to_string

   • Non-JSON strings compared correctly

6. test_json_null_vs_string_null

   • JSON null value comparison

7. test_json_empty_object_vs_null

   • {} != null (correctly different)

8. test_json_numeric_equivalence

   • {"value":42} == {"value":42.0}

Impact

Before (Buggy):

# These would fail comparison even though they're identical
original = '{"name":"test","value":42}'
candidate = '{ "value": 42, "name": "test" }'
# Result: FALSE NEGATIVE - reports as different

After (Fixed):

# Same example now correctly recognized as identical
original = '{"name":"test","value":42}'
candidate = '{ "value": 42, "name": "test" }'
# Result: CORRECT - reports as equivalent

Testing

✅ All 344 Java tests pass (7 skipped)
✅ All 26 comparator tests pass (18 existing + 8 new)
✅ No regressions - existing tests still pass
✅ Comprehensive coverage - edge cases, invalid JSON, nested objects

Why This Matters

1. Prevents False Negatives - Won't incorrectly reject valid optimizations
2. Cross-Language Consistency - Python fallback matches Java Comparator
3. Better User Experience - Fewer spurious failures in behavior verification
4. Robustness - Handles real-world JSON variations gracefully

🤖 Generated with Claude Code

[KRRT7]

@claude format and push

[claude]

Claude finished @KRRT7's task in 44s —— View job

---

Tasks

• Read repository setup and formatting guidelines
• Check current git status
• Attempt to run code formatter

Status

I attempted to run the code formatter, but I don't have permission to execute uv run ruff format commands.

The working tree is currently clean with no uncommitted changes, which suggests that either:

1. The code is already formatted correctly, or
2. The formatting needs to be done locally by someone with the appropriate permissions

What needs to be done

To format and push the code, you can run these commands locally:

# Format the code
uv run ruff format codeflash/ tests/

# Check for linting issues
uv run ruff check codeflash/ tests/

# If there are changes, commit and push
git add -A
git commit -m "chore: format code with ruff

Co-authored-by: Kevin Turcios <KRRT7@users.noreply.github.com>"
git push origin fix/java-comparator-json-comparison

Alternatively, if you'd like me to have the permissions to run these commands, you can update the --allowedTools or --allowedPrompts configuration for the Claude Code Action to include formatter commands.

--- | Branch: fix/java-comparator-json-comparison