fix: update Java Comparator to read from test_results table#1272
Merged
misrasaurabh1 merged 1 commit intoFeb 3, 2026
Conversation
The Comparator was reading from an `invocations` table, but Java instrumentation writes to a `test_results` table. This aligns the Comparator with the cross-language schema consistency requirement. Changes: - Update SQL query to SELECT from test_results table - Map columns: iteration_id + loop_index → call_id - Map return_value → resultJson for comparison - Construct method_id from test_class_name.function_getting_tested - Add parseIterationId() helper to extract numeric ID from string format - Set args_json and error_json to null (not captured in test_results schema) This enables behavior verification to work correctly by reading the data that instrumented tests actually write. Test results: All 336 Java tests pass (18 comparator tests + 318 others) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
misrasaurabh1
approved these changes
Feb 3, 2026
mashraf-222
pushed a commit
that referenced
this pull request
Feb 3, 2026
Added comprehensive integration tests to validate that the Java Comparator correctly reads from the test_results table schema. New test class: TestTestResultsTableSchema with 5 tests: - test_comparator_reads_test_results_table_identical Validates identical results are correctly compared - test_comparator_reads_test_results_table_different_values Detects when return values differ between original and candidate - test_comparator_handles_multiple_loop_iterations Tests multiple benchmark loops with different loop_index values - test_comparator_iteration_id_parsing Validates parseIterationId() correctly parses "iter_testIteration" format - test_comparator_missing_result_in_candidate Detects when candidate is missing results that exist in original Test features: - Creates actual test_results table with instrumentation schema - Tests full SQL integration path through Java Comparator - Validates column mapping: iteration_id → call_id, return_value → result_json - Uses @requires_java decorator to skip gracefully when Java unavailable - Documents expected schema for future developers - Prevents regressions if table name changes back to invocations These tests validate the fix in PR #1272 that updated the Comparator to read from test_results instead of invocations. Test results: 18 passed, 5 skipped (without Java) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
mashraf-222
pushed a commit
that referenced
this pull request
Feb 3, 2026
Resolved conflicts in pom.xml by accepting the omni-java version which has: - Updated artifactId: codeflash-test-fixture - Newer JUnit version: 5.10.0 - Simplified dependencies: junit-jupiter + junit-jupiter-params - Newer maven-surefire-plugin: 3.1.2 This brings the branch up to date with the latest changes from omni-java including the Comparator schema fix from PR #1272.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
test_resultstableProblem
The Java Comparator was reading from an
invocationstable with schema:call_id, method_id, args_json, result_json, error_jsonBut Java instrumentation writes to a
test_resultstable with schema:test_module_path, test_class_name, test_function_name, function_getting_tested, loop_index, iteration_id, runtime, return_value, verification_typeThis prevented behavior verification from working because the Comparator couldn't find any instrumented test data.
Solution
Updated
Comparator.javato read from thetest_resultstable that instrumentation actually writes to. This aligns with the requirement for consistent schema across all languages.Changes
codeflash-java-runtime/src/main/java/com/codeflash/Comparator.java
getInvocations() method:
SELECT ... FROM invocationstoSELECT ... FROM test_resultsloop_index + iteration_id→call_id(unique identifier)return_value→resultJson(JSON result to compare)test_class_name + "." + function_getting_tested→methodIdargs_json→null(not captured in test_results)error_json→null(not captured in test_results)New helper method:
parseIterationId(): Extracts numeric iteration from string format "iter_testIteration" (e.g., "1_0" → 1)call_idas(loop_index * 10000) + iteration_numberTesting
✅ All 336 Java tests pass (7 skipped)
✅ All 18 comparator tests pass
✅ Integration tests verify correct schema mapping
Impact
This fix is critical for Java optimization:
Notes
The
test_resultsschema doesn't captureargs_jsonorerror_json, so those fields are set tonullin the Invocation objects. This is acceptable because:🤖 Generated with Claude Code