Preserve BigDecimal precision during JSON ingestion#18976
Open
Vamsi-klu wants to merge 1 commit into
Open
Conversation
Author
|
Review request: @snleee @swaminathanmanish GitHub did not allow this account/integration to add requested reviewers through the reviewer section, so I am tagging the relevant reviewers here for visibility. This PR fixes #12312 by preserving JSON decimal values as Drafted-by: Codex (GPT-5); human-reviewed by Vamsi-klu before posting |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #12312
What changed
This PR preserves exact decimal precision when Pinot ingests JSON records through the built-in JSON ingestion path.
The change adds BigDecimal-aware parsing helpers to
JsonUtilsand routes the default JSON record reader, message decoder, and record extractor through those helpers. Numeric JSON values that cannot be represented exactly asdoublenow remainBigDecimaluntil Pinot's downstream type conversion logic maps them into the target column type.The implementation also keeps the custom extractor compatibility boundary explicit:
JSONRecordExtractornow receives BigDecimal-preserving maps by default.recordExtractorClasskeeps the historicalDoubleparsing behavior by default, so existing custom extractor code is not surprised by new numeric object types.preserveDecimalPrecision=true.Why it matters
JSON ingestion previously parsed decimal literals through Jackson's default floating-point path. That silently rounded values such as high-precision monetary amounts, identifiers encoded as decimal strings, or scientific measurements before Pinot had a chance to apply the configured schema conversion.
That behavior is especially risky because the corruption happens during ingestion and can be hard to detect later: queries see a stable but already-rounded value. Preserving the JSON decimal as
BigDecimalkeeps the source value intact until Pinot decides how to store it for the configured column.The custom extractor compatibility guard matters because many deployments use custom extractors that may inspect raw map values directly. Changing those extractors from
DoubletoBigDecimalwithout an opt-in could break otherwise unrelated ingestion jobs.Impact
Built-in JSON ingestion now keeps decimal precision for values that need it.
Existing ingestion pipelines that use Pinot's built-in JSON extractor get safer precision behavior without requiring table config changes.
Existing custom JSON record extractors keep their previous raw numeric value types unless they explicitly opt into the new precision-preserving map parsing mode.
No query execution behavior is changed by this PR. The change is limited to JSON input decoding and extraction.
Testing
./mvnw -Dmaven.repo.local=/tmp/m2-pinot-12312 -pl pinot-spi,pinot-plugins/pinot-input-format/pinot-json -Dtest=JsonUtilsTest,JSONRecordReaderTest,JSONMessageDecoderTest,JSONRecordExtractorTest test./mvnw -Dmaven.repo.local=/tmp/m2-pinot-12312 -pl pinot-spi,pinot-plugins/pinot-input-format/pinot-json spotless:apply checkstyle:check license:checkgit diff --checkThe tests cover BigDecimal parsing from strings and byte slices, JSON record reader precision preservation, default message-decoder precision preservation, built-in extractor behavior, legacy custom extractor behavior, and explicit custom extractor opt-in.
Drafted-by: Codex (GPT-5); human-reviewed by Vamsi-klu before marking ready for review