Skip to content

Preserve BigDecimal precision during JSON ingestion#18976

Open
Vamsi-klu wants to merge 1 commit into
apache:masterfrom
Vamsi-klu:fix-bigdecimal-json-ingestion
Open

Preserve BigDecimal precision during JSON ingestion#18976
Vamsi-klu wants to merge 1 commit into
apache:masterfrom
Vamsi-klu:fix-bigdecimal-json-ingestion

Conversation

@Vamsi-klu

@Vamsi-klu Vamsi-klu commented Jul 11, 2026

Copy link
Copy Markdown

Fixes #12312

What changed

This PR preserves exact decimal precision when Pinot ingests JSON records through the built-in JSON ingestion path.

The change adds BigDecimal-aware parsing helpers to JsonUtils and routes the default JSON record reader, message decoder, and record extractor through those helpers. Numeric JSON values that cannot be represented exactly as double now remain BigDecimal until Pinot's downstream type conversion logic maps them into the target column type.

The implementation also keeps the custom extractor compatibility boundary explicit:

  • The built-in JSONRecordExtractor now receives BigDecimal-preserving maps by default.
  • A user-supplied custom recordExtractorClass keeps the historical Double parsing behavior by default, so existing custom extractor code is not surprised by new numeric object types.
  • Custom extractors can opt into BigDecimal-preserving parsing with preserveDecimalPrecision=true.

Why it matters

JSON ingestion previously parsed decimal literals through Jackson's default floating-point path. That silently rounded values such as high-precision monetary amounts, identifiers encoded as decimal strings, or scientific measurements before Pinot had a chance to apply the configured schema conversion.

That behavior is especially risky because the corruption happens during ingestion and can be hard to detect later: queries see a stable but already-rounded value. Preserving the JSON decimal as BigDecimal keeps the source value intact until Pinot decides how to store it for the configured column.

The custom extractor compatibility guard matters because many deployments use custom extractors that may inspect raw map values directly. Changing those extractors from Double to BigDecimal without an opt-in could break otherwise unrelated ingestion jobs.

Impact

Built-in JSON ingestion now keeps decimal precision for values that need it.

Existing ingestion pipelines that use Pinot's built-in JSON extractor get safer precision behavior without requiring table config changes.

Existing custom JSON record extractors keep their previous raw numeric value types unless they explicitly opt into the new precision-preserving map parsing mode.

No query execution behavior is changed by this PR. The change is limited to JSON input decoding and extraction.

Testing

  • ./mvnw -Dmaven.repo.local=/tmp/m2-pinot-12312 -pl pinot-spi,pinot-plugins/pinot-input-format/pinot-json -Dtest=JsonUtilsTest,JSONRecordReaderTest,JSONMessageDecoderTest,JSONRecordExtractorTest test
  • ./mvnw -Dmaven.repo.local=/tmp/m2-pinot-12312 -pl pinot-spi,pinot-plugins/pinot-input-format/pinot-json spotless:apply checkstyle:check license:check
  • git diff --check

The tests cover BigDecimal parsing from strings and byte slices, JSON record reader precision preservation, default message-decoder precision preservation, built-in extractor behavior, legacy custom extractor behavior, and explicit custom extractor opt-in.


Drafted-by: Codex (GPT-5); human-reviewed by Vamsi-klu before marking ready for review

@Vamsi-klu Vamsi-klu marked this pull request as ready for review July 11, 2026 23:38
@Vamsi-klu

Copy link
Copy Markdown
Author

Review request: @snleee @swaminathanmanish

GitHub did not allow this account/integration to add requested reviewers through the reviewer section, so I am tagging the relevant reviewers here for visibility.

This PR fixes #12312 by preserving JSON decimal values as BigDecimal through Pinot's built-in JSON ingestion path, while keeping custom recordExtractorClass implementations on the legacy Double parsing behavior unless they opt in with preserveDecimalPrecision=true. Review from the ingestion/Kafka-decoder side would be especially useful to confirm the compatibility boundary and whether the opt-in behavior is appropriate for custom extractors.


Drafted-by: Codex (GPT-5); human-reviewed by Vamsi-klu before posting

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Data getting Truncated when using BigDecimal datatype

1 participant