Skip to content

JRuby isn't escaping strings correctly #870

Closed
#871
Closed
JRuby isn't escaping strings correctly#870
#871
Labels
jruby

Description

@nevans
nevans
opened on Oct 3, 2025

I've encounted a weird issue that I don't know how to debug. This is using JRuby 10.0.2.0. I used the version installed by ruby-build using rbenv install.

There's a rake task in ruby/net-imap that generates some code (mostly regular expressions and hashes) from RFC tables. First it parses the RFC and writes the data into a JSON file. Then it reads that JSON file and generates the code. It was broken into two parts so rake can conditionally run the generator only if the JSON or the generator itself has changed.

Anyway, it's not generating the JSON correctly, and I can't figure out why. To replicate:

From a clone of https://github.com/ruby/net-imap, run bundle exec rake saslprep_rb.

To dig deeper, run irb, and from that irb console, do the following:

load "./rakelib/string_prep_tables_generator.rb"
generator = StringPrepTablesGenerator.new
rfc_path  = generator.rfc_filename
rfc_text  = File.read rfc_path
parsed    = generator.send(:parse_rfc_text, rfc_text)

This parsed variable now holds the data that will be generated and written to disk and then reread in a later step. But a couple of strings toward the bottom show the issue:

irb(main):052> d_1 = parsed["titles"]["D.1"]
=> "Characters with bidirectional property \"R\" or \"AL\""
irb(main):053> d_1 == d_1.dump.undump
=> true
irb(main):054> d_1.bytes == d_1.dump.undump.bytes
=> true
irb(main):055> d_1.encoding == d_1.dump.undump.encoding
=> true
irb(main):060> puts JSON.generate d_1.dump.undump
"Characters with bidirectional property \"R\" or \"AL\""
=> nil
irb(main):061> puts JSON.generate d_1
"Characters with bidirectional property "R" or "AL\""
=> nil

Same with:

irb(main):065> d_2 = parsed["titles"]["D.2"]
=> "Characters with bidirectional property \"L\""
irb(main):067> puts JSON.generate d_2.dump.undump
"Characters with bidirectional property \"L\""
=> nil
irb(main):068> puts JSON.generate d_2
"Characters with bidirectional property "L\""
=> nil

What's going on with the quoting there?! What's different about the string and its dump.undump counterpart?

Activity

  1. nevans
    added 7 commits that reference this issue on Oct 3, 2025

    🐛 Workaround JRuby JSON encoding error

    eddd9b5

    🩹 Workaround JRuby JSON encoding error

    37e5d67

    🩹 Workaround JRuby JSON encoding error

    5cabbc7

    🩹 Workaround JRuby JSON encoding error

    1053ecd

    🩹 Workaround JRuby JSON encoding error

    5c6db36

    🩹 Workaround JRuby JSON encoding error

    fdb517d

    🩹 Workaround JRuby JSON encoding error

    d21f006
  2. byroot
    added
    jruby
    on Oct 4, 2025

  3. byroot commented on Oct 4, 2025

    @byroot
    byroot
    on Oct 4, 2025
    Member

    I'm sorry but I have a bit of a hard time understanding the issue (granted it's early here).

    Anyway you could reduce this further? E.g. dump the problematic string in a byte array so you can rebuild it in a simple standalone script or something?

    Because at that point it's not clear if there is a problem with how JSON created the string or if it's an issue with JRuby's dump/undump methods.

  4. nevans commented on Oct 4, 2025

    @nevans
    nevans
    on Oct 4, 2025
    Author

    I'm sorry but I have a bit of a hard time understanding the issue

    Me too! It's weird!

    Anyway you could reduce this further? E.g. dump the problematic string in a byte array so you can rebuild it in a simple standalone script or something?

    That's the part where I got stumped: the bad string and good string have the exact same byte array and the same encoding. I didn't print out the bytes because the string is exactly what it looks like: all of the characters are ASCII.

    There must be some underlying JRuby string representation differences, but I don't know how to debug that.

    I could probably eventually reduce the code to a bare-bones reproduction script, but... it was late on a Friday, and it can be reproduced with only a few lines (in the net-imap repo).

    Because at that point it's not clear if there is a problem with how JSON created the string or if it's an issue with JRuby's dump/undump methods.

    I used "dump.undump" to create an identical (surface level) string without the problematic (underlying hidden) state. Also, because #dump does something very similar to JSON string encoding, I wanted to see if it also exhibits the bug (it doesn't). So, I don't know if the bug is in JRuby's json encoder or if its handling of global regexp match vars ($1, $2, etc) created a string with corrupted internals... or something else entirely. But, other methods (like String#dump) seem unbothered by the string, so I'm guessing JRuby has different valid internal string representations and the bug is in the JSON encoder for one of them.

  5. byroot commented on Oct 4, 2025

    @byroot
    byroot
    on Oct 4, 2025
    Member

    Yeah, that's likely require too much JRuby specific knowledge for me to figure out, so I'll defer to @headius here, but I think you could help him help you by reducing the repro further when you get the time.

  6. nevans commented on Oct 4, 2025

    @nevans
    nevans
    on Oct 4, 2025
    Author

    I was gonna say, I'll get to that later. But I was able to try out my simple hypothesis while lying in bed using my phone (termux => ssh to laptop with JRuby). 🙂

    irb(main):001* 'String with text: "I\'m extracting this text."  This too.' =~
irb(main):002>   /text: (".*")  (\w.*)/
=> 12
irb(main):003> quoted = $1
=> "\"I'm extracting this text.\""
irb(main):004> other = $2
=> "This too."
irb(main):005> require "json"
=> true

irb(main):009> puts JSON.pretty_generate quoted: quoted.dump.undump, other: other.dump.undump
{
  "quoted": "\"I'm extracting this text.\"",
  "other": "This too."
}
=> nil
irb(main):010> puts JSON.pretty_generate quoted:, other:
{
  "quoted": ""I'm extracting this text.\"",
  "other": "This too."
}
=> nil

  7. nevans commented on Oct 4, 2025

    @nevans
    nevans
    on Oct 4, 2025
    Author

    So it looks to me like it is somehow related to the regexp match extraction. FWIW, global vars vs MatchData makes no difference:

    irb(main):011> match = 'String with text: "I\'m extracting this text."  This too.'.match(/text: (".*")  (\w.*)/)
=>
#<MatchData
...
irb(main):012> puts JSON.pretty_generate quoted: match[1], other: match[2]
{
  "quoted": ""I'm extracting this text.\"",
  "other": "This too."
}
=> nil

  8. tompng commented on Oct 4, 2025

    @tompng
    tompng
    on Oct 4, 2025
    Member

    Maybe it is related to shared substring

    irb(main):001> puts 'abcd"efg'[2, 4].to_json
"cd"e"
=> nil
  9. byroot
    closed this as completedin #871on Oct 5, 2025

  10. nevans commented on Oct 6, 2025

    @nevans
    nevans
    on Oct 6, 2025
    Author

    Thanks, @tompng! 🙂

  11. headius commented on Oct 6, 2025

    @headius
    headius
    on Oct 6, 2025
    Contributor

    Thanks for picking up the slack folks!

  12. nevans
    added a commit that references this issue on Oct 22, 2025

    🩹 Workaround JRuby JSON encoding error

    889e908
  13. nevans
    added a commit that references this issue on Nov 29, 2025

    🩹 Workaround JRuby JSON encoding error

    2b52b9a
  14. nevans
    added a commit that references this issue on Dec 10, 2025

    🩹 Workaround JRuby JSON encoding error

    7dabd1f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    jruby

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions