TLS server with an EC key fails on 0.19.x (chooseAlias doesn't match BCJSSE key types)

On 0.19.3, a TLS server with an EC certificate fails every handshake; RSA works. 0.16.2 works with both.

require "openssl"
require "socket"

key = ARGV[0] == "rsa" ? OpenSSL::PKey::RSA.new(2048) : OpenSSL::PKey::EC.generate("prime256v1")
cert = OpenSSL::X509::Certificate.new
cert.version = 2
cert.serial = 1
cert.subject = cert.issuer = OpenSSL::X509::Name.parse("/CN=localhost")
cert.public_key = key
cert.not_before = Time.now - 60
cert.not_after = Time.now + 3600
cert.sign(key, "SHA256")

ctx = OpenSSL::SSL::SSLContext.new(:TLSv1_2)
ctx.cert = cert
ctx.key = key
server = OpenSSL::SSL::SSLServer.new(TCPServer.new("127.0.0.1", 0), ctx)
Thread.new { c = server.accept; c.puts(c.gets); c.close }

client = OpenSSL::SSL::SSLSocket.new(TCPSocket.new("127.0.0.1", server.to_io.addr[1]),
                                     OpenSSL::SSL::SSLContext.new(:TLSv1_2))
client.connect
client.puts("ping")
client.gets
puts "OK #{client.ssl_version}"

jruby repro.rb rsa prints OK TLSv1.2; jruby repro.rb ec fails (JRuby 9.4.15.0, OpenJDK 21):

OpenSSL::SSL::SSLError: org.bouncycastle.tls.TlsFatalAlert: handshake_failure(40); [server #2 @…] found no selectable cipher suite among …
OpenSSL::SSL::SSLError: Socket closed

The same happens with TLS 1.3 and with P-384 keys.

Cause: 0.19 prefers BCJSSE (SecurityHelper.java#L556-L569). KeyManagerImpl.chooseAlias requires an exact key-type match (SSLContext.java#L1242-L1252), but BCJSSE asks for ECDHE_ECDSA (TLS 1.2) or EC/secp256r1 etc. (TLS 1.3), never plain EC, so no credential is found.

Normalising the key type in chooseAlias (strip /<group> and check the key's curve matches it; map ECDHE_ECDSA to EC, ECDHE_RSA/DHE_RSA to RSA, DHE_DSS to DSA) fixed it in a local build of 0.19.3 for P-256 and P-384 on TLS 1.2 and 1.3.

Clarification on when this fails:

Only when the server context selects BCJSSE, i.e. an explicit TLS version (SSLContext.new(:TLSv1_2) / ssl_version=), or always in the FIPS variant. A plain SSLContext.new keeps protocol "SSL" and uses SunJSSE, which works (that's why test_add_certificate_multiple_certs passes).
Affected server keys: EC (P-256, P-384) and Ed25519, on TLS 1.2 and 1.3. RSA works.

Fix and a regression test in #384.

Thanks Thomas, we'll have to take a closer look.

Custom alias choosing was added due BC-JSSE (to support add_certificate: ee71fd9) but maybe there's another path to approach that.

We've expected BC's JSSE engine to be mostly a drop in replacement for the Sun built-in provider but seems like incompatibilities are piling up - will have to investigate the behavioral differences.

Navigation Menu

Repository navigation

TLS server with an EC key fails on 0.19.x (chooseAlias doesn't match BCJSSE key types) #383

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

TLS server with an EC key fails on 0.19.x (chooseAlias doesn't match BCJSSE key types) #383

Description

Activity

toabctl commented on Oct 8, 2026

kares commented on Oct 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions