Bug #4421: [ext/openssl] Fix RSA public key encoding
http://redmine.ruby-lang.org/issues/show/4421

Author: Martin Bosslet
Status: Open, Priority: Normal
Category: ext, Target version: 1.9.3
ruby -v: ruby 1.9.2p180 (2011-02-18 revision 30909) [i686-linux]

When calling RSA#to_der and RSA#to_pem on RSA public keys, they currently=

get encoded using i2d_RSAPublicKey and PEM_write_bio_RSAPublicKey. This e=
ncoding
was specified in PKCS#1 and is specific to RSA. It is also not the defaul=
t =

encoding used by OpenSSL itself, which rather uses the generic format gen=
erated
by i2d_RSA_PUBKEY and PEM_write_bio_RSA_PUBKEY. This format is the same t=
hat is
used in a certificate's SubjectPublicKeyInfo, the advantage being that th=
e format
is generic and can be used to represent public keys of all kinds, includi=
ng RSA,
DSA and Elliptic Curve.

The attached patch will make use of the generic format for encoding RSA k=
eys. The
change should not cause compatibility problems, since RSA#initialize uses=
 several
fallback scenarios that cover public keys of both formats.

The fallbacks are also re-prioritized according to these changes.

Regards,
Martin


----------------------------------------
http://redmine.ruby-lang.org
SW5kZXg6IHJ1YnkvZXh0L29wZW5zc2wvb3NzbF9wa2V5X3JzYS5jCj09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT0KLS0tIHJ1YnkvZXh0L29wZW5zc2wvb3NzbF9w
a2V5X3JzYS5jCShyZXZpc2lvbiAzMDkzOCkKKysrIHJ1YnkvZXh0L29wZW5z
c2wvb3NzbF9wa2V5X3JzYS5jCSh3b3JraW5nIGNvcHkpCkBAIC0xNTgsMjMg
KzE1OCwyMyBAQAogCXJzYSA9IFBFTV9yZWFkX2Jpb19SU0FQcml2YXRlS2V5
KGluLCBOVUxMLCBvc3NsX3BlbV9wYXNzd2RfY2IsIHBhc3N3ZCk7CiAJaWYg
KCFyc2EpIHsKIAkgICAgKHZvaWQpQklPX3Jlc2V0KGluKTsKLQkgICAgcnNh
ID0gUEVNX3JlYWRfYmlvX1JTQVB1YmxpY0tleShpbiwgTlVMTCwgTlVMTCwg
TlVMTCk7CisJICAgIHJzYSA9IFBFTV9yZWFkX2Jpb19SU0FfUFVCS0VZKGlu
LCBOVUxMLCBOVUxMLCBOVUxMKTsKIAl9CiAJaWYgKCFyc2EpIHsKIAkgICAg
KHZvaWQpQklPX3Jlc2V0KGluKTsKLQkgICAgcnNhID0gUEVNX3JlYWRfYmlv
X1JTQV9QVUJLRVkoaW4sIE5VTEwsIE5VTEwsIE5VTEwpOworCSAgICByc2Eg
PSBkMmlfUlNBUHJpdmF0ZUtleV9iaW8oaW4sIE5VTEwpOwogCX0KIAlpZiAo
IXJzYSkgewogCSAgICAodm9pZClCSU9fcmVzZXQoaW4pOwotCSAgICByc2Eg
PSBkMmlfUlNBUHJpdmF0ZUtleV9iaW8oaW4sIE5VTEwpOworCSAgICByc2Eg
PSBkMmlfUlNBX1BVQktFWV9iaW8oaW4sIE5VTEwpOwogCX0KIAlpZiAoIXJz
YSkgewogCSAgICAodm9pZClCSU9fcmVzZXQoaW4pOwotCSAgICByc2EgPSBk
MmlfUlNBUHVibGljS2V5X2JpbyhpbiwgTlVMTCk7CisJICAgIHJzYSA9IFBF
TV9yZWFkX2Jpb19SU0FQdWJsaWNLZXkoaW4sIE5VTEwsIE5VTEwsIE5VTEwp
OwogCX0KIAlpZiAoIXJzYSkgewogCSAgICAodm9pZClCSU9fcmVzZXQoaW4p
OwotCSAgICByc2EgPSBkMmlfUlNBX1BVQktFWV9iaW8oaW4sIE5VTEwpOwor
CSAgICByc2EgPSBkMmlfUlNBUHVibGljS2V5X2JpbyhpbiwgTlVMTCk7CiAJ
fQogCUJJT19mcmVlKGluKTsKIAlpZiAoIXJzYSkgb3NzbF9yYWlzZShlUlNB
RXJyb3IsICJOZWl0aGVyIFBVQiBrZXkgbm9yIFBSSVYga2V5OiIpOwpAQCAt
MjYwLDcgKzI2MCw3IEBACiAJICAgIG9zc2xfcmFpc2UoZVJTQUVycm9yLCBO
VUxMKTsKIAl9CiAgICAgfSBlbHNlIHsKLQlpZiAoIVBFTV93cml0ZV9iaW9f
UlNBUHVibGljS2V5KG91dCwgcGtleS0+cGtleS5yc2EpKSB7CisJaWYgKCFQ
RU1fd3JpdGVfYmlvX1JTQV9QVUJLRVkob3V0LCBwa2V5LT5wa2V5LnJzYSkp
IHsKIAkgICAgQklPX2ZyZWUob3V0KTsKIAkgICAgb3NzbF9yYWlzZShlUlNB
RXJyb3IsIE5VTEwpOwogCX0KQEAgLTI4OSw3ICsyODksNyBAQAogICAgIGlm
KFJTQV9IQVNfUFJJVkFURShwa2V5LT5wa2V5LnJzYSkpCiAJaTJkX2Z1bmMg
PSBpMmRfUlNBUHJpdmF0ZUtleTsKICAgICBlbHNlCi0JaTJkX2Z1bmMgPSBp
MmRfUlNBUHVibGljS2V5OworCWkyZF9mdW5jID0gKGludCAoKikoY29uc3Qg
UlNBKiwgdW5zaWduZWQgY2hhcioqKSlpMmRfUlNBX1BVQktFWTsKICAgICBp
ZigobGVuID0gaTJkX2Z1bmMocGtleS0+cGtleS5yc2EsIE5VTEwpKSA8PSAw
KQogCW9zc2xfcmFpc2UoZVJTQUVycm9yLCBOVUxMKTsKICAgICBzdHIgPSBy
Yl9zdHJfbmV3KDAsIGxlbik7Cgo=