------=_NextPart_000_0272_01C7DF24.CE2A8BE0
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: 7bit

> -----Original Message-----
> From: list-bounce / example.com 
> [mailto:list-bounce / example.com] On Behalf Of Haze Noc
> Sent: Wednesday, August 15, 2007 10:05 AM
> To: ruby-talk ML
> Subject: HTTP headers and source
> 
> Ok guys, lets say i wanted to grab the source for google.com 
> or something... it wont allow if unless i send the correct 
> headers to spoof the program.. Can anyone give me a working 
> example of how to send headers and download a webpage source?
> 
> I tried looking through all of the docs and coming up with 
> something but i failed...
> 
> Thanks for any reply's
> --
> Posted via http://www.ruby-forum.com/.
>

With open-uri[0] you can open URIs just like local files. That would be
entirely sufficient to get the content of the index page of google.com, for
example. Instead of a simple URL you can also pass the open call a URI[1]
object, for which you can explicitly call headers if you need to.

You could then also also use Hpricot[2] to do all sorts of nifty HTML
parsing


[0] - http://www.ruby-doc.org/stdlib/libdoc/open-uri/rdoc/
[1] - http://www.ruby-doc.org/stdlib/libdoc/uri/rdoc/index.html
[2] - http://code.whytheluckystiff.net/hpricot/

Felix

------=_NextPart_000_0272_01C7DF24.CE2A8BE0
Content-Type: application/x-pkcs7-signature;
	name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
	filename="smime.p7s"

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIII3jCCAmYw
ggHPoAMCAQICEFJ3uelO4AJUNG7Np2ieTWkwDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UEBhMCWkEx
JTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQ
ZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA3MDMxMTEzMzk1OVoXDTA4MDMxMDEzMzk1
OVowSjEfMB0GA1UEAxMWVGhhd3RlIEZyZWVtYWlsIE1lbWJlcjEnMCUGCSqGSIb3DQEJARYYZndt
YWlsaW5nbGlzdHNAZ21haWwuY29tMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDoHQNMSMIi
1fb3BhJdz7Byt8PSeaa/Yx0wJ1BzLj7dJZK+8djBP06HvolZ9IYibnGerNg+LnxXT8tyGUZ0vw0Q
f3qSYtuGcYfdm44qVOAtV+fex14uBSmSj8i/QOL+710oSQdPGAJE1L+8N1bLvJAxznsF6tpD5GUZ
iPRY1HcloQIDAQABozUwMzAjBgNVHREEHDAagRhmd21haWxpbmdsaXN0c0BnbWFpbC5jb20wDAYD
VR0TAQH/BAIwADANBgkqhkiG9w0BAQUFAAOBgQCtXsHSAFBpcohm7/t5Vlak0eySI5c5cghVELhu
LyoNMdx+qCyKE4HJoWrr2+id/+YVPfiOaBsxsir3Aeg3DPuCODoafFYErm3OzJV13lRlwKWfrz01
vczScC2VvABk0ZvlQEHA54IJqiV8QoXntlQWaa2NYO3ggalUyaOeNmsk1TCCAy0wggKWoAMCAQIC
AQAwDQYJKoZIhvcNAQEEBQAwgdExCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUx
EjAQBgNVBAcTCUNhcGUgVG93bjEaMBgGA1UEChMRVGhhd3RlIENvbnN1bHRpbmcxKDAmBgNVBAsT
H0NlcnRpZmljYXRpb24gU2VydmljZXMgRGl2aXNpb24xJDAiBgNVBAMTG1RoYXd0ZSBQZXJzb25h
bCBGcmVlbWFpbCBDQTErMCkGCSqGSIb3DQEJARYccGVyc29uYWwtZnJlZW1haWxAdGhhd3RlLmNv
bTAeFw05NjAxMDEwMDAwMDBaFw0yMDEyMzEyMzU5NTlaMIHRMQswCQYDVQQGEwJaQTEVMBMGA1UE
CBMMV2VzdGVybiBDYXBlMRIwEAYDVQQHEwlDYXBlIFRvd24xGjAYBgNVBAoTEVRoYXd0ZSBDb25z
dWx0aW5nMSgwJgYDVQQLEx9DZXJ0aWZpY2F0aW9uIFNlcnZpY2VzIERpdmlzaW9uMSQwIgYDVQQD
ExtUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgQ0ExKzApBgkqhkiG9w0BCQEWHHBlcnNvbmFsLWZy
ZWVtYWlsQHRoYXd0ZS5jb20wgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBANRp19SwlGRbcelH
2AxRtupykbCEXn0tDY97Et+FJXUodDpCLGMnn5V7S+9+GYcdhuqj3bnOlmQawhRuRKx85o/oTQ9x
H0A4pgCjh3j2+ZSGXq3qwF5269kUo11uenwMpUtVfwYZKX+emibVars4JAhqmMex2qOYkf152+Va
xBy5AgMBAAGjEzARMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQEEBQADgYEAx+ySfk749Zal
Z2IqpPBNEWDQb41gWGGsJrtSNVwIzzD7qEqWih9iQiOMFw/0umScF6xHKd+dmF7SbGBxXKKs3Hnj
524ARx+1DSjoAp3kmv0T9KbZfLH43F8jJgmRgHPQFBveQ6mDJfLmnC8Vyv6mq4oHdYsM3VGEa+T4
0c53ooEwggM/MIICqKADAgECAgENMA0GCSqGSIb3DQEBBQUAMIHRMQswCQYDVQQGEwJaQTEVMBMG
A1UECBMMV2VzdGVybiBDYXBlMRIwEAYDVQQHEwlDYXBlIFRvd24xGjAYBgNVBAoTEVRoYXd0ZSBD
b25zdWx0aW5nMSgwJgYDVQQLEx9DZXJ0aWZpY2F0aW9uIFNlcnZpY2VzIERpdmlzaW9uMSQwIgYD
VQQDExtUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgQ0ExKzApBgkqhkiG9w0BCQEWHHBlcnNvbmFs
LWZyZWVtYWlsQHRoYXd0ZS5jb20wHhcNMDMwNzE3MDAwMDAwWhcNMTMwNzE2MjM1OTU5WjBiMQsw
CQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UE
AxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0EwgZ8wDQYJKoZIhvcNAQEBBQAD
gY0AMIGJAoGBAMSmPFVzVftOucqZWh5owHUEcJ3f6f+jHuy9zfVb8hp2vX8MOmHyv1HOAdTlUAow
1wJjWiyJFXCO3cnwK4Vaqj9xVsuvPAsH5/EfkTYkKhPPK9Xzgnc9A74r/rsYPge/QIACZNenpruf
ZdHFKlSFD0gEf6e20TxhBEAeZBlyYLf7AgMBAAGjgZQwgZEwEgYDVR0TAQH/BAgwBgEB/wIBADBD
BgNVHR8EPDA6MDigNqA0hjJodHRwOi8vY3JsLnRoYXd0ZS5jb20vVGhhd3RlUGVyc29uYWxGcmVl
bWFpbENBLmNybDALBgNVHQ8EBAMCAQYwKQYDVR0RBCIwIKQeMBwxGjAYBgNVBAMTEVByaXZhdGVM
YWJlbDItMTM4MA0GCSqGSIb3DQEBBQUAA4GBAEiM0VCD6gsuzA2jZqxnD3+vrL7CF6FDlpSdf0wh
uPg2H6otnzYvwPQcUCCTcDz9reFhYsPZOhl+hLGZGwDFGguCdJ4lUJRix9sncVcljd2pnDmOjCBP
ZV+V2vf3h9bGCE6u9uo05RAaWzVNd+NWIXiC3CEZNd4ksdMdRv9dX2VPMYIC+DCCAvQCAQEwdjBi
MQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoG
A1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECEFJ3uelO4AJUNG7Np2ie
TWkwCQYFKw4DAhoFAKCCAdgwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUx
DxcNMDcwODE1MTcxMjM3WjAjBgkqhkiG9w0BCQQxFgQUwJYhlz406RpVwOVGBzDFtyExgNgwZwYJ
KoZIhvcNAQkPMVowWDAKBggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAw
BwYFKw4DAgcwDQYIKoZIhvcNAwICASgwBwYFKw4DAhowCgYIKoZIhvcNAgUwgYUGCSsGAQQBgjcQ
BDF4MHYwYjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0
ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBAhBSd7npTuAC
VDRuzadonk1pMIGHBgsqhkiG9w0BCRACCzF4oHYwYjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRo
YXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVl
bWFpbCBJc3N1aW5nIENBAhBSd7npTuACVDRuzadonk1pMA0GCSqGSIb3DQEBAQUABIGAZ+dQdSLi
AcEZ+1w2t7wb+YGdL+32Zom02j33j4xrwdjDsJ/Eff1qgxsLDFcN7kE/yUtGHhXkk0wzjGTWfO0E
OuAizkKoBafURuwqnr5VVch5wPTFuwyeWOMLc60KrbSninJKGTAnDIkStMV3AVxDHA1/IMzV51eW
RZYZCkSlnJkAAAAAAAA=

------=_NextPart_000_0272_01C7DF24.CE2A8BE0--