Class: HTTPClient

Class | HTTPClient
In: | lib/httpclient/connection.rb lib/httpclient/auth.rb lib/httpclient/util.rb lib/httpclient/session.rb lib/httpclient/ssl_config.rb lib/httpclient/timeout.rb lib/httpclient.rb
Parent: | Object

The HTTPClient class provides several methods for accessing Web resources via HTTP.

HTTPClient instance is designed to be MT-safe. You can call a HTTPClient instance from several threads without synchronization after setting up an instance.

      clnt = HTTPClient.new
      clnt.set_cookie_store('/home/nahi/cookie.dat')
      urls.each do |url|
        Thread.new(url) do |u|
          p clnt.head(u).status
        end
      end

## How to use

At first, how to create your client. See initialize for more detail.

1. Create simple client.

        clnt = HTTPClient.new

2. Accessing resources through HTTP proxy. You can use environment variable ‘http_proxy’ or ‘HTTP_PROXY’ instead.

        clnt = HTTPClient.new('http://myproxy:8080')

### How to retrieve web resources

See get_content.

1. Get content of specified URL. It returns a String of whole result.

        puts clnt.get_content('http://dev.ctor.org/')

2. Get content as chunks of String. It yields chunks of String.

        clnt.get_content('http://dev.ctor.org/') do |chunk|
          puts chunk
        end

### Invoking other HTTP methods

See head, get, post, put, delete, options, propfind, proppatch and trace. It returns a HTTP::Message instance as a response.

1. Do HEAD request.

        res = clnt.head(uri)
        p res.header['Last-Modified'][0]

2. Do GET request with query.

        query = { 'keyword' => 'ruby', 'lang' => 'en' }
        res = clnt.get(uri, query)
        p res.status
        p res.contenttype
        p res.header['X-Custom']
        puts res.content

### How to POST

See post.

1. Do POST a form data.

        body = { 'keyword' => 'ruby', 'lang' => 'en' }
        res = clnt.post(uri, body)

2. Do multipart file upload with POST. No need to set extra header by yourself from httpclient/2.1.4.

        File.open('/tmp/post_data') do |file|
          body = { 'upload' => file, 'user' => 'nahi' }
          res = clnt.post(uri, body)
        end

### Accessing via SSL

Ruby needs to be compiled with OpenSSL.

1. Get content of specified URL via SSL. Just pass an URL which starts with ‘https://’.

        https_url = 'https://www.rsa.com'
        clnt.get_content(https_url)

2. Getting peer certificate from response.

        res = clnt.get(https_url)
        p res.peer_cert #=> returns OpenSSL::X509::Certificate

3. Configuring OpenSSL options. See HTTPClient::SSLConfig for more details.

        user_cert_file = 'cert.pem'
        user_key_file = 'privkey.pem'
        clnt.ssl_config.set_client_cert_file(user_cert_file, user_key_file)
        clnt.get_content(https_url)

### Handling Cookies

1. Using volatile Cookies. Nothing to do. HTTPClient handles Cookies.

        clnt = HTTPClient.new
        clnt.get_content(url1) # receives Cookies.
        clnt.get_content(url2) # sends Cookies if needed.

2. Saving non volatile Cookies to a specified file. Need to set a file at first and invoke save method at last.

        clnt = HTTPClient.new
        clnt.set_cookie_store('/home/nahi/cookie.dat')
        clnt.get_content(url)
        ...
        clnt.save_cookie_store

3. Disabling Cookies.

        clnt = HTTPClient.new
        clnt.cookie_manager = nil

### Configuring authentication credentials

1. Authentication with Web server. Supports BasicAuth, DigestAuth, and Negotiate/NTLM (requires ruby/ntlm module).

        clnt = HTTPClient.new
        domain = 'http://dev.ctor.org/http-access2/'
        user = 'user'
        password = 'user'
        clnt.set_auth(domain, user, password)
        p clnt.get_content('http://dev.ctor.org/http-access2/login').status

2. Authentication with Proxy server. Supports BasicAuth and NTLM (requires win32/sspi)

        clnt = HTTPClient.new(proxy)
        user = 'proxy'
        password = 'proxy'
        clnt.set_proxy_auth(user, password)
        p clnt.get_content(url)

### Invoking HTTP methods with custom header

Pass a Hash or an Array for extheader argument.

        extheader = { 'Accept' => '*/*' }
        clnt.get_content(uri, query, extheader)

        extheader = [['Accept', 'image/jpeg'], ['Accept', 'image/png']]
        clnt.get_content(uri, query, extheader)

### Invoking HTTP methods asynchronously

See head_async, get_async, post_async, put_async, delete_async, options_async, propfind_async, proppatch_async, and trace_async. It immediately returns a HTTPClient::Connection instance as a returning value.

        connection = clnt.post_async(url, body)
        print 'posting.'
        while true
          break if connection.finished?
          print '.'
          sleep 1
        end
        puts '.'
        res = connection.pop
        p res.status
        p res.content.read # res.content is an IO for the res of async method.

### Shortcut methods

You can invoke get_content, get, etc. without creating HTTPClient instance.

      ruby -rhttpclient -e 'puts HTTPClient.get_content(ARGV.shift)' http://dev.ctor.org/
      ruby -rhttpclient -e 'p HTTPClient.head(ARGV.shift).header["last-modified"]' http://dev.ctor.org/

### Methods

debug_dev debug_dev= default_redirect_uri_callback delete delete_async get get_async get_content head head_async new no_proxy no_proxy= options options_async post post_async post_content propfind propfind_async proppatch proppatch_async proxy proxy= put put_async redirect_uri_callback= request request_async reset reset_all save_cookie_store set_auth set_basic_auth set_cookie_store set_proxy_auth strict_redirect_uri_callback timeout_scheduler trace trace_async

### Included Modules

Util

### Classes and Modules

Module HTTPClient::DebugSocket
Module HTTPClient::SocketWrap
Module HTTPClient::Timeout
Module HTTPClient::Util
Class HTTPClient::AuthFilterBase
Class HTTPClient::BadResponseError
Class HTTPClient::BasicAuth
Class HTTPClient::ConfigurationError
Class HTTPClient::ConnectTimeoutError
Class HTTPClient::Connection
Class HTTPClient::DigestAuth
Class HTTPClient::LoopBackSocket
Class HTTPClient::NegotiateAuth
Class HTTPClient::ProxyAuth
Class HTTPClient::ReceiveTimeoutError
Class HTTPClient::SSLConfig
Class HTTPClient::SSLSocketWrap
Class HTTPClient::SSPINegotiateAuth
Class HTTPClient::SendTimeoutError
Class HTTPClient::Session
Class HTTPClient::SessionManager
Class HTTPClient::Site
Class HTTPClient::TimeoutError
Class HTTPClient::TimeoutScheduler
Class HTTPClient::WWWAuth

### Constants

NTLMEnabled | = | true
NTLMEnabled | = | false
SSPIEnabled | = | true
SSPIEnabled | = | false
SSLEnabled | = | true
SSLEnabled | = | false
VERSION | = | '2.1.4'
RUBY_VERSION_STRING | = | "ruby #{RUBY_VERSION} (#{RUBY_RELEASE_DATE}) [#{RUBY_PLATFORM}]"
LIB_NAME | = | "(#{$1}/#{$2}, #{RUBY_VERSION_STRING})"
PROPFIND_DEFAULT_EXTHEADER | = | { 'Depth' => '0' } |  | Default extheader for PROPFIND request.
NO_PROXY_HOSTS | = | ['localhost']

### Attributes

cookie_manager | [RW] |
WebAgent::CookieManager: | Cookies configurator.
follow_redirect_count | [RW] | How many times get_content and post_content follows HTTP redirect. 10 by default.
proxy_auth | [R] |
HTTPClient::ProxyAuth: | Proxy authentication handler.
request_filter | [R] | An array of request filter which can trap HTTP request/response. See HTTPClient::WWWAuth to see how to use it.
ssl_config | [R] |
HTTPClient::SSLConfig: | SSL configurator.
test_loopback_response | [R] | An array of response HTTP message body String which is used for loop-back test. See test/* to see how to use it. If you want to do loop-back test of HTTP header, use test_loopback_http_response instead.
www_auth | [R] |
HTTPClient::WWWAuth: | WWW authentication handler.

### Public Class methods

new(*args)

Creates a HTTPClient instance which manages sessions, cookies, etc.

HTTPClient.new takes 3 optional arguments for proxy url string, User-Agent String and From header String. User-Agent and From are embedded in HTTP request Header if given. No User-Agent and From header added without setting it explicitly.

      proxy = 'http://myproxy:8080'
      agent_name = 'MyAgent/0.1'
      from = 'from@example.com'
      HTTPClient.new(proxy, agent_name, from)

You can use a keyword argument style Hash. Keys are :proxy, :agent_name and :from.

      HTTPClient.new(:agent_name = 'MyAgent/0.1')

timeout_scheduler()

CAUTION: caller must aware of race condition.

### Public Instance methods

debug_dev()

Returns debug device if exists. See debug_dev=.

debug_dev=(dev)

Sets debug device. Once debug device is set, all HTTP requests and responses are dumped to given device. dev must respond to << for dump.

Calling this method resets all existing sessions.

default_redirect_uri_callback(uri, res)

A default method for redirect uri callback. This method is used by HTTPClient instance by default. This callback allows relative redirect such as

      Location: ../foo/

in HTTP header.

delete(uri, extheader = {}, &block)

Sends DELETE request to the specified URL. See request for arguments.

delete_async(uri, extheader = {})

Sends DELETE request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

get(uri, query = nil, extheader = {}, &block)

Sends GET request to the specified URL. See request for arguments.

get_async(uri, query = nil, extheader = {})

Sends GET request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

get_content(uri, query = nil, extheader = {}, &block)

Retrieves a web resource.

uri: | a String or an URI object which represents an URL of web resource.
query: | a Hash or an Array of query part of URL. e.g. { "a" => "b" } => ‘host/part?a=b’. Give an array to pass multiple value like ["a", "b"], ["a", "c"] => ‘host/part?a=b&a=c’.
extheader: | a Hash or an Array of extra headers. e.g. { ‘Accept’ => ’*/*’ } or [[‘Accept’, ‘image/jpeg’], [‘Accept’, ‘image/png’]].
&block: | Give a block to get chunked message-body of response like get_content(uri) { |chunked_body| … }. Size of each chunk may not be the same.

get_content follows HTTP redirect status (see HTTP::Status.redirect?) internally and try to retrieve content from redirected URL. See redirect_uri_callback= how HTTP redirection is handled.

If you need to get full HTTP response including HTTP status and headers, use get method. get returns HTTP::Message as a response and you need to follow HTTP redirect by yourself if you need.

head(uri, query = nil, extheader = {})

Sends HEAD request to the specified URL. See request for arguments.

head_async(uri, query = nil, extheader = {})

Sends HEAD request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

no_proxy()

Returns NO_PROXY setting String if given.

no_proxy=(no_proxy)

Sets NO_PROXY setting String. no_proxy must be a comma separated String. Each entry must be ‘host’ or ‘host:port’ such as; HTTPClient#no_proxy = ‘example.com,example.co.jp:443‘

‘localhost’ is treated as a no_proxy site regardless of explicitly listed. HTTPClient checks given URI objects before accessing it. ‘host’ is tail string match. No IP-addr conversion.

You can use environment variable ‘no_proxy’ or ‘NO_PROXY’ for it.

Calling this method resets all existing sessions.

options(uri, extheader = {}, &block)

Sends OPTIONS request to the specified URL. See request for arguments.

options_async(uri, extheader = {})

Sends OPTIONS request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

post(uri, body = nil, extheader = {}, &block)

Sends POST request to the specified URL. See request for arguments.

post_async(uri, body = nil, extheader = {})

Sends POST request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

post_content(uri, body = nil, extheader = {}, &block)

Posts a content.

uri: | a String or an URI object which represents an URL of web resource.
body: | a Hash or an Array of body part. e.g. { "a" => "b" } => ‘a=b’. Give an array to pass multiple value like ["a", "b"], ["a", "c"] => ‘a=b&a=c’. When you pass a File as a value, it will be posted as a multipart/form-data. e.g. { ‘upload’ => file }
extheader: | a Hash or an Array of extra headers. e.g. { ‘Accept’ => ’*/*’ } or [[‘Accept’, ‘image/jpeg’], [‘Accept’, ‘image/png’]].
&block: | Give a block to get chunked message-body of response like post_content(uri) { |chunked_body| … }. Size of each chunk may not be the same.

post_content follows HTTP redirect status (see HTTP::Status.redirect?) internally and try to post the content to redirected URL. See redirect_uri_callback= how HTTP redirection is handled.

If you need to get full HTTP response including HTTP status and headers, use post method.

propfind(uri, extheader = PROPFIND_DEFAULT_EXTHEADER, &block)

Sends PROPFIND request to the specified URL. See request for arguments.

propfind_async(uri, extheader = PROPFIND_DEFAULT_EXTHEADER)

Sends PROPFIND request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

proppatch(uri, body = nil, extheader = {}, &block)

Sends PROPPATCH request to the specified URL. See request for arguments.

proppatch_async(uri, body = nil, extheader = {})

Sends PROPPATCH request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

proxy()

Returns URI object of HTTP proxy if exists.

proxy=(proxy)

Sets HTTP proxy used for HTTP connection. Given proxy can be an URI, a String or nil. You can set user/password for proxy authentication like HTTPClient#proxy = ‘user:passwd@myproxy:8080‘

You can use environment variable ‘http_proxy’ or ‘HTTP_PROXY’ for it. You need to use ‘cgi_http_proxy’ or ‘CGI_HTTP_PROXY’ instead if you run HTTPClient from CGI environment from security reason. (HTTPClient checks ‘REQUEST_METHOD’ environment variable whether it‘s CGI or not)

Calling this method resets all existing sessions.

put(uri, body = nil, extheader = {}, &block)

Sends PUT request to the specified URL. See request for arguments.

put_async(uri, body = nil, extheader = {})

Sends PUT request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

redirect_uri_callback=(redirect_uri_callback)

Sets callback proc when HTTP redirect status is returned for get_content and post_content. default_redirect_uri_callback is used by default.

If you need strict implementation which does not allow relative URI redirection, set strict_redirect_uri_callback instead.

      clnt.redirect_uri_callback = clnt.method(:strict_redirect_uri_callback)

request(method, uri, query = nil, body = nil, extheader = {}, &block)

Sends a request to the specified URL.

method: | HTTP method to be sent. method.to_s.upcase is used.
uri: | a String or an URI object which represents an URL of web resource.
query: | a Hash or an Array of query part of URL. e.g. { "a" => "b" } => ‘host/part?a=b’ Give an array to pass multiple value like ["a", "b"], ["a", "c"] => ‘host/part?a=b&a=c‘
body: | a Hash or an Array of body part. e.g. { "a" => "b" } => ‘a=b’. Give an array to pass multiple value like ["a", "b"], ["a", "c"] => ‘a=b&a=c’. When the given method is ‘POST’ and the given body contains a file as a value, it will be posted as a multipart/form-data. e.g. { ‘upload’ => file } See HTTP::Message.file? for actual condition of ‘a file’.
extheader: | a Hash or an Array of extra headers. e.g. { ‘Accept’ => ’*/*’ } or [[‘Accept’, ‘image/jpeg’], [‘Accept’, ‘image/png’]].
&block: | Give a block to get chunked message-body of response like get(uri) { |chunked_body| … }. Size of each chunk may not be the same.

You can also pass a String as a body. HTTPClient just sends a String as a HTTP request message body.

When you pass an IO as a body, HTTPClient sends it as a HTTP request with chunked encoding (Transfer-Encoding: chunked in HTTP header). Bear in mind that some server application does not support chunked request. At least cgi.rb does not support it.

request_async(method, uri, query = nil, body = nil, extheader = {})

Sends a request in async style. request method creates new Thread for HTTP connection and returns a HTTPClient::Connection instance immediately.

Arguments definition is the same as request.

reset(uri)

Resets internal session for the given URL. Keep-alive connection for the site (host-port pair) is disconnected if exists.

reset_all()

Resets all of internal sessions. Keep-alive connections are disconnected.

save_cookie_store()

Try to save Cookies to the file specified in set_cookie_store. Unexpected error will be raised if you don‘t call set_cookie_store first. (interface mismatch between WebAgent::CookieManager implementation)

set_auth(domain, user, passwd)

Sets credential for Web server authentication.

domain: | a String or an URI to specify where HTTPClient should use this

          credential.  If you set uri to nil, HTTPClient uses this credential
          wherever a server requires it.

user: | username String.
passwd: | password String.

You can set multiple credentials for each uri.

      clnt.set_auth('http://www.example.com/foo/', 'foo_user', 'passwd')
      clnt.set_auth('http://www.example.com/bar/', 'bar_user', 'passwd')

Calling this method resets all existing sessions.

set_basic_auth(domain, user, passwd)

Deprecated. Use set_auth instead.

set_cookie_store(filename)

Sets the filename where non-volatile Cookies be saved by calling save_cookie_store. This method tries to load and managing Cookies from the specified file.

Calling this method resets all existing sessions.

set_proxy_auth(user, passwd)

Sets credential for Proxy authentication.

user: | username String.
passwd: | password String.

Calling this method resets all existing sessions.

strict_redirect_uri_callback(uri, res)

A method for redirect uri callback. How to use:

      clnt.redirect_uri_callback = clnt.method(:strict_redirect_uri_callback)

This callback does not allow relative redirect such as

      Location: ../foo/

in HTTP header. (raises BadResponseError instead)

trace(uri, query = nil, body = nil, extheader = {}, &block)

Sends TRACE request to the specified URL. See request for arguments.

trace_async(uri, query = nil, body = nil, extheader = {})

Sends TRACE request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

[Validate]