2 # $Id: LWP.pm,v 1.1.1.1 2003/08/02 23:39:28 takezoe Exp $
7 sub Version { $VERSION; }
10 require LWP::UserAgent; # this should load everything you need
18 LWP - The World-Wide Web library for Perl
23 print "This is libwww-perl-$LWP::VERSION\n";
28 The libwww-perl collection is a set of Perl modules which provides a
29 simple and consistent application programming interface (API) to the
30 World-Wide Web. The main focus of the library is to provide classes
31 and functions that allow you to write WWW clients. The library also
32 contain modules that are of more general use and even classes that
33 help you implement simple HTTP servers.
35 Most modules in this library provide an object oriented API. The user
36 agent, requests sent and responses received from the WWW server are
37 all represented by objects. This makes a simple and powerful
38 interface to these services. The interface is easy to extend
39 and customize for your own needs.
41 The main features of the library are:
47 Contains various reusable components (modules) that can be
48 used separately or together.
52 Provides an object oriented model of HTTP-style communication. Within
53 this framework we currently support access to http, https, gopher, ftp, news,
54 file, and mailto resources.
58 Provides a full object oriented interface or
59 a very simple procedural interface.
63 Supports the basic and digest authorization schemes.
67 Supports transparent redirect handling.
71 Supports access through proxy servers.
75 Provides parser for F<robots.txt> files and a framework for constructing robots.
79 Implements HTTP content negotiation algorithm that can
80 be used both in protocol modules and in server scripts (like CGI
85 Supports HTTP cookies.
89 A simple command line client application called C<lwp-request>.
94 =head1 HTTP STYLE COMMUNICATION
97 The libwww-perl library is based on HTTP style communication. This
98 section tries to describe what that means.
100 Let us start with this quote from the HTTP specification document
101 <URL:http://www.w3.org/pub/WWW/Protocols/>:
107 The HTTP protocol is based on a request/response paradigm. A client
108 establishes a connection with a server and sends a request to the
109 server in the form of a request method, URI, and protocol version,
110 followed by a MIME-like message containing request modifiers, client
111 information, and possible body content. The server responds with a
112 status line, including the message's protocol version and a success or
113 error code, followed by a MIME-like message containing server
114 information, entity meta-information, and possible body content.
118 What this means to libwww-perl is that communication always take place
119 through these steps: First a I<request> object is created and
120 configured. This object is then passed to a server and we get a
121 I<response> object in return that we can examine. A request is always
122 independent of any previous requests, i.e. the service is stateless.
123 The same simple model is used for any kind of service we want to
126 For example, if we want to fetch a document from a remote file server,
127 then we send it a request that contains a name for that document and
128 the response will contain the document itself. If we access a search
129 engine, then the content of the request will contain the query
130 parameters and the response will contain the query result. If we want
131 to send a mail message to somebody then we send a request object which
132 contains our message to the mail server and the response object will
133 contain an acknowledgment that tells us that the message has been
134 accepted and will be forwarded to the recipient(s).
136 It is as simple as that!
139 =head2 The Request Object
141 The libwww-perl request object has the class name C<HTTP::Request>.
142 The fact that the class name uses C<HTTP::> as a
143 prefix only implies that we use the HTTP model of communication. It
144 does not limit the kind of services we can try to pass this I<request>
145 to. For instance, we will send C<HTTP::Request>s both to ftp and
146 gopher servers, as well as to the local file system.
148 The main attributes of the request objects are:
154 The B<method> is a short string that tells what kind of
155 request this is. The most common methods are B<GET>, B<PUT>,
160 The B<uri> is a string denoting the protocol, server and
161 the name of the "document" we want to access. The B<uri> might
162 also encode various other parameters.
166 The B<headers> contain additional information about the
167 request and can also used to describe the content. The headers
168 are a set of keyword/value pairs.
172 The B<content> is an arbitrary amount of data.
176 =head2 The Response Object
178 The libwww-perl response object has the class name C<HTTP::Response>.
179 The main attributes of objects of this class are:
185 The B<code> is a numerical value that indicates the overall
186 outcome of the request.
190 The B<message> is a short, human readable string that
191 corresponds to the I<code>.
195 The B<headers> contain additional information about the
196 response and describe the content.
200 The B<content> is an arbitrary amount of data.
204 Since we don't want to handle all possible I<code> values directly in
205 our programs, a libwww-perl response object has methods that can be
206 used to query what kind of response this is. The most commonly used
207 response classification methods are:
213 The request was was successfully received, understood or accepted.
217 The request failed. The server or the resource might not be
218 available, access to the resource might be denied or other things might
219 have failed for some reason.
223 =head2 The User Agent
225 Let us assume that we have created a I<request> object. What do we
226 actually do with it in order to receive a I<response>?
228 The answer is that you pass it to a I<user agent> object and this
229 object takes care of all the things that need to be done
230 (like low-level communication and error handling) and returns
231 a I<response> object. The user agent represents your
232 application on the network and provides you with an interface that
233 can accept I<requests> and return I<responses>.
235 The user agent is an interface layer between
236 your application code and the network. Through this interface you are
237 able to access the various servers on the network.
239 The class name for the user agent is C<LWP::UserAgent>. Every
240 libwww-perl application that wants to communicate should create at
241 least one object of this class. The main method provided by this
242 object is request(). This method takes an C<HTTP::Request> object as
243 argument and (eventually) returns a C<HTTP::Response> object.
245 The user agent has many other attributes that let you
246 configure how it will interact with the network and with your
253 The B<timeout> specifies how much time we give remote servers to
254 respond before the library disconnects and creates an
255 internal I<timeout> response.
259 The B<agent> specifies the name that your application should use when it
260 presents itself on the network.
264 The B<from> attribute can be set to the e-mail address of the person
265 responsible for running the application. If this is set, then the
266 address will be sent to the servers with every request.
270 The B<parse_head> specifies whether we should initialize response
271 headers from the E<lt>head> section of HTML documents.
275 The B<proxy> and B<no_proxy> attributes specify if and when to go through
276 a proxy server. <URL:http://www.w3.org/pub/WWW/Proxies/>
280 The B<credentials> provide a way to set up user names and
281 passwords needed to access certain services.
285 Many applications want even more control over how they interact
286 with the network and they get this by sub-classing
287 C<LWP::UserAgent>. The library includes a
288 sub-class, C<LWP::RobotUA>, for robot applications.
292 This example shows how the user agent, a request and a response are
293 represented in actual perl code:
295 # Create a user agent object
297 $ua = LWP::UserAgent->new;
298 $ua->agent("MyApp/0.1 ");
301 my $req = HTTP::Request->new(POST => 'http://www.perl.com/cgi-bin/BugGlimpse');
302 $req->content_type('application/x-www-form-urlencoded');
303 $req->content('match=www&errors=0');
305 # Pass request to the user agent and get a response back
306 my $res = $ua->request($req);
308 # Check the outcome of the response
309 if ($res->is_success) {
312 print "Bad luck this time\n";
315 The $ua is created once when the application starts up. New request
316 objects should normally created for each request sent.
319 =head1 NETWORK SUPPORT
321 This section discusses the various protocol schemes and
322 the HTTP style methods that headers may be used for each.
324 For all requests, a "User-Agent" header is added and initialized from
325 the $ua->agent attribute before the request is handed to the network
326 layer. In the same way, a "From" header is initialized from the
329 For all responses, the library adds a header called "Client-Date".
330 This header holds the time when the response was received by
331 your application. The format and semantics of the header are the
332 same as the server created "Date" header. You may also encounter other
333 "Client-XXX" headers. They are all generated by the library
334 internally and are not received from the servers.
338 HTTP requests are just handed off to an HTTP server and it
339 decides what happens. Few servers implement methods beside the usual
340 "GET", "HEAD", "POST" and "PUT", but CGI-scripts may implement
341 any method they like.
343 If the server is not available then the library will generate an
344 internal error response.
346 The library automatically adds a "Host" and a "Content-Length" header
347 to the HTTP request before it is sent over the network.
349 For GET request you might want to add a "If-Modified-Since" or
350 "If-None-Match" header to make the request conditional.
352 For POST request you should add the "Content-Type" header. When you
353 try to emulate HTML E<lt>FORM> handling you should usually let the value
354 of the "Content-Type" header be "application/x-www-form-urlencoded".
355 See L<lwpcook> for examples of this.
357 The libwww-perl HTTP implementation currently support the HTTP/1.1
358 and HTTP/1.0 protocol.
360 The library allows you to access proxy server through HTTP. This
361 means that you can set up the library to forward all types of request
362 through the HTTP protocol module. See L<LWP::UserAgent> for
363 documentation of this.
366 =head2 HTTPS Requests
368 HTTPS requests are HTTP requests over an encrypted network connection
369 using the SSL protocol developed by Netscape. Everything about HTTP
370 requests above also apply to HTTPS requests. In addition the library
371 will add the headers "Client-SSL-Cipher", "Client-SSL-Cert-Subject" and
372 "Client-SSL-Cert-Issuer" to the response. These headers denote the
373 encryption method used and the name of the server owner.
375 The request can contain the header "If-SSL-Cert-Subject" in order to
376 make the request conditional on the content of the server certificate.
377 If the certificate subject does not match, no request is sent to the
378 server and an internally generated error response is returned. The
379 value of the "If-SSL-Cert-Subject" header is interpreted as a Perl
385 The library currently supports GET, HEAD and PUT requests. GET
386 retrieves a file or a directory listing from an FTP server. PUT
387 stores a file on a ftp server.
389 You can specify a ftp account for servers that want this in addition
390 to user name and password. This is specified by including an "Account"
391 header in the request.
393 User name/password can be specified using basic authorization or be
394 encoded in the URL. Failed logins return an UNAUTHORIZED response with
395 "WWW-Authenticate: Basic" and can be treated like basic authorization
398 The library supports ftp ASCII transfer mode by specifying the "type=a"
399 parameter in the URL. It also supports transfer of ranges for FTP transfers
400 using the "Range" header.
402 Directory listings are by default returned unprocessed (as returned
403 from the ftp server) with the content media type reported to be
404 "text/ftp-dir-listing". The C<File::Listing> module provides methods
405 for parsing of these directory listing.
407 The ftp module is also able to convert directory listings to HTML and
408 this can be requested via the standard HTTP content negotiation
409 mechanisms (add an "Accept: text/html" header in the request if you
412 For normal file retrievals, the "Content-Type" is guessed based on the
413 file name suffix. See L<LWP::MediaTypes>.
415 The "If-Modified-Since" request header works for servers that implement
416 the MDTM command. It will probably not work for directory listings though.
420 $req = HTTP::Request->new(GET => 'ftp://me:passwd@ftp.some.where.com/');
421 $req->header(Accept => "text/html, */*;q=0.1");
425 Access to the USENET News system is implemented through the NNTP
426 protocol. The name of the news server is obtained from the
427 NNTP_SERVER environment variable and defaults to "news". It is not
428 possible to specify the hostname of the NNTP server in news: URLs.
430 The library supports GET and HEAD to retrieve news articles through the
431 NNTP protocol. You can also post articles to newsgroups by using
432 (surprise!) the POST method.
434 GET on newsgroups is not implemented yet.
438 $req = HTTP::Request->new(GET => 'news:abc1234@a.sn.no');
440 $req = HTTP::Request->new(POST => 'news:comp.lang.perl.test');
441 $req->header(Subject => 'This is a test',
442 From => 'me@some.where.org');
443 $req->content(<<EOT);
444 This is the content of the message that we are sending to
449 =head2 Gopher Request
451 The library supports the GET and HEAD methods for gopher requests. All
452 request header values are ignored. HEAD cheats and returns a
453 response without even talking to server.
455 Gopher menus are always converted to HTML.
457 The response "Content-Type" is generated from the document type
458 encoded (as the first letter) in the request URL path itself.
462 $req = HTTP::Request->new(GET => 'gopher://gopher.sn.no/');
468 The library supports GET and HEAD methods for file requests. The
469 "If-Modified-Since" header is supported. All other headers are
470 ignored. The I<host> component of the file URL must be empty or set
471 to "localhost". Any other I<host> value will be treated as an error.
473 Directories are always converted to an HTML document. For normal
474 files, the "Content-Type" and "Content-Encoding" in the response are
475 guessed based on the file suffix.
479 $req = HTTP::Request->new(GET => 'file:/etc/passwd');
482 =head2 Mailto Request
484 You can send (aka "POST") mail messages using the library. All
485 headers specified for the request are passed on to the mail system.
486 The "To" header is initialized from the mail address in the URL.
490 $req = HTTP::Request->new(POST => 'mailto:libwww@perl.org');
491 $req->header(Subject => "subscribe");
492 $req->content("Please subscribe me to the libwww-perl mailing list!\n");
495 =head1 OVERVIEW OF CLASSES AND PACKAGES
497 This table should give you a quick overview of the classes provided by the
498 library. Indentation shows class inheritance.
500 LWP::MemberMixin -- Access to member variables of Perl5 classes
501 LWP::UserAgent -- WWW user agent class
502 LWP::RobotUA -- When developing a robot applications
503 LWP::Protocol -- Interface to various protocol schemes
504 LWP::Protocol::http -- http:// access
505 LWP::Protocol::file -- file:// access
506 LWP::Protocol::ftp -- ftp:// access
509 LWP::Authen::Basic -- Handle 401 and 407 responses
512 HTTP::Headers -- MIME/RFC822 style header (used by HTTP::Message)
513 HTTP::Message -- HTTP style message
514 HTTP::Request -- HTTP request
515 HTTP::Response -- HTTP response
516 HTTP::Daemon -- A HTTP server class
518 WWW::RobotRules -- Parse robots.txt files
519 WWW::RobotRules::AnyDBM_File -- Persistent RobotRules
521 Net::HTTP -- Low level HTTP client
523 The following modules provide various functions and definitions.
525 LWP -- This file. Library version number and documentation.
526 LWP::MediaTypes -- MIME types configuration (text/html etc.)
527 LWP::Debug -- Debug logging module
528 LWP::Simple -- Simplified procedural interface for common functions
529 HTTP::Status -- HTTP status code (200 OK etc)
530 HTTP::Date -- Date parsing module for HTTP date formats
531 HTTP::Negotiate -- HTTP content negotiation calculation
532 File::Listing -- Parse directory listings
533 HTML::Form -- Processing for <form>s in HTML documents
536 =head1 MORE DOCUMENTATION
538 All modules contain detailed information on the interfaces they
539 provide. The I<lwpcook> manpage is the libwww-perl cookbook that contain
540 examples of typical usage of the library. You might want to take a
541 look at how the scripts C<lwp-request>, C<lwp-rget> and C<lwp-mirror>
546 The following environment variables are used by LWP:
552 The C<LWP::MediaTypes> functions will look for the F<.media.types> and
553 F<.mime.types> files relative to you home directory.
563 These environment variables can be set to enable communication through
564 a proxy server. See the description of the C<env_proxy> method in
567 =item PERL_LWP_USE_HTTP_10
569 Enable the old HTTP/1.0 protocol driver instead of the new HTTP/1.1
570 driver. You might want to set this to a TRUE value if you discover
571 that your old LWP applications fails after you installed LWP-5.60 or
574 =item PERL_HTTP_URI_CLASS
576 Used to decide what URI objects to instantiate. The default is C<URI>.
577 You might want to set it to C<URI::URL> for compatiblity with old times.
583 The library can not handle multiple simultaneous requests yet. Also,
584 check out what's left in the TODO file.
586 =head1 ACKNOWLEDGEMENTS
588 This package owes a lot in motivation, design, and code, to the
589 libwww-perl library for Perl 4, maintained by Roy Fielding
590 E<lt>fielding@ics.uci.edu>.
592 That package used work from Alberto Accomazzi, James Casey, Brooks
593 Cutter, Martijn Koster, Oscar Nierstrasz, Mel Melchner, Gertjan van
594 Oosten, Jared Rhine, Jack Shirazi, Gene Spafford, Marc VanHeyningen,
595 Steven E. Brenner, Marion Hakanson, Waldemar Kebsch, Tony Sanders, and
596 Larry Wall; see the libwww-perl-0.40 library for details.
598 The primary architect for this Perl 5 library is Martijn Koster and
599 Gisle Aas, with lots of help from Graham Barr, Tim Bunce, Andreas
600 Koenig, Jared Rhine, and Jack Shirazi.
605 Copyright 1995-2001, Gisle Aas
606 Copyright 1995, Martijn Koster
608 This library is free software; you can redistribute it and/or
609 modify it under the same terms as Perl itself.
613 The latest version of this library is likely to be available from CPAN
616 http://www.linpro.no/lwp/
618 The best place to discuss this code is on the <libwww@perl.org>