added google url library

1 files changed, 185 insertions, 0 deletions

diff --git a/googleurl/README.txt b/googleurl/README.txt
new file mode 100644
index 0000000..d5f79a3
--- /dev/null
+++ b/googleurl/README.txt
@@ -0,0 +1,185 @@
+                       ==============================
+                       The Google URL Parsing Library
+                       ==============================
+
+This is the Google URL Parsing Library which parses and canonicalizes URLs.
+Please see the LICENSE.txt file for licensing information.
+
+Features
+========
+
+   * Easily embeddable: This library was written for a variety of client and
+     server programs in mind, so unlike most implementations of URL parsing
+     and canonicalization, it can be easily emdedded.
+
+   * Fast: hundreds of thousands of typical URLs can be parsed and
+     canonicalized per second on a modern CPU. It is much faster than, for
+     example, calling WinInet's corresponding functions.
+
+   * Compatible: When possible, this library has strived for IE7 compatability
+     for both general web compatability, and so IE addons or other applications
+     that communicate with or embed IE will work properly.
+
+     It supports Unix-style file URLs, as well as the more complex rules for
+     Window file URLs. Note that total compatability is not possible (for
+     example, IE6 and IE7 disagree about how to parse certain IP addresses),
+     and that this is more strict about certain illegal, rarely used, and
+     potentially dangerous constructs such as escaped control characters in
+     host names that IE will allow. It is typically a little less strict than
+     Firefox.
+
+
+Example
+=======
+
+An example implementation of a URL object that uses this library is provided
+in src/gurl.*. This implementation uses the "application integration" layer
+discussed below to interface with the low-level parsing and canonicalization
+functions.
+
+
+Building
+========
+
+The canonicalization files require ICU for some UTF-8 and UTF-16 conversion
+macros. If your project does not use ICU, it should be straightforward to
+factor out the macros and functions used in ICU, there are only a few well-
+isolated things that are used.
+
+TODO(brettw) ADD INSTRUCTIONS FOR GETTING ICU HERE!
+
+logging.h and logging.cc are Windows-only because the corresponding Unix
+logging system has many dependencies. This library uses few of the logging
+macros, and a dummy header can easily be written that defines the
+appropriate things for Unix.
+
+
+Definitions
+===========
+
+"Standard URL": A URL with an "authority", which is a hostname and optionally
+   a port, username, and password. Most URLs are standard such as HTTP and FTP.
+
+"File URL": A URL that references a file on disk. There are special rules for
+   this type of URL. Note that it may have a hostname! "localhost" is allowed,
+   for example "file://localhost/foo" is the same as "file:///foo".
+
+"FileSystem URL": A URL referring to a file reached via the FileSystem API
+   described at http://www.w3.org/TR/file-system-api/.  These are nested URLs,
+   with compound schemes of e.g. "filesystem:file:" or "filesystem:https:".
+   Parsed FileSystem URLs will have a nested inner_parsed() object containing
+   information about the inner URL.
+
+"Path URL": This is everything else. There is no standard on how to treat these
+   URLs, or even what they are called. This library decomposes them into a
+   scheme and a path. The path is everything following the scheme. This type of
+   URL includes "javascript", "data", and even "mailto" (although "mailto"
+   might look like a standard scheme in some respects, it is not).
+
+Design
+======
+
+The library is divided into four layers. They are listed here from the lowest
+to the highest; you can use any portion of the library as long as you embed the
+layers below it.
+
+1. Parsing
+----------
+At the lowest level is the parsing code. The files encompassing this are
+url_parse.* and the main include file is src/url_parse.h. This code will, given
+an input string, parse it into the most likely form of a URL.
+
+Parsing cannot fail and does no validation. The exception is the port number,
+which it currently validates, but this is a bug. Given crazy input, the parser
+will do its best to find the various URL components according to its rules (see
+url_parse_unittest.cc for some examples).
+
+To use this, an application will typically use ExtractScheme to determine the
+type of a given input URL, and then call one of the initialization functions:
+"ParseStandardURL", "ParsePathURL", or "ParseFileURL". This will result in
+a "Parsed" structure which identifies the substrings of each identified
+component.
+
+2. Canonicalization
+-------------------
+At the next highest level is canonicalization. The files encompasing this are
+url_canon.* and the main include file is src/url_canon.h. This code will
+validate an already-parsed URL, and will convert it to a canonical form. For
+example, this will convert host names to lowercase, convert IP addresses
+into dotted-decimal notation, handle encoding issues, etc.
+
+This layer will always do its best to produce a reasonable output string, but
+it may return that the string is invalid. For example, if there are invalid
+characters in the host name, it will escape them or replace them with the
+Unicode "invalid character" character, but will fail. This way, the program can
+display error messages to the user with the output, log it, etc.  and the
+string will have some meaning.
+
+Canonicalized output is written to a CanonOutput object which is a simple
+wrapper around an expanding buffer. An implementation called RawCanonOutput is
+proivided that writes to a raw buffer with a fixed amount statically allocated
+(for performance). Applications using STL can use StdStringCanonOutput defined
+in url_canon_stdstring.h which writes into a std::string.
+
+A normal application would call one of the four high-level functions
+"CanonicalizeStandardURL", "CanonicalizeFileURL", "CanonicalizeFileSystemURL",
+and CanonicalizePathURL" depending on the type of URL in question. Lower-level
+functions are also provided which will canonicalize individual parts of a URL
+(for example, "CanonicalizeHost").
+
+Part of this layer is the integration with the host system for IDN and encoding
+conversion. An implementation that provides integration with the ICU
+(http://www-306.ibm.com/software/globalization/icu/index.jsp) is provided in
+src/url_canon_icu.cc. The embedder may wish to replace this file with
+implementations of the functions for their own IDN library if they do not use
+ICU.
+
+3. Application integration
+--------------------------
+The canonicalization and parsing layers do not know anything about the URI
+schemes supported by your application. The parsing and canonicalization
+functions are very low-level, and you must call the correct function to do the
+work (for example, "CanonicalizeFileURL").
+
+The application integration in url_util.* provides wrappers around the
+low-level parsing and canonicalization to call the correct versions for
+different identified schemes.  Embedders will want to modify this file if
+necessary to suit the needs of their application.
+
+4. URL object
+-------------
+The highest level is the "URL" object that a C++ application would use to
+to encapsulate a URL. Embedders will typically want to provide their own URL
+object that meets the requirements of their system. A reasonably complete
+example implemnetation is provided in src/gurl.*. You may wish to use this
+object, extend or modify it, or write your own.
+
+Whitespace
+----------
+Sometimes, you may want to remove linefeeds and tabs from the content of a URL.
+Some web pages, for example, expect that a URL spanning two lines should be
+treated as one with the newline removed. Depending on the source of the URLs
+you are canonicalizing, these newlines may or may not be trimmed off.
+
+If you want this behavior, call RemoveURLWhitespace before parsing. This will
+remove CR, LF and TAB from the input. Note that it preserves spaces. On typical
+URLs, this function produces a 10-15% speed reduction, so it is optional and
+not done automatically. The example GURL object and the url_util wrapper does
+this for you.
+
+Tests
+=====
+
+There are a number of *_unittest.cc and *_perftest.cc files. These files are
+not currently compilable as they rely on a not-included unit testing framework
+Tests are declared like this:
+  TEST(TestCaseName, TestName) {
+    ASSERT_TRUE(a);
+    EXPECT_EQ(a, b);
+  }
+If you would like to compile them, it should be straightforward to define
+the TEST macro (which would declare a function by combining the two arguments)
+and the other macros whose behavior should be self-explanatory (EXPECT is like
+an ASSERT, but does not stop the test, if you are doing this, you probably
+don't care about this difference). Then you would define a .cc file that
+calls all of these functions.