@menu
* iconv:: Character set conversion routines
-* iconv architecture:: Architecture of Newlib iconv library
* iconv configuration:: Newlib iconv-specific configure options
-* Generating CCS tables:: How to generate CCS tables
-* Adding new converter:: Steps on adding a new converter
@end menu
@page
@include iconv/iconv.def
@page
-@node iconv architecture
-@section iconv architecture
-@findex iconv architecture
-@findex encoding
-@findex CCS
-@findex CES
-@findex iconv converter
-@*
-@itemize @bullet
-@item
-Encoding - a rule to represent computer text by means of bits and bytes.
-@item
-CCS (Coded Character Set) - a mapping from an abstract character set
-to a set of non-negative integers (character codes).
-@item
-CES (Character Encoding Scheme) - a mapping from a set of character codes
-units to a sequence of bytes.
-@end itemize
-
-@*
-Examples of CCS: ASCII, ISO-8859-x, KOI8-R, KSX-1001, GB-2312.@*
-Examples of CES: UTF-8, UTF-16, EUC-JP, ISO-2022-JP.
-
-@*
-The iconv library is used to convert an array of characters in one encoding
-to array in another encoding.
-
-@*
-From a user's point of view, the iconv library is a set of converters. Each converter
-corresponds to one encoding (e.g., KOI8-R converter, UTF-8 converter).
-Internally the meaning of converter is different.
-
-@*
-The iconv library always performs conversions through UCS-32: i.e., to convert
-from A to B, iconv library first converts A to UCS-32, and then USC-32 to B.
-
-@*
-Each encoding consists of CES and CCS. CCS may be represented as data tables
-but CES always implies some code (algorithm). Iconv uses CCS tables
-to map from some encoding to UCS-32. CCS tables are placed into
-the iconv/ccs subdirectory of newlib. The iconv code also uses CES
-modules which can convert some CCS to and from UCS-32. CES modules are placed
-in the iconv/ces subdirectory.
-
-@*
-Some encodings have CES = CCS (e.g., KOI8-R). For such encodings iconv uses
-special subroutines which perform simple table conversions (ccs_table.c).
-
-@*
-Among specialized CES modules, the iconv library has
-generic support for EUC and ISO-2022-family encodings (ces_euc.c and
-ces_iso2022.c).
-
-@*
-To enable iconv to work with CCS or CES-based encodings, the correspondent
-CES table or CCS module should be linked with Newlib. The iconv support
-can also load CCS tables dynamically from external files (.cct files from
-iconv/ccs/binary subdirectory). CES modules, on the other-hand, can't
-be dynamically loaded.
-
-@*
-Each iconv converter has one name and a set of aliases. The list of
-aliases for each converter's name is in the iconv/charset.aliases file.
-Note: iconv always normalizes converter names and aliases before using.
-
-@page
@node iconv configuration
@section iconv configuration
@findex iconv configuration
-@findex iconv converter
+@findex encoding
@*
To enable iconv, the --enable-newlib-iconv configuration option should be
-used when configuring newlib.
-
-@*
-To link a specific converter (CCS table or CES module) into Newlib, the
----enable-newlib-builtin-converters option should be used. A
-comma-separated list of converters can be passed with this option
-(e.g., ---enable-newlib-builtin-converters=koi8-r,euc-jp to link KOI8-R
-and EUC-JP converters). Either converter names or aliases may be used.
+used when configuring Newlib.
@*
-If the target system has a file system accessible by Newlib, table-based
-converters may be loaded dynamically from external files. The iconv
-code tries to load files from the iconv_data subdirectory of the directory
-specified by the NLSPATH environment variable.
+Iconv library is intended to perform conversions from one encoding to
+another encoding. Thus, the only user-visible abstraction is encoding.
+To enable particular encoding support user should enable it using
+Newlib's configure script options. Encoding's support is divided into
+two parts: "to" and "from". For example, if it is only wanted to have
+UTF-8 -> UCS-4 coding capabilities, "from" UTF-8 and "to" UCS-4 support
+should be enabled. In this case backward (UCS-4 -> UTF-8) conversion
+won't be possible (iconv_open will return error). Such division on "to"
+and "from" parts helps to save memory.
@*
-Since Newlib has no generic dynamic module load support, CES-based converters
-can't be dynamically loaded and should be linked-in.
+To enable encoding support --enable-newlib-iconv-encodings configure
+script option should be used. This option accepts a comma-separated list
+of encodins that should be enabled. Option enables each encoding in both
+("to" and "from") directions.
-@page
-@node Generating CCS tables
-@section Generating CCS tables
@*
-CCS tables are placed in the ccs subdirectory of the iconv directory.
-This subdirectory contains .cct and .c files. The .cct files are for
-dynamic loading whereas the .c files are for static linking with Newlib.
-Both .c and .cct files are generated by the 'iconv_mktbl' perl script
-from special source files (call them
-.txt files). The 'iconv_mktbl' script can be found in the iconv/ccs
-subdirectory. Input .txt files can be found at the Unicode.org site or
-other locations found on the web.
+--enable-newlib-iconv-from-encodings configure script option enables
+"from" support for each encoding that was passed to it.
@*
-The .c files are linked with Newlib if the correspondent 'configure' script
-option was given. This is needed to use iconv on targets without file system
-support. If a CCS table isn't configured to be linked, the iconv library
-tries to load it dynamically from a corresponding .cct file.
+--enable-newlib-iconv-to-encodings configure script option enables
+"to" support for each encoding that was passed to it.
@*
-The following are commands to build .c and .cct CCS table files from .txt
-files for several supported encodings.
+Example: if user plans only KOI8-R -> UTF-8, UTF-8 -> ISO-8859-5 and
+KOI8-R -> UCS-2 conversions, the most optimal way (minimal iconv's
+code and data will be linked) is to configure Newlib with
+--enable-newlib-iconv-encodings=UTF-8
+--enable-newlib-iconv-from-encodings=KOI8-R
+--enable-newlib-iconv-to-encodings=KOI8-R,ISO-8859-5
@*
-@itemize
-@item
-cp775:@*
-iconv_mktbl -Co cp775.c cp775.txt@*
-iconv_mktbl -o cp775.cct cp775.txt
-@end itemize
-
-@itemize
-@item
-cp850:@*
-iconv_mktbl -Co cp850.c cp850.txt@*
-iconv_mktbl -o cp850.cct cp850.txt
-@end itemize
-
-@itemize
-@item
-cp852:@*
-iconv_mktbl -Co cp852.c cp852.txt@*
-iconv_mktbl -o cp852.cct cp852.txt
-@end itemize
-
-@itemize
-@item
-cp855:@*
-iconv_mktbl -Co cp855.c cp855.txt@*
-iconv_mktbl -o cp855.cct cp855.txt
-@end itemize
-
-@itemize
-@item
-cp866@*
-iconv_mktbl -Co cp866.c cp866.txt@*
-iconv_mktbl -o cp866.cct cp866.txt
-@end itemize
-
-@itemize
-@item
-iso-8859-1@*
-iconv_mktbl -Co iso-8859-1.c iso-8859-1.txt@*
-iconv_mktbl -o iso-8859-1.cct iso-8859-1.txt
-@end itemize
-
-@itemize
-@item
-iso-8859-4@*
-iconv_mktbl -Co iso-8859-4.c iso-8859-4.txt@*
-iconv_mktbl -o iso-8859-4.cct iso-8859-4.txt
-@end itemize
-
-@itemize
-@item
-iso-8859-5@*
-iconv_mktbl -Co iso-8859-5.c iso-8859-5.txt@*
-iconv_mktbl -o iso-8859-5.cct iso-8859-5.txt
-@end itemize
-
-@itemize
-@item
-iso-8859-2@*
-iconv_mktbl -Co iso-8859-2.c iso-8859-2.txt@*
-iconv_mktbl -o iso-8859-2.cct iso-8859-2.txt
-@end itemize
+There is one more configue script option for iconv library:
+--enable-newlib-iconv-external-ccs. This options enables iconv's
+capabilities to work with external CCS files. Exteral CCS files are just
+conversion tables used by iconv. Without this option all conversion
+tables are linked-in and occupy a lot of ROM. If target system has
+some fyle-system, it can benefit using external CCS files which are
+loaded on iconv_open and unloaded on iconv_close. But this way require
+more RAM. Moreover, in current implementation, distinct copy of CCS file
+is loaded for each fore each opended iconv decriptor for the same
+encoding. This means that if, for example, two iconv descriptors for
+KOI8-R -> UCS-4BE and KOI8-R -> UTF-16BE are opened, two copies of
+koi8-r.cct file will be loaded.
-@itemize
-@item
-iso-8859-15@*
-iconv_mktbl -Co iso-8859-15.c iso-8859-15.txt@*
-iconv_mktbl -o iso-8859-15.cct iso-8859-15.txt
-@end itemize
-
-@itemize
-@item
-big5@*
-iconv_mktbl -Co big5.c big5.txt@*
-iconv_mktbl -o big5.cct big5.txt
-@end itemize
-
-@itemize
-@item
-ksx1001@*
-iconv_mktbl -Co ksx1001.c ksx1001.txt@*
-iconv_mktbl -o ksx1001.cct ksx1001.txt
-@end itemize
-
-@itemize
-@item
-gb_2312@*
-iconv_mktbl -Co gb_2312-80.c gb_2312-80.txt@*
-iconv_mktbl -o gb_2312-80.cct gb_2312-80.txt
-@end itemize
-
-@itemize
-@item
-jis_x0201@*
-iconv_mktbl -Co jis_x0201.c jis_x0201.txt@*
-iconv_mktbl -o jis_x0201.cct jis_x0201.txt
-@end itemize
-
-@itemize
-@item
-iconv_mktbl -Co shift_jis.c shift_jis.txt@*
-iconv_mktbl -o shift_jis.cct shift_jis.txt
-@end itemize
-
-@itemize
-@item
-jis_x0208@*
-iconv_mktbl -C -c 1 -u 2 -o jis_x0208-1983.c jis_x0208-1983.txt@*
-iconv_mktbl -c 1 -u 2 -o jis_x0208-1983.cct jis_x0208-1983.txt
-@end itemize
-
-@itemize
-@item
-jis_x0212@*
-iconv_mktbl -Co jis_x0212-1990.c jis_x0212-1990.txt@*
-iconv_mktbl -o jis_x0212-1990.cct jis_x0212-1990.txt
-@end itemize
-
-@itemize
-@item
-cns11643-plane1@*
-iconv_mktbl -C -p 0x1 -o cns11643-plane1.c cns11643.txt@*
-iconv_mktbl -p 0x1 -o cns11643-plane1.cct cns11643.txt
-@end itemize
-
-@itemize
-@item
-cns11643-plane2@*
-iconv_mktbl -C -p 0x2 -o cns11643-plane2.c cns11643.txt@*
-iconv_mktbl -p 0x2 -o cns11643-plane2.cct cns11643.txt
-@end itemize
-
-@itemize
-@item
-cns11643-plane14@*
-iconv_mktbl -C -p 0xE -o cns11643-plane14.c cns11643.txt@*
-iconv_mktbl -p 0xE -o cns11643-plane14.cct cns11643.txt
-@end itemize
-
-@itemize
-@item
-koi8-r@*
-iconv_mktbl -Co koi8-r.c koi8-r.txt@*
-iconv_mktbl -o koi8-r.cct koi8-r.txt
-@end itemize
-
-@itemize
-@item
-koi8-u@*
-iconv_mktbl -Co koi8-u.c koi8-u.txt@*
-iconv_mktbl -o koi8-u.cct koi8-u.txt
-@end itemize
-
-@itemize
-@item
-us-ascii@*
-iconv_mktbl -Cao us-ascii.c iso-8859-1.txt@*
-iconv_mktbl -ao us-ascii.cct iso-8859-1.txt
-@end itemize
-
-@*
-Source files for CCS tables can be taken from at least two places:
-
-@*
-@enumerate
-@item
-http://www.unicode.org/Public/MAPPINGS/ contains a lot of encoding
-map files.
-@item
-http://www.dante.net/staff/konstantin/FreeBSD/iconv/ contains original
-iconv sources and encoding map files.
-@end enumerate
-
-@*
-The following are URLs where source files for some of the CCS tables
-are found:
-
-@itemize
-@item
-big5:@*
-http://www.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/OTHER/BIG5.TXT
-@end itemize
-
-@itemize
-@item
-cns11643_plane14, cns11643_plane1 and cns11643_plane2:@*
-http://www.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/OTHER/CNS11643.TXT
-@end itemize
-
-@itemize
-@item
-cp775, cp850, cp852, cp855, cp866:@*
-http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/
-@end itemize
-
-@itemize
-@item
-gb_2312_80:@*
-http://www.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/GB/GB2312.TXT
-@end itemize
-
-@itemize
-@item
-iso_8859_15, iso_8859_1, iso_8859_2, iso_8859_4, iso_8859_5:@*
-http://www.unicode.org/Public/MAPPINGS/ISO8859/
-@end itemize
-
-@itemize
-@item
-jis_x0201, jis_x0208_1983, jis_x0212_1990, shift_jis@*
-http://www.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/JIS/JIS0201.TXT
-@end itemize
-
-@itemize
-@item
-koi8_r@*
-http://www.unicode.org/Public/MAPPINGS/VENDORS/MISC/KOI8-R.TXT
-@end itemize
-
-@itemize
-@item
-ksx1001@*
-http://www.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/KSC/KSX1001.TXT
-@end itemize
-
-@itemize
-@item
-koi8-u can be given from original FreeBSD iconv library distribution
-http://www.dante.net/staff/konstantin/FreeBSD/iconv/
-@end itemize
-
-@*
-Moreover, http://www.dante.net/staff/konstantin/FreeBSD/iconv/ contains a
-lot of additional CCS tables that you can use with Newlib (iso-2022 and
-RFC1345 encodings).
-
-@page
-@node Adding new converter
-@section Adding a new iconv converter
-@*
-The following steps should be taken to add a new iconv converter:
-
@*
-@enumerate
-@item
-Converter's name and aliases list should be added to
-the iconv/charset.aliases file
-@item
-All iconv converters are protected by a _ICONV_CONVERTER_XXX
-macro, where XXX is converter name. This protection macro should be added to
-newlib/newlib.hin file.
-@item
-Converter's name and aliases should be also registered in _iconv_builtin_aliases
-table in iconv/lib/bialiasesi.c. The list should be protected by
-the corresponding macro mentioned above.
-@item
-If a new converter is just a CCS table, the corresponding .cct and .c files
-should be added to the iconv/ccs/ subdirectory. The name of the files
-should be equivalent to the normalized encoding name. The 'iconv_mktbl'
-Perl script (found in iconv/ccs) may
-be used to generate such files. The file's name should be added to
-iconv/ccs/Makefile.am and iconv/ccs/binary/Makefile.am files and then
-automake should be used to regenerate the Makefile.in files.
-@item
-If a new converter has a CES algorithm, the appropriate file should be
-added to the
-iconv/ces/ subdirectory. The name of the file again should be equivalent
-to the normalized
-encoding name.
-@item
-If a converter is EUC or ISO-2022-family CES, then the converter
-is just an array with a list of used CCS (See ccs/euc-jp.c for example). This
-is because iconv already has EUC and ISO-2022 support. Used CCS tables should
-be provided in iconv/ccs/.
-@item
-If a converter isn't EUC or ISO-2022-based CCS, the following two functions
-should be provided (see utf-8.c for example):
-@enumerate -
-@item A function to convert from new CES to UCS-32;
-@item A function to convert from UCS-32 to new CES;
-@item An 'init' function;
-@item A 'close' function;
-@item A 'reset' function to reset shift state for stateful CES.
-@end enumerate
+Note: not evry encoding needs CCS tiles. For example, UTF-8, UTF-16,
+UCS-2, UCS-4 doesn't use such files at all.
@*
-All these functions are registered into a 'struct iconv_ces_desc' object.
-The name of the object should be _iconv_ces_module_XXX, where XXX is the
-name of the converter.
-@item
-For CES converters the correspondent 'struct iconv_ces_desc' reference should
-be added into iconv/lib/bices.c file.
+Note: CCS file contains a number of tables, and only several needed tables
+are loaded from this file. This means, that there is a possibility to save
+some "fyle-system space" not including unneeded tables to that CCS
+files. Such task may be performed using "mktbl.pl" Perl script
+destributed with iconv library.
@*
-For CCS converters, the corresponding table reference should be added into
-the iconv/lib/biccs.c file.
-@end enumerate
+Note: CCS files are searched by iconv_open in $NLSPATH/iconv_data/ directory.
OSDN Git Service
