1 <?xml version="1.0" encoding="UTF-8" standalone="no"?>
2 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
3 <html xmlns="http://www.w3.org/1999/xhtml"><head><title>Facets</title><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"/><meta name="keywords" content=" ISO C++ , library "/><meta name="keywords" content=" ISO C++ , runtime , library "/><link rel="home" href="../index.html" title="The GNU C++ Library"/><link rel="up" href="localization.html" title="Chapter 8. Localization"/><link rel="prev" href="localization.html" title="Chapter 8. Localization"/><link rel="next" href="containers.html" title="Chapter 9. Containers"/></head><body><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Facets</th></tr><tr><td align="left"><a accesskey="p" href="localization.html">Prev</a> </td><th width="60%" align="center">Chapter 8.
6 </th><td align="right"> <a accesskey="n" href="containers.html">Next</a></td></tr></table><hr/></div><div class="section" title="Facets"><div class="titlepage"><div><div><h2 class="title"><a id="std.localization.facet"/>Facets</h2></div></div></div><div class="section" title="ctype"><div class="titlepage"><div><div><h3 class="title"><a id="std.localization.facet.ctype"/>ctype</h3></div></div></div><div class="section" title="Implementation"><div class="titlepage"><div><div><h4 class="title"><a id="facet.ctype.impl"/>Implementation</h4></div></div></div><div class="section" title="Specializations"><div class="titlepage"><div><div><h5 class="title"><a id="id609115"/>Specializations</h5></div></div></div><p>
7 For the required specialization codecvt<wchar_t, char, mbstate_t> ,
8 conversions are made between the internal character set (always UCS4
9 on GNU/Linux) and whatever the currently selected locale for the
10 LC_CTYPE category implements.
12 The two required specializations are implemented as follows:
18 This is simple specialization. Implementing this was a piece of cake.
24 This specialization, by specifying all the template parameters, pretty
25 much ties the hands of implementors. As such, the implementation is
26 straightforward, involving mcsrtombs for the conversions between char
27 to wchar_t and wcsrtombs for conversions between wchar_t and char.
29 Neither of these two required specializations deals with Unicode
31 </p></div></div><div class="section" title="Future"><div class="titlepage"><div><div><h4 class="title"><a id="facet.ctype.future"/>Future</h4></div></div></div><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>
32 How to deal with the global locale issue?
33 </p></li><li class="listitem"><p>
34 How to deal with different types than char, wchar_t? </p></li><li class="listitem"><p>
35 Overlap between codecvt/ctype: narrow/widen
36 </p></li><li class="listitem"><p>
37 Mask typedef in codecvt_base, argument types in codecvt. what
38 is know about this type?
39 </p></li><li class="listitem"><p>
40 Why mask* argument in codecvt?
41 </p></li><li class="listitem"><p>
42 Can this be made (more) generic? is there a simple way to
43 straighten out the configure-time mess that is a by-product of
45 </p></li><li class="listitem"><p>
46 Get the ctype<wchar_t>::mask stuff under control. Need to
47 make some kind of static table, and not do lookup every time
48 somebody hits the do_is... functions. Too bad we can't just
49 redefine mask for ctype<wchar_t>
50 </p></li><li class="listitem"><p>
51 Rename abstract base class. See if just smash-overriding is a
52 better approach. Clarify, add sanity to naming.
53 </p></li></ul></div></div><div class="bibliography" title="Bibliography"><div class="titlepage"><div><div><h4 class="title"><a id="facet.ctype.biblio"/>Bibliography</h4></div></div></div><div class="biblioentry"><a id="id609240"/><p><span class="citetitle"><em class="citetitle">
55 </em>. </span><span class="author"><span class="firstname">Roland</span> <span class="surname">McGrath</span>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2007 FSF. </span><span class="pagenums">Chapters 6 Character Set Handling and 7 Locales and Internationalization. </span></p></div><div class="biblioentry"><a id="id609280"/><p><span class="citetitle"><em class="citetitle">
57 </em>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2002 . </span></p></div><div class="biblioentry"><a id="id609306"/><p><span class="citetitle"><em class="citetitle">
58 ISO/IEC 14882:1998 Programming languages - C++
59 </em>. </span><span class="copyright">Copyright © 1998 ISO. </span></p></div><div class="biblioentry"><a id="id609325"/><p><span class="citetitle"><em class="citetitle">
60 ISO/IEC 9899:1999 Programming languages - C
61 </em>. </span><span class="copyright">Copyright © 1999 ISO. </span></p></div><div class="biblioentry" title="The Open Group Base Specifications, Issue 6 (IEEE Std. 1003.1-2004)"><a id="id609344"/><p><span class="title"><em>
62 <a class="link" href="http://www.unix.org/version3/ieee_std.html">
63 The Open Group Base Specifications, Issue 6 (IEEE Std. 1003.1-2004)
65 </em>. </span><span class="copyright">Copyright © 1999
66 The Open Group/The Institute of Electrical and Electronics Engineers, Inc.. </span></p></div><div class="biblioentry"><a id="id609371"/><p><span class="citetitle"><em class="citetitle">
67 The C++ Programming Language, Special Edition
68 </em>. </span><span class="author"><span class="firstname">Bjarne</span> <span class="surname">Stroustrup</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley, Inc.. </span><span class="pagenums">Appendix D. </span><span class="publisher"><span class="publishername">
70 . </span></span></p></div><div class="biblioentry"><a id="id609409"/><p><span class="citetitle"><em class="citetitle">
71 Standard C++ IOStreams and Locales
72 </em>. </span><span class="subtitle">
73 Advanced Programmer's Guide and Reference
74 . </span><span class="author"><span class="firstname">Angelika</span> <span class="surname">Langer</span>. </span><span class="author"><span class="firstname">Klaus</span> <span class="surname">Kreft</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley Longman, Inc.. </span><span class="publisher"><span class="publishername">
75 Addison Wesley Longman
76 . </span></span></p></div></div></div><div class="section" title="codecvt"><div class="titlepage"><div><div><h3 class="title"><a id="std.localization.facet.codecvt"/>codecvt</h3></div></div></div><p>
77 The standard class codecvt attempts to address conversions between
78 different character encoding schemes. In particular, the standard
79 attempts to detail conversions between the implementation-defined wide
80 characters (hereafter referred to as wchar_t) and the standard type
81 char that is so beloved in classic <span class="quote">“<span class="quote">C</span>”</span> (which can now be
82 referred to as narrow characters.) This document attempts to describe
83 how the GNU libstdc++ implementation deals with the conversion between
84 wide and narrow characters, and also presents a framework for dealing
85 with the huge number of other encodings that iconv can convert,
86 including Unicode and UTF8. Design issues and requirements are
87 addressed, and examples of correct usage for both the required
88 specializations for wide and narrow characters and the
89 implementation-provided extended functionality are given.
90 </p><div class="section" title="Requirements"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.req"/>Requirements</h4></div></div></div><p>
91 Around page 425 of the C++ Standard, this charming heading comes into view:
92 </p><div class="blockquote"><blockquote class="blockquote"><p>
93 22.2.1.5 - Template class codecvt
94 </p></blockquote></div><p>
95 The text around the codecvt definition gives some clues:
96 </p><div class="blockquote"><blockquote class="blockquote"><p>
97 <span class="emphasis"><em>
98 -1- The class codecvt<internT,externT,stateT> is for use when
99 converting from one codeset to another, such as from wide characters
100 to multibyte characters, between wide character encodings such as
103 </p></blockquote></div><p>
104 Hmm. So, in some unspecified way, Unicode encodings and
105 translations between other character sets should be handled by this
107 </p><div class="blockquote"><blockquote class="blockquote"><p>
108 <span class="emphasis"><em>
109 -2- The stateT argument selects the pair of codesets being mapped between.
111 </p></blockquote></div><p>
112 Ah ha! Another clue...
113 </p><div class="blockquote"><blockquote class="blockquote"><p>
114 <span class="emphasis"><em>
115 -3- The instantiations required in the Table ??
116 (lib.locale.category), namely codecvt<wchar_t,char,mbstate_t> and
117 codecvt<char,char,mbstate_t>, convert the implementation-defined
118 native character set. codecvt<char,char,mbstate_t> implements a
119 degenerate conversion; it does not convert at
120 all. codecvt<wchar_t,char,mbstate_t> converts between the native
121 character sets for tiny and wide characters. Instantiations on
122 mbstate_t perform conversion between encodings known to the library
123 implementor. Other encodings can be converted by specializing on a
124 user-defined stateT type. The stateT object can contain any state that
125 is useful to communicate to or from the specialized do_convert member.
127 </p></blockquote></div><p>
128 At this point, a couple points become clear:
130 One: The standard clearly implies that attempts to add non-required
131 (yet useful and widely used) conversions need to do so through the
132 third template parameter, stateT.</p><p>
133 Two: The required conversions, by specifying mbstate_t as the third
134 template parameter, imply an implementation strategy that is mostly
135 (or wholly) based on the underlying C library, and the functions
136 mcsrtombs and wcsrtombs in particular.</p></div><div class="section" title="Design"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.design"/>Design</h4></div></div></div><div class="section" title="wchar_t Size"><div class="titlepage"><div><div><h5 class="title"><a id="codecvt.design.wchar_t_size"/><span class="type">wchar_t</span> Size</h5></div></div></div><p>
137 The simple implementation detail of wchar_t's size seems to
138 repeatedly confound people. Many systems use a two byte,
139 unsigned integral type to represent wide characters, and use an
140 internal encoding of Unicode or UCS2. (See AIX, Microsoft NT,
141 Java, others.) Other systems, use a four byte, unsigned integral
142 type to represent wide characters, and use an internal encoding
143 of UCS4. (GNU/Linux systems using glibc, in particular.) The C
144 programming language (and thus C++) does not specify a specific
145 size for the type wchar_t.
147 Thus, portable C++ code cannot assume a byte size (or endianness) either.
148 </p></div><div class="section" title="Support for Unicode"><div class="titlepage"><div><div><h5 class="title"><a id="codecvt.design.unicode"/>Support for Unicode</h5></div></div></div><p>
149 Probably the most frequently asked question about code conversion
150 is: "So dudes, what's the deal with Unicode strings?"
151 The dude part is optional, but apparently the usefulness of
152 Unicode strings is pretty widely appreciated. Sadly, this specific
153 encoding (And other useful encodings like UTF8, UCS4, ISO 8859-10,
154 etc etc etc) are not mentioned in the C++ standard.
156 A couple of comments:
158 The thought that all one needs to convert between two arbitrary
159 codesets is two types and some kind of state argument is
160 unfortunate. In particular, encodings may be stateless. The naming
161 of the third parameter as stateT is unfortunate, as what is really
162 needed is some kind of generalized type that accounts for the
163 issues that abstract encodings will need. The minimum information
164 that is required includes:
165 </p><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>
166 Identifiers for each of the codesets involved in the
167 conversion. For example, using the iconv family of functions
168 from the Single Unix Specification (what used to be called
169 X/Open) hosted on the GNU/Linux operating system allows
170 bi-directional mapping between far more than the following
171 tantalizing possibilities:
173 (An edited list taken from <code class="code">`iconv --list`</code> on a
174 Red Hat 6.2/Intel system:
175 </p><div class="blockquote"><blockquote class="blockquote"><pre class="programlisting">
176 8859_1, 8859_9, 10646-1:1993, 10646-1:1993/UCS4, ARABIC, ARABIC7,
177 ASCII, EUC-CN, EUC-JP, EUC-KR, EUC-TW, GREEK-CCIcode, GREEK, GREEK7-OLD,
178 GREEK7, GREEK8, HEBREW, ISO-8859-1, ISO-8859-2, ISO-8859-3,
179 ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8,
180 ISO-8859-9, ISO-8859-10, ISO-8859-11, ISO-8859-13, ISO-8859-14,
181 ISO-8859-15, ISO-10646, ISO-10646/UCS2, ISO-10646/UCS4,
182 ISO-10646/UTF-8, ISO-10646/UTF8, SHIFT-JIS, SHIFT_JIS, UCS-2, UCS-4,
183 UCS2, UCS4, UNICODE, UNICODEBIG, UNICODELIcodeLE, US-ASCII, US, UTF-8,
184 UTF-16, UTF8, UTF16).
185 </pre></blockquote></div><p>
186 For iconv-based implementations, string literals for each of the
187 encodings (i.e. "UCS-2" and "UTF-8") are necessary,
189 non-iconv implementations a table of enumerated values or some other
190 mechanism may be required.
191 </p></li><li class="listitem"><p>
192 Maximum length of the identifying string literal.
193 </p></li><li class="listitem"><p>
194 Some encodings require explicit endian-ness. As such, some kind
195 of endian marker or other byte-order marker will be necessary. See
196 "Footnotes for C/C++ developers" in Haible for more information on
197 UCS-2/Unicode endian issues. (Summary: big endian seems most likely,
198 however implementations, most notably Microsoft, vary.)
199 </p></li><li class="listitem"><p>
200 Types representing the conversion state, for conversions involving
201 the machinery in the "C" library, or the conversion descriptor, for
202 conversions using iconv (such as the type iconv_t.) Note that the
203 conversion descriptor encodes more information than a simple encoding
205 </p></li><li class="listitem"><p>
206 Conversion descriptors for both directions of encoding. (i.e., both
207 UCS-2 to UTF-8 and UTF-8 to UCS-2.)
208 </p></li><li class="listitem"><p>
209 Something to indicate if the conversion requested if valid.
210 </p></li><li class="listitem"><p>
211 Something to represent if the conversion descriptors are valid.
212 </p></li><li class="listitem"><p>
213 Some way to enforce strict type checking on the internal and
214 external types. As part of this, the size of the internal and
215 external types will need to be known.
216 </p></li></ul></div></div><div class="section" title="Other Issues"><div class="titlepage"><div><div><h5 class="title"><a id="codecvt.design.issues"/>Other Issues</h5></div></div></div><p>
217 In addition, multi-threaded and multi-locale environments also impact
218 the design and requirements for code conversions. In particular, they
219 affect the required specialization codecvt<wchar_t, char, mbstate_t>
220 when implemented using standard "C" functions.
222 Three problems arise, one big, one of medium importance, and one small.
224 First, the small: mcsrtombs and wcsrtombs may not be multithread-safe
225 on all systems required by the GNU tools. For GNU/Linux and glibc,
226 this is not an issue.
228 Of medium concern, in the grand scope of things, is that the functions
229 used to implement this specialization work on null-terminated
230 strings. Buffers, especially file buffers, may not be null-terminated,
231 thus giving conversions that end prematurely or are otherwise
234 The last, and fundamental problem, is the assumption of a global
235 locale for all the "C" functions referenced above. For something like
236 C++ iostreams (where codecvt is explicitly used) the notion of
237 multiple locales is fundamental. In practice, most users may not run
238 into this limitation. However, as a quality of implementation issue,
239 the GNU C++ library would like to offer a solution that allows
240 multiple locales and or simultaneous usage with computationally
241 correct results. In short, libstdc++ is trying to offer, as an
242 option, a high-quality implementation, damn the additional complexity!
244 For the required specialization codecvt<wchar_t, char, mbstate_t> ,
245 conversions are made between the internal character set (always UCS4
246 on GNU/Linux) and whatever the currently selected locale for the
247 LC_CTYPE category implements.
248 </p></div></div><div class="section" title="Implementation"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.impl"/>Implementation</h4></div></div></div><p>
249 The two required specializations are implemented as follows:
252 codecvt<char, char, mbstate_t>
255 This is a degenerate (i.e., does nothing) specialization. Implementing
256 this was a piece of cake.
259 codecvt<char, wchar_t, mbstate_t>
262 This specialization, by specifying all the template parameters, pretty
263 much ties the hands of implementors. As such, the implementation is
264 straightforward, involving mcsrtombs for the conversions between char
265 to wchar_t and wcsrtombs for conversions between wchar_t and char.
267 Neither of these two required specializations deals with Unicode
268 characters. As such, libstdc++ implements a partial specialization
269 of the codecvt class with and iconv wrapper class, encoding_state as the
270 third template parameter.
272 This implementation should be standards conformant. First of all, the
273 standard explicitly points out that instantiations on the third
274 template parameter, stateT, are the proper way to implement
275 non-required conversions. Second of all, the standard says (in Chapter
276 17) that partial specializations of required classes are a-ok. Third
277 of all, the requirements for the stateT type elsewhere in the standard
278 (see 21.1.2 traits typedefs) only indicate that this type be copy
281 As such, the type encoding_state is defined as a non-templatized, POD
282 type to be used as the third type of a codecvt instantiation. This
283 type is just a wrapper class for iconv, and provides an easy interface
284 to iconv functionality.
286 There are two constructors for encoding_state:
289 encoding_state() : __in_desc(0), __out_desc(0)
292 This default constructor sets the internal encoding to some default
293 (currently UCS4) and the external encoding to whatever is returned by
294 nl_langinfo(CODESET).
297 encoding_state(const char* __int, const char* __ext)
300 This constructor takes as parameters string literals that indicate the
301 desired internal and external encoding. There are no defaults for
304 One of the issues with iconv is that the string literals identifying
305 conversions are not standardized. Because of this, the thought of
306 mandating and or enforcing some set of pre-determined valid
307 identifiers seems iffy: thus, a more practical (and non-migraine
308 inducing) strategy was implemented: end-users can specify any string
309 (subject to a pre-determined length qualifier, currently 32 bytes) for
310 encodings. It is up to the user to make sure that these strings are
311 valid on the target system.
318 Strangely enough, this member function attempts to open conversion
319 descriptors for a given encoding_state object. If the conversion
320 descriptors are not valid, the conversion descriptors returned will
321 not be valid and the resulting calls to the codecvt conversion
322 functions will return error.
329 Provides a way to see if the given encoding_state object has been
330 properly initialized. If the string literals describing the desired
331 internal and external encoding are not valid, initialization will
332 fail, and this will return false. If the internal and external
333 encodings are valid, but iconv_open could not allocate conversion
334 descriptors, this will also return false. Otherwise, the object is
335 ready to convert and will return true.
338 encoding_state(const encoding_state&)
341 As iconv allocates memory and sets up conversion descriptors, the copy
342 constructor can only copy the member data pertaining to the internal
343 and external code conversions, and not the conversion descriptors
346 Definitions for all the required codecvt member functions are provided
347 for this specialization, and usage of codecvt<internal character type,
348 external character type, encoding_state> is consistent with other
350 </p></div><div class="section" title="Use"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.use"/>Use</h4></div></div></div><p>A conversions involving string literal.</p><pre class="programlisting">
351 typedef codecvt_base::result result;
352 typedef unsigned short unicode_t;
353 typedef unicode_t int_type;
354 typedef char ext_type;
355 typedef encoding_state state_type;
356 typedef codecvt<int_type, ext_type, state_type> unicode_codecvt;
358 const ext_type* e_lit = "black pearl jasmine tea";
359 int size = strlen(e_lit);
360 int_type i_lit_base[24] =
361 { 25088, 27648, 24832, 25344, 27392, 8192, 28672, 25856, 24832, 29184,
362 27648, 8192, 27136, 24832, 29440, 27904, 26880, 28160, 25856, 8192, 29696,
365 const int_type* i_lit = i_lit_base;
366 const ext_type* efrom_next;
367 const int_type* ifrom_next;
368 ext_type* e_arr = new ext_type[size + 1];
370 int_type* i_arr = new int_type[size + 1];
373 // construct a locale object with the specialized facet.
374 locale loc(locale::classic(), new unicode_codecvt);
375 // sanity check the constructed locale has the specialized facet.
376 VERIFY( has_facet<unicode_codecvt>(loc) );
377 const unicode_codecvt& cvt = use_facet<unicode_codecvt>(loc);
378 // convert between const char* and unicode strings
379 unicode_codecvt::state_type state01("UNICODE", "ISO_8859-1");
380 initialize_state(state01);
381 result r1 = cvt.in(state01, e_lit, e_lit + size, efrom_next,
382 i_arr, i_arr + size, ito_next);
383 VERIFY( r1 == codecvt_base::ok );
384 VERIFY( !int_traits::compare(i_arr, i_lit, size) );
385 VERIFY( efrom_next == e_lit + size );
386 VERIFY( ito_next == i_arr + size );
387 </pre></div><div class="section" title="Future"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.future"/>Future</h4></div></div></div><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>
388 a. things that are sketchy, or remain unimplemented:
389 do_encoding, max_length and length member functions
390 are only weakly implemented. I have no idea how to do
391 this correctly, and in a generic manner. Nathan?
392 </p></li><li class="listitem"><p>
393 b. conversions involving std::string
394 </p><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>
395 how should operators != and == work for string of
396 different/same encoding?
397 </p></li><li class="listitem"><p>
398 what is equal? A byte by byte comparison or an
399 encoding then byte comparison?
400 </p></li><li class="listitem"><p>
401 conversions between narrow, wide, and unicode strings
402 </p></li></ul></div></li><li class="listitem"><p>
403 c. conversions involving std::filebuf and std::ostream
404 </p><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>
405 how to initialize the state object in a
406 standards-conformant manner?
407 </p></li><li class="listitem"><p>
408 how to synchronize the "C" and "C++"
409 conversion information?
410 </p></li><li class="listitem"><p>
411 wchar_t/char internal buffers and conversions between
412 internal/external buffers?
413 </p></li></ul></div></li></ul></div></div><div class="bibliography" title="Bibliography"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.biblio"/>Bibliography</h4></div></div></div><div class="biblioentry"><a id="id610059"/><p><span class="citetitle"><em class="citetitle">
415 </em>. </span><span class="author"><span class="firstname">Roland</span> <span class="surname">McGrath</span>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2007 FSF. </span><span class="pagenums">
416 Chapters 6 Character Set Handling and 7 Locales and Internationalization
417 . </span></p></div><div class="biblioentry"><a id="id610099"/><p><span class="citetitle"><em class="citetitle">
419 </em>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2002 . </span></p></div><div class="biblioentry"><a id="id610125"/><p><span class="citetitle"><em class="citetitle">
420 ISO/IEC 14882:1998 Programming languages - C++
421 </em>. </span><span class="copyright">Copyright © 1998 ISO. </span></p></div><div class="biblioentry"><a id="id610144"/><p><span class="citetitle"><em class="citetitle">
422 ISO/IEC 9899:1999 Programming languages - C
423 </em>. </span><span class="copyright">Copyright © 1999 ISO. </span></p></div><div class="biblioentry" title="System Interface Definitions, Issue 7 (IEEE Std. 1003.1-2008)"><a id="id610163"/><p><span class="title"><em>
424 <a class="link" href="http://www.opengroup.org/austin">
425 System Interface Definitions, Issue 7 (IEEE Std. 1003.1-2008)
427 </em>. </span><span class="copyright">Copyright © 2008
428 The Open Group/The Institute of Electrical and Electronics
430 . </span></p></div><div class="biblioentry"><a id="id610190"/><p><span class="citetitle"><em class="citetitle">
431 The C++ Programming Language, Special Edition
432 </em>. </span><span class="author"><span class="firstname">Bjarne</span> <span class="surname">Stroustrup</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley, Inc.. </span><span class="pagenums">Appendix D. </span><span class="publisher"><span class="publishername">
434 . </span></span></p></div><div class="biblioentry"><a id="id610228"/><p><span class="citetitle"><em class="citetitle">
435 Standard C++ IOStreams and Locales
436 </em>. </span><span class="subtitle">
437 Advanced Programmer's Guide and Reference
438 . </span><span class="author"><span class="firstname">Angelika</span> <span class="surname">Langer</span>. </span><span class="author"><span class="firstname">Klaus</span> <span class="surname">Kreft</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley Longman, Inc.. </span><span class="publisher"><span class="publishername">
439 Addison Wesley Longman
440 . </span></span></p></div><div class="biblioentry" title="A brief description of Normative Addendum 1"><a id="id610275"/><p><span class="title"><em>
441 <a class="link" href="http://www.lysator.liu.se/c/na1.html">
442 A brief description of Normative Addendum 1
444 </em>. </span><span class="author"><span class="firstname">Clive</span> <span class="surname">Feather</span>. </span><span class="pagenums">Extended Character Sets. </span></p></div><div class="biblioentry" title="The Unicode HOWTO"><a id="id610302"/><p><span class="title"><em>
445 <a class="link" href="http://tldp.org/HOWTO/Unicode-HOWTO.html">
448 </em>. </span><span class="author"><span class="firstname">Bruno</span> <span class="surname">Haible</span>. </span></p></div><div class="biblioentry" title="UTF-8 and Unicode FAQ for Unix/Linux"><a id="id610326"/><p><span class="title"><em>
449 <a class="link" href="http://www.cl.cam.ac.uk/~mgk25/unicode.html">
450 UTF-8 and Unicode FAQ for Unix/Linux
452 </em>. </span><span class="author"><span class="firstname">Markus</span> <span class="surname">Khun</span>. </span></p></div></div></div><div class="section" title="messages"><div class="titlepage"><div><div><h3 class="title"><a id="manual.localization.facet.messages"/>messages</h3></div></div></div><p>
453 The std::messages facet implements message retrieval functionality
454 equivalent to Java's java.text.MessageFormat .using either GNU gettext
455 or IEEE 1003.1-200 functions.
456 </p><div class="section" title="Requirements"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.req"/>Requirements</h4></div></div></div><p>
457 The std::messages facet is probably the most vaguely defined facet in
458 the standard library. It's assumed that this facility was built into
459 the standard library in order to convert string literals from one
460 locale to the other. For instance, converting the "C" locale's
461 <code class="code">const char* c = "please"</code> to a German-localized <code class="code">"bitte"</code>
462 during program execution.
463 </p><div class="blockquote"><blockquote class="blockquote"><p>
464 22.2.7.1 - Template class messages [lib.locale.messages]
465 </p></blockquote></div><p>
466 This class has three public member functions, which directly
467 correspond to three protected virtual member functions.
469 The public member functions are:
471 <code class="code">catalog open(const string&, const locale&) const</code>
473 <code class="code">string_type get(catalog, int, int, const string_type&) const</code>
475 <code class="code">void close(catalog) const</code>
477 While the virtual functions are:
479 <code class="code">catalog do_open(const string&, const locale&) const</code>
480 </p><div class="blockquote"><blockquote class="blockquote"><p>
481 <span class="emphasis"><em>
482 -1- Returns: A value that may be passed to get() to retrieve a
483 message, from the message catalog identified by the string name
484 according to an implementation-defined mapping. The result can be used
485 until it is passed to close(). Returns a value less than 0 if no such
486 catalog can be opened.
488 </p></blockquote></div><p>
489 <code class="code">string_type do_get(catalog, int, int, const string_type&) const</code>
490 </p><div class="blockquote"><blockquote class="blockquote"><p>
491 <span class="emphasis"><em>
492 -3- Requires: A catalog cat obtained from open() and not yet closed.
493 -4- Returns: A message identified by arguments set, msgid, and dfault,
494 according to an implementation-defined mapping. If no such message can
495 be found, returns dfault.
497 </p></blockquote></div><p>
498 <code class="code">void do_close(catalog) const</code>
499 </p><div class="blockquote"><blockquote class="blockquote"><p>
500 <span class="emphasis"><em>
501 -5- Requires: A catalog cat obtained from open() and not yet closed.
502 -6- Effects: Releases unspecified resources associated with cat.
503 -7- Notes: The limit on such resources, if any, is implementation-defined.
505 </p></blockquote></div></div><div class="section" title="Design"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.design"/>Design</h4></div></div></div><p>
506 A couple of notes on the standard.
508 First, why is <code class="code">messages_base::catalog</code> specified as a typedef
509 to int? This makes sense for implementations that use
510 <code class="code">catopen</code> and define <code class="code">nl_catd</code> as int, but not for
511 others. Fortunately, it's not heavily used and so only a minor irritant.
512 This has been reported as a possible defect in the standard (LWG 2028).
514 Second, by making the member functions <code class="code">const</code>, it is
515 impossible to save state in them. Thus, storing away information used
516 in the 'open' member function for use in 'get' is impossible. This is
519 The 'open' member function in particular seems to be oddly
520 designed. The signature seems quite peculiar. Why specify a <code class="code">const
521 string& </code> argument, for instance, instead of just <code class="code">const
522 char*</code>? Or, why specify a <code class="code">const locale&</code> argument that is
523 to be used in the 'get' member function? How, exactly, is this locale
524 argument useful? What was the intent? It might make sense if a locale
525 argument was associated with a given default message string in the
526 'open' member function, for instance. Quite murky and unclear, on
529 Lastly, it seems odd that messages, which explicitly require code
530 conversion, don't use the codecvt facet. Because the messages facet
531 has only one template parameter, it is assumed that ctype, and not
532 codecvt, is to be used to convert between character sets.
534 It is implicitly assumed that the locale for the default message
535 string in 'get' is in the "C" locale. Thus, all source code is assumed
536 to be written in English, so translations are always from "en_US" to
537 other, explicitly named locales.
538 </p></div><div class="section" title="Implementation"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.impl"/>Implementation</h4></div></div></div><div class="section" title="Models"><div class="titlepage"><div><div><h5 class="title"><a id="messages.impl.models"/>Models</h5></div></div></div><p>
539 This is a relatively simple class, on the face of it. The standard
540 specifies very little in concrete terms, so generic
541 implementations that are conforming yet do very little are the
542 norm. Adding functionality that would be useful to programmers and
543 comparable to Java's java.text.MessageFormat takes a bit of work,
544 and is highly dependent on the capabilities of the underlying
547 Three different mechanisms have been provided, selectable via
549 </p><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>
552 This model does very little, and is what is used by default.
553 </p></li><li class="listitem"><p>
556 The gnu model is complete and fully tested. It's based on the
557 GNU gettext package, which is part of glibc. It uses the
558 functions <code class="code">textdomain, bindtextdomain, gettext</code> to
559 implement full functionality. Creating message catalogs is a
560 relatively straight-forward process and is lightly documented
561 below, and fully documented in gettext's distributed
563 </p></li><li class="listitem"><p>
566 This is a complete, though untested, implementation based on
567 the IEEE standard. The functions <code class="code">catopen, catgets,
568 catclose</code> are used to retrieve locale-specific messages
569 given the appropriate message catalogs that have been
570 constructed for their use. Note, the script <code class="code">
571 po2msg.sed</code> that is part of the gettext distribution can
572 convert gettext catalogs into catalogs that
573 <code class="code">catopen</code> can use.
574 </p></li></ul></div><p>
575 A new, standards-conformant non-virtual member function signature was
576 added for 'open' so that a directory could be specified with a given
577 message catalog. This simplifies calling conventions for the gnu
579 </p></div><div class="section" title="The GNU Model"><div class="titlepage"><div><div><h5 class="title"><a id="messages.impl.gnu"/>The GNU Model</h5></div></div></div><p>
580 The messages facet, because it is retrieving and converting
581 between characters sets, depends on the ctype and perhaps the
582 codecvt facet in a given locale. In addition, underlying "C"
583 library locale support is necessary for more than just the
584 <code class="code">LC_MESSAGES</code> mask: <code class="code">LC_CTYPE</code> is also
585 necessary. To avoid any unpleasantness, all bits of the "C" mask
586 (i.e. <code class="code">LC_ALL</code>) are set before retrieving messages.
588 Making the message catalogs can be initially tricky, but become
589 quite simple with practice. For complete info, see the gettext
590 documentation. Here's an idea of what is required:
591 </p><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>
592 Make a source file with the required string literals that need
593 to be translated. See <code class="code">intl/string_literals.cc</code> for
595 </p></li><li class="listitem"><p>
596 Make initial catalog (see "4 Making the PO Template File" from
597 the gettext docs).</p><p>
598 <code class="code"> xgettext --c++ --debug string_literals.cc -o libstdc++.pot </code>
599 </p></li><li class="listitem"><p>Make language and country-specific locale catalogs.</p><p>
600 <code class="code">cp libstdc++.pot fr_FR.po</code>
602 <code class="code">cp libstdc++.pot de_DE.po</code>
603 </p></li><li class="listitem"><p>
604 Edit localized catalogs in emacs so that strings are
607 <code class="code">emacs fr_FR.po</code>
608 </p></li><li class="listitem"><p>Make the binary mo files.</p><p>
609 <code class="code">msgfmt fr_FR.po -o fr_FR.mo</code>
611 <code class="code">msgfmt de_DE.po -o de_DE.mo</code>
612 </p></li><li class="listitem"><p>Copy the binary files into the correct directory structure.</p><p>
613 <code class="code">cp fr_FR.mo (dir)/fr_FR/LC_MESSAGES/libstdc++.mo</code>
615 <code class="code">cp de_DE.mo (dir)/de_DE/LC_MESSAGES/libstdc++.mo</code>
616 </p></li><li class="listitem"><p>Use the new message catalogs.</p><p>
617 <code class="code">locale loc_de("de_DE");</code>
620 use_facet<messages<char> >(loc_de).open("libstdc++", locale(), dir);
622 </p></li></ul></div></div></div><div class="section" title="Use"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.use"/>Use</h4></div></div></div><p>
623 A simple example using the GNU model of message conversion.
624 </p><pre class="programlisting">
625 #include <iostream>
626 #include <locale>
631 typedef messages<char>::catalog catalog;
633 "/mnt/egcs/build/i686-pc-linux-gnu/libstdc++/po/share/locale";
634 const locale loc_de("de_DE");
635 const messages<char>& mssg_de = use_facet<messages<char> >(loc_de);
637 catalog cat_de = mssg_de.open("libstdc++", loc_de, dir);
638 string s01 = mssg_de.get(cat_de, 0, 0, "please");
639 string s02 = mssg_de.get(cat_de, 0, 0, "thank you");
640 cout << "please in german:" << s01 << '\n';
641 cout << "thank you in german:" << s02 << '\n';
642 mssg_de.close(cat_de);
644 </pre></div><div class="section" title="Future"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.future"/>Future</h4></div></div></div><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>
645 Things that are sketchy, or remain unimplemented:
646 </p><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>
647 _M_convert_from_char, _M_convert_to_char are in flux,
648 depending on how the library ends up doing character set
649 conversions. It might not be possible to do a real character
650 set based conversion, due to the fact that the template
651 parameter for messages is not enough to instantiate the
652 codecvt facet (1 supplied, need at least 2 but would prefer
654 </p></li><li class="listitem"><p>
655 There are issues with gettext needing the global locale set
656 to extract a message. This dependence on the global locale
657 makes the current "gnu" model non MT-safe. Future versions
658 of glibc, i.e. glibc 2.3.x will fix this, and the C++ library
659 bits are already in place.
660 </p></li></ul></div></li><li class="listitem"><p>
661 Development versions of the GNU "C" library, glibc 2.3 will allow
662 a more efficient, MT implementation of std::messages, and will
663 allow the removal of the _M_name_messages data member. If this is
664 done, it will change the library ABI. The C++ parts to support
665 glibc 2.3 have already been coded, but are not in use: once this
666 version of the "C" library is released, the marked parts of the
667 messages implementation can be switched over to the new "C"
668 library functionality.
669 </p></li><li class="listitem"><p>
670 At some point in the near future, std::numpunct will probably use
671 std::messages facilities to implement truename/falsename
672 correctly. This is currently not done, but entries in
673 libstdc++.pot have already been made for "true" and "false" string
674 literals, so all that remains is the std::numpunct coding and the
675 configure/make hassles to make the installed library search its
676 own catalog. Currently the libstdc++.mo catalog is only searched
677 for the testsuite cases involving messages members.
678 </p></li><li class="listitem"><p> The following member functions:</p><p>
681 open(const basic_string<char>& __s, const locale& __loc) const
686 open(const basic_string<char>&, const locale&, const char*) const;
689 Don't actually return a "value less than 0 if no such catalog
690 can be opened" as required by the standard in the "gnu"
691 model. As of this writing, it is unknown how to query to see
692 if a specified message catalog exists using the gettext
694 </p></li></ul></div></div><div class="bibliography" title="Bibliography"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.biblio"/>Bibliography</h4></div></div></div><div class="biblioentry"><a id="id611002"/><p><span class="citetitle"><em class="citetitle">
696 </em>. </span><span class="author"><span class="firstname">Roland</span> <span class="surname">McGrath</span>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2007 FSF. </span><span class="pagenums">Chapters 6 Character Set Handling, and 7 Locales and Internationalization
697 . </span></p></div><div class="biblioentry"><a id="id611041"/><p><span class="citetitle"><em class="citetitle">
699 </em>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2002 . </span></p></div><div class="biblioentry"><a id="id611067"/><p><span class="citetitle"><em class="citetitle">
700 ISO/IEC 14882:1998 Programming languages - C++
701 </em>. </span><span class="copyright">Copyright © 1998 ISO. </span></p></div><div class="biblioentry"><a id="id611086"/><p><span class="citetitle"><em class="citetitle">
702 ISO/IEC 9899:1999 Programming languages - C
703 </em>. </span><span class="copyright">Copyright © 1999 ISO. </span></p></div><div class="biblioentry" title="System Interface Definitions, Issue 7 (IEEE Std. 1003.1-2008)"><a id="id611105"/><p><span class="title"><em>
704 <a class="link" href="http://www.opengroup.org/austin">
705 System Interface Definitions, Issue 7 (IEEE Std. 1003.1-2008)
707 </em>. </span><span class="copyright">Copyright © 2008
708 The Open Group/The Institute of Electrical and Electronics
710 . </span></p></div><div class="biblioentry"><a id="id611132"/><p><span class="citetitle"><em class="citetitle">
711 The C++ Programming Language, Special Edition
712 </em>. </span><span class="author"><span class="firstname">Bjarne</span> <span class="surname">Stroustrup</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley, Inc.. </span><span class="pagenums">Appendix D. </span><span class="publisher"><span class="publishername">
714 . </span></span></p></div><div class="biblioentry"><a id="id611171"/><p><span class="citetitle"><em class="citetitle">
715 Standard C++ IOStreams and Locales
716 </em>. </span><span class="subtitle">
717 Advanced Programmer's Guide and Reference
718 . </span><span class="author"><span class="firstname">Angelika</span> <span class="surname">Langer</span>. </span><span class="author"><span class="firstname">Klaus</span> <span class="surname">Kreft</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley Longman, Inc.. </span><span class="publisher"><span class="publishername">
719 Addison Wesley Longman
720 . </span></span></p></div><div class="biblioentry" title="API Specifications, Java Platform"><a id="id611218"/><p><span class="title"><em>
721 <a class="link" href="http://java.sun.com/reference/api/index.html">
722 API Specifications, Java Platform
724 </em>. </span><span class="pagenums">java.util.Properties, java.text.MessageFormat,
725 java.util.Locale, java.util.ResourceBundle
726 . </span></p></div><div class="biblioentry" title="GNU gettext tools, version 0.10.38, Native Language Support Library and Tools."><a id="id611237"/><p><span class="title"><em>
727 <a class="link" href="http://www.gnu.org/software/gettext">
728 GNU gettext tools, version 0.10.38, Native Language Support
731 </em>. </span></p></div></div></div></div><div class="navfooter"><hr/><table width="100%" summary="Navigation footer"><tr><td align="left"><a accesskey="p" href="localization.html">Prev</a> </td><td align="center"><a accesskey="u" href="localization.html">Up</a></td><td align="right"> <a accesskey="n" href="containers.html">Next</a></td></tr><tr><td align="left" valign="top">Chapter 8.
734 </td><td align="center"><a accesskey="h" href="../index.html">Home</a></td><td align="right" valign="top"> Chapter 9.
737 </td></tr></table></div></body></html>