In computing, a character encoding is used to represent a repertoire of characters by
some kind of an encoding system. Depending on the abstraction level and context, corresponding
code points and the resulting code space may be regarded as bit patterns, octets, natural
numbers, electrical pulses, etc. A character encoding is used in both computation, data
storage, and transmission of textual data. Terms such as character set, character map,
codeset or code page are sometimes used as near synonyms; however, these terms have related
but distinct meanings described in the article. Early character codes associated with the
optical or electrical telegraph could only represent a subset of the characters used
in written language, sometimes restricted to upper case letters, numerals and some punctuation
only. The low cost of digital representation of data in modern computer systems allows
more elaborate character codes which represent more of the characters used in many written
languages. Character encoding using internationally accepted standards permits worldwide interchange
of text in electronic form.
History Early binary repertoires include Bacon's cipher,
Braille, International maritime signal flags, and the 4-digit encoding of Chinese characters
for a Chinese telegraph code. Common examples of character encoding systems include Morse
code, the Baudot code, the American Standard Code for Information Interchange and Unicode.
Morse code was introduced in the 1840s and is used to encode each letter of the Latin
alphabet, each Arabic numeral, and some other characters via a series of long and short
presses of a telegraph key. Representations of characters encoded using Morse code varied
in length. The Baudot code, a 5-bit encoding, was created
by Émile Baudot in 1870, patented in 1874, modified by Donald Murray in 1901, and standardized
by CCITT as International Telegraph Alphabet No. 2 in 1930.
Fieldata, a 6- or 7-bit code, was introduced by the U.S. Army Signal Corp in the late 1950s.
IBM's Binary Coded Decimal was a 6-bit encoding scheme used by IBM in as early as 1959 in
its 1401 and 1620 computers, and in its 7000 Series, as well as in associated peripherals.
BCD extended existing simple 4-bit numeric encoding to include alphabetic and special
characters, mapping it easily to punch-card encoding which was already in widespread use.
It was the precursor to EBCDIC. ASCII was introduced in 1963 and is a 7-bit
encoding scheme used to encode letters, numerals, symbols, and device control codes as fixed-length
codes using integers. IBM's Extended Binary Coded Decimal Interchange
Code is an 8-bit encoding scheme developed in 1963.
The limitations of such sets soon became apparent, and a number of ad hoc methods were developed
to extend them. The need to support more writing systems for different languages, including
the CJK family of East Asian scripts, required support for a far larger number of characters
and demanded a systematic approach to character encoding rather than the previous ad hoc approaches.
Code unit A code unit is a bit sequence used to encode
the characters of a repertoire. With US-ASCII, code unit is 7 bits.
With UTF-8, code unit is 8 bits. With EBCDIC, code unit is 8 bits.
With UTF-16, code unit is 16 bits. With UTF-32, code unit is 32 bits.
Encodings associate their meaning with either a single code unit value or a sequence of
code units as one value. Unicode encoding model
Unicode and its parallel standard, the ISO/IEC 10646 Universal Character Set, together constitute
a modern, unified character encoding. Rather than mapping characters directly to octets,
they separately define what characters are available, their numbering, how those numbers
are encoded as a series of "code units", and finally how those units are encoded as a stream
of octets. The idea behind this decomposition is to establish a universal set of characters
that can be encoded in a variety of ways. To describe this model correctly one needs
more precise terms than "character set" and "character encoding." The terms used in the
modern model follow: A character repertoire is the full set of
abstract characters that a system supports. The repertoire may be closed, i.e. no additions
are allowed without creating a new standard, or it may be open, allowing additions. The
characters in a given repertoire reflect decisions that have been made about how to divide writing
systems into basic information units. The basic variants of the Latin, Greek, and Cyrillic
alphabets, can be broken down into letters, digits, punctuation, and a few special characters
like the space, which can all be arranged in simple linear sequences that are displayed
in the same order they are read. Even with these alphabets, however, diacritics pose
a complication: they can be regarded either as part of a single character containing a
letter and diacritic, or as separate characters. The former allows a far simpler text handling
system but the latter allows any letter/diacritic combination to be used in text. Ligatures
pose similar problems. Other writing systems, such as Arabic and Hebrew, are represented
with more complex character repertoires due to the need to accommodate things like bidirectional
text and glyphs that are joined together in different ways for different situations.
A coded character set specifies how to represent a repertoire of characters using a number
of integer values called code points. For example, in a given repertoire, a character
representing the capital letter "A" in the Latin alphabet might be assigned to the integer
65, the character for "B" to 66, and so on. A complete set of characters and corresponding
integers is a coded character set. Multiple coded character sets may share the same repertoire;
for example ISO/IEC 8859-1 and IBM code pages 037 and 500 all cover the same repertoire
but map them to different codes. In a coded character set, each code point only represents
one character, i.e., a coded character set is a function.
A character encoding form specifies the conversion of a coded character set's integer codes into
a set of limited-size integer code values that facilitate storage in a system that represents
numbers in binary form using a fixed number of bits. For example, a system that stores
numeric information in 16-bit units would only be able to directly represent integers
from 0 to 65,535 in each unit, but larger integers could be represented if more than
one 16-bit unit could be used. This is what a CEF accommodates: it defines a way of mapping
a single code point from a range of, say, 0 to 1.4 million, to a series of one or more
code values from a range of, say, 0 to 65,535. The simplest CEF system is simply to choose
large enough units that the values from the coded character set can be encoded directly.
This works well for coded character sets that fit in 8 bits and reasonably well for coded
character sets that fit in 16 bits. However, as the size of the coded character set increases,
this becomes less and less efficient, and it is difficult to adapt existing systems
to use larger code values. Therefore, most systems working with later versions of Unicode
use either UTF-8, which maps Unicode code points to variable-length sequences of octets,
or UTF-16, which maps Unicode code points to variable-length sequences of 16-bit words.
Next, a character encoding scheme specifies how the fixed-size integer code values should
be mapped into an octet sequence suitable for saving on an octet-based file system or
transmitting over an octet-based network. With Unicode, a simple character encoding
scheme is used in most cases, simply specifying whether the bytes for each integer should
be in big-endian or little-endian order. However, there are also compound character encoding
schemes, which use escape sequences to switch between several simple schemes, and compressing
schemes, which try to minimise the number of bytes used per code unit. See comparison
of Unicode encodings for a detailed discussion. Finally, there may be a higher level protocol
which supplies additional information that can be used to select the particular variant
of a Unicode character, particularly where there are regional variants that have been
'unified' in Unicode as the same character. An example is the XML attribute xml:lang.
The Unicode model reserves the term character map for historical systems which directly
assign a sequence of characters to a sequence of bytes, covering all of CCS, CEF and CES
layers. Character sets, maps and code pages
In computer science, the terms "character encoding", "character map", "character set"
and "code page" were historically synonymous, as the same standard would specify a repertoire
of characters and how they were to be encoded into a stream of code units – usually with
a single character per code unit. The terms now have related but distinct meanings, reflecting
the efforts of standards bodies to use precise terminology when writing about and unifying
many different encoding systems. Regardless, the terms are still used interchangeably,
with character set being nearly ubiquitous. A "code page" usually means a byte oriented
encoding, but with regard to some suite of encodings, where many characters share the
same codes in most or all those code pages. Well known code page suites are "Windows"
and "IBM"/"DOS", see Windows code page for details. Most, but not all, encodings referred
to as code pages are single-byte encodings IBM's Character Data Representation Architecture
designates with coded character set identifiers and each of which is variously called a "charset",
"character set", "code page", or "CHARMAP". The term "code page" does not occur in Unix
or Linux where "charmap" is preferred, usually in the larger context of locales.
Contrasted to CCS above, a "character encoding" is a map from abstract characters to code
words. A "character set" in HTTP parlance is the same as a character encoding.
"Legacy encoding" is a term sometimes used to characterize old character encodings, but
with an ambiguity of sense. Most of its use is in the context of Unicodification, where
it refers to encodings that fail to cover all Unicode code points, or, more generally,
using a somewhat different character repertoire: several code points representing one Unicode
character, or versa. Some sources refer to an encoding as legacy only because it preceded
Unicode. All Windows code pages are usually referred to as legacy, both because they antedate
Unicode and because they are unable to represent all 221 possible Unicode code points.
Character encoding translation As a result of having many character encoding
methods in use, many computer programs have been developed to translate data between encoding
schemes as a form of data transcoding. Some of these are cited below.
Cross-platform: Web browsers – most modern web browsers
feature automatic character encoding detection. On Firefox 3, for example, see the View/Character
Encoding submenu. iconv – program and standardized API to
convert encodings luit – program that converts encoding of
input and output to programs running interactively convert_encoding.py – Python based utility
to convert text files between arbitrary encodings and line endings.
decodeh.py – algorithm and module to heuristically guess the encoding of a string.
International Components for Unicode – A set of C and Java libraries to perform charset
conversion. uconv can be used from ICU4C. chardet – This is a translation of the Mozilla
automatic-encoding-detection code into the Python computer language.
The newer versions of the Unix file command attempt to do a basic detection of character
encoding. charset - C++ template library with simple
interface to convert between C++\user-defined streams. charset defined many character-sets
and allows you to use Unicode formats with support of endianness.
Unix-like: cmv – simple tool for transcoding filenames.
convmv – convert a filename from one encoding to another.
cstocs – convert file contents from one encoding to another for the Czech and Slovak
languages. enca – analyzes encodings for given text
files. recode – convert file contents from one
encoding to another utrac – convert file contents from one encoding
to another. Windows:
Encoding.Convert – .NET API MultiByteToWideChar/WideCharToMultiByte – Convert
from ANSI to Unicode & Unicode to ANSI cscvt – character set conversion tool
enca – analyzes encodings for given text files.
See also Alt code
Character encodings in HTML Character encoding – articles related to
character encoding in general Character sets – articles detailing specific
character encodings Hexadecimal representations
Mojibake – character set mismap. Mojikyo – a system that includes over 100,000
Chinese character drawings, modern and ancient, popular and obscure.
TRON, part of the TRON project, is an encoding system that does not use Han Unification,
instead it uses 'control codes' to switch between 16bit 'planes' of characters.
Universal Character Set characters Windows code page – various character set
encodings used by Microsoft Windows Charset sniffing – used in some applications
when character encoding metadata is not available Common character encodings
References
Further reading Mackenzie, Charles E.. Coded Character Sets,
History and Development. Addison-Wesley. ISBN 0-201-14460-3. External links
Character sets registered by Internet Assigned Numbers Authority
Jukka Korpela: Characters and encodings Unicode Technical Report #17: Character Encoding
Model