1 | UTF-8 decoder capability and stress test
|
---|
2 | ----------------------------------------
|
---|
3 |
|
---|
4 | Markus Kuhn <http://www.cl.cam.ac.uk/~mgk25/> - 2003-02-19
|
---|
5 |
|
---|
6 | This test file can help you examine, how your UTF-8 decoder handles
|
---|
7 | various types of correct, malformed, or otherwise interesting UTF-8
|
---|
8 | sequences. This file is not meant to be a conformance test. It does
|
---|
9 | not prescribes any particular outcome and therefore there is no way to
|
---|
10 | "pass" or "fail" this test file, even though the texts suggests a
|
---|
11 | preferable decoder behaviour at some places. The aim is instead to
|
---|
12 | help you think about and test the behaviour of your UTF-8 on a
|
---|
13 | systematic collection of unusual inputs. Experience so far suggests
|
---|
14 | that most first-time authors of UTF-8 decoders find at least one
|
---|
15 | serious problem in their decoder by using this file.
|
---|
16 |
|
---|
17 | The test lines below cover boundary conditions, malformed UTF-8
|
---|
18 | sequences as well as correctly encoded UTF-8 sequences of Unicode code
|
---|
19 | points that should never occur in a correct UTF-8 file.
|
---|
20 |
|
---|
21 | According to ISO 10646-1:2000, sections D.7 and 2.3c, a device
|
---|
22 | receiving UTF-8 shall interpret a "malformed sequence in the same way
|
---|
23 | that it interprets a character that is outside the adopted subset" and
|
---|
24 | "characters that are not within the adopted subset shall be indicated
|
---|
25 | to the user" by a receiving device. A quite commonly used approach in
|
---|
26 | UTF-8 decoders is to replace any malformed UTF-8 sequence by a
|
---|
27 | replacement character (U+FFFD), which looks a bit like an inverted
|
---|
28 | question mark, or a similar symbol. It might be a good idea to
|
---|
29 | visually distinguish a malformed UTF-8 sequence from a correctly
|
---|
30 | encoded Unicode character that is just not available in the current
|
---|
31 | font but otherwise fully legal, even though ISO 10646-1 doesn't
|
---|
32 | mandate this. In any case, just ignoring malformed sequences or
|
---|
33 | unavailable characters does not conform to ISO 10646, will make
|
---|
34 | debugging more difficult, and can lead to user confusion.
|
---|
35 |
|
---|
36 | Please check, whether a malformed UTF-8 sequence is (1) represented at
|
---|
37 | all, (2) represented by exactly one single replacement character (or
|
---|
38 | equivalent signal), and (3) the following quotation mark after an
|
---|
39 | illegal UTF-8 sequence is correctly displayed, i.e. proper
|
---|
40 | resynchronization takes place immageately after any malformed
|
---|
41 | sequence. This file says "THE END" in the last line, so if you don't
|
---|
42 | see that, your decoder crashed somehow before, which should always be
|
---|
43 | cause for concern.
|
---|
44 |
|
---|
45 | All lines in this file are exactly 79 characters long (plus the line
|
---|
46 | feed). In addition, all lines end with "|", except for the two test
|
---|
47 | lines 2.1.1 and 2.2.1, which contain non-printable ASCII controls
|
---|
48 | U+0000 and U+007F. If you display this file with a fixed-width font,
|
---|
49 | these "|" characters should all line up in column 79 (right margin).
|
---|
50 | This allows you to test quickly, whether your UTF-8 decoder finds the
|
---|
51 | correct number of characters in every line, that is whether each
|
---|
52 | malformed sequences is replaced by a single replacement character.
|
---|
53 |
|
---|
54 | Note that as an alternative to the notion of malformed sequence used
|
---|
55 | here, it is also a perfectly acceptable (and in some situations even
|
---|
56 | preferable) solution to represent each individual byte of a malformed
|
---|
57 | sequence by a replacement character. If you follow this strategy in
|
---|
58 | your decoder, then please ignore the "|" column.
|
---|
59 |
|
---|
60 |
|
---|
61 | Here come the tests: |
|
---|
62 | |
|
---|
63 | 1 Some correct UTF-8 text |
|
---|
64 | |
|
---|
65 | You should see the Greek word 'kosme': "κόÏμε" |
|
---|
66 | |
|
---|
67 | 2 Boundary condition test cases |
|
---|
68 | |
|
---|
69 | 2.1 First possible sequence of a certain length |
|
---|
70 | |
|
---|
71 | 2.1.1 1 byte (U-00000000): ""
|
---|
72 | 2.1.2 2 bytes (U-00000080): "Â" |
|
---|
73 | 2.1.3 3 bytes (U-00000800): "à " |
|
---|
74 | 2.1.4 4 bytes (U-00010000): "ð" |
|
---|
75 | 2.1.5 5 bytes (U-00200000): "ø" |
|
---|
76 | 2.1.6 6 bytes (U-04000000): "ü" |
|
---|
77 | |
|
---|
78 | 2.2 Last possible sequence of a certain length |
|
---|
79 | |
|
---|
80 | 2.2.1 1 byte (U-0000007F): ""
|
---|
81 | 2.2.2 2 bytes (U-000007FF): "ß¿" |
|
---|
82 | 2.2.3 3 bytes (U-0000FFFF): "ï¿¿" |
|
---|
83 | 2.2.4 4 bytes (U-001FFFFF): "÷¿¿¿" |
|
---|
84 | 2.2.5 5 bytes (U-03FFFFFF): "û¿¿¿¿" |
|
---|
85 | 2.2.6 6 bytes (U-7FFFFFFF): "ý¿¿¿¿¿" |
|
---|
86 | |
|
---|
87 | 2.3 Other boundary conditions |
|
---|
88 | |
|
---|
89 | 2.3.1 U-0000D7FF = ed 9f bf = "í¿" |
|
---|
90 | 2.3.2 U-0000E000 = ee 80 80 = "î" |
|
---|
91 | 2.3.3 U-0000FFFD = ef bf bd = "�" |
|
---|
92 | 2.3.4 U-0010FFFF = f4 8f bf bf = "ô¿¿" |
|
---|
93 | 2.3.5 U-00110000 = f4 90 80 80 = "ô" |
|
---|
94 | |
|
---|
95 | 3 Malformed sequences |
|
---|
96 | |
|
---|
97 | 3.1 Unexpected continuation bytes |
|
---|
98 | |
|
---|
99 | Each unexpected continuation byte should be separately signalled as a |
|
---|
100 | malformed sequence of its own. |
|
---|
101 | |
|
---|
102 | 3.1.1 First continuation byte 0x80: "" |
|
---|
103 | 3.1.2 Last continuation byte 0xbf: "¿" |
|
---|
104 | |
|
---|
105 | 3.1.3 2 continuation bytes: "¿" |
|
---|
106 | 3.1.4 3 continuation bytes: "¿" |
|
---|
107 | 3.1.5 4 continuation bytes: "¿¿" |
|
---|
108 | 3.1.6 5 continuation bytes: "¿¿" |
|
---|
109 | 3.1.7 6 continuation bytes: "¿¿¿" |
|
---|
110 | 3.1.8 7 continuation bytes: "¿¿¿" |
|
---|
111 | |
|
---|
112 | 3.1.9 Sequence of all 64 possible continuation bytes (0x80-0xbf): |
|
---|
113 | |
|
---|
114 | "
|
---|
115 | |
|
---|
116 | |
|
---|
117 | ¡¢£¤¥¦§¨©ª«¬®¯ |
|
---|
118 | °±²³´µ¶·¸¹º»¼½¾¿" |
|
---|
119 | |
|
---|
120 | 3.2 Lonely start characters |
|
---|
121 | |
|
---|
122 | 3.2.1 All 32 first bytes of 2-byte sequences (0xc0-0xdf), |
|
---|
123 | each followed by a space character: |
|
---|
124 | |
|
---|
125 | "À Á Â Ã Ä Å Æ Ç È É Ê Ë Ì Í Î Ï |
|
---|
126 | Ð Ñ Ò Ó Ô Õ Ö × Ø Ù Ú Û Ü Ý Þ ß " |
|
---|
127 | |
|
---|
128 | 3.2.2 All 16 first bytes of 3-byte sequences (0xe0-0xef), |
|
---|
129 | each followed by a space character: |
|
---|
130 | |
|
---|
131 | "à á â ã ä å æ ç è é ê ë ì í î ï " |
|
---|
132 | |
|
---|
133 | 3.2.3 All 8 first bytes of 4-byte sequences (0xf0-0xf7), |
|
---|
134 | each followed by a space character: |
|
---|
135 | |
|
---|
136 | "ð ñ ò ó ô õ ö ÷ " |
|
---|
137 | |
|
---|
138 | 3.2.4 All 4 first bytes of 5-byte sequences (0xf8-0xfb), |
|
---|
139 | each followed by a space character: |
|
---|
140 | |
|
---|
141 | "ø ù ú û " |
|
---|
142 | |
|
---|
143 | 3.2.5 All 2 first bytes of 6-byte sequences (0xfc-0xfd), |
|
---|
144 | each followed by a space character: |
|
---|
145 | |
|
---|
146 | "ü ý " |
|
---|
147 | |
|
---|
148 | 3.3 Sequences with last continuation byte missing |
|
---|
149 | |
|
---|
150 | All bytes of an incomplete sequence should be signalled as a single |
|
---|
151 | malformed sequence, i.e., you should see only a single replacement |
|
---|
152 | character in each of the next 10 tests. (Characters as in section 2) |
|
---|
153 | |
|
---|
154 | 3.3.1 2-byte sequence with last byte missing (U+0000): "À" |
|
---|
155 | 3.3.2 3-byte sequence with last byte missing (U+0000): "à" |
|
---|
156 | 3.3.3 4-byte sequence with last byte missing (U+0000): "ð" |
|
---|
157 | 3.3.4 5-byte sequence with last byte missing (U+0000): "ø" |
|
---|
158 | 3.3.5 6-byte sequence with last byte missing (U+0000): "ü" |
|
---|
159 | 3.3.6 2-byte sequence with last byte missing (U-000007FF): "ß" |
|
---|
160 | 3.3.7 3-byte sequence with last byte missing (U-0000FFFF): "ï¿" |
|
---|
161 | 3.3.8 4-byte sequence with last byte missing (U-001FFFFF): "÷¿¿" |
|
---|
162 | 3.3.9 5-byte sequence with last byte missing (U-03FFFFFF): "û¿¿¿" |
|
---|
163 | 3.3.10 6-byte sequence with last byte missing (U-7FFFFFFF): "ý¿¿¿¿" |
|
---|
164 | |
|
---|
165 | 3.4 Concatenation of incomplete sequences |
|
---|
166 | |
|
---|
167 | All the 10 sequences of 3.3 concatenated, you should see 10 malformed |
|
---|
168 | sequences being signalled: |
|
---|
169 | |
|
---|
170 | "Ààðøüßï¿÷¿¿û¿¿¿ý¿¿¿¿" |
|
---|
171 | |
|
---|
172 | 3.5 Impossible bytes |
|
---|
173 | |
|
---|
174 | The following two bytes cannot appear in a correct UTF-8 string |
|
---|
175 | |
|
---|
176 | 3.5.1 fe = "þ" |
|
---|
177 | 3.5.2 ff = "ÿ" |
|
---|
178 | 3.5.3 fe fe ff ff = "þþÿÿ" |
|
---|
179 | |
|
---|
180 | 4 Overlong sequences |
|
---|
181 | |
|
---|
182 | The following sequences are not malformed according to the letter of |
|
---|
183 | the Unicode 2.0 standard. However, they are longer then necessary and |
|
---|
184 | a correct UTF-8 encoder is not allowed to produce them. A "safe UTF-8 |
|
---|
185 | decoder" should reject them just like malformed sequences for two |
|
---|
186 | reasons: (1) It helps to debug applications if overlong sequences are |
|
---|
187 | not treated as valid representations of characters, because this helps |
|
---|
188 | to spot problems more quickly. (2) Overlong sequences provide |
|
---|
189 | alternative representations of characters, that could maliciously be |
|
---|
190 | used to bypass filters that check only for ASCII characters. For |
|
---|
191 | instance, a 2-byte encoded line feed (LF) would not be caught by a |
|
---|
192 | line counter that counts only 0x0a bytes, but it would still be |
|
---|
193 | processed as a line feed by an unsafe UTF-8 decoder later in the |
|
---|
194 | pipeline. From a security point of view, ASCII compatibility of UTF-8 |
|
---|
195 | sequences means also, that ASCII characters are *only* allowed to be |
|
---|
196 | represented by ASCII bytes in the range 0x00-0x7f. To ensure this |
|
---|
197 | aspect of ASCII compatibility, use only "safe UTF-8 decoders" that |
|
---|
198 | reject overlong UTF-8 sequences for which a shorter encoding exists. |
|
---|
199 | |
|
---|
200 | 4.1 Examples of an overlong ASCII character |
|
---|
201 | |
|
---|
202 | With a safe UTF-8 decoder, all of the following five overlong |
|
---|
203 | representations of the ASCII character slash ("/") should be rejected |
|
---|
204 | like a malformed UTF-8 sequence, for instance by substituting it with |
|
---|
205 | a replacement character. If you see a slash below, you do not have a |
|
---|
206 | safe UTF-8 decoder! |
|
---|
207 | |
|
---|
208 | 4.1.1 U+002F = c0 af = "À¯" |
|
---|
209 | 4.1.2 U+002F = e0 80 af = "à¯" |
|
---|
210 | 4.1.3 U+002F = f0 80 80 af = "ð¯" |
|
---|
211 | 4.1.4 U+002F = f8 80 80 80 af = "ø¯" |
|
---|
212 | 4.1.5 U+002F = fc 80 80 80 80 af = "ü¯" |
|
---|
213 | |
|
---|
214 | 4.2 Maximum overlong sequences |
|
---|
215 | |
|
---|
216 | Below you see the highest Unicode value that is still resulting in an |
|
---|
217 | overlong sequence if represented with the given number of bytes. This |
|
---|
218 | is a boundary test for safe UTF-8 decoders. All five characters should |
|
---|
219 | be rejected like malformed UTF-8 sequences. |
|
---|
220 | |
|
---|
221 | 4.2.1 U-0000007F = c1 bf = "Á¿" |
|
---|
222 | 4.2.2 U-000007FF = e0 9f bf = "à¿" |
|
---|
223 | 4.2.3 U-0000FFFF = f0 8f bf bf = "ð¿¿" |
|
---|
224 | 4.2.4 U-001FFFFF = f8 87 bf bf bf = "ø¿¿¿" |
|
---|
225 | 4.2.5 U-03FFFFFF = fc 83 bf bf bf bf = "ü¿¿¿¿" |
|
---|
226 | |
|
---|
227 | 4.3 Overlong representation of the NUL character |
|
---|
228 | |
|
---|
229 | The following five sequences should also be rejected like malformed |
|
---|
230 | UTF-8 sequences and should not be treated like the ASCII NUL |
|
---|
231 | character. |
|
---|
232 | |
|
---|
233 | 4.3.1 U+0000 = c0 80 = "À" |
|
---|
234 | 4.3.2 U+0000 = e0 80 80 = "à" |
|
---|
235 | 4.3.3 U+0000 = f0 80 80 80 = "ð" |
|
---|
236 | 4.3.4 U+0000 = f8 80 80 80 80 = "ø" |
|
---|
237 | 4.3.5 U+0000 = fc 80 80 80 80 80 = "ü" |
|
---|
238 | |
|
---|
239 | 5 Illegal code positions |
|
---|
240 | |
|
---|
241 | The following UTF-8 sequences should be rejected like malformed |
|
---|
242 | sequences, because they never represent valid ISO 10646 characters and |
|
---|
243 | a UTF-8 decoder that accepts them might introduce security problems |
|
---|
244 | comparable to overlong UTF-8 sequences. |
|
---|
245 | |
|
---|
246 | 5.1 Single UTF-16 surrogates |
|
---|
247 | |
|
---|
248 | 5.1.1 U+D800 = ed a0 80 = "í " |
|
---|
249 | 5.1.2 U+DB7F = ed ad bf = "í¿" |
|
---|
250 | 5.1.3 U+DB80 = ed ae 80 = "í®" |
|
---|
251 | 5.1.4 U+DBFF = ed af bf = "í¯¿" |
|
---|
252 | 5.1.5 U+DC00 = ed b0 80 = "í°" |
|
---|
253 | 5.1.6 U+DF80 = ed be 80 = "í¾" |
|
---|
254 | 5.1.7 U+DFFF = ed bf bf = "í¿¿" |
|
---|
255 | |
|
---|
256 | 5.2 Paired UTF-16 surrogates |
|
---|
257 | |
|
---|
258 | 5.2.1 U+D800 U+DC00 = ed a0 80 ed b0 80 = "í í°" |
|
---|
259 | 5.2.2 U+D800 U+DFFF = ed a0 80 ed bf bf = "í í¿¿" |
|
---|
260 | 5.2.3 U+DB7F U+DC00 = ed ad bf ed b0 80 = "í¿í°" |
|
---|
261 | 5.2.4 U+DB7F U+DFFF = ed ad bf ed bf bf = "í¿í¿¿" |
|
---|
262 | 5.2.5 U+DB80 U+DC00 = ed ae 80 ed b0 80 = "í®í°" |
|
---|
263 | 5.2.6 U+DB80 U+DFFF = ed ae 80 ed bf bf = "í®í¿¿" |
|
---|
264 | 5.2.7 U+DBFF U+DC00 = ed af bf ed b0 80 = "í¯¿í°" |
|
---|
265 | 5.2.8 U+DBFF U+DFFF = ed af bf ed bf bf = "􏿿" |
|
---|
266 | |
|
---|
267 | 5.3 Other illegal code positions |
|
---|
268 | |
|
---|
269 | 5.3.1 U+FFFE = ef bf be = "￾" |
|
---|
270 | 5.3.2 U+FFFF = ef bf bf = "ï¿¿" |
|
---|
271 | |
|
---|
272 | THE END |
|
---|