Unicode support in busybox There are several scenarios where we need to handle unicode correctly. Shell input We want to correctly handle input of unicode characters. There are several problems with it. Just handling input as sequence of bytes would break any editing. This was fixed and now lineedit operates on the array of wchar_t's. But we also need to handle the following problematic moments: * It is unreasonable to expect that output device supports _any_ unicode chars. Perhaps we need to avoid printing those chars which are not supported by output device. Examples: chars which are not present in the font, chars which are not assigned in unicode, combining chars (especially trying to combine bad pairs: a_chinese_symbol + "combining grave accent" = ??!) * We need to account for the fact that unicode chars have different widths: 0 for combining chars, 1 for usual, 2 for ideograms (are there 3+ wide chars?). * Bidirectional handling. If user wants to echo a phrase in Hebrew, he types: echo "srettel werbeH" Editors (vi, ed) This case is a bit similar to "shell input", but unlike shell, editors may encounder many more unexpected unicode sequences (try to load a random binary file...), and they need to preserve them, unlike shell which can afford to drop bogus input. more, less Need to correctly display any input file. Ideally, with ASCII/unicode/filtered_unicode option or keyboard switch. Note: need to handle tabs and backspaces specially (bksp is for manpage compat). cut, fold, watch May need ability to cut unicode string to specified number of wchars and/or to specified screen width. Need to handle tabs specially. sed, awk, grep Handle unicode-aware regexp match ls (multi-column display) ls will fail to line up columnar output if it will not account for character widths (and maybe filter out some of them, see above). OTOH, non-columnar views (ls -1, ls -l, ls | car) should NOT filter out bad unicode (but need to filter out control chars (coreutils does that). Note that unlike more/less, tabs and backspaces need not special handling. top, ps Need to perform filtering similar to ls. Filename display (in error messages and elsewhere) Need to perform filtering similar to ls. TODO: write an email to Asmus Freytag (asmus@unicode.org), author of http://unicode.org/reports/tr11/