[PATCH] Reset converter state after second wchar_t output (Bug 25734)

[Tom Honermann]

On 3/30/20 10:34 AM, Carlos O'Donell wrote:
> On 3/30/20 8:11 AM, Andreas Schwab wrote:
>> On Mär 27 2020, Carlos O'Donell wrote:
>>
>>> An input BIG5-HKSCS character may be converted into at most 2 whcar_t
>>> characters.
>> Could someone please file an interpretation request for POSIX, what
>> should happen in that two wchar_t case?  I think a case could be made
>> that mbrtowc should return -1/EILSEQ, but that also has implications for
>> mbsrtowcs, since that is defined in terms of repeated application of
>> mbrtowc.
> Tom had already reached out to ISO C WG14 to discuss this since there is
> harmonization there between C and POSIX.
>
> The current thinking in WG14 is that BIG5-HKSCS violates the definition
> of "wide character" because the wchar_t value cannot represent the
> original character in the locale and so the semantics underlying
> BIG5-HKSCS will not map to the current API designs.
>
> Florian raised a similar issue in May of 2019 and the general feedback
> at that time was that BIG5-HKSCS is simply not supported by ISO C.
> I expect the same answer from POSIX which is harmonized with ISO C in
> this case.
>
> If BIG5-HKSCS is not supported, then the standard will have nothing to
> say about which values can be returned after the first or second input
> bytes are read.

If this is determined to be the case, then my preference is for option 
(c) below as this has the advantage of adopting behavior that is already 
standard for mbrtoc16.

Tom.

>
> Options:
>
> (a) Do not change the current converter. We return 2 consumed bytes in
>      the first conversion, and 0 on the second.
>
> (b) Look ahead and split the conversion. We return 1 consumed in the
>      first conversion, and 1 on the second. This prevents us from returning
>      0 which may be interpreted as a L'\0'. This leads to false assumption
>      that the user could stop the conversion at this point and modify the
>      input.
>
> (c) Design something new. Return (size_t) -3 indicating a One-to-Many
>      conversion is happening and that there is more output to be generated.
>
> Do you still want me to file an interpretation request with POSIX?
>