View Issue Details

IDProjectCategoryView StatusLast Update
0020350mantisbtotherpublic2016-08-15 09:02
Reportertomop Assigned Todregad  
PrioritynormalSeverityminorReproducibilityalways
Status closedResolutionfixed 
Product Version1.2.6 
Target Version1.2.20Fixed in Version1.2.20 
Summary0020350: json_url() may break non-ASCII strings.
Description

core/json_api.php

$t_json = json_decode( utf8_encode( $t_data ) );

utf8_encode() must be called with ISO-8859-1 strings but JSON shall be written in Unicode (UTF-8, UTF-16, ...).

I'm using Source Control Integration plugin that calls json_url(). All of commit logs in Japanese was broken. And I tried to remove utf8_encode() then I got expected results.

TagsNo tags attached.

Activities

dregad

dregad

2015-12-07 10:13

developer   ~0052042

Last edited: 2015-12-08 03:05

I agree with you that JSON strings should be Unicode-encoded to begin with, so the call to utf8_encode() seems incorrect.

This was introduced by merging pull request https://github.com/mantisbt/mantisbt/pull/1 so I have asked the contributor to clarify why this was needed in the first place.

I believe this commit should be reverted.

tomop

tomop

2015-12-07 20:53

reporter   ~0052062

I commented to https://github.com/mantisbt/mantisbt/pull/1 .

If you want to support bad implementations to send not Unicode-encoded strings, change url_get() in core/url_api.php

  • detect encodings with "Content-Type" HTTP header.
  • use iconv() or mb_convert_encoding() with detected encoding to encode to UTF-8 from other encodings.

note:
mb_detect_encoding() and passing "auto" to mb_convert_encoding() may return wrong result.

tomop

tomop

2015-12-08 00:43

reporter   ~0052066

I tried to retrieve and parse logs from the contributor's repository.


$ curl 'https://api.github.com/repos/01-Scripts/01-Artikelsystem/commits?per_page=50' > commits.txt
$ file commits.txt
commits.txt: UTF-8 Unicode text
$ php -r 'var_dump(json_decode(file_get_contents("commits.txt")));' | grep 'ü'
(snip)
string(25) "Updatefile für 01article"
string(135) "0000692 Der Spaltenname 'text' und 'timestamp' ist ein reserviertes MySQL-Schlüsselwort
string(167) "#237 Verspäteter Bugfix. Ursprüngliche Änderung wurde nicht in die Installations .sql-Datei übernommen, sondern nur im Rahmen der Installationsroutine geändert..."
(snip)

...no problems without utf8_encode().

dregad

dregad

2015-12-08 05:35

developer   ~0052073

Michael (01-Scripts), the original contributor of this, change agreed to revert it [1]. Will do so shortly.

[1] https://github.com/mantisbt/mantisbt/pull/1#issuecomment-162825608

Related Changesets

MantisBT: master 7968aef0

2011-01-17 03:00

01-Scripts


Details Diff
Problems if the json-message contains special characters like german umlauts (äöü). Commit should fix that problem. Affected Issues
0020350
mod - core/json_api.php Diff File

MantisBT: master-1.2.x 7f4ba27f

2015-12-08 05:03

dregad


Details Diff
Revert "Problems if the json-message contains special characters [...]"

This reverts commit 031afb687097762a139878a04f0b39dbb02d32af.

As per the JSON specification in RFC 7159 section 8.1, "JSON text
SHALL be encoded in UTF-8, UTF-16, or UTF-32". [1]

It is therefore incorrect to apply utf8_encode() function to the data
received from the server since it is already in Unicode by definition.

Fixes 0020350

[1] https://tools.ietf.org/html/rfc7159#section-8.1

Conflicts:
library/adodb
library/phpmailer
Affected Issues
0020350
mod - core/json_api.php Diff File

MantisBT: master 47c512fc

2015-12-08 05:03

dregad


Details Diff
Revert "Problems if the json-message contains special characters [...]"

This reverts commit 7968aef0b2ea43456a66ee72f27e84cfc5999f99.

As per the JSON specification in RFC 7159 section 8.1, "JSON text
SHALL be encoded in UTF-8, UTF-16, or UTF-32". [1]

It is therefore incorrect to apply utf8_encode() function to the data
received from the server since it is already in Unicode by definition.

Fixes 0020350

[1] https://tools.ietf.org/html/rfc7159#section-8.1

Conflicts:
core/json_api.php
library/adodb
library/disposable
library/ezc/Base
library/ezc/Graph
library/phpmailer
Affected Issues
0020350
mod - core/json_api.php Diff File