View Issue Details

IDProjectCategoryView StatusLast Update
0020350mantisbtotherpublic2016-08-15 09:02
ReportertomopAssigned Todregad 
PrioritynormalSeverityminorReproducibilityalways
Status closedResolutionfixed 
Product Version1.2.6 
Target Version1.2.20Fixed in Version1.2.20 
Summary0020350: json_url() may break non-ASCII strings.
Description

core/json_api.php

$t_json = json_decode( utf8_encode( $t_data ) );

utf8_encode() must be called with ISO-8859-1 strings but JSON shall be written in Unicode (UTF-8, UTF-16, ...).

I'm using Source Control Integration plugin that calls json_url(). All of commit logs in Japanese was broken. And I tried to remove utf8_encode() then I got expected results.

TagsNo tags attached.

Activities

dregad

dregad

2015-12-07 10:13

developer   ~0052042

Last edited: 2015-12-08 03:05

View 2 revisions

I agree with you that JSON strings should be Unicode-encoded to begin with, so the call to utf8_encode() seems incorrect.

This was introduced by merging pull request https://github.com/mantisbt/mantisbt/pull/1 so I have asked the contributor to clarify why this was needed in the first place.

I believe this commit should be reverted.

tomop

tomop

2015-12-07 20:53

reporter   ~0052062

I commented to https://github.com/mantisbt/mantisbt/pull/1 .

If you want to support bad implementations to send not Unicode-encoded strings, change url_get() in core/url_api.php

  • detect encodings with "Content-Type" HTTP header.
  • use iconv() or mb_convert_encoding() with detected encoding to encode to UTF-8 from other encodings.

note:
mb_detect_encoding() and passing "auto" to mb_convert_encoding() may return wrong result.

tomop

tomop

2015-12-08 00:43

reporter   ~0052066

I tried to retrieve and parse logs from the contributor's repository.


$ curl 'https://api.github.com/repos/01-Scripts/01-Artikelsystem/commits?per_page=50' > commits.txt
$ file commits.txt
commits.txt: UTF-8 Unicode text
$ php -r 'var_dump(json_decode(file_get_contents("commits.txt")));' | grep 'ü'
(snip)
string(25) "Updatefile für 01article"
string(135) "0000692 Der Spaltenname 'text' und 'timestamp' ist ein reserviertes MySQL-Schlüsselwort
string(167) "#237 Verspäteter Bugfix. Ursprüngliche Änderung wurde nicht in die Installations .sql-Datei übernommen, sondern nur im Rahmen der Installationsroutine geändert..."
(snip)

...no problems without utf8_encode().

dregad

dregad

2015-12-08 05:35

developer   ~0052073

Michael (01-Scripts), the original contributor of this, change agreed to revert it [1]. Will do so shortly.

[1] https://github.com/mantisbt/mantisbt/pull/1#issuecomment-162825608

Related Changesets

MantisBT: master 7968aef0

2011-01-17 08:00:33

01-Scripts

Details Diff
Problems if the json-message contains special characters like german umlauts (äöü). Commit should fix that problem.
mod - core/json_api.php Diff File

MantisBT: master-1.2.x 7f4ba27f

2015-12-08 10:03:04

dregad

Details Diff
Revert "Problems if the json-message contains special characters [...]"

This reverts commit 031afb687097762a139878a04f0b39dbb02d32af.

As per the JSON specification in RFC 7159 section 8.1, "JSON text
SHALL be encoded in UTF-8, UTF-16, or UTF-32". [1]

It is therefore incorrect to apply utf8_encode() function to the data
received from the server since it is already in Unicode by definition.

Fixes 0020350

[1] https://tools.ietf.org/html/rfc7159#section-8.1

Conflicts:
library/adodb
library/phpmailer
mod - core/json_api.php Diff File

MantisBT: master 47c512fc

2015-12-08 10:03:04

dregad

Details Diff
Revert "Problems if the json-message contains special characters [...]"

This reverts commit 7968aef0b2ea43456a66ee72f27e84cfc5999f99.

As per the JSON specification in RFC 7159 section 8.1, "JSON text
SHALL be encoded in UTF-8, UTF-16, or UTF-32". [1]

It is therefore incorrect to apply utf8_encode() function to the data
received from the server since it is already in Unicode by definition.

Fixes 0020350

[1] https://tools.ietf.org/html/rfc7159#section-8.1

Conflicts:
core/json_api.php
library/adodb
library/disposable
library/ezc/Base
library/ezc/Graph
library/phpmailer
mod - core/json_api.php Diff File

Issue History

Date Modified Username Field Change
2015-12-06 09:01 tomop New Issue
2015-12-07 10:13 dregad Status new => acknowledged
2015-12-07 10:13 dregad Note Added: 0052042
2015-12-07 10:15 dregad Changeset attached => MantisBT master 7968aef0
2015-12-07 20:53 tomop Note Added: 0052062
2015-12-08 00:43 tomop Note Added: 0052066
2015-12-08 03:05 dregad Note Edited: 0052042 View Revisions
2015-12-08 05:35 dregad Note Added: 0052073
2015-12-08 10:23 dregad Assigned To => dregad
2015-12-08 10:23 dregad Status acknowledged => assigned
2015-12-08 10:23 dregad Product Version 1.2.19 => 1.2.6
2015-12-08 10:23 dregad Target Version => 1.2.20
2015-12-08 10:24 dregad Changeset attached => MantisBT master-1.2.x 7f4ba27f
2015-12-08 10:24 dregad Status assigned => resolved
2015-12-08 10:24 dregad Resolution open => fixed
2015-12-08 10:24 dregad Fixed in Version => 1.2.20
2015-12-08 10:25 dregad Changeset attached => MantisBT master 47c512fc
2016-08-15 09:02 dregad Status resolved => closed