View Issue Details

WikiRelated ChangesetsJump to Notes
IDProjectCategoryView StatusDate SubmittedLast Update
0010303mantisbtlocalizationpublic2009-04-08 09:452010-12-27 08:57
Reporterjreese Assigned Tojreese  
PriorityhighSeverityfeatureReproducibilityN/A
Status closedResolutionfixed 
Product Versiongit trunk 
Target Version1.2.0rc1Fixed in Version1.2.0rc1 
Summary0010303: Create and use string_api wrappers for multibyte string functions
Description

There are many places in the MantisBT codebase that utilize strtolower() or strlen() without regard for multibyte languages. We should create functions in string_api similar to the existing stringlower() that call mb*() if available, and fallback to the standard string function otherwise. We should then ensure that we use those functions throughout the entire codebase, which has not yet been done for string_lower().

Tagsutf8
Attached Files
patch1.patch (1,808 bytes)    
From 37c1afe39ecbaa57d18919c0e978c1ce721bdbdd Mon Sep 17 00:00:00 2001
From: Kirill Krasnov <krasnovforum@gmail.com>
Date: Wed, 22 Apr 2009 11:48:27 +0300
Subject: [PATCH] Test implement jreese strings wrapper

---
 changelog_page.php |    4 ++--
 roadmap_page.php   |    2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/changelog_page.php b/changelog_page.php
index 70167f6..c6730af 100644
--- a/changelog_page.php
+++ b/changelog_page.php
@@ -50,9 +50,9 @@ function print_version_header( $p_version_id ) {
 		$t_release_timestamp = version_get_field( $p_version_id, 'date_order' );
 
 		if ( (bool) $t_version_released ) {
-			$t_release_date = ' (' . mb_strtolower( lang_get('released') ) . ' ' . date( config_get( 'short_date_format' ), $t_release_timestamp ) . ')';
+			$t_release_date = ' (' . string_lower( lang_get('released') ) . ' ' . date( config_get( 'short_date_format' ), $t_release_timestamp ) . ')';
 		} else {
-			$t_release_date = ' (' . mb_strtolower( lang_get( 'not_released' ) ) . ')';
+			$t_release_date = ' (' . string_lower( lang_get( 'not_released' ) ) . ')';
 		}
 	} else {
 		$t_release_date = '';
diff --git a/roadmap_page.php b/roadmap_page.php
index 83e51ec..35c24f0 100644
--- a/roadmap_page.php
+++ b/roadmap_page.php
@@ -42,7 +42,7 @@
 		if ( config_get( 'show_roadmap_dates' ) ) {
 			$t_version_timestamp = $p_version_row['date_order'];
 
-			$t_scheduled_release_date = ' (' . mb_strtolower( lang_get( 'scheduled_release' ) ) . ' ' . date( config_get( 'short_date_format' ), $t_version_timestamp ) . ')';
+			$t_scheduled_release_date = ' (' . string_lower( lang_get( 'scheduled_release' ) ) . ' ' . date( config_get( 'short_date_format' ), $t_version_timestamp ) . ')';
 		} else {
 			$t_scheduled_release_date = '';
 		}
-- 
1.6.2.2.1669.g7eaf8
patch1.patch (1,808 bytes)   
patch1_correct.patch (1,799 bytes)    
From 643ae8689a205c43bb5b805201ec6c52395f31a0 Mon Sep 17 00:00:00 2001
From: Kirill Krasnov <krasnovforum@gmail.com>
Date: Wed, 22 Apr 2009 12:07:39 +0300
Subject: [PATCH] Test implement jreese strings wrapper

---
 changelog_page.php |    4 ++--
 roadmap_page.php   |    2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/changelog_page.php b/changelog_page.php
index 823f8f3..c6730af 100644
--- a/changelog_page.php
+++ b/changelog_page.php
@@ -50,9 +50,9 @@ function print_version_header( $p_version_id ) {
 		$t_release_timestamp = version_get_field( $p_version_id, 'date_order' );
 
 		if ( (bool) $t_version_released ) {
-			$t_release_date = ' (' . strtolower( lang_get('released') ) . ' ' . date( config_get( 'short_date_format' ), $t_release_timestamp ) . ')';
+			$t_release_date = ' (' . string_lower( lang_get('released') ) . ' ' . date( config_get( 'short_date_format' ), $t_release_timestamp ) . ')';
 		} else {
-			$t_release_date = ' (' . strtolower( lang_get( 'not_released' ) ) . ')';
+			$t_release_date = ' (' . string_lower( lang_get( 'not_released' ) ) . ')';
 		}
 	} else {
 		$t_release_date = '';
diff --git a/roadmap_page.php b/roadmap_page.php
index 21b405f..35c24f0 100644
--- a/roadmap_page.php
+++ b/roadmap_page.php
@@ -42,7 +42,7 @@
 		if ( config_get( 'show_roadmap_dates' ) ) {
 			$t_version_timestamp = $p_version_row['date_order'];
 
-			$t_scheduled_release_date = ' (' . strtolower( lang_get( 'scheduled_release' ) ) . ' ' . date( config_get( 'short_date_format' ), $t_version_timestamp ) . ')';
+			$t_scheduled_release_date = ' (' . string_lower( lang_get( 'scheduled_release' ) ) . ' ' . date( config_get( 'short_date_format' ), $t_version_timestamp ) . ')';
 		} else {
 			$t_scheduled_release_date = '';
 		}
-- 
1.6.2.2.1669.g7eaf8
patch1_correct.patch (1,799 bytes)   

Relationships

related to 0010271 closedjreese add warning if mb_string is missing 
parent of 0007400 closedsiebrand [all lang] When using UTF8 for encoding all reports some fields' contents are incorrectly truncated. 
parent of 0008572 closedjreese multibyte str pad problem 
parent of 0010057 closedatrol Indentation in email wrong when using non-us-ascii 

Activities

grangeway

grangeway

2009-04-14 10:13

reporter   ~0021519

?
grangeway

grangeway

2009-04-14 10:13

reporter   ~0021520

seems you can't post utf8 stuff into this db ;/
jreese

jreese

2009-04-14 10:20

reporter   ~0021521

Current work on my end is being done in my public 'strings' branch:
http://git.mantisforge.org/w/mantisbt/jreese.git?a=shortlog;h=refs/heads/strings
Kirill

Kirill

2009-04-15 02:35

reporter   ~0021542

@John, when this branch will be in master-branch?
Or ask me, How I can merge this branch to try in my mantis?
jreese

jreese

2009-04-15 08:56

reporter   ~0021554

Well, the work is not complete yet; I've only added a couple new wrappers. There is still a lot of work ahead to begin replacing all the calls in Mantis with the appropriate wrapper functions.

In the mean time, as I continue the work, you can test it with the following basic steps:



$ git remote add jreese git://git.mantisforge.org/mantisbt/jreese.git

$ git fetch jreese

$ git checkout -b strings jreese/strings

Kirill

Kirill

2009-04-16 10:58

reporter   ~0021577

I can checkout only string_api.php?
Kirill

Kirill

2009-04-22 04:45

reporter   ~0021616

I upload this file to my work mantis tracker.
and test change roadmap and changelog for task 0010145. All work perfectly.
Kirill

Kirill

2009-04-22 05:04

reporter   ~0021617

I upload patch for 0010145 or give me access to commit on your git
Kirill

Kirill

2009-04-22 05:08

reporter   ~0021618

Sorry. Delete my attach. Thin not work for your rep.
jreese

jreese

2009-06-09 13:02

reporter   ~0022065

Re-targeting for 1.2.x -- proposal branch available on MantisForge at http://git.mantisforge.org/w/mantisbt/jreese.git?a=shortlog;h=refs/heads/strings
jreese

jreese

2009-06-15 10:13

reporter   ~0022173

Fix committed to master branch, using the open source phputf8 library [1] to provide multibyte-aware versions of string functions.

[1] http://sourceforge.net/projects/phputf8
vboctor

vboctor

2009-06-15 12:09

manager   ~0022175

I wonder if we should trim the libraries we include by removing folders like "Tests". It just bloats our distribution with code that we don't use. I also remember there were once in ADODB or some other library some security issue in the test scripts.

The other option is to be strict about using the release script in preparing releases and updating such script to trim these directories along with packages/, our unit tests, etc.

Related Changesets

MantisBT: master 4fe8e027

2009-06-12 16:17

jreese


Details Diff		 Issue 0010303: Added phputf8 0.5 package. Affected Issues
0010303
add - library/utf8/tests/data/bench/strlen_native_20061001.txt Diff File
add - library/utf8/substr_replace.php Diff File
add - library/utf8/tests/data/bench/strlen_native_20060226.txt Diff File
add - library/utf8/tests/data/bench/strrpos_native_20061001.txt Diff File
add - library/utf8/tests/runtests.php Diff File
add - library/utf8/tests/index.php Diff File
add - library/utf8/utils/validation.php Diff File
add - library/utf8/docs/examples/1.php Diff File
add - library/utf8/tests/cases/utf8_ascii.test.php Diff File
add - library/utf8/docs/screen.css Diff File
add - library/utf8/tests/cases/utf8_str_split.test.php Diff File
add - library/utf8/docs/utf8.xhtml Diff File
add - library/utf8/tests/data/bench/strlen_mbstring_20061001.txt Diff File
add - library/utf8/utils/specials.php Diff File
add - library/utf8/ChangeLog Diff File
add - library/utf8/tests/cases/utf8_specials.test.php Diff File
add - library/utf8/mbstring/core.php Diff File
add - library/utf8/tests/data/bench/strlen_mbstring_20060226.txt Diff File
add - library/utf8/ucfirst.php Diff File
add - library/utf8/tests/cases/utf8_substr.test.php Diff File
add - library/utf8/tests/bench/strpos.php Diff File
add - library/utf8/tests/data/utf8.html Diff File
add - library/utf8/tests/data/bench/substr_mbstring_20060226.txt Diff File
add - library/utf8/tests/cases/utf8_ucfirst.test.php Diff File
add - library/utf8/tests/data/bench/substr_native_20061001.txt Diff File
add - library/utf8/docs/examples/tables.php Diff File
add - library/utf8/utf8.php Diff File
add - library/utf8/tests/cases/utf8_strpos.test.php Diff File
add - library/utf8/str_ireplace.php Diff File
add - library/utf8/utils/bad.php Diff File
add - library/utf8/tests/cases/utf8_stristr.test.php Diff File
add - library/utf8/native/core.php Diff File
add - library/utf8/tests/bench/strrpos.php Diff File
add - library/utf8/strcspn.php Diff File
add - library/utf8/tests/cases/utf8_strlen.test.php Diff File
add - library/utf8/tests/cases/utf8_strrpos.test.php Diff File
add - library/utf8/tests/cases/utf8_ucwords.test.php Diff File
add - library/utf8/tests/cases/utf8_validation.test.php Diff File
add - library/utf8/docs/print.css Diff File
add - library/utf8/ord.php Diff File
add - library/utf8/tests/data/bench/strpos_mbstring_20060226.txt Diff File
add - library/utf8/trim.php Diff File
add - library/utf8/tests/cases/utf8_unicode.test.php Diff File
add - library/utf8/tests/cases/utf8_str_ireplace.test.php Diff File
add - library/utf8/tests/cases/utf8_strtolower.test.php Diff File
add - library/utf8/tests/config.php Diff File
add - library/utf8/LICENSE Diff File
add - library/utf8/README Diff File
add - library/utf8/tests/bench/substr.php Diff File
add - library/utf8/tests/bench/strlen.php Diff File
add - library/utf8/ucwords.php Diff File
add - library/utf8/tests/cases/utf8_substr_replace.test.php Diff File
add - library/utf8/tests/cases/utf8_strspn.test.php Diff File
add - library/utf8/utils/patterns.php Diff File
add - library/utf8/tests/data/bench/strrpos_mbstring_20060226.txt Diff File
add - library/utf8/tests/cases/utf8_strrev.test.php Diff File
add - library/utf8/tests/data/bench/strpos_native_20061001.txt Diff File
add - library/utf8/str_split.php Diff File
add - library/utf8/TODO.tsk Diff File
add - library/utf8/tests/cases/utf8_strcasecmp.test.php Diff File
add - library/utf8/tests/data/bench/strpos_native_20060226.txt Diff File
add - library/utf8/tests/cases/utf8_strtoupper.test.php Diff File
add - library/utf8/strcasecmp.php Diff File
add - library/utf8/tests/cases/utf8_strcspn.test.php Diff File
add - library/utf8/tests/data/bench/strrpos_native_20060226.txt Diff File
add - library/utf8/tests/data/bench/strpos_mbstring_20061001.txt Diff File
add - library/utf8/tests/data/bench/substr_native_20060226.txt Diff File
add - library/utf8/tests/cases/utf8_trim.test.php Diff File
add - library/utf8/tests/cases/utf8_str_pad.test.php Diff File
add - library/utf8/utils/ascii.php Diff File
add - library/utf8/utils/unicode.php Diff File
add - library/utf8/tests/data/bench/strrpos_mbstring_20061001.txt Diff File
add - library/utf8/docs/phputf8.pod Diff File
add - library/utf8/tests/cli_reporter.php Diff File
add - library/utf8/utils/position.php Diff File
add - library/utf8/stristr.php Diff File
add - library/utf8/tests/data/bench/substr_mbstring_20061001.txt Diff File
add - library/utf8/tests/cases/utf8_bad.test.php Diff File
add - library/utf8/exp/regexunicode.php Diff File
add - library/utf8/strspn.php Diff File
add - library/utf8/tests/cases/utf8_ord.test.php Diff File
add - library/utf8/tests/cases/utf8_position.test.php Diff File
add - library/utf8/strrev.php Diff File
add - library/utf8/docs/makeDocs.pl Diff File
add - library/utf8/tests/benchconfig.php Diff File
add - library/utf8/str_pad.php Diff File

MantisBT: master e345f2f8

2009-06-12 16:58

jreese


Details Diff		 Fix 0010303: Use UTF-8 compatible string functions.

Use the phputf8 library's implementation of utf8_* as the preferred string
functions in the codebase.

Replaced:
- all usage of strlen() with utf8_strlen()
- all usage of strtolower() with utf8_strtolower()
- all usage of strtoupper() with utf8_strtoupper()
- all usage of substr() or mb_substr() with utf8_substr()
- all usage of str_pad() with utf8_str_pad()		 Affected Issues
0010303
mod - core/filter_api.php Diff File
mod - bug_graph_bystatus.php Diff File
mod - core/authentication_api.php Diff File
mod - signup.php Diff File
mod - file_download.php Diff File
mod - issues_rss.php Diff File
mod - core/html_api.php Diff File
mod - manage_tags_page.php Diff File
mod - core/relationship_api.php Diff File
mod - core/database_api.php Diff File
mod - core/custom_field_api.php Diff File
mod - core.php Diff File
mod - search.php Diff File
mod - changelog_page.php Diff File
mod - core/utility_api.php Diff File
mod - core/version_api.php Diff File
mod - core/columns_api.php Diff File
mod - core/lang_api.php Diff File
mod - core/php_api.php Diff File
mod - manage_proj_edit_page.php Diff File
mod - core/bug_api.php Diff File
mod - csv_export.php Diff File
mod - return_dynamic_filters.php Diff File
mod - core/user_api.php Diff File
mod - core/form_api.php Diff File
mod - bugnote_view_inc.php Diff File
mod - set_project.php Diff File
mod - core/custom_function_api.php Diff File
mod - make_captcha_img.php Diff File
mod - core/cfdefs/cfdef_standard.php Diff File
mod - roadmap_page.php Diff File
mod - core/tag_api.php Diff File
mod - manage_user_page.php Diff File
mod - bug_actiongroup_ext_page.php Diff File
mod - core/project_api.php Diff File
mod - core/plugin_api.php Diff File
mod - core/bugnote_api.php Diff File
mod - core/file_api.php Diff File
mod - core/session_api.php Diff File
mod - core/print_api.php Diff File
mod - core/projax_api.php Diff File
mod - print_bugnote_inc.php Diff File
mod - core/email_api.php Diff File
mod - browser_search_plugin.php Diff File
mod - core/graphviz_api.php Diff File
mod - admin/install_functions.php Diff File
mod - print_all_bug_options_update.php Diff File
mod - core/classes/MantisEnum.class.php Diff File
mod - print_all_bug_page_word.php Diff File
mod - manage_proj_cat_update.php Diff File
mod - core/string_api.php Diff File
mod - news_rss.php Diff File
mod - bug_actiongroup_page.php Diff File
mod - core/config_api.php Diff File