View Issue Details

IDProjectCategoryView StatusLast Update
0007623mantisbtdb mssqlpublic2023-05-04 16:23
ReporterRogierB Assigned Todregad  
PrioritynormalSeverityminorReproducibilityalways
Status closedResolutionduplicate 
Summary0007623: Can't store binary files into MSSQL database / can't attach files to bugs
Description

When the MSSQL Database server is used, it's not possible to store binary files in the database. The error shown after the Steps To Reproduce is:

Database query failed. Error received from database was 0000206: Operand type clash: text is incompatible with image for the query: INSERT INTO mantis_bug_file_table
(bug_id, title, description, diskfile, filename, folder, filesize, file_type, date_added, content)
VALUES
(22, '', '', ' b31736e21ea132368b6eeb66cd7edbc9', 'putty.exe', ' ', 421888, 'application/x-sdlc', '2006-12-01 12:43:04PM', 'MZ\0\0\0\0\0\0\0ÿÿ\0\0¸\0\0\0\0\

Text files are stored correctly into the database. I tried to set the mantis_bug_file_table content from image to text, but then only the first 4000 bytes are stored in the database.

In additional info is a description of how image / text / ntext should be implemented in MSSQL.

Steps To Reproduce
  • Install Mantis with MSSQL as database
  • Go to Report Issue
  • Attach a binary file (putty.exe in my case) to an issue to be created
  • Report the issue
Additional Information

Managing ntext, text, and image Data
The Microsoft® SQL Server™ ntext, text, and image data types are capable of holding extremely large amounts of data (up to 2 GB) in a single value. A single data value is typically larger than can be retrieved by an application in one step; some values may be larger than the virtual memory available on the client. Therefore, special steps are usually needed to retrieve these values.

If an ntext, text, and image data value is no longer than a Unicode, character, or binary string (4,000 characters, 8,000 characters, 8,000 bytes respectively), the value can be referenced in SELECT, UPDATE, and INSERT statements much the same way as the smaller data types. For example, an ntext column with a short value can be referenced in a SELECT statement select list the same way an nvarchar column is referenced. Some restrictions that must be observed, such as not being able to directly reference an ntext, text, or image column in a WHERE clause. These columns can be included in a WHERE clause as parameters of a function that returns another data type (such as ISNULL, SUBSTRING or PATINDEX) or in an IS NULL, IS NOT NULL, or LIKE expression.

Handling Larger Data Values
When the ntext, text, and image data values get larger, however, they must be handled on a block-by-block basis. Both Transact-SQL and the database APIs contain functions that allow applications to work with ntext, text, and image data block by block.

The database APIs follow a common pattern in the ways they handle long ntext, text, and image columns:

To read a long column, the application simply includes the ntext, text, or image column in a select list, and then binds the column to a program variable large enough to hold a reasonable block of the data. The application then executes the statement and uses an API function or method to retrieve the data into the bound variable one block at a time.

To write a long column, the application executes an INSERT or UPDATE statement with a parameter marker (?) in the place of the value to be placed in the ntext, text, or image column. The parameter marker (or parameter in the case of ADO) is bound to a program variable large enough to hold the blocks of data. The application goes into a loop where it first moves the next set of data into the bound variable, and then calls an API function or method to write that block of data. This is repeated until the entire data value has been sent.
Using text in row
In Microsoft SQL Server 2000, users can enable a text in row option on a table so it could store text, ntext, or image data in its data row.

To enable the option, execute the sp_tableoption stored procedure, specifying text in row as the option name and on as the option value. The default maximum size that can be stored in a row for a BLOB (binary large object: text, ntext, or image data) is 256 bytes, but values may range from 24 through 7000. To specify a maximum size that is not the default, specify an integer within the range as the option value.

text, ntext, or image strings are stored in the data row if the following conditions apply:

text in row is enabled.

The length of the string is shorter than the limit specified in @OptionValue

There is enough space available in the data row.
When BLOB strings are stored in the data row, reading and writing the text, ntext, or image strings can be as fast as reading or writing character and binary strings. SQL Server does not have to access separate pages to read or write the BLOB string.

If a text, ntext, or image string is larger than the specified limit or the available space in the row, pointers are stored in the row instead. The conditions for storing the BLOB strings in the row still apply though: There must be enough space in the data row to hold the pointers.
....

TagsNo tags attached.
Attached Files

Relationships

duplicate of 0022208 closeddregad File upload to MS-SQL not working 

Activities

RogierB

RogierB

2006-12-01 10:19

reporter   ~0013766

got this working myself changed in file_api.php :

        case DATABASE:
            $c_content = db_prepare_string( fread ( fopen( $p_tmp_file, 'rb' ), $t_file_size ) );

to
case DATABASE:
$c_content = fread ( fopen( $p_tmp_file, 'rb' ), $t_file_size

and also the following line:

    $query = "INSERT INTO $t_file_table
                    (" . $p_table . "_id, title, description, diskfile, filename, folder, filesize, file_type, date_added, content)
                  VALUES
                    ($c_id, '$c_title', '$c_desc', '$c_disk_file_name', '$c_new_file_name', '$c_file_path', $c_file_size, '$c_file_type', " . db_now() .", '$c_content')";

to

$query = "INSERT INTO $t_file_table
(" . $p_table . "_id, title, description, diskfile, filename, folder, filesize, file_type, date_added, content)
VALUES
($c_id, '$c_title', '$c_desc', '$c_disk_file_name', '$c_new_file_name', '$c_file_path', $c_file_size, '$c_file_type', " . db_now() .", 0x". bin2hex($c_content) .")";

updated the php.ini file and set mssql.textsize = 2147483647 and mssql.textlimit = 2147483647 (these values may be a 'bit' too large but it did the trick, besides php upload limit will also reject larger files). The purpose of setting those values is that else only the first 4096 bytes get stored in the server. Couldn't do this with ini_set('mssql.textsize','2147483647').

The bug_api.php also has an file upload routine, but i don't think it's being used, else it probably should also be updated.

rzg

rzg

2006-12-04 14:16

reporter   ~0013771

See 0006063. We came to the same conclusion/patch.

nurikabe

nurikabe

2007-01-07 17:24

reporter   ~0013905

As of 1.1.0a2 this is still an issue and RogierB's fix still works. Don't forget to change mssql.textsize and mssql.textlimit.

Ghazi Triki

Ghazi Triki

2009-12-28 05:35

reporter   ~0023998

It is still an issue in version 1.1.8.
I Got the message below when I compressed a Charles file .chls with winrar. I got the same problem when I tried to upalod it uncompressed. After the strange symbols I see the binary data.

atabase query failed. Error received from database was 0002006: MySQL server has gone away for the query: INSERT INTO mantis_bug_file_table
(bug_id, title, description, diskfile, filename, folder, filesize, file_type, date_added, content)
VALUES
(1744, '', '', 'fd2384f6ad0807858aaf9c91f75ea6b7', 'FindRide_Company.rar', '', 1158832, 'application/x-rar-compressed', '2009-12-28 11:30:20', 'Rar!\Z\0ϐs\0\0\r\0\0\0\0\0\0\0��t�:\0[�\0 �0\0�\�s�X�;5\0 \0\0\0FindRide_Company.chls\0�aP�����0Ԡ\"\"�\"��~�f.................

dregad

dregad

2012-08-28 03:25

developer   ~0032688

There have been several changes in file upload mechanism since version 1.2, as well as in the ADOdb abstraction layer module for mssql.

Please check if problem is still reproducible using a nightly build >= 18-Aug-2012, and report any issues you may encounter.

RogierB

RogierB

2012-08-28 04:07

reporter   ~0032690

@degrad, I will test this as soon as I get a chance, good job. (Luckily we are still using mantis bt after 6 years :) )

grangeway

grangeway

2014-02-17 13:26

reporter   ~0039443

Marking as Suspended

MSSQL support is currently known broken. We are going to be replacing the DB Layer in Mantis to fix this properly after the next release.

I'd strongly advise using MYSQL for now.

The new DB layer already contains the appropriate fixes for MS SQL Support.

grangeway

grangeway

2014-03-27 19:12

reporter   ~0039761

MSSQL support is currently known broken. We are going to be replacing the DB Layer in Mantis to fix this properly after the next release.

I'd strongly advise using MYSQL for now.

The new DB layer already contains the appropriate fixes for MS SQL Support.

DCOOMBER

DCOOMBER

2014-03-31 11:19

reporter   ~0039797

Is the release schedule for the new DB layer for MSSQL available?

grangeway

grangeway

2014-04-03 11:54

reporter   ~0039814

Dcoomber,

I plan to have completed testing of the new DB layer with MSSQL at some point over this weekend.

Would you be interested in helping to test the new layer?

Paul

DCOOMBER

DCOOMBER

2014-04-12 22:29

reporter   ~0040091

Paul,

I am willing to assist with testing of the new FB layer.

If possible, please PM me and we can discuss further.

David

Alyndri

Alyndri

2014-04-15 17:52

reporter   ~0040103

Last edited: 2014-04-15 18:30

Hi guys,

We have also got around the attachment issue in MSSQL with the latest version of mantis by altering the following:

Change these lines in core/file_api.php:

$query = "INSERT INTO $t_file_table
( $t_id_col, title, description, diskfile, filename, folder, filesize, file_type, date_added, content, user_id )
VALUES
( " . db_param() . ", " . db_param() . ", " . db_param() . ", "
. db_param() . ", " . db_param() . ", " . db_param() . ", "
. db_param() . ", " . db_param() . ", " . db_param() . ", "
. db_param() . ", " . db_param() . " )";
db_query_bound( $query, Array(
$t_id,
$p_title,
$p_desc,
$t_unique_name,
$t_file_name,
$t_file_path,
$t_file_size,
$p_file['type'],
$p_date_added,
$c_content,
(int)$p_user_id,
) );

Remove $c_content from the Array() and put it directly into the query parameters:

$query = "INSERT INTO $t_file_table
( $t_id_col, title, description, diskfile, filename, folder, filesize, file_type, date_added, content, user_id )
VALUES
( " . db_param() . ", " . db_param() . ", " . db_param() . ", "
. db_param() . ", " . db_param() . ", " . db_param() . ", "
. db_param() . ", " . db_param() . ", " . db_param() . ", "
. $c_content . ", " . db_param() . " )";
db_query_bound( $query, Array(
$t_id,
$p_title,
$p_desc,
$t_unique_name,
$t_file_name,
$t_file_path,
$t_file_size,
$p_file['type'],
$p_date_added,
(int)$p_user_id,
) );

grangeway

grangeway

2014-05-16 15:00

reporter   ~0040310

MantisBT currently supports Mysql and has support for other database engines.

The support for other databases is known to be problematic.

Having implemented the current database layer into Mantis 10 years ago, I'm currently working on replacing the current layer.

If you are interested in using Mantis with non-mysql databases - for example, Oracle, PGSQL or MSSQL, and would be willing to help out testing the new database layer, please drop me an email at paul@mantisforge.org

In the meantime, I'd advise running Mantis with Mysql Only to avoid issues.

Thanks
Paul

alex.volkov

alex.volkov

2015-09-08 17:38

reporter   ~0051406

The issue still exists and is due to an unnecessary varbinary/image encoding intended for inline queries. I examined all (4) instances of db_prepare_binary_string() calls, and in all cases the return value is now used in parameterized queries (i.e. with db_param()). Because of this, no encoding is necessary at least for the mssqlnative driver, and very likely for others as well.

I do not know if such encoding is necessary for PostgreSQL, but I propose to actually make the db_prepare_binary_string() function a no-op for the time being, until any broken use cases emerge. As it stands, this function appears to hurt more than it helps. If not made a no-op, then the function would need to be split down the road into two variants: one for inlined query values, and one for parameterized queries.

Attached here is the minimal patch needed for mssqlnative.

mssqlnative-binary-min.patch (826 bytes)   
diff -ur -U 5 mantisbt-1.3.0-beta.3\core\database_api.php mantisbt-1.3.0-beta.3-mspl1\core\database_api.php
--- mantisbt-1.3.0-beta.3\core\database_api.php	Sun Sep 06 14:10:14 2015
+++ mantisbt-1.3.0-beta.3-mspl1\core\database_api.php	Tue Sep 08 14:03:14 2015
@@ -802,11 +802,12 @@
 	global $g_db;
 	$t_db_type = config_get_global( 'db_type' );
 
 	switch( $t_db_type ) {
 		case 'mssql':
-		case 'mssqlnative':
+		# XXX: Is this function even needed now? All of its instances are used in parameterized SQL queries (db_param())
+		#case 'mssqlnative':  # Definitely do not need to encode varbinary in parameterized SQL queries with MS sqldrv v3.2 (php_sqlsrv_56_nts.dll)
 		case 'odbc_mssql':
 		case 'ado_mssql':
 			$t_content = unpack( 'H*hex', $p_string );
 			return '0x' . $t_content['hex'];
 			break;
dregad

dregad

2015-09-09 02:49

developer   ~0051412

Thanks for the research. I'll test this with PostgreSQL.

dregad

dregad

2023-04-22 10:58

developer   ~0067693

I believe this was fixed in 0022208. Closing as duplicate.