0011153: Truncated download

(0023939)
siebrand (developer)
2009-12-21 12:18

Showstopper 1.2.0.

(0024022)
vboctor (administrator)
2010-01-02 03:17

This bug seems to be related to the following checkin:
http://git.mantisbt.org/?p=mantisbt.git;a=commit;h=c416dd826b1da0c5093a0dfbe001018e6ffa90ff [^]

@grangeway, this is blocking 1.2.0 release. Will you be able to fix it or revert the change?

(0024037)
dhx (developer)
2010-01-05 03:52

Cannot reproduce with a 28MB file. Are you sure your PHP memory limits are set high enough? If you don't want to raise PHP memory limits, I strongly suggest looking at a new feature I added today in 0011291... X-Sendfile

Actually, just use X-Sendfile wherever possible because it's much higher performance. For small MantisBT installations with few small file uploads it won't make much of a difference though.

(0024451)
grangeway (developer)
2010-02-20 15:15

So erm, can anyone reproduce this? ;/

(0024454)
ejc (reporter)
2010-02-21 05:29
edited on: 2010-02-21 05:41

I think so, grangeway, with version 1.2.0RC2 on a Apache/2.2.3 (Debian) DAV/2 SVN/1.4.2 mod_ldap_userdir/1.1.11 PHP/5.2.0-8+etch13 mod_ssl/2.2.3 OpenSSL/0.9.8c.

The problem AFAIK was that the file downloads were "chunked" to (more or less) the closest 512KB chunk, i.e.: original 560KB file is downloaded as a 512KB file; or a 1600KB file is downloaded with a size of 1512KB.

Also we confirmed that the file is uploaded and saved properly to disk on the server. We don't use the database for it.

We walk-arounded it for a while with this little patch on nightly build file_download.php of feb-18:

//$chunksize = 4*131072;
$chunksize = 4*999999;

(0024455)
dhx (developer)
2010-02-22 00:31

Paul: What is the significance of choosing a chunk size of 4*131072? Otherwise known as: 4*((2^20)/8)

I gather it's something highly technical to do with heap sizes and reading files as efficiently as possible with respect to memory allocation, paging, etc?

(0024460)
grangeway (developer)
2010-02-22 17:49

ejc: does that include with the latest code in trunk (see attached patch to this bug), and are you reliably able to reproduce this with any file of approxy >512kb?

Paul

(0024487)
ejc (reporter)
2010-02-24 11:23

Paul: I think I can reliable reproduce this with exactly this platform: mantisbt-1.2.0rc2
+ http_api.php (from 18/feb nightly build)
+ file_download.php (from 18/feb nightly build, patched as said in 0011153:0024454).

However, with 1.2.0 released yesterday it works fine.

May you want more tests, just ask for them.

(0024502)
dhx (developer)
2010-02-24 22:49

Resolving this as it has been reported to be fixed in 1.2.0

If it's not working on a patched pre 1.2.0 release then it is likely a case of patches not being applied correctly, certain required patches missing, etc. We've made quite a few changes to file downloading since 1.2.0RC2 (where file downloads were more or less completely broken).

If you can still reproduce this problem on a fresh install of 1.2.0 (ie. not a hacked upgrade from 1.2.0RC2 by placing 1.2.0 files on top of your 1.2.0RC2 release), feel free to reopen this bug report.

(0024607)
valerio (reporter)
2010-03-03 06:42

I upgraded to 1.2.0 and the problem still remains
BUT
I did a lot of debugging and finally found that the problem is related
to php (5.1.6) file_get_contents() !!!!!
Running file_send_chunk with a CLI script in the last run of the while(true) loop file_get_contents crashes with a bus error on mmap().
(not logged in apache error_log :-( )

So the final solution is : system upgrade this afternoon ;-)

(0024730)
fman (reporter)
2010-03-12 13:47
edited on: 2010-03-12 13:54

I'm having same problem and I'm using php 5.2.1 on redhat.
I've tried with:
$chunksize = 4*999999;

and seems to work
is this the right solution ?

(0025131)
ngombe (reporter)
2010-04-15 09:54

I had the same problem with php 5.1.6.

I found 'child pid 24521 exit signal Bus error' in the apache log ...

In php.ini the MAX_POST_SIZE is set to 8Mo so I forced $chunksize = 8*1024*1024;

Now it's working ...

Related Changesets
MantisBT: master 5a84b45c Timestamp: 2010-01-03 22:19:05 Author: Paul Richards Committer: Paul Richards [ Details ] [ Diff ]	0011153: Truncated download Whilst I'm not sure this code can actually get hit, it's also wrong - it could be this is why the user is seeing a problem.
	mod - file_download.php	[ Diff ] [ File ]

MantisBT: master-1.2.x 98490560 Timestamp: 2010-01-03 22:19:05 Author: Paul Richards Committer: dhx Committer: David Hicks [ Details ] [ Diff ]	0011153: Truncated download Whilst I'm not sure this code can actually get hit, it's also wrong - it could be this is why the user is seeing a problem.
	mod - file_download.php	[ Diff ] [ File ]

Issue History
Date Modified	Username	Field	Change
2009-11-10 12:36	valerio	New Issue
2009-11-11 04:05	vboctor	Issue Monitored: grangeway
2009-11-11 04:06	vboctor	Status	new => acknowledged
2009-11-11 04:06	vboctor	Target Version	=> 1.2.x
2009-12-16 21:46	dhx	Issue Monitored: dhx
2009-12-21 12:18	siebrand	Priority	normal => urgent
2009-12-21 12:18	siebrand	Note Added: 0023939
2010-01-02 03:15	vboctor	Status	acknowledged => assigned
2010-01-02 03:15	vboctor	Assigned To	=> grangeway
2010-01-02 03:17	vboctor	Note Added: 0024022
2010-01-04 04:35	dhx	Relationship added	related to 0004943
2010-01-05 03:52	dhx	Note Added: 0024037
2010-01-05 10:22	dhoefling	Issue Monitored: dhoefling
2010-01-10 19:54	dhx	Changeset attached	master 5a84b45c =>
2010-01-10 20:02	dhx	Changeset attached	master-1.2.x 98490560 =>
2010-01-22 11:43	vr5	Issue Monitored: vr5
2010-02-20 12:43	ejc	Issue Monitored: ejc
2010-02-20 15:15	grangeway	Note Added: 0024451
2010-02-21 05:29	ejc	Note Added: 0024454
2010-02-21 05:41	ejc	Note Edited: 0024454	View Revisions
2010-02-22 00:31	dhx	Note Added: 0024455
2010-02-22 14:32	jreese	Target Version	1.2.0 => 1.2.x
2010-02-22 17:49	grangeway	Note Added: 0024460
2010-02-24 11:23	ejc	Note Added: 0024487
2010-02-24 22:49	dhx	Note Added: 0024502
2010-02-24 22:49	dhx	Status	assigned => resolved
2010-02-24 22:49	dhx	Resolution	open => fixed
2010-02-24 22:49	dhx	Target Version	1.2.x => 1.2.0
2010-03-03 06:42	valerio	Note Added: 0024607
2010-03-12 13:47	fman	Note Added: 0024730
2010-03-12 13:54	fman	Note Edited: 0024730	View Revisions
2010-04-15 09:54	ngombe	Note Added: 0025131

Notes
(0023939) siebrand (developer) 2009-12-21 12:18	Showstopper 1.2.0.

(0024022) vboctor (administrator) 2010-01-02 03:17	This bug seems to be related to the following checkin: http://git.mantisbt.org/?p=mantisbt.git;a=commit;h=c416dd826b1da0c5093a0dfbe001018e6ffa90ff [^] @grangeway, this is blocking 1.2.0 release. Will you be able to fix it or revert the change?

(0024037) dhx (developer) 2010-01-05 03:52	Cannot reproduce with a 28MB file. Are you sure your PHP memory limits are set high enough? If you don't want to raise PHP memory limits, I strongly suggest looking at a new feature I added today in 0011291... X-Sendfile Actually, just use X-Sendfile wherever possible because it's much higher performance. For small MantisBT installations with few small file uploads it won't make much of a difference though.

(0024451) grangeway (developer) 2010-02-20 15:15	So erm, can anyone reproduce this? ;/

(0024454) ejc (reporter) 2010-02-21 05:29 edited on: 2010-02-21 05:41	I think so, grangeway, with version 1.2.0RC2 on a Apache/2.2.3 (Debian) DAV/2 SVN/1.4.2 mod_ldap_userdir/1.1.11 PHP/5.2.0-8+etch13 mod_ssl/2.2.3 OpenSSL/0.9.8c. The problem AFAIK was that the file downloads were "chunked" to (more or less) the closest 512KB chunk, i.e.: original 560KB file is downloaded as a 512KB file; or a 1600KB file is downloaded with a size of 1512KB. Also we confirmed that the file is uploaded and saved properly to disk on the server. We don't use the database for it. We walk-arounded it for a while with this little patch on nightly build file_download.php of feb-18: //$chunksize = 4131072; $chunksize = 4999999;

(0024455) dhx (developer) 2010-02-22 00:31	Paul: What is the significance of choosing a chunk size of 4131072? Otherwise known as: 4((2^20)/8) I gather it's something highly technical to do with heap sizes and reading files as efficiently as possible with respect to memory allocation, paging, etc?

(0024460) grangeway (developer) 2010-02-22 17:49	ejc: does that include with the latest code in trunk (see attached patch to this bug), and are you reliably able to reproduce this with any file of approxy >512kb? Paul

(0024487) ejc (reporter) 2010-02-24 11:23	Paul: I think I can reliable reproduce this with exactly this platform: mantisbt-1.2.0rc2 + http_api.php (from 18/feb nightly build) + file_download.php (from 18/feb nightly build, patched as said in 0011153:0024454). However, with 1.2.0 released yesterday it works fine. May you want more tests, just ask for them.

(0024502) dhx (developer) 2010-02-24 22:49	Resolving this as it has been reported to be fixed in 1.2.0 If it's not working on a patched pre 1.2.0 release then it is likely a case of patches not being applied correctly, certain required patches missing, etc. We've made quite a few changes to file downloading since 1.2.0RC2 (where file downloads were more or less completely broken). If you can still reproduce this problem on a fresh install of 1.2.0 (ie. not a hacked upgrade from 1.2.0RC2 by placing 1.2.0 files on top of your 1.2.0RC2 release), feel free to reopen this bug report.

(0024607) valerio (reporter) 2010-03-03 06:42	I upgraded to 1.2.0 and the problem still remains BUT I did a lot of debugging and finally found that the problem is related to php (5.1.6) file_get_contents() !!!!! Running file_send_chunk with a CLI script in the last run of the while(true) loop file_get_contents crashes with a bus error on mmap(). (not logged in apache error_log :-( ) So the final solution is : system upgrade this afternoon ;-)

(0024730) fman (reporter) 2010-03-12 13:47 edited on: 2010-03-12 13:54	I'm having same problem and I'm using php 5.2.1 on redhat. I've tried with: $chunksize = 4*999999; and seems to work is this the right solution ?

(0025131) ngombe (reporter) 2010-04-15 09:54	I had the same problem with php 5.1.6. I found 'child pid 24521 exit signal Bus error' in the apache log ... In php.ini the MAX_POST_SIZE is set to 8Mo so I forced $chunksize = 810241024; Now it's working ...