Unable to post commands and error 504 when attaching files

[BCITMike]

IT configured up a mantis 1.0.8 server back in like 2007 running on Ubuntu 10.04. We have been using it pretty much since then with very little issues. A couple of times when Mantis started acting funny, I'd login to the box and find tons of backups taking up space and no space was left. Remove all the excessive backups, reboot, and mantis is back to working again. Unfortunately, I don't have the former IT person or resources to get someone else to fix this, so I have to see what I can figure out and get it back running.

Last week or so, Mantis stopped taking file uploads to attach to bug reports and displaying the message:

Code: Select all

APPLICATION ERROR #504
No file was uploaded. Please go back and Choose a file before pressing Upload

But we could still add new bugs and add comments.

This week, the problem has spread and now we cannot add comments or new bug reports. When trying to add a new bug report or comment, the browser never errors out, it just sits there like its waiting for the other side, forever.

I have only done basic checking so far:
- nothing in dmesg or messages or mysql.err
- syslog showed a password problem with our email server. I fixed it, rebooted mantis and the messages no longer appear, but I'm also not getting the email.

I don't think lack of space is the issue:

Code: Select all

root@Bacchus:/website/mantis# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              38G  8.0G   29G  23% /
none                  245M  200K  245M   1% /dev
none                  249M     0  249M   0% /dev/shm
none                  249M   40K  249M   1% /var/run
none                  249M     0  249M   0% /var/lock
none                  249M     0  249M   0% /lib/init/rw

Given that ubuntu never installed anything recent (it's end of life'd ages ago), I can't think of what changed to make this problem suddenly appear. Perhaps corruption somewhere? Broke a permissions or something?

What debug would be beneficial for trying to determine the root cause?

I don't know what the effort required to be able to install a fresh mantis on centos 7 and migrate/back/restore the database from the Ubuntu 10.04 server. I think I'll have to pursue that in parallel as fixing the older 1.0.8 mantis on EOL Ubuntu. It would be nice to be on a current, supported version than the old one, but it ran stable enough that we didn't feel we had to (until now).

Thanks very much in advance for any help, guidance or advice. I apologize for not full reading up and trying more things before requesting help, but I'm also under a time crunch and wouldn't complain if the problem/solution was told to me

[atrol]

Preface
1.0.8 is before the time I started using MantisBT, thus some of my advice might be wrong.
I don't have much time to have a deeper look at the old code.
Installing a newer version of MantisBT is quite easy, upgrading from 1.0.8 to latest stable 1.2.x is also not a big deal (at least if you are running an original MantisBT without any code changes)

- nothing in dmesg or messages or mysql.err

Check also web server error log (not sure, maybe it's under /var/www/logs/apache2 on Ubuntu 10.04)

- syslog showed a password problem with our email server. I fixed it, rebooted mantis and the messages no longer appear, but I'm also not getting the email.

This might be your main problem.
To find it out, you can temporary disable email notifications by adding the following line to file config_inc.php

Code: Select all

$g_enable_email_notification	= OFF;

[BCITMike]

Preface
1.0.8 is before the time I started using MantisBT, thus some of my advice might be wrong.
I don't have much time to have a deeper look at the old code.
Installing a newer version of MantisBT is quite easy, upgrading from 1.0.8 to latest stable 1.2.x is also not a big deal (at least if you are running an original MantisBT without any code changes)

- nothing in dmesg or messages or mysql.err

Check also web server error log (not sure, maybe it's under /var/www/logs/apache2 on Ubuntu 10.04)

- syslog showed a password problem with our email server. I fixed it, rebooted mantis and the messages no longer appear, but I'm also not getting the email.

This might be your main problem.
To find it out, you can temporary disable email notifications by adding the following line to file config_inc.php

Code: Select all

$g_enable_email_notification	= OFF;

Thanks for the reply. It looks like the mail was incorrect since May of last year. After fixing it, I got 5 backup summary emails from it (the server, not mantis specifically) for backup scripts that run on mysql from the previous 5 days of backups.

apache2's error.log contains the following:

Code: Select all

[Tue Jan 20 18:43:44 2015] [notice] caught SIGTERM, shutting down
PHP Deprecated:  Comments starting with '#' are deprecated in /etc/php5/apache2/conf.d/mcrypt.ini on line 1 in Unknown on line 0
[Tue Jan 20 18:43:56 2015] [notice] FastCGI: process manager initialized (pid 934)
[Tue Jan 20 18:43:56 2015] [notice] mod_python: Creating 8 session mutexes based on 150 max processes and 0 max threads.
[Tue Jan 20 18:43:56 2015] [notice] mod_python: using mutex_directory /tmp
[Tue Jan 20 18:43:56 2015] [notice] Apache/2.2.14 (Ubuntu) mod_fastcgi/2.4.6 PHP/5.3.2-1ubuntu4.5 with Suhosin-Patch mod_python/3.3.1 Python/2.6.5 mod_ssl/2.2.14 OpenSSL/0.9.8k mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations

I noticed in 'ps aux' output, that apache2 is ran by www-data, except for root for one instance. That seems fishy.

Code: Select all

root       927  0.0  2.3  42416 11724 ?        Ss   Jan20   0:02 /usr/sbin/apache2 -k start
www-data   934  0.0  0.6  23456  3120 ?        S    Jan20   0:00 /usr/sbin/fcgi-pm -k start
root       960  0.0  0.1   1788   572 tty1     Ss+  Jan20   0:00 /sbin/getty -8 38400 tty1
www-data   989  0.0  1.1  42416  6088 ?        S    Jan20   0:00 /usr/sbin/apache2 -k start
www-data   990  0.0  1.1  42416  6088 ?        S    Jan20   0:00 /usr/sbin/apache2 -k start
www-data   991  0.0  1.1  42416  6100 ?        S    Jan20   0:00 /usr/sbin/apache2 -k start
www-data   992  0.0  1.1  42416  6088 ?        S    Jan20   0:00 /usr/sbin/apache2 -k start
www-data   993  0.0  1.1  42416  6088 ?        S    Jan20   0:00 /usr/sbin/apache2 -k start

I've fixed the mcrypt.ini error warning and rebooted, but didn't help.

Getting the following in syslog after rebooting server. Not sure if this has always been there, or an indication mysql is unhappy:

Code: Select all

Jan 21 10:51:20 Bacchus /etc/mysql/debian-start[1223]: Upgrading MySQL tables if necessary.
Jan 21 10:51:20 Bacchus /etc/mysql/debian-start[1226]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
Jan 21 10:51:20 Bacchus /etc/mysql/debian-start[1226]: Looking for 'mysql' as: /usr/bin/mysql
Jan 21 10:51:20 Bacchus /etc/mysql/debian-start[1226]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Jan 21 10:51:20 Bacchus /etc/mysql/debian-start[1226]: This installation of MySQL is already upgraded to 5.1.41, use --force if you still need to run mysql_upgrade
Jan 21 10:51:20 Bacchus /etc/mysql/debian-start[1233]: Checking for insecure root accounts.
Jan 21 10:51:20 Bacchus /etc/mysql/debian-start[1237]: Triggering myisam-recover for all MyISAM tables

I'll look at upgrading mantis to see if that helps.

Thanks!!!

[BCITMike]

I enabled debug in mantis and checked syslog and noticed two errors:

Code: Select all

/etc/mysql/debian-start[21228]: mysql.columns_priv                                 OK
/etc/mysql/debian-start[21228]: mysql.db                                           OK
/etc/mysql/debian-start[21228]: mysql.event                                        OK
/etc/mysql/debian-start[21228]: mysql.func                                         OK
/etc/mysql/debian-start[21228]: mysql.general_log
/etc/mysql/debian-start[21228]: Error    : You can't use locks with log tables.
/etc/mysql/debian-start[21228]: status   : OK
/etc/mysql/debian-start[21228]: mysql.help_category                                OK
/etc/mysql/debian-start[21228]: mysql.help_keyword                                 OK
/etc/mysql/debian-start[21228]: mysql.help_relation                                OK
/etc/mysql/debian-start[21228]: mysql.help_topic                                   OK
/etc/mysql/debian-start[21228]: mysql.host                                         OK
/etc/mysql/debian-start[21228]: mysql.ndb_binlog_index                             OK
/etc/mysql/debian-start[21228]: mysql.plugin                                       OK
/etc/mysql/debian-start[21228]: mysql.proc                                         OK
/etc/mysql/debian-start[21228]: mysql.procs_priv                                   OK
/etc/mysql/debian-start[21228]: mysql.servers                                      OK
/etc/mysql/debian-start[21228]: mysql.slow_log
/etc/mysql/debian-start[21228]: Error    : You can't use locks with log tables.
/etc/mysql/debian-start[21228]: status   : OK
/etc/mysql/debian-start[21228]: mysql.tables_priv                                  OK
/etc/mysql/debian-start[21228]: mysql.time_zone                                    OK
/etc/mysql/debian-start[21228]: mysql.time_zone_leap_second                        OK
/etc/mysql/debian-start[21228]: mysql.time_zone_name                               OK
/etc/mysql/debian-start[21228]: mysql.time_zone_transition                         OK
/etc/mysql/debian-start[21228]: mysql.time_zone_transition_type                    OK
/etc/mysql/debian-start[21228]: mysql.user                                         OK

Is this normal or should I focus on these errors relating to mysql locks? I can see how a lock on mysql prevents adding notes and attachments.

[BCITMike]

More debug from /var/log/mysql/error.log does point to mysql issues...

Code: Select all

150121 11:31:40 [Note] Plugin 'FEDERATED' is disabled.
150121 11:31:40  InnoDB: Initializing buffer pool, size = 8.0M
150121 11:31:40  InnoDB: Completed initialization of buffer pool
150121 11:31:40  InnoDB: Started; log sequence number 0 1883923766
150121 11:31:40  InnoDB: Starting shutdown...
150121 11:31:45  InnoDB: Shutdown completed; log sequence number 0 1883923766
150121 11:31:45 [Note] Plugin 'FEDERATED' is disabled.
150121 11:31:45  InnoDB: Initializing buffer pool, size = 8.0M
150121 11:31:45  InnoDB: Completed initialization of buffer pool
150121 11:31:45  InnoDB: Started; log sequence number 0 1883923766
ERROR: 1064  You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'ALTER TABLE user ADD column Show_view_priv enum('N','Y') CHARACTER SET utf8 NOT ' at line 1
150121 11:31:45 [ERROR] Aborting

150121 11:31:45  InnoDB: Starting shutdown...
150121 11:31:50  InnoDB: Shutdown completed; log sequence number 0 1883923766
150121 11:31:50 [Note] /usr/sbin/mysqld: Shutdown complete

150121 11:31:50 [Note] Plugin 'FEDERATED' is disabled.
150121 11:31:50  InnoDB: Initializing buffer pool, size = 8.0M
150121 11:31:50  InnoDB: Completed initialization of buffer pool
150121 11:31:50  InnoDB: Started; log sequence number 0 1883923766
150121 11:31:50  InnoDB: Starting shutdown...
150121 11:31:55  InnoDB: Shutdown completed; log sequence number 0 1883923766
150121 11:31:55 [Note] Plugin 'FEDERATED' is disabled.
150121 11:31:55  InnoDB: Initializing buffer pool, size = 8.0M
150121 11:31:55  InnoDB: Completed initialization of buffer pool
150121 11:31:55  InnoDB: Started; log sequence number 0 1883923766
ERROR: 1050  Table 'plugin' already exists
150121 11:31:55 [ERROR] Aborting

150121 11:31:55  InnoDB: Starting shutdown...
150121 11:32:01  InnoDB: Shutdown completed; log sequence number 0 1883923766
150121 11:32:01 [Note] /usr/sbin/mysqld: Shutdown complete

150121 11:32:01 [Note] Plugin 'FEDERATED' is disabled.
150121 11:32:01  InnoDB: Initializing buffer pool, size = 8.0M
150121 11:32:01  InnoDB: Completed initialization of buffer pool
150121 11:32:01  InnoDB: Started; log sequence number 0 1883923766
150121 11:32:01 [Note] Event Scheduler: Loaded 0 events
150121 11:32:01 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.73-0ubuntu0.10.04.1-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
150121 11:32:05 [Warning] Statement may not be safe to log in statement format. Statement: ALTER TABLE general_log
   MODIFY event_time TIMESTAMP NOT NULL,
   MODIFY user_host MEDIUMTEXT NOT NULL,
   MODIFY thread_id INTEGER NOT NULL,
   MODIFY server_id INTEGER UNSIGNED NOT NULL,
   MODIFY command_type VARCHAR(64) NOT NULL,
   MODIFY argument MEDIUMTEXT NOT NULL
150121 11:32:05 [Warning] Statement may not be safe to log in statement format. Statement: ALTER TABLE slow_log
   MODIFY start_time TIMESTAMP NOT NULL,
   MODIFY user_host MEDIUMTEXT NOT NULL,
   MODIFY query_time TIME NOT NULL,
   MODIFY lock_time TIME NOT NULL,
   MODIFY rows_sent INTEGER NOT NULL,
   MODIFY rows_examined INTEGER NOT NULL,
   MODIFY db VARCHAR(512) NOT NULL,
   MODIFY last_insert_id INTEGER NOT NULL,
   MODIFY insert_id INTEGER NOT NULL,
   MODIFY server_id INTEGER UNSIGNED NOT NULL,
   MODIFY sql_text MEDIUMTEXT NOT NULL

[BCITMike]

Well, I feel embarrassed. The problem *was* the server running out of space... I was logged into the wrong server! Both an out of date spreadsheet as well as multiple mantis dirs on the server I was logged into hid this mistake. It was when I was porting over the httpd.conf stuff that I found out mantis wasn't running on that server.

EPIC fail. /me throws previous IT admin under the bus.


TL;DR: If you are getting error 504 when trying to add notes or attachments, ENSURE THE SERVER HAS AVAILABLE SPACE!