Deleting the null configs makes MantisBT work again, but the configuration is lost. It is also hard for users to discover the issue.

I looked into this further and here are the observations:

• The value in the database is not actually null, but the old php serialize() format.
• The reason the database ends in this state, is that there is at least one bad complex value that unserialize() can't decode.
• Once we hit one bad complex value, we don't even try the rest and assume there is a code bug. Fact is there are cases where for some reason the serialized value is invalid.
• I've fixed this by unserialize() + json_encode() for all entries. For the ones that fail, I just deleted them.

Having such invalid configs generates all sort of random errors in the apps, since other APIs gets a null value for config thresholds or workflows and passes these to APIs that don't expect nulls.

My recommendation here is to follow the model:

• attempt to upgrade all complex fields.
• when failed, show an error and continue.

If we want beta1 users to benefit from this, then we can add this as another step that detects complex fields with value starting with "a:" (for array) and upgrade them. If unserialize() fails, then delete.

To reproduce this create a 1.2.x instance with some of these complex configs, corrupt one of them and then upgrade to 1.3.

I'm curious as to how you got there in the first place, considering that the install_check_config_serialization() upgrade function should fail with an error if it detects an invalid config (leaving you with an incomplete upgrade).

Continuing after an error is incorrect, and IMO having the installer deleting an offending (corrupted) value may be a problem, so I think it's best to leave fixing such issues to manual correction by the admin.

What we should do is make it clear to the admin what why the error is occuring, and give them options (what they can do to fix it).

I'm not sure how this could be done in the installer, as by nature the options are fairly limited in terms of what we can display back to the admin in case of error. Maybe an admin check ?

The upgrade was done using upgrade_unattended script. I'm not sure whether the script stopped on first error or just ignore that this step is failing and continued other steps.

If we assume that the customer has a backup and that configs is not really the end of the world, then we may delete the bad items and provide the user with a list of what we have deleted after the fact. That wouldn't help with the upgrade_unattended script though.

The other issue that I'm not sure about is why unserialize() is sometimes not able to deserialize its own data. And why such configs didn't cause an issue before the upgrade. i.e. the blob was invalid. Then even with old 1.2.x code, the app should have got some bad value (e.g. false/null, etc) that would have impacted the logic of the app.

If we assume that the customer has a backup

Ahem. Well, you know... I thought I had one, but...
;-)

Joke aside, do you have some specific example(s) of corrupted entries ? I'd like to have a look at the upgrade_unattended script with a real test case (I never actually used it myself). It'd also be interesting to see what happens in 1.2.x with such corrupted data.

Here is a sample corrupt filter:

Corruption caused by special characters next to "+educa" - remove the "v8#" prefix and then test the result with unserialize().

v8#a:44:{s:13:"show_category";a:1:{i:0;s:1:"0";}s:13:"show_severity";a:1:{i:0;i:0;}s:11:"show_status";a:1:{i:0;i:0;}s:17:"highlight_changed";i:6;s:11:"reporter_id";a:1:{i:0;i:0;}s:10:"handler_id";a:1:{i:0;i:0;}s:10:"project_id";i:0;s:15:"show_resolution";a:1:{i:0;i:0;}s:10:"show_build";a:1:{i:0;s:1:"0";}s:12:"show_version";a:1:{i:0;s:1:"0";}s:11:"hide_status";a:1:{i:0;i:90;}s:12:"user_monitor";a:1:{i:0;i:0;}s:4:"sort";s:12:"last_updated";s:3:"dir";s:4:"DESC";s:8:"per_page";i:50;s:10:"match_type";i:0;s:8:"_version";s:2:"v8";s:10:"_view_type";s:8:"advanced";s:13:"sticky_issues";i:1;s:8:"platform";i:0;s:2:"os";i:0;s:8:"os_build";i:0;s:11:"start_month";i:0;s:9:"start_day";i:0;s:10:"start_year";i:0;s:9:"end_month";i:0;s:7:"end_day";i:0;s:8:"end_year";i:0;s:6:"search";s:0:"";s:16:"and_not_assigned";b:0;s:17:"do_filter_by_date";b:0;s:10:"view_state";i:0;s:17:"relationship_type";i:-1;s:16:"relationship_bug";i:0;s:14:"target_version";a:1:{i:0;s:1:"0";}s:10:"tag_string";s:23:"+multimidia,+educa��o";s:10:"tag_select";s:0:"";s:12:"note_user_id";a:1:{i:0;i:0;}s:13:"show_priority";a:1:{i:0;i:0;}s:16:"fixed_in_version";a:1:{i:0;s:1:"0";}s:12:"show_profile";a:1:{i:0;i:0;}s:13:"custom_fields";a:4:{i:4;a:1:{i:0;s:1:"0";}i:1;a:1:{i:0;s:1:"0";}i:3;a:1:{i:0;s:1:"0";}i:2;a:1:{i:0;s:1:"0";}}s:10:"profile_id";i:0;s:16:"_source_query_id";s:0:"";}

Just tested this a bit more in depth:

1. setup a new 1.2.x DB
2. set a complex config, e.g. by customizing workflow
3. manually edit the config (via phpMyAdmin) to make it corrupt

At this stage, the adm_config_report.php page will show the config as corrupted, and depending on error settings, a PHP notice will be displayed on some screens:

All Users   Test project    status_enum_workflow    complex     The configuration in the database is corrupted. administrator

SYSTEM NOTICE: 'unserialize(): Error at offset 312 of 548 bytes' in '/home/dregad/dev/mantisbt/core/config_api.php' line 161

4. run the upgrade (interactively) - fails with error

Schema step 193: UpdateFunction ( check_config_serialization )

SYSTEM NOTICE: 'unserialize(): Error at offset 312 of 548 bytes' in '/home/dregad/dev/mantisbt/core/install_helper_functions_api.php' line 644

BAD

check_config_serialization

Array

5. run upgrade_unattended.php - fails with error (on command-line, script exits with error code 1)

Post 1.0 schema changes

Connecting to database... OK

Schema UpdateFunction ( check_config_serialization )  - ERROR: check_config_serialization

1 schema upgrades executed.

Failed.

Based on the above, as far as I can tell the system works as designed, and no change is required.

Let me know your thoughts.