View Issue Details

IDProjectCategoryView StatusLast Update
0017336mantisbtinstallationpublic2017-02-01 22:47
Reportergrangeway Assigned Todregad  
PrioritynormalSeverityminorReproducibilityhave not tried
Status closedResolutionfixed 
Target Version1.3.6Fixed in Version1.3.6 
Summary0017336: Hide non-mysql experimental DB's for new installation whilst we get proven DB Layer into Mantis
Description

To protect our users and ensure a good experience, we should hide by default options to allow new installations for the experimental Database engines.

This will allow us to concentrate on implementing proven support for various database engines, whilst allowing existing users to carry on using their database engine if they are able to patch the code.

This is a temporary measure until we resolve the Database support completely to avoid users to hitting issues when running Mantis. Technical users that are willing to "take the risk" can uncomment the code.

By doing this, we can focus our efforts on supporting users on Mantis on the primary database engine i.e. mysql, and focus our efforts on implementing the new database layer, and avoid confusing new users with regards to our database support.

In the current state of play, we know that users are unable to run our 1.2 releases on pgsql for instance without issues. We know that the situation is improved in both 1.3 and 2.0 - note I currently say "improved" for both and not fixed for either.

If our main focus after 1.3 for 2.0 is to purely get the Database Layer implemented, and do a quick release with that functionality as soon as it is ready, that should be a short term commitment of 2-3 months maximum until a final release.

Technical users of mantis can then decide on whether they wish to attempt to run their chosen database engine on 1.3, or whether to help testing on 2.0. For those non-technical users of Mantis (who purely wants something that works for the task and are not skilled in php development), we can advise them on the best course of action for their individual needs.

In 1.3.1 or 2.0.0 (depending on where we agree to take things with database support), we can re-enable the hidden code as appropriate.

This would seem like a reasonable compromise to move forwards from the current position of two different routes to achieving support for other databases, and ensuring that end-users are not drawn into the debate.

My rationale for commenting engines is as follows:

  • // 'mssql' => 'Microsoft SQL Server',

In 1.2 and 1.3 the schema upgrades for this driver are broken (due to remove of compatibility code that we used to have in adodb). In addition, PHP have dropped this driver.

  • // 'mssqlnative' => 'Microsoft SQL Server Native Driver',

In 1.2 and 1.3 the schema upgrades for this driver are broken (due to remove of compatibility code that we used to have in adodb). This driver was added late in the 1.2 cycle (1.2.11)

  • // 'pgsql' => 'PostgreSQL',

This is broken in 1.2 ( due to booleans vs integers), improved in 1.3

  • // 'oci8' => 'Oracle',

This is broken in 1.2, radically improved by Damien in 1.3, but we still can't say is 100% - schema generation was broken due to 2 bugs a month ago, and there's some complex 'fix sql query' code generated to improve support for the oracle SQL Parser

  • // 'db2' => 'IBM DB2',

As a team, we've agreed so far we'll probably drop DB2 support moving forwards. If that turns out to be our final decision, and is not reverted, it would seem silly to allow new installations on this one

I'd then propose we agree a short timeframe (2 weeks ? 1 month?) where we either release a preview of a 2.0 build that passes test cases on all database engines, or (if a problem arises) use the new test cases to ensure that each database engine listed above passes and re-enable the appropriate engines, and release this as 1.3.1 as a quality product.

TagsNo tags attached.

Activities

vboctor

vboctor

2014-05-16 01:06

manager   ~0040293

  • I agree that we should drop DB2. Or any DBMS that we are willing to drop going forward.
  • I wouldn't remove any case that is better than 1.2.x but not perfect.
  • Changing DB layer is not going to be 1.3.x and is unlikely to land in a couple of month. So far it seems that it is taking us that much to land 1.3.0a1 or rc1.
  • I would prefer if we don't offer users multiple options for the same DBMS (MSSQL case). We should just offer one if at all possible.
grangeway

grangeway

2014-05-16 05:24

reporter   ~0040297

  • Changing DB layer is not going to be 1.3.x and is unlikely to land in a couple of month

Why is it unlikely to land in a couple of months?

dregad

dregad

2014-05-16 06:10

developer   ~0040299

I think we should have a clear definition of what "experimental" means here.

For me, that means MantisBT either does not install out of the box without fiddling with the generated SQL or the database schema (e.g. Oracle in 1.2), does not work (e.g. DB errors when using the software) or that we as a team are not able to support the platform (e.g. DB2)

By this definition, with the changes I've put in, Oracle and Postgres are not experimental anymore. I can't speak for MSSQL since I can't test on that platform.

Therefore I see no reason to effectively hide DB engines and prevent users from selecting them at install time, as it removes value from the 1.3 release in the name of possible future upgrade issues (which should be dealt with when the time comes).

The only one I would agree on, is DB2.

  • I would prefer if we don't offer users multiple options for the same DBMS (MSSQL case). We should just offer one if at all possible.

In reality, anyone using PHP >= 5.3 would not see 'mssql' for new installs [1] and upgrading users will get an error [2].

[1] https://github.com/mantisbt/mantisbt/blob/master/admin/install.php#L539
[2] https://github.com/mantisbt/mantisbt/blob/master/admin/install.php#L241

grangeway

grangeway

2014-05-16 07:22

reporter   ~0040300

"experimental" - I tried installing Oracle a month ago - it didn't install (I know you fixed some of those issues). I wouldn't call something that was broken a month ago as 'proven'.

When I was running my automated testing tool against PGSQL at the weekend (I didn't realise that I wasn't running MySQL, I got some error database layer errors back) - now, as this was a security testing tool, it could be that pgsql was kicking out invalid input (i.e. too long data for field) as opposed to a query error, but..

However, given the DB Layer is changing - that was an agreed goal of the team ages ago, and Victor even said we could have a short turn around between 1.3 and 2.0 with the new layer, It's not fair on users to state that we fully support database engines that have been known to cause issues, and potentially then give them more grief when they try to move from 1.3 to 2.0

You joined the Mantis Project 2-3 years ago now, I've been working on Mantis for 10 years now. I will do everything in my power to ensure that we get a 2.0 ASAP, and everything in my power that we do not pick up new users on Oracle/Pgsql and MSSQL(which as you know I'm passionate about) running 1.3 that may hit issues later on.

This pull request/patch was my attempt to allow existing users to run 1.3, and to protect new users whilst we evaluate our database support properly.

vboctor

vboctor

2014-05-16 12:09

manager   ~0040303

grangeway: Why is it unlikely to land in a couple of months?
Because the rate we are going and the number of times we are having a discussion about the same topics, like this one. We will never get anything done. The other point is that we haven't get release 1.3.0a1 after which we need to get to 1.3.0 stable before we push the next version out or checkin the db layer. That also we have discussed 10 times before. The slower it is for 1.3 going out, the further out the next version is going to be.... just saying...

grangeway: I wouldn't call something that was broken a month ago as 'proven'.
I don't buy this argument. Any product has bugs and it is typically shipped less then 30 days from such bugs being resolved.

Experimental vs. Released vs. Hidden
One option is to consider leaving "experimental" tag on dbs that had it in the past. i.e. don't advertise lower or higher support than 1.2.x. Kind of a compromise, though I think if the db installs fine, works fine, no worse than 1.2.x, and we are not cutting it, then it is OK not to have experimental.

grangeway

grangeway

2014-05-16 14:11

reporter   ~0040304

grangeway: Why is it unlikely to land in a couple of months?
Because the rate we are going and the number of times we are having a
discussion about the same topics, like this one. We will never get anything
done. The other point is that we haven't get release 1.3.0a1 after which we
need to get to 1.3.0 stable before we push the next version out or checkin
the db layer. That also we have discussed 10 times before. The slower it is
for 1.3 going out, the further out the next version is going to be.... just
saying...

Then we should accept no new features to 1.3 apart form bug fixes

In terms of non-mysql DB's, we know that the current DB layer has issues (hence replacing it). Whilst I accept the layer may have some bug fixes, advertising support for something that might be problematic for users is not fair on end users until we fix it properly.

Whilst i'm prepared to accept that technical users of Mantis can deal with those issues, and whilst i agree that the support in 1.3 may be better in 1.2, given I implemented the original ADODB support in Mantis, and have seen the issues that have come up over the last 10 years i'm not prepared to release a new version of Mantis that puts users in a similar position to the current users.

grangeway

grangeway

2014-05-16 14:14

reporter   ~0040305

You joined the Mantis Project 2-3 years ago now, I've been working on Mantis for 10 years now.
What's that supposed to mean ? "Shut up, I've been here longer so I know better than you" ? This is pathetic.

No actually, more, you've not seen the history of the multiple database support in Mantis. It's been problematic for a while, the reason people started looking to replace the ADODB layer before is due to the fact it's problematic. I'm not prepared to advertise support for mantis on other database engines whilst it's still problematic.

I've been there, hit the issues it's caused, and it's not fair on end users to continue.

rombert

rombert

2014-05-19 10:08

reporter   ~0040575

IMO it's not fair for end users to not get the updates we've been working on for 1.3 . I know for a fact that people are running 1.2.x on Postgres, and have tried running on Oracle. And marking a bunch of databases as working in 1.2.x and then hiding them for 1.3.x is not doing anyone any favours, we'd just be annoying our users.

So I am against hiding these until 1.4.x/2.0.x and in favour of releasing 1.3.x with the current DB layer approach and keeping all database engines as visible.

rombert

rombert

2014-05-19 10:21

reporter   ~0040576

One additional thought is that we might choose to do the following:

  • cut a a1/rc1/whatever release
  • gather feedback from users not using MySQL
  • take the decision at that point, based on the number and severity of bugs found VS what we can commit for fixing them for 1.3
grangeway

grangeway

2014-05-19 13:32

reporter   ~0040578

"I know for a fact that people are running 1.2.x on Postgres, and have tried running on Oracle."

The people that are running on them (in 1.2) have had to make custom modifications to get them to work.

For those users, they'd be unaffected by the change - they can still run as they do now, and can still upgrade as they do now.

The only difference is that if they have to make changes to get from their custom code in 1.2/1.3 to 2.x, is that they are already familiar with making those custom changes (and have the skills to do so)

I'm talking about new users that are looking to join mantis for the first time.

grangeway

grangeway

2014-05-19 13:49

reporter   ~0040579

Rombert, in the addition the patch I proposed re-instates the 'experimental' flag for non-mysql database engines.

This is something that has always been in the source code (and only recently removed).

I don't really see how you can mark a feature as non-experimental when also planning to scrap it in a month or two.

grangeway

grangeway

2014-05-19 18:23

reporter   ~0040583

Rombert, also, could you elaborate on what you meant by your additional thought?

atrol

atrol

2014-05-20 09:14

developer   ~0040594

I am not biased and hope that the following compromise is acceptable for all:

  • Remove "IBM DB2", "Microsoft SQL Server" and "Microsoft SQL Server Native Driver" from list in 1.3
  • Remove "PostgreSQL" and/or "Oracle" from list in 1.3 if there is any known open major database specific bug in 1.3.0rc<lastBeforeFinal>
  • Add hint 1 to "PostgreSQL" and hint 2 to "Oracle" in 1.3 if there is any known open minor database specific bug in 1.3.0 final.

1 not fully supported, check www.mantisbt.org/bugs for all open issues with category "db postgresql"
2 not fully supported, check www.mantisbt.org/bugs for all open issues with category "db oracle"

rombert

rombert

2014-05-20 09:38

reporter   ~0040597

Rombert, also, could you elaborate on what you meant by your additional thought?

I mean that we should rather wait for user feedback to declare things experimental. Having some early testers in the alpha stage is the best way to see what works and what not.

dregad

dregad

2014-05-20 11:42

developer   ~0040599

Thanks atrol for being more Swiss than the Swiss ;-)

Remove "IBM DB2"

OK, we can't support it anyway

"Microsoft SQL Server"

OK, it's deprecated in PHP 5.3 and in any case not displayed today, triggers errors at install if used

"Microsoft SQL Server Native Driver"

Not OK, this one benefits from many improvements in ADOdb and should (hopefully) work although TBH I can't test and grangeway won't. I propose to follow same logic as outlined for pgsql/oci8

  • Remove "PostgreSQL" and/or "Oracle" from list in 1.3 if there is any known open major database specific bug in 1.3.0rc<lastBeforeFinal>
  • Add hint 1 to "PostgreSQL" and hint 2 to "Oracle" in 1.3 if there is any known open minor database specific bug in 1.3.0 final.

OK

atrol

atrol

2014-05-20 12:08

developer   ~0040600

"Microsoft SQL Server Native Driver"
TBH I can't test and grangeway won't.

That's why I proposed to remove it from list.

grangeway

grangeway

2014-05-20 12:48

reporter   ~0040602

Atrol:

The issue is more if there are bugs in the schema (ADOdb has been knowing to generate dodgy schema versions).

The purpose of replacing the DB Layer is to ensure that the schema is to a known standard.

At the moment: our current release has problems: isn't to a standard.
At the moment: the code in 1.3 is better: but isn't necessarily to the future standard.

Therefore the upgrade path from 1.2 or 1.3 to a new db layer for non-MySQL is going to require people to be aware of this. Removing the "Experimental" wording that we've used against the driver and shown in 1.2: implies that we believe the driver layer no longer to be experimental.

I can't see how you can say that we'll release 2.0 within a month or 1.3: and that we'll include new DB Layer in that to fix historical issues and ensure we generate a consistent schema across our database engines: whilst at the same time removing the 'experimental' tag from all database engines.

Surely that implies we expect it to generate a supported upgrade path. At the moment: there's stuff being generated by ADODB in some cases (for ALL database engines (including MySQL) that I'd argue is incorrect behaviour)

For MySQL we obviously have to live with it: for Pgsql:oracle and MSSQL because they do not currently work correctly in 1.2 and are marked as experimental in 1.2: we have the flexibility to fix issues and ensure we are using working data types: index names that make sense going forwards.

And when I say the above: we know that other time ADODB has used different data types for different columns etc etc.

If you look back a few months: I believe Damien sent to the list difference between the bugtracker running on Mantisbt.org and that generated by a new installation. Most of those differences if I recall without looking were irrelevant and mainly due to removing ZEROFILL between our original pre-ADODB installation generation routine to moving towards ADODB and dropping ZEROFILL.

If you look more recently: we added a check to the installation checks folder to check that columns were UTF8 enabled in MySQL. This being something that users had to deal with manually (due to the nature of encodings)

I'm fairly confident that Oracle:Pgsql will work OK for users that are technical in 1.3 - what I'm definitely not confident about: and I think will be an issue is that ADODB has some legacy decisions (some of them correct: some wrong now 5-10 years down the line): however there is stuff going into 1.3 as new code (even now) that is relevant for trying to fix '1.3 to support oracle:pgsql': and the new code should probably not be included in 2.0. So as opposed to releasing a 1.3 now: saying that we are working on oracle:pgsql and doing it properly: we are both delaying 1.3: and making more work (in testing 2.0 and testing 1.3) - and lets before we get into a debate about whether or not the code going into 1.3 now that is needed to support 1.3 and can be backed out again to support 2.0 makes sense.

"- Add hint 1 to "PostgreSQL" and hint 2 to "Oracle" in 1.3 if there is any known open minor database specific bug in 1.3.0 final.

1 not fully supported: check www.mantisbt.org:bugs for all open issues with category "db postgresql"
2 not fully supported: check www.mantisbt.org:bugs for all open issues with category "db oracle" "

In terms of this: this doesn't address the point that users should be aware that we are planning on replacing the DB Layer with a fully tested layer in a month's time and could need to do some manual fixes when upgrading. Up until now we've handled that by having the 'experimental' tag listed against the drivers (as we know they are not fully tested and working) - if we are truly planning on getting 1.3 out the door as I've been told then following up with a 2.0 release shortly afterwards (to deal with the DB API changes) it would seem strange that people are proposing dropping the experimental tag.

I'll agree: that on reflection: putting in a Pull Request to hide the functionality was the wrong thing to do. However: we should make users fully aware that we plan on replacing the DB Layer shorter after the 1.3 release and testing Oracle:Pgsql:MSSQL thoroughly - this covers us for upgrade issues: and also makes it clear to users the status.

In terms of
"%3E "Microsoft SQL Server Native Driver" - Not OK: this one benefits from many improvements in ADOdb and should (hopefully) work although TBH I can't test and grangeway won't.""

My work instance of Mantis is already in a state where i'll need to upgrade manually due to ADOdb. There's no point spending time taking something that doesn't work currently: trying to patch it so it works: then replacing it in a month with something that we've tested properly - and finding out that users need to work out how to upgrade from the 'patched' version on their own.

It's a better use of our time ensuring that our new support for databases is known to be consistent across all platforms - and by consistent I mean that they behave the same way as much as possible through the abstraction layer in terms of schema's.

So for example- lets take an obvious problem/difference:

For the Data type within Mantis/ADODB of "I":

In MySQL: we currently make use of the UNSIGNED attribute for every column.

This means that we allow values from 0->4.... as opposed to -2^32 to 2^32 (obviously makes sense - everyone reading this is a coder)

In PGSQL the concept of UNSIGNED does not exist: that means the range is -2^32 to 2^32.

Now: in an ideal world: we'd provide a utility (and it would be possible to) migrate from any version of Mantis to another i.e. do a MYSQL->PGSQL or PGSQL->MYSQL migration for instance: and equally we'd expect the database backend to behave the same between platforms.

So taking the example above: the fact that pgsql does not support unsigned: and that for the most part we aren't going to need >2^32 (and if we do: we have our bigint type): I'd personally be inclined to 'block' numbers greater than 2^32 from being stored against an "I" Datatype column from the Mantis DB Layer.

This ensure compatibility between the database engines - as a side effect of this change: PHP silently switches between using FLOAT's and INT's once the value exceed 2^32 depending on OS Platform requirements.

Obviously: this is a more minor example: but I think you'll agree that we want the behaviour to be known and consistent.

Anyway: I'm still at work at the moment: so I've had to keep this note brief. I can provide a fuller reply later tonight.

grangeway

grangeway

2014-06-01 17:24

reporter   ~0040701

Damien,

We need to see if we can work out some wording that we are both happy with to put to users.

I'll try and catch you on skype and see if we can draft something out that fits both our goals.

I closed the pull request off I initially generated for this a few days ago.

Paul

Related Changesets

MantisBT: master-1.3.x c8312b84

2016-05-10 05:47

dregad


Details Diff
Hide DB2 for new installs

The MantisBT team have no access to a DB2 environment and no technical
knowledge of this database. Consequently, we are not able to support it
moving forward.

This commit removes 'db2' from the database types selection list in the
installer, reflecting the fact that this DB type is deprecated.

This commit should actually have gone in 1.3.0, but I forgot to merge
the branch back then...

Fixes 0017336
Affected Issues
0017336
mod - admin/install.php Diff File
mod - config_defaults_inc.php Diff File

MantisBT: master-1.3.x 3704e888

2016-05-10 05:51

dregad


Details Diff
Install: remove 'mssql' db type

This driver is only supported with PHP < 5.3 and we require 5.3.2.

Fixes 0017336
Affected Issues
0017336
mod - admin/install.php Diff File