mantisbt:importexport
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
mantisbt:importexport [2008/03/20 07:22] – giallu | mantisbt:importexport [2024/03/26 05:03] (current) – [Other] formatting xmllint code block dregad | ||
---|---|---|---|
Line 2: | Line 2: | ||
* **Author**: Gianluca Sforna (giallu) | * **Author**: Gianluca Sforna (giallu) | ||
- | * **Status**: < | + | * **Status**: < |
* **Associated Issue**: http:// | * **Associated Issue**: http:// | ||
+ | |||
===== Introduction ===== | ===== Introduction ===== | ||
A complete solution for copying a list of issues between two different Mantis instances is something asked from quite some time: | A complete solution for copying a list of issues between two different Mantis instances is something asked from quite some time: | ||
- | http:// | + | * http:// |
+ | * http:// | ||
For a greater flexibility, | For a greater flexibility, | ||
Line 22: | Line 24: | ||
* Log in instance B as project manager | * Log in instance B as project manager | ||
* Import the XML file and watch instance B populated with new bugs | * Import the XML file and watch instance B populated with new bugs | ||
- | |||
The first implementation will support all the issue fields (including extended info) and custom fields. | The first implementation will support all the issue fields (including extended info) and custom fields. | ||
Given the extensible nature of the XML format, this feature can be enhanced later to support notes, attachments and all other informations related to the issue. | Given the extensible nature of the XML format, this feature can be enhanced later to support notes, attachments and all other informations related to the issue. | ||
+ | |||
===== Design choices and issues ===== | ===== Design choices and issues ===== | ||
- | The main hurdle for this feature is to define the behavior | + | The main hurdle for this feature is to define the behaviour |
This includes: | This includes: | ||
Line 42: | Line 44: | ||
** Proposed solution ** | ** Proposed solution ** | ||
- | import will happen on a single project in instance B, so warn the user on export if "All project" | + | import will happen on a single project in instance B |
+ | |||
+ | ** Status ** | ||
+ | |||
+ | DONE | ||
=== User IDs === | === User IDs === | ||
Line 54: | Line 61: | ||
On import page, ask what to do with users. Options are: | On import page, ask what to do with users. Options are: | ||
* squash all ids to a single one in instance B | * squash all ids to a single one in instance B | ||
- | * attempt to find a matching user by username, then fallback | + | * attempt to find a matching user by username, then fall back to squashing id |
+ | |||
+ | ** Status ** | ||
+ | |||
+ | DONE | ||
+ | A function implementing the above behaviour exists, but composing a list of users is memory intensive (more than 32Mb to execute with the mantis DB) so I'm refraining from using it. | ||
+ | |||
+ | The current implementation tries by default to match usernames, then squash to the userID performing the import operation | ||
Line 70: | Line 84: | ||
* Replace them with fake links, pretending the linked bug or note was deleted | * Replace them with fake links, pretending the linked bug or note was deleted | ||
+ | ** Status ** | ||
+ | |||
+ | DONE (notes are not in the initial implementation, | ||
+ | |||
+ | |||
+ | |||
+ | === Categories === | ||
+ | ** Issue ** | ||
+ | |||
+ | Category is a required field, but nothing guarantees the target project on instance B will use the same ones as instance A. | ||
+ | |||
+ | ** Proposed solution ** | ||
+ | |||
+ | Add import options: | ||
+ | * skip import of issues with unknown categories | ||
+ | * create missing categories (dangerous...) | ||
+ | |||
+ | ** Status ** | ||
+ | |||
+ | PENDING | ||
Line 100: | Line 134: | ||
=== STEP 3 details === | === STEP 3 details === | ||
+ | < | ||
open XML file | open XML file | ||
- | | + | |
for each < | for each < | ||
create a new Bugdata object | create a new Bugdata object | ||
- | FIXME | + | populate data with < |
- | + | | |
- | | + | |
+ | |||
+ | for each element in (issues, notes) | ||
+ | load data from DB | ||
+ | search fields for links | ||
+ | for each link found | ||
+ | if link ID is known | ||
+ | replace it with known newID | ||
+ | else | ||
+ | replace it with a conventional string or with URL to item in instance A | ||
+ | </ | ||
+ | |||
+ | ===== Requirements ===== | ||
+ | |||
+ | |||
+ | The code being developed depends on: | ||
+ | * PHP 5 | ||
+ | * XMLReader module | ||
+ | * XMLWRiter module | ||
===== Other ===== | ===== Other ===== | ||
- | * The resulting xml file can be tested for validity with xmllint like: | + | * The resulting xml file can be tested for validity with xmllint like: < |
+ | xmllint --noout --dtdvalid mantis.dtd exported_issues.xml | ||
+ | </ | ||
+ | |||
+ | * Check memory consumption when a huge number of issues are exported and imported | ||
+ | * Add checks for required extensions - bool extension_loaded | ||
+ | * let user choose destination project on instance B | ||
- | '' | + | ===== Chat Logs ===== |
- | | + | < |
+ | Mar 16 22:27:57 vboctor giallu, | ||
+ | Mar 16 22:28:30 nuclear_eclipse hi vboctor | ||
+ | Mar 16 22:28:48 vboctor howdy | ||
+ | Mar 16 22:35:13 giallu back | ||
+ | Mar 16 22:38:17 vboctor giallu, | ||
+ | Mar 16 22:38:48 giallu have some time to discuss that now? | ||
+ | Mar 16 22:38:55 vboctor yes | ||
+ | Mar 16 22:39:17 giallu so, let's start from the DTD | ||
+ | Mar 16 22:39:43 nuclear_eclipse are you two talking about the DocBook manuals? | ||
+ | Mar 16 22:39:58 vboctor no, we are discussion an import/ | ||
+ | Mar 16 22:40:07 nuclear_eclipse ah, | ||
+ | Mar 16 22:40:16 giallu I just remembered seeing a dtd in bugzilla, so I read some documentation about DTD and adapted it to our purposes | ||
+ | Mar 16 22:40:36 giallu later I realized there exists XSD | ||
+ | Mar 16 22:41:11 vboctor ok, I guess this is not a big deal. Once we define the format we can make a DTD, XSD or both. | ||
+ | Mar 16 22:41:13 giallu but I don't really want to complicate more this stuff (and I'm on a schedule becase as you probably remember this is a sponsored issue) | ||
+ | Mar 16 22:41:31 giallu then | ||
+ | Mar 16 22:41:59 giallu the DTD defines the fields _and_ their order | ||
+ | Mar 16 22:42:03 vboctor I don't mind if we don't implement all the specified requirements. | ||
+ | Mar 16 22:42:04 thraxisp Are you writing import as well as export? | ||
+ | Mar 16 22:42:12 giallu thraxisp, | ||
+ | Mar 16 22:42:49 thraxisp A DTD will be a must for some one to use the import/ | ||
+ | Mar 16 22:43:18 giallu thraxisp, | ||
+ | Mar 16 22:43:24 vboctor thraxisp, | ||
+ | Mar 16 22:43:49 giallu I don't think it makes any difference if we have a DTD or XSD | ||
+ | Mar 16 22:44:14 giallu but having a defined format is a prerequisite for writing conversion stuff | ||
+ | Mar 16 22:44:22 thraxisp As I recall, a DTD defines the structure and semantics on the XML file. (XSD may do the same). | ||
+ | Mar 16 22:45:08 giallu thraxisp, | ||
+ | Mar 16 22:45:31 giallu like this field contains strings, integers in this range, booleans etc | ||
+ | Mar 16 22:47:06 vboctor XSD and DTD both define schema. | ||
+ | Mar 16 22:47:07 giallu vboctor, | ||
+ | Mar 16 22:47:50 vboctor why do the two orders have to match? | ||
+ | Mar 16 22:48:03 giallu if we want a different ordering it's an additional step we need to carry somewhere, but | ||
+ | Mar 16 22:48:07 vboctor the ordering returned by filter_get_bug_rows can change any time. | ||
+ | Mar 16 22:49:08 vboctor Don' | ||
+ | Mar 16 22:49:27 vboctor If you don't have this extra step, then a chance in the internal code will break the DTD right away. | ||
+ | Mar 16 22:49:32 giallu I wonder wht's the purpose of having a specific ordering, or having readable dates, etc. the file's purpose is meant to be a transport between two mantis instances | ||
+ | Mar 16 22:49:39 vboctor Also adding extra native fields will likely change your format as well. | ||
+ | Mar 16 22:50:06 vboctor Then why not use binary format? | ||
+ | Mar 16 22:50:09 vboctor or serialize? | ||
+ | Mar 16 22:50:29 vboctor The aim of using XML is to have a format that people can easily use to define their own tools based on a stable format. | ||
+ | Mar 16 22:50:35 giallu vboctor, | ||
+ | Mar 16 22:50:39 vboctor this can be export from product X and import into Mantis. | ||
+ | Mar 16 22:51:03 vboctor I think the schema shouldn' | ||
+ | Mar 16 22:51:19 vboctor there is no reason why field A must show up before field B. | ||
+ | Mar 16 22:51:30 giallu vboctor, | ||
+ | Mar 16 22:51:52 giallu so whatever we write as order in the DTD, we need to use on the XML | ||
+ | Mar 16 22:52:16 giallu of course, that's just a validation issue | ||
+ | Mar 16 22:52:30 giallu and the data import would be perfectly fine with any ordering | ||
+ | Mar 16 22:53:37 vboctor In my opinion, the validation should ideally be a step that we execute before importing start. | ||
+ | Mar 16 22:53:51 giallu so, for instance, I could ksort the array from filters and be done with this | ||
+ | Mar 16 22:54:34 vboctor I guess ordering is a not a big deal. | ||
+ | Mar 16 22:54:42 giallu yap. let's move on | ||
+ | Mar 16 22:55:05 vboctor I think the best option is to have a map which defines the name mapping / ordering, but it is not a big deal. | ||
+ | Mar 16 22:55:48 giallu so let's quicly summarize your comments: | ||
+ | Mar 16 22:55:56 giallu 1. The export has a lot of ids. These ids may mean different things to different people working with the data. I had the same problem when defining the SOAP API interface. What I ended up doing is to use an object reference concept. Hence, rather than just using an id, I use an id and a non-localized name to refer to an object. This applies to enumerations, | ||
+ | Mar 16 22:56:42 thraxisp A map might also help for foreign import where fields are missing. | ||
+ | Mar 16 22:56:58 giallu I guess I can use the same system as used in SOAP. I'll add that as needed though. is that OK? | ||
+ | Mar 16 22:58:47 vboctor what do you mean by as needed? | ||
+ | Mar 16 22: | ||
+ | Mar 16 22:59:08 vboctor What I think we should do now is to have < | ||
+ | Mar 16 22:59:34 vboctor Same for view status: < | ||
+ | Mar 16 23:00:11 vboctor The other option is to use < | ||
+ | Mar 16 23:00:38 giallu it means I could assume in a first instance that we are talking about two identical installations, | ||
+ | Mar 16 23:01:05 vboctor But you can't assume that for users, projects, etc. | ||
+ | Mar 16 23:01:07 giallu then we can expand to your proposal in order to support some euristic (is that the word?) | ||
+ | Mar 16 23:01:50 vboctor how are you planning to handle users, projects, etc? | ||
+ | Mar 16 23:01:51 giallu vboctor, | ||
+ | Mar 16 23:01:56 giallu i.e. | ||
+ | Mar 16 23:02:49 * vboctor checked the wiki page. | ||
+ | Mar 16 23:02:53 giallu define a squash-to userid during import and ask admin if everything has to be squashed to that user or just the not matching ids | ||
+ | Mar 16 23:02:58 vboctor your sample xml didn't include user names. | ||
+ | Mar 16 23:03:42 giallu for projects. I think I'm limiting the import to a _single_ project | ||
+ | Mar 16 23:04:12 giallu I don't really want to guess which project an issue belongs to | ||
+ | Mar 16 23:04:34 giallu vboctor, | ||
+ | Mar 16 23:04:58 giallu reporters? | ||
+ | Mar 16 23:05:14 vboctor I just think having < | ||
+ | Mar 16 23:05:29 vboctor by user names, i meant reporters, handlers, etc. | ||
+ | Mar 16 23:05:50 giallu so basically, everywhere we have an id... | ||
+ | Mar 16 23:06:08 CIA-11 mantisbt: | ||
+ | Mar 16 23:06:08 CIA-11 mantisbt: | ||
+ | Mar 16 23:06:08 CIA-11 mantisbt: | ||
+ | Mar 16 23:08:36 vboctor giallu, | ||
+ | Mar 16 23:09:24 giallu vboctor, | ||
+ | Mar 16 23:10:38 vboctor just end up with more code to maintain for no strong reason. | ||
+ | Mar 16 23:11:02 vboctor now we need to handle < | ||
+ | Mar 16 23:11:34 vboctor do you think it is a lot more work to export id/name combinations? | ||
+ | Mar 16 23:11:58 vboctor or is the logic to be intelligent about such combinations on import is what is worrying you? | ||
+ | Mar 16 23:12:36 giallu yeah. it's just more work in the import side (which is still a bit foggy in my head) | ||
+ | Mar 16 23:12:57 giallu I think I can do that | ||
+ | Mar 16 23:13:06 giallu just one more Q | ||
+ | Mar 16 23:13:33 giallu do you think your 4 years old note about using DOM XML still apply? | ||
+ | Mar 16 23:14:02 giallu if possible, I'd rather use some existing php module for the job | ||
+ | Mar 16 23:14:27 giallu though right now I'm writing it with custom code | ||
+ | Mar 16 23:14:35 CIA-11 mantisbt: | ||
+ | Mar 16 23:14:35 CIA-11 mantisbt: | ||
+ | Mar 16 23:14:35 CIA-11 mantisbt: | ||
+ | Mar 16 23:15:17 vboctor custom code is fine. | ||
+ | Mar 16 23:16:11 giallu yes. but I'm asking if it's fine to use some xml related functionality. The parsing is trickier | ||
+ | Mar 16 23:17:50 vboctor I don't understand your question, probably since I don't remember the comment you are referring to. | ||
+ | Mar 16 23:17:58 * vboctor checking... | ||
+ | Mar 16 23:18:13 giallu ah. you asked to _not_ use DOM XML | ||
+ | Mar 16 23:18:34 giallu becasue it was possibly not available everywhere | ||
+ | Mar 16 23:19:42 vboctor yep, | ||
+ | Mar 16 23:19:48 vboctor what is the status of DOM XML and PHP 5? | ||
+ | Mar 16 23:19:59 vboctor I remeber there was also Simple XML in PHP 5? | ||
+ | Mar 16 23:20:36 giallu right | ||
+ | Mar 16 23:21:02 vboctor Given that export / import is not a core feature, I am ok with using a PHP extension. | ||
+ | Mar 16 23:21:09 giallu great | ||
+ | Mar 16 23:21:17 vboctor Similar to twitter notifications that is dependent on curl extension. | ||
+ | Mar 16 23:21:37 giallu I' | ||
+ | Mar 16 23:22:21 giallu but I'm definitely looking for some cooked solution about import | ||
+ | Mar 16 23:22:43 vboctor consistency between export / import would be ideal. | ||
+ | Mar 16 23:23:06 vboctor you also need to be careful that PHP doesn' | ||
+ | Mar 16 23:23:47 giallu ok. I'm noting this on the wiki page | ||
+ | Mar 16 23:33:20 vboctor good | ||
+ | </ |
mantisbt/importexport.1206012169.txt.gz · Last modified: 2008/10/29 04:31 (external edit)