User Tools

  • Logged in as: anonymous (anonymous)
  • Log Out

Site Tools


mantisbt:importexport

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
mantisbt:importexport [2008/03/20 07:22] giallumantisbt:importexport [2024/03/26 05:03] (current) – [Other] formatting xmllint code block dregad
Line 2: Line 2:
  
    * **Author**: Gianluca Sforna (giallu)    * **Author**: Gianluca Sforna (giallu)
-   * **Status**: <del>Draft</del> | Work in Progress | <del>Implemented</del>+   * **Status**: <del>Draft</del><del>Work in Progress</del> | Implemented
    * **Associated Issue**: http://www.mantisbt.org/bugs/view.php?id=4220    * **Associated Issue**: http://www.mantisbt.org/bugs/view.php?id=4220
 +
  
 ===== Introduction ===== ===== Introduction =====
 A complete solution for copying a list of issues between two different Mantis instances is something asked from quite some time: A complete solution for copying a list of issues between two different Mantis instances is something asked from quite some time:
  
-http://www.mantisbt.org/bugs/view.php?id=4220+  * http://www.mantisbt.org/bugs/view.php?id=4220 
 +  * http://www.mantisbt.org/bugs/view.php?id=5310
  
 For a greater flexibility, the use of an XML format looks like the best option (XML export was also asked in http://www.mantisbt.org/bugs/view.php?id=4024). For a greater flexibility, the use of an XML format looks like the best option (XML export was also asked in http://www.mantisbt.org/bugs/view.php?id=4024).
Line 22: Line 24:
   * Log in instance B as project manager   * Log in instance B as project manager
   * Import the XML file and watch instance B populated with new bugs   * Import the XML file and watch instance B populated with new bugs
- 
  
 The first implementation will support all the issue fields (including extended info) and custom fields. The first implementation will support all the issue fields (including extended info) and custom fields.
  
 Given the extensible nature of the XML format, this feature can be enhanced later to support notes, attachments and all other informations related to the issue. Given the extensible nature of the XML format, this feature can be enhanced later to support notes, attachments and all other informations related to the issue.
 +
  
  
 ===== Design choices and issues ===== ===== Design choices and issues =====
  
-The main hurdle for this feature is to define the behavior when any piece of information is an internal reference to instance A.+The main hurdle for this feature is to define the behaviour when any piece of information is an internal reference to instance A.
  
 This includes: This includes:
Line 42: Line 44:
 ** Proposed solution ** ** Proposed solution **
  
-import will happen on a single project in instance B, so warn the user on export if "All project" is selected+import will happen on a single project in instance B 
 + 
 +** Status ** 
 + 
 +DONE 
  
 === User IDs === === User IDs ===
Line 54: Line 61:
 On import page, ask what to do with users. Options are: On import page, ask what to do with users. Options are:
   * squash all ids to a single one in instance B   * squash all ids to a single one in instance B
-  * attempt to find a matching user by username, then fallback to squashing id+  * attempt to find a matching user by username, then fall back to squashing id 
 + 
 +** Status ** 
 + 
 +DONE 
 +A function implementing the above behaviour exists, but composing a list of users is memory intensive (more than 32Mb to execute with the mantis DB) so I'm refraining from using it. 
 + 
 +The current implementation tries by default to match usernames, then squash to the userID performing the import operation
  
  
Line 70: Line 84:
   * Replace them with fake links, pretending the linked bug or note was deleted   * Replace them with fake links, pretending the linked bug or note was deleted
  
 +** Status **
 +
 +DONE (notes are not in the initial implementation, but code can support those)
 +
 +
 +
 +=== Categories ===
 +** Issue ** 
 +
 +Category is a required field, but nothing guarantees the target project on instance B will use the same ones as instance A.
 +
 +** Proposed solution **
 +
 +Add import options:
 +  * skip import of issues with unknown categories
 +  * create missing categories (dangerous...)
 +
 +** Status **
 +
 +PENDING
  
  
Line 100: Line 134:
  
 === STEP 3 details === === STEP 3 details ===
 +<code>
   open XML file   open XML file
-  create oldID => newID maps for: issues, users+  initialize oldID => newID array for: issues, notes
   for each <issue> element   for each <issue> element
     create a new Bugdata object     create a new Bugdata object
-FIXME +    populate data with <issue> fields 
-     +    save it into DB 
-    +    save oldID => newID pair 
 + 
 +  for each element in (issues, notes) 
 +    load data from DB 
 +    search fields for links 
 +    for each link found 
 +      if link ID is known 
 +        replace it with known newID 
 +      else 
 +        replace it with a conventional string or with URL to item in instance A 
 +</code> 
 + 
 +===== Requirements ===== 
 + 
 + 
 +The code being developed depends on: 
 +  * PHP 5 
 +  * XMLReader module 
 +  * XMLWRiter module
  
  
 ===== Other ===== ===== Other =====
  
-  * The resulting xml file can be tested for validity with xmllint like:+  * The resulting xml file can be tested for validity with xmllint like: <code> 
 +xmllint --noout --dtdvalid mantis.dtd exported_issues.xml 
 +</code> 
 + 
 +  * Check memory consumption when a huge number of issues are exported and imported 
 +  * Add checks for required extensions - bool extension_loaded  ( string $name  ) 
 +  * let user choose destination project on instance B 
  
-  ''xmllint --noout --dtdvalid mantis.dtd export_file.xml''+===== Chat Logs =====
  
-  Check memory consumption when a huge number of issues are exported+<file> 
 +Mar 16 22:27:57 vboctor giallu, I've submitted a note with my comments on the XML export. 
 +Mar 16 22:28:30 nuclear_eclipse hi vboctor 
 +Mar 16 22:28:48 vboctor howdy 
 +Mar 16 22:35:13 giallu back 
 +Mar 16 22:38:17 vboctor giallu, let me know your thoughts about the xml comments. 
 +Mar 16 22:38:48 giallu have some time to discuss that now? 
 +Mar 16 22:38:55 vboctor yes 
 +Mar 16 22:39:17 giallu so, let's start from the DTD 
 +Mar 16 22:39:43 nuclear_eclipse are you two talking about the DocBook manuals? 
 +Mar 16 22:39:58 vboctor no, we are discussion an import/export xml format for Mantis. 
 +Mar 16 22:40:07 nuclear_eclipse ah, ok, nm :P 
 +Mar 16 22:40:16 giallu I just remembered seeing a dtd in bugzilla, so I read some documentation about DTD and adapted it to our purposes 
 +Mar 16 22:40:36 giallu later I realized there exists XSD 
 +Mar 16 22:41:11 vboctor ok, I guess this is not a big deal.  Once we define the format we can make a DTD, XSD or both. 
 +Mar 16 22:41:13 giallu but I don't really want to complicate more this stuff (and I'm on a schedule becase as you probably remember this is a sponsored issue) 
 +Mar 16 22:41:31 giallu then 
 +Mar 16 22:41:59 giallu the DTD defines the fields _and_ their order 
 +Mar 16 22:42:03 vboctor I don't mind if we don't implement all the specified requirements.  However, whatever we implement should fall into a preset vision. 
 +Mar 16 22:42:04 thraxisp Are you writing import as well as export? 
 +Mar 16 22:42:12 giallu thraxisp, yap 
 +Mar 16 22:42:49 thraxisp A DTD will be a must for some one to use the import/export to interface to another system. 
 +Mar 16 22:43:18 giallu thraxisp, see http://www.mantisbt.org/wiki/doku.php/mantisbt:importexport for more details 
 +Mar 16 22:43:24 vboctor thraxisp, by DTD, you mean a DTD (or XSD) or specifically a DTD? 
 +Mar 16 22:43:49 giallu I don't think it makes any difference if we have a DTD or XSD  
 +Mar 16 22:44:14 giallu but having a defined format is a prerequisite for writing conversion stuff 
 +Mar 16 22:44:22 thraxisp As I recall, a DTD defines the structure and semantics on the XML file. (XSD may do the same). 
 +Mar 16 22:45:08 giallu thraxisp, right. additionally AFAICT with XSD you have more powerful definition rules 
 +Mar 16 22:45:31 giallu like this field contains strings, integers in this range, booleans etc 
 +Mar 16 22:47:06 vboctor XSD and DTD both define schema.  However, XSD is in XML format and is a newer standard compared to DTD. 
 +Mar 16 22:47:07 giallu vboctor, so, about your comment (2), I ordered the fields in the dtd with the same ordering I got them from filter_get_bug_rows 
 +Mar 16 22:47:50 vboctor why do the two orders have to match? 
 +Mar 16 22:48:03 giallu if we want a different ordering it's an additional step we need to carry somewhere, but 
 +Mar 16 22:48:07 vboctor the ordering returned by filter_get_bug_rows can change any time. 
 +Mar 16 22:49:08 vboctor Don't you get an array of issues, then you have your logic that translates the BugData into XML?  Hence, you have to specify the order anyway? 
 +Mar 16 22:49:27 vboctor If you don't have this extra step, then a chance in the internal code will break the DTD right away. 
 +Mar 16 22:49:32 giallu I wonder wht's the purpose of having a specific ordering, or having readable dates, etc. the file's purpose is meant to be a transport between two mantis instances 
 +Mar 16 22:49:39 vboctor Also adding extra native fields will likely change your format as well. 
 +Mar 16 22:50:06 vboctor Then why not use binary format? 
 +Mar 16 22:50:09 vboctor or serialize? 
 +Mar 16 22:50:29 vboctor The aim of using XML is to have a format that people can easily use to define their own tools based on a stable format. 
 +Mar 16 22:50:35 giallu vboctor, anyway, yes, I can force an ordering on the fields, it's just that right now I haven't done it ;) 
 +Mar 16 22:50:39 vboctor this can be export from product X and import into Mantis. 
 +Mar 16 22:51:03 vboctor I think the schema shouldn't require a specific order. 
 +Mar 16 22:51:19 vboctor there is no reason why field A must show up before field B. 
 +Mar 16 22:51:30 giallu vboctor, the xml validates _only_ if the order is the same 
 +Mar 16 22:51:52 giallu so whatever we write as order in the DTD, we need to use on the XML 
 +Mar 16 22:52:16 giallu of course, that's just a validation issue 
 +Mar 16 22:52:30 giallu and the data import would be perfectly fine with any ordering 
 +Mar 16 22:53:37 vboctor In my opinion, the validation should ideally be a step that we execute before importing start.  This is to avoid failing mid way during import.  Specially given that we don't use transactions.  Unless we are planning to use transactions. 
 +Mar 16 22:53:51 giallu so, for instance, I could ksort the array from filters and be done with this 
 +Mar 16 22:54:34 vboctor I guess ordering is a not a big deal. 
 +Mar 16 22:54:42 giallu yap. let's move on 
 +Mar 16 22:55:05 vboctor I think the best option is to have a map which defines the name mapping / ordering, but it is not a big deal. 
 +Mar 16 22:55:48 giallu so let's quicly summarize your comments: 
 +Mar 16 22:55:56 giallu 1. The export has a lot of ids. These ids may mean different things to different people working with the data. I had the same problem when defining the SOAP API interface. What I ended up doing is to use an object reference concept. Hence, rather than just using an id, I use an id and a non-localized name to refer to an object. This applies to enumerations, users, etc. 
 +Mar 16 22:56:42 thraxisp A map might also help for foreign import where fields are missing. 
 +Mar 16 22:56:58 giallu I guess I can use the same system as used in SOAP. I'll add that as needed though. is that OK? 
 +Mar 16 22:58:47 vboctor what do you mean by as needed? 
 +Mar 16 22:59:06 * nt (n=tuckern@spc1-pete6-0-0-cust391.asfd.broadband.ntl.com) has joined #mantishelp 
 +Mar 16 22:59:08 vboctor What I think we should do now is to have <reporter><id>1</id><name>administrator</name></reporter> 
 +Mar 16 22:59:34 vboctor Same for view status: <view_state><id>10</id><name>public</name></view_state> 
 +Mar 16 23:00:11 vboctor The other option is to use <view_state_id>10</view_state_id> and then have a section in the XML that defines all the possible view states. 
 +Mar 16 23:00:38 giallu it means I could assume in a first instance that we are talking about two identical installations, so states are the same 
 +Mar 16 23:01:05 vboctor But you can't assume that for users, projects, etc. 
 +Mar 16 23:01:07 giallu then we can expand to your proposal in order to support some euristic (is that the word?) 
 +Mar 16 23:01:50 vboctor how are you planning to handle users, projects, etc? 
 +Mar 16 23:01:51 giallu vboctor, no. the users are to be handled like drafted in the reqiurement page  
 +Mar 16 23:01:56 giallu i.e. 
 +Mar 16 23:02:49 * vboctor checked the wiki page. 
 +Mar 16 23:02:53 giallu define a squash-to userid during import and ask admin if everything has to be squashed to that user or just the not matching ids 
 +Mar 16 23:02:58 vboctor your sample xml didn't include user names. 
 +Mar 16 23:03:42 giallu for projects. I think I'm limiting the import to a _single_ project 
 +Mar 16 23:04:12 giallu I don't really want to guess which project an issue belongs to 
 +Mar 16 23:04:34 giallu vboctor, which names? 
 +Mar 16 23:04:58 giallu reporters? 
 +Mar 16 23:05:14 vboctor I just think having <project><id>5</id><name>Mantis</name></project> will allow us to match by name even if ids don't match. 
 +Mar 16 23:05:29 vboctor by user names, i meant reporters, handlers, etc. 
 +Mar 16 23:05:50 giallu so basically, everywhere we have an id... 
 +Mar 16 23:06:08 CIA-11 mantisbt: thraxisp * r5121 /branches/BRANCH_1_1_0/mantisbt/adm_config_report.php: 
 +Mar 16 23:06:08 CIA-11 mantisbt: partial fix for #0008976: Remote Code Execution in adm_config 
 +Mar 16 23:06:08 CIA-11 mantisbt:  - hide update form for those who can't change items 
 +Mar 16 23:08:36 vboctor giallu, yes.  I think it will make the format more flexible and we won't have to make breaking changes. 
 +Mar 16 23:09:24 giallu vboctor, well you know, the X in XML means we shouldn't have this problem ;) 
 +Mar 16 23:10:38 vboctor just end up with more code to maintain for no strong reason. 
 +Mar 16 23:11:02 vboctor now we need to handle <reporter_id> and <reporter> 
 +Mar 16 23:11:34 vboctor do you think it is a lot more work to export id/name combinations? 
 +Mar 16 23:11:58 vboctor or is the logic to be intelligent about such combinations on import is what is worrying you? 
 +Mar 16 23:12:36 giallu yeah. it's just more work in the import side (which is still a bit foggy in my head) 
 +Mar 16 23:12:57 giallu I think I can do that 
 +Mar 16 23:13:06 giallu just one more Q 
 +Mar 16 23:13:33 giallu do you think your 4 years old note about using DOM XML still apply? 
 +Mar 16 23:14:02 giallu if possible, I'd rather use some existing php module for the job 
 +Mar 16 23:14:27 giallu though right now I'm writing it with custom code 
 +Mar 16 23:14:35 CIA-11 mantisbt: thraxisp * r5122 /trunk/mantisbt/adm_config_report.php: 
 +Mar 16 23:14:35 CIA-11 mantisbt: partial fix for #0008980: Port: Remote Code Execution in adm_config 
 +Mar 16 23:14:35 CIA-11 mantisbt:  - hide change form from unauthorized users 
 +Mar 16 23:15:17 vboctor custom code is fine. 
 +Mar 16 23:16:11 giallu yes. but I'm asking if it's fine to use some xml related functionality. The parsing is trickier 
 +Mar 16 23:17:50 vboctor I don't understand your question, probably since I don't remember the comment you are referring to. 
 +Mar 16 23:17:58 * vboctor checking... 
 +Mar 16 23:18:13 giallu ah. you asked to _not_ use DOM XML 
 +Mar 16 23:18:34 giallu becasue it was possibly not available everywhere 
 +Mar 16 23:19:42 vboctor yep, just read it now. 
 +Mar 16 23:19:48 vboctor what is the status of DOM XML and PHP 5? 
 +Mar 16 23:19:59 vboctor I remeber there was also Simple XML in PHP 5? 
 +Mar 16 23:20:36 giallu right 
 +Mar 16 23:21:02 vboctor Given that export / import is not a core feature, I am ok with using a PHP extension. 
 +Mar 16 23:21:09 giallu great 
 +Mar 16 23:21:17 vboctor Similar to twitter notifications that is dependent on curl extension. 
 +Mar 16 23:21:37 giallu I'm probably leaving the exporter as is unless I find it unmaintenable 
 +Mar 16 23:22:21 giallu but I'm definitely looking for some cooked solution about import 
 +Mar 16 23:22:43 vboctor consistency between export / import would be ideal. 
 +Mar 16 23:23:06 vboctor you also need to be careful that PHP doesn't run out of memory when exporting / importing lot of issues
 +Mar 16 23:23:47 giallu ok. I'm noting this on the wiki page 
 +Mar 16 23:33:20 vboctor good 
 +</file>
mantisbt/importexport.1206012169.txt.gz · Last modified: 2008/10/29 04:31 (external edit)

Driven by DokuWiki