View Issue Details

IDProjectCategoryView StatusLast Update
0014694mantisbtmigrationpublic2013-11-22 11:02
Reporteramby-adm Assigned Todregad  
PrioritynormalSeveritymajorReproducibilityhave not tried
Status closedResolutionduplicate 
Summary0014694: Cannot import xml/csv files
Description

Hi

I have read the posts regarding the importing xml files in Mantis BT, but none of the solution matches the issue. The error I am getting is:

SYSTEM WARNING: 'XMLReader::read() [xmlreader.read]: file:///C:/Windows/Temp/php2425.tmp:4: parser error : Extra content at the end of the document' in 'C:\websites\mantisbt\plugins\XmlImportExport\ImportXml.php' line 77

SYSTEM WARNING: 'XMLReader::read() [xmlreader.read]: <date>2007-06-01 09:09</date>' in 'C:\websites\mantisbt\plugins\XmlImportExport\ImportXml.php' line 77

SYSTEM WARNING: 'XMLReader::read() [xmlreader.read]: ^' in 'C:\websites\mantisbt\plugins\XmlImportExport\ImportXml.php' line 77

SYSTEM WARNING: 'XMLReader::read() [xmlreader.read]: An Error Occured while reading' in 'C:\websites\mantisbt\plugins\XmlImportExport\ImportXml.php' line 77

Importing file, please wait...
SYSTEM WARNING: 'XMLReader::read() [xmlreader.read]: An Error Occured while reading' in 'C:\websites\mantisbt\plugins\XmlImportExport\ImportXml.php' line 88

I read the file and the code is as per normal.
I need to get issues from ITSM and transfer to Mantis urgently. Please help!

TagsNo tags attached.
Attached Files
mantis_sample_import.xml (1,558 bytes)   
<?xml version="1.0" encoding="utf-8" ?>

  <author>administrator</author>
  <date>2007-06-01 09:09</date>
  <bugs>
   <bug>
	 <additional_information></additional_information>
     <date_submitted>29/05/2005</date_submitted>
     <custom name="Date">18/06/2007</custom>
     <bugnotes>
       <bugnote>
         <id>0</id>
         <reporter_id>1</reporter_id>
         <note>Feedback 1</note>
         <view_state>10</view_state>
         <date_submitted>01/01/2007</date_submitted>
         <last_modified>01/01/2007</last_modified>
       </bugnote>
     </bugnotes>
     <project_id>1</project_id>
     <category>Category 1</category>
     <reporter_id>1</reporter_id>
     <status>50</status>
	 <summary>Test reminder with deadline empty</summary>
     <description>Test</description>
   </bug>
   
   <bug>
	 <additional_information></additional_information>
     <date_submitted>29/06/2009</date_submitted>
     <custom name="Date">18/05/2009</custom>
     <bugnotes>
       <bugnote>
         <id>0</id>
         <reporter_id>2</reporter_id>
         <note>Feedback 2</note>
         <view_state>10</view_state>
         <date_submitted>01/06/2009</date_submitted>
         <last_modified>01/04/2009</last_modified>
       </bugnote>
     </bugnotes>
     <project_id>2</project_id>
     <category>Category 2</category>
     <reporter_id>2</reporter_id>
     <status>50</status>
	 <summary>Test reminder with deadline empty also</summary>
     <description>Test</description>
   </bug>
   
  </bugs>
mantis_sample_import.xml (1,558 bytes)   
ImportXml.php (6,794 bytes)   
<?php
# MantisBT - A PHP based bugtracking system
# Copyright (C) 2002 - 2011  MantisBT Team - mantisbt-dev@lists.sourceforge.net
# MantisBT is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 2 of the License, or
# (at your option) any later version.
#
# MantisBT is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with MantisBT.  If not, see <http://www.gnu.org/licenses/>.

require_once( 'ImportXml' . DIRECTORY_SEPARATOR . 'Mapper.php' ); //directory_separator == \ 
require_once( 'ImportXml' . DIRECTORY_SEPARATOR . 'Issue.php' );

class SourceData { //created a class
	public $version;
	public $urlbase;
	public $issuelink;
	public $notelink;
	public $format;

	public function get_issue_url( $issue_id ) {
		return $this->urlbase . 'view.php?id=' . $issue_id; //concanation
	}

	public function get_note_url( $issue_id, $note_id ) {
		return $this->urlbase . 'view.php?id=' . $issue_id . '#c' . $note_id;
	}
}

/**
  * Perform import from an XML file
  */
class ImportXML { //new class defined
	private $source_; // can only be accessed by the class ImportXML
	private $reader_; //initial value they get is NULL if you don't define any values
	private $itemsMap_;
	private $strategy_;
	private $fallback_;

	// issues specific options
	private $keepCategory_;
	private $defaultCategory_;

	/**
	  * Constructor
	  *
	  * @param string $filename name of the file to read
	  * @param string $strategy - conversion strategy; one of "renumber", "link" or "disable"
	  * @param string $fallback - alternative conversion strategy when "renumber" does not apply
	  */
	public function __construct( $filename, $strategy, $fallback, $keepCategory, $defaultCategory ) {
		$this->source_ = new SourceData;
		$this->reader_ = new XMLReader( ); //XMLReader is a class defined in php, so _reader is an object of the class 
		$this->itemsMap_ = new ImportXml_Mapper; //class defined in Mapper.php
		$this->strategy_ = $strategy;
		$this->fallback_ = $fallback;
		$this->keepCategory_ = $keepCategory;
		$this->defaultCategory_ = $defaultCategory;

		$this->reader_->open( $filename['tmp_name'] );
	}

	/**
	 * Perform import from an XML file
 *
	 * @param string $p_filename name of the file to read
	 * @param string $p_strategy conversion strategy; one of "renumber", "link" or "disable"
	 */
	public function import( ) {
		// Read the <mantis> element and it's attributes
		while( $this->reader_->read( ) && $this->reader_->name == 'mantis' ) {
			$this->source_->version = $this->reader_->getAttribute( 'version' );
			$this->source_->urlbase = $this->reader_->getAttribute( 'urlbase' );
			$this->source_->issuelink = $this->reader_->getAttribute( 'issuelink' );
			$this->source_->notelink = $this->reader_->getAttribute( 'notelink' );
			$this->source_->format = $this->reader_->getAttribute( 'format' );
		}

		echo 'Importing file, please wait...';

		// loop through the elements
		while( $this->reader_->read( ) ) {
			switch( $this->reader_->nodeType ) {
				case XMLReader::ELEMENT: //first element

					/* element start */
					$t_element_name = $this->reader_->localName;
					$t_importer = $this->get_importer_object( $t_element_name );
					if( !is_null( $t_importer ) ) {
						$t_importer->process( $this->reader_ );
						$t_importer->update_map( $this->itemsMap_ );
					}
					break;
			}
		}

		echo " Done\n";

		// replace references in bug description and additional information
		$importedIssues = $this->itemsMap_->getall( 'issue' );
		printf( "Processing cross-references for %s issues...", count( $importedIssues ) );
		foreach( $importedIssues as $oldId => $newId ) {
			$bugData = bug_get( $newId, true );

			$bugLinkRegexp = '/(^|[^\w])(' . preg_quote( $this->source_->issuelink, '/' ) . ')(\d+)\b/e';
			// replace links in description
			preg_match_all( $bugLinkRegexp, $bugData->description, $matches );
			if ( is_array( $matches[3] && count( $matches[3] ) > 0 ) ) {
				$content_replaced = true;
				foreach ( $matches[3] as $old_id ) {
					$bugData->description = str_replace( $this->source_->issuelink . $old_id, $this->getReplacementString( $this->source_->issuelink, $old_id ), $bugData->description);
				}
			}
			// replace links in additional information
			preg_match_all( $bugLinkRegexp, $bugData->additional_information, $matches );
			if ( is_array( $matches[3] && count( $matches[3] ) > 0 ) ) {
				$content_replaced = true;
				foreach ( $matches[3] as $old_id ) {
					$bugData->additional_information = str_replace( $this->source_->issuelink . $old_id, $this->getReplacementString( $this->source_->issuelink, $old_id ), $bugData->additional_information);
				}
			}
			if ( $content_replaced ) {
				// only update bug if necessary (otherwise last update date would be unnecessarily overwritten)
				$bugData->update( true );
			}
		}

		// @todo: replace references within bugnotes
		echo " Done\n";
	}

	/**
	 * Compute and return the new link
	 *
	 */
	private function getReplacementString( $oldLinkTag, $oldId ) {
		$linkTag = config_get( 'bug_link_tag' );

		$replacement = '';
		switch( $this->strategy_ ) {
			case 'link':
				$replacement = $this->source_->get_issue_url( $oldId );
				break;

			case 'disable':
				$replacement = htmlFullEntities( $oldLinkTag ) . $oldId;
				break;

			case 'renumber':
				if( $this->itemsMap_->exists( 'issue', $oldId ) ) {
					// regular renumber
					$replacement = $linkTag . $this->itemsMap_->getNewID( 'issue', $oldId );
				} else {
					// fallback strategy
					if( $this->fallback_ == 'link' ) {
						$replacement = $this->source_->get_issue_url( $oldId );
					}
					if( $this->fallback_ == 'disable' ) {
						$replacement = htmlFullEntities( $oldLinkTag ) . $oldId;
					}
				}
				break;

			default:
				echo "Unknown method";
		}

		//echo "$oldId -> $replacement\n"; // DEBUG
		return $replacement;
	}

	private function get_importer_object( $p_element_name ) {
		$importer = null;
		switch( $p_element_name ) {
			case 'issue':
				$importer = new ImportXml_Issue( $this->keepCategory_, $this->defaultCategory_ );
				break;
		}
		return $importer;
	}
}

/** candidates for string api **/

/**
 * Convert each character of the passed string to the
 * corresponding HTML entity.
 */
function htmlFullEntities( $string ) {
	$chars = str_split( $string );
	$escaped = array_map( 'getEntity', $chars );
	return implode( '', $escaped );
}

function getEntity( $char ) {
	return '&#' . ord( $char ) . ';';
}
ImportXml.php (6,794 bytes)   
mantis_working.xml (975 bytes)   
<?xml version="1.0" encoding="utf-8" ?>
<mantis>
  <issue>
      <id></id>
      <project_id>9</project_id>
      <reporter_id>13</reporter_id>
      <handler_id></handler_id>
      <priority_id>30</priority_id>
      <severity_id>50</severity_id>
      <reproducibility_id>70</reproducibility_id>
      <status_id>10</status_id>
      <resolution_id>10</resolution_id>
      <projection_id>10</projection_id>
      <category_id>85</category_id>
      <date_submitted>1287472975</date_submitted>
      <last_updated>1287472975</last_updated>
      <eta_id>10</eta_id>
      <os></os>
      <os_build></os_build>
      <platform></platform>
      <view_state_id>10</view_state_id>
      <summary>Tester 5</summary>
      <due_date>1</due_date>
      <profile_id>0</profile_id>
      <description>DESCRIPTION</description>
      <steps_to_reproduce></steps_to_reproduce>
      <additional_information></additional_information>
    </issue>
</mantis>
mantis_working.xml (975 bytes)   
mantis_working_005.xml (1,277 bytes)   
<?xml version="1.0" encoding="utf-8" ?>
<mantis>
  <issue>
	 <id></id>
     <additional_information>INC700016793199</additional_information>
	 <incident_number>UP98</incident_number>
	 <CustomerReference>To remove the wording "UAT Server" from the banner</CustomerReference>
	  <incident_number>UP98</incident_number>
      <project_id>8</project_id>
      <reporter_id>12</reporter_id>
      <handler_id></handler_id>
      <priority_id>30</priority_id>
      <severity_id>50</severity_id>
      <reproducibility_id>70</reproducibility_id>
      <status_id>10</status_id>
      <resolution_id>10</resolution_id>
      <projection_id>10</projection_id>
      <category_id>84</category_id>
      <date_submitted>1287472974</date_submitted>
      <last_updated>1287472974</last_updated>
      <eta_id>10</eta_id>
      <os></os>
      <os_build></os_build>
      <platform></platform>
      <view_state_id>10</view_state_id>
      <summary>Tester with more info after adding 2 custom fields and patching</summary>
      <due_date>1</due_date>
      <profile_id>0</profile_id>
      <description>DESCRIPTION</description>
      <steps_to_reproduce></steps_to_reproduce>
      <additional_information></additional_information>
    </issue>
</mantis>
mantis_working_005.xml (1,277 bytes)   

Relationships

duplicate of 0012013 closeddregad mantisbt Improvements for plugin ImportExportXml (and required core changes) 
related to 0014713 closedvboctor Plugin - CsvImport Cannot import custom fields in the csv files 

Activities

amby-adm

amby-adm

2012-09-07 02:46

reporter   ~0032808

In the end it just says - 0 issues imported

dregad

dregad

2012-09-12 12:20

developer   ~0032825

The problem is not reproducible with the information you provided.
Please be more specific (mantis version, platform, etc), and provide a sample import file

amby-adm

amby-adm

2012-09-16 21:37

reporter   ~0032849

Hi

The mantis info:

MantisBT Version - 1.2.10
Schema Version-183

Attaching the sample import file above. Thanks a lot!

amby-adm

amby-adm

2012-09-16 22:58

reporter   ~0032850

The file ImportXml.php which shows error is also uploaded. Please help!

amby-adm

amby-adm

2012-09-16 23:41

reporter   ~0032851

I changed the name 'temp_name' to 'mantis' in the file ImportXml.php, and the file being imported is also 'mantis.xml'. However, it shows:

SYSTEM WARNING: 'XMLReader::open() [xmlreader.open]: Empty string supplied as input' in 'C:\websites\mantisbt\plugins\XmlImportExport\ImportXml.php' line 66

SYSTEM WARNING: 'XMLReader::read() [xmlreader.read]: Load Data before trying to read' in 'C:\websites\mantisbt\plugins\XmlImportExport\ImportXml.php' line 78

Importing file, please wait...
SYSTEM WARNING: 'XMLReader::read() [xmlreader.read]: Load Data before trying to read' in 'C:\websites\mantisbt\plugins\XmlImportExport\ImportXml.php' line 89

Done
Processing cross-references for 0 issues... Done

Please help!

dregad

dregad

2012-09-18 06:25

developer   ~0032858

Had a quick look at the uploaded XML file, seems to be a different format from that generated when exporting with "XML Export" link in view issues. This is what I get.

<?xml version="1.0" encoding="UTF-8"?>
<mantis version="1.2.12-dev" urlbase="http://localhost/mantis/dev/" issuelink="#" notelink="~" format="1">
<issue>
<id>157</id>
<project id="1">Test Public Project</project>
<reporter id="25">Test Administrator</reporter>
<priority id="30">normal</priority>
<severity id="50">minor</severity>
<reproducibility id="70">have not tried</reproducibility>
<status id="10">new</status>
<resolution id="10">open</resolution>
<projection id="10">none</projection>
<category id="1">General</category>
<date_submitted>1344241932</date_submitted>
<last_updated>1347533321</last_updated>
<eta id="10">none</eta>
<view_state id="10">public</view_state>
<summary>tes 14549</summary>
<due_date>1</due_date>
<description>xxx</description>
</issue>

Although I do get some errors with the above too, but I don't have enough time to investigate in details atm.

amby-adm

amby-adm

2012-09-18 06:29

reporter   ~0032859

Hi i just uploaded the working file. This updates, but the custom fields do not get updated.

amby-adm

amby-adm

2012-09-18 06:30

reporter   ~0032860

The mantis_working_005 is a sample, the additional_information and incident_number do not get updated. What do I do?

dregad

dregad

2012-09-18 08:46

developer   ~0032862

After a bit of research it appears that the custom fields import is a known issue, which has been fixed in the "master" branch (still under development).

You may decide to use master instead of 1.2.10, or to try and back-port dhx's commits to your version.

amby-adm

amby-adm

2012-09-19 02:42

reporter   ~0032874

Ok I will try doing that. Thanks a lot!

dregad

dregad

2012-09-19 04:21

developer   ~0032876

If you decide to backport the 1.3 changes, please report what you did here (preferably as a github pull request, git patch or unified diff).