Content importing & migration

[ahathaway]

This is a fairly massive issue for those of us with pre-existing content in other wikis or CMS, or who wish to convert others from wiki X to DekiWiki. I would also argue that Deki's adoption rate and success will be determined in no small part by the ease with which wiki content can be imported (to allow wiki fans to easily migrate to DekiWiki).

I have a fairly large TWiki wiki, and at present, I'm reduced to cutting and pasting HTML from TWiki to DekiWiki. New lines get added, links have to be manually removed and recreated one-by-one, and files have to be uploaded. Needless to say, I've had more fun watching paint dry...and then crack and flake off with age. Of course, the revision history is lost, but that's a complicated issue that I don't expect DekiWiki to be able to do much about (TWiki uses RCS for revision tracking). But there must--or should--be an easier way.

I know you all offer services to help migrate from MediaWiki to DekiWiki. If you were willing to expose that functionality, that alone would be a major help (I believe there are ways to translate TWiki to MediaWiki markup). Ultimately, though, any means of importing multiple HTML pages is desperately needed, it seems to me.

Anything that smooths and accelerates the migration path helps MindTouch dig not only into wiki-less organizations, but also allows wiki-using organizations to start reaping the benefits of DekiWiki with minor hassle, making DekiWiki a much easier sell.

[AaronF]

I agree Aaron. BTW, you have a great name. Unfortunately it's a matter of available cycles. We simply haven't had them to provide a DekiWiki importer. Also, in the next month there are going to be some significant changes to DekiWiki that will make it even more divergent from MediaWiki, db schemas, etc. In about a month DekiWiki will be able to import an HTML page. What we'll do is offer code bounties for importers at that time.

The community largely dictates what we spend cycles on; so, the more people pester us about this the higher value we'll place on it. Thanks for your interest Aaron because this is something I want us to do too, but unless I have droves of folks pestering us about it it's difficult for me to justify us spending cycles on it. Anyway, let's revisit this in a month or so and hopefully by then we'll have loads of people asking about it here.

[ahathaway]

As my cut-and-paste fingers grow ever wearier, I figured I'd bring this issue up again. I suspect that this is a feature that more and more customers will require.

Confluence is offering an open-sourced import tool (Universal Wiki Converter (http://confluence.atlassian.com/display/CONFEXT/UWC+User+Documentation)) for converting a wide array (ex: MediaWiki, JotSpot, MoinMoin, TWiki, Dokuwiki, TikiWiki, etc.) of wiki content to Confluence content. They have made the source available here (http://svn.atlassian.com/svn/public/contrib/confluence/universal-wiki-converter), and while I couldn't find mention of a specific license (neither on the site nor in the code), their documentation suggests that this is reusable code.

Their tool is java-based, but might it not be the worst place to start.

One huge caveat: this depends, on a fully functional REST API in DekiWiki, so this may be a down-the-road project.

[ahathaway]

Here's another tool that could be extended (and that doesn't come from the competition):


WikiAPI - The generic Java Wiki API (http://jwikiapi.sourceforge.net/)

"The WikiAPI provides a standard java api for accessing wikis from a Java Application. The WikiAPI currenly includes support for MediaWiki and TWiki, however there are more to come.

Along with the library are various utilities which demonstrate or facilitate its usage. Including a java console app to copy documents from one wiki to another."

There's an included example class (http://jwikiapi.svn.sourceforge.net/viewvc/jwikiapi/src/WikiSync.java?revision=37&view=markup) for syncing TWiki to MediaWiki that could be extended or altered for DekiWiki.

[bstrackany]

FYI there's an importer.php file in the maintenance/ directory that works. I was able to dump a bunch of content from an Excel spreadsheet into individual .wiki files & use importer.php to load it into DekiWiki. HTH.

[mads]

Would be very nice if you could import content from other wiki's

[AaronF]

Converting from other wikis to DekiWiki has become a frequently requested feature. I was hoping someone in the community would put together a tool for this. I thought someone had written something for Mediawiki. MindTouch would even offer a bounty for this if someone were interested in implementing it. Any takers?

[sjb]

What happened to the importer.php tool mentioned by bstrackany. I don't see it in hayes?

Is there any way to just upload a raw HTML page using a script or even from some php page?

[bstrackany]

What happened to the importer.php tool mentioned by bstrackany. I don't see it in hayes?

Is there any way to just upload a raw HTML page using a script or even from some php page?

Ah, that would stink if it disappeared ... I use that thing all the time for our massive wiki.

I could post or host it somewhere but ofc I don't know if it'll work at all with Hayes.

[SteveB]

I'm guessing here, but it might not be too hard to adapt it since the DB format for pages has only changed slightly and the content formatting of Gooseberry is compatible with Hayes. We actually don't upgrade the contents at all until a page is edited again.

[scott.harman]

I'd love to see an importer from mediawiki (or any of the others)
We've come to the conclusion that MediaWiki is fine for Developers and tech-savvy engineers, however for regular helpdesk and overseas support staff it's all a bit bewildering -
I've got about 30mb worth of debugging information and configuration guides (that's just raw text) which I'd quite like to poke into my dekiwiki test installation to see if it flies before I show it to our IT department

Ideally we'd have importers for pdf manuals, open office, and rtf - i think it's somethink my management would happily pay for

[SteveB]

There is quite a bit of interest for doing a mediawiki importer. I started collecting some preliminary requirements on a wiki page:
http://wiki.opengarden.org/User:SteveB/Converting_Mediawiki_to_Deki_Wiki

Unfortunately, my work plate is a bit too full at this time to start hacking on it.

[scott.harman]

Bleurgh
Trying to write some xsl to convert the Wiki export to valid sql to dump into the db. slight problem is half our work is based on xml extensions - so that buggers things up somewhat
i've got a headache and think i might go back to working directly with a mysql dump of my mw wiki
it's messy and 'orrible when you've got xml enscapsulated within xml. there's a lesson in there somewhere.
going back to the 'hole' cover of 'hungry like the wolf' ;)

[SteveB]

Scott,

If you figure out a way to import content, please let us know. There seems to be a lot of interest in this. If we can in any way to make it easier, please suggest away. Thanks!

[scott.harman]

doing loads of nifty sql, will pick a fixed date/timestamp to use, as working them out is crushing my skull
I'm trying to work out the easiest way to extract all of the categories, but suspect this will be easier to do manually
however would be nice just to get each of the primary root categories from the sql
about to test with my first article on a fresh vmware image to see how it copes with native wikitext (suspect it will fail to render)

[scott.harman]

messy, messy, messy.
i've resorted to vanilla sql, and am doing it with about 8 seperate commands.
now need to convert from wikitext to html.

the below is incorrect, but close enough for the purposes:
create table new_pages select page_id, 0 as page_namespace, page_title, 0 as page_text, 0 as page_comment, 0 as page_user_id, '' page_timestamp,page_counter, page_is_redirect, 0, page_is_new, page_touched, 99999999999999-page_touched as page_inverse_timestamp, 0 as page_use_cache, '' as page_toc, 0 as page_tip, 0 as page_parent, '' as page_restriction_id, 'application/x.deki-text' as page_content_type from qw_page where page_namespace = 0 order by page_id;

missing page random etc... unfortunately I've only got a console history of 50 commands, so it's a bit messy and painful.

fundamentally, take qw_page into a new table, with generic values for comments and text. assign a transition user as the owner of all imported material.

create a table consisting of the latest revision for your articles:
create table new_text select rev_page, max(rev_text_id) from qw_revision group by rev_page;
alter table new_text change `max(rev_text_id)` rev_text_id int(8) unsigned;
alter table new_text add page_text mediumtext;

update new_text left join qw_text on rev_text_id = old_id set page_text = old_text;

update new_pages left join new_text on page_id = rev_page set new_pages.page_text = new_text.page_text where page_id is not null;

there's a fair amount of cleaning to do, and moving to an html based format is a bit of an arse, but never mind.
just need an automated way to clean the wiki text if there are any suggestions.
at the moment it's manual. digging out multiple categories is a complete waste of time - I'll move to a totally heirarchical system - i.e. choose the first listed category, but move things manually around the system, too complicated to automate yet.

[scott.harman]

answer me this:
<code>
(1431,0,'AAF','[[Category:General]] <br /> [[Category:databases]] <br /> [[Category:EDL]] <br /> [[Category:Video editing]] <br /> \'\'\'The Advanced Authoring Format\'\'\' <br /> <br /> The Advanced Authoring Format ([[AAF]]) is a multimedia file format that enables content creators to easily exchange digital media and metadata across platforms, and between systems and applications. The [[AAF]] simplifies project management, saves time, and preserves valuable metadata that was often lost when transferring media between applications in the past. <br /> <br /> AAF and MXF - A complementary pair <br /> The AAF Association, along with the Pro-MPEG Forum and the SMPTE, is also a co-creator of [[MXF]]. [[AAF]] and [[MXF]] combine to offer a well-rounded, complementary solution to the complex problems involved in the interchange and movement of media throughout the post production and distribution processes. AAF is primarily intended for post production interchange and supports external content references and downstream processing (such as effect, fades, etc.); whereas, [[MXF]] is primarily intended for storage, broadcast and play-out interchange of complete programs. <br /> <br /> For more information: <br /> <br /> [The AAF association] <br /> http://www.aafassociation.org/index.html','',3,'20061010153111',68,0,0,0,0,'2006 1010153111','79938989846888',0,'','0',0,NULL,'appl ication/x.deki0702+xml'),</code>

here's my sql - will write some regex in perl to fix the bits, but it won't render when I dump this entry into the pages table.
any ideas why?
I've changed all carriage returns to <br /> entries - what else am I missing construction wise?

[SteveB]

Is the page just not showing up? Are there any error messages in the API logs?

Also, the content type should be 'application/x.deki-text' otherwise the [[link]] notation won't be converted to actual links.

[scott.harman]

changed it to x.desk as suggested so it's parsing the wikilinks correctly (ish - they won't exist yet and will need to be changed)
no errors in the logs - page is showing, but it's just not interpreting the html I've written in.

i.e. the <br /> tags and the font tags are not rendering at all - they're being displayed as regular text data

I've compared an sql dump of some pages I've created with similar formatting - is it possible that it's only being rendered if within <p> tags?
As the pages I've created should legitimately be rendered correctly as html, as simply dropping the text into an html file with body tags renders as I'd expect

[SteveB]

No, the <p> tags are not mandatory. The text is simply loaded into a <html><body>TEXT HERE</body></html> frame for rendering.

Is an error message displayed when the contents are shown? What does it look like in the editor?

To clarify, the <br /> elements are converted into &lt;br /&gt;?

[scott.harman]

no, in my db export they're not encoded as such - will give it a shot - either way it's an easy thing to check/replace

once i've got this working i'll see if i can script it.
i've got a perl script that should work, but i need to unstitch the raw data from the sql, and can't think of an elegant way to do this.
the perl script i've got (wikitext-convert) won't alter what i expect inline so I'll have to fix that.

cheers for that

scott

[NilsLien]

Hi.

I see this post has been quiet for a long time.

I have a suggestion, or rather a question, in connection with exporting content from TWiki and importing it to DekiWiki.

In TWiki there is a plugin called PublishContrib.
http://twiki.org/cgi-bin/view/Plugins/PublishContrib

This can export TWiki pages to a chosen format; currently HTML, pdf or an archive zip/tgz.

And in DekiWiki I have the understanding that it is possible to create pages via the API using the POST:contents function. Especially useful when using the curl tool, like one example I found here:
http://forums.opengarden.org/showthread.php?t=1061&page=3#29
It seems to be handling import of XHTML files, judging by the original poster.

Could it be as simple as just exporting TWiki pages to HTML and importing each file to DekiWiki with curl, using the Deki API?

Of course you would have to recreate the TWiki topic hierarchy of each Twiki Web, but that shouldn't be too difficult, as the file structure in TWiki is already organized hierarchically.

[SteveB]

It could be that simple. Note that if you used a slight more powerful language (e.g. Java/C#), you could probably achieve this more easily than via curl. The API works with simple HTTP requests, which are available in almost any language.

[NilsLien]

Yes, I was thinking of using perl, but still use curl to do the job.

If I recreate the TWiki hierarchy structure first and saving it to a hash or a file or whatever, I could then loop over it and use the libcurl library in perl to POST each TWiki HTML file to DekiWiki with the same structure.

But this is an effort I don't have the opportunity to do right away. Fortunately we don't have so many wiki pages in TWiki yet, so I'm just going to do the old cut&paste from TWiki to DekiWiki for now. That works like a charm, even keeping the formatting and styles, thanks to the good WYSIWYG editor in DekiWiki.

[sharonn]

Let me share with you my concerns for adopting a Wiki for my corporate.
Besides addressing the day-to-day management issues such as backups, there's the longer-term issue of what if development stops? How will I migrate FROM Deki Wiki?
Is it a thing I need to factor in when considering a Wiki for my organization? I think it is.
Think of it like a shop's return policy. It's on the store's best interest to have one. Since (and I'm making an educated guess here) wiki adoption in corporate is still at its infancy, providing such means makes the decision to adopt one WIki over another much easier.

[SteveB]

sharonn,

You're most definitively right worrying about data lock-in. This is where open source software, such as deki wiki, has already a leg up, because YOU have access to the source so YOU can be in control of the data.

But, I still have a better answer for you. Deki Wiki stores all your data in HTML, not in wiki text. First, wiki text is not standard and it's gimped (and should die a horrible death, but I digress). Second, because the rest of the world already works in HTML, you can easily migrate your data out of the wiki again. And still, it gets better! Since Deki Wiki has a fully API, you won't have to do SQL spelunking either! Instead, you can sit back and use almost any scripting/programming language for this task. But chances are that someone has already done so and would be willing to share their work. Another benefit of a healthy and growing community. :)

[scott.harman]

Hi guys,
I've been battering away at my data (due to loads of embedded sql scripts and xml within my existing pages i'm using vanilla sql dumps instead)
I'm most of the way there - but Lucene refuses to index certain pages.

What parameters would I need to set to allow lucene to index the pages on import?

example:


INSERT INTO `pages` (`page_id`, `page_namespace`, `page_title`, `page_text`, `page_comment`, `page_user_id`, `page_timestamp`, `page_counter`, `page_is_redirect`, `page_minor_edit`, `page_is_new`, `page_touched`, `page_inverse_timestamp`, `page_usecache`, `page_toc`, `page_tip`, `page_parent`, `page_restriction_id`, `page_content_type`, `page_random`) VALUES

(1430,0,'AAF_Path','Category: Tips<p>&nbsp;</p><p>&nbsp;</p><h2> <em>AAF 3.0 and 3.5 software.</em> </h2><p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p>3.0 software stores the AAF\'s in a network available location. The location of this is stored in the seatproperties table. Each seat will run up and read this location so it knows where to go to read\\write AAFs<p>&nbsp;</p><p>&nbsp;</p>3.5 software stores the AAF information in the MySQL database. For the seat to read this it needs to extract this information from the database then build a temp AAF locally. The location for the temporary AAF is set in the QIO registry. This should be left as the default which is c:\\data\\user\\aaffiles<p>&nbsp;</p><p>&nbsp;</p>We need to keep an eye on this location as it maybe possible for temp aafs not to be tidied up.<p>&nbsp;</p><p>&nbsp;</p>A thing to watch: When a 3.5 seat loads a clip will look for AAF data in the ISA manager first, and if it is not found then it will examine the filepath. This has the potential to cause problems when running an upgraded/mixed system as old AAF files hanging about can be applied to new clips that don\'t have AAF files and that can cause the seat to hang. Test this by looking at a browse pic of the clip - this doesn\'t use the AAF data.<p>&nbsp;</p><p>&nbsp;</p>Also:<p>&nbsp;</p><p>&nbsp;</p>----<p>&nbsp;</p><p>&nbsp;</p>*<em>BUG:If the path you enter in the AAF-Path isn\'t available to a 3.5 seat, it will hang for about 30s (while Windows looks for the path) when you drag a clip out of the server bin.</em><p>&nbsp;</p><p>&nbsp;</p>*<em>BUG: Currently a 3.5 system will complain on boot up if there is no entry in the seatproperties table for the AAF path. On new systems you will need to put a null entry (\"\") in here until this is resolved. The seat will still boot but there will be a nasty message appear. See bug:</em><p>&nbsp;</p><p>&nbsp;</p>----<p>&nbsp;</p><p>&nbsp;</p>3.0 and 3.5 If upgrading from 3.0 to 3.5 or using a mix of seats, you will need to have both methods available to the seat. 3.5 will store all AAF\'s in the ISA database but it can all read older 3.0 AAF\'s from the network location. The seat will have to find this path from the seatproperties table.',0,0,'',72,0,0,0,'20061031172458',799389688 27541,0,'',0,0,'','application/x.deki0702+xml',0),



As you can probably see, I've fudged some values - does the page need a user_id and touched associated with it for initial import maybe?
I'd rather not use that as then I can logically separate the fresh imported data from the new data.

I might fiddle with that today.

[mystikweb]

Hi there, I am looking to see if anyone has had success in migrating a Twiki site into Deki, I would love to hear from you and discuss what you did, how long it took etc...

Surely someone has attempted this by now.....

[SteveB]

Did you try searching for "twiki" on these forums?

[mystikweb]

Yeah a few times, and found a few small things, but nothing really stood out as a way of doing it. Our version of Twiki os fairly old, so the plugin one of the users mentioned to export twiki content out will not work.

The only thing I could think of would be to use something like wget to recurseively save the pages in xhtml format as they are rendered on the client.

Then write a script to strip out the bits not required, map the structure to a file or xml document, then parse that and use the api to recurseively post the new documents.

But surely someone has done something like this already?? rather than having to re-invent the wheel?

[SteveB]

Hmm, no the page you found is the one I remembered as well. As far as I know, nobody has done an importer yet. Might be worthwhile starting a new thread on this to see if anyone has begun doing work on this.

[dashu]

Has any work been done on a tool or simple process to convert an html based web site to DekiWiki?

[SteveB]

Not to my knowledge. Although, web.html goes quite a way to do that for you.

For instance, consider this page:
http://wiki.opengarden.org/User:SteveB/Web.Html

It contains the home page of wikipedia using:

{{ web.html("http://en.wikipedia.org/wiki/Main_Page", "//_:body") }}

Now, if you did this instead:

{{SUBST: web.html("http://en.wikipedia.org/wiki/Main_Page", "//_:body") }}

It would substitute the script with it's result when save. Ergo, you just made a copy of the page!

Web.Html is available in "Jay Cooke". RC1 is out today!

[phillfri]

SteveB - I want to be able to do this with an html file residing on a network disk (e.g. a dynamic training document), but I can't seem to get web.html to resolve a network file name uri. Is this possible?

[hgorina]

Dear all,

In my humble opinion, one-way importer is not doing much long-term. When recommending DEKI to my company I really need to know that I will be able to extract the information and convert it to other WIKI if need be. I can see that DEKI has really excellent potential, and is user friendly, but WIKI concept in the production environment is being a KNOWLEDGE repository. My management is concerned (and, for once, I agree) that we might be locking our "know how" in "proprietary" format.

I think it would be really easier for us to adopt )and for me to sell it to management) if we knew that we have a "safety net" of being able to easily migrate not only IN but OUT.

[brigettek]

Since Deki functionality is exposed entirely through an API (http://wiki.developer.mindtouch.com/MindTouch_Deki/API_Reference), you can import and export your data as needed. For example, you can use GET: pages/{pageid}/contents (http://wiki.developer.mindtouch.com/Deki_Wiki/API_Reference/GET%3apages%2f%2f%7bpageid%7d%2f%2fcontents) to retrieve the page data in various formats, GET: pages/{pageid}/revisions (http://wiki.developer.mindtouch.com/Deki_Wiki/API_Reference/GET%3apages%2f%2f%7bpageid%7d%2f%2frevisions) to retrieve revisions, GET: pages/{pageid}/files (http://wiki.developer.mindtouch.com/Deki_Wiki/API_Reference/GET%3apages%2f%2f%7bpageid%7d%2f%2ffiles) to retrieve files attached to a page, etc. Because of this API, Deki has a lot more export potential than a lot of other products.

[vdubya]

This is not import per se, but filtering out useless MS Word tags would also go a long way towards users being able to add content via cutting and pasting. I am finding that a LOT of manual pruning must be done to the html source to get rid of a lot of junk. Can that be filtered when pasted?

[hgorina]

Saving as RTF before copy/paste retains formatting and strips out all junk
It worked for me

[vdubya]

Saving as RTF before copy/paste retains formatting and strips out all junk
It worked for me
Agree and have done that myself at times. Don't think my end users will be as patient/tolerant. What I am curious about is whether this could be developed as an optional paste method. I am finding a good percentage of contributions to my site are pasted...and pasted with junk.

[SteveB]

FCKeditor has a "Paste from Word" option that can be enabled. This will then show up a paste-window which will do the MS-attribute pruning.

[theillien]

Where does this stand? I've read through the thread and I'm not clear on whether a migration tool for various wikis will come to fruition. We use Tiki here at work. I might even be able to put a bug in peoples' ears to pay for the feature if necessary (no guarantees).

-Mathew

[SteveB]

There are conversion tools for MediaWiki (http://wiki.developer.mindtouch.com/index.php?title=MindTouch_Deki/FAQ/Migration/How_do_I...Migrate_from_MediaWiki_to_Deki_Wiki%3F) , MoinMoin (http://www.kleinfelter.com/moin-to-deki-import), Lotus Notes (http://www.ask.be/Lotus_Notes_2_Dekiwiki_Converter), ScreenSteps (http://www.mindtouch.com/blog/2008/08/14/screensteps-how-documentation-gets-done/), and others. So, it has been done before and more will do it. As for Tiki, you'll have to either do it yourself or find some likeminded individuals. In the above list, the only converter built by us was for MediaWiki. The rest was done by community members like yourself.

[rickmeyer]

Has there been any movement on this? I would like to convert from a Trac wiki to the Deki wiki but need to retain the latest version of the data and attachments.

[SteveB]

The latest converter being added is for Atlassian Confluence. It should be beta-testable next month.

Not sure what is involved for doing a Trac converter.

[sabadosj]

The latest converter being added is for Atlassian Confluence. It should be beta-testable next month.

Not sure what is involved for doing a Trac converter.

The lack of import/export capability out of the shrink-wrap of the product is a deterrent for companies looking to hop to new solutions. And assuring customers or perspective customers that you have a way to export their knowledge in a usable generic form at any time is also extremely important as well.

Please tell me there has been some developments in this area since this last post...

[SteveB]

We currently offer to customer conversion from MediaWiki and soon Confluence. For small wikis, the end-user can do it themselves, but our customers usually convert over 30,000 pages of content. For sites that large, there is a lot of weirdly formatted content that requires in-depth knowledge of the conversion process. So your mileage may vary if you attempt to do it yourself.

[mickdavidson]

The latest converter being added is for Atlassian Confluence. It should be beta-testable next month.

Not sure what is involved for doing a Trac converter.

I've been looking for info on this here and in the rest of MT's website/FAQs etc - but I can't find anything. I think this is something we're going to have to do fairly soon, by mid April at the latest, so any info about this would be great.
Cheers,

[SteveB]

We had to scrap the old converter (referring to my post from 2008). A new one is being put through it paces already. For now, you'll have to download and compile it yourself. A full build of the converter will be included with "Olympic", our next major release.

[mickdavidson]

Steve, thanks, that's good news. When is Olympic due to be released?

[mickdavidson]

We had to scrap the old converter (referring to my post from 2008). A new one is being put through it paces already. For now, you'll have to download and compile it yourself. A full build of the converter will be included with "Olympic", our next major release.

Steven,
Thanks for the update. Having already carried out a search of the MT website for this and not finding anything about it, where can I find a version to download and compile?
Cheers,

[SteveB]

Sorry, there isn't any. The bits are provided "as-is". And since it's not yet released, we also don't provide support for it. This is more intended for those who like get their hands (really) dirty, coupled with some investigative work.

[mickdavidson]

Steven, ok, thanks. We have a bit of time now, so we're going to wait until Olympic is released and take it from there. Cheers.

[openseo]

It looks like there is now
http://developer.mindtouch.com/en/docs/MindTouch/Tools/mindtouch.import
and an API http://developer.mindtouch.com/en/ref/MindTouch_API/POST%3Asite%2F%2Fimport

the mediawiki importer has been deprecated http://developer.mindtouch.com/User:deprecated/*_Archive/Migrating_from_MediaWiki_to_MindTouch