Search not working in codebase since > 2 weeks

[merktnichts]

Sorry for bugging, but as the Hayes .d release is near, I'm a bit worried about bug #2343 (http://bugs.opengarden.org/view.php?id=2343).

The problem is that searching doesn't work at all since several SVN versions and the bug hasn't been touched by a developer although it's assigned.
This makes me wonder if there's no general problem and it's a problem about my database. Could PeteE, Steve or Roy comment on this, please?

If it's a local problem (which I don't think as Mozhechkov from QA experienced the same), I'd be glad to get hints for troubleshooting/fixing it, as this seems to be the very last issue stopping us from going public with our Deki installation.

I just updated sources (r5373@tp (orig r5267)) and ran the Lucene updater (which I'm compiling/copying/executing every morning since the bug occured) by executing /var/www/deki-hayes/bin/mindtouch.deki.luceneupdater.sh, but today there's a change - a new error message from the updater:

Error deleting lucene index at: http://localhost/@api/deki/luceneindex/clear?apikey=xxx&dream.in.verb=DELETE
Error:<?xml-stylesheet type='text/xsl' href='/@api/host/resources/error.xslt'?><error><status>404</status><title>Not Found</title><message>resource not found</message><uri>http://localhost/@api/deki/luceneindex/clear?apikey=xxx&amp;dream.in.verb=DELETE</uri></error>


Nevertheless, the search still doesn't work and I'm a bit worried that on the one hand the bug will be rolled out to the public on friday and on the other hand that we still can't go public, as this strongly depends on a working search function.

Help! :) Btw, it's running on Debian, if that's important to mention.

[merktnichts]

Btw, I know the problem is with SVN only, as I checked searching with my database by temporary installing 1.8.1c (in a copy of my VM). The search works aok there. After going forward to latest SVN again, search didn't work anymore.

That was just to check if my database is to blame.

[royk]

hey merktnichts, this is pete's expertise - he was gone last week and i know he's been very busy catching up with his work items this week (he's actually the one responsible for pushing out the OSS *and* the VM releases on friday). i'm sure he'll take a look at it once he wraps up some of his other work items.

[merktnichts]

Alright, I just wanted to bring it back into your guys' focus before the release will be out on friday. Thanks for commenting, I'm gonna wait and keep on doing my daily morning search function sports. ;)

[PeteE]

Btw, I know the problem is with SVN only, as I checked searching with my database by temporary installing 1.8.1c (in a copy of my VM). The search works aok there. After going forward to latest SVN again, search didn't work anymore.

That was just to check if my database is to blame.

merknichts - thanks for reminding me ;) I'll get this bug resolved today!

pete

[PeteE]

Error deleting lucene index at: http://localhost/@api/deki/luceneind...in.verb=DELETE
Error:<?xml-stylesheet type='text/xsl' href='/@api/host/resources/error.xslt'?><error><status>404</status><title>Not Found</title><message>resource not found</message><uri>http://localhost/@api/deki/luceneindex/clear?apikey=xxx&amp;dream.in.verb=DELETE</uri></error>


merktnichts - the error you're seeing is a regression from some changes we made yesterday to support "multi-tenant" mode (multiple wikis served by a single dekihost). I've updated the mindtouch.deki.luceneupdater.exe so it should work now. Can you get latest from SVN and try again? If you're using the VM you should be able to just:


sudo updateWiki.sh

Otherwise you'll need to rebuild the deki C# bits and update your .dll, .exe's in /var/www/deki-hayes/bin/

Thanks!
pete

[merktnichts]

merktnichts - the error you're seeing is a regression from some changes we made yesterday to support "multi-tenant" mode (multiple wikis served by a single dekihost). I've updated the mindtouch.deki.luceneupdater.exe so it should work now.
Yup, the luceneupdater runs now without an error.

Did you also mean to fix the "search is not working" bug? Unfortunately that one still exists.

Request URI:
http://localhost/@api/deki/site/search?dream.out.format=php&dream.in.host=xxx&dream.in.origin=xxx&q=deki&limit=26&offset=0&format=search

Server response:
Array
(
[error] => Array
(
[message] => error querying the lucene index service with query: deki title:deki^4 description:deki^3 tags:deki^2 comments:deki
[status] => 500
[title] => Internal Error
[uri] => http://xxx/@api/deki/site/search?dream.out.format=php&dream.in.host=xxx&dream.in.origin=xxx&q=deki&limit=26&offset=0&format=search
)

)

[PeteE]

Yup, the luceneupdater runs now without an error.

Did you also mean to fix the "search is not working" bug? Unfortunately that one still exists.


merktnichts - Can you send me a stack trace from the deki-api.log. I'm having a hard time reproducing that issue on my end.

When you update from SVN and recompile the C# code, are you copying the .dll and .exe file into your wiki's bin dir and restarting dekihost? I'm just trying to get an idea for how you're updating...

Thanks,
pete

[merktnichts]

Uhm... Guess what happened:

I was writing a step by step list for you and followed my list religiously, cause I didn't want to provide wrong or half-hearted information. I cleared the logs and was about to prepare a nice package for you. When I was at point 13 of my 13 points list ("13. type "deki" in search field, hit "search""): BANG! Search was listing the "deki" hits. :rolleyes:

So I can trash my 13-points-built-with-love-list again (*grrrr*@myself) and you can put a checkmark next to the bug. ;)

Hooray for PeteE, you're a legend! Everything's working as it should, as it seems. Amazing!

(Maybe Mozhechkov can verify that it's fixed for him as well?) :)

[PeteE]

hahaha, just glad it's working for you :) Thanks so much for all your hard work!

[alex25]

Hi,
I have the same problem in "Deki Wiki 1.9.0b (rev. 8921)"
and when I try to run /var/www/deki-hayes/bin/mindtouch.deki.luceneupdater.sh
I get the following error message:
Cannot open assembly mindtouch.deki.luceneupdater.exe.

when I try to manually run /var/www/deki-hayes/bin/mindtouch.deki.luceneupdater.exe
I get:
** (/var/www/deki-hayes/bin/mindtouch.deki.luceneupdater.exe:27852): WARNING **: The class MindTouch.Tasking.Result`1 could not be loaded, used in mindtouch.dream, Version=1.3.99.188, Culture=neutral

Unhandled Exception: System.TypeLoadException: Could not load type 'MindTouch.Tasking.Result`1' from assembly 'mindtouch.dream, Version=1.3.99.188, Culture=neutral'.
at <0x00000> <unknown method>
at LuceneIndexUpdater.Program.Main (System.String[] args) [0x00000]

[PeteE]

alex25 - Did you try updating your search index in the control panel? Go to Admin:Settings and click the "Rebuild Index" button. That will re-index everything.

The mindtouch.deki.luceneupdater.exe is a deprecated tool.

If you rebuild, wait a minute, then try searching and don't get any results, check your /etc/dekiwiki/mindtouch.deki.startup.xml file and look for this:


<path.store>/var/www/dekiwiki/bin/cache/luceneindex/$1</path.store>


Make sure that path allows your your apache user to write to it.

Thanks,
pete

[alex25]

alex25 - Did you try updating your search index in the control panel? Go to Admin:Settings and click the "Rebuild Index" button. That will re-index everything.

The mindtouch.deki.luceneupdater.exe is a deprecated tool.

If you rebuild, wait a minute, then try searching and don't get any results, check your /etc/dekiwiki/mindtouch.deki.startup.xml file and look for this:


<path.store>/var/www/dekiwiki/bin/cache/luceneindex/$1</path.store>


Make sure that path allows your your apache user to write to it.

Thanks,
pete

Thanks pete,
the rebuild button worked perfectly.

Alex

[dashu]

That solution did not work for me. I clicked on the rebuild search index in the control panel and I can see that the time stamp on the /var/www/dekiwiki/bin/cache/luceneindex/default/segments is updated, but when I type something in the search box I get "Sorry, there were no exact matches to your query." when the thing I am searching for is in the title of one of my pages.

How does the search index normally get rebuilt? Is it something automatic or do I have to click the button once a day?

[PeteE]

Did you wait 60 seconds for the indexing to kick in? The index gets updated every time you modify a page, add a comment, add a tag, etc. so you don't need to click that button every day.

[bohappa]

Search terms appear to be connected with OR, which isn't what I want. I want to use Dekiscript to define popular searches, for example, wiki.search("tag tag tag tag"), but I'm not getting restrictive results. I'm getting more and more content.

Any suggestions on how I can make restrictive searches?

[SteveB]

Use AND between the terms that you're looking for instead of just spaces. Lucene uses OR to concatenate multiple search terms.

See
http://wiki.opengarden.org/Deki_Wiki/Search

[rtowne]

I am running 1.9.0b (rev. 9143) running on: Linux 2.6.18-5-686, PHP 5.2.0-8+etch10, mySQL 14.12 distribution 5.0.32, and Mono 1.2.2.1 -- from latest VM.

The Lucene filters for Word 97/2003 work, but while Word 2007 filters appear to be there (docx2txt) and enabled it doesn't seem to pickup. Are there filters for office 2007 that have been verified by anyone?

The contents of docx2txt:

#!/bin/sh

# save stdin to a file since unzip doesn't work on streams
TEMP=`mktemp`
dd of=$TEMP 2> /dev/null

TEMPDIR=`mktemp -d`
unzip -oq $TEMP -d $TEMPDIR # Extract the file
tr "<" "\012" < $TEMPDIR/word/document.xml | grep ^w:t \
| cut '-d>' -f2, | uniq
rm $TEMP
rm -r $TEMPDIR #Delete excess (formatting) folders
deki-hayes:/var/www/deki-hayes/bin/filters#


Thanks,
Robert

[PeteE]

Robert - This might be because the unzip program isn't installed (will be fixed in the next VM release).

Try this:



apt-get install unzip


Then try uploading the document again (and waiting 60 seconds) for lucene to process it.

Thanks,
pete

[bohappa]

Thanks for the search tip. I made a few pages using Dekiscript, e.g. {{wiki.search("tag:How_to AND tag:admin"), and it worked very well.

Thanks again for quick, useful help. :)

[rtowne]

cool, thanks Pete - that worked for Word 2007. is there somewhere i could have reviewed a log that would have pointed me to the fact that it was failing on the zip?

i don't have filters for the additional non-Word products in the config file but will post back here if I find anything (e.g., powerpoint, infopath, excel, etc).

Many thanks!

[PeteE]

cool, thanks Pete - that worked for Word 2007. is there somewhere i could have reviewed a log that would have pointed me to the fact that it was failing on the zip?

i don't have filters for the additional non-Word products in the config file but will post back here if I find anything (e.g., powerpoint, infopath, excel, etc).

Many thanks!

There currently isn't any good logging. I usually just run the converter manually to see if it's giving me output.


cat /var/www/dekiwiki/attachments/{pageid}/{filename} | /var/www/dekiwiki/bin/filters/{program}


I've filed a bug to log any error messages: http://bugs.opengarden.org/view.php?id=3860

Also, these links have some info on indexing files:

http://wiki.opengarden.org/index.php?title=Deki_Wiki/FAQ/File_Management/How_do_I...Index_file_attachments%3F
http://wiki.opengarden.org/index.php?title=Deki_Wiki/Community_Contributions/Other/Enabling_indexing_of_Microsoft_Word%2C_Excel%2C_Po werpoint%2C_and_OpenOffice_Writer_documents

Thanks,
pete