17 replies to this topic

[Autocrat]

Okay... now I use a CMS, and it has a Search functionality included... though it does have a few drawbacks (being that it is not particualrly compatible with one of the content types).... and it can either do a full search on certain content types... or that it can search based on KeyWord data (supplied when adding content of any type).

So I thought I'd look into an external supplier... and instantly thought of Google.
Which resulted in a rather large shock.
They only provide an AJAX version... which means it will only work if JS is enabled on the visitors browser.
Does this not mean that certain visitors will not get to see the search option due to security or software reasons?
This strikes me as being rather dangerous and questionable.


So I thought I'd look around to see what other things I could find... and most of them appear to be external resources...
You basically plugin a piece of form code, then all the work, and the results, are done on the suppliers site.
This leaves you open to whatever they choose to display on their resulsb page (including for some a large nubmer of adverts for exactly the type of thing the visitor is searchign for!!!).



So I am now at a loss!

I'm not a programmer (I may fudge, but have little skill)... and I do not really have the time to faff around with it.... and I don't want JS based stuff.... so what are the alternatives?

Do others use site-search scripts... and if so, which ones?

Have you seen good site-searches on otehr sites that you use?

Or should I throw myself in to the KeyWord (taxonomy) system and pray the clients actually do the correct thing?


All ideas and thoughts welcome (apart from the ones saying I should learn to program... I know I need to learn... but having the time is a bit of an issue).

Edited by Autocrat, 07 January 2008 - 08:12 AM.

[EGOL]

I know someone at NPS and they used the Google Search appliance. Its really expensive but delivers great results. Here is what they say on Google's profile page

“We used to get 30 to 40 complaints a day, plus lots of ‘I can’t find this’ inquiries, which took a lot of our staff time to deal with,” he recalls. In addition, he says, the existing solution “wasn’t capable of the managing and reporting capabilities that I knew we needed. But I knew we weren’t going to build it ourselves. Once we implemented the search and integrated it with the feedback page the email inquiries went from an average of 40 a day to 2.”

...

[Autocrat]

I'm not questioning it's ability (though it is reliant on the existing index... so it may take a while for new content to be found and listed)...

I'm mroe concenred about the accessibility issues that some of these thigns raise, adn the danger of loosing visitors to adverts that may appear on others (including Googles free version), and that some of htem seem to be a little stringent (I've seen some that limit to the top 5 only results???).


So, the paid for Gogle one has it's advantages and is useful... particularly that stats. Good to know.
But is it better in regards to the code/accessibility? I can see that it doesn't include adverts (unless you want to include your own account, which is a nice touch).

What others are there?

[joedolson]

I'm not questioning it's ability (though it is reliant on the existing index... so it may take a while for new content to be found and listed)...

Actually, the Google seach appliance is a completely different service from the custom search, Autocrat --- and it's NOT dependent on their search index. It is, however, intended for enterprise level clients and is out of the price range of most website. Google Enterprise: Search Appliance

Yes, I absolutely agree that the AJAX only interface for Google's site search is a serious issue.

Instead of the AJAX site search, you can use the Custom search (Custom Search), which displays results either in an iFrame or in a new page. (I think.) I've never implemented it in a site, so I'm not sure what the flexibility actually is, but it IS available without AJAX or JavaScript being required.

[yannis]

Adding a script to do search on a website is not an easy task, entailing 'stop words', cron jobs and the like. There are some good scripts on sourceforge to get you started but my recommendation is to rather try and improve on the CMS search code. It will probably be easier. Check also if the Drupal Search module can be ported to your CMS, is actually very powerful and can do folkskonomy.

Yannis

[Autocrat]

What I'm after is something that will crawl the site itself...

Which seems to be a problem if you do not have any actual "files", do not use file extensions, and use friendly urls (so purely dynamic content, presented in the format of " http: / www. yoursite . com / index.php / section1 / thispage " seems to be a major pain!).

The only one I've found that seems to come close is TSEP ... yet even that seems to be failing to find the actual pages (it can find files like a dream... but the dynamic stuff seems invisible, to all the plugins and crawler scripts).


Taxonomy isn't realy of use... I can already make use of such a feature inthe current CMS, and it simply isn't robust enough (as I cannot count on clients entering in relevant terms, nor on users entering the same terms as the client). Thus the wish for acutal indexing (which can be done on a regular basis and means faster result returns etc.).


Maybe I'm simply being overly optermistic.

[Brad]

>>What I'm after is something that will crawl the site itself...

Look at Fluid Dynamics Search Engine. I think the limit of it's index is 10,000 pages but it works great.

>>Google

I'm not convinced that Google's regular site search is much good anymore. Too many pages without enough link juice end up in the supplemental index.

Yahoo might be better but frankly for FREE site search from a big SE MSN may produce as good results but better co-branding.

[eKstreme]

Try ht://dig for self-hosted search. Or Atomz or if you don't care to use Google or Yahoo's site searches.

I used to use Atomz a while back and it was wonderful. Not sure what it's like these days. Also, I find Yahoo!'s site search tool (called Search Builder if you don't care to use Google or Yahoo's site searches.

I used to use Atomz a while back and it was wonderful. Not sure what it's like these days. Also, I find Yahoo!'s site search tool (called [url="http://www.cre8asiteforums.com/redirect/jump.php?url=omorp%2Fm%2Fmoc.oohay.hcraes.redliub%2F%2F%3Aptth"]Search Builder) much better for the webmaster than Google's CSE.

Failing that, give me a shout and I'll send you a little script I wrote.

Pierre

[yannis]

Try the 'snoopy class' for crawling and capturing the posts. (You can download from sourceforge). You will need also to pick up a suitable script for keyword extraction.

Yannis

[Autocrat]

Well, I found... " isearch2"... and it was a doddle to use... and it worked.
Within 15 minutes (including upload and following instructions), I had a simple site indexed... and searchable.
It even let me decide what value to put on body words, header words, titles and urls (damn nice!).

Of course, at the moment it's on it's own page... but I'm sure I can turn it into a block for a site


...eKstreme...
I will also go and look at dig and atmoz.
Thank you for hte kind offer

...yannis...
Now that sounds like I'd haveto do some form of coding (or at least udnerstand coding)... which is part of the problem (that being don't have much of a clue).
Still, I will go and have a look, as I might learn a few thigns whilst tinkering.

[phaithful]

Sounds like you found your solution, but if it turns out otherwise, you might want to consider Google Custom Search Business Edition.

It's something that I've been looking into since we need to replace our Google Appliance. It's not quite the same Google Custom Search that Joe posted.

Essentially here you specify to the Google bot specifically the pages you'd like it to spider, external or internal (you can let the bot through your firewall). Then all you need to do is put the form field on your pages, and when a search request comes in, it'll send a request to Google and pass you back information as an XML feed which you can display any which way you want. Similar to how the Google Appliance works.

There is of course a cost, but not nearly as much as the Google Appliance:

Custom Search Business Edition is available in a number of plans:

• Search less than 5,000 web pages: $100 per year
• Search less than 50,000 web pages: $500 per year
• Search less than 100,000 web pages: $850 per year
• Search less than 300,000 web pages: $2250 per year

Edited by phaithful, 08 January 2008 - 12:52 AM.

[Autocrat]

Interesting... I will go look at that, as well as the suggestion by ...Brad... (sorry, I failed to see your previous post!).

[jazzylee77]

I'm looking at adding a self hosted search. And no, I don't want to use Google's index based tool or pay for what the big sites use. This would be for a dynamic php site. I actually would like to be able to point to search results in a related blog. A quick look at isearch2 shows the same url for any result. (at least that is the way it is set up on their site).

I'm also checking out a free one called sphider. It creates search result urls like search.php?query=cre8asite&search=1 which I'm sure I can make friendlier with modrewrite.

So obviously I'm thinking 3 things. Make it easier for visitors to find things on my site, make it easier for me to point to those things, and (here is where I get uncertain) make those pages friendlier to search engines.

What do you think? Dupe content penalty?

[jazzylee77]

I found what Matt Cutts had to say.


from that page:

The new webmaster guideline ...“Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don’t add much value for users coming from search engines.

I can see how this could be abused, but I'm thinking the way I want to implement it might be allowed.

I'm not going to auto-generate pages, but link to useful searches of my sites products. If someone is searching for blue widgets and they are available from different manufacturers and not otherwise a category on my site, I think this would be a useful page to land on from a search engine. In this sites specific case it would be very useful.

Question is, how will Google see it?

Edited by jazzylee77, 17 January 2008 - 05:27 PM.

[Autocrat]

I'm not sure I follow...
You want to have search functionality on your site.

You want it to work for your site, and possibly for other sites as well?

That way if someone does a search on your site and you don't have any matching content, it would find results from another site?
If so, that sounds rather intriguing.

Alternatively, if you are suggesting that you have pages on your site that would be listed in search-engines, and take people to serch results... I'd (me, personally) woudl go mad.
There are more than enough sites that do that as it is, and they always seem naff... and it's annoyign going from a search engine... to a search engine.
(of course, thats just my view!)


As for hte Isearch2... I'm quite happy with it... I can include the simple search easily enough....
and will incorporate the advanced search into a sepreate page in my CMS... and then tell robots/htaccess to ignore, not touch the isearch dir... no-one needs links to the results page, as they will be taken there from the search (if I could find a way of pushing the results into one of my CMS pages, I'd be ecstatic).

[jazzylee77]

That's not quite what I'm saying. If using sphider you do a search, you can copy the url of those search results. So if I wanted to tell someone where all of a kind of product are, then I can do the search, copy the link and send it to them.

And yes I could also publish those links with the possibility they may be crawled and indexed. I agree those pages that show search results to other sites are annoying. But I plan to limit the spider to just my product pages, so it would show a specialized index that should prove useful to someone looking for a type of item. "Here are all the blue widgets that are otherwise scattered around my site"

I guess I could do a search, copy the html results, edit it a bit and add that to a regular page. But the problem with that is; it would be static and I would have to go back and update it with new or dropped products.

[Autocrat]

Do you have a static site?
If not, you may be able to do what you are wanting without the search stuff (depending on the site of course).
Things like Categorisation, Taxonomy etc. goes a long way towards such things.

I'm opting for the Search as some of my sites are communities with different news topics/events etc... so no real categorisation as they are so diverse... thus the ability to search for specific terms (though it appears Isearch2 has a problem with multiple words???).

[jazzylee77]

No, this site is a dynamic mess. A mix of perl datafeed script and php database.

As far as Sphider goes, I've given up for now on fully directing the crawl. By using advanced search I can exclude terms etc. It has and, or, phrase and "-" and probably more that I don't know about.

The program is light and quick when it comes to returning results. I so far have 19,569 links and 64,158 keywords indexed and the results page loads instantly.

The biggest downside is the knowledgeable ones are not very active at the forum, though I suspect I'll see answers eventually.

Edited by jazzylee77, 21 January 2008 - 01:21 AM.