Tuesday, May 07, 2013

Advanced Database Crawler / Sitecore Search Contrib Update


This is a post about a maintenance update to the SitecoreSearchContrib project, which is hosted on GitHub now.

The version got bumped to 2.1, which means that besides bug fixes, I was able to include some frequently requested features.
Simply put, I would not be able to do that without Sitecore developer community support, so big thanks goes to Nick Wesselman, Mark Stiles, Adam Brauer, Kam Figy, @motoyugota, Kyle Linden (sorry if I forgot someone).
Here are some highlights of what was changed:
Besides this, I performed brief testing on 6.6.0. The codebase compiles with warnings, however the Crawler and Searcher work as expected.
If you need to module for your Sitecore 6.6.0 solution, give it a shot. If you have any issues, please submit them here
Finally, and this is what I should have done in the beginning, the Crawler and Searcher components of this project are now distributed via NuGet, so simply issue the following commands in the Package Manager Console to get the install the module:
PM> Install-Package scSearchContrib.Crawler
PM> Install-Package scSearchContrib.Searcher

And one last thing - the project now has home hosted on GitHub Pages:
http://sitecorian.github.io/SitecoreSearchContrib/
Enjoy!

14 comments:

Stephen Hynds said...

Hi Alex,

We've recently upgraded some sites to 6.6 Update 4 and have had issues with indexes and the Searcher. We have a 2 server/3 database environment - One server for authoring pointed at the master and a pre-live database, and one server pointed at the live database. We're finding that after a few updates of the index on the live server the index will suddenly stop updating.

We can copy the index from Authoring to Live and publish a few items which will then make it across to the index, but as I say - they suddenly then stop getting added.

I downloaded latest versions of the code, compiled and updated our code to reference it. I then deployed this updated code to a test version of the site (on the same physical servers pointed at the same databases) but the problem persists.

We've had some conversations with Sitecore support but obviously they don't support this project. So I was wondering if you might have run into any similar issues with this?

Thanks!

Stephen

Alex Shyba said...

Hi Stephen,

I doubt that index update issues are realated to scsearchcontrib. The project just extends the crawling mechanism, but it does not change the way indexing is triggered.
Could you please give me the ticket ID so I can review with support?
In meantime, I suggest adding more Log.Info entries into the AdvancedDatabaseCrawler methods, so you can see whether the crawler is triggered at all. If it's not triggered, it's probably the IndexingProvider that is failing.
I frequently see cases when too many rows in the web db history table would cause IndexingProvider to choke, so I would check that first.

Hope it helps.

Stephen Hynds said...

Hi Alex,

We think we've solved it by changing the update interval to 1 minute rather than 5 minutes which it was set at. Since then it does seem to have resolved itself.

Out of interest - what would you consider as too many rows in the web db history table? We're looking at 468,000 on web, but only 96,000 on master which seems like quite a difference!

Thanks for the suggestions.

Stephen

brian @ BrainJocks said...

Alex, I just downloaded the latest and will be using it against a 6.6 implementation. Any chance you solved the "root item not found" issue when an index is rooted to a possibly "not yet published" item?

Alex Shyba said...

Hi Brian,

No I don't believe it was possible to override. We def solved it in 7 though. Will take another look next week to confirm for the module.

Mariella Scofield said...

Hi Alex,

I downloaded the latest version of ADC and pluged it into my sitecore website implementation that was using the old version of ADC. I am getting an error that it can;t find the methods:
AddFieldValueClause();
ApplyFullTextClause();
ApplyIdFilter();
ApplyLanguageClause();
ApplyLocationFilter();
ApplyRelationFilter();
ApplyTemplateFilter();

in the SearchParam.cs. And indeed i don't see them in the project latest version. Did u stop using it? I see some people on the net talking about them but I just can;t find them on the code I have... could you point me out where i could find those methods?

Thanks,
mariella.

Alex Shyba said...

Hi Mariella,

Thanks for giving the module a try!

In the latest version those methods were moved out to the QueryBuilder class:

QueryBuilder.BuildFullTextQuery(this.FullTextQuery, index),
QueryBuilder.BuildRelationFilter(this.RelatedIds),
QueryBuilder.BuildIdFilter(templateFieldName, this.TemplateIds),
QueryBuilder.BuildLocationFilter(this.LocationIds),
QueryBuilder.BuildFieldQuery(BuiltinFields.Database, this.Database),
QueryBuilder.BuildLanguageQuery(this.Language)

I did not expect you will be using those directly, outside of the search parameters.

Hope it helps.

If you have any questions, please submit them here:
https://github.com/sitecorian/SitecoreSearchContrib/issues

Alex Shyba said...

To Everyone:

Please submit your questions or bug reports about the module here.

Mariella Scofield said...

Thank you Alex for answering me. I really appreciate it. I am not sure as well why they are using those methods. The code was written by a consulting company. I want to just have it running then I will refactor it. If I have any other question, I will post it in the link you provided.

Thanks,
Mariella.

Ray Wang said...

Hi Alex,

Do we know if this Sitecore Search Contrib works well with Sitecore 7?

Cheers

Ray

Ray Wang said...

Hi Alex,

Just wondering if Sitecore Search Contrib works well with Sitecore 7?

Cheers
Ray

Mariella Scofield said...

Hi Alex,

our website uses filter with ADC module. I wanted to know if it is possible to something like this: an user chooses 2013 as a document year and 'awards' and 'market trend' as a document type. he should see ALL 2013 awards documents and ALL 2013 market trend documents. I tried all the filter examples it comes with your module, but I always get ALL 2013 documents (of all types), AND ALL awards and document types that are 2013.. Also, could you explain what "RelatedIds" and "LocationIds" means....?

Thanks,
Mariella.

Mariella Scofield said...

Hi Alex,

our website uses filter with ADC module. I wanted to know if it is possible to something like this: an user chooses 2013 as a document year and 'awards' and 'market trend' as a document type. he should see ALL 2013 awards documents and ALL 2013 market trend documents. I tried all the filter examples it comes with your module, but I always get ALL 2013 documents (of all types), AND ALL awards and document types that are 2013.. Also, could you explain what "RelatedIds" and "LocationIds" means....?

Thanks,
Mariella.

Alex Shyba said...

To Everyone: Regarding 7.0 compatibility and migration questions/concerns, please post here.