Solr in Coldfusion 9

Some tips for using Coldfusion 9 Solr search:

• Lesson Number 1: Make sure you review ALL the files under your collections \conf folder, many of them need to be commented out or changed from default values. Here is an incomplete list of what I changed:
• Commented out words in protwords.txt
• Deleted content in spellings.txt
• Updated synonyms.txt, deleted the test words, and added new synonyms based on the vocabulary specific to my application's domain
• schema.xml:
• Updated default query operator from AND to OR
• solrconfig.xml
• commented out "solr", "rocks" and other similar configurations
• Enable term highlighting (see http://help.adobe.com/en_US/ColdFusion/9.0/Developing/WSe9cbe5cf462523a0-5bf1c839123792503fa-8000.html \), but it's still not quite there. Still looking for a complte solution for this one.
• In Coldfusion Admin, change Solr buffer limit from default value 40 to 80 (http://bloggeraroundthecorner.blogspot.com/2009/08/tuning-coldfusion-solr-part-1.html)
• Ensure "Coldfusion 9 Solr Service" is started and set to "Automatic" startup type
• To access Solr collection from remote Coldfusion servers, 
• For the remote CF servers: change Solr server name in Coldfsuion Admin, DATA & Service > Solr Server > Solr Host Name
• For the Solr Host: Allow remote server to have access to its Solr instance. Configure Jetty to listen on all incoming IP address. Coldfusion's default configuration is allow only 127.0.0.1, thus disabling remote CF server's access to it's hosted Solr instance. To change this, just open jetty.xml in {coldfusion home}\solr\ect, and comment out the host line which restricts to listen on 127.0.0.1 only (interestingly, I found this solution from this link: http://helpx.adobe.com/coldfusion/kb/coldfusion-9-limit-access-solr.html, which claims Coldfusion's default configuration is to listen on all ip addressed
• Web access to Solr admin: http://{Your CF Server Name}:8983/solr/
• Last and a big one, when you try to update multiple indexes, you will see error like this:

Error_opening_new_searcher_exceeded_limit_of_maxWarmingSearchers4_try_again_later

Stack Trace:
...
org.apache.solr.common.SolrException: Error_opening_new_searcher_exceeded_limit_of_maxWarmingSearchers4_try_again_later

Error_opening_new_searcher_exceeded_limit_of_maxWarmingSearchers4_try_again_later

request: http://localhost:8983/solr/.../update?commit=true&waitFlush=false&waitSearcher=false&wt=javabin&version=1
    at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:424)
    at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:243)
    at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)
    at coldfusion.tagext.search.SolrUtils.commitToServer(SolrUtils.java:1024)
    at coldfusion.tagext.search.SolrUtils.addDocument(SolrUtils.java:664)
    at coldfusion.tagext.search.IndexTag.doQueryUpdate(IndexTag.java:929)
    at coldfusion.tagext.search.IndexTag.doStartTag(IndexTag.java:254)
    at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2722)
…

This error will be consistently showing up in production environment with even very light traffic, and became a big show stopper for the usage of Solr collection provided by Coldfusion (does not seem to be a Solr problem, instead a problem of Coldfusion integration problem). 

Root cause seems to be that Coldfusion commits every index update, and causing excessive Solr searcher warming. Combining with misconfigured default Solr configuration, this problem starts to appear for a collection of only a few hundred entries.

In Coldfusion 10, this problems seems like easier to solve by using the newly introduced "autoCommit" attribute, which should always be "no" (i.e. never use the default value). Then I guess configuring the "autoCommit" in solr config file will solve the above problem in Coldfusion 10.

However, production environment that is already in CF 9 is stuck. My solution for it is a combination attack:

• Manually throttle cfindex update by adding a sleep after every update
• Wrap try catch block around the cfindex update, then add a longer sleep time in the catch block, and add retry logic to try update again after the sleep
• Database flag of index entries, this flag will be set only after index is updated successfully. A scheduled CF task will scan this flag to identify missed updates (due to the above error), and try to update index again

The above approach so far allows me to update thousands of indexes without any problem.

Coldfusion Webservice Input Name Changed to in0

Symptom: Coldfusion webservices sometimes complain about input parameter "In0"

Environment: Coldfusion 9 on Windows

Solution: Login to Coldfuion 9 Administrator portal, Go to Server Settings > Caching, click on "Clear Template Cache Now" and "Clear Component Cache Now". This seems to trigger to re-compile of webservices WSDL thus fixed the problem.


However:

1. I don't know what is the root cause of this problem. Google turn-up some similar complains, but could not find any official answers on this
2. The solution which required human interaction does not help in production environment
3. The solution does not solve the problem permanently, sooner or later I'll hit the same error, and has to go back to the server and clear cache again

Flex Builder - TODO, FIXME in "Tasks" view

Here is a plugin that adds support for TODO comments in "Tasks" view.
http://www.richinternet.de/blog/index.cfm?entry=911D4B57-0F0D-5A73-AF6F4D4D04099757

It was created for Flex Builder 2 in 2006, but I tested, it also works for Flex Builder 3.

Here is link to the request on Adobe's JIRA: FlexBuilder should parse //TODO comments into the normal Task panel . I have voted for this issue.

However, Dirk obviously did a good job in his plugin, such that Adobe decided to defer and closed the issue!?

MS SQL Column Alias

T-SQL used in Microsoft SQL Server seem to have very limited support for column alias. If you have a complex calculated column that you want to re-use in GROUP, SELECT or another expression, it will be desirable to use alias to reference the same definition everywhere so that we can have a clear and easier to understand SQL statement.


Here is an article that is very helpful in this matter: Avoid Transact-SQL's Column Alias Limitations

Key points:

• MS SQL only allow alias in SELECT and ORDER BY
• Use sub-query, can help in some cases to re-use column alias
  

Flex Builder Error 1046: "Type was not found or was not a compile-time constant"

Problem:
While editing a big .as file in Flex Builder 3, the compile is broken suddenly. Got this error:

1046: Type was not found or was not a compile-time constant: ?????.

Cause:
Seems like Flex editor tend to wrongly remove import statement at the top of .as file without user consent.

Solution:
Disable Flex Builder feature:
Windown > Preferences > Flex > Editors > ActionScript Code, uncheck "Remove unused imports when organizing"

SQL 2005 on Windows XP SP3 - MSXML6 Error

Problem:
With a fresh Windows XP SP3, I cannot install SQL 2005. The error message in log file is:

Property(S): SupportedOSMessage = Installation of this product failed because it is not supported on this operating system. For information on supported configurations, see the product documentation.
Property(S): ShortCutText = MSXML 6 Service Pack 2 (KB973686)
Property(S): DialogTitle = MSXML 6 Service Pack 2 (KB973686) Setup
Property(S): ProductName = MSXML 6 Service Pack 2 (KB973686)
Property(S): ShortName = MSXML 6 Service Pack 2 (KB973686)
Property(S): WrongPackage = This MSXML6 Service Pack 2 (KB973686) package is not supported on the current processor type.
Property(S): DialogPatchTitle = MSXML 6 Service Pack 2 (KB973686) Patch
Property(S): SystemFolder = C:\WINDOWS\system32\

...

MSI (s) (04:A8) [09:38:47:725]: Product: MSXML 6 Service Pack 2 (KB973686) -- Configuration failed.

MSI (s) (04:A8) [09:38:47:725]: Windows Installer reconfigured the product. Product Name: MSXML 6 Service Pack 2 (KB973686). Product Version: 6.20.2003.0. Product Language: 1033. Reconfiguration success or error status: 1603.

Solution:
Luckily, I am not the first one who saw this problem. This problem seems to be around for a while now. There is a nice Microsoft KB article for this exact problem, and the solution is to use "Windows Installer CleanUp utility" to remove existing MSXML then install SQL 2005.

And, IT WORKS!!

Reference: MSKB 968749 http://support.microsoft.com/kb/968749

Blue Screen Trying to Install Windows XP SP3

During last few months, I have been through a couple of Windows XP SP3 installation. They are mostly eventless. One of the worst was a computer that hung after reboot, but after a forced power recycle that system seem to be fine.

Just when I am about to conclude that Windows XP SP3 update is pretty good, I got the famous Blue Screen of Death.

While trying to install Windows XP SP3 on my computer, I got an "Access is denied" error message. So that was not too scary, I typed the error message along with keyword "Windows XP SP3" in Google and found an MSKB 949377 which provided a solution to this exact problem.

Now, following KB 949377, I downloaded full Windows XP SP3 package and subinacl.exe, and created Reset.CMD. But, I tried it 3 times, everytime I ran Reset.CMD, it gave me Blue Screen of Death with error message about registry access. So, I did another Google search with keywords: "windows xp sp3 access denied subinacl registry blue screen". Now "I'm feeling lucky". The first hit provides a solution that works perfectly for me. Here is link to Jason's blog post that solved my problem: http://www.mrfloppysa.com/wordpress/?p=13

The trick is to remove the HKLM line in RESET.CMD. The new RESET.CMD file would be like this:

cd /d "%ProgramFiles%\Windows Resource Kits\Tools"
REM subinacl /subkeyreg HKEY_LOCAL_MACHINE /grant=administrators=f /grant=system=f
subinacl /subkeyreg HKEY_CURRENT_USER /grant=administrators=f /grant=system=f
subinacl /subkeyreg HKEY_CLASSES_ROOT /grant=administrators=f /grant=system=f
subinacl /subdirectories %SystemDrive% /grant=administrators=f /grant=system=f
subinacl /subdirectories %windir%\*.* /grant=administrators=f /grant=system=f
secedit /configure /cfg %windir%\repair\secsetup.inf /db secsetup.sdb /verbose

C++/CLI Warning C4945

Reason: This warning was generated because I referenced multiple C# projects in the C++/CLI project, and they all have "copy local" set to true.

Solution: Change reference to the C# projects, set all "copy local ..." properties to false.

A hotfix is available, which does not seem to be in SP1 as of 7/7/2008. So this fix probably will never be publicly available. This proves again C++/CLI is not on the top of TODO list for Visual Studio team.

Reference:
Microsoft KB 922271: http://support.microsoft.com/kb/922271

Reboot Remote Computer

"Remote Desktop" makes it easy to manage Windows computers remotely.

But, if you want to reboot the remote computer, you will be frustrated. Because there is no "Shut Down" button on the start menu. The trick is to click on the "Windows Security" menu which will bring up the "Windows Security" dialog which gives you the options to "Shut Down..." computer. You can also use the shortcut: Ctrl+Del+End to access the "Windows Security" dialog.

Delay Load DLL and __HrLoadAllImportsForDll

Here are a few tips for delay loaded DLLs based on my experience with Visual Studio 2003 (Visual C++ 7.1)

1. Delay load can be easily setup using the Visual Studio IDE by the following steps:
• Open project property page: Linker > Input
• In Additional Dependencies, add library: delayimp.lib;
• In Delay Loaded DLLs, add all the DLLs you want to delay load.
  TIP #1: if you have more than one DLLs, the file names must be separated by ";", e.g.: rapi.dll;ceutil.dll. If you forget this, and try something like: rapi.dll ceutil.dll, then the linker will treat the whole string as the file name of a single file and fail to find a file with that name. There will be a subtle warning during build: LINK : warning LNK4199: /DELAYLOAD:rapi.dll ceutil.dll ignored; no imports found from rapi.dll ceutil.dll, which most likely will be ignored.
2. Detect if the delay loaded DLLs exists on the host computer:
   The whole purpose of delay loaded DLLs is to allow our program to be more flexible. It can be loaded on computers that do not have certain DLLs, and it can choose to use those DLLs when they are present. So, there is always a need to know if we can load the delay loaded DLLs on the host computer. __HrLoadAllImportsForDll is the recommended function by MSDN documentation to do the job.
   TIP #2:Do not use __HrLoadAllImportsForDll because it use memcmp to compare DLL file names byte by byte which is too strict. File names in Windows system is not case sensitive, so the file names can be the same even if memcmp fails. This function will return module not found error code even when the DLL files are installed on host computer. I end up lift the implementation from delayhlp.cpp, and replace memcmp with _tcsicoll which is case insensitive.
   

Lost in Space

While using XmlSerializer in .NET 2.0, I found that spaces in my string values are lost during a round-trip of serialization and de-serialization. Looking at the serialized XML string, I can find that space is preserved during serialization. The XML looks like this:

<Text> </Text>

But after de-serialization the string variable Text only has value string.Empty. After a little research, I found that using the following code segment to de-serialize can avoid the above problem:

//variable xmlString contains the XML string to be de-serialized.
//varialbe result of type T is to be de-serialized from the string
T result;
XmlSerializer s = new XmlSerializer(typeof(T));
StringReader r = new StringReader(xmlString);
XmlReaderSettings settings = new XmlReaderSettings();
//white space are especially important for string types
//if "IgnoreWhitespace" is true, a string value equal to " "
//may be lost during serialization and deserialization.
//the line below is the key of this posting
settings.IgnoreWhitespace = false;
XmlReader reader = XmlReader.Create(r, settings);
result = (T)s.Deserialize(reader);

For comparison, below is my old de-serialization code that causes lost of space:

//variable xmlString contains the XML string to be de-serialized.
//varialbe result of type T is to be de-serialized from the string
T result;
XmlSerializer s = new XmlSerializer(typeof(T));
StringReader r = new StringReader(xmlString);
result = (T)s.Deserialize(r);

srand() and threads in MFC

This is an old problem I only run into recently: You should call srand() for each and every thread that calls rand().

In its multi-thread version, the CRT implemenation in VC++6.0 will save seed to thread local storage. So, it is necessary to seed the rand() function in every new thread your program created. Otherwise, the random number sequence is same as if seeded by 1. This poses 2 serious questions:

1. How can I write a class that is thread safe, and caller won't have to worry about calling srand() whenever they create a new thread? I do not have a solution yet. The best I can think of is to ask callers, when not sure, always call srand() whenever a new thread is created;
   
2. A second question is what if user creates multiple threads in a batch, this will make the recommended intialization code: srand( (unsigned) time(NULL)); fail, because the threads are created within 1 second, and you will get the same sequence in all your threads. After some searching, I found this discussion thread to be quite helpful: It proposed to multiply time by the thread ID to seed rand().