Wednesday, August 28, 2013

OpenCV in Visual Studio 2012

It's surprisingly easy, just took about 15 minutes to see some example code running.

Steps to create the first OpenCV project using Visual Studio 2012
  1. Download: opencv.org > OpenCV for Windows
  2. Run the downloaded executable and decompress to d:\opencv (to help the examples below)
  3. Open Visual Studio 2012, create new project  Visual C++ > Win32 > Win32 Console Application
  4. Add "Include Directories": Project Properties > Configuration Properties > C/C++ > General > Additional Include Directories, add: D:\opencv\build\include
  5. Change to static link (optional, but if this is not done, then the library below need to be changed slightly): Project Properties > Configuration Properties > C/C++ > Code Generation > Runtime Library, select Multi-threaded (Debug)
  6. Add "Library Directories": Project Properties > Configuration Properties > Linker > General > Additional Library Directories, add:D:\opencv\build\x86\vc11\staticlib (or  D:\opencv\build\x86\vc11\lib if you don't want static link)
  7. Add lib files: Project Properties > Configuration Properties > Linker > Input >Additional Dependencies, add the libraries listed below. (Note: 1. comctl32 is a Windows library; 2. replace "246" to version number you have; 3. files ending with "d" is debug version, for release configuration, remove it)
IlmImfd.lib
libjasperd.lib
libjpegd.lib
libpngd.lib
libtiffd.lib
opencv_calib3d246d.lib
opencv_contrib246d.lib
opencv_core246d.lib
opencv_features2d246d.lib
opencv_flann246d.lib
opencv_gpu246d.lib
opencv_haartraining_engined.lib
opencv_highgui246d.lib
opencv_imgproc246d.lib
opencv_legacy246d.lib
opencv_ml246d.lib
opencv_nonfree246d.lib
opencv_objdetect246d.lib
opencv_ocl246d.lib
opencv_photo246d.lib
opencv_stitching246d.lib
opencv_superres246d.lib
opencv_ts246d.lib
opencv_video246d.lib
opencv_videostab246d.lib
zlibd.lib
comctl32.lib


Now, for a quick sanity check, run one of the sample code from opencv.org (the example I used is "Hough Circle Transform" http://docs.opencv.org/doc/tutorials/imgproc/imgtrans/hough_circle/hough_circle.html
 
Environment:
  • Windows 7
  • Visual Studio 2012
  • OpenCV 2.4.6

Monday, August 26, 2013

Selenium WebDriver

Two lessons learned

  • IE driver need to start with "ignoreZoomSettings" true
  • JavascriptExecutor return for JS object

IE driver need to start with "ignoreZoomSettings" true

Got this error when using WebDriver for IE: "
org.openqa.selenium.remote.SessionNotFoundException". Turns out I was using IE to visit the same site I'm testing, and I adjusted zooming to 120% there. Fix for this problem is easy, simply change IE web driver creation to this:


DesiredCapabilities caps = DesiredCapabilities.internetExplorer();
caps.setCapability("ignoreZoomSetting", true);
driver = new InternetExplorerDriver(caps);

JavascriptExecutor return for JS object

What can be returned from JavaScriptExecutor.executeScript? According to the documentation, these are the types that it handles: Boolean, Long, String, List, or WebElement or null. I did a little experiment, and pleasantly surprise by the fact that array of JSON can be handled too. The return will be ArrayList of Maps. Calling obj.toString (Java side) will actually give me expected JSON string too.
Figure: Return Object as Shown in Eclipse Variables View

Thursday, August 22, 2013

Selenium WebDriver sendKey is very slow in IE

sendKey for IE is very slow. Turns out there is an easy fix. Simply download 32 bit IE driver (guess it's my bad used 64 bit driver before).

Environment:
Windows 7 Pro 64 bit
IE 10
Selenium 2

Saturday, August 17, 2013

Make Custom Fields in Coldfusion Solr Collections not Searchable

How

Here is how I did it, might not be the best way, but it works:

  • Update Solr schema
    • Change custom fields definition to: type: string, indexed: false
    • Comment out all the copyField definitions for the custom fields
  • Purge all Solr collections and reindex all of them
Details of the XML Changes
Changes of custom field definitions
Before:
<field name="custom1"   type="text"   indexed="true" stored="true" required="false" />
<field name="custom2"   type="text"   indexed="true" stored="true" required="false" />
<field name="custom3"   type="text"   indexed="true" stored="true" required="false" />
<field name="custom4"   type="text"   indexed="true" stored="true" required="false" />
After:
<field name="custom1"   type="string"   indexed="false" stored="true" required="false" />
<field name="custom2"   type="string"   indexed="false" stored="true" required="false" />
<field name="custom3"   type="string"   indexed="false" stored="true" required="false" />
<field name="custom4"   type="string"   indexed="false" stored="true" required="false" />
Changes for the copyField definitions
Before:
<copyField source="contents_*"  dest="contents" />
<copyField source="custom*"  dest="contents" />
<copyField source="title"  dest="contents" />
<copyField source="contents_*"  dest="contentsExact" />
<copyField source="custom*"  dest="contentsExact" />

After
<copyField source="contents_*"  dest="contents" />
<!--
<copyField source="custom*"  dest="contents" />
-->
<copyField source="title"  dest="contents" />
<copyField source="contents_*"  dest="contentsExact" />
<!-- 
<copyField source="custom*"  dest="contentsExact" />
-->
Environment
Coldfusion 10

Why


Why do I need to make custom fields not searchable?

Migrating Coldfusion collections from Verity to Solr is never straightforward. See my other post: Solr in Coldfusion 9 for some lessons I learn during migration.

Fast forward to last week (a couple of months after the production servers are migrated), another surprise surfaced. Some search will result in random text that does not appear to be relevant. After some investigation, I finally found out it's due to unexpected behavior of custom fields.

Custom field in Verity was not seem to be searchable. After migration to Solr, the custom fields are indexed and became searchable. It may be a good thing for people who want it to be searchable. But, in my case they were used for storage only, contains database ID and other contents that should not be used for search purpose.

Parting Words

It took me nearly half a day to finally figure out how to do it. Hopefully it can save someone a few hours of frustration. I started by naively changing only custom fields to index: false, and turns out they are still searchable. Google search did not turn up anything helpful, except a few posts asking the exact question I was asking. Finally after exploring the Solr admin pages, and looking at schema of the collections, I found out that they are also copied into searchable fields, and type: text will also trigger analyzer on it.

Tuesday, July 30, 2013

ExtJS multiselect problem when shift click to select entries

Environment

IE  10; ExtJS 4.1

Problem

I am using ExtJS "multiselect" control. When trying to select multiple rows using Shift+Click, I'm getting a messy screen (Figure 1). The data behind the scene is correct, but the look is disturbing. Ideally, it should look like Figure 2. 
Figure 1. Screenshot showing the problem

Figure 2. Screenshot showing the expected behavior

Solution

Solution to this problem is relatively trivial, but not obvious. Just add the following attribute to the multiselect control, you will get behavior like Figure 2. I found the solution by looking up example hosted on Sencha web site, and found the only difference is in this "ddReorder" attribute. Once it's added, the selection behavior is as expected. Drag drop reorder is harmless for what I am working on.
ddReorder: true 

Friday, July 19, 2013

Adobe Flex RSL Error in IE

This is another case which I know the cause and the solution for a problem. But I don't understand why it would cause a problem.

Environment:
  - IE10, Win7, Flash Player 11.8 debugger
  - Google Chrome  28.0 with Flash Player 11.8

Symptom:
  Flex application stopped  working in IE. Always getting this error message: "RSL Error 1 of 2". RSL is short for Runtime Shared Library. Using ServiceCapture, I do see the main SWF loaded, and the first RSL loaded, but no the rest.
  The same Flex application works perfectly in Google Chrome.

Cause:
  In the the IE Security setting, "Enable Protected Mode" is unchecked.


Solution:
  Go to IE > Tools > Internet Options > Security, make sure "Enable Protected Mode" is checked. Then restart IE. Voila, it all works again. In retrospect, I might turned it off when I was trying to set up Selenium test framework.

Question:
  It's still beyond me why "Protected Mode" seems to make IE 10 more compatible. Just a few days back I had another problem with Office Web application due to the exact same cause. An extra IE windows will always popup when I open an Office 2013 application in Excel. The problem is gone after I enabled protected mode.

----
Flex: a RIA web application development framework based on Adobe Flash technology

Tuesday, July 09, 2013

Upgrading Coldfusion 9.0.0 to 9.0.1 with HotFix 2

Instructions from Adobe are not clear or simply wrong sometimes.
Here is actual steps I took:
Environment: Windows XP, CF 9.0.0 Server Installation, IIS

  • Update Coldfusion from 9.0.0 to 9.0.1
  • Apply Cumulative Hot Fix 2 | Coldfusion 9.0.1 (http://helpx.adobe.com/coldfusion/kb/cumulative-hot-fix-2-coldfusion-1.html)
    • Download and extract files:
    • Go to your {ColdFusion-Home}/lib directory and make a backup of lib folder (I just copy and paste the folder in the same place)
    • Apply update JAR file
      • In the ColdFusion Administrator, select System Information page by clicking the "i" icon in the upper-right corner. 
      • In the Update File text box, browse and select chf9010002.jar located under CF901/lib/updates directory. 
      • Click Submit Changes. 
    • Stop ColdFusion instance. 
    • Update CFIDE
      • Go to {CFIDE-HOME} and make a backup of CFIDE folder (I just copy and paste the folder in the same place)
      • Go to extracted CFIDE folder (typically CFIDE-901\CFIDE), copy all files into current CFIDE folder and overwrite existing files
    • Update WEB-INF
      • Go to {ColdFusion-Home}/wwwroot/WEB-INF directory and make a backup of WEB-INF folder (I just copy and paste the folder in the same place)
      • Go to extracted WEB-INF folder (typically CF901\CF901\WEB-INF\WEB-INF), copy all file into the current WEB-INF folder and overwrite existing files
    • Update lib folder (it's already backed up at the begining)
      • Go to extract lib folder (typically CF901\CF901\lib), copy all files (exclude the "updates" folder) into the current lib folder and overwrite existing files
    • Start Coldfusion instance
To Uninstall: 
  • Stop Coldfusion
  • Delete CFIDE, WEB-INF and lib folders, rename the copied folders back to their original name
  • Start Coldfusion

Tuesday, April 30, 2013

Breaking change in Coldfusion 10 vs Coldfusion 9

Here is one breaking change:
  • DeserializeJSON of null value
Test Code to show the change:
result = deserializeJSON('{"test": null}');
writedump(result);
writeoutput(result.test);

In CF9, result.test is string value: "null".
In CF10, the above code will have error, because result.test is "undefined".

Debugging DotNet component on Coldfusion Server

Here is a recipe to setup Visual Studio debugging of remote Coldfusion server's .NET application.

Environment: Visual Studio 2012, Coldfusion 9
Setup:
  1. Coldfusion Server: 
    1. install "Remote Tools for Visual Studio 2012 Update 2"
    2. Start "Remote Debugger Configuration Wizard" and configure accordingly
    3. Start "Remote Debugger" As Administrator
  2. Visual Studio computer:
    1. Click Debug > Attach to Process
    2. In "Qualifier" type the Coldfusion server's computer name
    3. Check "Show processes from all users"
    4. Click "Refresh"
    5. Select "JNBDotNetSide.exe"
    6. Click "Attach"

Done!

To see it in action, set a break point in your code, load a Coldfusion page that invokes your .NET application, Visual Studio debugger should break accordingly.

Wednesday, April 03, 2013

test embedding Skydrive Spreadsheet

Above is an example to embed skydrive document to blog. Only catch? Use the "JavaScript" instead of the "Embed Code" version. It works beautifully, impressive.

Wednesday, March 20, 2013

ExtJS Store Shared Between Multiple ComboBox Controls

Environment: ExtJS 4.1.3, auto loaded store, shared by multiple comboboxes

Problem:
Combobox with auto-complete behaves strangely. Sometimes, the list is obviously filtered down by something else.

Cause:
The same data store instance is shared by multiple combobox on the page. "doQuery" function in the combobox will try to reserve the store's filter that is not created by the combobox itself (see ComboBox.js > clearFilter, and doQuery implementation). This design allows the auto-complete filter of different dropdown pickers to interfere with each other.

Solution:
1. create custom control, then override "doQuery" function to call store.clearFilter()
Pros:
- Clean, reusable design
Cons
- Validation still has cross contamination of conflicting filtering
- In cases you want to preserve an existing filter, this solution need more tweaks or configuration input to make it working

2. Declare store of the combobox using the following style: store: {type: 'myStoreAlias'}, instead of store: 'myStoreName'. This way it will ensure each combobox has their own instance of the store, thus not interfering each other's filtering.
Pros:
- No other hidden surprises lurking around
Cons:
- Pretty much have to abandon the store: 'myStoreName' completely, because in a module design, you just don't know which combobox may live in the same page with another one and using the same store
- Network and memory overhead to maintain multiple copies of the same data

Wednesday, March 13, 2013

Coldfusion Error: "Value can not be converted to requested type."

Symptom: All of a sudden the test cases that worked before, stopped working. Error message is: "Value can not be converted to requested type.". Wasted hours trying to diff and check typos with no avail. Finally, surrender, and Googled the following article published in 2006:

ColdFusion Query Error: Value Can Not Be Converted To Requested Type

http://www.bennadel.com/blog/194-ColdFusion-Query-Error-Value-Can-Not-Be-Converted-To-Requested-Type.htm

So, it is an old problem that never go away. Everything clicks now. Database schema was just updated with a few new columns. "SELECT *" combined with CFQUERYPARAM seems to have some caching issue and everything just messed up from there.

Solution? remove SELECT *, change to use explicit column names. The problem simply goes away for good.

Friday, March 01, 2013

HTML + JavaScript Charting Solutions

  • Ext JS, Example (good for intranet application)
  • d3.js (extremely popular at this moment)
  • Dojo Charting
  • YUI Chart
  • Google Chart (Big down side, at least at this time, is that Google will receive every single data point you want to plot. Not a good thing for any data that requires privacy!)
  • Flotr2
  • Flash charts? lots of good choices there, but not accessible on mobile clients, and in danger of being completely out in a few years?

Tuesday, February 19, 2013

Support JSONP in Coldfusion

Let say you already have a Coldfusion page that serves JSON data to your AJAX web application, now you want to support JSONP from cross domain client call. Here is the code snippet that will do it:

Solr in Coldfusion 9


Some tips for using Coldfusion 9 Solr search:
  • Lesson Number 1: Make sure you review ALL the files under your collections \conf folder, many of them need to be commented out or changed from default values. Here is an incomplete list of what I changed:
    • Commented out words in protwords.txt
    • Deleted content in spellings.txt
    • Updated synonyms.txt, deleted the test words, and added new synonyms based on the vocabulary specific to my application's domain
    • schema.xml:
      • Updated default query operator from AND to OR
    • solrconfig.xml
      • commented out "solr", "rocks" and other similar configurations
    • Enable term highlighting (see http://help.adobe.com/en_US/ColdFusion/9.0/Developing/WSe9cbe5cf462523a0-5bf1c839123792503fa-8000.html \), but it's still not quite there. Still looking for a complte solution for this one.
  • In Coldfusion Admin, change Solr buffer limit from default value 40 to 80 (http://bloggeraroundthecorner.blogspot.com/2009/08/tuning-coldfusion-solr-part-1.html)
  • Ensure "Coldfusion 9 Solr Service" is started and set to "Automatic" startup type
  • To access Solr collection from remote Coldfusion servers, 
    • For the remote CF servers: change Solr server name in Coldfsuion Admin, DATA & Service > Solr Server > Solr Host Name
    • For the Solr Host: Allow remote server to have access to its Solr instance. Configure Jetty to listen on all incoming IP address. Coldfusion's default configuration is allow only 127.0.0.1, thus disabling remote CF server's access to it's hosted Solr instance. To change this, just open jetty.xml in {coldfusion home}\solr\ect, and comment out the host line which restricts to listen on 127.0.0.1 only (interestingly, I found this solution from this link: http://helpx.adobe.com/coldfusion/kb/coldfusion-9-limit-access-solr.html, which claims Coldfusion's default configuration is to listen on all ip addressed
  • Web access to Solr admin: http://{Your CF Server Name}:8983/solr/
  • Last and a big one, when you try to update multiple indexes, you will see error like this:
Error_opening_new_searcher_exceeded_limit_of_maxWarmingSearchers4_try_again_later

Stack Trace:
...
org.apache.solr.common.SolrException: Error_opening_new_searcher_exceeded_limit_of_maxWarmingSearchers4_try_again_later

Error_opening_new_searcher_exceeded_limit_of_maxWarmingSearchers4_try_again_later

request: http://localhost:8983/solr/.../update?commit=true&waitFlush=false&waitSearcher=false&wt=javabin&version=1
    at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:424)
    at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:243)
    at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)
    at coldfusion.tagext.search.SolrUtils.commitToServer(SolrUtils.java:1024)
    at coldfusion.tagext.search.SolrUtils.addDocument(SolrUtils.java:664)
    at coldfusion.tagext.search.IndexTag.doQueryUpdate(IndexTag.java:929)
    at coldfusion.tagext.search.IndexTag.doStartTag(IndexTag.java:254)
    at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2722)
…

This error will be consistently showing up in production environment with even very light traffic, and became a big show stopper for the usage of Solr collection provided by Coldfusion (does not seem to be a Solr problem, instead a problem of Coldfusion integration problem). 

Root cause seems to be that Coldfusion commits every index update, and causing excessive Solr searcher warming. Combining with misconfigured default Solr configuration, this problem starts to appear for a collection of only a few hundred entries.

In Coldfusion 10, this problems seems like easier to solve by using the newly introduced "autoCommit" attribute, which should always be "no" (i.e. never use the default value). Then I guess configuring the "autoCommit" in solr config file will solve the above problem in Coldfusion 10.

However, production environment that is already in CF 9 is stuck. My solution for it is a combination attack:
    • Manually throttle cfindex update by adding a sleep after every update
    • Wrap try catch block around the cfindex update, then add a longer sleep time in the catch block, and add retry logic to try update again after the sleep
    • Database flag of index entries, this flag will be set only after index is updated successfully. A scheduled CF task will scan this flag to identify missed updates (due to the above error), and try to update index again
The above approach so far allows me to update thousands of indexes without any problem.

Wednesday, October 10, 2012

Downloadify Stopped Working in Google Chrome

Downloadify uses Flash's download functionality to enable client side download. It saves server side roundtrip.  There is also nice user extensions using Downloadify to export ExtJS grid on the client side. HTML 5 does have some limited support of client side download, but it suffers inconsistent support between browsers.

Ok, after all the justification for using it, Downloadify stopped working for Google Chrome recently. And here are the observations:
  • Environment: Windows, Google Chrome is up-to-date
  • Download still works in IE and Firefox;
  • Download only works in Chrome if I'm testing using Intranet URL. But it does not work if the URL looks like Internet address (i.e. http://localhost works, but http://10.10.10.1 won't work);
  • IE, Firefox are on Flash 11.3, and Chrome is running Flash 11.4
Solution:
Not really a solution. It's more like a workaround.
  1. In Chrome address bar type: "about:plugins"
  2. Click on "Details" button to expand into more detailed view
  3. Find "Flash" section, and disable the Flash player 11.4 (something called "PepperFlash"\"pepflashplayer.dll")
  4. Make sure there is another Flash player listed here, which should be Adobe's distribution

Root Cause?
I can only guess. Still not sure why and how to fix it properly.
Turns out Google Chrome has it's own Flash distribution. (Only guessing) It seems to be different from Adobe's official one, and somehow has higher default security settings.

itemId and id in ExtJS

The story: 
Here is a lesson for not reading document thoroughly. My very first ExtJS project goes smoothly until one day I created multiple tabs, and then closed some of them. All of sudden the layout completely messed up. In the debug console, there is null reference error. Call stack is deep in the ExtJS library.

Fast forward, after several hours of digging around, I found out I was using "id" config for several ExtJS controls. Then, close tab action will cause all the controls on the other tabs with the same "id" got deleted from DOM.

Solution:
Use "itemId" instead of "id" config for controls.

Comparison to ASP.NET
Came from ASP.NET world, I guess that is why I was using "id" config without any second thought. Below is a comparison of similar concepts in ASP.NET

ExtJS ASP.NET _
itemId  ID  (Recommended) HTML element's ID is automatically generated to ensure uniqueness, usually by concatenating ID of the container hierarchy. Programmer only need to ensure ID is unique within it's container.
id  ClientID  (Not recommended) Rendered HTML element will use this property as their ID. It's programmer's responsibility to ensure it is unique across the whole page (think when you starting to have multiple instances of the same control on a page).

(To check out more ExtJS related posts from me, click here: http://developertips.blogspot.com/search/label/ExtJS)

Monday, March 12, 2012

ASP.NET Exception: "Session state has created a session id, but cannot save it ..."

Symptom:
  • Only happens for new session
  • Exception Message:
    A first chance exception of type 'System.Web.HttpException' occurred in System.Web.dll Additional information: Session state has created a session id, but cannot save it because the response was already flushed by the application.
  •  Stack Trace: 
[HttpException (0x80004005): Session state has created a session id, but cannot save it because the response was already flushed by the application.]   System.Web.SessionState.SessionIDManager.SaveSessionID(HttpContext context, String id, Boolean& redirected, Boolean& cookieAdded) +3691007   System.Web.SessionState.SessionStateModule.DelayedGetSessionId() +199   System.Web.SessionState.SessionStateModule.ReleaseStateGetSessionID() +25   System.Web.SessionState.SessionStateModule.OnReleaseState(Object source, EventArgs eventArgs) +874   System.Web.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +80   System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +270 

Cause:
  • I was able to narrow down the cause of this problem to setting page's Buffer="false"
  • Reasoning would be that without buffering, all content are pushed to client immediately, and somehow, ASP.NET decided to set cookie at a very late stage where it can no longer write to HTTP header to set SessionID cookie
Solution:
Set Buffer to true is not a solution for me, because I will need to flush content to deliver real-time progress information.
Based on some understanding, a solution quickly came up with some Google search. Simply add the following line to Global.asax.cs > Session_Start, the problem is resolved.
string sessionID = Session.SessionID;

Wednesday, February 15, 2012

Faster Flex Builder Compile Time

To increase Flex Builder compile time during development, the following tweak helped my Flex 3 builder to go from 30 seconds to just a few seconds for a medium sized project.
Open project property, under Flex Compiler -> Additional compiler arguments, add " -incremental -keep-generated-actionscript"

It definitely helps me go from compile once every hour, to every few minutes. Such that I can get immediate feedback through compiler errors or warnings.