Lucas kanade vlib on c6678

Joseph,

Can you provide details on the problems/trouble you see when trying to combine them?  Can you post your project? 

Regards,

Travis

Hi travis,

thank you for the quick response. First, I want to verify that I have the right code for the LK. Since the documentation is non existent I need to clarify a few points. I get a gray image directly from a hardware source so I only need to apply the LK. My image is 720x288 and I get 100 points to track :

  static unsigned char input_field[720*288];
  static unsigned char previous_input_field[720*288];
  static S16    GRADX[720 * 288];
  static S16    GRADY[720 * 288];
  static S16    GRADMAG[720 * 288];
     
  static S16 X[100];
  static S16 Y[100];
  static S16 mX[100];
  static S16 mY[100];
  static U08    BUFFER[384];
  
  for (int i = 0; i < nTrackedPoints; i++)
  {
    mX[i] = X[i];
    mY[i] = Y[i];
  }
   
  VLIB_xyGradientsAndMagnitude(input_field, GRADX + 720 + 1, GRADY + 288 + 1, GRADMAG + 720 + 1, 720, 288);
  VLIB_trackFeaturesLucasKanade_7x7(input_field, previous_input_field, GRADX, GRADY, 720, 288, nTrackedPoints, X, Y, mX, mY, 10, BUFFER);

The questions I have are the following:

Q1. The BUFFER size as indicated in the documentation needs to have a size of 384. Is that correct, irrespectively of the image size?

Q2. In the test sample for the LK the gradient and magnitude is only calculated for the first arguement i.e. supplied image. Is that correct?

Q3. Does the first argument passed in the LK function corresponds to the current frame and the second to the previous frame or vice versa?

Q4. Regarding the call to VLIB_xyGradientsAndMagnitude. What is the correct size for GRADX, GRADY, GRADMAG. I have used the total image size. Is that correct. If not what would be the correct size for each for an image of 720 by 288 .

Q5. The 2nd, 3rd and 4th arguement passed to xyGradientsAndMagnitude in the LK test case is GRADX+720, GRADY+288 and GRADMAG+ 720, but in the test case for the xyGradientsAndMagnitude itself is just GRADX, GRADY, GRADMAG. Which is the correct way to call that function (especially for GRADMAG).

Q6. In the test case for the LK function the points passed are in the form of :

    S16    X[NFEATURES] = { 15.3f * 16, 25.2f * 16, 49.0f * 16, 20.2f * 16 };

Why do they need to be multiplied by 16. My points are not floating points numbers?

Regarding the pyramid implementation I have 3 more questions. The code I am using is something like this (I have made changes for clarity):

  U08    calculatedOutData_Pyramid8[360 * 144 + 180 * 72 + 90 * 36];
  VLIB_imagePyramid8(input_field,  720, 288,  calculatedOutData_Pyramid8);
  U08* pCalculatedOutData_Pyramid ;
  pCalculatedOutData_Pyramid = calculatedOutData_Pyramid8;
  
  S32 piramidSizeX = 90;
  S32 piramidSizeY = 36;
  pCalculatedOutData_Pyramid = pCalculatedOutData_Pyramid + (720*288);
  pCalculatedOutData_Pyramid_previous = pCalculatedOutData_Pyramid_previous + (720*288);
  
  //Image Pyramid processing
  for (int i = 0; i < 3; i++)
  {
      pCalculatedOutData_Pyramid = pCalculatedOutData_Pyramid - (piramidSizeX*piramidSizeY);
      pCalculatedOutData_Pyramid_previous = pCalculatedOutData_Pyramid_previous - (piramidSizeX*piramidSizeY);
      VLIB_xyGradientsAndMagnitude(*pCalculatedOutData_Pyramid, GRADX + piramidSizeX + 1, GRADY +   piramidSizeY + 1, GRADMAG + piramidSizeX + 1, piramidSizeX, piramidSizeY);
      VLIB_trackFeaturesLucasKanade_7x7(*pCalculatedOutData_Pyramid, *pCalculatedOutData_Pyramid_previous, GRADX, GRADY, piramidSizeX, piramidSizeY, nTrackedPoints, X, Y, mX, mY, 10, BUFFER);
   
      piramidSizeX= piramidSizeX*2;
      piramidSizeY= piramidSizeY*2;
      for (int i = 0; i < nTrackedPoints; i++)
      {
         X[i] =mX[i];
         Y[i] = mY[i]; 
      }

}
  pCalculatedOutData_Pyramid_previous = pCalculatedOutData_Pyramid;
  
  //Actual Image
  VLIB_xyGradientsAndMagnitude(input_field, GRADX + 720 + 1, GRADY + 288 + 1, GRADMAG + 720 + 1, 720, 288);
  VLIB_trackFeaturesLucasKanade_7x7(input_field, previous_input_field, GRADX, GRADY, 720, 288, nTrackedPoints, X, Y, mX, mY, 10, BUFFER);

Q7. The image pyramid function outputs 3 pyramids. Do I need to call the LK 4 times like I do in the code above?

Q8. I am not sure where to plugin the mX,mY output from each stage (loop iteration). Its not clear from the test case.

Q9. In the first call to the LK I am supplying points calculated from a frame of size 720x288 but the input image is the output of the pyramid algorithm i.e. coordinates will not match. That does not sound correct. What would be the way to do it i.e. calculate the points on the equivalent pyramid level (calculate for corresponding points on each pyramid image).

Thanks again for your response. Is there any chance of getting something that may resemble a user guide for this library. I have seen references to SPRU00 but I am not able to locate it. I demo for the pyramid would also be helpfull.

Regards

Joseph

Joseph,

I am looking into this, and will send you an update later today with whatever I find. Thanks for waiting.

Sudhakar

Joseph,

Here are some answers to your questions.

1. This is correct. The last argument 'BUFFER' to the function VLIB_trackFeaturesLucasKanade_7x7() should be 384 bytes irrespective of the image size, and two byte aligned.

2. The gradient and magnitude are calculated for the first argument, and it is the previous frame. I will anyway confirm this with development team and let you know if I am wrong.

3. The first argument is the previous frame, and the second argument is current frame. X and Y are known feature co-ordinates of previous frame (first argument), and mX and mY are unknown feature co-ordinates of current frame (second argument). I will let you know if I am wrong.

4. On calling VLIB_xyGradientsAndMagnitude(), output GRADX, GRADY and GRADMAG will be updated for width * (height - 2) pixels. For 720 by 288 input image, you must use sizes of at least 720 * 286 for GRADX, GRADY, GRADMAG when you pass GRADX, GRADY and GRADMAG as arguments to avoid out of range memory accesses. For the last argument, passing height - 1 is safer to avoid out of range memory access for calculating GRADY and GRADMAG.

5. VLIB_xyGradientsAndMagnitude() will process input image starting from pixel at row two, column two (pixel input_field[width + 1]), to find GRADX, GRADY and GRADMAG. This is done for width * (height - 2) pixels starting from input_field[width + 1]. So, the first GRADMAG value found is for input_field[width + 1]. This is assigned to the first location of GRADMAG passed, i.e., GRADMAG[0]. Instead, if you want this to be assigned to GRADMAG[width + 1] so that input_field and GRADMAG pixels are in correspondance, pass GRADMAG[width + 1] as argument, and assignment will start from GRADMAG[width + 1]. This is same for GRADX and GRADY. So, passing GRADX+720+1, GRADY+720+1, GRADMAG+720+1 is a better option (please note that you are passing GRADMAG+720+1 and not GRADMAG+288+1). But in this case, 720*286 pixels are updated from GRADMAG+720+1. So, GRADX, GRADY and GRADMAG can be defined with sizes of 720 * 288, just like input_field. Also, passing height - 1 instead of height (last argument) to this function is safe to avoid out of range memory access.

6. I will check this with the development team and get back to you.

7. The VLIB_imagePyramid8() function generates a pyramid of three subsampled images. The largest subsampled image (Level 1) is stored first, followed by smaller ones. In our case, first 360*144 pixels correspond to first image generated. The next 180*72 pixels form the smaller image. The smallest image is the last 90*36 pixels in the output.
You need to call KL tracker three times if you are doing a tracking on every image of the pyramid.
I have made changes to your implementation to give you an idea of how it can be used:

U08    calculatedOutData_Pyramid8[360 * 144 + 180 * 72 + 90 * 36];
VLIB_imagePyramid8(input_field,  720, 288,  calculatedOutData_Pyramid8);
U08* pCalculatedOutData_Pyramid ;
pCalculatedOutData_Pyramid = calculatedOutData_Pyramid8;
 

S32 piramidSizeX = 360; // The first image of the image pyramid is of size ((width/2)*(height/2))
S32 piramidSizeY = 144;
// pCalculatedOutData_Pyramid = pCalculatedOutData_Pyramid + (360 * 144 + 180 * 72 + 90 * 36); // Not sure why you needed this. I am commenting it
// pCalculatedOutData_Pyramid_previous = pCalculatedOutData_Pyramid_previous + (360 * 144 + 180 * 72 + 90 * 36); // I am commenting this as well
 
//Image Pyramid processing
for (int i = 0; i < 3; i++)
{     

VLIB_xyGradientsAndMagnitude(*pCalculatedOutData_Pyramid_previous, GRADX + piramidSizeX + 1, GRADY +   piramidSizeX + 1, GRADMAG + piramidSizeX + 1, piramidSizeX, piramidSizeY);
VLIB_trackFeaturesLucasKanade_7x7(*pCalculatedOutData_Pyramid_previous, *pCalculatedOutData_Pyramid, GRADX, GRADY, piramidSizeX, piramidSizeY, nTrackedPoints, X+i*nTrackedPoints, Y+i*nTrackedPoints, mX+i*nTrackedPoints, mY+i*nTrackedPoints, 10, BUFFER);

pCalculatedOutData_Pyramid = pCalculatedOutData_Pyramid + (piramidSizeX*piramidSizeY); // Move to next smaller image in the pyramid
pCalculatedOutData_Pyramid_previous = pCalculatedOutData_Pyramid_previous + (piramidSizeX*piramidSizeY);
    
piramidSizeX = piramidSizeX/2;
piramidSizeY = piramidSizeY/2;
for (int j = 0; i < nTrackedPoints; i++) // update for next frame
    {
     X[i*nTrackedPoints+j] = mX[i*nTrackedPoints+j];
     Y[i*nTrackedPoints+j] = mY[i*nTrackedPoints+j];
    }
}
pCalculatedOutData_Pyramid_previous = pCalculatedOutData_Pyramid;

8. As per changes made above, if you are using nTrackedPoints number of features on every image of pyramid, then total features you use per frame (input_field) is 3*nTrackedPoints. So, you can update first nTrackedPoints in first iteration of the for loop above, and so on. So, size of X, Y, mX and mY is 3*nTrackedPoints. Please see above call to VLIB_trackFeaturesLucasKanade_7x7() where I have added an offset to X, Y, mX, mY.

9. In the first call, you get co-ordinates of features mX and mY for a 720 x 288 image. You need to find corresponding co-ordinates of three subsampled images that you get from pyramid before you pass them to VLIB_trackFeaturesLucasKanade_7x7() with pyramid images as inputs. You can find the co-ordinates and store them in an array X of size 3*nTrackedPoints like I have done above. You can find corresponding co-ordinates as follows:

Say X_input_field and Y_input_field are feature co-ordinates of input image.
Define X, Y, mX, mY of size 3*nTrackedPoints each
int i;
for(i=0; i<nTrackedPoints; i++) {
X[i] = X_input_field[i] >> 1; // Level one subsampling - input image is halved in width and height
Y[i] = Y_input_field[i] >> 1;
X[nTrackedPoints + i] = X_input_field[i] >> 2; // Level two subsampling
Y[nTrackedPoints + i] = Y_input_field[i] >> 2;
X[2*nTrackedPoints + i] = X_input_field[i] >> 3; // Level three subsampling
Y[2*nTrackedPoints + i] = Y_input_field[i] >> 3;
}
The X and Y can now be used to call KL function three times as I did above.

Thank you.

Sincerely,

Sudhakar Ayyasamy

Hi Sudhakar,

thank you for responding to my questions. I went through what you said and basically what I understood can be summarized in this piece of code:

  U08 calculatedOutData_Pyramid8[360 * 144 + 180 * 72 + 90 * 36];

  S32 piramidSizeX = 720;   S32 piramidSizeY = 288;   

VLIB_imagePyramid8(input_field,  piramidSizeX, piramidSizeY,  calculatedOutData_Pyramid8);   

U08* pCalculatedOutData_Pyramid ;

  pCalculatedOutData_Pyramid = calculatedOutData_Pyramid8;

  VLIB_xyGradientsAndMagnitude(input_field_previous, GRADX + piramidSizeX + 1, GRADY + piramidSizeX + 1, GRADMAG + piramidSizeX + 1, piramidSizeX, piramidSizeY-1);
  VLIB_trackFeaturesLucasKanade_7x7(input_field_previous, input_field, GRADX, GRADY, piramidSizeX, piramidSizeY, nTrackedPoints, X, Y, mX, mY, 10, BUFFER);  

int i;

  for (i = 0; i < 3; i++)

  {

   piramidSizeX= piramidSizeX/2;

   piramidSizeY= piramidSizeY/2;

   for (int j = 0; j < nTrackedPoints; j++) {     

X[i*nTrackedPoints+j] = mX[i*nTrackedPoints+j] >> 1;

    Y[i*nTrackedPoints+j] = mY[i*nTrackedPoints+j] >> 1;

            }

   VLIB_xyGradientsAndMagnitude(*pCalculatedOutData_Pyramid, GRADX + piramidSizeX + 1, GRADY + piramidSizeX + 1, GRADMAG + piramidSizeX + 1, piramidSizeX, piramidSizeY-1);    VLIB_trackFeaturesLucasKanade_7x7(*pCalculatedOutData_Pyramid_previous, *pCalculatedOutData_Pyramid, GRADX, GRADY, piramidSizeX, piramidSizeY, nTrackedPoints, X+i*nTrackedPoints, Y+i*nTrackedPoints, mX+i*nTrackedPoints, mY+i*nTrackedPoints, 10, BUFFER);  

  }

  //i=2

  for (int j = 0; j < nTrackedPoints; j++)

        {

   mX[i*nTrackedPoints+j] = mX[i*nTrackedPoints+j] << 3;    

mY[i*nTrackedPoints+j] = mY[i*nTrackedPoints+j] << 3;         }

     pCalculatedOutData_Pyramid_previous = pCalculatedOutData_Pyramid;

 input_field_previous = input_field;

So I have 3 questions.

Q1. Does the code above matches what you meant in your previous response(Basically 4 calls to LK with the inputs shown above and 3*nTracked points)? If yes, then my mX,mY need to be converted back to the original image coordinates in order to be usable (as shown in the code above), is that correct?

Q2. Once I have the mX, mY vectors I need to filter them so that I can reject the points that are not trackable any more. Currenlty, I filter negative values and values that fall outside of the window 720-7-1, 288-7-1. Obviously there is another type of check which I am not performing. Is there another function call that I can use in order to filter those 2 vectors (I have asked that question in another post too) and exclude the points that can't be tracked. If not what do I need to filter against?

Q3. There is another function for this algorithm called VLIB_trackFeaturesLucasKanade_7x7_track_error. This returns a track_error array. Can this be used to check the sanity of the returned points? If yes what is the threshold beyond which a point can be disgarded.

If this tracker performs as expected we will probably go on with this project. But since our algorithm has more stages than a simple LK it will involve more vlib functions. We also want to evaluate ORB. Is there any reference manual that explains this library in more depth. Is SPRUG00 the same as the user guide found in the installation. If not can we get this document?

Thanks again

Regards

Joseph

Joseph,
Regarding Q1, the first call to VLIB_trackFeaturesLucasKanade_7x7() function (before the for loop) uses X, Y points of input_field_previous to find mX, mY of input_field. I would suggest you keep mX, mY for next frame.
Now, pCalculatedOutData_Pyramid is the image pyramid for input_field. This will have three images, so total feature points is 3*nTrackedPoints in addition to mX and mY. For this, let's say we have arrays mX_pyramid, mY_pyramid each of size 3*nTrackedPoints. The first nTrackedPoints of these arrays are the feature points of largest image at level 1 in the pyramid, and last nTrackedPoints are the feature points of smallest pyramid image. So, you can keep (mX, mY) and (mX_pyramid, mY_pyramid) separate.
Each (mX,mY) will correspond to three points in the (mX_pyramid,mY_pyramid). That is, for example, mX_pyramid[i], mX_pyramid[nTrackedPoints+i], mX_pyramid[2*nTrackedPoints+i] will correspond to mX[i].
You can directly find the feature points (mX_pyramid, mY_pyramid) of each image of the pyramid from (mX, mY).
Here is a rough implementation:

U08 calculatedOutData_Pyramid8[360 * 144 + 180 * 72 + 90 * 36];
S32 piramidSizeX = 720;   S32 piramidSizeY = 288;   

while(input_field!=null) {
    VLIB_imagePyramid8(input_field,  piramidSizeX, piramidSizeY,  calculatedOutData_Pyramid8);   
    U08* pCalculatedOutData_Pyramid ;
    pCalculatedOutData_Pyramid = calculatedOutData_Pyramid8;

    VLIB_xyGradientsAndMagnitude(input_field_previous, GRADX + piramidSizeX + 1, GRADY + piramidSizeX + 1, GRADMAG + piramidSizeX + 1, piramidSizeX, piramidSizeY-1);
    VLIB_trackFeaturesLucasKanade_7x7(input_field_previous, input_field, GRADX, GRADY, piramidSizeX, piramidSizeY, nTrackedPoints, X, Y, mX, mY, 10, BUFFER);

    // Get feature points for pyramid images from mX, mY
    for(i=0; i<nTrackedPoints; i++) {
        mX_pyramid[i] = mX[i] >> 1; // Feature points of level 1 image.
        mY_pyramid[i] = mY[i] >> 1;
        mX_pyramid[nTrackedPoints + i] = mX[i] >> 2; // Feature points of level 2 image
        mY_pyramid[nTrackedPoints + i] = mY[i] >> 2;
        mX_pyramid[2*nTrackedPoints + i] = mX[i] >> 3; // Feature points of level 3 image
        mY_pyramid[2*nTrackedPoints + i] = mY[i] >> 3;
    }

    // Other required operations using pCalculatedOutData_Pyramid and mX_pyramid,my_pyramid

    for(i=0; i<nTrackedPoints; i++) { // make current frame previous
        X[i] = mX[i];
        Y[i] = mY[i];
    }

    input_field_previous = input_field;
}

This avoids the use of for loop to compute features from pCalculatedOutData_Pyramid_previous using KL function.
I believe your goal is finding the feature points mX,mY and mX_pyramid,mY_pyramid for every frame. If yes, you can use the above method with one call to KL function for each frame. Please let me know why you want to use four calls to KL using pCalculatedOutData_Pyramid_previous to calculate the same values.
Thanks.

Sudhakar

Dear Sudhakar,

lets suppose that I have a large displacement between my input_field and input_field_previous. If I apply the LK on those 2 frames I will probably end up with no points. Then what? Why would I need a pyramid in the first place if I could just half the points.It does not make sense. Unless the arguments to the LK are different.

I don't think there is a point in this discussion anymore. I HAVE WASTED 10 DAYS IN THIS FORUM trying to get a simple answer on how to call 3 functions. Something that should have been in a document in the first place. Since I cannot afford to invest more time in this endeavour, I will make a final request for a WORKING sample on how to apply the LK in an image pyramid, and filter the output points accordingly. I wouldn't normally ask for sample code but since the documentation is non existent I see no other option to figure out how to use this library. If you can't supply me with that, we will just move to alternative solutions and certainly away from TI. Thank you again for your time. I apologise for being impolite but 10 days in this forum and 20 days before is a lot of time. 

In case you are allowed to send me a sample and a guide, my company email can be found in my forum account.

Regards

Joseph

Joseph,

I apologize for the inconvenience caused as we currently have very few people who have worked on VLIB and they are currently unavailable. I will update you regarding the filtering option you are looking for and provide more information about what VLIB_trackFeaturesLucasKanade_7x7_track_error() performs once they are back next week. I also will check with them on working samples that combine LK and image pyramid functions.

From your response, I understand that you are looking for a way to track points with larger displacements between two frames. In this case, we will start by tracking those points at level 3 of pyramid, and go up to level 1, then finally to original resolution to fine tune the features. If this is what you are looking for, here is a rough implementation that you can try to see if it works on your side.

while(input_field!=NULL) {
	U08 calculatedOutData_Pyramid8[360 * 144 + 180 * 72 + 90 * 36];
	S32 piramidSizeX = 720;   S32 piramidSizeY = 288;
	VLIB_imagePyramid8(input_field,  piramidSizeX, piramidSizeY,  calculatedOutData_Pyramid8);   
	U08* pCalculatedOutData_Pyramid;

	/* Initially point to the end of pyramids */
	pCalculatedOutData_Pyramid = calculatedOutData_Pyramid8 + [360 * 144 + 180 * 72 + 90 * 36];
	pCalculatedOutData_Pyramid_previous = pCalculatedOutData_Pyramid_previous + [360 * 144 + 180 * 72 + 90 * 36];

	/* We will start from the smallest image (deepest level) in pyramid, and go towards original resolution */
	piramidSizeX = 90;
	piramidSizeY = 36;

	int i;
	/* X,Y will have feature co-ordinates of input_field_previous. mX, mY are to have feature co-ordinates of input_field.
	Since we are starting with deepest level in pyramid, the initial estimates of mX, mY are same as the features of level 3 image of input_field_previous
	These will be updated on first call to  VLIB_trackFeaturesLucasKanade_7x7 */
	for(i=0; i<nTrackedPoints; i++){
		mX[i] = X[i] >> 3;
		mY[i] = Y[i] >> 3;
	}

	/* Three iterations starting from level 3 of pyramid up to level 1 */
	for (i = 3; i > 0; i--) {
		/* X_pyramid, Y_pyramid will have feature co-ordinates of level i of input_field_previous */  
		for (int j = 0; j < nTrackedPoints; j++) {     
			X_pyramid[j] = X[j] >> i;
			Y_pyramid[j] = Y[j] >> i;
		}

		/* Move pointers to the beginning of level i image of the pyramid */
		pCalculatedOutData_Pyramid = pCalculatedOutData_Pyramid - (piramidSizeX * piramidSizeY);
		pCalculatedOutData_Pyramid_previous = pCalculatedOutData_Pyramid_previous - (piramidSizeX * piramidSizeY);

		/* Estimates mX, mY are updated at level i. Input features are X_pyramid, Y_pyramid */		
		VLIB_xyGradientsAndMagnitude(pCalculatedOutData_Pyramid_previous, GRADX + piramidSizeX + 1, GRADY + piramidSizeX + 1, GRADMAG + piramidSizeX + 1, piramidSizeX, piramidSizeY-1);
		VLIB_trackFeaturesLucasKanade_7x7(pCalculatedOutData_Pyramid_previous, pCalculatedOutData_Pyramid, GRADX, GRADY, piramidSizeX, piramidSizeY, nTrackedPoints, X_pyramid, Y_pyramid, mX, mY, 10, BUFFER);

		/* Update pyramid image size for next iteration */
		piramidSizeX = piramidSizeX << 1;
		piramidSizeY = piramidSizeY << 1;

		/* mX, mY refined at this level i are scaled to become estimates for next iteration */ 
		for (int j = 0; j < nTrackedPoints; j++) {     
			mX[j] = mX[j] << 1;
			mY[j] = mY[j] << 1;
		}
	}

	/* Refine mX,mY fourth time for original resolution for fine tuning */
	VLIB_xyGradientsAndMagnitude(input_field_previous, GRADX + piramidSizeX + 1, GRADY + piramidSizeX + 1, GRADMAG + piramidSizeX + 1, piramidSizeX, piramidSizeY-1);
	VLIB_trackFeaturesLucasKanade_7x7(input_field_previous, input_field, GRADX, GRADY, piramidSizeX, piramidSizeY, nTrackedPoints, X, Y, mX, mY, 10, BUFFER);
	/* mX, mY will now have features for input_field */

	/* Current frame and its features become previous */	
	pCalculatedOutData_Pyramid_previous = pCalculatedOutData_Pyramid;
	input_field_previous = input_field;
	for(i=0; i<nTrackedPoints; i++){
		X[i] = mX[i];
		Y[i] = mY[i];
	}
}

I am also attaching VLIB API Reference Guide that we have for VLIB 2.1. 5001.[SPRUG00C] VLIB_2_1_API.pdf

Thank you.

Sincerely,

Sudhakar Ayyasamy

Joseph,

Here are few confirmations:

Q2. In the test sample for the LK the gradient and magnitude is only calculated for the first arguement i.e. supplied image. Is that correct? (Posted on Feb 27)
The gradient and magnitude are calculated for the first argument, and it is the previous frame.

Q3. Does the first argument passed in the LK function corresponds to the current frame and the second to the previous frame or vice versa? (Posted on Feb 27)
The first argument is the previous frame, and the second argument is current frame. X and Y are known feature co-ordinates of previous frame (first argument), and mX and mY are unknown feature co-ordinates of current frame (second argument).

Q3. There is another function for this algorithm called VLIB_trackFeaturesLucasKanade_7x7_track_error. This returns a track_error array. Can this be used to check the sanity of the returned points? If yes what is the threshold beyond which a point can be disgarded. (Posted on Mar 4)
VLIB_trackFeaturesLucasKanade_7x7_track_error() can be used to filter out bad feature points (ie not trackable or went out of scene). Error given out by this API is the sum of absolute differences of corresponding pixels in two windows. Window 1 is in image1 centered at previous location of feature point and window 2 is present in image2 centered at new location of feature point. You can choose appropriate threshold for this to discard a bad feature point. Window size here is 7x7. track_error array length is equal to the number of feature points.

Thank you.

Sincerely,
Sudhakar Ayyasamy

Hi Sudhakar,

three points that I need to clarify from your previous answer:

P1. for the basic function (VLIB_trackFeaturesLucasKanade_7x7) there is no other way to discard 'bad' points other than negative values and coordinates that fall outside the image , is that correct? Many other libraries use min eigen values as an error measure. I suppose that cannot be used.

P2. If SAD is used in the trackFeaturesLucasKanade_7x7_track_error what is a reasonable value for a threshold in this case.

Thank you responding

Regards

Joseph

Joseph,

Here is what I have now:

P1. A popular method to drop bad features is defined as par to of KLT algorithm, which is based on affine transform of rectangular window at final location. This feature of dissimilarity measure of feature points is currently under development as part of VLIB library. As per current situation the user can only use trackFeaturesLucasKanade_7x7_track_error to get translation errors.

P2.  We don't have a fixed/reasonable value for threshold as we have left it for the user to find that based on use case.

Additional information based on past queries:
Q6. In the test case for the LK function the points passed are in the form of :
S16    X[NFEATURES] = { 15.3f * 16, 25.2f * 16, 49.0f * 16, 20.2f * 16 };
Why do they need to be multiplied by 16. My points are not floating points numbers? (Posted on Feb 27)
The values X, Y, mX, mY (feature point co-ordinates of previous frame and current frame) as passed as 16 bit integer values, but are processed as SQ11.4 values. This format is mentioned in the header file containing the function's prototype. This means that of the 16 bits, bits 0-3 (LSB) are fraction bits, bits 4-14 are integer bits, bit 15 (MSB) is sign bit. The sign has no significance as pixel locations are always positive. Fraction bits are used to provide fractional pixel accuracy to feature locations. Accuracy here is 1/16. So, if you want to pass an X value 5 to the function, this must occupy bits 4-14 in the the value you pass as argument. So you can multiply by 16 so that you actually pass 0x0050 as four LSB bits are for fraction.

Thank you.

Sincerely,
Sudhakar Ayyasamy

Hi Sudhakar,

Finally a worthy explanation for a VLIB feature! Thanks!

I have some open issues regarding the feature tracking:

The outX and outY buffers hold the new pixel coordinates in shift by 4 format? Meaning, I need to divide the result by 16 to get the accurate result?

I have this part of a program where I run Canny edge detect on an image, then send the results to the lucas-kanade feature tracker, the results I get from outX and outY are mostly 0 (after subtracting from the original location) but I know for sure that there is a motion between the frames).

Is there something I'm doing wrong?

P.S- I tried using harris score but it didn't find the correct corners.

Void TrackObjects(UInt8* A, UInt8* B, Int32 width, Int32 height, Int32 XShift, Int32 YShift)
{
    Int32 i,j,numItems = 0;
	Int32 scratchSize = 160*width;
    // initialize memory
    memset(pBufGradX, 0, width * height * sizeof(Int16));
    memset(pBufGradY, 0, width * height * sizeof(Int16));
    memset(pBufMag,   0, width * height * sizeof(Int16));
    memset(pBufOut,   0, 200000/*(width * height * 6 + scratchSize) * sizeof(UInt8)*/);

    Int16* gradX  = pBufGradX;
    Int16* gradY  = pBufGradY;
    Int16* gradM = pBufMag;
    UInt8* scratch = pBufOut;
    Int16* X = 		(Int16*)&pBufOut[(scratchSize)*sizeof(Int16*)];
    Int16* Y = 		(Int16*)&pBufOut[(scratchSize+(width*height))*sizeof(Int16*)];
    Int16* outX = 	(Int16*)&pBufOut[(scratchSize+(width*height*2))*sizeof(Int16*)];
    Int16* outY = 	(Int16*)&pBufOut[(scratchSize+(width*height*3))*sizeof(Int16*)];
    Int16* OUTM = 	(Int16*)&pBufOut[(scratchSize+(width*height*4))*sizeof(Int16*)];
    UInt8* OUT = 	(UInt8*)&pBufOut[(scratchSize+(width*height*5))*sizeof(Int16*)];

    DoCanny(A, width, height, (Int16*)OUT, high, low);
    for(i=3;i<32;i++)
    {
    	for(j=6;j<36;j++)
    	{
        	if(OUT[i*width+j]==255)
    		{
    			X[numItems]=j<<4;
    			Y[numItems]=i<<4;
    			outX[numItems]=X[numItems]+XShift;
    			outY[numItems]=Y[numItems]+YShift;
    			numItems++;
    		}
    	}
    }
    VLIB_trackFeaturesLucasKanade_7x7(A, B, gradX, gradY, width,
    		height, numItems, X, Y, outX, outY, LK_MAX_ITER, scratch);

    for(i=0;i<height;i++)
    {
    	for(j=0;j<width;j++)
    	{
        	if(outX[i*width+j]>=0 && outY[i*width+j]>=0)
    		{
    			outX[i*width+j]=outX[i*width+j]>>4-X[i*width+j]>>4;
    			outY[i*width+j]=outX[i*width+j]>>4-Y[i*width+j]>>4;
    		}
        	else
        	{
    			outX[i*width+j]=0;
    			outY[i*width+j]=0;
        	}
    	}
    }
}

int DoCanny(UInt8 *img, UInt16 width, UInt16 height, Int16 *cannyResult, Uint8 hi, Uint8 lo)
{
    Int32   *numItems;
    numItems  = (Int32 *) memalign(32, sizeof(numItems));
    // initialize memory
    memset(pBufGradY, 0, width * height * sizeof(Int16));
    memset(pBufMag,   0, width * height * sizeof(Int16));
    memset(pBufOut,   0, width * height * sizeof(UInt8));

    Uint8* buffOut = pBufOut + width * (CONV_WIND_SIZE - 1) / 2 + (CONV_WIND_SIZE - 1) / 2;
    Int16* gradX  = cannyResult + width * (CONV_WIND_SIZE - 1) / 2 + (CONV_WIND_SIZE - 1) / 2;
    Int16* gradY  = pBufGradY + width * (CONV_WIND_SIZE - 1) / 2 + (CONV_WIND_SIZE - 1) / 2;
    Int16* gradM = pBufMag   + width * (CONV_WIND_SIZE - 1) / 2 + (CONV_WIND_SIZE - 1) / 2;
    Int32 outputHeight = height - (CONV_WIND_SIZE - 1);
    Int32 outputWidth = width - (CONV_WIND_SIZE - 1);
    Uint32* listptr = (Uint32*)rgb;

    Int32 pixelsCount = width*(outputHeight)-6;
    int offset = 0;
    int change = width*7;
    while(offset<pixelsCount)
    {
    	VLIB_conv_7x7_i8_c8s(&img[offset],&pBufOut[offset],change,width,gaussian_7x7,8); //pixelNums=width*height
    	offset+=change;
    }
    gradX += (width+1);
    gradY += (width+1);
    gradM += (width+1);
    outputHeight-=2;
    VLIB_xyGradientsAndMagnitude(pBufOut,gradX,gradY,gradM,width,outputHeight);

    int i;
    for( i=0; i < width * 5; i++ ) {
    	pBufOut[i] = 0;
    	pBufOut[width * (height - 5) + i] = 0;
    }

    buffOut = pBufOut + width * (CONV_WIND_SIZE - 1) / 2 + (CONV_WIND_SIZE - 1) / 2;
    outputHeight -= 2;
    buffOut +=(2 * width + 2);
    VLIB_nonMaximumSuppressionCanny(gradM,gradX,gradY,buffOut,outputWidth,width,outputHeight);
    VLIB_doublethresholding(gradM,buffOut,listptr,numItems,outputWidth,width,outputHeight,lo,hi,0);
    if(*numItems<=0 || *numItems>1000000)
    {
    	return 0;
    }
    VLIB_edgeRelaxation(buffOut,listptr,numItems,width);
    align_free(numItems);

    memset(cannyResult, 0, width * height * sizeof(UInt8));
    IMG_thr_le2min_8((const UInt8*)buffOut, (UInt8*)cannyResult, width, height, MIN_LIGHT_VAL);
    return 1;
}

Thanks,

Yoel

Yoel,
Yes, you are right. You need to divide outX and outY by 16 to find the co-ordinates in pixels.

I have few questions/suggestions regarding your implementation:
1. You need to have gradX and gradY of image A before you can pass them to VLIB_trackFeaturesLucasKanade_7x7(). Otherwise, gradX, gradY all have initialized values (zero) when you pass them to VLIB_trackFeaturesLucasKanade_7x7().

2. What are XShift and YShift? Are they estimated displacement? Are they in pixels?

3. X, Y, outX and outY are arrays of length numItems. They only contain the co-ordinates of features, and are not images. So, accessing anything like outX[i*width+j] is out of range and would give undesired results for i>0. You would have to reduce the nested for loop below VLIB_trackFeaturesLucasKanade_7x7() call into just:

for(i=0; i<numItems; i++) {
    if(outX[i]>=0 && outY[i]>=0)
    {
        outX[i]=outX[i]>>4-X[i]>>4;
        outY[i]=outX[i]>>4-Y[i]>>4;
    }
    else
    {
        outX[i]=0;
        outY[i]=0;
    }
}

Thanks.

Sudhakar

Hey Sudhakar,

Thanks for the quick reply!

1. You are right, it's leftovers from the harris score I previously used.

2. Yes, in pixels. Right now they are constant zero.

3. Again, you are correct, here is the new function:

Void TrackObjects(UInt8* left, UInt8* right, Int32 width, Int32 height, Int32 XShift, Int32 YShift)
{
    Int32 i,j,numItems = 0;
	Int32 scratchSize = 160*width;
    // initialize memory
    memset(pBufGradX, 0, width * height * sizeof(Int16));
    memset(pBufGradY, 0, width * height * sizeof(Int16));
    memset(pBufMag,   0, width * height * sizeof(Int16));
    memset(pBufOut,   0, 200000/*(width * height * 6 + scratchSize) * sizeof(UInt8)*/);

    Int16* gradX  = pBufGradX;
    Int16* gradY  = pBufGradY;
    Int16* gradM = pBufMag;
    UInt8* scratch = pBufOut;
    Int16* X = 		(Int16*)&pBufOut[(scratchSize)*sizeof(Int16*)];
    Int16* Y = 		(Int16*)&pBufOut[(scratchSize+(width*height))*sizeof(Int16*)];
    Int16* outX = 	(Int16*)&pBufOut[(scratchSize+(width*height*2))*sizeof(Int16*)];
    Int16* outY = 	(Int16*)&pBufOut[(scratchSize+(width*height*3))*sizeof(Int16*)];
    UInt8* OUTM = 	(UInt8*)&pBufOut[(scratchSize+(width*height*4))*sizeof(Int16*)];
    UInt8* OUT = 	(UInt8*)&pBufOut[(scratchSize+(width*height*5))*sizeof(Int16*)];

    /* Compute the gradient using the library function */
    DoCanny(left, width, height, (Int16*)OUT, high, low);
    memset(pBufGradX, 0, width * height * sizeof(Int16));
    memset(pBufGradY, 0, width * height * sizeof(Int16));
    VLIB_xyGradients(left, gradX + width + 1, gradY + width + 1, width, height);
    for(i=3;i<32;i++)
    {
    	for(j=6;j<36;j++)
    	{
        	if(OUT[i*width+j]==255)
    		{
    			X[numItems]=(j+2)<<4; // Shift because of the Canny process
    			Y[numItems]=(i+5)<<4; // Shift because of the Canny process
    			outX[numItems]=X[numItems]+XShift;
    			outY[numItems]=Y[numItems]+YShift;
    			numItems++;
    		}
    	}
    }
    VLIB_trackFeaturesLucasKanade_7x7(left, right, gradX, gradY, width,
    		height, numItems, X, Y, outX, outY, LK_MAX_ITER, scratch);

    for(j=0;j<numItems;j++)
    {
    	if(outX[j]>=0 && outY[j]>=0)
    	{
    		outX[j]=outX[j]>>4-X[j]>>4;
    		outY[j]=outY[j]>>4-Y[j]>>4;
    	}
    	else
    	{
    		outX[j]=0;
    		outY[j]=0;
    	}
    }
}

4. Is there a way to see a displacement in sub pixels? Meaning, can the algorithm take into account the intensity of the pixel?

Yoel

Yoel,

The code now looks fine to me. Just to be safe, you can shift XShift and YShift to left by four in the nested for loop as follows:

outX[numItems]=X[numItems]+XShift<<4; // Need to convert XShift to Q4 format if XShift is non-zero
outY[numItems]=Y[numItems]+YShift<<4;

4. Can you please explain your question in detail? Sub-pixel displacements become significant when you continuously track features from frame to frame so that tracking is more accurate. You cannot see sub-pixel displacements in one image.

Thanks.

Sudhakar

Hi Sudhakar,

Thanks for the update, I made those changes in the code.

4. That's exactly what I want to do, track an object very accurately so I need to know the sub-pixel displacement. How do you suggest doing it? Do you average displacement on several consecutive frames or can you do it with only 2?

Yoel

Yoel,

outX and outY will be updated with sub-pixel accuracy when you pass them to VLIB_trackFeaturesLucasKanade_7x7(). You can pass them directly as they are, as inputs X and Y during next function call for next frame. Is this what you are asking?

Sudhakar

Sudhakar,

First let me thank you for your excellent support!

This means that shifting the outX and outY values by 4 is a mistake because that way you don't get the sub-pixel displacement, just full pixel, am I wrong? If this is the case, how do you suggest looking at the output?

Yoel

Yoel,

You are right. When you get outX and outY updated after calling VLIB_trackFeaturesLucasKanade_7x7(), the values provide the new feature co-ordinates, and not the displacement. Bits 0-3 provide sub-pixel accuracy. To get the displacement with sub-pixel accuracy, you can do outX[i] - X[i] without any shifting. The least four bits provide sub-pixel displacement. You can do outX>>4 only if you need the pixel location without sub-pixel accuracy.

For example, say numItems is 1, and initial location of the point you need to track in image A is at row 5 column 7. So, X[0] = 7, Y[0] = 5. You need to convert them into Q4 format first, so you will pass X[0] = 7<<4 = 0x0070 and Y[0] = 5<<4 = 0x0050. Also, you must provide the initial estimates of outX and outY. So, outX[0] = 7<<4 and outY[0] = 5<<4. You will pass this into the VLIB_trackFeaturesLucasKanade_7x7() function.  Now, If the displacement is say, 2.25 pixels towards right in horizontal direction and 1.5 pixels down in vertical direction, the updated outX and outY will have values outX[0] = 0x0094 and outY[0] = 0x0068. If you right shift these, you will get pixel co-ordinates without sub-pixel accuracy, which are 9 and 6. So, to preserve sub-pixel displacement, you should not right shift outX and outY.

Thanks.

Sudhakar

Hi Sudhakar,

I have two things that I need to clarify as I am unable to track anything reliably for more than a couple of frames by using the tracking error as a way to disgard 'bad' points.

P1. Do I need to filter my image prior to feeding it to VLIB_trackFeaturesLucasKanade_7x7. If that is the case do I need to filter both images i.e. the one before I calculate the gradient and the one that I use as is? Will it improve the performance of the tracking in any case? For the record, the tracking test I tried does not use pyramids just 2 function calls so its pretty much down to these 2 functions.

P2. Since, in order to solve the optical flow equation within  trackFeaturesLucasKanade_7x7, you need to check if the image derivative matrix(or Gradient matrix or G matrix or....), for a particular point, is invertible. Therefore, you must somehow be rejecting some points (checking the determinant maybe). But the outx, outy vector have the same size as the x,y. Can you please clarify for me how is it possible not reject points at all? Or is there a way to identify those points. And as it is widely known affine tracking alone is not adequate.

I have run several experiments with other libraries and if points are not rejected as I describe the tracking performance is horrible which is pretty much what I get with vlib.

and a new (final) question

Is there a way to run the vlib library in a  vc++ project so that I can verify the output of trackFeaturesLucasKanade_7x7 against some other known library. We actually have our own implementation of lucas kanade by now, but we could do with the extra speed that vlib provides.Thanks in advance.

Regards

Joseph

Joseph,

Attached are the PC compiled libraries for VLIB release 3.0.1.0.  One is compiled for X86 64bit version and the other for 32bit version. You can use the headers with same release.

7043.release_32-bit.zip

8400.release_64-bit.zip

I will get back to you as soon as I find answers to your first two questions.

Thank you.

Sudhakar

Hi Sudhakar,

thanks for the libraries it was a major major help. I have managed to setup a vc++ project and run vlib side by side with 2 other implementations (opencv, and the one from the link found in vlib manuals). Although, the 2 libraries give similar results they don't match with the output of VLIB_trackFeaturesLucasKanade_7x7. I am calling them within the same project. I don't user pyramids just 2 images. The displacements are in the range 0 - 3. Therefore I must be missing some step. I use opencv's good features to track to find the points and do the necessary conversions before passing them to vlib. I only use the following 2 function calls for vlib (given I don't have to calculate input points):

VLIB_xyGradientsAndMagnitude(currentGrayFrame, GRADX+640+1, GRADY+640+1 , GRADMAG+640+1 , 640, 480);

VLIB_trackFeaturesLucasKanade_7x7(previousGrayFrame,  currentGrayFrame, GRADX, GRADY, 640, 480, inputPointsSize, X, Y, mX, mY, 10, BUFFERLK); // BUFFERLK size is 384 or maybe not?

Is there any step I am missing. The library seems to work for trivial displacements on test images. 

UPDATE

I have also noticed that you acknowledged as correct, in a previous post not addressed to me, a size for the scratch buffer used in the LK of:

 when I asked if 384 is the correct size you confirmed it. Can you confirm which is the correct size and if this depends on the image size. This actually seems to effect the results. 

Since the answers I have received or read so far are highly inconsistent, I will ask for a reference sample for one more time, just in case. I have serious doubts that this particular function produces accurate results for anything other than simple test images. A working piece of code (not the ideal test case) will prove me wrong. This is my last post on this subject. I really cannot understand the situation around this library. If TI does not want people to use it, they can at least warn us about its status, rather than flooding the internet with articles and presentations praising its performance. Highly unprofessional. Can you please confirm that it can produce reasonably accurate results when tracking moving objects in a real time situation i.e. moving car.(failure to respond in this question will be perceived as a negative response) Thanks again. 

Regards

Joseph

Joseph,

1. You are not missing any steps between VLIB_xyGradientsAndMagnitude() and VLIB_trackFeaturesLucasKanade_7x7(). You can use the gradients obtained from VLIB_xyGradientsAndMagnitude() on VLIB_trackFeaturesLucasKanade_7x7() directly.

2. For trackFeaturesLucasKanade_7x7(), the scratch buffer size should be 384 bytes, and two byte aligned. If more than 384 bytes is allocated for scratch buffer, only 384 bytes are used from the address you pass as the argument. So, it can be more than 384 bytes, but not less.

3. The trackFeaturesLucasKanade_7x7() is very good in tracking feature points such as Harris corners. So, if you can find such feature points on a moving object (car, for example), the function should be able to track them. The demo video is available here: http://www.youtube.com/watch?v=LlLVom3fsww

Please tell us what your expectations are with respect to using trackFeaturesLucasKanade_7x7? Like what you are trying to track (feature points/object as a whole), how fast the object is moving, frame rate, is the camera moving, how you define the initial feature points, frame width and height, etc?

As you have mentioned before, using image pyramids will improve the performance. Can you try tracking using image pyramids and see if there are improvements? You can start from Level 3 and go up to Level 0, applying tracker function at each level (you can use the implementation I provided earlier).

Thank you.

Sincerely,

Sudhakar Ayyasamy

Hi Sudhakar,

I have tried to compare the performance of trackFeaturesLucasKanade_7x7 against 2 other libraries. The displacements I am tracking are 2 pixels on one axis and 1 pixel on the other, therefore the pyramid will not provide any improvement, since the displacements are SMALL. The biggest problem I have, as I mentioned before, is that I get a lot of displacement values which are clearly wrong. It is expected. What is not expected is the lack of a way to filter them out. The tracking error is also inadequate and since there is no other way other function to assist in this, tracking something with reasonable accuracy is out of the question. 

The link shows a video that displays motion vectors. It does not track anything. The points are probably reset after each frame, you can actually observe points 'flying' from one point to another in the background.

Anyway, we have wasted enough time with this library,  I hope other people will be warned by this post . Extrimely disappointing attitude from TI. Thank you for your effort.

Regards

Joseph

Same here...

Yoel

Joseph,

Sorry that the tracking function doesn't meet your needs.  At this point, there aren't resources available to enhance/modify this function.  Best thing we can offer here is either put you in touch with one of our 3rd parties that may be able to develop something for your specific need.

Regards,

Travis

Hi Travis,

As I wrote earlier, the problem with vlib is mainly its poor documentation. A reference guide similar to the one imglib has would be perfect. Regarding VLIB_trackFeaturesLucasKanade_7x7, as I wrote in a previous post, I have tested its performance against other libraries using common input points. It can actually track the same points that the other libraries track. But since it has no way of disgarding 'bad points', the tracking result is affected by them. The algorithm implementation that I am currently working on uses other vlib functions, not trackFeaturesLucasKanade_7x7. We have replaced it with our own implementation. We could really benefit from its source code though, in case we can add a check for disgarding 'bad points', since we really need every speed improvement we can get. That would be appreciated greatly. Our implementation provides good tracking results but its slower than trackFeaturesLucasKanade_7x7. Thank you for your consideration.

Regards

Joseph

Joseph,

In the demo video I sent earlier, only the following sanity checks are used to delete feature points from one frame to next:

1. Points that move close to edges of frame (within 10 pixels from edges) are deleted

2. Points that moved too far (70 pixels in x or y direction) from one frame to next are deleted

Thanks.

Sudhakar

Hi Sudhakar,

well my problem is with values that are incorrectly tracked temporarily and are not far away to be disgarded but close enough to offset the tracking. Can you verify my finding that vlib implements the Inverse Compositional Lucas Kanade? If not then which algorithm does it implement because it does not look like it implements the Forward Additive LK. If yes, is there any other way that I can implement that will help me disgard incorrectly tracked features(I know I cannot use the determinant of the gradient matrix in the IC LK)? I don't want to use something like affine consistency check because as it will consume too much cpu power.  Thanks again.

Regards

Joseph

Joseph,

Sorry. Unfortunately, we don't have vision experts at present who have worked on this particular function of vlib to help answer these questions. I will however forward the questions you have to the vlib team. As Travis mentioned, if needed, we can get the help of a third party to assist you in finding out if its possible to implement a way to discard incorrectly tracked features on top of current LK implementation. Thank you.

Sincerely,

Sudhakar Ayyasamy

Hi,

We have the same problem as Joseph, which is that the Lucas kanade result is very poor in VLIB.

We also tried all kinds of experiments and doubted the there exist bug in the function of VLIB.

Can you send me the source code of VLIB? Or the part of VLIB_trackFeaturesLucasKanade_7x7 to fix the bug.

thx.

Hi Sen,

since I have spent over a month trying to get this to work some things I noticed may help you. First, the LK function does not have a bug. That I can verify. Not an obvious one at least(there is one if you use debug and older cg tools but if you use the version they recommend its fine). There are a few things you can do to see if you get better performance. First use good features to track to supply a good set of points. Do not supply too many. For a window of 150x200px max 50. Start with less and see what works better. If you supply too many some are not that easy to track and they will ruin your tracking. Then when you detect that a point is out of bounds set its coordinates to zero. Do not use a seperate variable to flag it.In this way the function will ignore it next time. Also if you know that the images you use to test have small displacements then you should be fine. If you have larger displacements use a pyramid. The VLIB_imagePyramid8 function will supply you with 3 levels of pyramid. Use only the first level and it can track pretty much any rigid object you will try even when they turn and change shape. It also works when your camera is moving. If you still can't track well enough then get the source and zero the coordinates wherever there is a check for out of bound points within each iteration (i use 10 iterations). That works pretty well. As a final advice, use the TI c intrinsics library to set up a project to compare it with other libraries, it helps a lot. I hope this helps.

BR

Joseph

Joseph,

Thank you for reporting your find. We will start working on the documentation part and make sure VLIB is well documented for future releases. Thanks again.

Sincerely,

Sudhakar Ayyasamy

thank you for your kind replay.

I have test the several functions as below:

1)read a CIF image as first frame, simulate movement of 5 pixels in verticle  direction as secord frame.

    a) calcute 100 points in grid of  10 x 10, the accurate ratio of opticle flow tracking result is about 30%

    b) calculate harris corner, use threshold filter and nonMaxSuppress, the right number is about 10 in total 61 corners.

    c) calcute 100 points in grid of  10 x 10,  use pyramid  opticle flow tracking, the accurate ratio is almost 0.

2) simulate movement of 1-3 pixels in verticle  direction as secord frame,  the corresond result is better.

3) read 2 successive frames of real scene,   the corresond result is worse.

4) in my application, the point is about 25-100 in a small region about  from 20x20 to 50x50, the result is worse of course.

If necessory,  i will post the source code and the test images tomorrow.

If the problem is coordinates out of bound in iteration, we have to check in the iteration in the vlib source code.

Even we do so, the accurate ratio of the result will not get better, which will exclude some wrong reslults only.

Maybe we can modify the iteration time from 10 to 30 to improve the tracking reslut, but this need modify the vlib source code of vlib too.