count lines of code (cloc) in multiple PHP source folders

Over the years, I have used many tools that generate CLOC reports (how many lines of code you have written). The one I really like is cloc project on sourceforge . The tool is written in perl and is quite flexible.

Our problem is that we have a web application and that means

• we need to ignore certain folders when counting lines. Examples would be compiled templates, generated minified files, third party libraries and such.
•  We need to create custom definitions for some files (like .tmpl is our view templates and .inc are our PHP include files) Without custom definitions, the tool would count .inc as "PASCAL" file and ignore .tmpl files !!!
• We need to sum the individual reports over. We have dependencies and the project is split across different source trees and not everything is under one root.

Ignoring folders/files with cloc is dead simple.  Just pass a file with files and directories in --exclude-list-file.

To generate custom definitions (so .tmpl are counted as view templates and .inc as PHP includes)  first use cloc.pl to write standard definition files using --write-lang-def switch. Later on we will modify this file and ask cloc to use our definition instead of the standard one.

Exclude list

Exclude list is a file with one entry per line.

rjha@mint13 ~/code/github/sc/deploy/apps/cloc $ cat cloc-ignore 
/home/rjha/code/github/sc/web/css/bundle-full.css
/home/rjha/code/github/sc/web/css/bundle.css
/home/rjha/code/github/sc/web/js/bundle.js
/home/rjha/code/github/sc/web/js/bundle-full.js
/home/rjha/code/github/sc/web/compiled

creating custom definitions file

First generate the definitions of extensions used by cloc tool. Then we add our custom definitions to this file. Later on we will use this file to supply file definitions to cloc tool. Just copy the  existing matching definitions for your custom file types.

 ./cloc-1.56.pl --write-lang-def=cloc.def

Add to this file following definitions

View Template
    filter remove_html_comments
    filter call_regexp_common HTML
    extension tmpl
    3rd_gen_scale 1.0


PHP Include
    filter remove_matches ^\s*#
    filter remove_matches ^\s*//
    filter call_regexp_common C
    filter remove_inline #.*$
    filter remove_inline //.*$
    extension inc
    3rd_gen_scale 1.0

script to sum reports

rjha@mint13 ~/code/github/sc/deploy/apps/cloc $ cat cloc.sh 
# web folder - use an ignore list
# web folder - needs custom definitions

./cloc-1.56.pl  --exclude-list-file=./cloc-ignore --read-lang-def=./cloc.def   /home/rjha/code/github/sc/web --report-file=sc.web.report
./cloc-1.56.pl  /home/rjha/code/github/sc/lib --report-file=sc.lib.report
./cloc-1.56.pl  /home/rjha/code/github/webgloo/lib/com/indigloo --report-file=webgloo.report
# sum the reports 
./cloc-1.56.pl  --read-lang-def=./cloc.def  --sum-reports *.report
#remove tmp
rm *.report

Here we are running cloc on three separate folders. First one uses an ignore list and a custom definition file (created earlier).  second and third are standard cloc reports.

Finally we use --sum-reports option to produce the final report across three different source trees.

rjha@mint13 ~/code/github/sc/deploy/apps/cloc $ ./cloc.sh 
     239 text files.
     239 unique files.                                          
      54 files ignored.
Wrote sc.web.report
      94 text files.
      94 unique files.                              
       0 files ignored.
Wrote sc.lib.report
      41 text files.
      41 unique files.                              
       0 files ignored.
Wrote webgloo.report

http://cloc.sourceforge.net v 1.56
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
PHP                            226           4795           1340          13256
PHP Include                     32            243             31            902
CSS                              1            219             33            850
Javascript                       1            249             99            823
View Template                   54            185              5            681
HTML                             4             19              0             46
XML                              2              0              0             44
-------------------------------------------------------------------------------
SUM:                           320           5710           1508          16602
-------------------------------------------------------------------------------

infinite scrolling but on steroids

we use infinite scrolling plugin by paul irish (https://github.com/paulirish/infinite-scroll) on 3mik now. However Problem with the vanilla plugin is that it assumes that the next URL only depends on current page number. Only  a naive pagination scheme would depend on only one variable.

As you go deeper into pages, you have to scan more and more records before you can arrive at the records you are interested in. For this reason,   real world pagination scheme rely on two variables (see these slides from percona conference  and this blog post )

• current page number
• last record id (of previous page ) or first record id (of next page)

An example would be the instagram API

To Apply infinite scrolling pattern to pages with such 2 variable pagination scheme, you need to do additional work. So here is our fork of infinite scroll plugin that documents this use case and provides sample server  PHP scripts.

Debian Linux with 3D graphics in vmware fusion 4.1

Joostlek has a nice post about enabling 3D graphics in linux guests on vmware.  I followed this same post for installing 3D graphics support for my Debian wheezy desktop.  I wanted to see the gnome shell in action after I heard so many (*bad*) things about it!

I had to install some extra packages and build dependencies for Mesa and if you also installed wheezy from net installer then these "extra" steps will be useful for you. Also his post describes building it on Ubuntu. These two points are the only difference from his post. I have vmware fusion 4.1.3 running on a 13" macbook pro.

Following packages are from above post


sudo apt-get install autoconf libtool xutils-dev xorg-dev flex bison libx11-xcb-dev libxcb-glx0-dev g++ git

And here are the extra packages that were missing in my net installer

•  sudo aptitude install  libxcb-dri2-0-dev
•  sudo aptitude install  libxcb-xfixes0 libxcb-xfixes0-dev
•  sudo aptitude install llvm
•  sudo aptitude install libxml2 libxml2-dev
•  sudo aptitude install python-libxml2
•  sudo aptitude install build-essential

You also have to install build dependencies for Mesa. Check the Debian How to build Mesa page.

•  sudo apt-get build-dep mesasudo apt-get build-dep mesa

Rest of the steps remain the same as in above post.  If you are using Debian wheezy after 21st June 2012 then you do not need to download a new kernel. DRM is enabled for your kernel. Just skip that step.

After the restart, I could log into gnome shell. The default leave a lot to desire. So I had to tweak my desktop for next  hour or so.  Here is the screenshot of my Debian wheezy  desktop running gnome-3 shell

lint like static code analysis tools for PHP

PHP is dynamically typed language and that means the error detection present in statically typed languages is absent. If you have an error in your script, it will not be detected till you have actually run it(apart from parse errors). The big, shine in your eyes errors are never a problem because they are detected quite easily during development. It is the silent types (doing X when you meant Y and X is a perfectly legit operation) that are more difficult.

Like, I will show you a piece of code I had re-factored  a while back

 
       static function find($subjectId,$objectId,$verb) {
            $mysqli = MySQL\Connection::getInstance()->getHandle();

            //sanitize input
            settype($subjectId,"integer");
            settype($objectId,"integer");
            settype($verb,"integer");

            $sql = " select count(id) as count from sc_bookmark " ;
            $sql .= " where subject_id = %d and object_id = %d and verb = %d ";
            $sql = sprintf($sql,$loginId,$itemId,$verb);

            $row = MySQL\Helper::fetchRow($mysqli, $sql);
            return $row;
        }

 

Here loginId was changed to subjectId in parameter list but I forgot to change the name of parameter I was passing to MySQL.  This used to result in a notice that was going to logs but no sign of program failure outside. That led me to say, enough is enough and I need to have a good static analysis tool for my PHP code.


There are many posts dedicated to the topic of PHP static analysis, sort of things that lint does for c code. The aim is to catch errors from just scanning the PHP code. When I started looking around, everyone talked about their favorite static analysis tool but no one gave me hard data or how it actually looks on your terminal.

I looked  at following tools

1. PHP code sniffer (phpcs)
2. PHP on command line with minus eel option (-l)
3. PHP mess detector (phpmd)
4. pfff/scheck

After a quick evaluation I (emphasized)  found phpmd to be the best of the lot.  Why?


One more tool worth checking should be https://github.com/sebastianbergmann/php-code-coverage
However I did not look at this one.

PHP code sniffer

phpcs looked too verbose out of the box. For 20 line perfectly legit code it generates 20 errors, most of that to do with style guidelines. Look at the output, I believe real  error will be buried under tons of such warnings. Sure, I did not look at how to tweak it and all but I believe most of the time first impressions are confined to out of box experience. Also, it could not detect the error I had. (error inside a class method)

FILE: /home/rjha/code/github/sc/lib/com/indigloo/sc/util/Asset.php
--------------------------------------------------------------------------------
FOUND 16 ERROR(S) AND 1 WARNING(S) AFFECTING 11 LINE(S)
--------------------------------------------------------------------------------
  2 | ERROR   | Missing file doc comment
  4 | ERROR   | Missing class doc comment
  4 | ERROR   | Opening brace of a class must be on the line after the
    |         | definition
  6 | ERROR   | Missing function doc comment
  6 | ERROR   | Opening brace should be on a new line
 17 | ERROR   | Expected "if (...) {\n"; found "if(...) {\n"
 20 | ERROR   | No space found after comma in function call
 24 | ERROR   | Expected "if (...) {\n"; found "if(...) {\n"
 28 | WARNING | Line exceeds 85 characters; contains 103 characters
 31 | ERROR   | Expected "if (...) {\n"; found "if(...) {\n"
 31 | ERROR   | No space found after comma in function call
 33 | ERROR   | No space found after comma in function call
 33 | ERROR   | No space found after comma in function call
 36 | ERROR   | Expected "if (...) {\n"; found "if(...) {\n"
 36 | ERROR   | No space found after comma in function call
 38 | ERROR   | No space found after comma in function call
 38 | ERROR   | No space found after comma in function call
--------------------------------------------------------------------------------

php -l (minus eel)

looks too limited. It could not detect errors inside class methods.

pfff/scheck

pfff is a project from facebook and it is a suite of tools. There are some interesting ocaml based tools there, like code map generator etc. scheck is the tool to do static checks. However pfff looks to be in alpha stage right now.

php mess detector

This could detect the problem inside a class method. Also, the tool is quick and you can decide on the kind of rules you would like to run.

rjha@mint ~/code/github/sc $ phpmd  lib/com/indigloo/sc/util/Asset.php text codesize,design,naming,unusedcode

/home/rjha/code/github/sc/lib/com/indigloo/sc/util/Asset.php:13 Avoid variables with short names like $ts. Configured minimum length is 3.

Nice looking Debian Wheezy Desktop

Well, if you do not mind 1000 extra packages and 100 MB of extra memory then I would suggest you install Linux Mint LMDE edition. The nice thing about mint LMDE is that you get a nice looking desktop with the choice of MATE or Cinnamon. The fonts and rendering looks nice out of the box.

However, if you have just started with the Debian net installer and just installed your desktop then the rendering and fonts may be a bit out of shape! You may wonder if your 1300$ LCD screen is not seen by Debian at all! Fret not! With little patience and work, we can make our Debian desktop look real nice and polished.

Fix font configuration

We first copy the ubuntu font config files in Debian wheezy following the steps outlined in 

http://noz3001.wordpress.com/2011/07/01/ubuntu-font-rendering-on-debian-wheezy/

After that we install microsoft ttf fonts 

$sudo aptitude install  ttf-mscorefonts-installer

Now we reload the font config and clear the old font caches

$sudo fc-cache -fv 

$sudo dpkg-reconfigure fontconfig-config 

$sudo dpkg-reconfigure fontconfig

Fix font aliasing and rendering

The following combination works for LCD screens

System | Preferences | Customize L&F  (or whatever is equivalent for your DM of choice)

• Turn subpixel hinting to RGB
• Turn on font aliasing
• Set font hinting to  slight

Change Mouse cursor

$sudo aptitude install dmz-cursor-theme

$ sudo update-alternatives --config x-cursor-theme

Preferences | customize L&F | Mouse cursor 

select DMZ white theme and you get a sane mouse instead of default x-cursor (circa 1991)

Install Flash

 Download 64 bit flash player from Adobe site, unpack

 $sudo cp libflashplayer.so /usr/lib/mozilla/plugins/.

 restart iceweasel, try viewing a youtube video.

Change login screen

By default wheezy is using gdm3 and the gdm3 login screens are, well, yuck! To change gdm3 login screen, one "simple way" is to create a new symlink for /usr/share/images/desktop-base/login-background.svg that points to your new login splash background. Please note that only the .png files work. The .jpg files would not work in this scheme.

Install lightdm

This provides a much better looking login screen. 

$sudo aptitude install lightdm 

verify that lightdm is the new display manager in /etc/X11/default-display-manager

To configure

$dpkg-reconfigure lightdm 

To change lightdm login screen background (greeter background) change background in /etc/lightdm/lightdm-gtk-greeter.conf (PNG images) 

After all these changes you will have a shiny looking Debian Wheezy Desktop :D Enjoy'

VMWare tools on Debian Wheezy inside Fusion

I have an early 2011 intel macbook and I run linux VM using Vmware fusion 4.x. Earlier I was using an Ubuntu VM for development that had 2.6.x kernel. Now, I wanted to upgrade my development VM to Debian wheezy for assorted reasons.  One of them was having a rolling distribution because I do not want to full upgrade my installation all the time.

The problem with installing Debian wheezy or any other linux 3.x series kernel in vmware fusion is that the vmware tools shipped with fusion is not compatible with linux 3.0 and 3.2 kernels. The VMware tools only work with 2.6.x kernel headers. Now this is a big headache as I need to mount my macbook directories inside my VM. Other nuisance is that your mouse will be captured inside vm window if you do not install the tools. in short, I want a life with vmware tools.

So open-vm-tools came to my rescue. This is one of the moments in life when you did not know about existence of something that is so essential to your existence! All my life of running linux VM on macbook and I did not know that this project exists! As it happens open-vm-tools are rolled with Debian so installing them is super easy.

First install the prerequisites

$aptitude install linux-headers-`uname -r` libx11-6 libx11-dev xorg libxtst6 psmisc build-essential ia32-libs ia32-libs-gtk

The Debian wiki mentions x-window-system and x-window-system-core but those packages are now provided by xorg. Now install the open-vm-tools packages using $sudo aptitude install

• open-vm-tools 
• open-vm-dkms      
• open-vm-tools-dev  
• open-vm-toolbox 

After the installation, try querying the vmware tools modules

%modinfo vmxnet

%modinfo vmhgfs

if the required kernel modules are not loaded then we have to do so via dkms (Dynamic kernel module support). To install required kernel modules

% dkms add open-vm-tools/2011.12.20

% dkms build open-vm-tools/2011.12.20

% dkms install open-vm-tools/2011.12.20

Last number in red is the module version, you can get this number by using  

$sudo aptitude show open-vm-tools 

Look for version string (after the + sign)

Run modinfo again and verify that we have the required modules. Shut down the VM. Map shares using VMware fusion settings. Start the VM. Now we should have VMware tools loaded.

$vmware-hgfsclient 

The output should match with the shared folders

To mount the shared folders, you can do something like. Here Public is the folder name that we shared via VMware fusion. We mount this folder at /mnt/hgfs.

$sudo mount -t  vmhgfs -v -o rw  .host:/Public /mnt/hgfs

Send Email using SendGrid PHP library on Github

We are hosting the 3mik.com boxes on Rackspace right now and  we get 40,000 emails/month free of cost  from one of their partners, SendGrid. We use SendGrid to send  emails from our application written in PHP.  The SendGrid documentation suggests using SMTP relays and swiftmail-4.x library.  However, SendGrid also has a PHP front-end library published on github that makes it super easy to send emails.

Here is one complete test script

    require_once($_SERVER['WEBGLOO_LIB_ROOT']. '/ext/sendgrid-php/SendGrid_loader.php');
    set_error_handler('offline_error_handler');
    $sendgrid = new SendGrid('your-sendgrid-login', 'your-sendgrid-password');
    $mail = new SendGrid\Mail();


     $mail->addTo('foo@gmail.com')->
       setFrom('foo@3mik.com')->
       setSubject('Sendgrid github PHP library test')->
       setText('Hello World! from sendgrid library')->
       setHtml('Hello World! from sendgrid github lib');  

    $response = $sendgrid->web->send($mail);
    // Error Handling:- 
    // sendgrid->web method uses curl_exec CURLOPT_RETURNTRANSFER set. 
    // This means you will get FALSE
    // when the send method fails at network level.
    // you get JSON response back when curl is able to communicate with the server
    // success from API is returned as
    // {"message":"success"}
    // Error from API is returned as
    // {"message": "error", "errors": ["Bad username / password"]}
    print_r($response);



?>

Though all SendGrid documentation exhorts you to use SMTP, I do not see a problem in using the WEB method.  As per this SO question, http://stackoverflow.com/questions/6193702/sendgrid-smtp-or-curl , " The  web API actually works faster than SMTP, as you only need to make a single cURL request to us to send a message, whereas with SMTP there's a lot of back-and-forth TCP chatter for connection, HELO, and such." 


Error checks are crucial part of any code. Here is my complete sendgrid wrapper complete with error checks. I do not want to throw exceptions because I want clients to handle the error their own way.

Upload to s3 using aws php sdk

Here is a quick code sample

    error_reporting(-1);
    require_once 'sdk.class.php';

    // UPLOAD FILES TO S3
    // Instantiate the AmazonS3 class
    $options = array("key" => "aws-key" , "secret" => "aws-secret") ;

    $s3 = new AmazonS3($options);
    $bucket = "media1.3mik.com" ;

    $exists = $s3->if_bucket_exists($bucket);
    if(!$exists) {
        trigger_error("S3 bucket does not exists \n" , E_USER_ERROR);
    }

    $name = "cows-and-aliens.jpg" ;
    $path = "/home/rjha/Pictures/Cows-And-Aliens-1280x960.jpg";

    echo " upload file $name from $path \n" ;
    $sblob = file_get_contents($path);
    $response = $s3->create_object($bucket, $name, array('body' => $sblob , 'acl' => AmazonS3::ACL_PUBLIC));
    if($response->isOk()){
        echo "done" ;
    } else {
        echo "error";
    }

Add caching headers to existing S3 Object

Problem :- How to add caching headers to an existing amazon S3 object?

Background:- Yes, I know. I should have done it when I had uploaded those files. However they are already there and now we need to add caching control headers to our already uploaded objects.

Simple solution :- Use cloud Berry explorer or some such tool. 

Programmer's solution:- Looks like using aws-sdk copy_object does the trick.

Code sample

    error_reporting(-1);
    require_once 'sdk.class.php';

    // copy S3 object
    // Instantiate the AmazonS3 class
    $options = array("key" => "aws-key" , "secret" => "aws-secret") ;


    $s3 = new AmazonS3($options);
    $bucket = "bucket.3mik.com" ;


    $exists = $s3->if_bucket_exists($bucket);
    if(!$exists) {
        trigger_error("S3 bucket does not exists \n" , E_USER_ERROR);
    }

    $name = "cows-and-aliens.jpg" ;
    echo " change headers for $name  \n" ;
    $source = array("bucket" => $bucket, "filename" => $name);
    $dest = array("bucket" => $bucket, "filename" => $name);

    //caching headers
    $offset = 3600*24*365;
    $expiresOn = gmdate('D, d M Y H:i:s \G\M\T', time() + $offset);
    $headers = array('Expires' => $expiresOn, 'Cache-Control' => 'public, max-age=31536000');

    $meta = array('acl' => AmazonS3::ACL_PUBLIC, 'headers' => $headers);

    $response = $s3->copy_object($source,$dest,$meta);
    if($response->isOk()){
        printf("copy object done \n" );

    }else {
        printf("Error in copy object \n" );
    }

How to integrate Facebook login with your website to authenticate users

Say you have a website and you would like to authenticate people before they can do some actions. Now, many people may not be very willing to fill out your registration form and do a login. Now if you allow people to use their existing Facebook or twitter Login for authentication then you are removing a big barrier from trying out your site.


This post  deals with server side integration of Facebook Login into your website (or online app!)  I am going to present it in a very sequential fashion.


We only cover the server side integration. If you are looking for javascript integration then stop reading now :) I believe server integration is a better scheme as it gives you better control. So if you are running your own website and would like to authenticate users with Facebook login then dive straight in.

Register with Facebook developers to get App ID and App Secret Key

First head over to  https://developers.facebook.com/apps  and register your application. Click on create new app button if you do not have any. The registration is straightforward.

• App display name - chose carefully because this is the application name presented to people who are trying to access your site using their Facebook login.
• App domain - Here you can put multiple domains + sub domains. Sub domains are especially useful if you plan to test on your local machine. Like in my case I have created two entries
• mint.3mik.com (to test from my local linux mint VM)
• www.3mik.com
• Site URL - put your website url here - like http://www.3mik.com
• Mobile web - like http://m.3mik.com
• You do not have to register App on Facebook, Native iOs app and native android app if you do not want to. I just wanted the website integration and I checked off the other options.
• After a successful registration, note down your App ID and App secret key

Create a  link on your website  to launch Facebook oAuth Dialog box

When users come to your site you have to present a link to launch Facebook oAuth dialog box. You can put a simple link or wrap it up in a nice book. When users come to your site, they see a Login with Facebook link.

The best resource for Facebook login server side integration   is  

• http://developers.facebook.com/docs/authentication/

To create such a link that will launch Facebook authentication dialog box, You need

1. Your Facebook App ID 
2. Your callback URL (where Facebook will redirect after a successful login) - Do not worry too much about this right now. Just add a URL that belongs to your site.

Here is how I am creating it

<?php
    $stoken = Util::getMD5GUID();
    $gWeb->store("fb_state",$stoken);
    $fbAppId = Config::getInstance()->get_value("facebook.app.id");
    $host = "http://".$_SERVER["HTTP_HOST"];
    $fbCallback = $host."/callback/fb2.php" ;
    
    $fbDialogUrl = "https://www.facebook.com/dialog/oauth?client_id=".$fbAppId ;
    $fbDialogUrl .= "&redirect_uri=".urlencode($fbCallback)."&scope=email&state=".$stoken ;
?>

  <a href="<?php echo $fbDialogUrl; ?>"> Login with Facebook</a>

some explanation is in order for parameters supplied to Facebook dialog URL

• fbAppId - is simply the Facebook application ID
• redirect_uri is where Facebook should redirect the user (A callback for us) 
• scope - if you just want the basic information back then you do not have to use this parameter. in Our case we want Facebook to return email together with basic information hence we pass this extra parameter, scope=email. You can pass in a comma separated list.
• state token. This is to prevent CSRF attacks. We generate a random token and keep it in our server session. We pass the same token as state parameter to Facebook. Facebook will return this state token as it is to our callback URL. There we can retrieve this token and compare it with the token stored in our session.

Now when users come to your login page, they will see a login with Facebook link. When they click it they will go to Facebook oAuth dialog page. After their authentication they will be redirected to our callback page.

Callback Page to process Facebook response

Most of the example code I have seen on parsing Facebook response is pathetic including Facebook's own sample. They are suppressing errors in their sample code. Real world applications will need a robust error handling and user information processing. 

Error Handler 

We want to trap any errors, log it and then redirect the users to login page where they can view the error. Also we log the errors so we can see why our integration is not working.  Here is our Error Handler

function login_error_handler($errorno,$errorstr,$file,$line) {

    if(error_reporting() == 0 ) {
        // do nothing for silenced errors
        return true ;
    }
    
    switch($errorno) {

        case E_STRICT :
            return true;
        case E_NOTICE :
        case E_USER_NOTICE :
            Logger->error(" $file :: $line :: $errorstr");
            break ;

        case E_USER_ERROR:
            Logger->trace($file,$line,$errorstr,'TRACE');
            $_SESSION["form.errors"] = array($errorstr);
            header('Location: /user/login.php');
            exit(1);
   

        default:
            Logger->trace($file,$line,$errorstr,'TRACE');
            $_SESSION["form.errors"] = array("Error happened during login");
            header('Location: /user/login.php');
            exit(1);
            
    }
    
    //do not execute PHP error handler
    return true ;
}

We use the above error handler in our callback code. Now coming to callback, following is our scheme to process Facebook response

• Facebook can return error and error descriptions to callback. If Facebook returned an error then we have to process those errors and show them to users. I suggest reading the Facebook document here.
• If there is no code then user has accessed the callback page directly and we need to launch the Dialog box
• If you have a code and state returned by Facebook is same as stored by us in session (to prevent CSRF attacks) then we need to 
• Issue a request to Facebook using the returned code and our App secret key 
• Parse the data returned by Facebook
• Use this data to start our own login session

The code in isolation will not make much sense. Here is the code to do error checks and issue a request to Facebook to get user data using our App secret key.

 
    //set special error handler for callback scripts 
    include ($_SERVER['APP_WEB_DIR'].'/callback/error.inc');
    set_error_handler('login_error_handler');
   
    $fbAppId = Config::getInstance()->get_value("facebook.app.id");
    $fbAppSecret = Config::getInstance()->get_value("facebook.app.secret");

    $host = "http://".$_SERVER["HTTP_HOST"];
    $fbCallback = $host. "/callback/fb2.php";
  
    $code = NULL;
    if(array_key_exists('code',$_REQUEST)) {
        $code = $_REQUEST["code"];
    }
 
    $error = NULL ;
    if(array_key_exists('error',$_REQUEST)) {
       $error = $_REQUEST['error'] ;
       $description = $_REQUEST['error_description'] ;
       $message = sprintf(" Facebook returned error :: %s :: %s ",$error,$description);
       trigger_error($message,E_USER_ERROR);

       exit ;
     }


     if(empty($code) && empty($error)) {
        //see how to launch an FB dialog again on Facebook URL given above
     }

    //last state token
    $stoken = $gWeb->find('fb_state',true);
 
    if(!empty($code) && ($_REQUEST['state'] == $stoken)) {
    
    //request to get access token
    $fbTokenUrl = "https://graph.facebook.com/oauth/access_token?client_id=".$fbAppId ;
    $fbTokenUrl .= "&redirect_uri=" . urlencode($fbCallback). "&client_secret=" . $fbAppSecret ;
    $fbTokenUrl .= "&code=" . $code;
  
    $response = file_get_contents($fbTokenUrl);
    $params = null;
    parse_str($response, $params);

    $graph_url = "https://graph.facebook.com/me?access_token=".$params['access_token'];
    $user = json_decode(file_get_contents($graph_url));
    processUser($user);

   }
   else {
    $message = "3mik.com and Facebook statedo not match.possible CSRF.";
    trigger_error($message,E_USER_ERROR);
  }

How to process the User information

Now you have the user information. At this point you can just dump the $user variable to see what all you get back from Facebook. From here on it depends on what you want to do. A simple scheme could be to

• Look at the $user->id  
• see if you have encountered this $user->id before 
• You can issue a get_or_create($user) to store this user information in your DB
• Start a session on your website for this user

There are many other hairy details like merging accounts, logout etc. However, I hope this post has given you sufficient information to get started.

Update:- 
Here is the code to  process user information

 


function processUser($user) {
    // exisitng record ? find on facebook_id
    // New record - create login + facebook record
    // start login session  
    $id = $user->id;
    $name = $user->name;
    $firstName = $user->first_name ;
    $lastName = $user->last_name ;
    $link = $user->link ;
    $gender = $user->gender ;
    $email = $user->email ;

    // do not know what facebook will return
    // we consider auth to be good enough for a user
    if(empty($name) && empty($firstName)) {
        $name = "Anonymous" ;
    }

    $message = sprintf("Login:Facebook :: id %d ,email %s \n",$id,$email); 
    Logger::getInstance()->info($message);

    $facebookDao = new \com\indigloo\sc\dao\Facebook();
    $loginId = $facebookDao->getOrCreate($id,$name,$firstName,$lastName,$link,$gender,$email);


    if(empty($loginId)) {
        trigger_error("Not able to create login for facebook user",E_USER_ERROR);
    }

    \com\indigloo\sc\auth\Login::startFacebookSession($loginId,$name);
    header("Location: / ");
}

Here you can store your Facebook users in a DB table. There you can check if you already have this Facebook ID. If you are using MYSQL then please do not use int(11) for storing facebook.ID, use string.

You can see a working example at our site : http://www.3mik.com/user/login.php
Questions/Issues/Comments about this post? - send to my blogger-id@gmail

Find and move files on macosx lion hard disk using perl File::Find

There are times in your life when doing a bit of programming can really be a convenience, a time saving convenience. These are the times when even your wife would agree that being a programmer has its advantages.

Now a couple of aeons ago I had a G4 macbook running panther. When I switched from that machine, I just dumped that hard disk on a western digital external drive. Then I was using a Thinkpad and again the Thinkpad disk got dumped on same WD external hard drive when I moved to my current MBP.

Problem now is that locating photos or music on this WD external drive can be a pain. I had really created a deeply nested "I am organized" directory structure on my old G4 and Thinkpad. (The argument is settled in my mind now, never created directories more than 2 level deep no matter what people say. Search is always faster than locating a file by traversal. All your little ontological schemes are totally arbitrary and you are certain to forget your arbitrary "conventions" after 3 years)

I am looking for a way to "flatten" this nested directory tree and move all the photos scattered on this disk in one place. Ditto for music and PDF files and whatever I care about. I want to create a new directory structure that is flat and for that I need to find files in old tree and move them to new tree. Find files on a hard disk and move to a new directory tree. what can be simpler?

First I try bash find and mv command using xargs

find /Volumes/Elements/iBook/Music/ -name *.MP3 -print0 | xargs -0 -I {} mv {} /Volumes/Elements/Music/ibook/{}

The problem with this scheme is

• There can be two files with same name but different content (result of importing from 2 digital cameras)
• Extracting base name can be difficult when file contains spaces (none of the suggested trick worked for bash shipped with my mac osx lion)
• You may want to run some rules on source as well as target, with Bash programming is difficult

find and mv with xargs will work for simple cases and I suggest using them. However for my find and move case I found perl File::Find to be a better fit. 

• Perl File:Find is fast - no complaints with speed
• I can access perl and all the programming logic, like attaching a counter to file name etc. Programming perl is preferable to programming bash
• With perl File::Find and closures, I can easily reuse the logic across different sources
• World is full of perl File::Find ready made examples

with about 20 mins. of work I was able to whip up my scripts and find and moves files in desired directory structure. Now I can browse the dump of my old hard disks easily and decide on what to keep and what to throw away

#!/usr/bin/perl
use strict;
use warnings;
use File::Basename;
use File::Find;

our $count = 0 ;

my $ibook_dir = "/Volumes/Elements/iBook";
my $thinkpad_dir = "/Volumes/Elements/Thinkpad-R60";

my $wanted = make_wanted(\&move_media,'/Volumes/Elements/media');
find($wanted, $thinkpad_dir);


#http://www.perlmonks.org/?node_id=109068

sub make_wanted { 
 my $wanted= shift;                      # get the "real" wanted function
    my @args= @_;                           # "freeze" the arguments
    my $sub= sub { $wanted->( @args);  };   # generate the anon sub
    return $sub;                            # return it
}


sub move_media {
 my @args = @_ ;
 my $file = $File::Find::name;
 
 $file =~ s,/,\\,g;
 #return unless -f $file;
 return if $file =~ /THUMBS/i ;
 return unless $file =~ /\.mp3|\.avi|\.mov|\.mpeg|\.mpg|\.mpeg4|\.mp4|\.3gp|\.3gpp|\.h264|\.wmv|\.flv/i;

 #replace backslash with slash
 # we are getting backslash from File::Find on macosx
 $file =~ s/\\/\//g;
 my $mvname = fileparse($file);
 #quote source - otherwise mv command fails
 print "mv \"$file\"  \"$args[0]/$mvname\" \n" ;
}

sub move_photo {
 my @args = @_ ;
 my $file = $File::Find::name;
 
 $file =~ s,/,\\,g;
 #return unless -f $file;
 return if $file =~ /THUMBS/i ;
 return if $file =~ /\.svn/ ;
 return if $file =~ /gloodev/ ;
 return if $file =~ /pgsem/i ;
 return if $file =~ /DMC/i ;

 return unless $file =~ /\.JPG|\.jpg|\.jpeg|\.JPEG/;

 $file =~ s/\\/\//g;
 my $x = fileparse($file);
 $count++ ;
 my $mvname = $count."_".$x;
 #quote source - otherwise mv command fails
 print "mv \"$file\"  \"$args[0]/$mvname\" \n" ;
}

1. fileparse routine is to get base name out of full file name
2. we have a global count variable declared in "our namespace"
3. mv cannot handle spaces in names so we need to quote such file names
4. The closure to create custom perl File find functions that accept our parameters from outside is taken from perl monk site

Reddit pagination scheme and addressing concerns about mysql performance and SEO

Pagination of search results or of blog pages is a simple technique yet you need to be aware of few performance concerns especially if your backend is an RDBMS like MySQL. The simplest scheme goes like

• Get a page size
• Get a page offset (or page number)
• sort on some column
• use LIMIT with offset

The queries you form are like,

SELECT post.title, post.summary from t_post post order by post.created_on DESC LIMIT 100,20

where 100 is the offset and 20 is result size.  The problem with such queries is the offset. The performance is fine for smaller offsets but for bigger offsets (first number after LIMIT) you are examining a lot of rows even with an index. Naturally that has a performance impact. check LIMIT performance optimization on mysql performance blog.

So what to do? The remedy is to drop the offset from queries. You needed the offset to locate the starting rows (pointer to your results) and you need something else to locate the range of rows for you. The solution is to use WHERE clause to locate rows. You should check this nice presentation by Surat Bhati and Rick James done during Percona 2009 conference.

www.percona.com/ppc2009/PPC2009_mysql_pagination.pdf 

To locate the right range using where clause you need a column that can sort your data. For date based sorting an auto increment key is a nice proxy.

SEO concerns

Suppose you had 3 pages and you served rows 1-20 from /page/3, rows 21-40 from /page/2 and rows 41-60 from /page/1. After inserting one more "page of data", /page/1 will fetch records 61-80, /page/2 will fetch 41-60, /page/3 will fetch 21-40 etc. The problem here is that all the pages have changed as a result of inserting "one more page". You can run into crawler indexing issues if that happens. (Every day every URL on your site is changing)

A better scheme is to use a script with query parameters to generate such summary pages (Check slashdot)
or use static pagination URL (check Digg)

Also, it may not be advisable to go on ad infinitum and you can refuse to show more results after user has browsed a certain number of pages (deep into your site)

Page link rendering concerns 


You will typically see two schemes. First is  a simple Next and Previous base scheme and second is rendering links to all the individual pages (1,2,3...10) with a total count. I believe the modern way is to use just the previous and Next links. Displaying total number of records is an absolute dud and user would any day prefer  to locate results directly via search than browsing.

Reddit site pagination

Now, armed with all this knowledge, lets see why reddit uses a  pagination scheme like (URL of type http://www.reddit.com/?count=25&after=t3_nu9we)

I know reddit source code is out there in open but I am not going to read it. Let's take an educated guess here. Most likely after parameter is in base36 and you can just use the php base_convert function to see the value in decimal. This number is used in where clause to locate the start row with a standard page size.

 count is keeping track of how deep you are and at some point the site will refuse to paginate further. Again, I am not going to click link after link but you can do the verification :D

Spell check an entire web site

Okay so here is the problem
Q. How do you spell check an entire web site?


Now in true reductionist programmer's spirit you can ask a simpler question. How do I spell check one web page? If you are on a linux machine then answer is pretty simple




 $links -dump    | spell  


links is a text browser and from links man pages


-dump


              Write formatted document to stdout


so links -dump solves a big headache of parsing all that html to produce text output. If you are working with Java, you can use swing toolkit to do a similar trick. You also have a google gdata class that can do html to text. However links is neat and I have no intentions to programmatically do what I can get ready made from a tool. I am here to just write glue scripts and be done.


Now we have to solve the problem of getting a list of links from a starting page (or host base address). Here again YMMV but I decided to use perl www mechanize module. You can also use LinkExtor or fetch page and parse using sed or do a regex matching or whatever. Point is, you can get a list of web pages from a starting address and then you can feed these pages to links, get a text dump and feed that to spell.


This solution totally works using duct tape and utilizing existing tools in true unix fashion. However you can be done spell checking an entire website without spending much effort from your side.

rjha @mbp ~/code/misc $ cat web-links.pl 
use strict ; 
use warnings;
use WWW::Mechanize;

    my $mech = WWW::Mechanize->new( autocheck => 1 );
    my $base = $ARGV[0];
    # @todo null test for $url 
    $mech->get($base);
    # only consider url with converse in them
    my @links = $mech->find_all_links('tag' => 'a');
            
    for my $link (@links) {
            printf "%s \n", $link->url;
    }

Above perl script will dump all the links into a file from where a shell script will pick them and feed to links -dump and spell.

rjha @mbp ~/code/misc $ cat spell.sh 
#!/bin/bash
FILE=tmplinks
perl web-links.pl $1 > $FILE

while read line
do
    echo $line
    links -dump $line | spell
    echo 
done < $FILE

Thats it! First we learnt to spell check a web page and then we extended the scheme to spell check all the web pages of a web site. spelling errors can look unprofessional on a website and with this trick you make sure you can see the obvious ones!

Compiling php 5.3.7 with FPM on mac osx lion

My work machine is a new MBP 13 with mac osX lion. This machine had php 5.x installed in factory but I wanted to compile the latest PHP (5.3.x). Here are the required steps. One thing you have to be aware of is that pre-requisites may vary depending on what all module you want to bake in. I wanted mcrypt and GD2. GD2 has dependency on libpng and libjpeg. Also, I wanted to use mysql native driver.

Before compiling set the compiler flags to be 64 bit x86. Download required libraries (libmcrypt, libpng, libjpeg) from respective sites and do a configure make sequence.

Install dependencies

• php-5.3.7.tar.gz
• libmcrypt-2.5.8.tar.gz
• libpng-1.5.4.tar.gz
• jpegsrc.v8c.tar.tar.gz

Use following information to compile and install our dependencies

  CFLAGS="-arch x86_64"   
 echo $CFLAGS   
 ./configure --prefix=/usr/local make make install  

mysql was already installed on mac osx lion and I went with the default install of mysql. After compiling and installing the dependencies we compile PHP tar ball. Note that we supply required library paths to configure script. Use the --enable-fpm switch to enable baking in FPM module.

  ./configure --prefix=/usr/local --enable-fpm --with-mysql=mysqlnd   
 --with-mysqli=mysqlnd --with-openssl --with-pcre-regex --with-mhash   
 --with-curl --with-mcrypt=/usr/local/lib --with-gd   
 --with-png-dir=/usr/local --with-jpeg-dir=/usr/local

After successful install, check modules using $php -m. CLI, CGI, JSON etc. are default with 5.3.7 source bundle. install nginx from source. Nginx normally has only one dependency, PCRE. Now both nginx and php with fpm are installed.

 PHP layout

After successful install, PHP executable will be in /usr/local/bin and php.ini is in /usr/local/lib. php-fpm is in /usr/local/sbin and php-fpm config files are in /usr/local/etc

Start/stop NGINX, php-fpm and mysql

  sudo /usr/local/sbin/nginx -- to start nginx 
sudo /usr/local/sbin/nginx -s stop -- to stop nginx 
sudo /usr/local/sbin/php-fpm to start fpm 
sudo kill `cat /usr/local/var/run/php-fpm.pid` to kill fpm 
sudo /usr/local/mysql/bin/mysqld_safe to start mysql

Faster Netbeans startup

There are two easy ways for faster Netbeans startup

1. Un-install/Deactivate ununsed plugin from Tool|Plugin Menu

2. Go to /etc/conf and open netbeans.conf file, change the Java Heap size there

change -J-Xms32m To -J-Xms512m

anonymous upload using vsftpd

Alright. I need to run an FTP server on my UBUNTU slice @ slicehost. The content is not very sensitive so I am okay with anonymous login but with a password. I had to make some changes to the default /etc/vsftpd.conf to make it work. My requirement is to allow anonymous users to upload content and then that content should be readable by other local accounts (change file permissions after vsftpd has uploaded them)

To allow anonymous upload and file mode changes with vsftpd on ubuntu I had to make following changes from default install

• create a new directory called pub inside /home/ftp , make sure this directory is writable and not owned by ftp user, see man page of vsftpd
• Adjust file_mode and anon_umask
• Create an email password file

Here is the full configuration file

1:  listen=YES 
2:  anonymous_enable=YES  
3:  write_enable=YES  
4:  anon_upload_enable=YES  
5:  anon_root=/home/ftp  
6:  anon_mkdir_write_enable=YES  
7:  dirmessage_enable=YES  
8:  xferlog_enable=YES  
9:  connect_from_port_20=YES  
10:  ftpd_banner=Welcome to indigloo art FTP service.  
11:  secure_chroot_dir=/var/run/vsftpd  
12:  pam_service_name=vsftpd  
13:  rsa_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem  
14:  rsa_private_key_file=/etc/ssl/private/ssl-cert-snakeoil.key  
15:  file_open_mode=0777  
16:  anon_umask=022  
17:  secure_email_list_enable=YES  
18:  email_password_file=/home/rjha/vsftpd.passwords  

Installing Apache portable runtime (APR) for tomcat 6.0.x

Tomcat can now use Apache Portable Runtime and using APR results in performance boost.
Below we detail the steps to get tomcat native library installed on a linux system
(I am using Red hat but steps should be similar for other distros)

Download and install APR

wget http://apache.norhex.com/apr/apr-1.3.3.tar.gz
    mkdir x
    mv apr-1.3.3.tar.gz x/
    408  tar -zxvf apr-1.3.3.tar.gz
    409  cd apr-1.3.3
    416  gcc -v
    417  ./configure --prefix=/opt/apr
    418  make
    419  make install

Download and compile openssl tarball corresponding to your system
(Latest version maybe incompatible with your system)

       424  openssl version -a
      426  wget http://www.openssl.org/source/openssl-0.9.8b.tar.gz
      427  tar -zxvf openssl-0.9.8b.tar.gz
      429  cd openssl-0.9.8b
      431  ./Configure
      433  ./config --prefix=/opt/apr --openssldir=/opt/openssl shared
      434  make
      435  make test
      436  make install
 

Download and install sun JDK .

You have to put /usr/java/latest/bin in your path to pick SUN JDK ahead of GCJ.

 
       466  uname -a
      467  java -version
      468  links http://java.sun.com/javase/downloads/index.jsp
      chmod +x jdk-6u10-linux-i586-rpm.bin
      411  ./jdk-6u10-linux-i586-rpm.bin
      418  /usr/java/latest/bin/java -version
      419  echo $PATH
      423  vi .bash_profile
      424  . .bash_profile
      425  echo $PATH
      426  java -version


  

Download and install tomcat native source

  
   443  wget http://tomcat.heanet.ie/native/1.1.10/source/tomcat-native-1.1.10-src.tar.gz
  444  tar -zxvf tomcat-native-1.1.9-src.tar.gz
  445  cd tomcat-native-1.1.10-src/jni/native
  464  ./configure  --help
  440  ./configure --prefix=/opt/apr/ --with-ssl=/opt/apr --with-apr=/opt/apr
           --with-java-home=/usr/java/latest --with-java-platform=2
   460  make
   467  make install
   468  ls /opt/apr/lib/
   469  ldd /opt/apr/lib/libtcnative-1.so.0
   

if configure complains about not finding openSSL then set LD_LIBRARY_PATH to include /opt/apr/lib.

  454  echo $LD_LIBRARY_PATH
  455  LD_LIBRARY_PATH=/opt/apr/lib
  456  export LD_LIBRARY_PATH
 

Put tomcat native library in LD_LIBRARY_PATH before starting tomcat. Start tomcat and check catalina log file to verify that tomcat is using APR.

  
# setup native library in path
LD_LIBRARY_PATH=/opt/apr/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH
/usr/local/apache-tomcat-6.0.16/bin/startup.sh

 
   

Making jar files dependency graph using jarjar and dot

Our project has grown quite a bit over the past couple of years and with it the library folder containing third party jars . So much so that we are not sure now as to what is used and what is not. So we decided to do a bit of cleanup and remove the unused jars. However before we can do that we need to know which ones are used by code and which ones are not.

I have not been able to locate a tool that can print out run time dependencies of java code on a jar file. Anyway I am not sure if that can be done at all with static analysis. Leaving aside run time dependencies there are some tools that can print out the static dependencies between two jar files. After some googling and general goofing I settled on jarjar (http://code.google.com/p/jarjar/)

jarjar can print the dependencies between any two jar files (see http://sixlegs.com/blog/java/depfind.html). jarjar can print dependencies between multiple jar files also. Just separate the jar file names with a colon (UNIX) or semi colon (MS-WIN32) What I wanted was the multiple file version of command , A panacea to print dependencies between all my jar files in lib folder.

I made a batch file with all the jar file names (Using a perl script to print the required batch file) along the lines of
/>java -jar jarjar-1.0rc7.jar find jar activation-1.1.jar;ant-1.6.5.jar;antlr-2.7.2.jar.............. 70 more files.

I used the following script to generate my batch file

$argc = @ARGV;
if($argc != 1 ) {
print " USAGE : $perl read-dir.pl \n ";
exit(0);
}


$mydir = $ARGV[0] ;

opendir(dhandle, $mydir) || die("Cannot open directory");
@thefiles= readdir(dhandle);
closedir(dhandle);

print " java -jar jarjar-1.0rc7.jar --find --level=jar " ;

foreach $f (@thefiles) {
unless ( ($f eq ".") || ($f eq "..") || ($f eq ".svn" )) {
print "$f:";
}
}

The batch file runs for a while and then dies with an ArrayOutOfBoundException. Now I did not want to tackle any problems in jarjar so I took a workaround. Since jarjar can print dependencies between any two jars , I decided to use that fact in my solution. I just wrote a perl script that would store all the jar files in a folder and compare a pair of them at a time using jarjar. If we find any dependency we print it on console. (script attached below) The output of this script can be saved as a "dot" file.

The script ran for quite some time but in the end it produced dependencies of all jar files in the folder in dot-ready format. dot is a nifty program to print graphs that comes bundled with graphviz. (Graphviz website) if you print the dependencies in a format that dot can understand then you can use dot on this output to produce a graph. That was the intention. If you install graphviz then dot should already be in your system path.

Just run dot over the outfile file like, ( This link from oreilly linux dev center may be useful )

D:/project/lib/> dot -Tpng -o graph.png .dot


And now you have the dependencies graph. If you are not satisfied with quality then you can try OmniGraffle see link ( http://paco.to/blog/000040.html). However for Omnigraffle make sure

• You save the output of perl script as a dot file (extension dot like jars-data.dot)
• OmniGraffle will run on a Mac only

And here is the perl script to do dependency analysis

$argc = @ARGV;
if($argc != 1 ) {
    print " USAGE : $perl jarjar.pl <dirname> \n ";
    exit(0);
}


$mydir = $ARGV[0] ;

opendir(dhandle, $mydir) || die("Cannot open directory");
@files= readdir(dhandle);
closedir(dhandle);

# make extra copy of  files
@alljars = @files ;

# program may end abruptly for all possible combnations
# To control the jar combinations we can look for files
# starting with following alphabets only
#

%allowed = qw/o p q r s t u v w x y z/;


foreach $file (@files) {
    $starts_with  = substr($file,0,1);
    # process  $file that starts with allowed letter and ends  with jar
    if ( $file=~/jar$/ && exists($allowed{$starts_with})) {
        # go over all jar files
        foreach $jar(@alljars) {
            if ( $jar=~/jar$/ ) {
                # compare this jar against the file
                $command = "java -jar jarjar-1.0rc7.jar  find jar $file  $jar \n ";
                #print $command ;
                #open command pipe
                open(CMD, "$command|");
                $output = <CMD>;
                close(CMD);
                # print  output in graphviz dot format 
                if(length($output) > 0 ) {
                    print " \"$file\" ->  \"$jar\" \n " ;
                }

           
            }
        }
    }
       
   
}

exit(0);

Load testing stand alone classes with Apache JMeter

We want to collect timing information for some of your class methods. The kind of question we are interested in is, how long method A or method B takes. We would also like to subject our methods to multi thread load and analyze the performance under load. We also want pretty graphs of performance tests plus wish to save all the individual sample in some file.

Instead of doing this task by writing our own instrumentation and system.currentTimeMillis(), we decided to take a different path. Since we are already using apache JMeter for our web application load testing , we started looking for ways to do load testing of our class methods using apache JMeter. Finally, the task turned out to be much simpler than anticipated.

First up, you would like to create some custom samplers that apache JMeter can use. Below we present one such custom sampler.

public class JPACreateTest implements JavaSamplerClient {

public Arguments getDefaultParameters() {
    Arguments params = new Arguments();
    return params;
}

public SampleResult runTest(JavaSamplerContext context) {
    SampleResult results = new SampleResult();
    String subject = null ;
    String message = null ;
    String userName = " jpa console" ;

    try {
        Date now = new Date();
        String t = DateFormat.getDateTimeInstance().format(now);
        subject = " JPA create test @ " + t;
        message = " JPA create message @ " + t ;

        results.sampleStart();
        // call your methods
        MessageService messageService = new MessageServiceImpl();
        messageService.createNewMessage(userName, subject , message);
        results.sampleEnd();

        results.setSuccessful(true);

    } catch(Exception ex) {
        results.setSuccessful(false);
        ex.printStackTrace();
    }
    return results ;


}

public void setupTest(JavaSamplerContext arg0) {
// no initialization required

}

public void teardownTest(JavaSamplerContext arg0) {
// no cleanup required

}

Implement Java Sample Client interface and return a Sample Result. Now to use this Sampler in JMeter please follow these steps:

• Put your sampler implementation in JMeter classpath. Better thing is to package the application class files inside a jar and drop the jar in JMeter/lib/ext folder.
• Adjust the value of property user.classpath to include all the jars that are used by your application jar file. JMeter.properties file

Now you have created a sampler, put the jar file containing sampler inside apache JMeter lib/ext folder and all the dependencies of sampler jar is known to JMeter via user.classpath property. Now re-start JMeter UI. To add your sampler,

• Right click context Menu
• Add Sampler - Java Request
• Select your Sampler from the drop down

Two Links have been especially useful

• DEV2DEV Article on BEA - Using JMeter to performance test web services
• Apache korea - JMeter extension scenario

using sphinx freetext search engine on windows xp

This entry deals with using (testing) the open source full text search engine sphinx on a win32 box. Official sphinx document as well as other links and tutorials for sphinx are Linux oriented. So a step by step newbie sort of guide would be helpful , I believe. First go to the sphinx website and download the zip file containing binaries. If you feel you are up to compiling sphinx on win32 boxes then you should not read newbie guides anyway! Assumption is that you already have a working mysql installation on your windows box.

http://www.sphinxsearch.com/downloads.html

Download the zip file and extract in some folder of your choice. First step in our process is to create a sphinx.conf file. This sphinx.conf file will be used by all the programs. For simplicity's sake, lets say we have a table that has only 3 columns, id, title and description. we want to index text in title and description fields. Our test table is named "test" and it sits in a database called testdb.

Below is how our config file looks like,

source test {
type = mysql
sql_host = localhost
sql_user = gloo
sql_pass = gloo
sql_db = testdb
sql_port = 3306

# indexer query

sql_query = \ SELECT \ id, title,description FROM test;

# document info query
sql_query_info = SELECT * FROM test WHERE id=$id
}

index test {
source = test
path = ./test-index
morphology = stem_en
min_word_len = 3
min_prefix_len = 0
min_infix_len = 3
}

searchd {
port = 3312
}

You have to know a bit about sphinx config file. see this link on the sphinx web site for all the options. Now we are ready to index our test table. To do so just go to the folder where you extracted the sphinx binaries. You should see three programs, indexer,searchd and search as shown in the picture below.

Just go to command line and type command "indexer". Make sure that your mysql service is running before you try to use the indexer. Indexer will list out all possible options. You can run it with the path to your configuration file. Again , read the sphinx documentation for full information. If you type in the indexer command, the indexer will run and create index of your data. you can verify this by looking inside your folder ( or wherever you set the path to index files) . some new files should have been created.

So now, we have used the indexer and created the indexes out of data. But how do we use these indexes? What we do here is start the sphinx search daemon that other programs and API can connect to. It is also possible to build sphinx as mysql plug-in. But lets go with the sphinx searchd daemon for now. start the searchd daemon.

sphinx searchd daemon is now listening for connections. To test, now use the search client program. Later we will see how to use the PHP API to do searches.
You will find the following links useful

• IBM developer works article on sphinx
• acts_as_sphinx rails plug-in article on data noise