Sunday, December 02, 2012

Add folders to rackspace cloud file container with php and cURL


We use rackspace cloudfile container to publish our css and javascript files to Akamai CDN. They (rackspace)  has a nice one click deployment of cloud files to the CDN. Assuming you are new to this, the recipe goes like
  • Access your rackspace cloud account
  • Create a new container in rackspace cloud files
  • Upload your files to that container
  • Select container and in the bottom panel - just select publish to CDN.
You can map a CNAME record in your DNS server to make the the long rackspace CDN URL manageable (or something that your application understands)

The css and javascript files refer to assets (logos/ sprites/ images and such) and we need to upload those as well to the CDN enabled container. However we do not want to do this manually and we wish to create and upload file + assets  bundle as part of build process (automatic and not manual)

Rackspace provides PHP libraries to make such tasks as creating a container and uploading files to them easier. However I planned to take the plain cURL route because 

  1. Number of files is not large
  2. The files themselves are not large 
  3. I was not in a mood to download and install yet another library and learn the API

So I just rolled my sleeve and wrote this PHP script that would take a local folder and upload its content in a pseudo_directory hierarchy to rackspace cloud file container. YMMV but sometimes we all need quick and dirty and then this script can come in handy. 

Relevant links





    error_reporting(-1);

    function do_upload($ch2,$auth,$fname) {

        $headers = array();
        $grab = array("X-Auth-Token");
        $host = $auth["X-Storage-Url"];

        foreach($auth as $name => $value) {
            if(in_array($name,$grab)) {
                array_push($headers, "$name: $value");
            }
        }


        // Content-Type
        $finfo = finfo_open(FILEINFO_MIME_TYPE);
        $mime = finfo_file($finfo, $fname);
        finfo_close($finfo);
        $etag = md5_file($fname);

        array_push($headers, "Content-Type: $mime");
        array_push($headers, "ETag: $etag");

        $fp = fopen($fname, "r");
        $fsize = filesize($fname);
        // asset is container name
        // upload in a pseudo_dir structure
        // /asset/css/...
        //
        $url = $host. "/asset/".$fname ;
        printf("HTTP PUT %s  to => %s \n",$fname,$url); 

         $options = array(
            CURLOPT_TIMEOUT => 60 ,
            CURLOPT_RETURNTRANSFER => 1 ,
            CURLOPT_FOLLOWLOCATION => 1,
            CURLOPT_HTTPHEADER => $headers,
            CURLOPT_URL => $url,
            CURLOPT_VERBOSE => false,
            CURLOPT_HEADER => 1,
            CURLOPT_PUT => 1 ,
            CURLOPT_INFILE => $fp,
            CURLOPT_INFILESIZE => $fsize);



        // print_r($headers); 
        // Do a PUT operation

        curl_setopt_array($ch2, $options);
        $response = curl_exec ($ch2);
    }


    function do_auth() {

        $ch = curl_init();

        $host = "https://identity.api.rackspacecloud.com/v1.0" ;
        $apiKey = "xxxxxxx" ;
        $user = "yyyyyyyyy" ;

        $headers = array(
            "X-Auth-Key: $apiKey " ,
            "X-Auth-User: $user");
           $options = array(
            CURLOPT_TIMEOUT => 60 ,
            CURLOPT_RETURNTRANSFER => 1 ,
            CURLOPT_FOLLOWLOCATION => 1,
            CURLOPT_HTTPHEADER => $headers,
            CURLOPT_URL => $host,
            CURLOPT_VERBOSE => false,
            CURLOPT_HEADER => 1);

        curl_setopt_array($ch, $options);
        $response = curl_exec ($ch);
        curl_close($ch);

        list($headers, $body) = explode("\r\n\r\n", $response, 2);

        $lines = explode("\n",$headers);
        $auth = array();
        $grab = array("X-Storage-Token", "X-Storage-Url","X-Auth-Token");

        foreach($lines as $line ) {
            $parts = explode(" ",$line);
            $name = $parts[0] ;
            $name = trim($name,": ");

            if(in_array($name,$grab)) {
                $auth[$name] = trim($parts[1]);
            }

        } 

        return $auth ;

    }


     $auth = do_auth();
    printf(" \n **** parsed auth headers for PUT **** \n");
    print_r($auth);

    $ch2 = curl_init();
    // get all files in css dir
    $files = array();

    // load everything from local css folder into
    // /asset/css/local-file-path
    foreach (new RecursiveIteratorIterator(new RecursiveDirectoryIterator('css')) as $path) {
        $filename = sprintf("%s",$path);
        $pos = strrpos($filename,"/");
        if($pos !== false) {
            $last = substr($filename,$pos+1);
            if($last == '.' || $last == '..') {
                printf(" ignore file :: %s \n",$filename); 
            }else {
                array_push($files,$filename);
            }
        }
    }

    foreach($files as $file){
        do_upload($ch2,$auth,$file);
    }

    curl_close($ch2);


 
       



Sunday, September 16, 2012

count lines of code (cloc) in multiple PHP source folders



Over the years, I have used many tools that generate CLOC reports (how many lines of code you have written). The one I really like is cloc project on sourceforge . The tool is written in perl and is quite flexible.

Our problem is that we have a web application and that means

  • we need to ignore certain folders when counting lines. Examples would be compiled templates, generated minified files, third party libraries and such.
  •  We need to create custom definitions for some files (like .tmpl is our view templates and .inc are our PHP include files) Without custom definitions, the tool would count .inc as "PASCAL" file and ignore .tmpl files !!!
  • We need to sum the individual reports over. We have dependencies and the project is split across different source trees and not everything is under one root.

Ignoring folders/files with cloc is dead simple.  Just pass a file with files and directories in --exclude-list-file.

To generate custom definitions (so .tmpl are counted as view templates and .inc as PHP includes)  first use cloc.pl to write standard definition files using --write-lang-def switch. Later on we will modify this file and ask cloc to use our definition instead of the standard one.

Exclude list

Exclude list is a file with one entry per line.


rjha@mint13 ~/code/github/sc/deploy/apps/cloc $ cat cloc-ignore 
/home/rjha/code/github/sc/web/css/bundle-full.css
/home/rjha/code/github/sc/web/css/bundle.css
/home/rjha/code/github/sc/web/js/bundle.js
/home/rjha/code/github/sc/web/js/bundle-full.js
/home/rjha/code/github/sc/web/compiled




creating custom definitions file

First generate the definitions of extensions used by cloc tool. Then we add our custom definitions to this file. Later on we will use this file to supply file definitions to cloc tool. Just copy the  existing matching definitions for your custom file types.


 ./cloc-1.56.pl --write-lang-def=cloc.def

Add to this file following definitions

View Template
    filter remove_html_comments
    filter call_regexp_common HTML
    extension tmpl
    3rd_gen_scale 1.0


PHP Include
    filter remove_matches ^\s*#
    filter remove_matches ^\s*//
    filter call_regexp_common C
    filter remove_inline #.*$
    filter remove_inline //.*$
    extension inc
    3rd_gen_scale 1.0




script to sum reports


rjha@mint13 ~/code/github/sc/deploy/apps/cloc $ cat cloc.sh 
# web folder - use an ignore list
# web folder - needs custom definitions

./cloc-1.56.pl  --exclude-list-file=./cloc-ignore --read-lang-def=./cloc.def   /home/rjha/code/github/sc/web --report-file=sc.web.report
./cloc-1.56.pl  /home/rjha/code/github/sc/lib --report-file=sc.lib.report
./cloc-1.56.pl  /home/rjha/code/github/webgloo/lib/com/indigloo --report-file=webgloo.report
# sum the reports 
./cloc-1.56.pl  --read-lang-def=./cloc.def  --sum-reports *.report
#remove tmp
rm *.report




Here we are running cloc on three separate folders. First one uses an ignore list and a custom definition file (created earlier).  second and third are standard cloc reports.
Finally we use --sum-reports option to produce the final report across three different source trees.



rjha@mint13 ~/code/github/sc/deploy/apps/cloc $ ./cloc.sh 
     239 text files.
     239 unique files.                                          
      54 files ignored.
Wrote sc.web.report
      94 text files.
      94 unique files.                              
       0 files ignored.
Wrote sc.lib.report
      41 text files.
      41 unique files.                              
       0 files ignored.
Wrote webgloo.report

http://cloc.sourceforge.net v 1.56
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
PHP                            226           4795           1340          13256
PHP Include                     32            243             31            902
CSS                              1            219             33            850
Javascript                       1            249             99            823
View Template                   54            185              5            681
HTML                             4             19              0             46
XML                              2              0              0             44
-------------------------------------------------------------------------------
SUM:                           320           5710           1508          16602
-------------------------------------------------------------------------------





Friday, September 14, 2012

infinite scrolling but on steroids



we use infinite scrolling plugin by paul irish (https://github.com/paulirish/infinite-scroll) on 3mik now. However Problem with the vanilla plugin is that it assumes that the next URL only depends on current page number. Only  a naive pagination scheme would depend on only one variable.

As you go deeper into pages, you have to scan more and more records before you can arrive at the records you are interested in. For this reason,   real world pagination scheme rely on two variables (see these slides from percona conference  and this blog post )


  • current page number
  • last record id (of previous page ) or first record id (of next page)

An example would be the instagram API

To Apply infinite scrolling pattern to pages with such 2 variable pagination scheme, you need to do additional work. So here is our fork of infinite scroll plugin that documents this use case and provides sample server  PHP scripts.




Sunday, June 24, 2012

Debian Linux with 3D graphics in vmware fusion 4.1

Joostlek has a nice post about enabling 3D graphics in linux guests on vmware.  I followed this same post for installing 3D graphics support for my Debian wheezy desktop.  I wanted to see the gnome shell in action after I heard so many (*bad*) things about it!

I had to install some extra packages and build dependencies for Mesa and if you also installed wheezy from net installer then these "extra" steps will be useful for you. Also his post describes building it on Ubuntu. These two points are the only difference from his post. I have vmware fusion 4.1.3 running on a 13" macbook pro.

Following packages are from above post


sudo apt-get install autoconf libtool xutils-dev xorg-dev flex bison libx11-xcb-dev libxcb-glx0-dev g++ git

And here are the extra packages that were missing in my net installer


  •  sudo aptitude install  libxcb-dri2-0-dev
  •  sudo aptitude install  libxcb-xfixes0 libxcb-xfixes0-dev
  •  sudo aptitude install llvm
  •  sudo aptitude install libxml2 libxml2-dev
  •  sudo aptitude install python-libxml2
  •  sudo aptitude install build-essential


You also have to install build dependencies for Mesa. Check the Debian How to build Mesa page.


  •  sudo apt-get build-dep mesasudo apt-get build-dep mesa


Rest of the steps remain the same as in above post.  If you are using Debian wheezy after 21st June 2012 then you do not need to download a new kernel. DRM is enabled for your kernel. Just skip that step.

After the restart, I could log into gnome shell. The default leave a lot to desire. So I had to tweak my desktop for next  hour or so.  Here is the screenshot of my Debian wheezy  desktop running gnome-3 shell






Monday, June 18, 2012

lint like static code analysis tools for PHP

PHP is dynamically typed language and that means the error detection present in statically typed languages is absent. If you have an error in your script, it will not be detected till you have actually run it(apart from parse errors). The big, shine in your eyes errors are never a problem because they are detected quite easily during development. It is the silent types (doing X when you meant Y and X is a perfectly legit operation) that are more difficult.

Like, I will show you a piece of code I had re-factored  a while back


 
       static function find($subjectId,$objectId,$verb) {
            $mysqli = MySQL\Connection::getInstance()->getHandle();

            //sanitize input
            settype($subjectId,"integer");
            settype($objectId,"integer");
            settype($verb,"integer");

            $sql = " select count(id) as count from sc_bookmark " ;
            $sql .= " where subject_id = %d and object_id = %d and verb = %d ";
            $sql = sprintf($sql,$loginId,$itemId,$verb);

            $row = MySQL\Helper::fetchRow($mysqli, $sql);
            return $row;
        }

 



Here loginId was changed to subjectId in parameter list but I forgot to change the name of parameter I was passing to MySQL.  This used to result in a notice that was going to logs but no sign of program failure outside. That led me to say, enough is enough and I need to have a good static analysis tool for my PHP code.


There are many posts dedicated to the topic of PHP static analysis, sort of things that lint does for c code. The aim is to catch errors from just scanning the PHP code. When I started looking around, everyone talked about their favorite static analysis tool but no one gave me hard data or how it actually looks on your terminal.

I looked  at following tools

  1. PHP code sniffer (phpcs)
  2. PHP on command line with minus eel option (-l)
  3. PHP mess detector (phpmd)
  4. pfff/scheck

After a quick evaluation I (emphasized)  found phpmd to be the best of the lot.  Why?


One more tool worth checking should be https://github.com/sebastianbergmann/php-code-coverage
However I did not look at this one.


PHP code sniffer

phpcs looked too verbose out of the box. For 20 line perfectly legit code it generates 20 errors, most of that to do with style guidelines. Look at the output, I believe real  error will be buried under tons of such warnings. Sure, I did not look at how to tweak it and all but I believe most of the time first impressions are confined to out of box experience. Also, it could not detect the error I had. (error inside a class method)



FILE: /home/rjha/code/github/sc/lib/com/indigloo/sc/util/Asset.php
--------------------------------------------------------------------------------
FOUND 16 ERROR(S) AND 1 WARNING(S) AFFECTING 11 LINE(S)
--------------------------------------------------------------------------------
  2 | ERROR   | Missing file doc comment
  4 | ERROR   | Missing class doc comment
  4 | ERROR   | Opening brace of a class must be on the line after the
    |         | definition
  6 | ERROR   | Missing function doc comment
  6 | ERROR   | Opening brace should be on a new line
 17 | ERROR   | Expected "if (...) {\n"; found "if(...) {\n"
 20 | ERROR   | No space found after comma in function call
 24 | ERROR   | Expected "if (...) {\n"; found "if(...) {\n"
 28 | WARNING | Line exceeds 85 characters; contains 103 characters
 31 | ERROR   | Expected "if (...) {\n"; found "if(...) {\n"
 31 | ERROR   | No space found after comma in function call
 33 | ERROR   | No space found after comma in function call
 33 | ERROR   | No space found after comma in function call
 36 | ERROR   | Expected "if (...) {\n"; found "if(...) {\n"
 36 | ERROR   | No space found after comma in function call
 38 | ERROR   | No space found after comma in function call
 38 | ERROR   | No space found after comma in function call
--------------------------------------------------------------------------------




php -l (minus eel)

looks too limited. It could not detect errors inside class methods.

pfff/scheck

pfff is a project from facebook and it is a suite of tools. There are some interesting ocaml based tools there, like code map generator etc. scheck is the tool to do static checks. However pfff looks to be in alpha stage right now.

php mess detector

This could detect the problem inside a class method. Also, the tool is quick and you can decide on the kind of rules you would like to run.



rjha@mint ~/code/github/sc $ phpmd  lib/com/indigloo/sc/util/Asset.php text codesize,design,naming,unusedcode

/home/rjha/code/github/sc/lib/com/indigloo/sc/util/Asset.php:13 Avoid variables with short names like $ts. Configured minimum length is 3.




Saturday, May 19, 2012

Nice looking Debian Wheezy Desktop

Well, if you do not mind 1000 extra packages and 100 MB of extra memory then I would suggest you install Linux Mint LMDE edition. The nice thing about mint LMDE is that you get a nice looking desktop with the choice of MATE or Cinnamon. The fonts and rendering looks nice out of the box.

However, if you have just started with the Debian net installer and just installed your desktop then the rendering and fonts may be a bit out of shape! You may wonder if your 1300$ LCD screen is not seen by Debian at all! Fret not! With little patience and work, we can make our Debian desktop look real nice and polished.



Fix font configuration

We first copy the ubuntu font config files in Debian wheezy following the steps outlined in 
After that we install microsoft ttf fonts 
$sudo aptitude install  ttf-mscorefonts-installer

Now we reload the font config and clear the old font caches

$sudo fc-cache -fv 
$sudo dpkg-reconfigure fontconfig-config 
$sudo dpkg-reconfigure fontconfig

Fix font aliasing and rendering

The following combination works for LCD screens
System | Preferences | Customize L&F  (or whatever is equivalent for your DM of choice)

  • Turn subpixel hinting to RGB
  • Turn on font aliasing
  • Set font hinting to  slight


Change Mouse cursor

$sudo aptitude install dmz-cursor-theme
$ sudo update-alternatives --config x-cursor-theme

Preferences | customize L&F | Mouse cursor 
select DMZ white theme and you get a sane mouse instead of default x-cursor (circa 1991)

Install Flash


 Download 64 bit flash player from Adobe site, unpack
 $sudo cp libflashplayer.so /usr/lib/mozilla/plugins/.
 restart iceweasel, try viewing a youtube video.

Change login screen

By default wheezy is using gdm3 and the gdm3 login screens are, well, yuck! To change gdm3 login screen, one "simple way" is to create a new symlink for /usr/share/images/desktop-base/login-background.svg that points to your new login splash background. Please note that only the .png files work. The .jpg files would not work in this scheme.

Install lightdm

This provides a much better looking login screen. 

$sudo aptitude install lightdm 

verify that lightdm is the new display manager in /etc/X11/default-display-manager

To configure
$dpkg-reconfigure lightdm 

To change lightdm login screen background (greeter background) change background in /etc/lightdm/lightdm-gtk-greeter.conf (PNG images



After all these changes you will have a shiny looking Debian Wheezy Desktop :D Enjoy'






VMWare tools on Debian Wheezy inside Fusion

I have an early 2011 intel macbook and I run linux VM using Vmware fusion 4.x. Earlier I was using an Ubuntu VM for development that had 2.6.x kernel. Now, I wanted to upgrade my development VM to Debian wheezy for assorted reasons.  One of them was having a rolling distribution because I do not want to full upgrade my installation all the time.

The problem with installing Debian wheezy or any other linux 3.x series kernel in vmware fusion is that the vmware tools shipped with fusion is not compatible with linux 3.0 and 3.2 kernels. The VMware tools only work with 2.6.x kernel headers. Now this is a big headache as I need to mount my macbook directories inside my VM. Other nuisance is that your mouse will be captured inside vm window if you do not install the tools. in short, I want a life with vmware tools.

So open-vm-tools came to my rescue. This is one of the moments in life when you did not know about existence of something that is so essential to your existence! All my life of running linux VM on macbook and I did not know that this project exists! As it happens open-vm-tools are rolled with Debian so installing them is super easy.

First install the prerequisites


$aptitude install linux-headers-`uname -r` libx11-6 libx11-dev xorg libxtst6 psmisc build-essential ia32-libs ia32-libs-gtk

The Debian wiki mentions x-window-system and x-window-system-core but those packages are now provided by xorg. Now install the open-vm-tools packages using $sudo aptitude install

  • open-vm-tools 
  • open-vm-dkms      
  • open-vm-tools-dev  
  • open-vm-toolbox 


After the installation, try querying the vmware tools modules

%modinfo vmxnet
%modinfo vmhgfs

if the required kernel modules are not loaded then we have to do so via dkms (Dynamic kernel module support). To install required kernel modules

% dkms add open-vm-tools/2011.12.20
% dkms build open-vm-tools/2011.12.20
% dkms install open-vm-tools/2011.12.20

Last number in red is the module version, you can get this number by using  
$sudo aptitude show open-vm-tools 
Look for version string (after the + sign)



Run modinfo again and verify that we have the required modules. Shut down the VM. Map shares using VMware fusion settings. Start the VM. Now we should have VMware tools loaded.

$vmware-hgfsclient 
The output should match with the shared folders

To mount the shared folders, you can do something like. Here Public is the folder name that we shared via VMware fusion. We mount this folder at /mnt/hgfs.

$sudo mount -t  vmhgfs -v -o rw  .host:/Public /mnt/hgfs






Tuesday, May 01, 2012

Send Email using SendGrid PHP library on Github

We are hosting the 3mik.com boxes on Rackspace right now and  we get 40,000 emails/month free of cost  from one of their partners, SendGrid. We use SendGrid to send  emails from our application written in PHP.  The SendGrid documentation suggests using SMTP relays and swiftmail-4.x library.  However, SendGrid also has a PHP front-end library published on github that makes it super easy to send emails.

Here is one complete test script


    require_once($_SERVER['WEBGLOO_LIB_ROOT']. '/ext/sendgrid-php/SendGrid_loader.php');
    set_error_handler('offline_error_handler');
    $sendgrid = new SendGrid('your-sendgrid-login', 'your-sendgrid-password');
    $mail = new SendGrid\Mail();


     $mail->addTo('foo@gmail.com')->
       setFrom('foo@3mik.com')->
       setSubject('Sendgrid github PHP library test')->
       setText('Hello World! from sendgrid library')->
       setHtml('Hello World! from sendgrid github lib');  

    $response = $sendgrid->web->send($mail);
    // Error Handling:- 
    // sendgrid->web method uses curl_exec CURLOPT_RETURNTRANSFER set. 
    // This means you will get FALSE
    // when the send method fails at network level.
    // you get JSON response back when curl is able to communicate with the server
    // success from API is returned as
    // {"message":"success"}
    // Error from API is returned as
    // {"message": "error", "errors": ["Bad username / password"]}
    print_r($response);



?>

Though all SendGrid documentation exhorts you to use SMTP, I do not see a problem in using the WEB method.  As per this SO question, http://stackoverflow.com/questions/6193702/sendgrid-smtp-or-curl , " The  web API actually works faster than SMTP, as you only need to make a single cURL request to us to send a message, whereas with SMTP there's a lot of back-and-forth TCP chatter for connection, HELO, and such." 


Error checks are crucial part of any code. Here is my complete sendgrid wrapper complete with error checks. I do not want to throw exceptions because I want clients to handle the error their own way.





Sunday, March 18, 2012

Upload to s3 using aws php sdk

Here is a quick code sample






    error_reporting(-1);
    require_once 'sdk.class.php';

    // UPLOAD FILES TO S3
    // Instantiate the AmazonS3 class
    $options = array("key" => "aws-key" , "secret" => "aws-secret") ;

    $s3 = new AmazonS3($options);
    $bucket = "media1.3mik.com" ;

    $exists = $s3->if_bucket_exists($bucket);
    if(!$exists) {
        trigger_error("S3 bucket does not exists \n" , E_USER_ERROR);
    }

    $name = "cows-and-aliens.jpg" ;
    $path = "/home/rjha/Pictures/Cows-And-Aliens-1280x960.jpg";

    echo " upload file $name from $path \n" ;
    $sblob = file_get_contents($path);
    $response = $s3->create_object($bucket, $name, array('body' => $sblob , 'acl' => AmazonS3::ACL_PUBLIC));
    if($response->isOk()){
        echo "done" ;
    } else {
        echo "error";
    }






Add caching headers to existing S3 Object


Problem :- How to add caching headers to an existing amazon S3 object?
Background:- Yes, I know. I should have done it when I had uploaded those files. However they are already there and now we need to add caching control headers to our already uploaded objects.

Simple solution :- Use cloud Berry explorer or some such tool. 

Programmer's solution:- Looks like using aws-sdk copy_object does the trick.

Code sample







    error_reporting(-1);
    require_once 'sdk.class.php';

    // copy S3 object
    // Instantiate the AmazonS3 class
    $options = array("key" => "aws-key" , "secret" => "aws-secret") ;


    $s3 = new AmazonS3($options);
    $bucket = "bucket.3mik.com" ;


    $exists = $s3->if_bucket_exists($bucket);
    if(!$exists) {
        trigger_error("S3 bucket does not exists \n" , E_USER_ERROR);
    }

    $name = "cows-and-aliens.jpg" ;
    echo " change headers for $name  \n" ;
    $source = array("bucket" => $bucket, "filename" => $name);
    $dest = array("bucket" => $bucket, "filename" => $name);

    //caching headers
    $offset = 3600*24*365;
    $expiresOn = gmdate('D, d M Y H:i:s \G\M\T', time() + $offset);
    $headers = array('Expires' => $expiresOn, 'Cache-Control' => 'public, max-age=31536000');

    $meta = array('acl' => AmazonS3::ACL_PUBLIC, 'headers' => $headers);

    $response = $s3->copy_object($source,$dest,$meta);
    if($response->isOk()){
        printf("copy object done \n" );

    }else {
        printf("Error in copy object \n" );
    }




Wednesday, February 29, 2012

How to integrate Facebook login with your website to authenticate users


Say you have a website and you would like to authenticate people before they can do some actions. Now, many people may not be very willing to fill out your registration form and do a login. Now if you allow people to use their existing Facebook or twitter Login for authentication then you are removing a big barrier from trying out your site.


This post  deals with server side integration of Facebook Login into your website (or online app!)  I am going to present it in a very sequential fashion.


We only cover the server side integration. If you are looking for javascript integration then stop reading now :) I believe server integration is a better scheme as it gives you better control. So if you are running your own website and would like to authenticate users with Facebook login then dive straight in.


Register with Facebook developers to get App ID and App Secret Key

First head over to  https://developers.facebook.com/apps  and register your application. Click on create new app button if you do not have any. The registration is straightforward.


  • App display name - chose carefully because this is the application name presented to people who are trying to access your site using their Facebook login.
  • App domain - Here you can put multiple domains + sub domains. Sub domains are especially useful if you plan to test on your local machine. Like in my case I have created two entries
    • mint.3mik.com (to test from my local linux mint VM)
    • www.3mik.com
  • Site URL - put your website url here - like http://www.3mik.com
  • Mobile web - like http://m.3mik.com
  • You do not have to register App on Facebook, Native iOs app and native android app if you do not want to. I just wanted the website integration and I checked off the other options.
  • After a successful registration, note down your App ID and App secret key

Create a  link on your website  to launch Facebook oAuth Dialog box

When users come to your site you have to present a link to launch Facebook oAuth dialog box. You can put a simple link or wrap it up in a nice book. When users come to your site, they see a Login with Facebook link.

The best resource for Facebook login server side integration   is  

To create such a link that will launch Facebook authentication dialog box, You need
  1. Your Facebook App ID 
  2. Your callback URL (where Facebook will redirect after a successful login) - Do not worry too much about this right now. Just add a URL that belongs to your site.
Here is how I am creating it


<?php
    $stoken = Util::getMD5GUID();
    $gWeb->store("fb_state",$stoken);
    $fbAppId = Config::getInstance()->get_value("facebook.app.id");
    $host = "http://".$_SERVER["HTTP_HOST"];
    $fbCallback = $host."/callback/fb2.php" ;
    
    $fbDialogUrl = "https://www.facebook.com/dialog/oauth?client_id=".$fbAppId ;
    $fbDialogUrl .= "&redirect_uri=".urlencode($fbCallback)."&scope=email&state=".$stoken ;
?>

  <a href="<?php echo $fbDialogUrl; ?>"> Login with Facebook</a>

some explanation is in order for parameters supplied to Facebook dialog URL

  • fbAppId - is simply the Facebook application ID
  • redirect_uri is where Facebook should redirect the user (A callback for us) 
  • scope - if you just want the basic information back then you do not have to use this parameter. in Our case we want Facebook to return email together with basic information hence we pass this extra parameter, scope=email. You can pass in a comma separated list.
  • state token. This is to prevent CSRF attacks. We generate a random token and keep it in our server session. We pass the same token as state parameter to Facebook. Facebook will return this state token as it is to our callback URL. There we can retrieve this token and compare it with the token stored in our session.

Now when users come to your login page, they will see a login with Facebook link. When they click it they will go to Facebook oAuth dialog page. After their authentication they will be redirected to our callback page.


Callback Page to process Facebook response

Most of the example code I have seen on parsing Facebook response is pathetic including Facebook's own sample. They are suppressing errors in their sample code. Real world applications will need a robust error handling and user information processing. 

Error Handler 

We want to trap any errors, log it and then redirect the users to login page where they can view the error. Also we log the errors so we can see why our integration is not working.  Here is our Error Handler



function login_error_handler($errorno,$errorstr,$file,$line) {

    if(error_reporting() == 0 ) {
        // do nothing for silenced errors
        return true ;
    }
    
    switch($errorno) {

        case E_STRICT :
            return true;
        case E_NOTICE :
        case E_USER_NOTICE :
            Logger->error(" $file :: $line :: $errorstr");
            break ;

        case E_USER_ERROR:
            Logger->trace($file,$line,$errorstr,'TRACE');
            $_SESSION["form.errors"] = array($errorstr);
            header('Location: /user/login.php');
            exit(1);
   

        default:
            Logger->trace($file,$line,$errorstr,'TRACE');
            $_SESSION["form.errors"] = array("Error happened during login");
            header('Location: /user/login.php');
            exit(1);
            
    }
    
    //do not execute PHP error handler
    return true ;
}





We use the above error handler in our callback code. Now coming to callback, following is our scheme to process Facebook response


  • Facebook can return error and error descriptions to callback. If Facebook returned an error then we have to process those errors and show them to users. I suggest reading the Facebook document here.
  • If there is no code then user has accessed the callback page directly and we need to launch the Dialog box
  • If you have a code and state returned by Facebook is same as stored by us in session (to prevent CSRF attacks) then we need to 
    • Issue a request to Facebook using the returned code and our App secret key 
    • Parse the data returned by Facebook
    • Use this data to start our own login session

The code in isolation will not make much sense. Here is the code to do error checks and issue a request to Facebook to get user data using our App secret key.





 
    //set special error handler for callback scripts 
    include ($_SERVER['APP_WEB_DIR'].'/callback/error.inc');
    set_error_handler('login_error_handler');
   
    $fbAppId = Config::getInstance()->get_value("facebook.app.id");
    $fbAppSecret = Config::getInstance()->get_value("facebook.app.secret");

    $host = "http://".$_SERVER["HTTP_HOST"];
    $fbCallback = $host. "/callback/fb2.php";
  
    $code = NULL;
    if(array_key_exists('code',$_REQUEST)) {
        $code = $_REQUEST["code"];
    }
 
    $error = NULL ;
    if(array_key_exists('error',$_REQUEST)) {
       $error = $_REQUEST['error'] ;
       $description = $_REQUEST['error_description'] ;
       $message = sprintf(" Facebook returned error :: %s :: %s ",$error,$description);
       trigger_error($message,E_USER_ERROR);

       exit ;
     }


     if(empty($code) && empty($error)) {
        //see how to launch an FB dialog again on Facebook URL given above
     }

    //last state token
    $stoken = $gWeb->find('fb_state',true);
 
    if(!empty($code) && ($_REQUEST['state'] == $stoken)) {
    
    //request to get access token
    $fbTokenUrl = "https://graph.facebook.com/oauth/access_token?client_id=".$fbAppId ;
    $fbTokenUrl .= "&redirect_uri=" . urlencode($fbCallback). "&client_secret=" . $fbAppSecret ;
    $fbTokenUrl .= "&code=" . $code;
  
    $response = file_get_contents($fbTokenUrl);
    $params = null;
    parse_str($response, $params);

    $graph_url = "https://graph.facebook.com/me?access_token=".$params['access_token'];
    $user = json_decode(file_get_contents($graph_url));
    processUser($user);

   }
   else {
    $message = "3mik.com and Facebook statedo not match.possible CSRF.";
    trigger_error($message,E_USER_ERROR);
  }




How to process the User information

Now you have the user information. At this point you can just dump the $user variable to see what all you get back from Facebook. From here on it depends on what you want to do. A simple scheme could be to


  • Look at the $user->id  
  • see if you have encountered this $user->id before 
  • You can issue a get_or_create($user) to store this user information in your DB
  • Start a session on your website for this user

There are many other hairy details like merging accounts, logout etc. However, I hope this post has given you sufficient information to get started.

Update:- 
Here is the code to  process user information

 


function processUser($user) {
    // exisitng record ? find on facebook_id
    // New record - create login + facebook record
    // start login session  
    $id = $user->id;
    $name = $user->name;
    $firstName = $user->first_name ;
    $lastName = $user->last_name ;
    $link = $user->link ;
    $gender = $user->gender ;
    $email = $user->email ;

    // do not know what facebook will return
    // we consider auth to be good enough for a user
    if(empty($name) && empty($firstName)) {
        $name = "Anonymous" ;
    }

    $message = sprintf("Login:Facebook :: id %d ,email %s \n",$id,$email); 
    Logger::getInstance()->info($message);

    $facebookDao = new \com\indigloo\sc\dao\Facebook();
    $loginId = $facebookDao->getOrCreate($id,$name,$firstName,$lastName,$link,$gender,$email);


    if(empty($loginId)) {
        trigger_error("Not able to create login for facebook user",E_USER_ERROR);
    }

    \com\indigloo\sc\auth\Login::startFacebookSession($loginId,$name);
    header("Location: / ");
}




Here you can store your Facebook users in a DB table. There you can check if you already have this Facebook ID. If you are using MYSQL then please do not use int(11) for storing facebook.ID, use string.

You can see a working example at our site : http://www.3mik.com/user/login.php
Questions/Issues/Comments about this post? - send to my blogger-id@gmail

Tuesday, January 10, 2012

Find and move files on macosx lion hard disk using perl File::Find

There are times in your life when doing a bit of programming can really be a convenience, a time saving convenience. These are the times when even your wife would agree that being a programmer has its advantages.

Now a couple of aeons ago I had a G4 macbook running panther. When I switched from that machine, I just dumped that hard disk on a western digital external drive. Then I was using a Thinkpad and again the Thinkpad disk got dumped on same WD external hard drive when I moved to my current MBP.

Problem now is that locating photos or music on this WD external drive can be a pain. I had really created a deeply nested "I am organized" directory structure on my old G4 and Thinkpad. (The argument is settled in my mind now, never created directories more than 2 level deep no matter what people say. Search is always faster than locating a file by traversal. All your little ontological schemes are totally arbitrary and you are certain to forget your arbitrary "conventions" after 3 years)

I am looking for a way to "flatten" this nested directory tree and move all the photos scattered on this disk in one place. Ditto for music and PDF files and whatever I care about. I want to create a new directory structure that is flat and for that I need to find files in old tree and move them to new tree. Find files on a hard disk and move to a new directory tree. what can be simpler?

First I try bash find and mv command using xargs



find /Volumes/Elements/iBook/Music/ -name *.MP3 -print0 | xargs -0 -I {} mv {} /Volumes/Elements/Music/ibook/{}




The problem with this scheme is

  • There can be two files with same name but different content (result of importing from 2 digital cameras)
  • Extracting base name can be difficult when file contains spaces (none of the suggested trick worked for bash shipped with my mac osx lion)
  • You may want to run some rules on source as well as target, with Bash programming is difficult

find and mv with xargs will work for simple cases and I suggest using them. However for my find and move case I found perl File::Find to be a better fit. 

  • Perl File:Find is fast - no complaints with speed
  • I can access perl and all the programming logic, like attaching a counter to file name etc. Programming perl is preferable to programming bash
  • With perl File::Find and closures, I can easily reuse the logic across different sources
  • World is full of perl File::Find ready made examples

with about 20 mins. of work I was able to whip up my scripts and find and moves files in desired directory structure. Now I can browse the dump of my old hard disks easily and decide on what to keep and what to throw away




#!/usr/bin/perl
use strict;
use warnings;
use File::Basename;
use File::Find;

our $count = 0 ;

my $ibook_dir = "/Volumes/Elements/iBook";
my $thinkpad_dir = "/Volumes/Elements/Thinkpad-R60";

my $wanted = make_wanted(\&move_media,'/Volumes/Elements/media');
find($wanted, $thinkpad_dir);


#http://www.perlmonks.org/?node_id=109068

sub make_wanted { 
 my $wanted= shift;                      # get the "real" wanted function
    my @args= @_;                           # "freeze" the arguments
    my $sub= sub { $wanted->( @args);  };   # generate the anon sub
    return $sub;                            # return it
}


sub move_media {
 my @args = @_ ;
 my $file = $File::Find::name;
 
 $file =~ s,/,\\,g;
 #return unless -f $file;
 return if $file =~ /THUMBS/i ;
 return unless $file =~ /\.mp3|\.avi|\.mov|\.mpeg|\.mpg|\.mpeg4|\.mp4|\.3gp|\.3gpp|\.h264|\.wmv|\.flv/i;

 #replace backslash with slash
 # we are getting backslash from File::Find on macosx
 $file =~ s/\\/\//g;
 my $mvname = fileparse($file);
 #quote source - otherwise mv command fails
 print "mv \"$file\"  \"$args[0]/$mvname\" \n" ;
}

sub move_photo {
 my @args = @_ ;
 my $file = $File::Find::name;
 
 $file =~ s,/,\\,g;
 #return unless -f $file;
 return if $file =~ /THUMBS/i ;
 return if $file =~ /\.svn/ ;
 return if $file =~ /gloodev/ ;
 return if $file =~ /pgsem/i ;
 return if $file =~ /DMC/i ;

 return unless $file =~ /\.JPG|\.jpg|\.jpeg|\.JPEG/;

 $file =~ s/\\/\//g;
 my $x = fileparse($file);
 $count++ ;
 my $mvname = $count."_".$x;
 #quote source - otherwise mv command fails
 print "mv \"$file\"  \"$args[0]/$mvname\" \n" ;
}

  1. fileparse routine is to get base name out of full file name
  2. we have a global count variable declared in "our namespace"
  3. mv cannot handle spaces in names so we need to quote such file names
  4. The closure to create custom perl File find functions that accept our parameters from outside is taken from perl monk site



© Life of a third world developer
Maira Gall