Mac OSX like screen capture with ksnapshot in KDE

ksnapshot is a nifty tool for capture windows and screens. However I don't want to click the menu , bring up ksnapshot and then select options in menu to do screen grab. I really liked the way it is done on Mac OSX. Press Command +4 and you have a rectangular screen grab.

So I tuned my ksnapshot to do the same on KDE desktop. command +4 should open a rectangular screen grab and command +P should open a full screen print.

ksnapshot can be called with --region or --fullscreen.
so all we need to do is to map above key combos against ksnapshot [ --region | --fullscreen]
Here is how to do it

KDE desktop | settings |  shortcust and gestures

How to keep in sync with git repository of another person on github or bitbucket

suppose rjha94 has a repo on bitbucket.org called dl (rjha94/dl - copy #1). Now you also want a copy of this repository on your machine. First thing you have to do is to fork this repo using your bitbucket.org account.

 #1. fork this repository on bitbucket.org first (fork rjha94/dl as srj0408/dl )
 #2. to get this code on your local machine, you can just clone your fork of repo.

 $git clone srj0408/dl (get the actual clone URL on bitbucket interface)

when you forked, you made a copy at hosting server (copy #2) . when you cloned, you made a copy of your server repo on your local m/c (copy #3) . from a git point of view, all these copies are equally valid (there is no central or one true copy). so you have rjha94/dl that you forked into srj0408/dl on server. Then you cloned the same on your local m/c creating a third copy.

All these copies can be independent of each other. Like you can make changes on your local m/c that no one knows about. Same way, the original repo (copy #1) can be changed. Now it could be that some new changes have come to rjha94/dl. how can you get them into the repo on your local m/c (copy #1 -> copy #3) and push it to your own server repo srj0408/dl (copy #3 -> copy #2) ?

 #3 To get rjha94/dl repo changes to the repo on your local m/c
$git checkout master
$git remote add rjha94/dl ssh://git@bitbucket.org/rjha94/dl.git

check out your local master branch and then add a new remote URL called rjha94/dl that points to the original server repo your forked (copy #1) . Then to merge the new changes from rjha94/dl repo (copy #3)

$git fetch rjha94/dl
$git merge remotes/rjha94/dl/master

This would pull changes from rjha94/dl repo and merge into your local copy. (copy #1 -> copy #3) To get these changes into your server repo copy

$git commit -m "merged changes from upstream on 25-apr-14"
$git push origin

Doing this will push the changes you just merged into your local copy into your server repo (copy #3 -> copy #2) where origin is a shortcut (alias) for your own server copy.

Time series database survey for IoT and m2m devices

This is a survey of time series databases available for use, both the cloud offerings as well as "install on our own machines" solutions. The requirement we have is

• store high velocity time series data (frequent data arriving from one node)
• store data from lot of nodes 
• compute aggregates (sum over a days worth of data)
• Grouping functions (average, STDEV) 
• Analyze the data for patterns etc.

No one is paying me to write this so I will stay clear of jargons like, slice and dice, Cubes and all that b.s. in plain simple terms, we are receiving data from lot of devices very frequently, so first problem is simply storing a lot of data. Mysql and other RDBMS are not optimized for storing such time series data.  That is problem #1.

Another problem is that it may not be prudent to fetch all the raw data points for certain queries later on. Let's say that you want to watch the trend over a month then just fetching all the raw datapoints may be a overkill. What you instead would like to do is to just fetch 30 data points, each an average over a day's worth of datapoints. Now, creating such buckets (rollups) on demand would be expensive operation, so we need to push data into such buckets (rollup) as and when they arrive. That is problem #2, a good solid support for whatever rollup I would like to create. For data arriving at millsecond intervals that can just be one minute! 

There is actually a rollup hierarchy. say, data is arriving at 5 minute intervals and then you make rollup of an hour (average over 12 datapoints) . Further you would like to make a rollup of a day (averaged over 24 datapoints of previous bucket) etc. 

Then we also need aggregates. We would like to sum over datapoints for a particular interval for reporting. (say Rainfall over a day). 

For IoT/m2m kind of use cases, you also need to detect patterns in real times (this is apart from the threshold alerts). Then we would like to analyze the data and perform statistical opeartions on it.

RRDTOOL

http://oss.oetiker.ch/rrdtool/

Nice circular buffer

Expects data at requried intervals

Language bindings available

Good fit for small numer of metric

KairosDB

https://code.google.com/p/kairosdb/

forked from openTSDB

storing metric in HBase/ Cassandra

Good storage facility, allows tagging of data 

However Data model is very limited. 

Aggreates are calculated during query time and can be a performance drag

No support for automatic rollup

OpenTSDB 

http://opentsdb.net/

Looks very married to the Graphs

Good for computer metric cases 

Does not look a good fit for device case 

(where data dictionary is device dependent)

Graphite

http://graphite.wikidot.com/

Cloud offerings

Xivey a.k.a  pachcube a.k.a whatever-it-was

http://www.xively.com/

Good PR buzz

Good ecosystem

support is a black hole if you are in Asia

Rollup supported (in their own way)

Good provisioning and device activation support

Device side things are unnecessarily complicated

support for average function only (haven't found others yet)

Librato

https://metrics.librato.com/

Digi m2m cloud

http://www.etherios.com/

Tempo-DB

https://tempo-db.com/

I think all cloud based application would run into limitation for serious applications.  Also, there is no way others can do your analytic for you. For the moment, my strategy is to prototype on xively and then switch to influxdb (or maybe another on-my-machine solution). For realtime analytic, look at 

amazon Kinesis or numPy with HDF5. The debate is far from settled.

OSX Maverick and Vmware fusion 4.1

 My copy of Vmware fusion 4.1 works with OSX maverick. 

It is quite understandable why vmware would spread FUD to sell more copies of newer versions of Fusion. However, we had already burnt the vmware support bridge when we had updated to 3.x kernels. Long live open-vm-tools!

Tcpflow Network traffic capture in three simple steps

Three simple steps to capture network traffic capture on your linux box

• On the Debian Box, just install tcpflow using apt-get
• $ifconfig to figure out the network interface (mine was eth0)
• $sudo tcpflow -c -e -i eth0

HTTP Traffic monitor and capture tools

Here is a quick list of tools that can capture and monitor http traffic between your browser and server. yeah, I know wireshark exists. it is just that I still do not know how to use it!

1. tcpflow - works at card level. Much better if you can script things on your own box.
2. Fiddler - Nice tool but Windows only (who is going to download mono to install a http capture tool on linux box?)
3. httpry - Not tried yet, on the list though!
4. TCPCatcher - Would need Java. Downloadable as a stanadlone jar.
5. Charles Proxy - commercial (50$)  but people have lot of praise for the tool.
6. WebScarab - Our security team was using it. Ugly as hell but does the job.

Eight queen problem in Java

Here is a solution to eight queen puzzle in Java. You should note the following

• This is a brute force solution for NxN board
• The way we solve it is - first solve it for eight rook - i.e. first take care of horizontal and vertical lines only (the way a rook moves) and then omit the solutions having collision on diagonals 
• To solve the N-rook problem we generate all possible permutations of N (corresponding to the fact that one queen occupies one column) - so this solution is not at all going to scale
• The only way to learn anything in life is to do it yourself - even though this is a simple brute force solution or whatever - doing it gives me more pleasure than reading other's elegant solutions.

Next step - is probably to port this to javascript and generate the boards using HTML5 canvas. 

/*
 * 8-queen problem  using brute force searching
 * This solutions uses following strategies 
 *
 * 1 - fix one queen in one column and generate all
 * non-conflicting permutations - N! in total  
 * this is akin to solving the N-rook problem
 * 2- eliminate from N! permutations - that do not 
 * pass the additional diagonal test 
 * 
 * @author Rajeev Jha
 * @version 1.0
 *
 */

import java.util.Set;
import java.util.HashSet;

public class queen8 {

    private int[] columns ;
    private char[] colNames ;
    private int size ;
    private int solutions ;
    
    public queen8(int N) { 
        this.columns = new int[N] ; 
     for(int i=0 ; i < N ; i++) this.columns[i] = i+1 ;

        this.colNames = new char[N] ; 
        char a = 'A' ;
     for(int i=0 ; i < N ; i++) this.colNames[i] = (char) (a + i) ;

        this.size = N ;
        this.solutions = 0 ;
    }

    private void solve() {
        this.generate(this.size);
    }

    /* 
     * permutation generator using a backtracking algo
     * from http://www.cs.princeton.edu/~rs/talks/perms.pdf */

    private void generate(int N){
     int c ;
        /* factorial 1, 1! case,just one possibility,print that ..*/
        if ( N == 0 ) test_diagonal(); 
        //algorithm adjusted for zero-based indexes ..
        for(c = 0 ; c < N ; c++){
            swap(c,N-1);
            generate(N-1);
            swap(c,N-1);
        }
    
    } 

    /* swap for permutation */
    private void swap(int x, int y){
     int tmp = this.columns[x] ;
        this.columns[x] = this.columns[y] ;
        this.columns[y] = tmp ;
    
    }

    /* for position of a queen in a column (a permutation)
     * diagonal positions are given by moving 
     * one row up in next column and one row down in 
     * previous column */

    private void test_diagonal(){
        int x ;

     for(int i = 0 ; i < this.columns.length ; i++) {
            x = this.columns[i] ;
            for(int j = i+1, k = 1; j < this.columns.length ; j++, k++) {
                if((x+k) == this.columns[j]) return ;
                if((x-k) == this.columns[j]) return ;
            }
        }

        // diagonal test passed
        print_board();
    }

    private void print_board() {

     for(int i = 0 ; i < this.columns.length ; i++) {
            System.out.print(this.colNames[i]);
            System.out.print(this.columns[i] + " ");
        }

        System.out.println();
        this.solutions++ ;

    }

    private int getNumSolutions() {
        return this.solutions ;
    }
   
    public static void main(String[] args) throws Exception {
        queen8 board = new queen8(8);
        board.solve();
        System.out.println(" \n Total " + board.getNumSolutions() + " solutions " );

    }
}

And here are the solutions

rjha@mint13 ~/code/fun $ javac queen8.java 
rjha@mint13 ~/code/fun $ java -classpath . queen8 
A4 B2 C7 D3 E6 F8 G5 H1 
A5 B2 C4 D7 E3 F8 G6 H1 
A3 B5 C2 D8 E6 F4 G7 H1 
A3 B6 C4 D2 E8 F5 G7 H1 
A4 B7 C5 D3 E1 F6 G8 H2 
A5 B7 C1 D3 E8 F6 G4 H2 
A4 B6 C8 D3 E1 F7 G5 H2 
A3 B6 C8 D1 E4 F7 G5 H2 
A5 B3 C8 D4 E7 F1 G6 H2 
A5 B7 C4 D1 E3 F8 G6 H2 
A4 B1 C5 D8 E6 F3 G7 H2 
A3 B6 C4 D1 E8 F5 G7 H2 
A6 B4 C2 D8 E5 F7 G1 H3 
A5 B2 C6 D1 E7 F4 G8 H3 
A6 B4 C7 D1 E8 F2 G5 H3 
A1 B7 C4 D6 E8 F2 G5 H3 
A6 B2 C7 D1 E4 F8 G5 H3 
A6 B8 C2 D4 E1 F7 G5 H3 
A5 B8 C4 D1 E7 F2 G6 H3 
A4 B8 C1 D5 E7 F2 G6 H3 
A4 B7 C1 D8 E5 F2 G6 H3 
A4 B2 C7 D5 E1 F8 G6 H3 
A2 B5 C7 D4 E1 F8 G6 H3 
A5 B7 C1 D4 E2 F8 G6 H3 
A2 B7 C5 D8 E1 F4 G6 H3 
A1 B7 C5 D8 E2 F4 G6 H3 
A5 B1 C4 D6 E8 F2 G7 H3 
A6 B4 C1 D5 E8 F2 G7 H3 
A6 B3 C7 D2 E8 F5 G1 H4 
A2 B7 C3 D6 E8 F5 G1 H4 
A5 B1 C8 D6 E3 F7 G2 H4 
A1 B5 C8 D6 E3 F7 G2 H4 
A3 B6 C8 D1 E5 F7 G2 H4 
A7 B5 C3 D1 E6 F8 G2 H4 
A6 B3 C1 D7 E5 F8 G2 H4 
A7 B3 C1 D6 E8 F5 G2 H4 
A5 B7 C2 D6 E3 F1 G8 H4 
A3 B6 C2 D7 E5 F1 G8 H4 
A6 B2 C7 D1 E3 F5 G8 H4 
A7 B3 C8 D2 E5 F1 G6 H4 
A5 B3 C1 D7 E2 F8 G6 H4 
A2 B5 C7 D1 E3 F8 G6 H4 
A3 B6 C2 D5 E8 F1 G7 H4 
A6 B1 C5 D2 E8 F3 G7 H4 
A8 B3 C1 D6 E2 F5 G7 H4 
A2 B8 C6 D1 E3 F5 G7 H4 
A3 B7 C2 D8 E6 F4 G1 H5 
A6 B3 C7 D2 E4 F8 G1 H5 
A4 B2 C7 D3 E6 F8 G1 H5 
A1 B6 C8 D3 E7 F4 G2 H5 
A7 B1 C3 D8 E6 F4 G2 H5 
A6 B3 C7 D4 E1 F8 G2 H5 
A3 B8 C4 D7 E1 F6 G2 H5 
A7 B4 C2 D8 E6 F1 G3 H5 
A4 B6 C8 D2 E7 F1 G3 H5 
A2 B6 C1 D7 E4 F8 G3 H5 
A3 B6 C2 D7 E1 F4 G8 H5 
A7 B2 C6 D3 E1 F4 G8 H5 
A2 B4 C6 D8 E3 F1 G7 H5 
A3 B6 C8 D2 E4 F1 G7 H5 
A8 B4 C1 D3 E6 F2 G7 H5 
A4 B8 C1 D3 E6 F2 G7 H5 
A6 B3 C1 D8 E4 F2 G7 H5 
A2 B6 C8 D3 E1 F4 G7 H5 
A4 B7 C3 D8 E2 F5 G1 H6 
A4 B8 C5 D3 E1 F7 G2 H6 
A3 B5 C8 D4 E1 F7 G2 H6 
A7 B4 C2 D5 E8 F1 G3 H6 
A5 B7 C2 D4 E8 F1 G3 H6 
A4 B2 C8 D5 E7 F1 G3 H6 
A4 B1 C5 D8 E2 F7 G3 H6 
A5 B1 C8 D4 E2 F7 G3 H6 
A5 B2 C8 D1 E4 F7 G3 H6 
A8 B2 C4 D1 E7 F5 G3 H6 
A7 B2 C4 D1 E8 F5 G3 H6 
A3 B7 C2 D8 E5 F1 G4 H6 
A3 B1 C7 D5 E8 F2 G4 H6 
A8 B2 C5 D3 E1 F7 G4 H6 
A3 B5 C2 D8 E1 F7 G4 H6 
A3 B5 C7 D1 E4 F2 G8 H6 
A5 B2 C4 D6 E8 F3 G1 H7 
A6 B3 C5 D8 E1 F4 G2 H7 
A5 B8 C4 D1 E3 F6 G2 H7 
A4 B2 C5 D8 E6 F1 G3 H7 
A4 B6 C1 D5 E2 F8 G3 H7 
A5 B3 C1 D6 E8 F2 G4 H7 
A6 B3 C1 D8 E5 F2 G4 H7 
A4 B2 C8 D6 E1 F3 G5 H7 
A6 B3 C5 D7 E1 F4 G2 H8 
A6 B4 C7 D1 E3 F5 G2 H8 
A4 B7 C5 D2 E6 F1 G3 H8 
A5 B7 C2 D6 E3 F1 G4 H8 
 
 Total 92 solutions 

Using wordpress export data with PHP simpleXML

I had a site running in word press. This was a 256 MB slice and WP 3.2+, I must say (in a relative sense of course) is not light on resources. So I decided to move this site to my own code. That also meant moving the word press data to my own schema. So I took an XML dump using word press export tool and imported it back using my own scripts that use PHP and SimpleXML.

XML from Wordpress export tool has namespaces and multiple elements of same name so I reckoned my skeleton script can be of use to someone. Here we try to grab the  title, publication date, link (permalink), categories, tags and content from original wordpress post.

The code follows

 
error_reporting(-1);
    libxml_use_internal_errors(true);

    function process_post($title,$category,$tags,$createdOn) {
        if(empty($content)) { return ; }
        // process post

    }


    // start:script 
    // wp.xml contains dump of wordpress posts

    if (file_exists('wp.xml')) {
        $doc = simplexml_load_file('wp.xml');

        if($doc === false) {
            echo "Failed loading XML\n";
            foreach(libxml_get_errors() as $error) {
                echo "\t", $error->message;
            }
        }

    } else {
        echo('Failed to open wp.xml.');
        exit ;
    }


    foreach($doc->channel->item as $item) {

        $title = $item->title ;
        // content and other elements can be  wrapped inside 
        // a separate namespace. To deal with such elements we 
        // use item->children on the namespace given in wp.xml 

        $ns_wp = $item->children("http://wordpress.org/export/1.1/");
        $attachment = $ns_wp->attachment_url ;

        if(empty($attachment)) {
            $ns_content = $item->children("http://purl.org/rss/1.0/modules/content/");
            $content =  (string) $ns_content->encoded;
            $link = $item->link ;

            $pubDate = $item->pubDate ;
            $createdOn = date("Y-m-d", strtotime($pubDate));

            $tags = "" ;
            $category = "" ;

            // tags and category
            // we can have multiple category elements inside an item

            foreach($item->category as $elemCategory) { 

                if(strcmp($elemCategory["domain"],"category") == 0 ) {
                    $category = $category." ".$elemCategory["nicename"] ;
                } 

                if(strcmp($elemCategory["domain"],"post_tag") == 0 ) {
                    $tags = $tags." ".$elemCategory["nicename"] ;
                } 
            }

            printf("title = %s, category = %s ,tags = %s , pub_date = %s  \n",$title,$category,$tags,$createdOn);
            process_post($title,$category,$tags,$createdOn);
        }

    }

Export Nokia 5800 contacts to Gmail on macosx

So I have an old Nokia 5800 express phone and there is no Ovi suite for macosx. There is something called Nokia multimedia transfer app but there is no option to save contacts with that. Even Ovi was butt-ugly, I do not remember if I ever tried synchronizing my contacts with Ovi suite.

Now my phone is dying on me and I wanted to transfer all the contacts online to Gmail in a hassle free manner. Thankfully, I found the youlu app in symbian store. ( http://www.youlu.com )

youlu is quick, simple and easy. I installed it via Ovi store app on my phone. The download was around ~0.84MB. The installation was smooth. Afterwards,

• you synchronize your contacts to youlu.com website.
• Download contacts from youlu in csv format
• Upload csv file to gmail

problem solved. Thanks youlu :)

KDE 4.10 looks nice and sophisticated

After reading rave reviews of KDE 4.10 I was tempted to install it. I had one mint 13 VM and some kind souls have already published instructions to pull the required packages from Kubuntu repository (all hail the mighty Internet :D )

The install was rather painless and 3D support also worked without a hitch :)  what surprised me most  is the fact that how polished the latest KDE desktop looks! ( I have rather fond memories of my KDE 1.0 desktop).

Polished is the right word. The only pain on eyes could be some strange looking icons.  Performance wise, it works fine  on my 1 GB VM.  There are some niggles that I would rather put to my limited RAM.

 I do not hate Gnome 3.6 with a passion but they have clearly taken a more tablet oriented  route while people on desktop have been left to work with ugly extensions.  The last panel extension I had was so ugly that finally I switched to tint2 for a panel + dock.

So It was a nice change to see an integrated nice looking dock and an icon for showing desktop.  Customization were easy and I was able to make whatever changes I wanted without reading manuals. So for a moment I am sticking to KDE as desktop of choice on Linux.

Options to run PHP scripts as daemons

PHP is mostly used as a front-end scripting language in a stateless fashion. What if you need to run a
PHP script like a daemon? An example use case would be a gearman worker or a PHP client for a message queue.

First, why is this an issue? You can just write a PHP program and start it from command line (that is what we will do in DEV env). However in real life this is an issue because the program started from command line

•  + Should survive reboots
•  + Should have ability to handle signals
•  + Should spawn/kill workers depending on system load

What are our options? Let me list them below. My money would be on supervisord.

1) Write your own program that can do monitoring and spawn more child.

•  + Example in PHP power programming
•  + Example in Steven's Unix programming

Also see

• + http://stuporglue.org/writing-a-daemon-with-php/
• + http://www.re-cycledair.com/php-dark-arts-daemonizing-a-process

The problems is that to pull it off you really need to be an accomplished Unix programmer.

2) using init.d standard scripts . init.d script would call a shell script that will start our PHP worker

Problems

1.  - we have to understand / do infrastructure pieces
2.  - graceful handling of signals
3.  - restart when child dies

3) DJB's daemon tools at http://cr.yp.to/daemontools.html

people swear by it. However I have not used it.


4) nohup and screen technique - Run scripts in GNU screen and come out of SSH.

1.  + issue: no supervision - job may do bad thing and die 
2.  + some watch script is needed. Too flaky.

5) python daemons at http://pypi.python.org/pypi/python-daemon/

Also see sander marechal's script (python2)
http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/

6) Try upstart with respawn on Ubuntu
 + should work, atleast theoretically. That is how I was running php-fcgi daemon in Ubuntu Lucid.

7) Gearman Manager
PECL extension did not work for me. I have used PEAR version in my DEV enviroment.

7) Perp
http://b0llix.net/perp/

8) supervisord
http://supervisord.org/

@see also using gearman with supervisord
http://stackoverflow.com/questions/8217848/running-gearman-workers-in-the-background

9) libslack daemon (used by kestrel) - http://libslack.org/daemon/


10) Others

phpdeamon - http://phpdaemon.net/
Fat controller - http://fat-controller.sourceforge.net/getting-started.html

Daemonize - http://software.clapper.org/daemonize/
restartd - https://launchpad.net/ubuntu/+source/restartd/

PEAR system_deamon

• http://kevin.vanzonneveld.net/techblog/article/create_daemons_in_php/
• http://pear.php.net/package/System_Daemon

PHP-daemon
https://github.com/shaneharter/PHP-Daemon

Add folders to rackspace cloud file container with php and cURL

We use rackspace cloudfile container to publish our css and javascript files to Akamai CDN. They (rackspace)  has a nice one click deployment of cloud files to the CDN. Assuming you are new to this, the recipe goes like

• Access your rackspace cloud account
• Create a new container in rackspace cloud files
• Upload your files to that container
• Select container and in the bottom panel - just select publish to CDN.

You can map a CNAME record in your DNS server to make the the long rackspace CDN URL manageable (or something that your application understands)

The css and javascript files refer to assets (logos/ sprites/ images and such) and we need to upload those as well to the CDN enabled container. However we do not want to do this manually and we wish to create and upload file + assets  bundle as part of build process (automatic and not manual)

Rackspace provides PHP libraries to make such tasks as creating a container and uploading files to them easier. However I planned to take the plain cURL route because 

1. Number of files is not large
2. The files themselves are not large 
3. I was not in a mood to download and install yet another library and learn the API

So I just rolled my sleeve and wrote this PHP script that would take a local folder and upload its content in a pseudo_directory hierarchy to rackspace cloud file container. YMMV but sometimes we all need quick and dirty and then this script can come in handy. 

Relevant links

• http://docs.rackspace.com/files/api/v1/cf-devguide/content/Uploading_a_Storage_Object-d1e3067.html
• http://docs.rackspace.com/files/api/v1/cf-devguide/cf-devguide-20121116.pdf

    error_reporting(-1);

    function do_upload($ch2,$auth,$fname) {

        $headers = array();
        $grab = array("X-Auth-Token");
        $host = $auth["X-Storage-Url"];

        foreach($auth as $name => $value) {
            if(in_array($name,$grab)) {
                array_push($headers, "$name: $value");
            }
        }


        // Content-Type
        $finfo = finfo_open(FILEINFO_MIME_TYPE);
        $mime = finfo_file($finfo, $fname);
        finfo_close($finfo);
        $etag = md5_file($fname);

        array_push($headers, "Content-Type: $mime");
        array_push($headers, "ETag: $etag");

        $fp = fopen($fname, "r");
        $fsize = filesize($fname);
        // asset is container name
        // upload in a pseudo_dir structure
        // /asset/css/...
        //
        $url = $host. "/asset/".$fname ;
        printf("HTTP PUT %s  to => %s \n",$fname,$url); 

         $options = array(
            CURLOPT_TIMEOUT => 60 ,
            CURLOPT_RETURNTRANSFER => 1 ,
            CURLOPT_FOLLOWLOCATION => 1,
            CURLOPT_HTTPHEADER => $headers,
            CURLOPT_URL => $url,
            CURLOPT_VERBOSE => false,
            CURLOPT_HEADER => 1,
            CURLOPT_PUT => 1 ,
            CURLOPT_INFILE => $fp,
            CURLOPT_INFILESIZE => $fsize);



        // print_r($headers); 
        // Do a PUT operation

        curl_setopt_array($ch2, $options);
        $response = curl_exec ($ch2);
    }


    function do_auth() {

        $ch = curl_init();

        $host = "https://identity.api.rackspacecloud.com/v1.0" ;
        $apiKey = "xxxxxxx" ;
        $user = "yyyyyyyyy" ;

        $headers = array(
            "X-Auth-Key: $apiKey " ,
            "X-Auth-User: $user");
           $options = array(
            CURLOPT_TIMEOUT => 60 ,
            CURLOPT_RETURNTRANSFER => 1 ,
            CURLOPT_FOLLOWLOCATION => 1,
            CURLOPT_HTTPHEADER => $headers,
            CURLOPT_URL => $host,
            CURLOPT_VERBOSE => false,
            CURLOPT_HEADER => 1);

        curl_setopt_array($ch, $options);
        $response = curl_exec ($ch);
        curl_close($ch);

        list($headers, $body) = explode("\r\n\r\n", $response, 2);

        $lines = explode("\n",$headers);
        $auth = array();
        $grab = array("X-Storage-Token", "X-Storage-Url","X-Auth-Token");

        foreach($lines as $line ) {
            $parts = explode(" ",$line);
            $name = $parts[0] ;
            $name = trim($name,": ");

            if(in_array($name,$grab)) {
                $auth[$name] = trim($parts[1]);
            }

        } 

        return $auth ;

    }


     $auth = do_auth();
    printf(" \n **** parsed auth headers for PUT **** \n");
    print_r($auth);

    $ch2 = curl_init();
    // get all files in css dir
    $files = array();

    // load everything from local css folder into
    // /asset/css/local-file-path
    foreach (new RecursiveIteratorIterator(new RecursiveDirectoryIterator('css')) as $path) {
        $filename = sprintf("%s",$path);
        $pos = strrpos($filename,"/");
        if($pos !== false) {
            $last = substr($filename,$pos+1);
            if($last == '.' || $last == '..') {
                printf(" ignore file :: %s \n",$filename); 
            }else {
                array_push($files,$filename);
            }
        }
    }

    foreach($files as $file){
        do_upload($ch2,$auth,$file);
    }

    curl_close($ch2);


 
       

count lines of code (cloc) in multiple PHP source folders

Over the years, I have used many tools that generate CLOC reports (how many lines of code you have written). The one I really like is cloc project on sourceforge . The tool is written in perl and is quite flexible.

Our problem is that we have a web application and that means

• we need to ignore certain folders when counting lines. Examples would be compiled templates, generated minified files, third party libraries and such.
•  We need to create custom definitions for some files (like .tmpl is our view templates and .inc are our PHP include files) Without custom definitions, the tool would count .inc as "PASCAL" file and ignore .tmpl files !!!
• We need to sum the individual reports over. We have dependencies and the project is split across different source trees and not everything is under one root.

Ignoring folders/files with cloc is dead simple.  Just pass a file with files and directories in --exclude-list-file.

To generate custom definitions (so .tmpl are counted as view templates and .inc as PHP includes)  first use cloc.pl to write standard definition files using --write-lang-def switch. Later on we will modify this file and ask cloc to use our definition instead of the standard one.

Exclude list

Exclude list is a file with one entry per line.

rjha@mint13 ~/code/github/sc/deploy/apps/cloc $ cat cloc-ignore 
/home/rjha/code/github/sc/web/css/bundle-full.css
/home/rjha/code/github/sc/web/css/bundle.css
/home/rjha/code/github/sc/web/js/bundle.js
/home/rjha/code/github/sc/web/js/bundle-full.js
/home/rjha/code/github/sc/web/compiled

creating custom definitions file

First generate the definitions of extensions used by cloc tool. Then we add our custom definitions to this file. Later on we will use this file to supply file definitions to cloc tool. Just copy the  existing matching definitions for your custom file types.

 ./cloc-1.56.pl --write-lang-def=cloc.def

Add to this file following definitions

View Template
    filter remove_html_comments
    filter call_regexp_common HTML
    extension tmpl
    3rd_gen_scale 1.0


PHP Include
    filter remove_matches ^\s*#
    filter remove_matches ^\s*//
    filter call_regexp_common C
    filter remove_inline #.*$
    filter remove_inline //.*$
    extension inc
    3rd_gen_scale 1.0

script to sum reports

rjha@mint13 ~/code/github/sc/deploy/apps/cloc $ cat cloc.sh 
# web folder - use an ignore list
# web folder - needs custom definitions

./cloc-1.56.pl  --exclude-list-file=./cloc-ignore --read-lang-def=./cloc.def   /home/rjha/code/github/sc/web --report-file=sc.web.report
./cloc-1.56.pl  /home/rjha/code/github/sc/lib --report-file=sc.lib.report
./cloc-1.56.pl  /home/rjha/code/github/webgloo/lib/com/indigloo --report-file=webgloo.report
# sum the reports 
./cloc-1.56.pl  --read-lang-def=./cloc.def  --sum-reports *.report
#remove tmp
rm *.report

Here we are running cloc on three separate folders. First one uses an ignore list and a custom definition file (created earlier).  second and third are standard cloc reports.

Finally we use --sum-reports option to produce the final report across three different source trees.

rjha@mint13 ~/code/github/sc/deploy/apps/cloc $ ./cloc.sh 
     239 text files.
     239 unique files.                                          
      54 files ignored.
Wrote sc.web.report
      94 text files.
      94 unique files.                              
       0 files ignored.
Wrote sc.lib.report
      41 text files.
      41 unique files.                              
       0 files ignored.
Wrote webgloo.report

http://cloc.sourceforge.net v 1.56
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
PHP                            226           4795           1340          13256
PHP Include                     32            243             31            902
CSS                              1            219             33            850
Javascript                       1            249             99            823
View Template                   54            185              5            681
HTML                             4             19              0             46
XML                              2              0              0             44
-------------------------------------------------------------------------------
SUM:                           320           5710           1508          16602
-------------------------------------------------------------------------------

infinite scrolling but on steroids

we use infinite scrolling plugin by paul irish (https://github.com/paulirish/infinite-scroll) on 3mik now. However Problem with the vanilla plugin is that it assumes that the next URL only depends on current page number. Only  a naive pagination scheme would depend on only one variable.

As you go deeper into pages, you have to scan more and more records before you can arrive at the records you are interested in. For this reason,   real world pagination scheme rely on two variables (see these slides from percona conference  and this blog post )

• current page number
• last record id (of previous page ) or first record id (of next page)

An example would be the instagram API

To Apply infinite scrolling pattern to pages with such 2 variable pagination scheme, you need to do additional work. So here is our fork of infinite scroll plugin that documents this use case and provides sample server  PHP scripts.

Debian Linux with 3D graphics in vmware fusion 4.1

Joostlek has a nice post about enabling 3D graphics in linux guests on vmware.  I followed this same post for installing 3D graphics support for my Debian wheezy desktop.  I wanted to see the gnome shell in action after I heard so many (*bad*) things about it!

I had to install some extra packages and build dependencies for Mesa and if you also installed wheezy from net installer then these "extra" steps will be useful for you. Also his post describes building it on Ubuntu. These two points are the only difference from his post. I have vmware fusion 4.1.3 running on a 13" macbook pro.

Following packages are from above post


sudo apt-get install autoconf libtool xutils-dev xorg-dev flex bison libx11-xcb-dev libxcb-glx0-dev g++ git

And here are the extra packages that were missing in my net installer

•  sudo aptitude install  libxcb-dri2-0-dev
•  sudo aptitude install  libxcb-xfixes0 libxcb-xfixes0-dev
•  sudo aptitude install llvm
•  sudo aptitude install libxml2 libxml2-dev
•  sudo aptitude install python-libxml2
•  sudo aptitude install build-essential

You also have to install build dependencies for Mesa. Check the Debian How to build Mesa page.

•  sudo apt-get build-dep mesasudo apt-get build-dep mesa

Rest of the steps remain the same as in above post.  If you are using Debian wheezy after 21st June 2012 then you do not need to download a new kernel. DRM is enabled for your kernel. Just skip that step.

After the restart, I could log into gnome shell. The default leave a lot to desire. So I had to tweak my desktop for next  hour or so.  Here is the screenshot of my Debian wheezy  desktop running gnome-3 shell

lint like static code analysis tools for PHP

PHP is dynamically typed language and that means the error detection present in statically typed languages is absent. If you have an error in your script, it will not be detected till you have actually run it(apart from parse errors). The big, shine in your eyes errors are never a problem because they are detected quite easily during development. It is the silent types (doing X when you meant Y and X is a perfectly legit operation) that are more difficult.

Like, I will show you a piece of code I had re-factored  a while back

 
       static function find($subjectId,$objectId,$verb) {
            $mysqli = MySQL\Connection::getInstance()->getHandle();

            //sanitize input
            settype($subjectId,"integer");
            settype($objectId,"integer");
            settype($verb,"integer");

            $sql = " select count(id) as count from sc_bookmark " ;
            $sql .= " where subject_id = %d and object_id = %d and verb = %d ";
            $sql = sprintf($sql,$loginId,$itemId,$verb);

            $row = MySQL\Helper::fetchRow($mysqli, $sql);
            return $row;
        }

 

Here loginId was changed to subjectId in parameter list but I forgot to change the name of parameter I was passing to MySQL.  This used to result in a notice that was going to logs but no sign of program failure outside. That led me to say, enough is enough and I need to have a good static analysis tool for my PHP code.


There are many posts dedicated to the topic of PHP static analysis, sort of things that lint does for c code. The aim is to catch errors from just scanning the PHP code. When I started looking around, everyone talked about their favorite static analysis tool but no one gave me hard data or how it actually looks on your terminal.

I looked  at following tools

1. PHP code sniffer (phpcs)
2. PHP on command line with minus eel option (-l)
3. PHP mess detector (phpmd)
4. pfff/scheck

After a quick evaluation I (emphasized)  found phpmd to be the best of the lot.  Why?


One more tool worth checking should be https://github.com/sebastianbergmann/php-code-coverage
However I did not look at this one.

PHP code sniffer

phpcs looked too verbose out of the box. For 20 line perfectly legit code it generates 20 errors, most of that to do with style guidelines. Look at the output, I believe real  error will be buried under tons of such warnings. Sure, I did not look at how to tweak it and all but I believe most of the time first impressions are confined to out of box experience. Also, it could not detect the error I had. (error inside a class method)

FILE: /home/rjha/code/github/sc/lib/com/indigloo/sc/util/Asset.php
--------------------------------------------------------------------------------
FOUND 16 ERROR(S) AND 1 WARNING(S) AFFECTING 11 LINE(S)
--------------------------------------------------------------------------------
  2 | ERROR   | Missing file doc comment
  4 | ERROR   | Missing class doc comment
  4 | ERROR   | Opening brace of a class must be on the line after the
    |         | definition
  6 | ERROR   | Missing function doc comment
  6 | ERROR   | Opening brace should be on a new line
 17 | ERROR   | Expected "if (...) {\n"; found "if(...) {\n"
 20 | ERROR   | No space found after comma in function call
 24 | ERROR   | Expected "if (...) {\n"; found "if(...) {\n"
 28 | WARNING | Line exceeds 85 characters; contains 103 characters
 31 | ERROR   | Expected "if (...) {\n"; found "if(...) {\n"
 31 | ERROR   | No space found after comma in function call
 33 | ERROR   | No space found after comma in function call
 33 | ERROR   | No space found after comma in function call
 36 | ERROR   | Expected "if (...) {\n"; found "if(...) {\n"
 36 | ERROR   | No space found after comma in function call
 38 | ERROR   | No space found after comma in function call
 38 | ERROR   | No space found after comma in function call
--------------------------------------------------------------------------------

php -l (minus eel)

looks too limited. It could not detect errors inside class methods.

pfff/scheck

pfff is a project from facebook and it is a suite of tools. There are some interesting ocaml based tools there, like code map generator etc. scheck is the tool to do static checks. However pfff looks to be in alpha stage right now.

php mess detector

This could detect the problem inside a class method. Also, the tool is quick and you can decide on the kind of rules you would like to run.

rjha@mint ~/code/github/sc $ phpmd  lib/com/indigloo/sc/util/Asset.php text codesize,design,naming,unusedcode

/home/rjha/code/github/sc/lib/com/indigloo/sc/util/Asset.php:13 Avoid variables with short names like $ts. Configured minimum length is 3.

Nice looking Debian Wheezy Desktop

Well, if you do not mind 1000 extra packages and 100 MB of extra memory then I would suggest you install Linux Mint LMDE edition. The nice thing about mint LMDE is that you get a nice looking desktop with the choice of MATE or Cinnamon. The fonts and rendering looks nice out of the box.

However, if you have just started with the Debian net installer and just installed your desktop then the rendering and fonts may be a bit out of shape! You may wonder if your 1300$ LCD screen is not seen by Debian at all! Fret not! With little patience and work, we can make our Debian desktop look real nice and polished.

Fix font configuration

We first copy the ubuntu font config files in Debian wheezy following the steps outlined in 

http://noz3001.wordpress.com/2011/07/01/ubuntu-font-rendering-on-debian-wheezy/

After that we install microsoft ttf fonts 

$sudo aptitude install  ttf-mscorefonts-installer

Now we reload the font config and clear the old font caches

$sudo fc-cache -fv 

$sudo dpkg-reconfigure fontconfig-config 

$sudo dpkg-reconfigure fontconfig

Fix font aliasing and rendering

The following combination works for LCD screens

System | Preferences | Customize L&F  (or whatever is equivalent for your DM of choice)

• Turn subpixel hinting to RGB
• Turn on font aliasing
• Set font hinting to  slight

Change Mouse cursor

$sudo aptitude install dmz-cursor-theme

$ sudo update-alternatives --config x-cursor-theme

Preferences | customize L&F | Mouse cursor 

select DMZ white theme and you get a sane mouse instead of default x-cursor (circa 1991)

Install Flash

 Download 64 bit flash player from Adobe site, unpack

 $sudo cp libflashplayer.so /usr/lib/mozilla/plugins/.

 restart iceweasel, try viewing a youtube video.

Change login screen

By default wheezy is using gdm3 and the gdm3 login screens are, well, yuck! To change gdm3 login screen, one "simple way" is to create a new symlink for /usr/share/images/desktop-base/login-background.svg that points to your new login splash background. Please note that only the .png files work. The .jpg files would not work in this scheme.

Install lightdm

This provides a much better looking login screen. 

$sudo aptitude install lightdm 

verify that lightdm is the new display manager in /etc/X11/default-display-manager

To configure

$dpkg-reconfigure lightdm 

To change lightdm login screen background (greeter background) change background in /etc/lightdm/lightdm-gtk-greeter.conf (PNG images) 

After all these changes you will have a shiny looking Debian Wheezy Desktop :D Enjoy'

VMWare tools on Debian Wheezy inside Fusion

I have an early 2011 intel macbook and I run linux VM using Vmware fusion 4.x. Earlier I was using an Ubuntu VM for development that had 2.6.x kernel. Now, I wanted to upgrade my development VM to Debian wheezy for assorted reasons.  One of them was having a rolling distribution because I do not want to full upgrade my installation all the time.

The problem with installing Debian wheezy or any other linux 3.x series kernel in vmware fusion is that the vmware tools shipped with fusion is not compatible with linux 3.0 and 3.2 kernels. The VMware tools only work with 2.6.x kernel headers. Now this is a big headache as I need to mount my macbook directories inside my VM. Other nuisance is that your mouse will be captured inside vm window if you do not install the tools. in short, I want a life with vmware tools.

So open-vm-tools came to my rescue. This is one of the moments in life when you did not know about existence of something that is so essential to your existence! All my life of running linux VM on macbook and I did not know that this project exists! As it happens open-vm-tools are rolled with Debian so installing them is super easy.

First install the prerequisites

$aptitude install linux-headers-`uname -r` libx11-6 libx11-dev xorg libxtst6 psmisc build-essential ia32-libs ia32-libs-gtk

The Debian wiki mentions x-window-system and x-window-system-core but those packages are now provided by xorg. Now install the open-vm-tools packages using $sudo aptitude install

• open-vm-tools 
• open-vm-dkms      
• open-vm-tools-dev  
• open-vm-toolbox 

After the installation, try querying the vmware tools modules

%modinfo vmxnet

%modinfo vmhgfs

if the required kernel modules are not loaded then we have to do so via dkms (Dynamic kernel module support). To install required kernel modules

% dkms add open-vm-tools/2011.12.20

% dkms build open-vm-tools/2011.12.20

% dkms install open-vm-tools/2011.12.20

Last number in red is the module version, you can get this number by using  

$sudo aptitude show open-vm-tools 

Look for version string (after the + sign)

Run modinfo again and verify that we have the required modules. Shut down the VM. Map shares using VMware fusion settings. Start the VM. Now we should have VMware tools loaded.

$vmware-hgfsclient 

The output should match with the shared folders

To mount the shared folders, you can do something like. Here Public is the folder name that we shared via VMware fusion. We mount this folder at /mnt/hgfs.

$sudo mount -t  vmhgfs -v -o rw  .host:/Public /mnt/hgfs

Send Email using SendGrid PHP library on Github

We are hosting the 3mik.com boxes on Rackspace right now and  we get 40,000 emails/month free of cost  from one of their partners, SendGrid. We use SendGrid to send  emails from our application written in PHP.  The SendGrid documentation suggests using SMTP relays and swiftmail-4.x library.  However, SendGrid also has a PHP front-end library published on github that makes it super easy to send emails.

Here is one complete test script

    require_once($_SERVER['WEBGLOO_LIB_ROOT']. '/ext/sendgrid-php/SendGrid_loader.php');
    set_error_handler('offline_error_handler');
    $sendgrid = new SendGrid('your-sendgrid-login', 'your-sendgrid-password');
    $mail = new SendGrid\Mail();


     $mail->addTo('foo@gmail.com')->
       setFrom('foo@3mik.com')->
       setSubject('Sendgrid github PHP library test')->
       setText('Hello World! from sendgrid library')->
       setHtml('Hello World! from sendgrid github lib');  

    $response = $sendgrid->web->send($mail);
    // Error Handling:- 
    // sendgrid->web method uses curl_exec CURLOPT_RETURNTRANSFER set. 
    // This means you will get FALSE
    // when the send method fails at network level.
    // you get JSON response back when curl is able to communicate with the server
    // success from API is returned as
    // {"message":"success"}
    // Error from API is returned as
    // {"message": "error", "errors": ["Bad username / password"]}
    print_r($response);



?>

Though all SendGrid documentation exhorts you to use SMTP, I do not see a problem in using the WEB method.  As per this SO question, http://stackoverflow.com/questions/6193702/sendgrid-smtp-or-curl , " The  web API actually works faster than SMTP, as you only need to make a single cURL request to us to send a message, whereas with SMTP there's a lot of back-and-forth TCP chatter for connection, HELO, and such." 


Error checks are crucial part of any code. Here is my complete sendgrid wrapper complete with error checks. I do not want to throw exceptions because I want clients to handle the error their own way.

Upload to s3 using aws php sdk

Here is a quick code sample

    error_reporting(-1);
    require_once 'sdk.class.php';

    // UPLOAD FILES TO S3
    // Instantiate the AmazonS3 class
    $options = array("key" => "aws-key" , "secret" => "aws-secret") ;

    $s3 = new AmazonS3($options);
    $bucket = "media1.3mik.com" ;

    $exists = $s3->if_bucket_exists($bucket);
    if(!$exists) {
        trigger_error("S3 bucket does not exists \n" , E_USER_ERROR);
    }

    $name = "cows-and-aliens.jpg" ;
    $path = "/home/rjha/Pictures/Cows-And-Aliens-1280x960.jpg";

    echo " upload file $name from $path \n" ;
    $sblob = file_get_contents($path);
    $response = $s3->create_object($bucket, $name, array('body' => $sblob , 'acl' => AmazonS3::ACL_PUBLIC));
    if($response->isOk()){
        echo "done" ;
    } else {
        echo "error";
    }