php
Showing posts with label php. Show all posts

Tuesday, May 14, 2013

Using wordpress export data with PHP simpleXML

I had a site running in word press. This was a 256 MB slice and WP 3.2+, I must say (in a relative sense of course) is not light on resources. So I decided to move this site to my own code. That also meant moving the word press data to my own schema. So I took an XML dump using word press export tool and imported it back using my own scripts that use PHP and SimpleXML.

XML from Wordpress export tool has namespaces and multiple elements of same name so I reckoned my skeleton script can be of use to someone. Here we try to grab the  title, publication date, link (permalink), categories, tags and content from original wordpress post.

The code follows



 
error_reporting(-1);
    libxml_use_internal_errors(true);

    function process_post($title,$category,$tags,$createdOn) {
        if(empty($content)) { return ; }
        // process post

    }


    // start:script 
    // wp.xml contains dump of wordpress posts

    if (file_exists('wp.xml')) {
        $doc = simplexml_load_file('wp.xml');

        if($doc === false) {
            echo "Failed loading XML\n";
            foreach(libxml_get_errors() as $error) {
                echo "\t", $error->message;
            }
        }

    } else {
        echo('Failed to open wp.xml.');
        exit ;
    }


    foreach($doc->channel->item as $item) {

        $title = $item->title ;
        // content and other elements can be  wrapped inside 
        // a separate namespace. To deal with such elements we 
        // use item->children on the namespace given in wp.xml 

        $ns_wp = $item->children("http://wordpress.org/export/1.1/");
        $attachment = $ns_wp->attachment_url ;

        if(empty($attachment)) {
            $ns_content = $item->children("http://purl.org/rss/1.0/modules/content/");
            $content =  (string) $ns_content->encoded;
            $link = $item->link ;

            $pubDate = $item->pubDate ;
            $createdOn = date("Y-m-d", strtotime($pubDate));

            $tags = "" ;
            $category = "" ;

            // tags and category
            // we can have multiple category elements inside an item

            foreach($item->category as $elemCategory) { 

                if(strcmp($elemCategory["domain"],"category") == 0 ) {
                    $category = $category." ".$elemCategory["nicename"] ;
                } 

                if(strcmp($elemCategory["domain"],"post_tag") == 0 ) {
                    $tags = $tags." ".$elemCategory["nicename"] ;
                } 
            }

            printf("title = %s, category = %s ,tags = %s , pub_date = %s  \n",$title,$category,$tags,$createdOn);
            process_post($title,$category,$tags,$createdOn);
        }

    }





Tuesday, February 26, 2013

Options to run PHP scripts as daemons


PHP is mostly used as a front-end scripting language in a stateless fashion. What if you need to run a
PHP script like a daemon? An example use case would be a gearman worker or a PHP client for a message queue.

First, why is this an issue? You can just write a PHP program and start it from command line (that is what we will do in DEV env). However in real life this is an issue because the program started from command line
  •  + Should survive reboots
  •  + Should have ability to handle signals
  •  + Should spawn/kill workers depending on system load
What are our options? Let me list them below. My money would be on supervisord.

1) Write your own program that can do monitoring and spawn more child.
  •  + Example in PHP power programming
  •  + Example in Steven's Unix programming

Also see
The problems is that to pull it off you really need to be an accomplished Unix programmer.

2) using init.d standard scripts . init.d script would call a shell script that will start our PHP worker

Problems
  1.  - we have to understand / do infrastructure pieces
  2.  - graceful handling of signals
  3.  - restart when child dies

3) DJB's daemon tools at http://cr.yp.to/daemontools.html

people swear by it. However I have not used it.


4) nohup and screen technique - Run scripts in GNU screen and come out of SSH.

  1.  + issue: no supervision - job may do bad thing and die 
  2.  + some watch script is needed. Too flaky.

5) python daemons at http://pypi.python.org/pypi/python-daemon/

Also see sander marechal's script (python2)
http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/

6) Try upstart with respawn on Ubuntu
 + should work, atleast theoretically. That is how I was running php-fcgi daemon in Ubuntu Lucid.

7) Gearman Manager
PECL extension did not work for me. I have used PEAR version in my DEV enviroment.

7) Perp
http://b0llix.net/perp/

8) supervisord
http://supervisord.org/

@see also using gearman with supervisord
http://stackoverflow.com/questions/8217848/running-gearman-workers-in-the-background

9) libslack daemon (used by kestrel) - http://libslack.org/daemon/


10) Others

phpdeamon - http://phpdaemon.net/
Fat controller - http://fat-controller.sourceforge.net/getting-started.html

Daemonize - http://software.clapper.org/daemonize/
restartd - https://launchpad.net/ubuntu/+source/restartd/

PEAR system_deamon


PHP-daemon
https://github.com/shaneharter/PHP-Daemon

Sunday, September 16, 2012

count lines of code (cloc) in multiple PHP source folders



Over the years, I have used many tools that generate CLOC reports (how many lines of code you have written). The one I really like is cloc project on sourceforge . The tool is written in perl and is quite flexible.

Our problem is that we have a web application and that means

  • we need to ignore certain folders when counting lines. Examples would be compiled templates, generated minified files, third party libraries and such.
  •  We need to create custom definitions for some files (like .tmpl is our view templates and .inc are our PHP include files) Without custom definitions, the tool would count .inc as "PASCAL" file and ignore .tmpl files !!!
  • We need to sum the individual reports over. We have dependencies and the project is split across different source trees and not everything is under one root.

Ignoring folders/files with cloc is dead simple.  Just pass a file with files and directories in --exclude-list-file.

To generate custom definitions (so .tmpl are counted as view templates and .inc as PHP includes)  first use cloc.pl to write standard definition files using --write-lang-def switch. Later on we will modify this file and ask cloc to use our definition instead of the standard one.

Exclude list

Exclude list is a file with one entry per line.


rjha@mint13 ~/code/github/sc/deploy/apps/cloc $ cat cloc-ignore 
/home/rjha/code/github/sc/web/css/bundle-full.css
/home/rjha/code/github/sc/web/css/bundle.css
/home/rjha/code/github/sc/web/js/bundle.js
/home/rjha/code/github/sc/web/js/bundle-full.js
/home/rjha/code/github/sc/web/compiled




creating custom definitions file

First generate the definitions of extensions used by cloc tool. Then we add our custom definitions to this file. Later on we will use this file to supply file definitions to cloc tool. Just copy the  existing matching definitions for your custom file types.


 ./cloc-1.56.pl --write-lang-def=cloc.def

Add to this file following definitions

View Template
    filter remove_html_comments
    filter call_regexp_common HTML
    extension tmpl
    3rd_gen_scale 1.0


PHP Include
    filter remove_matches ^\s*#
    filter remove_matches ^\s*//
    filter call_regexp_common C
    filter remove_inline #.*$
    filter remove_inline //.*$
    extension inc
    3rd_gen_scale 1.0




script to sum reports


rjha@mint13 ~/code/github/sc/deploy/apps/cloc $ cat cloc.sh 
# web folder - use an ignore list
# web folder - needs custom definitions

./cloc-1.56.pl  --exclude-list-file=./cloc-ignore --read-lang-def=./cloc.def   /home/rjha/code/github/sc/web --report-file=sc.web.report
./cloc-1.56.pl  /home/rjha/code/github/sc/lib --report-file=sc.lib.report
./cloc-1.56.pl  /home/rjha/code/github/webgloo/lib/com/indigloo --report-file=webgloo.report
# sum the reports 
./cloc-1.56.pl  --read-lang-def=./cloc.def  --sum-reports *.report
#remove tmp
rm *.report




Here we are running cloc on three separate folders. First one uses an ignore list and a custom definition file (created earlier).  second and third are standard cloc reports.
Finally we use --sum-reports option to produce the final report across three different source trees.



rjha@mint13 ~/code/github/sc/deploy/apps/cloc $ ./cloc.sh 
     239 text files.
     239 unique files.                                          
      54 files ignored.
Wrote sc.web.report
      94 text files.
      94 unique files.                              
       0 files ignored.
Wrote sc.lib.report
      41 text files.
      41 unique files.                              
       0 files ignored.
Wrote webgloo.report

http://cloc.sourceforge.net v 1.56
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
PHP                            226           4795           1340          13256
PHP Include                     32            243             31            902
CSS                              1            219             33            850
Javascript                       1            249             99            823
View Template                   54            185              5            681
HTML                             4             19              0             46
XML                              2              0              0             44
-------------------------------------------------------------------------------
SUM:                           320           5710           1508          16602
-------------------------------------------------------------------------------





Friday, September 14, 2012

infinite scrolling but on steroids



we use infinite scrolling plugin by paul irish (https://github.com/paulirish/infinite-scroll) on 3mik now. However Problem with the vanilla plugin is that it assumes that the next URL only depends on current page number. Only  a naive pagination scheme would depend on only one variable.

As you go deeper into pages, you have to scan more and more records before you can arrive at the records you are interested in. For this reason,   real world pagination scheme rely on two variables (see these slides from percona conference  and this blog post )


  • current page number
  • last record id (of previous page ) or first record id (of next page)

An example would be the instagram API

To Apply infinite scrolling pattern to pages with such 2 variable pagination scheme, you need to do additional work. So here is our fork of infinite scroll plugin that documents this use case and provides sample server  PHP scripts.




Monday, June 18, 2012

lint like static code analysis tools for PHP

PHP is dynamically typed language and that means the error detection present in statically typed languages is absent. If you have an error in your script, it will not be detected till you have actually run it(apart from parse errors). The big, shine in your eyes errors are never a problem because they are detected quite easily during development. It is the silent types (doing X when you meant Y and X is a perfectly legit operation) that are more difficult.

Like, I will show you a piece of code I had re-factored  a while back


 
       static function find($subjectId,$objectId,$verb) {
            $mysqli = MySQL\Connection::getInstance()->getHandle();

            //sanitize input
            settype($subjectId,"integer");
            settype($objectId,"integer");
            settype($verb,"integer");

            $sql = " select count(id) as count from sc_bookmark " ;
            $sql .= " where subject_id = %d and object_id = %d and verb = %d ";
            $sql = sprintf($sql,$loginId,$itemId,$verb);

            $row = MySQL\Helper::fetchRow($mysqli, $sql);
            return $row;
        }

 



Here loginId was changed to subjectId in parameter list but I forgot to change the name of parameter I was passing to MySQL.  This used to result in a notice that was going to logs but no sign of program failure outside. That led me to say, enough is enough and I need to have a good static analysis tool for my PHP code.


There are many posts dedicated to the topic of PHP static analysis, sort of things that lint does for c code. The aim is to catch errors from just scanning the PHP code. When I started looking around, everyone talked about their favorite static analysis tool but no one gave me hard data or how it actually looks on your terminal.

I looked  at following tools

  1. PHP code sniffer (phpcs)
  2. PHP on command line with minus eel option (-l)
  3. PHP mess detector (phpmd)
  4. pfff/scheck

After a quick evaluation I (emphasized)  found phpmd to be the best of the lot.  Why?


One more tool worth checking should be https://github.com/sebastianbergmann/php-code-coverage
However I did not look at this one.


PHP code sniffer

phpcs looked too verbose out of the box. For 20 line perfectly legit code it generates 20 errors, most of that to do with style guidelines. Look at the output, I believe real  error will be buried under tons of such warnings. Sure, I did not look at how to tweak it and all but I believe most of the time first impressions are confined to out of box experience. Also, it could not detect the error I had. (error inside a class method)



FILE: /home/rjha/code/github/sc/lib/com/indigloo/sc/util/Asset.php
--------------------------------------------------------------------------------
FOUND 16 ERROR(S) AND 1 WARNING(S) AFFECTING 11 LINE(S)
--------------------------------------------------------------------------------
  2 | ERROR   | Missing file doc comment
  4 | ERROR   | Missing class doc comment
  4 | ERROR   | Opening brace of a class must be on the line after the
    |         | definition
  6 | ERROR   | Missing function doc comment
  6 | ERROR   | Opening brace should be on a new line
 17 | ERROR   | Expected "if (...) {\n"; found "if(...) {\n"
 20 | ERROR   | No space found after comma in function call
 24 | ERROR   | Expected "if (...) {\n"; found "if(...) {\n"
 28 | WARNING | Line exceeds 85 characters; contains 103 characters
 31 | ERROR   | Expected "if (...) {\n"; found "if(...) {\n"
 31 | ERROR   | No space found after comma in function call
 33 | ERROR   | No space found after comma in function call
 33 | ERROR   | No space found after comma in function call
 36 | ERROR   | Expected "if (...) {\n"; found "if(...) {\n"
 36 | ERROR   | No space found after comma in function call
 38 | ERROR   | No space found after comma in function call
 38 | ERROR   | No space found after comma in function call
--------------------------------------------------------------------------------




php -l (minus eel)

looks too limited. It could not detect errors inside class methods.

pfff/scheck

pfff is a project from facebook and it is a suite of tools. There are some interesting ocaml based tools there, like code map generator etc. scheck is the tool to do static checks. However pfff looks to be in alpha stage right now.

php mess detector

This could detect the problem inside a class method. Also, the tool is quick and you can decide on the kind of rules you would like to run.



rjha@mint ~/code/github/sc $ phpmd  lib/com/indigloo/sc/util/Asset.php text codesize,design,naming,unusedcode

/home/rjha/code/github/sc/lib/com/indigloo/sc/util/Asset.php:13 Avoid variables with short names like $ts. Configured minimum length is 3.




Tuesday, May 01, 2012

Send Email using SendGrid PHP library on Github

We are hosting the 3mik.com boxes on Rackspace right now and  we get 40,000 emails/month free of cost  from one of their partners, SendGrid. We use SendGrid to send  emails from our application written in PHP.  The SendGrid documentation suggests using SMTP relays and swiftmail-4.x library.  However, SendGrid also has a PHP front-end library published on github that makes it super easy to send emails.

Here is one complete test script


    require_once($_SERVER['WEBGLOO_LIB_ROOT']. '/ext/sendgrid-php/SendGrid_loader.php');
    set_error_handler('offline_error_handler');
    $sendgrid = new SendGrid('your-sendgrid-login', 'your-sendgrid-password');
    $mail = new SendGrid\Mail();


     $mail->addTo('foo@gmail.com')->
       setFrom('foo@3mik.com')->
       setSubject('Sendgrid github PHP library test')->
       setText('Hello World! from sendgrid library')->
       setHtml('Hello World! from sendgrid github lib');  

    $response = $sendgrid->web->send($mail);
    // Error Handling:- 
    // sendgrid->web method uses curl_exec CURLOPT_RETURNTRANSFER set. 
    // This means you will get FALSE
    // when the send method fails at network level.
    // you get JSON response back when curl is able to communicate with the server
    // success from API is returned as
    // {"message":"success"}
    // Error from API is returned as
    // {"message": "error", "errors": ["Bad username / password"]}
    print_r($response);



?>

Though all SendGrid documentation exhorts you to use SMTP, I do not see a problem in using the WEB method.  As per this SO question, http://stackoverflow.com/questions/6193702/sendgrid-smtp-or-curl , " The  web API actually works faster than SMTP, as you only need to make a single cURL request to us to send a message, whereas with SMTP there's a lot of back-and-forth TCP chatter for connection, HELO, and such." 


Error checks are crucial part of any code. Here is my complete sendgrid wrapper complete with error checks. I do not want to throw exceptions because I want clients to handle the error their own way.





Sunday, March 18, 2012

Upload to s3 using aws php sdk

Here is a quick code sample






    error_reporting(-1);
    require_once 'sdk.class.php';

    // UPLOAD FILES TO S3
    // Instantiate the AmazonS3 class
    $options = array("key" => "aws-key" , "secret" => "aws-secret") ;

    $s3 = new AmazonS3($options);
    $bucket = "media1.3mik.com" ;

    $exists = $s3->if_bucket_exists($bucket);
    if(!$exists) {
        trigger_error("S3 bucket does not exists \n" , E_USER_ERROR);
    }

    $name = "cows-and-aliens.jpg" ;
    $path = "/home/rjha/Pictures/Cows-And-Aliens-1280x960.jpg";

    echo " upload file $name from $path \n" ;
    $sblob = file_get_contents($path);
    $response = $s3->create_object($bucket, $name, array('body' => $sblob , 'acl' => AmazonS3::ACL_PUBLIC));
    if($response->isOk()){
        echo "done" ;
    } else {
        echo "error";
    }






© Life of a third world developer
Maira Gall