Cvuorinen.net

Running Angular tests in headless Chrome

Fri, 05 May 2017 00:00:00 +0000

Angular has some great tooling for running tests, namely Karma and Protractor. By default (at least when using Angular CLI) they run using Chrome. So when you execute the tests from command-line, it will pop open a browser window where the tests execute. This works well enough, but sometimes you either don’t want to see that browser window pop open or you are running the tests in an environment where there is no graphical environment (on a CI server or a Docker container for example).

There is nothing new in running Karma tests without a browser window, you have been able to do it with PhantomJS by installing the karma-phantomjs-launcher. PhantomJS has been good enough solution for this, but you might encounter some issues every now and then and need to add some additional polyfills etc. But Chrome now has the ability to run in headless mode since version 59, so you can use it to run tests without needing to install any additional packages and with a more standard environment.

Conf your Karma

Whether you have a Karma config generated with Angular CLI or one that you have created manually, you can use a config option called customLaunchers to create a new launcher based on an existing one by defining additional flags for it. This works equally with older AngularJS v1.x projects as well as newer Angular v2.x-4.x projects. To use Chrome in headless mode, you need to add the following section to your karma.conf.js

customLaunchers: {
  ChromeHeadless: {
    base: 'Chrome',
    flags: [
      '--headless',
      '--disable-gpu',
      // Without a remote debugging port, Google Chrome exits immediately.
      '--remote-debugging-port=9222',
    ],
  }
}

Note, depending on your Chrome version, the --disable-gpu flag might not be needed.

Then you can replace whatever you had in the browsers section (either ‘Chrome’ or ‘PhantomJS’ etc.) with ChromeHeadless. That’s it, after that you can enjoy running your tests without any browser window popping up.

E2E tests with Protractor

Running E2E tests in headless mode has been a bit more difficult, since it has not worked very nicely with PhantomJS. Basically your only option has been to run Chrome in Xvfb (that’s X virtual framebuffer in case you were wondering). But now it’s as simple as adding a few lines to your protractor.conf.js to also run your E2E tests in headless mode. You need to add the following options under the capabilities key (where you should already have browserName: 'chrome'):

chromeOptions: {
  args: [ "--headless", "--disable-gpu", "--window-size=800x600" ]
}

See this Gist for full examples of both karma.conf.js and protractor.conf.js (both files generated by Angular CLI with only the changes described above).

Go forth and test all the things

Now you can just sit back and enjoy running all your tests without any distracting browser windows, or more importantly you can run them in your Continuous Integration server like Travis CI or Jenkins etc.

For more information, see Getting Started with Headless Chrome on Google Developers.

Using RxJS Observables with AngularJS 1

Thu, 05 May 2016 00:00:00 +0000

Reactive programming with RxJS has become quite popular lately in the frontend world, partly because it is included in Angular 2 and many people have already started learning it even though a stable version is not out yet. But if you are currently working with Angular 1, you don’t have to wait until you start using Angular 2 in your production apps to start using RxJS, since RxJS itself is stable and can be used with any framework.

Whether you already know RxJS and the reactive programming concepts, or just want to learn more about them before you start using Angular 2 for real, this post will show how you can integrate RxJS with Angular 1 and get into reactive programming right now.

Reactive programming in itself is a complex topic, but there are many great tutorials about it online so I will not cover it here in much detail. Some level of understanding about reactive programming concepts and Rx is assumed to be able follow along this post.

Rx Bindings for AngularJS

There is an official library for integrating RxJS with Angular 1 called rx.angular.js. This library works as a bridge between RxJS observables and AngularJS, making it easy to work with Observables in an Angular 1 application.

The main features of the library are:

Trigger digest cycle on a scope when an observable emits a value
Create observables from scope watch expressions
Create observable functions on a scope
Convert scope events to observables

Here is a small example where we create an autocomplete input field that can be used to search GitHub usernames:

angular.module('app', ['rx']) .directive('autocomplete', function () { return { scope: {}, template: '<input ng-model="val" ng-change="update(val)" />' + '<ul class="suggestions">' + '<li ng-repeat="suggestion in suggestions">' + '<a href="https://github.com/{{ suggestion }}"' + ' target="_blank">{{ suggestion }}</a>' + '</li>' + '</ul>', link: function (scope) { scope.$createObservableFunction('update') .debounce(400) .map(keyword => keyword.trim()) .filter(keyword => keyword.length > 0) .distinctUntilChanged() .map(keyword => searchGitHub(keyword)) .switch() .digest(scope, 'suggestions') .subscribe(); } }; function searchGitHub(term) { var apiUrl = 'https://api.github.com/search/users?q='; return Rx.Observable.create(function (observer) { fetch(apiUrl + term) .then(res => res.json()) .then(json => { observer.onNext(json); observer.onCompleted(); }).catch(observer.onError); }) .map(json => { return json.items.map(item => item.login); }); } });

To create an observable that emits the input elements values, we create an observable function called update to the scope usig the $createObservableFunction scope method that rx.angular.js added. We then call that function in the template from the input element’s ng-change. You could also use Rx.Observable.fromEvent(element, 'change') if you have a reference to the input element, but in this case I wanted to showcase the features rx.angular.js offers and how they can be used with normal Angular directives etc.

After we have the input values as an observable, we need to create the autocomplete logic and make the network requests. So, here is a break down of what the code does:

We wait until user has stopped typing with debounce
Then trim any trailing whitespace
Filter out empty values
Only continue if value changed from previous (no new request if only added whitespace that was then trimmed)
Search GitHub using Fetch API and convert the response to an Observable
Since we mapped the keyword to the search function that returned an Observable, we now have an Observable of Observables. So we use switch to flatten it back to a single Observable that will emit the search results. This will also discard the previous inner Observable if it was still running, making sure we get the results in correct order even if the network requests don’t complete in the same order that they were sent.
We then use digest method (added by rx.angular.js) to assign the emitted values to a property on the scope. This is needed since we need to trigger a digest cycle because we used Fetch instead of $http service.
Finally we have an empty subscription. It’s empty since we already assigned the emitted value to the scope, but it’s needed because Observables are lazy so nothing will happen if there are no subscriptions.

Here is a link to a working JS Bin: jsbin.com/fekimuz

Async pipe

If you are familiar with Angular 2 development, you might know about the AsyncPipe (pipes are for Angular 2 what filters are for Angular 1). The Async pipe allows binding asynchronous values in the view, like so:

{{ observable | async }}

It works by creating a subscription to the observable that updates the view value when the Observable emits a new value. It also works with Promises (by adding a “then” callback that sets the view value to the resolved value).

There isn’t really anything magical about this implementation and I decided to make an Async filter so that I could use the same approach in Angular 1 applications, so I wrote angular1-async-filter.

One of the key differences between Angular 1 and Angular 2 is the way the change detection works. In Angular 2 the pipes get a reference to the component change detector, that they can use to tell the change detection system that the component needs checking. In Angular 1 filters don’t have a way to get a handle of the current scope so that they would be able to trigger a digest cycle.

It was quite easy to make the Async filter work with $q promises, as they will automatically trigger a digest cycle, but RxJS Observables don’t have that kind of integration out of the box. You can of course use the rx.angular.js bindings mentioned above, and call .digest() on the Observable in your controller/service to make it work, but I started wondering if there was any way to pass the scope into the filter as a parameter. And turns out you can just use this as an argument in an expression to pass in the scope.

So I just added a check if a second argument was passed in and then use it to trigger a digest cycle in the listener when a new value is emitted. To use the Async filter without rx.angular.js, you need to pass in the scope like so:

{{ observable | async:this }}

You can install the Async filter to your own Angular 1 project from npm like this:

npm install angular1-async-filter --save

Async filter example

As an example, here is the same autocomplete directive but this time using the Async filter instead of rx.angular.js:

angular.module('app', ['asyncFilter']) .directive('autocomplete', function () { return { scope: {}, template: '<input />' + '<ul class="suggestions">' + '<li ng-repeat="suggestion in suggestions | async:this">' + '<a href="https://github.com/{{ suggestion }}"' + ' target="_blank">{{ suggestion }}</a>' + '</li>' + '</ul>', link: function (scope, element) { scope.suggestions = Rx.Observable .fromEvent(element.find('input'), 'keyup') .map(e => e.target.value) .debounce(400) .map(keyword => keyword.trim()) .filter(keyword => keyword.length > 0) .distinctUntilChanged() .map(keyword => searchGitHub(keyword)) .switch(); } }; function searchGitHub(term) { var apiUrl = 'https://api.github.com/search/users?q='; return Rx.Observable.create(function (observer) { fetch(apiUrl + term) .then(res => res.json()) .then(json => { observer.onNext(json); observer.onCompleted(); }).catch(observer.onError); }) .map(json => { return json.items.map(item => item.login); }); } });

Most of it is the same as before, here are the key differences:

We create the Observable using the fromEvent method and pass in a reference to the input element. Since we are not using any Angular features here, there are no digests cycles happening on every key stroke.
Since the Observable will now emit the native DOM event, we need to dig out the actual input field string value (from the target.value property on the event).
After that it’s exactly the same as in the first example, except that there is no subscription and we store a reference to the final observable in a property on the scope (scope.suggestions).
Finally, in the view we use the Async filter on suggestions, which will subscribe to it and update the view when a new value is emitted. We also need to pass the current scope to it using async:this so that it can trigger a digest cycle, otherwise we would not see anything appear in the ng-repeat.

Working JS Bin: jsbin.com/kelutu

Conclusion

As you can see, there is no reason you can’t use RxJS with your existing Angular 1 applications. You just need to understand how the Angular digest cycle works so that you know when you need to trigger it. rx.angular.js offers many ways you can integrate Observables to your Angular code base. My Async filter is a more simple approach, but it should cover the basic situation where you need to update the view at the end of an Observable pipeline. You could also go the most simple route and not add any additional dependencies at all, and just .do(() => $scope.$applyAsync()) to trigger a digest cycle when an Observable emits a value. You can always start simple and add more libraries if you need more features.

Actually, the Async filter is not even limited to just working with RxJS observables, you could quite easily use it with other reactive libraries like Bacon.js or Kefir.js. You would just need to change the .subscribe method to use .onValue instead (pull requests are welcome to make it more interoperable).

Photo credits: Highway 101 by kromped licensed under CC BY-NC 2.0 & modified by me.

Building a Deploy Button

Mon, 19 Oct 2015 00:00:00 +0000

I recently got a Olimex A20-OLinuXino-LIME mini computer as a speaker gift for speaking at the Bulgaria PHP Conference. It’s quite similar to a Raspberry Pi, it has few USB ports, Ethernet port, HDMI output and some GPIO pins to hook up some electronics. Quite a lot of GPIO pins actually, a lot more than on Raspberry Pi. And it runs Debian Linux from an SD card. So it’s perfect for doing some hardware hacking and I have been building something on it last week.

I decided to build a deploy button for a project I’m working on at work. We are using Jenkins CI server and Capistrano for deploying. So the idea was to show build status from Jenkins with two LEDs, green when last build has succeeded and red when last build has failed. And blink the LED while a build is in progress, just as the icons in Jenkins web interface. And then pressing a button would execute a deploy using Capistrano (if the last build was successful).

There is a Python library for interfacing with the GPIO pins and I also found a nice Python library for interacting with Jenkins. There is also an experimental PHP extension for the OLinuXino GPIO interface, but I thought this would be a nice little project to brush up my Python skills, since I haven’t used it that much so I wanted to do it in Python.

If you are interested in building something similar on a Raspberry Pi, there is a PHP library for the GPIO access and also a Javascript one, so there is a lot to choose from.

Wiring

The GPIO pins on the OLinuXino are very small and close together, so a regular female jumper lead did not fit. I had to take away the plastic enclosure from the connectors and put some tape for insulation to be able to fit them on the pins. Not that difficult, but a little inconvenient.

Otherwise the wiring was easy as I already had some experience from building similar circuits with the Raspberry Pi.

Case

I diced to build a case for the whole thing using Legos. We have used Legos with our electronics projects I have done with my daughter, so I had some ideas to add a lid over the button for example.

It came out quite nice and I was able to fit the button and the LEDs quite firmly using just Legos and nothing else. Could be better of course, but I’m happy with the end result and it’s good enough for the use case.

Programming

The main program logic is quite simple, here is an excerpt that has the main loop:

while True:
    if buttonPressed() and build.success():
        deploy()

    build.check()

    setLeds(build)

    time.sleep(1)

You can see all of the code here.

Here are some images of the end result:

HDMI and mini-USB ports can be accessed by opening the hatches.

Bulgaria PHP Conference 2015 & the PHP Community

Mon, 05 Oct 2015 00:00:00 +0000

A week ago I attended the first Bulgaria PHP Conference that was organized on September 26th and 27th 2015 in Sofia. Even though it was the first PHP conference in Bulgaria, the event was a real success. The venue was spectacular, the atmosphere was laid-back but still enthusiastic and energetic and the speaker line-up was very impressive.

I presented a talk about AngularJS, yes a JavaScript talk even though it was a PHP conference. The organizers said that they wanted to have a diverse set of talks that includes something that you don’t come across at every PHP conference. AngularJS is the most popular JS framework at the moment, so many of us PHP developers get/have to work with it. There were also talks about topics such as testing, security, monitoring, graph databases, mentoring, data handling, API design and many others.

I could write a brief summary of all the talks I saw, or I could iterate over the things I learned, or tell you more about how well the conference was organized (but others have already done those). I thought instead that I would write about a topic that has been on my mind since the event. And that’s the PHP community.

Both local and global communities have formed around many different technologies. Communities where developers share experiences, ask questions and help each other. Especially around open source technologies, there are many communities where the important things are not money, fame or who is the best, but instead sharing, openness and helping others without expectations or hidden agendas. And at least from my own experiences I can say that it is great to be a part of the global PHP community. Even though many of the speakers are well-known from different conferences around the world, or from popular open source projects, or work for some of the worlds largest tech companies, the atmosphere is warm and relaxed and nobody seems to think that they are “above others”.

Everyone is willing to help others, and you could clearly see that in the so-called “hallway track” of the conference, which many thinks is one of the best things about attending conferences. It means for example that anybody can come and ask for help about testing from the creator of the most popular PHP testing framework. And he will sit down, take out his laptop and show you and teach you. Even though people are willing to pay money for his books and workshops, it doesn’t stop him (and others) from helping fellow developers during the conference, just for the joy of helping others and being a part of the community.

One of the most important and rewarding things about attending conferences is networking with other people. Nowadays the talks are often recorded and published online, so you can watch them for free later. But the conversations with other like-minded people and forming real life-long friendships are the best things that a conference can offer. This also makes it possible that helping others is not only limited to the hallway track at the conference venue, but people can later chat and ask questions online through various platforms, such as Twitter, IRC, forums etc. It’s great to know that if needed, you can ask for testing related advice from a testing expert, or about new version of PHP from the people who took part in building it, or some tips for handling large quantities of data from a developer that works with the analytics data of some of the world’s largest websites.

Helping others within the PHP community is not limited to only technical things. Recently the PHP community has for example collected money for the family of a PHP developer that passed away unexpectedly to cover funeral expenses and donated money to a rent deposit for a core PHP developer when his family was struggling financially and had to live in a trailer for a while. No one had to donate money but the community members just wanted to help “one of their own”.

Since many of us are working using open source tools, that have in many cases been developed by volunteers without any monetary compensation for their work, it is important that we take part and give back. It can mean participating in the development of these tools by for example fixing bugs or writing documentation, or mentoring other developers, or whatever you can help with. It can be rewarding even in it’s own right, but this way we can all take part in the global PHP community and be a part of something bigger than any one of us can be by ourselves or within the company we work for. And not limited only to the PHP community, but with all the open source communities we take part in.

PHP is definitely not the worlds best programming language from a technical view-point, but it is popular because it’s approachable to new developers, available in most hosting environments and well suited for web development. In addition to these, one of the biggest strengths of PHP in my opinion is the community around it. A community that I can proudly say I’m a part of.

Photos borrowed from BGPHP (with permission), go check out more at flickr.com/photos/bgphp

What is Clean Code and why should you care?

Sat, 19 Apr 2014 00:00:00 +0000

Clean code is something that I have been interested in for a while now, and plan to write a series of blog posts about the different concepts related to clean code. In this introduction post to the series I will talk a little bit about what clean code actually is and also try to answer the question why should you care about clean code.

What is Clean Code?

Clean code is subjective and every developer has a personal take on it. There are some ideas that are considered best practice and what constitutes as clean code within the industry and community, but there is no definitive distinction. And I don’t think there ever will be.

After reading a few books on the topic, giving it some thought and delivering a couple of talks on the subject, if I had to summarize what clean code means in one sentence, I would say that for me:

Clean code is code that is easy to understand and easy to change.

Ok, that sounds nice, but what does it really mean? Let’s break that sentence apart and examine the individual points behind it.

Easy to understand means the code is easy to read, whether that reader is the original author of the code or somebody else. It’s meaning is clear so it minimizes the need for guesswork and possibility for misunderstandings. It is easy to understand on every level, specifically:

It is easy to understand the execution flow of the entire application
It is easy to understand how the different objects collaborate with each other
It is easy to understand the role and responsibility of each class
It is easy to understand what each method does
It is easy to understand what is the purpose of each expression and variable

Easy to change means the code is easy to extend and refactor, and it’s easy to fix bugs in the codebase. This can be achieved if the person making the changes understands the code and also feels confident that the changes introduced in the code do not break any existing functionality. For the code to be easy to change:

Classes and methods are small and only have single responsibility
Classes have clear and concise public APIs
Classes and methods are predictable and work as expected
The code is easily testable and has unit tests (or it is easy to write the tests)
Tests are easy to understand and easy to change

As I stated in the introduction, I plan to write a series of posts that cover these topics in more detail.

Why should you care about Clean Code?

As Robert C. Martin stated in his book Clean Code: A Handbook of Agile Software Craftsmanship, “Clean code is code that has been taken care of. Someone has taken the time to keep it simple and orderly. They have paid appropriate attention to details. They have cared.” But why should you care? What’s wrong with code that just works?

You should care because code is (almost) never written just once and then forgotten. Most of the time you, or someone else, need to work on the code. And to be able to work on it efficiently you need to understand the code.

And because people need to understand the code we write, we can say that the code we write is not intended only for the computer but also for humans.

Programming is the art of telling another human what one wants the computer to do.
— Donald Knuth

If you write clean code, then you are helping your future self and your co-workers. You are reducing the cost of maintenance of the application you are writing. You are making it easier to estimate the time needed for new features. You are making it easier to fix bugs. You are making it more enjoyable to work on the code for many years to come. Essentially you are making the life easier for everyone involved in the project.

Now, I’m not saying you should get obsessed about clean code. Your code needs to provide value, so you can’t spend countless hours making it perfect. Clean code usually doesn’t happen on first try anyway, so you need to adopt a mindset that you will always strive to improve the code you are working on. You then need to decide when it is good enough and move on.

Conclusion

In this post I have tried to explain what clean code means to me and also hopefully convinced you that you should also care about clean code (in case you didn’t previously).

My writing productivity has not been that good lately, but hopefully I will get around to writing more posts soon. So check back if you are interested. Also any feedback and comments would be greatly appreciated, either here or Twitter.

To close things up, I would like to share a few more of my favourite quotes and tweets I have come across lately.

If you want your code to be easy to write, make it easy to read
— Robert C. Martin

Clean code always looks like it was written by someone who cares. There is nothing obvious you can do to make it better.
— Michael Feathers

Beautiful code, like beautiful prose, is the result of many small decisions. The right method length here, the right object name there. — DHH (@dhh) March 21, 2014

"Code is like humor. When you *have* to explain it, it’s bad" - @housecor — About Programming (@abt_programming) March 24, 2014

“The cheapest time to refactor code is right now.” Great advices by @bugroll http://t.co/GGEQ8Bc3e1 #refactoring #cleancode — Luca Guidi (@jodosha) March 20, 2014

The cost of ownership for a program includes the time humans spend to understand it. Humans are costly, so optimize for understandability. — Mathias Verraes (@mathiasverraes) April 18, 2014

Dev time = 60% reading and 40% writing. Code refactor can reduce reading by 20%, improving 8 hours/week, or 416 hours/year. #refactor — Refactoring 101 (@refactoring101) April 7, 2014

CQRS? Or did you mean (bumper) cars? – My PHPBenelux 2014 experience

Fri, 31 Jan 2014 00:00:00 +0000

I recently attended the PHPBenelux 2014 conference with a co-worker of mine. It was the 5th anniversary edition of PHPBenelux and the second time I attended the conference. It was held January 24th & 25th at hotel Ter Elst in Antwerp, Belgium.

In this post I will recap my experience at the conference and highlight what I found most interesting, fun or otherwise noteworthy.

Model Storming

Friday started with half day workshops. We attended a work shop called Model Storming by Mathias Verraes. It was a very interesting workshop that focused on building a domain model in small teams. We didn’t use any digital tools for this, but instead consumed massive amounts of Post-It notes of various colours. We received some good advice what kind of things you should think about and ask from customers (or “domain expert”) when designing software and some nice insight into Domain-driven design.

Mathias also presented a quick introduction to a concept called CQRS towards the end of the workshop. This is a very interesting concept. CQRS stands for Command Query Responsibility Segregation. Basically it means that there is a separate model for writing things and a separate model for reading things. CQRS combined with a concepts like Event Sourcing and Task-based UI provides some interesting capabilities where the UI sends commands to the write model, the write model turns them into events that get stored into an event store. The read model can then create all kinds of different representations of the data, even afterwards because the events can be replayed. CQRS seems interesting and I intend to learn more about it, in case you are interested you can check out some links here. The slides Mathias showed us can also be found here.

Conference time

After the workshop, the “main” conference began. It featured only one keynote (at the beginning) and after that 3 tracks of presentations for the friday afternoon and whole saturday. This year there was also an UnConference, which turned out to be quite popular and I think almost, if not all, of the available UnCon slots were taken.

The keynote was called Mentoring Developers by Elizabeth M Smith. Mentoring is a quite interesting topic and Elizabeth was clearly passionate about the topic. The talk was an interesting mix of the social and professional aspects of mentoring and stories from her personal experience. I enjoyed the talk a lot and it served as a good reminder that I really should get into this mentoring business myself.

CQRS? Or did you mean cars?

Stijn Vannieuwenhuyse (who works with Mathias Verraes and was also helping out at the workshop) did an UnCon talk titled CQRS? Or did you mean cars?. The story behind the title is that some time ago when they first heard about CQRS it was so new that when they Googled for it, Google asked “Did you mean cars?”. This talk recapped most of the information about CQRS that Mathias presented in the morning, but offered some more in-depth code examples so it was very interesting.

Functional Design Patterns Under the Hood

We spent the rest of the afternoon learning Functional Application Design in PHP by Michael John Burgess, Refactoring with Design Patterns by Benjamin Eberlei and PHP Performance: Under The Hood by Davey Shafik. All of these talks were interesting and well presented and we learned a lot.

Bumper cars!

After the last sessions of the day, it was time for the Conference Social. This year the organizers had really out done themselves and the social was a real success. There were the famous Belgian fries and always tasty Belgian beer and fun and games of various kind, including pinball, air-hockey, arcade games, mechanical bull rodeo and to top it all, bumper cars! The bumper cars were so much fun, but the most important thing in any social event are the people. It was great to meet so many friends and make new friends in such a relaxed and fun atmosphere. Really enjoyed it a lot.

Day two: Hemoglobin and Hobgoblins

In a multi track conference there will almost always be at least few slots where you really want to see more than one session. The organizers really wanted people to wake up early for day two, since all three of the first sessions were really interesting talks with great speakers. We decided to go see Models and Service Layers; Hemoglobin and Hobgoblins by Ross Tuck. It was a great talk with some very good points, but you really had to pay attention to keep up. The first slide really says it all. Ross also talked a little about CQRS towards the end of the presentation, are we starting to see a pattern here? There was also a link to a nice PHP library for CQRS by Benjamin Eberlei.

After recovering from the first talk of the day, we went to see The seven deadly sins of Dependency Injection by Stephan Hochdörfer and after that Application monitoring with Heka and statsd by Jordi Boggiano.

Clean Code at the UnCon

After lunch it was time for my UnConference talk Clean Code. The room was full (I like to think it was all me, but the fact that Matthew Weier O’Phinney was right after me might also have something to do with it) and I think it went quite well. I really wish I would have had a little more time to prepare the talk and the slides, but I got some good feedback so I will keep working on it and maybe submit as a real talk to another conference someday. After my session Matthew demoed Apigility which seems great for building good APIs.

Extracting and Closing

Last sessions we went to see were Social human architecture for beginners by Sebastian Schürmann which concentrated more on the human factor and Extract Till You Drop by Mathias Verraes which was a really nice live coding demo of refactoring legacy code.

After the last sessions it was time for the closing remarks, raffles and post-conference social. The organizing team gave us some insight what it is like to create such an event and also thanked the sponsors. There were some great prizes in the raffles or competitions like a 3D printer, a quadro-copter with video camera, few Raspberry Pis and also some elePHPants as is accustomed. Was not my lucky day as I did not win anything.

After the closing it was time for another social and some more bumper cars! Bumper cars were still fun and the BBQ food was really good. Enjoyed it a lot and made some new friends, so again it was a success.

Retrospective

All in all, the trip was a great success in my mind and I really enjoyed it. There were great presentations and I learned a lot, I got some speaking experience and some good feedback from my UnCon talk and had lots of fun with existing and new friends. What more can you ask for?

I have only mentioned the talks I went to see, but there were many more. You can find all of the talks and slides for most of them through joind.in. I think some of the talks were also filmed, so keep an eye out for videos later. Or you could also try googling for previous recordings as some of the talks have been presented before.

If you want to be a part of the PHP community, I can’t recommend enough that you attend the next PHP Benelux conference (and why not some others also if you haven’t already). The PHP community is so welcoming and there are so many great people it is really worth the time, money and effort. Already waiting for the next one, hope to see you there.
-Carl.

What's new in PHP 5.5

Sun, 04 Aug 2013 00:00:00 +0000

So, why another “PHP 5.5 new features” post? I was doing research on the subject for a tutorial at work so I was digging through all the resources I could find on the subject anyway, so I decided to write a post about it. And also because even though there are many great posts already on this topic, I found that none of them were that comprehensive to list all interesting changes that come with PHP 5.5. The PHP manual also has a quite comprehensive documentation about everything that has changed with PHP 5.5 here, but it is scattered across many pages with some not so relevant information (for most PHP devs) in there also. The official ChangeLog also lists everything that has changed, but it obviously does not go into that much detail on any of the topics.

Generators

I think the most discussed new feature has been generators. So, what generators actually are and what are they good for?

Generators look like normal functions or methods, except for the yield keyword. When calling yield the generator will return the given value to the caller but retain it’s state and continue where it left off when it is called next time. Generators also implement the Iterator interface, so they can be used with foreach loops for example. The usual examples given about generators are something like a replacement for PHP range() function or a Fibonacci number generator, I will not repeat them here (you can find these by reading some of the articles linked in this post), but instead present an equally useful example of a FizzBuzz implementation:

class FizzBuzz {
    public function printValues($num = 100)
    {
        foreach ($this->getValues($num) as $value) {
            echo $value . PHP_EOL;
        }
    }

    private function getValues($num)
    {
        for ($i = 1; $i <= $num; $i++) {
            $value = ($i % 3) ? '' : 'Fizz';
            $value .= ($i % 5) ? '' : 'Buzz';
            yield $value ?: $i;
        }
    }
}

$fb = new FizzBuzz();
$fb->printValues();

One thing to keep in mind is that the yield keyword does not work the same way as the return keyword. The return value of calling a generator is actually an object of a PHP internal class called Generator. The Generator class implements the Iterator interface as already stated above, so this means that if you want to get values from a generator outside of a foreach loop, you will have to manually call the current() and next() methods on it (as defined by the interface).

I suppose generators will be used mainly as iterators with foreach, where the generator will be called for each iteration and the yielded value will then be available in the loop, like in the example above. The main benefit of using generators like this is that if you have large set of data that you need to iterate over, you do not have to create one huge array to hold all the data but instead you can process it one piece at the time to keep memory usage under control. Situations like this could be processing some large data set from a database or a file to generate reports or graphs for example. Another use case would be if you do not initially know how many records you need to process, so you can only yield as many as you need and then break out of the loop. Generators can effectively be used to create infinite data sets (like the Fibonacci sequence), although I could not come up with any valid real world use case for it (apart from some advanced maths). Sebastian Bergmann also has written a blog post about using Generators as Data Providers with PHPUnit.

Generators also allow for two way communication between the calling code and the generator, i.e. you can send values back when the execution returns to the generator. This can be done by calling the send() method on the Generator instance. The sent value will be a “return” value of the yield expression. It is also possible to combine sending and receiving values within a generator with the same yield expression, but that could lead to some pretty hard to comprehend code quite easily. You can also throw exceptions into the generator in addition to throwing them from the generator. Throwing from the generator works as expected and the same way you normally throw an exception from a function or method. Throwing into the generator is done by calling the throw() method on the Generator object. You can then catch it inside the generator with a try-catch block around the yield.

Generators that you send data to can also be called coroutines, but that is a quite advanced topic and I will not cover it with anymore detail here, if you are interested to learn more about generators and coroutines, I suggest you read this post by Nikita Popov (developer of this feature in PHP core), the RFC of generators (also by Nikita Popov) and/or this post by Anthony Ferrara (also a PHP core contributor).

Password hashing API

With version 5.5 PHP finally has an easy to use and secure password hashing API, which is a big deal security wise. This allows people that are not very experienced with hashing and cryptography to securely store passwords without having to learn some complicated stuff about salts, cost, stretching etc. If you have already been storing passwords securely by either doing it with your own code, or by using some third party library, this means from now on you can use the simple PHP password hashing API and not have to do it yourself or add any dependencies to do it.

Generating a secure hash is as simple as this:

$hash = password_hash($password, PASSWORD_DEFAULT);

Under the hood the new password hashing API uses bcrypt for hashing and has automatic salting and also otherwise sane default values, so it is secure out of the box on most systems. There really is no reason to use any other hashing mechanism from now on in my opinion. Instead of using PASSWORD_DEFAULT, you can also specify the algorithm to use with PASSWORD_BCRYPT. Currently these are the only valid algorithms, but new ones will be added later as needed, and also the default algorithm may change in the future. An optional third parameter can also be used to provide some options for the hashing algorithm, such as salt and cost (for bcrypt). Note on the bcrypt cost parameter, it has a pretty good default value but it really is hardware dependent. So if you are worried about security, you should try what is a good cost value for your production hardware. An example script to do this can be found at the PHP manual page.

To verify if a password is valid when a user logs in, it is equally simple:

if (password_verify($password, $hash)) {
    /* Valid password */
}
else {
    /* Invalid password */
}

There is also a third function in this new API, called password_needs_rehash. This can be used to check if a password needs to be rehashed in case you have changed the options or hashing algorithm. With this functionality you can keep your software secure in the future as the hardware speeds increase you will eventually have to tweak the cost parameter, or change the algorithm entirely if a better one is introduced.

Here is an example of the usage:

if (password_needs_rehash($hash, $algorithm, $options)) {
    /* Hash password with new algorithm or options */
    /* Store the new hash */
}

If you are not yet ready to upgrade to PHP 5.5, but you like the new password hashing API and would like to use it in your application right now, there is a library that you can use for just this purpose called password_compat. It has the same API as the official PHP 5.5 implementation and it was created by the same developer as the PHP core implementation (Anthony Ferrara). This means that you can use it as long as your app is running with PHP version below 5.5, and when you upgrade to 5.5 it will start using the PHP core implementation automatically and you can remove the dependency to password_compat.

One thing to note is that if you use the PASSWORD_DEFAULT algorithm option, it is recommended to use VARCHAR(255) to store the hashes and not limit the size to current bcrypt hash size explicitly. This is to avoid issues in the future if the default algorithm is changed to something that creates longer hashes.

OPCache

PHP 5.5 has a built-in Opcode cache called OPcache. It is actually the previously called Zend Optimizer+ that was closed source software, but Zend open sourced it to be included in PHP core. An opcode cache increases performance of PHP applications with zero effort on the application side (by storing precompiled bytecode in shared memory so scripts will not have to be parsed on each request), i.e. you get faster app without doing anything. That is a no-brainer and also the reason you should be using an opcode cache even if you are still running some previous version of PHP. The OPcache extension can be installed as an extension to older PHP versions too, or you can of course use APC.

Even though OPcache is built-in with PHP 5.5 (compiled by default with PHP), it is not enabled by default so you will need to enable it. This can be done by simply editing the php.ini file as described in the manual. The way that different repositories for popular Linux distributions handle the installation/enabling this extension remains to be seen, it might be enabled by default or it might need to be enabled separately by installing some other package like php-opcache etc.

Performance wise the new Zend Optimizer+ has been a little bit better than APC. Here is a benchmark chart with PHP 5.5 without any opcode cache, with APC and with OPcache (generated from this chart that was linked in the RFC of OPcache)

PHP Opcode cache benchmark

If you have been using APC’s data caching API you can use the APCu (short for APC user cache) that is a fork of APC with the opcode cache functionality removed. It is still in beta but a stable version should be available soon.

Something that you might need to consider when enabling OPcache is your deployment strategy. OPcache does not use inodes as keys in the shared memory (like APC), but instead it is realpath based. This means that if you override a file, it will start using the updated file immediately. This causes a situation where a running script might start execution on your old code base but start using files from a new code base in the middle of request if deployment happened at the same time. Whether or not you have to worry about situations like these depends on how much traffic your site/application has, how often you deploy, how critical the application is etc. so it’s really up to you to decide. This issue can be handled in many ways, Rasmus Lerdorf has posted a quite detailed description how they handle deploys atomically at Etsy by using a symlinked document root directory and a custom Apache module and PHP extension to resolve the symlink.

Finally

Finally there is… finally. The new finally keyword can be used with a try-catch block to execute certain code regardless if an exception was thrown or not. Actually you don’t even need the catch part, you can put finally after try if you are not anticipating any exceptions but want to make sure that some code is run even if an uncatched exception is thrown.

Here is a small example:

$db = mysqli_connect();
try {
    //the function may throw exceptions which we can not handle
    call_some_function($db);
} catch (SomeException $e) {
    error($e);
} finally {
    mysqli_close($db);
}

In the example above, the code inside the finally block will be called in all of these situations:

The call to call_some_function() executes successfully without throwing any exceptions.
An exception of type SomeException is thrown and catched and the code inside the catch block is executed.
An uncaught exception is thrown.

So the finally is quite useful in case you need some code to execute regardless of what happens. The obvious use cases for this are things like cleanup, close resource handles like file pointers and external connections.

list() in foreach

Foreach now supports the use of list() that allows easier handling of nested arrays. Basically you can map the elements from the second level array into variables, all in the foreach statement itself.

Maybe an example will demonstrate this better, so here is a small script that iterates over some csv data:

$csvData = '1;Foo;bar
2;baz;
5;Lorem;ipsum';

foreach (parseCsv($csvData) as list($id, $name, $type)) {
    echo 'Id: ' . $id . "\t" 
        . 'Name: ' . $name . "\t" 
        . 'Type: ' . $type . "\n";
}

function parseCsv($csv, $separator = ';')
{
    foreach (explode("\n", $csv) as $csvLine) {
        yield str_getcsv(trim($csvLine), $separator);
    }
}

The output from this script would be:

Id: 1   Name: Foo       Type: bar
Id: 2   Name: baz       Type:
Id: 5   Name: Lorem     Type: ipsum

The main use cases for this would be looping over nested arrays where the second level array always has the same known structure. So something like looping over SQL result sets or csv file data come to mind first. It doesn’t really matter where the data comes from, as long as it is in this kind of tabular format where you have any number of “rows” and each row contains same “columns”. You can then use this syntax to avoid using something like $row[1] that really has no meaning as to what data it holds.

empty(anything)

The empty() call (language construct, not a function) now supports arbitrary expressions, which means “just about anything you can throw at it” in more common terms, including function and method calls. Yay for this! No more “PHP Fatal error: Can’t use function return value in write context…”. It’s about time this finally works the way that everyone not knowing any better expects it to work.

Here are a few examples:

var_dump(empty("foo")); // false

var_dump(empty($foo)); // true

$bar = 1;
var_dump(empty($bar)); // false

var_dump(empty(--$bar)); // true

function foo() {
    return 1;
}
var_dump(empty(foo())); // false

class Foo {
    const CLASS_CONSTANT = true;
    public function bar() {
        return 0;
    }
    public static function baz() {
        return null;
    }
}
$foo = new Foo();
var_dump(empty($foo->bar())); // true

var_dump(empty(Foo::CLASS_CONSTANT)); // false

var_dump(empty(Foo::baz())); // true

var_dump(empty(function() { return -1; })); // false

$func = function() { return; };
var_dump(empty($func())); // true

$x = 1;
$y = 2;
var_dump(empty($x + $y)); // false

var_dump(empty(2 - 2)); // true

array_column()

The new array_column() function is a convenient way to get a single column of data from a multidimensional array. Similar to PDOStatement::fetchColumn() and what other database access layers provide for database record retrieval, array_column() can do the same in PHP. Nothing that special about it, you can do the same yourself quite easily, it’s just convenient to have it in the core and might be useful in many situations. You can also select what column you want to use as the index of the returned array by providing it as a third parameter.

Here is a small example:

$array = [
    [
        'id' => 2135,
        'first_name' => 'John',
        'last_name' => 'Doe'
    ],
    [
        'id' => 3245,
        'first_name' => 'Sally',
        'last_name' => 'Smith'
    ],
    [
        'id' => 5342,
        'first_name' => 'Jane',
        'last_name' => 'Jones'
    ],
];

$names = array_column($array, 'first_name');
print_r($names);

$namesById = array_column($array, 'last_name', 'id');
print_r($namesById);

And the output from that would be:

Array
(
    [0] => John
    [1] => Sally
    [2] => Jane
)
Array
(
    [2135] => Doe
    [3245] => Smith
    [5342] => Jones
)

The author of this function, Ben Ramsey, has also released a small userland library that you can use with previous versions of PHP if you can’t update right away but want to use it.

::class

A new ::class syntax allows the retrieval of the fully qualified class name of any namespaced classes that have been imported/aliased with a use statement.

Here is a simple example:

use Some\Really\Long\Namespacing\AndAlsoQuiteLongClassName as Foo;

echo 'Foo: ' . Foo::class;

And it will output:

Foo: Some\Really\Long\Namespacing\AndAlsoQuiteLongClassName

It is quite handy so you will not have to type (or copy-paste) the fully qualified class names around when you already have the class imported/aliased with a use statement.

The main use cases for this are any implementations where you have to provide a class name as a string. Some examples are PHPUnit’s mock builder, Doctrine entity manager, PHP’s built in reflection API and also with PHP functions such as class_exists() and is_subclass_of() etc. I think I will personally be using this most with the Doctrine entity manager since I am currently working with a Symfony project, and I am quite looking forward to using it since it is quite annoying to have to repeat the whole namespace every time like this: $this->em->getRepository('Acme\\DemoBundle\\Entity\\SomeEntity'); when you could just write $this->em->getRepository(Entity\SomeEntity::class); if you have a use statement like this: use Acme\DemoBundle\Entity;

The ::class syntax also works with other existing OOP keywords, such as self, static and parent, for example: self::class and parent::class. Although this does not provide any new functionality since you can get the equivalent of those in previous PHP versions with __CLASS__ and get_parent_class(), the new syntax is cleaner and more consistent so it’s a pretty nice thing in my opinion.

DateTimeImmutable

The new DateTimeImmutable class allows creating DateTime like objects that cannot be modified. This is mainly useful when using a single DateTime object multiple times, for example passing it as a parameter to other methods, performing calculations on it or displaying it in another timezone. Using DateTimeImmutable means you will not have to worry about whether the object has been changed somewhere along the way or not.

The DateTimeImmutable works almost just like regular DateTime, but there are a few gotchas. First one is that DateTimeImmutable does not inherit from DateTime so if you have a method that requires a DateTime object, you can not give it a DateTimeImmutable. The other thing is that, as stated before, DateTimeImmutable cannot be modified, but it returns a new object instead. The regular DateTime object returns $this to allow method chaining, but if you are not chaining anything you can just call a method to change the object, like for example $date->modify('+1 day'). With DateTimeImmutable you can also call it the same way but that will not change anything, you will have to assign the return value to a variable and that will be a new object and not the original, so the example would have to be changed to $date = $date->modify('+1 day').

Array and String Dereferencing

Constant array and string dereferencing, also known as constructor dereferencing and literal dereferencing, basically allows plucking one element from an array or one letter from a string right when creating the array or string. Again an example illustrates this better than any words I can come up with:

echo array('Foo', 'Bar', 'Baz')[1] . PHP_EOL; // Output: Bar

echo 'abc'[2] . PHP_EOL; // Output: c

You can of course use a variable at the referencing part (inside the brackets), which is kind of the only way this feature would make any sense. Although I still have a hard time coming up with any real world use cases for this. As I understand, the main reason behind implementing this feature was “because some other languages have it”. So I guess there are some use cases for this out there.

GD improvements

There are a few improvements to the GD image manipulation extension. Support for image flipping and cropping with functions imageflip(), imagecrop() and imagecropauto(). Support for reading and creating WebP images was also added with functions imagecreatefromwebp() and imagewebp().

Other misc new features/functions

Here is a list of some other miscellaneous new features and functions that I will not cover here with any more details.

Non-scalar Iterator keys in foreach
Apache 2.4 support on Windows
boolval() - Get the boolean value of a variable
cli_set_process_title() - Set PHP CLI process’ title that’s visible in top or ps
hash_pbkdf2() - Generate a PBKDF2 key derivation of a supplied password
CURLFile - cURL file upload class
IntlCalendar

Deprecations and other removals

With all the great new stuff, also some old have been cleaned out. Here is a list of the most notable deprecations and removals:

ext/mysql deprecated. This means that from now on when connecting to a database an E_DEPRECATED error will be generated. It can still be used for now, but it is advisable to use mysqli or PDO instead.
Dropped Windows XP and 2003 support. Yea. Anybody still running modern PHP apps on those?
preg_replace() /e modifier deprecated. preg_replace_callback() can be used instead.
Logo GUIDs removed i.e. php_logo_guid() and zend_logo_guid()

I have tried to cover most of the relevant stuff here. You can check a more comprehensive list of all the little things that have changed at the PHP manual migration guide. Both new and removed.

Conclusion

PHP 5.5 has quite a lot of new and handy features and also a lot of nice little additions/modifications to existing features that make our lives as PHP developers a little bit easier. When doing the research about these new features for the tutorial and this blog post, I have come to the conclusion that the PHP RFCs (Request For Comments) is a really good place to learn about the new stuff. They often offer quite a lot of background about the changes or the new feature and almost always also give some real world use cases for them. Nikita Popov has put up a nice list of new features in PHP 5.5 that links to their respective RFCs. This has been really helpful and I have learned a lot when reading those. From now on I think I will always check the RFCs of the new features when a new version of PHP comes out.

This post turned out to be quite long, if you have read all the way here, hopefully you have learned something new. The next step is to try these out by yourself. I know I learned a lot by playing around with these new features. If you are familiar with Vagrant, you can have a virtual machine up and running with PHP 5.5 in just a few minutes, for example by using the awesome PuPHPet service where you can configure the VM on the website really easily.

Installing SonarQube with Jenkins integration for a PHP project

Mon, 08 Jul 2013 00:00:00 +0000

In this second part of my Continous Integration setup I will detail the steps required to install SonarQube (previously called just Sonar, renamed to SonarQube with 3.6 release just a few days ago) and integrate it with the Jenkins server from the previous post so SonarQube will run a daily analysis of our PHP project. In the previous post I covered the installation of Jenkins on a CentOS server and integrated it with GitHub, so if you do not have Jenkins set up you might want to start there.

So why do we need Sonar if Jenkins is already setup with all the static code analysis reports as in the “Template for Jenkins Jobs for PHP Projects” ? Jenkins is great for running builds and it can be setup in a way that it generates useful reports about the source code, but Sonar has been specifically designed for this. It has a great user interface and gives so much better statistics. You can really easily see what direction the project is going and how much technical debt you are accumulating. You can drill down to different parts of the project and compare them, you can see how the code base has changed over time and compare different versions etc. This is especially useful with large projects that have a long lifespan so you can better keep track of code quality. You can also take action on the issues Sonar has found by assigning them to people. All the dashboards are customizable with loads of different widgets and all the widgets are also customizable so you can really make it your own and make the data that you care about the most easily accessible. SonarQube really takes the “static” out of static code analysis and makes it dynamic and interactive. You have to see it in action to really experience it, so I suggest you either watch a screencast or click around in the live demo.

Installing SonarQube

First, install SonarQube using yum by running these commands:

$ sudo wget -O /etc/yum.repos.d/sonar.repo http://downloads.sourceforge.net/project/sonar-pkg/rpm/sonar.repo
$ yum install sonar

Although this post focuses on CentOS installation, you can find packages for most popular Linux distributions.

Next we have to edit the Sonar config file /opt/sonar/conf/sonar.properties to change the db to MySQL. You can of course use whatever database you like, but I had MySQL already installed on the machine so it was an obvious choice. Just comment out the line about the embedded H2 database (which is not intended for production use) and uncomment the MySQL line, save and exit. After that you also have to create an empty database called sonar and create a user named sonar that has access to the database.

This can be done by running the following commands on mysql:

CREATE DATABASE sonar CHARACTER SET utf8 COLLATE utf8generalci; 
CREATE USER 'sonar' IDENTIFIED BY 'sonar';
GRANT ALL ON sonar.* TO 'sonar'@'%' IDENTIFIED BY 'sonar';
GRANT ALL ON sonar.* TO 'sonar'@'localhost' IDENTIFIED BY 'sonar';
FLUSH PRIVILEGES;

I also had to change the default port of Sonar since I had Nginx with PHP running on the same server and php-fpm was already using port 9000 (which I discovered after Sonar refused to start and I had to take a look at the Sonar log file in /opt/sonar/logs/sonar.log). So I changed Sonar to use port 9090 (in the Sonar config file).

After that you can go ahead and start Sonar using the following command:

$ sudo service sonar start

Note, do not use the script in /opt/sonar/bin… to start Sonar (instructed in Sonar documentation) as root user because you will run into file permission issues later since the init.d script runs it as the user “sonar” (I had to learn this the hard way, so just trying to save you the trouble).

If you have a firewall that blocks access by default, you may need to open the port 9000 (or 9090 if you had to change the port like me) to be able to access Sonar from outside. We had iptables running so I just added a rule to allow 9090 and restarted iptables.

If everything went fine, SonarQube should be running and we can point a browser to: http://serverAddress:9000 (or http://serverAddress:9090 in my case).

Securing Sonar

You can log in with the default account “admin” with password “admin”. The first thing you should do is of course change the password on the page under “Administrator” > “My Account”. After that you can disable access from anonymous users in the “Settings” > “Configuration” > “Security” page by selecting True in the “Force user authentication” setting.

Installing SonarQube Runner

Installation instructions can be found here: http://docs.codehaus.org/display/SONAR/Installing+and+Configuring+SonarQube+Runner

First download and uncompress the file (check the above link for the latest version) and create a symbolic link to this specific version by executing the following commands:

$ cd /opt
$ sudo wget http://repo1.maven.org/maven2/org/codehaus/sonar/runner/sonar-runner-dist/2.2.2/sonar-runner-dist-2.2.2.zip
$ sudo unzip sonar-runner-dist-2.2.2.zip
$ sudo ln -s sonar-runner-2.2.2 sonar-runner

Next we need to edit the configuration file and update database information and server URL. In our case the file is /opt/sonar-runner/conf/sonar-runner.properties and you need to set the correct sonar.host.url, uncomment the line for MySQL as the database, uncomment the database username and password. You will also have to uncomment and update the login details if you selected the “Force user authentication” setting in the previous section.

Next we need to create a new environment variable called SONAR_RUNNER_HOME and add the sonar runner bin directory to the PATH environment variable, this can be done with the following command:

$ sudo echo -e '#!/bin/bash\nexport SONAR_RUNNER_HOME=/opt/sonar-runner\nexport PATH=$PATH:$SONAR_RUNNER_HOME/bin' > /etc/profile.d/sonar-runner.sh

And if you don’t want to log out and back in again for this to take effect, you can run these commands to add the environment variables for the current session:

$ export SONAR_RUNNER_HOME=/opt/sonar-runner
$ export PATH=$PATH:$SONAR_RUNNER_HOME/bin

Now the runner has been installed, and you can test it by executing this command:

$ sonar-runner -v

If everything worked correctly, you should receive the Sonar Runner version number as output.

Installing PHP environment for SonarQube

SonarQube relies on some external tools for PHP analysis, so we need to make sure everything is installed on our system and also install the PHP plugin for Sonar. SonarQube requires PHP Depend, PHPMD, PHP_CodeSniffer and PHPUnit which you already might have installed on your system if you followed my previous post and installed these for the build tasks. If not, check http://phpqatools.org/ for a quick install of everything in one command. SonarQube also requires Xdebug, so we will need to install that with the following command:

$ yum install php-pecl-xdebug

Next log into SonarQube and head to “Settings” > “System” > “Update Center”. Click “Available Plugins” and click “PHP” in the list of available languages and then click Install. After that we need to restart Sonar by executing the following command:

$ sudo service sonar restart

Integrating SonarQube with Jenkins

Now that we have SonarQube all set up, it’s time to integrate code analysis with our Continuous Integration set up from the previous post. For this we need to go back to Jenkins and install the SonarQube Jenkins Plugin. Log into Jenkins as administrator and go to “Manage Jenkins” > “Manage Plugins”, click the “Available” tab and type “sonar” to the filter input in the top right, select Jenkins Sonar Plugin and install it and restart Jenkins.

Then we need to tell Jenkins where our Sonar installation is, so head on to “Manage Jenkins” > “Configure System” and find the “Sonar Runner” section and click “Add Sonar Runner”. Give your Sonar Runner a name (I called mine “Default”) and point it to the SONAR_RUNNER_HOME directory (/opt/sonar-runner). Then find the “Sonar” section and click “Add Sonar”. Give your Sonar installation a name (I called mine “SonarQube”), click “Advanced” and fill in your server URL, login account and database details. Finally hit Save.

Next create a file called sonar-project.properties in the root directory the project you want to analyse with SonarQube. This file is needed to configure the SonarQube Runner. You can find an example file for a PHP project here: https://github.com/SonarSource/sonar-examples/blob/master/projects/languages/php/php-sonar-runner-unit-tests/sonar-project.properties. Set the properties according to your project and save and commit.

Now we need to add a build step to run Sonar analysis on each build. Open the project in Jenkins and click “Configure” on the left sidebar. Scroll down to the “Build” section and click “Add build step” and select “Invoke Standalone Sonar Analysis”. If you used the default name and location for the sonar-project.properties file, you will not have to specify it’s path. In case you selected the “Force user authentication” setting when securing Sonar, you will have to provide login credentials for the runner. This can be done in the “Project properties” text input field by setting the sonar.login and sonar.password properties (not sure if this is really required since they are already in the Sonar runner configuration file, but I added them anyway). Then Save.

Now everything is set up and you can try it out by running a build.

Optimizing and finishing up

Since Jenkins was already running all the static code analysis with PHPMD, PHP Depend and PHP_CodeSniffer etc. in my Ant build, there really is no need for Sonar to run the same stuff again. Luckily there are properties that you can set in the sonar-project.properties file to disable generating the reports again and only analyze existing reports. More details can be found here: http://docs.codehaus.org/display/SONAR/PHP+Plugin#PHPPlugin-Reusingexistingreports

Another thing is that if you have the failonerror parameter in the Ant build.xml file set to “true” for unit tests, Jenkins will not run Sonar at all if any tests fail, since the build already failed in the first step. So I removed the attribute because it really is not needed, at least not in my case since I have the “XUnit test results report” Post-build action configured (from the Template for Jenkins Jobs for PHP Projects) and it will mark the build as failed when there are failed unit tests, but the good thing in this case is that it will do so only after all the build steps have been executed, allowing Sonar runner to run as well.

Installing Jenkins CI server with GitHub integration for a PHP project

Sun, 30 Jun 2013 00:00:00 +0000

Here are the details how to install Jenkins CI server on a CentOS server (version 6.4 in this case) and set it up with GitHub integration so pushing to GitHub automatically triggers a build. Our project is a PHP project so the build will have PHP related stuff and we are going to use Ant as the build system.

Install Jenkins

First open https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+on+RedHat+distributions for installation instructions, here are the relevant points in short; Check java version by running:

$ java -version

If you have the OpenJDK version you are good to go (I did). If not, follow instructions in the page linked above to install a compatible Java runtime.

Then it’s time to proceed with the Jenkins install. Just add the Jenkins repo and install it with yum by running these commands:

$ sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo 
$ sudo rpm --import http://pkg.jenkins-ci.org/redhat/jenkins-ci.org.key 
$ sudo yum install jenkins

And it’s done. Then start Jenkins by running the command:

$ sudo service jenkins start

If you have a firewall that blocks access by default, you may need to open the port 8080 to be able to access Jenkins. We had iptables running so I just added a rule to allow 8080 and restarted iptables. Done.

Next you can check that Jenkins is running by pointing a browser to http://serverAddress:8080

If you are greeted by “Welcome to Jenkins!”, everything is fine. If not, then I can’t help you since I was lucky enough to get it right the first time, sorry.

Securing Jenkins

As you may have noted, there was no login or anything, so we need to do some securing so that not everybody can mess with our precious CI setup. Navigate to Manage Jenkins in the left sidebar and then Configure Global Security. Check Enable security, but do not save yet (otherwise Jenkins will lock you out). Under Security Realm select your preferred authentication method, I selected “Jenkins’s own user database” and then select the preferred authorization type, I selected “Logged-in users can do anything”. If you also choose to use Jenkins’s own user database, make sure to keep the “Allow users to sign up” checked for now. Then hit save.

Jenkins will now kick you out, so to speak, and present you with a login screen. If you followed my example and chose the same authentication method, you should see “Create an account” link. Follow that and create a new account. After you have successfully created a new account, you are automatically logged in. Then go back to “Configure Global Security” and disable “Allow users to sign up” to prevent anonymous users from creating accounts.

GitHub integration

Next we will integrate Jenkins with GitHub to automatically trigger builds in Jenkins when new code has been pushed to GitHub or new pull requests are opened.

For this we will need to install a few plugins. Go to “Manage Jenkins” and then “Manage Plugins”. Select the Available tab and scroll or filter to find a plugin called “GitHub Plugin” and check it, click “Install without restart”. On the next page you will see the progress of the install, you can check the “Restart Jenkins when installation is complete and no jobs are running” so you will not have to restart it manually.

Next go to “Manage Jenkins” and then “Configure System”. Halfway down the page, you will find Git plugin with a button “Git installations…”. If you have git installed by yum etc. you will not have to change anything here. Otherwise install git and type the install dir here. At the bottom you can find a section called GitHub Web Hook. Select “Let Jenkins auto-manage hook URLs” and add your GitHub username (or create a specific user in GitHub for Jenkins) and API key. After that, hit Save.

Finally we need to configure a Service Hook in GitHub. Go to your projects settings page and click “Service Hooks”. Scroll down until you find “Jenkins (GitHub plugin)” and click it. Add the url of your jenkins server followed by /github-webhook/ (i.e. http://serverAddress:8080/github-webhook/) and Save.

Create a job for the project

In Jenkins you will need to create a job for something you wish to build periodically. The job represents our project in GitHub that we wish to build. Click “New Job” and fill in a job name (it’s going to be easier if you use only alphanumeric characters because Jenkins will create a folder with this name in the filesystem) and then select free-style software project.

After creating a job, you will be presented with the job configuration view. You will need your GitHub repository URL. Paste the HTTPS URL to the “GitHub project” field at the top and then under “Source Code Management” select Git and paste the SSH URL into the text field. Under “Build Triggers” check “Build when a change is pushed to GitHub” and save.

Now the job has been created but as you might have noticed, Jenkins is still not able to access the repository, at least if it is private. You will need to set up SSH keys in the Jenkins home directory. My setup did not allow me to “su” as user jenkins, so I had to do this as root. I created a new key normally for the root user, saved it to GitHub (I used the per repository “Deploy Keys” option) and then connected to the repository from command line (so that a known_hosts file was created). Then I copied the key files and the known_hosts file to /var/lib/jenkins/.ssh and chown’d them to jenkins:jenkins. After that the job configuration page warning did not appear anymore and everything was working fine. Note. you must pay attention to the file permissions of the files inside .ssh folder (and the folder itself) when moving from somewhere else. If you are not familiar with SSH keys or GitHub “Deploy Keys” feature, here a few helpful links:

Generating SSH Keys at GitHub Help
Managing Deploy Keys at GitHub Developer

Create a build file

Now that we have a job for our project, we need to have that job do something, e.g. build. Jenkins supports Ant and Maven builds by default, as well as executing bare command line commands. We are going to use Ant, so first we need to install it. I followed the steps outlined in this page to get it installed: http://xmodulo.com/2013/03/how-to-install-apache-ant-on-centos.html. After that go to Jenkins configuration “Manage Jenkins” and then “Configure System” and click the “Ant installations…” button in the Ant section. Add a new Ant and give it a name (I named mine “Default”) and point it to the ANT_HOME directory and Save.

Next we need to create a build file for our project. What tasks you want to do in a build really depends on the project, but some obvious choices for a PHP project are things like execute unit tests, analyse code coverage of test, generate API documentation and run static code analysis (like PHP_CodeSniffer, PHP_Depend, PHP Mess Detector, PHP Copy/Paste Detector and PHPLOC). A great resource for PHP related Jenkins setup can be found in http://jenkins-php.org/. They also have a sample build.xml file that has configuration for all of the above tasks. You can start with that and modify it to your liking and the requirements for your project. The build.xml file should be placed at the root folder of the project, and pushed to GitHub. You can of course play around with it and test different tasks before committing. Also make sure that all the tools you are going to use in the build are installed on the server. A good resource on these is The PHP Quality Assurance Toolchain at http://phpqatools.org/

Configure Jenkins to execute the build

After we have a working build.xml committed to GitHub, we need to configure Jenkins to execute the build. Open the “Configure” page under the job created previously. Scroll down to “Build” section and click “Add build step” and select “Invoke Ant”. In the “Targets” input, type the build targets you wish to execute. This really depends on your build file, but in case of the sample build file it should be “build” (or “build-parallel”).

Now when you push new changes to GitHub Jenkins will automatically download the new code and execute the build tasks. Way to go! If you get a blue dot beside the project name in the Jenkins home page, your build was successful. If you get a red dot, you were not so lucky and you got some fixing to do. Either some of your build tasks failed to run at all, or you tests didn’t pass etc. You can check the console output of all the executed tasks by clicking the build and the “Console Output” on the left sidebar. And if you would like to have a green dot on successful builds rather than a blue one, there is a plugin for that.

Continuous Inspection of code quality

Now that we have Continuous Integration set up, I am going to focus on installing Sonar for running Continuous Inspection of code quality and integrating it with Jenkins.

Validating data with triggers in MySQL

Tue, 14 May 2013 00:00:00 +0000

MySQL triggers can be used to create some validation conditions that are a little bit more complex than what can be achieved with basic data types and unique index for example. The reason why data validation is better kept at the database level rather than application level is that in case the same data source is used by multiple applications, or even multiple interfaces within the same application, is that you can rely on the data being consistent and valid regardless the validation logic on the application side, which might not always be consistent across different implementations.

So why triggers are ideal for this kind of logic? Triggers can be executed before data is inserted or updated into the database, and you have the values that would be inserted to the database at your disposal, as well as the old values of the row in case of an update. The insert or update can also be prevented from actually executing from the trigger with an error message. This makes it an ideal place to consistently enforce some validation logic in my opinion.

There are many great tutorials about MySQL triggers, so I will not write about that. If you are not familiar with triggers, I suggest you first read a tutorial or a few, or check out the MySQL manual.

SQL Error: Cannot add or update a row

So we can do the validation in an IF statement inside the trigger, but how can we cancel the insert/update and throw an error? MySQL 5.5 introduced the handy SIGNAL operator that can be used to do just that. It allows to set a specific error condition (or SQLSTATE) and a custom error message. This is exactly what we need in this case, as it will return a native MySQL error and thus also prevent the insert or update clause (as long as we use it in a trigger that is specified to run before insert or update).

But what about MySQL 5.1 (or even 5.0)? If you are still running old versions, the SIGNAL statement will trigger a generic MySQL syntax error about the SIGNAL statement when creating the trigger so you can’t use that. A simple trick is to call a non existing stored procedure to trigger a MySQL error. If you use some generic name for the non existing procedure, like throw_error(), you will not be getting a very useful error message so it can be quite confusing especially for people that do not know about this trigger and the constraint it enforces. This can be improved a little bit by using the error message in the actual call, so the error message becomes a little bit more relevant.

Example

So here is an example of a case where a table has a foreign key to another table and the relationship between them is one-to-many, but the table also has an active boolean field (0|1) and there should always be only one active row per relationship.

DELIMITER $$

CREATE TRIGGER example_before_insert_allow_only_one_active
     BEFORE INSERT ON example_tbl FOR EACH ROW
     BEGIN
          IF NEW.active = 1 AND (SELECT COUNT(id) FROM example_tbl 
               WHERE active=1 AND foreign_key_id=NEW.foreign_key_id) > 0
          THEN
               SIGNAL SQLSTATE '45000'
                    SET MESSAGE_TEXT = 'Cannot add or update row: only one active row allowed per type';
          END IF;
     END;
$$

CREATE TRIGGER example_before_update_allow_only_one_active
     BEFORE UPDATE ON example_tbl  FOR EACH ROW
     BEGIN
          IF NEW.active = 1 AND (SELECT COUNT(id) FROM example_tbl
               WHERE id<>NEW.id AND active=1 AND foreign_key_id=NEW.foreign_key_id) > 0
          THEN
               SIGNAL SQLSTATE '45000'
                    SET MESSAGE_TEXT = 'Cannot add or update row: only one active row allowed per type';
          END IF;
     END;
$$

Here is a sample output from an update statement that does not pass the validation logic:

mysql> UPDATE example_tbl SET active=1 WHERE id=2;
ERROR 1644 (45000): Cannot add or update row: only one active row allowed per type

Here is the same example for older MySQL versions that do not support the SIGNAL statement. Here we are using the same error message, but this time we are trying to call it as a stored procedure. Not that pretty, but it works.

DELIMITER $$

CREATE TRIGGER example_before_insert_allow_only_one_active
     BEFORE INSERT ON example_tbl FOR EACH ROW
     BEGIN
          IF NEW.active = 1 AND (SELECT COUNT(id) FROM example_tbl
               WHERE active=1 AND foreign_key_id=NEW.foreign_key_id) > 0
          THEN
               CALL `'Cannot add or update row: only one active row allowed per type'`;
          END IF;
     END;
$$

CREATE TRIGGER example_before_update_allow_only_one_active
     BEFORE UPDATE ON example_tbl  FOR EACH ROW
     BEGIN
          IF NEW.active = 1 AND (SELECT COUNT(id) FROM example_tbl
               WHERE id<>NEW.id AND active=1 AND foreign_key_id=NEW.foreign_key_id) > 0
          THEN
               CALL `'Cannot add or update row: only one active row allowed per type'`;
          END IF;
     END;
$$

And here is the sample output from an update statement:

mysql> UPDATE example_tbl SET active=1 WHERE id=2;
ERROR 1305 (42000): PROCEDURE database.'Cannot add or update row: only one active row allowed per type' does not exist

Conclusion

Data validation should be consistent across all applications or interfaces using given data source to enforce data integrity and validity. This can be hard to achieve especially if multiple developers use the same data source in different applications. Keeping the validation logic at the data source itself, in this case in a MySQL database, makes sure that the same validation logic is applied always and across all applications and even command line users accessing the database. Triggers are an ideal place to create such validation when basic data type and index rules are not enough.

Of course this does not mean you can forget about data validation in your application, you should always filter or validate user submitted data and give users meaningful error messages.