Here is an example…

├── ./app
└── ./plugins
    ├── ./plugins/plugin_a
    ├── ./plugins/plugin_b
    └── ./plugins/plugin_c

The Problem

Unfortunately, it’s not a perfect world, and some platforms are not as well architected. Alternatively, they may be architected that way, but you may still want some files above that isolated area. This typically poses a problem. How can we effectively combine several components, each with its own tree of files, into one working copy?

What if we’re stuck with four projects like this – each with no top level “plugin” directory?

├── ./admincp
├── ./includes
│   └── ./includes/functions_something.php
├── ./index.php
└── ./something.php

├── ./admincp
│   └── ./admincp/security.php
├── ./images
│   └── ./images/secure.png
└── ./includes
    ├── ./includes/functions_security.php
    └── ./includes/xml
        └── ./includes/xml/product-security.xml

These two projects have to fit into the core platform’s filesystem tree in order to work. We’ve been dealing with projects made up of a dozen or so smaller projects, and it was a nightmare to manage. Initially, we had one large repository, which was fine, but as soon as you want to start re-using any component it becomes problematic. Symbolic links can work, but we didn’t want to have to maintain that configuration for a growing number of projects, on my computer as well all staff.

The other headache of a large repository (or project) is the number of files you need to work with. The context of a given task becomes much larger, and it’s harder to find the files that are relevant.

Enter Union Filesystems

Our first stab at this was looking for a sync program to handle syncing multiple directories into a master one. That seemed reasonable, but was still not ideal. After more searching, we came across union filesystems. Specifically, AUFS: Another Union Filesystem. Now, there may be other better ones at some point, but that’s beyond the scope of this article…

AUFS is a type of filesystem that allows you to stack a series of directories into a single target. Depending on your development platform, you may not be able to do it. I made the plunge of switching to Ubuntu a while back for this very reason: if you are developing for Linux systems, you should be building them on a Linux system. It’s possible to install for other Linux type systems, but very cautious.

Here is the command…

mount -t aufs -o br=/src1:/src2:/src3 none /path/to/stacked/dir

You can essentially pull in those 3 source directories, stack them at /path/to/stacked/dir, and run your application from that directory. Directories stack from right to left, so anything on the left will take precedence. You can also specify how the stacked directory behaves when modified. This is where it gets very cool.

Let’s say we have 3 plugin directories: plugin1, plugin2, plugin3, and our base system: platform. Let’s say we want to perform a fresh install of platform to instance/.

Here is the command...

mount -t aufs -o br=local:plugins/plugin1:plugins/plugin2:plugins/plugin3/vendor/platform none instance

I've shortened the paths for the sake of readability (you should be using full paths), but that's what the command would look like. I've introduced a new folder as well called local which would contain all of our changes on it. We can use this for several things:

• Versioning our local installation
• Having a folder only contain system-created files (log files, uploads, etc.)

AUFS creates a bunch of .wh..wh* files in the furthest-left directory. This keeps track of any deleted files. This is also very cool: we can delete files from our live instance, and the filesystem reflects that, but the source files are never actually deleted.

The other handy use-case of this is deploying an instance of a platform for users to screw around on without any harm to the source system itself. If they screw something up, you can restore them easily enough. You can also run a ton of sites from a single set of source files.

For local development, it makes sense to have your VCS ignore the .wh* files and use it for project configuration and other local project artifacts that you want versioned.

Workflow

Being able to split up repositories may offer some changes to your existing workflow. It allows you to version more things than you typically could, which means you may end up with more small repositories.

When developing, I'd load the source files in your editor, and never directly edit the stacked directory. This allows you to work on very small projects, only having to worry about what is needed and relevant to that project.

The only flaw we've run into is the mount command doesn't always pick up new files quick enough. You may find yourself re-running the mount command as part of your workflow. To me that is a easy to tradeoff that I'm happy to take.

Also, be warned that these mounts are not permanent unless you either run them at start-up, or put them in your fstab file.

Rapid Project Development

If you often build projects consisting of a platform and multiple components, you can quickly generate new "builds" (stacked deploys) and mess around with them efficiently.

To be honest, I have no idea how this would perform in a production environment. If you give it a try, let me know how it goes.

Not a Replacement for Goood Architecture

For the primary use case I've described, this is sort of an anti-pattern. Versioning and packaging systems handle this problem far better. However, be aware that not all platforms were created with modularity in mind (or they failed), in which case traditional tools do not work very well.

I wouldn't do this for any of my newer projects (er-platforms), except perhaps as experimentation, and I much prefer the use of git submodules, or even a packaging system such as Composer (more on that later!).

This has already saved us piles of time, so hopefully somebody else finds it useful, or at least interesting.

Ambitious, right? After searching for a while, I couldn’t find anything like this out there. Just today I decided to take another stab at it, and actually came up with an elegant solution. No plugins or hacky measures required.

Create a new bookmark and set the keyword as something short that you would like to use. I used cd. The way smart bookmarks, or keywords, work is you enter keyword everything else into the address bar, and everything following the keyword gets put into the %s placeholder from the bookmark URL.

To navigate to a relative path, we’d use the following bookmark URL: javascript:window.location="%s"

Now, whenever you type something like cd .. or cd /admin into your address bar, you’ll navigate to the correct relative path. If you’re super lazy like me, you can use Ctrl+L to bring the address bar into focus.

It’s all about the small victories.

Our forum brand, ForumOps works on a lot of vBulletin forums, primarily running on 3.8.x. We ran into quite an annoying "exploit" (perhaps not technically, but it’s caused us problems) that older communities should be interested in.

Whenever you copy/paste an image into the editor, it gets converted into an inline image. This makes people say “wow!” because it’s so much more convenient than using the attachment system or than using a 3rd party image uploading service. vBulletin’s old WYSIWYG editor uses inline images, which get inserted directly into the post with base64 encoding. There are a few major concerns here…

However, in order for this to cause problems, you need to have the post character limit disabled. This is quite common, since without a technical explanation, why would you want to limit the post length? Some people post long articles and are frustrated by it. Usually, the inline image is harmless… people who copy/paste rich content typically only paste in small images or icons, which jut work.

If you post a large image, and have the character limit disabled, it can cause a myriad of problems.

Database – Let’s say an average post is 1000 characters. That is approximately 1000 bytes, or roughly 1 kilobyte (assuming they are standard English characters). Once you paste an image in, that post could now be 1MB, or even 10MB (depending on multiple PHP settings). 1MB is a fairly reasonable image size that we see quite often. With a single image embedded in the post, that post is now takes up 1000x space. If you had a few images, or a larger image, it could be up to 100,000x larger! People tend to see performance problems once they get a few hundred thousand posts, and especially once they get into the millions. If you have a few posts, you will be hitting those limits a lot faster than you’d expect. This can create some obvious database issues with performance and maintainability. This may even cause your post table to crash and give you a ton of hard to debug problems. A DBA will notice the table is huge, but will just assume it’s because of all the post content.

vBulletin Code – The biggest problem we ran into was with the BB code parser. Instead of it having to parse 1kb of text, it now has ~1000x more text to process, which is brutally slow. This makes viewing that broken thread impossible since it will likely time out, or just hang forever (long enough for people to give up, anyway). If it’s buried in a popular thread, it may even be near impossible to remove, since it’s hard to delete posts without actually going into the specific thread.

Now, our client was lucky since the problem was ran into on accident. A user had posted a 5MB image, and we received complaints about certain parts of the site being broken (BB Code parser showing new posts, and in a custom moderator section). A malicious user could easily use this to bring the site down rapidly, giving the moderators little to no chance of fighting back.

The Solution – If you insist on disabling your post limit, please consider setting it high (even very high; say 500,000) since users could still post a huge post and cause issues. Images are a little trickier since it’s unintentional. However, people are set in their ways, so lets at least try to mitigate the root problem…

If you create a new plugin using the "newpost_process" hook, and use the following code, it should prevent attacks.

if (strpos($post['message'], 'data:image') !== false and strpos($post['message'], 'base64') !== false) {
    $maxSize   = 1024*500;
    $totalSize = 0;
    $matches   = null;
 
    preg_match_all(
    	'#data\:image/(gif|png|jpg|jpeg);base64,([0-9a-zA-Z\+\/\=]+)\[/(img|IMG)\]#', 
        $post['message'], 
        $matches,
        PREG_SET_ORDER
    );
 
    foreach ($matches as $match) {
        $totalSize += strlen($match[2]);
    }
 
    if ($totalSize > $maxSize) {
        $dataman->errors[] = sprintf(
            trim("
            	You've copy-pasted some images in your post which are %s too large.
            	Please remove them and add them as attachments instead.
            "),
    	    vb_number_format(strlen($post['message']) - $maxSize, 0, true)
        );
    }
}

This checks for the existence of inline images within a post, and can limit them to a specific file size. In the above code, I have it set at 500kb, which is probably a little high. Feel free to tweak the error message the user will receive.

I have no idea which versions this affects. I know it works on 3.x, and perhaps some of the earlier 4.x versions before the editor was swapped out. If anyone can verify it for specific versions, I’ll be sure to mend my post.

Hope this helps someone out there!

My experience is strictly in back-end development, so I try to keep up with the experts in the global PHP community. Due to the nature my past clients, I’ve also grown to specialize in performance and security. If you’re looking for a mentor, or even to mentor others, this is a great opportunity to learn new things and meet other people who face similar problems to you every day.

We meet every Wednesday morning at 8am-9am, inside Accelerate Okanagan. The format is an hour a week before work to be consistent enough to continuously re-enforce what you learn, and also give you the chance to quickly apply new things while they are fresh. There is already a core group of about 4 of us, but we’d love to see some new faces out!

Mobile web applications are an interesting example. A large portion of the code is downloaded and executed by remote clients, so we tend to pay more attention to total file-size, number of HTTP requests, and ultimately how much code gets executed on the other end. If we are graded on our work, surely this is what will be looked at. There are benchmarking tools such as Yahoo YSlow and Google Page Speed which help you analyze your site and suggest techniques to apply. Naturally, everyone wants an A or a high rating.

About a week ago, I was looking over one of my projects, and wondered what the hell happened to the code; it was awful. White-space was missing in a lot of areas, CSS was condensed (in most places), and even some of the PHP caching code was pretty hackish. Since I was having to rebuild a large component of the project for other reasons, I made an effort to clean up these concerning areas.

If you can optimize your architecture or design as you go, then by all means do it. What you shouldn’t do; however, is try and remove things that affect readability in favor of performance. I think there are two acceptable ways to do this… as part of a build/deployment script, or as a filter in production. Your development copy should always be for humans. Developer time is often significantly more expensive than server resources. Let’s look at some of the various resources we need to deal with…

CSS / JavaScript

Tools like minify are awesome for this. Simply pass your external CSS/JS files as a query parameter, and it generates a combined and minified version of the resources so the client only downloads one file. You can also use this or similar tools for inline CSS/JavaScript, though those can usually be avoided unless you’re dealing with truly dynamically generated code.

If you’re using Zend Framework, check out these view helpers to use Minify with Zend_View_Helper_HeadScript and Zend_View_Helper_HeadLink. Similarly, if you’re using Symfony2, check out the builtin component, assetic.

Whitespace / HTML File Size

Make your templates as readable as possible. There are two ideal choices to deal with this: gzip/deflate compression or a simple filter that removes extraneous white-space. Here’s an example of how you could set up compression. Add any of the following lines to your

<IfModule mod_deflate.c>
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/x-javascript
</IfModule>

Each particular line would enable deflate compression for the associated mime type. You can usually expect around 2/3 to 3/4 of your filesize to disappear when compressed. The other approach, which was a filter, would look like this:

$response = trim(preg_replace('/>\s+</', '><', $response));

This is blatantly taken from Twig‘s spaceless filter. Note, there isn’t much point in using both of these techniques. It’s best to benchmark both (resulting file-size, server compression time, client decompression time) and decide for yourself.

Images

Although not in scope of this post, you can do similar processing to images. There are command-line tools available to compress images, or remove meta data from them to help eliminate a few percent of file-size. This is definitely not the low hanging fruit, but interesting nonetheless. Don’t spend too much time optimizing specific images when it’s possible to run batch processing across all of your images at deploy or update time.

Do yourself and your peers a favor, write code with them in mind.

Web Application Flow

A typical web application’s job is to turn a request into a response. In order for this to appear responsive, applications should generate that response as quickly as possible – I usually find anything under half a second acceptable. If we were to build the application asynchronously, we’d delegate any complex logic, service calls, or other expensive tasks. These tasks can usually be broken down into two categories…

Preparing Data For This Page

If a page requires us to perform complex queries, pull data from web services, etc. then that will likely add considerable delays to page generation. The usual response is caching, but people tend to naturally still use it in a synchronous fashion. Let’s take a twitter feed as an example (showing latest 10 tweets).

You’ll see code like this quite often:

public function getTweets($username) {
    if ($tweets = $this->_cache->get("tweets_$username")) {
        return $tweets;
    }
 
    $tweets = $this->_twitter->get($username);
    $this->_cache->set("tweets_$username", $tweets);
 
    return $tweets;
}

It checks to see if our cache entry exists, and returns it if it does. If not, it calls the routine that fetches the tweets from Twitter, and caches that result (assuming our default cache expiration). Let’s say this is 30 minutes. The problem with this approach is every 30 minutes, it needs to download the information again, making the user wait while the page loads. The other major problem is that when Twitter is down (or for whatever reason, we are unable to connect), the page is broken, because the cache has expired.

A much better approach is to never even attempt to do this work in the main application flow. Instead, let’s have a cron job run every 30 minutes that downloads this information and caches it with no expiration. This way, even if Twitter is down, the cached copy is always expected to work. You could still use the above code as a safeguard, but with this approach it should never have to be used as a fallback. You’d only want to ever update the cache if you successfully retrieve your information.

This model works with almost any data you can cache. You’ll need to decide if you’d rather only perform these tasks on demand (when a user visits a page), or if you want to do a little extra work to ensure the data is always available. Depending on the traffic of the site, or the amount of unique cacheable items you are working with, you’ll have to weigh the benefits.

Processing Needed Later

The other category usually comes from requests that actually need some sort of action performed. One concept we’re going to look at which really helps scale your applications is a work queue or job queue. With a queue in place, we can defer execution of actions so we can quickly complete our request to response cycle. Just because something is delayed, it doesn’t mean there will be a lapse in our accurate data. By simply delaying something for 1, or even 0 seconds, it can be run immediately, but from another process.

The way this works is we need to have a few different processes in place:

Job Queue

Storage of tasks to be executed. Example: print spooler

Job Worker

A process that reads tasks from the queue, and executes them. Workers can be spread across multiple physical machines, if necessary.

Client

A client would be our application, which adds items to the job queue. Similar to workers, we could have multiple applications using the same queue.

Let’s look at some examples…

Sending Mail

Using PHP to send mail is a relatively simple task. If your server can directly send mail, then it is probably very quick, too. A lot of servers will use some sort of mail queue internally to ensure mail gets sent at a steady pace and doesn’t overload the server. However, if you are sending mail via remote web servers (for example, using GMail), then it will take a lot longer, especially if you are sending lots of mail.

Instead of making the user wait a few extra seconds, we can instead toss the mail into a queue (which should be very quick) and not worry about it. Then, one of our mail queue workers can process it as soon as they are available. Depending on your set up, this could be instant, or it could be very delayed. It’s up to you to decide how important the timing of your mail delivery is.

Rebuilding Stats, Caches, etc.

Another common scenario is performing clean-up processing after making changes to your data. Delaying execution until another process can pick it up is usually harmless to the user, and gets them to their next task that much faster.

Different Types of Queues

There are two very common implementations or approaches you can take.

Running as a Daemon

If you need to minimize time between queuing and execution, or more importantly, between scheduled execution and actual execution, you should look at something like Gearman or beanstalkd . These applications run as daemon and can immediately process queued tasks as they are ready to be executed. However, since it requires software to be running, it may not be available or possible in some hosting environments, and takes a little more work to set up. I’d argue, for the accuracy, it will faster than the alternatives since it is not having to continually poll to get new data.

Running Scheduled Tasks

If you are limited, a simple option is to set up a scheduled task for your worker. It should be a simple PHP script that connects to your queue (you could store it in the database), and processes items as they are available. If you want things to run as quickly as possible, you can make your cron job run more frequently, however this may also burden your system depending how frequently you poll, and how intensive your scripts are.

I’m going to follow this post up with some examples of implementing this with both a database-based queue, running as a scheduled task, and also with either gearman or beanstalkd.

When I’m looking at software to use, whether it’s development tools, content management systems, or other applications, there is almost always a direct correlation between quality of product and size of community. When I talk about “product”, I don’t necessarily mean the actual downloaded package or even the code.

I’ll admit, I’m usually quick to point out poorly designed software. As a developer, I hate WordPress with a passion, but because it has such a strong community, the ecosystem around it is incredible. I can get by simply by installing plugins and themes, and hopefully not ever have to touch any code. If I get stuck, with anything, there is so much community, support and documentation on the web that it’s a very positive and usually fruitful experience.

This past week I’ve been working with another seemingly popular CMS. I thought it would be easy to work through whatever issues I had, because I’ve heard of it before, and you know, if it’s anything like WordPress I’ll have no problem… do a quick search, find a plugin, configure, profit. I very quickly found a screencast by one of the few (only?) companies supporting the product. It was outdated, and on the version I was working with, did not work correctly. I immediately asked on Twitter and my extended network. Nothing.

For examples sake, let’s say this product was technically far superior to WordPress. It doesn’t matter. No amount of technical superiority will ever amount to anything if it has no strong community to back it.

Since the rise of social networks, forums, GitHub || SouceForge (they need some love, too), the global software community has become a lot more united. It’s not just the geeks on mailing lists and IRC anymore. If you’re developing software for the public, embrace community. Community wins.

“Once a user Likes a page, by design, the landing page for future visits will be the Wall tab. Thanks for the report.”

This is not what you’d expect – since it only appears to happen when the application is the default tab. I’d expect it would always redirect back to the tab you had open. They can still satisfy their above quote this way.

We couldn’t come up with any fixes, but found a few workarounds:

• Don’t use it as the default tab
• Point your marketing material to the actual tab and not the generic page URL

I also came up with a potential solution the next morning, which should work, though not tested:

• Add some JS code (pseudocode): if window.top.href == ‘generic page URL’, set window.top.href = ‘page URL with specified application tab’

Hope this helps someone!

Facebook has always been a bit of a pain with things like FBML, but now that page tabs support iframes, I thought I’d revisit it a little. It turns out that you can simply use your localhost (or any private) address, and the iframe will load. However, there are some major problems in the authentication process with this method. I moved on, and continued to do things the old fashioned way with a remote development server, and set up all my application settings accordingly.

However, since Facebook simply needs to check the URL, and expect valid responses once and a while, you can edit your local hosts file and map the remote hostname to 127.0.0.1. This way, when Facebook performs its checks, the remote server responds properly, but you can still rapidly develop locally with the iframe pointing to your local development server.

Example, assuming our development server is set up at dev.somefbapp.mycompany.com, we’d configure the app with that (public) address. Then, when we want to test things more rapidly, we can enable this in our hosts file:

127.0.0.1    dev.somefbapp.mycompany.com

This seems to be working flawlessly for me as of today (July 28th, 2011) and on PHP SDK 3.2. Hopefully this brings a little light to someones day who gets as frustrated as I do with Facebook’s documentation.

<?php
 
namespace SyndExtension;
 
use SilexApplication;
use SilexExtensionInterface;
 
class ZendDb implements ExtensionInterface
{
    public function register(Application $app)
    {
        $app['autoloader']->registerPrefix('Zend_', $app['zend.class_path']);
 
        $app['db'] = $app->share(function() use ($app) {
            return Zend_Db::factory($app['db.adapter'], array(
                'host'     => $app['db.host'],
                'dbname'   => $app['db.dbname'],
                'username' => $app['db.username'],
                'password' => $app['db.password']
            ));
        });
    }
}

Create that under ./src/Synd/Extension/ZendDb.php. To use it, you’ll need to add Synd to your autoloader, and then register the extension in your application:

$app['autoloader']->registerNamespace('Synd', 'src');
 
$app->register(new SyndExtensionZendDb(), array(
    'db.adapter'      => 'mysqli',    
    'db.dbname'       => 'your_db',
    'db.host'         => 'localhost',
    'db.username'     => 'root',
    'db.password'     => '',
    'zend.class_path' => '/path/to/zf/library'
));

Please note that this is a quick and dirty bridge, and is for ZF1 not ZF2. If I’m breaking any best practices with naming and combining libraries, be sure to let me know. I’m hoping to explore the Zend_Db and Zend_Form combination since I use them heavily in my apps. This provides a very lightweight infrastructure in order to throw up quick proof of concept applications.

Hopefully this helps someone… or at least gives enough familiarity for my ZF buddies to start looking at Silex / Symfony2.