21 Feb

Mailinator(tm) Blog – How Mailinator compresses email by 90%

Given the title of this article, the first thing that should pop into your mind is probably – “well, use a compression algorithm – right?”.

Right! Well, yes, well, not exactly. Read on.

via Mailinator(tm) Blog – How Mailinator compresses email by 90%. A fun journey through algorithms to find a solution to getting some awesome compression stats.

15 Feb

Backups, Automated and Off Site

One of the biggest issues in running a server1 is making sure if everything disappears you can be up and running as quickly as possible. So how do I do it?

Simple answer is I use a cron job that runs every day and does daily, weekly and monthly database and file system backups and then pushes those to Amazon S3. I rolled my own bash script to perform the backups and after a few months of both testing and improving it’s ready to be shown off.

The script is extremly simple:

  1. Import config settings from a file
  2. Dump MySQL Databases, gzip and move the file to your backup folder
  3. Dump PostgreSQL Databases, gzip and move the file to your backup folder
  4. Dump MongoDB Databases, gzip and move the file to your backup folder
  5. Tar and gzip the local webroot and move the file to your backup folder
  6. Delete daily backup files older than 7 days from the backup folder
  7. If Monday
    1. Copy just created database and webroot backups to be weekly backups
    2. Delete weekly backup files older than 28 days from the backup folder
  8. If First of Month
    1. Copy just created database and webroot backups to be monthly backups
    2. Delete monthly backup files older than 365 days from the backup folder
  9. Use S3 Tools to essentially rsync the backup folder with an Amazon S3 Bucket

It’s clean, quick and above all has worked without fail for several months now. The slowest part of the process is uploading the files to S3 which has never taken that terribly long. It’s also repeating the mantra from my earlier post of “tar it then sync”.

This method is simple and it seems to work great for most single server setups. I haven’t optimized the database dumps, mainly because that is highly dependent upon your particular use of each. If you have multiple servers or separate database and web servers, why are you taking sys admin advice from me?

It’s available on GitHub: S3_Backup


  1. I use a virtual host from Linode for this site and a few others, they are great. 

15 Feb

Incubaid Research – Rediscovering the RSync Algorithm

Don’t walk the folder and ‘rsync’ each file you encounter. A small calculation will show you how bad it really is.

Suppose you have 20000 files, each 1KB. Suppose 1 rsync costs you about 0.1s (reading the file, sending over the signature, building the stream of updates, applying them). This costs you about 2000s or more than half an hour.

System administrators know better:they would not hesitate: “tar the tree, sync the tars, and untar the synced tar”.

Suppose each of the actions takes 5s (overestimating) you’re still synced in 15s.

via Incubaid Research – Rediscovering the RSync Algorithm. The right way to synch two remote file systems.

26 Jan

Apple Outsider – Hollywood Still Hates You

Hollywood continues to completely ignore that lesson. It continues to punish the people who play by the rules with an insufferable customer experience. This is the sole reason piracy is up and profits are down: because doing it right totally sucks. And that’s apparently how the studios want it.

via Apple Outsider – Hollywood Still Hates You. It bears repeating, the vast majority of piracy is people just trying to get content the easiest way.

26 Jan

Michael Tsai – PDFpen and iCloud

It’s no longer possible to write a single app that takes advantage of the full range of Mac OS X features. Some APIs only work inside the Mac App Store. Others only work outside it. Presumably, this gap will widen as more new features are App Store–exclusive, while sandboxing places greater restrictions on what App Store apps are allowed to do.

via Michael Tsai – PDFpen and iCloud. My largest long-term fear of OSX is that Apple will slowly turn off the ability for applications to be useful without using the App Store and thus some Apps may just not exist anymore (SuperDuper is the easy example).

24 Jan

inessential.com – Fantastical and language detection

I like this. The best Mac developers have been famous for taking the extra steps. Most people won’t need this — but those who do it will delight.

via inessential.com – Fantastical and language detection. That is practically the definition of great software, causing your users delight in the everyday workings.

06 Jan

Mike’s Lookout – SPDY of the Future Might Blow Your Mind Today

Despite its coolness, there is an aspect of SPDY that doesn’t get much press yet (because nobody is doing it). Kudos for Amazon’s Kindle Fire for inspiring me to write about it. I spent a fair amount of time running network traces of the Kindle Fire, and I honestly don’t know quite what they’re doing yet. I hope to learn more about it soon. But based on what I’ve seen so far, it’s clear to me that they’re taking SPDY far beyond where Chrome or Firefox can.

The big drawback of the previous picture of SPDY is that it requires sites to individually switch to SPDY. This is advantageous from a migration point of view, but it means it will take a long time to roll out everywhere. But, if you’re willing to use a SPDY gateway for all of your traffic, a new door opens. Could mobile operators and carriers do this today? You bet!

Check out the next picture of a SPDY browser with a SPDY gateway. Because SPDY can multiplex many connections, the browser can now put literally EVERY request onto a single SPDY connection. Now, any time the browser needs to fetch a request, it can send the request right away, without needing to do a DNS lookup, or a TCP handshake, or even an SSL handshake. On top of that, every request is secure, not just those that go to SSL sites.

via Mike’s Lookout – SPDY of the Future Might Blow Your Mind Today. The pictures give a really good sense of what is going on.

28 Dec

The Kernel – The golden age of the developer

There’s never been a better time to be a developer. Thanks to an unprecedented range of open-source software, learning resources and useful web services at our disposal, we can learn new languages, get help, collaborate with others and, if our ideas win traction, there’s now a multitude of investors waiting in the wings to help us build companies around our products.

This is not to say that our work is easy. Standards must remain high. But the resources available offer us the opportunity to move faster and make even more progress. The nature of innovation means that many of our ideas will not succeed, making determination vital for seeing ideas through. But the opportunity is here, my friends. We are the kingmakers.

The good news is that this golden age has made you the developer you are and will continue to help you. The even better news is that you have the chance now to, in the slightly emetic language of the Valley, “pay it forward”.

via The Kernel – The golden age of the developer. I’m trying to do more of this moving forward even in little bits and pieces I think help me stand out as a developer and it’s good to help the community which provides the basis of so much of my daily income (CakePHP, jQuery, etc).

16 Dec

The Year of C.E.O. Failures Explained – NYTimes.com

Last spring, I taught a class at the Columbia Business School called “What Makes a Hit a Hit—and a Flop a Flop.” I focused on consumer-tech success stories and disasters.

I distinctly remember the day I focused on products that were rushed to market when they were full of bugs — and the company knew it (can you say “BlackBerry Storm?”). I sagely told my class full of twentysomethings that I was proud to talk to them now, when they were young and impressionable — that I hoped I could instill some sense of Doing What’s Right before they became corrupted by the corporate world.

But it was too late.

To my astonishment, hands shot up all over the room. These budding chief executives wound up telling me, politely, that I was wrong. That there’s a solid business case for shipping half-finished software. “You get the revenue flowing,” one young lady told me. “You don’t want to let your investors down, right? You can always fix the software later.”

You can always fix the software later. Wow.

That’s right. Use your customers as beta testers. Don’t worry about burning them. Don’t worry about souring them on your company name forever. There will always be more customers where those came from, right?

That “ignore the customer” approach hasn’t worked out so well for Hewlett-Packard, Netflix and Cisco. All three suffered enormous public black eyes. All three looked like they had no idea what they were doing.

Maybe all of those M.B.A.’s pouring into the workplace know something we don’t. Maybe there’s actually a shrewd master plan that the common folk can’t even fathom.

But maybe, too, there’s a solid business case to be made for factoring public reaction and the customer’s interest into big business decisions. And maybe, just maybe, that idea will become other C.E.O.s’ 2011 New Year’s resolution.

via NYTimes.com – The Year of C.E.O. Failures Explained. I’m not certain if business school teach that only thing matters is the profit you can make or if it is the result of something else. However, business schools seem to create an environment that rewards not making happy customers, not doing the ethical thing, not doing the thing that protects the environment down the road. One of the ways in which Apple succeeds is by releasing products when they are fully finished and not half-baked.

06 Dec

Ars Technica – Google Earth, other mobile apps leave door open for scripting attacks

In the rush to create mobile apps that work across the leading smartphones and tablets, many developers have leaned heavily on web development tools and use embedded browsers as part of their packaged applications. But security researchers have shown that relying on browser technology in mobile apps—and even some desktop apps—can result in hidden vulnerabilities in those applications that can give an attacker access to local data and device features through cross-site scripting.

via Ars Technica – Google Earth, other mobile apps leave door open for scripting attacks. Oops, just because it doesn’t look like a browser doesn’t mean it doesn’t suffer the same security holes.