15 Feb

Backups, Automated and Off Site

One of the biggest issues in running a server1 is making sure if everything disappears you can be up and running as quickly as possible. So how do I do it?

Simple answer is I use a cron job that runs every day and does daily, weekly and monthly database and file system backups and then pushes those to Amazon S3. I rolled my own bash script to perform the backups and after a few months of both testing and improving it’s ready to be shown off.

The script is extremly simple:

  1. Import config settings from a file
  2. Dump MySQL Databases, gzip and move the file to your backup folder
  3. Dump PostgreSQL Databases, gzip and move the file to your backup folder
  4. Dump MongoDB Databases, gzip and move the file to your backup folder
  5. Tar and gzip the local webroot and move the file to your backup folder
  6. Delete daily backup files older than 7 days from the backup folder
  7. If Monday
    1. Copy just created database and webroot backups to be weekly backups
    2. Delete weekly backup files older than 28 days from the backup folder
  8. If First of Month
    1. Copy just created database and webroot backups to be monthly backups
    2. Delete monthly backup files older than 365 days from the backup folder
  9. Use S3 Tools to essentially rsync the backup folder with an Amazon S3 Bucket

It’s clean, quick and above all has worked without fail for several months now. The slowest part of the process is uploading the files to S3 which has never taken that terribly long. It’s also repeating the mantra from my earlier post of “tar it then sync”.

This method is simple and it seems to work great for most single server setups. I haven’t optimized the database dumps, mainly because that is highly dependent upon your particular use of each. If you have multiple servers or separate database and web servers, why are you taking sys admin advice from me?

It’s available on GitHub: S3_Backup


  1. I use a virtual host from Linode for this site and a few others, they are great. 

11 Jan

Gigantt Blog – The GitHub Job Interview

That’s why I’m advocating the GitHub job interview. Open-Source projects are a fantastic way to collaborate with people you don’t know too well. And GitHub in particular, with its ease of forking and pull-requests is just the best (and biggest) platform for open-source collaboration.

Here’s what you do. You come up with a cool idea of an open-source project. This becomes your company’s development sandbox. Candidates are asked to then contribute to the project in some way. You want to see them code? Ask them to develop a module. You want to see them tackle a bug? Ask them to choose one from the bug-list. This works for every aspect of development work. You can design features together. You can gauge their communication skills. You can see how well they handle reviews. You can ask them to document their work and see how well they can write. But above all, you’re not taking advantage of anyone, and true developers probably won’t mind investing time into an open-source effort. 

Choose your GitHub project wisely. It should be something relatively fun. It ought to use the same technology stack your company uses. And it should be relatively simple to grasp, because the point is not to be investing too much time training people you’re not yet hiring.

via Gigantt Blog – The GitHub Job Interview. This sounds like a really solid way to do a job interview.

29 Sep

Zach Holman – Scaling GitHub’s Employees

We do things differently at GitHub: we work out of chat rooms, we don’t enforce hours, and we have zero managers. People work on what they want to work on. Product development is driven by whoever wants to drive product.

I’m GitHub employee number nine, and although I wasn’t there at the beginning, I’ve been hearing and reading the same things since even before I was hired a year and a half ago: GitHub really has a great work environment, but it’s not going to scale as they grow. The common sentiment was once GitHub grew past five employees, they’ll have to start changing their strategy.

Once we made it to five employees, we heard the same thing about ten employees. Once we made it to ten employees, then they talked about twenty. Then thirty. Today we’re at forty employees, and, if anything, we’re even happier with our way of working than ever before.

via Zach Holman – Scaling GitHub’s Employees. Pretty cool read, the focus is on automating tasks so new employees don’t need to know a lot to get started and working, having teams that focus primarily on one single part of GitHub and reducing complexity.

28 Jun

GitHub – Linguist

From time to time we get requests asking us to add support for new highlighting lexers, recognize additional extensions as certain languages, or ignore a directory from a repo’s stats graph.

The code for these concerns was scattered around the app. I decided to unify and package them all up into a single library. Now it’s open source.

So if you notice an unrecognized extension or you’re really into some obscure language that isn’t supported yet, now is your chance to help contribute back.

via GitHub – Linguist. Awesome job GitHub, nice to let the community help build out detection for specific languages and frameworks.

22 Jun

GitHub – Announcing GitHub for Mac

Pull requests, merge button, fork queue, issues, pages, wiki –– all awesome features that make sharing easier. But those things are only great after you’ve pushed your code to GitHub.

Today we’re happy to announce GitHub for Mac.

via GitHub – Announcing GitHub for Mac. It’s very nice looking and does a fair bit of abstraction towards the intricacy of dealing with Git on a daily basis. Should be an awesome tool for people who need to use a version control system but aren’t sure how to use Git. GitHub for Mac also uses the wonderful open source project Chamelon, which lets you build an app that targets the Mac and iOS devices with the same code base.

22 Feb

Andrew Vos’s Blog – Amount of profanity in git commit messages per programming language

Last weekend I really needed to write some code. Any code. I ended up ripping just under a million commit message from GitHub.

The plan was to find out how much profanity I could find in commit messages, and then show the stats by language. These are my findings:

via Andrew Vos’s Blog – Amount of profanity in git commit messages per programming language. How both fascinating and funny.

18 Oct

Trying Out Some New Technologies: WordPress Child Themes and GitHub

I recently moved this site over to a new host (MediaTemple in this case) and along with that I decided to start with a new theme and try out some new (for me) technologies along the way.

The first, is WordPress Child Themes. WordPress Child Themes basically enable you to extend a theme to your own liking, while allowing the parent theme to be updated along the way. That’s bad way of saying; you can make changes to the theme without editing or worrying about the parent theme. The old theme was a customized version of Viligance and I ran into the problem of Viligance was being updated and I wanted to apply the updates however I couldn’t because I had customized the theme so any updates I applied would break all the changes and tweaks that had been added in.

Child themes are WordPress’s answer to this sort of problem and I’ve already found them imnessely useful. Erudite didn’t support favicons, Bit.ly short urls, OpenID Delegate Server, etc. Now it supports all of those and more in the future. Most of that probably didn’t mean something to you but the basic idea is that you can add custom style sheets, add custom templates, interject code where ever a WordPress plugin can and a lot more. If you are interested in WordPress Child themes, two places to check out: ThemeShaper – How To Modify WordPress Themes The Smart Way and ThemeShaper – Sample Theme Options. The first is a good guide on building a basic child theme, the second walks you through adding an options page to your theme.

So, that was the first new technology, the other is GitHub. GitHub is built around two ideas, Git is an awesome tool for programmers and coding is a social experience. Both of these differ from most of my experience with programming. I’m used to SVN and have used it almost exclusively over the years. Programming as well even while working on a team was built typically around working one person at a time on a particular task or area of the project. Git and GitHub are designed to change both of those.

Unfortunately GitHub makes it so easy that I’ve found myself becoming lazy. It feels a lot harder to contribute to non-GitHub projects because it often requires signing up for their custom bug tracker, learning the patch process, and waiting longer before the patch is accepted. That extra friction is sometimes enough to prevent me from submitting a fix, and that’s not good for the project.

Ease of contribution is clearly an important factor for open source and other community-driven projects (just look at Wikipedia). As GitHub continues to grow, are more projects going to feel pressure to switch? I think they will, and I’m looking forward to it. Better software is good for everyone.

via HipChat Blog – GitHub is making me lazy but I like it. So I’m going with the flow at first I had the code for my child theme posted on a public SVN repository but I’m going to make it even easier for people to play with and see what I’m doing. It’s now on GitHub: http://github.com/jtyost2/Erudite-Child-Theme.

Let the hardcore forking action commence.