Tag: data

Categories
Uncategorized

Tweet Nest: Archive Your Tweets

I’ve had a renewed interest in my data lately, and that’s manifested itself in making sure I’ve got copies of my data. Those bits and pieces we so easily create on other web site? I want them. I want them “here”, where here is under my own control, on my own site, my own server, not just living in the cloud somewhere, at the whim of some 3rd party.

Twitter - Since Sept 2006 See, I’ve been using Twitter for a long time… since September 2006 supposedly. I’d love to go back and see what my first dozen tweets look like, but Twitter doesn’t allow that. I’ve been kicking myself for not saving all of that data since the beginning, but with micro-content you tend to think that a. It’s tiny, so it doesn’t matter, and b. It’ll always be there. We’ve learned (time and time again) that this isn’t the case.

About 2 years ago I was doing some Drupal work and set up an aggregator to ingest my Twitter feed, which managed to back up a large portion of my tweets, but not all of them. It was more experimenting with Drupal than trying to create a good backup. Still, I was slightly happier knowing I had some sort of archive. I still wanted something that would display my own content (tweets) on my own site.

I then found Tweet Nest. Developed by Andy Graulund (@graulund) it’s a simple open source PHP/MySQL application that does just what I wanted. It grabs your tweets from Twitter using the API, and stores and displays them on your own site. Perfect!

Tweet Nest : @raster The install was pretty painless, and I actually spent more time customizing the CSS (and I’m still not happy with what I came up with.) Of course two days after I installed it, it appeared to stop working. But alas, no worries, it was just a “rate limit freakout” with the Twitter API. After that, things have been smooth.

Tweet Nest also did a great job of grabbing all of my tweets as far back as October 2009. Not quite back to September 2006, but I’m starting to think that will never happen. But, I did manage to get tweets as far back as March 2008, thanks to another service called Backupify. Now I’ve got to find a good way to load the data into Tweet Nest, and I’ll have most (but not all) of my tweets stored on my own site.

You can find my archived tweets at rasterweb.net/micro, and just like my recent Delicious/Scuttle/bookmarks exploration, I now feel that much better about my data. (But I’m not stopping there. The next project may take quite a bit more hacking on my part, but it’s another one that’s long overdue.)

So go check out Tweet Nest if you want an application on your own server, or Backupify if you just want a nice, secure backup.

Categories
Uncategorized

Visitor Stats

I’m typically not one to obsess over statistics related to the visitors of this web site, but I figured I’d take a look and see what the numbers might tell me.

Stats by Browser

Firefox is the winner here, which wasn’t too surprising to me. I was surprised to see Chrome ahead of Safari though, even if by a small percentage. Of course I’m saddened that Internet Explorer even shows up at all. At least Chrome beat IE. Also, welcome SeaMonkey user!

Stats by Operating System

Too many Windows users! I’m kidding… sort of. I mean, I tend to write about things related to Macs, Linux, and open source… none of which are Windows. My assumption is that the people with corporate jobs who are forced to use Windows machines at work (but use Macs and Linux machines at home) tend to browse this site while on the job. Yeah, that makes sense. Also, welcome to both Android users, and that lone Playstation Portable user!

Stats by Browser / Operating System

Firefox / Windows users take the lead! So at least some Windows users are smart enough to install Firefox. Not surprising, as I think it’s been at least a few years since even die-hard Microsoft fans realized that Firefox is a better/safer browsing experience than Internet Explorer. As for IE being second? Those are obviously the corporate drones who do not have privs to install Firefox. At least Safari made it into the list once… Also, welcome Mozilla Compatible Agent / iPhone (what is that exactly!?) and your friend using Mozilla / Linux!

Stats by Screen Resolution

Hooray for more pixels! Is no one visiting this site on their shiny new Netbook? I spent so much time optimizing for 800×600 when I got my Eee PC.. all for naught! Those 1920×1080 screens must be all the designers I know. And that 320×396? That’d be the iPhone.

So join me in welcoming all visitors to the site… be they Chrome / Windows users, or iPad users, 1024×600 screen resolution users. Welcome!

Categories
Uncategorized

Data goes in, Data comes out.

Data Portability

When you’re building a web application (you know, like Twitter, or Flickr, or Facebook, or Last.FM, or Posterous) you obviously want people to put data into it, and you’re going to make it easy to do.

But are you making it easy for people to get their data out of it?

Here’s my simple rules for building a web application:

  1. Make it easy for users to get their data into your system.
  2. Make it easy for users to get their data out of your system.

That’s it. Do number 1, then do number 2, Do not stop in between number 1 and number 2.

If you’re convinced your application is amazing, you shouldn’t have to rely on lock-in to keep people using it. They’ll praise it openly, they’ll become your fans, they’ll shout it to their friends. Anything else is bullshit.

Categories
Uncategorized

Scuttle rides again!

Because I always fear the CloudFail™, and because I like hacking on open source projects and interesting APIs, and because I like to store my own data… I’ve got Scuttle up and running again.

p2url: Bookmarks I started using del.icio.us in 2003 to store my bookmarks. I loved it. Over time del.icio.us had some issues with reliability and there were times when I could not access my bookmarks that were stored in del.icio.us. This made me sad. Around 2005 an open source project named Scuttle appeared, which was basically a clone of del.icio.us you could run on your own server. I ran it on my own server, and put together some code to make it sync up with del.icio.us. (At some point the name changed from “del.icio.us” to “Delicious” but I am calling it del.icio.us here for historical reasons.)

For years this ran on my own home server, and served as a great backup for my bookmarks in del.icio.us. This was all very easy because Scuttle implemented (most of) the del.icio.us API, so syncing the data between the two was so trivial, even a hacker like me could cobble a bit of code together to make it work.

As an interesting sidenote, while del.icio.us ended up being pretty darn reliable (Yahoo! eventually acquired it) others were not so lucky. Ma.gnolia was another social bookmarking site. Ma.gnolia died a tragic death, and lost all data. See http://en.wikipedia.org/wiki/Gnolia#January_2009_total_data_loss for details. This is why having an API is important. This is why having a backup is important. This is why you should trust yourself with your own data.

I’ve collected a few links for the blog posts I wrote about Scuttle, del.icio.us, and related sites.

(Most of the bits above can also be found on the page at: http://p2url.com/bookmarks/about.php/, and while you are welcome to browse through http://p2url.com/bookmarks/ as it is a public site, I’m the only on who will be adding bookmarks to it, because it’s there for me. If you want your own, I encourage you to go make it happen. Let me know if you need help.

Categories
Uncategorized

The (Low Cost) Red DIT Station

If you’re on location with the RED, it’s a good idea to have a Digital Imaging Technician Station (or DIT Station) which will allow you to backup up all the footage you are shooting while you are in the field, and will also allow you to view the footage and make sure it’s all good before you leave the field.

If you’ve got an unlimited budget, I’d recommend the top of the line MacBook Pro (which can easily run $3,000-$4,000+ depending on how you deck it out) as well as a few external drives. Look at the G-Technology offerings, as they have various solutions depending on your budget and needs. The G-Raid mini is a nice, small, bus-powered drive with a quad interface (FW400, FW800, USB2, eSata.) Get a few of these. You could go with the 1TB, but the 500GB might be fine. You want your data in two places, and not “RAID two places” but “two different physical drives” places. The RED Drive holds 320GB, so if you are dumping a full RED Drive, you could just dump it to a 500GB drive (well, two drives) and mark it as “DONE” and move on. Oh, and since we have gobs of money, pick up an ExpressCard with eSata on it to put in your new MacBook Pro.

RED DIT Station
For the “dumping” of data, you can either use something like R3D Data Manager which does all sorts of checksums for the data, can backup to multiple locations at once, and has other nice features, or you can do a standard Finder copy. R3D Data Manager is $79, which is not too bad… again, what is your footage worth? If you want to fly in the Finder, and can review everything you dump, that can work too.

Speaking of software, RED has provided a number of excellent tools for free. Grab REDCINE-X, RED ALERT! and whatever else you may need at RED’s support site. REDCINE-X is a nice little app for checking the footage you just shot.

So where’s the “Low Cost” part come in? Well, let’s say you don’t have the budget for the top of the line MacBook Pro… that’s fine, get the bottom of the line MacBook Pro (at least get a Pro, you want Firewire 800!) and with that get some drives. I still really like the G-Raid minis, but you could probably find USB bus-powered drives that are cheaper (and slower!) if needed, but again, we do want as much speed as we can afford. (Bus-powered drives are nice because you never know where power will be when on location… if there is power at all.)

If you are shooting to CF Cards, do yourself a favor and get a Sandisk Extreme FireWire Reader (about $65) which will dramatically speed up transferring of footage compared to a cheap USB Card Reader. (The low end MacBook Pro has just one Firewire port, but you can plug the Firewire CF Card Reader into the the back of the G-Raid mini, which has two ports.

RED DIT

So we’ve go the MacBook Pro, which is $1,300 before adding AppleCare (but here’s a tip, you can get a refurb, plus AppleCare for it, for about $1,295) add in two G-Raid minis (the 500GB) at about $450, and a CF Card Reader for $65. We come in under $2,000 for this set up, which is not the ULTIMATE portable RED DIT Station, but it’s a Low Cost one, that is effective enough to get the job done.