Over at the BloggerCon site is an interesting post about visions for the next generation of blogging tools. Read through the comments for some neat ideas. Some of them are very doable today, while some of them just seem crazy or don’t make sense. That’s ok, they’re all just ideas, and they’re from users. Hopefully all the weblog tool makers get a look at this list, or are open to feature requests from their users.

As for me, I tend to build my own tools, and I use my own tools. So the feature requests have a much shorter path from user to developer. ;)

I recently did some work on a site that runs on a Windows server (gasp!) using IIS (yikes!) and ASP (eek!) If you know me, you know these are not my favored technologies. Nonetheless, we had a job to do.

The job was to add syndication feeds. We first added an RSS feed. Why did we choose to add an RSS feed first instead of an Atom feed? Well, not being familiar enough with ASP, we were not easily able to properly encode everything, In some cases data coming from SQL might need to be encoded, and our code didn’t always do the right thing. So we were faced with creating an RSS file that was occasionally invalid. We did not like this, but didn’t have another solution. We also knew that most of the RSS aggregators out there don’t mind invalid XML when it’s in the form of RSS. Still, while we were not pleased with the situation, we compromised.

At this point there was still not an Atom feed. We just couldn’t allow ourselves to bring an occasionally-invalid Atom feed to life, not until we discovered Server.HTMLEncode, which took care of the nasty characters that needed encoding. Once we had this final piece in place, we felt that it was OK to implement an Atom feed, and figured it would be valid (fingers were crossed of course.)

But wait, this wasn’t the final piece. There was one more, the dates and times… See, as an old Perl hacker, we’re used to using strftime to make any date/time we need. Since I couldn’t find a strftime function in ASP, I looked to the web and found A Customizable Date Formatting Routine which is close to strftime, well, close enough for my needs, but had some bugs. Anyway, we grabbed it, and it worked. More or less. (Ken Schaefer has pointed me to an update, though I still have to drop it in place.)

Where were we? Ah yes, the dates and times. Well, at first we just fudged the GMT which created valid feeds, but was lame. A little change in SQL eventually fixed that. Of course Atom requires a modified time, and it looks like it requires a created time, but if the created time is missing it should be considered the same as the modified time, and then there’s the time an item was issued as well, which is not required. I think.

Anyway, I’m pretty sure we’ve got it figured out now. The feeds both validate. Hopefully that will remain true. I think the dates and times even make sense now.

I’m starting to really like ASP… Ha! Just kidding!

On the bright side, I finally found Server.HTMLEncode, which makes creating a (more-often-that-not) valid RSS feed using ASP a little easier for me.

Who knows? Maybe I’ll attempt to use my (limited) ASP kung-foo skillz to create an Atom feed. ;)

What to do with that bad XML? Aaron suggests I beat it with a stick. A stick made out of either HTML::Parser, or XML::Simple, or perhaps Tidy. At least I think that’s what he said…

As usual, Aaron is about 25 feet above me on this stuff, so I’ll take me some time to investigate his suggestions… I suppose I could just use the Universal Feed Parser, of course it’s written in Python. I might have to make some exceptions here…

Aaron also does some crazy transformations of XHTML 1.1 to Atom, and there’s some hackery titled atom03-to-rss as well.

Speaking of atom2rss, or as it’s called, the Feed Normalizer, is an Atom translator (to RSS) that Phillip Pearson put together.

Of course it’s written in Python… No matter. I’m starting to think this is the year I actually Dive into Python… I’ll let you know next month.

As we all know, XML can be easily parsed with an XML parser. Right? Right… So what happens when XML is not really XML. Well, as we all know, when XML is not XML you resort to text and regular expressions. It’s one of the dirty secrets of XML. And hey, I’m not the only one who uses regex to parse XML. There’s also the speed/memory issue, but right now I’m just concerned with the not-really-XML part of it.

The Universal Feed Parser tries to use XML, and if that fails, does the regex dance.

If XML parsing fails due to well-formedness errors in the feed… …it will automatically fall back to the 2.x-style parser based on regular expressions.

If you’ve processed a form of XML commonly known as RSS, you might have run into these issues before, because there are feeds that are not well-formed, and therefor invalid, and if you want to be picky, they aren’t really XML… Perl needs a module that does the “try it as XML, and fall back on regex if it ain’t” module. Why? Because once again I figured I could just use something like XML::DOM to deal with an RSS file, which is supposed to be XML, but when you’ve got an & instead of an & in there, it all blows up. (Hmmm, perhaps we should go the other way around, create a pre-filter that takes in XML, fixes all the errors making it valid XML, and then passes it on to the XML parser! Could this be done?)

I guess I’ll blame the developers creating the software that creates the invalid XML/RSS. Want more secrets? I’m probably one of them. Most of the code that creates my RSS feeds, and Atom feed is a bunch of perl with home-brewed templates, and regular expressions… Why? Why don’t I use the proper tools? Laziness, lack of… whatever, it doesn’t matter. People are going to do it this way, and even though you would think RSS is simple and you could create valid markup, we don’t always do that. Sure, I’ve implemented feed checking into my system, as I don’t want to be a wonk that outputs garbage, but I still have to deal with the garbage out there, and damn is it frustrating.

To rephrase “Be liberal in what you accept, and conservative in what you send” I’d say: “Garbage in” is bad but “garbage out” is worse…

Is there hope? Well, there’s always hope, right? Will Atom save the day, doing what RSS can’t always do? It would be nice, but I’m just not sure… Should we rely on software that requires well-formed XML, and can fall back on plain old regular expressions if needed? I don’t know… I tend to think that’s a hack we shouldn’t need, but only time will tell…

See the following bit, Google spurns RSS for rising blog format, where there’s a quote from Dave:

“A good way to provide feedback to the Google people is to switch away from them,” Winer wrote on his site, citing a blogger who had suggested RSS supporters bolt from Blogger. “Let them make the connection that the day they started playing unfair, is the day the users started moving away.”

Meanwhile, Dave had this bit about one of my comments:

In a comment on the Cadenhead site, a guy named Pete says: “Just a reminder, you don’t have to use Google.” Perfect. A good way to provide feedback to the Google people is to switch away from them. Let them make the connection that the day they started playing unfair is the day the users started moving away. Companies always respond to this kind of input. It’s where users have the most power.

Just to put things in context (since I am that guy named Pete) I wasn’t specifically recommending that people move away from using Google because of this RSS/Atom controversy. I was recommending that people move away from Google when they become uncomfortable with any of their practices. Don’t sit on your backside and complain about Google, do something about it. There are alternatives, use another search engine. Do the others not have the features you like? Suggest them! Though you might be surprised by what some of the others can do. When’s the last time you used a search engine that wasn’t Google? If you have to really thing about that question, it might be time to switch. Remember, users are customers, and the customers are a big part of what made Google so successful. The whole “don’t be evil” thing is a good guideline, but I often think once a company becomes popular and grows to a certain size, evilness will creep in, it’s just inevitable, you just can’t please everyone, and the more customers you have, the larger the percent that might think you are evil.

Is Google the next Microsoft? Let’s hope not, but once again, if you want to look at a company that people continually complain about, Microsoft is it. I can’t tell you how many co-workers complain about some Windows problem they’ve had at home, and whine about Microsoft. How many times have I heard of people who use Windows all day at work, but when they get home use a Mac, or Linux because they actually want to enjoy using a computer… Sure, they’ll argue that Macs cost more, or they don’t really know how to use Linux, but again, the reality is, you have the choice of what kind of computer you use in your own home, don’t you?

Just remember this: You’ve always got a choice… You can choose what products to buy, what companies to support, what operating systems to run, and what search engines to use…

We hear a lot of talk about Linux on the desktop, and how this is the year it will really happen. Heck, I’ve heard that IBM has it’s own desktop Linux distro with 15,000 internal beta users. I’ve no doubt that Linux on the desktop will continue to improve and get more popular, as will Mac OS X, and I think both will happen at the expense of Windows. That’s just my opinion of course, and what I’m here to talk about is Linux under the desktop.

What is Linux under the desktop? It’s the practice of sticking a server under your desk to get the job done. In some companies they’ve got a lot of Windows servers, and they do all these official things like email, file/print services, DNS, etc. But when some geek type needs something done that can’t be easily done by Windows, they stick an old PC under their desk, load up a Linux distro, and installs the tools needed to get the job done.

Over time, these machines become useful, or even critical, and you need to move them into the server room along side all of the Windows boxes. And then gradually, over time, the Linux boxes outnumber the Windows boxes, and guess what? They’re more reliable, and they’re cheaper to put in place, and as long as you have people who know what they are doing, they’re easier to maintain.

That’s the plan anyway…

Oh yeah, what about Linux on the desktop? It’s coming… I hear this is the year it will really happen!

As PowerBook (and iBook) users know all to well, those damn power adapters can go bad. Well, mine finally got to the point of not working. So for the last few weeks or so I’ve been without the use of pbox, our lovely old PowerBook G3 Wallstreet.

I looked on ebay and at some of the 3rd party suppliers of power adapters, but because I’m what you might call frugal, and a hacker, I took matters into my own hands, and in my own hands I put some tools. Pliers, utility knife, wire cutters, and some duct tape. Ah, there’s always room for duct tape…

So now the power adapter works again. I did managed to lose a tiny resistor in the hackery of it all, but as the saying goes “We got power!” I mean, what could that little resistor be doing anyway? Sure, there is a chance I might get an electrical shock when checking email, or launching Firebird, I mean FireFox, might cause it to burst into flames. Oh well, such is the price you pay for attempting to keep up with the fast pace of technology on a limited budget…

Dave is meeting with the IE team.

I want to talk about how the browser can be made more useful to people who use RSS and who write weblogs. I’m going to ask for features that work for all blogging software and all aggregators, foolish me, maybe I’m the only one who thinks we all do better if everyone has a chance to compete.

Um, Dave, you do realize that the next version of IE will be a pay-to-play affair. As IE for Windows will only be included when you purchase Longhorn, and IE for Mac will only be available if you purchase MSN. So if you really want to help “all blogging software and all aggregators” and want us all to have “a chance to compete” will you also be meeting with the Mozilla folks, the Opera folks, the Safari folks?

The funny thing is, if Dave wanted to make the browser work better for these things, he might do well to look at the Mozilla project, which has a huge pile of extensions at MozDev, many of which aim to make the browser known as Mozilla a better blogging and aggregator component. If Mozilla has these components, and they are done right, won’t Microsoft sit up and take notice, and do something about it?

Dave is getting closer… He’s released a new service for people who want to be sure their feeds are in good shape.

I say getting closer because it’s a step in the right direction. It’s aimed toward people who want to make sure the feeds they have online are valid. That’s a good thing.

I went a step further. The software used to publish this site attempts to validate the feeds produced at publish time. Meaning, I create an entry, the feeds are produced, and then they are checked for validity. If I screw up and write some invalid markup, I know it right away, before the feeds ever get published.

Here’s an example: Let’s say I accidentally paste in some HTML and end up with an extraneous </a> in my text. When I render out my pages, I’d see an error like this: line 37, column 149: XML Parsing error: <unknown>:37:149: mismatched tag.

Now, here’s the interesting part. The feedvalidator doesn’t see any problems with the RSS 2.0 feed, it reports ‘No errors or warnings’ for it, but the Atom feed is the one that returns the error. Since I screwed up and created invalid markup, I’d expect an error. You can make your own call here as to whether having an unmatched </a> is screwed up or not. (Add stuff about XML, strictness, etc. if you wish.)

Oh, we also run the HTML output through Tidy to check for well-formedness, and lo and behold that stray </a> is reported by Tidy as: line 117 column 148 - Error: unexpected </a> in <p>.

Now all this does not mean that errors won’t slip through the cracks every now and then, but it makes it that much harder to let them slip through. Since we’re calling existing applications and libraries, the whole process of adding in these checks took very little time. The majority of time was spent installing the software and figuring out how it worked. There’s also less than a dozen lines of code to actually do the checks and report the errors. I know that this can’t be built into all publishing system quite so easily, but it is getting easier to do these sorts of things everyday.

I ended up getting the W3C’s MarkUp Validator installed properly on Mac OS X 10.2.8. There were just some minor ‘gotchas’ along the way. The best advice I can give is to READ! Yes, RTFM, and then read it again…

My software installs usually go like this: Attempt an install, with a basic skimming of the docs, and if it doesn’t work, start really reading the docs…

I ended up installing gettext, then libiconv again (as suggested) and then Text::Iconv (following David Wheeler’s notes) and that fixed the big problem….

As for the second problem, well, that’s my own stupid fault. See, I edited a config file using pico, and I forgot that when you copy text, and then paste it, it turns tabs into spaces. Since name/value pairs need to be separated by a tab, this broke things. That’s just one of those sanity check things so I think I get a free pass on that one.

So in summary, I’ve now got the W3C’s MarkUp Validator running locally on Mac OS X. You can just feel the exciting building!

I finally got around to installing the W3C’s MarkUp Validator locally. (Yes, more validators!) It’s now running on Red Hat Linux 7.2 inside the firewall, for use by the internal folks. I also attempted an install on Mac OS X 10.2.8 (using this guide) but I’m not quite there yet. I’ll be trying again…

Anyway, the install on Linux was not too difficult, requiring OpenSP, libiconv, and some perl modules. (Let’s put it this way, it’s much easier than a Bugzilla install.) I’m glad I finally did this, as another project I’m working on needs a validator installed locally.

Chris Gulker notes that Microsoft makes use of Linux, just like all those average Joe’s who use Google on a daily basis.

Linux is like that, people tend to benefit from it, often without even knowing it. I think that’s got to change, and 2004 might be the year that happens…

Dave Winer has a new RSS Validator. It appears to be using the Feed Validator source code (coming soon to SourceForge, but available today at feedvalidator.org) as the backend.

The odd thing is, it parses Atom 0.3 feeds and reports them as RSS. This was reported to Dave yesterday. The service appears to be in place to promote RSS as a syndication format, and I have no problem with that. For many uses RSS is great, and exactly what should be used. In other cases Atom is a good choice.

Obviously though, an Atom feed is not valid RSS, but the RSS Validator does not tell you this. Dave could have just disconnected the Atom stuff, and made it a pure RSS validator, but I think the high road would have been to issue a message like so:

You have attempted to validate an Atom feed. This system is for the validation of RSS feeds. If you wish to validate an Atom feed, please use the Feed Validator.

Now, if Dave really thinks RSS is superior to Atom, or has advantages Atom does not (and I’m sure it does) he would have no problem doing this.

On the other hand, if the desire is to promote RSS, and pretend Atom does not exist, well, I don’t have a good suggestion for that one…

Syndic8 now supports Atom. This is good, I think the Syndic8 guys will bring to light some of the issues that other Atom folks might not have dealt with in the syndicated feed world.

You know you’ve been spending too much time on web development when the alarm clock goes off and you wonder why it didn’t send the correct mime-type.

What exactly the correct mime type for an alarm clock is I’m not sure… perhaps audio/mp3 or something…

So if you wake up and immediately think about debugging your alarm clock or checking it’s error log, you might want to go back to sleep.