Archive for category Work

Week 127

(idea cribbed from BERG, week count done by Wolfram Alpha because I am too lazy)

"[T]he beauty of reading a page of de Selby is that it leads one inescapably to the happy conclusion that one is not, of all nincompoops, the greatest."
The Third Policeman*

Mix of a week. Started with some edits to the XML parser I wrote for Financial Institution Client as part of a project that decrypts (GPG) a large set of XML documents, skims them for the parts we're interested in, dumps the relevant bit into an Expression Engine database and then transforms all that via XSL to display a customizable dashboard (jQuery UI) on the front end. How's that for a keyword-rich blog post? Tangentially speaking, this week reminded me that of all the languages, technologies, whatever you like that I work in, XSL is probably the easiest one to make a great mess in. It looks just like XML and HTML, how hard could it be? A little bit of XPath knowledge and you're good to go. To make a complete mess. When you write an infinite loop in a normal language, it's pretty obvious: you sit there a while, the computer starts to get noisy, the lights go dim. After a few dozen times, you realize what you've done. XSL is (like) a functional programming language. And with all that recursion, it's easy to make a computer do something Big Number of times. Just harder to spot.

Also did some final-mile edits on a social network application from Slim Kiwi built on top of Pinax (and Django).  Hoping it's truly "final-mile" as the project is really cool and I'm quite proud of it. There's a ton of geolocation and mapping going on and a fair number of other bright things happening (the bright ideas being supplied by other team members and the bright implementation by Django, obviously).  I was able to roll some of the geo search and Google Maps integration right into a site for Community Trust Bank, a project from Lightfin Studios (you can see the location stuff at the branch & ATM finder). That's the second bank site I've built with Lightfin, the other being the much-closer-to-home (but harder to spell) Piscataqua Bank (built in .NET) which got some updates this week as well.

At the end of last week I started to ramp up on my second Django site with Lightfin. Not much to say about it yet except that it integrates with a third-party API which reminds me of one thing: I hate SOAP (the overly-verbose web service format, not the cleaning product so beloved by my ancestors they enslaved a leprechaun to endorse it). Please don't ever use it. I'd rather parse faxes by hand. If you're stuck dealing with SOAP in Python, Suds seems to be the best parsing package out there. I'm sure there was other stuff going on, but the only thing I can think of is some cleanup I did of an old ASP site and the less said, the better.

* Indie/ hipster required disclaimer: I am re-reading The Third Policeman, I read it well before it showed up in Lost, so cram it.


Microformat Proposal: Coding Experience

When I'm working, even in a language I know well, I often search for how to do something; either because I don't know or because I feel there's a better way (as @ed_atwell says, "I don't know, but I bet my friends Larry and Sergei do). My personal system for filtering code search results looks something like:

  1. Blogs I trust
  2. Personal blogs
  3. Development sites (e.g.,, etc.)
  4. Mailing lists and newsgroups1
  5. Forums
  6. Expert Sexchange

Regardless of where it comes from, there's no way to know if it's right. It's human nature to use the first thing that works (if under deadline, even the first thing that kinda works will do). As Jeff Atwood has pointed out (twice) , the danger is you might be copying off the paper of someone dumber than you2. Because of this, I'd like to propose a microformat (assuming one doesn't already exist, given I didn't bother to check with Larry and Sergei) to indicate an author's experience with a language.

Immediate disclaimer: I realize this is a programming solution to a human nature problem and those never work, but bear with me, because my hope isn't to fix the problem, but to provide some metadata that will let machines do the work for us so we can stay lazy. Given that is in line with Newton's First Law, this will obviously be a huge success.

The format doesn't need to be very complicated. In fact, I'd prefer if it just provided a few bits of raw data that could be remixed by search engines however they see best. The data provided would stay the same but the algorithms could be tweaked for better results (though that would require feedback), providing an incentive for search engines to consume the format. Make the data something rough, broad and quick to fill out, like years of experience with the language and a simple measure of number of lines written (e.g., none, 10, 100, 1,000, 10,000, a whole bunch). There are any number of issues with using Lines of Code (LoC) as a metric (mainly that an idiot can say in 1,000 lines what a smarter person can say in 10), but if the ranges are broad enough, it should dampen the effect.

Bolt this format onto syntax highlighting engines; this blog, for example, uses WP-Syntax to format the few, poor code samples I provide— one more panel in the plugin admin that allowed me to store a hash of [language name, years, lines of code] would allow the plugin to provide that information in any page using the languages and output a visible box on the page so inexperienced users who come to the page and see my code could know it was terrible without knowing it was terrible. Add it into the syntax formatters for popular forum software (and allow users to specify their experience) and every code argument in a forum post becomes a little easier to follow.

The format doesn't tell you if a snippet is correct, it just gives you some background information (assuming the author is honest in their self-reporting). The danger would be users trusting a snippet blindly because the author has 10 (bad) years of experience (a sort of "Appeal to authority") while better code from "newer" users goes ignored. That's a human nature problem and obviously you can't solve those with programming (/broad wink).

1. I'd rank these higher, especially official groups for languages and systems except for two reasons:

  1. They tend to be so ill-formatted and the ability to follow threads varies wildly from site to site
  2. The advice can be good but dated: it's easy to find perfectly legitimate Python answers from 2000 or so. While the answer is fine, it's possible there's a newer idiom and in a language like Python, where there's "one right way", the right way will be the way that the language has been optimized to work.

2. Basically unrelated story that I've crammed in because I always tell it because it cracks me up: in high school, we had to go to the local public high to take the SATs. The person sitting next to me scribbled furiously throughout the test and was always the first one finished (which frustrated me to no end). When we were walking out, he turned to us and said, "Dude, I just made pretty pictures with the bubbles."

Django/ Pinax: Problems With Login() in Unit Tests

This is the first in what promise to be a number of "Stupid Django Tricks" where the "stupid" is me and not Django. I was having a good deal of trouble creating unit tests for authenticated views (i.e., pages that require a user to be logged in) for the Pinax project I've been working on. I dug up two problems, one of which is on Pinax and one that's entirely on me:

  1. Pinax's file does not provide a setting for AUTHENTICATION_BACKENDS, so the test client's login method doesn't know how to log your user in. Specify "AUTHENTICATION_BACKENDS = ('django.contrib.auth.backends.ModelBackend',)" in your settings file. Actually, I lied. That's the default value for the setting; having gone back and re-run my tests without it specified, everything works, which means the only idiot here is the guy who . . .
  2. Don't create users by specifying the password directly in the declaration (e.g., user = User(username='Dummy', password='goodluck')). Use the set_password() User method to properly set the password.

I've run into a fair number of issues working in Django where Google wasn't helpful. I think 90% of those issues were because no one else was dumb enough to make such an obvious mistake. The other 10% were typos.

YUI Rich Text Editor in Django Admin

This ain't exactly rocket science, but it took me an embarrassing amount of time to get there, so I'm posting the code for next time. This will turn a given textarea in your admin area into a WYSIWYG. It's got a fairly small feature set, but that's only because I've stripped most of them out. You can add them back in by taking a look at the documentation. Per the Django docs, create an admin folder under one of your templates directories, then add subfolders for the app and model (though you can do just one if you want it to apply to all forms in the app or do neither to apply to all apps and models in your site) and add this as "change_form.html" (it took me an extra 10 minutes to get this done because I was sure it should be named "" in spite of copious amounts of documentation that said otherwise):

  2. {% extends "admin/change_form.html" %}
  4. {% block extrahead %}{{ block.super }}
  5. <!-- Skin CSS file -->
  6. <link rel="stylesheet" type="text/css" href="">
  7. <!-- Utility Dependencies -->
  8. <script type="text/javascript" src=""></script>
  9. <script type="text/javascript" src=""></script>
  10. <!-- Needed for Menus, Buttons and Overlays used in the Toolbar -->
  11. <script src=""></script>
  12. <script type="text/javascript" src=""></script>
  13. <script type="text/javascript" src=""></script>
  14. <!-- Source file for Rich Text Editor-->
  15. <script src=""></script>
  16. <script type="text/javascript">
  17. // YUI editor
  18. var editor = new YAHOO.widget.Editor("id_content", {
  19. handleSubmit: true,
  20. toolbar: {
  21. buttonType: 'advanced',
  22. buttons: [
  23. { group: 'fontstyle', label: 'Font',
  24. buttons: [
  25. { type: 'select', label: 'Arial', value: 'fontname',
  26. disabled: true,
  27. menu: [
  28. { text: 'Arial', checked: true },
  29. { text: 'Verdana' }
  30. ]
  31. },
  32. { type: 'spin', label: '10', value: 'fontsize', range: [
  33. 10, 16 ], disabled: true },
  34. { type: 'color', label: 'Font Color', value: 'forecolor', disabled: true }
  35. ]
  36. },
  37. { type: 'separator' },
  38. { group: 'textstyle', label: 'Font Style',
  39. buttons: [
  40. { type: 'push', label: 'Bold CTRL + SHIFT + B', value: 'bold' },
  41. { type: 'push', label: 'Italic CTRL + SHIFT + I', value: 'italic' },
  42. { type: 'push', label: 'Underline CTRL + SHIFT + U', value: 'underline' }
  43. ]
  44. },
  45. { type: 'separator' },
  46. { group: 'indentlist', label: 'Indenting and Lists',
  47. buttons: [
  48. { type: 'push', label: 'Indent', value: 'indent',
  49. disabled: true },
  50. { type: 'push', label: 'Outdent', value: 'outdent',
  51. disabled: true },
  52. { type: 'push', label: 'Create a Bulleted List', value:
  53. 'insertunorderedlist' },
  54. { type: 'push', label: 'Create a Numbered List', value:
  55. 'insertorderedlist' }
  56. ]
  57. },
  58. { type: 'separator' },
  59. { group: 'insertitem', label: 'Link',
  60. buttons: [
  61. { type: 'push', label: 'HTML Link CTRL + SHIFT + L',
  62. value: 'createlink', disabled: true }
  63. ]
  64. }
  65. ]
  66. }
  67. }
  68. );
  69. editor._defaultToolbar.buttonType = 'advanced';
  70. editor.render();
  71. </script>
  72. {% endblock %}
  74. {% block bodyclass %}{{ block.super }} yui-skin-sam{% endblock %}

Like I said, not rocket science. It adds some CSS & JavaScript includes (which are remotely-hosted, so you don't even have to worry about media roots or how it works locally vs. live) and then a bit to add a class to the body tag for the YUI skin.


Expression Engine if Clauses

This is the kind of thing that's not worth a blog post except some day it might save one person hours of frustration. Expression Engine apparently doesn't like it when if statements either span multiple lines or when the trailing curly brace is pushed to a new line. I can't quite run down which it is, but it's not all that important: if your if clause isn't behaving as expected, make sure it's all on one line without any extraneous whitespace.

PHP Excel Exporter

A few times a year a client needs to export something from a database table to Excel. There's a simple hack to do it in most any language. There are actually a few, but having come up as a web developer, my preferred trick is to just build an HTML table and serve it as Excel by setting the mime type header. Having done this dozens of times, I finally formalized this into a simple PHP class tonight to save myself some time and figured I might as well share it.

The bad news: because I am lazy, it relies on an old data connection class I wrote years ago when I was even less bright than I am now. The thing's so ugly I posted it somewhere else because I am too ashamed to host it here. You can rip that out and use whatever you prefer by just changing the logic in _get_table() below. If you do choose to use my old data-class.php, be aware it expects 4 constants, DB_SERVER, DB_USER, DB_USER_PASS, DB_NAME to create a connection to the database.

Here's the exporter code itself Update: I moved the code to snipplr because this WordPress plugin doesn't handle newline characters very well.

The simplest use is to instantiate an object, tell the exporter what you want to appear in the header row in the spreadsheet (by setting column_heads to an array of values) and then calling export(), passing it the SQL query that gets the data. If the number of fields in your query doesn't match the number of heads in column_heads, the resulting HTML will be a mess. You will understand if the code assumes you never make such mistakes. Here's a code example:

  2. $e = new ExcelExporter();
  3. $e->column_heads = array("First Name", "Last Name");
  4. echo $e->export("SELECT first_name, last_name FROM table");

Quick notes:

  • Control the Excel filename in my example by setting $e->filename("something-else.xls")
  • Add a timestamp to every file (useful for making sure the filename is always unique) by setting $e->timestamp_file = true
  • When you're trying to implement this and it's not working and having to say yes to the popup and let the file open in Excel is driving you crazy, set $e->debug = true and it will skip the Excel headers, sending the output to the browser

The big gotcha that works well for me but might not for you: there's a hook in the code that passes every data column through _format_field(). In my current class, this looks for any field with "_date" in the column name, assumes that field is a Unix timestamp and transforms the value into a m/d/y date. If you live in the other 99% of the world where people format their dates un-Americanly, well, you can do that like this: $e->date_format("d/m/y") or whatever other crazy date/ time format you like.

If you think that behavior stinks, rip it out. Alternatively, you can modify it or subclass this code (like "client-xyz-exporter extends ExcelExporter" for every client who lives in Excel) and change _format_field() to do whatever you want in a one-off sort of way. This is not high art, it's just a faster way of making someone happy (if you can imagine the kind of person whose life is improved by additional spreadsheets).

Windows GPG Front-End

I'm doing some work with GPG encryption and I always like to have a visual/ gui front-end to use to make sure I haven't screwed something up in my command line adventures. I came across Cryptophane today and it seems like a nice way to keep track of my particular Alice and Bob. The only problem I ran across was that my GPG install was in a non-standard place and Cryptophane doesn't look in the registry (I'm pretty sure GPG writes to it). The error wasn't immediately clear and there's no online help (though a .chm is provided), so I thought I'd post this for anyone else who runs into a similar problem. My shortcut target now looks like this:
"G:\Program Files\Cryptophane\Cryptophane.exe" --gpg-path "G:\Program Files\GNU\GnuPG\gpg.exe"

Your paths may vary, etc.

"Life's Good", Indeed

I am providing this post-mortem both as a public service and so I don't punch in the screen of my brand-new LG monitor. I picked up a new 22" monitor at Best Buy yesterday after losing a monitor to the power surges from the ice storm (what I ought to have done is replace the APC unit that's now cost me an external drive enclosure and a monitor, but that's another gripe). Almost 24 hours later, the monitor is actually running as expected. Let me preface this by saying I appreciate my setup might be a little different from what the folks at LG bothered to test on:

  • Mac hardware running Windows
  • it's a second monitor
  • NVidia and UltraMon are both fighting for control of the setup

All the same, this was a more painful hardware process than I remember going through. It's a goddamn display, not an ultrasound machine. Nothing should be more plug-and-play. And yet, when I plugged it all in with the existing VGA connection from the old monitor, none of the native resolutions were available. I installed the software and drivers, but it still couldn't figure itself out. The LG software could identify my primary monitor, but the software would not allow me to use any of the features because it could only run on an LG device. Even when it figured out there was an LG display somewhere, no dice. I'm assuming it has to be the primary display. Also, for a company that's done such a good job of becoming an international player, it's disappointing the English was as broken as the software displaying it.

I gave up on LG and updated the NVidia Control Panel, hoping newer versions would have the widescreen resolutions. Playing with the new control panel only blew things up worse: reversed the primary and the secondary and then screwed up Ultramon so badly the taskbar was displaying a box for every background process running on the box under my user account. It also saw fit to snap the resolution on both monitors to something fun. So, reboot.

My last gasp, and it included a fair bit of gasping as I've been screwing off work due to a wrenched back, was to find a DVI cable and try that. Nothing. No second monitor, no avowed knowledge of a second monitor in the control panel. For no good reason except I'm my own IT department, rebooted. Success! Sort of. Both monitors are working during bootup, both are correctly identified in the control panel, both have their proper native resolutions dialed in and programs are being popped up on the LG. Just one issue: it's power light is in orange standby mode, the screen is off and no amount of pressing the power button will convince it to turn on or off. This is when one has to say, "I can tell a convincing lie to the returns desk if need be" and show inanimate objects just who the hell runs the show around here. Out comes the power cable. The funny little capacitor manages to keep the light orange for five brave seconds before giving up the ghost. I yell, "Clear", jam the power cable back in as painfully as possible and here we are, all of us with a better understanding of the pecking order. And then I removed the startup item LG installed ("forte display manager") to keep it from doing it's infinite loop dance with NVidia for control of the display.

So, the public service announcement: if you get a new wide-screen LG monitor, get a DVI cable with it. Hook them up. Don't install the LG software (though you'll most likely need the drivers). I suppose if you just have one monitor, you'll be fine either way, but you'll only find this post if something goes wrong.


Making .NET 2.0 Spit JSON

I just finished translating a set of PHP web services into C# for a national car insurer and came across two things worth mentioning, if only for my future reference.

Each service needed an XML and a JSON version. I knew I didn't want to write code to hand-roll duplicate sets of responses, but I also didn't think it was worth knocking up a formatter class for two response formats. I created the XML responses and decided I'd transform them to JSON via XSL. Just before I got started on that, I remembered anything halfway smart I think of has already been done. Thus, xml2json.

Sinfully proud of myself, I wired up the JSON responses and pointed the pre-existing test site at my new services. And: nothing. Knowing I'm perfect, I fiddled with the site's JavaScript for an embarrassingly long time before looking at the output of my JSON services. I'd created a bunch of web services that returned a string type and the JSON looked gorgeous in the unit tests, so where was the problem? Oh yeah:

  2. <string>{json:"help, i'm trapped in here"}</string>

There's a lot I love about .NET's web services, mainly how the busywork is taken care of behind the scenes. Except there's no easy way to get back there and change the plumbing (to mix a metaphor) so it spits out strings instead of XML. .NET 3.5 has native JSON serialization, but this had to be 2.0-compatible. I found a number of possible solutions, most of which relied on the Ajax toolkit. Having worked with it, I have two objections to the toolkit:

  1. It's really heavy, so you'd better need it
  2. It feels like black magic: include the proper fake files, say the incantations just right, things may work

The toolkit was too much of an elephant gun for this ant. But the alternative was to create my own custom response handler, which felt both egotistical and like a good way to make a mess. While weighing the various options, all of them ugly, it hit me:

  2. [WebMethod]
  3. public void MyJSONService()
  4. {
  5. Context.Response.Write(myJsonString);
  6. Context.Response.Flush();
  7. Context.Response.End();
  8. }

A hack can be something that works even though you don't understand it, something you do because you're too lazy to do it correctly or something less than beautiful that solves the problem with a minimum of fuss. I'd like to think this was a #3. It's not ideal and it's not a good solution for a large system, but if you're just trying to get .NET 2.0 to send back strings without it wrapping everything in an XML safety envelope, this works without requiring two tons of library or chicken blood + a full moon.

CSS Print Tests or How to Go Crazy after a Few Hundred Revisions

I have been building tests for CSS 2.1 and CSS 3 print conformance since last October, in partnership with Revenution and HP. It's been a terrific learning experience. Previous to going out on my own, I worked at a world-class web development consultancy, where I'd developed much of our approach to CSS and browser conformance, so I felt confident in my ability to churn out these tests by the dozen. It hasn't been so.

It's easy enough to build pages in CSS and to work around browser bugs. You code, load in a browser, tweak, refresh, etc. Take the user agent out of the equation and things become difficult. Not only is it tough to check your work after you build something, but it's hard to conceive of an approach to the task when you're trying to build a test for a CSS property that doesn't exist, especially if you haven't ever run into a situation where you thought, "Know what I wish CSS had?" (Actually, those situations are even worse, because I can never remember what I was doing to get to that point, so I wind up still having to come up with a test with the bonus of a scratching feeling in the back of my noggin.)

Not that I'm totally alone: the W3C representatives are fantastic. I used to think I was thorough: when you come into a meeting with a client who wants to build an app and start getting feedback like, "We hadn't even thought of that," you're covering your bases.When you break a big application down into a 100+ use cases, you know how to get to the details. This is a level-of-magnitude difference. It makes you frustrated with the nuance and vagueness in language. Early on I would get upset when tests came back just because of the instructions, feeling it should have been obvious what I meant. Doubly frustrating because my high school freshman English teacher (think Dead Poets Society without any self-murder) had spent time teaching us how to be clear: one of our first assignments was to write the instructions for tying your shoe. He then performed everyone's assignment in the class. We went 0 for 8, but created some fantastic knots. Obviously, the word "obvious" has no place in instructions.

My other companion in the journey is Prince; not a short man from Minnesota, but an application that simulates a CSS2.1/ CSS3-compliant (if unsexy) user agent. Given I'm still building tests to show how a  CSS2.1/ CSS3-compliant user agent should work and given the spec still changes occasionally, Prince can't get everything right. Which means it's like a travel guide from a logic problem in To Mock a Mockingbird ("if you're traveling with a man who always lies and his brother who never lies . . . "). I bought that book thinking it was about how to mock objects in unit tests. When I found out it was a book of logic problems that made me feel stupid, I had flashbacks to one of our last assignments in that freshman English class: LSAT word problems. While it taught us (or at least me) a good deal about how to wring meaning from obscurity (poetry), I always felt it was less-than-coincidental our teacher left for law school after that one year.

Either way, thank you Steven Muller for making me a better test case writer. Sorry I never got around to just being the writer you were hoping for (note the dangling participle and consider the day you had us rip the entire grammar section out of our text book).