Week 132

"Well, the Devil made me do it the first time/ Second time I done it on my own."
Billy Joe Shaver, "Black Rose"

Never got around to the last couple of weekly updates. There wasn't much of interest, technology-wise, and they burned the hell out of me. It's twice as fun to be burnt-out busy when they paychecks are weeks or months away. Every 4-6 months I go through a transition period where one or two big projects is winding up and new work is starting to trickle in (including a Django project for Frontline). The trick is dealing with the maintenance work and one-off edits for clients during this period. I'm really struggling with the continuous stream of interruptions. When I started on my own I was mad to reply to everything as fast as humanly possible and get the work done in much the same time. Not a great idea. While I've gotten better about resource planning, I still can't shake the feeling I need to reply rightawayohquicknow or lose a client. I know none of my clients stick with me because I can turn around an email quickly. Anyone could do that. I know it's for the quality of my work and I know I have credit in the bank with (most of) them, but I still drive myself nuts.

Email's the problem right now. It's always the first tab in my browser and then I have a Gmail plugin/ setting that updates the tab with an inbox count, in the interests of further self-abuse. When I worked for someone else and email became a problem, I'd just shut it off. Someone wanted me, they could call. And if that got to be a problem, I set the phone to make-busy. Not the thing that sends 'em straight to voicemail (I probably checked my voicemail less than 20 times in 8 years-- never occurred to me), the thing that dumped the call. Walk over if things are that bad. But now that I'm my own boss, I can't do it. I need to find some way to shut off email without obsessing over it. Still working on that. It's not an original observation (I think it's the theme song of the freelancer), but I want to make more days lie Saturday. Michelle works on Saturday morning, so my work week is from late morning Monday (theoretically— it's been more like early morning Monday recently) through 1pm on Saturday. And it feels like 50% of the work I do gets done in the quiet of Saturday when no one expects a response out of me (and don't start emailing me now). All I need is two of me: one to do the communicating and one of me to do the work and keep them away from each other before they go out drinking and get in a fight.

On a different subject, it's always interesting to be on the other side of the fence like I am right now on a project for Big Financial Institution. We've done a lot of work to paper over some of the cracks in their data team's process, but we're running into Newton's Fourth Law: "For every effort you make to hide the incompetence of someone you depend on, they will redouble their effort." This is a Big Company. I'm not mentioning it to impress anyone. I'm mentioning it because I can't imagine working for their data team and sleeping more than an hour a night. We've been (theoretically, mind you) getting the same thing delivered quarterly for almost a year now. Counting all the test runs, we've probably had a dozen data drops from them. And it's painfully clear this stuff is not delivered by script. It's 12 XML files with the same data fields every time for the same funds. Is it scripted? Hell no. Best I can tell, there's a guy with a button somewhere. And best I can tell, he was dropped on his head as a child. Maybe that's what makes me decent at my job, because I always hear Jim Clancy in my head: "There's a difference between a job done and a job done right." At some point, stay late. Whether they pay you for it or try to kick you out for being there late, stay and figure out all the questions in the process, find the correct answers one time and then make a damn computer do it. Because then you'll never have to look up the answers again, which means you won't have a chance to mess it up. And if you tell me there are too many moving parts to script the process, that just means you don't understand the process. Break it down into a bunch of steps, write some God-awful-looking code that works and be done with it. No one's ever going to see it, they're just going to be happy with you. And if they fire you because you made yourself redundant, good. I guess some people can work at a job where everyone agrees not to work too hard, but that's a Devil's Bargain: you grow fat and lazy and dull and then the only jobs you can work are ones just like it. And they don't hire much. Because people don't leave and because the company isn't growing.

It pains me some people are (theoretically) in the same industry as me but have no curiosity about it. True anywhere, I just happen to see it here. Which is why when people ask what I do, I tell them I'm a valet at the drive-in.

Tags:

Week 129

"There's no sense in a man picking out the worst name he can find for everything."
Dashiell Hammett, Red Harvest

On Wednesday, I attended the NH UPA presentation on focus groups. I spent a lot of the time wondering about the effect of group dynamics on end products. The immediately striking moment came no more than a half hour after the presenter said to always watch for the person who chooses to sit opposite you as they intend to challenge everything, a volunteer sat right down and played his role. Reminded me of a zillion client meetings with one of my old co-workers who had a gift for making an entire room uncomfortable before he ever opened his mouth. It was obvious from his body language he was going to argue with anything if he could be arsed to pay attention.

How does this drive design and development when the answers you get are colored by the dynamics of the group? While it works for consumer products with wide appeal, does it work for software? Does it work for consumer  products with wide appeal? Whether a focus group improves or hurts the end result compared to what a lone tinkerer would have built in their garage, it's hard to imagine the final product isn't watered-down. Large groups of stakeholders are used to define software because you can't trust developers (who are almost as literal-minded as the computers they work with), but that doesn't make gathering marketing folks, VPs, and whomever had free time around lunch on Thursday the best way to develop an application. I just can't shake the feeling so many projects get derailed by someone trying to find something to say in a meeting. It was an adjustment from high school to college, going from a place where teachers were more than ready to tell you you were wrong to tenured college professors who'd let any fool capable of raising a paw offer an answer, no matter how wrong. "Ok, run with that." Please don't.  I never saw evidence that if you let someone talk long enough they'll get around to the right answer, and that's when the problem had a definite right answer. Let someone talk long enough and they wind up talking about themselves. No matter what promises they walked in the door with, how they were going to be the voice of the front-line customer service techs, after a couple of minutes of rambling the answers are what they think.[1]

I was contacted by a company I work with about bailing out a .NET project gone South. It's written in .NET4 and the fact I never got around to installing VisualStudio 2008, much less 2010, tells you when my Microsoft Empower for ISVs subscription ran out: last week. Plus you only get two years in the program, so now I get to sign up for MAPS, which requires you to take a 10 question quiz to prove . . . you can use a search engine, I guess. The Microsoft Partner site is built like they're trying to keep you away. It opened 4 or 5 windows (IE-only, of course) all over the screen (somehow managing to throw them across two monitors-- by looking for the bottom left and right corners of my screen, I guess), two of which sat and watched me, apparently. It finally opened the test window, only to have it throw a 500 error and not even show a friendly error page suggesting anyone gave a shit. Even after I completed the test today, the link to purchase a license was broken, asking for a network login on the server. I had to discover some other click path to find a way in to the store. Maybe that's how they really test partner candidates.

I added django-axes to the resident management project I'm working on. Axes tracks failed login attempts; normally you'd use it for making sure no one's trying to break into your site. In this case, I'm more interested in seeing who isn't able to login since we're physically mailing usernames and passwords out to users and we will not have email addresses for the users until after they've signed in and updated their accounts. Even though I needed to customize it a bit to fit our needs (mainly because I'm not using the generic Django login view for reasons which might not have the greatest of justification), it was easy to fit in and customize. Given how little work it is to add, I'm making it part of my standard Django deployments going forward as it provides a level of insight into hack attempts and protection against brute force attacks that any project should have.

An "Ah, duh" moment yesterday: added a line in the deployment script on my current project to run syncdb on each update. Not exactly rocket science, not exactly the best idea ever, but it ensures the deployment server at least has all the database tables. Sounds obvious, but it's a testament to Django that I rip through small tasks fast enough I close the ticket as done without ever thinking, "Hey, I just added x new tables to the database, better make sure the server has them before anyone tries to QA."

1. If you don't recognize this tune from this graf, it's "The Snob's Waltz", an aria sung only by the rarest of divas. The song is notable for having no conclusion.

Tags:

Week 128

Not much changed from last week, mainly working to push the social networking site out the door (while working on a proposal for another Django/ Pinax project). This week did mark the first time I actually got a Fabric script working (not that it's hard, just that my heart hadn't been in it before now). The script isn't amazing, it just connects to the server, grabs the most recent source code and runs a couple of commands to restart the server, but it does it all without me having to leave the local command line and none of it happens if the unit tests don't pass. The unit test bit is handy because other team members are willing to make the trade-off of sitting through the tests for the convenience of the automated deployment. I like Fabric enough already that I'm searching around for other problems I can solve with it.

Had an unhappy start to the end of the week when a live Django  site started barfing this morning. It wouldn't load any pages and kept complaining about too many MySQL connections, even when requesting static files. Waiting to hear back from the host to see if they have any insight, but I made some changes that seemed to solve the problem. From the logs, it looked like the site was being indexed by one or more search engines that were still looking for the old URLs. Our 404 page was dynamic and ran some queries to render the page. I changed it to a purely static template and the load cooled down. With some rewrite rules for the old URLs and some additional page caching, things should be ok.

Tags:

Week 127

(idea cribbed from BERG, week count done by Wolfram Alpha because I am too lazy)

"[T]he beauty of reading a page of de Selby is that it leads one inescapably to the happy conclusion that one is not, of all nincompoops, the greatest."
The Third Policeman*

Mix of a week. Started with some edits to the XML parser I wrote for Financial Institution Client as part of a project that decrypts (GPG) a large set of XML documents, skims them for the parts we're interested in, dumps the relevant bit into an Expression Engine database and then transforms all that via XSL to display a customizable dashboard (jQuery UI) on the front end. How's that for a keyword-rich blog post? Tangentially speaking, this week reminded me that of all the languages, technologies, whatever you like that I work in, XSL is probably the easiest one to make a great mess in. It looks just like XML and HTML, how hard could it be? A little bit of XPath knowledge and you're good to go. To make a complete mess. When you write an infinite loop in a normal language, it's pretty obvious: you sit there a while, the computer starts to get noisy, the lights go dim. After a few dozen times, you realize what you've done. XSL is (like) a functional programming language. And with all that recursion, it's easy to make a computer do something Big Number of times. Just harder to spot.

Also did some final-mile edits on a social network application from Slim Kiwi built on top of Pinax (and Django).  Hoping it's truly "final-mile" as the project is really cool and I'm quite proud of it. There's a ton of geolocation and mapping going on and a fair number of other bright things happening (the bright ideas being supplied by other team members and the bright implementation by Django, obviously).  I was able to roll some of the geo search and Google Maps integration right into a site for Community Trust Bank, a project from Lightfin Studios (you can see the location stuff at the branch & ATM finder). That's the second bank site I've built with Lightfin, the other being the much-closer-to-home (but harder to spell) Piscataqua Bank (built in .NET) which got some updates this week as well.

At the end of last week I started to ramp up on my second Django site with Lightfin. Not much to say about it yet except that it integrates with a third-party API which reminds me of one thing: I hate SOAP (the overly-verbose web service format, not the cleaning product so beloved by my ancestors they enslaved a leprechaun to endorse it). Please don't ever use it. I'd rather parse faxes by hand. If you're stuck dealing with SOAP in Python, Suds seems to be the best parsing package out there. I'm sure there was other stuff going on, but the only thing I can think of is some cleanup I did of an old ASP site and the less said, the better.


* Indie/ hipster required disclaimer: I am re-reading The Third Policeman, I read it well before it showed up in Lost, so cram it.

Tags:

On Optimization

It's strange what you can get used to: the current social network site I'm working on has a page with 216 database queries on it. Used to be I'd get the hives if I hit a dozen queries on a page.

"216! Did you know databases let you bring back more than one row at a time nowadays?"

Yes. The project is in Django (and built on top of Pinax), so it's the ORM making all those queries, not me. It's one of those social network site pages that aggregates activity from everyone you follow. It also shows details about them, how far they are away from you and any comments on the item, so there's only so small I can make it while coloring inside the lines of the mapping system. I've already fallen back to raw SQL for one of the elements (there are a couple of places, and sure to be more in the future, where we return a list of the database ids of all your friends so we can use them as part of " AND id in (x, y, z)" queries. Doing that through Django resulted in one query to the database for every friend you have. Given this was causing a slowdown when I'm the only user of the site and I only have 3 friends (one is another tester and the other two are dogs I know, so it's kind of a "Bob" situation (specifically the dog part and not the rest)), I had a suspicion that wasn't going to scale. Modified that, added some caching, got smarter about some lookups (I thought I'd only hit the db once no matter how many times I referred to a model's property in a function) and things are back to running smoothly.

"216!"

Hey, it was 1066 when I started a day ago. Or something close to that. I've got 1066 on the brain because I've been thinking about William of Orange and before you say--

"Write code for a job and think about William of Orange in your spare time. You must be a hit with the ladies."

--that, let me point out it was in reference to a Dutch Oven joke. That has to count for something.

"Undoubtedly. Perhaps 'lady killer' is more literal than figurative in your case."

Regardless, given the nature of the screen, aggregating a dozen types of activities from an arbitrary number of users, I don't think the current solution is the long-term answer, so I buttoned it up as best I could.

"As best you could? Implement the long-term solution now."

That would be solving a problem I don't have (c.f., "premature optimization", "YAGNI"). Given the data for this screen is derived from other objects in the system anyway, I think the long-term solution is to move this data into a nosql store (here's an example of using CouchDB in Django now and future updates to Django should improve support for this kind of thing). It's important to remember traffic issues fall under the title Good Problems to Have. While I'd love to spend a couple of days implementing this rightnowyespleasecani, if the overall project never takes off, it would be unfair to ask the client to pay for something they didn't ask for and never needed.

"216!"

I'm already obsessing over it on my own. Why do you think you're here?

Tags:

Microformat Proposal: Coding Experience

When I'm working, even in a language I know well, I often search for how to do something; either because I don't know or because I feel there's a better way (as @ed_atwell says, "I don't know, but I bet my friends Larry and Sergei do). My personal system for filtering code search results looks something like:

  1. Blogs I trust
  2. Personal blogs
  3. Development sites (e.g., 4guysfromrolla.com, etc.)
  4. Mailing lists and newsgroups1
  5. Forums
  6. Expert Sexchange

Regardless of where it comes from, there's no way to know if it's right. It's human nature to use the first thing that works (if under deadline, even the first thing that kinda works will do). As Jeff Atwood has pointed out (twice) , the danger is you might be copying off the paper of someone dumber than you2. Because of this, I'd like to propose a microformat (assuming one doesn't already exist, given I didn't bother to check with Larry and Sergei) to indicate an author's experience with a language.

Immediate disclaimer: I realize this is a programming solution to a human nature problem and those never work, but bear with me, because my hope isn't to fix the problem, but to provide some metadata that will let machines do the work for us so we can stay lazy. Given that is in line with Newton's First Law, this will obviously be a huge success.

The format doesn't need to be very complicated. In fact, I'd prefer if it just provided a few bits of raw data that could be remixed by search engines however they see best. The data provided would stay the same but the algorithms could be tweaked for better results (though that would require feedback), providing an incentive for search engines to consume the format. Make the data something rough, broad and quick to fill out, like years of experience with the language and a simple measure of number of lines written (e.g., none, 10, 100, 1,000, 10,000, a whole bunch). There are any number of issues with using Lines of Code (LoC) as a metric (mainly that an idiot can say in 1,000 lines what a smarter person can say in 10), but if the ranges are broad enough, it should dampen the effect.

Bolt this format onto syntax highlighting engines; this blog, for example, uses WP-Syntax to format the few, poor code samples I provide— one more panel in the plugin admin that allowed me to store a hash of [language name, years, lines of code] would allow the plugin to provide that information in any page using the languages and output a visible box on the page so inexperienced users who come to the page and see my code could know it was terrible without knowing it was terrible. Add it into the syntax formatters for popular forum software (and allow users to specify their experience) and every code argument in a forum post becomes a little easier to follow.

The format doesn't tell you if a snippet is correct, it just gives you some background information (assuming the author is honest in their self-reporting). The danger would be users trusting a snippet blindly because the author has 10 (bad) years of experience (a sort of "Appeal to authority") while better code from "newer" users goes ignored. That's a human nature problem and obviously you can't solve those with programming (/broad wink).

1. I'd rank these higher, especially official groups for languages and systems except for two reasons:

  1. They tend to be so ill-formatted and the ability to follow threads varies wildly from site to site
  2. The advice can be good but dated: it's easy to find perfectly legitimate Python answers from 2000 or so. While the answer is fine, it's possible there's a newer idiom and in a language like Python, where there's "one right way", the right way will be the way that the language has been optimized to work.

2. Basically unrelated story that I've crammed in because I always tell it because it cracks me up: in high school, we had to go to the local public high to take the SATs. The person sitting next to me scribbled furiously throughout the test and was always the first one finished (which frustrated me to no end). When we were walking out, he turned to us and said, "Dude, I just made pretty pictures with the bubbles."

Cheating at The Beatles: Rock Band

My favorite thing about the game is the harmonizing and the way it increases the feeling that you're really in a band, but if you're all about the score (or bereft of friends), feel free to take advantage of these two pieces of information:

  1. The different singers do not have to sing different parts
  2. The multiple scores are based solely on having multiple microphones

In this case, 1 + 2 equals, "If you stick three microphones in front of your face and start wailing, you'll be credited as three singers, including the double and triple bonuses". Kids, you're only cheating yourselves.

Django/ Pinax: Problems With Login() in Unit Tests

This is the first in what promise to be a number of "Stupid Django Tricks" where the "stupid" is me and not Django. I was having a good deal of trouble creating unit tests for authenticated views (i.e., pages that require a user to be logged in) for the Pinax project I've been working on. I dug up two problems, one of which is on Pinax and one that's entirely on me:

  1. Pinax's settings.py file does not provide a setting for AUTHENTICATION_BACKENDS, so the test client's login method doesn't know how to log your user in. Specify "AUTHENTICATION_BACKENDS = ('django.contrib.auth.backends.ModelBackend',)" in your settings file. Actually, I lied. That's the default value for the setting; having gone back and re-run my tests without it specified, everything works, which means the only idiot here is the guy who . . .
  2. Don't create users by specifying the password directly in the declaration (e.g., user = User(username='Dummy', password='goodluck')). Use the set_password() User method to properly set the password.

I've run into a fair number of issues working in Django where Google wasn't helpful. I think 90% of those issues were because no one else was dumb enough to make such an obvious mistake. The other 10% were typos.

Monty Back, Rommel Still Dead

Bob Montgomery is doing the color for today's Red Sox game and I can't figure out how to feel about it. Monty and the late Ned Martin were the voice of the Red Sox (on WSBK TV38) when I was growing up and it's strangely transporting to hear him again. He's done some Pawsox games, but it's hearing him back, like Jake Taylor (a fellow catcher) getting one last chance with the parent club. And it's like he's never left: same dulcet tones, knows the team, not a sign of age (unless you get a look at the tombstone of a gut he's developed in retirement).

It wasn't until about year 3 of the Don Orsillo Experience that I realized I'd seriously undervalued Sean McDonough. Orsillo is fine, but he's a generic Connecticut School of Broadcasting voice. Close your eyes and he could be talking about the Kansas City Royals. Sean McDonough's only sin for me (beyond the too-goofy adulation of Remy) was not being Ned Martin. Hearing Monty makes me feel like I'm ten, I've got a whole summer in front of me and there's nothing to worry about for the foreseeable future*. And that shit will get you killed.

* Of course, there were no World Series wins back then either

Free to a Harmonix Home, Rock Band Idea

Why doesn't Rock Band allow you to create additional cities and venues? Nothing fancy, just the ability to set a city name and country, then create some venues. Venues would just let you select from the existing arenas and clubs1 (the 3D animation tool for user-created venues feels like more of an RB3 thing). It seems like an obvious idea for selling more content: allow for users to download cities from other users or Harmonix, but require them to have x downloaded songs to be able to use the city. Maybe the venue creation could have a recommended genre for what types of songs to choose from a user's collection, but not require specific songs.

Except in one case: if a label wanted to set up a "city" that contained historic venues a group played at on their rise to stardom and require you to buy various tracks to use them, that seems like a really cool way for labels to increase artists' sales in Rock Band (or Guitar Hero). It'd be like a low-cost version of Rock Band: Beatles for any group that cared to take the time/ money to get the venues created.

1. I am, of course, ignoring the legal issues that could arise from letting people create venues with names like "This place in my hometown sucks b@!!s", but it's my post and I'll do so if I want to.