Empiricism Is Not a Matter of Faith

“Hurrah, this is it!” you exclaim as you set down the most recent issue of Computational Linguistics. “This Zigglebottom Tagger is exactly what I need!” A gleeful smile crosses your face as you imagine how your system will improve once you replace your tagger from graduate school with the clearly superior Zigglebottom method. You rub your hands together and page through the article looking for a way to obtain the tagger, but nothing is mentioned. That doesn’t dampen your enthusiasm, so you search the Web, but still nothing turns up. You persist though; those 17 pages of statistically significant results really are impressive. So you e-mail Zigglebottom asking for the tagger. Some days, or perhaps weeks, later, you get a hesitant reply saying: “We’re planning to release a demo version soon, stay tuned . . . ” Or perhaps: “We don’t normally do this, but we can send you a copy (informally) once we clean it up a bit . . . ” Or maybe: “We can’t actually give you the tagger, but you should be able to re-implement it from the article. Just let us know if you have any questions . . . ” Still having faith, and lacking any better alternative, you decide to re-implement the Zigglebottom Tagger. Despite three months of on-and-off effort, the end result provides just the same accuracy as your old tagger, which is nowhere near that reported in the article. Feeling sheepish, you conclude you must have misunderstood something, or maybe there’s a small detail missing from the article. So you contact Zigglebottom again and explain your predicament. He eventually responds: “We’ll look into this right away and get back to you . . . ” A year passes. You have the good fortune to bump into Zigglebottom at the Annual Meeting of the Association for Computational Linguistics (ACL). You angle for a seat next to him during a night out, and you buy him a few beers before you politely resume your quest for the tagger. Finally, he confesses rather glumly: “My student Pifflewhap was the one who did the implementation and ran the experiments, and if he’d only respond to my e-mail I could ask him to tell you how to get it working, but he’s graduated now and is apparently too busy to reply.” After a fewmore beers, Zigglebottom finally agrees to give you the tagger: “I’ll send you the version of the code I have, no promises though!” And true to his word, what he sends is incomplete and undocumented. It doesn’t compile easily, and it’s engineered so that a jumble of programs must be run in an undisclosed kabalistic sequence known only to (perhaps) the elusive Pifflewhap. You try your best to make it work every now