May 2013
2 posts
2 tags
March 2013
1 post
4 tags
February 2013
3 posts
4 tags
Parsing tables in MS Word with Python
I’ve been meaning to post this for a while and was reminded when I saw a post from Anthony DeBarros about using xlrd to parse an Excel document.
A FOIA request for voting records for Delaware state legislators returned a ton of MS Word documents, each with a table of votes for a single legislator. The State House uses a Lotus Notes database and actually stores their voting records in this...
January 2013
2 posts
2 tags
“In God we trust, all others bring data.”
– William Edwards Deming (via an awesome free stats book you can download from Stanford).
November 2012
1 post
5 tags
Locked out of the server on Election Day
Election Day at The News Journal was a pretty solid success.
We used PHP and internet duct tape (iframes) to scrape and display results for big races live on our homepage and results for all races on another landing page.
Unlike the Primary Election, our cron job ran smooth all night and my stress level wasn’t through the roof.
Click here to check out a “still-live” version...
October 2012
2 posts
4 tags
Linking Leaflet maps to DataTables
Along with likely every other newsroom in the country, we’ve been having a lot of fun with campaign finance data around here. Last week, we tore into Gov. Jack Markell and GOP challenger Jeff Cragg’s reports and were experimenting with how to present the data online.
Our initial idea was just a map because the story focused heavily on geography after Markell asserted that most of the...
September 2012
3 posts
5 tags
The News Journal's new PANDA
The News Journal now has a PANDA, a “newsroom data appliance” that will help fuel our reporting and hopefully make lives easier.
Needless to say, I’m pretty stoked. At the NICAR conference last February, I attended a presentation by the developers and knew right away that our newsroom would benefit from a PANDA. It took us a little while to track down a local server we could...
1 tag
2 tags
Found this on a DE government site today
Oh IE, how the world hates you.
<HTML>
<HEAD> <meta HTTP-EQUIV="REFRESH" ...>
</HEAD>
<BODY>
</BODY>
</HTML> <!-- - Unfortunately, Microsoft has added a clever new - "feature" to Internet Explorer. If the text of - an error's message is "too small", specifically - less than 512 bytes, Internet Explorer returns - its own error message. You can turn that...
August 2012
1 post
July 2012
1 post
2 tags
Making Excel only use visible cells
I’m cleaning and preparing school test score data for later this week. We are ranking schools for each subject and grade level based on average test score. The somewhat annoying challenge in excel is getting the built-in RANK() function to work on filtered data. When trying to input a range of scores into the RANK function, excel is including the hidden rows. The solution is courtesy of...
June 2012
4 posts
4 tags
My new favorite tool: IronSpread
Two very smart and generally awesome students at MIT created an Excel plug-in called IronSpread that allows you to write Python scripts for the popular spreadsheet program, and I’m now in love with it.
I just started playing with the plug-in this week and have already applied it to a couple projects. In short, it lets you use a shell to manipulate cell values or run a python script from...
1 tag
Still a better love story than Twilight
howdoiputthisgently:
1 tag
LinkedIn hacked: Is your password among the...
After hearing about LinkedIn being hacked, I dug around a little to find out how to check if your password was among the stolen ones. All the passwords are still stored as hashed values, but if you have a simple password, it could be easy to figure out and apparently many of the easier ones have already been cracked.
So, here’s a little python script I pulled together from a few different...
May 2012
2 posts
6 tags
HOW-TO: Using pop-ups to embed DocumentCloud notes
For a story that ran this past weekend, I was asked to figure out how we could incorporate notes created in DocumentCloud - a tool for journalists that is invaluable for organizing, annotating and publishing documents.
We had more than 15 highlights from roughly 175 pages of emails between developers and state officials regarding a riverfront project in Wilmington that we wanted to share within...
“AND NOW, this 25th day of April, 2012, upon consideration of the pervasive...”
– Thomas K. Kistler, President Judge of the Court of Common Pleas of Centre County, Pa. (via officialssay)
April 2012
3 posts
8 tags
IT'S FINALLY LIVE
Almost two months ago, my managing editor called our web designer and I into a meeting to discuss how we planned to deal with putting together a voters guide for next month’s school board election and the General Election in the fall. We decided we wanted something interactive and encompassing, and I decided I didn’t want to spend all my time creating new voters guides because by the...
7 tags
Dealing with Dates and Times in SQL
A fellow reporter handed me eight years worth of parking ticket data that he acquired from the City of Wilmington a few weeks ago. As databases go, this was a pretty fun and easy one to work with, mainly because it was basic. There were no car makes and models or driver names in the data, so our focus was easy to find, we were going to look at where and when the more than 600,000 tickets were...
6 tags
Keystone Awards
The Citizens’ Voice got some great news today about the 2012 Professional Keystone Press Awards. The tab that I was fortunate enough to call my first employer tied Bloomsburg as the overall Division III sweepstakes winner. Click here to read the staff roundup.
Apart from helping out with the first-place staff award for our coverage of the September flood, I snagged both first and second...
March 2012
2 posts
3 tags
“I expected to have to go through the offense category and change “snorted blow...”
– Matthew Doig from the Sarasota Herald-Tribune talking about how clean a database was that the paper used for “Unfit for Duty,” a nine-day series on how officials handle serious incidents of misconduct by Florida’s law enforcement officers.
This chart uses data scraped from the BLS...
into a Google Spreadsheet using the ImportHtml function.
Delaware Unemployment
...
February 2012
4 posts
5 tags
#NICAR12: Day one highlights
NICAR day one has been a whirlwind, but I’ve already learned a ton and had quite a bit of fun. Here are the quick highlights from the day:
NewsCamp: Investigating text in the wild
I was super excited for this panel. Picture having to analyze hundreds of government documents. And, we’re not talking about a tab-delimited table; just raw text.
Sarah Cohen from Duke University walked...
3 tags
More Google Charts: Mapping the family
Scroll over the states to see which family members live there.
3 tags
More Google Charts: Fun with dispatch data
google.load("visualization", "1", {packages:["corechart"]});
google.setOnLoadCallback(drawChart);
function drawChart() {
var data = new google.visualization.DataTable();
data.addColumn('string', 'Day of the week');
data.addColumn('number', 'Number of total dispatches in 2011');
...
Fun with Google APIs
Finally playing around with Google’s Javascript API’s. Lots of fun to be had.
google.load("visualization", "1", {packages:["corechart"]});
google.setOnLoadCallback(drawChart);
function drawChart() {
var data = new google.visualization.DataTable();
data.addColumn('string', 'Date');
data.addColumn('number', 'Number of dispatches');
...
January 2012
1 post
5 tags
Moving to a new job
This happened faster than I ever imagined.
Today was the official announcement in The Citizens’ Voice newsroom. My friend Andy and I will be leaving in a couple weeks to take jobs at the News Journal in Wilmington, Del.
For those of you who don’t follow my life too closely (i.e. most everyone except my parents and Krista), I’ve worked at the Voice for roughly eight months and...
December 2011
3 posts
3 tags
6 tags
'Peggy' called back!
Over the past several weeks, I’ve been trying to reach someone at the Federal Emergency Management Agency to give me a status update on my Freedom of Information Act request for a database.
I submitted the request on Oct. 4 and received a response saying the agency was taking a 10-day extension. After that, nothing. As of today, the request is on the verge of being several weeks past the...
6 tags
Putting boots on the ground
These past few months, I’ve been consistently reminded why the world needs “traditional” reporters and media institutions.
While media continues to transition into a hard-to-predict, amorphous blob of new technology and evolving mindsets, it’s difficult not to hear folks talking about the death of newspapers and traditional media.
I got the chance last week to watch Page...
November 2011
7 posts
3 tags
4 tags
Open Records: Fighting for information on...
There are more than 200 cameras watching the residents of Wilkes-Barre, and city officials say they make the city safer.
The problem, though, is the city says it doesn’t keep track - or plan to keep track - of any statistics that would show whether the cameras are effective. In its mind, the city’s word that the cameras are effective is all the public needs.
That’s where we...
3 tags
USA Today, Chris Schnaars use jQuery DataTables... →
Schnaars gives a great explanation of how USA Today built its interactive database of salaries for college football coaches. It’s awesome and will probably suck up a lot of my weekend.
4 tags
Chronicle of Higher Ed. uses Storify well to track...
The Chronicle of Higher Education put together a Storify post following the Penn State sex abuse story.
Check it out below the jump.
3 tags
“It’s kind of like watching commercials about yourself constantly.”
– Incumbent District Attorney candidate Jackie Musto Carroll, the subject of a massive attack ad campaign, talking about cyber bullying.
An open records victory →
A proposal to allow agencies to be able to tell people records don’t exist if they are exempt from FOIA finally failed.
What a horrid proposal.
4 tags
October 2011
5 posts
5 tags
Definitely need to take advantage of Storify more...
Storify released some major changes to its website today that make creating social media stories even easier.
For anyone not familiar, Storify lets you create stories that incorporate posts from Twitter, Facebook and other social media and photo websites. The stories have a really clean layout and are pretty darn neat.
I’ve only put together one sizable post using Storify, a recap of The...
3 tags
Stumbling through federal contracts
You can’t immediately say it’s wrong, but when a government official gets a job at a company which his office provided a lucrative government contract, it makes you wonder.
I found this interesting tidbit while wading through the Federal Emergency Management Agency’s list of active contracts (pdf).
FEMA awarded a more than $30 million contract to a company called OST, Inc. The...
2 tags
“Spell check will never put a squiggly under my name.”
– Reasons to be happy.
3 tags
Time to learn some new music
I’ve decided that I need to learn to play many new songs on the bass. I’ve got a partial list going, but I need some help adding to it.
In no particular order, here’s what I have so far:
Higher Ground - Red Hot Chili Peppers
Maxwell Murder - Rancid
Shaft - Incubus
YYZ - Rush
Sing a Simple Song - Sly and the Family Stone
John the Fisherman - Primus
What else should I...
August 2011
9 posts
7 tags
Mom, don't freak out
I was in one of those positions yesterday, where it probably would have been a good idea to be armed.
Before going any further, I’ll say that I’ve seen a dead body and covered stabbings and shootings - including two on my own street when I was a Citizens’ Voice intern last summer. I almost never feel wary walking into a crime scene or weird situation. After all, when I get sent...
Busting out the ol' social media methods
I used Storify for the first time to help recap all of the Citizens’ Voice’s social media reporting on Hurricane Irene.
It’s kind of long, so I posted it after the jump.
5 tags
Python and free GIS software FTW!
Friend and coworker Andy Staub put together a great piece about how Wilkes-Barre’s mayor was hiring his family members for city jobs. With the help of a few in the newsroom, he put together a spreadsheet of all of the hires since 2004.
We used that to create a Fusion Table map of all the hires. What was evident almost right away was that a lot of the hires came from the mayor’s own...
