RailsConf is coming back to the Baltimore Convention Center, so here's an update of a list I made last year for attendees. The list is optimized for people staying near the convention center (this is not necessarily the list I would give you if you were staying with me or had ready access to a car)
Your best bet in this direction is the Federal Hill neighborhood which has a lot of activity, bars, and nice restaurants. During the day you can climb to the top of Federal Hill itself, and then visit the incredibly awesome American Visionary Art Museum: this is the ideal art museum for hackers, because it celebrates self-taught artists.
On Wednesday or Thursday evening I recommend checking out Illusions Magic Bar & Lounge. If you're around on Friday or Saturday night, there's an awesome magic show featuring an upside-down straitjacket escape. There's nothing like this place back where you come from!
Right next to the convention center you'll find Geppi's Entertainment Museum: all about pop culture, comic books, toys, etc.
Maryland's signature dish is the crab cake, and one of the best versions is made at the Faidley's stand in Lexington Market which is worth visiting to soak up all the vibrant activity of a city market.
Definitely visit the Mt. Vernon cultural district, a long walk north from the harbor. It's a beautiful neighborhood with brownstone homes and great restaurants. The Brewer's Art bar was apparently named one of the "Best Bars in America" by Esquire, and they brew an excellent ale called Resurrection. Ask them for a flight of all the beers they brew there.
If you have a car or bike, you may also want to check out the Hampden neighborhood which tends to get a lot of attention by people writing articles like this one - John Waters recently described it as a mix of "hipster culture and redneck culture".
The Inner Harbor is our ubertouristy area, but it's very nice if you've never been there. Besides our great National Aquarium, you can each catch a water taxi from there to the Fells Point neighborhood farther east, which has a ton of bars and restaurants and cool shops and coffee shops.
If you have access to a car or taxi, after stopping in Fells Point you might want to visit Canton, one of the city's technology hubs. The Beehive coworking facility is well-worth a visit if you have time to kill before or after the conference.
Besides the RailsConf 5K taking place, there's a red brick path going all the way around the harbor that makes for a good running route. Check out the Bmoreonrails recommended running route (at the bottom of the page).
If are a Wire fan with a car, you may want to visit some of the shooting locations which I have catalogued previously in "The Wire tour".