This really drives me nuts. When San Franciscans do it, it sounds arrogant and self-indulgent. However, it's even worse when San Joseans refer to our smaller neighbor up north as "the city." Every time we do this we are continuing to perpetuate a nickname that is degrading to San Jose, and many other Bay Area cities like Oakland.
Leigh Weimers is a former columnist from the San Jose Mercury News that has recently passed away (his memorial service is Monday, 3:30 at St. Joseph's Basilica). One of his few regrets was: "I'm sorry I haven't gotten more San Joseans to stop calling San Francisco "the city." It's "a city" -- so is San Jose -- but not "the city." That's so last century. (Source)
I'm hoping that Sal Pizarro, who has taken over Leigh's position, will take the baton and continue to fight to get that reference out of San Josean's vocabulary. I think the way things are going in San Jose, we can do it, but like anything it will take some time and conscious effort.