Write What You Know...Baloney!

Someone recently spoke at a writers' group I attended and really pushed the philosophy to these writers to "write what you know." While I understand that, in theory, and oftentimes do just that in my own writing, I don't believe that's a hard-and-fast "rule." Think about it...to write only what you know, it'd get pretty boring pretty quick. But this speaker was adamant. I mean, VERY adamant. So, of course, me being...well, me, I opened up a dialogue with him after his presentation. Here's a summary of the conversation:
Me: So, you really think writers should only write about topics they know?
Him: Yes.
Me: *pausing to give him time to elaborate...he doesn't, so I continue* I don't believe that's necessarily true. I think you should write about what you're passionate about.
Him: Oh, not so. If you ever want to be published, you should write what you know. It makes the prose more authentic.
Me: *arching a single brow* Really?
Him: *nodding emphatically* Yes. That's why so many are unpublished--they're trying to write about things they don't know about
Me: But what about passion?
Him: Passion only takes you so far
Me: Interesting. But I still don't agree with that
Him: *heavy sigh* Why not?
Me: Well, if that's the case, and I'm only writing what I know, I'm a child trafficker, voodoo priestess, helicopter pilot, US Marshal, gangster, FBI agent, pastor, alligator-lover, reporter, sheriff, investigator, contractor, PR officer, radio DJ...oh, and an arsonist and murderer too.
Him: *eyes wide* Excuse me?
Me: Well, if what I've written is only what I know, then that would be my description
Him: *smiling sweetly and shaking his head* Ah, I see. These are from stories you've written
Me: Yes
Him: *smiling wider* Do you already have an agent?
Me: Yes, one of the most respected in my industry
Him: And this agent hasn't suggested you stick to topics you know?
Me: Nope
Him: *his smile is so wide, it might split his face* Well, you'd probably get published if you did. That's my point.
Me: Um, that description I gave you a second ago?
Him: yes?
Me: They're all the hero/heroines of my books. The published ones
Him: *smile gone, eyes wide* You have a book published?
Me: 10 out, one set to release in August, another set to release next February, and I just signed a contract for 3 more to release in 2012 and 2013. How about you?
Him: I have two published.
Me: That's awesome. Who's your publisher?
Him: *puffing chest out* Tate Publishing. And you?
Me: B&H Publishing and an imprint of TOR
Him: *eyes roaming over the room* Oh, I see someone I need to talk with. Nice chatting with you. **and he walks off**

So...what about you? What're your thoughts of writing what you know only?