From 1981 to 1984, Peter Davison played the fifth incarnation of the titular hero of Doctor Who, a role he has reprised for more than a decade now in regular audio plays released by Big Finish Productions. During a panel at New York Comic Con, the actor shared with hundreds of fans his history with the hit sci-fi franchise and his thoughts on the new series.

“This is the only [convention] I’ve done in New York,” Davison revealed to the packed room before jumping into his experiences with fans.

“When I come to a Who convention, I quite often get given books people have written, stories people have written,” he said. “I got one at a recent convention which turned out to be a kind of Fifty Shades of Grey take on it. Fascinating it was.”

“I’ve always loved the fandom,” Davison continued. “You do seem to be an extraordinarily tolerant bunch of people, and I mean this in the nicest way, because it’s every kind of facet of the human condition that you see at every kind of convention.”

When asked about his experience with science fiction before he joined Doctor Who, Davison replied, “I did love science fiction, but I’m talking about H.G. Wells and John Wyndham. That’s really the only kind of science fiction I’d actually read. But I always watched Doctor Who. I’m the first actor [cast to play the Doctor] who grew up with Doctor Who. I think I was 12 when it started, so I watched it all the way through. Even when I became an actor, I remember thinking it’d be great to get a part in Doctor Who one day, never suspecting at all that I’d actually be playing the Doctor.

“So it was a great thrill and indeed it was quite a challenge to come to terms with the fact that I would be playing a sort of icon from my childhood, which is why I didn’t accept the job straight away. It seemed such an extraordinary idea that I should be playing it!”

The discussion turned to the subject of “Time Crash,” the Children in Need charity mini-episode written by Steven Moffat that featured the Fifth Doctor meeting the Tenth Doctor, allowing Davison to team up with his future son-in-law David Tennant.

“I know Steven Moffat quite well, and I remember we were sitting in the garden at my house,” Davison recalled. “And he said to me, ‘How would you feel about taking part in a Doctor Who special for Children in Need?’ And I looked at him and said, ‘Why do you even have to ask?’ So he wrote this, I thought, very good short script that was both about the Tenth Doctor looking back at being the Fifth Doctor, and also about David remembering looking at me on TV. We did it all in one day.”

Comparing the classic series to the current one, Davison explained the difference in how writers approached the show. “Now Doctor Who is one of the BBC’s premier prestige programs. It really wasn’t in those days,” he said. “It was very successful and it sold to many countries and made the BBC lots of money, but it was never considered a premier drama series. So the people who wrote for Doctor Who quite often were people who wrote for a detective series one week, Doctor Who one week, and then maybe a hospital drama the week after. They weren’t really driven by their love of science fiction. Except my last story ‘The Caves of Androzani’ was by Robert Holmes, and it was fantastic.

“Whereas what I feel now is the difference is largely that [the new] Doctor Who is written by people with a passion for what they’re doing. A comfort for me about that, and the other classic Doctors, is these people were all sitting out there as kids in the audience and watching us do Doctor Who; they’re all fantastic fans. Steven Moffat can tell you, just by looking at one single black-and-white picture of any Doctor Who story from the classic series, not only what episode it is, he can tell you how far into the episode that scene was. I’ve seen him do it! It’s quite extraordinary – it’s disturbing as well!”

And is there any chance, as some rumors have claimed, the BBC may put away Doctor Who again after the 50th anniversary?

“I can’t see any sign of it ending,” Davison said. “It got a very low point before it was canceled the last time. I always knew when it was canceled that it would come back. A certain amount of time would pass and then someone would think it was a good idea to bring it back.”

Every 10-year anniversary for Doctor Who has involved a reunion of previous Doctors joining a story with the latest incarnation. Might Davison return for the 50th anniversary in 2013?

“Every day I check the phone to see if Steven Moffat has called me,” he said with a laugh. “I don’t know what’s happening next year, I have nothing to report. I’m sure it will be something fantastic! But I don’t know what. I think if [the classic Doctors] aren’t invited, I’m going to make my own rival video. I’ll do my own 50th anniversary special. Colin Baker’s prepared to work for nothing!”

The floor opened up to fan questions, and included Davison taking a photo of a cosplayer in a very accurate Fifth Doctor outfit. When one fan asked how Davison felt about unraveling Tom Baker’s famous scarf in his debut episode, he answered, “I loved every minute of it!”

Other fans wanted to know if Davison thought the Doctor could’ve had a romantic relationship with any of his companions. “Nyssa? Maybe that’s too easy a choice,” he replied. “Nicola Bryant? Yeah, she stole my regeneration scene.”

One of the last fans to reach the microphone asked which companion in the history Doctor Who did Davison think stood out the most. “I have to say, Rose Tyler. I think she changed the show and how we see and relate to companions. Those first seasons, she’s the most important character, more important in some ways than the Doctor. You’re seeing the show from her point of view. Really great companion.”