Jesse Schell On the History of VR, the Rise of MR, & the Potential of Digital Worlds

ICYMI, this year marks the 10-year anniversary of Reality Labs. We originally marked the occasion in April with a blog post covering some of the highlights from the past decade. The momentum continued with this week’s launch of Meta Quest 3S—the easiest way to jump into mixed reality. And today, we’re continuing the celebration with a new five-week series that shines a spotlight on the developers who make the magic possible.

Each week, we’ll sit down with different XR pioneers to talk about their time in the industry, some lessons learned, and what they’re most looking forward to in the future. First, we caught up with game designer, author, and CEO of Schell Games Jesse Schell on his own origin story and the birth of I Expect You To Die.

Jesse Schell: I first started working in VR—I guess it would have been 1992, something like that. That was the first time VR really entered the marketplace. It had been in research labs before that, but this was the first time there were products, third-party headsets available, all that sort of thing, and that was just exciting. It was something that I had heard about and I thought it seemed really interesting. I ended up going to Carnegie Mellon University where I was studying computer networking. I had to do a thesis, and there was a team trying to create a networked virtual reality art museum.

JS: And this was a really way-ahead-of-its-time kind of project, and so myself and a few other students got together and said, “Hey, you know, maybe we could help work on this.” My thesis was in network architectures for massively multiplayer virtual reality worlds. It would have been a little early there for that to be happening in 1993, but that was kind of what we were thinking about and working on. And it was very exciting because it was so new—no one had ever tried anything like this before. We were actually able to put an exhibit in the Guggenheim in New York City, which was very exciting.

JS: A short while after I graduated and was working at another job, I heard that Disney had this virtual reality lab. I’d seen some of their work down at Disney World, and then I saw a job posting at their studio. I was very interested, and it turned out that I had exactly the right kind of background for what they needed, so I started working there. That would have been in the mid ’90s because Disney started building a thing called DisneyQuest, which was a chain of virtual reality theme parks. That was really how I got drawn into the VR world.

JS: I mean, the way it was back then—we were definitely working with the future. The machines we were working with cost hundreds of thousands of dollars. It was incredible. Some of the machines we were working on were half-a-million-dollar machines. It was way out there.

It was exciting to be a part of something where we could see where it’s going, but it’s exciting to be there so early on and so exploratory. But at the same time, that was very lonely because there wasn’t anybody to compare notes with. You know, I’d go to the Game Developers Conference, and the things people were talking about there were so different than the problems we were dealing with day-to-day. It was exciting to be a pioneer, but it was also pretty lonely, way out in the woods on the frontier.

Ten years ago was exciting because it was the first time there was a headset that was going to go to the consumer market. The headsets we’d been looking at were for the enterprise market, and this was now going to be something that people would use at home. And that, for me, that was exciting.

I had a nice bridge in between those two things because I started teaching virtual reality classes at Carnegie Mellon in the early 2000s. My students were working with VR pretty consistently all the way through the gap between DisneyQuest and when VR started to show up in the marketplace. I worked on literally over a thousand student projects that were creating virtual reality worlds. When it finally showed up in the market, I thought, “Oh, OK. I think I know what to do. I think I know how this works.”

JS: Actually, the network architecture of the Toontown MMO was partly based on the thesis that I had done. Being able to sort of bring some of that to bear, you know, in a real game—a real game that still runs today—that was nice.

JS: I always get the most excited when there’s an opportunity to build something that no one’s been able to make before—just a way to make the world a better place by doing something new that no one’s ever done before. That’s always absolutely exciting, and there’s so many different ways that can work. It can happen through entertainment. Entertainment experiences can be really powerful for people. It can be through education—new ways to help people get better and improve themselves, that can be exciting. There’s a lot of different ways to do things that just improve the world.

JS: I think it’s very exciting, the shift from pure, immersive VR into the realm of augmented reality and mixed reality. It’s exciting for a lot of reasons. Partly, there’s a much bigger audience, I think, for mixed reality experiences. There are a lot of people who are intimidated by being totally immersed in a virtual world because of the way you can lose touch with your physical surroundings—that’s intimidating for them. Now, MR lets you get some of the best of both worlds, where you’re still in the physical world in your real place but there’s some magical things here with you. And that’s very appealing, I think, to a lot of people. So it’s exciting to sort of see that happen. It’s technically incredibly difficult, but the technology is starting to make things possible. There’s a lot of big questions about exactly what this is for, and a lot of people are exploring different things in that area—but we’re all learning it together.

JS: It’s definitely optical tracking—without a doubt, the ability to track both headset, hands, and controllers—to be able to do that optically, that’s just changed everything. That’s essentially the only real change that we’ve had since the ’90s. Back then, we used to track everything magnetically, and it was a big hassle because you had to hang a big antenna and it was complicated and weird. To be able to do this optically — it just gives you this freedom because once you can do that, now you don’t have to have wires.

Being able to do things without wires makes everything so much more freeing. And we’ve seen that. Once there was hand tracking and a wireless headset, that’s when we saw the Quest just take off like a rocket Other headsets before then, they were there, but they weren’t really taking off. It’s really made the difference and gotten so many more people to do VR and mixed reality on a regular basis.

JS: I think Apple has shown that eye tracking can help make an experience excellent, so I’m expecting that’s going to continue to be normal and a big part of things. I do think that while, for light experiences, hands-free has its place, for more serious experiences, for gamer experiences, the controllers we have now are probably going to stay pretty standard for quite some time. And that happens sometimes: You get a controller, and you get it right. Look at the computer mouse—when did that last change? It hasn’t changed a whole lot because it works. It does what it’s supposed to do. So I expect the controllers we have are probably going to hang around a long time.

JS: I mean, the lighter and the smaller the controllers can be, the better. The main problem with the ring is sometimes it’s in the way and it bumps things. So, yeah, I think the less intrusive they can be, the better.

JS: One story I think of a lot is that whenever there’s a new medium, people have a hard time figuring how to make new things, right? They always try and just copy what came before. You can go back and look at when film was invented. People said, “Oh, great. Now we can tell stories with film. We can go to a play and point the camera at the play and that’s our movie.” Right? But now it turns out there’s a lot more to a movie than just pointing a camera at a play.

The same thing happened when the touchscreen showed up. People said, “Great, we’re just going to take PC games and dump them on the touchscreen.” But that didn’t really work. People had to invent games that were touch-oriented in order to really use the medium.

We saw the same thing with VR. One of the ways we experienced it really up close and personal was when we were trying to build VR prototypes at Schell Games, and some of the developers were not really enthusiastic about it. They didn’t like the restrictions that they felt in VR.

I talked to one developer who was working on a prototype. I said, “Make sure you don’t do too much camera motion because it can make people a little uncomfortable sometimes. So maybe just do some teleporting in this experience.” He totally disregarded what I had said, and he moved the camera all over the place and it was uncomfortable. I said, “Hey, I asked you not to do this. Why did you do it this way?” He said, “See this is the problem with VR. When you put on the headset, you want to fly all around like a superhero and instead you get tied to a chair—and nobody wants that.”

We all looked at each other, and we were like, “Oh! Superhero getting tied to a chair!” Like, that happens actually all the time but nobody ever made a game about it. That was the moment we came up with I Expect You To Die—because we thought about those moments where James Bond is trapped and he’s got to get out of this deathtrap.

JS: It was a moment of taking the strength of the medium and leaning in. Flatscreen video games are always about running and running and running and running—that’s just what you do. You run and run and run. The nice thing in VR is you have this much greater intimacy, this ability to do careful manipulation of objects, and that’s something that normal games just can’t do. So that led us to asking, “What if we build a game that leaned into that?”

Of course, it’s different nowadays. People are getting more comfortable with motion. We’re learning better techniques for motion. So people are moving through worlds. But when I look at the strength of VR as a medium, it’s much more about manipulation and intimacy than it is about just running endlessly. That’s the story I think of a lot when it comes to the success we had with that game. It all kind of came from that moment of asking, “What can this system do really, really well?”

JS: That was a funny one for us. Not every game transitions well into VR—that’s just how it is. So we were looking at the question of Among Us, which on one level it seems like, well, that shouldn’t work. It’s a two-dimensional game and it’s about talking, so how would that work? It’s 2D and it involves a lot of typing. But as we thought about it more and more, we realized: This is a game that’s sneaking up on people, and it’s a game about watching your back. So there was that whole notion of doing a task with your hands, but then having to look behind you and literally watch your back. It was like, “Whoa, that actually feels really good. That feels like that’s partly what this is for.”

And then, of course, the fact that you have the integrated microphones—now you can have integrated speech and have these debates about who was the Impostor. All of that felt like it could work.

It all came back to that question: What is the essence of Among Us? It actually translates really well into the physicality of VR.

That’s one of the things we always say: Anything you do in VR or mixed reality, it always starts with the body because that’s what makes the medium different is that you have this bodily presence. And if you’re not doing anything that meaningfully involves your body, why are you doing it in VR? Why don’t you just do it on a flat screen?

JS: I can think of a few moments. Once I heard about it, it was very intuitive to me. There was an issue of Mondo 2000 magazine, which was an early ’90s techno futurist magazine, and they were running ads for VR headsets. So this is the first exposure I really had to the existence of this as a medium as far as I remember. And this notion of being able to not just look at a world, but be within a world—that just immediately made sense to me. The power of the potential immersion of that concept made instant sense to me.

Another moment I can think of was when I was trying to get people in our studio to understand the medium and get excited about it. There was this early Oculus experience that wasn’t actually interactive. You were sitting at this desk, and there were a lot of interesting things on the desk, interesting things in the room. And then you could look outside the window and there were these giant Tetris blocks falling down. And I realized that I wasn’t even interacting and I felt so immersed—and I had to stop and think about it. Like, why? Why do I feel immersed?

And it was just because the experience was doing so many things right. It was lining up my physical body with my virtual body, right? I’m sitting down while I do it, and I’m sitting at a desk in the experience. There were so many interesting things to look at, and when there are interesting things to look at, it makes the whole environment seem more real. By putting interesting things out the window, it makes you look around. And one of the things about the way we build immersion in our brains is the act of turning, right? Looking around.

What’s actually happening is our brain is doing a three-dimensional scan of the environment and building a model of it. That’s part of what makes it feel real. When you have experiences that make you gradually turn around and look at details, it starts to look and feel very real. And then it had a very strong audio environment as well. It was just doing everything right, and I remember being in there and thinking, “If something this simple can feel this real, then think of being able to build on that as a foundation and then turning that into something more interactive.” That was just a moment. I remember thinking, “OK, I have no doubt about the power of the immersion of this as a medium.”

JS: The mixed reality stuff that’s happening, I think that’s very exciting. I think that’s going to be really interesting. The thing that’s going to change everything the most is going to be AI, and there’s going to be this crossover between AI and virtual reality that I think people aren’t going to expect because the power that AI brings is going to be these in-game characters that behave in a realistic way.

They speak coherently. They can have a conversation with you, and they can have real emotions, and that’s going to all seem very real. And combining that with the immersion of VR so that these characters are in the room, in the space, in the place with you—I think we’re going to see a whole new medium of story-based games that are largely about talking.

One of the genres I’ve been predicting is something I’ve been calling adaptive in-home story games. Imagine a mixed reality game. You put on your headset, and you’re just looking inside of your house, and the doorbell rings. So you get up, and you go to your physical door. You open the physical door, and standing outside is a virtual character, who says, “Hey, I need to come inside.” They’ve got a bag of groceries. And you follow them into your kitchen because the headset knows how your whole house is laid out. So the character starts putting these items down on the counter and says, “I need you to help me.” And you start helping them. You cut the virtual bread loaf and slice vegetables and get everything ready. And they’re talking to you about this problem and how you need to kind of bake these things to make an item to use as bait. So you’re helping them do all this, and then they put the virtual items in your physical oven to cook.

While that’s happening, the character explains to you what’s going on. “The problem is up in your attic. There are these creatures, and you and I, we’re going to go up there and we’re going to get them, but first we need to finish getting the bait ready.”

We’ll be able to build these stories that use your own home as the basis, where we know what the story is going to be but the characters are now improvising based on the way your house works. We’re going to make this story work where you live, and it’s going to seem very real, and the characters are going to seem very real. Those are the sorts of experiences I think are going to be possible within the next few years. I think that’s going to be the biggest change: sort of living characters that are going to be in your own home.

JS: One thing we didn’t talk about that I think is really important is that the video game world tends to focus on males ages 16 to 30. That’s a lot of their focus. And the opportunity for this to be a technology that brings in so many more people is tremendous. In particular, one of the things that we’re seeing is younger people. We’re seeing some of the biggest growth in the VR space among younger people ages 10 to 15. There’s a lot of enthusiasm, and it makes sense because kids are very physically oriented, right? They’re happy. They love running around. They’ve got strong imaginations. So this is a really natural medium for them, and up ’til now there hasn’t been a lot of focus on that.

We’re already seeing growth in that direction. You look at the popularity of games like Gorilla Tag and Among Us. They have huge popularity with these younger players. I think that’s going to be a big trend. And as we move into the realm of mixed reality, I think it’s going to be an even bigger trend. Those are definitely things that are on my mind about where the future is going.

JS: Certainly for developers, the opportunity is there to be able to create content that is less traditional. For example, we created Lost Recipes, a virtual reality game about cooking in historical places. And there are people we see all the time who say, “Oh, I don’t normally play games, but this—this is what I wanted. This is meaningful to me.” So it’s exciting because the opportunity is there to create things for people.

I personally love creating games for audiences that don’t expect them because they’re so grateful. We were able to create a game once that was designed for emergency room doctors. No one else can play it because if you don’t have a medical degree, there’s no way you can play this. It was a training game, designed to help them, but you could see how much they enjoyed it because we were able to take what they’re good at, what they care about, and make that the key to success in the game world.

So for devs, there’s a huge amount of opportunity there. One of the things that’s been exciting and a little scary is the notion of moderation. These worlds can be exciting because they can be so intimate and so powerful, but at the same time, if you have bad actors who are going to try and be jerks in this space, it can be really intense. With Among Us VR, we’re doing what we can to make it moderated and make it safe, and it’s been exciting to see all the progress we’ve been able to make in terms of using AI systems and other systems in order to make these places safe and welcoming for everyone. We’re seeing more and more that that’s become the norm—it’s becoming what’s expected that you need this level of safety because otherwise it can be intimidating for people.

The good news is that things are definitely trending towards safe, welcoming worlds. There’s more work to do, but that definitely feels like the direction things are going.

And one of the things that I always think is really powerful is that you don’t have a lot of control in the physical world about how you present — what your body and your overall self is like. That’s part of what’s exciting about these worlds is that you can represent yourself exactly how you want it to be. For a lot of people, that can be really liberating and freeing to be able to show the world: “Here’s how I see myself.”