Dev Talks

The Role of UX in Robotics and AI

Vincit Dev Talks
March 2nd 2020
Vincit Dev Talks is a quarterly tech meetup held in Southern California. It’s designed to bring directors down from their towers and developers out from their cubicles. We feature speeches on a variety of topics ranging from marketing and business development to software engineering. Talks will be brief but informative, leaving time to chat and enjoy the free food and drinks. Check out vincitdevtalks.com for upcoming events.
In this talk from our February 2020 event, Hamid Badiozamani and Di Le from BrainCorp discuss how their company designs and builds robots that have to deal with the complexity that comes with interacting with humans while performing their designated tasks.
Hamid: Thank you, Rachel. Thank you.
Di: Thank you. Thank you very much. I'm sorry to make you be the tethered one.
Hamid: No, it's all good.
Di: I now feel like I should take most of this [crosstalk 00:00:10].
Hamid: No, no, no. That's good. That's good. We should probably turn on that camera too.
Di: By the way-
Hamid: Hang on, hang on. I'll do it.
Di: What?
Hamid: Turn on the camera. I forgot about that.
Di: We're doing a private home video. That sounds bad. That sounds naughty but at least you guys are in it. By the way, if you guys stay till the end when we do the robot demo, I'd love to do a crowd selfie with you guys. We always say that we're going to do that but we always forget because it looks like people are about to leave, so we'll see what happens today. But honestly thank you so much everybody for being here. Coming from San Diego, we really are aware of how bad LA and Orange County traffic is, which is probably why we flew. But it means a lot that there are so many people here today. I do want to know who is here today. Is there anyone here that works in AI, deep learning, ML, self driving car people. Feel free to participate. Any robotics people?
Hamid: Really.
Di: Wow.
Hamid: All right.
Di: [inaudible 00:01:11] might have that.
Hamid: How many designers here? Two. No, that's two hands up from the same person.
Di: No, that's two hands. What?
Hamid: Two people. Oh my God. Three people.
Di: Four, no, no, no.
Hamid: Come on.
Di: There's a girl back there. She's being shy.
Hamid: Ah, okay.
Di: The one in the white jacket.
Hamid: Floor design. Wait a minute. Our raw speakers were designers. They're not raising their hands. Where are they?
Di: No, they're UI engineers.
Hamid: They're UI, well, they're easy. You have to do design.
Di: Okay, okay, okay.
Hamid: All right.
Di: Well, how many general engineers, developers? Okay? Developers. Ah.
Hamid: Wow, okay.
Di: Here's your people.
Hamid: There's my people.
Di: Well, technically front end development. I guess I could throw my hand in there too.
Hamid: Yeah, a little.
Di: Jack of all trades. Well, as technologists, people that are preparers and also spectators, when we talk about automation, AI, robotics, software development, and just technology in general, looking at our landscape, it's really hard to not notice how the evolution of technology is just changing at an unprecedented rate, much like it did with the advent of the internet when we became so connected and we could see all of each other's cats and dogs eating ice cream or doing whatever weird things that cats do. And then once again, with the mass adoption of mobile devices and phones and liking each other's posts and all of that stuff. But we're at this event horizon right now with automation and especially intelligence. Whenever someone thinks about intelligence, they think of general intelligence where it's Ex Machina and we'll jump into that.
Di: But there's a lot of intelligence and AI that permeates throughout our entire lives. It's changing the way people commute. It's changing the way people communicate, conduct business, changing the way that we consume. So today we're going to talk about an opportunity that isn't talked about frequently enough. It is a mobile application. It is especially in consumer software, but not necessarily in our industry, which is robotics. And that's the opportunity of designing intelligence systems with humans at the center of the conversation, humans in focus. I mean, you guys hear this all the time, human centered design, but for some reason when it comes to AI and intelligence and robotics, it becomes this novel idea, kind of the way it was for software in the early days. What do we mean when we talk about robots though?
Hamid: Well, for me, for my part, even after my years in the industry, when I think of a robot, I always think of an android and I think it has a lot to do with the media. When we consume movies, if you ever notice, they normally take a character that could be a human and they simply replace their apparatus, their hardware, with an electronic thing. And so even the emoji icon for robot is an android. But in reality... My projector is turned off.
Di: It'll happen a lot. Just roll with it.
Hamid: All right. There we go. Robots have been part of our lives for quite some time without us really noticing it. And what I'd like to discuss... Well, let me go back up a little bit. You can think of robotics as being part of three major branches and these aren't hard delineations, but for the most part, if you look at them by technical concern, they fall into these three branches. The first one is robotic arms. These are typically on a platform bolted to the ground. They're multi-jointed, a number of, yes. Just like that. Pick and place. Yep. And their primary technical concern is repeatability and accuracy. Like 3D printers or CNC machines, pick and place robots, things like that. On the other end of the spectrum, we have what I like to call fluid dynamic robots and these are robots that are suspended in some kind of fluid or air.
Hamid: Talking about drones, submarine drones, maritime robots and they're mostly concerned with where they are in the world, because regardless of what motor commands you send to them, they're going to move. And so in real time you're going to have to continually update where you are. And then we have the third category, which is kind of in the middle, ground-based robots, and those are further subdivided into two separate, similar but very distinct categories, AGVs and AMRs.
Hamid: AGV stands for automated guided vehicle. These have been in use for a very long time in warehouses. If you visit an Amazon warehouse, for example, these are the types of robots you'll see. The entire building is essentially a robotic system because there are signals and lines and magnetic strips, I guess would be the best way to describe them that are inlaid either-
Di: QR codes.
Hamid: Yeah. Into the ground or on the ceiling. And what these robots do is they follow these lines and they go from point A to point B and that's it. That's their job. And of course you can program them to efficiently move around the manufacturing environment or the warehousing environment to get your merchandise to where it needs to go.
Di: Yeah. And a lot of these robots have singular tasks. It's like move from here to here, pick up or deliver.
Hamid: Or drop off. [crosstalk 00:00:06:15]. Yeah. And then on the other hand you have AMRs, which is what we work on. These types of robots stand for autonomous mobile robots. And their primary difference is that all of their brains and decision making is onboard. They make decisions, they adapt to a changing environment, they will avoid obstacles. They basically think for themselves. And you'll see later on in the demo, the reason is because we need to have these things work in an environment that is novel to them. And that's kind of our advantage. And it's the big technological leap that I think we're at an inflection point of. And so with AMRs we have an opportunity and a risk. And that risk is that they operate in public spaces where you have multiple levels of human interaction.
Hamid: So when we talk about designing human centered design, we're talking about not just the operator of the robot, whose job it is to make sure that the robot runs safely, or the customer who's purchasing the robot and needs to determine whether the robot is actually fit for the job at hand, but we're also talking about end users, innocent bystanders who may not have ever seen the particular robot.
Hamid: And a specific example would be shoppers at a Walmart. And our robots might be cleaning. Those shoppers, they have nothing to do with the robot, and yet they occupy the same area and they need to be able to negotiate how to navigate around it. They need to not be freaked out. And so what we want to talk to you a little bit about today is how we do that, specifically, how we design our robots with humans at the center.
Di: Because they are entering our world. It is a human based world. Another user that oftentimes in developing robots and autonomous systems that aren't thought about are people operating other types of machinery, other types of existing infrastructure, whether manual or autonomous that are now impacted by our machine, where now their success or our success are relying on each other.
Di: When you brought up humanoid robots and we're going to be showing a clip soon, but when you bring up humanoid robots, I can't help but think of Terminator and then Skynet, and I know some of you are thinking this because I see it as I'm looking around in your eyes. [inaudible 00:08:44] really excited. First floor scrubbers and vacuums and delivery robots, and then next thing you know [crosstalk 00:08:50].
Hamid: That's how it starts, man. They got us pegged.
Di: Okay. Well, before we get into the granularity of designing for autonomous vehicles, for the sake of today's talk, we will be focusing on robotics, but that's with consciousness knowing that artificial intelligence is the brain that drives that vessel. So let's assume not necessarily that Skynet happened. Let's teleport into a little bit of the dystopian future and see if we did have robot overlords, what would their perspective of the design of humans as a species would be like. I hope the audio works.
Hamid: The audio is not working. Ah, technical difficulties. This is what they would think of us. They'd be like, you guys can't get your audio/video ready. Yeah. Should I bring it closer? All right.
Di: Humans are like one giant technical difficulty.
Hamid: We only have like four slides, so if you want to just hit it there and we'll just signal you and say, hey, next slide.
Di: We've only got like four slides.
Hamid: Four slides. Yeah.
Di: If you guys can't tell, we're not very good with the whole slide thing.
Hamid: Yeah. We don't prepare either, so that's why the quality of the talk isn't going to, okay, go ahead.
Di: Emiril Lagasse back.
Hamid: Aw, what's going on with the audio?
Di: It's not working.
Hamid: The audio is not working. They're having a conversation. Maybe we can have the subtitles on or something. I don't know. What is going on?
Di: This is so funny.
Hamid: You're going to have to, yeah, it's from the anthology-
Di: It's from the show Three Robots and we have these humanoid robots and they're looking at this hamburger. One goes, wait, wait, wait, wait, wait. So they stick this where? He goes, they stick it into their orifice.
Hamid: Their intake orifice.
Di: Their intake orifice, where their rocky pegs turn this into a paste and then it gets consumed into a vat of acid and that's how they create energy.
Hamid: And they're like, well why why would they do that? We just have a fusion battery. We all have, yeah.
Di: [inaudible 00:10:57] everyone has a fusion battery.
Hamid: Or why wouldn't you just put it, instead of the rocky pegs, just put everything in a vat of acid and then digest it that way. Anyway, you'll have to watch it. Love Death & Robots on Netflix. Highly recommended. Yes.
Di: Sorry about that. See? Humans, one big technical difficulty.
Hamid: That's right.
Di: But the funny part about this, ,and why we love this clip and I will gladly send it to anyone that wants to see it after, is from talking about our orifices, things coming in and out all over the place, which they said about me. You know, I already started with making a home video, so I don't want you guys to think some kind of way, but between the vat of acid, how we generate power, to this one point where one of the human robots goes, who designed them anyway? And then the other robot goes well, we checked their creator signature, there was none. We checked their code, there was none. So rumor has it they all came from a warm soup.
Di: And then another robot jokes, and he's like, no, I think some ancient deity created them all. The reason why we love this clip is it's so representative and a mirror of how we design, we as humans designing technology and robotics. One-to-one replications of human tasks, binary actions that either pass or fail, and everything that they were critiquing about humanity and our design and our technical difficulties and inefficiencies came from a very robot-centered technical perspective. The same way, when we design software and robotics, we're looking at it most of the time from a technological, engineering perspective. What's missing is that human perspective, and why? Because it's a human world. It's really a moment to sit and reflect a second. I'm not trying to freak everyone out, saying, yeah, robots are going to take over. But it's an opportunity to look at how the impact may be if we at this event horizon right now could make choices differently.
Hamid: Yeah. In talking about altruism and altruism, I think many would think doesn't always keep the lights on. And I would challenge that. And we're not too far away from Disneyland and we study Disney quite a bit, and talk about a guy who was about ROI, you don't get much bigger than Disney. When he built Disneyland, he had an opportunity to build all of his rides at the same time and he would've gotten faster to the end of the park and could have opened faster and would have had presumably a few more months, maybe even a year, of ROI because the park would be open.
Hamid: But he chose to go a different route because he understood his people. He understood what people wanted to see, his customers, what they wanted to experience, and what he ended up doing was building his castle first. He finished the castle at the center of the park first so that all of his Imagineers, himself even, and all of his crew members would know that when they build a ride, that castle is at the center. That's how it should be. Similarly with robotics, we need to make sure that humans are at the center because it's going to introduce a much better adoption rate and ROI in the long run.
Di: And it would be irresponsible as purveyors of intelligent systems and automation to not acknowledge that the current social economic climate around automation is one that is surrounded with apprehension, fear, and not necessarily fear against the technology, but maybe fear on the impact that technology will have on one's own identity, one's own career. When we think about our own jobs, designers, developers, and we're technologists here, we're at the pinnacle of it, how much of our own tasks could be automated? I mean, when we look at the neural nets that are being trained now, there's so much of UX work and a lot of data points that we could take from previous existing systems and maybe do UX better than humans can right now. This apprehension is so prevalent that ex-presidential candidate, Andrew Yang, the core platform that he was running on was this discussion of automation taking jobs and saying that's why Donald Trump won.
Di: Whether that's true or not, this is something that is already being discussed. So when we talk about ROI and we talk about this introduction of the technology, this barrier to entry is the biggest part. And when I talk about barrier to entry, I'm not just talking about the cost of sensors and how much the mechanical engineering or the software engineering is. I'm talking about how much it costs to just integrate a disruptive technology into an industry that hasn't seen disruption. A great example of this is Uber. We remember Uber, just not that many years ago, came into the market, started lobbying right away and took on the industry of public transportation head on. They were met with violence. It was met with huge political debate, and I think even until today, they're still paying for a lot of that money invested and that loss of ROI just based on how they were disruptive to this industry and how people were really apprehensive and reluctant to adopt that, even though for the common good, it was better.
Di: So automation could be better for the common good but if people are afraid of it, they won't accept that. And we're starting to see that now with self-driving trucks and truckers, factory workers and automation. So knowing that we're approaching the design of a technology with common perception and fear... Fear, that feeling, that emotion, fear. What do you do in the face of fear? Well, one, you could flight. We all shut down, go build yogurt shops.
Hamid: Boba shop for you.
Di: Boba shops [inaudible 00:17:04] popcorn for me. But you know, we're not going to do that. Technological evolution will never change, innovation will persist. What are the things, what can we do with fear? Well, you can change fear. You can change the perception of what you're trying to introduce. You could cater to it. It's okay to be a little soft. It's okay to pamper people a little bit when you're delivering new tech and leverage fear. Fear's a great emotion. It's kept us alive for many, many, many years and allowed us to procreate. So we'll touch on each of these topics, but talking about change first.
Hamid: Yeah, one of the best examples, I think, of overcoming fear of adoption is through iterative introduction of a feature. A great example would be when I bought my car, it came with a standard cruise control. You know, you hit the stock, it maintains the speed that you're at and that's all it does. It'll happily ram into the next obstacle that's in front of you if you're not paying attention. A few months later, a software update came and now we have adaptive cruise control, which uses the ultrasound sensors that are in front of the car to slow down and match the speed of the car that's in front of it.
Hamid: A year later I get another update and now I have lane keeping. So at every step of the way, I use the same way to activate it. And I'm already familiar with half of, if not more than half, of the feature already. And that really helps. Now, I must admit, the very first time I tried it, I was still a little nervous. There was a car, is this going to stay on the road or not?
Di: [inaudible 00:18:48].
Hamid: I will never do that. I am definitely not a distracted driver at all when I engage autopilot. Never, never, never. But yeah, it really helped ease adoption. And I think that that iterative design is one of the best ways that we can change the fear, is by-
Hamid: ... is one of the best ways that we can change the fear is by showing an example of it's the same thing, it's just a little bit better. Then it gets a little bit better. Then it gets a little bit better, rather than this big bang of, "Here we have a self-driving car."
Di: We're going to touch on this a lot, but just borrowing from legacy design. In order to design, it doesn't mean invent something entirely new. It means take something and then slowly kind of incrementally change it. We borrowed a lot from the legacy of car designs. Automobiles have been with us forever. It would've been naive to not borrow from perceptions of how people expect a large machine to move.
Di: This floor scrubber here was the first robot that we at Brain Corporation built, and it was for the cleaning professional industry, sanitation industry. Multi-million dollar industry. Hadn't been touched by automation or much disruption by technology in decades. We're entering this industry that hasn't been disrupted by technology and where the dominant core of demographic of professionals in that industry are not necessarily always technically affluent. They have a set of tasks that they have to go through every day, and the last thing they want is to be bothered by looking at a new sexy robot that they have to figure out how to use. There's that barrier to entry again. Do I have to learn how to use your technology?
Di: The choice of not making a Tron-looking robot was very conscious. We made the floor scrubber look like a floor scrubber, so if I, a cleaning professional, were to walk up to this machine, immediately it's familiar. I've been using this machine for decades. There's a steering wheel. I know where my hand should go. Talking about fear and perception of automation, it doesn't look like a robot. When I as a cleaning professional look at it, there's a seat positioned above the machine.
Di: When we were borrowing from legacy designs of floor cleaning devices, there were many different types. You have push behind, ones that were remote control, ones that you could move in many different ways, but it was very conscious to pick something where the seat was positioned above the robot, so there's always a place for a human. There's always that thought that, "Okay, the human has final control." It's the same product, the same tool just with added automation benefits.
Di: It's the same way that we conduct mapping, the way that we map as opposed to just having this thing immediately run off out of the box. It's with the owner of the robot moving the robot through the space exactly how they would clean the floor. Then that iterative design. Once we went to the delivery robot, there's still a steering wheel. There's a platform for the human, but now that platform and that positioning is a little bit less. People got a little bit more familiar with robots. Robots are a little more prevalent now, so we can make that platform a little hidden. Then all the way down to these small vacuum robot, there's a handle there. It doesn't stick out the way a manual handle does. It recedes into the body.
Hamid: Yeah. Speaking about, again, humans. Di touched on the operator and the seed and the familiarity of the operator with that type of robot, but there's also something to be said about all the other end users who happen to be bystanders when this robot is operating. Because they need to be able to predict what this robot is doing, and if we built something from the ground up to look like the sexy, Tron-looking, Optimus Prime-looking thing, they don't know. Maybe this robot is going to take off and hover over my head. They have no idea, so it was deliberate for us to design this in such a way that people can see there's a steering wheel. People immediately, even if you've never seen this robot, if you look at it and you see it driving by itself, you'll notice the steering wheel turn. You have an idea of how this thing is going to behave and how it's going to move.
Di: Yeah, and borrowing from, again the legacy automotive design, even for the small vacuum robot that we're going to be demoing today, when I was doing the sound and the lighting testing, testing different colors that people [inaudible 00:23:22] say amber with blinking lights. Unless you're a racer like I was and was into clear hyper-blinkers [crosstalk 00:04:32]. But-
Hamid: Those were the days.
Di: Yeah those were the days.
Hamid: Fast and furious days.
Di: If amber could be perceived by most people that it translated direction intent. Same thing with the reversing of the robot. We kept the white lights at the back for the pattern. Small nuances like that. It's really important in the familiarity to really establish how people interface with the robot. When I say interface, I mean more than a UI. It's like the UI is the smallest window of how we interface with technology when you think about social cues and how humans communicate, before we ever see a word. That's like how fast I approach you.
Hamid: Whoa, whoa, whoa, whoa.
Di: How quickly I get in your face. Do you know where I'm about to turn? Where am I looking? Stuff like that.
Hamid: Yeah. Talking about that, what's cool about robotics, what I really enjoy is a dimension the beyond the UI, because physical spaces can be navigated. You can use the kinetics of the robot to communicate what it's doing, and he experience of the robot moving, how it moves, how it reacts to obstacles versus how it reacts to people. For example, if it's supposed to be cleaning, one of the challenges is that you have to be able to get close to the walls. Obstacles that are static and they're not moving, the robot will try to get as close as possible without touching it. Whereas with people who are moving, we want to give them some safe space. We want to make sure that they feel safe around our robots. It's an 800-pound robot. It will go right through a wall, that big one over there and this one over here. We've had three of our buffest guys get in front of it, then one person sat on it. It just steamrolled right through them, so it's definitely a-
Di: No engineers were hurt in filming.
Hamid: Yes, right.
Di: So we're talking, so all of these things help to change people's perception of what a robot looks like. What is automation? What is AI? But sometimes it's okay to be a little nice. When we talk about like the design of a robot and choosing something that doesn't look like a humanoid, scary Ex Machina Terminator looking thing, there's a lot of, if we look at hospitality robots, there's a lot of robots out there that don't look like Skynet robots jumping over boxes that are more agile than us.
Di: UB Tech is a great company that does this really well. They have humanoid robots that can walk up and downstairs, but it doesn't freak you out. The way you move doesn't make you think that they're going to chase you down. They have very intricate animatronics in their digits, but then off the top of their cruiser robotic is a giant gumball dome that has beautiful animations on the eyes, friendly voice controls, friendly sounds, and it's very welcoming. The whole head and everything is an interface. Google's self-driving hard did this very well. Very, very meticulously designed a car that did not look like the Tesla Roadster. I don't know why I said Tesla Roadster.
Hamid: [inaudible 00:26:36].
Di: [inaudible 00:26:37]. Maybe because you're a Tesla person, but it looks like a ride at Disneyland. Like if you [crosstalk 00:26:46].
Hamid: Sorry, Di. I've got to interject, because she made fun of me. She makes fun of my Tesla. She's a Porsche driver. She calls it a grocery getter just so you know.
Di: He blames ... He says that I called it a grocery getter. I said no. Well, anyway.
Hamid: Anyway.
Di: But catering and maybe coddling who you're introducing this technology with. In the early days of web, Twitter did this really well too stepping away from robotics of it. There were 414 messages were cute animations. They were kind of one of the first companies to make friendly messages that didn't sound like a blue screen of death when your windows machine went up.
Hamid: Yeah, that is an intimidating thing. Let's talk about leveraging, leveraging the fear.
Di: Fear is good.
Hamid: Like greed is good. Greed is good. Fear is good. I think that's a little counterintuitive. Wouldn't you say?
Di: No, fear is fantastic. For us, especially in automation, children are taught at a very young age when you cross the street to look both ways. Don't walk in front of moving cars. When you see bright lights and when you see more things, think about stop sign, big red signs. There's something about fear that we could leverage that almost trains your end user how to behave around our technology. A study that I did unrelated to robotics but around security, when it came to, say, a Spotify app or any type of musical app, people are annoyed to end with two factor identification. What do I care if someone steals my playlist? But when it came to the banking, when it came to credit cards, when it came to their own identity, two factor authentication. Let me text you. You have to reset your password if you update your OS, completely fine. That's again the perception. What are people apprehensive about and how do we leverage that?
Di: For us, the emergency stop on the top. It is the same emergency stuff that's utilized on fork lifts, that's utilized on industrial printing machines. There's a lot of standardized infrastructure around safety, around machinery that was really useful to leverage and use in the design of the products.
Hamid: Yeah. With these robots, as I touched on earlier, safety is really the name of the game. With physical machines like this, you really have to, even this thing, if it fell down an escalator on top of somebody unsuspecting, it would be a bad day. One of the things that we do with safety is we have to deal with trade offs. The safest robot in the world is one that doesn't move at all, just to take an extreme. Yeah, it's just completely static. Obviously, that's not a viable product. We have this trade off of performance versus safety.
Hamid: I'll go into a little bit of the technical challenges. When I did mobile development and before that web development, one of the nice things was that it was a very discreet and well-defined technical problem. When you went to a server, you got a 404 error, it was really 404 as an integer that was discreet and you got it. In our world though, the world that we live in, it's noisy. Our sensors are noisy. There is a probabilistic approach that we have to take from a technical standpoint, lots of Gaussian distributions. Maybe I'm over here, maybe I'm over there. There's a chance I'm over here and maybe not.
Di: Is this a human, or is it not a human?
Hamid: Yeah. Using this probabilistic approach, we obviously have to make these trade offs in performance, and let me go through some of the sensors that we chose, because we studied this carefully in conjunction with user-centered design. So I'm going to-
Di: First of all, the sensor is something I want to highlight is that to an end user-
Hamid: I'm telling it.
Di: We'll [inaudible 00:30:49] off each other.
Hamid: All right.
Di: I'll be left out. When we chose placement of sensors to an end user, and I don't know if you've seen them, but all of the sensors are almost invisible to the public end user. It's not something that's prominent. It's not something that we highlight. It's not something like, "Hey, look how fancy we are. Steal our sensor."
Hamid: Yeah, and yet the sensors, wow, this one looks works a lot better. Is that why you did this?
Di: No. Anyway.
Hamid: So the sensors were integrated in such a way as to provide better coverage for this, depending on the size of the machine. Again the form factor of the machine was very much in conjunction with user design. One of the sensors that I want to talk about a little bit is this center sensor. This is a what's called a depth camera. You might be familiar with the Connect, for example, if you ever wave your hand and play video games. These are time of flight sensors. There are different types of depth cameras. The ones that we're using for our big robot is a time of flight sensor. It blasts out a piece of infrared beam and then measures how it takes to hit certain obstacles at certain locations. It creates this image where, in this particular visualization, the darker colors are closer and the lighter colors are farther back. Now, one of the things that we noticed very quickly is that you're going to see noise. If you can pause it real quick. Nope, we went through it. Hang out. Let's do it one more time.
Di: [inaudible 00:13:17].
Hamid: Sorry. That's all right. So you see these black dots over here? This is noise. This is sensor. This is right up in your face. Clearly there's nothing there but a straight infrared beam. So with a probabilistic model, we have to account for how likely is it that this is an actual obstacle.
Di: And then just to explain how the different sensors work, LIDARs scan, so imagine a single laser beam going back and forth. The time of flight camera where he's talking about [inaudible 00:32:46] camera, we create a 3D visualization of the world. It's like a burst of light. Same way when you take a picture, and it gets light back, and then it creates an image of the world. Do you want me to take over?
Hamid: Yeah, please. Oh, the other one. There we go. These noises were easy to fix and easy to overcome. But the user experience when you see a different type of noise that you're going to witness here in a moment, one that permanesces, it stays there, is that the robot is stopping for no reason. These types of noises we have to filter out and we have to make decisions about and say, "Okay, how big does it have to be before we tell the robot, 'Ignore that. That's not really there. Keep going.'" If we were too aggressive with it, then the robot performs great, but it might hit things. If we don't, then the robots stop at any second. Let's look at this here. Can you pause it for just a moment.
Di: Yep.
Hamid: This is what's called a cost map. We have a number of sensors. The front depth camera was one of the ones that I mentioned earlier. We also have a planar LIDAR, which is on plane the horizontal plane with the ground, and then we have a slanted LIDAR, which is there to look at cliffs and make sure that the robot doesn't go downstairs or escalators and hurts someone at the bottom. We take all of these sensors, not just the LIDARs and depth cameras. We also have IMUs, gyroscopes that tell us what the angular velocity is and what the acceleration is, and we take all of this and we create a map of the world.
Hamid: This is a top-down view of that map. It's an amalgamation of all of the sensory input where obstacles are marked. What You're seeing here in green is the footprint of the robot, its own image of itself, where it is in the world, and what you're seeing here in green is what it was trained to do. In blue is what it wants to do. It needs to be able to follow that path as well and as close as possible, except for when you have obstacles in the way, in which case it needs to adapt, and it needs to move around. So go ahead and hit play.
Hamid: This happens to be. You keep doing that. This happens to be the exact same replay of that depth camera, the values that we get from that depth camera. Here's the aisle. We turn left, and then later on you're going to see ... You notice there's no noise pixels here in front of it, despite the fact that there was over there. We filtered those successfully. But as we go down this aisle, I think this is a Walmart or something. As you go down this aisle and you're cleaning away, you'll notice down here a spec of noise is going up here, and you will not be able to navigate around it. There it is right there. Right there. Now it's trying to get around it. Maybe I can fit over here. Maybe I can fit on the other side, whereas it was trained to go right through it initially. Now, can you do ... Now the next slide. This is a 3D visualization of that same replay.
Di: We call this a point cloud.
Hamid: Yeah, it's a point cloud of the cost map. I'm going to rotate and zoom around here for just a minute. Again, we're going to make a left turn. We'll see this one that actually takes a little bit longer to do, but we'll see the noise pixels towards the end. Actually, yeah. If you can ... There we go. Here we are. We've turned. We've made our turn, and we're going to move down. I rendered this on my laptop, which doesn't have a GPU, so I apologize for the poor frame rate, but probably would have been better to do that.
Di: At what minute is that going to be?
Hamid: I would say probably about three quarters of the way there is where we're going to see it. There we go. It's starting to come in. I can feel it, man. There it is right there.
Di: Right there.
Hamid: So you see? You see these pixels up here? They're just floating. They're suspended in air, and it's one of the initial sort of lessons that we learned was to be able to make assumptions that things that are hanging by themselves on earth at least are probably ghost pixels, what we call ghost pixels, and you're safe to filter them out. Anyway, that's that on the ... Let's see ... Trade offs.
Di: So at this point, you guys might be wondering, "Okay, point cloud, great. Sounds so great. What does design have to do with this? What does this matter?" So I wanted to show this video right here where you guys see this is the wall. See what happens when I'm tethered?
Hamid: Yeah.
Di: This is the wall right there.
Hamid: Here.
Di: This right here is a big thing, this ominous thing that the robot sees, and it's like, "What the hell is that?" When we first started deploying our robots in the stores, especially on the East Coast, consistently we were getting these obstacles. It's just like, "What is this?" It's like a huge slug blocking the aisle. This is where UX and R&D come together magically. I'm shooting footage, we're looking at the environment, and then at one point I was standing in front of the store, and I look downed down, then I looked up. On the East coast, because it's winter, there's ...
Di: And on the east coast because it's winter, there's giant infrared heaters, invisible to human eyes, blinding to sensor type eyes where it's causing the robot to stop all the time. Because it influences light, so it's like right here there's a body of something that I can't see. Multiply that factor with that during winter time, because it's around the holidays, so in the front of the store, what they do? They bring beautiful, sparkly, shiny ornaments to put there and reflect all that light back at the robot.
Hamid: Yeah. And that, and it confuses the sensors and obviously we have to account for that and no amount of filtering is going to help you with that. That's just a gigantic blob that's on the ground. But fortunately we were able to diagnose it because we would go there. There's really nothing there. And it wasn't until Di had checked it out that we were able to figure out, okay, it's really part of the infrastructure that we did.
Di: And how many ended up solving for that was, okay, well if the robot only sees this periodically at this height, what are the other sensors seeing? And then accumulating all those factors together and saying, "Okay, probabilistically you are safe to move forward, but proceed with caution."
Hamid: Yeah. The other thing was, one of the things that also helped, was to tilt the camera a little bit further up. We had always tilted it down thinking that we would use our depth cameras also to be able to detect cliffs. But because of factors like this, we ended up saying, "No, let's move it a little bit higher so that we can see overhanging obstacles and be able to avoid those as well. And also to get rid of some of the noise.
Di: So we've kept you here for a long time. I know we promised you a demo. There's always a trade off between safety and performance. There's always a trade off between disruption, innovation and then apprehension to change, and I think there's real opportunity here to really factor in, what does it need to design for humans? What is human nature? What do humans perceive? How do we have adoption increase, which also benefits the ROI of business?
Hamid: Yeah. I really believe that with robotics and AI we're at the same inflection point. I just have the same feeling that we're at this inflection point where it's going to become much more ubiquitous than it already is. With AMRs in particular, working around us everyday people. And so we have an opportunity as developers and as engineers and as designers to shape that direction a little bit. We're able to give it a little bit of a nudge.
Hamid: Much the same way that when the internet was in its infancy, we had a very dedicated group of people who had a philosophy that software and information should be free. And of course talking about the Free Software Foundation, GNU, Linux, all of the tools that came out of that, every developer that was out there, they really set the tone for this and they made it possible through their altruism for a huge amount of ROI. Think about how many companies use Linux, how many companies use Amazon which runs on Linux, or Postgres as a database, whatever. And we have that same opportunity with robotics. And that's why I think one of the things that I would like to share with you is a pledge that we urge anyone who wants to get into this field to sign, and that is the lethal weapons autonomous pledge.
Di: There's always two sides to the cap. Where one side it's just like the internet should be entirely free. Technology will advance and people need to and will adopt and laissez faire, whatever happens, happens. Then there's the other side, where we kind of stand on, which is like, okay, as purveyors of this technology we should be responsible about what we're building. Who are we building it for? What is it doing? What is it meant for? And I mean, both of us signed this. The CEO of DeepMind, Google aside, a lot of people have signed this pledge to not build an autonomous weapons, but-
Hamid: Yeah. Not build Skynet. And I know that sounds weird, but-
Di: Don't build Skynet.
Hamid: But I think we can make it just as taboo as say, chemical weapons and biological weapons and-
Di: Don't upset the robot.
Hamid: All right. So, with that, we're going to do a demo. Before we do a demo, who here has a question? Oh yes sir.
Speaker 1: So I'm not really into robotics. Just out of curiosity, why did you guys choose light rather than sonar grams?
Hamid: We actually started out testing with sonar. And now every sensor has its trade offs. Time of flight sensors have their trade offs, structured light sensors have their trade offs. RGB cameras, your standard 2D cameras that you might be able to use for deep learning and classifying and navigating, those have their trade offs in that when you turn off the lights, they can't work. So every sensor has its trade off. The thing with ultrasound is that... Whoa, whoa, whoa. Sorry.
Di: That was me, sorry.
Hamid: No worries. The thing with ultrasound is that it doesn't go very far and it's not very precise in terms of the shape of objects. If you can think about your car when you pull into a garage. If your car has that ultrasound array in front of the bumper, you get kind of this amorphous shape. You don't get a very accurate pixel perfect kind of thing. And that's bad for a cleaning robot because you need to be able to, as I said earlier, get close to walls and really clean around the corners. So we did consider that. The other thing is the advantage of sonar over all of the other methods is that they can detect glass. So when you get close to glass, sonar will definitely work whereas the time of flight sensors will not. It'll just go right through the glass. So we address that with bumpers for example on this one. Does that answer your question?
Speaker 1: Yeah.
Di: And sonar's just really noisy. If you think that every line of code you write requires a lot of processing time, it requires a lot of... I think the robot's doing bad things on there.
Hamid: What are you guys doing?
Di: If you kick the robot it will go backwards but then it will also start scanning the environment to be safe again.
Hamid: You have to kick it in the front, sorry. That's where the battery is. If the lithium leaks, I think we're all in trouble.
Di: You got to stop saying these things when we're speaking and people are filming us, because this is going to end up on Twitter. Skynet building robots. They make bad things out of lithium ion batteries. In the back, you.
Hamid: Question.
Speaker 2: [inaudible 00:44:32].
Hamid: That is already accounted for so the sensor manufacturers themselves are very aware of these limitations and they work with software updates to be able to recognize situations where this happens. The problem is with the edge cases. And those ornaments bring out those edge cases boy, because they are put in essentially a random order. Whereas something that is nicely structured and ordered and predictable like this, we can train our neural networks to classify. Whereas with ornaments that are in all sorts of weird shapes, put down by whoever in an arbitrary fashion, that's much more difficult to do.
Di: The other thing too is if you notice it's not just an even plane of glass where it's all reflection. There's beatings right here and there's also a ledge on the floor, so that helps kind of create this boundary and shapes where it helps it obey the space within the environment too. So the first thing that you just saw right there is one of our mapping methods, where an individual can teach a route that's like, "Okay, I want you to follow this path and follow it to your best judgment, but these are the specific areas I want you to clean." The second demo I'm going to show you right now is if you were to just create a boundary. So you drove it in a big square and you said, "Okay, this is the area I want you to clean." And then the robot's AI will create the best path within that boundary.
Hamid: Just setting it up as you talk. The reason we did this was because this robot is designed to work in large office environments where you have sort of a monotonous task. If you have to teach the robot every single aisle and cubicle that it's going down, it would take you a very long time. And instead we want the machine to do the work, and this was again a user experience [inaudible 00:46:27] where you just have to go around the room that you want to clean and the robot will determine the best path to go.
Di: And then warning: the vacuum is going to turn on.
Hamid: And the vacuum is really loud.
Di: Just so that you guys believe that it actually does vacuum and we didn't just bring a-
Hamid: A cute little robot.
Di: ... shiny, 30kg moving thing. So the first thing it's going to do is drive the perimeter and then after it drives the perimeter, notches the map to what it knows of the environment, it'll generate its own path. Maybe while it's driving around we could do a-
Hamid: Yeah, some more questions over the vacuum. Yeah. Another question sir.
Speaker 1: Yeah. Could you guys experiment too with pressure plates. Like if something from the front sticks out a few inches and has a pressure plate.
Hamid: Well, the bumper is kind of like that. The bumper-
Speaker 1: It's got a sensor?
Hamid: Yes it does. It has a limit switch that essentially you push. We had a question in the back. Yes sir.
Speaker 3: [inaudible 00:47:22]
Di: Should we turn off the vacuum? Okay, you guys know what got here.
Hamid: Okay, go ahead.
Speaker 3: [inaudible 00:47:29]
Hamid: Sure.
Speaker 3: [inaudible 00:47:34] is happening so what do you think is the biggest kind of [inaudible 00:47:38]?
Hamid: What do we think the biggest untapped market is for mobile robots?
Speaker 3: [inaudible 00:00:47:47].
Hamid: Well, if I knew that I would be starting a company that did that and I wouldn't tell you about it. No, we...
Di: Maybe I can help [inaudible 00:47:59]. This is [inaudible 00:48:02], purely based on opinion, no further AI. I honestly think when it comes to hospitality roll outs, especially whether in an airport or in a hospital space, when we look at industries where people's attention and their tasks are the most strained, I mean oftentimes... I have friends that are nurses and they're not spending most of their time tending to patients, but they're bringing things like juice, extra water, extra blankets. So what if we could have an automated system that could deliver that to patients and then free up nurses' times to attend to patients.
Di: Same thing I feel about levels of education in a library system, accessing books, getting data, getting stuff that you need for school, finding out where your classes are. I went to UCSC and I spent the first two months just trying to understand where all my courses were and where my classes were. It's like guided robots, things that require [inaudible 00:49:00].
Hamid: These are great ideas Di, I can't believe you haven't shared them with me. I would've started a company with you doing these. This is great. Wow, look at this, huh? Yeah.
Di: Any given Sunday, any given Sunday. We would show this but the questions are a lot more engaging and [crosstalk 00:11:16].
Hamid: Yes sir. The beautiful hair I love your hair, man. That's awesome.
Speaker 4: Thank you. I've still got it. Do you think there's a reason why it's not being marketed that way? Instead of it complimenting someone's job, it's more of a hostile takeover. Especially in the blue collar section where it's like robots are replacing the workers, whereas you guys are kind of favoring it to complimenting.
Di: So, we have a position in one way. I don't want to get all Donald Trump like fake news and screw the media. I think in some levels, to be honest, it is displacing certain tasks. Certain tasks will be automated, but we also have to elevate and really highlight, "Okay. Well, if these tasks are automated, what other tasks can be brought to the forefront that leaves people more time to conduct?" When we were initially doing research with janitors, especially in Japan... So in Japan where this was the target market for, they have a decreasing population. Most of the population is going to be seniors and retirement. The introduction to retirement was through the janitorial industry. So you have a population of people that are aging that in a janitorial setting, sanitation profession, it's really, really physically hard to do a lot of the tests that they have to do.
Di: So we've kind of looked at the space and said, "Okay, that's what one task we definitely can help alleviate so that they can get through to other tasks." They also think it's more of this design development conversation like this that needs to take place. Often times these conversations are designed in a silo, talking about how you should design ethically and then engineering talking about technical feasibility and looking to the future. But I don't think there's been many conversations with those in combination yet, and it's just starting. It's new.
Hamid: Yeah. And these robots aren't going to be able to replace a human and yet. It's a long ways away. You still need operators to help this thing out. They still call for help if they get stuck. What it does do is alleviate sort of the mundane tasks that someone is supposed to do. And that's really our goal here is to be able to elevate the level of tasks so that these guys... I've been on deployments at three in the morning at Target. Boy these crews come up from 3:00 to 12:00 PM they work, and they're making sure that the store is ready for all of us to go shop. And it's hard, it's grueling, and they have to make certain deadlines, especially before the store opens. So if they've got something that's helping them do this while they re-shelve, while they put out new inventory, it's fantastic for them.
Di: You had a question over there, sir.
Speaker 5: Yes. In terms of educational surveys and studies to consult for [inaudible 00:52:04] build your robot and get the right educational transcripts, information [inaudible 00:52:13].
Di: In the beginning a lot of it was spent doing the jobs and working alongside, and really talking to the people in this industry that we're disrupting. Bringing people in that created the manual versions of these machines that we wanted to operate. Understanding the impact. Understanding what is it that people actually needed or wanted. What were people are afraid of? And even initially when I bring in the robots for testing from [inaudible 00:52:43], people still say, is that robot going to take my job? And I go, "No, it's not that smart." You know, I wish.
Hamid: Yeah. It's like Moravec's-
Di: Not I wish, but... such a taboo. [crosstalk 00:52:53] But I think we take for granted how complex humans are. Just for us to stay balanced and walk downstairs, the amount of neurons firing just to make all this happen, it's extremely complicated.
Hamid: Yeah, it's a thing that was posited actually in the 80s called Moravec's paradox. This guy Moravec said, "The robots are coming for all the high level tasks, so the accountants and the stock analysts should be the ones that were worried because that's what AI is going to replace. Those really highly complicated things are very easy for them to do. But the toddlers who can stand up and navigate an environment, a tremendous amount of effort has to be made technically to make that work."
Hamid: So it's a weird thing in artificial intelligence, and intelligence systems in general, that things that we take for granted we don't even think about. They're completely natural or so difficult to actually program and implement.
Di: Yes, sir.
Speaker 6: I have a question about the three different types of systems that you showed earlier, the floor scrubbers, the auto [inaudible 00:53:57] platform and the vacuum. Assuming that you're actually selling, you've partnered with a lot of different OEMs of floor scrubbers and I think that the whole [inaudible 00:54:11] lots of OEMs for things like pallet jacks, like you want to adapt your own delivery platform and make things like that. But with this was this developed specifically with SoftBank, which means we're not going to see similar versions from other companies?
Di: This was developed in partnership with SoftBank first, but our hope is that, I think from green side what our hope is, we're not in the business of telling people what kind of robot they should build. Our position is we're going to empower the builders, we're going to make great navigation software, you decide what you want to build with it. And it just happened that SoftBank came first and said, "I think that there's a market for a small vacuum that is not quite the size of a Roomba but like a little bigger that could clean more spaces."
Speaker 1: But if another company came to Green Club and said, "We want to make something like that", are your hands tied to a certain extent, like SoftBank has some [crosstalk 00:55:02]
Di: Maybe not exactly like this, but if it was a different form factor, a different function. Those conversations are always ongoing. And that's the hope is that we can partner with different OEMs, different companies, different types of builders, and eventually open it up to different people to make different robots. And we talked to our CEO, and since the beginning when I joined four years ago, he was like, "I want to see robots everywhere." But not in that tone. More like, "I want to see robots everywhere." Today, I swear. I am so stoked with how many people stay back. This is really cool.
Hamid: Yeah. Thank you.
Speaker 7: Do you guys mind if we grab a selfie with you?
Hamid: Yeah, let's do it. All right.
Speaker 7: We've been wanting to do this for so long.
Hamid: They're going to flip us off in the selfie as we're doing this.
Speaker 7: Okay. One, two, three. Thank you guys.
Hamid: Thank you guys. Thank you. Thanks for coming out. Thank you. Thank you, Rachel.

Want invites to cool events and things?

Boom, Newsletter sign up form.