Speaker 0 00:00:00 At a summit, we are creating human, like emission for robots.
Speaker 1 00:00:08 Hello everyone. And welcome to the <inaudible> robot industry podcast. I'd like to welcome the listeners from all over the world from Kalamazoo, Michigan, Teaneck, New Jersey and Myrtle beach, South Carolina. And I'm thrilled to have Henrik Olson from vivid. And if you don't know Henrik, or if you've not heard about vivid, they manufacture one of the leading 3d cameras, which enables robotics to do the things that they do from picking plays to Ben picking and machine tending. For some examples, ZipID and Henrik are located in Oslo Norway. So let me tell you a little bit about Henrik. He's an experienced founder with a demonstrated history of working in the industrial automation industry. His leading 3d vision companies, Vivek was from its incubation at SINTEF, which is one of Scandinavia's largest independent research organizations to a fast growing company in the 3d robot and automation field internationally he's seasoned and award senior researcher in machine vision, pattern recognition, 3d cameras, optics, robotics programming, and parallel processing. He graduated from Norwegian university of science and technology called NTNU in engineering, cybernetics, and robotics. Thanks Henrik for coming on the 83 robot industry podcast. This is going to be a bit different because much of what you supply is kind of on the visual side, especially someone on a webinar or YouTube or at a trade event. So I'm looking forward to the conversation.
Speaker 0 00:01:37 Thank you. Thank you for that nice introduction and thank you for having me. So it's going to be great to talk about, uh, the thing I'm so passionate about name, the name <inaudible>. So yeah,
Speaker 1 00:01:47 Before we dive into the industry and the tech, can you answer a few questions for those in the audience that may not be exclusively vision experts and they might be headed on a walk or, or, uh, at the gym or whatever. So I often, so we're going to have to visualize some of the things. So my first question to you is can you explain what a 3d camera is and why is it different from a two D camera?
Speaker 0 00:02:09 Sure. So, um, yes, I will try to do it, uh, very visually without an English, not all here, but otherwise. Yeah. Um, so think of it to the camera. We all know that, right? So it's, um, it's, uh, that's a flat representation of the world. Uh, so you have the, um, the X and Y data, like, uh, a plane. So to say, so, so that is actually, uh, it is a projection, so you'll stand and take an image. And that is a projection of what you see. So in a 3d camera, um, we introduce that depth dimension. So if you think of it for every pixel, you also have the depth, the data, the distance from the camera origin, uh, to the objects that you are imaging. Um, and in that sense you have X, Y and Zed, which, where the Zed represented depth. It's more of a true representation of what you actually see it because we, as humans, we see in three dimensions and the 3d camera, uh, gives, uh, a picture that is represented in three dimensional.
Speaker 2 00:03:21 Thank you for that. Also, I'm going to ask you to explain to the audience, because I think it's important to have a podcast who may not know what is a point cloud.
Speaker 0 00:03:29 Yeah. So, so that is what we called the picture, um, the 3d picture. So a point cloud is done points in space. Um, so it's a digital representation. And if you think of the image we talked about earlier, uh, for every pixel in the, in the sensor. So to say, um, instead of just having the, the color information that you would have in a regular 2d color image, you have also X, Y and Zed coordinates, and those coordinates are the distance and positions of surface points of the objects that you are imaging. So together, all of these points represent a point cloud. And that point cloud is then, um, all these pounds are on the surfaces of all the objects that you are taking a picture of, if that was understandable.
Speaker 2 00:04:25 No, that's a great, great, um, uh, explanation. Thank you. And I, I was also going to ask you another kind of basic one Oh one 3d vision question about where is the vision algorithm created and how do we make it better?
Speaker 0 00:04:39 Yeah. So that is a, is a good question. So, um, of course there's several types of algorithms. So to say, um, one thing that, that is what we do at saver, this is, um, the vision algorithm in order to create this point cloud. So, and then you go into the measurement technology on how to do 3d imaging and so on. And that is an algorithm itself, which create a point cloud as an output. And when you have the point cloud, then it depends on what you're going to use that point cloud for, uh, you need to do, uh, make an algorithm, uh, to do that. So let's say you want to pick objects, you have a robot and you want to pick objects, and then you have a 3d camera, for instance, from CVS or some others to take an image, you get the point cloud, and then you want to try to detect objects in the point cloud.
Speaker 0 00:05:30 And you can for instance, have a CAD model of the object. And then you search in the point cloud, try to match that CAD model with the points in space there. Um, and if you get the match, then you can tell that match to the robot and it can go in and pick it, that algorithm to do all of that is, is a vision algorithm. Um, and it's kind of representing the brain, uh, of humans. You know, we, we see the world or I see the world, or then we have the brains that tell them that hand to go and pick something, tells it where it is in space and so on. So, so it's several algorithms, but, um, yeah, we, we, at CVA, we work on creating the best point clouds and then our customers, again, they, they, um, utilize that point child to do their so-called vertical integration, whether it's picking assembly, uh, dimension control, uh, inspection or guiding of robots or whatnot.
Speaker 3 00:06:29 And so how did you get started in the 3d vision space? Like you went to school, you, and this was part of your, um, education.
Speaker 0 00:06:37 Yeah. Uh, it was, I, I, uh, did a robotics education and I got, uh, pretty early. We had, um, a project, uh, um, we were seven students that went together to create a robot that was competing in, in this, um, uh, real competition in France. Um, and they had to make, um, a robot that was going to do that company is now, and it was bursting balloons. I was like two robots competing. Um, and I was, uh, me and a friend. We did a vision system and the brain, so we had to, and then I got really inspired, uh, in that field. This is early days, it was in 2000. Um, then after school, I chose to, um, start working in SINTEF, uh, where I joined, um, the mission, uh, group there. Um, and we worked very closely with the robotics group in the same in research Institute. So I was, uh, actually part of building up a group that was particularly oriented what robot vision. And, um, yeah, we, we worked with all types of 3d cameras, histones, like time-of-flight or laser or steady, there'll be a Chanel, also structured light, which is a core technology behind the civic camera. And, yeah, so it was around 15 years of research before we started, um, Civit me and a colleague in 2015
Speaker 3 00:08:11 And your motivation was to make the best 3d camera available.
Speaker 0 00:08:16 And we worked, uh, actually we, we did lots of, um, the unlicensed part in regards to the division algorithm we talked about. Uh, uh, so, and then we bought 3d cameras, um, or, or systems from, from the net. And it was always a little problematic because, uh, the quality was not asked you that we wanted, of course there existed some really high-end systems, uh, like the metrology scanners and so on, but that's also extremely expensive and they were very slow. Um, so mostly we chose cheaper, uh, camera assistance, which is pretty normal to do in, in research or in, in, um, in, um, nurses. Um, and then also we were inspired by the connect. The first connect came out at that time. Um, um, and we saw that there was, there was a need for a better 3d camera. And we had during different projects, I worked for a time for, for ASI NASA, uh, creating a time of flight camera, uh, that was, uh, going to be used in space, uh, things like that.
Speaker 0 00:09:22 So we had explored different technologies and follow the development for a long time. Um, and we saw the customer need, like our industrial clients. They came to us, asked if he could do this kind of inspection or picking task or whatnot. And, uh, and, um, we tried different cameras. Wasn't good enough. And eventually we started out to strategic project where we wanted to create, um, a new camera that kind of unifying the world of the high-end metrology scanners with more of the low-end Kinnock dish type of system that was fast and affordable. So we could get high quality really fast and to an affordable price such that it could open this field of robotics, uh, within material handling that we saw were emerging. So now we see that, um, uh, several companies CBD included is offering these solutions now. And we see there is, there is a wave of automation happening on top of these cameras in regards to peaking placing assembly on and yeah. All of these tasks. So, so that was, um, the start.
Speaker 1 00:10:34 Well, that's great to hear that history. So w now in your, in your work, in the industry, do you work with integrators or with, uh, like OEM customers or is it a mix of both?
Speaker 0 00:10:43 So it's a mix. Um, it depends, but since we, since we, uh, only provide the point cloud, so I'd say the image, uh, then our customers, mainly that they have competence in doing the other, which now algorithms, uh, the detection of object or, or the vertical integration. So, so, uh, whether it's be a system integrator or, or an OEM that creates, uh, an off the shelf system for, for some sort of task, um, we are done, uh, a supplier of 3d cameras to, to, to those customers. So it's very similar to, to the 2d camera industry. They legislated to the camera manufacturers and they, they sell their cameras and people use them for different tasks, but the two, the manufacturer doesn't sell to the end customer. So let's say it sells via some, some integrator or where I'm watching take care. So that's, that's similar for us.
Speaker 1 00:11:44 What are some of the challenges that your customers then your manufacturers and integrators are facing? Is it ease of use reliability speed or is it just the pick?
Speaker 0 00:11:54 Yeah, so that, that's a, that's a good question. And I'll also say a huge, uh, topic. Um, but also of course, why I think, uh, this, uh, automation is so exciting. Um, it's, um, it it's so, so many challenges, but, uh, the thing is that it's, it's really frustrating also because this is easy for humans, right? To, to do, to stand in front of a bean and peak object from a bean. Everyone can do it. It's, it's really simple for us, but for a robot, this is ridiculously hard. Um, um, and there's lots of reasons for that. Um, and it's about, um, uncertainty in all aspects. So the uncertainty in the vision system, in the gripper, in the motion planning and the robot, and of course the complexity or creating a software brain to control all of that. Uh, so, so all of that together needs to, to, to work in order, uh, for, for such a system to be robust and, uh, and can be, uh, actually scaling in, in the industry.
Speaker 0 00:13:03 Uh, so a lot of this is, is done in research and there are systems out there already. Um, but, but it's not, uh, it's still in the early beginnings, I would say, um, in regards to human level capabilities. So it's just the, the sheer complexity of all parts involved. And, and we are focusing on one of those complexities and that is, uh, the 3d camera itself. Um, and that ability to get, um, good data on all types of materials and surfaces. So, so that you can image all those millions of skews, you will meet in the warehouse or in the, in the manufacturing plant. Yeah. Because, uh, the cameras out there today, you also see everything there. We can't see everything, but we can see a lot. And, uh, and that's what we working on, uh, to, to improve all the time, um, and get more and more and, and make those robots capable to, to give them this human, like, vision that we set out to do to do.
Speaker 3 00:14:07 And so by good data, you really mean good, good, good vision, good pictures. Yeah. And you've said that pick and place is not solved, and that's what you mean, right. Is that this is, yeah,
Speaker 0 00:14:17 It is. I mean, just, just take, take a look at a logistics industry. And I, and now in the last year, uh, of course, due to COVID as well, um, there was a huge, uh, rise in, in, um, online shopping and, um, Amazon alone, they hired 400,000 people to, to, uh, to do peaking in the warehouses. And if this was sold, then of course it would be loads or robots doing that work. Right. Uh, clearly it's not because, uh, we are, um, we are putting people steel. So also in the logistics industry, the, the, um, the, uh, the automation, the intelligence sticks, that's, that's come pretty far with, with, uh, the moving of, of stuff inside, you know, the, uh, shelves are, uh, they're small robots moving the shelves, or they're picking from small boxes and, and, and so on, but they moved that to a human picker that, that takes them.
Speaker 0 00:15:22 If you have ordered something online, the human picker picks, um, maybe it's a, you're a cell phone and some cables keep big start and put it in the package packet and send it. So, so that part, um, the industry is now going hard on, on automating that part. And, um, and then, um, yeah, w you wouldn't need better vision because when all those robots are going to do the human tasks, they need to see more like us because we have ridiculously good lesion. Right. So who wouldn't, we wouldn't, um, go around with and accept, um, at worker that almost not seeing, uh, can't pick 60% of what you have in the bin. It's like, it wouldn't work. It's, it's, it's, that's not a sustainable business. So, I mean, there needs to be, uh, improvements there. And, um, um, we're not there yet, but, uh, we are up the good start and, um, yeah, lots of things happening. And, uh, and of course a lot can be already automated, but, uh, but, uh, there's still a long way.
Speaker 2 00:16:38 So you've got both hardware and software in your life when it comes to vision. Is there one that's harder than the other, like, have you solved the hardware's part and you're working on the software part? Yeah,
Speaker 0 00:16:49 No way. Uh, I think these things goes hand in hand and, um, um, when you're have been in the industry for a while, you understand that it takes some time to, to, to call something solved. It's, there's always, um, uh, things that can be improved. So I think, uh, uh, in regards to hardware, some of the things that we have worked a lot on, um, which is, uh, they are very complex and unimportant, uh, for, for high-end industrial, uh, three division is, is these things around, uh, thermal and mechanical stability changes, um, that happened you to like, take a warehouse again, eh, in, in nighttime, uh, the temperature goes down and in daytime, maybe it's a different parts of the world can be up to 40 degrees in the warehouse. Uh, if it's next to open door, where, where, uh, in the winter time, cold air is streaming in every time a truck goes by things like that.
Speaker 0 00:17:53 So, so then, and these things changes, everything changes, as you can see there, this is atomic forces. Uh, so the lens change, the, the, um, account exchange, everything changed and then being a high-end calibrated system and accurate system, uh, you need to, to be able to cope with these things. Um, so that's something that I've worked a lot on and that's, uh, that's your hardware. Uh, and then you can add software on top of it, in order to, to, as we do, we have so-called floating calibration, where we measure everything that happens inside, and we try to adapt based on, on, uh, yeah. Uh, readings from sensors inside and so on. So, so that is a, and then of course, we need to build, um, smartness on top of the camera to, to understand what is noise, what is real, uh, that's something we have worked a lot on in regards to shiny and reflective surfaces, which also is a, it's a big problem.
Speaker 2 00:18:50 I watched the, uh, your YouTube. It was very, I thought it was really good for, I'll ask the listeners to check that out. Maybe later on after the podcast, I was going to ask you a question about, uh, the glass, the optical glass in the camera. Of course, you got two lenses because you're, uh, you're recording in 3d. And how important is, is the glass in your camera?
Speaker 0 00:19:10 That's, uh, that's interesting. We have, um, we, we apply a sister, um, measurement principle called structured light. So that means, um, in contrary to, to StereoVision with maybe a mainland know about, uh, which is also what we have as humans, you know, the two eyes, um, by the principal instructor light, one of the other, the lenses, uh, is from a projector. So there's projector behind the lens. Um, and if you think of a projector, it's some sort of an inverse camera where instead of, uh, receiving light, it's sending out light. So we sending out light, uh, um, since it's structured light where, uh, it's coded, so we know what we are sending out, and then we'll look in the camera, which has a baseline and a projected difference there. Uh, and then we can look at how that, um, projection changes and then it can infer the 3d information.
Speaker 0 00:20:09 So understanding, uh, understandably since we are, uh, projecting and we need a very high accuracy, uh, it sets a lot of demands for, for the, uh, the optics and the glass. And also in regards to what I talked about earlier about thermal and mechanical stability, we need to have ruggedized, uh, systems. Um, and so we put a lot of effort into the design and, uh, the village, of course, in the detail there, how to make, um, yeah, how to make a system that can be stable B uh, industrial grade and usable lead for, for, uh,
Speaker 4 00:20:49 In an industrial environment for years and years.
Speaker 0 00:20:51 So, so, uh, the glass is very important. Absolutely.
Speaker 4 00:20:55 Can you tell the audience what you mean by true to reality? Yeah,
Speaker 0 00:20:58 I'm sure that that is a, it's a hard one, but, uh, I'll start with, um, I mean, if you think of the, um, the, um, mathematical foundation, so to say, um, we have a term called a currency, and I think it's something that is, um, is confused a lot, uh, uh, when you hear people talk about it, but I currency has two components, and that is one component, which is precision, um, glom that is true in us. And procession is about the noise. Uh, or if you take a surface and it images surface, there will be some random noise on the surface. Like not every pixel is perfectly placed where it should be. It's a little over the surface, little under, you know, there there's there's noise there. That is the noise you can have when people talk about now that talk about like, this is like this blurriness or, or, or a graininess of an image.
Speaker 0 00:21:53 So, so in 3d, that's kind of like the pre-session or, or how, uh, when they take a lot of images, uh, how close to each other are each points, and then you have trueness and trueness is about, are you representing the reality? Correct. Uh, so, so that means, uh, sizes, artist sizes, uh, exactly what they are in real life, uh, are the rotation correct? Uh, is the absolute distance, because you measure always in 3d, you measure from an origin and that's inside the 3d camera. And then what is that correct? Absolute distance to the object that you are looking at. So all of that is inside the trueness. So if you have good trueness, you have a, is there truth to reality? And then of course, the combination of, of the trueness and precision together gives you a currency. So then that, uh, that is low noise and high trueness.
Speaker 0 00:22:51 So this is very important to understand when to use a camera and what sort of applications, um, and a good example is, um, the first connect that came out, it had the, it wasn't so good, the quality or that data, but what it was made for, it worked quite okay for, and that was guests to recognition in front of your TV. And in that particular case, what you want to do is you want to recognize a gesture, and it doesn't matter how much, I mean, if your hand is at 1.1 meter from the camera, or 1.2 meter, it's not so important, it's more important that you see what you do. So, so, and then that is the precision that you can actually distinguish the fingers, or, uh, but you're not placing them right in space, but immediately when you introduce a robot and you want to use the data to interact with the objects, of course, trueness is very, very important because if you see an object and it's placed a little wrongly, maybe it's a little to the side and a little bigger or rotated, uh, than in reality, then you can crash when you try to grip it.
Speaker 0 00:24:00 And that, that is a huge problem, actually. So, so a true reality means that it's correct in regards to all of these factors.
Speaker 2 00:24:11 That's great. Thank you for that clarity on that. Um, I see from some of the videos that I researched, uh, in preparation for the podcast, I see the vivid, uh, mounted on a robot arm. Is that kind of the natural state, is that preferred or are you just seeing a lot more of that application?
Speaker 0 00:24:27 Yeah, that's something, um, it's, uh, it's used sometimes, but mostly, uh, the industry has been, um, mounting cameras stationary, um, because it's, um, it's kind of a little more straightforward to do that. Uh, the thing with alarm is that it gives a lot of, uh, appealing benefits if you can, uh, can do it. Uh, so one thing is that you can get closer and a lot of the problems, um, with, with, uh, 3d in particular is you want high resolution, you want high quality images. Um, you want it over the working range, you are working out. And if you have a, if you have a long working range, then, then, uh, that the degradation is pretty quickly going, uh, worse, uh, based on distance. So if you have it on arm, you can always have the same distance. So to say, because, uh, you, uh, whether you do a deep palletization task, you take the image at the same height.
Speaker 0 00:25:30 Even if you have taken the first layer of boxes, you go a little closer to the next image and so on. So, so you get closer, then you don't, uh, so annoyed by the ambient light and, and you can minimize the occlusion, the shadowing, because you can see from different sides. And, uh, it's also very economic that you can use instead of having one camera at the peak and the place position, um, or yeah, you can use the alarm camera to do everything. So why aren't the industry using it more because it's also complex. Um, so that's something we have worked a lot on and now we're really starting with two cameras. Uh, we have reduced the size significantly. Uh, so it's more easily mountable. It doesn't, uh, um, restrict the motion. So the robots are March, uh, um, uh, it's, it's also a mechanically stable things like that, but it's important when you have maybe vibrations or things happen when you actually rate and so on the robot.
Speaker 0 00:26:29 So, so I think it's, um, if it was, uh, more available, the industry would use it more. And that's what we are trying to do, uh, because it has so many benefits, um, and just think of mobile manipulators that are coming more and more, uh, having an, um, the, the flexibility, when you have an arm camera to like peeking from shelves, you couldn't Mount a station in camera on all the shelves, right? So you, you, you need to have some, uh, on the robot moving around and then the flexibility of looking at different shelves and so on. So, so I think, uh, it's something that we will see more and more often. And we at CVA, we believe so much in it that we have, uh, positioned us to, to deliver those kinds of system to the industry.
Speaker 2 00:27:15 So that's one of your focuses on the autonomous robot industry as well?
Speaker 0 00:27:19 Yes, I, I, yeah. Um, so that, it's about the flexibility and the, uh, what you can do with it and, um, um, yeah, all this, uh, factories of the future, uh, where everything changes is dynamic. Um, and so on then, then, uh, there's this stationary, sepsis is a, they have their beauties, but, but, uh, um, they are not as flexible.
Speaker 2 00:27:48 Well, I think the future is mobile that's for sure. What, what, what are some of the challenges that still remain in the industry?
Speaker 0 00:27:55 So there still is a, like we talked about, uh, of course, uh, this, uh, thing about 3d quality on all types of materials and surfaces, that's, that's a biggie, uh, in general, uh, there are objects that are not possible to image as of today. Um, especially transparent that that's, um, kind of like this Holy grail thing to, to do, uh, but still, uh, something we have worked a lot on is shiny and reflective. Uh, we have come pretty far on that, but there's still challenges. And then of course, um, in general, this thing about, uh, that you want, um, a big working area distance, and you want high resolution and great image quality, and, and typically a bigger field of view makes it harder to achieve all of this. So you want like super great quality and you want it in a smaller and better package. So it's, um, uh, so, um, and then if you put on nine on top of that, you know, then, then, uh, that's some of the challenges.
Speaker 2 00:29:04 So those are kind of those things that we've all been dealing with. Everybody wants it smaller or bigger.
Speaker 0 00:29:08 Yeah. Like better, smaller, bigger, faster, better. Um, um, and you kind of hitting your head into some of the, uh, physical challenges, limitations. And, uh, so there's lots of innovation going on and then, uh, ways to get around it, but it's, it's, it's rather lean in improving. What we see, uh, going forward is, is, uh, more on arm, uh, more, better quality of the vision, uh, as in that's an important factor. Uh, and that also one thing I've seen though is also this thing about, um, with CA inflict some more high end, uh, industrial grade vision system and AI combined, um, because there has been, there is a lot of AI going on and I think that will be the future. Uh, it's still early, early for that in, in regards to picking and placing. But, but, um, if you take our, uh, yeah, ourself, we, we, uh, we have great vision in, uh, in combination combination where with our brain. So I think, uh, we need to offer the robots the same, you know, we need to give them great vision as well. Um, uh, so, so I think that that's something that will be important for them to be able to do this in a virtual picking. We want them to do
Speaker 1 00:30:34 Henrik. Thanks for coming on the podcast. Um, if someone's interested in getting in touch with you or learning more about <inaudible>,
Speaker 0 00:30:42 So the easiest would be to just contact us on a cvs.com. Uh, we have, uh, like a contact data and so on. And of course, if you want to get, hold on me, I'm a pretty active on LinkedIn, so you can contact me there, or, yeah.
Speaker 1 00:30:59 Great. Well, thank you. Our sponsor for this episode is Earhart automation systems, Earhart builds and commissions turnkey automation solutions for their worldwide clients. With over 80 years of precision manufacturing, they understand the complex world of robotics, automated manufacturing, and project management, delivering world-class custom automation on time and on budget contact one of their sales engineers to see what Earhart can build for you and their
[email protected] and Earhart is spelled E H R H a R D T. I'd like to thank and acknowledge our partner. <inaudible> the association for advancing automation. Eight three is the umbrella association for the RIA, the AIA MCMA and three Mexico. And these four associations combined represent almost 1300 automation manufacturers, component suppliers and systems integrators, and users, research groups, and consulting firms throughout the world that are driving automation forward. I'd like to thank and recognize our partner painted robot painted robot builds and integrates digital solutions.
Speaker 1 00:32:06 They're a web development firm that offers SEO and digital social marketing, and can set up and connect CRM and other ERP tools to unify marketing sales and operations, and
[email protected]. And if you'd like to get in touch with us at the robot industry podcast, just reach out to me, Jim Beretta on LinkedIn. We'll see you next time. Thanks for listening. Be safe out there. Today's podcast was produced by customer traction, industrial marketing, and I'd like to thank my nephew, Chris gray for the music, Chris Colvin, audio production, my partner, Janet, and our partners. Eighty-three painted robot at our sponsor Earhart automations.
