Following the piloting process of Advanced Manufacturing Labs for H/VET through the Collaborative Learning Factory (hereafter CLF), the EXAM4.0 partners we have piloted 16 technologies embedded in Industry 4.0.
The following image shows the overall structure of the piloting process.
Labs for Advanced Manufacturing-CLF
The present report is the one out of 16 I4.0 technology described within the “Industry 4.0 technologies in labs” section, specifically on #3 Virtual Reality (VR) and Augmented reality (AR)
DEFINITION AND APPLICATION OF VR/AR IN INDUSTRY
Virtual Reality, VR, is a technology used for simulation, creating a different vision and perception. In VR, the user experiences a virtual world by wearing a head mounted display, HMD. The virtual world, a 3D artificial environment, replaces the real world and creates a new reality for the user. VR is beneficial for education in many aspects, it is for example possible to access environments that effectively can transfer knowledge or accelerate the learning process. The following picture is an example that displays the view, the artificial reality, in a VR-HMD. This application is created at Curt Nicolin Gymnasiet and it is a virtual version of a lab at the school.
Figure 1: A VR-application created at Curt Nicolin Gymnasiet. Source: Curt Nicolin Gymnasiet
AR, Augmented Reality, does not create a new reality in the way that VR does. AR is a set of addons to human senses that provides an extra layer of information, thus expanding the user’s reality by, for example, adding artificial objects or information to the user’s environment. AR can, just like VR, be utilized through a HMD, but it is also commonly used with other equipment such as smartphones or tablets. The following picture is an
example of the view in a pair of Augmented Reality glasses, an artificial arrow extends the reality and gives the user information regarding which tool should be used.
For industries, the use of those technologies provides several advantages; it automatizes new workers’ first steps with virtual reality (VR) tutorials for instance. It can help in everyday support for newcomers using augmented reality (AR) tele assistance and it is the easiest way to implement digital twin technologies in the higher levels with mixed reality interfaces (AR/ XR). There are some new tools and applications available within AR that are beneficial for both industries and institutions. Following are some examples of beneficial Augmented Reality features.
- Looking at 3D-objects digitally
- To get assistance virtually (which reduces traveling, thus expenses)
- Using simulations to show products
- Have digital meetings with colleagues or customers
Regarding industry, four main applications can be described:
- Training: So far, the new worker starter pack was composed of a mobile and a laptop but more and more industries have developed training programs using VR headsets. For instance, Accenture has acquired 60.000 new HMD for training and they will become part of the welcome pack. The purpose is double, on the one hand it will be used for training, on the other hand, it will be used for going deeper in the metaverse
industry.
Figure 2: View of an AR user in Curt Nicolin Gymnasiet’s lab. Source: Court Nicolin Gymnasiet
Remote collaboration: Using virtual meetings but in VR worlds. Many enterprises are developing a user-friendly and easy solution for meeting and collaboration. Recently Facebook has invested more than 10.000.000$ just for creators of the new Facebook platform called Horizon. With this kind of platform the meetings will gain more human interactions and we will be able to see all kinds of kinesthetic reactions.
Figure 3: Workroom meeting in VR. Source: https://www.cnet.com/tech/computing/virtual-mark-zuckerberg-showed-me-facebooks-new-vr-workplace-solution/
Assembly industry and real time tutorials. AR/XR HMDs are perfectly able to recognize the environment and its objects. Even more, they can project over that object extra information that could make the installation of difficult parts easier. This is the case of the T2AR program in NASA where astronauts follow the instructions over different and complex maintenance of the ISS installations.
Figure 4: Astronaut performing routine maintenance. Source :NASA
Warehouse logistics: whenever we talk about AR we immediately think about HMD but the most extended hardware with AR (and XR) capabilities are not HMD but mobile phones and tablets. If those devices are ARCore and ARkit compatible, they can show AR scenarios. Warehouse logistics have developed, using these simple tablets, several apps where even the non-expert user can simulate how any piece of furniture would look in a certain spot at home. If we go further, using VR simulations we can virtually enter any place, any house, any facility without even having it developed or constructed. Underneath we can see an example from AR furniture.
Figure 4: : Amazon room showing how a chair would look like in that room. Source: Marketing4commerce.net
VR/AR in HVET/VET labs
VR/AR and XR provide the ability to enter in a virtual scenario that can be shared among different actors for collaborative learning.
Most of the time in VET and HVET facilities there is a teacher that has certain knowledge about a topic that could benefit the VET and HVET students in different cities. Sharing those skills would require different approaches that vary from using just a simple slide show presentation (which is not very effective) to using any of the video meeting tools.
First of all, the breakthrough of VR is the ability to have meetings around a 3D animated object in real time, with people from different cities. This provides the perfect environment for bringing third party actors to educational environments. We can easily ask any facility technician to share his or her skills with different teachers and students in real time, without moving from the workplace and using ad hoc modelled 3D objects.
Secondly, the AR tele assistance is being used for showing real life equipment from real life workplaces without moving the students or putting them in dangerous environments or situations.
Finally, the mixed reality ability to interact with AR objects in real life provides the perfect learning environments for teaching and showing real time effects over real life objects.
Again, the tele assistance provided by most of the XR and AR head mounted display (HMD) leads to a better collaboration from people of different locations.
We have defined three levels of Virtual Reality education within VET/HVET labs.
- THE FIRST LEVEL: It is to learn how to handle the equipment and operate the interface. Examples of this could be how to install the physical VR equipment in a LAB, run updates and troubleshoot problems etc. This ensures that the students could work independently with this technology in the future. These are basic skills to move on to the next level.
- THE SECOND LEVEL: It is to use applications within VR as a learning methodology instead of other more traditional methods, such as reading books. An example of a beneficial feature is simulation. These simulations can, for example, represent production processes and instructions or information. There are also a lot of games that are relevant to different educational objectives available on the market. An example of
valuable games is where the participants can work in a team to solve problems and hence improve their collaboration skills. These applications and simulations contribute to the acceleration of learning. Students that use these applications get a lot of valuable education such as hands-on learning, with machines, before actually working with eal equipment. This reduces the risks that arise when working with real machines and
equipment. - THE THIRD LEVEL: It comprises learning on how to create games and applications for VR. This is currently done in education at Curt Nicolin Gymnasiet. The software used for this purpose, at CNG, is called Unreal Engine. This software has been used in order to create multiple VR-applications, including the one displayed in the first picture of the report. We have defined two levels of Augmented Reality education within VET/HVET labs.
- THE FIRST LEVEL: It is to learn how to handle the equipment and manoeuvre the interface. This level is almost indistinguishable from the first level of VR education. The first levels comprise the knowledge on how to use the interface of the AR-equipment, install updates, troubleshoot for problems etc. It also includes the skill to be able to find and install applications that are relevant for different purposes. This technical expertise is vital in order to move on to the next level.
- THE SECOND LEVEL: It is to, more comprehensively, integrate AR as a part of the education. The following are some examples of ways to do this:
- Get expert assistance digitally from another place.
- Get digital artificial instructions.
- Visualize 3D-objects and use digital drawings.
Headsets, such as the HoloLens 2, are sufficient tools to use during learning processes. There are various applications customized for this technology, some of them worth adapting into education.
Digitalisation is emerging and it is now more important than ever, because of the current Covid-19 pandemic, to use digital tools in an advantageous manner. It is possible to get distance -assistance by using AR. The person that wears the headset can call an expert, teacher or instructor. The expert will, on their screen, see the same thing as the user, by an integrated camera on the front of the headset. The expert can then add artificial objects, such as arrows or marks, which will appear in the field of view in real time for the user, augmenting reality and simplifying learning. This tool will also be beneficial for the CLF. Partners of the CLF are located at different geographic locations, but working together within the same learning factory. This digital AR-tool can minimize the distance between the partners in the sense of transforming knowledge.
AR has the possibility to provide the user with artificial guides. This is greatly beneficial to understand production chains, assembling of components, how to operate machines etc. The user will wear a headset or use the application through a smartphone. Artificial objects, information and various forms of support are added in the user’s field of view. It could for example be simulations that display how to move, remove, rotate etc. an object or artificial text with information that appears just in time when it is needed.
Augmented Reality could also be used to look at 3D-objects or display virtual drawings, which contribute to a paperless production.
Integration of VR/AR in Curt Nicolin Gymnasiet LAB
There are three VR labs implemented at Curt Nicolin Gymnasiet. These labs are equipped with various Virtual Reality headsets from different brands. Augmented Reality is implemented in education as well. AR equipment is more movable, thus not fixed into a specific lab. AR technologies are used in different labs, e.g in the workshop, at the school as a tool for educational improvement. Each of the “levels”, described in the previous section, are used in education at Curt Nicolin Gymnasiet, with regard to both Virtual Reality and Augmented Reality technologies.
Technologies and software used within the labs:
- Oculus quest 2
- Hololens 1 & 2
- HTC Vive
- Vavle index
- Unreal Engine
- Blender
Integration of VR/AR in Tknika’s lab
The technology is used in the format of simulators, equipment that reproduces a production process and trains the student in it and then finishes their training in a real environment, achieving an acceleration of learning.
Figure 6: : Virtual Reality Painting environment. Source: TKNIKA
The second level of integration is when the learning classroom is integrated with the virtual environment, in this way there is no break between the spaces and we can be in the real or virtual space at any time.
Figure 7: : Virtual Reality Painting environment as seen by the user. Source: TKNIKA
Among the integrated software and hardware we use the following tools and devices:
- Soldamatic: for welding simulations
- SimSpray: for painting simulations
- Innvison and InnXr: for VR meetings in different cities
- Some HMD: Oculus quest2, HTC vive, Oculus Rift and others
- Some AR and XR headsets: Microsoft hololens 1 and 2
- Unity: as development IDE
- Blender: as modeling, sculpting and animation software
- Motion captor: for body tracking and movement capture
Role of the VR/AR in the EXAM4.0 CLF
The CLF that is going to be launched has divided its production process into 4 stages (product design, process engineering, production and assembly) as can be seen in the following image. Within these stages, VR/AR is going to be incorporated in the process engineering, production and assembly stage.
Figure 8: Exam 4.o manufacturing process. Source: Author’s creation
The role of VR/AR in the CLF is to ensure good communication and collaboration between the partners of the learning factory in order to be able to fully exchange data and information. VR/AR applications are used in order to have meetings and discuss the products of the CLF.
Figure 8: Meeting of Exam4.0 cosortium during process engineering. Source: Author’s creation
For example, the picture above shows a meeting with the consortium when Augmented Reality was used in order to show the features of the re-designed Clipper for the group. The Clipper is a flexible component that is a part of the EXAM robot produced in the CLF.
Moreover, it will also have an important role in the production and assembly part, since through VR / AR we can implement machine manuals, maintenance manuals, assembly manuals …
Benefits of using VR/AR in EXAM’s CLF
When it comes to inserting VR / AR into the CLF we can talk about four main benefits:
- Acceleration of the learning process, with better monitoring and greater control over it. It is not only running the simulation, it also includes powerful feedback from several parametrized items so the students can perform better and know where the mistakes are made.
- Reduction of consumable material with consequent cost reduction. Using any of the virtual technologies the consumption of material is zero. We can simulate the use of the most expensive materials without truly using them at all. This is an eco-friendlier way of working and it prepares our students for the real world environment.
- Increase in the physical security of students as well as of the facilities involved in the processes. Using these technologies, we can simulate even catastrophic events such as heavy industry machinery fails or fire simulations. The safe environment leads the students to perform better in case of those stressful situations.
- To ensure excellent communication, exchange of data and information. The CLF is a collaboration using different partners’ labs in order to create a Learning Factory. Communication and exchange of data is therefore an essential part of this collaboration, it is required for the functionality of the CLF. VR/AR is a useful tool in order to achieve this. Aforementioned methods, workroom meetings in VR, tele assistance etc., are useful in order to achieve this.
Competences addressed with digital workplace
VR/AR education is very beneficial for competence training because of the variety of simulations and applications that it can provide. The competencies addressed in VR/AR therefore vary depending on the educational objective focused on.
However, here are some examples of competencies addressed within the scope of VR/AR in the EXAM 4.0 CLF:
- Communication, cooperation and teamwork: Visualisation applications have been used during the pilot of the CLF to set high quality information exchange and communication amongst the partners. These applications will later be used as a part of the CLF to ensure that the different partners can exchange data, information and knowledge in an adequate way.
- Technical expertise and IT knowledge: Users will get both technical expertise and IT knowledge by working with the visualisation equipment and its appurtenant applications.
- Problem solving: The partners of the CLF have used AR in order to look at the robot’s components. This part of the project required problem solving, because there were many features in the robot that were difficult to produce.
COLLABORATION OPPORTUNITIES OPENED BY VR-AR
VR / AR is a great tool to work collaboratively. On the one hand, work meetings between different schools could be created through VR. That way, students from different countries could work together and get to know each other better. This practice could be very good as a previous knowledge exercise for a later exchange.
On the other hand, as seen in figure 9, the AR is a very helpful support when it comes to designing meetings. By means of AR all the members of the meeting can observe the piece while giving different opinions about it.
Finally, due to the digital twin, where the entire manufacturing process can be digitized, and the data analytic with the IioT, where data is collected in real time, students from centres that do not have enough resources to have the production machines, could work in a VR environment as if they were working on a real machine.