The traditional route to knowledge is to read a book from a library. We’re investigating how we can go beyond this and embed knowledge directly into the perception of the user, right where action happens and performance is required.
Wearables thereby act as gateways, mediating between objective reality and its enhancements with visual, auditive, haptic, etc. overlays. When done well, they help turn sensorimotor perception into experience.
This requires two types of world knowledge, i.e., data about the workplace and data about the activity pursued. While the first is rather stable, the latter is dynamic and much more rapidly. We’re researching both representation and implementation, working on both standards as well as development toolkits and frameworks.
On entering Audiomotion Studios, visitors will encounter a ginormous space for motion capture (MoCap) with over 160 Vicon cameras mounted on the rigs (see picture). The Audiomotion Studio is the largest performance capture stage in Europe. Quite a number of actor and animal movements can be recorded at the same time for production of accurate animations. Behind the MoCap space there are several rooms with green screens and Motion Control Crane for filming. PAL’s PhD candidate Yu Huang, Dr Fridolin Wild, and John Twycross went to see Brian Mitchell, the Managing Director, to explore possibilities for further collaboration on volumetric video capture.
The work of the Performance Augmentation Lab was recognised by the University with the Research Excellence Award for the academic year 2018/19. Fridolin Wild and John Twycross were awarded by the Pro-Vice Chancellor for Research and Global Partnerships, Prof Dr Linda King, in the category Interdisciplinary/
This will help to advance our capacity for creating and simulating digital twins of real life characters and their action sequences for training, media, and entertainment purposes. Gaps in current research have been identified in the areas of body and face scanning (the process of converting a human actor and their expressions into a virtual model ready for animation). While there is a significant body of work completed by video game and Hollywood film producers there is a significant lack of disseminated knowledge by commercial companies. This project aims to address this gap in knowledge. Through experimentation and practice based production we will refine and disseminate a robust workflow that will enable photo-real full body capture with movement and facial expression digitisation and reproduction. We seek to collaborate with industry on this.
Just in time for Halloween, we have finalised work on a major release of the ‘WEKIT.one’, our next-generation app for wearable experiences in knowledge-intensive training.
The development of the experience capturing software has been led by members of the Performance Augmentation Lab. This is one of the first of such tools that allows the generation of content to be done completely within AR. Using a HoloLens and other wearable sensors, the software guides experts to record immersive training procedures using all available AR content. Blending 2D and 3D instruction into the workplace creates a far richer and more interactive training experience.
The expert works through the procedure, capturing their actions, thoughts, and guiding instruction step by step. We are able to capture their movement in and around the workplace, their hand positions, and even some additional biophysical signals, such as heart rate variability or galvanic skin resistance. With just the technology at hand, trainees can now visualise the expert, listen to live guidance, and have access to on-demand knowledge about the task at hand.
To now we have seen experts in the field of aircraft maintenance, radiology and astronaut training use this software and, in 2019, we aim to establish new collaboration within the university, within Oxford and abroad – most imminently with the European Space Agency.
At the Augmented World Expo in Munich, we have been exhibiting the WEKIT project last week, showcasing the breakthrough achievements for our augmented reality and wearables solution in the space industry, aviation, and medicine. On stand #217, we exhibited the different versions of the e-textile garment (and its underlying sensor harness) as well as the WEKIT.one software solution. The director of our lab, Dr Fridolin Wild, gave a keynote presentation in the enterprise track about AR experience capturing and sharing for Training 4.0, explaining the technologies of the project and the findings of the pilot trials reported so far in a series of articles and papers.
On October 1st, our artists in residence inaugurated their ATLAS solo exhibition in The Glass Tank. The exhibition features an artwork that is part real and part virtual, and that has been grown over the course of a year from our collaboration. The artists, Yann Deval and Marie-Ghislaine Losseau, were supported by the European Union with 15k EUR (via the VERTIGO project)
to team up with the top notch funded ICT projects. In our case, this paired with the WEKIT project (wearable experience for
knowledge intensive training, funded with 2.7M EUR, where we sign responsible for the scientific coordination).
The artwork allows to engage the public in the more complicated research of WEKIT – the artists provide an interpretation of what our new information and communication technologies mean, in an accessible format, understandable by kids and adults alike.
The AR smart glasses and wearable technologies used in the WEKIT project and in the ATLAS residency will change personal computing as we know it. Augmented Reality (AR) Smart Glasses will be be the next generation of personal computing, likely to disrupt the world on a similar scale as smart phones did over the past decades.
WEKIT’s focus is on augmented training – investigating what Work 4.0 will look like for an Industry 4.0 workplace. We developed a 3D learning activity authoring tool and a re-enactment client. Together they allow to very quickly create augmented reality learning experiences. We test them in three scenarios: we train astronauts on a battery swap on a mars rover, testing it on the actual rover in a mars terrain surface simulator. We test it also with airplane maintenance engineers in Norway and with
medical doctors in Genoa. We are waiting for the trials to finish, but expect our findings to show high levels of acceptance and usability of these new technologies, while at the same time improving effectiveness in training (reducing error rates, increasing efficiency). Our previous findings already point in that direction and have led to numerous high impact publications.
Atlas is an interactive and scenographic exhibition situated between visual and digital arts. The project has been produced in collaboration between Wekit and artists, Yann Deval and Marie Losseau, in the context of the Vertigo Residencies programme.
During this exhibition, the Glass Tank will host a mixture of real models and interactive virtual elements. The artists will also occupy the space, using parts of the gallery as a working studio so visitors can watch the project develop.
Using the latest in augmented reality technology Atlas invites spectators to interact with holograms in the world around them by building virtual cities. These cities take on a life of their own, with or without the interactions of users, just like living organisms…
To produce this exhibition and help bring it to the Glass Tank, the artists Yann Deval and Marie G. Losseau have been working in collaboration with Dr Fridolin Wild from the Performance Augmentation Laboratory at Oxford Brookes University. The project began with a 3D scan of a hand built wooden house and a hololens. Using the hololens, users can sew virtual seeds which cause the now holographic houses to grow. Since early 2018 when the residency began at Brookes, the prototype has been demonstrated at MCCS, Molenbeek with a group of children. The children helped to build hundreds more wooden houses which added to the ATLAS universe. Since then it has been displayed at the Venice Architecture Biennale, in Texas at SXSW, at Ircam / Centre Pompidou in Paris and at Ars Electronica in Austria.
Overall, the work seeks to provoke a reflection on urbanism, architecture, and the influences of these things on our lifestyles.
Venues where the prototype has been played in the previous months:
MCCS Molenbeek – Brussels / Belgium
EU@SXSW – Austin / Texas
Venice Architecture Biennale – Venice / Italy
Ircam / Centre Pompidou – Paris / France
Ars Electronica – Linz / Austria
Launched on 1 December 2015, WEKIT is an ambitious European research and innovation project supported under Horizon 2020 to develop and test within three years a novel way of industrial training enabled by smart wearable technology and Augmented Reality. With 2.7 million euros of EU funding, thirteen WEKIT partners representing academia, including Oxford Brookes University, and industry from six countries in Europe will build a ground-breaking industrial-strength learning technology platform and unique methodology to capture expert experience and share it with trainees in the process of enabling immersive, in-situ, and intuitive learning. In this way, WEKIT will bring learning content and technical documentation to life via task-sensitive Augmented Reality, making industrial training more efficient, affordable and engaging. WEKIT conducts pilots in astronaut training (ground-ground and ground-space), aircraft maintenance and preflight checks, and diagnostic training of radiologists.
The temporal residency at Oxford Brookes centred in on the ‘artistic exploration of holograms’. Holograms are 3D content that cannot be captured appropriately in 2D video or imagery. Moreover, holographic exhibitions provide ample opportunity for visual research, exploring reality as a medium for the fine arts. Smart Glasses and Augmented Reality turn reality into a medium, the characteristics of which, however, still need to be determined through artistic exploration.
These are the highlights from the talk “Making Augmented Reality: from Spatial Interaction to Astronaut Training” that our PhD candidate Alla Vovk gave in Kiev, Ukraine at Sensorama Lab. The talk covered the history of augmented reality and how human thinking evolved with the technological advancements of head-worn displays. Alla talked about the switch from 2D thinking to 3D thinking in spatial computing. The talk also reviewed PAL’s work and looked at the existing AR applications and concept implementations.
Alla also taught a two-day intensive course in AR with four lectures and nine workshops following up the lecture.
1. Introduction to the Course. How to Transform Knowledge to Experience.
2. Experience Engineering: Creating Narratives and Scenarios for Augmented Reality Using the Method of Design Thinking
3. Spatiality of Human Life: User Interface for Augmented Reality
4. Making Augmented Reality Work: How to Develop for Microsoft HoloLens in Unity so it Finally Does Make Sense.
1. “Building to think”— a set of quick exercises to get into a working mood and boost creativity in the class
2. “Empathy: standing in the shoes (or lying on the gurneys) of others” — reconfigure everyday experience
using the method of Stanford d.school
3. “An end to old ideas” — scenario-based prototyping for designing complex systems
4. “Beyond the individual” — understand your audience: watching what people don’t do, listening to what they don’t say
5. “Prototyping in the wild” — shifting to real-world applications: human-oriented thinking for AR.
6. “Microsoft HoloLens is my friend” — integration of the prototype with the Mixed Reality Toolkit
7. “Think Spatial” — executing the idea using your brain, heart, hands and bunch of prototyping materials
8. “Doing more with less” — 3D Mesh Simplification for Microsoft HoloLens
9. “Make my dreams come true” — software development in Unity and Visual Studio with assets and scripts
The WEKIT project was selected as ‘Highly Commended’ at the Innovation Excellence Award 2018 of The Stationers’ Company.
“Human brains learn by getting feedback. The judges were impressed with WEKIT, a system to enable more efficient workplace training using various wearable tracking devices and motion sensors, coupled with augmented reality, to give useful and readily assimilated feedback.” (Innovation Excellence Award 2018, The Stationers’ Company).
The Warwick Business School has a Knowledge Innovation Network and PAL’s Dr Fridolin Wild gave the opening keynote at this year’s autumn workshop, speaking about Holographic Training and other little wonders of an Industry 4.0. In the talk, Dr Wild explored how smart glasses and wearable technologies can be used for knowledge intensive training and as job performance aids, sharing the floor with Jeremy Dalton of PwC, head of VR/AR. The workshop included several case studies and demos, including of Severn Trent Water and Kazendi, as well as a guided your to the Warwick Manufacturing Group Innovation Labs.
At the beginning of September I attended the third partner meeting for the LAAR project. This is the project I’m working on which looks at using learning analytics and augmented reality to improve work-based learning. New to the project, this was my first opportunity to meet the team. It was really nice to finally meet everyone in person and see the full extent of the work produced. Video conferencing is a great way to stay in touch during a project but it always feels a little impersonal compared to face-to-face. In person ideas are easier to get across, conversations flow more naturally and people are more efficient, I believe.
The event was hosted by team members from the IT University of Copenhagen. What a great university in a beautiful city! The university building really impressed me, large modern open and made of glass. We meet in an open glass-box room with a view down of the whole campus. The building boasted some really fantastic hipster coffee which is apparently sometimes served by heads of departments – a way for them to connect with students and staff alike. The canteen food was also superb, I was spoilt compared to the UK.
In the evening we all ate out together. This was the point when I really understood why EU projects like this are a great thing. All the tensions from the days discussions, a good sign of intellectuals invested in their work, were now gone. To sit around and talk to these academics about news, the current politics climate, their individual countries, academia and beyond, was a treat. It was nice to hear about the similarities and little differences we have between countries (and ofcourse conversation always end up on Brexit at the moment).
I’m now really looking forward to the next meet in December, my chance to meet up with the team again and visit Brussels.
The paper first-authored by our research visitor Carla Bonato Marcolin won the best-paper award at the 15th International Conference on Information Systems and Technology Management (CONTECSI’18). The paper reports findings of a novel LSA-based tool for text analysis, including capturing, cleaning and formatting publicly available data from consumer reviews. The tool and findings help hotel managers in the analysis of large amounts of text data, thereby facilitating decision-making.