SIGGRAPH 2023

Technology 01 Spatial Reality Display "ELF-SR2"

Larger 27 inch 4K Model, ELF-SR2 , Provides a Realistic, Glasses-free 3D Visualization Experience for Professionals
Spatial Reality Display ”ELF-SR2” is a larger 27 inch 4K display that provides highly realistic, three-dimensional content without the use of special glasses or VR headsets. Highlights of the new offering include an upgraded high-speed vision sensor, image quality enhancing technologies and installation flexibility. Additionally, the ELF-SR2 enables more robust functionality through its support of applications and development. It is optimized for industrial design, surgical planning, architecture, engineering, construction, signage, retail, software/application development, game developers and entertainment applications. The ELF-SR2 not only empowers creators to develop 3DCG content but also provides a platform to showcase their creations to others. Thanks to its strong features, it can be utilized in various B2B verticals and environments, catering to a wide range of professional applications.

Immersive Spatial Imagery

The display’s visual fidelity is amplified by its immersive depth of field and detailed resolution, in addition to its larger size. New generation high-speed vision sensors enable high-speed processing, as well as low latency for reduced motion blur and crosstalk. 10-bit processing supports a wide color gamut that covers Adobe RGB at approximately 100% for accurate color reproduction. The super resolution engine provides upscaling from 2K to 4K. Additionally, the new model offers color moire correction to better address fine details, patterns and lines. The ELF-SR2’s enhanced facial tracking and recognition senses a viewer’s eyes to provide a natural and comfortable visual experience, while the wide viewing angle enables consistency and accuracy from numerous vantage points.

Versatile Software Development Environment

The ELF-SR2 promotes application compatibility, as well as efficient and streamlined development through software development kits (SDKs). With support for leading SDKs including Unity and, Unreal Engine, it also allows for development with OpenGL, DirectX 11/12 and OpenXR (coming later this year). The Spatial Reality Display enables simplified VR and AR digital content creation. A new Spatial Reality Display App Select website will also be available, where users can easily find applications compatible with the display, as well as relevant case studies and information. In addition, Sony will offer an intuitive Spatial Reality Display Player app, which supports various 3D file formats to easily show 3D projects on the display.

Adaptable to your environment

The easy-to-use ELF-SR2 Spatial Reality Display’s updated sensors and high-speed hardware processing provide accommodation for a wider variety of PCs to support different use cases and needs. With a detachable stand, the portable display can be installed in multiple different environments and configurations. It also accommodates the Video Electronics Standards Association (VESA) mounting standard for further flexibility and compatibility.

Spatial Reality Display and Autodesk Maya demonstration at SIGGRAPH

At this SIGGRAPH 2023, Sony will be conducting a demonstration of the ELF-SR2. In addition to the variety of supported applications, our new DCC tool plugin application, specifically designed for Maya, will be shown as part of this demonstration. With the ELF-SR2 and this plugin application, users will be enabled to preview their creations in 3D without the need for glasses, which will boost creation efficiency and precision throughout the creative process. We look forward to seeing you at SIGGRAPH to experience this demonstration.

Technology 02 Automatic Lip-sync in Anime Production

Synchronizing lip movements to the speech audio requires manual and tedious work and is highly time-consuming. Our goal is to automate the lip-syncing process and consequently make the production more efficient. We first train a model that classifies phoneme from input speech using temporal convolution network. Our model is first trained with Japanese speech data and subsequently fine-tuned with speech data used in animations, especially with vocal tracks. We have further incorporated soft cross entropy loss on vowels while training the model, to make lip movements smooth. The model is lightweight and there is no need for expensive GPU to process the lip-sync, which even runs on cheap laptop PC. Our lip-sync tool was delivered to A-1 Pictures and used in the production of TV anime “Visual Prison” and anime movie “Uta no Prince-sama”.

Fig. Efficient & rapid R&D cycle of Automatic Lipsync

Technology 03 Enhancing games with cutting-edge AI to unlock new possibilities for game developers and players.

We are evolving Game-AI beyond rule-based systems by using deep reinforcement learning to train robust and challenging AI agents in gaming ecosystems. This technology enables game developers to design and deliver richer experiences for players. The recent demonstration of Gran Turismo Sophy^TM, a trained AI that beat world champions in the PlayStation^TM game Gran Turismo^TM SPORT, embodies the excitement and possibilities that emerge when modern AI is deployed in a rich gaming environment. As AI technology continues to evolve and mature, we believe it will help spark the imagination and creativity of game designers and players alike.

Can an AI outrace the best human Gran Turismo drivers in the world? Meet Gran Turismo Sophy and find out how the teams at Sony AI, Polyphony Digital Inc., and Sony Interactive Entertainment worked together to create this breakthrough technology. Gran Turismo Sophy is a groundbreaking achievement for AI, but there’s more: it demonstrates the power of AI to deliver new gaming and entertainment experiences.

• Sony AI
• Blog Series
• The Team Behind GT Sophy - Part 3 - Training a Champion
• The Team Behind GT Sophy - Part 2 - Off To The Races
• The Team Behind GT Sophy - Part 1 - Start Your Engines
• Gran Turismo Sophy

Technology 05 Cutting-edge technology meets culture in Kyoto

As a research institute at the intersection of cutting-edge technology and human potential, we - Sony Computer Science Laboratories, Inc. - are in a position to shape the evolution of society. In an everchanging world, the need for solutions to complex problems is higher than it has ever been. Our mission is to make use of the vast potential of our technology, which has computer science at its core, to solve the numerous issues that face humanity. With an understanding of the duality of hopes and anxieties that arise from rapid technological advances, we pledge to pursue the eternal values that define human nature and lead humanity to a prosperous future. We aim to contribute to humanity and society by investigating and implementing desirable solutions in a wide range of social sectors, including industry, education, health, creativity, and culture.

In Kyoto, we are now taking on the challenge of creating new Kimono - traditional Japanese clothes - by incorporating cutting-edge technology into the tradition of Nishijin (Kyoto’s famed weaving district) textiles. There are many manufacturing processes involved in the production of textiles, including design creation, pattern paper creation, yarn dyeing, and weaving. As a first step, we have produced Nishijin textiles based on designs created using fine-tuning diffusion model that generates images from keywords/sentence, and continuously we are exploring ways to fuse traditional craftsmanship with cutting-edge technology in other manufacturing processes as well.

Information technology is rapidly advancing at an unprecedented level to automate and streamline human actions. What we are witnessing is a wave of social change on a scale comparable to the Industrial Revolution. As a research organization, we believe that the time has come for us to develop the basic technologies that will lead this progress, while at the same time seriously consider the compatibility of technological progress and human happiness. It is not just about efficiency and automation, but also of senses of fulfillment and permanence that people can feel, and the inheritance and development of traditions. We call this "Yutakasa (abundance)". Yutakasa is a timeless principle that transcends mere material and monetary wealth and has the power to link technological progress with the well-being of humankind. It is the enduring driving force behind our research activities.