Real-time Moving Object Detection Technology and Trial of Stone Location Delivery at a Curling Venue

Tetsuro Tokunaga, Yoshihide Tonomura, Jun Shimamura,
Fumihisa Hamamura, and Hiroaki Kakimoto

Abstract

NTT Service Evolution Laboratories is researching and developing tracking technology combined with video understanding as part of efforts to develop advanced video services. This technology features both high recognition performance and a short processing time and shows promise for application to public event viewings and stadium solutions. This article introduces a trial conducted at the 2017 Sapporo Asian Winter Games in which this technology was applied to deliver stone location information to spectators at a curling venue.

Keywords: moving object detection, image processing, curling

1. Introduction

NTT Service Evolution Laboratories is researching and developing tracking technology for real-time recognition and tracking of objects in video with the objective of developing more advanced stadium solutions and public viewing of events. The aim here is to achieve tracking technology that has both a short processing time and high-accuracy object recognition to provide in real time meaningful information to spectators of competitive matches, audiences of theatrical performances, and other event spectators. We can envision various types of applications using tracking technology combined with a video understanding system, for example, recognizing and tracking an athlete or the ball during a sports match and overlaying information as in augmented reality, and recognizing and tracking the motion of an actor’s hands and adding stage effects. We can expect new video services to be offered in this way.

In this article, we outline the real-time moving object detection technology that we developed targeting curling stones and describe the trial we conducted at the curling competition held at the 2017 Sapporo Asian Winter Games using a system incorporating this technology.

2. Demonstration experiments on providing new spectator experiences at Asian Winter Games

The City of Sapporo and NTT held demonstration experiments on providing new spectator experiences using advanced information and communication technology at the 2017 Sapporo Asian Winter Games held in Sapporo, Hokkaido, from February 19 to 26. The targeted sports were figure skating, ice hockey, and curling, and these demonstration experiments used an official smartphone application (app) for the 2017 Sapporo Asian Winter Games to showcase new spectator services that could not be experienced by the traditional means of watching television or observing the sports at the venue.

When viewers watch a curling match, their ability to accurately understand the stone’s current location is tied to having an accurate understanding of the state of the match. However, since it is impossible to obtain an overhead view of the ice-based curling sheet (playing surface) and house (the circular target on the sheet) from spectator seats inside the venue, it has been difficult to accurately grasp the locations of the stones. To address this problem, we aimed to provide a new spectator experience service enabling a spectator inside the curling venue to grasp the state of the current match by delivering stone location information to the smartphones of spectators.

To deliver stone location information that is useful in understanding the state of the match, the stone location must be obtained during the match as the match progresses. We therefore developed technology for detecting in real time the location of a stone as a moving object and conducted a trial on the use of moving object detection technology to deliver stone location information to spectators at a curling venue (Fig. 1).

Fig. 1. Demonstration experiment (curling) on providing a new spectator experience at 2017 Sapporo Asian Winter Games.

3. Real-time moving object detection technology targeting curling stones

To achieve low-cost detection of the locations of all stones present within the effective area between the hog line and back line without making modifications to players or their equipment, we decided to make use of existing venue stones and cameras. The camera above the sheet of a match in progress has a limited shooting range and cannot be controlled freely, so we used a camera above a sheet not currently in use and oriented it to overlook and shoot the effective area (Fig. 2).

Fig. 2. Curling house area.

We developed technology for inputting video taken in this way and for detecting in real time the team color and location coordinates of each stone. The process flow for each video frame is shown in Fig. 3 and described below:

(1) Projection transformation: A projection transformation is applied to the oblique view of the input image shown in Fig. 1 to achieve an overhead image. Subsequent processing is limited to the effective area after this projection transformation to reduce computational complexity.

(2) Foreground extraction: A background image less any stones or athletes is shot, the foreground is extracted by background difference processing, and the result is passed to the next process as stone candidate areas.

(3) Extraction by color: Only two colors—red and yellow—were used for the colors of stone surfaces in this trial, so two types of color patterns were recorded from stone images prepared beforehand. This process compares these red and yellow patterns with the stone candidate areas, extracts those areas with similar colors, and passes the result to the next process.

(4) Extraction by shape: The stone candidate areas input to this process would include athlete uniforms if their colors were reddish or yellowish. To therefore exclude such candidate areas and detect only stones, this process detects the outline of each input stone candidate area and uses its shape and size to judge whether that candidate area corresponds to a stone.

(5) Coordinate correction: The above projection transformation assumes a flat target, so any object having height as in the case of a stone surface will be transformed to a location offset from the actual location. This process corrects each detected location to estimate a more accurate location and finally outputs sets of stone colors and stone coordinates as program results for this frame.

Fig. 3. Process flow per frame of real-time automatic stone-location detection program.

4. Stone location delivery system for spectators at curling venue and trial

We constructed a stone location delivery system based on real-time moving object detection technology that we developed. This system delivers stone location information to a smartphone app every time a throw is completed, enabling spectators at a curling venue to understand where the stones in a match are currently located. In this trial, a function for recognizing the time that a throw is completed (the time at which all stones are stationary) was not implemented as part of the program but simply achieved by having the operator determine that time visually.

The basic configuration of this system for one curling sheet is shown in Fig. 4 and summarized below.

Fig. 4. Configuration of curling-stone location delivery system.

As previously described, video signals from existing camera facilities in the curling venue are distributed and used as input video to the proposed system. In curling, however, the stone-throwing direction changes every end^*, so the effective area enclosed by the input image must be switched for each end. The system therefore installs a switcher so that the video input to the video analysis personal computer (PC) can be selected from the video output from two cameras.

Now, once the operator visually determines that a throw has been completed and operates the system accordingly, the video analysis PC detects the location of that stone from the input video using real-time moving object detection technology and passes that result to the delivery control PC.

Next, the operator at the delivery control PC visually checks the received stone location information for errors, corrects errors if any exist, and uploads the result to the delivery server. The smartphone app accesses the delivery server every few seconds. If the app detects an update on the stone location information at that time, it obtains that information and updates its display content.

This system trial was conducted during the curling competition held February 18–24. The system operated without problems during this period and received a favorable response from users and curling associations.

*	End: In curling, one round of play by each team in a game. One end is completed when each team throws 8 stones, for a total of 16 stones.

5. Accuracy of stone location detection results in trial

We evaluated the accuracy of the stone location detection results using video of 21 matches of the men’s and women’s curling competitions at the 2017 Sapporo Asian Winter Games. The stone location after being visually checked and corrected if necessary by the operator of the delivery control PC was taken to be correct location information.

For each match, the number of stones corrected by the operator of the delivery control PC, the total number of stones placed in the effective area and targeted for location detection, and the stone correction rate are listed in Table 1. The stone correction rate per match fell in the range from 1.12% to 25.98%, resulting in a value of 11.12% for all matches. This means that the probability of having no correction needed by the operator of the delivery control PC was approximately 90%.

Table 1. Stone correction rate per match.

Plots of corrected stone locations on the curling sheet are shown in Fig. 5 by stone color. A comparison of the two sets of plots indicates that the number of corrections is higher for red stones and that many of these red-stone corrections appear on the red circle of the house. We surmise that the close resemblance of the house color and the red stone color led to errors in detection.

Fig. 5. Corrected stone locations (by stone color).

6. Future development

In this article, we described real-time moving object detection technology targeting curling stones, a stone location delivery system incorporating this technology, and a system trial conducted at the 2017 Sapporo Asian Winter Games. In the trial, the stone locations were detected with an accuracy rate of about 90%. We were able to provide a new spectator experience service that delivered stone location information to spectators at the curling venue without any problems during the period of these games.

Going forward, we plan to continue our research and development efforts to enhance tracking accuracy using moving-object locus information and to expand the number of recognizable moving objects. In this way, we seek to provide new video services featuring, for example, new ways of viewing sports and new types of staging in entertainment.

	Tetsuro Tokunaga Research Engineer, Natural Communication Project, NTT Service Evolution Laboratories. He received a Bachelor of Liberal Arts and Master of Informatics from Yokohama National University, Kanagawa, in 2002 and 2004. He joined NTT Network Service Systems Laboratories in 2004. He was with NTT Resonant from 2008 to 2012. His current research interests include web technology, computer vision, and design thinking. He was a co-developer on a team that participated in 2003 in the youth division in the Exploratory IT Human Resources Project (The MITOH Program) initiated by the Information-technology Promotion Agency of Japan. He is a member of the Information Processing Society of Japan and the Database Society of Japan.
	Yoshihide Tonomura Senior Research Engineer, Natural Communication Project, NTT Service Evolution Laboratories. He received an M.S. in electronics engineering from Nagaoka University of Technology, Niigata, in 2004 and a Ph.D. from Tokyo Metropolitan University in 2010. Since joining NTT, he has been researching and developing multimedia data transmission technologies. From 2011 to 2012, he was a visiting scientist at MIT Media Lab, Massachusetts, USA. He received the SUEMATSU-Yasuharu Award from the Institute of Electronics, Information and Communication Engineers (IEICE) in 2016 and the 26th TAF TELECOM System Technology Award from the Telecommunications Advancement Foundation in 2011.
	Jun Shimamura Senior Research Engineer, Planning Section, NTT Service Evolution Laboratories. He received a B.E. in engineering science from Osaka University in 1998 and an M.E. and Ph.D. from Nara Institute of Science and Technology in 2000 and 2006. He joined NTT Cyber Space Laboratories in 2000. He was with NTT Communications from 2006 to 2009 and NTT Media Intelligence Laboratories from 2010 to 2015. His research interests include computer vision and mixed reality. He received the 2005 TAF TELECOM System Technology Award from the Telecommunications Advancement Foundation. He is a member of IEICE and the Institute of Image Information and Television Engineers.
	Fumihisa Hamamura Associate Manager, Strategic Business Development Division, NTT. He received a B.A. in environment and information studies from Keio University, Kanagawa, in 2005 and joined the NTT Group the same year. Since 2012, he has been engaged in new business and service development with ICT in the healthcare and sports market.
	Hiroaki Kakimoto Strategic Business Development Division, NTT. He received a B.E. and M.E. in computer science from Nagoya Institute of Technology, Aichi, in 2008 and 2010. He joined NTT Communications in 2010 and was engaged in corporate sales and development of a live broadcasting service. He has been with NTT since 2015 and has been involved in business development and collaboration support for NTT Group companies in sportstech, healthtech, and other fields.

↑ TOP