Real-time Moving Object Detection Technology and Trial of Stone Location Delivery at a Curling Venue

1. Introduction

NTT Service Evolution Laboratories is researching and developing tracking technology for real-time recognition and tracking of objects in video with the objective of developing more advanced stadium solutions and public viewing of events. The aim here is to achieve tracking technology that has both a short processing time and high-accuracy object recognition to provide in real time meaningful information to spectators of competitive matches, audiences of theatrical performances, and other event spectators. We can envision various types of applications using tracking technology combined with a video understanding system, for example, recognizing and tracking an athlete or the ball during a sports match and overlaying information as in augmented reality, and recognizing and tracking the motion of an actor’s hands and adding stage effects. We can expect new video services to be offered in this way.

In this article, we outline the real-time moving object detection technology that we developed targeting curling stones and describe the trial we conducted at the curling competition held at the 2017 Sapporo Asian Winter Games using a system incorporating this technology.

2. Demonstration experiments on providing new spectator experiences at Asian Winter Games

The City of Sapporo and NTT held demonstration experiments on providing new spectator experiences using advanced information and communication technology at the 2017 Sapporo Asian Winter Games held in Sapporo, Hokkaido, from February 19 to 26. The targeted sports were figure skating, ice hockey, and curling, and these demonstration experiments used an official smartphone application (app) for the 2017 Sapporo Asian Winter Games to showcase new spectator services that could not be experienced by the traditional means of watching television or observing the sports at the venue.

When viewers watch a curling match, their ability to accurately understand the stone’s current location is tied to having an accurate understanding of the state of the match. However, since it is impossible to obtain an overhead view of the ice-based curling sheet (playing surface) and house (the circular target on the sheet) from spectator seats inside the venue, it has been difficult to accurately grasp the locations of the stones. To address this problem, we aimed to provide a new spectator experience service enabling a spectator inside the curling venue to grasp the state of the current match by delivering stone location information to the smartphones of spectators.

To deliver stone location information that is useful in understanding the state of the match, the stone location must be obtained during the match as the match progresses. We therefore developed technology for detecting in real time the location of a stone as a moving object and conducted a trial on the use of moving object detection technology to deliver stone location information to spectators at a curling venue (Fig. 1).

Fig. 1. Demonstration experiment (curling) on providing a new spectator experience at 2017 Sapporo Asian Winter Games.

3. Real-time moving object detection technology targeting curling stones

To achieve low-cost detection of the locations of all stones present within the effective area between the hog line and back line without making modifications to players or their equipment, we decided to make use of existing venue stones and cameras. The camera above the sheet of a match in progress has a limited shooting range and cannot be controlled freely, so we used a camera above a sheet not currently in use and oriented it to overlook and shoot the effective area (Fig. 2).

Fig. 2. Curling house area.

We developed technology for inputting video taken in this way and for detecting in real time the team color and location coordinates of each stone. The process flow for each video frame is shown in Fig. 3 and described below:

Fig. 3. Process flow per frame of real-time automatic stone-location detection program.

4. Stone location delivery system for spectators at curling venue and trial

We constructed a stone location delivery system based on real-time moving object detection technology that we developed. This system delivers stone location information to a smartphone app every time a throw is completed, enabling spectators at a curling venue to understand where the stones in a match are currently located. In this trial, a function for recognizing the time that a throw is completed (the time at which all stones are stationary) was not implemented as part of the program but simply achieved by having the operator determine that time visually.

The basic configuration of this system for one curling sheet is shown in Fig. 4 and summarized below.

Fig. 4. Configuration of curling-stone location delivery system.

As previously described, video signals from existing camera facilities in the curling venue are distributed and used as input video to the proposed system. In curling, however, the stone-throwing direction changes every end^*, so the effective area enclosed by the input image must be switched for each end. The system therefore installs a switcher so that the video input to the video analysis personal computer (PC) can be selected from the video output from two cameras.

Now, once the operator visually determines that a throw has been completed and operates the system accordingly, the video analysis PC detects the location of that stone from the input video using real-time moving object detection technology and passes that result to the delivery control PC.

Next, the operator at the delivery control PC visually checks the received stone location information for errors, corrects errors if any exist, and uploads the result to the delivery server. The smartphone app accesses the delivery server every few seconds. If the app detects an update on the stone location information at that time, it obtains that information and updates its display content.

This system trial was conducted during the curling competition held February 18–24. The system operated without problems during this period and received a favorable response from users and curling associations.

5. Accuracy of stone location detection results in trial

We evaluated the accuracy of the stone location detection results using video of 21 matches of the men’s and women’s curling competitions at the 2017 Sapporo Asian Winter Games. The stone location after being visually checked and corrected if necessary by the operator of the delivery control PC was taken to be correct location information.

For each match, the number of stones corrected by the operator of the delivery control PC, the total number of stones placed in the effective area and targeted for location detection, and the stone correction rate are listed in Table 1. The stone correction rate per match fell in the range from 1.12% to 25.98%, resulting in a value of 11.12% for all matches. This means that the probability of having no correction needed by the operator of the delivery control PC was approximately 90%.

Table 1. Stone correction rate per match.

Plots of corrected stone locations on the curling sheet are shown in Fig. 5 by stone color. A comparison of the two sets of plots indicates that the number of corrections is higher for red stones and that many of these red-stone corrections appear on the red circle of the house. We surmise that the close resemblance of the house color and the red stone color led to errors in detection.

Fig. 5. Corrected stone locations (by stone color).

6. Future development

In this article, we described real-time moving object detection technology targeting curling stones, a stone location delivery system incorporating this technology, and a system trial conducted at the 2017 Sapporo Asian Winter Games. In the trial, the stone locations were detected with an accuracy rate of about 90%. We were able to provide a new spectator experience service that delivered stone location information to spectators at the curling venue without any problems during the period of these games.

Going forward, we plan to continue our research and development efforts to enhance tracking accuracy using moving-object locus information and to expand the number of recognizable moving objects. In this way, we seek to provide new video services featuring, for example, new ways of viewing sports and new types of staging in entertainment.