Brunswick Corporation filed U.S. Patent Application “Person Detection in a Marine Environment” on 5 May 2016. The patent application (US 2017/0323154 A1) was published by the U.S. Patent Office on 9 November 2017. As of late January 2018, the patent has yet to be granted.

The basics of Brunswick’s patent application teaches the use of computer vision (an image sensor on the vessel to capture an image) as the basis for a detecting the presence of people in a marine environment. An image sensor captures the scene, the system scans the image using an Artificial Neural Network (ANN) that has been trained to detect patterns in the image associated with one or more predefined objects such a swimmer near the propeller. The ANN outputs information regarding the presence or absence of such predefined objects within the image. Then powertrain computers respond by alerting the operator, slowing the vessel, shifting the drives to neutral, or other appropriate actions. In effect, the system acts as a sensor based propeller guard. We long ago coined a phrase for these types of systems, Virtual Propeller Guards.

We first coined the phrase, Virtual Propeller Guard, back in 2002 for use with sensor based systems designed to detect people in the water near a boat at risk for being struck by the propeller. These sensors were used to intelligently prevent propeller strikes by alerting the operator people were in the water near the vessels, sounding an alarm, shifting drives to neutral, shutting down marine drives, prevent the engine from starting, or other appropriate actions based on conditions. Over time it became obvious “Virtual Propeller Guards” also applied to sensor based detection systems monitoring possible presence of people in the water near a vessel, such as virtual lanyards, ladder switches, swim gate switches, and even traditional lanyard kill switches. Over time, Virtual Propeller Guards expanded to include Operator Presence Detection (OPS), and Voluntary Egression Detection such as swim gate switches and swim ladder switches.

Brunswick’s ‘154 patent application teaches:

1. Using the technology to detect people near the stern of the vessel by attaching cameras (image, infrared, sonar, or other types) to the upper rear side of the drive or to the stern
2. Using a similar system to detect the presence or absence of an operator at the helm
3. Using a similar system to look forward off the bow to detect people, debris, or small vessels
4. Using a similar system to look around the sides of the vessel for people in the water

We will further discuss the patent, then present evidence of our previous posting of materials teaching of a similar device several years ago (prior art).

---

Table of Contents

1. First Page of Brunswick’s Virtual Propeller Guard Patent Application
2. Discussion of Brunswick’s Virtual Propeller Guard Patent Application
3. Brunswick’s Virtual Propeller Guard Patent Application Claims
4. Volvo Penta Image Averaging Patent
5. Boiler Plate / Fluff Language in the Brunswick Virtual Propeller Guard Patent Application
6. Brunswick I-JET Lab
7. Our Prior Art
8. Volvo Penta Prior Art Statement
9. A New Reference in This Field

---

First Page of Brunswick’s Virtual Propeller Guard Patent Application

A copy of the first page of the patent application was cropped and reassembled into the image below.

---

Discussion of Brunswick’s Virtual Propeller Guard Patent Application

At first glance we are struck by the similarity of the image in the Abstract of the patent application filed by Brunswick’s President, George Buckley, and others back in 2001, US Patent 6,273,771.

U.S. Patent 6,273,771 focused on Brunswick’s CANBUS network’s ability to tie many systems on the vessel together. The current invention builds upon this connectivity. Years later, Mercury Marine sometimes refers to their CANBUS as SmartCraft.

The current patent application spends a lot time talking about the need to “train” the Artificial Neural Network with “Positive” images (those showing the presence of what you want the network to detect) and “Negative” images (those showing the absence of what you want the network to detect). The patent application repeatedly teaches to use images taken in the marine environment with similar “field of view” and “view angle” (angle the camera looks from).

Figure 4A and Figure 4B in Brunswick’s ‘154 patent application (see left) provide an indication of the field of view and view angle for some typical applications at the stern.

Per the inventors, currently available swimmer detection systems perform poorly at detecting people or other objects in the water around a marine vessel. (section 0022).

While we agree “swimmer detection systems” are not currently available off the shelf for OEM use on recreational boats, swimmer detection systems have been around for decades. We have been encouraging the industry to apply and adapt those concepts for a long time. Among existing swimmer detection systems are those detecting the presence of swimmers in a swimming pool, those detecting individual swimmers drowning in crowded public pools, those detecting terrorists/intruders in the water near a vessel, those detecting swimmers at considerable distances in open water, and those for use in aerial open water rescue operations.

Section 0023 notes “There has been a long felt need in the marine vehicle and marine sporting applications of a system and method of reliably detecting presence of people or other predetermined objects, such as floating debris or other obstacles, within a marine environment.” Glad to see Brunswick finally publicly recognizing it.

Section 0023 notes the detection system could be integrated with a display system such as the Mercury Marine Vessel View display and a speaker could be added at the helm to provide an audio output.

Section 0033 notes the detection information output from the Artificial Neural Network (ANN) could include a probability value for the predefined object being within the image.

Section 0034 says the ANN could be trained using a supervised training algorithm such as Ground Truth Training. Per our understanding, Ground Truth has to do with the reliability of your training set of images. Meaning are you positive which training images really have a person in them or not. That should be pretty easy to control.

Section 0037 notes fractions of larger images may be submitted to ANN such as by subdividing a larger image into smaller pieces, each of which may contain a portion of a larger object.

Section 0038 notes the training images may traditional photographs or single frames captured by a video camera.

Section 0042 notes training the Artificial Neural Network may account for:

1. Various viewing angles caused by the motion of the vessel
2. Various water conditions such as small lakes vs. larger waves in ocean environments
3. Various applications of marine vessels such as skiing, tubing on smaller vessels vs larger yachts that might encounter a small paddie boat off the bow

Section 0043 notes ANN may be trained to detect one or more certain predefined objects in a marine environment more generally. In this instance training could include many images of the specific object from various viewing angles and in different marine environments. While this section may be addressing boat operator presence detection the same concept might lead to improved floating log detection that could address some of the outboard motor broke off and flipped into he boat issues AND to improved detection of snorkelers and divers.

Section 0045 notes the image sensor could have a fisheye lens (sees a broad area, but the image may be warped) and the image could be digitally cropped and de-warped before processing by an ANN not trained on fisheye images. They also note the ANN would likely be trained on the same type of images it will be processing such as digital light images, infrared images, sonar images, or radar images.

Section 0046 notes infrared detection can be challenging in some water environments because water distorts and obscures the temperature of the object to be detected. As a result, for certain marine environments, visual light camera sensors may be preferable and provide superior performance compared to infrared detectors.

Section 0048 begins to talk about the output of the system. Detection information can be used to alert the operator or control one or more marine drives.

Section 0049 notes powertrain computers could treat detection information as an input to the engine control algorithm such as by using a lookup table to determine the response. For example the marine drive could immediately be shifted to neutral if an output of 95 percent probability of a person being being near the drive. Similarly the detection of a high probability of a person just off the bow when underway could result in the drives being shifted to neutral or even to reverse in some situations.

Section 0050 notes the output from the image computer associated with the helm image sensor would be treated differently than that of the stern sensors. In the instance of the helm sensor, the detection of the ABSENCE of a person at the helm for a number of frames or predetermined time could cause the powertrain computers to slow down the marine drive or shift them to neutral depending on the situation.

They note the control algorithm could include a filtering mechanism that progressively applies certain actions after certain thresholds have been reached. While the patent does not provide this example, it is as if if the person is gone from the helm while underway for a couple seconds the system might sound a buzzer, gone a little longer they might slow down the marine drives, gone a little longer they might shift the drives to neutral. We note those decisions might also be partially based on vessel speed and/or engine RPM at that time.

---

Brunswick’s Virtual Propeller Guard Patent Application Claims

Brunswick’s ‘154 patent application includes two independent claims. All other claims are based upon these two independent claims, Claim #1 and Claim #13.

The 2 independent claims in Brunswick’s patent application
 
Claim #1. An object detection system for a marine vessel having at least one marine drive, the object detection system comprising: at least one image sensor positioned on the marine vessel and configured to capture an image of a marine environment on or around the marine vessel; a processor; and an image scanning module executable on the processor that receives the image as input, the image scanning module including an artificial neural network trained to detect patterns within the image of the marine environment associated with one or more predefined objects, and to output detection information regarding a presence or absence of the one or more predefined objects within the image of the marine environment.
 
Claim #13. A method of detecting a presence of one or more objects in a marine environment of a marine vessel, the method comprising: training an artificial neural network to detect patterns associated with a predefined object in images of a marine environment; recording an image with at least one image sensor positioned on the marine vessel; providing the image as input to the artificial neural network; and outputting detection information regarding a presence or absence of the object within the image.

The swimmer detection portion of Claim #1 basically claims a swimmer detection for a boat with at least one sensor to capture an image of the water around the boat, a computer, an image scanning module (turns the image into digital format), an Artificial Neural Network trained to recognize patterns within these images associated with swimmers, and an output telling you if a swimmer is in the water near the boat or not. The claim is a little broader to include detection of predefined objects on the vessel (such as an operator at the helm) or other predefined objects in the water such as debris or a small boat in front of larger vessels.

The swimmer detection portion of Claim #13 has to do with training the Artificial Neural Network (ANN) to detect objects in images in the marine environment, then the ANN outputs detection information.

As is often the case, the whole patent basically hinges on being able to get the patent examiner to grant Claim #1. Patent examiners are notorious for rejecting most claims as they were originally filed. That does not mean the patent will not eventually be issued. It means the first claim will likely be narrowed in scope from the form in which it was originally proposed.

Inventors want broad claims that define a large space belonging only to them (or their firm). Examiners want to issue narrow claims, only granting a small unique space to the inventor (or to the inventor’s firm) if they meet all the requirements for a patent. The give and take between the two sides, typically between the patent examiner and the inventor’s firm’s patent attorney, will likely eventually result in something patentable in the middle. It then becomes the inventor’s firm’s decision if a patent with these narrower claims is really worth paying the fees to have the patent issued plus the maintenance fees in future years. That is just the way it is.

---

Volvo Penta Image Averaging Patent

Back in 2012 Volvo Penta was awarded US Patent 8,195,381 for a Safety System for Marine Vessels. Volvo Penta was later issued a related patent, US Patent 8,271,155. Both patents focused on GPS anchors (digital anchors) but the patents also included the use of infrared sensors to detect people in the water.

Volvo talked about using a moving average difference system (detecting the difference in a small region of an image with the images before and after the image being analyzed) as being a superior method to using Neural Networks. The quote below comes from Volvo Penta’s ‘381 patent:

Volvo Penta U.S. Patent 8,195,381
 
“Computation of a moving average is a most reliable approach to detecting a presence of the one or more persons in the water in comparison to neural network-type detection or template comparison detection which are also within the scope of the present invention.”

While we understand rivals disagreeing about the best way to do things, it is interesting Brunswick does not reference the Volvo Penta patents or mention the moving average technique.

Volvo Penta also promoted using gyroscopes to stabilize the image sensors at the stern to help prevent all the bouncing around reflected in the images. We later claimed the use of optical image stabilization in this situation.

We noted Volvo’s patent took 6 years to issue. That is a sign of a difficult path and a firm really wanting to push through a patent in this particular field (the digital anchor application may have been the focus of much of Volvo Penta’s hopes for a patent). Hopefully Brunswick’s recent ‘154 application will not require that long to be issued in some form.

---

Boiler Plate / Fluff Language in the Brunswick Virtual Propeller Guard Patent Application

Brunswick fleshed out the invention they describe, but much of the patent is boilerplate language. It broadly talks about possible approaches such as Sections 0031 and 0032 on Storage Devices.

Storage system (item 34) in Brunswick’s current patent application is where the image scanning software and the trained neural network are stored.

[0031] The storage system 34 can comprise any storage media, or group of storage media, readable by processor 36, and capable of storing software 32. The storage system 34 can include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Storage system 34 can be implemented as a single storage device but may also be implemented across multiple storage devices or sub-systems. Storage system 34 can further include additional elements, such a controller capable of communicating with the processor 36.
 

[0032] Examples of storage media include random access memory, read only memory, magnetic discs, optical discs, flash memory, virtual memory, and non-virtual memory, magnetic sets, magnetic tape, magnetic disc storage or other magnetic storage devices, or any other medium which can be used to storage the desired information and that may be accessed by an instruction execution system, as well as any combination or variation thereof, or any other type of storage medium. Likewise, the storage media may be housed locally with the processor 36, or may be distributed in one or more servers, which may be at multiple locations and networked, such as in cloud computing applications and systems. In some implementations, the storage media can be a non-transitory storage media. In some implementations, at least a portion of the storage media may be transitory.

They basically took a lot of space to say a lot of nothing. We probably do that sometimes too.

---

Brunswick I-JET Lab

In December 2017 Brunswick announced the Brunswick Innovation Laboratory at University of Illinois at Urbana-Champaign was expanding and held a Grand Opening in a new building know known as the Brunswick I-JET Lab. We bring this up here because the college environment would be a great place to further develop the Virtual Propeller Guard invention Brunswick describes in their US 2017/0323154 patent application.

We have long encouraged College Student projects in this exact field as seen on our “Research Projects for Senior Design Classes, Masters Thesis Projects, Industrial Design & Other Researchers” page. We hope Brunswick takes advantage of their increased ties to the University of Illinois to exploit these opportunities.

---

Our Prior Art

While we (PropellerSafety.com, previously known as the Propeller Guard Information Center) are glad to see Brunswick developing Virtual Propeller Guard projects, we continue to ask them to cite our work when they do. We just went through this last year with Brunswick’s Active Trim control patent.

While it is always up to patent examiners to determine if certain patent claims are unique, we suggest Patent Claim #1 of Brunswick’s current US2017/03231154 A1 Patent Application could struggle against our prior art.

As mentioned above the swimmer detection portion of the 1st claim of Brunswick’s current patent application basically claims a swimmer detection system for a boat with at least one sensor to capture an image of the water around the boat, a computer, an image scanning module (turns the image into digital format), an Artificial Neural Network trained to recognize patterns within these images associated with swimmers, and an output indicating if a swimmer is in the water near the boat or not.

The full claim is a little broader to include the ability to detect predefined objects on the vessel or in the water, allowing the patent to also cover an Operator Presence Detection system, a debris detection system, as well as detecting small vessels off the bow of larger boats.

Note, Brunswick filed their current patent application on 5 May 2016.

Back to just the swimmer portion of the 1st claim, below is a COPY of a 20 June 2010 entry on our Propeller Guard Technologies Page.

************************************************************
Copy (the posted date and “trainable neural networks” were bolded in this copy for emphasis)

Posted 20 June 2010 – Virtual Propeller Guard Detects Absence of Water
 

In the past, we and others have suggested virtual propeller guards based on various means of directly detecting the presence of people in the water (infrared, sonar, image detection, etc.)
 

Why not just try to detect water, then act when water is NOT detected. For example, detect water in the danger area behind a boat just about to be started by scanning the area using infrared sensors to recognize if everything is the same temperature (at times when water temperature is not real close to the surface temperature of humans). If contiguous areas of non-water are discovered, run algorithms or a trainable neural network on them to detect the presence of people. If people are in the water, sound an alarm and prevent the engine from being started.
 

This same method (detecting the non-presence of water) could be used as one of multiple variables to help prevent false positives in other virtual propeller systems. It could be more strongly relied on on days when water temperatures were further from human body temps.

***************************************************************
If anybody does not believe the date provided above (20 June 2010 as the public disclosure date) and on the original entry, these materials were moved to our new PropellerSafety.com website in 2011. They were originally posted in a “Propeller Guard Information Center” portion of our RBBI.com website. Those materials can still be viewed on the Internet Archive (archive.org). Below is a January 4, 2011 capture of that page by the Internet Archive. Anybody can find that same page using the URL at the top in conjunction with the capture date. I added the red box around a portion of the text in this copy for emphasis.

If that was not enough, we posted another invention a year earlier that even more closely resembles Brunswick’s current invention.

On 2 February 2009 we posted three Propeller Guard Information Center (PGIC) inventions, made public disclosure of them, and place all parts not previously patented by others into the Public Domain for all to use. One of those inventions was a System for Reducing False Alarms in Virtual Propeller Guard Systems Using Infrared Sensors.

An image of this post as captured by the Internet Archive on 13 February 2009 (just 11 days after we posted it) is supplied below. The red boxes and arrow were added to this copy for emphasis. We basically put forth the concepts previously put forward by Brunswick in the patent literature at that time, then claimed additional use of one or more methods to reduce false alarms and improve accuracy of the detection system. The last item on the list, marked by the red arrow, is “Use of Neural Networks allowing the system to learn”.

Both historical Propeller Guard Information Center invention posts can be seen today at their respective dates on our Propeller Guard Technologies Prior to 2011 page or on the Internet Archive at the URLs and capture dates on their respective images above.

It is no secret to Brunswick that we are a source of prior art in the field of Virtual Propeller Guards. I promoted detecting people in the water at MerCruiser in the early 1990s. In 2003, Brunswick filed a “Related Art Statement” regarding one of the early Staerzl heat sensor /detector patents U.S. Patent 7,105,800. The two page document discusses multiple references attributed to us in this field that were then posted on rbbi.com. These articles were later moved to our new PropellerSafety.com website in 2011.

Brunswick’s 2003 letter notes our work “constitutes the closest art of which the applicant(s) is /are aware relating to the invention …” They go on to identify 4 specific documents on our RBBI.com site which the examiner later listed as “Other References” on the patent. Brunswick goes on to expound us proposing numerous types of sensors for these applications and us stating the system be electronically capable of shifting the gears to neutral, setting off an alarm, or handling the situation in some other way.

Brunswick has long known we are working in this area and that we are one of the leading sources to be searched for the current developments in the field of Virtual Propeller Guards. For example, while not specifically mentioning recreational boat applications, many swimmer detection technologies and papers are identified on our Pre 2011 Propeller Guard Technologies page.

---

Volvo Penta Prior Art Statement

If you do not feel our prior art creates problems for Brunswick’s first patent claim, we refer you to Volvo Penta’s digital anchor patent, US Patent 8,195,381 we discussed earlier. Volvo’s patent issued about 3 years before Brunswick’s ‘154 patent application was filed. while Volvo’s patent focused on digital anchor technologies, it also included a Virtual Propeller Guard. We repeat a quote from that patent below.

Volvo Penta U.S. Patent 8,195,381
 
“Computation of a moving average is a most reliable approach to detecting a presence of the one or more persons in the water in comparison to neural network-type detection or template comparison detection which are also within the scope of the present invention.”

Volvo Penta is openly saying their approach to identifying swimmers near the vessel in a marine environment (moving average detection) is superior to “neural-network type detection”.

In 2006, when Volvo Penta filed their ‘381 patent, Volvo Penta wrote their approach is superior to the neural network approach which Brunswick came along 9 years later in 2015 and tries to patent.

---

A New Reference in This Field

We suggest those interested in using neural networks in the detection of swimmers or floaters in open water (rivers, lakes, oceans) review a late 2017 paper from Korea.

Efficient Swimmer Detection Algorithm Using CNN-based SVM.
by Dasol Hong and Yoon Kim.
Journal of The Korea Society of Computer and Information.
Vol.22. No. 12. pgs.79-85. December 2017.

The authors promote the use of a Convolutional Neural Network (CNN) specialized for small object sizes to reduce the number of errors from waves being detected as swimmers (False Positives) or swimmers being detected as waves (False Negatives).

The web site provides the abstract in English, and an opportunity to purchase the full paper (in Korean). Several mobile apps can provide a rough machine translation via OCR (Optical Character Recognition).

---