Obstacle Avoidance of Flapping?Wing Air Vehicles Based on Optical Flow and Fuzzy Control

，，，gyuan，*

1.School of Automation and Electrical Engineering，University of Science and Technology Beijing，Beijing 100083，P.R.China；2.Institute of Artificial Intelligence，University of Science and Technology Beijing，Beijing 100083，P.R.China

Abstract: The flapping-wing air vehicle（FWAV）is a kind of bio-inspired robot whose wings can flap up and down like bird and insect wings. A vision-based obstacle avoidance method for FWAVs is proposed in this paper. First，the Farneback algorithm is used to calculate the optical flow field of the first-view video frames taken by the on-board image transmission camera. Based on the optical flow information，a fuzzy obstacle avoidance controller is then designed to generate the FWAV steering commands. Experimental results show that the proposed obstacle avoidance method can accurately identify obstacles and achieve obstacle avoidance for FWAVs.

Key words：dense optical flow；monocular vision；obstacle avoidance；flapping-wing air vehicle；fuzzy control

0 Introduction

The flapping-wing air vehicle（FWAV），in?volving bionics，aerodynamic analysis，mechanical structure analysis，energy，communication，control and other multidisciplinary technologies，is a new type of unmanned aerial vehicle that mimics the flight mode of birds or insects［1］to achieve flight［2-3］.The strengths of FWAVs are small size，high flight efficiency，low noise and strong flight maneuverabil?ity［4］. Therefore，FWAVs are promising to be used in military reconnaissance，field exploration，natu?ral disaster rescue and other fields.

Obstacle avoidance is one of the indispensable capabilities for autonomous flight［5-7］. A FWAV has quite limited load capabilities，so common sensors such as laser rangers and radars are not suitable.With the advantages of light weight，low power con?sumption and rich information，vision sensors have proved to be a good choice for obstacle avoidance of FWAVs［8］. However，designing a practical obstacle avoidance system for FWAVs is still a problem to be solved.

One way of achieving obstacle avoidance is to use stereo vision. For example，the DelFly II［9］of the Delft University of Technology in The Nether?lands contains an on-board stereo vision system which has two CMOS cameras running simultane?ously at 30 Hz. DelFly II successfully avoids the walls of the indoor office space many times during the 72 s flight test. The experiment clearly demon?strates the potential of stereo vision for autonomous obstacle avoidance of FWAVs. The defections of DelFly II include short flight time and poor imaging quality of the visual perception system，so it cannot be used for more complex tasks.

The optical flow method，inspired from in?sects［10］，is also an effective navigation means of FWAVs. Researchers adopt spherical cameras to obtain translational optical flow，and use it as the in?put of hovering flight and vertical landing control of an vertical take-off and landing unmanned aerial ve?hicle［11］. Edge-FS is a vision algorithm to determine velocity and depth［12］. A FWAV with an embedded STM32F4 is used as the experimental platform and the feasibility of Edge-FS is demonstrated on this platform. Optical flow is utilized to measure altitude for feedback control of a flapping-wing microrobot of 101 mg［13］. Ryu et al. from Seoul National Uni?versity designed a FWAV with first-person view module and proposed an algorithm based on the time-to-contact（TTC）. TTC was calculated from the optical flow［14-15］. Experiments demonstrated that the proposed algorithm could realize simple tar?get tracking and obstacle avoidance.

Fuzzy control technology based on fuzzy theory is an advanced technology in modern industrial auto?mation control［16-18］. It simulates human approximate reasoning as well as decision-making process，and is a fuzzy set control theory based on control language rules and fuzzy reasoning［19-20］. The complex struc?ture of the FWAV makes it difficult to establish an accurate mathematical model，so a fuzzy control method is suitable for controlling FWAVs.

The main contributions of this paper are as fol?lows：（1）Designing an obstacle avoidance method for FWAVs based on dense optical flow and fuzzy control；（2）implementing real experiments to veri?fy the effectiveness of the proposed obstacle avoid?ance method.

The structure of this paper is organized as fol?lows. Section 1 introduces the principle and calcula?tion process of Farneback optical flow，as well as the principle of fuzzy control and the design process of the fuzzy controller. Section 2 describes the pa?rameters of the FWAV，image transmission camera and flight control module. In Section 3，the control system of the FWAV is tested and the obstacle avoidance experiment results are presented. Section 4 draws the conclusions and extends future work.

1 Algorithm for Obstacle Avoid?ance

1.1 Optical flow

1.1.1 Farneback dense optical flow

Optical flow is the instantaneous velocity of the pixel movement of a space moving object on the im?age plane［21］. It was put forward by Gibson in The Perception of the Visual World［22］in 1952. There are two basic assumptions：（1） When a target moves between different frames，its brightness will not change；（2）The pixel displacement between adjacent frames ought to be small. Then the basic constraint equation of optical flow can be obtained［23］

whereI(x，y，t) means the intensity of pixel point(x，y) in the image plane at timetand dx；dythe small displacements of pixel between the frame at timetandt+dtinx-axis andy-axis，respectively.

There is only one constraint equation，and there are two unknown variables in Eq.（2），so we cannot directly calculate the optical flow vector. In the literatures，there are many methods to calculate the optical flow［24-26］. This paper computes the opti?cal flow by the method proposed by Farneback［27］.

Unlike the traditional sparse optical flow meth?ods that only calculate the optical flow of feature points，the Farneback method，a dense optical flow methods computes the optical flow of all pixels on the image. The sparse optical flow methods，like the pyramid L-K method，sometimes fail to detect the corners of obstacles so that the FWAV fails to avoid obstacles. The pyramid L-K method is com?pared with the Farneback method. The result of the pyramid L-K method is shown in Fig.1，the trunk is not detected and the result of the Farneback method is shown in Fig.2.

Fig.1 Results of the pyramid L-K method detection of ob?stacles

1.1.2 Obstacle detection

When the FWAV moves forward，the camera transmits images to the ground station. Each image is divided into the left and the right parts，and the obstacle avoidance system calculates the optical flow field of the left and the right parts，seperately.

The flight of the FWAV will not be affected by obstacles on the edge of the image，so we only need to calculate the optical flow in the central area of each image，and compute the normalisation of the difference of optical flow values between the left and the right fields［28］

whereFL，FRare the average vector moduli；WL，WRthe sums of optical flow vector moduli，and the subscript L，R indicate that the variable is the value of the left or the right optical flow field.There is no obstacle whenis very small. Therefore，ifewill be very large，soeis set to 0 when（εis a threshold value）.

Ife＞0，the FWAV needs to turn right；ife＜0，the FWAV needs to turn left；ife=0，the FWAV keeps going straight. Therefore，the differ?ence valueein Eq.（3）is set to the input of the fuzzy controller.

1.2 Fuzzy control

1.2.1 Theory of fuzzy control

The fuzzy control process is divided into three steps：Fuzzification，fuzzy logic inference and de?fuzzification judgment［29］. It is completed by the fuzz?ifier，fuzzy inference engine and defuzzifier［30］. The structure of a fuzzy controller is shown in Fig.3.

Fig.3 Structure of a fuzzy controller

1.2.2 Fuzzy controller design

The fuzzy controller designed in this paper has two inputs and one output，where the inputs areein Eq.（3）and its derivative term?，and the output is the FWAV control signalu. The symbol ofurepre?sents the turning direction of the FWAV，and the amplitude ofurepresents the turning amplitude.

The fuzzy subsets ofe，?，uare［NB，NM，NS，ZO，PS，PM，PB］，and the quantificational field do?main is［-3，-2，-1，0，1，2，3］. Note that“N”means the outputu＜0 ，and the FWAV will turn left；“ZO”means the outputu=0，and the FWAV will go straight；“P”means the outputu＞0，and the FWAV will turn right；“S”“M”and“B”repre?sent the outputs，that is，the steering amplitudes of the FWAV. In order to obtain the corresponding fuzzy subset ofe，?，u，they must be converted to the interval［-3，3］. The transformation equations are described as follows

Fig.4 Membership functions

After obtaining the fuzzy subsets ofe1and，the fuzzy subset ofu1can be obtained through the fuzzy rule table which is derived from experiences.The rule-table of fuzzy inference system （FIS）aboutu1in this paper is as follows.

In Table 1，“N”means that if the output ＜0 ，the FWAV will turn left；“P”means that if the out?putu＞0 ，and the FWAV will turn right.“S”，“M”and“B”represent the outputs，that is，the steering amplitudes of the FWAV.

The fuzzy operation relationships that are formed by 49 conditional statements can be obtained from Table 1. The calculation process of one condi?tional statement“ife1，?，thenu1”is as follows.

Table 1 Rule?table of FIS about u1

Step 1Calculate the 7×7 matrixD1

whereMe1，Me?1are the matrices of the fuzzy subset ofe1ande?1.

Step 2Transform matrixD1into 49-dimen?sional column vectorDT1

Step 3Calculate the 49×7 fuzzy relationship matrixR1

After finding the matrices corresponding to all 49 conditional sentences，take the union of the ma?trices（take the larger element of them）and finally get the fuzzy relationship matrixR.

Based onDT1andR，the matrix ofu1can be calculated byu1=DT1×R. Therefore，we can get the fuzzy subset ofu1.

After obtaining the fuzzy subset ofu1，it must be defuzzified and divided byKuto obtain the true output value of the fuzzy controller. If the fuzzy sub?set ofuis composed by fuzzy subsetX1and fuzzy subsetX2，the value ofu1can be defuzzified by

wherex1，x2are the corresponding values of the fuzzy subsetsX1andX2，respectively，when the membership degree is 1；μX1andμX2the member?ships of fuzzy subsetX1and fuzzy subsetX2，respec?tively. The controller outputucan be calculated by Eq.（6）.

The entire control system is a closed loop as shown in Fig.5.

Fig.5 Diagram of control system block diagram

2 Hardware Preparation

2.1 FWAV platform

As shown in Fig.6，the FWAV adopts an Xwing structure with two pairs of wings. The FWAV weighs 18 g and has a wingspan of 27 cm. The fuse?lage is very light because it is made from hard plas?tic，and the wings are made from polyvinyl chloride.The flight of the FWAV is driven by a 6 mm hollow cup motor with a speed of 4 200 r/min，and the transmission ratio is 1∶25.4. The power supply is provided by a 3.7V-70 mA·h lithium battery. The flight speed of the FWAV is about 5 m/s，and the wing flapping frequency is 16 Hz. Due to its light body weight and small wing area，the FWAV has weak wind resistance and can only fly indoors or out?doors in windless weather.

Fig.6 FWAV platform

As shown in Fig.7，the FWAV is equipped with a video transmission camera module to trans?mit images to the ground station. The camera weighs 4.6 g，the size of the captured image is 640 pixel×480 pixel，and the FOV is 45°. It com?municates with the ground station through 5.8 GHz wireless signal.

Fig.7 Camera data transmission module

The ground station is a laptop，whose configu?ration is shown in Table 2.

Table 2 Configuration of the ground station

The ground station software is written by C++ based on the Qt5 platform. The image pro?cessing part is based on the OpenCV open-source function library. The main interface of the ground station software is shown in Fig.8.

Fig.8 Main interface of the ground station software

2.2 Flight control module

Flight control is realized by a self-developed flight control module，which is shown in Fig.9. The parameters are as follows：Length 20.45 mm，width 15.74 mm，weight 6.24 g. The flight control module contains a 32-bit low-power chip STM32F1 03C8T6 based on ARM Cortex-M3 architecture，voltage regulator module xc6206，two 5 V servo output ports，and a 5 V communication interface.The circuit board uses 5 V power supply. During the flight，the FWAV sends image information to the ground station through the camera data transmis?sion module，and the ground station sends control commands to the data transmission module to con?trol the steering of the FWAV.

Fig.9 Flight control module

The ground station sends the control signal to the FWAV through HC-12 wireless serial communi?cation module，as shown in Fig.10. The whole ex?periment system of obstacle avoidance is shown in Fig.11.

Fig.10 HC-12 wireless serial communication module

Fig.11 Experiment system

3 Experimental Results

3.1 Control system test

Before conducting the FWAV obstacle avoid?ance experiment，a video is taken with a camera in hand，which is used as an input to the controller to test whether the controller’s output has the desired effect.

The parameters of the fuzzy controller are as follows：emax=1，=10，umax=1，Ke=3，Ke?=0.3，Ku=3 and the thresholdε=400，which is de?termined by the flight experiments in a background monotonic environment.

As shown in Fig.12，we use a tree as an obsta?cle and walk towards it from far to near and recordFL，FR. The controller outputs results during the process，as shown in Fig.13.

At 3—5 s the obstacle is close and in the right half of the field，thus the controller outputuis nega?tive. The video is taken by a person walking with a hand-held camera，so the optical flow vector is small，and the fuzzy subset ofuis almost NS.

Fig.12 Recorded sequential images (testing)

However，the output has unexpected glitch atu＞0，since the selected background color is not mo?notonous. To solve the problem，we filter the output signalu，calculate the average value of it every five frames and output every five frames，as shown in Fig.14.

Fig.14 Control signal output every five frames

3.2 Experimental results

In the experiments，two boxes are used as ob?stacles for the FWAV and the obstacles are shown in Fig.15. We conduct two experiments to verify whether the FWAV can successfully avoid obsta?cles in different situations.

During the first obstacle avoidance experiment，the distance between the two obstacles is set to 1.0 m. When the FWAV turn left to avoid the first obstacle，the second obstacle is at the right side of the FWAV，causing the FWAV to continue to turn left. The first flight path of the FWAV is shown in Fig.16. The recorded images are shown in Fig.17and the data of experiment is shown in Fig.18.

Fig.15 The obstacles

Fig.16 The first flight path of the FWAV

Fig.17 The recorded images of the first flight

Fig.18 The data of the first flight

Since the obstacles always appear in the right half of the FWAV，FL＜FRandu＜0. The FWAV avoids the first obstacle before timeT1，and the differenceeis large during the periods 0.2—0.4 s and 1.6—1.8 s because the FWAV is close to the obstacles.

During the second obstacle avoidance experi?ment，the distance between the two obstacles is set to 1.5 m. When the FWAV turns left to avoid the first obstacle，the second obstacle is at the left of the FWAV，causing the vehicle to turn right. The second flight path of the FWAV is shown in Fig.19，the recorded images are shown in Fig.20 and the data of experiment is shown in Fig.21.

Fig.19 The second flight path of the FWAV

Fig.20 The recorded images of the second flight

Before timeT2，FL＜FRandu＜0，so the FWAV turns left to avoid the first obstacle. After timeT2，FL＞FRandu＞0，then the FWAV turns right to avoid the second obstacle. The differ?enceeis large during the periods 1.2—1.4 s and 2.4—2.6 s because the FWAV is very close to the obstacles. During 2.7—2.9 s the outputuis unrea?sonably less than 0，which is due to the complex background.During 3—3.7 s，the FWAV has avoid?ed obstacles，but the outputuis still large because there is a basketball hoop which is incorrectly con?sidered as an obstacle by the obstacle avoidance sys?tem.

The above two experiments show that the steering of the FWAV is not affected by wind and load capacity，and the proposed obstacle avoidance method is feasible and effective. However，the con?troller may be susceptible to the influence of the complex environment.

Fig.21 The data of the second flight

4 Conclusions

In this paper，aiming at the obstacle avoidance problem of FWAV，a vision-based obstacle avoid?ance method is proposed based on the Farneback op?tical flow algorithm and fuzzy control. In addition，an X-wing aerial vehicle is modified as the experi?mental platform for experiments. Experimental re?sults demonstrate the effectiveness of the proposed obstacle avoidance method. The proposed method does not require complex mathematical modelling of the FWAV and is simple as well as easy to imple?ment. In future work，we will add effective filters to the control system to make it achieve better perfor?mance under complex backgrounds，and use neural network to train the controller parameters to achieve more precise control.