Basic theories and methods of target’s height and distance measurement based on monocular vision

1. Introduction

Mobile robots have been widely used in various fields such as surveillance, visual navigation, automatic image interpretation, human-computer interaction, and virtual reality [1−2]. The machine vision technique tends to simulate visual functions of human eyes for extracting information from video sequences and recognizing the morphology and motion of three-dimensional scenes and targets [3−5].

Machine vision plays a vital role for the performance of mobile robots [6−8]. Like human beings relying on vision to obtain most of the information, mobile robot vision systems use cameras to capture and process the information. The machine vision technology is therefore crucial for the robot to properly process its received information. In machine vision, 3D information acquisition remains a challenging problem in the field of autonomous navigation [9], 3D scene reconstruction [10], visual measurements [11, 12] and industrial automation [13].

Existing methods of machine vision can be roughly divided into two categories. The methods in the first category try to acquire the target depth information, including binocular perception methods [14], monocular camera calibration methods [15, 16], target depth information estimation methods [17, 18] and single camera plus plane mirror-based methods [19, 20]. While the methods of the second category are based on the sensor positioning technology.

The binocular perception method simulates human stereo vision, which obtains the target depth information from the difference of viewing angles between the direct view of the left and right eye [13]. Yang et al. [14] used a binocular camera stereo matching method for gesture recognition. In this method, parallax is obtained from the relative information between two cameras, which is then used to calculate the depth information of the target. However, the accuracy of this method depends on the performance of the camera, such as illumination and baseline length (i.e., the distance between two cameras). Further, this method has a limited capability for real applications due to high complexity and a relatively large data processing load.

Camera calibration methods can be divided into the traditional approach [21, 22] and self-calibration approach [15, 16]. Abdal-Aziz and Karara [23] proposed a camera calibration method based on direct linear transformation. Tsai et al. [24] devised a radial uniform constraint calibration method. Zhang et al. [25] introduced a camera calibration method for active vision. These traditional methods could achieve high accuracy of calibration. However, these methods require specific calibration references. In the calibration process, due to the limitations of equipment, it is still impossible to accurately record the corresponding coordinates of target in the world coordinate and image coordinate system. Further, the accuracy of coordinate conversion fluctuates accordingly.

The deep learning-based methods [17, 18] employ neural networks and large-scale data learning to obtain the depth information of targets. Liu et al. [26] designed a Full Convolutional Network (FCN) to perform a single-eye estimation scene depth map. Eigen et al. [27] developed deep neural networks to estimate depth cues in RGB images. He et al. [28] conducted deep learning on a large-scale fixed focal length data set to synthesize a variable focal length data set, thus generating deep cavity in the image. Zhang et al. [29] proposed a deep guidance and regularization (HGR) learning framework for end-to-end monocular depth estimation. Although these methods can be used to estimate target depth, it is difficult to obtain accurate depth of the target, which limits its application to mobile robots.

In recent years, monocular depth estimation has been widely applied in the field of simultaneous localization and mapping (SLAM). Loo et al. [30] proposed CNN-SVO where the three-dimensional point information obtained from monocular depth estimation was used to improve SVO. Tateno et al. [31] proposed real-time monocular dense SLAM. With the help of a depth estimator, this method is robust and accurate. David et al. [32] devised a multi-scale architecture to simultaneously predict depth, surface normal and semantic labels. This method can capture many image details without relying on super pixel segmentation. Liu et al. [26] proposed a deep convolutional neural field model for depth estimation. In this method, unary and binary potentials of continuous CRF are learned in a unified deep network. The model is based on a fully convolutional network with a super pixel pooling scheme. Similar methods were also designed in [33, 34], which combine the CNNs and CRF technology. In these methods, two-layer CRF are used to enhance the collaborative operation of global and local prediction values to obtain the final depth prediction results. Cao et al. [35] converted the depth estimation problem to a pixel-level classification problem. The continuous ground truth depth value is first discretized into multiple depth intervals, then a label is assigned to each interval according to the depth value range, and the depth estimation problem is transformed into a sub-classification problem. Finally, the depth is obtained using a fully convolutional depth residual network. Zheng et al. [36] proposed a layout-conscious convolutional neural network (LA-Net) for accurate monocular depth estimation, in which a multi-scale layout map is employed as a structural guide to generate a consistent layout depth map.

The monocular camera plus flat mirror schemes [19, 20] are methods designed for observing the fish target in a water tank. These methods could be used to effectively solve 3D occlusion tracking of the fish in a water tank. However, they are not suitable for mobile robots or moving targets when realizing 3D tracking of the target.

The non-visual sensor [37, 38] uses information including sound waves, infrared, pressure, and electromagnetic induction to sense the distance of an external target to the robot, thus obtaining the depth information. To work out the 3D coordinates of the target, this approach requires visual imaging and sensors, and this increases the data processing burden of the robot and requires synchronization between the vision and the sensor. As a result, such an approach is generally not efficient in mobile robots.

In this paper, we propose a target’s height and distance measurement method. In the proposed method, we adopt an operation of “self-variant” to achieve the measurement of target information in monocular vision, which requires no additional supplement devices. Our proposed method can accurately obtain the target height and distance information of the target. Such a method can overcome the shortcomings of traditional target depth measurement methods, and help robots to obtain target information measurements and track in a moving environment. This paper analyzes the insolvability of “self-invariance” and the solvability of “self-change” through theoretical and experimental analysis. The main contributions of this work are as follows:

1) A target’s height and distance measurement method is proposed of “self-change” and a relationship model of parameters is established including the target distance, target height, camera height, image size, target image size and camera parameters, etc.

2) The experiment proves that under the conditions of unknow target height and “self-invariant”, the target distance cannot be calculated even there are multiple views available.

3) It is theoretically proved that under the condition of “self-variant”, the target distance can be calculated using two different views.

The above contributions provide a theoretical basis for researchers engaged in target measurements in the field of computer vision, and make it possible for scholar to use a single image for target depth estimation. At the same time, the target depth can only be calculated when the height or distance of the two images is changed.

The rest of the paper is organized as follows. Section 2 introduces the basic concept of “self-change”. Section 3 provides the relationship model between the distance and height of the target when it is in the visual center. The infinite solution analysis of “self-invariance” is given in Section 4. Section 5 presents the solvability analysis of “self-change”, In Section 6, experimental verification is carried out while Section 7 concludes the work.

2. Target’s height and distance measurement method of "self-change"

For monocular vision, the target distance should be properly addressed to achieve target tracking and location. Based on our previous research [39], it is easy to achieve target tracking and positioning when the target height is known beforehand. For real applications, however, it is difficult to obtain the height of the target. In this case, certain machine learning technique is required to estimate the target’s height.

The performance of monocular vision is mainly affected by following parameters: 1) the focal length of the camera, which is denoted as $f$(mm), 2) the size of the photosensitive film of CMOS, which is denoted as $w\times l\left({\mathrm{m}\mathrm{m}}^2\right)$, 3) the height of the camera, recorded as ${\mathrm{h}}_{\mathrm{r}}$, 4) the target distance, denoted as $\mathrm{m}$, and 5) the height of target AB, denoted as ${\mathrm{h}}_{\mathrm{o}}$. We call the first two parameters as internal parameters while the other three as external parameters.

Definition 1： The term of “self-change” refers to the change of camera’s height ${\mathrm{h}}_{\mathrm{r}}$ or the target distance m. For example, the robot may change the height of the camera ${\mathrm{h}}_{\mathrm{r}}$ during the process of stretching or bending its legs. The target’s distance $\mathrm{m}$. can be changed while it moves forward or backward. Here, we call ${\mathrm{h}}_{\mathrm{r}}$ and $\mathrm{m}$ as variable external parameters.

The proposition opposite to “self-change” is “self-invariance”. It refers to the cases that the camera’s height ${\mathrm{h}}_{\mathrm{r}}$ and the target’s distance $\mathrm{m}$ keep unchanged. It should be noted that the target height ${\mathrm{h}}_{\mathrm{o}}$ cannot be changed, so we call it an invariable external parameter.

To carry out target’s height and distance measurements through "self-change", we first should understand typical relationship models of the target height and distance.

3. Relational model of target height and depth when the target is in the center of the vision

To solve the problem of target depth measurement when the target is in the visual center, we first need to define the term of “target is located in the visual center”.

Definition 2: From the point of view of the 3D imaging process, the so-called “target is located at the center of vision” is that the line passing through the highest and lowest vertexes of the target can intersect the main optical axis of the camera in space. In the two-dimensional plane of target image, it means the line or extension line between the highest and lowest points of the target image passing through the center point of the visual image.

According to the position of the main optical axis of the camera, we divide the relationship model of the target distance and target height into the following categories: 1) the model of imaging geometry when the target vertex falls on the main optical axis of the camera; 2) the model of imaging geometry when the target vertex is higher than the main optical axis of the camera; 3) the model of imaging geometry when the target vertex is lower than the main optical axis of the camera; and 4) the relationship model of the target distance and target height of the angular bisecting image.

Before introducing these models, we first explain the concept of the depression angle.

Definition 3: The so-called depression angle is the angle between the main optical axis and the horizontal line when the camera is shooting downwards, marked as $\alpha$.

As shown in Figure 1, if we assume that the horizontal line of the camera is FE and the main optical axis is $FB{}^{\prime }$, then the depression angle $\alpha$ is $\mathrm{\angle }EFB{}^{\prime }$. To measure the horizontal distance between target AB and the camera, we first set the coordinates of the center point of the image plane to be $\mathrm{C}{}^{\prime }(w/2,l/2)$. The size of the digitized image is $M\times N\left({\mathrm{p}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{l}}^2\right)$, and the digital coordinates of image CD of target AB after image segmentation are $C(X_C,Y_C)$ and $\mathrm{D}(X_D,Y_D)$. Then, we can calculate the length $d$ of the target image CD, the distance $d_1$ from point D to the center point of the image plane ${\mathrm{C}}^{\prime }$, and the distance $d_2$ from point C to the center point ${\mathrm{C}}^{\prime }$ of the image plane as:

In the following sections, we should introduce the relationship models.

Figure 1. Two-dimensional geometric principle of camera imaging.

3.1. Model of camera imaging geometry with the target vertex falling on the main optical axis of the camera

When the highest point of the target just falls on the main optical axis of camera, it means that the target vertex is at the center of vision. To measure the horizontal distance $\mathrm{m}$ between target AB and the camera, we use the bottom position of the camera as the center of the coordinate system, the horizontal line of the machine vision direction as X axis, and the line perpendicular to the OX axis on the ground plane as Y axis. Also, we set the camera position as point F, then the coordinate system shown in Figure 2 is established with the side of OF as Z axis. Suppose that the image plane is ${\text{π}}_1$, the ground plane is ${\text{π}}_2$, and the projected image plane ${\text{π}}_1$ is CD after the target AB is imaged by the camera. Obviously, points A, B, C, D, O and F are all on the plane where XOZ is located. Based on this, we can easily derive the relationship between the length $d$, target distance $m$, target height $h_o$, and camera height $h_r$:

From the basic principle of camera imaging, we know that since the vertex B of target |AB| falls on the main optical axis, the image C of point B must be the center of the image plane ${\pi }_1$. Then, we can calculate the target distance $m$ as:

Figure 2. Schematic diagram of imaging with the target vertex falling on the main optical axis

3.2. Model of imaging geometry when the target vertex is higher than the main optical axis of the camera

When the target vertex is higher than the main optical axis of the camera, then in the image plane ${\text{π}}_1$, the vertex C of the target image CD is higher than the center point of the image plane as shown in Figure 3. Suppose that the main optical axis of the camera and target AB intersect at point ${\mathrm{B}}^{\prime }$, and set $\left|\mathrm{A}{\mathrm{B}}^{\prime }\right|=\mathrm{x}$, then $\left|{\mathrm{B}}^{\prime }\mathrm{B}\right|={\mathrm{h}}_{\mathrm{o}}-\mathrm{x}$. Consequently, we can easily obtain the following relationship of these parameters:

Finally, the relationship between the target height $h_o$ and the target distance $m$ can be written as:

Figure 3. Schematic diagram of imaging when the target vertex is higher than the main optical axis of the camera.

3.3. Model of imaging geometry when the target vertex is lower than the main optical axis of the camera

When the target vertex is lower than the main optical axis of the camera, then in the image plane ${\text{π}}_1$, the vertex C of the target image CD is lower than the center point of the image plane as shown in Figure 4. Suppose that the main optical axis of the camera and the extended line of target AB intersects at point $B^{\prime }$, and set $\left|\mathrm{A}{B}^{\prime }\right|=\mathrm{x}$, then $\left|B^{\prime }\mathrm{B}\right|={\mathrm{x}-\mathrm{h}}_{\mathrm{o}}$. Since the coordinates of CD in the image plane ${\text{π}}_1$ are $\mathrm{C}({\mathrm{X}}_{\mathrm{C}},{\mathrm{Y}}_{\mathrm{C}})$ and $\mathrm{D}({\mathrm{X}}_{\mathrm{D}},{\mathrm{Y}}_{\mathrm{D}})$, and $C^{\prime }$ is the center point of the image plane, the simulation coordinates of $C^{\prime }$ are $C^{\prime }(w/2,l/2)$. We can express the target distance as:

Figure 4. Schematic diagram of imaging when the target vertex is lower than the main optical axis of the camera.

3.4. Special model: relationship model of the target’s distance and height of the angle bisecting image

Definition 4：In a certain image acquisition process, if the main optical axis of the camera bisects the angle between two endpoints of the target and the camera. This kind of image is called the angle bisection image (ABI).

As shown in Figure 1, the main optical axis bisects $\mathrm{\angle }AFB$, so we can easily derive the relationship between the depression angle $\alpha$ and the CD length $d$:

In Eq. (7), $f$ is the focal length of the camera, $\tan (\cdot )$ is the tangent function, ${\alpha }_1={\tan }^{-1}\left(h_r/m\right)$, ${\alpha }_2={\tan }^{-1}\left((h_r-h_o)/m\right)$, and ${\tan }^{-1}(\cdot )$ is the arctangent function.

Fig. 5 is a function diagram of the relationship between the depression angle and the target image length obtained with the target AB height 680 (mm), distance (mm), focal length of 50 (mm), and camera height of (mm). It can be seen from Figure 5 that when the depression angle $\alpha$ gradually increases from zero, the CD length d gradually decreases until it reaches a certain value. After that, it gradually increases. In other words, $d$ has a minimum value. Taking the derivative of Eq. (7), we can easily know that when $\alpha =({\alpha }_1+{\alpha }_2)/2$, $d$ reaches the minimum value. This is due to:

Therefore, when the main optical axis is the bisector of $\mathrm{\angle }AFB$ (the green part in Figure 1), $d$ reaches the minimum value. According to the angle bisector theorem in geometry, we can get the following equation:

Based on Eq. (9), we can obtain the relationship between the target’s distance and height of ABI as:

Figure 5. Relationship between the depression angle $\alpha$ and target image CD length $d$.

4. The solvability analysis of “self-invariance”

The “self-invariant” solvability analysis is to jointly solve above models without changing the camera’s height $h_r$ and the target’s distance $m$ to see whether the target distance $m$ and the target height $h_o$ can be solved. Before analyzing the solvability, we first agree on a joint solution symbol. The unknown variables for the joint solution of several equations are marked as “$\bigwedge \{\cdot \}$”. For example, $\bigwedge \{\left(1\right),\left(2\right)\}$ means Eqs. (1) and (2) are used to perform the joint solution.

In Section 2, we derive four relational models. To discuss whether their joint form has a solution, we will first introduce a basic theory.

4.1. Infinite solutions of multi-model combination of a single frame image

The so-called single frame image multi-model combination refers to an image in which a single frame image satisfies two or more model conditions. For example, an ABI image does not only satisfy Eq. (10), but also satisfy Eqs. (5-1) and (5-2). To discuss whether there is a solution to the combination, we first introduce a basic theory.

4.1.1. Basic theory

For given camera parameters (including the camera height ${\mathrm{h}}_{\mathrm{r}}$, focal length $f$, photosensitive film size $w\times l\left({\mathrm{m}\mathrm{m}}^2\right)$ and image size $M\times N\left({\mathrm{p}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{l}}^2\right)$), can the target distance $\mathrm{m}$ and target height $h_o$ be solved by single-frame image multi-model combination? In other words, does $\bigwedge \{\left(5\right),\left(10\right)\}$ have a unique solution?

It seems that$\bigwedge \{\left(5\right),\left(10\right)\}$ may have a unique solution. There are two unknown variables in Eq. (5), i.e., the target distance m, height $h_o$, and the distance $\mathrm{x}$ between the intersection of the main optical axis, the target and the lowest point of the target. Formula (5) has two equations (5-1) and (5-2), and Formula (10) has these three variables. With three equations and three variables, it seems that the target distance and height could be solved. However, $\bigwedge \{\left(5\right),\left(10\right)\}$ has infinite solutions. In the following, we shall give a theorem about infinite solutions to joint multiple models of a single image.

Theorem 1: For any frame of the target image CD, under the condition of the same shooting height, images obtained from different depression angles will have different target heights and distances.

Proof: The equivalent meaning of the above theorem is that, given the height $h_o$of target AB, the horizontal distance from the camera is $\mathrm{m}$ and the obtained image is CD when the depression angle is $\alpha$. Under the condition that the camera height $h_r$ remains unchanged, change the depression angle and shoot. Let us set a new depression angle ${\alpha }_1$, that is, $\alpha \Rightarrow {\alpha }_1$. Then, there must be a new target $A_1{B}_1$, so that the image $C_1{D}_1$ of the new target is identical to the original target image CD, which is recorded as $CD\equiv C_1{D}_1$.To prove this theorem, we only need to prove its equivalence.

When shooting the target AB, the main optical axis is $FB{}^{\prime }$, the depression angle is $\alpha$, and the intersection of the main optical axis and image plane ${\text{π}}_1$ is $C{}^{\prime }$. Obviously, $C{}^{\prime }$ is the center point of the image. Suppose the angle between the two endpoints of targets A and B and the camera is $\beta$. That is, $\mathrm{\angle }AFB=\beta$, which is shown by the black line in Figure 6.

Figure 6. Principle of target imaging when rotating the main optical axis.

The conclusion of the above theorem is that $CD\equiv C_1{D}_1$, which means that the length of the target image does not change, and the position of the target image in the image does not change. This also means that the distance will not change between the two endpoints of the target image and the center point of the image plane ${\text{π}}_1$. In the following section, we will employ a construction method to prove this conclusion.

The bind straight lines AF and BF to the active axis $F{B}^{\prime }$ (that is, $\mathrm{\angle }AF{B}^{\prime }$ and $\mathrm{\angle }BF{B}^{\prime }$) remain unchanged, and $F{B}^{\prime }$ can rotate around point F on the XOZ plane. When $\alpha \Rightarrow {\alpha }_1$, the intersection points of the straight-line AF and the OX axis is $A_1$, and a straight line parallel to AB is made through point $A_1$, intersecting the main optical axis $F{B}^{\prime }$ at $B_1^{\prime }$ and BF at $B_1$ respectively. This is shown in the red part in Figure 6. Since straight lines AF and BF are bound to $F{B}^{\prime }$, the image CD equivalent to AB is also bound to the AFB tripod. During rotation, $\mathrm{\angle }AFB$, $\mathrm{\angle }AF{B}^{\prime }$ and $\mathrm{\angle }BF{B}^{\prime }$ always remain the same, i.e., $\beta ={\beta }_1$, $\mathrm{\angle }AF{B}^{\prime }=\mathrm{\angle }{A}_{1}F{B}_1^{\prime }$ and $\mathrm{\angle }BF{B}^{\prime }=\mathrm{\angle }{B}_{1}F{B}_1^{\prime }$ are true. Also, since the focal length $f$ of the camera is fixed during the rotation, the length of the target image $\left|CD\right|$ and the position information of the target image CD in the image will not change. Therefore, the target $A_1{B}_1$ obtained is the new target, and the new target image $C_1{D}_1$ is identical to the original target image CD. Therefore, $CD\equiv C_1{D}_1$ is established. Proof completed.

According to Figure 6, we can clearly know that when $\alpha \ne {\alpha }_1$, $m\ne m_1$ and $h_o\ne h_{o1}$ hold. Of course, we can also see that the distance $m$ is a monotonically increasing function of the height $h_o$. This theorem shows that the single-frame image multi-model combination cannot solve the two variables of the target distance $\mathrm{m}$ and height $h_o$. This is due to that a single frame image can only obtain the information of the target image CD, including the length of CD and the position information of CD in the image. We can only obtain the values of $\mathrm{\angle }AFB$, $\mathrm{\angle }AF{B}^{\prime }$ and $\mathrm{\angle }BF{B}^{\prime }$ through this information. There are infinite number of targets satisfying this condition.

To further illustrate the conclusion of "the infinite solutions of the single-frame image multi-model joint", we take $\bigwedge \{\left(5\right),\left(10\right)\}$ as an example and the details are described below.

4.1.2. $\bigwedge \{\left(5\right),\left(10\right)\}$ infinite solution analysis

$\bigwedge \{\left(5\right),\left(10\right)\}$ represents the problem of existence of the solution value of ABI. Therefore, the angle bisector image satisfies the condition that the main optical axis of the camera is lower than the target vertex and satisfies the angle bisector theorem in geometry.

In Eq. (11), $d_1$ is the distance from the center point of the image plane ${\text{π}}_1$ to point D of the image CD, and $d_2$ is the distance from the center point of image plane ${\text{π}}_1$ to point C of the image CD. Both $d_1$ and $d_2$ can be worked out by image segmentation and CD coordinates are obtained. Then, the digital transfer to the simulation method is shown in Figure 3. $f$ and $h_r$ represent the focal length and height of the camera, where $x$ represents the distance from the intersection point of the target and the optical axis to the ground, respectively, both of which are known beforehand. Therefore, there are only three unknown variables in the equation system, namely $\mathrm{m}$, $h_o$, $x$, and the equation system has three equations. From the number of equations and the number of variables, it seems that the equation can give the unique target height and target distance. Unfortunately, this equation also has an infinitely number of solutions.

The specific theoretical basis can be explained according to Theorem 1. We regard (11-1), (11-2), and (11-3) as three curved surfaces. In Figure 7, the subgraphs (a), (b) and (c) are three curved surfaces at the target height $h_o$. It is a cross-sectional view of 580, 480, and 380. In each sub-graph, the three curves can intersect at one point. In other words, as the target height $h_o$ changes, a unique target distance value m can be obtained. The sub-figure (d) in Figure 7 is a cross-sectional view of $x=0$. It can be seen from the sub-figure that the three curved surfaces intersect in a curve, not a point, and the target distance m increases monotonically along with the target height $h_o$. This is consistent with our Theorem1.

Figure 7. A sample diagram of $\bigwedge \{\left(5\right),\left(10\right)\}$ with infinite solutions of the ABI image.

4.2. Infinite solutions of the multi-frame image joint

In Section 4.1, we have analyzed the infinite solutions of the single-frame image multi-model joint, that is, it is impossible to solve the target height and distance from a single frame image. So, can multiple images be used for the joint solution? For example, can we solve the target height and distance by combining different depression angle images? Let us first conduct theoretical analysis on this issue.

4.2.1. Basic theory

The multi-frame and multi-model joint situations with different depression angles are as follows. 1)$\bigwedge \{\left(3\right),\left(5\right)\}$: represents a model when the target vertex happens to fall on the main optical axis and the target vertex is higher than the main optical axis. 2) $\bigwedge \{\left(3\right),\left(6\right)\}$ represents a model when the target vertex happens to fall on the main optical axis and the target vertex is lower than the main optical axis. 3) $\bigwedge \{\left(5\right),\left(6\right)\}$ represent a model when the target vertex is higher than the main optical axis and the target vertex is lower than the main optical axis. Since the ABI image is also a model with the target vertex higher than the main optical axis, we will not discuss the joint problem of the ABI image and other models.

$\bigwedge \{\left(3\right),\left(5\right)\}$ and $\bigwedge \{\left(3\right),\left(6\right)\}$ are three equations with three unknown variables $(m,h_o,x)$, so the three unknown equations can be solved. In $\bigwedge \{\left(5\right),\left(6\right)\}$, there are four equations with four unknown variables $(m,h_o,x_1,x_2)$, where $x_1$ is the distance from the lowest point of the target to the intersection point of the main optical axis of the camera and the target AB when the target vertex is higher than the main optical axis of the camera, and $x_2$ is the distance from the lowest point of the target to the intersection of the camera’s main optical axis and the target AB when the target vertex is lower than the camera's main optical axis. The four variables can also be solved. However, no matter for which set of equations, the unique target depth information cannot be found without knowing the target height. We first present a theorem as follows.

Theorem 2: Under the condition that the camera height $h_r$ the target height $h_o$ and the distance $m$ are unchanged, the target height obtained by the image taken at any depression angle is equivalent to the target distance relationship model.

Proof:

Suppose that shooting a target at a depression angle $\alpha ={\alpha }_1$, the image obtained is $I_1$, and the relationship model between the corresponding target height and distance is denoted as $R_1(m,h_o)$. When shooting a target at a depression angle $\alpha ={\alpha }_2$, the image obtained is $I_2$, and the relationship model between the corresponding target height and distance is denoted as $R_2(m,h_o)$. From a geometric point of view, on the plane $mO{h}_o$, two curves (or straight lines) $R_1$ and $R_2$ appear. On the same plane, there are only three kinds of relationships between $R_1$ and $R_2$: 1) $R_1$ and $R_2$ intersect but do not overlap. 2) $R_1$ and $R_2$ never intersect. 3) $R_1$ and $R_2$coincide.

Under the condition that the target height $h_o$ and distance $m$ are unchanged, if we can prove that $R_1$ and $R_2$ coincide, then it is proved that the target height obtained by the image taken at any depression angle is equivalent to the target distance relationship model. Here, we use an elimination method to prove.

If we substitute $(m,h_o)$ into $R_1$ and $R_2$ respectively, and $R_1$ and $R_2$ are true, then it indicates that "$R_1$ and $R_2$ never intersect" will not hold. Below we use the method of proof by contradiction to prove that "$R_1$ and $R_2$ intersect but do not overlap" does not hold.

Assume that "$R_1$ and $R_2$ intersect but do not overlap" is true, that is, there is at least one point $(m^{\prime },h_o^{\prime })$ to make $R_1(m^{\prime },h_o^{\prime })$ true but $R_2(m^{\prime },h_o^{\prime })$ is invalid. Suppose that the target image corresponding to $R_1(m^{\prime },h_o^{\prime })$ is $I_1^{\prime }$. Based on Theorem 1, we know that $I_1\equiv I_1^{\prime }$ holds. At this time, ${m\ne m}^{\prime }$ and $h_o\ne h_o^{\prime }$ hold. This contradicts our condition that “the target height $h_o$ and distance $m$ remain unchanged”. In other words, our hypothesis that “$R_1$ and $R_2$ intersect but do not overlap” does not hold. Therefore, only one case that “$R_1$ and $R_2$ coincide” holds. Proof completed.

It can be explained by Theorem 2 that the multi-frame image joint has infinite solutions. That is, the equations $\bigwedge \{\left(3\right),\left(5\right)\}$, $\bigwedge \{\left(3\right),\left(6\right)\}$,$\bigwedge \{\left(5\right),\left(6\right)\}$ all have infinite solutions. In the following, we take $\bigwedge \{\left(5\right),\left(6\right)\}$ as an example to further illustrate that $\bigwedge \{\left(5\right),\left(6\right)\}$ has infinite solutions.

4.2.2. $\bigwedge \{\left(5\right),\left(6\right)\}$ infinite solution analysis

Since Eq. (3) is a special form of Eq. (5) and (6), the infinite solutions of $\bigwedge \{\left(5\right),\left(6\right)\}$ discussed is representative to a certain extent. Eq. (5) is the model of the imaging geometry when the target vertex is higher than the main optical axis of the camera. Eliminate the variable x in the Eq. (5) to obtain

Since $d=d_1+d_2$, then

Eq. (6) is the model of the imaging geometry when the target vertex is lower than the main optical axis of the camera. Eliminate the variable $x$ in the equation (6) to obtain

Since $d=d_1-d_2$, then

Since equations (13) and (15) are different models, the lengths of the target image in equations (13) and (15) are different, and the distances are also different from the lowest point and highest point of the target image to the center point of the image plane. To distinguish, we record the length of the target image, the distance from the lowest point of the target image to the center point of the image plane, and the distance from the highest point to the center point of the image plane in equation (15) as $d^{\prime }$, $d_1^{\prime }$ and $d_2^{\prime }$. Then, $\bigwedge \{\left(5\right),\left(6\right)\}$ can be simplified to

When the height of the camera in Figure 8 is 1040mm and the focal length of the camera is 50mm, the image CD is obtained by shooting, and the coordinates (in pixel) are $C\left(2009,3000\right)$, $D\left(3511,3000\right)$, $C^{\prime }\left(1995,3000\right)$, $D{}^{\prime }\left(3497,3000\right)$. After calculating, we will get (in mm) $d=5.85799$, $d_1=5.838299$, $d_2=0.0195$, $d^{\prime }=5.8578$, $d_1^{\prime }=5.8929$ and $d_2^{\prime }=0.0351$. In Figure 8, the two curves in the middle overlap completely. This further explains $\bigwedge \{\left(5\right),\left(6\right)\}$ has infinite solutions.

Figure 8. $\bigwedge \{\left(5\right),\left(6\right)\}$ infinite solutions example graph.

5. Solvability analysis of “self-change”

From Section 1 we know that “self-change” is to change the two variable external parameters $h_r$ and $m$. We will analyze the solvability of these two external parameters as follows.

5.1. Solvability analysis of changing the external parameter ${\mathit{h}}_{\mathit{r}}$

In case a soft robot tracking the target, the external parameter $h_r$ can be changed during the machine learning process. The soft robot can change the external parameter $h_r$ by stretching its height. For example, the soft robot can stand on tiptoe or stretch its neck to observe the target. It can also bend down or shorten the neck to observe the target. In these cases, the external parameter $h_r$ will change. Suppose that the target image obtained by the robot at the height $h_r$ is $I$, and the target image obtained after extending or shortening $\mathrm{\Delta }h$ is $I^{\prime }$, as shown in Figure 9. Then, the external parameter $h_r^{\prime }$ is

We stipulate $\overrightarrow{OZ}$ as the positive direction, then when the robot is shortened, $\mathrm{\Delta }h$ is negative, and when the robot is extended, $\mathrm{\Delta }h$ is positive. Similarly, when the main optical axis is higher than the target vertex, $d_2$ is negative, When the main optical axis is lower than the target vertex, $d_2$ takes a positive value. In this way, we can substitute Eq. (17) into Eq. (16), and obtain joint solving equations with changing external parameters as follows:

Figure 9. The imaging geometry principle of changing the external parameter $h_r$.

$d^{\prime }$,$d_1^{\prime }$,$d_2^{\prime }$ in Eq. (18) represent the length of the target image $C^{\prime }{D}^{\prime }$ in the image $I^{\prime }$, the distance from $C^{\prime }$ to the image center, and the distance from $D^{\prime }$ to the image center, respectively, as shown in the red part in Figure 9. Then, we analyze whether there is a solution to the equation set (18). We denote the curve Eq. (18-1) as $\xi \left(I\right)$, and the curve Eq. (18-2) as $\xi \left(I^{\prime }\right)$. We take the partial derivative of $h_o$ on both sides of Eq. (18-1):

Since $d_1{d}_2\ll f^2$, Eq. (19) can be simplified to

The numerator part of Eq. (20) is $d{h}_r+fm>0$. So, the denominator part determines the sign of the function $\phi (m,h_r,h_o)$. When $m>\left(f{h}_o\right)/\left(2d\right)$, it is clear that $\phi (m,h_r,h_o)$ is a monotonically increasing function with respect to $h_r$, as shown in Figure 9. Since $h_r^{\prime }<h_r$, the curvature of the curve $\xi \left(I^{\prime }\right)$ is smaller than that of the curve $\xi \left(I\right)$. The intercept of the curve $\xi \left(I\right)$ on the $h_o$ axis is $fd{h}_r(h_r-h_o)$, and the intercept of the curve $\xi \left(I^{\prime }\right)$ on the $h_o$ axis of the axis is $f{d}^{\prime }{h}_r^{\prime }(h_r^{\prime }-h_o)$. This is due to $d<d^{\prime }\ll h_r^{\prime }<h_r$, so$fd{h}_r(h_r-h_o)>f{d}^{\prime }{h}_r^{\prime }(h_r^{\prime }-h_o).$ That is, the intercept of the curve $\xi \left(I^{\prime }\right)$ on the $h_o$ axis is smaller than the intercept of the curve $\xi \left(I\right)$ on the $h_o$ axis, therefore, the curves of $\xi \left(I\right)$ and $\xi \left(I^{\prime }\right)$ have a unique intersection in the $m>\left(f{h}_o\right)/\left(2d\right)$ range. When $m<\left(f{h}_o\right)/\left(2d\right)$, function ${\displaystyle \frac{\partial m}{\partial h_o}}\left(h_r\right)<0$ and is monotonously decreasing, but the curvature of the curve $\xi \left(I^{\prime }\right)$ is still smaller than the curvature of the curve $\xi \left(I\right)$. Since the intercept of the curve $\xi \left(I^{\prime }\right)$ on the $h_o$ axis of the axis is smaller than the intercept of the curve ξ(I) on the $h_o$ axis, the curve ξ(I) and the curve $\xi \left(I^{\prime }\right)$ have no intersection within the range of 0 < $m<\left(f{h}_o\right)/\left(2d\right)$.

In summary, we can easily draw a simple graph of curves $\xi \left(I\right)$ and $\xi \left(I^{\prime }\right)$, as shown in Figure 10. From Figure 10, we can see that there is only a unique intersection point between curves $\xi \left(I\right)$ and $\xi \left(I^{\prime }\right)$ in the range of $m>0$, that is, there is a unique solution to the equation set (18).

Figure 10. Approximate diagram of curves $\xi \left(I\right)$ and $\xi \left(I^{\prime }\right)$ when the external parameter $h_r$ changes.

5.2. Solvability analysis of changing external parameter $\mathit{m}$

In case of a non-soft robot (e.g., a sliding robot and an unmanned car) tracking a target, it is difficult to learn the target’s height by changing its own height $h_r$, but it is able to learn the target height by changing the external parameter $m$. We can move forward or slide back to change the external parameter $m$. Suppose that when the sliding robot is at height $h_r$ and the distance from the target is m, the target image obtained is $I$, and the target image $I^{\prime }$ is obtained after sliding forward or backward by $\mathrm{\Delta }m$. As shown in Figure 11. the external parameter $m^{\prime }$ is

Figure 11. The imaging geometry principle of changing the external parameter $m$.

We stipulate that $\overrightarrow{OX}$ is the positive direction. Then, when the robot approaches $\mathrm{\Delta }m$ to the target, take the positive value, and when the robot moves backward, take the positive value of ∆m. In the target image, the $\overrightarrow{DC}$ direction is specified as the positive direction. Then, when equation (22) is substituted into equation (16), we have

$d^{\prime }$, $d_1^{\prime }$ and $d_2^{\prime }$in Eq. (22) represent the length of the target image $C^{\prime }{D}^{\prime }$ in the image $I^{\prime }$, the distance from $C^{\prime }$ to the image center, and the distance from $D^{\prime }$ to the image center, respectively, as shown in red in Figure 11.

Next, we analyze whether there is a solution to the equation set (22). Similarly, let us denote the curve formula (22-1) as $\xi \left(I\right)$ and the curve Eq. (22-2) as $\xi \left(I^{\prime }\right)$, which are shown in Figure 12. We take the partial derivative of $m$ on both sides of equation (22-1)：

When $m>\left(f{h}_o\right)/\left(2d\right)$, the curvature of the curve $\xi \left(I\right)$ increases monotonically. From Equation (23), it is difficult to judge the changing trend of the function ${\phi }^{-1}(m,h_r,h_o)$ with respect to the variable m. Therefore, we take the partial derivative of m with respect to the function ${\phi }^{-1}(m,h_r,h_o)$ as

When $\left(f{h}_o\right)/\left(2d\right)<m<(d^2{h}_r+f^2{h}_o)/\left(df\right)$, $\partial {\phi }^{-1}/\partial m>$0, the curve ${\phi }^{-1}(m,h_r,h_o)$ is a monotonically increasing function with respect to $m$, and the curvature of curve $\xi \left(I\right)$ is larger than the curvature of curve $\xi \left(I^{\prime }\right)$. When $>(d^2{h}_r+f^2{h}_o)/\left(df\right)$ ,$\partial {\phi }^{-1}/\partial m<0$, curve ${\phi }^{-1}(m,h_r,h_o)$ is a monotonically decreasing function with respect to $m$, and the curvature of curve $\xi \left(I\right)$ is smaller than the curvature of curve $\xi I^{\prime }$ at this time. Therefore, $m=(d^2{h}_r+f^2{h}_o)/\left(df\right)$ is the turning point of curve $\xi \left(I\right)$, which are marked as the dark red dot in Figure 12.

Since the intercept of the curve $\xi \left(I\right)$ on the $h_o$ axis changes from $fd{h}_r(h_r-h_o)$) to $f{d}^{\prime }{h}_r(h_r-h_o)$, and when the observer approaches the target, the target distance decreases while the target image becomes larger, namely $m>m^{\prime }$，$d<d^{\prime }$. Therefore, $fd{h}_r(h_r-h_o)<f{d}^{\prime }{h}_r(h_r-h_o)$. The curve $\xi \left(I^{\prime }\right)$ is at the upper left of the curve $\xi \left(I\right)$, as shown by the red dashed line in Figure 12. The red dashed line and the black curve will not intersect. It should be noted that $\mathrm{\Delta }m$ in Figure 11 is a negative number since $m^{\prime }=m+\mathrm{\Delta }m$. Therefore, it is equivalent to shifting the curve $\xi \left(I^{\prime }\right)$ to the right along the m axis $\left|\mathrm{\Delta }m\right|$ to get the red solid curve in Figure 12. Although $d^{\prime }$ is greater than $d$, $(d^{\prime }-d)\ll \mathrm{\Delta }m$. Therefore, the curves $\xi \left(I^{\prime }\right)$ and $\xi \left(I\right)$ will have two intersection points in the first quadrant, which are marked as blue points in Figure 12. That is, the solvability of the changing external parameter $m$ is true.

Figure 12. Approximate diagram of curves $\xi \left(I\right)$ and $\xi \left(I^{\prime }\right)$ when the external parameter $m$ changes.

6. Experimental Verification

To verify the correctness of our theory, we carried out the following experiments: 1) the target’s distance measurement experiment based on the change of the external parameter $h_r$ and 2) the target ’s height measurement based on the change of the external parameter $m$.

6.1. Preparation

6.1.1. Experimental instrument

The experimental instrument is shown in Figure 13. The instrument is mainly composed of the camera and rangefinder. The rangefinder is used to measure the distance between the target and camera. The overhead control handle is used to control the depression angle when shooting. The rotator is used to adjust the direction of the camera so that the camera faces the target. The rotating console is used to control the horizontal rotation angle of the rotator, so that the lens can be aligned with the target. The variable-length bracket is used to control the height of the camera, and the telescopic range of the tripod bracket is 0.71-2.10 m.

Figure 13. Experimental instrument.

To eliminate the error caused by the inaccurate target segmentation, we paint the target-strip steel pipe in red color. Such a target can be easily segmented by the color segmentation algorithm [40]. The lengths of the four target steel pipes are 120, 100, 80 and 50cm, respectively. The camera used in our experiment is SONY ILCE-7M2 with LAOWA Camera Lenses, which can reduce the error of target segmentation caused by distortion. The detailed parameters of the camera are shown in Table 1.

6.1.2. Experimental data

We collect two sets of experimental data, and capture four steel pipes. Each has four camera heights and four different distances, a total of 16 sets of images. We take them as the target’s distance data based on the change of the external parameter ${\mathit{h}}_{\mathit{r}}$ (as shown in Table 2) and the target’s height data based on the change of the external parameter $m$ (as shown in Table 3). During the shooting process, we start shooting downwards from the axis slightly higher than the target vertex, so that each group of images has 5 cases: the main optical axis is higher than the target vertex, the main optical axis is slightly lower than the target vertex, the main optical axis falls near the midpoint of the target, the main optical axis is slightly higher than the lowest point of the target, and the main optical axis is lower than the lowest point of the target. These cases are labeled 1, 2, 3, 4, and 5 as shown in the first column in Figure 14.

Figure 14. Sample image and target segmentation map.

6.2. Experimental results

To verify our theory and eliminate unnecessary errors caused by inaccurate segmentation, we manually segment the obtained images, which are shown in column (b) in Figure 14. The experimental results of these two kinds of the target’s height and distance are shown below.

6.2.1. Results of the target’s distance experiments based on the change of the external parameter ${\mathit{h}}_{\mathit{r}}$

The meaning of the data in Table 2 is that the target height is ${\mathrm{h}}_{\mathrm{o}}$ and the distance $m$ between the target and camera is unknown. We need to learn the distance and the camera height $h_r$ is known. In this case, we first take a target image. Then, we change the height of the camera and take another target image. The camera change amount is $\mathrm{\Delta }h$, and this change amount is known beforehand. The height and distance of the target are obtained through the joint solution of the two images. We denote the target height and distance obtained by solving $\bigwedge \{i,\mathrm{j}\}$ as ${(h}_o^{(i,j)},m^{(i,j)})$.

We conducted experiments on the data in Table 2, and the experimental results are shown in Tables 4 and 5. Out of 50 trials with a target height of 50 cm (as shown in Table 4), the minimum error of the target height is 0.1 mm, and the maximum error is 11.6 mm. The minimum error of the target distance is 1.1324 mm, and the maximum error is 195.7346 mm. The average error of the target height is 2.54 mm, and the average error of the target distance is 33.414 mm.