Andre Chia, Cheslav Balash

Edith Cowan University, Perth, WA 6028, Australia

Abstract Currently joysticks are used for ROVs’ interface, though they prove to be tedious and there have been cases of severe damage to the vehicle and subsea architecture due to operator fatigue and attention loss.There is anecdotal evidence to suggest that the ease of manoeuvring with a more intuitive interface design would minimise these damage risks and psychological pressures related to operation.In this study we explored options for a more intuitive physical modality of ROV’s interface.Specifically, we modelled alternative controllers and their operating functions in a 3D environment to provide insight on how they could be used.Three possible conceptual ideas were then proposed, which include the joystick with gesture controllers, the steering wheel with the trackball and the steering wheel with gesture controllers.To further strengthen the ideas presented within this paper and reduce the likelihood of authors’ bias, industry experts were consulted, and a formal survey targeting ROV pilots was conducted.To establish the most superior design, a trade-off analysis was conducted by developing a weighted ranking system based on the supporting and opposing arguments and the design’s intuitiveness, with human psychology being used as another means of justification.

Keywords: ROV; interface; Locomotion; Manipulator.

1.Introduction

Remotely operated vehicles (ROV) are waterproof enclo- sures that are used to carry out subsea intervention, histor- ically performed by commercial divers [ 1 , 2 ].When ROVs were first introduced, each thruster was controlled by its own rheostat: a variable resistor that controls the current [3] .In the case of the thruster control, it would ultimately change vehi- cle speed and direction by adjusting the thruster’s rotational speed [ 1 , 3 ].A system of this nature would make the control of the vehicle extremely difficult as multiple rheostats would need to be adjusted to make a simple change in the vehi- cle’s position.Technology gradually evolved, and the joystick was implemented with thruster scaling and vectoring, allow- ing gradual vehicle acceleration and turning.Compared to the original control method, the joystick allowed more enhanced vehicular control and finer locomotion [1] .

A joystick is a peripheral input device that is made up of a vertical stick, which pivots about its base [4] .Gener- ally, joysticks provide control via two axes of motion: for- ward/backward and left/right.There are many iterations of the device, some with varied means of determining the stick position; however, the working principles are largely simi- lar: the controller constantly reports the stick’s position back to the controlling program, which then executes the relevant commands to trigger the desired response [4] .Joysticks have been used for many years in various applications including controlling aircrafts, earth movers and wheelchairs; it is also an increasingly popular choice in gaming for virtual machine control.Depending on the application, buttons and triggers are also added to make other functions more readily accessible to the user, for instance allowing pitch and yaw functions in vehicle locomotion [1] .For a comprehensive review on the historical development of ROV manipulator controllers and current state of art, the reader can refer to [5] .

While the joystick provides a significant improvement in the control of ROVs compared to its predecessors, it still proves to be tedious and there have been cases of severe damage to the vehicle and subsea architecture due to opera- tor fatigue and attention loss.There is anecdotal evidence to suggest that the ease of manoeuvring with a more intuitive interface design would minimise these damage risks and psy- chological pressures related to operation.Currently there is a number of auxiliary technologies considered for improved movement controls (e.g.intellectual support systems, assist- ing AUVs) [ 6 , 7 ].In this study we focused on conceptually exploring options for a more intuitive physical modality of ROV’s interface.

2.Proposed Approach

As the problem at hand is very open-ended and it deals with evolving ideas, a Soft System Methodology (SSM) cycle [8] was adopted as follows:

As a starting point, a review of relevant enabling tech- nologies was completed, with arguments both for and against each type.Each controller, along with its operating functions, was then modelled in a 3D environment to provide insight on how the controllers could be used.To further strengthen the ideas presented within this paper and reduce the likelihood of authors’ bias, industry experts were consulted, and a for- mal survey targeting ROV pilots was conducted.The survey results were analysed using ‘IBM SPSS’ and the relevant cor- relations were explored.To establish the most superior design, a trade-off analysis was conducted by developing a weighted ranking system based on the supporting and opposing argu- ments and the design’s intuitiveness; with human psychology being used as another means of justification.

3.Overview of Enabling Technologies

The idea of joystick-controlled cars has been around for over 15 years but has not been adopted in any mass-produced cars due to its impracticalities; a study by [9] has shown that steering wheels are superior to joysticks for lane track- ing.Though the comparison between ROVs and cars may not be completely adequate, there is some merit behind the increased average tracking deviation from joysticks in gen- eral.The same study also suggested that some level of force feedback is beneficial in both lane tracking and evasive ma- noeuvring

Joysticks are used in many vehicles that require three axes of control and have a separate throttle control, including air- craft and fighter jets.The setup for ROVs utilises the fore/aft movement for acceleration, meaning pitch controls must be relocated [1] .

Compact controllers were first introduced into the gaming market with consumers responding well to them due to their simple and easy control.However, in many specialised games other means of control are used: the joystick for flying and the steering wheel for driving [10] .

Steering wheels are used in most vehicles that have two axes of motion, though the steering wheel itself has only one axis [9] .A steering wheel design should have a good operator response, due to the familiarity of users to the system [10] .

Fig.1.Position control layout.

The keyboard and mouse are used in many computer games, they but were not specifically designed for gaming or vehicle control.Gamers respond well to the setup of the mouse and control keys as the latency between the gamer’s input and the desired action is small.If the gamer makes a mistake, then a quick recovery on their behalf can fix it [10] —such a fast-responding system in a real-life application would be difficult; an ROV with this type of control would have significant thruster lag, so it is easy to make an over- correction or under-correction when a mistake is made [1] .

The gesture controller was designed with virtual reality to enable gamers to carry out physical movements to interact with new games [11] .These controllers track the movement of the user’s hands and arms, whilst also giving them access to analogue sticks and customisable buttons to make other interactions with the games [11] .Given the good consumer response to interact with the game on such a physical level, a similar control system for an ROV may be justified.

The trackball has a similar function to the joystick; how- ever, its movement is more representational of how a 3D ob- ject would move when influenced by external force [12] —having a design like this may work well in tandem with an- other type of technology.

4.Alternative Solution Development

4.1.Controller Layout

Figs.1–7 show options of the control layouts for the vari- ous controllers considered as possible control interface alter- natives.Note that the functions can be inverted or customised to accommodate for pilots’ personal preferences.

Fig.2.Joysticks control layout.

4.2.Possible Conceptual Designs

There preliminary designs were created to stimulate dis- cussion with industry experts:

[i] Joystick with Detachable Gesture Controllers

The joystick is the original design, however with suitable thruster vector scaling, the control of the ROV can become more natural and intuitive.The design can be coupled with de- tachable gesture controllers to allow an easier handling of the manipulators and other tools on board.Both the joystick and the gesture controllers will provide force feedback to enhance the operators’ experience and interaction.The gesture con- trollers will have force-feedback triggers to adjust the hands of the manipulators.

[ii] Steering Wheel and Trackball

Fig.3.Gamepod control layout.

The steering wheel is superior at lane tracking when com- pared to the joystick, so the steering wheel can be used to navigate the ROV left and right, whilst accelerating and brak- ing with pedals.This coupled with the trackball will enable the operator to control the pitch, yaw and roll functions.The manipulators can be controlled by the trackball when the ve- hicle is stationary.

[iii] Steering Wheel with Detachable Gesture Controller

The same concept as [i], but instead of using a trackball system the steering wheel will have detachable gesture con- trollers of the manipulators.The function buttons on the steer- ing wheel will provide the operator with roll and pitch func- tions.Full-lock steering will provide lateral movement and the paddle shifters will provide up/down movement.

4.3.ROV Pilots Surveying

An internet-based survey amongst ROV pilots from three companies were distributed (ECU ethics approval number 19153CHIA).All identifiable information on individuals has either not been obtained or omitted from this paper.

The survey for ROV pilots and designers is outlined below.

[1]Whatisyourjobposition?

Fig.4.Steering wheel control layout.

Fig.5.Keyboard and mouse control layout.

[2]HowlonghaveyoubeenworkingwithROVs?

[3]ForeachoftheROVfunctionsbelow,rateoutof10,theimportanceofbeingabletointuitivelycontrolthatparticularfunction.[N.B.1=LowestImportance]

X&Ymotion

Zmotion

Yaw

Pitch

Roll

[4]Foreachofthemanipulatorfunctionsbelow,rateoutof10,theimportanceofbeingabletointuitivelycon-trolthatparticularfunction.[N.B.1=LowestImpor-tance]

Y&Zmotion

Armrotation

Armextension

Jointmovement

Grabbing

[5]Doyouagreethatsomelevelofforcefeedbackisben-eficialinROVcontrol?

Table 1 ROV locomotion ?number of surveyees who selected each weighting

[6]Doyouexperiencefatigueorattentionlossafterpro-longeduse?

[7]WhatdoyoufindisthehardestpartofoperatinganROV?

[8]WhatarethemostimportantfeaturesyouwouldlookforinanintuitivecontrolinterfaceforanROV?

Fig.6.Gesture control layout.

Table 2 Manipulator locomotion ?number of surveyees who selected each weighting

The survey was completed by 20 ROV pilots, ranging from 5 ?30 years of experience.Tables 1 and 2 show the number of pilots who selected each weighting for questions [3] and [4] respectively.It is important to recognise that with the limited sample size, no accurate conclusions can be made about the ROV pilot population.However, combining the re- sults with the literary research and the expert interactions may provide some merit.

The results show that almost 100% of the respondents se- lected an importance weighting of ten for theX,Y,Zandyawmotion in ROV locomotion.There are a few inconsis- tent selections with the weightings forpitchandrollmotion, however they have a median weighting of four and two re- spectively.Looking at the manipulator locomotion;Y,Z,jointmovementandgrabbingall have an importance weighting of ten selected by most of the respondents.Arm extension is the exception, with weightings ranging from four to ten and having a median of ten.

80% of respondents agreed that some level of force feed- back would be beneficial for control; noting that for a few, their comments stated their response was purely about manip- ulator control.15% even suggested it would not be beneficial for ROV locomotion at all due to the increased strain on the wrist.90% of respondents believed they experience fa- tigue or attention loss after prolonged use, mostly due to the intense concentration required and the repetitive tasks.The respondents’ responses to the hardest part of operating an ROV were mixed.For many, the lack of situational awareness and visibility are the hardest things to deal with, some draw- ing comparisons to walking around whilst looking through straws.Others stated their biggest issue was controlling the locomotion of the ROV or dealing with, what they thought were, ‘ridiculous client requests’.Finally, what the respon- dents thought were the most important features in an intuitive control interface came down to a few things; most saying they wanted something simple, not overloading the user with un- necessary information.Others stated they wanted better auto- matic positioning, station keeping, camera and sonar systems implemented; improving situational awareness and visibility.A couple of respondents also believed that touch screens or panels should not be used, under any circumstances.

Fig.7.Trackball control layout.

‘IBM SPSS’ was used to analyse the correlation between the‘yearsofexperience’and the‘selectedweightings’.The results shown in Table 3 reveal the Pearson correlations be- tween each of the variables, with the column titled as‘1’showing the relevant correlations between the variables.The values in column‘1’lie between ?0.3 and 0.3 implying thatthere is no significant correlation between‘yearsofexperi-ence’and the‘selectedweightings’(based on [13] ).The data suggests that amongst the respondents, the number of years of experience does not seem to determine a particular weight- ing of importance for a particular locomotive function (as the scatter plot in Fig.8 does not have a linear trend).

Table 3 Pearson correlations of survey data

Fig.8.Weightings selected by ROV pilots versus their experience in years.

4.4.Trade-Off Analysis

The weighting for vehicle and manipulator of locomotion shown in Tables 4 and 5 respectively were based on the im- portance of being able to intuitively control that particular function.XandYmotion in vehicle locomotion andYandZmotion in manipulator locomotion have been coupled for simplicity, due to usual linked control operation.Two sets of weightings are shown: based on the literary research and survey responses, which were then averaged.

Table 4 TOA ?vehicle locomotion weighting

X&Y,Zandyawmotion are the most used functions and require a high level of precision to control them [14] , hence they were each given a weighting often.Pitchandrollare less frequently used and when they are, they do not require finite control [1] .These functions were weighted asfiveandtworespectively.For manipulator control,Y&Z,armrota-tion,jointmovementandgrabbingall require the most intu- itive control interface to be able to accurately perform general manipulator tasks [1] .These functions received a weighting often.Armextensionis not a required function, however it can provide the operator with extra reach [1] .The same results can be achieved by utilising joint movement or by moving the vehicle closer to the infrastructure being manipulated, for this reason it was weighted aseven.

Table 5 TOA ?manipulator locomotion weighting

Table 6 TOA ?vehicle locomotion performance score

Tables 6 and 7 show how well the controllers can perform vehicle and manipulator locomotive tasks.Each function is scored out of ten with one being the lowest.

The joystick is more capable at controlling the vehicle than manipulators.For the vehicle control ofX&Y,Zandyawmo- tion, the performance score is equal, due to the type of move- ment required of the control device.These functions scored aneightfor performance because of the well-defined control actions that need to be carried out by the pilot to complete said functions.The human-joystick interaction works as most pilots would expect it to, moulded from their past interac- tions with the physical environment.For example, pushing the joystick forward, causes the vehicle travel forward.They scored lower than a perfect score due to the relatively small range of motion of the joystick, increasing the likelihood of unintentional movement; however, this can be improved with appropriate acceleration vector scaling.Vehiclepitchandrollfunctions are less natural and harder to control on the small analogue stick; for this reason, they have each been given a performance score ofsix.The analogue sticks have a much shorter length, making finite control extremely difficult.Ma- nipulator functions,Y&Zmotion andarmrotationhave been given an equal performance score of seven for the same rea- son mentioned above.Armextensionhas been given asixdue to the more difficult control operations on the small analogue stick.Jointmovementis a tedious task to perform on the joy- stick as all joints cannot be moved simultaneously.Instead a switch needs to be activated to change over the control so it was given athreefor its performance.The controller activates thegrabbingfunction when the pressure sensitive trigger is depressed, given the short travel distance of the trigger, finite control is difficult; therefore, it was scored afour.

The position controller was designed solely for the control of the manipulators and cannot properly carry out vehicle locomotive functions, hence why it only scored aonefor all vehicle control.All of the manipulator functions scored aneight, apart fromgrabbingwhich scored aseven.This is because the degrees of motion of the replica arm are identical to the manipulator itself, so when the replica arm is moved the manipulator arm mimics it.The tip of the replica arm is pressure sensitive which controls thegrabbingfunction, it scored less than the rest due to its decreased ability for finite control.

The gamepad requires the same type of pilot input for all vehicle locomotive tasks, which is why they all have an equal score.It scored asixdue to the small range of motion of the analogue sticks and triggers.The gamepad scores worse in manipulator control as the control layout can be confusing and again the small analogue sticks make control more difficult.All manipulator functions scored afive, except forgrabbingwhich scored afour.

The steering wheel is designed for vehicle control and does not work well with manipulator control.This controller scored anineforX&Ymotion as both the wheel and accelerator can be controlled with great precision.It scored asixforZmotionpitchandrollagain because of the small range of mo- tion of the analogue sticks and paddle shifters.This controller obtained a perfect score oftenforyawas the 180 degrees of rotation allows for high precision control.For manipulator control it scores atwofor all functions exceptarmrotationin which it scored aneight.This is due to the awkward controls and small range of motion of the analogue stick.Rotating the wheel allows for accurate control ofarmrotation.

Table 7 TOA ?manipulator locomotion performance score

Table 8 TOA ?Vehicle locomotion weighted performance score

Table 9 TOA ?manipulator locomotion weighted performance score

The keyboard and mouse provide little options for finite control, scoring athreefor all locomotive functions.It is even more difficult to control the manipulators due to the lack of ability to properly define the controls in a well-placed manner; it was scored atwofor all manipulator functions.

The gesture controller, similarly to the gamepad, uses two small analogue sticks for vehicle control; scoring asixforX&Y,Z,yaw,pitchandrollfunctions.There are sensors which track arm and hand movement, making it much better for manipulator control.The manipulator functions similarly to the human arm, meaning that tasks can be acted out and the manipulators can mimic the actions.This controller scored a perfecttenfor all manipulator tasks, apart fromarmextensionwhich scored afive, as this function requires the use of the triggers.

Finally, the trackball works similarly to the joystick in ve- hicle control forX&Yandyawmotion, scoring aneightfor its performance.Zmotion,pitchandrollare controlled by the rotation of the ball.The ball can rotate indefinitely in any direction allowing for high precision control, which is why it scored aninefor those functions.For manipulator control, the trackball is similar to the joystick forX&Zmotion andjointmovement, scoring asevenandthreerespectively.Armrotation,armextensionandgrabbingare controlled by the rotation of the ball, all scoring aninedue to the high level of control.

Tables 8 and 9 show the weighted performance score for each type of controller.The weighted performance scores were calculated with the following equations:

No single controller scored the highest in both vehicle and manipulator control, but as in typical operations there is one pilot for ROV locomotion and one for manipulator control, it is not necessary for both controllers to be of the same type.In vehicle control, the trackball scored the highest with a weighted performance score of 308.5 out of a maximum score of 365.In manipulator control, the gesture controller scored the highest, with a weighted performance score of 442.5 out of a maximum score of 485.

4.5.Proposed Design Concepts

The debate on whether or not to add resistive feedback to the controller is still inconclusive, due to the relatively small number of survey responses and the somewhat contradictory research done by [9] .The proposed suggestion is to have a controller that has an easily customisable force feedback fea- ture, where the pilot can toggle it off and on or adjust how much feedback they receive.The ‘smart panel’ with a uni- versal hub should be used so that pilots can chose their own control layout, with the ability to revert back to the old sys- tem if they do not like the new controller.At this point, the chosen vehicle controller is the trackball ( Fig.7 ), and the cho- sen device to control the manipulator is the gesture controller ( Fig.6 ).The sensors placed on the arms can be improved to provide force feedback; however, they may be unnecessary weight for the pilot to carry.Instead, they could be devel- oped with a customisable haptic (vibrational) feedback.The proposed design concepts by no means, final designs, but a mere starting point to further considerations and the possible action for change.

Future work could also include the programming of the controllers to test their functions and to deal with any un- foreseen issues.Research into the visual components of the user interface could be done, such as new camera and sonar systems; possibly integrating augmented reality so that the pilots can still operate in muggy conditions.Current auto- positioning systems can be reviewed, and further research conducted to determine how they can be improved or rein- vented.

Declaration of Competing Interest

The Authors have no interests to declare.

Acknowledgments

The authors are most thankful to Paul Colley (Total Marine Technology), Graeme Gordon (DOF Subsea), and Professor Geoffrey Roy (Edith Cowan University) for their interest in the project and invaluable shared insights and feedback.