Enable voice control wherever a textbox/OSK exists? HOW??

Iceman_jkh · 12-15-2007, 03:25 PM

Does anyone know how I can enable Voice Control whenever the OSK/text Cursor appears?

I dont want to control RR with it, only enter text and have it auto trigger whever there is an opportunity to type.

EG:
Enter address in Satnav
Enter track title in music search
Fill in Album/Artist/Genre in CdRip plugin
Enter a phone number in Phoco/Freefone/RRmobile

Is there an elegant/correct way to have this occur? something like trapping every call to OSK, or detecting wherever there is a text cursor/input field on the screen?
Any other ideas?

Thanks
Ice

guino · 12-16-2007, 11:25 AM

Only by a means of a plugin you'd be able to check if RR is in either NUMOSK.SKIN or OSK.SKIN and have the voice control activated... I'd suppose you could also have a timer in NUMOSK.SKIN and OSK.SKIN which executes a command to activate the voice, and then have another command in the ESC/ENTER buttons of the OSK so it disables the voice control.

Iceman_jkh · 12-17-2007, 12:58 AM

Is there any way to detect if the pointer has become a text cursor ?
and doing it that way?

EL CAMINO · 12-17-2007, 01:34 AM

try this. simple very simple script.

Code:

#Include <Misc.au3>
AutoItSetOption("WinTitleMatchMode", 3)
$test = "test"
If WinExists($test) Then Exit 
AutoItWinSetTitle($test)

$rr=ObjCreate("RoadRunner.sdk")
if not IsObj($rr) then Exit
$oMyError = ObjEvent("AutoIt.Error","MyErrFunc")

HotKeySet("!{esc}", "_exit") ;for testing only, exit ALT+ESC disable when no need for it.

While 1
	If  $oMyError.windescription then 
		if Not WinExists("RoadRunner") then ExitLoop ;exit if RR is not Running.
	EndIf
		
    if MouseGetCursor() = 5 And _IsPressed("01") then
      MsgBox(4096, "", " text mode")
       $rr.execute("YOUR_COMMAND") ;place here your command
	EndIf
Sleep (3000)                ;push and hold 3 sec for executing.
WEnd
;==============================================================================================
;AI's error reporting Object
;==============================================================================================

Func MyErrFunc()
    Local $err = $oMyError.number
    If $err = 0 Then $err = -1
    SetError($err) 
Endfunc

Func _exit()
Exit
EndFunc

pRoFiT · 12-17-2007, 02:06 AM

Perfect thread.....Subscribed to thread

El Camino, do you have the same code for vb.net

I think i can create the roadrunner.sdk object like this

Code:

Dim rr As Object = CreateObject("RoadRunner.sdk")

hmm, strange but neat at the same time. If i do that roadrunner starts. so what..., i would be compiling a new program that would include RR? Or a new program that would include my features but call/open RR?

Okay tested a little more. So That will give me access to send to RR? And if my program is run without rr running, then rr will start. Otherwise RoadRunner can LOAD my program while entering a skin that would need my programs enteraction. Then you can have the skin close my program on exit.....hmmm.

Iceman_jkh, I would love to see what you end up with. I have a program that i'm secretly working on

that needs the same kind of control of RR.

pRoFiT · 12-17-2007, 02:38 AM

Hope the forum lets me double post

Anyways after a couple seconds i can send searchs to the search box like this.
Created a new form with button1 and textbox1

Code:

Public Class Form1

    Dim rr As Object = CreateObject("RoadRunner.sdk")

    Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
    End Sub

    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        Dim i As Integer
        Dim ch As String ' must be string to send :)
        For i = 0 To TextBox1.Text.Length - 1
            ch = TextBox1.Text.Chars(i)
            rr.execute(ch)
        Next i
        rr.execute("Enter")
    End Sub
End Class

Simple VB.net code

Iceman_jkh · 12-17-2007, 04:57 AM

Thanks El Camino.

EDIT: I got it to compile - turns out it must be saved as a .au3 file before it allows compilation (DOH!)

Basically, if the cursor becomes a text cursor, I want:
a) Audio to mute/pause
b) Mic to turn on
c) Accept commands/phonetic alphabet...until;
d) ENTER/command word is said (which is the equivalent of pressing ENTER)
e) turn mic off - as ENTER should change the cursor back to a pointer (not text cursor).
f) Audio resumes/unmutes

Profit> would also love to see what you are working on (in secret :P)

I'm hoping this little 'plugin' will make the voice control actually viable and useful!

Cheers
Ice

montgomeryt · 12-17-2007, 07:27 AM

Quote: Originally Posted by Iceman_jkh

Thanks El Camino.

Can you step me through that script a little. I ve only written 1 very VERY basic script before. I couldnt even see how to get this to compile.

EDIT: I got it to compile - turns out it must be saved as a .au3 file before it allows compilation (DOH!)

Basically, if the cursor becomes a text cursor, I want:
a) Audio to mute/pause
b) Mic to turn on
c) Accept commands/phonetic alphabet...until;
d) ENTER/command word is said (which is the equivalent of pressing ENTER)
e) turn mic off - as ENTER should change the cursor back to a pointer (not text cursor).
f) Audio resumes/unmutes

Profit> would also love to see what you are working on (in secret :P)

I'm hoping this little 'plugin' will make the voice control actually viable and useful!

Cheers
Ice

Ice - Won't you still have to issue the "letters" command or "numbers" command since they are in the commands.cfg or could you load a different config file for just the osk ? Also, doesn't the voice module stop listening after a successful command is issued and you have to prompt again ? I know I'm asking more questions than I'm helping but wanted to think it all the way through.

I use the voice control in mine all the time but I have a steering wheel control mapped to enable the voice prompt and then mute the mic after a set amount of time. Works out pretty well - I don't normally issue just letter and number commands because of what I asked you above.

Iceman_jkh · 12-17-2007, 07:48 AM

OK, now that I can run it , Ive managed to understand whats going on

Heres what ive put together.
Purely a concept at the moment. Credit goes to El Camino for the main code!

Code:

#Include <Misc.au3>
AutoItSetOption("WinTitleMatchMode", 3)
$test = "test"
$SpeechOn = -1

If WinExists($test) Then Exit 
AutoItWinSetTitle($test)

$rr=ObjCreate("RoadRunner.sdk")
if not IsObj($rr) then Exit
$oMyError = ObjEvent("AutoIt.Error","MyErrFunc")

HotKeySet("!{esc}", "_exit") ;for testing only, exit ALT+ESC disable when no need for it.

While 1
	If  $oMyError.windescription then 
		if Not WinExists("RoadRunner") then ExitLoop ;exit if RR is not Running.
	EndIf
	
	If MouseGetCursor() = 5 And _IsPressed("01") then
		;Place code here.. to tell Voice control to listen
	    ;$rr.execute("My_Commands")
		$rr.execute("next") ;Just a test - so you can see it switch to the next track - delete this line in real version!
	   $SpeechOn= 1 ; Tells the app that it is about to receive speech commands
	EndIf
	
Sleep (800)                ;push and hold for 0.8 seconds to engage Voice control
	
	While MouseGetCursor() = 5 And $SpeechOn = 1
		; While we are waiting for the speech input, we make sure that the cursor stays as a TEXT CURSOR and is in speech mode
		; Just sit here and do nothing. This means the app will wait till the 
		; ..user finishes talking and issues their "ENTER" command
		; Might have to add a trap for AND Not(_IsPressed("0D")) to the end of this While loop
	WEnd
	
	If Not(MouseGetCursor() = 5) And $SpeechOn = 1 Then ;As soon as the user presses ENTER or gives the ENTER command
	
		$SpeechOn= 0 ; Turn off voice commanding - ignore incoming commands/turn mic off
		$rr.execute("prev") ;Just a test - so you can see it switch to the prev track - delete this line in real version!
		;$rr.execute("My_Commands") any other commands you wish to run
	EndIf
	
			
WEnd
;==============================================================================================
;AI's error reporting Object
;==============================================================================================

Func MyErrFunc()
    Local $err = $oMyError.number
    If $err = 0 Then $err = -1
    SetError($err) 
Endfunc

Func _exit()
Exit
EndFunc

Compile it to an autoit script.
Hold left click for 0.8 seconds on a text box - should skip to next track.
Pressing enter, moving cursor off text box,etc should skip to prev track.

Basically all this does is prove that you can recognise such events and then trigger commands as a result

Next step is getting it to control/interface to the voice commands aspect of RR.

Ice

Iceman_jkh · 12-17-2007, 07:57 AM

Quote: Originally Posted by montgomeryt

Ice - Won't you still have to issue the "letters" command or "numbers" command since they are in the commands.cfg or could you load a different config file for just the osk ? Also, doesn't the voice module stop listening after a successful command is issued and you have to prompt again ? I know I'm asking more questions than I'm helping but wanted to think it all the way through.

I use the voice control in mine all the time but I have a steering wheel control mapped to enable the voice prompt and then mute the mic after a set amount of time. Works out pretty well - I don't normally issue just letter and number commands because of what I asked you above.

Im not that advanced at the voice control prog as you are mate.
But reading the .xml I can see what you are asking.
Hopefully we can create a smaller version of the .xml with only letters,numbers,etc using the grammar compiler. To be used specifically for this purpose. Even better would be causing the voice recognition to only use the mini .xml file when triggered in this way... and use the normal grammar file if invoked any other way. Get my drift?

Could even use PowerVoice from 0l33l (i tihnk it was). All I really want is a prog that writes what I speak into the text box.

EL CAMINO · 12-17-2007, 02:02 PM

@Profit
sorry man, all i know is a little bit of Autoit, i'm lost in Vb.net, but i see you have it working by now, hope to see that secret program soon.

@iceman
i need to play more with voice control, my end gives me errors when enabled, cut be my end(nlite) so yea i can help you out with this, but if you know the commands let me know and we can get it to work for sure.

Iceman_jkh · 12-17-2007, 03:02 PM

Just gotta find the rr.execute() commands needed for:

a) Pausing Audio - $rr.Execute("Pause")
b) Switching the voice control .xml file to the mini version (optional) - ?
c) Sending the HOTKEY to RR/Voicecontrol application - ? Cant seem to get $rr.Execute("SENDKEY;{F10}") to work
(turning the Mic on)
d) Sending the ENTER key to RR's OSK/TextBox - $rr.Execute("Enter")
(submitting the text to RR and turning the mic off)

....If I use $rr.Execute("Enter") inside the playlist search in BMV2 it works nicely, howver if I use it inside CDRIP to name the Artist/Album/etc, $rr.Execute("Enter") takes me to the main menu.

Is there any other way to trigger the enter command?

f) Resume audio - $rr.Execute("Resume")

EL CAMINO · 12-17-2007, 10:46 PM

a)done i guess.
b)well dont know how voice uses this xml, but if is just mater of changing the file, file move would work nicely.
c)

Code:

$Voice = WinGetHandle("RoadRunner", "")
WinActivate($Voice, "")
Send("{F10}")

d)dont know about this one.
e)??
f)what about if music is paused before all this process? if you use resume or play will start playing no matter what, when finish, maybe use

Code:

 $Status = $rr.getinfo("STATUS")

to poll the status info, and according to that, resume or not after this process.

Iceman_jkh · 12-18-2007, 04:51 AM

Oops.. there was no (e) . I'm a little overzealous to get this working I guess

Good thinking regarding the status polling. Fixed it now.

Ive figured (d) out. When you click the text cursor (for 0.8 seconds in the text field) to trigger the Voice recognition to turn on, the process of sending the F10 key is actually taking focus away from the RR CDRIP app. This causes the LOADING screen to be seen. Pressing ESC goes back to the main menu and the entire app is still running correctly (thinking that it has been told to stop voice recognition etc).

Test: I replaced the send F10 to RR with send "5" to Calculator, the app worked perfectly and CDRIP DID NOT lose focus from the OSK screen.

[All testing was done on BMV2.]

Any other ways to trigger the Voice recognition plugin to ON?
Can the plugin be recoded to include this option?
I would almost prefer it if the plugin wasnt waiting for a hotkey, purely turning the Mic input ON/OFF.

I tried the following commands to send the F10 key - all result in LOADING screen:

1. ControlSend("RoadRunner","","","{F10}")

2. $Voice = WinGetHandle("RoadRunner", "")
WinActivate($Voice, "")
Send("{F10}")

Anyway, heres my current code:

Code:

#Include <Misc.au3>
AutoItSetOption("WinTitleMatchMode", 3)
$test = "test"
$SpeechOn = -1
$PlayStatus = -1
If WinExists($test) Then Exit 
AutoItWinSetTitle($test)

$rr=ObjCreate("RoadRunner.sdk")
if not IsObj($rr) then Exit
$oMyError = ObjEvent("AutoIt.Error","MyErrFunc")

HotKeySet("!{esc}", "_exit") ;for testing only, exit ALT+ESC disable when no need for it.

While 1
	If  $oMyError.windescription then 
		if Not WinExists("RoadRunner") then ExitLoop ;exit if RR is not Running.
	EndIf
	
	If MouseGetCursor() = 5 And _IsPressed("01") then
		;Mouse has been clicked where the User wants to dictate
		If ($rr.getinfo("STATUS") = "Play") Then	;Check the previous play/pause/stop status
			$rr.Execute("pause") ;Pause music		;if playing, then pause the music, else skip
			$PlayStatus = 1
		EndIf
		
		;ControlSend("RoadRunner","","","{F10}") ; this does the same as the 3 code lines below - but is slightly less obtrusive
		;the above line tries to tell the Voice Recog plugin to start accepting dictation
		
		$Voice = WinGetHandle("RoadRunner", "") ;trigger hotkey line 1
		WinActivate($Voice, "");trigger hotkey line 2
		Send("{F10}");trigger hotkey line 3
		
		$SpeechOn= 1 ; Tells the app that it is about to receive speech commands
	EndIf
	
Sleep (800)                ;push and hold for 0.8 seconds to engage Voice control
	

	While (MouseGetCursor() = 5 And $SpeechOn = 1) Or (Not(_IsPressed("01")) And Not(MouseGetCursor() = 5) And $SpeechOn = 1)  
		;Above line checks that EITHER:
		;a)The cursor is a text cursor (Ibeam) and it doesnt care if its being left clicked; or
		;b) The cursor is NOT a text cursor (Ibeam) but thats OK, as long as the user isnt left clicking on anything - purely moving the mouse around!
		
		; While we are waiting for the speech input, we make sure that the cursor stays as a TEXT CURSOR and is in speech mode
		; Just sit here and do nothing. This means the app will wait till the 
		; ..user finishes talking and issues their "ENTER" command
		
	WEnd
	
	If Not(MouseGetCursor() = 5) And $SpeechOn = 1 Then ;As soon as the user gives the ENTER command - This needs work!
		;Place code here for
		;turn mic off /hotkey - mic has auto-off it seems
		$rr.Execute("ENTER") ;press enter/submit text - doesnt always work
		If ($rr.getinfo("STATUS") = "PAUSE") AND ($PlayStatus = 1) Then ;at the top we checked if RR was playing/paused/stopped, if playing this resumes, else skips over
		$rr.Execute("resume") ;resume audio
		$PlayStatus = -1
		EndIf
	
		$SpeechOn = 0 ; Turn off voice commanding - ignore incoming commands/turn mic off
		
	EndIf
	
			
WEnd
;==============================================================================================
;AI's error reporting Object
;==============================================================================================

Func MyErrFunc()
    Local $err = $oMyError.number
    If $err = 0 Then $err = -1
    SetError($err) 
Endfunc

Func _exit()
Exit
EndFunc

EDIT: changed code to use the '$voice' method (simply changed commenting), as method 1 didnt always work (ControlSend("RoadRunner","","","{F10}"))

Ice

montgomeryt · 12-18-2007, 08:47 AM

@ICE

Any other ways to trigger the Voice recognition plugin to ON?
Can the plugin be recoded to include this option?
I would almost prefer it if the plugin wasnt waiting for a hotkey, purely turning the Mic input ON/OFF.

I agree with turning the mic on/off. I had to switch to using the hotkey because the mic was always on and the voice control was in listen mode. I noticed that outside noise or conversation that may have resembled commands would get executed.

When I left work last night I tested what I was talking about. In order to type letters using the current voice commands the system had to be prompted for every letter - press a button letter "a" press a button letter "b" and so on. IMO the voice command system needs to be more like a Dragon Dictate software or the Microsoft voice dictation. Then the system could disseminate entire words or phrases. This would also facilitate the use of playing artists, or albums on command. I know there is a plugin (SmartPlaylist) that does this but if the voice commands plugin can be "re-written" may want to consider that as well.