OCX: Merlin speaks and listens but generates no events

[Anonymous]

ello Everybody,

I am using msAgent and need speech recognition. Merlin behaves very well, speaks, performs,recognizes
commands and responds to 'hide'. But does not seem to generate any events: speech or otherwise.

The OCX ProgID is Agent.Control.2.
The events code was generated as per Firefly instructions on OCX
The control Interface code was generated from Powerbasic 7 COM browser.
The test program was adapted from the Powerbasic 7 COM demo.

No event messages are coming through. (I am just inserting message boxes for detection).

Is there any additional coding required to capture those events?
What could be going wrong?

[Jose Roca]

Is there any additional coding required to capture those events?

Yes. You have to cnnect with the events. For example:

Code Select Expand


GLOBAL dwCookie AS DWORD

Function FORM1_WM_CREATE ( _
                        hWndForm As Dword _  ' handle of Form
                        ) As Long

Local hr As Dword
hr = Agent_AgentEvents1_SetEvents(ObjPtr(DISPATCH_FORM1_AGENT), dwCookie)

End Function


That is, if you're using DISPATCH_FORM1_AGENT. If you're using another name change it.

And not forget to disconnect from the events when finished:

Code Select Expand


Function FORM1_WM_DESTROY ( _
                         hWndForm      As Dword _  ' handle of Form
                         ) As Long
                         
Local hr As Dword
If dwCookie Then hr = Agent_AgentEvents1_ReleaseEvents(ObjPtr(DISPATCH_FORM1_AGENT), dwCookie)

End Function


[Anonymous]

Many thanks Jose,

That has got me another stage further.

But only Events 27 and 28 are coming through (ListenStart and ListComplete ), when the scroll lock
is pressed and released.
I also noticed that the CharacterID is a null string so I wonder if more hitching-up code will be
necessary?

This is my WM_CREATE using your instructions:

 AgentTest 1
' This intialises Merlin and makes him speak as per PB7  Demo code
' No further commands are given until WM_DESTROY when Merlin is deallocated.
 Local hr As Dword
 hr = Agent_AgentEvents1_SetEvents(ObjPtr(DISPATCH_FORM1_OCXCONTROL1), dwCookie)

[Jose Roca]

I think you're mixing things. If you include an OCX control in your form, name it Agent and use DISPATCH_FORM1_AGENT then you can't use the PB example as it is. You have to connect to the events using the object or dispatch variable used to create the instance of MSAgent. If you're following the PB example the you will need to use ObjPtr(oA).

The code below will show Merlin and connect with the events. You can put message boxes in events like Click and StarDrag and see as the event fires and the CharacterID contains "Merlin" when you click or drag the character.

Code Select Expand


Function FORM1_WM_CREATE ( _
                        hWndForm As Dword _  ' handle of Form
                        ) As Long

Local hr As Dword
hr = Agent_AgentEvents1_SetEvents(ObjPtr(DISPATCH_FORM1_AGENT), dwCookie)

Local v As Variant
Local v1 As Variant    
Local v2 As Variant                
Local oCharacters As Dispatch
Local oCharEx As Dispatch
v1 = -1 As Long
Object Let DISPATCH_FORM1_AGENT.Connected = v1
Object Get DISPATCH_FORM1_AGENT.Characters To v
Set oCharacters = v
v1 = "Merlin"
v2 = "Merlin.acs"
Object Call oCharacters.Load(v1, v2)
Object Call oCharacters.Character(v1) To v
Set oCharEx = v
v1 = 0
Object Call oCharEx.Show(v1)


End Function


'------------------------------------------------------------------------------------------------------------------------
Function FORM1_WM_DESTROY ( _
                         hWndForm      As Dword _  ' handle of Form
                         ) As Long
                         
                         
Local hr As Dword
If dwCookie Then hr = Agent_AgentEvents1_ReleaseEvents(ObjPtr(DISPATCH_FORM1_AGENT), dwCookie)
                         

End Function


[Anonymous]

Thank you again Jose

I am now receiving all the events.

The next problem is to access the UserInput data when AgentEvents1_Command is called.

VBscript does it this way.
 '--------------------------------------
 Sub AgentCtl_Command(byval UserInput)

 HeardWhat = UserInput.Voice

 If (UserInput.Confidence <= -60) then
   BadAcceptanceRec
   Exit Sub
 Else...
 '--------------------------------------

What is the equivalent access method for PB?
The data seems to be well hidden somewhere.

[Jose Roca]

In the Agent_AgentEvents_Command you get a reference to the IAgentCtlCommand interface in the UserInput variable. To be able to call the methods and properties of this interface using PB Automation you can do the following:

Code Select Expand


Local oUserInput As Dispatch
Local vRes As Variant
TB_MakeDispatchVariant UserInput, vRes
Set oUserInput = vRes
vRes = Empty

Then:

Code Select Expand


Local strHeardWhat As String
Object Get oUserInput.Voice to vRes
strHeardWhat = Variant$(vRes)

Local lConfidence As Long
Object Get oUserInput.Confidence to vRes
lConfidence = Variant#(vRes)
If lConfidence <= -60 Then ...

Set oUserInput = Nothing


[Anonymous]

Thank you very much Jose,

We now have speech recognition.

but there is another problem:

After a few commands, sometimes on the first command, the
OBJECT GET oUserInput.whatever  tries to reference protected memory
such as 0x00000008, causing the application to be terminated by Windows.

I have isolated the problem to either of the OBJECT GET lines.

I have about 16 commands available such as "one" "two" "three".

This is the code I am using with your instructions:
I have commented out some lines for debugging to show that it
crashes on OBJECT GET oUserInput.Confidence

  Local UserInput As Dword
  UserInput = Variant#(@pv[0])
'  ===============================================
'  *** Put your code here ***
Global strHeardWhat As Asciiz*128
Local lConfidence As Long
Local oUserInput As Dispatch
Local vRes As Variant
TB_MakeDispatchVariant UserInput, vRes
Set oUserInput = vRes

'If IsFalse IsObject(oUserInput) Then Function=0: beep: Exit Function


vRes = Empty
'---
Object Get oUserInput.Confidence To vRes
'lConfidence = Variant#(vRes)

strHeardWhat = "???"

If lconfidence>-60 Then
'Object Get oUserInput.Name To vRes
'strHeardWhat = Variant$(vRes)
End If
SendMessage HWND_FORM1_TEXT1, %WM_SETTEXT, 0,VarPtr(strHeardWhat)
Set oUserInput = Nothing
'  ===============================================
  Function = 0  ' %S_OK
End Function
' ************************************************

[Jose Roca]

Check UserInput. Before TB_MakeDispatchVariant put  IF IsFalse UserInput THEN Exit Function to see if it that solves the problem.

[Anonymous]

No it is still accessing protected memory locations.

[Jose Roca]

Without being able to test it myself it is difficult to guess where can be the problem.

Below are wrapper functions to call the methods and properties of the UserInput interface.

To use them (discard all the previous code except the first four lines below):

Local pv As Variant Ptr, pvapi As VARIANTAPITYPE Ptr
pv = pdispparams.VariantArgs : pvapi = pv

Local UserInput As Dword
UserInput = Variant#(@pv[0])

If IsFalse UserInput Then Exit Function

Local hr As Long
Local strName As String
Local lConfidence As Long

hr = IAgentCtlUserInput_get_Confidence(UserInput, lConfidence)
hr = IAgentCtlUserInput_get_Name(UserInput, strName)

The result code is returned in hr.

Code Select Expand


' ********************************************************************************************
' IAgentCtlUserInput interface
' IID: {C4ABF875-8100-11D0-AC63-00C04FD97575}
' Number of functions: 11
' Attributes: 4160 [&H1040]  [Dual] [Dispatchable]
' ********************************************************************************************

' ********************************************************************************************
' [get_]Count property
' Interface name: IAgentCtlUserInput
' Vtable offset: 28
' DispID: 1610743808 [&H60020000]
' ********************************************************************************************
FUNCTION IAgentCtlUserInput_get_Count ( _
   BYVAL pthis AS DWORD PTR, _                         ' VT_PTR <*VT_UNKNOWN>
   BYREF pCount AS INTEGER _                           ' *VT_I2 <INTEGER>  [out]
   ) AS LONG                                           ' VT_HRESULT <LONG>

   LOCAL HRESULT AS LONG
   CALL DWORD @@pthis[7] USING IAgentCtlUserInput_get_Count(pthis, pCount) TO HRESULT
   FUNCTION = HRESULT

END FUNCTION
' ********************************************************************************************

' ********************************************************************************************
' [get_]Name property
' Interface name: IAgentCtlUserInput
' Vtable offset: 32
' DispID: 1610743809 [&H60020001]
' ********************************************************************************************
FUNCTION IAgentCtlUserInput_get_Name ( _
   BYVAL pthis AS DWORD PTR, _                         ' VT_PTR <*VT_UNKNOWN>
   BYREF pName AS STRING _                             ' *VT_BSTR <DYNAMIC UNICODE STRING>  [out]
   ) AS LONG                                           ' VT_HRESULT <LONG>

   LOCAL HRESULT AS LONG
   CALL DWORD @@pthis[8] USING IAgentCtlUserInput_get_Name(pthis, pName) TO HRESULT
   pName = ACODE$(pName)
   FUNCTION = HRESULT

END FUNCTION
' ********************************************************************************************

' ********************************************************************************************
' [get_]CharacterID property
' Interface name: IAgentCtlUserInput
' Vtable offset: 36
' DispID: 1610743810 [&H60020002]
' ********************************************************************************************
FUNCTION IAgentCtlUserInput_get_CharacterID ( _
   BYVAL pthis AS DWORD PTR, _                         ' VT_PTR <*VT_UNKNOWN>
   BYREF pCharacterID AS STRING _                      ' *VT_BSTR <DYNAMIC UNICODE STRING>  [out]
   ) AS LONG                                           ' VT_HRESULT <LONG>

   LOCAL HRESULT AS LONG
   CALL DWORD @@pthis[9] USING IAgentCtlUserInput_get_CharacterID(pthis, pCharacterID) TO HRESULT
   pCharacterID = ACODE$(pCharacterID)
   FUNCTION = HRESULT

END FUNCTION
' ********************************************************************************************

' ********************************************************************************************
' [get_]Confidence property
' Interface name: IAgentCtlUserInput
' Vtable offset: 40
' DispID: 1610743811 [&H60020003]
' ********************************************************************************************
FUNCTION IAgentCtlUserInput_get_Confidence ( _
   BYVAL pthis AS DWORD PTR, _                         ' VT_PTR <*VT_UNKNOWN>
   BYREF pConfidence AS LONG _                         ' *VT_I4 <LONG>  [out]
   ) AS LONG                                           ' VT_HRESULT <LONG>

   LOCAL HRESULT AS LONG
   CALL DWORD @@pthis[10] USING IAgentCtlUserInput_get_Confidence(pthis, pConfidence) TO HRESULT
   FUNCTION = HRESULT

END FUNCTION
' ********************************************************************************************

' ********************************************************************************************
' [get_]Voice property
' Interface name: IAgentCtlUserInput
' Vtable offset: 44
' DispID: 1610743812 [&H60020004]
' ********************************************************************************************
FUNCTION IAgentCtlUserInput_get_Voice ( _
   BYVAL pthis AS DWORD PTR, _                         ' VT_PTR <*VT_UNKNOWN>
   BYREF pVoice AS STRING _                            ' *VT_BSTR <DYNAMIC UNICODE STRING>  [out]
   ) AS LONG                                           ' VT_HRESULT <LONG>

   LOCAL HRESULT AS LONG
   CALL DWORD @@pthis[11] USING IAgentCtlUserInput_get_Voice(pthis, pVoice) TO HRESULT
   pVoice = ACODE$(pVoice)
   FUNCTION = HRESULT

END FUNCTION
' ********************************************************************************************

' ********************************************************************************************
' [get_]Alt1Name property
' Interface name: IAgentCtlUserInput
' Vtable offset: 48
' DispID: 1610743813 [&H60020005]
' ********************************************************************************************
FUNCTION IAgentCtlUserInput_get_Alt1Name ( _
   BYVAL pthis AS DWORD PTR, _                         ' VT_PTR <*VT_UNKNOWN>
   BYREF pAlt1Name AS STRING _                         ' *VT_BSTR <DYNAMIC UNICODE STRING>  [out]
   ) AS LONG                                           ' VT_HRESULT <LONG>

   LOCAL HRESULT AS LONG
   CALL DWORD @@pthis[12] USING IAgentCtlUserInput_get_Alt1Name(pthis, pAlt1Name) TO HRESULT
   pAlt1Name = ACODE$(pAlt1Name)
   FUNCTION = HRESULT

END FUNCTION
' ********************************************************************************************

' ********************************************************************************************
' [get_]Alt1Confidence property
' Interface name: IAgentCtlUserInput
' Vtable offset: 52
' DispID: 1610743814 [&H60020006]
' ********************************************************************************************
FUNCTION IAgentCtlUserInput_get_Alt1Confidence ( _
   BYVAL pthis AS DWORD PTR, _                         ' VT_PTR <*VT_UNKNOWN>
   BYREF pAlt1Confidence AS LONG _                     ' *VT_I4 <LONG>  [out]
   ) AS LONG                                           ' VT_HRESULT <LONG>

   LOCAL HRESULT AS LONG
   CALL DWORD @@pthis[13] USING IAgentCtlUserInput_get_Alt1Confidence(pthis, pAlt1Confidence) TO HRESULT
   FUNCTION = HRESULT

END FUNCTION
' ********************************************************************************************

' ********************************************************************************************
' [get_]Alt1Voice property
' Interface name: IAgentCtlUserInput
' Vtable offset: 56
' DispID: 1610743815 [&H60020007]
' ********************************************************************************************
FUNCTION IAgentCtlUserInput_get_Alt1Voice ( _
   BYVAL pthis AS DWORD PTR, _                         ' VT_PTR <*VT_UNKNOWN>
   BYREF pAlt1Voice AS STRING _                        ' *VT_BSTR <DYNAMIC UNICODE STRING>  [out]
   ) AS LONG                                           ' VT_HRESULT <LONG>

   LOCAL HRESULT AS LONG
   CALL DWORD @@pthis[14] USING IAgentCtlUserInput_get_Alt1Voice(pthis, pAlt1Voice) TO HRESULT
   pAlt1Voice = ACODE$(pAlt1Voice)
   FUNCTION = HRESULT

END FUNCTION
' ********************************************************************************************

' ********************************************************************************************
' [get_]Alt2Name property
' Interface name: IAgentCtlUserInput
' Vtable offset: 60
' DispID: 1610743816 [&H60020008]
' ********************************************************************************************
FUNCTION IAgentCtlUserInput_get_Alt2Name ( _
   BYVAL pthis AS DWORD PTR, _                         ' VT_PTR <*VT_UNKNOWN>
   BYREF pAlt2Name AS STRING _                         ' *VT_BSTR <DYNAMIC UNICODE STRING>  [out]
   ) AS LONG                                           ' VT_HRESULT <LONG>

   LOCAL HRESULT AS LONG
   CALL DWORD @@pthis[15] USING IAgentCtlUserInput_get_Alt2Name(pthis, pAlt2Name) TO HRESULT
   pAlt2Name = ACODE$(pAlt2Name)
   FUNCTION = HRESULT

END FUNCTION
' ********************************************************************************************

' ********************************************************************************************
' [get_]Alt2Confidence property
' Interface name: IAgentCtlUserInput
' Vtable offset: 64
' DispID: 1610743817 [&H60020009]
' ********************************************************************************************
FUNCTION IAgentCtlUserInput_get_Alt2Confidence ( _
   BYVAL pthis AS DWORD PTR, _                         ' VT_PTR <*VT_UNKNOWN>
   BYREF pAlt2Confidence AS LONG _                     ' *VT_I4 <LONG>  [out]
   ) AS LONG                                           ' VT_HRESULT <LONG>

   LOCAL HRESULT AS LONG
   CALL DWORD @@pthis[16] USING IAgentCtlUserInput_get_Alt2Confidence(pthis, pAlt2Confidence) TO HRESULT
   FUNCTION = HRESULT

END FUNCTION
' ********************************************************************************************

' ********************************************************************************************
' [get_]Alt2Voice property
' Interface name: IAgentCtlUserInput
' Vtable offset: 68
' DispID: 1610743818 [&H6002000A]
' ********************************************************************************************
FUNCTION IAgentCtlUserInput_get_Alt2Voice ( _
   BYVAL pthis AS DWORD PTR, _                         ' VT_PTR <*VT_UNKNOWN>
   BYREF pAlt2Voice AS STRING _                        ' *VT_BSTR <DYNAMIC UNICODE STRING>  [out]
   ) AS LONG                                           ' VT_HRESULT <LONG>

   LOCAL HRESULT AS LONG
   CALL DWORD @@pthis[17] USING IAgentCtlUserInput_get_Alt2Voice(pthis, pAlt2Voice) TO HRESULT
   pAlt2Voice = ACODE$(pAlt2Voice)
   FUNCTION = HRESULT

END FUNCTION
' ********************************************************************************************


[Anonymous]

Yes! Yes!

The wrapper functions have done the trick. No crashes this time.

I patched them in at the top of the events code.

but am I right in assuming that this code cannot be generated in Firefly or PB com browser? Do I need to do it manually for other interfaces?

There doesnt seem to be anything on MSAgent in the PB forums. Perhaps these techniques ought to be more widely publicised.

Many thanks Jose.

[Jose Roca]

These wrappers use the technique of Direct Virtual Table calls and have been generated by a new version of my COM browser, not yet finished. FireFly only generates code to host visual OCXs using ATL71.DLL and events code (using an stripped down version of my browser), but no wrapper functions. This is adequate in most cases.

I like low level COM programming because I have full control of what I'm doing, whereas when using PB Automation I don't know exactly what code has generated the compiler and therefore I don't know if it is my fault or a bug. However, to use this technique you need to have a good knowledge of the use of pointers and indirection, so I make wrappers to put inside them the low level calls and let you to use them as regular functions and procedures.

In your case I don't think that you will have more problems. The only potential one are in the RequestStart and RequestComplete events, that return a reference to the IAgentCtlRequest interface.

Since MSAgent can be used without hosting it in a window, I will prepare wrapper functions for all the interfaces and an adaptation of the example provided by PB to show how to use them. You will be able to use them both with FireFly or without it.