Advanced Iris Tracking Software

[s-bbrackett]

I have recently encountered a problem in my attempts to create my project. I'm trying to create a program that uses the iris tracking software to map the users eye gaze. Essentially I'm trying to:
-Find the users gaze
-Mark where it is
-Draw/Display it on the viewed image
Currently I have the basic eye tracking program working but I'm confused as to what the next steps would be. If you could provide any help as to what I would do next, I would be extremely grateful.

[kwagyeman]

Hi, I can't quite tell what you'd like me to help with. The current eye tracking program tells you where the eye is looking and marks where it is on the image... that said, you have to mount the camera very close to the person's eyes.

Can you go into more details?

[s-bbrackett]

Hi there,
I'm sorry if my first reply didn't make much sense. What I'm trying to achieve is a program that makes you give the program an image, tracks where you're looking on that image, and then proceeds to mark where you were looking on the given image. In short, it's a program that tracks and marks the users gaze at any given point. I guess I'm confused as to what the code would be needed in order to draw the gaze of the user on the given image. You also said that the current program already marks the image with the users gaze. Could perhaps send me the code that does this? If you could help me I would be extremely grateful.

Thanks for the time

[kwagyeman]

The current method draws the position of the persons pulpils given the cameras field of view.

You can to determine where the person is looking though. This is beyond just the OpenMV Cam. So, you want to send the detected pupil positions from the OpenMV Cam out to a serial port. Once you do this... then you need to send that data to the PC. After which, process it and draw on the screen.

So, you're going to need to some knowledge of geometry to setup a vector transformation to run the x/y position the camera sees into an x/y position on the PC. I guess... start with this. Try mounting the camera in a fixed position and working back the match to transform the coordinates.

https://www.google.com/search?q=coordin ... e&ie=UTF-8

[s-bbrackett]

Thank you so much for all the help. I have just one last question. How do I send the data out the port, and where(what software) do I send it to? Once again thank you so much for your time and help.

[kwagyeman]

Just use the USB_VCP module or UART module on the camera. There's documentation online about how to send serial data.

[s-bbrackett]

Thank you for your response. However, I guess my next question would be, how do I set these steps up and what would the code look like? I have the camera sending me coordinates of only the eye right now but I'm rather perplexed as to what would the next steps look like. Could you please clarify this in more detail for me.
Thanks again for all the help

[kwagyeman]

I have a script for this on my desktop... Will post later tonight. Also, this question has been asked before on the forum. Just search a bit to find it.

[s-bbrackett]

Thank you very much for your help with this issue.

[s-bbrackett]

Hey,
so I took a look your work with the French class on the computer mouse controlling software. I guess the best way to visualize my project is to think of something like that, but it draws the current location of the eye gaze on the screen. If you could help me out with the Code for this I would appreciate it greatly.

[kwagyeman]

I can give you the OpenMV Cam code. As for the PC side code... Um not sure what library to use for drawing anywhere on screen.

I know how to write Qt desktop programs. I recommend using Qt for doing the desktop program.

[s-bbrackett]

That would be awesome. If you could send me those files and perhaps tell me how to work the QT code I would be extremely grateful. Thanks again for your help.

[kwagyeman]

Attached are the starting programs.

As for Qt... just google getting started with that. It's not something you learn in a week though. It took me two years to get good.

[s-bbrackett]

Once again thanks for your help. I just wanted to know, could you send the serial data over to C# and does the MV cam function like a Arduino, where it runs its last saved program if it's unplugged from the computer and plugged into a battery?

Thanks

[kwagyeman]

Yes and yes.

[s-bbrackett]

Hey,
I have recently come up with the question about whether I can use two OpenMV cams at once. I wanted to use this to have one camera showing a forward facing live time shot, while the other watches the persons eye and records it on the view of the first one. If this is possible please let me know.
Thanks

[kwagyeman]

Sure, you just have to do the protocol interfacing between the boards.

[s-bbrackett]

Is there an easy way to do that?

[kwagyeman]

Async serial is the easiest way to send commands from one board to another. You have to define the command protocol. You can also just use GPIO wires too if you want to sync things.

[s-bbrackett]

Would I need to write any code for that and if so, what libraries would I need to use?

[kwagyeman]

Yes, you'd have to write code...

So, when you talk about having two OpenMV Cam's you're looking at having to communicate between them. We provide code to send and receive bytes via the UART. But, you need some protocol on top of the data stream to send commands, staemachines on the camera to process data, etc.

If you don't need to send much data between cams then you can use GPIOs instead if your just trying to synchronize them. However, you have to write sometypr of statemachine.

Alternatively, you can also just send all data from the cameras to the PC and do all the processing on the PC. Then you can keep the program on the OpenMV Cam simple.

[s-bbrackett]

So I could send UART data between the two and essentially have the one with the live feed running while the other marks on its feed?

[kwagyeman]

Not exactly sure what you want to do...

One camera is looking at the person... And one at the screen the person is looking at? I thought you planned to write a PC app... So, what is the second camera for? The PC app should be able to track what the user is looking at given the coordinate stream from the first camera.

[s-bbrackett]

I was just thinking if I wanted to have the person look at something other than the screen, then I would need the other cam.

[s-bbrackett]

Now for reference, is there a library I need to access to send data via UART from one board to another? Also, you mentioned something about examples of sending data online, what are those examples called?
Once again, thanks a million

[s-bbrackett]

Forget that last post, I got it working. My other question is if I use UART to communicate between the two cameras, do I use the pins or no? Also, do I write the UART read function in a different program or in the same as the one I'm using to send the data? If different how do I separate the programs from each other?

Thanks

[s-bbrackett]

Essentially, if you remember the code you sent me last time, how would I read that data via UART.

[kwagyeman]

Yeah, so, you just use this method here: http://docs.micropython.org/en/v1.9.3/p ... b.UART.any

And then do: http://docs.micropython.org/en/v1.9.3/p ... T.readchar

When there's a byte.

For a statemachine example, checkout the Pixy emulation examples.

[s-bbrackett]

Thank you so much. Now just for my own ideas, you said that the two cameras can be run at the same time right? However, since there's only one IDE running at a time, how to I run the UART packaging code and the UART shipping code at the same time. Sorry about my newbie questions!

[s-bbrackett]

Also, is there a way of using the COMS to communicate data at the same time or quickly switching between them? Basically, can a just have the cameras read the COMS for data or do I need to have some sort of bussing method?

[kwagyeman]

OpenMV IDE supports connecting to one camera on purpose with the main feature set. You can use the Open terminal feature to connect to multiple cameras for debugging though on a second/third/etc. serial port.

Realistically, once you have two cameras in the system your skill level is going to need to be rather good to deal with communications issues. I really can't teach you this over the forums.

As for sending image data back and forth. Um, I thought you'd just route everything to the PC. If so, then the cameras are just generators.

[s-bbrackett]

WOW that really helped thank you se much! For the open terminal though, how to I run code on it?

[kwagyeman]

Please see our YouTube channel. I posted a video about the feature.

[s-bbrackett]

Thank you those videos really help. Now just to recap, there's no easier way to send eye coordinate data from one camera to the other and mark it on the other's display without using the coms correct? Once again thank you for your help.

[kwagyeman]

Serial is the easiest. Please see the Picture Emulation UART example. It shows how to do serial.

You'd have to do something like serial for any Microcontroller. It's a fundamental communication step.

[s-bbrackett]

The code for reading serial is on there right?

[s-bbrackett]

By the way, there is nothing blocking your cam's serial port right? I keep getting an issue on Visual Studios saying that the port to the cameras is blocked.

[s-bbrackett]

I have the following as C# code in VS for receiving the Cams data:


mySerialPort.BaudRate = 9600;
mySerialPort.Parity = Parity.None;
mySerialPort.StopBits = StopBits.One;
mySerialPort.DataBits = 8;
mySerialPort.Handshake = Handshake.None;
mySerialPort.RtsEnable = true;

[kwagyeman]

Does another program have the serial port open?

Note that there are two serial ports on the camera. A hardware UART and a USB VCP port. OpenMV IDE uses the USB VCP port to talk to the camera. If you are trying to use OpenMV IDE and your program at the same time then you have to talk to the camera via the hardware UART in I/O pins 4 and 5 which you can connect to the PC via an FTDI chip.

Alternatively, you can develop your code using OpenMV IDE and the hardware UART and then switch to the software USB VCP uart after you've gotten things working to reduce complexity.

[s-bbrackett]

Thank you for that, just so you know my MVcam is connected to the computer using a microUSB cable and it is being run on the penMV IDE. Meanwhile, I'm trying to get the code to Visual studios but it keeps saying that access is denied to the camera. Note visual studios detects that there's an object in my port, it just can't access it. The code on the camera is the one you sent me awhile ago, but the VS code is as follows. I don't know serial very well so please tell me if something isn't matching up! Thank again

using System;
using System.IO.Ports;

class PortDataReceived
{
public event SerialDataReceivedEventHandler DataReceived;
public static void Main()
{
SerialPort mySerialPort = new SerialPort("COM4");

mySerialPort.BaudRate = 115200;
mySerialPort.Parity = Parity.None;
mySerialPort.StopBits = StopBits.One;
mySerialPort.DataBits = 8;
mySerialPort.Handshake = Handshake.None;
mySerialPort.RtsEnable = true;

mySerialPort.DataReceived += new SerialDataReceivedEventHandler(DataReceivedHandler);

mySerialPort.Open();

Console.WriteLine("Press any key to continue...");
Console.WriteLine();
Console.ReadKey();
mySerialPort.Close();
}

private static void DataReceivedHandler(
object sender,
SerialDataReceivedEventArgs e)
{
SerialPort sp = (SerialPort)sender;
string indata = sp.ReadExisting();
Console.WriteLine("Data Received:");
Console.Write(indata);
}
}

[kwagyeman]

As previously mentioned... OpenMV IDE uses the camera's USB VCP port. If you want to talk to the camera when OpenMV IDE is connected please connect an FTDI chip to the OpenMV Cam's hardware UART. Pins 4/5. Unfortunately, we have no solution for more than one USB VCP port to the PC. This is a limit of the USB hardware on the STM32 chip.

[s-bbrackett]

Would an Arduino work?

[kwagyeman]

Sure, just turn it into a serial port pass through...

[s-bbrackett]

Awesome thank you

[s-bbrackett]

So I just write the code to receive and package data for Arduino and I should all work?

[kwagyeman]

As determined by your skill...

I get the feeling your trying to work on something out of your depth here.

I can help. But, you're going to really want to exercise your Googling muscle and try to do as much as you can before coming to the forums.

[s-bbrackett]

I will do that. However, thank you so much for your help. Yo have gotten me very far and your posts have taught me a lot. Thanks again for putting up with all of my posts and questions!

[s-bbrackett]

After a bit of work I have three questions for you. So, if you recall the code you sent me earlier for the UART iris tracking, do I need to add anything else to that if I want to send serial data to an Arduino. Secondly does the camera uses pins 4 and 5 for serial? Lastly, if the cam and Arduino are connected and my code is correct on both ends, this should work when I run the openMV's IDE, right?

Thanks for your time and help

[kwagyeman]

1. Um, just write code for the Arduino.
2. Yes, pins 4 and 5.
3. OpenMV IDE connects using the USB port which is different from the hardware UART. So, you can run both at the same time.

Note, if you just want to get data off the OpenMV Cam and have that go to a program on your PC just use the "print()" command in your python script and any program that opens the serial port the OpenMV Cam is on will receive the print command output.

[s-bbrackett]

So even if I have a micro-USB cable attached to the computer, the other IDE could still read the print statement?