Metadata and Video and Why We Should Work With Students in Project Development

One of the puzzles I have been working on this year is how to attach metadata to video. Yes, I understand that I can attach search terms to a chunk of video -- that's not the problem. I also know I can use the captions as search terms. But both of those scenarios are unsatisfactory to me.

If I attach metadata to the entire chunk of video -- a 50 minute video of a professor teaching a math class, for example, it doesn't help the student find that one part where she is talking about how to solve quadratic equations. To find that, the student has to fast forward through the entire 50 minutes, and hopefully will be able to find it on the first pass. What I want is to be able to search for "quadratic equation" and have the video queue up all the places where the professor used that term.

And what if the professor isn't speaking? What if she is signing ASL? I still want to find all the places where she uses that term. And if she fingerspells it one place, and uses a sign somewhere else, I want to find both places.

AND -- here's the hardest part -- I don't want this to require post production work. Because post production means money means we won't be able to do this all of the time. And it has to be screw-up proof too. For the obvious reasons.

So, one day this 4th year RIT illustration student stops by my office. He is working as a note taker, one of the access services we offer to students at NTID (and RIT too.) And he is taking notes for a student who is sighted, but unable to read. So this notetaker, who's name is Ben Rubin, is drawing notes for the student. Here is some of Ben's work. Very cool and very interesting, but it gets better.

Ben go a hold of a of a special pen called Pulse SmartPen. This devise allows Ben to record the spoken words of the professor as he is taking notes and link particular parts of the lecture to particular parts of the notes. In other words, Ben can insert a bookmark on the fly while he is drawing notes. Then he uploads the notes (and the audio file) to a website, and when his student wants to review what the professor was teaching about while Ben was drawing his notes, he just clicks on the website and hears the audio. Which is very cool, if you hear.

So Ben got to thinking about how sometimes there is also an interpreter in the room for the Deaf students. And what if, Ben said to me, we could video tape the interpreter and use the audio track to sync to his notes and make it so you could click on something and instead of hearing the audio, a little window popped up and you saw the interpreter signing what the professor was saying.

What if indeed!

So I told my programmers that I love this idea and I want to include it as a Major Design Project next year. And I met with the head of Access Services to ask him if we could use a couple or three of his notetakers next year. And he said yes. So this isn't really solving the problem of inserting metadata into video, but it is a hack that might work if we can figure it out. More to come!