10.0 – Technology and Interaction Design
As Alphaland is being aimed at children we want it to be both simple to use and relatable to the way they learn the subject at school. This is why we chose to explore both voice recognition and an interactive whiteboard system. We want the user to be able to both say the word and write letters as this is our main learning objective.
10.1 Voice Recognition
Firstly we looked at ways to program the voice recognition. We had initially hoped to program this in Adobe Flash but we found that there was a lot of problems associated with programming voice recognition in this way for example the program would only recognize the voice of the person who created it.
We then looked at programming it in JavaScript with HTML5 or Processing. We found that both worked in terms of the voice recognition and we got samples working in both. However we found the Java Script could not identify simple words such as “cat” but rather complex sentences, which is not what we were looking for. We also have a very limited knowledge of Java Script to continue with this. Therefore we felt Processing would be the optimum choice as we have a better understanding of the subject and were able to use this to carry out actions such as playing animations.
The Processing example (10.2) uses Voce, which is a speech synthesis and recognition library that is cross platform and has a very small API (Application Programming Interface). It uses CMU Sphinx4 and Free TTS. It uses a grammar file in a JSGF format. This must be added to the sketches library for the code to work.
CMU Sphinx4 is an open source library written in Java.
FreeTTS is a free open source Speech Syntheses library also written in Java.
Both work on multiple platforms.
JSGF – Java Speech Grammar Format – a textual representation of grammars for use in speech recognition.
The example uses the words “play” and “pause “ to control an animation. The animation must be added to the sketch folder for it to play. The user will speak into a wireless headset the command they choose. If they say play the animation will play and if they say pause it will pause. The sample code for this example is outlined below (10.2). This code will be altered to fit our project.
10.2 Voice Recognition Example, Wizard Animation
import voce.*;
import processing.video.*;
Movie theMov;
boolean isPlaying;
boolean isLooping;
void setup() {
size(540,400);
theMov = new Movie(this, “wizard animation.mov”);
theMov.loop();
isPlaying = true;
voce.SpeechInterface.init(sketchPath+”/code/”, true, true, sketchPath+”/data/”, “test”);
}
void draw() {
background(0);
image(theMov, 0,0);
while (voce.SpeechInterface.getRecognizerQueueSize() > 0){
String s = voce.SpeechInterface.popRecognizedString();
println(“You said: ” + s);
voce.SpeechInterface.synthesize(s);
if(s.equals(“pause”)){
theMov.stop();
isPlaying = false;
}
if(s.equals(“play”)) {
theMov.loop();
}
}
}
void movieEvent(Movie m) {
m.read();
}
10.3 Point and Click Interaction
We want the user to be able to select and draw with the use of their controller. We want the user to be carrying out the action of drawing similarly to how they would at school. We investigated our different options.
10.4 Reactivision (reacTIVision)
Reactivision allows you to create a tangible multi-touch surface using symbols called fiducial markers (See figure 10.4.1). These can be attached to physical objects as to track their movement and position. It also allows for multi-touch finger tracking. It is ideal for table based tangible user interfaces. It is cross platform, open source and is compatible with Windows, MacOS X and Linux. It recognizes over 200 symbols and is free to download from:
http://sourceforge.net/projects/reactivision/files/reacTIVision/reacTIVision-1.4/reacTIVision-1.4-mac.zip/download?use_mirror=heanet&download= (Note this is the version for MacOS X. Other versions are available from: http://reactivision.sourceforge.net/
Figure 10.4.1
Reactivision would be possoble to implement with our project however the symbols are very large and would mean the design of the pen would be too big for a small child to hold in their hand. This would make the teaching the child to write the letter element of the game redundant as they would not be physically holding a pan like they would in school.
10.5 Colour Tracking
Colour Tracking involves using a webcam to track the movement and location of a coloured object as to carry out actions onscreen. This is the technology used by Sony with the PlayStation Move. This can be carried out in processing with examples readily available from both our assigned textbooks, Learning Processing by author Daniel Shiffman and online.
Having watched some examples I found that one of the main problems with this was that as the camera searches for the pen (with a specific colour attached) it also searches for other objects of the same colour.
However this is a very cost effective route to consider as it only requires a web cam, a coloured object and open source software, which is available, online.
10.6 SMART Table 442i
The SMART Table 442i (See Figure 10.6.1) has been specifically build with teaching young students in mind. It is a 42” LCD multi-touch interactive table that allows the user to carry out both individual and group activities. It supports USB devices and has built in volume controls and Wi-Fi. It has a sturdy design and is easy for young children to use. This would be an ideal and extremely simple solution for our game. However having researched this online the SMART Table 442i is an extremely expensive piece of technology with the prices varying between £4000 and £5000 sterling. This means that it would not be a possible route for us to consider.
Figure 10.6.1
10.7 Wiimote Whiteboard
Another idea to consider is creating a multipoint interactive whiteboard using a Nintendo Wii remote and an infrared light pen. This uses the camera in the Wii remote to track the movement of the infrared pen along a projected surface. A demonstration of this can be seen at http://www.youtube.com/watch?v=5s5EvhH y7eQ by Johnny Chung Lee from Carnegie Mellon University.
To do this you can both purchase or build your own infrared pen using an infrared LED a push button switch and a power supply E.G a battery. You then project the image from the computer onto a surface (in our case a table) and set up the Wii remote so as it can see the whole screen keeping in mind the remote has a field of vision of about 45o. Then using software available from http://johnnylee.net you can carry out a four-point touch calibration as you do with any interactive whiteboard. This allows you to interact with your computer using the pen as you would with a mouse.
This is quite inexpensive as we have most of the hardware needed to carry it out. The pen is also quite small and would be easy for a child to hold and operate. We have considered that there may be issues with the child blocking the view of the Wii remote or the projection and are considering projecting from below a glass or Perspex surface with the Wii remote set up above the surface. This may mean using mirrors to angle the projection correctly.
10.8 Chosen Technology
After considering the pros and cons of each technology we have decided to use the Wiimote Whiteboard as we find it to be the most efficient and cost effective. Having set up the equipment and tested it we have found it to be easy to set up and can run on any computer provided it could be connected to a projector through a male-to-male VGA cable. This does not require an audio connection, as the user will be wearing a headset. This was decided, as it is vital that the user can clearly comprehend the sounds and are not distracted by outside noises. We hope to have this operating through Bluetooth as wires may restrict the user and may be an issue regarding health and safety when working with small children.
10.8 Time-lapse Code
For our main menu screen we hope to use time-lapse code to create a changing background. We want the animation to loop between a nighttime and daytime background over a set time if the animation is left idle. Having looked into this we have found that there are time codes available in processing. These are:
- day()
- hour()
- millis()
- minute()
- month()
- second()
- year()
We hope to use these to implement this and are currently working on the following example to turn a circle from green to red:
void setup(){
size(600,600);
}
void draw() {
background(0);
ellipseMode(CENTER);
int m = millis(); // Values from 0 – 59
int a = millis();
ellipse(50,50,300,300);
if (m<=10000){
fill(255,0,0);
}else{
fill(0,255,0);
}
if(a<= 20000){
fill(255,0,0);
}else{
fill(0,255,0);
}
}
10.9 Mouse Click Detection
The user interacts with the project the same way in which they would interact with a normal computer i.e. through mouse clicks. The following code is an example of using mouse clicks in Processing.
int value = 0;
import ddf.minim.*;
import ddf.minim.signals.*;
import ddf.minim.analysis.*;
import ddf.minim.effects.*;
Minim minim;
AudioPlayer player;
void setup(){
size(600,600);
minim = new Minim(this);
player = minim.loadFile(“Hall Of Fame – The Script feat. will.i.am (Lyrics).mp3”);
}
void draw() {
fill(value);
ellipseMode(CENTER);
ellipse(300, 300, 100, 100);
}
void mouseClicked() {
if (value == 0) {
value = 255;
player.play();
println(“playing…”);
} else {
value = 0;
player.pause();
println(“paused”);
}}
Laura Traynor 