Start with English, Aim for Greatness
Group: Electical and Electronics Group
Key word: Big Data, Artificial Intelligence, Internet of Things
Table of Contents
I.Project Motivation and Objectives
(I)Creative Motivation
(II)Project Purpose
II.Product Features and Creative Aspects
(I)Product Features
(II)CreativeAspects
III.Research Methodology
(I)Design Description
(II)Research Steps
IV.Theoretical Basis and Design Principles
(I)Theoretical Framework
A.Speech Recognition technology
B.Google Could Spreadsheet
C.MQTT Protocol
D.ESP32 Development Board
(II)Methods Comparison
V.Product Functions and Operation Guide
(I)English Quiz App Features
A.Listening Practice
B.Speaking Practice
C. English-to-Chinese Translation
D.Quiz Review
E.Add Custom Questions
(II)Connected Tic-Tac-Toe Game Functionality
A.Game Settings
B.Game Results
(II)Voice-Controlled Bluetooth Switch
VI.Development Process
(I)Problem Discussion
(II)Feture Suggestions
(III)Project Conclusion
Start with English, Aim for Greatness
I.Project Motivation and Objectives
(I)Research Motivation
In today's society, English has become an essential language that everyone must learn. To enhance our English learning abilities, we independently developed and English quiz application that helps users strengthen their skills in listening, speaking, reading, and writing. As Taiwan's well-known badminton player Lee Yang humorously said: "I say English very poor, I'm pround P-O-R-N-D(My English is terrible, but I'm...(proud)"(2025 Lee Yang) Although his grammar wasn't perfect, his confidence was admirable. His courage to speak despite imperfections serves as a reminder: mistakes is okay – what matters is the determination to improve. This is the spirit behind our project: to help learners bulid confidence and develop the skills they need to achieve their ideal English proficiency.
In the Department of Computer Science, English comminication skills are especially important, both for reading technical materials and for future global collaboration. We used this project as an opportunity to train ourselves in all four language areas. During development, we applied Google Speech Recognition technology as the foundation. Later, we realized this technology could be extended in new directions, leading us to explore and integrate Big Data, Artificial Intellgence (AI), and the Internet of Things (IoT) – forming our project's core: the "Big Three" or what we creatively call "(Great Tech Trio)".
(II)Project Objectives
In this internship project, we aimed to apply Speech Recognition technology and Google Docs integration to create an English learning application that enhances listening, speaking, reading, and writing skills. Additionally, we used MQTT communication to add an interactive game (Tic-Tac-Toe), and integrated Bluetooth voice control to expand the functionality, creating a well-rounded and engaging smart system.Our project objectives can be summarized in three key areas:
A.Project Deliverables:
(A)Develop a functional application
(B)Acquire app development and software design skills
(C)Understand real-world software implementation processes
B.Gain Competition Experience
(A)Prepare and present our project for external exhibitions or contests
(B)Creat visual and written documentation of our results
(C)Strengthen our public presentation and teamwork abilities
C.Train Teamwork and Collaboration Skills
(A)Practice effective task delegation and time management
(B)Collaborate to solve technical problems
(C)Consolidate research data and co-author the final report
II.Product Features and Creative Aspects
(I)Product Features
One of the key feature of this project is that our application was developed using App Inventor 2(AI2), a platform created by the Massachusetts Institute of Technology(MIT) with America. What makes this app stand out is designed specifically for mobile devices, rather than desktop computers. This significantly expands accessibility and user convenience. In today's society, we can't guarantee that every user owns a computer, but we can safely say that most people own a smartphone. By choosing to bulid a mobile-friendly application, we effectively meet the needs of a broader range of users and ensure greater usability in real-life situations.
This project is divided into three major parts. Part 1focuses on the development of an English quiz practice system, using App Inventor as the main development plarform. The program uses speech recognition technology as the input method. User responses are sent to Google Forms, which are then connected to a Google Sheets cloud spreadsheet. The results are then sent back to the app and dispalyed to the user. This system allows learners to practice vocabulary, phrases, and sentence structures effectively through interactive voice-based exercises.
Part 2 incorporates a casual game element by implementing a Tic-Tac-Toe game. The rules are simple and intuitive:
1.Players use voice commands to select a position on the board, such as "center" or "bottom left."
2.The system recognizes the instruction and places the player's "O" or "X" accordingly.
3.The first player to connect three in a row wins; if all positions are filled without a winner, the game ends in a draw.
This features uses the MQTT (Message Queuing Telemetry Transport) protocol to bulid a connected multiplayer architecture, allowing two users to play remotely. In addition to English learning, users can enjoy casual gameplay through voice interaction with friends, adding both fun and engagement to the experience.
Part 3 moves toward an Internet of Things (IoT) application. By integrating Bluetooth communication, the mobile app connects to an Arduino ESP32 board, enabling voice-controlled power switching. This practical application demonstrates how smart living technology can be implemented, step by step – turning the dream of cloud-based intelligent living into reality. Through these components, our project transitions from a basic quiz application into what we call the "Great Tech Trio": Big Data, Artificial Intelligence, and the Internet of Things – achieving our goal of creating a meaningful and innovatives smart-learning solution.
(II)Creative Characteristics
 |
| Figure 1 Speech Recognition-Driven Integration of Our Three Core Projects |
At the core of this project lies speech recognition technology, which connects all three components of our system. We chose to use Google Speech Recognition for development due to it's higher sensitivity and accuracy compared to other solutions we tested during the development phase. To manage our quiz data, we used Google Sheets, a cloud-based spreadsheet tool that typically supports functions like budgeting or grade calculations. In our project, it serves as a lightweight and convenient quesiton database. Unlike MySQL, which requires a more complex setup, Google Sheets provides an easy-to-use solution for storing English quiz content. It supports exercises in listening, speakingm reading, and writing, and functions as a reliable source for both testing and review materials. For our IoT component, we implemented a voice-activated Bluetooth power switch using the ESP32 microcontroller. The ESP32 is a programmable chip that integrates Wi-Fi and Bluetooth, effectively acting as an enhanced version of the Arduino Uno. It allows us to upload code for various hardware control applications – such as controlling electrical appliances or small modules like servo motors and ultrasonic sensors – enabling even broader potential for smart home applications. In addition, we utilized MQTT (Message Queuing Telemetry Transport), a lightweight messaging protocol that operates with a publisher-subscriber model. Much like the old BB Call system, one end sends data while the other receives it. In our project, we combined speech recognition with MQTT to design a two-player Tic-Tac-Toe game, allowing users to enjoy an interactive and relaxing experience while continuing to learn English.
III.Research Methodology
(I)Design Description
 |
| Figure 2 Project Architecture Diagram |
This project consists of three main subsystems, further divided into twelve funcational modules. As shown in Figure 2, the diagram illustrates the overall system architecture. Users interact with the system via a tablet interface, where thry can select from a main menu that leads to three core programs: 1.English Quiz
2.Tic-Tac-Toe Game
3.IoT Home Control
1.English Quiz
The English quiz system includes five key functions:
(A)Listening Practice
Display a list of English words with their Chinese meanings. When a word is tapped, it's pronumciation is played aloud, allowing users to practice listening skills, especially for unfamiliar vocabulary.
(B)Speaking Practice
Users tap to start the speaking test. The system uses speech recognition to identify the spoken word, evaluates the answer, and proceeds to the next question if the answer is correct.
(C)English-to-Chinese Translation
The app displays English text, and the user is required to translate it into Chinese. After submitting the answer, the system checks for accuracy and continues if correct.
(D)Quiz Review
Presents a Chinese word or phrase, and the user must input the correct English equivalent. The answer is submitted, evaluated, and the test continues with the next question if correct.
(E)Add New Questions
Allows users to add unfamiliar words into the quiz database for future practice, reinforcing vocabulary learning through repetition and review.
2.Tic-Tac-Toe Game
In this module, users can click the "Start Connection" button (button left) to initiate the multiplayer mode. After a few seconds, the status will change to "Disconnected", indicating that the game is ready. Players use voice commands to choose the position on the grid (e.g., "center" or "buttom left"). The system recognizes the speech input, places the mark, and checks for a win condition. If one player connects three in a row, they win; otherwisem the game results in a draw.
3.IoT Home Control
In this section, users select a Bluetooth device from a list, then pair it with the system. They can use voice commands to control electrical switches, turning devices on or off. This feature supports energy saving and improves convenience in home automation.
(II)Research Steps
To effectively mange the overall progress of our project, we developed a detailed project timeline as shown in Figure 3. We beagn by visiting the library to collect relevant technical books, academic journals, and periodicals related to our topic. In parallel, we utilized various online resources and forums to gather information on software programming and hardware development. Once sufficieng data had been collected, we proceeded to plan and conceptualize the design framework and technical tools required for our project, titled: "From English Quzi Application to the Great Tech Trio" This included identifying the appropriate design methods, development tools, and hardware components needed to implement the system.
 |
| Figure 3 Project Architecture Diagram |
 |
Figure 4 Time Allocation and Work Item (Gantt Chart) |
IV.Theoretical Basis and Design Principles
(I)Theoretical Framework
(A)Speech Recognition
Speech recognition is a technology that enables computers to convert spoken language into written text by analyzing audio input. In this project, we adopted Google Speech Recognition as the core engine for voice input processing. Within the English vocabulary quiz module, speech recognition is primarily used for spoken word identification. By integrating Google's speech recognition API, we were able to reduce common issues such as misinterpretation of words or detection of unintended input. This significantly enhances the accuracy and user experience of the speaking practice commponent.
(B)Google Cloud Spreadsheet
We created a lightweight question bank using Google Sheets, which serves as the backend data storage for our English quiz system. Through App Inventor 2, the app retrieves the quiz questions from the spreadsheet and displays them on the user interface. When the user answers by voice, the system uses Google Speech Recognition to convert the audio into text. The recognized word is then compared with the correct answer stored in Google Sheets. User responses are also submitted through a Google Form, Which automatically sends the data back to the spreadsheet for recordkeeping. The system compares the recognized iput with the correct vocabulary stored in the sheet. If the words match, the answer is marked correct; otherwise, it is marked incorrect.
(C)MQTT Protocol
MQTT (Message Queuing Telemetry Transport) is a lightweight messaging protocol developed by IBM in 1999. It operates on a publish-subscribe model between a server (broker) and clients, and has since become an open international standard maintained by OASIS. Designed specifically for the Internet of Things (IoT), MQTT is ideal for device-to-device communication due to it's lightweigh architecture and low latency, making it an efficient data transmission bridge in IoT environments. In our project, we integrated MQTT into the Tic-Tac-Toe multiplayer game as the core communication protocol. Rather than using direct peer-to-peer (P2P) connections between two tablets, which would restrict gameplay to only those two devices and within a limited range, MQTT allows for more flexibe and scalable communication. With MQTT, the game state (such as placing an "O" or "X") is sent to the broker, which then relays the message to all subscribed devices. This allows players to interact across long distances – for example, one user can be in Taipei while the other is in Kaohsiung – as long as both devices are connected to the internet and subscribed to the MQTT broker. This architecture removes both device and distance limitations, enabling a more robust and versatile multiplayer experience. Furthermore, MQTT is not limited to two devices. The system can be scaled to include multiple devices or users, allowing for greater interactivity and future expansion in various IoT-based applications.
(D)ESP32 Development Board
We chose to use ESP32 in our project due to it's compact size and integrated features. Compared to traditional Arduino boards paired with external Bluetooth modules, which tend to be bulky, complex to assemble, and more costly, ESP32 offers a more efficient and cost-effective solution. One of the key advantages of ESP32 is that it comes with-in Bluetooth functionality, eliminating the need for an additional Bluetooth reducing both cost and potential compatibility issues. When using a separate Arduino and Bluetooth module, we often encountered a problem where the tablet showed a successful Bluetooth connection, but the module itself failed to establish communication – causing unnecessary complications. In our design, we use ESP32 to receive binary signals (1 for ON, 0 for OFF) generated from voice commands via App Inventor 2. These binary signals are then transmitted to the ESP32, which controls a relay module to switch the power ON or OFF.
Figure 5 below shows our custom-designed Bluetooth-controlled power outlet circuit diagram.
 |
| Figure 5 Bluetooth-Controlled Power Socket Circuit Diagram |
(II)Method Comparison
In our Tic-Tac-Toe game, we chose to use MQTT instead of HTTPS as the communication protocol. The main reason is that MQTT is lightweight and optimized for low-bandwidth, low-power environments, which makes it a more suitable choice for real0time applications like multiplayer games. In contrast, HTTPS, while secure and well-structured, comes with a more complex and resource-intensive overhand, making it less ideal for simple IoT interactions. For our IoT control system, we utilized the Bluetooth feature of the ESP32 for device connectivity, rather than relying on Wi-Fi. Bluetooth was selceted due to it's simplicity, lower power consumption, and suitablilty for short-range control applications, such as switching electrical devices on and off through voice commands. Additionally, using Bluetooth avoids some of the network setup complications associated with Wi-Fi, making the user expericence smoother and more stable.
 |
| Figure 6 Project Development Flowchart |
V.Product Functions and Operation Guide
(I)English Quiz App Features
(A)Listening Practice
 |
| Figure 7 English Quiz App - Overall Function Showcase |
1.Tap "Listening Practice"
2.Tap "Start Quiz"
3.Tap "Play Question Audio" to begin the listening test
4.Tap "Submit Answer" to send your response for evaluation
5.Tap "Next Question" to continure the quiz  |
| Figure 8 English Quiz App - Listening Practice Showcase |
(B)Speaking Practice
 |
| Figure 9 English Quiz App - Speaking Practice Showcase |
1.Tap "Speaking Practice"
2.Tap "Start Quiz"
3.Tap "Voice Recognition" to begin the speaking test
4.Tap "Submit Answer" to send your response for evaluation
5.Tap "Next Question" to continure the quiz
(C)English-to-Chinese Translation
1.Tap "English-to-English Translation"
2.Tap "Start Quiz"
3.Tap the text input field and enter your translated answer (English → Chinese)
4.Tap "Submit Answer" to send your response for evluation
5.Tap "Next Question" to continure the quiz
 |
| Figure 10 English Quiz App - English-to-Chinese Translation Showcase |
(D)Quiz Review
 |
| Figure 11 English Quiz App - Quiz Review Showcase |
1.Tap "Quiz Review"
2.Tap on any word displayed in the list to listen to it's pronunciation
3.Use this feature to reinforce vocabulary that you are unfamiliar with or find more difficult
(E)Add Custom Questions
1.Tap "Add New Question"
2.Tap the "Chiese Meaning" input field and enter the word's meaning in Chinese
3.Tap the "English Word" input field and enter the corresponding English word
4.Tap "Confirm Add" to add the new vocabulary item to the quiz database
5.If any input is incorrect, tap "Clear All" to erase all entered content from the fields
 |
| Figure 12 English Quiz App - Add Custom Question Showcase |
 |
| Figure 13 English Quiz App - Cloud-Based Question Data Showcase |
(II)Connected Tic-Tac-Toe Game Functionality
(A)Game Settings
When the game begins, the "Connect" button is used to establish a connection with the MQTT server to enable multiplayer gameplay. The connection status is displayed on the screen in real time. To start a new game, simply tap the "Reset" button to restart the game session.
 |
| Figure 14 Connected Tic Tac Toe - Game Setup Showcase |
(B)Game Results
Table 1. Game Performance Display
|
|
|
Player O places a mark in the top-left corner using voice recognition
|
Player X Places a mark in top-right corner using voice recognition
|
|
|
|
|
Player O Place a mark in the bottom-right corner using voice recognition
|
Player X places a mark in center using voice recognition
|
|
|
|
|
Player O places a mark in the bottom-left corner using voice recognition
|
Player X places a mark in the left side using voice recognition
|
|
|
|
|
Player O places a mark in the bottom using voice recognition
|
Displays the game result
|
(III)Voice-Controlled Bluetooth Switch
After selecting a Bluetooth device from the list, users can control the power switch wirelessly via voice commands. This enables hands-free, smart control over connected applications. A deteiled overview of the feature functionality is shown in Table 2 below.
Tanle 2. Voice-Controlled Bluetooth Switch - Result Dispaly Table
|
|
|
When the command "Turn on the light" is spoken, the Bluetooth module sends an ON signal to activate the power switch
|
|
|
|
When the command "Turn off the light" is spoken, the Bluetooth module sends an OFF signal to deactivate the power switch
|
VI.Development Process
(I)Problem Discussion
At the begining of the design process, we used Winfows Forms (as shown in Figure 5) and developed the program using Visual Basic.NET (VB.NET). However, during the testing phase, we encountered several issues – such as the program failing to open, incorrect speech recognition results (e.g., speaking in Chinese but receiving English ouput), or multiple or inaccurate result when speaking English. After spending a significant amount of time troubleshooting and attempting to fix these problems, we decided to switch to MIT App Inventor 2, developed by the Massachusetts Institute of Technology. Compared to the text-based nature of VB.NET, which relies heavily on English syntax and can feel dry and complex, App Inventor 2's block-based programming interface is much more intuitive and beginner-friendly. While it still required time and effort to build each function by stacking blocks, the process was smoother and more visually understandable.
 |
| Figure 15 Visual Basic.NET language Diagram |
(II)Future Suggestions
While this project has achieved many of it's goals, there are still some imperfections and areas for future enhancement:
(A)English Quiz App:
Some questions many appear repeatedly, and the quiz currently lacks a pause feature or a final score summary. Additionally, when an answer is incorrect, the app does not highlight which part of the answer was wrong, making it harder for users to learn from their mistakes.
(B)Tic-Tac-Toe Game:
The voice input for moves is limited to specific phrases like "top-left" or "left". It doesn't support alternative expressions such as "left side" or "bottom area", reducing the flexibility and naturanlness of speech input.
(C)IoT Smart Home Control:
The speech recognition for Bluetooth control is not always reliable, sometimes failing to excute commands or experiencing significant delays. These issues affect the user experience and need to be optimized.
These are key areas we plan to improve in the future to make the project more complete, user-friendly, and robust.
(III)Project Conclusion
In this project, we successfully developed an English quiz application using MIT App Inventor, which retieves question data from Google Sheets. With the integration of speech recognition and Google Forms, users can answer questions through voice iput, enabling slef-directed English language practice. We also completed development of an interactive game (Tic-Tac-Toe). Players can place their moves using voice commands, and the game utilizes MQTT protocol for real-time multiplayer connectiveity. This not only modernizes a traditional game but also adds a fun and engaging element to a language application. Lastly, we implemented a voice-controlled power switch, using Bluetooth communication to connect with the ESP32 module, which controls a relay inside an electrical outlet. This feature allows users to control appliances easily, wirelessly, and hands-free, bringing us one step closer to the vision of a smart phone. Through this project, we demonstrated how combining speech recognition technology with language learing can result in a simple yet effective English learning tool. With the added features of casual gaming and IoT-based home control, users can enjoy both entertainment and convenience during their free time – enhancing their English speaking skills while experiencing the joys of smart living.
Writer: Roy Yan
Comments
Post a Comment