The circuit for the switches

The circuit for the lights

The Software System:

The software system is where I spend most of my time and efforts. The whole system is built on ROS and is a mixture of Python and C++. Below is a general overview of what messages pass between what nodes and how everything is connected.

Four main nodes communicate with each other asynchronously that control the flow of the pinball machine when it is run in autonomous mode. These nodes are Input_Output.ino, track_metal.cpp, run_low_level.py, and GUI.py. When not running in autonomous mode, the node track_metal.cpp may be omitted.

Input_Output.ino is the script uploaded to the Arduino that handles all the literal switching on/off of the pinball components and reading in from any playfield switches. These items include flippers, bumpers, slingshots, lights, switches, stand-up targets, a spinner, a start button, and an autonomy on/off switch. The Input_Output.ino node has two major functions: turning on items on the playfield and relaying the status of switches back to the computer. 

track_metal.cpp  node takes care of locating the ball in real-time, locating the flippers at set increments, and making a decision of when to flip the flippers. Additionally, the track_metal.cpp node takes in a single input from the Input_Output.ino node of the status of the physical autonomy switch to know when it should be flipping the flippers.

Locating the ball in track_metal.cpp is done in 6 steps:

1. Take a picture of the playfield when the flippers are down and there is no ball in the frame.
2. Convert that frame to grayscale.
3. Compare every new frame to the original frame by using a binary threshold. This is possible because each frame has a single pixel color value that defines it.
4. Enlarge anomalies found in the binary comparison using cv::dilate().
5. Retrieve a list of contours from this binary comparison using cv::findContours().
6. Loop through the list. Any contour’s area that is in-between some threshold will be classified as a ball.

This method for finding pinballs on the playfield has both advantages and disadvantages over attempting to detect circles in the image. The first major advantage with this method is speed. After preliminary testing, trying to find circles in the image proved to have too much delay in the system and the pinballs would move too fast to be detected at high speeds. Secondly, when a pinball is moving at high speeds, it appears on the camera with a large amount of blur surrounding it. This could greatly throw off the detection of circles as the shape no longer appears circular. The major disadvantage of this was the frequent mislabeling of other objects as pinballs. Many times when lights would turn on or other things would move on the playfield they would be categorized as a ball. This, however, proved not to be a major issue as they would not typically appear in the flipping area. If we were to include velocity tracking over time of a specific pinball, this would have been a greater issue.

Locating the flippers in track_metal.cpp is done in 6 steps:

1. Create a HSV threshold that the flippers must fall between. Because they are bright yellow, it can be a relatively narrow hue threshold.
2. Enlarge anomalies found in the binary comparison using cv::dilate().
3. Retrieve a list of contours from this binary comparison using cv::findContours().
4. Find the centroid of each contour in the list.
5. Sort the centroids by their distance to the center of the frame.
6. Return the two most central contours as long as their areas are between a certain threshold. They will be the two flippers.

This method for finding the flippers was extremely effective and accurate. The only issue we had when implementing this on the real system was the introduction of the yellow acrylic. This acrylic fell into almost the same HSV color space that the flippers did. The above-mentioned steps, however, take the center two contours, which the flippers always were.

Lastly, the track_metal.cpp node tells the system to flip the flippers when the detected pinball falls into a desired “flip zone” that is attached to the flippers. The “flip zone” is created by initializing a circle on the center of rotation of each flipper, turning it into a N-gon with a custom function, then removing most of the points of that N-gon to create a wedge. This method allowed the user to tune the center of the circle location, the radius of the circle, and which points to remove from the circle. This tuning allowed certain shots on the playfield to be hit more frequently or less frequently and favoring different types of playstyles.

run_low_level.py  node is responsible for, in software, keeping track of the status of all playfield components. This includes physical components like the switches and lights, but also digital components like the score, mode, and bonus multipliers. Every switch on the playfield has a ROS callback that triggers when the switch is hit by a pinball. These switch callbacks all mostly do the same thing such as updating the score and turning on a light. Some of the callbacks, like the switch that activates multiball, are slightly more complicated. 

In addition to keeping track of all the playfield components in near-real-time, the run_low_level.py node is in charge of telling the Arduino to turn lights and solenoids on and off. To do this effectively, we make use of a python scheduling package called apscheduler. This package allows the developer to provide a ‘datetime’ formatted time to schedule a command or function to run. It does not, however, allow for a developer to say they want a command to be run ‘x seconds in the future’. We wrapped this functionality into run_low_level.py to allow for lights to be turned on and told to turn off in ‘x seconds’. 

run_low_level.py also provides some functionality to keep track of a list of high scores for manual play. After each game, users can enter their name and it will check their score against a list internally. If they make it into the top ten players, their name is slotted into the list. To store and retrieve this list, the python pickle package was used for writing a serialized list to a file. Additionally, the list is initialized to a set of default scores if the user has not created the list before.

GUI.py  is extremely simple compared to the others listed. The only thing that GUI.py does is display on the screen the current score, bonus, and play mode. All three of these enter GUI.py by a ROS topic that updates the GUI object. The GUI was built using the python Tkinter package. This package is incredibly simple to use and provides a sleek look.