Choosing a Display

The quality and size of your display, next to system latency, is perhaps the single most critical component of your physical rig when designing a PlayMotion based experience. Two main factors are physical size and orientation. For instance, consider a projector aimed down 90 degrees to evenly illuminate the surface of a tabletop cut to precisely the right size. Or, consider an array of 12 networked plasma displays positioned along a long corridor. Our purpose here is to open your mind to the design possibilities available to you when you look beyond your desktop or laptop monitor.

Definition: A display is a device which radiates energy on the visible spectrum wavelength in order to transmit information from a machine to the human brain via the optical cortex. Displays can range in complexity from a simple lightbulb (think: one pixel, one bit - on & off) to as complex as a holographic video projector (multidimensional, multicolor).

Ramesh Raskar is credited with the concept that displays in aggregate, seen as a unified structure, can be considered as windows that enable us to penetrate through the present timespace / physical paradigm and to peer into the extra-dimensional world of machine, code and spirit.

The capabilities of a display can be measured and compared across several major axes:

• type,
• orientation,
• physical size,
• resolution,
• aspect ratio,
• bit depth,
• color gamut,
• brightness,
• contrast,
• latency, and of course
• cost.

type:
Major classifications include vector vs. raster, and emissive vs. reflective. Examples include CRTs, LCD panels, OLEDs, plasma panels, rear projection, front projection, video matrices (many individual displays managed by a video control system to function as a single cohesive display), and LED wall / billboards.

orientation:
Classically, displays are oriented in landscape fashion, with the image plane situated vertically so that the vector drawn from the eyes of a standing or sitting person will be perpendicular to the display surface. When creating your PlayMotion experience, we invite you to consider other orientations. For instance, projection based PlayMotion experiences are generally best viewed with the projection coming all the way to the floor, in contrast to the standard home theatre setup of 3-8 feet up. Another very compelling orientation from a design standpoint is the table top display, where the display is placed horizontally on a flat surface. Finally, we invite you to consider the effects of projecting onto three-dimensional or sculptural surfaces, which can produce quite surprising results when explicitly factored into the software design.

physical size:
For small raster screens, consider the color video display on your cellphone. Large LED screens have been installed as large as 240 feet wide, and high definition projections have illuminated entire facades of city buildings. Much of the power of a PlayMotion style experience derives from a 1:1 ratio of virtual world to real world, which we call a human scale experience. If you wish to design in this fashion, you need to strongly consider a display capable of fitting one or more full sized humans within its addressable area.

resolution:
number of addressable pixels used to synthesize an image. Resolutions are commonly defined to match media content standards. XGA is a fairly common 4:3 aspect ratio standard for computers which is 1024 pixels wide by 768 pixels tall. 1080 HD is the new standard of excellence, which is 1920 pixels wide by 1080 pixels tall running at 60Hz. It takes a decent GPU to render complex geometry in real-time at HD resolutions.

aspect ratio:
for rectilinear displays, the ratio of width to height in terms of physical dimension. Traditional aspect ratios include 4:3 (classic television) and 16:9 (cinema, and HD). Many large public displays are created in non-traditional aspect ratios. A simple and effective means of jolting participants out of the "this is video, I'll sit and watch" syndrome is to rotate the display 90 degrees. Seeing a video wall 16 feet tall by 9 feet wide is a good call to attention. :)

bit depth:
The theoretical range of color values that the display will accept. The simplest display is one bit (on or off, white or black). Typical computer displays are 32-bit (8 bits per channel of red, green, and blue, plus an "alpha" or transparency channel; each channel has 8 addressable bits, which translates to 256 available levels of intensity. When all three channels are combined (red 0-255, green 0-255, and blue 0-255), all possible combinations result in 16.7 million possible colors.

color gamut:
while bit depth determines what signals may be recevied by the display device, color gamut defines what range of colors is practically reproducible by the device in the real world. Displays with broader color gamuts are said to be of higher fidelity, though brightness, resolution, and contrast also play significant factors.

brightness:
how much light energy can be pumped out of the display. This is typically measured in lumens, and is measured at a fixed distance from the display surface while driving a maximum brightness signal (for instance, 100% white on a 32-bit display)

contrast:
the difference between the brightest bright reproducible, and the darkest black. Higher contrast translates to more realism. Low contrast can result in the "glowing screen" phenomenon.

latency:
how long it takes the display to update from time of signal reception. Also, how long it takes any given pixel to update from one color to another. Displays said to be "video capable" need minimum refresh rates of 24 frames per second, or latency of less than 0.04 seconds (1/24th of a second).

cost:
Cost is a major factor in any installation. A generally good way to think about display cost is either cost per square foot of coverage, or cost per lumen of brightness. Generally the hands down winner in these comparisons are projectors, though these devices do have their limitations, most notably their need for a relatively dark environment. On the other end of the spectrum, some of the most expensive displays in the world are massive LED walls manufactured by the likes of Barco and Daktronics; while these displays can run into tens of millions of dollars, they also have the highest brightness of anything on the market, making them ideal for daylight and outdoor applications.

Once you've selected your display, its time to consider the location of your camera.

Selecting a Camera Location

There are three distinct physical "components" which comprise a PlayMotion environment: the camera, the display and the player(s). The arrangement of these three things relative to one another alters which portion and angle of the player(s) is seen by the camera. This information will impact which are the best computer vision methods to retrieve consistent data about the player(s) actions and intentions.

Following are a few typical configurations. In each case it is assumed that the player are facing the display and the camera is facing the player.

The Traditional camera -- player -- display surface
In this arrangement the camera view is of the back of the player. In a rear projection setting, co-locating the camera and the projector gives strong optical alignment of the camera's view and the projector's display. If background subtraction is used, there is a clear mapping from the player's projected shadow to the signal generated by the vision pipeline, causing the shadow to be the player's avatar. This configuration and its best practices are embodied in the PlayMotion Reference Environmental Specification.

The Laptop player -- camera/display surface
Here the camera and display surface are close to one another. The name comes from the fact that most laptops sold today come with a camera mounted just above or below the LCD display. In this configuration the camera sees the front of the player. If the camera's view or the segmentation image is displayed as a layer of the graphical output, this orientation emulates a "magic mirror", where the real world is augmented with game elements.

The Pass-Thru camera -- display surface -- player
In this configuration, the camera sees the display and the (front of) the player superimposed together. This setup requires a display that is not opaque in order for the camera to be able to detect the player on the opposite side of the display. While similar to The Laptop, this configuration provides more accurate registration near the display, allowing tracking of fingers, hands or eyes relative to the display.

This list is not intended to be comprehensive, but rather a taste of the options available in physical configuration. Consider, for example, the case when the camera is not fixed in place. With 1 or more degrees of freedom, one could switch between configurations mentioned above, or even use the camera's motion as an input vector.

Now that you've decided on a display type and camera location for your application, lets explore the effects of environmental lighting.

Accounting for Local Lighting

Since cameras provide the input signal for PlayMotion, it's important for their image quality to be optimal. Some aspects of the image quality, such as focus, are straightforward to adjust. The impact of light in the game play space, however, is a much more complex issue.

Varying the light where your game is played is likely to change the way you use PlayMotion to generate control signals. Poor, non-uniform or non-constant lighting can have unpleasant effects on segmentation algorithms. Some of the techniques available in PlayMotion are stronger against adverse lighting than others. But, as always, there is no free lunch. These more robust techniques have their own drawbacks -- they may be more computationally expensive or could provide poorer quality (or just plain different) segmentation than other methods.

Natural light from the sun, candles, fire, etc is notoriously inconsistent. This is obviously true for flames, but remember that the movement of clouds and the time of day have a huge impact on the way that the sun lights an environment. Given the intensity of natural sunlight, amazingly small amounts of even indirect sunlight (as in, an open shade 20 feet away) can cause vast irregularities in the input signal. Optimal environments will be completely sealed from natural sunlight (or, optionally, set up for outdoor use during nighttime)

Some electric lights, such as incandescent, neon and fluorescent bulbs provide a stable, even light that is almost always compatible with all of PlayMotion's vision algorithms. On the other hand, mercury-vapor and halogen lamps create erratic light that may produce visible artifacts in your camera image. Another thing to observe is that many modern lights fluctuate on a given frequency. While this is generally imperceptible to the human eye, it may appear as a pulsing brightness on your incoming camera signal.

Finally, there's the light from your display surface, aka visual feedback. Reflection of the light emitted by a computer monitor probably will not have a negative impact on the camera signal. If the camera is facing the display, however, the light in a (presumably large) part of the camera's view will be changing constantly. Similarly, high-brightness displays such as projectors or LED arrays can create significant dynamic in the overall illumination of your environment. Traditionally, PlayMotion uses an open-loop feedback scheme where the camera influences the display, but the display has no impact on what the camera detects. If the camera image is changed by the displayed image, we close the feedback loop. One possible way to avoid this scenario is to use a camera that filters out visible light, meaning that the displayed image will not appear in the camera view. This idea is implemented in the PMSP, an industrial class infra-red sensor array available for purchase from PlayMotion.

Now that you've considered your display type, chosen your camera location, and accounted for local lighting conditions, its time to get into the nitty gritty: configuring the filter graph.

Optionally, you can use the default filter graph and jump straight into development by running the Calibrate Utility and delving into the included Sample Applications.

Configuring the Filter Graph

After adjusting your homography, you can call up the menu and enter the Filter Graph mode of the Calibrate Utility, where where you can configure each of the filters defined in the active input filter graph. Simultaneously, Calibrate shows the real time effects of the decisions you make on the overall clarity of the camera input signal.

By changing parameters for the image segmentors and/or image processing filters, you can ensure that the raw camera image is transformed into an optimal segmented output image to be used by the vision services in your game.

Typically this involves configuring the image source(s), image segmentor(s), and any image processing filters.

Image Sources
For Lumenera cameras, webcams, PMSPs or other cameras defined as the source in the filter graph, you can mirror the output about the X- and/or Y-axes to correct for camera orientation. Additionally, you can adjust the Lumenera cameras' exposure to achieve the desired mean intensity level in the video stream.

Image Segmentors
For background subtraction filters, such as bgSubtract, you can adjust their thresholds to make the segmentation more accurate. A lower threshold makes the background subtraction more lenient in classifying a given pixel as non-zero ("true"). Sometimes in the segmented image, people's shapes can appear to be incomplete, or disconnected. Lowering the threshold can correct this problem. Too low of a threshold, however, will result in many false positives, or pixels classified as non-zero which should be zero.

Image Processing Filters
Usually the output data of the segmentor filters needs to be adjusted before it is fit for use in computer vision algorithms. Several image processing filters can be used to accomplish this and can be configured in Calibrate.

One common image processing filter is the quadWarper. Typically when the camera sees beyond the extents of the projected PlaySpace, a quadWarper can be used to warp the region of the camera's sensed image that falls within the projector's display frustum to match the extents of the display. When a quadWarper is defined for the source, you can move the corners or sides of a quadrilateral to set the area of interest within the camera's raw image, which will be warped into to the size of the segmented output image.

Another common image processing filter is the medianFilter. Many times there is noise in the output of the image segmentors, which can be removed by a median filter. You can set the size of the filter as well as the number of passes the filter makes to smooth out the segmentation.

In addition to configuring the image source, image segmentors, and image processing filters, the filter graph mode of Calibrate lets you adjust the screen real world width and height dimensions. These have no direct affect on the output segementation image, but are used in the vision library and can be useful for scaling objects within your games to match real world scale. These values should be set to the dimension of your display surface in real-world units.

NOTE: to add or arrange filters in the vision pipeline, you must manually edit the .corners file.

Configuring the Vision Services

In addition to configuring the parameters of the filter graph, the Calibrate Utility is your command center for configuring and visualizing most all of PlayMotion's vision services. From within Calibrate, easily switch between service configuration modes by pressing the ctrl + left / right arrow keys. (helpful hint: print the Calibrate Quick Reference Card)

Each service has its own real-time visualization to help convey what data the service provides to your application. Most services also have tunable parameters accessible via an on-screen menu (within Calibrate, TAB toggles menu on/off; arrow up/down navigates between parameters; and arrow left/right adjusts individual values).

Choose a service below to begin your journey, or visit our Services Matrix for some wild-style browsing:

Initialize VisionInput

The VisionInput class is the center of the PlayMotion runtime. It marshalls the vision pipeline, and generates the input signals used in each update cycle of a game's logic. Accessors for all of the parts of the pipeline belong to the VisionInput class.

So in order to do anything with PlayMotion, you must first initialize the system:

#python
playmotion.init()

or

//c++
VisionInput* input = new VisionInput();

The PlayMotion runtime can be run in its own thread, providing asynchronous access to the data generated by the vision pipeline and decoupling the input and update units of the game. To do this, call

//c++
input->start();

Note the start() call works in Python as well, but the wrappers currently do not relinquish the Global Interpreter Lock, so the performance gain is minimal. Watch the changelog for updates!

Once you've initialized VisionInput, its time to subscribe to services.

Subscribe to Services

Some computer vision algorithms are quite computationally intensive. To ensure that the games you develop are as responsive as possible, PlayMotion's computer vision services are provided on an as-needed basis. When a game requires a specific type of processing, it can request the vision pipeline to compute the given data and gain access to it.

Controlling which services should and should not be run is the job of the VisionSubscription. Like a magazine subscription, you ask for an item and it begins "arriving" in the next time the vision pipeline is updated. Similarly, you can cancel subscription to one or more of the services. Their processing will be stopped and the data they provide will no longer be available.

Say we want to know how far it is from the upper lefthand corner of the segmentation image to the nearest segmented pixel (i.e. non-zero value). First, create a VisionSubscription, and ask for the distance transform service.

#python
mySubscription = playmotion.VisionSubscription()
mySubscription.subscribe(playmotion.SERVICE_DISTTRANSFORM)

After the next update of the vision pipeline, the functions associated with the distance transform service will provide useful data:

#python
playmotion.update()
d = playmotion.getDistanceFromEdge(0,0)

When we're done with the service, we can manually remove it from the subscription to save processing cycles.

#python
mySubscription.unsubscribe(playmotion.SERVICE_DISTTRANSFORM)

The subscription is automatically cleared on release of the VisionSubscription object.

This is how you call it once. But the PlayMotion input signal is a real-time video stream. Next, we'll find out how to request new vision data on a per-frame basis.

Request New Vision Data

When you want the latest data from the vision pipeline, you must ask for an update:

#python
playmotion.update()

Generally, this call is made exactly once per update cycle within your game. In a single-threaded environment, this actually invokes the vision pipeline's update routine, producing new data.

If VisionInput::start() has been called, though, the vision thread is a tight loop effectively containing only the pipeline update. In this case, the update() call in the application thread acts as a "buffer-flip" in the vision pipeline, fetching the last processed frame, and it's accompanying service data. We use the same function call for these two purposes to make the transition between single- and multi-threaded environments seamless.

Once you've updated the data, you're ready to perform some signal analysis, and get the desired data from the subscribed services.

Signal Analysis with Services

The ultimate purpose of the filter graph is to transform the input video signals into one or more image data streams that can be analyzed to produce numerical input values to a game each frame. Many analysis techniques are provided by PlayMotion's vision services.

Generally, these techniques fall into one of two categories: static analysis, where only data from a single frame is analyzed, and dynamic analysis, where subsequent frames are compared in some meaningful way in order to generate time-based data abstractions.

#python
dist = playmotion.getDistanceFromEdge(0,0)

#python
#x1, y1 indicates the point we wish to query, in world space 

v = cv.cvPoint2D32f(0,0) #out parameter (pointer)
w = 16 #worldWidth
h = 9  #worldHeight

testpoint = playmotion.xformPointToSegmentation(x1,y1,w,h)
dist = playmotion.getDistanceFromEdge(testpoint,v)
result = Vec3(v.x, 0, -v.y)

#python
playmotion.captureSnapshot("PointLeft")

#python
playmotion.captureSnapshot("StandOnOneFoot")

#python
playmotion.getScalarDifference("PointLeft")

#python
diff = playmotion.getScalarDifference()

#python
gList = playmotion.getMotionLocations(0)
detected = not gList

Do Your Own Analysis

You may find that the information you want is not available through PlayMotion's services. We welcome developers to perform their own computer vision processing, using the direct matrix output stream from the filter graph as your input. If you're particularly proud of your technique or would like to see it optimizied by inclusion in the runtime, please share it on the forums.

The images used by the PlayMotion vision pipeline are stored in the popular IplImage matrix structure, which makes them perfectly suited for third party analysis using OpenCV. Below you'll find a simple example of how to do analysis beyond what's available from PlayMotion.

#python
from opencv.cv import cvCountNonZero

sImg = playmotion.getSegmentationImage1()
w = playmotion.getSegmentationWidth()
h = playmotion.getSegmentationHeight()

occupiedArea = float(cvCountNonZero(sImg)) / (w * h)

What's generated here is the percentage of the segmentation image that is occupied by "good" (i.e. non-zero) pixels, as a number between 0 and 1. There are plenty of much more interesting things that can be done with PlayMotion's input signal and OpenCV.

Another excellent (and elegant) example of inline vision processing is provided in the Sample Application: Heads. In this case additional processing is computed within Python to find a more accurate Head object than the PlaySDK natively provides.

Of course, you don't have to use OpenCV to do your processing. Other computer vision primitives libraries are avialable, and you can directly manipulate the data if you choose.

See also:

• Sample Application: Heads
• OpenCV: Open Source Computer Vision Resources
• A Python Interface to OpenCV

  
  

All About Coordinate Spaces

The canonical coordinate spaces for raster images and the 3D virtual spaces in which most video games exist differ significantly.

The coordinate description for images, which will be referred to as vision space is an integer lattice (no values like 0.5) with origin in the upper lefthand corner. The positive x direction is rightward across the screen, and the positive y direction points down.

The 3D graphics for the game live in what is called 3D world space. World space coordinates are floating point triples (x,y,z). Axes are typically arranged to obey the "right hand rule". In Panda, they are ordered X,Z,Y where Z is depth normal to the display surface.

Often, PlayMotion games employ a fixed virtual camera in the 3D world that faces the origin. In these cases, world space can be considered a plane orthogonal to the camera's view with its origin in the center. For this discussion, let's assume the positive x direction is to the right and positive y is up, as in the Cartesian plane. For simplicity, let's also assume that the virtual camera's extent is [-1,1] in both axes.

Points in these two spaces have a slightly complicated relationship to one another because of the differences in the coordinate representations. Interpreting values from the vision pipeline in the virtual world requires coordinate space transformation. Some shortcuts for such transformations are available through the runtime.

Here's a simple example using the distance transform service. Let x and y be the 2D coordinates of a point in the world space plane. The function getDistanceFromEdge(), operates in vision space, so we must transform the point to its world space equivalent.

#python
visionSpacePoint = playmotion.xformPointToSegmentation(x,y)

All that we are doing here is making sure that the world point corresponds to the correct point in vision space. If the world point is in the bottom right corner of the screen, then the point should be in the bottom right corner of the segmentation image too. If the world space point is in the middle, then the vision space point should be in the middle as well.

Now that we have this information, we can simply make the call

#python
visionSpaceDistance = playmotion.getDistanceFromEdge(visionSpacePoint)

The distance returned is relative to the vision space, but we need it in a world space context. To accomplish this, we can just use another xform...() function

#python
worldDistance = playmotion.xformScalarToWorld(visionSpaceDistance)

Now we have an accurate distance value in units that are meaningful in our world.

Note that the xform...() functions do not require a subscription to any vision service.

Technical Reference:

• Functions to convert between vision space & world space

Framework

This is the simplest of all the Sample Apps. In a nutshell, it:

• sets up the PlayMotion services
• creates a card in Panda3D on which to place the segmentation as a texture
• maps the texture to the card
• sets up a task in Panda3D to update the texture with each frame refresh

segmentation of PMSP signal on screen.

location:

filename: PMFramework.py
directory: C:\PlayMotion\PlaySDK\demos\Framework

related links:

• Using the PMVE Runtime

Heads

Heads is a very basic demo which:

• shows how one can call the Humans Service to find a head, and
• maps the results to an on-screen text indicator beacon.

a red "H" indicates the headpoint as returned by the getHumans service. This segmentation image is delivered via a PMSP, a high performance machine vision input device available for purchase from PlayMotion.

location:

filename: pm-heads.py
directory: C:\playmotion\PlaySDK\demos\Heads

related links:

• The Humans Service
• Blob Labeling / GetHumans
• Accessor Functions Requiring SERVICE_HUMANS
• The Human Model in PlayMotion

Heads-Advanced

Heads-Advanced takes the Heads demo and upgrades it with some internal logic:

• shows how one can call the Humans Service to find heads
• directly accesses the IplImage structure from within Python, then
• utilizes an interpolation function to find a more accurate head reading, and
• maps both results to on screen text indicator beacons.

a yellow "V" indicates the interpolated head, and a red "H" indicates the raw head as returned by the getHumans service. This segmentation image is delivered via a PMSP, a high performance machine vision input device available for purchase from PlayMotion.

location:

filename: PMHeadsAdvanced.py
directory: C:\playmotion\PlaySDK\demos\Heads

related links:

• The Humans Service
• Blob Labeling / GetHumans
• Accessor Functions Requiring SERVICE_HUMANS
• The Human Model in PlayMotion
• Performing your own Signal Analysis

Hands

Hands shows how one can call the Humans service to see hands, and map those to an on screen render. It uses the same basic logic as the Heads demo, and adds support for multiple Human detections on-screen simultaneously.

location:

filename: PMhands.py
directory: C:\PlayMotion\PlaySDK\demos\Hands

related links:

• The Human Model in PlayMotion
• The Humans Service
• Blob Labeling / GetHumans
• Accessor Functions Requiring SERVICE_HUMANS

8ball

8ball demonstrates how a very basic physics engine can have the player's virtual shadow inserted into the scene and uses the Edge Normals service to calculate collisions between virtual billiard balls and real world silouhettes.

8ball calculates rigid body physics collisions between segmentation edges of physical bodies, and the surfaces of the virtual billiard balls

any number of players can interact simultaneously with the virtual objects

location:

filename: PM8ball.py
directory: C:\PlayMotion\PlaySDK\demos\8ball\

related links:

• Edge Normals Service
• Accessor Functions Requiring SERVICE_EDGENORMALS

Chakra

Chakra is an art piece which connects visualizations of the body's energetic centers with detected individuals in the PlaySpace using the Humans service. It is optimized for a rear projection environment, where the rendered chakra symbols act as a sort of 1:1 spiritual mirror for the humans playing in front of it. Two sets of images are available, depending on the directory set in the config.py file.

location:

filename: PMchakra.py
directory: C:\PlayMotion\PlaySDK\demos\Chakra

related links:

• Humans Service
• Chakra on Wikipedia

Tank92

VERSION 1.1 PREVIEW

Tank92 is a prototype videogame-style experience which takes advantage of the Moment Service as its primary form of player input.

filename: scorchmain.py
directory: C:\playmotion\PlaySDK\demos\Scorch\src\

related links:

• The Moment Service
• Accessor Functions Requiring SERVICE_MOMENT

Maxwell

VERSION 1.1 PREVIEW

Maxwell is a dynamic real-time visualization of electromagnetic forces, using variable charges attached to player's hands.

Features include:

• a real-time visualization of Maxwell's Equation
• interpolation of hand movement
• building / decaying charge magnitude based upon how long an individual charge is attached to / detached from an actively detected hand
• keyboard configurable variables to adjust performance / resolution.

location:

filename: max.py
directory: C:\playmotion\PlaySDK\demos\Maxwell

related links:

• Wikipedia: Maxwell's Equations

Big

VERSION 1.1 PREVIEW

Big is a sample application demonstrating a floor firing environmental setup. It is inspired by the humongous toy keyboard installed in the floor of the toy store in the movie Big (1988).

External Links:

• WikiPedia: Piano Key Frequencies
• WikiPedia: Twelth root of two
• WikiPedia: Piano Acoustics: The Railsback Curve

Islands

VERSION 1.1 PREVIEW

Islands is an elegant demonstration which accesses the Distance Transform Service to morph the 2D camera input signal into a 3D model output in real-time.

the colorized distance transform image of a human in the scene

the Panda3D camera begins rotation to gain perspective on the scene

the final 3D island as derived from the original distance transform data

location:

filename: islands.py
directory: C:\playmotion\PlaySDK\demos\DistTran\src
author: Jordan Killpack

related links:

• The Distance Transform Service
• Accessor Functions Requiring SERVICE_DISTTRANSFORM

Filter graph basics

The purpose of PlayMotion's filter graph is to transform the source images into a signal that is meaningful to your program and/or PlayMotion's computer vision services. If you've used Photoshop, then you probably know a bit about image filtering. Filters are software processes that are applied to each pixel of an image to create a new image. Often, the resultant image bears strong resemblance to the original, merely extracting features or slicing the image into discrete segments. Other filters perform complex mathematical operations on the image, providing output that bears little or no resemblance to the original from a human's perspective.

The image processing filters can be thought of as little factories. Each factory takes a number of images (possibly zero) and some parameters as input and produce a number of images as output. Consider this cartoon:

A conveyor belt brings images (black fields with the letter P in white) in from the left. The filter inverts the color palette, producing black Ps on white fields rolling out on a conveyor belt to the right. These factories can be linked together, with the product one factory feeding directly into another as materials. To accomplish this, the first filter in the chain is declared to be the "source" of the next filter. In the next cartoon

a factory creates white Ps on black fields out of thin air (notice that there is no input image stream). These images roll off the assembly line and move straight to the input of the "invert" filter described above. It is easy to imagine very long lines of factories moving images around to produce a final result.

Some filters take multiple images as input. This is analogous to factories needing more than one kind of good to build a product. Similarly, some filters have multiple images as output. We can extend the metaphor here to to factories that create multiple products simultaneously.

Segmentation Methods

Image segmentation is any process whereby an image is divided into parts (or segments). These parts need not be polygonal (arbitrary shapes are possible) or connected (two or more different parts of an image can be part of a single segment). Computer vision makes heavy use of image segmentation to add extra information to images, such as identifying body parts, grouping pixels of similar color, etc.

The segment classes identified by PlayMotion's segmentation methods are only two. Hence these algorithms effectively transforms an image from any arbitrary color depth into 1-bit color depth. Typically, we consider the zero ("false") values to be "non-input" or regions not explicitly influencing the system, while the non-zero ("true") values are the "interesting", user-generated pixels which we'll mostly focus on.