CMU Sensor Frame

The Sensor Frame's most ingenious technical choice was using Micron IS32 OpticRAM 64K DRAM chips as image sensors. These were standard memory chips in ceramic DIP packages with glass windows — designed for UV-EPROM-style applications, not imaging. McAvinney realized their photosensitivity meant they could serve as crude 2D cameras. Each cell was written to '1', light exposure caused charged cells to leak to '0' at different rates depending on illumination, and the pattern was read back as a shadow image. Four such sensors in the frame's corners tracked finger silhouettes, with angle-side-angle trigonometry computing precise X,Y positions.

Canonical pinch-to-zoom — two fingers moving apart to scale an object — was demonstrated on the Sensor Frame with coordinated graphics in 1985. Patent drawings show two fingers scaling the frequency of a displayed waveform (Figs 15a, 15b of US Patent 4,746,770). Other demonstrated gestures included two-finger knob rotation, marquee-style object selection, and amplitude scaling. The gestures were recognized in software running on a host computer at approximately 30Hz refresh rate.

In October 1985, Steve Jobs visited CMU — months after being ousted from Apple, in the process of founding NeXT. According to CMU's The Link magazine (Summer 2017), Jobs signed a non-disclosure agreement before being allowed to tour the Sensor Frame lab and see the multi-touch technology in action. The visit is cited as evidence that Apple had direct, documented exposure to advanced multi-touch at CMU nearly 22 years before the iPhone's launch.

With NASA SBIR Phase II funding (contract NAS9-18686, circa 1991-92), McAvinney developed the Sensor Cube — a 3D volumetric variant that could detect finger approach angle in three dimensions. Each finger became a virtual joystick with 3D control at the point of contact. NASA was interested for spacecraft crew interfaces and telerobotics control where physical buttons were impractical.