• Zack Lee

Week3 - Project Ideas in 3 scales

Updated: Feb 19, 2020

This week, I was asked to blog post the project ideas in 3 scales.

Although my initial idea was to build a hardware musical instrument just like my past project Beste which I mentioned in my previous post, I leaned towards using another medium because its limitations seemed to outweigh the advantages.


- Tangible interface


- Inflexible interface

- Hard to rebuild and distribute to the public

- Laborious fabrication

- High cost

Therefore, I spend many hours researching on finding a better medium to use for creating a more intuitive and accessible music creation environment.

First Idea: A projected AR MIDI controller

Projected AR is a technology that directly overlays digitals projections onto the physical world. Unlike smartphones or wearable AR, projected AR does not typically require a device to mediate and project imagery. This creates the possibility of shared AR experiences and mixed reality experiences that integrate tightly with the environments in which they are installed.

Image from CMU, Future Interfaces Group: Desktopography


- Flexible interface

- Shared AR experience

- No need to wear AR device

- Can use both hands


- Flat touch interface

- Requires projector and camera

- Environment-dependent (Not portable)

- Hard to distribute to the public

While I do like the idea of using a table as a touch interface, I didn't like it not being really accessible for most people in real life because it would require the installation of a projector and camera. And if having the big flexible interface is so important, there is an even cheaper and easier option like the following 24-inch multi-touch screen monitor from Dell.

Dell 24 Touch Monitor: P2418HT

Second Idea: An AR app for headgears


- Flexible interface

- Portable

- Distributable to the public (Although limited)

- Can use both hands


- Flat or air touch interface

- Requires wearing a headgear

- Hard to share the experience with others

- Expensive

While I personally believe a wearable AR app to be close to an ideal music creation environment, I find the technology is not there yet for most people considering the headgears' bulky size and expensive price. Since my goal is to create a more accessible music creation environment, I want to create something that can be used by many people at the moment.

Third Idea: An AR app for smartphones


- Flexible interface

- Portable

- Distributable to public

- Can offer shared AR experience

- Inexpensive


- Flat or air touch interface

- Cannot use both hands to use the AR interface

I find this idea the most accessible approach among the 3 ideas since most people carry a smartphone these days. The only problem compared to the other 2 ideas was not being able to use both hands to interact with the AR interface since people would use one hand to hold their phone to view the AR world. To cope with this issue, my plan is to create a simple AR musical instrument that can be played using one hand and the other hand that holds the phone can also be used to press the button on the phone screen to record or clear the playing of the musical instrument. (Similar to how my mobile music creation app Azurite works)

Also, If multiple people use the app in the same space, they will see and hear each other's instrument playing so they can jam in the AR world. I was inspired by an AR drawing app called Just a Line by Google which allows multiple people to draw lines and share the experience in the same place. I basically want to create something similar to this app but instead of drawing lines, people would play musical instruments and jam with others in real-time to create music together and have a fun experience.

The milestones

  • Feb 19 - Feb 25: Research and try out development tools to find out which one to use. (Unity, openFrameworks, WebAR, ARCore, ARKit, C++, Python, Lua, Swift, Java)

  • Feb 26 - Mar 3: Detect a flat surface and place an AR object on it.

  • Mar 4 - Mar 10: Detect touching an AR object with fingertips and get the touch positions.

  • Mar 11 - Mar 17: Design and build a simple AR musical instrument that can be played.

  • Mar 18 - Mar 24: Fix bugs and improve the overall quality. (UI/UX/Sound/Animation)

  • Mar 25 - Mar 1: Let people try the app and collect feedback. (What can be improved?)

  • Apr 1 - Apr 7: Add recording/clearing the instrument play feature. (UI/UX)

  • Apr 8 - Apr 14: Continue implementing the feature.

  • Apr 15 - Apr 21: Add a multiple-user sharing feature. (UI/UX)

  • Apr 22 - Apr 28: Continue implementing the feature.

  • Apr 29 - May 5: Fix bugs and improve the overall quality. (UI/UX/Sound/Animation)

  • May 6: Final Presentation.

26 views0 comments

Recent Posts

See All

Software Engineer | UX Designer | Creative Technologist