Implementing Pure Data in Unity for iOS using Heavy

The smartphone has been around for less than ten years, yet by August 2015, two thirds of the UK population were using one on a daily basis [1]. The smartphone has revolutionised how people process and receive information and international usage produces sensational statistics such as, some 60,000 photos are shared globally every second, amounting 5 billion photos a day [2], gobbling up storage space.

Across internet access, emails, music streaming, social media, storage of precious memories in the form of photos and videos, playing games and even making telephone calls, user requirements of the smartphone are multiple and varied, blurring the boundary between work and play and creating a huge supply and demand for better user experience.

As an interdisciplinary sound designer, interested in the use of smartphone technology for game driven dynamic audio experiences, I am aware that the smartphone has yet to realise it’s full potential in its uptake of converging technologies. In my case, I first started using Unity around two years ago. I love how easy it is to build for multiple platforms and the potential it provides for rapid prototyping. It’s the same feeling I got when I was first introduced to Pure Data: whose visual approach allows for quick prototyping of creative dynamic audio. Having devoured Peter Brinkmann’s book, Making Musical Apps, this led me to the use of LibPd with Xcode.

While the processing power of mobile technologies is ever improving, the problem of device storage capacity still raises problems. Often, when when exhibiting my mobile game Hedra, when people go to download it, they discover they have no space left available on their device (despite the game’s meagre 30mb size). As a developer, I am keen to keep the size of my apps to a minimum and this leads to my interest in Pure Data.

When there are already excellent audio tools for game audio such as Wwise or FMOD, why should I want to go using Pure Data? Normally when setting out to achieve a sense of variation and dynamism using sampled audio, large amounts of sound files must be packaged up with the game, thereby increasing its size. This is where Pure Data shines, allowing for the procedural generation of sound without the demand for storage. In addition, there is no graphical interface which could shape the user’s choice of interaction design, creating the possibility for much more unique potential sound design. For some great examples of procedural audio created using Pure Data Andy Farnell’s work serves as a great starting point.

I came across LibPd4unity which allowed for the fairly straightforward integration of Pure Data in Unity games for OS X and Android, and it worked quite nicely but alas, it did not work on iOS. So I was off on the search for something that would do the job of integrating Pure Data in Unity for iOS. The missing link : Heavy, a cloud-based service which compiles Pure Data patches to portable C code, javascript and a range of binaries.

In this tutorial I aim to show how to set up a simple oscillator in Pure Data which can be controlled from an iOS app built in Unity using Enzien Audio’s Heavy. Heavy is an online tool which generates high-performance code from Pure Data patches.

Building a Pure Data patch and using the Heavy compiler

1. Build a patch! Take special care to only use objects supported by Heavy. Their list is still growing so for the most up to date object support refer to the list of supported objects
2. To expose receive objects within Unity they must be annotated with @hv_param. Internal receives that don’t require controlling from Unity will function as normal without this extra annotation
3. Specify the receive’s range, within the receive object, as follows:
   

   [r frequency @hv_param 200 2000 400]

   where 200 is the minimum, 2000 is the maximum and 400 is the default initialisation value

4. Using Unity’s AudioMixer to host the generated plugin requires that an adc~ object be included in the patch even if there is no audio input from Unity. Here is an example of a patch set up to work with this method

Compiling using Heavy

1. Go to Enzien Audio’s website, register if you haven’t already done so.
2. Go to “My Patches”
3. Enter a name for your patch and click + New Patch
4. Choose the Pure Data patch that you wish to compile, click Compile.
5. You should now see a list of generated files ready to download you can also test functionality using the webplayer, you will be using the Unity OS X and source files

Building a Static Library in Xcode.

In this step you will compile a static library to allow the audio plugin to work on iOS.

1. Download the generated Unity OS X files and unzip
2. Download the Unity Source files and unzip
3. Open the plugin Xcode project located in the Xcode folder within the heavy source files you just dowloaded
4. Go to Editor > Add Target
5. Select iOS > Framework & Library > Cocoa Touch Static Library
6. Name the product
7. In the project view set Build Settings > Architectures > Build Active Architecture Only to NO
8. Set Valid Architectures to arm64 armv7 armv7s armv6
9. Set the Active Scheme to the static library just created and the destination to Generic iOS Device
10. Press the play button to build the library
11. Find the generated files in Library > Developer > Xcode > DerivedData > Hv_Simplesynth_plugin-(A random bunch of letters) > Build > Products > Debug-iphoneos 
12. Copy the .a file and the .h file from within include > Hv_Simplesynth_plugin into Assets > Plugins > iOS once you have setup your Unity project.

Setting up Unity

1. Head over to Unity, you will need Unity 5 as this method depends on its Audio Mixer feature
2. Create a new project
3. Create a new folder called Plugins in the Assets folder and within it create a new folder called iOS
4. Copy the .h and .a files generated by XCode earlier into the new iOS folder
5. Head back to the original source files downloaded from Heavy go to the folder named source and drag its contents into the iOS folder in Unity (Assets > Plugins > iOS)
6. In the Unity OS X files downloaded from Heavy copy  AudioPlugin_Hv_(X).bundle to the Assets folder 
7. Open the AudioMixer Window by going to Window > AudioMixer
8. Add a new Mixer, (it can be named anything)
9. Go to the Master channel and click Add, select your Audio Plugin from the bottom of the list, it should now appear in the inspector window where you can alter the parameters you have exposed
10. Create a new game object and add an Audio Source component
11. On the Audio Source set the Output to the mixer channel that’s just been set up, it will be listed under the name of your mixer as “Master”

Expose the plugin parameters of your Heavy plugin within the newly created Audio Mixer to allow for runtime manipulation. Unity have made a handy video explaining how this is done:

Building the app, some tweaks and running in Xcode.

1. Build your project for iOS, generating an XCode project
2. Before running the project a few changes must be made in order to link the AudioPlugins with the generated project
3. Open the generated Xcode project
4. Navigate to Unity-iPhone > Classes and import “AudioPluginInterface.h” to UnityAppController.h
5. To the function preStartUnity{} in UnityAppController.mm add the line:
   

   UnityRegisterAudioPlugin(&UnityGetAudioEffectDefinitions);

    

6. Run the app on your device

Huzzah!

I would like to thank Martin and Joe from Enzienaudio for their help and advice on a couple of details that helped find some missing links. I would also like to acknowledge the work detailed on this thread on the Unity forum.

Sources

[1] http://www.deloitte.co.uk/mobileuk/assets/pdf/Deloitte-Mobile-Consumer-2015.pdf

[2] http://www2.deloitte.com/uk/en/pages/press-releases/articles/3-point-5-million-photos-shared-every-minute.html

[3] http://forum.unity3d.com/threads/native-audio-plugin-bundle-import-settings-for-ios-build.315590/

 

 

Advertisement

Implementing libpd4unity in Unity 5

Following on from a tutorial I ran for the Interactive Sound Environments students at the University of Edinburgh, in March 2016, I thought it may be helpful to centralise my notes about how to use libpd4unity in your own projects.

Libpd4unity is a way of embedding Pure Data within your games allowing you to design a highly flexible and dynamic sound engine of your own.

While previously it was only possible to run libpd4unity in Unity Pro due to restrictions on the use of plugins, Unity 5 changed this, making the whole process much more accessible.

Implementing libpd in Unity

1. Download the libpd4unity files from github
2. Create a new project or open an existing one
3. Open the libpd4unity folder you downloaded
4. Copy LibPdFilterRead.cs, PluginUtils.cs, Plugins (folder), LibPD (folder) and Streaming Assets (folder) into your Assets folder of the Unity project.
5. Copy your Pd patch and any abstractions into StreamingAssets > PdAssets
6. If there isn’t already an AudioListener component on any of the game objects e.g. Main Camera, then create an empty game object in your scene, name it LibPd for easy reference.
7. Add an AudioListener component (if there’s not already one), this will act at the dac~ object in Pd. Any spatialisation must be manually programmed, so the position of the Audio Listener has no effect on your Pd sounds unlike when using Unity’s inbuilt sound source components or middleware such as FMOD or Wwise.

   

   Adding an AudioListener in the inspector view

8. Drag the LibPdFilterRead.cs script onto the game object where the audio listener component is placed (e.g the LibPD or Main Camera game object). This is where the patch is loaded from. Be careful to only include one instance of LibPdFilterRead.cs in your scene.
9. Write the name of the Pd patch you copied into the Streaming Assets folder in the “Name of Patch” field in inspector. Note: This is case sensitive so be sure to copy the name exactly, including the .pd extension.

Adding the LibPdFilterRead script and naming the patch

If your Pd patch requires no control to trigger sounds, when you press play you should now hear some sound!

Communicating with Pure Data from Unity.

Pure Data patches really come to life through user interaction, communicating with your patch is straight forward.

At the top of the C# script from which you would  like to send a value to your Pd patch, add in the following: 

using LibPDBinding;

If you wish to send a float to Pd use:

LibPD.SendFloat(“yourPdReceiveName”, yourValue);

If you wish to send a bang to Pd use:

LibPD.SendBang(“yourPdReceiveName”);

A quick example:

Say you want to control the frequency of the oscillator in this patch:

Create a new C# script called “oscControl” and place it on the LibPD object 

Creating a new C# script

Right click on the script and choose Edit Script, this will open MonoDevelop

Replace the default code with:

using UnityEngine;
using System.Collections;
using LibPDBinding;
public class oscControl : MonoBehaviour {

public float frequency;

void Update () {

LibPD.SendFloat ("frequency", frequency);

}

}

In the inspector you should now be able to alter the frequency of the oscillator at runtime (when you press Play).

Changing the frequency of the oscillator from the inspector window.

General Tips

If you find your project crashing, it could be because some Pd objects aren’t supported, therefore to avoid problems, test your patch regularly once integrated in Unity and stick mostly to pd-vanilla objects.

Don’t use [print] objects in your Pd patches.

The above method will work for OS X standalone builds and Android (as long as you don’t use any abstractions). It won’t work for iOS … but more on this coming soon!

Credits

https://github.com/patricksebastien/libpd4unity

http://www.thefuntastic.com/2014/04/the-misadventures-of-unity-and-puredata-libpd/

http://twobigears.com/labs/unity-and-libpd/