Sunday, January 7, 2018

Augmented Reality Vuforia Demo for Android & Unity


Demonstration of Augmented Reality using Vuforia libraries, on 2 separate builds for Android (Android Studio , Unity 2017.2)

3D AR object tracking using ImageTarget & Virtual Buttons

Unity native AR integration requires Android v7 / iOS ARKit,
however adding Vuforia AR libraries allows backlevel Android support (back to v4.1 as of Unity 2017.2)

Please see the transcript for links & background info.

Transcript + timecodes

Hi, This is Morte from PCAndVR,
with a video demonstrating Augmented Reality using Vuforia libraries, on builds for Android (using Android Studio & Unity 2017.2)

If you're not familiar with Augmented Reality (AR), it means using a device to synchronise virtual objects into the sensory field (e.g. visual/auditory) through which we experience the real world.
A key feature of AR is locking a virtual reference frame into a live camera image of either a specific image target, a 3D object, or a surface (ground plane).  This reference frame becomes a stable centre point for generated 3D objects to exist such that they appear synchronised with the real world in the device's camera image.
Because an AR object or image target can be tracked in 3D space, it actually provides a form of head tracking for Mixed Reality (XR) if you use a mobile Virtual Reality (VR) head mount with a camera opening such as Samsung's Gear VR or custom Google Cardboard headsets.

Without head tracking, VR headsets can only provide 6DoF (degrees of freedom) rotation from a single point, and no translational movement in virtual space, adding to motion sickness.
AR devices can be something as simple as a recent model mobile phone or tablet with camera.
Head mounted AR devices can be more useful for hands-free operation and keeping virtual objects in the field of view, but typically there are weight and portability issues with tethered head mount devices, and image quality limitations on current untethered glass-frame devices.

This video will demo 3D AR object tracking using ImageTarget & Virtual Buttons, as they are reliable and simple to implement.
First up, we have the Android Studio built app, which largely contains the same feature set as the Unity app, albeit using simpler models and older image targets.
The 1st Android Studio example uses the stones image target.  The added 3D object is a teapot, originating from the famous 1975 Utah teapot 3D model.

Using the app we can show the teapot from various distances, angles, and even zoom in on the inside of the spout, with the teapot visually appearing fairly stable against the stones image target.

The image targets are printed on A4 paper from files in the supplied library, and are designed to be high contrast, with numerous well defined edges and non-repeating features.  This allows detection and tracking even if only a portion of the image is visible in the camera field.

The 2nd Android Studio example uses the wood background image target, with four virtual buttons.   Each button gets mapped with a virtual rectangle surrounding it to detect occlusion by real world objects, effectively meaning the button has been pressed.  In this case, it changes the colour of the teapot to match the colour of the virtual button covered by my fingers: red, blue, yellow, and green.

Next, we move to the Unity build of the same type of example features, with image targets and non-interactive 3D objects, and virtual buttons to allow interactive 3D object behaviour using the same image targets.

Here we have a standing astronaut, which can jitter if the AR device is not held steady, an effect worsened by the model's height, and relatively small target image size
next is an oxygen cylinder which is significantly more stable due to the 3D model being smaller and positioned closer to the target surface,
and an animated drone, which while small, can jitter in the hovering position above the target surface.

I coded a custom 3D image of my own showing an empty gamelevel platform, using the stones image target and downloaded .xml file, but I could have used any suitable custom image with the Vuforia web xml generator.

Finally, the Unity build's virtual buttons feature shows the astronaut waving after a button press,
the oxygen cylinder displaying an infographic dropdown visual,
the drone emitting a blue projection field.
and the fissure changing from white steam to dark red.

In concluding, we saw some limitations of smaller, less complex image targets in terms of feature detection & target tracking,
with resultant visual instability of mapped 3D objects, worsening further above the surface of the target,
and objects disappearing at low incident angles to the target, where feature detection is not possible.  
Both these limitations can be mitigated somewhat through use of 3D object targets or larger complex multiple image targets on 3D objects,
since the detection should still clearly see features of one target even if those of another become occluded or at a low angle.

For further background info & links, please see the transcript for this video.

And that's it, so thanks for watching!

Background info

If you are thinking of doing your own Vuforia AR builds for Android,
The 1st demo uses an app built on Android Studio using the Vuforia Android SDK:

A number of books & online tutorials exist, but can be confusing due to the deprecated legacy Android Vuforia library API which used QualComm. & QCAR. naming.
Now Vuforia library naming is standalone, and Android native builds require Android Studio, replacing Eclipse.
However it still required significant manual tweaking and customisation for me to convert deprecated code from legacy Eclipse format into Gradle-based Android Studio framework.
It is really not worth pursuing Vuforia libraries for Android Studio since the Vuforia Unity SDK makes the entire process simple and modularised, utilising drag n drop gameobjects.

It is already far easier to go straight to Unity for both Augmented, Virtual, & mixed reality  (XR) features.
Unity native AR integration requires Android v7 / iOS ARKit,
however adding Vuforia AR library support allows backlevel Android support (back to v4.1 as of Unity 2017.2)
Don't bother with the downloadable legacy Vuforia library .unitypackage for Unity 5.x.x or earlier, there are too many deprecations to contend with.

Aside from using the image targets from the supplied library, creating your own image targets for use in an app consists of uploading an image to your Vuforia web account, and processing them for the appropriate platform. 
Internally, Vuforia converts images into feature points, downloaded from the site as an xml file used in the app as a form of 2D vector UV mapping. 
These feature points are used for tracking in conjunction with the Vuforia image detection & processing algorithms, and any chosen 3D objects can be overlaid onscreen, locked to the detected target so that movement of the device's camera shows the chosen 3D objects from the appropriate angle and distance, matching the real world target.

This is a general introduction to Vuforia:

This link helps you get started with Vuforia in Unity:

Some tips & tricks for Vuforia on Unity:

Vuforia's legal doco allows for free use of a subset of features during app development:

Vuforia requires setting up a Licence for each Unity project/application you develop

If you use the supplied example code from the .unitypackage in Asset Store (for Unity 2017.2 or later), be sure to use the link here in the notes to understand how to successfully apply custom image targets without them being overriden by the sample code:

Google Daydream View VR requires specific compatible Android v7 phones such as Google's Pixel 2 or Samsung Note 8/Galaxy S8, but reference headsets do not allow rear camera visibility, so no AR or XR on those.

Sunday, November 12, 2017

Marblesared on Google Play (finally)

Well it took a while but finally it is there for actual sale in the Google Play store, with a free demo, and a separate closed beta release for all you beta testers out there.


It took this long partly due to multi-factored complications with bundle package naming restrictions.
 - The original beta was called com.pcandvr.marblesared.
 - My automated bundle naming code needed to distinguish alpha/beta/production bundles when publishing, so I couldn't leave that name as is for beta, & couldn't customise the naming just for production due to other G.P. restrictions.
 - Google Play does not allow changing a 'free' app to a 'paid' app, so I couldn't make com.pcandvr.marblesared the prod paid bundle
 - Google Play lets you 'unpublish' an app bundle so that it no longer appears in Google Play to users as something they can download, but it does not let you delete the app bundle.  Ever.  Users who do not uninstall the app are free to continue to use that app on their devices.
- Google Play does not let you re-use old app bundle names for new apps, even after you unpublish the old app.  That is understandable - users should never face the issue of an update to an old app resulting in a different new app on their device.

So, ultimately, I ended up with three new app bundle names to meet a repeatable naming scheme:
com.pcandvr.marblesared_beta       (free)
com.pcandvr.marblesared_demo     (free)
com.pcandvr.marblesared_prod      (paid)

Some annoying news for beta testers - you'll need to uninstall your current Marblesared app, and reinstall the new beta here:

I looked briefly again at Apple iTunes/App Store & Microsoft UWP
Both Apple & Microsoft Developer programs require payment of 100USD (150AUD) every year.
Steam was a 100USD fee per title, but it was a once-off cost per title.

With current lack of sales of Marblesared, across, Steam, & Patreon,
coupled with the recent news of Microsoft dropping Windows Phone as a delivery platform,
balanced against significant losses already for Steam & PCAndVR business set-up,
those are additional costs I simply can't justify just at the moment - I will reconsider when I have more titles to publish, or a separate income stream to sustain that additional cost factor.

Wednesday, November 8, 2017

Simple, low cost cooling solution for Samsung Gear VR S6

After more than a year of my Samsung Galaxy S6 phone overheating after only about 10 minutes of play in my Innovator Edition Gear VR (SM-R321) headset, regardless of setting flight mode, wireless on/off, screen brightness, power saving, do-not-disturb mode, per these tips

I was losing enthusiasm and running out of $$$ to waste on the followup S7 & S8 and newer Gear VR headset.
I did get the latest Gear VR (SM-R324) headset + Touch Controller bundle for a reasonable price on ebay,
but I was still left with the problem of overheating, which apparently is especially bad on the Galaxy S6.
I was also quite dispirited in the wallet area upon learning that Google Daydream View would require a Google Pixel 2 or equivalent latest OLED phone (like the Samsung Galaxy S8).

I have read about numerous solutions specific to certain Gear VR models, and/or phone models, including clip-on fan mounts, heatsinks which required pulling the back cover and battery out of the phone, custom drilled or 3d-printed fan mounts and crazy looking aluminium foil heatsink rigs, and highly risky frozen coolant gel packs strapped to the phone (these people don't understand the concept of condensation occurring even outside a sealed cold gel pack, simply due to ambient humidity).

I decided the best & safest route was a very simple air-cooling solution with a fan small enough to fit just inside the gear vr faceplate, but not too small that airflow would be insignificant & ineffective. 
Being powered by the same usb connector as the gear VR itself would be nice but I decided against attempting to pull apart the headset for fear of irreversible damage.

Instead, I used three cheap items to make this a battery pack free-swivel-around-play-till-you-drop solution,
which can also be a mains-powered play-without-moving-too-much solution.
I am most impressed with the fact not only does the new Gear VR Touch Controller work with the Galaxy S6,
there is also no hardware dependency on the Gear VR headset - it works just fine on my old Innovator Edition SM-R321 model.  

So, on top of the Samsung Gear VR + Galaxy S6 + Touch Controller + micro USB charger cable

I got the following items:
- standard USB-A male to female 1.8M cable
- 60x60X10mm 5V USB Cooler Fan
- dual port 1A + 2.1A power pack

For the mains powered option, I got a dual-port USB mains adapter with at least 2A output on each port.
The official Samsung USB 5V @ 2A mains adapter seems to be the only thing which actually puts the S6 into 'fast charging' mode.

I wanted this solution to be VERY simple & easy to mount/dismount the phone from the Gear VR headset, and to plug/unplug the power source.
I also wanted the power source to be mounted on the headset somehow but the lightest power packs were too low on the mAh rating, and the best power-per-buck  vs low weight tradeoff got me a 6600mAh pack which even at 216g was still a bit too heavy to consider strapping to the actual headset, especially if I wanted to avoid neck strain and/or lie back on a pillow or lounge.
Since the overheating on the Galaxy S6 occurs primarily just under the camera lens, I positioned the fan so it would hang down just below the lens protrusion with the S6 mounted, and swing back up over to rest on the top of the Gear VR when unmounting the S6.

I achieved this of course with the amazingly high tech adhesive technology of ..... (wait for it)

sticky tape.

And here's the result, giving endless VR playtime, no overheating, cool-to-the-touch even after one hr solid play (eyestrain prevented further testing).

You can see in the side-on shots, the spinning fan pushes itself away from the phone face, which is fine, since that allows the airflow to disperse across the phone backplate for more effective thermal displacement.

I will definitely be able to enjoy VR again now - although Gear VR does have that unfortunate degree-of-freedom limitation where physical motion-in-space of the headset and controller are not represented, as they are with both the Oculus Rift VR headset and new touch controllers... 
Also the CPU/GPU limit on phones doesn't allow the same level of detail you can generate with nVidia GTX high end cards.  
I only have a GTX 970 & Intel 3770K and they are perfectly adequate for Oculus Rift VR.

Tuesday, October 31, 2017

Marblesared Build 2017/10/31 - All Hallow's Eve edition

Marblesared Splash image blog link

Hey Players,
Here is the All-Hallows Eve edition 31/10/2017 build of "Marblesared" (Marbles Are Red) for PC and Android.

I can give the link or Steam Key for the full game to beta testers or reviewers on request,
or to Patreon Patrons.
Otherwise it is available on Steam,, Facebook Gameroom for USD2.99.
Free Demo version with 6x4 levels x skills rather than 33x4 also available.

Link to Marbleared blog page

Thursday, October 5, 2017

Marblesared Early Release on Steam for Windows - 20171005

Marblesared Splash image blog link

Hey Players,
Marblesared has been [Early] Released on Steam today for the Windows PC platform!
Marblesared Steam

No game updates per se,
Steam's early release is the same as the 20170920 version I announced ...  on 20th Sep :-)

If anyone wants a Steam key for the PC version, lemme know, you deserve it for being beta testers.

You don't need a Steam key for the Android platform, it is still sitting in free beta mode for now in Google Play.

One day soon I hope to have a MacOSX build (which I can add to Steam) & iOS build
- but with OSX 10.13 High Sierra released on 25th Sep & 10.13.1 beta bug fixes already out to devs,
and iOS11 upgrades being pushed out to iDevices as we speak, with numerous battery power drain issues noted,
the question of build stability is up in the air.
Thanks to Microsoft libraries for IL2CPP & UWP SDK not liking Unity3D at present,

Microsoft Win10 / Universal Windows Platform (UWP) builds & MS App Store remain a pipe dream, even tho normal windows 32bit builds work just fine on windows 10.

You've Played Hard - you can give Rolling Balls a well earned rest for now :-)


Tuesday, September 19, 2017

Marblesared Game Trailer on Youtube

Totally forgot I had only tweeted/FB posted this Youtube link to my Marblesared Game Trailer
without posting here!

This is an early draft of a game trailer I am planning for Steam Direct Early Release.

Many thanks to the brilliance of Edvard Greig for his 1875 work "I Dovregubbens hall" (In the Hall of the Mountain King) as performed by the Czech National Symphony Orchestra, commissioned by Musopen as public domain, found on

Marblesared Build 2017/09/20 - Pre-Steam Early Release edition

Marblesared (Marbles Are Red) Marblesared Splash image blog link

Hey Players,
Here is the Pre-Steam Early Release edition 20/09/2017 build of "Marblesared" (Marbles Are Red) for PC and Android.

I can give out the link to the full game to beta testers or reviewers on request,
but once it goes into Steam Early Release, the only free option will be the Demo version with 6x4 levels x skills rather than 33x4

Thanks & praise be to the beta testers yet again, for discovering numerous bugs in Tutoriballs,
caused by the NPC bean AI navigation now being too 'smart' to follow instructions.

What is "Marblesared" ?
It is a new take on the old hand-held "marble-in-a maze" type games commonly available many decades before electronic hand held games existed,
where you had to get a marble from one corner of the maze to the other, or variations on the theme.

Marblesared is designed to make the most of Android device built-in accelerometers,
enabling you to play simply by tilting your Android mobile device as if it were a physical container holding a marble.
It can also be played on PC with keyboard (or gamepad) controlling the marble.

How to play ?
- Roll your red marble to the red goal, through mazes, obstacles & bonuses.
- Help a little yellow bean get to the yellow goal.
The yellow bean can auto-navigate, and even has a jetpack for speeding to the yellow goal after your red marble unlocks it.
Only your marble can help the bean reach that yellow goal.
How? By lowering obstacles, by NOT tripping bad items, by NOT directly running over the bean and Smooshing it,
and ideally, by doing it in as little time as possible if you are competitive and want to brag a bit :-)

The target audience is early teens and older, due to the bean character being "smooshed",
and humour content (very bad puns),
There is no elite skill barrier - anyone capable of using a PC or Android tablet can play - albeit some better than others :-)

Current Features:
- Animated intro/splash screen with sound effects.
- Help/Instructions (Please read first as introduction on How to Play).
- Tutoriballs (Animated Tutorials) - preview on by default, appearing as a zoomable PIP inset image
    synced to 1st appearance of each new bad/bonus item so players learn how to use them
- 33 hand-designed levels for each of 4 skills for Play testing (only 6 levels on Demo):
        - "BabyBall" skill (No Bonuses or Bad Items),
        - "Normball" skill (Bonus Items - e.g. Actuator/Triggers with Drop-down walls)
        - "Hardball" skill (Bonus/Bad Items - e.g. lava pit, spiky balls, bomb, smartmine)
        - "Maballs!" skill (Bonus/Bad Items AND electrified goal for yellow bean requiring player goal entry to disable)
- Bonus items (Green), include: Actuator Pad, Wall, Bean Shield, Extra Time, Bean Powerup, PortalEntry, PortalExit,
- Bad items (Red), include: Actuator Pad, Wall, Compactor, Slicer, Spike Plate, Lava Pit, Weight , Bomb, SmartMine
- Bad/Bonus actuators exist primarily to activate the other bad/bonus items.
- Reset Actuator (Purple) clear Bad Items in Hardball, MaBalls! skill
- Player setup/name/skill/Score reset for New games (New Player games start by default at Level 1 on BabyBall skill)
- Continued games option to start at any level up to highest played for current player
- Score/countdown per level & local saved high score table for each player & top 10 player scores
- Auto Next Level countdown on each level completion.
- Game settings including Debug on/off, Sound on/off, Controller input selection for both PC & Android using inbuilt default/keyboard/accelerometer/gamepad
- SoundFX setting enable/disable
- Additional "Settings" for improved graphics quality (shaders, reflections, transparency) - use Vertex Material & disable texture for fastest performance

Next on Todo list:
- Steam Early Release
- Adding more levels manually
- Marketing/Promotion
- option to upload player total scores to internet with top 1000 player scores worldwide
- Algorithm for auto-generation of levels,
- Make it a bit more "pretty", add in music, etc...

The [expanded] app size is currently about 40Mb for Android, 30Mb for PC version, 
Download size is 23Mb for the Android .APK, 11Mb for the PC version executable installer ).

Link to my "PCAndVR" Google Drive shared directory for Windows PC download :
MortePCAndVR Google Drive shared directory "Marblesared"
The shared directory contains self-extracting Setup executable for PC, Release Notes,
and an Install-README text file with more detail on PC & Android version install/uninstall.
The PC version is a generic Windows 32bit .exe so it works on WinXP through to Windows 10 (AMD/Intel x86/x64 only)
No Windows 10 "Universal" build for ARM/Atom yet - coming soon as I get Win10+Universal SDK.

Once the game goes into early release for sale,
The link for PC will only be available to patrons & registered beta testers (see patreon link below).
Android users who install now will get a free ride on beta updates
until the game is converted into Google Play Store release mode :D

If your Android alpha version is not auto-updating past the old v20170511 or v20170512 version to the new Beta build,
try uninstalling, then reinstalling from the Google Play Store link.
If you are getting game hangs, clear the Marblesared application data/cache in Android application manager
If that doesn't work, drop me a line on one of my new social media channels.

Play Hard & Roll Ball!

Link & QR code for Google Play Store: