WorldSpace Rendering (and lack thereof)

UGUI provides users to put canvasses in WorldSpace RenderMode. This allows for 2D UserInterfaces to be rendered in 3D space. Common examples are markers in 3D space or maybe text balloons above 3D characters. In the UI Toolkit this is simply not supported. There is no way to render an UIDocument in 3D space.

This leads to us having a hybrid system where the 2D UI is rendered using the UI Toolkit, yet the 3D UI is rendered using UGUI. The main problem here is the inconsistency of the UI since we need to implement (some) things twice as we want one coherent UI in both 2D and 3D. An example of our usage of this hybrid system is shown in the image below:

There's a lot going on in this image. The UI Toolkit is used to render all the 2D UI like the bars at the top and bottom and the speech bubble. The 3D UI, however, is rendered by the old UGUI system. Fortunately, in this particular screenshot, there are no shared UI components between the 2D and 3D UI.

If we, for example, want a button with rounded edges (let's say 5 pixels) in both UI, we need to implement and maintain two different files. We also need to maintain two different theming systems. With UGUI we created our own system which relies on custom components (MonoBehaviours) that respond to events that are raised by some ThemeService. But with the UI toolkit we simply replace the theme in the PanelSettings and we are done. This is just one example of the cost of implementing new (UI) features and the maintenance thereof which is sluggish and not really cost-effective.

So... we wanted to solve this issue of WorldSpace rendering because it saves us not just a lot of time, but also the frustration of getting UI between UGUI and UI Toolkit implementation on a 1-to-1 match.

The next step is to be able to register input. What's the use of a UI without input anyway!? So in order the register actions like clicks or taps (for mobile) and drags there needs to be some kind of EventSystem. It took us quite a while to find an easy and performant solution for this since the registering of input in the UI Toolkit is encapsulated in lots of private/protected/internal interfaces and classes. We could of course access all of this using reflection but that would tank the performance big time, not to mention unsupported reflection magic with IL2CPP -- especially since Dephion will mainly use this in mobile apps. Thus, we needed to find a different solution.

To fix it we implemented a very simple RayCaster, which checks if the top object (with the least distance) has a WorldSpaceUI component. If so, we take the RaycastHit TextureCoord and we feed it into our event system. This way, we know exactly where the user clicked on the mesh, and based on this position we are able to determine IF the user actually clicked something of interest on the WorldSpaceUI. We then feed this information into the default UI Toolkit PanelEventHandler and voila, we have input! (We can't seem to find docs about the PanelEventHandler class)

if (Physics.RaycastNonAlloc(ray, _hits, _maxRaycastDistance) > 0)
            {
                ISelectable currentlySelected = null;
                foreach (var hit in _hits.Where(h => h.collider).OrderBy(h => h.distance))
                {
                    if (hit.collider.gameObject.TryGetComponent<WorldSpaceUIDocument>(out var doc))
                    {
                        currentlySelected = doc.PickTopElement(hit, out var hitPos);
                        currentHitPosition = hitPos;
                        if (currentlySelected != null)
                            break;
                    }
                }
             }

Now we finally have a working 3D UI document and are thus able to create one coherent UI across 2D and 3D space. Awesome, right!?

But, there's always a but... after some testing we quickly found out that, although it works, it does not support "clicking through" transparent areas. Since we RayCast onto a MeshCollider, we don't actually know what we clicked, we just feed that position into the UI Toolkit PanelEventHandler.

The image below showcases this exact problem (we enabled "Convex" mode on the MeshCollider of the UI to emphasize the colliders):

Fixing this was quite a mission. As mentioned before, we did not want to commit the crime of using run-time reflection to achieve our goals. With the current solution, we use the default PanelEventHandler to process the position of the input. But of course, the collider is not permitting the 2D Physics RayCaster to click through the collider in the front. And thus, we are unable to click the buttons in the UI that's rendered in the back.

However, to fix this we are required to reimplement or re-engineer most of the UI Toolkits PanelEventHandler and EventSystem to get functions like hover, clicks, drags and other controls. This is a disadvantage we are willing to take and invest time in.

The solution was found after long debugging sessions and decompiling the UI toolkit source code. An interface called "IPanel", which is implemented by the UIDocument, exposes a public function called "Pick". This function basically traverses the entire VisualTree, in a recursive fashion, and checks if some position is inside the bounding box of VisualElements.

We reimplemented this function, as a nice Extension Method to the IPanel interface to check whether we actually clicked anything on the canvas, if not we take the next RayCastHit that has a WorldSpaceUI document and repeat the action there. We do this until we hit something. (The method below, can most likely still be optimized!)

 /// <summary>
        /// recursive picking function
        /// </summary>
        /// <param name="root">root visual element</param>
        /// <param name="point">position</param>
        /// <param name="picked">list of picked elements</param>
        /// <returns>new root element</returns>
        private static VisualElement PerformPick(
            VisualElement root,
            Vector2 point,
            List<VisualElement> picked = null)
        {
            if (root.resolvedStyle.display == DisplayStyle.None || root.pickingMode == PickingMode.Ignore && root.hierarchy.childCount == 0)
                return (VisualElement)null;
            Vector2 local = root.WorldToLocal(point);
            bool flag = root.ContainsPoint(local);

            VisualElement visualElement1 = (VisualElement)null;
            for (int key = root.hierarchy.childCount - 1; key >= 0; --key)
            {
                VisualElement visualElement2 = PerformPick(root.hierarchy[key], point, picked);
                if (visualElement1 == null && visualElement2 != null)
                {
                    if (picked == null)
                        return visualElement2;
                    visualElement1 = visualElement2;
                }
            }

            if (((!root.visible ? 0 : (root.pickingMode == PickingMode.Position ? 1 : 0)) & (flag ? 1 : 0)) != 0)
            {
                picked?.Add(root);
                if (visualElement1 == null)
                    visualElement1 = root;
            }

            return visualElement1;
        }