- #1
Killtech
- 344
- 35
- TL;DR Summary
- drafting a concept for a framework to describe the world as perceived through acoustic eye, and rendering it via a 3d engine
Disclaimer: This will be a lengthy post and it will take me a little time to outline how this is related to this forum, so please hang in with me here please – and for everyone that makes it to the end, thanks for taking the time.
So let’s start with the premise that we want to make a 3d game where the protagonist is blind and relies entirely on his sharp hearing sense for orientation, like Marvel’s Daredevil – or more realistically a bat. So the objective is to visualize/render the world as it is perceived from the protagonist’s perspective. In term of a game it would be a huge win to depict just like 80% accurately (the number is symbolic) but enough to highlight the real uniqueness of that view.
Now sonic waves have enough similarities to light waves to reuse an existing 3d engine to visualize that world. In particular one could theoretically employ acoustic lensing effect to build a sonic wave detector with some spatial resolution that works like an acoustic analog to an eye. Sure, it’s a stretch that a human ear would be capable of that but for the context of a game let’s go with the premise anyway.
Having in mind how a 3d rendering engines work it would require having all objects internally modeled as they appear acoustically and more importantly where they are positioned for the acoustic eye – because that’s what will be pipelined to the 3d engine to do the render. And here is the problem: since sonic waves have still quite different properties then light does, the positioning will be off in some cases. And this is also very much needed if a technology like ray tracing is to be employed – since those rays are then sonic in nature. But to do all calculation with real positions and then transforming everything to perceived positions every single frame would be very costly in term of performance.
Hence the idea to try explore the option to skip real positioning altogether and do everything within perceived position space… which comes with a long list of interesting challenges and implications.
So let’s start in clarifying what the perceived position actually is. So the rules of projective geometry apply to a acoustic eye in just the same way as they do for a optical one. Distance from the eye makes objects appear smaller the further away they are – but here it’s not the actual distance which is relevant for this effect but rather the travel time of the acoustic wave. Hence, the idea of this approach is to use this definition for the perceived distance. I will deliberately leave out a lot of details like if this should be defined as a one-way or two-way distance because in practice this gets a lot more complicated when the medium is not very uniform… and in context of a game we just want to focus on the cases where it remains simple enough to realize it (and in the other cases we might employ other tricks). Still, it’s furthermore quite important to build the entire framework well enough to even be able understand which interesting effects there are to visualize.
With that settled it’s funny to explore how different some things are perceived acoustically (and what should be covered by our game). For example, sound travels very slowly in comparison… so as with light very distant objects (like stars) are actually images from the past. But for a bat this creates a lot of funny effects in normal life situations already. Let’s assume for now the medium is perfectly at rest to our bat. Looking/listening at a distant incoming object it appears as it was in the past but as it closes in will be perceived closer to its actual present state. So what that means is that everything happening on that object will be perceived to happen in accelerated pace to catch up with the present as it approaches our bat. In particular observing an on-board clock it will look like it runs faster. The opposite will be true when the object passes by and moves away showing again a past picture of itself. Due to the definition of perceived distances it would also mean that distances on the object in the direction facing the observer will be squeezed for incoming objects and stretched for outgoing ones – while movement parallel to the observer will cause the object to appear sheared along the perceived depth axis (since the front (I.e. side facing the bat) of the object appears closer to its present state while its back is more in the past).
Taking a closer look at the role of the medium, things get a bit more exotic. For one gravity acting on it creates a gradient in air pressure and thus in the speed of sound. So this has the consequence that two wave fronts starting in parallel won’t necessarily maintain and can actually intersect after some time. Keeping in mind how perceived distances are defined this actually means that the parallel postulate isn’t satisfied in the perceived world – i.e. its geometry is inherently not Euclidean. This of course is a no go for any 3d engine which builds exclusively on Euclidean geometry. Therefore some trickery will be required to visualize any kind of effects resulting from an inhomogeneous medium.
For example a interesting aspect for visualization of the above mention perception may be that sound leaving earth’s surface at an angle will be deflected ever downwards due to the gradient in air pressure/speed of sound until it eventually turns around and meets the ground again. This means however that looking (hearing) upwards into the sky one should be able to perceive sounds from far away. So together this should create the perception that Earth surface is concave rather than convex, giving the bat an impression of living on the inside of a sphere – a bit like in a Dyson sphere.
I have talked with various mathematicians about this and the agreement was so far that differential geometry should be able provide the mathematical framework to describe the perceived world. This would be done by equipping the set of real world coordinates with this new perceived metric defining a new distance measure on the set to make it a proper manifold. Because the new metric should be still topologically equivalent (i.e. the identity function on the set is should be a non-isometric diffeomorphism) to the original real metric (in most scenarios relevant for the game) a transformation of physics into a formulation of how it’s perceived acoustically should be possible (meaning it should be potentially possible to calculate physics entirely in that new framework – by using said diffeomorphism to pullback physical laws). Problem is that no one so far could point me towards papers doing something quite like this. However, given that this would share a common mathematical calculus with general relativity and also aims to do physics in it, I thought it might be worthwhile to post this here for feedback and ideas.
So let’s start with the premise that we want to make a 3d game where the protagonist is blind and relies entirely on his sharp hearing sense for orientation, like Marvel’s Daredevil – or more realistically a bat. So the objective is to visualize/render the world as it is perceived from the protagonist’s perspective. In term of a game it would be a huge win to depict just like 80% accurately (the number is symbolic) but enough to highlight the real uniqueness of that view.
Now sonic waves have enough similarities to light waves to reuse an existing 3d engine to visualize that world. In particular one could theoretically employ acoustic lensing effect to build a sonic wave detector with some spatial resolution that works like an acoustic analog to an eye. Sure, it’s a stretch that a human ear would be capable of that but for the context of a game let’s go with the premise anyway.
Having in mind how a 3d rendering engines work it would require having all objects internally modeled as they appear acoustically and more importantly where they are positioned for the acoustic eye – because that’s what will be pipelined to the 3d engine to do the render. And here is the problem: since sonic waves have still quite different properties then light does, the positioning will be off in some cases. And this is also very much needed if a technology like ray tracing is to be employed – since those rays are then sonic in nature. But to do all calculation with real positions and then transforming everything to perceived positions every single frame would be very costly in term of performance.
Hence the idea to try explore the option to skip real positioning altogether and do everything within perceived position space… which comes with a long list of interesting challenges and implications.
So let’s start in clarifying what the perceived position actually is. So the rules of projective geometry apply to a acoustic eye in just the same way as they do for a optical one. Distance from the eye makes objects appear smaller the further away they are – but here it’s not the actual distance which is relevant for this effect but rather the travel time of the acoustic wave. Hence, the idea of this approach is to use this definition for the perceived distance. I will deliberately leave out a lot of details like if this should be defined as a one-way or two-way distance because in practice this gets a lot more complicated when the medium is not very uniform… and in context of a game we just want to focus on the cases where it remains simple enough to realize it (and in the other cases we might employ other tricks). Still, it’s furthermore quite important to build the entire framework well enough to even be able understand which interesting effects there are to visualize.
With that settled it’s funny to explore how different some things are perceived acoustically (and what should be covered by our game). For example, sound travels very slowly in comparison… so as with light very distant objects (like stars) are actually images from the past. But for a bat this creates a lot of funny effects in normal life situations already. Let’s assume for now the medium is perfectly at rest to our bat. Looking/listening at a distant incoming object it appears as it was in the past but as it closes in will be perceived closer to its actual present state. So what that means is that everything happening on that object will be perceived to happen in accelerated pace to catch up with the present as it approaches our bat. In particular observing an on-board clock it will look like it runs faster. The opposite will be true when the object passes by and moves away showing again a past picture of itself. Due to the definition of perceived distances it would also mean that distances on the object in the direction facing the observer will be squeezed for incoming objects and stretched for outgoing ones – while movement parallel to the observer will cause the object to appear sheared along the perceived depth axis (since the front (I.e. side facing the bat) of the object appears closer to its present state while its back is more in the past).
Taking a closer look at the role of the medium, things get a bit more exotic. For one gravity acting on it creates a gradient in air pressure and thus in the speed of sound. So this has the consequence that two wave fronts starting in parallel won’t necessarily maintain and can actually intersect after some time. Keeping in mind how perceived distances are defined this actually means that the parallel postulate isn’t satisfied in the perceived world – i.e. its geometry is inherently not Euclidean. This of course is a no go for any 3d engine which builds exclusively on Euclidean geometry. Therefore some trickery will be required to visualize any kind of effects resulting from an inhomogeneous medium.
For example a interesting aspect for visualization of the above mention perception may be that sound leaving earth’s surface at an angle will be deflected ever downwards due to the gradient in air pressure/speed of sound until it eventually turns around and meets the ground again. This means however that looking (hearing) upwards into the sky one should be able to perceive sounds from far away. So together this should create the perception that Earth surface is concave rather than convex, giving the bat an impression of living on the inside of a sphere – a bit like in a Dyson sphere.
I have talked with various mathematicians about this and the agreement was so far that differential geometry should be able provide the mathematical framework to describe the perceived world. This would be done by equipping the set of real world coordinates with this new perceived metric defining a new distance measure on the set to make it a proper manifold. Because the new metric should be still topologically equivalent (i.e. the identity function on the set is should be a non-isometric diffeomorphism) to the original real metric (in most scenarios relevant for the game) a transformation of physics into a formulation of how it’s perceived acoustically should be possible (meaning it should be potentially possible to calculate physics entirely in that new framework – by using said diffeomorphism to pullback physical laws). Problem is that no one so far could point me towards papers doing something quite like this. However, given that this would share a common mathematical calculus with general relativity and also aims to do physics in it, I thought it might be worthwhile to post this here for feedback and ideas.