Last night I dreamed I was standing among a family of burning giraffes. I felt the heat piercing my skin, the stench of burning flesh in my nostrils, and the fumes frying my eyes so much that I had to squint just to see. I was completely convinced. It was only upon waking that I realized how wrong my sense of reality had been.

This is an essential problem: our reality is never observed indubitably, it is perceived.

Reality as we know it is not fully real. But it is real enough (for now).

Consider how few tools we are given. We have exactly five senses, and no more. A platypus, on the other hand, can detect the electric fields generated by muscle contractions in its prey, a sense called electroreception. Imagine what we might do with it. The following example may help illustrate this:

Suppose I visit a friend who asks whether I like his new haircut. I say yes, it looks great. With electroreception, he reads the deep contractions in my muscles and knows immediately that I am lying. His haircut is, in fact, ugly, and now he knows it. In this way, we have removed dishonesty from human interaction. Court testimony, negotiation, things that ought to be truthful, can no longer be done “asynchronously”. A written paper can lie in ways a living voice cannot. Perhaps then scientific peer review becomes a live event, some sort of academic sermon that has to be performed before publication.

However, we were not given the gift of electroreception, but our lack of senses may be what drove us to build something remarkable in its place.

We invented mathematics, not just as a counting tool, but as a language for reasoning about the invisible. We built instruments that extend our perception with it: microscopes, telescopes, sensors that detect what no eye could see unaided. We developed statistics to tell us how much to trust what we observe, and the scientific method to ask whether our hypotheses adhere to reality. These are like prosthetics for us limited creatures, but so far, they have worked extraordinarily well. No other species we know of has come close to this kind of understanding.

But yet, these tools only provide mere approximations (extraordinarily good ones, but still approximations). Our models of the universe fit the data we have gathered, but not the universe itself.

This brings us back to the giraffes. How confident can we be, standing in what we call the waking world, that we are not still dreaming? We are doing the best we can with what we have. But who is to say it is enough?

This post is my attempt to unpack this idea.

What Does Dimension Mean?

A common intuition is that dimension indicates how many numbers is needed to specify a point. For example, a point on a line needs 1, a point in the plane needs 2, and a point in (our) space needs 3. This is the topological dimension.

But there’s another way to think about it, based on scaling. If you double the side length of a line segment, you get xxMATHPROTECTxx1xx copies. Doubling the side of a square gives you xxMATHPROTECTxx2xx copies, and doubling the side of a cube yields xxMATHPROTECTxx3xx copies.

Clearly, there’s a pattern. If scaling by a factor xxMATHPROTECTxx4xx gives xxMATHPROTECTxx5xx self-similar copies, then:

xxMATHPROTECTxx6xx

For ordinary shapes this gives exactly the integer dimension we expect, but for self-similar fractals, xxMATHPROTECTxx7xx does not have to be an integer.

The Hausdorff Dimension

This scaling argument is made rigorous by the Hausdorff dimension, introduced by Felix Hausdorff in 1918.

Instead of scaling up (like doubling the side lengths of a square), we can instead think about scaling down. Suppose we take a shape and break it into xxMATHPROTECTxx8xx smaller copies of itself, each scaled down by a factor xxMATHPROTECTxx9xx. In 1D, lengths scale by xxMATHPROTECTxx10xx. In 2D, areas scale by xxMATHPROTECTxx11xx. In 3D, volumes scale by xxMATHPROTECTxx12xx. So in general, for a xxMATHPROTECTxx13xx-dimensional object, “size” scales by xxMATHPROTECTxx14xx. That means each small copy has xxMATHPROTECTxx15xx times the size of the original. If we say the original shape had size xxMATHPROTECTxx16xx, then since there are xxMATHPROTECTxx17xx smaller copies making up the whole, we must have:

xxMATHPROTECTxx18xx

Solving for xxMATHPROTECTxx19xx get us:

xxMATHPROTECTxx20xx

The Sierpiński Triangle: xxMATHPROTECTxx21xx

Take an equilateral triangle. Divide it into four congruent smaller triangles and remove the central one. Repeat forever on each remaining triangle.

At each step, each triangle becomes xxMATHPROTECTxx22xx copies, each scaled by xxMATHPROTECTxx23xx. Plugging in:

xxMATHPROTECTxx24xx

It’s more than a curve but less than a surface. Its area must be zero, since at each iteration keeps only xxMATHPROTECTxx25xx of what remained, so after xxMATHPROTECTxx26xx steps the area is xxMATHPROTECTxx27xx, but it’s not 1-dimensional either. The number xxMATHPROTECTxx28xx captures exactly how much of the plane it occupies.

We can also construct this shape via a game:

1. Pick the three vertices of a triangle.
2. Start at any point inside.
3. Pick a random vertex, move halfway toward it, plot the point.
4. Repeat forever.

Despite the randomness, this procedure converges to the Sierpiński triangle.

Imaginary Cubes

This is the part of the Numberphile video that blew my mind.

The Sierpiński tetrahedron is the 3D analogue of the Sierpiński triangle: shrink a tetrahedron to half size, make 4 copies, connect them at their corners, and repeat. Its Hausdorff dimension is:

xxMATHPROTECTxx29xx

This is weird. The shape is clearly a 3D construction, yet it has the same dimension as a flat surface? It gets weirder.

When you project the Sierpiński tetrahedron onto a plane perpendicular to one of its axes, its shadow exactly fills a square. Every point in the square is covered, with no gaps and no overlaps. A fully 3D hollow fractal casting a completely solid 2D shadow.

We’ll call this an imaginary cube.

Here’s an interactive demo of a Sierpiński tetrahedron¹. Drag it around and see if you can find angle where it looks like a full 2D square.

This the kind of beautiful mathematics I can’t stop thinking about.

Change Your Mind

I’m a firm believer in changing your mind. When arguing, I find that a lot of people have their egos attached to the argument. Winning an argument raises their egos and losing drops it. In cases like this, arguments end up being a fight between humans. I find that this approach is flawed. Instead, arguments should be a battle between ideas, with the humans administering the fight.

If, while arguing, one finds that the opposing idea seems to be winning, it’s in their best interest to change their mind, since, upon doing so, means that they have now taken a more “winning” position. Aligning with this new position means that they have a higher chance in winning subsequent arguments, so long as the original “winning” points are still kept. Ideally, if everyone follows this heuristic, then all humans will then converge on the “ultimate” idea, and they can then be sure that this idea is better than all other ones (until an even better idea is formed); however, in practice, this is likely not the case. Oftentimes, winning is not always aligned with truth, and arguments aren’t good tests for truthful ideas. Ideas can win because they appeal to rhetoric, exploit biases, are more simple, etc. Also, it’s very possible that the audience simply lacks background knowledge. Here, arguments win just because they exploit the fact that humans perceive victory, instead of impartially judging of an argument’s evidential strength. In a way, Goodhart’s Law is a great application here: when winning an argument becomes the target, truth ceases to be the measure. In general, humans aren’t perfect, and argument environments aren’t perfect¹. Not everyone has equal information, and they don’t always argue in good faith or are exposes to the same global counterarguments. “Convergence” is thus oftentimes local, which is why I think super smart communities of people can still disagree for long periods. Also, some ideas are super hard to argue in general, such as ethics, metaphysics. Humans just don’t have the tools to reliably obtain indubitable evidence in these disciplines.

Nonetheless, I do believe that generally following this heuristic would increase the probability of humans moving toward more robust beliefs over time.

Updating on Bad Arguments

I would also like to reject an interesting epistemological argument I saw recently about updating on bad arguments. It argues that although intuitively it may make sense that hearing a bad argument for some idea should not change your degree of belief in the view, it actually turns out that hearing a bad argument when you expected to hear a good one should provide evidence against the view. It uses a probabilistic argument as follows:

The Argument:

Let xxMATHPROTECTxx0xx be the hypothesis my friend is arguing for, and suppose my prior xxMATHPROTECTxx1xx. Let xxMATHPROTECTxx2xx denote the event that my friend provides good evidence for xxMATHPROTECTxx3xx. The law of total probability suggests that

xxMATHPROTECTxx4xx

If I believed in advance that xxMATHPROTECTxx5xx, then since the prior is a weighted average of the two conditional probabilities, it follows that

xxMATHPROTECTxx6xx

In Bayesian updating, the law of total expectation implies that the prior is equal to the expectation of the posterior, which implies that observing xxMATHPROTECTxx7xx must push the posterior below the prior if observing xxMATHPROTECTxx8xx would have pushed it above, and thus, my friend’s failure in providing a good argument provides evidence against xxMATHPROTECTxx9xx.

Counterargument:

However, I disagree with this idea. I think that a bad argument being provided should be considered as no observation rather than as negative evidence. The above argument assumes that xxMATHPROTECTxx10xx, but I claim that instead, we should consider that xxMATHPROTECTxx11xx. If that holds, then we have that xxMATHPROTECTxx12xx and thus, no update is needed. My friend providing a bad argument should result in an update to my own belief about my friend’s reliability on this particular topic rather than a negative update on the idea itself.

Arguments are Hard

Arguments are very noisy systems. Humans are also very noisy. Still, I do think that arguments, even when treated as imperfect, are still one of our best tools for refining our beliefs, but the important thing is that people do refine their beliefs. We should always consider the noisiness in such a system and the reliability of the arguer among other things, but we should always be willing to revise our positions when the ideas themselves (not the recitation of it) demand it. We should try to detach our egos and identities from the beliefs we believe.

I will aim to explain the solution to the Three Gods Riddle.

We have 3 Gods: T, F, and R.
T always tells the truth. F always lies. R provides a random answer each time.
Each of these Gods will only give you one response, either “Ozo” or “Ulu”, signifying either Yes or No. Unfortunately, we do not know which is which. We are allowed to ask 3 questions total to any of the Gods. How can we deduce which God is which?

The key to this problem is asking the Gods a conditional statement. Consider the following question:

If I asked whether 1+1=2, would you answer Ozo?

Suppose Ozo = Yes. If T was asked whether xxMATHPROTECTxx0xx, it knows it would respond with the affirmative, Ozo. When asked whether or not it would answer Ozo to the question, his final response is Ozo. If F was asked whether xxMATHPROTECTxx1xx, it knows it would respond with the negative, Ulu. When asked whether or not it would answer Ozo to the question, his final response would be the affirmative, since it is a liar. Thus, both T and F would respond with Ozo.

Suppose Ozo = No. If T was asked whether xxMATHPROTECTxx2xx, it knows it would respond with the affirmative, Ulu. When asked whether or not it would answer Ozo to the question, it would respond with the negative, Ozo.
If F was asked whether xxMATHPROTECTxx3xx, it knows it would respond with the negative, Ozo. When asked whether or not it would answer Ozo to the question, it would respond with the negative, Ozo.

Thus, regardless of what Ozo means, when asked the carefully constructed question above, both T and F would respond with Ozo.

To further see the power of such a question, consider another question:

If I asked whether 1+1=3, would you answer Ozo?

It can be shown that regardless of what Ozo and Ulu mean, both T and F would respond with Ulu. In this way, we can deduce the truth value of the antecedent of our conditional, xxMATHPROTECTxx4xx or xxMATHPROTECTxx5xx, based on the response of either T or F.

If they respond with Ozo, then we know the antecedent is true. If they respond with Ulu, the antecedent is false.

Now, we need to utilize this property to our advantage.

First, ask the alien in the middle the following question:

If I asked you whether the alien on the left is R, would you answer Ozo?

If they respond with Ozo, we have two scenarios: either we have asked this question to R, or we asked it to T/F.
Assume this question was asked to either T or F. Due to our powerful property, we can then deduce the truth value of our antecedent: whether the alien on the left is R. Since they responded with Ozo, then the alien to the left must be R.

Assume this question was asked to R. His response does not matter, but under this assumption, the alien in the middle is R. Under both exhaustive assumptions, we can deduce that the alien to the right is not R.

Similarly, if they respond with Ulu, we again have the two scenarios: either we have asked this question to R, or we asked it to T/F.
Assume this question was asked to either T or F. We can then deduce the truth value of our antecedent: whether the alien on the left is R. Since they responded with Ulu, then the alien to the left is not R.
Assume this question was asked to R. His response does not matter, but under this assumption, the alien in the middle is R. Under both exhaustive assumptions, we can deduce that the alien to the left is not R.

Thus, the answer to this first question will reveal to us a God that is not R, a God that is either T or F. Then we can ask another one of our carefully constructed conditionals to this God to deduce the truth value of its antecedent.

For our second question, we will ask the following question to that God:

If I asked “are you F?” would you answer Ozo?

If they answer Ulu, then we know it is T that responded. If they answer Ozo, then we know it is F that responded.

For our third and final question, we will ask the same God the following:

If I asked “is the center alien R?” would you answer Ozo?

If they answer Ulu, then we know that the center alien is not R. If they answer Ozo, then we know that the center alien is R.

We can deduce one god is either T or F (from Q1). We can identify which of T or F it is (from Q2). We can determine if the middle god is R or not (Q3). Therefore, we can deduce all roles by elimination.

I aim to provide a brief introduction of this topic here and hope to build to the Erdős–Ko–Rado Theorem.

Brief Introduction

In graph theory, we have a set of vertices and a set of edges with each edge “connecting” two vertices. Formally, we say that a graph xxMATHPROTECTxx0xx, where xxMATHPROTECTxx1xx is a set of vertices and xxMATHPROTECTxx2xx is a set of edges. In graph theory, edges only involve two vertices.

A hypergraph is a basic generalization of graphs. Each edge can “connect” any number of vertices. Formally, a hypergraph xxMATHPROTECTxx3xx where xxMATHPROTECTxx4xx is a set of vertices and xxMATHPROTECTxx5xx. Drawing hypergraphs quickly becomes extremely difficult, so it’s worth thinking of them as “set systems” instead of visualizable graphs (hence the name “Combinatorics on Set Systems”).

We say a hypergraph is xxMATHPROTECTxx6xx-uniform if every edge is connected to xxMATHPROTECTxx7xx vertices. In other words, xxMATHPROTECTxx8xx. Here, this can also be thought of as collections of xxMATHPROTECTxx9xx-element subsets. This also allows us to characterize graphs as just xxMATHPROTECTxx10xx-uniform hypergraphs. A canonical example of a xxMATHPROTECTxx11xx-uniform hypergraph is the Fano plane.

We can then define a family xxMATHPROTECTxx12xx is intersecting if:

xxMATHPROTECTxx13xx

Note that xxMATHPROTECTxx14xx denotes the set xxMATHPROTECTxx15xx.

If you imagine a graph where vertices represent xxMATHPROTECTxx16xx sized sets and edges represent “disjointness”, then this intersecting idea represents independent sets. If edges represent disjointness, then an “independent set” (vertices that share no edges) implies that no two sets are disjoint, which means that they all intersect. This is often called a Kneser Graph.

Now, fix an element xxMATHPROTECTxx17xx. If we let xxMATHPROTECTxx18xx, then we have that every pair here intersects, and xxMATHPROTECTxx19xx. This is called a star.

It turns out, this is the largest possible size, and this is precisely the Erdős–Ko–Rado Theorem (1961).

Erdős–Ko–Rado Theorem (1961)

The theorem states: If xxMATHPROTECTxx20xx, then any intersecting family xxMATHPROTECTxx21xx satisfies

xxMATHPROTECTxx22xx

Moreover, if xxMATHPROTECTxx23xx, equality holds only for stars.

Note that we need xxMATHPROTECTxx24xx because if xxMATHPROTECTxx25xx, any two sets of size xxMATHPROTECTxx26xx must intersect by Pigeonhole Principle, meaning the entire set of xxMATHPROTECTxx27xx is intersecting.

The intuition behind the theorem is that if a family is not “anchored” by a single element (like a star), the sets are forced to “spread out” across the domain. This spreading increases the likelihood of disjointedness, preventing the family from reaching that maximum size.

In the future, I may include a full proof, but I hope that this brief introduction was sufficient in showcasing the complexity and overall vibe of hypergraph theory. Moving from graphs (xxMATHPROTECTxx28xx) to hypergraphs (xxMATHPROTECTxx29xx) causes the difficulty in studying these structures to explode, but it allows us to come up with so many fascinating results. To me, generalizing edges in this way was like opening up Pandora’s box.

Samuel van Hoogstraten - Old Man at a Window (c. 1653)

Pieter Bruegel the Elder - The Peasant and the Nest Robber (1568)

Johannes Vermeer - The Art of Painting (c. 1666-1668)

Jacobus Vrel - Woman at a Window (c. 1654-1662)

Rembrandt van Rijn - Portrait of an Old Woman (1654)

Let’s define a simple PDE to show how this works:

xxMATHPROTECTxx0xx

This is known as the 1-D heat equation. To solve this, the idea is to essentially define a neural network with differentiable activation functions xxMATHPROTECTxx1xx which approximates the solution. Let’s denote xxMATHPROTECTxx2xx. Since the network is differentiable, we can compute xxMATHPROTECTxx3xx and xxMATHPROTECTxx4xx easily. We now need to train the parameters xxMATHPROTECTxx5xx such that we minimize the residual, that is, we minimize the error between xxMATHPROTECTxx6xx and xxMATHPROTECTxx7xx. Mathematically, this is:

xxMATHPROTECTxx8xx

where xxMATHPROTECTxx9xx are sampled points in the domain and xxMATHPROTECTxx10xx is the PDE residual (difference between LHS and RHS of the PDE) at those points.

We also need to learn the initial condition, so we can define another loss function as follows:

xxMATHPROTECTxx11xx

Now, we define our final loss:

xxMATHPROTECTxx12xx

and all we need to do is to optimize xxMATHPROTECTxx13xx such that xxMATHPROTECTxx14xx is minimized. Note that if we had boundary conditions, we would add yet another loss, xxMATHPROTECTxx15xx, to account for that as well.

In practice, to solve this, I defined a fully connected neural network with input xxMATHPROTECTxx16xx and output xxMATHPROTECTxx17xx, given by:

xxMATHPROTECTxx18xx

where each xxMATHPROTECTxx19xx and xxMATHPROTECTxx20xx is applied element-wise.

The network parameters are optimized by minimizing the loss function above using the Adam optimizer. I also used PyTorch’s autograd.grad() feature to compute the derivatives. Here are my results (at xxMATHPROTECTxx21xx), it’s crazy how this literally works:

References

1. Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378, 686–707. https://doi.org/https://doi.org/10.1016/j.jcp.2018.10.045

The Generator takes a random noise vector xxMATHPROTECTxx2xx as input and produces a 64×64 image, xxMATHPROTECTxx3xx. The Discriminator receives an image (either real from the dataset or generated by xxMATHPROTECTxx4xx) and outputs a probability xxMATHPROTECTxx5xx indicating whether the image is real (from the dataset) or fake (produced by the Generator).

Training alternates between improving the Discriminator and the Generator. The Discriminator is trained to maximize the probability of assigning the correct label to both real and fake images. Its loss function is:

xxMATHPROTECTxx6xx

Here, xxMATHPROTECTxx7xx is a real image from the dataset, and xxMATHPROTECTxx8xx is a fake image produced from noise.

The Generator, on the other hand, tries to fool the Discriminator by producing images that it classifies as real. Its loss function is:

xxMATHPROTECTxx9xx

Training proceeds iteratively. First, I sample a random noise vector, generate a fake image, and evaluate it with the Discriminator to compute the loss_fake. Then, I pass a real image through the Discriminator to compute the loss_real. The total Discriminator loss is the sum of these two. After updating xxMATHPROTECTxx10xx, I sample new noise, generate another fake image, and compute the Generator loss by treating the fake image as if it were real, encouraging the Generator to improve.

Over time, this adversarial process improves both models. The Discriminator becomes better at distinguishing real from fake, and the Generator becomes better at producing realistic images. Eventually, the Generator can produce samples that are nearly indistinguishable from the real dataset, and we use it to generate new images.

Now, let’s see what this is like in practice. I have a dataset of ~200k celebrity faces (Liu et al., 2015), and I have a deep convolutional model for both the Generator and Discriminator. This is therefore called a DCGAN (Deep Convolutional GAN) (Radford et al., 2016). Here are the architectures:

DCGAN Generator Architecture

DCGAN Discriminator Architecture

As mentioned before, I used the Binary Cross-Entropy (BCE) loss function to train both the generator and discriminator. This is defined as:

xxMATHPROTECTxx11xx

where xxMATHPROTECTxx12xx is the predicted probability and xxMATHPROTECTxx13xx is the ground truth label.

I used the Adam optimizer for both the generator and the discriminator, with learning rate xxMATHPROTECTxx14xx and momentum parameters xxMATHPROTECTxx15xx, xxMATHPROTECTxx16xx. These are commonly used hyperparameters in GAN training.

I ended up training the models with a batch size of 128 across 20 epochs. It took quite a while, but after 15 epochs, here are the results:

The generated faces are starting to form. The model is slowly learning how to create features like eyes, hair, facial structure, jaw, etc. I think at this point, I had a Discriminator loss of 0.3211 and a Generator loss of 3.1851, which is pretty standard. Ideally, these losses should keep fluctuating. One model cannot win (because that spoils the adversarial part of the training).

Interestingly, however, after the full 20 epochs, we get the following:

Upon inspecting the losses, we have a Discriminator loss of 0.000 and a Generator loss of 64.0911. It seems like the Generator decided to output this one weird image all the time, and the Discriminator was not fooled. This is known as mode collapse. This happens when the Generator finds one output that consistently fools the Discriminator (until it doesn’t anymore) (Salimans et al., 2016). I will try to figure out how to fix this later.

References

1. Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative Adversarial Networks. https://arxiv.org/abs/1406.2661
2. Liu, Z., Luo, P., Wang, X., & Tang, X. (2015, December). Deep Learning Face Attributes in the Wild. Proceedings of International Conference on Computer Vision (ICCV).
3. Radford, A., Metz, L., & Chintala, S. (2016). Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. https://arxiv.org/abs/1511.06434
4. Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., & Chen, X. (2016). Improved Techniques for Training GANs. https://arxiv.org/abs/1606.03498