This work is licensed under the Creative Commons Attribution 4.0 International License.
In memoriam of Valentin Afraimovich (1945-2018), a visionary scientist, respected colleague, generous mentor, and loyal friend
Introduction
In the famous book "The intelligent investor", Benjamin Graham had advised not wasting time at forecasting how the market will perform in the future, as markets are fickle and market prices are largely meaningless. Instead, he taught that the intelligent investor should focus at the diligent financial evaluation of company’s market value, buying stocks that are clearly underpriced in the market. These opportunities are hard to find, but worth waiting for [1]. Nowadays, we have enough publicly available historical data on stock market prices and the adequate mathematical methods in our hands to confirm or deny the conclusions of the father of value investing and security analysis.
We have investigated the recent five year stock price data for the components of the Standard & Poor’s 500 (S&P 500) selected by a committee that assesses the company’s market capitalization (must be greater than or
equal to $6.1 bln USD), liquidity, domicile, public float, sector classification, financial viability, and length of time publicly traded and stock exchange. The companies selected by the S&P 500 committee are representative of the industries in the United States economy.
The efficient-market hypothesis (EMH) states that asset prices fully reflect all available information [2] though it is impossible to ascertain what a stock should be worth under an efficient market, since investors value stocks differently. Random events are entirely acceptable under an efficient market, and it is unknown how much time prices need to revert to fair value, so that the price remains uncertain at every moment of time and over any time horizon. Therefore, it is important to ask whether EMH undermines itself in its allowance for random occurrences or environmental eventualities. In our work, we suggest a conceptually novel answer resolving this paradox by demonstrating that uncertainty of a stock state in phase space (specified by the stock daily return and its time derivative, roughness) always contain a quantifiable information component that can neither be predicted from any available data (whether historical, new, or hidden "insider" information), nor has any repercussion for the future stock behavior, but which holds on to the ephemeral present only ("ephemeral information", see Sec. 4). Although the price remains profoundly uncertain, uncertainty does not affect the global ability of financial market to correct itself by instantly changing to reflect new public information.
We study predictability of market states and stock prices of individual companies by reconstructing a discrete model of S&P 500 phase space, in which every component of S&P 500 is characterized by the daily return and roughness of the stock, and the market state corresponds to a certain distribution of stock in phase space.
The novelty of our approach is twofold.
First, in Sec. 4.1, we have developed the novel concept of predictability of states in phase space. Our approach is based on the idea that with some probability a state might be a t-day precursor of another state in phase space. The introduced measure of t-day predictability of a state is a sum of all information components from its t-day precursors. From such a point of view, Markov chains are characterized by the maximum predictability, as any state of a Markov chain is a predictor for any other state of the chain for all t with probability 1. The total amount of predictable information for any state of the Markov chain equals the total information content of the chain.
Second, in Sec. 4.2, we have proposed the novel methodology for a quantitative assessment of the amounts of predictable and unpredictable information in any stochastic process whenever an empirical transition matrix between the states observed over a certain time horizon becomes available. The proposed technique is an extension of the famous Ulam’s method [6] for approximating invariant densities of dynamical systems.
The following conclusions can be drawn from the present study:
Long-range forecasts based on observed stock price time series might be efficient only for the frequently observed events in market phase space.
Short-range forecasts (1-3 days ahead) of rear events (such as market crushes) might be efficient if and only if the predicted states are resulted from the processes of exponential decay (growth) in time.
Stocks have different ’mobility’ in phase space. Mobility of a stock is usually irrelevant to the market capitalization weight of the company. The time series for highly mobile stocks might contain more predictable information for the efficient forecasting of the future states in phase space than the time series corresponding to the less mobile stocks.
Interestingly, the relations between predictable and unpredictable information in the stock price time series S&P 500, as well as between the amounts of predictable information on the future state of a stock available from the historic time series and from the present state alone are resembling those of unfair coin tossing, in which each state repeats itself with the probability 0 ≤ p ≤ 1.
Data source, data validation and preparation
Data on the individual stock prices of S&P 500 constituents for five years from 08/12/2013 to 08/09/2018 (i.e., 1,259 trading days in total) were acquired using The Investor’s Exchange API, the Python script is available at https://github.com/CNuge/kaggle-code/blob/master/stock_data We have chosen the largest data set publicly available for free and verifiable from different data sources. The data set contains price of the stock at market open, highest price reached in the day, lowest price in the day, price of the stock at market close, number of shares traded labeled by their ticker names for the most of S&P 500 companies during the studied period. We have used individual stock prices at market close since this information was always present in the data set. Moreover, in order to preserve consistency of our data, only 468 companies were considered; those selected were present in S&P 500 index throughout the entire observation period, and comprise 98.78% of S&P 500 total market weight. The other 37 constituents were excluded from the study because their presence in the index was inconsistent, and their stock prices were corrupted. The data set has been verified using Yahoo Finance. Computations were made using Python’s numerical libraries, such as NumPy and Pandas, Maple and Matlab to make sure the consistency of computations and eliminate possible errors.
The discrete model of Standard & Poor’s 500 phase space
We use the classical concept of phase space as a collection of all states specified by the observed values of the one-day investment return (as a ’position’):
$$R\left( t \right)=\frac{\text{Price}\left( t+1 \right)-\text{Price}\left( t \right)}{\text{Price}\left( t \right)},$$
where Price(t) is the price of the stock at market close, and t runs over trading days;
and its time derivative,
$$\dot{R}\left( t \right)=R\left( t+1 \right)-R\left( t \right),$$
known as roughness (as a ’momentum’) [3]. A stock is represented by a point in phase space accordingly its daily return and roughness. The stock’s price evolving over time traces a path in phase space (a trajectory) representing the set of states in a phase plot (see Fig. 1). The state of the market corresponds to a certain distribution and coherent movements of points in phase space.
Fig. 1
The smoothed phase space trajectories (individual stock prices at market close, one observation per day). a.) The Amazon company stocks (from 12/18/2013 to 01/10/2014); b.) The Apple company stocks (from 02/21/2018 to 03/14/2018). The regions of negative return are colored in red; the regions of positive returns are green colored.
For the sake of computational feasibility, the 5-year range of return values (minimum −0.53404886 and maximum of 0.3433584) as well as 5-year range of roughness values (minimum −0.59823906 and maximum 0.7059597) were divided into 50 intervals of equal lengths (of 0.01754815 and 0.0260839, respectively). The discrete model of market phase space was reconstructed as the ordered set of 2500 distinct cells, $\left( r,\dot{r} \right)\in \left\{ R \right\}\times $$\left\{ {\dot{R}} \right\}\text{where }\left\{ R \right\}\,\,\,\text{and }\left\{ {\dot{R}} \right\}$are the sets of intervals of returns and roughness, respectively.
Our approach to reconstructing market phase space is substantially different from the previous works, in which a phase space was discussed as a metaphor of either all possible market events "where the transfer of rights to real estate takes place and decision-making processes occur" [4], or a collection of recurrence plots containing the information on the systemic trajectory repetition in exchange market times series [5].
Methods
Methods discussed in the present section can be used for a quantitative assessment of the amounts of predictable and unpredictable information in any stochastic process – whether Markovian or not – whenever an empirical transition matrix between the states observed over a certain time horizon becomes available. The proposed methodology resembles the rigorous numerical scheme for approximating invariant densities of dynamical systems, Ulam’s method [6], and can be adopted to work with diverse, multimodal data sets.
Predictability of stock behavior in market phase space
The degree to which a correct prediction of the future value of a company stock or other financial instrument traded on an exchange can be made is an important open question for an intelligent investor. The stock market is prone to the sudden dramatic declines of stock prices driven by panic as much as by underlying economic factors [7]. Although the behavior of stock prices might be highly uncertain, the amount of uncertainty can be quantified, and the relevant degree of predictability can be assessed from publicly available information.
Information content of cells in market phase space
The amount of uncertainty associated to an event is related to the probability distribution of the event. Once the event X has been observed, some amount of uncertainty is removed, and the relevant amount of information is released. A highly uncertain event X, occurring with small probability P(X)≪1, contains more information than frequent events. The information content of the event X occurring with probability P(X) is
$$J\left( X \right)=-{{\log }_{2}}P\left( X \right)$$
measured in bits [8]. The information content of tossing a fair coin where the probabilities of heads and tails are p = 0.5 equals −log2 0.5 = 1 bit.
T-days mutual information between cells in market phase space
Mutual information measuring the dependence between two random variables had been introduced by Cover and Thomas [9] as a measure of predictability of stochastic states in time,
$$I\left( t \right)=\sum\limits_{\left\{ X,Y \right\}}{P\left( X\xrightarrow{t}Y \right)}{{\log }_{2}}\frac{P\left( X\xrightarrow{t}Y \right)}{P\left( X \right)P\left( Y \right)},$$
where $P\left( X\overset{t}{\mathop{\to }}\,Y \right)$is the empirical probability found by dividing the number of times the transition between the X and Y cells occurred precisely in t days by the total number of observed transitions from X, P(X) and P(Y) are the marginal probabilities (of observing the stock in X and Y, independently of each other), and summation is performed over all possible pairs of cells X and Y. If the transition probability $P\left( X\overset{t}{\mathop{\to }}\,Y \right)$is statistically independent of the cells X and Y, the amount of mutual information associated with such a pair of cells is zero. Mutual information decreases monotonically with time for a stationary Markov process [9].
T-days precursors for market events
Bayes’ theorem describes the probability of an event, based on prior knowledge of conditions that might be related to the event [10]. Given that a stock is in the phase space elementary cell X, it may be a t-days precursor of that in t days the stock will be in the cell Y with the following probability:
$${{P}_{t}}\left( X\left| Y \right. \right)=\frac{P\left( X\overset{t}{\mathop{\to }}\,Y \right)P\left( X \right)}{P\left( Y \right)}$$
where $P\left( X\overset{t}{\mathop{\to }}\,Y \right)$is the observed probability of transition of a stock from the phase space cell X to Y precisely in t days; P(X), P(Y) are the marginal probabilities of X and Y, respectively. Pt(X|Y) can be interpreted as a density of the t-days precursors for the event Y in phase space. If Pt(X|Y) is the same as P(X), then Y is unpredictable (i.e., any X is a precursor for Y). The total amount of available information about the forthcoming event Y can be gained from observing all t-day precursors X is measured by the Kullback-Leibler (KL) divergence [9],
$${\mathscr{P}_{t}}\left( Y \right)=\sum\limits_{\left\{ X \right\}}{{{P}_{t}}}\left( X\left| Y \right. \right){{\log }_{2}}\frac{{{P}_{t}}\left( X\left| Y \right. \right)}{P\left( X \right)}=\sum\limits_{\left\{ X \right\}}{P\left( Y \right)}\frac{P\left( X\overset{t}{\mathop{\to }}\,Y \right)}{P\left( Y \right)}{{\log }_{2}}\frac{P\left( X\overset{t}{\mathop{\to }}\,Y \right)}{P\left( Y \right)}.$$
The KL–divergence (6) has the form of relative entropy [9] that vanishes if and only if the density of the t- day precursors for the event Y in phase space Pt(X|Y) is identical to P(X). This implies that observation of stock in X is statistically independent of its observation in Y t days later, i.e. X is not a precursor for Y. Relative entropy was proposed as a measure of predictability in [11, 12], and we consider the relative entropy (6) as a measure of predictability for the forthcoming event Y t days ahead.
If the discussed process was Markovian, the marginal probability P(X) is the major eigenvector of the transition matrix $P\left( X\overset{t}{\mathop{\to }}\,Y \right),$and the probability (5) is Pt(X|Y) = 1, for all t and any Y. For a Markov chain, any state X is a predictor for any other state Y for all t with probability 1 (5). Then the total amount of predictable information (6) for any state Y of the Markov chain equals
$${\mathscr{P}_{\text{M}}}=-\sum\limits_{\left\{ X \right\}}{{{\log }_{2}}}P\left( X \right)=\sum\limits_{\left\{ X \right\}}{J\left( X \right)},$$
the total information content of the chain. Markovian processes are characterized by the maximum degree of predictability.
Predictable and unpredictable information estimated from an empirical transition matrix
The empirical transition probability $P\left( X\overset{t}{\mathop{\to }}\,Y \right)$constitutes a finite state discrete Markov chain in S&P 500 phase space. Every bit of information characterizing uncertainty of a state in a Markov chain consists of three independent information components,
$$H=\mathscr{D}+\mathscr{U}+\mathscr{E},$$
where $\mathscr{D}$("downward causation" [13]) characterizes our capability to predict the forthcoming state of the chain from the past states, $\mathscr{U}$("upward causation" [13]) characterizes our capability to guess the future state from the present one, and the amount of information $\mathscr{E}$("ephemeral information" [14]) can neither be predicted from the past, nor it’s repercussions can be traced in the future.
For example, tossing a unfair coin, in which each state (’heads’ or ’tails’) repeats itself with the probability
0 ≤ p ≤ 1, is described by the Markov chain with the transition probabilities $\mathbf{T}\left( p \right)=\left( \begin{matrix}p & 1-p \\1-p & p \\\end{matrix} \right)$(see Fig. (2. a)). For p = 1 and p = 0, the Markov chain generates the constant sequences of symbols, viz., . . 0,1,0,1,0, (for p = 0), and . . 1,1,1,1,1, or . . 0,0,0,0,0 (for p = 1). When p = 1/2, the Markov chain Fig. 2.a represents tossing a fair coin. The densities of both states are equal, π = [1/2,1/2], so that throwing a unfair coin to choose
Fig. 2
a.) The state diagram for tossing a unfair coin, in which each state (’heads’ or ’tails’) repeats itself with the probability 0 ≤ p ≤ 1. b.) The information components $(\mathscr{D})$(p), the past-future mutual information (excess entropy [14]); $(\mathscr{U})$ (p), the conditional mutual information available at the present state of the chain and relevant to the future states; $(\mathscr{E})$ (p), the ephemeral information existing only in the present state of the chain, being neither a consequence of the past, nor of consequence for the future.
between ’heads’ and ’tail’ reveals a single bit of information as quantified by Shannon’s entropy, $H\left( {{X}_{t}} \right)=$$-\sum{_{i=1}^{2}}{{\pi }_{i}}{{\log }_{2}}{{\pi }_{i}}=-{{\log }_{2}}\frac{1}{2}=1,$for any value of p [13]. 2
The information associated to the downward causation process,
quantifies the amount of uncertainty that is released from observation of the sequence of past states of the chain. When p = 0 or p = 1, the series is stationary, and the forthcoming state is determined by observing the past states, so that $\mathscr{D}\left( 0 \right)=D\left( 1 \right)=1\,\text{bit}\text{.}$When tossing a fair coin (p = 1/2), the information component (9) vanishes $\left( \mathscr{D}\left( {1}/{2}\; \right)=0 \right),$and the forecast of a future state by observing the historical sequence of symbols loses any predictive power (see Fig. (2. b)).
Although our capability to predict the forthcoming state by observing the historical sequences weakens as p > 0 or p < 1, we can now guess the future state of the chain directly from the presently observed symbol (by alternating / repeating the current symbol with the probability p > 0, or p < 1, respectively). The related information component (’upward causation’) quantifying the goodness of such a guess is the mutual information between the present and future states of the chain conditioned on the past [13], $\mathscr{U}=H\left( {{X}_{t+1}}\left| {{X}_{t-1}} \right. \right)-H\left( {{X}_{t+1}}\left| {{X}_{t-1}} \right. \right),$viz.,
Our capability to predict the future symbol from the present alone (as quantified by the mutual information (10)) increases as p > 0 (p < 1) till p ≈ 0.121 (p ≈ 0.879) when the effect of destructive interference between the obviously incompatible guesses on alternating the current symbol at the next step (for p > 0) and on repeating the current symbol (for p < 1) causes the attenuation and complete cancellation of this information component in the case of fair coin tossing $\left( \mathscr{U}\left( {1}/{2}\; \right)=0 \right)$(see Fig. (2. b)).
The remaining conditional entropy, $\mathscr{E}\equiv H\left( \left. {{X}_{t}}\, \right|{{X}_{t+1}},{{X}_{t-1}} \right),$quantifies the portion of uncertainty that can neither be predicted from the historical data, nor having repercussions for the future, viz.,
Since all information shared between the past and future states in Markov chains goes only through the present, the mutual information between the past states of the chain and its future states conditioned on the present moment is always trivial: I(Xt−1;Xt+1|Xt) = 0 [13, 14].
Results
We can gain insight into predictability and the related amounts of predictable and unpredictable information in the market events and daily stock prices of individual companies by analyzing the empirical t-days transition matrices $P\left( X\overset{t}{\mathop{\to }}\,Y \right)$and the density of states P(X) in market phase space.
Visualizing the stock market crashes, rallies, and market tumbling on shocking events
The state of the US economy can be visualized by daily snapshots of the S&P 500 stocks in phase space as displayed in Fig. 3. Standard & Poor’s calculates the market capitalization weights (currently ranging from 0.00753 for the National Weather Service to 3.963351 for the Apple company) using the number of shares available for public trading. Movements in the prices of stocks with higher market capitalization (the share price times the number of shares outstanding) had a greater impact on the value of the index than do companies with smaller market caps. In Fig. 3, we indicate the capitalization weights of companies by the radii of circles and jet-colors for convenience of the reader.
Fig. 3
The daily snapshots of S&P 500 stocks in phase space during a.) a stock market crush; b.) a stock market rally; c.) & e.) and d.) & f.) a market tumbling phenomenon. The regions of phase space characterized by the positive/ negative values of return are colored by ’green’ and ’red’, respectively).
Fig. 4
a.) The color - coded histogram indicating the number of visits of the phase space cells throughout the entire period of observations; b.) The color - coded histogram representing the information content of phase space cells; c.) The predictability of states based on 1-day precursors; d.) The predictability of states based on 24-day precursors.
Market crashes reveal themselves as a dramatic decline of stock prices across the market when the most of stocks appear in the ’red zone’ of phase space (Fig. 3.a) characterized by the negative return and often negative roughness values, resulting in a significant loss of paper wealth. During the periods of sustained increases in the prices of stocks (i.e., market rallies), the stocks overwhelmingly move to the ’green zone’ of phase space, with positive returns and often positive roughness (Fig. 3.b).
Market tumbling on shocking events constitutes a 2-day synchronization phenomenon (Fig. 3.c & e; d & f). On the first day, many stocks across the market get synchronized on the zero–return value that becomes visible as a vertical line emerging on the phase space snapshots (Fig. 3.c & d). On the second day, the stock market would become essentially volatile, plunging and snapping back due to coherent sells-off / ramping of stocks that might be affected by the announced news. Indeed, the stock drops may result in the rise of stock prices for corporations competing against the affected corporations. Interestingly, a certain degree of coherence in the apparently volatile market movements might persist for a day as it seen from the continued alignment of stocks in phase space visible on the second day of the market recovery process (Fig. 3.e & f).
The first tumbling event shown on Fig. 3.c& e had occurred on Thursday, 07/18/2014, when a Malaysia Airlines plane (flight MH17) headed from Amsterdam to Kuala Lumpur carrying 298 passengers had been shot down by a Russian military unit invading eastern Ukraine. The second tumbling is observed on August 4,2014 (see Fig. 3. d & f) at the coincidence of alarming events, including Argentina’s default on bond payments, more sanctions against Russia backed by EU and US in response to the downing of flight MH17, and the Islamic State seizure of fifth Iraqi oil field. Within that week, the S&P 500 lost 2.69%, the Dow fell 2.75%, and the Nasdaq slid 2.18%.
Phase portrait , t–days predictability of states, and t–days mutual information in S&P 500 phase space
Not all cells in market phase space are visited equally often by stock prices. The central region, about the zero-return zero-roughness equilibrium point was the most visited of all (see Fig. 4.a) during five years of
observations. This observation is in line with the Mean reversion theory suggesting that asset prices and returns eventually return back to the long-run mean or average of the entire data set [15]. Consequently, the information content is minimal for the frequently visited cells located in the central region, but is high for the periphery cells rarely visited by stocks (Fig. 4.b). The phase space cells corresponding to market crushes are characterized by the maximum information content. Uneven attendance of cells in S&P 500 phase space results in the inequitable degree of predictability of the corresponding events.
The predictability of states calculated as the KL–divergence (6) between the density of t- days precursors for an event and the marginal density of states P(X) across phase space changes with time (see Fig. 4.c & d). Summarizing on short-range predictions based on observation of all possible 1–day precursors for every cell in phase space, we obtain (see Fig. 4.c) that the most predictable states are aligned along the ’main diagonal’ (R ∝ Ṙ) of the phase portrait in S&P 500 phase space, roughly corresponding to the exponential growth (decay) processes with a single unstable equilibrium at the zero-return zero-roughness point (0,0). The states out of the ’main diagonal’ shown in Fig. 4.c cannot be predicted efficiently.
Long-range forecasts predicting the behavior of stocks for more than 3 days in advance follow the different predictability pattern shown in Fig. 4.d. The efficiently predictable states for long-range forecasts are located overwhelmingly in the central region of the phase portrait, about the zero-return zero-roughness unstable equilibrium point. In Fig. 4.d, we have presented the predictability pattern based on 24-day precursors. This observation also supports the general wisdom of the Mean reversion theory [15]. Long-range predictions of infrequent events (such as market crushes) situated on the periphery of the phase diagram should be seen as a rough guide, as the accuracy of such predictions falls considerably around the few days mark.
The decay rate of time-dependent mutual information is an important characteristic of the fluctuations of information flow in a complex system [16]. In the case of small fluctuations of information flow, the time–dependent mutual information decays linearly in time; but the decay might be slower than linear, in the case of the large fluctuations of the information flow in the system [16]. Mutual information decays monotonically with time for Markov processes [9]. In formal languages, the mutual information between two symbols, as a function of the number of symbols between the two, decays exponentially in any probabilistic regular grammar, but can decay like a power law for a context-free grammar [17].
It is worth mentioning that the time decay of mutual information in time (4) calculated over S&P 500 phase space is very heterogeneous and even non-monotonous. The value of mutual information plunges within the first three days to approximately 7 Kbits and then gets stuck there, at least up to 100-day’s mark (see fig. 5). The observed non-monotonous decay of mutual information may indicate the presence of the long-lasting large fluctuations of information flow that might be attributed to stock market bubbles resulted from groupthink and herd behavior [18].
Fig. 5
T–days mutual information of S&P 500.
Predictable and unpredictable information in company stock prices. Stocks of different mobility
An intelligent investor is interested in diversification of the portfolio across stocks to reduce the amount of unsystematic risk of each security. Unsystematic risk (related to the occurrence of desirable events) might be associated to the high shares of unpredictable (ephemeral) information in daily stock prices. To quantify the risk levels in market phase space, we have investigated the shares of predictable and unpredictable information in the transitions of stocks between the cells in phase space for all 468 studied companies from the S&P 500 (Fig. 6).
Fig. 6
The shares of transition information predictable from the historic time series ($(\mathscr{D})$), from the present observation ($(\mathscr{U})$), and ephemeral information $(\mathscr{E})$ vs. Shannon’s entropy H measuring uncertainty of daily return for every studied company. The lines represent the polynomial trends for the empirically observed relations.
Below, we report the results obtained for 1-day transitions of company stock prices, planning the detailed report on the long-term transitions for the future publications.
We have analyzed the empirical 1-day transition matrices for the stocks of individual companies from the S&P 500 list observed in the last five years. Every transition matrix gives us three values (D,U,E) quantifying the predictable and unpredictable information components in uncertainty of a stock state with respect to one-day transitions. In Sec.4.2, we discussed that the transition matrices for unfair coin tossing can be parameterized by the state repetition probability, so that for every value of 0 ≤ p ≤ 1, we get a triplet (D,U,E)(p) shown in Fig. 2. While working with the S&P 500 data, we have parameterized the (D,U,E) – triplets by uncertainty of daily stock price assessed by its Shannon entropy H producing the curves shown in Fig. 7.
Fig. 7
Uncertainty of daily stock price measured by Shannon’s entropy H along with the information components, $\mathscr{D},\,\mathscr{U}$and $\mathscr{E},$for 468 major companies from the S&P 500.
The functional analogy between the state repetition probability 0 ≤ p ≤ 1 for tossing an unfair coin and uncertainty of daily stock price of a company can be intuitively understood in terms of stock “mobility” in phase space. The abundance of trajectory patterns pertinent to highly mobile stocks provide more valuable information for the efficient prediction of the future states. On the contrary, from the forecasting perspective, the behavior of low mobile stocks is prone to high unsystematic risk of unpredictable events like tossing a fair coin.
The empirically observed relations shown in Fig. 6 demonstrate that the amount of predictable information in daily transition of stocks $\left( \mathscr{D}+\mathscr{U} \right)$grows while the amount of unpredictable information $(\mathscr{E})$decays monotonously with Shannon’s entropy H measuring uncertainty of the daily stock price for every studied company. Unpredictable information associated to unsystematic risk is maximum (exceeding the both predictable information components) for the companies characterized by the lowest entropy values. In contrast to it, the amount of unpredictable information practically vanishes for the companies characterized by the highest uncertainty of daily stock prices, which can be mostly resolved by observation the historic time series as indicated by the dominance of the information component $\mathscr{D}$(see Fig. 7).
Although entropy is commonly associated with the amount of disorder, or chaos in a system, our conclusion does not appear as a paradox judged from the everyday perspective. Stocks have different ’mobility’ in phase space as quantified by inequitable uncertainty of daily return (Shannon’s entropy) H in market phase space. In statistics, entropy is related to the abundance of ’microscopic configurations’ (i.e, trajectories of stocks updated daily) that are consistent with the observed ’macroscopic quantities’ (i.e., the density of cells obtained over the whole period of observation) [19]. Highly mobile stocks increase the degree to which the probability of visiting a cell is spread out over different possible trajectories in market phase space, providing more data valuable for the efficient forecasting the future states; the more such trajectories are available to the stock with appreciable probability, the greater it’s entropy. On the contrary, stocks of low mobility characterized by the low levels of Shannon’s entropy do not accumulate enough data for an efficient predictions of the future states.
The shares of information components in complex systems are naturally related to each other. The amount of unpredictable information decreases when the amount of predictable information increases; information predictable from the present state of a system alone is minimum when the future state can be determined from observation of the historic data.
Interestingly, there are remarkable similarities between the information relations observed across the S&P 500 stock market and in unfair coin tossing Fig. (2.a) discussed by us in Sec. 4.2. In Fig. 8.a.) and b.), we have presented the relations between the amounts of unpredictable $(\mathscr{E})$and predictable information $(\mathscr{D}\,+\,\mathscr{U})$ and between $\mathscr{D}\,\text{and }\mathscr{U}$in unfair coin tossing: these relations are linear and parabolic, respectively. The corresponding relations between the information components observed across the S& P 500 stock market are shown in Fig. 8.c.) and d.).
Fig. 8
A "unfair coin" of the S& P 500 stock market. a.) Unpredictable $(\mathscr{E})$vs. predictable information $(\mathscr{D}\,+\,\mathscr{U})$in unfair coin tossing Fig. (2. a); b.) Predictable information available from the present state alone $(\mathscr{U})$vs. predictable information available from the past series $(\mathscr{D})$in unfair coin tossing Fig. (2. a); c.) Unpredictable $(\mathscr{E})$vs. predictable information $(\mathscr{D}\,+\,\mathscr{U})$in the S& P 500; d.) Predictable information available from the present state alone $(\mathscr{U})$vs. predictable information available from the past series $(\mathscr{D})$in the S& P 500.
Following the analogy between the behavior of stock in S&P 500 phase space and tossing a unfair coin, we may say that stocks highly mobile in phase space characterized by the high entropy values correspond to the
marginal values $p\underset{\scriptscriptstyle\thicksim}{>}0,\,\,\text{or }p\underset{\scriptscriptstyle\thicksim}{<}1$for the probability to repeat ∼∼><a symbol in unfair coin tossing. In both cases, the abundance of patterns in symbolic time series provide more valuable information for the efficient prediction of the future states. On the contrary, from the forecasting perspective, the behavior of ’low mobile’ stocks prone to high unsystematic risk of undesirable events is somewhat similar to tossing a fair coin.
Discussion and Conclusion
We have reconstructed the discrete model of S&P 500 phase space and studied predictability of the cells and individual S&P 500 constituents. The certain distributions and coherent movements of points in phase space are associated to the particular states of the market.
Inhomogeneous density of states in S&P 500 phase space results in their inequitable predictability: more frequent market events are predicted more efficiently than relatively rare events, especially in the long-run. The heterogeneous and even non-monotonous decay of mutual information in time might be attributed to strong fluctuation of information flow across S&P 500 phase space arisen due to stock market bubbles.
Mobility of a stock in phase space can be quantified by entropy measuring uncertainty of the daily return of the company. Highly mobile stocks characterized by the high entropy values provide more data valuable for the efficient forecasting the future states and therefore are associated to lower unsystematic risk than the stocks of ’low mobility’. High stock mobility is usually irrelevant to the market capitalization weight of the company. WEC Energy Group Inc., providing electricity and natural gas to 4.4 mln customers across four states through its brands, is characterized by the highest value of entropy (HWEC = 8.377 bit) among all S&P 500 constituents. Entergy Corporation, an integrated energy company engaged in electric power production and retail electric distribution operations, and the CMS Energy Corporation share the second and third places with HWEC = 8.352 bit and HCMS = 8.279 bit, respectively. For comparison with Apple Inc, the leader of the S&P 500 market capitalization weight (3.963351), it’s on the 120th place with HAAPL = 7.5526 bit.
We confirm a great deal of wisdom shared with the common investors by Benjamin Graham in his famous book [1]. Reliable long-range predictions of rare events, such as market crushes, are hardly possible. Companies that offer a large margin of safety are not always those of the highest S&P 500 market capitalization weight. In the last five years, the companies of low unsystematic risk are found in the energy industry, forming the bedrock of our society.
We believe that our work would help to resolve a logical paradox related to EMH on that if no investor had any clear advantage over another, would there be investors who have consistently beat the market, such as Benjamin Graham and his disciple, Warren Buffett?
