More

Why does a z factor of 1 get turned into 0.999998 in my output

Why does a z factor of 1 get turned into 0.999998 in my output


Just built a model that uses 3D Analyst - Surface Volume. I am trying to use a z factor of 1. Whenever I run the model my output table shows that a Z factor of 0.999998 was used instead of 1. I know this is a very small difference but even so, it is skewing my results somewhat. Really what I would like to know, is there a good explanation of why it gets changed to 0.999998 or is there a way to make it stay at 1?


Thanks again for your input. When I've tried changing the type of raster, the tool grays out the z factor all together and locks it in at 0.999998. So… I'm guessing there is really no way around this using integer or floating point data. I even tried converting to integer and using the raster calculator to multiply the data by 1000 to preserve the decimals in integer form, but was still stuck with the 0.999998 z factor. No dice.


I have to agree with the comments, its about how you want to represent that number. If you want to store number 1 as a floating point, ArcGIS stored this number as 0,9999… It seems that this Z factor argument is stored that way. I think there's no way to store it as integer. In fact, 0,9999 is equal 1.

I must add a picture I was saw few days ago. It fits to this question.


Why is it called Latent Vector?

I just learned about GAN and I'm a little bit confused about the naming of Latent Vector.

First, In my understanding, a definition of a latent variable is a random variable that can't be measured directly (we needs some calculation from other variables to get its value). For example, knowledge is a latent variable. Is it correct?

And then, in GAN, a latent vector $z$ is a random variable which is an input of the generator network. I read in some tutorials, it's generated using only a simple random function:

then how are the two things related? why don't we use the term "a vector with random values between -1 and 1" when referring $z$ (generator's input) in GAN?


I'm going to approach this a little differently starting from roughly the same place. Here I am going to use a resonant $lambda$/2 20m dipole driven by 100 W as the model.

Let's compute the current at the feed point of a dipole at resonance, this is found with the input power (100 watts) and the feed point impedance which for our dipole is assumed to be the theoretical 73 $Omega$ :

Therefore the driving voltage can be calulated with Ohm's Law:

$ V_ ext = I cdot R = 1.17 mathrm A_ ext cdot 73 Omega = 85.44 :mathrm V_ ext $ (unmodulated signal)

The voltage at the end of the dipole would require us to calculate the Q and solve the following:

Trying to minimize the hand-waving, we can use some approximations from transmission line theory to give us the Q. (See Edmund Laport's Radio Antenna Engineering for a complete (and math heavy) explanation) To do this we need the characteristic impedance of the dipole (considered as a transmission line). That is given by:

$ Z_ <0>= 276 cdot log_<10>frac

= 972.31 Omega $

Where $l$ is the total length of the dipole and $p$ is the radius of the conductor (all in the same units). I am going to ignore calculating the exact length here, we know it's approximately 5% shorter than the real wavelength to make up for velocity factor and end effects. This next bit leans on transmission line theory and can turn into a bag of snakes, if you want to know more about where these equations come from, check the reference quoted above. $Q$ here is the ratio of the voltage of the direct wave and the reflected wave:

and $m$ is calculated from the feed point impedance $R$ and the characteristic impedance $Z_0$:

When I calculate $ Z_0 $, I am going to assume our dipole is made with 3mm wire. Now to crank through the numbers:

Now we can solve for $ V_ ext $:

Again, this is the RMS voltage we should convert to peak voltage:

$ 568 :mathrm V_ ext cdot sqrt <2>= pm 804 :mathrm V_ ext $

This is all for 100W, if we instead plug 1500W into the above math, we come up with

$ 4397 :mathrm V_ ext : ext : pm 6200 :mathrm V_ ext $

That's a pretty hefty jolt. So getting back to the OP's other questions, the input power has a substantial effect on the voltage. The rest of the factors are all the same as for maximizing antenna efficiency (resonance, conductor size, etc.)

EDIT: Most of the above equations come from the section on Circuital Design in the reference listed above. The book is more math heavy than typical amateur radio references, but not as bad as some of the more modern engineering texts. It's slow going, but a worthwhile read.


Good Password Hashing Functions

PBKDF2 comes from PKCS#5. It is parameterized with an iteration count (an integer, at least 1, no upper limit), a salt (an arbitrary sequence of bytes, no constraint on length), a required output length (PBKDF2 can generate an output of configurable length), and an "underlying PRF". In practice, PBKDF2 is always used with HMAC, which is itself a construction built over an underlying hash function. So when we say "PBKDF2 with SHA-1", we actually mean "PBKDF2 with HMAC with SHA-1".

  • Has been specified for a long time, seems unscathed for now.
  • Is already implemented in various framework (e.g. it is provided with .NET).
  • Highly configurable (although some implementations do not let you choose the hash function, e.g. the one in .NET is for SHA-1 only).
  • Received NIST blessings (modulo the difference between hashing and key derivation see later on).
  • Configurable output length (again, see later on).
  • CPU-intensive only, thus amenable to high optimization with GPU (the defender is a basic server which does generic things, i.e. a PC, but the attacker can spend his budget on more specialized hardware, which will give him an edge).
  • You still have to manage the parameters yourself (salt generation and storage, iteration count encoding. ). There is a standard encoding for PBKDF2 parameters but it uses ASN.1 so most people will avoid it if they can (ASN.1 can be tricky to handle for the non-expert).

Unity just means 1, so they have presumably normalized their values so that they all sum to 1 instead of whatever their "natural" total is. I could imagine a few specialized normalization schemes, but this is typically done by dividing, and that's what I would assume in the absence of a more detailed description. If they had normalized so that the values summed to 100 instead, they'd be expressing it as a percentage.

Suppose there is a substance made of three chemicals: 5L of Chemical A, 2L of Chemical B, and 3L of Chemical C. You could do a similar normalization and say that each litre of substance contains 0.5L of A, 0.2L of B, and 0.3L of C (each value has been divided by 10, the total, so all the values together sum to one). If you normalized to 100 instead of unity, then you could also say that the substance is 50% A, 20% B, and 30% C.


They mainly differ in the link function.

In Probit: $Pr(Y=1 mid X) = Phi(X'eta)$ (Cumulative normal pdf)

In other way, logistic has slightly flatter tails. i.e the probit curve approaches the axes more quickly than the logit curve.

Logit has easier interpretation than probit. Logistic regression can be interpreted as modelling log odds (i.e those who smoke >25 cigarettes a day are 6 times more likely to die before 65 years of age). Usually people start the modelling with logit. You could use the likelihood value of each model to decide for logit vs probit.

A standard linear model (e.g., a simple regression model) can be thought of as having two 'parts'. These are called the structural component and the random component. For example:
$ Y=eta_0+eta_1X+varepsilon ext varepsilonsimmathcal(0,sigma^2) $ The first two terms (that is, $eta_0+eta_1X$) constitute the structural component, and the $varepsilon$ (which indicates a normally distributed error term) is the random component. When the response variable is not normally distributed (for example, if your response variable is binary) this approach may no longer be valid. The generalized linear model (GLiM) was developed to address such cases, and logit and probit models are special cases of GLiMs that are appropriate for binary variables (or multi-category response variables with some adaptations to the process). A GLiM has three parts, a structural component, a link function, and a response distribution. For example:
$ g(mu)=eta_0+eta_1X $ Here $eta_0+eta_1X$ is again the structural component, $g()$ is the link function, and $mu$ is a mean of a conditional response distribution at a given point in the covariate space. The way we think about the structural component here doesn't really differ from how we think about it with standard linear models in fact, that's one of the great advantages of GLiMs. Because for many distributions the variance is a function of the mean, having fit a conditional mean (and given that you stipulated a response distribution), you have automatically accounted for the analog of the random component in a linear model (N.B.: this can be more complicated in practice).

The link function is the key to GLiMs: since the distribution of the response variable is non-normal, it's what lets us connect the structural component to the response--it 'links' them (hence the name). It's also the key to your question, since the logit and probit are links (as @vinux explained), and understanding link functions will allow us to intelligently choose when to use which one. Although there can be many link functions that can be acceptable, often there is one that is special. Without wanting to get too far into the weeds (this can get very technical) the predicted mean, $mu$, will not necessarily be mathematically the same as the response distribution's canonical location parameter the link function that does equate them is the canonical link function. The advantage of this "is that a minimal sufficient statistic for $eta$ exists" (German Rodriguez). The canonical link for binary response data (more specifically, the binomial distribution) is the logit. However, there are lots of functions that can map the structural component onto the interval $(0,1)$, and thus be acceptable the probit is also popular, but there are yet other options that are sometimes used (such as the complementary log log, $ln(-ln(1-mu))$, often called 'cloglog'). Thus, there are lots of possible link functions and the choice of link function can be very important. The choice should be made based on some combination of:

  1. Knowledge of the response distribution,
  2. Theoretical considerations, and
  3. Empirical fit to the data.

Having covered a little of conceptual background needed to understand these ideas more clearly (forgive me), I will explain how these considerations can be used to guide your choice of link. (Let me note that I think @David's comment accurately captures why different links are chosen in practice.) To start with, if your response variable is the outcome of a Bernoulli trial (that is, $ or $1$), your response distribution will be binomial, and what you are actually modeling is the probability of an observation being a $1$ (that is, $pi(Y=1)$). As a result, any function that maps the real number line, $(-infty,+infty)$, to the interval $(0,1)$ will work.

From the point of view of your substantive theory, if you are thinking of your covariates as directly connected to the probability of success, then you would typically choose logistic regression because it is the canonical link. However, consider the following example: You are asked to model high_Blood_Pressure as a function of some covariates. Blood pressure itself is normally distributed in the population (I don't actually know that, but it seems reasonable prima facie), nonetheless, clinicians dichotomized it during the study (that is, they only recorded 'high-BP' or 'normal'). In this case, probit would be preferable a-priori for theoretical reasons. This is what @Elvis meant by "your binary outcome depends on a hidden Gaussian variable". Another consideration is that both logit and probit are symmetrical, if you believe that the probability of success rises slowly from zero, but then tapers off more quickly as it approaches one, the cloglog is called for, etc.

Lastly, note that the empirical fit of the model to the data is unlikely to be of assistance in selecting a link, unless the shapes of the link functions in question differ substantially (of which, the logit and probit do not). For instance, consider the following simulation:

Even when we know the data were generated by a probit model, and we have 1000 data points, the probit model only yields a better fit 70% of the time, and even then, often by only a trivial amount. Consider the last iteration:

The reason for this is simply that the logit and probit link functions yield very similar outputs when given the same inputs.

The logit and probit functions are practically identical, except that the logit is slightly further from the bounds when they 'turn the corner', as @vinux stated. (Note that to get the logit and the probit to align optimally, the logit's $eta_1$ must be $approx 1.7$ times the corresponding slope value for the probit. In addition, I could have shifted the cloglog over slightly so that they would lay on top of each other more, but I left it to the side to keep the figure more readable.) Notice that the cloglog is asymmetrical whereas the others are not it starts pulling away from 0 earlier, but more slowly, and approaches close to 1 and then turns sharply.

A couple more things can be said about link functions. First, considering the identity function ($g(eta)=eta$) as a link function allows us to understand the standard linear model as a special case of the generalized linear model (that is, the response distribution is normal, and the link is the identity function). It's also important to recognize that whatever transformation the link instantiates is properly applied to the parameter governing the response distribution (that is, $mu$), not the actual response data. Finally, because in practice we never have the underlying parameter to transform, in discussions of these models, often what is considered to be the actual link is left implicit and the model is represented by the inverse of the link function applied to the structural component instead. That is:
$ mu=g^<-1>(eta_0+eta_1X) $ For instance, logistic regression is usually represented: $ pi(Y)=frac <1+exp(eta_0+eta_1X)>$ instead of: $ lnleft(frac<1-pi(Y)> ight)=eta_0+eta_1X $

For a quick and clear, but solid, overview of the generalized linear model, see chapter 10 of Fitzmaurice, Laird, & Ware (2004), (on which I leaned for parts of this answer, although since this is my own adaptation of that--and other--material, any mistakes would be my own). For how to fit these models in R, check out the documentation for the function ?glm in the base package.

(One final note added later:) I occasionally hear people say that you shouldn't use the probit, because it can't be interpreted. This is not true, although the interpretation of the betas is less intuitive. With logistic regression, a one unit change in $X_1$ is associated with a $eta_1$ change in the log odds of 'success' (alternatively, an $exp(eta_1)$-fold change in the odds), all else being equal. With a probit, this would be a change of $eta_1 ext< >z

V Conclusion

…a city is not at its fundamental level optimizable. A city’s dynamism derives from its inefficiencies, from people and ideas colliding unpredictably. (Badger, 2018)

The excerpt above is from a New York Times article describing the tech world’s attempt to improve cities by optimizing their functions it paraphrases a quote from UC-Berkeley professor Nicholas de Monchaux who reminds us that efficiency is overrated. The pursuit of real-time efficiency means that we are trying to optimize components of complex systems that we do not fully understand – and may understand even less due to our attempts. A city is not a machine to be engineered for high speeds: a city is a complex ecosystem of networks and flows (Batty, 2012, 2013b). Fast geographic data is a potent accelerant that should only be applied to this ecosystem in a judicious and discriminating manner.

The real challenge facing humanity is not speeding up but rather slowing down the flow of people, material and energy through cities (Townsend, 2013). A first step is to stop viewing friction as an enemy to be vanquished: friction can be a friend (Miller, 2017b). In Terra Nova: The New World after Oil, Cars, and Suburbs, Eric Sanderson (2013) describes a vision of city organization based on ecological principles that embrace good frictions and reduce bad frictions. Good frictions are those at the interfaces between natural systems and human systems where resources (including land) leave the natural world and wastes re-enter. Bad frictions are those that slow exchanges and innovation within human systems. Physical friction is also bad, to avoided by replacing roads with rail. Whether you accept the details of his vision or not, Sanderson shows there are ways we can organize human systems other than the unquestioned pursuit of a fast, frictionless world.


How accurate is GPS for speed measurement?

As with positioning, the speed accuracy of GPS depends on many factors.

The government provides the GPS signal in space with a global average user range rate error (URRE) of &le0.006 m/sec over any 3-second interval, with 95% probability.

This measure must be combined with other factors outside the government's control, including satellite geometry, signal blockage, atmospheric conditions, and receiver design features/quality, to calculate a particular receiver's speed accuracy.


3 Answers 3

One important factor is soil fertility:

Java’s soils are very fertile because of periodic enrichment by volcanic ash.

In contrast, with the two nearby large Indonesian islands of Borneo (shared with Brunei and Malaysia) and Sumatra:

Borneo, the world’s third largest island, has exemplary rainforest soils: shallow and nutrient poor. The abundance of rain in these ancient ecosystems has leached the soil for millions of years.

In contrast to overpopulated Java the neighbouring island of Sumatra still provides huge unused land reserves. However, by far not all of these reserves can be regarded as real agricultural potentials, e.g. for resettlement projects. Especially the poor soils often prove an agricultural handicap. Besides soil fertility the existing vegetation has to be considered. Thus, for example, the so called "alang alang grass savannas" in general show better potentialities than forest areas, while most of the swamps prove rather unsuitable for agricultural development.

(I don't think it makes sense to compare Java with islands like Honshu or Madagascar that are in other countries with entirely different histories.)

The most fundamental reason with respect to other parts of tropical Asia is the earlier adoption of wet rice cultivation in Java. Grigg points out that the only places in Asia that had greater population densities in the late 19th century were China and Japan. Wet rice is just an amazingly productive form of agriculture, especially by traditional standards, before the age of synthetic fertilizers.

Japan is a different case. It is only recently that Java's population has surpassed Japan's, and that is due to Japan having some of the lowest fertility rates in the world. Java is still at an earlier stage of its demographic transition, but over time its population growth is gradually slowing down.

In Clifford Geertz' classic volume Agricultural Involution: the Processes of Ecological Change in Indonesia (1963), the author argues that the Javanese form of wet rice cultivation has a very high potential to "absorb labor". While Geertz' view has been criticized – for an enlightening discussion see Wood (2020), Chapter 6 (preview available on Google Books) – it is a good starting point towards answering your question. Wood writes:

It is fundamental to Geertz's view that different agricultural systems have different capacities for labor absorption and involution. According to Geertz, wet rice farming as practiced in East and Southeast Asia probalby has the highest capacity to absorb labor of any form of traditional agriculture.

Thus, the particular attributes of wet rice farming systems in the tropics and subtropics allow these systems to support, and indeed in the views of some authors demand, high population densities. High density, high intensity wet rice systems are found throughout south and southeastern Asia.

However, as other responders to this question remarked, the properties of the physical environment are also critical factors in the potential for a high density wet rice system. Growing season temperature and rainfall will influence potential productivity, but in tropical and temperate monsoon Asia, soil fertility is probably more important. So the best examples of high density rice systems are found not just in the young, fertile volcanic soils of Java, but also along the major river valleys and deltas of the region, e.g., Mekong, Red, Chao Phraya, Irrawaddy, Pearl, Yangtze, Ganges, and in the fertile but narrow volcanic valleys of Japan. As noted above, the other islands you mention:

Madagascar, Borneo, Sulawesi, Honshu (Japan), and Sri Lanka

are characterized by older and more complex geology with significantly less fertile soils, and in the case of Japan, substantial areas with cool to cold temperate climate where rice agriculture would have been limited to a single crop per year, if at all.


Footnotes

Text files can obviously be translated into numbers this is how they are stored and transmitted. Can’t text files be processed electronically? Again, the answer has to be yes, conditional on what one means by processed. The ability of computer algorithms to process and generate speech (text) has dramatically improved since we first discussed soft and hard information. Whether it can be interpreted and coded into a numeric score (or scores) is a more difficult question. A numeric score can always be created. The question is how much valuable information is lost in the process. We call this process the hardening of information, and we will discuss it below.

A firm’s sales revenue or their stock return is an example of hard information. There is wide agreement as to what it means for a firm to have had sales of $10 million last year or the firm’s stock price to have risen by 10%. However, if we say the owner of the firm is trustworthy, there is less agreement about what this means and why it is important. Definitions of trustworthiness may differ across agents and the context under which one evaluates trustworthiness may be relevant.

This distinction is reminiscent of the difference between the approach we take when we teach first-year graduate econometrics and the way empirical research is done in practice. In Econometrics 101, we assume we know the dependent variable, the independent variables, and the functional form. The only unknown is the precise value of the coefficients. In an actual research project, we have priors about the relationships between important economic concepts, but we don’t know how to measure precisely the concepts behind the dependent and independent variables nor the functional form. Only after collecting the data and examining the preliminary results do we understand how the variables are related. This leads us to modify our hypothesis and often requires the collection of additional data or a change in our interpretation of the data. The research process helps us see and understand the missing context.

A typical example is the relationship-based loan officer. The loan officer has a history with the borrower and, based on a multitude of personal contacts, has built up an impression of the borrower’s honesty, creditworthiness, and likelihood of defaulting. Based on this view of the borrower and the loan officer’s experience, the loan is approved or denied. Uzzi and Lancaster (2003) provide detailed descriptions of interactions between borrowers and loan officers.

In Bikhchandani, Hirshleifer, and Welch’s (1992) study of informational cascades, they model sequential decisions in which agents see the (binary) decisions of prior agents, but not the information upon which the decision is made. This reduction (hardening) of information leads to agents ignoring their own (soft) information and following the crowd.

The authors’ description of trade credit markets during this period is strikingly similar to Nocera’s (2013) description of the U.S. consumer lending market of the 1950s.

The precursor to Dun and Bradstreet, the Mercantile Agency, was founded in 1841 ( Carruthers and Cohen 2010b). The precursor to Standard and Poor’s, the History of Railroads and Canals in the United States by Henry Poor, was founded in 1860.

Carruthers and Cohen (2010b, pp. 5–6) paraphrase Cohen (1998) saying, “… what went into credit evaluations was a variable and unsystematic collection of facts, judgments and rumors about a firm, its owner’s personality, business dealings, family and history… . what came out was a formalized, systematic and comparable rating of creditworthiness … .”.

CRSP began with a question from Louis Engel, a vice president at Merrill Lynch, Pierce, Fenner, and Smith. He wanted to know the long-run return on equities. He turned to Professor James Lorie at the University of Chicago, who didn’t know either but was willing to find out for them (for a $50,000 grant). The process of finding out led to the creation of the CRSP stock return database. That neither investment professionals nor academic finance knew the answer to this question illustrates how far we have come in depending on hard information, such as stock returns. Professor Lorie described the state of research prior to CRSP in his 1965 Philadelphia address: “Until recently almost all of this work was by persons who knew a great deal about the stock market and very little about statistics. While this combination of knowledge and ignorance is not so likely to be sterile as the reverse—that is, statistical sophistication coupled with ignorance of the field of application—it nevertheless failed to produce much of value.” In addition to CRSP, he talks about another new data set: Compustat (sold by the Standard Statistics Corporation), which had 60 variables from the firm’s income statement and balance sheet.

For small business loans, the size of the fees is independent of the size of the loan. Thus, the percentage fee declines with the loan size ( Petersen and Rajan 1994). The lowering of transactions costs, especially through digital delivery and automation, can be particularly important in microfinance lending, where the loan amounts are very small ( Karlan et al. 2016).

Causation can also run in the opposite direction. Greater competition, which can arise from deregulation, for example, increases the pressure to lower costs and thus to transform the production process to depend more on hard information.

Subprime mortgage loans are less standardized and more informationally sensitive than normal mortgages, because, sometimes, borrowers are not able to provide full disclosure of their income ( Mayer, Pence, and Sherlund 2009).

Karolyi (2017) finds that the relationship lies with individuals, not firms. After exogenous changes in leadership (the death or retirement of a CEO), firms are significantly more likely to switch to lenders with whom the new CEO has a relationship (see also Degryse et al. 2013). This is one reason why firms that rely on soft information in securing debt capital care about the fragility of the banks from which they borrow ( Schwert 2017).

Using a randomized control trial, Paravisini and Schoar (2015) evaluate the adoption of credit scores in a small business lending setting. They find that using credit scores improves the productivity of credit committees (e.g., less time is spent on each file).

Friedman (1990) argues that this is one advantage of a market versus a planned economy. He argues that all information relevant to a consumer or producer about the relative supply of a good is contained in the price. Thus, it is not necessary for a supplier to know whether the price has risen because demand has risen or supply has fallen. The supplier only needs to know that the price has risen, and this will dictate her decision of how much to increase production. Friedman’s description of a market economy depicts a classic hard information environment.

In the 2008 financial crisis, a large number of investment-grade securities defaulted. The magnitude of the defaults suggested a problem with the rating process (see Benmelech and Dlugosz 2009a, 2009b). Observers in industry, academics, and government suggested possible sources of the problem and potential solutions. Intriguingly, the defaults experience was very different in the corporate bond market (debt of operating companies) compared with the structured finance market (e.g., RMBS). Defaults in the corporate bond market spiked in 2009, but the peak is not drastically different than the peak in prior recessions (see Vazza and Kraemer 2016, chart 1). The peak in defaults in the structured finance market in 2009 was dramatically larger (see South and Gurwitz 2015, chart 1). That the collapse of the housing market hit the structured finance securities more aggressively suggests that a part of the problem with the rating process resides uniquely in the structured finance segment of the market. For an operating company, a low cost of capital is an advantage, but not its only or predominant source of competitive advantage. For a securitization structure, a lower cost of capital is one of its few sources of a “competitive advantage.” Thus, a bank might change which mortgages are placed into a securitization if this change would increase the fraction of the securitization rated AAA and thus lower the cost of capital. An auto-manufacturing firm is unlikely to close plants or close down a division solely to achieve a higher credit rating. The costs of altering the business to improve a credit score are higher and the benefits are (relatively) lower for an operating firm. This may be why we saw relatively fewer defaults in the corporate bond sector relative to the securitized sector. This issue prompted the credit rating agencies to consider different rating scales for structured finance versus corporate debt ( Kimball and Cantor 2008).

Hu, Huang, and Simonov (2017) see the same behavior in the market for individual loans. The theoretical importance of nonlinearities in the mapping of inputs (hard information) to outputs (decisions) is discussed in Jensen (2003). In his examples, incentives to misstate one’s information are smaller if the payoff function is linear. Small changes in the reported information have only small changes in the manager’s payoff.

There may also be strategic reasons to avoid a transparent mapping between the numbers and the credit rating. The business model of credit rating agencies relies on market participants being unable to replicate the ratings at a lower cost than the agency. If the mapping were a direct function of easily accessible inputs (e.g., the income statement and balance sheet) and nothing else, some clever assistant finance or accounting professor would figure out the function. This is one reason that the early credit reporting agencies publicly released only a fraction of their information in the form of a credit score. For additional fees, users could review a more complete report ( Carruthers and Cohen 2010a, 2014).

Guiso, Sapienza, and Zingales (2013) find that borrowers feel less obligated to repay an underwater mortgage if the mortgage has been sold in the marketplace.

Brown et al. (2012) find that loan officers use discretion to smooth credit, but there is limited information in discretionary changes. Degryse et al. (2013) provide evidence that soft information helps predict defaults over public information (e.g., financial statements), but discretionary actions do not predict default. Gropp, Gruendl, and Guettler (2012) show that the use of discretion by loan officers does not affect the performance of the bank portfolio. Puri, Rocholl, and Steffen (2011) document the widespread use of discretion inside a German savings bank but find no evidence that loans approved based on discretion perform differently than those approved not based on discretion. Cerqueiro, Degryse, and Ongena (2011) find that discretion seems to be important in the pricing of loans but that it only plays a minor role in the decision to lend.

This is an imperfect solution when the loan officer has an incentive and the ability to manipulate the inputs, just as the borrower might. The loan officers in Berg, Puri, and Rocholl (2016) work for a bank that uses an internal credit score to evaluate loans. They show that loan officers repeatedly enter new values of the variables into the system until a loan is approved. Not only are they able to get loans approved that were originally rejected, but they also learn the model’s cutoffs and thus what is required for loan approval. These results suggest that even hard information decision-making algorithms, which are transparent and depend on data subject to the control of either participant (local decision maker or the target of the decision), are subject to the Lucas critique (see Section 2.4).

A variety of possible costs are embedded in the transmission of information in an organization. Theories of costly communication, where soft information may be more costly to communicate across hierarchies ( Becker and Murphy 1992 Radner 1993 Bolton and Dewatripont 1994) theories of loss of incentives to collect, process, and use soft information like in Aghion and Tirole (1997), because of the anticipation of being overruled by one’s superior and strategic manipulation of information like in Crawford and Sobel (1982) and Dessein (2002), offer three different, but related, explanations. In all these theories, those who send the information make it noisier and less verifiable if their preferences are not aligned with those who are receiving it and, ultimately, have the final authority to make the decision.

Rajan and Zingales (1998) argue that ownership is not the only way to allocate power in an organization. Another, and, in some cases, a better way, is through access. Access is the ability to work with or use a critical resource, though not necessarily a physical resource that can be owned. In financial institutions (and increasingly in other firms), this resource is often information.

Although these papers all examine geographical distance, they are different in nature. Petersen and Rajan (2002) document that distance between lenders and borrowers increased because of improvements in information technology. Degryse and Ongena (2005) study the relationship between the competitiveness of the lending market and the distance between the borrower, their lender, and other potential competitors (banks). Mian (2006) suggests that greater distance not only decreases the incentives of a loan officer to collect soft information but also makes it more costly to produce and communicate soft information. DeYoung, Glennon, and Nigro (2008) document the relationship between the use of hard information using credit scoring technologies and an increase in borrower-lender distances. Finally, Agarwal and Hauswald (2010) study the effects of distance on the acquisition and use of private information in informationally opaque credit markets. They show that borrower proximity facilitates the collection of soft information, which is reflected in the bank’s internal credit assessment.

Starting in the early eighties, the number of banks in the United States began declining by over 50%, with most of the fall occurring in the first decade ( Petersen and Rajan 2002, figure 4 Berger and Bouwman 2016, figure 8.1). The decline in the total number of banks is completely driven by the decline of small banks defined by those with gross total assets less than $1 billion. The number of large banks has grown. The decline in small banks is driven, in part, by the technology and the shift to hard information and also by deregulation (Strahan and Kroszner 1999). The growing reliance on hard information and automated decision-making and the associated cost savings created pressure to reduce regulations on bank expansion. In turn, diminishing regulatory restrictions raised the value of capturing cost savings by shifting to production processes that rely on hard information and enabled greater economies of scale.

Even in markets that we think are dominated by hard information and thus where we would expect distance not to be relevant, research has sometimes found a preference for local investments. Mutual fund managers tend to hold a higher concentration in shares of local firms, because access to soft information of local firms is cheaper ( Coval and Moskowitz 1999, 2001). The effect is strongest in small and highly levered firms.

If the local underwriters have soft information that nonlocal underwriters do not have, and they can thus sell the bonds at higher prices, they should be able to extract larger fees. Oddly, they do not. Local underwriters charge lower fees relative to nonlocal underwriting, suggesting that local competition limits their pricing power.

They use the measure of distance between banks and borrowers from Petersen and Rajan (2002) to classify whether industries are hard- or soft-information intensive. Industries where the distance between borrowers and lenders is larger are classified as hard information environments.

A plant may be located far away in terms of geographical distance, but monitoring may be easier when direct flights are available between the cities in which headquarters and plants are located.

Analogously, firms attempted to alter the financial information they reported in response to the introduction of credit ratings in an effort to increase their access to credit in the late nineteenth century ( Carruthers and Cohen 2010b, footnote 36).

The literature began by simply counting positive and negative words, which proved to be more complicated than one would have initially guessed. The language of finance is not as simple as we think ( Longhran and McDonald 2011). For example, the sentence “The Dell Company has 100 million shares outstanding” would have been classified as an extremely positive sentence by the early dictionaries, since “company,” “share,” and “outstanding” are coded as positive words (Engelberg 2008). The Hoberg and Phillips (2010) method is similar, but they are interested in a very different question. They use text-based analysis of firms’ 10-Ks to measure the similarities of firms involved in mergers and thus predict the success of the mergers. Mayew and Venkatachalam (2012) took this idea one step further and examined the information embedded in the tone of managers’ voices during earning calls.

Loss of information is not only due to the effect of hardening the information. A change in the compensation structure of agents may also affect the use of information. In a controlled experiment, Agarwal and Ben-David (2018) study the impact that changing the incentive structure of loan officers to prospect new applications has on the volume of approved loans and default rates. They find that after the change, loan officers start relying more on favorable hard information and ignoring unfavorable soft information. The results highlight how incentives dictate not just what information is collected but also what role it plays in the decision. Another form of loss of information is due to the portability of soft information. For example, Drexler and Schoar (2014) show that when loan officers leave, they generate a cost to the bank, because leaving affects the borrower-lender relationship. As the departing loan officers have no incentives to voluntarily transfer the soft information, borrowers are less likely to receive new loans from the bank in their absence.

Appearance also played a role in the early credit reports collected by the Mercantile Agency. The agency’s instructions to their agents stated “… give us your impressions about them, judging from appearances as to their probable success, amount of stock, habits, application to business, whether they are young and energetic or the reverse …” ( Carruthers and Cohen 2010b, p. 12).

Mollick (2014, p. 2) defines crowdfunding as “… the efforts by entrepreneurial individuals and groups … to fund their ventures by drawing on relatively small contributions from a relatively large number of individuals using the internet, without standard financial intermediaries.”

Participants contribute capital in exchange for a product or so they may participate in supporting an event or creative endeavor. The first is a form of trade credit (prepaying for a product) and in most examples is more akin to market research than equity funding, since the existence and the quality of the product are often uncertain.

Newman (2011) has raised the concern that “… crowdfunding could become an efficient, online means for defrauding the investing public … .”

Investors “… rely on highly visible (but imperfect) proxies for quality such as accumulated capital, formal education, affiliation with top accelerator programs, advisors, etc.” ( Catalini and Hui 2018, p. 1).

The Mercantile Agency, the precursor to Dun and Bradstreet’s, also worried about the tendency of some subscribers, who had purchased access to their reports, relying too heavily on the ratings, as opposed to visiting their offices and inspecting the underlying data ( Carruthers and Cohen 2010b).

The evidence that human brokers factor their client’s characteristics into the investment decision is not reassuring. A retail investor’s asset allocation significantly depends more on who their broker is (e.g., broker fixed effects) than the investors own characteristics (e.g., risk tolerance, age, financial knowledge, investment horizon, and wealth see Foerster et al. 2017).

Algorithms are written by humans, so they may embody the same behavioral biases that human advisors have ( O’Neil 2016, D’Acunto, Prabhala, and Rossi 2017) as well as the biases of those who design the algorithms or which may be inherent in the data ( O’Neil 2016).


Watch the video: Z-Factor - Thorns