“Private” is a word that appeared in the late 14th century that came from Latin “privatus”, “set apart, belonging to oneself” (not to the state), used in contrast to publicus, communis; originally pp. stem of privare “to separate, deprive,” from privus”one’s own, individual. In short, it points to something that you have sovereign over. Thus, the notion of the “right of privacy” as “self-control right” is not too far away from its original meaning.
2. Right to Privacy in legal literature
On the other hand, in the legal literature, right to privacy is often referred to as “right to be let alone.” This comes from the Warren and Brandeis’s famous paper “Right to Privacy” that quotes Justice Cooley’s “Law of Torts”. The phrase however is often mistaken to be pointing to the right to the solitude, which I think is not the case. Let me quote from Justis Cooley’s paper.
Personal immunity. The right to one’s person may be said to be a right of complete immunity: to be let alone. The corresponding duty is, not to inflict an injury, and not, within such proximity as might render it successful, to attempt the infliction of an injury. In this particular the duty goes beyond what is required in most cases; for usually an unexecuted purpose or an unsuccessful attempt is not noticed. But the attempt to commit a battery involves many elements of injury not always present in breaches of duty; it involves usually an insult, a putting in fear, a sudden call upon the energies for prompt and effectual resistance. There is very likely a shock to the nerves, and the peace and quiet of the individual is disturbed for a period of greater or less duration. There is consequently abundant reason in support of the rule of law which makes the assault a legal wrong, even though no battery takes place. Indeed, in this case the law goes still further and makes the attempted blow a criminal offense also. 
So, “right to be let alone” is synonymous to the “personal immunity”. If you read what follows, you will see that “person” includes not only the physical body but also the soul. In other words, “Right to be let alone” is “the immunity of one’s body and soul” and is very far from the impression that non-legal folks gets from the phrase “right to be let alone.”
Warren and Brandeis follows this and they define the right to privacy as a very basic human right of sovereignty over one’s private property, both tangible and intangible, under the common law, that such rights to free speech, copyright, etc. are derivative right on the privacy. The consumer privacy bill of right that the Obama legislation put forward in 2012  follows this as well. Indeed, we can almost interpret right to privacy as the right to personal freedom also known as liberty.
Warren and Brandeis then goes on to discuss the difference between the right to privacy and the right to property, copyright, slander and of libel, and discusses why being intentional is not necessary in its infringement, and under what condition such right is limited . It is a short paper that I highly recommend you to read. The online version can be found at the MIT site.
3. Right to privacy in a narrower sense and confusion arising from it
Thus, both direction derived the same result. Right to privacy is the sovereign right to over one’s private tangible and intangible property including one’s body and soul. In another word, it is a basic human right of personal freedom (liberty).
Note however, much of it actually can be protected by more specific laws, and where there is a defined law, it should be dealt with it. The remainder after all those area covered by specific laws seems to be often referred to as the right to privacy in narrow sense. This is a negative way of defining it. Trying to do it in a positive statement is utterly impossible. This is one of the reason why it is so hard to define the right to privacy in narrower sense. I think “right to control personal image through the control of the use and release of personal information” covers good part of it, but not all. Note that personal data protection is just a tool to protect this right, and not something we should be aiming at. It is the privacy that needs to be protected, not the data per se.
Further, the fact those specific rights deducted differs from a jurisdiction to another. This, I feel, is one of the reason that making it difficult to bring different jurisdiction to the same page.Whereas the free world probably has very similar notion on the personal freedom (liberty), which is depicted as the rectangle in the following figure, the remainder after removing the area covered by specific laws are all different.
In addition, in some jurisdiction like EU, Right to respect for private and family life is defined. This is yet another subset of the right to privacy in the broader sense, but due to the similarity in the wording, it often is referred to as the right to privacy as well adding further confusion.
Often, it is said that “privacy cannot be defined.” This is not true. Defining it in the narrow sense using “positive expression” is almost impossible, but it can be defined in the “negative expression.” However, when we get to it in the concrete term, the right to privacy in the narrower sense in concrete differs from jurisdiction to jurisdiction resulting in tension and confusion. It is better to define it in the abstract term, and to make the concrete guidance, extract the subset such as “self-image control right” and put it into a statute though it may overlap with existing laws. In addition, it would probably help to create a logical model of what constitutes a violation / infringement etc. as a clearer guidance. I have started it here, which is a very early work in progress, but I would much appreciate your comments on it as well.
 I am not a lawyer, so I feel a little inappropriate and intimidated to refer to these document, but they are too important not to mention and analyse.
 Warren and Brandeis, “The Right to Privacy”, Harvard Law Review, Vol. IV December 15, 1890 No. 5
 Thomas McIntyre Cooley, “Law of Torts”, Callaghan, 1888
 「These considerations lead to the conclusion that the protection afforded to thoughts, sentiments, and emotions, expressed through the medium of writing or of the arts, so far as it consists in preventing publication, is merely an instance of the enforcement of the more general right of the individual to be let alone. 」
 This notion of equality being the identity of the concession of certain quantity of sovereignty over oneself can also be found in the speech of Enjolras in Les Misrables by Victor Hugo as follows:
From a political point of view, there is but a single principle; the sovereignty of man over himself. This sovereignty of myself over myself is called Liberty. Where two or three of these sovereignties are combined, the state begins. But in that association there is no abdication. Each sovereignty concedes a certain quantity of itself, for the purpose of forming the common right. This quantity is the same for all of us. This identity of concession which each makes to all, is called Equality. (from Victor Hugo, “Les Miserables” vol. V, Book First, Chapter V. )
Having been working on Digital Identity since 2000.
Co-author of various identity related specifications like OpenID Connect, JSON Web Token.
Chair of the OpenID Foundation (2011-)
Vice Chair of the OpenID Foundation (2010),
Founder of OpenID Foundation Japan (2008-),
Trustee of Kantara Initiative (2009-).