Basic idea: “geometry asks, “Given a geometric object X, what is its group of symmetries?” Representation theory reverses the question to “Given a group G, what objects X does it act on?” and attempts to answer this question by classifying such X up to isomorphism.” from https://math.berkeley.edu/~teleman/math/RepThry.pdf
A Lie group is in abstract terms a manifold, which obeys the group axioms. A representation is a special type of map $R$ from this manifold to the linear operators of some vector space. The map must obey the condition $$ R(gh) = R(g) R(h), $$ where $g$ and $h$ are elements of the group. This means the map must preserve the product structure of the group and the mathematically notion for such a map is homomorphism.
In practice a representation is a map that maps each element of the abstract group onto a matrix. (Matrices are linear operators over a vector space.)
(There are other representations, where the group elements aren't given as matrices, but in physics matrix representations are most of the time sufficient).
Take note that it is possible to introduce a more general notion, called a realisation of the group. A realisation maps the group elements onto the (not necessarily linear) operators over an arbitrary (no necessary vector-)space. However, in physics we usually only deal with representations, because our physical objects live in vector spaces.
One way to label representations is by using the Casimir Operators.
Another possibility is given by the Weyl character formula. This formula allows to compute the “character” of a group. A “character” is a function that yields a number for each group element. Thus, one can compute which representation one is dealing with by computing this “character” function. If two representations that could be constructed very differently are actually the same their character functions are the same.