|
@@ -23,53 +23,51 @@
|
|
|
% file structure/format <-> datatypes. länger beschreiben: e.g. File formats to store dna
|
|
% file structure/format <-> datatypes. länger beschreiben: e.g. File formats to store dna
|
|
|
% 3.2.1 raus
|
|
% 3.2.1 raus
|
|
|
|
|
|
|
|
-\section{Compression aproaches}
|
|
|
|
|
|
|
+\section{Compression Aproaches}
|
|
|
The process of compressing data serves the goal to generate an output that is smaller than its input \cite{dict}.\\
|
|
The process of compressing data serves the goal to generate an output that is smaller than its input \cite{dict}.\\
|
|
|
-In many cases, like in gene compressing, the compression is idealy lossless. This means it is possible with any compressed data, to receive the full information that were available in the origin data, by decompressing it. Lossy compression on the other hand, might excludes parts of data in the compression process, in order to increase the compression rate. The excluded parts are typicaly not necessary to transmit the origin information. This works with certain audio and pictures files or with network protocols like \ac{UDP} which are used to transmit video/audio streams live \cite{rfc-udp, cnet13}.\\
|
|
|
|
|
-For storing \acs{DNA} a lossless compression is needed. To be preceice a lossy compression is not possible, because there is no unnecessary data. Every nucleotide and its exact position is needed for the sequence to be complete and usefull.\\
|
|
|
|
|
|
|
+In many cases, like in gene compressing, the compression is idealy lossless. This means, it is possible to receive the full information that was available in the origin data by decompressing any kind of compressed data. Lossy compression on the other hand might exclude parts of data in the compression process, in order to increase the compression rate. The excluded parts are typically not necessary to transmit the original information. This works with certain audio and picture files, or with network protocols like \ac{UDP} which are used to transmit video/audio streams live \cite{rfc-udp, cnet13}.\\
|
|
|
|
|
+For storing \acs{DNA} a lossless compression is needed. To be precise a lossy compression is not possible, because there is no unnecessary data. Every nucleotide and its exact position are needed for the sequence to be complete and useful.\\
|
|
|
Before going on, the difference between information and data should be emphasized.\\
|
|
Before going on, the difference between information and data should be emphasized.\\
|
|
|
% excurs data vs information
|
|
% excurs data vs information
|
|
|
-Data contains information. In digital data clear, physical limitations delimit what and how much of something can be stored. A bit can only store 0 or 1, eleven bit can store up to $2^{11}$ combinations of bit and a 1 \acs{GB} drive can store no more than 1 \acs{GB} data. Information on the other hand, is limited by the way how it is stored. What exactly defines informations, depends on multiple factors. The context in which information is transmitted and the source and destination of the information. This can be in form of a signal, transfered from one entity to another or information that is persisted so it can be obtained at a later point in time.\\
|
|
|
|
|
|
|
+Data contains information. In digital data , clear physical limitations delimit what and how much of something can be stored. A bit can only store 0 or 1, eleven bit can store up to $2^{11}$ combinations of bit and a 1~\acs{GB} drive can store no more than 1~\acs{GB} of data. Information on the other hand is limited by the way it is stored. What exactly defines information, depends on multiple factors. The context in which information is transmitted and the source and destination of the information. This can be in form of a signal, transferred from one entity to another, or information that is persisted, so it can be obtained at a later point in time.\\
|
|
|
% excurs information vs data
|
|
% excurs information vs data
|
|
|
-For the scope of this work, information will be seen as the type and position of nucleotides, sequenced from \acs{DNA}. To get even more preceise, it is a chain of characters from a alphabet of \texttt{A, C, G, and T}, since this is the \textit{de facto} standard for digital persistence of \acs{DNA} \cite{isompeg}.
|
|
|
|
|
-The boundaries of information, when it comes to storing capabilities, can be illustrated by using the example mentioned above. A drive with the capacity of 1 \acs{GB} could contain a book in form of images, where the content of each page is stored in a single image. Another, more resourceful way would be storing just the text of every page in \acs{UTF}-16 \cite{isoutf}. The information, the text would provide to a potential reader would not differ. Changing the text encoding to \acs{ASCII} and/or using compression techniques would reduce the required space even more, without loosing any information.\\
|
|
|
|
|
|
|
+For the scope of this work, information will be seen as the type and position of nucleotides, sequenced from \acs{DNA}. To be even more precise, it is a chain of characters from an alphabet of \texttt{A, C, G, and T}, since this is the \textit{de facto} standard for digital persistence of \acs{DNA} \cite{isompeg}.
|
|
|
|
|
+When it comes to storing capabilities, the boundaries of information, can be illustrated by using the example mentioned above. A drive with the capacity of 1~\acs{GB} could contain a book in form of images, where the content of each page is stored in a single image. Another, more resourceful way would be storing just the text of every page in \acs{UTF}-16 \cite{isoutf}. The information the text would provide to a potential reader would not differ. Changing the text encoding to \acs{ASCII} and/or using compression techniques would reduce the required space even more, without losing any information.\\
|
|
|
% excurs end
|
|
% excurs end
|
|
|
-For \acs{DNA} a lossless compression is needed. To be precise a lossy compression is not possible, because there is no unnecessary data. Every nucleotide and its position is needed for the sequenced \acs{DNA} to be complete. For lossless compression two mayor approaches are known: the dictionary coding and the entropy coding. Methods from both fields, that aquired reputation, are described in detail below \cite{cc14, moffat20, moffat_arith, alok17}.\\
|
|
|
|
|
|
|
+For \acs{DNA} a lossless compression is needed. To be precise a lossy compression is not possible, because there is no unnecessary data. Every nucleotide and its position is needed for the sequenced \acs{DNA} to be complete. For lossless compression, two mayor approaches are known: the dictionary coding and the entropy coding. Methods from both fields, that acquired reputation, are described in detail below \cite{cc14, moffat20, moffat_arith, alok17}.\\
|
|
|
|
|
|
|
|
-\subsection{Dictionary coding}
|
|
|
|
|
|
|
+\subsection{Dictionary Coding}
|
|
|
|
|
|
|
|
\label{k4:dict}
|
|
\label{k4:dict}
|
|
|
-Dictionary coding, as the name suggest, uses a dictionary to eliminate redundand occurences of strings. Strings are a chain of characters representing a full word or just a part of it. For a better understanding this should be illustrated by a short example:
|
|
|
|
|
|
|
+Dictionary coding, as the name suggest, uses a dictionary to eliminate redundant occurrences of strings. Strings are a chain of characters representing a full word or just a part of it. For a better understanding, this should be illustrated by a short example:
|
|
|
% demo substrings
|
|
% demo substrings
|
|
|
-Looking at the string 'stationary' it might be smart to store 'station' and 'ary' as seperate dictionary enties. Which way is more efficient depents on the text that should get compressed.
|
|
|
|
|
|
|
+Looking at the string 'stationary' it might be smart to store 'station' and 'ary' as separate dictionary entries. Which way is more efficient depends on the text that should get compressed.
|
|
|
% end demo
|
|
% end demo
|
|
|
-The dictionary should only store strings that occur in the input data. Also storing a dictionary in addition to the (compressed) input data, would be a waste of resources. Therefore the dicitonary is part of the text. Each first occurence is left uncompressed. Each occurence of a string, after the first one, points either to to its first occurence or to the last replacement of its occurence.\\
|
|
|
|
|
-\ref{k4:dict-fig} illustrates how this process is executed. The bar on top of the figure, which extends over the full widht, symbolizes any text. The squares inside the text are repeating occurences of text segments.
|
|
|
|
|
-In the dictonary coding process, the square annotated as \texttt{first occ.} is added to the dictionary. \texttt{second} and \texttt{third occ.} get replaced by a structure \texttt{<pos, len>} consisting of a pointer to the position of the first occurence \texttt{pos} and the length of that occurence \texttt{len}.
|
|
|
|
|
-The bar at the bottom of the figure shows how the compressed text for this example would be structured. The dotted lines would only consist of two bytes, storing position and lenght, pointing to \texttt{first occ.}. Decompressing this text would only require to parse the text from left to right and replace every \texttt{<pos, len>} with the already parsed word from the dictionary. This means jumping back to the parsed position stored in the replacement, reading for as long as the length dictates, copying the read section, jumping back and pasting the section.\\
|
|
|
|
|
|
|
+The dictionary should only store strings that occur in the input data. Also storing a dictionary in addition to the (compressed) input data, would be a waste of resources. Therefore the dictionary is part of the text. Each first occurrence is left uncompressed. Each occurrence of a string, after the first one, points either to to its first occurrence or to the last replacement of its occurrence. Which method is used depends on the algorithm.\\
|
|
|
|
|
+\ref{k4:dict-fig} illustrates how this process is executed. The bar on top of the figure, which extends over the full width, symbolizes any text. The squares inside the text are repeating occurrences of text segments.
|
|
|
|
|
+In the dictionary coding process, the square annotated as \texttt{first occ.} is added to the dictionary. \texttt{Second} and \texttt{third occ.} get replaced by a structure \texttt{<pos, len>} consisting of a pointer to the position of the first occurrence \texttt{pos} and the length of that occurrence \texttt{len}.
|
|
|
|
|
+The bar at the bottom of the figure shows how the compressed text for this example would be structured. The dotted lines would only consist of two bytes, storing position and lenght, pointing to \texttt{first occ.}. Decompressing this text would only require parsing the text from left to right and to replace every \texttt{<pos, len>} with the already parsed word from the dictionary. This means jumping back to the parsed position stored in the replacement, reading for as long as the length dictates, copying the read section, jumping back and pasting the section.\\
|
|
|
% offsets are volatile when replacing
|
|
% offsets are volatile when replacing
|
|
|
|
|
|
|
|
\begin{figure}[H]
|
|
\begin{figure}[H]
|
|
|
\centering
|
|
\centering
|
|
|
\includegraphics[width=15cm]{k4/dict-coding.png}
|
|
\includegraphics[width=15cm]{k4/dict-coding.png}
|
|
|
- \caption{Schematic sketch, illustrating the replacement of multiple occurences done in dictionary coding.}
|
|
|
|
|
|
|
+ \caption{Schematic sketch, illustrating the replacement of multiple occurrences done in dictionary coding.}
|
|
|
\label{k4:dict-fig}
|
|
\label{k4:dict-fig}
|
|
|
\end{figure}
|
|
\end{figure}
|
|
|
|
|
|
|
|
\label{k4:lz}
|
|
\label{k4:lz}
|
|
|
\subsubsection{The LZ Family}
|
|
\subsubsection{The LZ Family}
|
|
|
-The computer scientist Abraham Lempel and the electrical engineere Jacob Ziv created multiple algorithms that are based on dictionary coding. They can be recognized by the substring \texttt{LZ} in its name, like \texttt{LZ77 and LZ78} which are short for Lempel Ziv 1977 and 1978 \cite{lz77}. The number at the end indictates when the algorithm was published. Today LZ78 is widely used in unix compression solutions like gzip and bz2. Those tools are also used in compressing \ac{DNA}.\\
|
|
|
|
|
|
|
+The computer scientist Abraham Lempel and the electrical engineer Jacob Ziv created multiple algorithms that are based on dictionary coding. They can be recognized by the substring \texttt{LZ} in their name; like \texttt{LZ77 and LZ78} which are short for Lempel Ziv 1977 and 1978 \cite{lz77}. The number at the end indicates when the algorithm was published. Today, members of the LZ family are widely used in compression implementations like rar, zip, gzip and bz2 \cite{rfcgzip}. Some of those are also used to compress \ac{DNA}.\\
|
|
|
|
|
|
|
|
-\acs{LZ77} basically works, by removing all repetition of a string or substring
|
|
|
|
|
-and replacing them with information where to find the first occurence and how long it is. Lempel and Ziv described restricted the pointer in a range to integers. Today a pointer, length pair is typically stored in two bytes. One bit is reseverd to indicate that the next 15 bit are a position, lenght pair. More than 8 bit are available to store the pointer and the rest is reserved for storing the length. Exact amounts depend on the implementation \cite{rfc1951, lz77}.
|
|
|
|
|
|
|
+\acs{LZ77} basically works, by removing all repetitions of a string or substring and replacing them with information where to find the first occurrence and how long it is. The distance between the first occurrence and a replacement is limited, because each pointer has a static amount of storage available. A pointer, length pair is typically stored in two bytes. One bit is reseverd to indicate that the next 15 bit are a position, lenght pair. More than 8 bit are available to store the pointer and the rest is reserved for storing the length. Exact amounts depend on the implementation \cite{rfc1951, lz77}.
|
|
|
% rewrite and implement this:
|
|
% rewrite and implement this:
|
|
|
-%This method is limited by the space a pointer is allowed to take. Other variants let the replacement store the offset to the last replaced occurence, therefore it is harder to reach a position where the space for a pointer runs out.
|
|
|
|
|
-
|
|
|
|
|
-Unfortunally, implementations like the ones out of LZ Family, do not use probabilities to compress and are therefore not in the main scope for this work. To strenghten the understanding of compression algortihms this section will remain. Also it will be usefull for the explanation of a hybrid coding method, which will get described later in this chapter.\\
|
|
|
|
|
|
|
+%This method is limited by the space a pointer is allowed to take. Other variants let the replacement store the offset to the last replaced occurrence, therefore it is harder to reach a position where the space for a pointer runs out.
|
|
|
|
|
|
|
|
|
|
+Unfortunally, implementations like the ones out of LZ Family, do not use probabilities to compress and are therefore not in the main scope for this work. To strengthen the general understanding of compression algortihms and because it is a part of hybrid coding implementations, this section remains.\\
|
|
|
|
|
|
|
|
\subsection{Shannons Entropy}
|
|
\subsection{Shannons Entropy}
|
|
|
-The founder of information theory Claude Elwood Shannon described entropy and published it in 1948 \cite{Shannon_1948}. In this work he focused on transmitting information. His theorem is applicable to almost any form of communication signal. His findings are not only usefull for forms of information transmition.
|
|
|
|
|
|
|
+The founder of information theory Claude Elwood Shannon described entropy and published his work in 1948 \cite{Shannon_1948}. Here, he focused on transmitting information. His theorem is applicable to almost any form of communication signal. His findings are not only useful for forms of information transmission.
|
|
|
|
|
|
|
|
% todo insert Fig. 1 shannon_1948
|
|
% todo insert Fig. 1 shannon_1948
|
|
|
\begin{figure}[H]
|
|
\begin{figure}[H]
|
|
@@ -79,10 +77,10 @@ The founder of information theory Claude Elwood Shannon described entropy and pu
|
|
|
\label{k4:comsys}
|
|
\label{k4:comsys}
|
|
|
\end{figure}
|
|
\end{figure}
|
|
|
|
|
|
|
|
-Altering \ref{k4:comsys} would show how this can be applied to other technology like compression. The Information source and destination are left unchanged, one has to keep in mind, that it is possible that both are represented by the same physical actor.\\
|
|
|
|
|
|
|
+Altering \ref{k4:comsys} would show how this can be applied to other technology like compression. The information source and destination are left unchanged; one has to keep in mind, it is possible that both are represented by the same physical actor.\\
|
|
|
Transmitter and receiver would be changed to compression/encoding and decompression/decoding. Inbetween those two, there is no signal but instead any period of time \cite{Shannon_1948}.\\
|
|
Transmitter and receiver would be changed to compression/encoding and decompression/decoding. Inbetween those two, there is no signal but instead any period of time \cite{Shannon_1948}.\\
|
|
|
|
|
|
|
|
-Shannons Entropy provides a formula to determine the 'uncertainty of a probability distribution' in a finite field.
|
|
|
|
|
|
|
+Shannon's Entropy provides a formula to determine the ``uncertainty of a probability distribution'' in a finite field.
|
|
|
|
|
|
|
|
\begin{equation}\label{eq:entropy}
|
|
\begin{equation}\label{eq:entropy}
|
|
|
%\resizebox{.9 \textwidth}{!}
|
|
%\resizebox{.9 \textwidth}{!}
|
|
@@ -99,7 +97,7 @@ Shannons Entropy provides a formula to determine the 'uncertainty of a probabili
|
|
|
% \label{k4:entropy}
|
|
% \label{k4:entropy}
|
|
|
%\end{figure}
|
|
%\end{figure}
|
|
|
|
|
|
|
|
-He defined entropy as shown in figure \eqref{eq:entropy}. Let X be a finite probability space. Then $x\in X$ are possible final states of an probability experiment over X. Every state that actually occurs, while executing the experiment generates information which is meassured in \textit{Bits} with the part of the equation displayed in \ref{eq:info-in-bit} \cite{delfs_knebl,Shannon_1948}:
|
|
|
|
|
|
|
+He defined entropy as shown in figure \eqref{eq:entropy}. Let X be a finite probability space. Then $x\in X$ are possible final states of a probability experiment over X. Every state that actually occurs, while executing the experiment generates information which is measured in binary digits \textit{bits} for short with the part of the equation displayed in \ref{eq:info-in-bit} \cite{delfs_knebl,Shannon_1948}:
|
|
|
|
|
|
|
|
\begin{equation}\label{eq:info-in-bit}
|
|
\begin{equation}\label{eq:info-in-bit}
|
|
|
log_2(\frac{1}{prob(x)}) \equiv - log_2(prob(x)).
|
|
log_2(\frac{1}{prob(x)}) \equiv - log_2(prob(x)).
|
|
@@ -112,6 +110,8 @@ He defined entropy as shown in figure \eqref{eq:entropy}. Let X be a finite prob
|
|
|
% \label{f4:info-in-bit}
|
|
% \label{f4:info-in-bit}
|
|
|
%\end{figure}
|
|
%\end{figure}
|
|
|
|
|
|
|
|
|
|
+%Noteable here is, that with \textit{Bits} a unit for the information entropy is meant. Even though they store the same form of data, no indications could be found, that there is a direct connection to the binary digit (\,bit)\, that describes the physical unit to store iformation in computer science.
|
|
|
|
|
+
|
|
|
%todo explain 2.2 second bulletpoint of delfs_knebl. Maybe read gumbl book
|
|
%todo explain 2.2 second bulletpoint of delfs_knebl. Maybe read gumbl book
|
|
|
|
|
|
|
|
%This can be used to find the maximum amount of bit needed to store information.\\
|
|
%This can be used to find the maximum amount of bit needed to store information.\\
|
|
@@ -119,7 +119,7 @@ He defined entropy as shown in figure \eqref{eq:entropy}. Let X be a finite prob
|
|
|
|
|
|
|
|
\label{k4:arith}
|
|
\label{k4:arith}
|
|
|
\subsection{Arithmetic coding}
|
|
\subsection{Arithmetic coding}
|
|
|
-This coding method is an approach to solve the problem of wasting memeory due to the overhead which is created by encoding certain lenghts of alphabets in binary \cite{ris76, moffat_arith}. For example: Encoding a three-letter alphabet requires at least two bit per letter. Since there are four possilbe combinations with two bit, one combination is not used, so the full potential is not exhausted. Looking at it from another perspective and thinking a step further: Less storage would be required, if there would be a possibility to encode more than one letter in two bit.\\
|
|
|
|
|
|
|
+This coding method is an approach to solve the problem of wasting memory due to the overhead which is created by encoding certain lengths of alphabets in binary \cite{ris76, moffat_arith}. For example: Encoding a three-letter alphabet requires at least two bit per letter. Since there are four possilbe combinations with two bit, one combination is not used, so the full potential is not exhausted. Looking at it from another perspective and thinking a step further: Less storage would be required, if there was a possibility to encode more than one letter in two bit.\\
|
|
|
Dr. Jorma Rissanen described arithmetic coding in a publication in 1976 \cite{ris76}. % Besides information theory and math, he also published stuff about dna
|
|
Dr. Jorma Rissanen described arithmetic coding in a publication in 1976 \cite{ris76}. % Besides information theory and math, he also published stuff about dna
|
|
|
This works goal was to define an algorithm that requires no blocking. Meaning the input text could be encoded as one instead of splitting it and encoding the smaller texts or single symbols. He stated that the coding speed of arithmetic coding is comparable to that of conventional coding methods \cite{ris76}.
|
|
This works goal was to define an algorithm that requires no blocking. Meaning the input text could be encoded as one instead of splitting it and encoding the smaller texts or single symbols. He stated that the coding speed of arithmetic coding is comparable to that of conventional coding methods \cite{ris76}.
|
|
|
|
|
|
|
@@ -141,10 +141,10 @@ The coding algorithm works with probabilities for symbols in an alphabet. From a
|
|
|
\end{equation}
|
|
\end{equation}
|
|
|
}
|
|
}
|
|
|
% math and computers
|
|
% math and computers
|
|
|
-Bevore getting into the arithmetic coding algorithm, the following section will go over some details on how digital fractions are handled by computers. This knowledge will be helpfull in understanding how arithmetic coding works.\\
|
|
|
|
|
-In computers, arithmetic operations on floating point numbers are processed with integer representations of given floating point number \cite{ieee-float}. The number 0.4 + would be represented by $4\cdot 10^{-1}$.\\
|
|
|
|
|
-A interval would be represented by natural numbers between 0 and 100 and $... \cdot 10^-x$. \texttt{x} starts with the value 2 and grows as the intgers grow in length, meaning only if a uneven number is divided. For example: Dividing a uneven number like $5\cdot 10^{-1}$ by two, will result in $25\cdot 10^{-2}$. On the other hand, subdividing $4\cdot 10^y$ by two, with any negativ real number as y would not result in a greater \texttt{x} the length required to display the result will match the length required to display the input number \cite{witten87, moffat_arith}.\\
|
|
|
|
|
-Binary fractions are limited in from of representing decimal fractions. This is due to the fact that every other digit, adds zero or half of the value before. In other terms: $b \cdot 2^{-n}$ determines the value of $b \in {0,1}$ at position n behind the decimal point.\\
|
|
|
|
|
|
|
+Before getting into the arithmetic coding algorithm, the following section will go over some details on how digital fractions are handled by computers. This knowledge will be helpful in understanding how arithmetic coding works.\\
|
|
|
|
|
+In computers, arithmetic operations on floating point numbers are processed with integer representations \cite{ieee-float}. The number 0.4 for example would be represented by $4\cdot 10^{-1}$.\\
|
|
|
|
|
+An interval would be represented by natural numbers between 0 and 100 and $... \cdot 10^-x$. \texttt{x} starts with the value 2 and grows as the integers grow in length; meaning only if a uneven number is divided. For example: Dividing an uneven number like $5\cdot 10^{-1}$ by two, will result in $25\cdot 10^{-2}$. On the other hand, subdividing $4\cdot 10^y$ by two, with any negative real number as y would not result in a greater \texttt{x}. The length required to display the result will match the length required to display the input number \cite{witten87, moffat_arith}.\\
|
|
|
|
|
+Binary fractions are limited in form of representing decimal fractions. This is due to the fact that every other digit adds zero or half of the value before. In other terms: $b \cdot 2^{-n}$ determines the value of $b \in {0,1}$ at position n behind the decimal point.\\
|
|
|
|
|
|
|
|
|
|
|
|
|
%todo example including figure
|
|
%todo example including figure
|
|
@@ -156,8 +156,8 @@ Binary fractions are limited in from of representing decimal fractions. This is
|
|
|
\label{k4:arith-unscaled}
|
|
\label{k4:arith-unscaled}
|
|
|
\end{figure}
|
|
\end{figure}
|
|
|
|
|
|
|
|
-The encoding of the input text, or a sequence is possible by projecting it on a binary encoded fraction between 0 and 1. To get there, each character in the alphabet is represented by an interval between two fractions, in the space between 0.0 and 1.0. In \ref{k4:arith-unscaled} this space is illustraded by the line in the upper center, with a scaling form 0.0 on the left, to 1.0 on the right side. The interval for each symbol is determined by its distribution, in the input text (interval start) and the the start of the next character (interval end). The sum of all intervals will result in one \cite{moffat_arith}.\\
|
|
|
|
|
-In order, to remain in a presentable range, the example in \ref{k4:arith-unscaled} uses an alphabet of only three characters: \texttt{A, C and G}. For the sequence \texttt{AGAC} a probability distribution as shown in the upper left corner and listed in \ref{t:arith-prob} was calculated. The intervals resulting from this probabilities, are visualized by the three sections marked by outwards pointing arrows at the top. The interval for \texttt{A} extends from 0.0 until the start of \texttt{C} at 0.5, which extends to the start of \texttt{G} at 0.75 and so on.\\
|
|
|
|
|
|
|
+The encoding of the input text, or a sequence is possible by projecting it on a binary encoded fraction between 0 and 1. To get there, each character in the alphabet is represented by an interval between two fractions, in the space between 0.0 and 1.0. In \ref{k4:arith-unscaled} this space is illustrated by the line in the upper center, with a scaling from 0.0 on the left, to 1.0 on the right side. The interval for each symbol is determined by its distribution, in the input text (interval start) and the start of the next character (interval end). The sum of all intervals will result in one \cite{moffat_arith}.\\
|
|
|
|
|
+In order to remain in a presentable range, the example in \ref{k4:arith-unscaled} uses an alphabet of only three characters: \texttt{A, C and G}. For the sequence \texttt{AGAC} a probability distribution as shown in the upper left corner and listed in \ref{t:arith-prob} was calculated. The intervals resulting from these probabilities are visualized by the three sections marked by outwards pointing arrows at the top. The interval for \texttt{A} extends from 0.0 until the start of \texttt{C} at 0.5, which extends to the start of \texttt{G} at 0.75 and so on.\\
|
|
|
|
|
|
|
|
\label{t:arith-prob}
|
|
\label{t:arith-prob}
|
|
|
\sffamily
|
|
\sffamily
|
|
@@ -169,32 +169,32 @@ In order, to remain in a presentable range, the example in \ref{k4:arith-unscale
|
|
|
\toprule
|
|
\toprule
|
|
|
\textbf{Symbol} & \textbf{Probability} & \textbf{Interval}\\
|
|
\textbf{Symbol} & \textbf{Probability} & \textbf{Interval}\\
|
|
|
\midrule
|
|
\midrule
|
|
|
- A & $\frac{2}{4}=0.11$ & [0.0, 0.5[ \\ %${x\in \mathbb{Q} | 0.0 <= x < 0.5}$\\
|
|
|
|
|
- C & $\frac{1}{4}=0.71$ & [0.5, 0.75[ \\ %${x\in \mathbb{Q} | 0.5 <= x < 0.75}$\\
|
|
|
|
|
- G & $\frac{1}{4}=0.13$ & [0.75, 1.0[ \\ %${x\in \mathbb{Q} | 0.75 <= x < 1.0}$\\
|
|
|
|
|
|
|
+ A & $\frac{2}{4}=0.11$ & $[ 0.0,$ $ 0.5 ) $ \\ %${x\in \mathbb{Q} | 0.0 <= x < 0.5}$\\
|
|
|
|
|
+ C & $\frac{1}{4}=0.71$ & $[ 0.5,$ $ 0.75 ) $ \\ %${x\in \mathbb{Q} | 0.5 <= x < 0.75}$\\
|
|
|
|
|
+ G & $\frac{1}{4}=0.13$ & $[ 0.75,$ $ 1.0 ) $ \\ %${x\in \mathbb{Q} | 0.75 <= x < 1.0}$\\
|
|
|
\bottomrule
|
|
\bottomrule
|
|
|
\end{longtable}
|
|
\end{longtable}
|
|
|
\end{footnotesize}
|
|
\end{footnotesize}
|
|
|
\rmfamily
|
|
\rmfamily
|
|
|
|
|
|
|
|
-In the encoding process, the first symbol read from the sequence determines a interval, its symbol is associated with. Every following symbol determines a subinterval, which is formed by subdividing the previous interval into sections proportional to the probabilities from \ref{t:arith-prob}.
|
|
|
|
|
-Starting with \texttt{A}, the most left interval in \ref{k4:arith-unscaled} is subdivided into intervals visulaized below. Leaving a available space of $[0.0, 0.5)$. From there the interval, representing \texttt{G} is subdivided, and so on until the last symbol \texttt{C} is processed. This leaves a interval of $[0.40625, 0.421275)$. This is marked in \ref{k4:arith-unscaled} with a red line. Since the interval is comparably small, in the illustration it seems like a point in the interval is marked. This is not the case, the red line shows the position of the last mentioned interval.\\
|
|
|
|
|
|
|
+In the encoding process, the first symbol read from the sequence determines a interval that its symbol is associated with. Every following symbol determines a subinterval, which is formed by subdividing the previous interval into sections proportional to the probabilities from \ref{t:arith-prob}.
|
|
|
|
|
+Starting with \texttt{A}, the most left interval in \ref{k4:arith-unscaled} is subdivided into intervals visualized below. Leaving an available space of $[0.0, 0.5)$. From there, the interval representing \texttt{G} is subdivided, and so on until the last symbol \texttt{C} is processed. This leaves an interval of $[0.40625, 0.421275)$. This is marked in \ref{k4:arith-unscaled} with a red line. Since the interval is comparably small, in the illustration it seems like a point in the interval is marked. This is not the case, the red line shows the position of the last mentioned interval.\\
|
|
|
%To encode a text, subdividing is used, step by step on the text symbols from start to the end
|
|
%To encode a text, subdividing is used, step by step on the text symbols from start to the end
|
|
|
-To store the encoding result in as few bits as possible, only a single number,between upper and lower end of the last intervall will be stored. To encode in binary, the binary floating point representation of any number inside the interval, for the last character is calculated.\\
|
|
|
|
|
-For this example, the number \texttt{0.41484375} in decimal, or \texttt{0.0110101} in binary, would be calculated.\\
|
|
|
|
|
|
|
+To store the encoding result in as few bits as possible, only a single number between the upper and the lower end of the last interval will be stored. To encode in binary, the binary floating point representation of any number inside the interval for the last character is calculated.\\
|
|
|
|
|
+In this example, the number \texttt{0.41484375} in decimal, or \texttt{0.0110101} in binary, would be calculated.\\
|
|
|
%todo compression ratio
|
|
%todo compression ratio
|
|
|
To summarize the encoding process in short \cite{moffat_arith, witten87}:\\
|
|
To summarize the encoding process in short \cite{moffat_arith, witten87}:\\
|
|
|
|
|
|
|
|
\begin{itemize}
|
|
\begin{itemize}
|
|
|
\item The interval representing the first character is noted.
|
|
\item The interval representing the first character is noted.
|
|
|
\item Its interval is split into smaller intervals, with the ratios of the initial intervals between 0.0 and 1.0.
|
|
\item Its interval is split into smaller intervals, with the ratios of the initial intervals between 0.0 and 1.0.
|
|
|
- \item The interval representing the second character is choosen.
|
|
|
|
|
- \item This process is repeated, until a interval for the last character is determined.
|
|
|
|
|
- \item A binary floating point number is determined wich lays in between the interval that represents the represents the last symbol.\\
|
|
|
|
|
|
|
+ \item The interval representing the second character is chosen.
|
|
|
|
|
+ \item This process is repeated until an interval for the last character is determined.
|
|
|
|
|
+ \item A binary floating point number is determined wich lays in between the interval that represents the last symbol.\\
|
|
|
\end{itemize}
|
|
\end{itemize}
|
|
|
|
|
|
|
|
% its finite subdividing because of the limitation that comes with processor architecture
|
|
% its finite subdividing because of the limitation that comes with processor architecture
|
|
|
-For the decoding process to work, the \ac{EOF} symbol must be be present as the last symbol in the text. The compressed file will store the probabilies of each alphabet symbol as well as the floatingpoint number. The decoding process executes in a simmilar procedure as the encoding. The stored probabilies determine intervals. Those will get subdivided, by using the encoded floating point as guidance, until the \ac{EOF} symbol is found. By noting in which interval the floating point is found, for every new subdivision, and projecting the probabilies associated with the intervals onto the alphabet, the origin text can be read \cite{witten87, moffat_arith, ris76}.\\
|
|
|
|
|
|
|
+For the decoding process to work, the \ac{EOF} symbol must be present as the last symbol in the text. The compressed file will store the probabilities of each alphabet symbol as well as the floatingpoint number. The decoding process executes in a similar procedure as the encoding. The stored probabilities determine intervals. Those will get subdivided by using the encoded floating point as guidance until the \ac{EOF} symbol is found. By noting in which interval the floating point is found for every new subdivision, and projecting the probabilities associated with the intervals onto the alphabet, the origin text can be read \cite{witten87, moffat_arith, ris76}.\\
|
|
|
% rescaling
|
|
% rescaling
|
|
|
|
|
|
|
|
\begin{figure}[H]
|
|
\begin{figure}[H]
|
|
@@ -206,25 +206,26 @@ For the decoding process to work, the \ac{EOF} symbol must be be present as the
|
|
|
\end{figure}
|
|
\end{figure}
|
|
|
|
|
|
|
|
% finite percission
|
|
% finite percission
|
|
|
-The described coding is only feasible on machines with infinite percission \cite{witten87}. As soon as finite precission comes into play, the algorithm must be extendet, so that a certain length in the resulting number will not be exceeded. Since digital datatypes are limited in their capacity, like unsigned 64-bit integers which can store up to $2^64-1$ bit or any number between 0 and 18,446,744,073,709,551,615. That might seem like a great ammount at first, but considering a unfavorable alphabet, that extends the results lenght by one on each symbol that is read, only texts with the length of 63 can be encoded (62 if \acs{EOF} is exclued) \cite{moffat_arith}. For the compression with finite percission, rescaling is used. This method works by scaling up the intervals which results from subdividing. With that. The process for this is illustrated in \ref{k4:arith-scaled}. The red lines indicate the final interval.
|
|
|
|
|
|
|
+The described coding is only feasible on machines with infinite precision \cite{witten87}. As soon as finite precision comes into play, the algorithm must be extended, so that a certain length in the resulting number will not be exceeded. This is due to the fact that digital datatypes are limited in their capacity for example, the unsigned 64-bit integers which can store up to $2^64-1$ bit or any number between 0 and 18,446,744,073,709,551,615. That might seem like a great amount at first, but considering a unfavorable alphabet that extends the results lenght by one on each symbol that is read, only sequences with the length of 63 can be encoded (62 if \acs{EOF} is exclued) \cite{moffat_arith}. For the compression with finite percission, rescaling is used. This method works by scaling up the intervals which result from subdividing. The upscaling process is illustrated in \ref{k4:arith-scaled}. The vertical lines illustrate the interval of each step. The smaller, black lines between them indicate which previous section was scaled up. The red lines indicate the final interval and the letters at the bottom indicate which symbol gets encoded in this step.
|
|
|
|
|
|
|
|
\label{k4:huff}
|
|
\label{k4:huff}
|
|
|
-\subsection{Huffman encoding}
|
|
|
|
|
|
|
+\subsection{Huffman Encoding}
|
|
|
% list of algos and the tools that use them
|
|
% list of algos and the tools that use them
|
|
|
-D. A. Huffmans work focused on finding a method to encode messages with a minimum of redundance. He referenced a coding procedure developed by Shannon and Fano and named after its developers, which worked similar. The Shannon-Fano coding is not used today, due to the superiority in both efficiency and effectivity, in comparison to Huffman. % todo any source to last sentence.
|
|
|
|
|
|
|
+D. A. Huffman's work focused on finding a method to encode messages with a minimum of redundance. He referenced a coding procedure developed by Shannon and Fano, named after its developers, which worked similar. The Shannon-Fano coding is not used today due to the superiority of Huffman's algorithm in both efficiency and effectivity \cite{moffat_arith}.\\
|
|
|
Even though his work was released in 1952, the method he developed is in use today. Not only tools for genome compression but in compression tools with a more general ussage \cite{rfcgzip}.\\
|
|
Even though his work was released in 1952, the method he developed is in use today. Not only tools for genome compression but in compression tools with a more general ussage \cite{rfcgzip}.\\
|
|
|
-Compression with the Huffman algorithm also provides a solution to the problem, described at the beginning of \ref{k4:arith}, on waste through unused bit, for certain alphabet lengths. Huffman did not save more than one symbol in one bit, like it is done in arithmetic coding, but he decreased the number of bit used per symbol in a message. This is possible by setting individual bit lengths for symbols, used in the text that should get compressed \cite{huf52}.
|
|
|
|
|
-As with other codings, a set of symbols must be defined. For any text constructed with symbols from mentioned alphabet, a binary tree is constructed, which will determine how the symbols will be encoded. As in arithmetic coding, the probability of a letter is calculated for given text. The binary tree will be constructed after following guidelines \cite{alok17}:
|
|
|
|
|
|
|
+Compression with the Huffman algorithm also provides a solution to the problem, described at the beginning of \ref{k4:arith}; of waste through unused bit for certain alphabet lengths. Huffman did not save more than one symbol in one bit, like it is done in arithmetic coding, but he decreased the number of bit used per symbol in a message. This is possible by setting individual bit lengths for symbols used in the text that should get compressed \cite{huf52}.
|
|
|
|
|
+As with other codings, a set of symbols must be defined. For any text constructed with symbols from mentioned alphabet, a binary tree is constructed, which will determine how each individual symbols will be encoded. The binary tree will be constructed after following guidelines \cite{alok17}:
|
|
|
% greedy algo?
|
|
% greedy algo?
|
|
|
\begin{itemize}
|
|
\begin{itemize}
|
|
|
\item Every symbol of the alphabet is one leaf.
|
|
\item Every symbol of the alphabet is one leaf.
|
|
|
\item The right branch from every knot is marked as a 1, the left one is marked as a 0.
|
|
\item The right branch from every knot is marked as a 1, the left one is marked as a 0.
|
|
|
- \item Every symbol got a weight, the weight is defined by the frequency the symbol occurs in the input text. This might be a fraction between 0 and 1 or an integer. In this scenario it will described as the first.
|
|
|
|
|
- \item The less weight a leaf has, the higher the probability is, that this node is read next in the symbol sequence.
|
|
|
|
|
- \item The leaf with the lowest probability is most left and the one with the highest probability is most right in the tree.
|
|
|
|
|
|
|
+ \item Every symbol got a weigh. The weight is defined by the frequency the symbol occurs in the input text. This might be a fraction between 0 and 1 or an integer. In this scenario it will described as the first.
|
|
|
|
|
+ \item The less weight a leaf has, the higher is the probability, that this node is read next in the symbol sequence.
|
|
|
|
|
+ \item Pairs of the lowest weighting nodes are formed. This pair will from there on be represented by a node which weight is equal to the sum of the weight of its child nodes.
|
|
|
|
|
+ \item Higher weighting nodes are positioned left, lower ones right.
|
|
|
\end{itemize}
|
|
\end{itemize}
|
|
|
%todo tree building explanation
|
|
%todo tree building explanation
|
|
|
-A often mentioned difference between Shannon-Fano and Huffman coding, is that first is working top down while the latter is working bottom up. This means the tree starts with the lowest weights. The nodes that are not leafs have no value ascribed to them. They only need their weight, which is defined by the weights of their individual child nodes \cite{moffat20, alok17}.\\
|
|
|
|
|
|
|
+An often-mentioned difference between Shannon-Fano and Huffman coding is that the first is working top down while the latter is working bottom up. Meaning the first Shannon-Fano is starting with the highest probabilities while Huffman starts with the lowest \cite{moffat20, alok17}.\\
|
|
|
|
|
|
|
|
Given \texttt{K(W,L)} as a node structure, with the weigth or probability as \texttt{$W_{i}$} and codeword length as \texttt{$L_{i}$} for the node \texttt{$K_{i}$}. Then will \texttt{$L_{av}$} be the average length for \texttt{L} in a finite chain of symbols, with a distribution that is mapped onto \texttt{W} \cite{huf52}.
|
|
Given \texttt{K(W,L)} as a node structure, with the weigth or probability as \texttt{$W_{i}$} and codeword length as \texttt{$L_{i}$} for the node \texttt{$K_{i}$}. Then will \texttt{$L_{av}$} be the average length for \texttt{L} in a finite chain of symbols, with a distribution that is mapped onto \texttt{W} \cite{huf52}.
|
|
|
\begin{equation}\label{eq:huf}
|
|
\begin{equation}\label{eq:huf}
|
|
@@ -257,7 +258,7 @@ The average length for any symbol encoded in \acs{ASCII} is eight, while only us
|
|
|
\end{footnotesize}
|
|
\end{footnotesize}
|
|
|
\rmfamily
|
|
\rmfamily
|
|
|
|
|
|
|
|
-The exact input text is not relevant, since only the resulting probabilities are needed. To make this example more illustrative, possible occurences are listed in the most right column of \ref{t:huff-pre}. The probability for each symbol is calculated by dividing the message length by the times the symbol occured. This and the resulting probabilities on a scale between 0.0 and 1.0, for this example are shown in \ref{t:huff-pre} \cite{huf52}.\\
|
|
|
|
|
|
|
+The exact input text is not relevant, since only the resulting probabilities are needed. To make this example more illustrative, possible occurrences are listed in the most right column of \ref{t:huff-pre}. The probability for each symbol is calculated by dividing the message length by the times the symbol occured. This and the resulting probabilities on a scale between 0.0 and 1.0, for this example are shown in \ref{t:huff-pre} \cite{huf52}.\\
|
|
|
Creating a tree will be done bottom up. In the first step, for each symbol from the alphabet, a node without any connection is formed .\\
|
|
Creating a tree will be done bottom up. In the first step, for each symbol from the alphabet, a node without any connection is formed .\\
|
|
|
|
|
|
|
|
\texttt{<A>, <T>, <C>, <G>}\\
|
|
\texttt{<A>, <T>, <C>, <G>}\\
|
|
@@ -372,7 +373,7 @@ With this simple rules, the alphabet can be compressed too. Instead of storing c
|
|
|
BGZF extends this by creating a series of blocks. Each can not extend a limit of 64 Kilobyte. Each block contains a standard gzip file header, followed by compressed data.\\
|
|
BGZF extends this by creating a series of blocks. Each can not extend a limit of 64 Kilobyte. Each block contains a standard gzip file header, followed by compressed data.\\
|
|
|
|
|
|
|
|
\subsubsection{CRAM}
|
|
\subsubsection{CRAM}
|
|
|
-The improvement of \acs{BAM} \cite{cram-origin} called \acs{CRAM}, also features a block structure \cite{bam}. The whole file can be seperated into four sections, stored in ascending order: File definition, a CRAM Header Container, multiple Data Container and a final CRAM EOF Container.\\
|
|
|
|
|
|
|
+The improvement of \acs{BAM} \cite{cram-origin} called \acs{CRAM}, also features a block structure \cite{bam}. The whole file can be separated into four sections, stored in ascending order: File definition, a CRAM Header Container, multiple Data Container and a final CRAM EOF Container.\\
|
|
|
The complete structure is displayed in \ref{k4:cram-struct}. The following paragrph will give a brief description to the high level view of a \acs{CRAM} fiel, illustrated as the most upper bar. Followed by a closer look at the data container, which components are listed in the bar, at the center of \ref{k4:cram-struct}. The most in deph explanation will be given to the bottom bar, which shows the structure of so called slices.\\
|
|
The complete structure is displayed in \ref{k4:cram-struct}. The following paragrph will give a brief description to the high level view of a \acs{CRAM} fiel, illustrated as the most upper bar. Followed by a closer look at the data container, which components are listed in the bar, at the center of \ref{k4:cram-struct}. The most in deph explanation will be given to the bottom bar, which shows the structure of so called slices.\\
|
|
|
|
|
|
|
|
\begin{figure}[H]
|
|
\begin{figure}[H]
|
