%SUMMARY
%- ABSTRACT
%- INTRODUCTION
%# BASICS
%- \acs{DNA} STRUCTURE
%- DATA TYPES
% - BAM/\ac{FASTQ}
% - NON STANDARD
%- COMPRESSION APPROACHES
% - SAVING DIFFERENCES WITH GIVEN BASE \acs{DNA}
% - HUFFMAN ENCODING
% - PROBABILITY APPROACHES (WITH BASE?)
%
%# COMPARING TOOLS
%- 
%# POSSIBLE IMPROVEMENT
%- \acs{DNA}S STOCHASTICAL ATTRIBUTES 
%- IMPACT ON COMPRESSION

% todo: use this https://www.reddit.com/r/bioinformatics/comments/7wfdra/eli5_what_are_the_differences_between_fastq_and/

% bigger picture - structure chapters like this:
% what is it/how does it work
% where are limits (e.g. BAM)
% what is our focus (and maybe 'why')

\section{File Formats used to Store DNA}
\label{chap:filetypes}
As described in previous chapters \ac{DNA} can be represented by a string with the buildingblocks A,T,G and C. Using a common fileformat for saving text would be impractical because the ammount of characters or symbols in the used alphabet, defines how many bits are used to store each single symbol.\\
The \ac{ascii} table is a characterset, registered in 1975 and to this day still in use to encode texts digitally. For the purpose of communication bigger charactersets replaced \ac{ascii}. It is still used in situations where storage is short.
% todo grund dass ascii abgelöst wurde -> zu wenig darstellungsmöglichkeiten. Pro heute -> weniger overhead pro character
Storing a single \textit{A} with \ac{ascii} encoding, requires 8 bit (\,excluding magic bytes and the bytes used to mark \ac{EOF})\ . Since there are at least $2^8$ or 128 displayable symbols. The buildingblocks of \ac{DNA} require a minimum of four letters, so two bits are needed
% cout out examples. Might be needed later or elsewhere
% \texttt{00 -> A, 01 -> T, 10 -> G, 11 -> C}. 
In most tools, more than four symbols are used. This is due to the complexity in sequencing \ac{DNA}. It is not 100\% preceice, so additional symbols are used to mark nucelotides that could not or could only partly get determined. Further a so called quality score is used to indicate the certainty, for each single nucleotide, that is was sequenced correctly as what was stored.\\
More common everyday-usage text encodings like unicode require 16 bits per letter. So settling with \ac{ascii} has improvement capabilitie but is, on the other side, more efficient than using bulkier alternatives like unicode.\\

Several people and groups have developed different fileformats to store genomes. Unfortunally for this work, there is no defined standard filetype or set of filetypes, therefore one has to gather information by themselve. In order to not go beyond scope, this work will focus only on fileformats that fullfill following criteria:\\
\begin{itemize}
  \item{the format has reputation, either through a scientific paper, that prooved its superiority to other relevant tools or through a broad ussage of the format.}
  \item{the format should no specialize on only one type of \ac{DNA}.}
  \item{the format mainly stores nucleotide seuqences and does not neccesarily include \ac{IUPAC} codes besides A, C, G and T \autocite{iupac}.}
  \item{the format is open source. Otherwise improvements can not be tested, without buying the software and/or requesting permission to disassemble and reverse engineer the software or parts of it.}
  \item{the compression methode used in the format is based on probabilities.}
\end{itemize}

Information on available formats where gathered through various Internet platforms \autocite{ensembl, ucsc, ga4gh}. 
Some common fileformats found:
\begin{itemize}
% which is relevant? 
  \item{BED}  % \autocite{bed}
  \item{CRAM} % \autocite{cram}
  \item{FASTA} % \autocite{}
  \item{\ac{FASTQ}} % \autocite{}
  \item{GFF} % \autocite{}
  \item{SAM/\ac{BAM}} % \autocite{}
  \item{twoBit}% \autocite{}
  \item{VCF}% \autocite{}

\end{itemize}
% src: http://help.oncokdm.com/en/articles/1195700-what-is-a-bam-fastq-vcf-and-bed-file
Since methods to store this kind of Data are still in development, there are many more filetypes. The few, mentioned above are used by different organisations and 
%todo calc percentage 
are backed by scientific papers.\\
Considering the first criteria, by searching through anonymously accesable \ac{ftp} servers, only two formats are used commonly: FASTA or its extension \ac{FASTQ} and the \ac{BAM} Format. %todo <- add ftp servers to cite

\subsection{\ac{FASTQ}}
% todo add some fasta knowledge
Is a text base format for storing sequenced data. It saves nucleotides as letters and in extend to that, the quality values are saved.
\ac{FASTQ} files are split into multiples of four, each four lines contain the informations for one sequence. The exact structure of \ac{FASTQ} format is as follows:
\texttt{
Line 1: Sequence identifier aka. Title, starting with an @ and an optional description.\\
Line 2: The seuqence consisting of nucleoids, symbolized by A, T, G and C.\\
Line 3: A '+' that functions as a seperator. Optionally followed by content of Line 1.\\
Line 4: quality line(s). consisting of letters and special characters in the \ac{ascii} scope.}\\

The quality values have no fixed type, to name a few there is the sanger format, the solexa format introduced by Solexa Inc., the Illumina and the QUAL format which is generated by the PHRED software. 
The quality value shows the estimated probability of error in the sequencing process.
[...]

\subsection{Sequence Alignment Map}
% src https://github.com/samtools/samtools
\ac{SAM} often seen in its compressed, binary representation \ac{BAM} with the fileextension \texttt{.bam}, is part of the SAMtools package, a uitlity tool for processing SAM/BAM and CRAM files. The SAM/BAM file is a text based format delimited by TABs. It uses 7-bit US-ASCII, to be precise Charset ANSI X3.4-1968 as defined in RFC1345. The structure is more complex than the one in \ac{FASTQ} and described best, accompanied by an example:

\begin{figure}[ht]
  \centering
  \includegraphics[width=15cm]{k3/sam-structure.png}
  \caption{SAM/BAM file structure example}
  \label{k_datatypes:bam-struct}
\end{figure}


