API Reference¶

This is the complete API reference of the Genestack Client Library. For a more gentle introduction, you can read the Getting Started with the Genestack Python Client Library section.

Application Wrappers¶

Application¶

class genestack_client.Application(connection, application_id=None)¶

Bases: object

Create a new application instance for the given connection. The connection must be logged in to call the application’s methods. The application ID can be specified either as an argument to the class constructor or by overriding the APPLICATION_ID attribute in a child class.

get_response(method, params=None, trace=True)¶

Invoke one of the application’s public Java methods and return Response object. Allow to access to logs and traces in code, if you need only result use invoke()

Parameters:	method (str) – name of the public Java method params (tuple) – arguments that will be passed to the Java method. Arguments must be JSON-serializable. trace (bool) – request trace from server
Returns:	Response object
Return type:	Response

invoke(method, *params)¶

Invoke one of the application’s public Java methods.

Parameters:	method (str) – name of the public Java method params – arguments that will be passed to the Java method. Arguments must be JSON-serializable.
Returns:	JSON-deserialized response

upload_file(file_path, token)¶

Upload a file to the current Genestack instance. This action requires a special token that can be generated by the application.

Parameters:	file_path (str) – path to existing local file. token – upload token
Return type:	None

DataImporter¶

class genestack_client.DataImporter(connection)¶

Bases: object

A class used to import files to a Genestack instance. If no parent is specified, the files are created in the special folder Imported files

Required and recommended values can be set by arguments directly or passed inside a Metainfo:

create_bed(name="Bed", url="some/url")

# is equivalent to:
metainfo = Metainfo()
metainfo.add_string(Metainfo.NAME, "Bed")
metainfo.add_external_link(Metainfo.DATA_LINK, "some/url", text="link name")
create_bed(metainfo=metainfo)

However, do not pass the same value both through the arguments and inside a metainfo object.

Genestack accepts both compressed and uncompressed files. If the protocol is not specified, file:// will be used. Special characters should be escaped except s3://. Links to Amazon S3 storage should be formatted as in s3cmd.

Supported protocols:

file://:
- test.txt.gz
- file://test.txt
- file%20name.gz
ftp://
- ftp://server.com/file.txt
http:// https://
- http://server.com/file.txt
ascp://
- ascp://<user>@<server>:file.txt
s3://
- s3://bucket/file.gz
- s3://bucket/file name.gz

If you are uploading a local file, a Raw Upload intermediary file will be created on the platform.

AFFYMETRIX_ANNOTATION = 'affymetrixMicroarrayAnnotation'¶: Affymetrix microarray annotation type

AGILENT_ANNOTATION = 'agilentMicroarrayAnnotation'¶: Agilent microarray annotation type

INFINIUM_ANNOTATION = 'methylationArrayAnnotation'¶: Infinium microarray annotation type

MICROARRAY_ANNOTATION_TYPES = ('agilentMicroarrayAnnotation', 'affymetrixMicroarrayAnnotation', 'TSVMicroarrayAnnotation', 'methylationArrayAnnotation')¶: Supported microarray annotation types

TSV_ANNOTATION = 'TSVMicroarrayAnnotation'¶: TSV (GenePix etc) microarray annotation type

create_bam(parent=None, name=None, url=None, organism=None, strain=None, reference_genome=None, metainfo=None)¶

Create a Genestack Aligned Reads file from a local or remote BAM file. name, url and organism are required fields. They can be specified through the arguments or via a Metainfo instance.

Parameters:	parent (str) – accession of parent folder (if not provided, files will be created in the `Imported files` folder) name (str) – name of the file url – URL of a BAM file; the index will be created at initialization organism (str) – organism strain – strain reference_genome (str) – reference genome accession metainfo (Metainfo) – metainfo object
Returns:	file accession
Return type:	str

create_bed(parent=None, name=None, reference_genome=None, url=None, metainfo=None)¶

Create a Genestack BED Track from a local or remote BED file. name and url are mandatory fields. They can be specified through the arguments or via a Metainfo instance.

Parameters:	parent (str) – accession of parent folder (if not provided, files will be created in the `Imported files` folder) name (str) – name of the file reference_genome (str) – accession of reference genome url (str) – URL or local path to file metainfo (Metainfo) – metainfo object
Returns:	file accession
Return type:	str

create_dbnsfp(parent=None, url=None, name=None, organism=None, metainfo=None)¶

Create a Genestack Variation Database file. name and url are required fields. They can be specified through the arguments or via a Metainfo instance.

Parameters:	parent (str) – accession of parent folder (if not provided, files will be created in the `Imported files` folder) url (str) – URL or local path name (str) – name of the file organism (str) – organism metainfo (Metainfo) – metainfo object
Returns:	file accession
Return type:	str

create_dictionary(parent=None, name=None, url=None, term_type=None, metainfo=None, parent_dictionary=None)¶

Create a Dictionary file from a local or remote file. owl, obo, and csv formats are supported. name and url are required fields. They can be specified through the arguments or via a Metainfo instance.

Parameters:	parent (str) – accession of parent folder (if not provided, files will be created in the `Imported files` folder) name (str) – name of the file url (str) – URL of a file term_type (str) – dictionary term type metainfo (Metainfo) – metainfo object parent_dictionary (str) – accession of parent dictionary
Returns:	file accession
Return type:	str

create_expression_levels(parent=None, unit=None, name=None, url=None, metainfo=None)¶

Create a Expression Levels file from a local or remote expression levels file. name, url and unit are required fields. They can be specified through the arguments or via a Metainfo instance.

Parameters:	parent (str) – accession of parent folder (if not provided, files will be created in the `Imported files` folder) name (str) – name of the file url (str) – URL of the file unit (str) – unit of expression, e.g. `TPM`, `FPKM` metainfo (Metainfo) – metainfo object
Returns:	file accession
Return type:	str

create_gene_expression_signature(parent=None, name=None, url=None, organism=None, metainfo=None)¶

Create a Gene Expression Signature file from a local or remote gene expression signature file. name, url and organism are required fields. They can be specified through the arguments or via a Metainfo instance.

Parameters:	parent (str) – accession of parent folder (if not provided, files will be created in the `Imported files` folder) name (str) – name of the file url – URL of a file organism (str) – organism name metainfo (Metainfo) – metainfo object
Returns:	file accession
Return type:	str

create_gene_list(parent=None, name=None, url=None, organism=None, metainfo=None)¶

Create a Gene List file from a local or remote gene list file. name, url and organism are required fields. They can be specified through the arguments or via a Metainfo instance.

Parameters:	parent (str) – accession of parent folder (if not provided, files will be created in the `Imported files` folder) name (str) – name of the file url – URL of a file organism (str) – organism name metainfo (Metainfo) – metainfo object
Returns:	file accession
Return type:	str

create_genome_annotation(parent=None, url=None, name=None, organism=None, reference_genome=None, strain=None, metainfo=None)¶

Create a Genestack Genome Annotation file from a local or remote file. name and url are required fields. They can be specified through the arguments or via a Metainfo instance.

Parameters:	parent (str) – accession of parent folder (if not provided, files will be created in the `Imported files` folder) url (str) – URL or local path name (str) – name of the file organism (str) – organism reference_genome (str) – reference genome accession strain (str) – strain metainfo (Metainfo) – metainfo object
Returns:	file accession
Return type:	str

create_infinium_microarray_data(parent, name=None, urls=None, method=None, metainfo=None)¶

Create a Genestack Infinium Microarrays Data inside a folder. We can’t use create_microarray_data method because ‘microarrayData’ importer can have only one source file, while infinium assay has two. So we invoke ‘infinium MicroarrayData’ importer with two links for BioMetaKeys.DATA_LINK key in metainfo.

Infinum microarrays available only for humans so we have no ‘organism’ key in arguments.

Parameters:	parent (str) – accession of parent folder name (str) – name of the file urls (list) – list of urls method (str) – method metainfo (Metainfo) – metainfo object
Returns:	file accession
Return type:	str

create_mapped_reads_count(parent=None, name=None, url=None, reference_genome=None, metainfo=None)¶

Create a Mapped Reads Count file from a local or remote mapped reads count file. name and url are required fields. They can be specified through the arguments or via a Metainfo instance.

Parameters:	parent (str) – accession of parent folder (if not provided, files will be created in the `Imported files` folder) name (str) – name of the file url – URL of a file reference_genome (str) – reference genome accession metainfo (Metainfo) – metainfo object
Returns:	file accession
Return type:	str

create_microarray_annotation(annotation_type, parent=None, name=None, url=None, metainfo=None)¶

Create a Dictionary file from a local or remote microarray annotation file. name and url are required fields. They can be specified through the arguments or via a Metainfo instance.

Parameters:	annotation_type (str) – type of annotation being loaded, an element of `MICROARRAY_ANNOTATION_TYPES` parent (str) – accession of parent folder (if not provided, files will be created in the `Imported files` folder) name (str) – name of the file url – URL of a file metainfo (Metainfo) – metainfo object
Returns:	file accession
Return type:	str

create_microarray_data(parent, name=None, urls=None, method=None, organism=None, metainfo=None)¶

Create a Genestack Microarray Data inside an folder. name and urls are required fields. They can be specified through the arguments or via a Metainfo instance.

Parameters:	parent (str) – accession of parent folder name (str) – name of the file urls (list) – list of urls method (str) – method organism (str) – organism metainfo (Metainfo) – metainfo object
Returns:	file accession
Return type:	str

create_reference_genome(parent=None, name=None, description='', sequence_urls=None, annotation_url=None, organism=None, assembly=None, release=None, strain=None, metainfo=None)¶

Create a Genestack Reference Genome from a collection of local or remote FASTA sequence files, and a GTF or GFF annotation file. name, sequence_urls, organism and annotation_url are required fields. They can be specified through the arguments or via a Metainfo instance.

Parameters:

parent (str) – accession of parent folder (if not provided, files will be created in the Imported files folder)
name (str) – name of the file
description (str) – experiment description
sequence_urls (list) – list urls or local path to sequencing files.
annotation_url (str) – url to annotation file
organism (str) – organism
assembly (str) – assembly
release (str) – release
strain (str) – strain
metainfo (Metainfo) – metainfo object

Returns:

create_report_file(parent=None, name=None, urls=None, metainfo=None)¶

Create a Genestack Report File from a local or remote data file. name and urls are required fields. They can be specified through the arguments or via a Metainfo instance.

Parameters:	parent (str) – accession of parent folder (if not provided, files will be created in the `Imported files` folder) name (str) – name of the file urls (list or str) – URL or list of URLs of local file paths metainfo (Metainfo) – metainfo object
Returns:	file accession
Return type:	str

create_unaligned_read(parent=None, name=None, urls=None, method=None, organism=None, metainfo=None)¶

Create a Genestack Unaligned Reads file from one or several local or remote files. Most common file formats encoding sequencing reads with quality scores are accepted (FASTQ 33/64, SRA, FASTA+QUAL, SFF, FAST5). name and urls are required fields. They can be specified through the arguments or via a Metainfo instance.

Parameters:	parent (str) – accession of parent folder (if not provided, files will be created in the `Imported files` folder) name (str) – name of the file urls (list) – list of urls method (str) – method organism (str) – organism metainfo (Metainfo) – metainfo object
Returns:	file accession
Return type:	str

create_vcf(parent=None, name=None, reference_genome=None, url=None, metainfo=None)¶

Create a Genestack Variants file from a local or remote VCF file. name and url are required fields. They can be specified through the arguments or via a Metainfo instance.

Parameters:	parent (str) – accession of parent folder (if not provided, files will be created in the `Imported files` folder) name (str) – name of the file reference_genome (str) – accession of reference genome url (str) – URL or local path to file metainfo (Metainfo) – metainfo object
Returns:	file accession
Return type:	str

create_wig(parent=None, name=None, reference_genome=None, url=None, metainfo=None)¶

Create a Genestack Wiggle Track from a local or remote WIG file. name and url are required fields. They can be specified through the arguments or via a Metainfo instance.

Parameters:	parent (str) – accession of parent folder (if not provided, files will be created in the `Imported files` folder) name (str) – name of the file reference_genome (str) – accession of reference genome url (str) – URL or local path to file metainfo (Metainfo) – metainfo object
Returns:	file accession
Return type:	str

load_raw(file_path)¶

Create a Genestack Raw Upload file from a local file, and return the accession of the created file.

Parameters:	file_path (str) – existing file path
Returns:	accession
Return type:	str

FilesUtil¶

class genestack_client.FilesUtil(connection, application_id=None)¶

Bases: genestack_client.Application

An application to perform file management operations on Genestack.

add_checksums(app_file, expected_checksums)¶

Add expected MD5 checksum to the metainfo of a CLA file. Expected checksums are calculated in the following way:

The number of checksums equals number of entries in storage. For instance, a Reference Genome file has 2 entries (annotation and sequence files).

If there are multiple files in one entry, they will be concatenated in the same order as they were PUT to storage by the initialization script.

If a file is marked for testing, then after initialization its metainfo will contain both expected and actual checksum values.

Parameters:	app_file – accession of application file expected_checksums – collection of MD5 checksums
Returns:	None

add_metainfo_string_value(accession_list, key, value)¶

Add a string value to the metainfo of specified files.

Parameters:	accession_list (list[str]) – list of files to be updated key (str) – metainfo key value (str) – string
Return type:	None

add_metainfo_values(accession, metainfo, skip_existing_keys=True, replace_existing_keys=False)¶

Add metainfo to a specified file. By default, metainfo keys that are already present in the file will be skipped.

Parameters:

accession – accession of the file to update
metainfo (Metainfo) – metainfo object containing the metainfo to add
skip_existing_keys (bool) – ignore metainfo keys that are already present in the file’s metainfo (default: True)
replace_existing_keys (bool) – replace the existing metainfo value for the metainfo keys that are already present in the file’s metainfo (default: False)

Return type:

None

collect_initializable_files_in_container(accession)¶

Recursively search for all initialisable file in container.

Parameters:	accession (str) – accession of container
Returns:	list of accessions
Return type:	list

collect_metainfos(accessions)¶

Get complete metainfo of a list of files.

Parameters:	accessions (list[str]) – list of accessions
Returns:	list of metainfo objects
Return type:	list[Metainfo]

count_file_children(container_accession)¶: Count children of a container (not recursive). :param container_accession: accession of container :type container_accession: str :return: number of children :rtype int:

create_folder(name, parent=None, description=None, metainfo=None)¶

Create a folder.

Parameters:	name (str) – name of the folder parent (str) – if not specified, create folder in the user’s private folder description (str) – description of the folder (goes into the metainfo) metainfo (Metainfo) – additional `Metainfo`. Description and accession should be specified either via arguments or in a metainfo object (but not in both).
Returns:	accession of created folder

find_file_by_name(name, parent=None, file_class='com.genestack.api.files.IFile')¶

Finds file with specified name (ignore case!) and type. If no file is found None is returned. If more than one file is found the first one is returned. If the parent container is not found, the corresponding exceptions are thrown.

Parameters:	name (str) – file name parent (str) – parent accession, private folder is default file_class (str) – File class to be returned, default IFile
Returns:	file accession
Return type:	str

find_files(file_filter, sort_order='DEFAULT', ascending=False, offset=0, limit=2000)¶

Search for files with file_filter and return dictionary with two key/value pairs:

'total': total number (int) of files matching the query

'result': list of file info dictionaries for subset of matching files

(from offset to offset+limit). See the documentation of get_infos() for the structure of these objects.

Parameters:	file_filter (FileFilter) – file filter sort_order (str) – sorting order for the results, see `SortOrder` ascending (bool) – should the results be in ascending order? (default: False) offset (int) – search offset (default: 0, cannot be negative) limit (int) – maximum number of results to return (max and default: 100)
Returns:	a dictionary with search response
Return type:	dict[str, int\|list[dict[str, str\|dict]]]

find_or_create_folder(name, parent=None)¶

Return the folder accession if it already exists, and create it otherwise. If more than one folder is found the first one is returned.

Parameters:	name (str) – display name parent (str) – parent accession, use home folder if None
Returns:	accession of folder
Return type:	str

find_reference_genome(organism, assembly, release)¶

Returns the accession of the reference genome with the specified parameters: organism, assembly, release. If more than one or no genome is found, the corresponding exceptions are thrown.

Parameters:	organism (str) – organism assembly (str) – assembly release (str) – release
Returns:	accession
Return type:	str
Raises:	`GenestackServerException` if more than one genome, or no genome is found

get_file_children(container_accession)¶

Return accessions of files linked to current container.

Parameters:	container_accession (str) – accession of container
Returns:	list of accessions
Return type:	list

get_folder(parent, *names, **kwargs)¶

Find a subfolder (by name) in a folder passed as an accession, returning accession of that subfolder. If several names are provided, treat them as a path components for the sub-sub-…-folder down the folder hierarchy, returning accession of that deepmost folder:

fu.get_folder('GS777', 'RNASeq') looks for subfolder with name “RNASeq” in folder with accession “GS777”, and returns accession of that “RNASeq” subfolder;
fu.get_folder('GS777', 'Experiments', 'RNASeq') looks for subfolder with name “Experiments” in a folder with accession “GS777”, then looks for “RNASeq” in “Experiments”, and returns the accession of “RNASeq”.

If create=True is passed as a kwarg, all the folders in names hierarchy will be created (otherwise GenestackException is raised).

Parameters:	parent (str) – accession of folder to search in names – tuple of “path components”, a hierarchy of folders to find create* (bool) – whether to create folders from `names` if they don’t exist or not; default is `False` (raise `GenestackException` if any folder doesn’t exist)
Returns:	accession of found (or created) subfolder
Return type:	str
Raises:	GenestackException – if no name is passed, or folder with required name is not found (and shouldn’t be created)

get_home_folder()¶

Return the accession of the current user’s home folder.

Returns:	accession of home folder
Return type:	str

get_infos(accession_list)¶

Returns a list of dictionaries with information about each of the specified files. This will return an error if any of the accessions is not valid. The order of the returned list is the same as the one of the accessions list.

The information dictionaries have the following structure:

accession

owner

name

isDataset

application

id

initializationStatus

isError

id

permissionsByGroup (the value for each key is a dictionary with group accessions as keys)

groupNames

ids

time

fileCreation

initializationQueued

initializationStart

initializationEnd

fileCreation

lastMetainfoModification

Parameters:	accession_list (list) – list of valid accessions.
Returns:	list of file info dictionaries.
Return type:	list[dict[str, object]]

get_metainfo_values_as_string_list(accessions_list, keys_list=None)¶

Retrieve metainfo values as lists of strings for specific files and metainfo keys. The function returns a dictionary.

Parameters:	accessions_list – accessions of the files to retrieve keys_list – metainfo keys to retrieve (if `None`, all non-technical keys are retrieved for each file)
Type:	accessions: list[str]
Type:	keys: list[str]\|None
Returns:	a two-level dictionary with the following structure: accession -> key -> value list
Return type:	dict[str, dict[str, list[str]]]

get_metainfo_values_as_strings(accessions_list, keys_list=None)¶

Retrieve metainfo values as strings for specific files and metainfo keys. Metainfo value lists are concatenated to string using ‘, ‘ as delimiter. The function returns a dictionary.

Parameters:	accessions_list – accessions of the files to retrieve keys_list – metainfo keys to retrieve (if `None`, all non-technical keys are retrieved for each file)
Type:	accessions: list[str]
Type:	keys: list[str]\|None
Returns:	a two-level dictionary with the following structure: accession -> key -> value
Return type:	dict[str, dict[str, str]]

get_public_folder()¶

Return the accession of the Public folder on the current Genestack instance.

Returns:	accession of `Public` folder
Return type:	str

get_special_folder(name)¶

Return the accession of a special folder.

Available special folders are described in SpecialFolders

Parameters:	name (str) – special folder name
Returns:	accession
Return type:	str
Raises:	GenestackException: if folder name is unknown

link_file(accession, parent)¶

Link a file to a folder.

Parameters:	accession (str) – file accession parent (str) – parent folder accession
Return type:	None

link_files(children_to_parents_dict)¶

Link files to containers.

Parameters:	children_to_parents_dict – dictionary where keys are accessions of the files to link, and values are lists of accessions of the containers to link into
Type:	dict
Return type:	None

mark_for_tests(app_file)¶

Mark Genestack file as test one by adding corresponding key to metainfo. Test file will calculate md5 checksums of its encapsulated physical files during initialization.

Parameters:	app_file – accession of file
Returns:	None

mark_obsolete(accession)¶

Mark Genestack file as obsolete one by adding corresponding key to metainfo.

Parameters:	accession – accession of file
Returns:	None

remove_metainfo_value(accession_list, key)¶

Delete a key from the metainfo of specified files.

Parameters:	accession_list (list[str]) – list of files to be updated key (str) – metainfo key
Return type:	None

rename_file(accession, name)¶

Rename a file.

Parameters:	accession (str) – file accession name (str) – name
Return type:	None

replace_metainfo_string_value(accession_list, key, value)¶

Replace a string value in the metainfo of specified files.

Parameters:	accession_list (list[str]) – list of files to be updated key (str) – metainfo key value (str) – string
Return type:	None

replace_metainfo_value(accession_list, key, value)¶

Replace a value in the metainfo of specified files.

Parameters:	accession_list (list[str]) – list of files to be updated key (str) – metainfo key value (MetainfoScalarValue) – metainfo value
Return type:	None

unlink_file(accession, parent)¶

Unlink a file from a folder.

Parameters:	accession (str) – file accession parent (str) – folder accession
Return type:	None

unlink_files(children_to_parents_dict)¶

Unlink files from containers.

Parameters:	children_to_parents_dict (dict[str, list[str]]) – dictionary where keys are accessions of the files to unlink, and values are lists of accessions of the containers to unlink from
Return type:	None

GroupsUtil¶

class genestack_client.GroupsUtil(connection, application_id=None)¶

Bases: genestack_client.Application

find_group_by_name(name)¶

Finds group with specified name. If there are no groups or more than one group with this name, an exception is thrown.

Parameters:	name (str) – group name
Returns:	group accession
Return type:	str

ShareUtil¶

class genestack_client.ShareUtil(connection, application_id=None)¶

Bases: genestack_client.Application

Application that acts as a facade for sharing-related operations.

class Permissions¶

Bases: object

Supported permission values that can be used in ShareUtil.share_files() and ShareUtil.share_folder() methods.

VIEW¶: Allows finding files via search and reading files’ content

EDIT¶: Allows finding files via search, reading files’ content and modifying files’ metainfo

SHARE¶: Allows finding files via search, reading files’ content and sharing them with other groups. This permissions type only allows sharing by group members from the same organization as the file owner. When sharing, non-owners are only allowed to set permissions that are the same or narrower that they currently have for the given file.

get_available_sharing_groups()¶

Find groups that the current user can share files with, which means that he is either a sharing user or an administrator of these groups.

Returns:	dictionary in format ‘group accession’ -> ‘group name’
Return type:	dict

safe_share_files(file_accessions, group_accession, permissions, destination_folder=None)¶

Same as share_files() but does not throw an exception in case some of the given files cannot be shared (i.e. the current user doesn’t own them or doesn’t have the ShareUtil.Permissions.SHARE permission).

Parameters:

file_accessions (str | collections.Iterable[str]) – accession or an iterable of accessions of files to be shared
group_accession (str) – accession of the group to share the files with
permissions (str | collections.Iterable[str]) – permissions that should be assigned to the provided files. Must consist of ShareUtil.Permissions values
destination_folder (str) – accession of the folder to link shared files into. Typically this parameter should be used for linking files into group folders, which is currently impossible to do using the FilesUtil.link_file() method. No links will be created if this parameter is equal to None

share_files(file_accessions, group_accession, permissions, destination_folder=None)¶

Share files with the given permissions with the given groups. Available permission values are listed in the ShareUtil.Permissions class.

Parameters:

file_accessions (str | collections.Iterable[str]) – accession or an iterable of accessions of files to be shared
group_accession (str) – accession of the group to share the files with
permissions (str | collections.Iterable[str]) – permissions that should be assigned to the provided files. Must consist of ShareUtil.Permissions values
destination_folder (str) – accession of the folder to link shared files into. Typically this parameter should be used for linking files into group folders, which is currently impossible to do using the FilesUtil.link_file() method. No links will be created if this parameter is equal to None

Raises:

GenestackServerException – if some of the given files cannot be shared by the current user (i.e. he doesn’t own them or doesn’t have the ShareUtil.Permissions.SHARE permission).

share_files_for_edit(file_accessions, group_accession, destination_folder=None)¶

Share files with editing permissions. Editing permissions include viewing permissions and also allow modifying metainfo and linking/unlinking files (only applicable to containers and datasets).

This method is equivalent to calling safe_share_files() method with ShareUtil.Permissions.EDIT permission.

Parameters:

file_accessions (str | collections.Iterable[str]) – accession or an iterable of accessions of files to be shared
group_accession (str) – accession of the group to share the files with
destination_folder (str) – accession of the folder to link shared files into. Typically this parameter should be used for linking files into group folders, which is currently impossible to do using the FilesUtil.link_file() method. No links will be created if this parameter is equal to None.

share_files_for_view(file_accessions, group_accession, destination_folder=None)¶

Share files with viewing permissions. Viewing permissions include finding the shared files and running tasks that access their content.

This method is equivalent to calling safe_share_files() method with ShareUtil.Permissions.VIEW permission.

Parameters:

file_accessions (str | collections.Iterable[str]) – accession or an iterable of accessions of files to be shared
group_accession (str) – accession of the group to share the files with
destination_folder (str) – accession of the folder to link shared files into. Typically this parameter should be used for linking files into group folders, which is currently impossible to do using the FilesUtil.link_file() method. No links will be created if this parameter is equal to None.

share_folder(folder_accession, group_accession, permissions, destination_folder=None)¶

Recursively share the given folder, its subfolders and files inside them. Files that cannot be shared by the current user will be skipped.

This method is useful for sharing folders with a lot of files because calling share_files() may result in a timeout. This method shares files in chunks and may take significant time to complete.

Parameters:

folder_accession (str) – accession of the folder
group_accession (str) – accession of the group to share the files with
permissions (str | collections.Iterable[str]) – permissions that should be assigned to the provided files. Must consist of ShareUtil.Permissions values
destination_folder (str) – accession of the folder to link shared files into. Typically this parameter should be used for linking files into group folders, which is currently impossible to do using the FilesUtil.link_file() method. No links will be created if this parameter is equal to None

FileInitializer¶

class genestack_client.FileInitializer(connection, application_id=None)¶

Bases: genestack_client.Application

Wrapper class around the File Initializer application.

initialize(accessions)¶

Start initialization for the specified accessions. Missed accession and initialization failures are ignored silently.

Parameters:	accessions (list[str]) – list of accessions
Return type:	None

load_info(accessions)¶

Takes as input a list of file accessions and returns a list of dictionaries (one for each accession) with the following structure:

accession: (str) file accession

name: (str) file name if the file exists

status: (str) initialization status

The possible values for status are:

NoSuchFile

NotApplicable

NotStarted

InProgress

Complete

Failed

Parameters:	accessions (list[str]) – list of accessions
Returns:	list of dictionaries
Return type:	list

TaskLogViewer¶

class genestack_client.TaskLogViewer(connection, application_id=None)¶

Bases: genestack_client.Application

A wrapper class for the Task Logs Viewer application. This application allows you to access the initialization logs of a file.

print_log(accession, log_type=None, follow=True, offset=0)¶

Print a file’s latest task initialization logs to stdout. Raises an exception if the file is not found or has no associated initialization task. By default the output stdout log is shown. You can also view the stderr error log. follow=True will wait until initialization is finished. Incoming logs will be printed to the console.

Parameters:	accession – file accession log_type – stdout or stderr follow – if enabled, wait and display new lines as they appear (similar to `tail --follow`) offset – offset from which to start retrieving the logs. Set to -1 if you want to start retrieving logs from the latest chunk.

Expression Navigator¶

class genestack_client.expression_navigator.ExpressionNavigatorforGenes(connection, application_id=None)¶

APPLICATION_ID = 'genestack/expressionNavigator'¶

PKG_DESEQ = 'DESeq2'¶

PKG_EDGER = 'edgeR'¶

create_file(groups, r_package='DESeq2', organism=None)¶

Create an expression navigator file from RNA-seq gene counts files. Each group is described by a dictionary with the following keys:

accessions: list of accessions of the raw gene counts files for this group

name (optional): group name

description (optional): group description

Parameters:	groups – list of dictionaries describing the groups for differential expression. See above for the dictionary structure. r_package – name of R package to use for differential expression (either `edgeR` or `DESeq2`) organism – organism
Returns:	accession of the created Expression Navigator file

class genestack_client.expression_navigator.ExpressionNavigatorforIsoforms(connection, application_id=None)¶

APPLICATION_ID = 'genestack/expressionNavigator-isoforms'¶

create_file(groups, fragment_bias_corr=True, multi_mapping_corr=True, organism=None)¶

Create an expression navigator file from RNA-seq isoform FPKM counts files. Each group is described by a dictionary with the following keys:

accessions: list of accessions of the isoform counts files for this group

name (optional): group name

description (optional): group description

Parameters:	groups – list of dictionaries describing the groups for differential expression. See above for the dictionary structure. fragment_bias_corr (bool) – apply correction for fragment bias multi_mapping_corr (bool) – apply correction for reads with multiple mappings organism – organism
Returns:	accession of the created Expression Navigator file

class genestack_client.expression_navigator.ExpressionNavigatorforMicroarrays(connection, application_id=None)¶

APPLICATION_ID = 'genestack/expressionNavigator-microarrays'¶

create_file(groups, normalized_microarray_file, microarray_annotation, organism=None)¶

Create an Expression Navigator file from a normalized microarray file. Each group is described by a dictionary with the following keys:

accessions: list of accessions of the source microarray files for this group

name (optional): group name

description (optional): group description

is_control (optional): boolean value indicating whether the group is a control group

Parameters:	groups – list of dictionaries describing the groups for differential expression. See above for the dictionary structure. normalized_microarray_file – accession of normalized microarray file microarray_annotation – accession of the microarray annotation file organism – organism
Returns:	accession of the created Expression Navigator file

DatasetsUtil¶

class genestack_client.DatasetsUtil(connection, application_id=None)¶

APPLICATION_ID = 'genestack/datasetsUtil'¶

BATCH_SIZE = 100¶

add_dataset_children(accession, children)¶

Add new files to a dataset.

Parameters:	accession (str) – dataset accession children (list[str]) – list of children accessions to add to the dataset

add_file_to_datasets(file_accession, dataset_accessions)¶

Add given file to several datasets.

Parameters:	file_accession (str) – file accession dataset_accessions (list[str]) – accessions of the datasets

create_dataset(name, dataset_type, children, parent=None, dataset_metainfo=None)¶

Create a dataset.

Parameters:	name (str) – name of the dataset dataset_type (str) – type of the dataset (children files interface name, must extend IDataFile) children (list[str]) – list of children accessions parent (str) – folder for the new dataset, ‘My datasets’ if not specified dataset_metainfo (Metainfo) – metainfo of the created dataset
Returns:	dataset accession
Return type:	str

create_empty_dataset(name, dataset_type, parent=None, dataset_metainfo=None)¶

Create an empty dataset.

Parameters:	name (str) – name of the dataset dataset_type (str) – type of the dataset (children files interface name, must extend IDataFile) parent (str) – folder for the new dataset, ‘My datasets’ if not specified dataset_metainfo (Metainfo) – metainfo of the created dataset
Returns:	dataset accession
Return type:	str

create_subset(accession, children, parent=None)¶

Create a subset from dataset’s children.

Parameters:	accession (str) – dataset accession children (list[str]) – list of children accessions to create a subset parent (str) – folder for the new dataset, ‘My datasets’ if not specified
Returns:	accession of the created subset
Return type:	str

get_dataset_children(accession)¶

Return generator over children accessions of the provided dataset.

Parameters:	accession (str) – dataset accession
Returns:	generator over dataset’s children accessions

get_dataset_size(accession)¶

Get number of files in dataset.

Parameters:	accession (str) – dataset accession
Returns:	number of files in dataset
Return type:	int

merge_datasets(datasets, parent=None)¶

Create a new dataset from the given datasets.

Parameters:	datasets (list[str]) – list of source datasets accessions parent (str) – folder for the new dataset, ‘My datasets’ if not specified
Returns:	accession of the created dataset
Return type:	str

remove_dataset_children(accession, children)¶

Remove children from dataset.

Parameters:	accession (str) – dataset accession children (list[str]) – list of children accessions to remove from the dataset

SampleLinker (Beta)¶

class genestack_client.samples.SampleLinker(connection, application_id=None)¶

Application for linking data files to samples.

It operates with the following concepts:

A study is a dataset (collection) of samples.
A sample is a file that contains common metainfo that can be attached to files with data.
When linking data files and samples, data files must be uploaded and put into an upload dataset. This dataset simplifies operations on these files in Genestack and provides data versioning. Upload dataset is linked to the study.
When uploading files to the upload dataset, they are put inside this dataset and initialized. Each file’s metainfo will contain a link to the according sample.

A typical workflow might look like this:

A study with samples is created via the Study Design application inside Genestack.
Study number is generated and exported via the Study Design API.
An upload dataset is created and linked to the provided study.
Files with data are uploaded, linked to samples and initialized via the ‘import_data’ method.
If some data files are considered corrupted or invalid, they can be removed using the ‘unlink_data’ method.
When all required data files are uploaded, data can be made visible to others by releasing the upload dataset using the ‘release’ method.

NOTE: This API is currently in Beta stage and is a subject to change, so no backwards compatibility is guaranteed at this point.

APPLICATION_ID = 'genestack/sample-linker'¶

create_upload_dataset(study_number, file_type, **kwargs)¶

Create a dataset that will later be used to hold uploaded data files.

This method accepts additional parameters required for creating files inside Genestack. These parameters depend on the file type:

“ExpressionLevels”: no additional parameters.

Example:

sample_linker.create_upload_dataset(
    study_number=1,
    file_type='ExpressionLevels'
)

Supported file types:

“ExpressionLevels”: expression data
“MappedReadCounts”: deprecated, use “ExpressionLevels” instead

Parameters:	study_number (int) – number of the study that contains samples for uploaded files. file_type (str) – type of files that will be uploaded kwargs – additional options that are needed when creating a file. Options content depends on the type of the created file.
Returns:	accession of the created dataset
Return type:	str

import_data(samples, upload_dataset_accession)¶

Create data files inside the upload dataset and link them to the specified samples.

Created files are initialized upon creation.

NOTE: This method can only handle 100 files at a time, so in case of uploading more files than that they must be uploaded in batches of this size.

Example:

sample_linker.import_data(
    samples={
        'sampleId1': ['http://data_url1', 'http://data_url2'],
        'sampleId2': ['http://more.data']
    },
    upload_dataset_accession='GSF000123'
)

This call will return the following dictionary:

{
    'sampleId1': ['GSF0001', 'GSF0002'],
    'sampleId2': ['GSF0003']
}

Parameters:	samples (dict[str, list[str]]) – mapping from sample id to a list of URLs that point to data. upload_dataset_accession (str) – accession of the upload dataset that will hold the created data files.
Returns:	mapping from sample id to a list of accessions of the created data files.
Return type:	dict[str, list[str]]

release(group_name, upload_dataset_accession)¶

Release the provided dataset. Releasing a dataset means that all data files are ready and can be shared with the outer world.

This method is idempotent and can be run multiple times in case of errors.

Parameters:	group_name (str) – name of the group that the provided dataset will be shared with. upload_dataset_accession (str) – accession of the dataset that holds the uploaded data files.

unlink_data(file_accessions, upload_dataset_accession)¶

Remove uploaded data files from the given dataset and unlink them from their samples. Links to samples are always removed but actual files may not be removed from the system.

Removing a file that isn’t present in the dataset is a no-op and will not throw an exception.

Parameters:	file_accessions (list[str]) – accessions of data files that should be unlinked. upload_dataset_accession (str) – accession of the upload dataset that holds the provided data files.

Command-Line Applications¶

CLApplication¶

class genestack_client.CLApplication(connection, application_id=None)¶

Bases: genestack_client.Application

Base class to interact with Genestack command-line applications. The APPLICATION_ID is mandatory. You can either pass it as an argument to the class constructor or override it in a child class. Source files and parameters are application-specific.

change_command_line_arguments(accession, params)¶

Change the command-line arguments strings in a file’s metainfo. params is a list of command-line strings. Note that the syntax of command-line argument strings is application-specific. The only way for you to know which command-line strings to provide it is to look at the Parameters metainfo field of a CLA file that has the correct parameters specified through the graphical user interface of the application.

If the file is not found, does not have the right file type or is already initialized, an exception will be thrown.

Parameters:	accession (str) – file accession or accession list params (list) – list of commandlines to be set
Returns:	None

create_file(source_files, name=None, params=None, calculate_checksums=False, expected_checksums=None, initialize=False)¶

Create a native Genestack file with the application and return its accession. If a source file is not found or is not of the expected type, an exception will be thrown.

Parameters:	source_files (list) – list of source files accessions name (str) – if a name is provided, the created file will be renamed params – custom command-line arguments strings; if None, the application defaults will be used. params – list calculate_checksums (bool) – a flag used in the initialization script to compute checksums for the created files expected_checksums (dict) – Dict of expected checksums (`{metainfo_key: expected_checksum}`) initialize – should initialization be started immediately after the file is created?
Returns:	accession of created file
Return type:	str

replace_file_reference(accession, key, accession_to_remove, accession_to_add)¶

Replace a file reference on the file.

If the file is not found or is not of the right file type, the corresponding exceptions are thrown. If accession_to_remove or accession_to_add is not found, an exception will be thrown.

Parameters:	accession – file accession or accession list key – key for source files accession_to_remove – accession to remove accession_to_add – accession to add
Returns:	None

start(accession)¶

Start file initialization. If the file is not found or is not of the right file type, an exception will be thrown.

Parameters:	accession (str) – file accession or accession list
Returns:	None

AffymetrixMicroarraysNormalizationApplication¶

class genestack_client.AffymetrixMicroarraysNormalizationApplication(connection, application_id=None)¶

API Reference¶

Application Wrappers¶

Application¶

DataImporter¶

FilesUtil¶

GroupsUtil¶

ShareUtil¶

FileInitializer¶

TaskLogViewer¶

Expression Navigator¶

DatasetsUtil¶

SampleLinker (Beta)¶

Command-Line Applications¶

CLApplication¶

AffymetrixMicroarraysNormalizationApplication¶

AgilentMicroarraysNormalizationApplication¶

AlignedReadsQC¶

AlignedReadsSubsamplingApplication¶

ArrayQualityMetricsApplication¶

BWAApplication¶

BowtieApplication¶

BsmapApplication¶

BsmapApplicationWG¶

ConcatenateVariantsApplication¶

CuffquantApplication¶

DoseResponseApplication¶

EffectPredictionApplication¶

FastQCApplicaton¶

FilterByQuality¶

FilterDuplicatedReads¶

GOEnrichmentAnalysis¶

GenePixMicroarraysNormalizationApplication¶

HTSeqCountsApplication¶

InfiniumMicroarraysNormalizationApplication¶

IntersectApplication¶

IntersectGenomicFeaturesMapped¶

IntersectGenomicFeaturesVariants¶

L1000MicroarraysNormalizationApplication¶

MarkDuplicated¶

MergeMappedReadsApplication¶

MethratioApplication¶

NormalizationApplication¶

QiimeMicrobiomeAnalysis¶

RemoveDuplicated¶

SingleCellRNASeqAnalysisApplication¶

SingleCellRNASeqVisualiserApplication¶

SubsampleReads¶

TargetedSequencingQC¶

TestCLApplication¶

TophatApplication¶

TrimAdaptorsAndContaminants¶

TrimLowQualityBases¶

TrimToFixedLength¶

UnalignedReadsQC¶

VariantsAssociationAnalysisApplication¶

VariationCaller2Application¶

VariationCallerApplication¶

VariationMergerApplication¶

Genestack Objects¶

Metainfo¶

Metainfo scalar values¶

File filters¶

Genome Queries¶

File Types¶

File Permissions¶

Users and Connections¶

Connection¶

settings.User¶

Helper methods¶

get_connection¶

make_connection_parser¶

get_user¶

Exceptions¶

GenestackBaseException¶

GenestackException¶

GenestackServerException¶

GenestackAuthenticationException¶

GenestackResponseError¶

GenestackConnectionFailure¶

Others¶