Literature clients (indra.literature)

indra.literature.get_full_text(paper_id, idtype, preferred_content_type='text/xml')[source]

Return the content and the content type of an article.

This function retreives the content of an article by its PubMed ID, PubMed Central ID, or DOI. It prioritizes full text content when available and returns an abstract from PubMed as a fallback.

Parameters:
  • paper_id (string) – ID of the article.
  • idtype ('pmid', 'pmcid', or 'doi) – Type of the ID.
  • preferred_content_type (Optional[st]r) – Preference for full-text format, if available. Can be one of ‘text/xml’, ‘text/plain’, ‘application/pdf’. Default: ‘text/xml’
Returns:

  • content (str) – The content of the article.
  • content_type (str) – The content type of the article

indra.literature.id_lookup(paper_id, idtype)[source]

Take an ID of type PMID, PMCID, or DOI and lookup the other IDs.

If the DOI is not found in Pubmed, try to obtain the DOI by doing a reverse-lookup of the DOI in CrossRef using article metadata.

Parameters:
  • paper_id (str) – ID of the article.
  • idtype (str) – Type of the ID: ‘pmid’, ‘pmcid’, or ‘doi
Returns:

ids – A dictionary with the following keys: pmid, pmcid and doi.

Return type:

dict

Pubmed client (indra.literature.pubmed_client)

Search and get metadata for articles in Pubmed.

indra.literature.pubmed_client.expand_pagination(pages)[source]

Convert a page number to long form, e.g., from 456-7 to 456-457.

indra.literature.pubmed_client.get_abstract(pubmed_id, prepend_title=True)[source]

Get the abstract of an article in the Pubmed database.

indra.literature.pubmed_client.get_article_xml[source]

Get the XML metadata for a single article from the Pubmed database.

indra.literature.pubmed_client.get_id_count(search_term)[source]

Get the number of citations in Pubmed for a search query.

Parameters:search_term (str) – A term for which the PubMed search should be performed.
Returns:The number of citations for the query, or None if the query fails.
Return type:int or None
indra.literature.pubmed_client.get_ids[source]

Search Pubmed for paper IDs given a search term.

Search options can be passed as keyword arguments, some of which are custom keywords identified by this function, while others are passed on as parameters for the request to the PubMed web service For details on parameters that can be used in PubMed searches, see https://www.ncbi.nlm.nih.gov/books/NBK25499/#chapter4.ESearch Some useful parameters to pass are db=’pmc’ to search PMC instead of pubmed reldate=2 to search for papers within the last 2 days mindate=‘2016/03/01’, maxdate=‘2016/03/31’ to search for papers in March 2016.

PubMed, by default, limits returned PMIDs to a small number, and this number can be controlled by the “retmax” parameter. This function uses a retmax value of 100,000 by default that can be changed via the corresponding keyword argument.

Parameters:
  • search_term (str) – A term for which the PubMed search should be performed.
  • use_text_word (Optional[bool]) – If True, the “[tw]” string is appended to the search term to constrain the search to “text words”, that is words that appear as whole in relevant parts of the PubMed entry (excl. for instance the journal name or publication date) like the title and abstract. Using this option can eliminate spurious search results such as all articles published in June for a search for the “JUN” gene, or journal names that contain Acad for a search for the “ACAD” gene. See also: https://www.nlm.nih.gov/bsd/disted/pubmedtutorial/020_760.html Default : True
  • kwargs (kwargs) – Additional keyword arguments to pass to the PubMed search as parameters.
indra.literature.pubmed_client.get_ids_for_gene[source]

Get the curated set of articles for a gene in the Entrez database.

Search parameters for the Gene database query can be passed in as keyword arguments.

Parameters:hgnc_name (str) – The HGNC name of the gene. This is used to obtain the HGNC ID (using the hgnc_client module) and in turn used to obtain the Entrez ID associated with the gene. Entrez is then queried for that ID.
indra.literature.pubmed_client.get_ids_for_mesh(mesh_id, major_topic=False, **kwargs)[source]

Return PMIDs that are annotated with a given MeSH ID.

Parameters:
  • mesh_id (str) – The MeSH ID of a term to search for, e.g., D009101.
  • major_topic (bool) – If True, only papers for which the given MeSH ID is annotated as a major topic are returned. Otherwise all annotations are considered. Default: False
  • **kwargs – Any further PudMed search arguments that are passed to get_ids.
indra.literature.pubmed_client.get_issns_for_journal[source]

Get a list of the ISSN numbers for a journal given its NLM ID.

Information on NLM XML DTDs is available at https://www.nlm.nih.gov/databases/dtd/

indra.literature.pubmed_client.get_metadata_for_ids(pmid_list, get_issns_from_nlm=False, get_abstracts=False, prepend_title=False)[source]

Get article metadata for up to 200 PMIDs from the Pubmed database.

Parameters:
  • pmid_list (list of str) – Can contain 1-200 PMIDs.
  • get_issns_from_nlm (bool) – Look up the full list of ISSN number for the journal associated with the article, which helps to match articles to CrossRef search results. Defaults to False, since it slows down performance.
  • get_abstracts (bool) – Indicates whether to include the Pubmed abstract in the results.
  • prepend_title (bool) – If get_abstracts is True, specifies whether the article title should be prepended to the abstract text.
Returns:

Dictionary indexed by PMID. Each value is a dict containing the following fields: ‘doi’, ‘title’, ‘authors’, ‘journal_title’, ‘journal_abbrev’, ‘journal_nlm_id’, ‘issn_list’, ‘page’.

Return type:

dict of dicts

indra.literature.pubmed_client.get_metadata_from_xml_tree(tree, get_issns_from_nlm=False, get_abstracts=False, prepend_title=False, mesh_annotations=False)[source]

Get metadata for an XML tree containing PubmedArticle elements.

Documentation on the XML structure can be found at:
Parameters:
  • tree (xml.etree.ElementTree) – ElementTree containing one or more PubmedArticle elements.
  • get_issns_from_nlm (bool) – Look up the full list of ISSN number for the journal associated with the article, which helps to match articles to CrossRef search results. Defaults to False, since it slows down performance.
  • get_abstracts (bool) – Indicates whether to include the Pubmed abstract in the results.
  • prepend_title (bool) – If get_abstracts is True, specifies whether the article title should be prepended to the abstract text.
  • mesh_annotations (bool) – If True, extract mesh annotations from the pubmed entries and include in the returned data. If false, don’t.
Returns:

Dictionary indexed by PMID. Each value is a dict containing the following fields: ‘doi’, ‘title’, ‘authors’, ‘journal_title’, ‘journal_abbrev’, ‘journal_nlm_id’, ‘issn_list’, ‘page’.

Return type:

dict of dicts

indra.literature.pubmed_client.get_title(pubmed_id)[source]

Get the title of an article in the Pubmed database.

Pubmed Central client (indra.literature.pmc_client)

indra.literature.pmc_client.extract_paragraphs(xml_string)[source]

Returns list of paragraphs in an NLM XML.

This returns a list of the plaintexts for each paragraph and title in the input XML, excluding some paragraphs with text that should not be relevant to biomedical text processing.

Relevant text includes titles, abstracts, and the contents of many body paragraphs. Within figures, tables, and floating elements, only captions are retained (One exception is that all paragraphs within floating boxed-text elements are retained. These elements often contain short summaries enriched with useful information.) Due to captions, nested paragraphs can appear in an NLM XML document. Occasionally there are multiple levels of nesting. If nested paragraphs appear in the input document their texts are returned in a pre-ordered traversal. The text within child paragraphs is not included in the output associated to the parent. Each parent appears in the output before its children. All children of an element appear before the elements following sibling.

All tags are removed from each paragraph in the list that is returned. LaTeX surrounded by <tex-math> tags is removed entirely.

Note: Some articles contain subarticles which are processed slightly differently from the article body. Only text from the body element of a subarticle is included, and all unwanted elements are excluded along with their captions. Boxed-text elements are excluded as well.

Parameters:xml_string (str) – String containing valid NLM XML.
Returns:List of extracted paragraphs from the input NLM XML
Return type:list of str
indra.literature.pmc_client.extract_text(xml_string)[source]

Get plaintext from the body of the given NLM XML string.

This plaintext consists of all paragraphs returned by indra.literature.pmc_client.extract_paragraphs separated by newlines and then finally terminated by a newline. See the DocString of extract_paragraphs for more information.

Parameters:xml_string (str) – String containing valid NLM XML.
Returns:Extracted plaintext.
Return type:str
indra.literature.pmc_client.filter_pmids(pmid_list, source_type)[source]

Filter a list of PMIDs for ones with full text from PMC.

Parameters:
  • pmid_list (list of str) – List of PMIDs to filter.
  • source_type (string) – One of ‘fulltext’, ‘oa_xml’, ‘oa_txt’, or ‘auth_xml’.
Returns:

PMIDs available in the specified source/format type.

Return type:

list of str

indra.literature.pmc_client.get_xml(pmc_id)[source]

Returns XML for the article corresponding to a PMC ID.

indra.literature.pmc_client.id_lookup(paper_id, idtype=None)[source]

This function takes a Pubmed ID, Pubmed Central ID, or DOI and use the Pubmed ID mapping service and looks up all other IDs from one of these. The IDs are returned in a dictionary.

bioRxiv client (indra.literature.biorxiv_client)

A client to obtain metadata and text content from bioRxiv (and to some extent medRxiv) preprints.

indra.literature.biorxiv_client.get_collection_dois(collection_id, min_date=None)[source]

Get list of DOIs from a biorxiv/medrxiv collection.

Parameters:
  • collection_id (str) – The identifier of the collection to fetch.
  • min_date (Optional[datetime.datetime]) – A datetime object representing an cutoff. If given, only publications that were released on or after the given date are returned. By default, no date constraint is applied.
Returns:

The list of DOIs in the collection.

Return type:

list of dict

indra.literature.biorxiv_client.get_collection_pubs(collection_id, min_date=None)[source]

Get list of DOIs from a biorxiv/medrxiv collection.

Parameters:
  • collection_id (str) – The identifier of the collection to fetch.
  • min_date (Optional[datetime.datetime]) – A datetime object representing an cutoff. If given, only publications that were released on or after the given date are returned. By default, no date constraint is applied.
Returns:

A list of the publication entries which include the abstract and other metadata.

Return type:

list of dict

indra.literature.biorxiv_client.get_content_from_pub_json(pub, format)[source]

Get text content based on a given format from a publication JSON.

In the case of abstract, the content is returned from the JSON directly. For pdf, the content is returned as bytes that can be dumped into a file. For txt and xml, the text is processed out of either the raw XML or text content that rxiv provides.

Parameters:
  • pub (dict) – The JSON dict description a publication.
  • format (str) – The format, if available, via which the content should be obtained.
indra.literature.biorxiv_client.get_formats(pub)[source]

Return formats available for a publication JSON.

Parameters:pub (dict) – The JSON dict description a publication.
Returns:A dict with available formats as its keys (abstract, pdf, xml, txt) and either the content (in case of abstract) or the URL (in case of pdf, xml, txt) as the value.
Return type:dict
indra.literature.biorxiv_client.get_pdf_xml_url_base(content)[source]

Return base URL to PDF/XML based on the content of the landing page.

Parameters:content (str) – The content of the landing page for an rxiv paper.
Returns:The base URL if available, otherwise None.
Return type:str or None
indra.literature.biorxiv_client.get_text_from_rxiv_text(rxiv_text)[source]

Return clean text from the raw rxiv text content.

This function parses out the title, headings and subheadings, and the content of sections under headings/subheadings. It filters out some irrelevant content e.g., references and footnotes.

Parameters:rxiv_text (str) – The content of the rxiv full text as obtained from the web.
Returns:The text content stripped out from the raw full text.
Return type:str
indra.literature.biorxiv_client.get_text_from_rxiv_xml(rxiv_xml)[source]

Return clean text from the raw rxiv xml content.

Parameters:rxiv_xml (str) – The content of the rxiv full xml as obtained from the web.
Returns:The text content stripped out from the raw full xml.
Return type:str
indra.literature.biorxiv_client.get_text_url_base(content)[source]

Return base URL to full text based on the content of the landing page.

Parameters:content (str) – The content of the landing page for an rxiv paper.
Returns:The base URL if available, otherwise None.
Return type:str or None

CrossRef client (indra.literature.crossref_client)

indra.literature.crossref_client.doi_query(pmid, search_limit=10)[source]

Get the DOI for a PMID by matching CrossRef and Pubmed metadata.

Searches CrossRef using the article title and then accepts search hits only if they have a matching journal ISSN and page number with what is obtained from the Pubmed database.

Return a list of links to the full text of an article given its DOI. Each list entry is a dictionary with keys: - URL: the URL to the full text - content-type: e.g. text/xml or text/plain - content-version - intended-application: e.g. text-mining

indra.literature.crossref_client.get_metadata[source]

Returns the metadata of an article given its DOI from CrossRef as a JSON dict

Elsevier client (indra.literature.elsevier_client)

For information on the Elsevier API, see:
indra.literature.elsevier_client.check_entitlement(doi)[source]

Check whether IP and credentials enable access to content for a doi.

This function uses the entitlement endpoint of the Elsevier API to check whether an article is available to a given institution. Note that this feature of the API is itself not available for all institution keys.

indra.literature.elsevier_client.download_article(id_val, id_type='doi', on_retry=False)[source]

Low level function to get an XML article for a particular id.

Parameters:
  • id_val (str) – The value of the id.
  • id_type (str) – The type of id, such as pmid (a.k.a. pubmed_id), doi, or eid.
  • on_retry (bool) – This function has a recursive retry feature, and this is the only time this parameter should be used.
Returns:

content – If found, the content string is returned, otherwise, None is returned.

Return type:

str or None

indra.literature.elsevier_client.download_article_from_ids(**id_dict)[source]

Download an article in XML format from Elsevier matching the set of ids.

Parameters:<id_type> (str) – You can enter any combination of eid, doi, pmid, and/or pii. Ids will be checked in that order, until either content has been found or all ids have been checked.
Returns:content – If found, the content is returned as a string, otherwise None is returned.
Return type:str or None

Save raw text files based on a search for papers on ScienceDirect.

This performs a search to get PIIs, downloads the XML corresponding to the PII, extracts the raw text and then saves the text into a file in the designated folder.

Parameters:
  • query_str (str) – The query string to search with
  • folder (str) – The local path to an existing folder in which the text files will be dumped
  • do_extract_text (bool) – Choose whether to extract text from the xml, or simply save the raw xml files. Default is True, so text is extracted.
  • max_results (int or None) – Default is None. If specified, limit the number of results to the given maximum.
indra.literature.elsevier_client.extract_paragraphs(xml_string)[source]

Get paragraphs from the body of the given Elsevier xml.

indra.literature.elsevier_client.extract_text(xml_string)[source]

Get text from the body of the given Elsevier xml.

indra.literature.elsevier_client.get_abstract(doi)[source]

Get the abstract text of an article from Elsevier given a doi.

indra.literature.elsevier_client.get_article(doi, output_format='txt')[source]

Get the full body of an article from Elsevier.

Parameters:
  • doi (str) – The doi for the desired article.
  • output_format ('txt' or 'xml') – The desired format for the output. Selecting ‘txt’ (default) strips all xml tags and joins the pieces of text in the main text, while ‘xml’ simply takes the tag containing the body of the article and returns it as is . In the latter case, downstream code needs to be able to interpret Elsever’s XML format.
Returns:

content – Either text content or xml, as described above, for the given doi.

Return type:

str

indra.literature.elsevier_client.get_dois[source]

Search ScienceDirect through the API for articles and return DOIs.

Parameters:
  • query_str (str) – The query string to search with.
  • year (Optional[str]) – The year to constrain the search to.
  • loaded_after (Optional[str]) – Date formatted as ‘yyyy-MM-dd’T’HH:mm:ssX’ to constrain the search to articles loaded after this date. Example: 2019-06-01T00:00:00Z
Returns:

dois – The list of DOIs identifying the papers returned by the search.

Return type:

list[str]

indra.literature.elsevier_client.get_piis(query_str)[source]

Search ScienceDirect through the API for articles and return PIIs.

Note that ScienceDirect has a limitation in which a maximum of 6,000 PIIs can be retrieved for a given search and therefore this call is internally broken up into multiple queries by a range of years and the results are combined.

Parameters:query_str (str) – The query string to search with
Returns:piis – The list of PIIs identifying the papers returned by the search
Return type:list[str]
indra.literature.elsevier_client.get_piis_for_date[source]

Search ScienceDirect through the API for articles and return PIIs.

Parameters:
  • query_str (str) – The query string to search with.
  • year (Optional[str]) – The year to constrain the search to.
  • loaded_after (Optional[str]) – Date formatted as ‘yyyy-MM-dd’T’HH:mm:ssX’ to constrain the search to articles loaded after this date. Example: 2019-06-01T00:00:00Z
Returns:

piis – The list of PIIs identifying the papers returned by the search.

Return type:

list[str]

indra.literature.elsevier_client.search_science_direct[source]

Search ScienceDirect for a given field with a query string.

Users can specify which field they are interested in and only values from that field will be returned. It is also possible to restrict the search either to a specific year of publication or to papers published after a specific date.

Parameters:
  • query_str (str) – The query string to search with.
  • field_name (str) – A name of the field of interest to be returned. Accepted values are: authors, doi, loadDate, openAccess, pages, pii, publicationDate, sourceTitle, title, uri, volumeIssue.
  • year (Optional[str]) – The year to constrain the search to.
  • loaded_after (Optional[str]) – Date formatted as ‘yyyy-MM-dd’T’HH:mm:ssX’ to constrain the search to articles loaded after this date.
Returns:

all_parts – The list of values from the field of interest identifying the papers returned by the search.

Return type:

list[str]

NewsAPI client (indra.literature.newsapi_client)

This module provides a client for the NewsAPI web service (https://newsapi.org/). The web service requires an API key which is available after registering at https://newsapi.org/account. This key can be set as NEWSAPI_API_KEY in the INDRA config file or as an environmental variable with the same name.

NewsAPI also requires attribution e.g. “powered by NewsAPI.org” for derived uses.

indra.literature.newsapi_client.send_request(endpoint, **kwargs)[source]

Return the response to a query as JSON from the NewsAPI web service.

The basic API is limited to 100 results which is chosen unless explicitly given as an argument. Beyond that, paging is supported through the “page” argument, if needed.

Parameters:
  • endpoint (str) – Endpoint to query, e.g. “everything” or “top-headlines”
  • kwargs (dict) – A list of keyword arguments passed as parameters with the query. The basic ones are “q” which is the search query, “from” is a start date formatted as for instance 2018-06-10 and “to” is an end date with the same format.
Returns:

res_json – The response from the web service as a JSON dict.

Return type:

dict

Deft Tools (indra.literature.adeft_tools)

This file provides several functions helpful for acquiring texts for Adeft disambiguation.

It offers the ability to get text content for articles containing a particular gene. This is useful for aquiring training texts for genes genes that do not appear in a defining pattern with a problematic shortform.

General XML processing is also provided that allows for extracting text from a source that may be either of Elsevier XML, NLM XML or raw text. This is helpful because it avoids having to know in advance the source of text content from the database.

indra.literature.adeft_tools.filter_paragraphs(paragraphs, contains=None)[source]

Filter paragraphs to only those containing one of a list of strings

Parameters:
  • paragraphs (list of str) – List of plaintext paragraphs from an article
  • contains (str or list of str) – Exclude paragraphs not containing this string as a token, or at least one of the strings in contains if it is a list
Returns:

Plaintext consisting of all input paragraphs containing at least one of the supplied tokens.

Return type:

str

indra.literature.adeft_tools.get_text_content_for_gene(hgnc_name)[source]

Get articles that have been annotated to contain gene in entrez

Parameters:hgnc_name (str) – HGNC name for gene
Returns:text_content – xmls of fulltext if available otherwise abstracts for all articles that haven been annotated in entrez to contain the given gene
Return type:list of str
indra.literature.adeft_tools.get_text_content_for_pmids(pmids)[source]

Get text content for articles given a list of their pmids

Parameters:pmids (list of str) –
Returns:text_content
Return type:list of str
indra.literature.adeft_tools.universal_extract_paragraphs(xml)[source]

Extract paragraphs from xml that could be from different sources

First try to parse the xml as if it came from elsevier. if we do not have valid elsevier xml this will throw an exception. the text extraction function in the pmc client may not throw an exception when parsing elsevier xml, silently processing the xml incorrectly

Parameters:xml (str) – Either an NLM xml, Elsevier xml or plaintext
Returns:paragraphs – Extracted plaintext paragraphs from NLM or Elsevier XML
Return type:str
indra.literature.adeft_tools.universal_extract_text(xml, contains=None)[source]

Extract plaintext from xml that could be from different sources

Parameters:
  • xml (str) – Either an NLM xml, Elsevier xml, or plaintext
  • contains (str or list of str) – Exclude paragraphs not containing this string, or at least one of the strings in contains if it is a list
Returns:

The concatentation of all paragraphs in the input xml, excluding paragraphs not containing one of the tokens in the list contains. Paragraphs are separated by new lines.

Return type:

str