Scribd Logo
Upload

Welcome to Scribd!

  • Poincaré Embeddings For Learning Hierarchical Representations

    Uploaded by

    jimakosjp
    31 views29 pages
    The document proposes Poincaré embeddings, which use hyperbolic geometry to capture hierarchical structures in data. Poincaré embeddings parameterize items in the Poincaré disk and optimize their positions with Riemannian stochastic gradient descent. This allows hierarchical relationships to be reconstructed accurately, as demonstrated on taxonomy embedding and graph link prediction tasks, where Poincaré embeddings achieved better results than previous methods.

    Original Description:

    Original Title

    poincareembeddings-170704090828.pdf

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    The document proposes Poincaré embeddings, which use hyperbolic geometry to capture hierarchical structures in data. Poincaré embeddings parameterize items in the Poincaré disk and optimize their positions with Riemannian stochastic gradient descent. This allows hierarchical relationships to be reconstructed accurately, as demonstrated on taxonomy embedding and graph link prediction tasks, where Poincaré embeddings achieved better results than previous methods.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    31 views29 pages

    Poincaré Embeddings For Learning Hierarchical Representations

    Uploaded by

    jimakosjp
    The document proposes Poincaré embeddings, which use hyperbolic geometry to capture hierarchical structures in data. Poincaré embeddings parameterize items in the Poincaré disk and optimize their positions with Riemannian stochastic gradient descent. This allows hierarchical relationships to be reconstructed accurately, as demonstrated on taxonomy embedding and graph link prediction tasks, where Poincaré embeddings achieved better results than previous methods.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    of 29

    Poincaré Embeddings for

    Learning Hierarchical
    Representations
    July 4, 2017
    Tatsuya Shirakawa
    ABEJA Inc.
    CONFIDENTIAL

    Tatsuya Shirakawa
    CONFIDENTIAL

    Today’s Paper

    Paper Stats
    • Guys from FAIR
    • Sumitted to arXiv at May 26, 2017
    https://arxiv.org/abs/1705.08039
    • Sumitted to NIPS2017?

    Key Contributions
    • Introducing hyperbolic geometry
    (Poincaré disk model) into word/graph
    embeddings paradigm
    • Automatically capture hierarchical
    structure of data
    • Achieved incredible better results than
    previous works.
    CONFIDENTIAL

    Agenda

    1. Problems
    2. Hyperbolic Geometry
    3. Poincaré Embeddings
    (and Some Incredible Results)

    4
    CONFIDENTIAL

    Problems

    5
    CONFIDENTIAL

    The Problem

    Find good representation(embedding) of items such


    that underlying hierarchical relation structure are
    well reconstructed
    CONFIDENTIAL

    Taxonomy Embedding

    Embed nouns in WordNets such that related nouns


    are close in embedded space

    http://www.nltk.org/book_1ed/ch02.html
    7
    CONFIDENTIAL

    Graph Link Prediction

    Embed nodes in given graph such that missing


    links are well-reconstructed

    http://ml.cs.tsinghua.edu.cn/~jiaming/publications/

    8
    CONFIDENTIAL

    Back Theory

    9
    CONFIDENTIAL

    Hyperbolic Geometry

    • Geometry with negative curvature


    • Many models (realizations):
    - Poincaré half space model
    - Poincaré disk model
    …
    each is isometric

    10
    CONFIDENTIAL

    Poincaré Half Space Model

    • Defined on upper half space


    with metric

    • Distance btw points is

    11
    CONFIDENTIAL

    Tree representation in H

    • Tree structure is well


    represented in Poincaré
    half space

    12
    https://arxiv.org/abs/1006.5169
    CONFIDENTIAL

    Poincaré Disk Model

    • A realization of hyperbolic geometry


    • Defined on
    equipped with metric of

    • Distance btw points is

    M.C. Escher's Circle Limit III, 1959


    13
    CONFIDENTIAL

    Relation to Poincaré Half Space Model

    (for simplicity: 2-dim, identify as )

    https://arxiv.org/abs/1006.5169
    14
    CONFIDENTIAL

    Why not Euclidean Space?

    • Euclidean space is too narrow to embed


    hierarchical (tree) structures

    Surface Area Volume


    / # of leaf nodes / # of nodes

    Euclidean Ball O(R^n) O(R^n)

    b-ary tree O(b^R) O(b^R)

    ※ R=radius of ball or depth of tree

    15
    CONFIDENTIAL

    Why Hyperbolic Space?

    • b-array tree can be interpreted as discrete


    analogue of Poincaré disk

    16
    CONFIDENTIAL

    Conclusion Here

    • Hyperbolic space is far more appropriate than


    Euclidean space to represent hierarchical
    structure

    • Many equivalent models


    - Poincaré half space model
    - Poincaré disk model
    …

    17
    CONFIDENTIAL

    Prior Works around hyperbolic geometry


    applications
    • R. Kleinberg, “Geographic routing using hyperbolic
    spaces”, 2007
    • M. Boguna et al., “Sustaining the internet with
    hyperbolic mapping”, 2010
    • P. D. Hoff et al., “Latent space approaches to social
    network analysis”, 2016
    • A. B. Adcock et al., “Tree-like structure in large social
    and information networks’, 2013
    • D. Krioukov et al., “Hyperbolic geometry of complex
    networks”, 2010

    18
    CONFIDENTIAL

    Poincaré Embeddings

    19
    CONFIDENTIAL

    Proposed Method

    1. Parametrize each item in Poincaré ball

    2. Optimize them by Riemannian optimization


    under metric of
    CONFIDENTIAL

    Riemannian SGD

    1. Compute stochastic (Euclidean) gradient

    2. Correct metric

    3. Apply GD

    4. Project onto space

    21
    CONFIDENTIAL

    Taxonomy Embedding

    Embed nouns in WordNets such that related nouns


    are close in embedded space

    http://www.nltk.org/book_1ed/ch02.html
    22
    CONFIDENTIAL

    Objective Function

    Maximize

    randomly chosen 10 negative samples

    Reconstruction setting:
    - D is full relations
    Prediction setting
    - D is subset of full relations

    23
    CONFIDENTIAL

    Result

    24
    CONFIDENTIAL

    25
    CONFIDENTIAL

    Graph Link Prediction

    Embed nodes in given graph such that missing


    links are well-reconstructed

    http://ml.cs.tsinghua.edu.cn/~jiaming/publications/

    26
    CONFIDENTIAL

    Objective Function

    Minimize the cross entropy of probability

    27
    CONFIDENTIAL

    Result

    28
    CONFIDENTIAL

    Summary

    • Poincaré embeddings automatically capture


    hierarchical structure from data

    • Riemannian SGD provides the way to optimize


    Poincaré embeddings

    • Achieved quite good results on word/graph


    embedding tasks

    29

    You might also like