Comparação Difusa De Texto

pip install fuzzywuzzy

Utilização simples:

>>> from fuzzywuzzy import fuzz
>>> fuzz.ratio("this is a test", "this is a test!")
    96

O pacote foi construído em cima de difflib. Porque não usar isso? Além de ser um pouco mais simples, ele tem uma série de diferentes métodos de correspondência (como insensibilidade de ordem token, correspondência parcial de strings) que o tornam mais poderoso na prática. As funções process.extract são especialmente úteis: encontrar as melhores cadeias e rácios de correspondência a partir de um conjunto. Dos seus readme:

Razão Parcial

>>> fuzz.partial_ratio("this is a test", "this is a test!")
    100

Token Sort Ratio

>>> fuzz.ratio("fuzzy wuzzy was a bear", "wuzzy fuzzy was a bear")
    90
>>> fuzz.token_sort_ratio("fuzzy wuzzy was a bear", "wuzzy fuzzy was a bear")
    100

Token Set Ratio

>>> fuzz.token_sort_ratio("fuzzy was a bear", "fuzzy fuzzy was a bear")
    84
>>> fuzz.token_set_ratio("fuzzy was a bear", "fuzzy fuzzy was a bear")
    100

Processo

>>> choices = ["Atlanta Falcons", "New York Jets", "New York Giants", "Dallas Cowboys"]
>>> process.extract("new york jets", choices, limit=2)
    [('New York Jets', 100), ('New York Giants', 78)]
>>> process.extractOne("cowboys", choices)
    ("Dallas Cowboys", 90)

Existe um módulo na biblioteca-padrão (chamado difflib) isso pode comparar strings e retornar uma pontuação com base em sua similaridade. O SequenceMatcher as aulas devem fazer o que querem.

Editar: pequeno exemplo da linha de comandos de python:

>>> from difflib import SequenceMatcher as SM
>>> s1 = ' It was a dark and stormy night. I was all alone sitting on a red chair. I was not completely alone as I had three cats.'
>>> s2 = ' It was a murky and stormy night. I was all alone sitting on a crimson chair. I was not completely alone as I had three felines.'
>>> SM(None, s1, s2).ratio()
0.9112903225806451

HTH!

fuzzyset é muito mais rápido do que fuzzywuzzy (difflib) tanto para indexar como para procurar.

from fuzzyset import FuzzySet
corpus = """It was a murky and stormy night. I was all alone sitting on a crimson chair. I was not completely alone as I had three felines
    It was a murky and tempestuous night. I was all alone sitting on a crimson cathedra. I was not completely alone as I had three felines
    I was all alone sitting on a crimson cathedra. I was not completely alone as I had three felines. It was a murky and tempestuous night.
    It was a dark and stormy night. I was not alone. I was not sitting on a red chair. I had three cats."""
corpus = [line.lstrip() for line in corpus.split("\n")]
fs = FuzzySet(corpus)
query = "It was a dark and stormy night. I was all alone sitting on a red chair. I was not completely alone as I had three cats."
fs.get(query)
# [(0.873015873015873, 'It was a murky and stormy night. I was all alone sitting on a crimson chair. I was not completely alone as I had three felines')]

Atenção: tenha cuidado para não misturar unicode e bytes no seu fuziset.

A tarefa chama-seIdentificação da paráfrase que é uma área activa de investigação no processamento da Linguagem Natural. Liguei vários artigos de última geração, muitos dos quais podem encontrar código de código aberto no GitHub.

Note que toda a pergunta respondida assume que há alguma similaridade de cadeia/superfície entre as duas sentenças enquanto na realidade duas sentenças com pouca similaridade de cadeia podem ser semanticamente semelhantes.

import skipthoughts
model = skipthoughts.load_model()
vectors = skipthoughts.encode(model, X_sentences)

Isto converte as suas frases (X_sentences) em vetores. Mais tarde você pode encontrar a semelhança de dois vetores por:

similarity = 1 - scipy.spatial.distance.cosine(vectors[0], vectors[1])

Onde estamos assumindo vetor [0] e Vetor1 são o vector correspondente a X_sentences[ 0], X_sentences1 que querias encontrar os resultados deles.

Existem outros modelos para converter uma sentença para um vetor que você pode encontrar aqui .