Search moodle.org's
Developer Documentation

See Release Notes

  • Bug fixes for general core bugs in 3.11.x will end 14 Nov 2022 (12 months plus 6 months extension).
  • Bug fixes for security issues in 3.11.x will end 13 Nov 2023 (18 months plus 12 months extension).
  • PHP version: minimum PHP 7.3.0 Note: minimum PHP version has increased since Moodle 3.10. PHP 7.4.x is supported too.
/lib/ -> tokeniserlib.php (source)

[Source view]

Based on Drupal's search.module version 1.224 {@link http://cvs.drupal.org/viewcvs/drupal/drupal/modules/search/search.module?view=markup}

Copyright: (C) 2001-3001 Eloy Lafuente (stronk7) {@link http://contiento.com}
License: http://www.gnu.org/copyleft/gpl.html GNU GPL v3 or later
File Size: 406 lines (17 kb)
Included or required:0 times
Referenced: 0 times
Includes or requires: 0 files

Defines 5 functions

  tokenise_text()
  tokenise_split()
  tokenise_simplify()
  tokenise_expand_cjk()
  tokenise_truncate_word()

Functions
Functions that are not part of a class:

tokenise_text($text, $stop_words = array()   X-Ref
This function process the text passed at input, extracting all the tokens
and scoring each one based in their number of ocurrences and relation with
some well-known html tags

param: string  $text the text to be tokenised.
param: array   $stop_words array of utf-8 words than can be ignored in
param: boolean $overlap_cjk option to split CJK text into some overlapping
param: boolean $join_numbers option to join in one unique token sequences of numbers
return: array one sorted array of tokens, with tokens being the keys and scores in the values.

tokenise_split($text, $stop_words, $overlap_cjk, $join_numbers)   X-Ref
Splits a string into tokens


tokenise_simplify($text, $overlap_cjk, $join_numbers)   X-Ref
Simplifies a string according to indexing rules.


tokenise_expand_cjk($matches)   X-Ref
Basic CJK tokeniser. Simply splits a string into consecutive, overlapping
sequences of characters (MINIMUM_WORD_SIZE long).


tokenise_truncate_word(&$text)   X-Ref
Helper function for array_walk in search_index_split.
Truncates one string (token) to MAXIMUM_WORD_SIZE